Kubernetes Monitoring with Prometheus, Grafana & Loki

Why do kubectl logs and kubectl top fall short once a cluster runs more than a handful of services? This module explains the difference between monitoring (what is happening?) and observability (why is it happening?) and introduces the stack you will build over the next four days: Prometheus for metrics, Grafana for visualization and alerting, Loki with Grafana Alloy for logs. By the end, you will have a clear picture of the architecture and understand which component handles which task.

You install the kube-prometheus-stack via Helm chart, which deploys Prometheus, Alertmanager, Grafana, Node Exporter, and kube-state-metrics in a single step. At its core is the included Prometheus Operator, whose Custom Resource Definitions (ServiceMonitor, PodMonitor, PrometheusRule) you then explore in detail. By the end, you configure automatic service discovery that detects new scrape targets as soon as a service is deployed.

The Grafana Operator manages dashboards and datasources as Kubernetes resources, ideal for GitOps workflows with ArgoCD or Flux. Installation via Helm chart is faster to set up but requires manual dashboard maintenance. You install both variants, import a dashboard via GrafanaDashboard CR and via ConfigMap, and then decide which approach fits your team.

Now it gets practical: you verify the scrape configuration of the Prometheus Operator, import a first dashboard into Grafana, and analyze CPU, RAM, and network metrics from your cluster. Along the way, you learn the four metric types (counter, gauge, histogram, summary) through real examples and understand when each type is the right choice.

How does Loki collect logs without indexing them, and why does that make the stack so storage-efficient? You install Loki in your Kubernetes cluster, configure Grafana Alloy as the log collector, and see how log streams are structured, collected, and displayed in Grafana. The focus is on the architecture (Distributor, Ingester, Querier) and the interplay with Prometheus metrics.

Promtail reached its end-of-life on March 2, 2026. Grafana Alloy takes over as the unified telemetry collector for logs, metrics, and traces. You migrate an existing Promtail configuration to Alloy, use the built-in convert command, and configure initial pipeline stages for log transformation. Along the way, you see how Alloy's modular dataflow model and OpenTelemetry compatibility make it significantly more flexible than its predecessor.

How do you monitor an application that doesn't export its own metrics? You configure individual scrape jobs via ServiceMonitor resources, instrument a sample application with the Prometheus client, and create custom metrics for application-specific KPIs. Best practices for naming conventions, label cardinality, and efficient metric design round out the session.

You write PromQL queries for real monitoring scenarios: vector selectors with label matchers, range queries for trend analysis, and aggregations across multiple services. You also learn about the breaking changes in Prometheus 3.0: the .* regex operator now matches newlines, and metric names support UTF-8 characters. Both can silently break existing queries and alerting rules. By the end, you optimize slow queries using recording rules and understand when subqueries make sense.

A dashboard nobody uses is a waste of time. That is why you build a Kubernetes namespace dashboard from scratch here: with template variables for namespace and service filters, so one dashboard works for all teams instead of maintaining a separate one per service. You work with transformations for calculated fields (e.g., error rate derived from two metrics), set annotations for deployment markers, and learn when a stat panel is a better choice than a time series panel. Since Grafana 12: dashboard-as-code enables dashboard versioning directly via Git (Git Sync, initially experimental).

An alert triggered at 3 AM must be immediately understandable. You configure the Prometheus Alertmanager with routing rules, receiver integrations (Slack, email, PagerDuty), and alert templates. In parallel, you set up Grafana Alerts and compare when each system is the better choice. Your first concrete alerts: CPU overload, pod restarts, and disk pressure.

How does Loki scale from a small cluster to enterprise level? You analyze the architecture in detail: Distributor for distribution, Ingester for buffering, Querier for queries, and Storage for persistence. The session shows how to choose between single-binary and microservices mode, which storage backends (S3, GCS, filesystem) fit which scenarios, and how the new horizontal compactor in Loki 3.6 speeds up large delete requests.

You write LogQL queries for typical troubleshooting scenarios: log stream selectors with label filters, parsers (json, logfmt, pattern) for structured extraction, and metric queries (e.g., rate over error logs) to generate metrics directly from logs. You then combine these queries with Prometheus metrics in a shared Grafana dashboard.

Metrics show that a service is slow. Logs show why. Here you combine both: a correlated dashboard in Grafana that displays Prometheus metrics and Loki logs side by side. You configure data links between metrics and logs so that clicking a metric outlier directly shows the corresponding log entries. The result is a stack that gives you the complete context of an incident.

How do you define a Service Level Objective that is neither too lenient nor too strict? You measure SLIs (latency, error rate, availability) with Prometheus queries and formulate SLOs with error budgets. For implementation, you use Sloth or the Grafana SLO feature to automatically generate recording rules and alerting rules. The module wraps up with a discussion of real-world examples: which SLOs make sense for an API gateway, a batch job, and a database?

Resource demand in Prometheus depends less on cluster size than on the number of active time series and their label combinations. You identify a high-cardinality label in prepared metrics, adjust a relabeling rule, and compare the resulting series count. From that observation, you determine when one server is sufficient and when federation, sharding, Thanos, or Mimir should be evaluated.

Your PromQL query takes 30 seconds, Grafana shows timeouts, Loki eats memory: typical symptoms of an untuned stack. For Prometheus, you adjust memory limits, scrape intervals, and sharding. For Grafana, you enable query caching and optimize panel queries. For Loki, you manage log retention, compression strategies, and chunk encoding. By the end, you know the key tuning levers for each of the three stack components.

Your cluster reports high latency, Prometheus shows data gaps, a Loki ingester is OOM. Three real scenarios that you diagnose step by step: from the alert through dashboard analysis to the root cause. Along the way, you develop a systematic troubleshooting approach and collect dos and don'ts that make the difference in production.

In the final project, you apply everything learned over the four days: working in groups, you build an end-to-end dashboard that integrates metrics and logs, configure alerts for critical scenarios, and define at least one SLO with error budget. You present the results to the class, receive feedback, and take home concrete blueprints for your own cluster.