Most logging and monitoring tools you send data to. Prometheus, by contrast, is built around pulling: it reaches out to the things it monitors on a schedule and asks for their current numbers. That one design decision shapes almost everything else about how it’s deployed and configured, and this guide covers the architecture that follows from it.

Scraping: a pull instead of a push

Prometheus periodically sends an HTTP request to every system it monitors and reads back a snapshot of that system’s current metric values. Prometheus calls this scraping, and each thing it scrapes is a target. A scrape carries only the values at that moment; there’s no history in the payload; Prometheus is what turns a sequence of scrapes, taken at intervals, into a history.

The pull model has a practical consequence: anything Prometheus monitors needs to expose an HTTP endpoint (/metrics by convention) that returns its current values in a format Prometheus understands, rather than Prometheus needing an inbound listener for every possible source pushing data at it.

Targets: instrumented apps and exporters

A target falls into one of two categories:

• Directly instrumented applications. Your own service, using one of Prometheus’s official client libraries, tracks and exposes its own metrics natively at a /metrics endpoint.
• Exporters, for anything that doesn’t (or can’t) expose Prometheus metrics itself: a database, a Linux host, a network appliance. An exporter is a small process that runs alongside the thing being monitored, translating its native stats into Prometheus’s format. node_exporter for host-level metrics (CPU, memory, disk, network) and cAdvisor for per-container metrics are two of the most widely used.

Because the exposition format is deliberately simple text (covered in the next guide), writing an exporter, or even hand-rolling metrics without a client library at all, is realistic even for something with no official Prometheus support.

Finding targets: static config or service discovery

Prometheus needs a list of targets to scrape. For a handful of stable hosts, listing them by hand in Prometheus’s configuration file is fine. For anything larger or more dynamic, machines and containers that come and go, that quickly becomes impractical to maintain by hand. Prometheus instead integrates with service discovery mechanisms, DNS, the Kubernetes API server, cloud provider APIs, and others, to build and continuously update its target list automatically as the environment changes.

Storage: the local time-series database

Scraped samples land in Prometheus’s own local time-series database (TSDB) on disk. By default, Prometheus retains 15 days of data; that’s a configurable flag (--storage.tsdb.retention.time), not a hard limit, but 15 days is enough for the overwhelming majority of real troubleshooting, since most incidents are compared against minutes or hours ago, rarely months. For long-term retention or a centralized store across many Prometheus servers, Prometheus can also be configured to forward (remote-write) samples to an external storage system with different trade-offs than its own local TSDB.

What sits around the core server

A few components complete a typical setup:

• Alertmanager is a separate component Prometheus hands firing alerts to. It deduplicates, groups, and routes them to email, Slack, PagerDuty, or whatever else you configure, decoupling “detecting a problem” from “deciding who gets notified and how.”
• Grafana (or Prometheus’s own bundled web UI, or other tools) is how you’d typically explore and visualize the data; Prometheus itself focuses on collecting and storing metrics reliably, not on being the dashboarding layer.
• PromQL, Prometheus’s query language, is what both Grafana’s dashboards and Prometheus’s own alerting rules are built on. Alerting and dashboarding aren’t two different systems bolted together; they’re two consumers of the same query language against the same data. Later guides in this series cover the data model and PromQL directly.

Putting the pieces together

   ┌──────────────┐        scrapes over HTTP        ┌──────────────┐
   │  Prometheus  │ ───────────────────────────────▶ │    Target    │
   │    server    │                                    │ (/metrics)   │
   └──────┬───────┘                                    └──────────────┘
          │ stores samples in
          ▼
   local TSDB (default 15d retention)
          │
   ┌──────┴───────────────┬───────────────────────┐
   ▼                       ▼                       ▼
firing alerting rules   PromQL queries          remote write
   │                    (Grafana, ad hoc)        (optional, external storage)
   ▼
Alertmanager → email / Slack / PagerDuty / ...

Everything downstream, alerts, dashboards, ad hoc debugging, reads from the same TSDB that scraping fills. The rest of this series starts with what actually gets stored in it.