Monitoring and Observability

Monitoring and Observability

Knowing the system is broken — and why

Monitoring tells you whether a system is working. Observability tells you why it is not. Production systems require both: dashboards and alerts for operational awareness, plus the ability to drill into logs, metrics, and traces to diagnose problems. This page covers the three pillars of observability — metrics, logs, and traces — along with the primary tooling for each.

ℹ️ Observability and security monitoring share the same plumbing: the metrics and logs you collect for reliability are the same signals a SIEM correlates for detection. The difference is the question asked — “is it healthy?” vs. “is it compromised?” — which is why this page pairs with SIEM, SOC, and Threat Detection.

The Three Pillars of Observability

Pillar	What it captures	Primary tools
Metrics	Numeric measurements over time (CPU %, request rate, error count)	Prometheus, Grafana, Datadog, Zabbix
Logs	Timestamped text records of events	Elastic Stack, Loki, Graylog, Splunk
Traces	Request paths across distributed services	Jaeger, Zipkin, OpenTelemetry

Infrastructure Monitoring

Prometheus

Prometheus is an open-source monitoring system and time-series database. It scrapes metrics from HTTP endpoints on a pull model — targets expose a /metrics endpoint and Prometheus fetches it on a configurable interval.

Architecture

Prometheus server — scrapes and stores metrics
Exporters — adapters that expose metrics for systems that don’t natively support Prometheus (Node Exporter for Linux hosts, mysqld_exporter for MySQL, etc.)
Alertmanager — receives alert notifications from Prometheus and routes them to email, PagerDuty, Slack, etc.
Pushgateway — accepts metrics pushed from short-lived jobs (batch processes, cron jobs)

Installation

# Download and install Prometheus
wget https://github.com/prometheus/prometheus/releases/latest/download/prometheus-2.52.0.linux-amd64.tar.gz
tar xf prometheus-*.tar.gz
sudo mv prometheus-*/prometheus /usr/local/bin/
sudo mv prometheus-*/promtool /usr/local/bin/

# Node Exporter (exposes Linux host metrics)
wget https://github.com/prometheus/node_exporter/releases/latest/download/node_exporter-1.8.0.linux-amd64.tar.gz
tar xf node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/

# Run node_exporter as a systemd service
sudo useradd -rs /bin/false node_exporter

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl http://localhost:9100/metrics | head -20   # verify metrics endpoint

Prometheus configuration

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s       # how often to scrape targets
  evaluation_interval: 15s   # how often to evaluate rules

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets:
          - 'web1:9100'
          - 'web2:9100'
          - 'db1:9100'

  - job_name: 'nginx'
    static_configs:
      - targets: ['web1:9113']   # nginx-prometheus-exporter

  # Service discovery via file (dynamic target list)
  - job_name: 'dynamic-hosts'
    file_sd_configs:
      - files: ['/etc/prometheus/targets/*.json']
        refresh_interval: 30s

promtool check config /etc/prometheus/prometheus.yml   # validate config

PromQL — Prometheus Query Language

# Current CPU usage (all CPUs, averaged)
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory available (bytes)
node_memory_MemAvailable_bytes

# HTTP request rate (requests/second over last 5 minutes)
rate(http_requests_total[5m])

# Error rate (percentage)
rate(http_requests_total{status=~"5.."}[5m])
  / rate(http_requests_total[5m]) * 100

# 95th percentile request latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Disk usage percentage
(node_filesystem_size_bytes - node_filesystem_free_bytes)
  / node_filesystem_size_bytes * 100

# Instant vector: current value of a metric
up                                          # 1 if target is up, 0 if down

# Filter by label
http_requests_total{job="nginx", method="GET"}

# Range vector: values over a time window
node_cpu_seconds_total[5m]

Alerting rules

# /etc/prometheus/rules/alerts.yml
groups:
  - name: host-alerts
    rules:
      - alert: HighCPUUsage
        expr: >
          100 - (avg by(instance)
            (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on "
          description: "CPU usage is % for 5+ minutes"

      - alert: DiskSpaceLow
        expr: >
          (node_filesystem_size_bytes - node_filesystem_free_bytes)
            / node_filesystem_size_bytes * 100 > 85
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on "
          description: "Disk  is % full"

      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance  is down"

Python application metrics with prometheus-client

from prometheus_client import Counter, Histogram, Gauge, start_http_server
import time

# Define metrics
REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status_code']
)
REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency',
    ['endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
)
ACTIVE_CONNECTIONS = Gauge(
    'active_connections',
    'Number of active connections'
)

# Instrument a function
def handle_request(method, endpoint):
    start = time.time()
    ACTIVE_CONNECTIONS.inc()
    try:
        # ... process request ...
        status = 200
        REQUEST_COUNT.labels(method=method, endpoint=endpoint,
                             status_code=status).inc()
    finally:
        ACTIVE_CONNECTIONS.dec()
        REQUEST_LATENCY.labels(endpoint=endpoint).observe(time.time() - start)

# Expose metrics on port 8000
start_http_server(8000)

Grafana

Grafana is the standard visualization layer for Prometheus metrics (and many other data sources: InfluxDB, Elasticsearch, PostgreSQL, CloudWatch, etc.).

# Install Grafana (Debian/Ubuntu)
sudo apt install -y apt-transport-https software-properties-common wget
# Store the signing key in a keyring (apt-key is deprecated)
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key \
    | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" \
    | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install grafana
sudo systemctl enable --now grafana-server

# Access UI at http://localhost:3000 (default: admin/admin)

Grafana as code (provisioning)

Rather than clicking through the UI, dashboards and data sources can be defined in YAML files that Grafana loads at startup:

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://localhost:9090
    isDefault: true
    editable: false

Dashboard JSON files can be exported from the UI and stored in /etc/grafana/provisioning/dashboards/.

Datadog

Datadog is a commercial monitoring-as-a-service platform. The agent is installed on hosts and ships metrics, logs, and traces to Datadog’s SaaS platform.

# Install Datadog agent (Debian/Ubuntu)
DD_API_KEY=<your-api-key> DD_SITE="datadoghq.com" \
    bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

sudo systemctl status datadog-agent
sudo datadog-agent status             # detailed agent status

# /etc/datadog-agent/datadog.yaml (partial)
api_key: <YOUR_API_KEY>
site: datadoghq.com
hostname: web-prod-01
tags:
  - env:production
  - role:web
  - team:platform
logs_enabled: true

Zabbix

Zabbix is a mature open-source enterprise monitoring platform with a web UI, agents, SNMP polling, and built-in alerting. It is common in enterprise environments and excels at monitoring network devices and legacy systems.

# Zabbix agent installation (on monitored hosts)
sudo apt install zabbix-agent2
sudo systemctl enable --now zabbix-agent2

# Configure agent
# /etc/zabbix/zabbix_agent2.conf:
# Server=<zabbix-server-ip>
# ServerActive=<zabbix-server-ip>
# Hostname=<this-host-fqdn>

Logs Management

The ELK Stack (Elastic Stack)

The Elastic Stack — Elasticsearch, Logstash, and Kibana (plus Beats for collection) — is the most widely deployed log aggregation platform.

Component	Role
Elasticsearch	Search and analytics database; stores and indexes logs
Logstash	Log pipeline: ingest, parse, transform, forward
Kibana	Web UI for searching, visualizing, and alerting on logs
Filebeat	Lightweight log shipper that tails log files
Metricbeat	Ships system/service metrics to Elasticsearch

Filebeat configuration

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    fields:
      service: nginx
      environment: production

  - type: log
    enabled: true
    paths:
      - /var/log/myapp/*.log
    multiline:
      type: pattern
      pattern: '^\d{4}-\d{2}-\d{2}'   # start of log line
      negate: true
      match: after

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  username: "filebeat_user"
  password: "${FILEBEAT_PASSWORD}"
  ssl.certificate_authorities: ["/etc/ssl/certs/ca.crt"]

setup.kibana:
  host: "https://kibana:5601"

Logstash pipeline

# /etc/logstash/conf.d/nginx.conf
input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][service] == "nginx" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    geoip {
      source => "clientip"
    }
    mutate {
      convert => { "bytes" => "integer" }
      convert => { "response" => "integer" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://elasticsearch:9200"]
    index => "nginx-%{+YYYY.MM.dd}"
    user => "logstash_writer"
    password => "${LOGSTASH_PASSWORD}"
  }
}

Loki

Loki (by Grafana Labs) is a horizontally scalable log aggregation system designed to work alongside Prometheus and Grafana. Unlike Elasticsearch, Loki indexes only labels (not the full log content), keeping storage costs low. Logs are queried using LogQL.

# Install Promtail (the log shipper for Loki)
wget https://github.com/grafana/loki/releases/latest/download/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail

# /etc/promtail/config.yaml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: ['localhost']
        labels:
          job: nginx
          host: web-prod-01
          __path__: /var/log/nginx/*.log

  - job_name: systemd-journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: unit

LogQL queries

# Show all nginx logs from the last hour
{job="nginx"}

# Filter to error lines
{job="nginx"} |= "ERROR"

# Parse structured logs and filter
{job="myapp"} | json | level = "error"

# Count log lines per minute
count_over_time({job="nginx"}[1m])

# Rate of error logs
rate({job="nginx"} |= "500" [5m])

Graylog

Graylog is an open-source log management platform with a built-in web UI, pipelines for log processing, and alerting. It uses MongoDB for metadata and Elasticsearch/OpenSearch for log storage. A common choice for on-premises enterprise deployments.

Splunk

Splunk is a commercial log management and SIEM platform. It is widely used in enterprise security operations. Splunk’s query language (SPL) is powerful for security analytics.

# Basic SPL queries
index=nginx | stats count by status
index=nginx status=500 | head 20
index=auth "Failed password" | stats count by src_ip | sort -count

Application Monitoring

Application monitoring tracks the behavior of running code: request rates, error rates, latency distributions, database query times, and external service call durations.

OpenTelemetry

OpenTelemetry (OTel) is the CNCF standard for instrumentation. It provides vendor-neutral APIs, SDKs, and a collector that can export to any backend (Jaeger, Zipkin, Datadog, Grafana Tempo, etc.).

pip install opentelemetry-sdk opentelemetry-exporter-otlp

# Instrument a Python application with OpenTelemetry
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Configure the tracer
resource = Resource.create({
    "service.name": "myapp",
    "service.version": "1.2.3",
    "deployment.environment": "production",
})
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

# Instrument application code
def process_order(order_id):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.source", "web")

        with tracer.start_as_current_span("validate_order"):
            validate(order_id)          # creates a child span

        with tracer.start_as_current_span("charge_payment") as payment_span:
            try:
                charge(order_id)
            except PaymentError as e:
                payment_span.set_status(
                    trace.Status(trace.StatusCode.ERROR, str(e))
                )
                raise

OpenTelemetry Collector

The Collector receives telemetry, processes it, and exports to one or more backends. Configure it in otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: 'myapp'
          static_configs:
            - targets: ['localhost:8000']

processors:
  batch:
    timeout: 1s
  memory_limiter:
    limit_mib: 512

exporters:
  # The standalone `jaeger` exporter was removed from the Collector — modern
  # Jaeger ingests OTLP natively, so export OTLP to its gRPC port (4317).
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
  # The `logging` exporter was renamed to `debug`.
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheus]

Jaeger

Jaeger is an open-source distributed tracing system originally built by Uber. It stores and visualizes traces collected from instrumented applications.

# Run Jaeger all-in-one for development (Docker)
docker run -d --name jaeger \
    -p 6831:6831/udp \
    -p 16686:16686 \
    -p 14268:14268 \
    -p 4317:4317 \
    jaegertracing/all-in-one:latest

# Access UI at http://localhost:16686

Jaeger concepts:

Trace — the complete journey of a request through all services
Span — a single named, timed operation within a trace
Service — a logical component; appears in the Jaeger UI service list
Sampling — only a fraction of traces are recorded (head or tail sampling)

New Relic and Datadog APM

Commercial Application Performance Monitoring (APM) platforms provide auto-instrumentation with minimal code changes:

# Datadog APM — install ddtrace, then run with ddtrace-run
pip install ddtrace
# Launch application:
# ddtrace-run python app.py

# Or instrument manually
from ddtrace import tracer

@tracer.wrap(service="myapp", resource="process_order")
def process_order(order_id):
    # ... function body ...
    pass

Alerting best practices

Effective alerting reduces alert fatigue and ensures on-call engineers respond only to actionable problems.

Principle	Description
Alert on symptoms, not causes	“5xx error rate > 1%” beats “CPU > 80%” — one directly affects users
Minimum alert duration	Require a condition to persist (e.g., `for: 5m`) before firing
Severity levels	Distinguish `critical` (page on-call now) from `warning` (review next business day)
Runbook links	Every alert annotation should link to a runbook explaining how to respond
Regular review	Remove alerts that never fire or always fire without action

Alertmanager configuration

# /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alerts@example.com'

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - matchers:
        - severity = critical
      receiver: 'pagerduty'
    - matchers:
        - severity = warning
      receiver: 'slack'

receivers:
  - name: 'default'
    email_configs:
      - to: 'team@example.com'

  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/...'
        channel: '#alerts'
        title: ''
        text: ''

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '<pagerduty-integration-key>'

inhibit_rules:
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal: ['instance']    # suppress warnings when critical fires for same instance

Key takeaways

Monitoring vs. observability: monitoring answers “is it working?”, observability answers “why not?” — you need both, built on the three pillars: metrics, logs, and traces.
Prometheus pulls metrics from /metrics endpoints (exporters bridge systems that don’t speak it natively), stores them as time series, and queries with PromQL; Grafana visualizes; Alertmanager routes alerts. Provision both as code, not by clicking.
Logs scale differently by tool: the Elastic Stack indexes full content (powerful, heavier), while Loki indexes only labels (cheap, Grafana-native); Graylog and Splunk round out the on-prem/enterprise options.
Traces follow a request across services. OpenTelemetry is the vendor-neutral CNCF standard for instrumentation; export to Jaeger, Tempo, or a commercial APM.
Alert on symptoms, not causes (“5xx rate > 1%” beats “CPU > 80%”), require a sustained for: duration, separate critical (page now) from warning, and link every alert to a runbook — alert fatigue is itself an outage risk.

References

Prometheus documentation. https://prometheus.io/docs/
Grafana documentation. https://grafana.com/docs/grafana/latest/
Grafana Loki documentation. https://grafana.com/docs/loki/latest/
Elastic Stack documentation. https://www.elastic.co/guide/index.html
OpenTelemetry documentation. https://opentelemetry.io/docs/
Jaeger tracing. https://www.jaegertracing.io/docs/
Google SRE Book — Monitoring Distributed Systems. https://sre.google/sre-book/monitoring-distributed-systems/

Related course pages: SIEM, SOC, and Threat Detection · Wazuh SIEM · Incident Response

🛠️ Maintenance note: the download URLs pin versions (Prometheus 2.52.0, Node Exporter 1.8.0) that move — note Prometheus has since released a 3.x line, so bump these each term. The OpenTelemetry Collector config schema continues to evolve (exporter and processor names change between releases), so re-verify the collector pipeline against the installed version.