Loki, Tempo, and Pyroscope

Relevant source files

• infrastructure/ansible/playbooks/monitoring.yaml
• kubernetes/apps/home-automation/home-assistant/ks.yaml
• kubernetes/apps/observability/gatus/app/externalsecret.yaml
• kubernetes/apps/observability/gatus/app/grafana-dashboard.yaml
• kubernetes/apps/observability/gatus/app/helmrelease.yaml
• kubernetes/apps/observability/gatus/app/kustomization.yaml
• kubernetes/apps/observability/gatus/app/resources/config.yaml
• kubernetes/apps/observability/gatus/ks.yaml
• kubernetes/apps/observability/kube-prometheus-stack/app/externalsecret.yaml
• kubernetes/apps/observability/kube-prometheus-stack/app/kustomization.yaml
• kubernetes/apps/observability/kube-prometheus-stack/app/resources/alertmanager.yaml
• kubernetes/apps/observability/kube-prometheus-stack/app/scrapeconfig.yaml
• kubernetes/apps/observability/kube-prometheus-stack/ks.yaml
• kubernetes/apps/observability/loki/app/helmrelease.yaml
• kubernetes/apps/observability/loki/app/kustomization.yaml
• kubernetes/apps/observability/loki/ks.yaml
• kubernetes/apps/observability/pyroscope/app/helmrelease.yaml
• kubernetes/apps/observability/pyroscope/app/kustomization.yaml
• kubernetes/apps/observability/pyroscope/ks.yaml
• kubernetes/apps/observability/tempo/app/helmrelease.yaml
• kubernetes/apps/observability/tempo/app/kustomization.yaml
• kubernetes/apps/observability/tempo/ks.yaml

This section details the log aggregation, distributed tracing, and continuous profiling components of the observability stack. These services complement the metrics-focused Prometheus/Grafana implementation to provide full-spectrum visibility into the cluster’s operations.

Loki: Log Aggregation

Loki is deployed in SingleBinary mode, providing a simplified architecture suitable for home-ops while maintaining high performance via TSDB indexing.

Implementation and Storage

Loki is configured to use the tsdb shipping store with a v13 schema kubernetes/apps/observability/loki/app/helmrelease.yaml57-65 It utilizes filesystem storage for both the index and the chunks, backed by a 50Gi openebs-hostpath PersistentVolume kubernetes/apps/observability/loki/app/helmrelease.yaml56-81

Retention and Compaction

The system enforces a 14-day global retention period kubernetes/apps/observability/loki/app/helmrelease.yaml49 The compactor component manages this lifecycle, using a dedicated working directory for retention tasks and the filesystem for delete requests kubernetes/apps/observability/loki/app/helmrelease.yaml42-45

Alerting and Rules

Loki integrates with the cluster’s Alertmanager. It uses a sidecar container (ghcr.io/home-operations/k8s-sidecar) to discover and load alerting rules from across all namespaces kubernetes/apps/observability/loki/app/helmrelease.yaml98-104 These rules are processed by the internal ruler and sent to the alertmanager-operated service kubernetes/apps/observability/loki/app/helmrelease.yaml67-75

Data Flow: Log Ingestion to Alerting

The following diagram illustrates how logs move from ingestion to the generation of alerts.

Loki Log Processing Pipeline

Sources: kubernetes/apps/observability/loki/app/helmrelease.yaml35-75kubernetes/apps/observability/loki/app/helmrelease.yaml98-104

Tempo: Distributed Tracing

Tempo provides high-volume trace storage and discovery, primarily receiving data via the OpenTelemetry Protocol (OTLP).

Configuration

Tempo is configured with a 24-hour retention period for traces kubernetes/apps/observability/tempo/app/helmrelease.yaml41 It exposes both gRPC (port 4317) and HTTP (port 4318) OTLP receivers to ingest spans from instrumented applications kubernetes/apps/observability/tempo/app/helmrelease.yaml44-50

Storage and Persistence

Similar to Loki, Tempo uses a local storage backend targeting /var/tempo/traceskubernetes/apps/observability/tempo/app/helmrelease.yaml52-56 It is backed by a 10Gi openebs-hostpath volume kubernetes/apps/observability/tempo/app/helmrelease.yaml92-95

Tempo Entity Mapping

Feature	Code Entity / Value	File Reference
Ingestion Protocol	otlp (grpc/http)	kubernetes/apps/observability/tempo/app/helmrelease.yaml44-50
Retention Policy	24h	kubernetes/apps/observability/tempo/app/helmrelease.yaml63
Storage Class	openebs-hostpath	kubernetes/apps/observability/tempo/app/helmrelease.yaml94
Ingester Config	max_block_duration: 5m	kubernetes/apps/observability/tempo/app/helmrelease.yaml59

Sources: kubernetes/apps/observability/tempo/app/helmrelease.yaml33-96

Pyroscope: Continuous Profiling

Pyroscope is the profiling component of the stack, used to analyze application performance (CPU, memory, etc.) over time.

Deployment

Pyroscope is deployed via the Grafana Helm chart in the observability namespace kubernetes/apps/observability/pyroscope/app/helmrelease.yaml11-19 The current implementation uses a single replica kubernetes/apps/observability/pyroscope/app/helmrelease.yaml34

Ingress and Access

Access to the Pyroscope UI is managed through an HTTPRoute resource, which exposes the service at pyroscope.cloudjur.com via the envoy-internal gateway kubernetes/apps/observability/pyroscope/app/helmrelease.yaml39-46

Sources: kubernetes/apps/observability/pyroscope/app/helmrelease.yaml1-47

Integration with Grafana Stack

The components are tightly integrated with the kube-prometheus-stack to provide a unified observability experience.

Service Monitoring

Both Loki and Tempo export their own metrics for self-monitoring.

• Loki: Enables serviceMonitor and provides pre-configured Grafana dashboards via the sidecar pattern, labeled for discovery by Grafana kubernetes/apps/observability/loki/app/helmrelease.yaml105-117
• Tempo: Configured with a serviceMonitor scraping every 30 seconds kubernetes/apps/observability/tempo/app/helmrelease.yaml65-75

Alerting Integration

Loki’s structuredConfig explicitly points to the cluster’s Alertmanager service: http://alertmanager-operated.observability.svc.cluster.local:9093kubernetes/apps/observability/loki/app/helmrelease.yaml70 This allows log-based alerts to follow the same routing logic as metric-based alerts defined in the Alertmanager configuration kubernetes/apps/observability/kube-prometheus-stack/app/resources/alertmanager.yaml4-33

Observability Stack Integration

Sources: kubernetes/apps/observability/loki/app/helmrelease.yaml70-81kubernetes/apps/observability/tempo/app/helmrelease.yaml92-95kubernetes/apps/observability/kube-prometheus-stack/app/resources/alertmanager.yaml40-53