Cloudjur

Kubernetes-Security-Tooling

5 min read

Kubernetes Security Tooling

Relevant source files

The Kubernetes security stack in this repository provides a multi-layered defense and observability strategy. It encompasses real-time eBPF runtime security, automated vulnerability scanning, policy enforcement, and a centralized reporting dashboard. These tools ensure that the cluster remains compliant with security best practices and provides visibility into potential risks.

Security Architecture Overview

The security ecosystem is integrated into the GitOps workflow, with Flux CD managing the lifecycle of each component. Data flows from scanning and enforcement agents into the PolicyReport CRD format, which is then aggregated by the Policy Reporter for visualization and alerting.

Security Tooling Data Flow

This diagram illustrates how security findings from various operators are unified into a centralized reporting interface.

Security Data Flow
```mermaid
graph TD
subgraph "Scanning_Enforcement"
    [trivy-operator] -->|"Generates"| VULN["VulnerabilityReports"]
    [kyverno] -->|"Generates"| POL["PolicyReports"]
    [tetragon] -->|"Generates"| EBPF["eBPF Tracing Events"]
end

subgraph "Aggregation_Layer"
    VULN --> [trivy-operator-polr-adapter]
    [trivy-operator-polr-adapter] -->|"Converts to"| POL_CRD["PolicyReport CRDs"]
    POL --> POL_CRD
end

subgraph "Visualization_Alerting"
    POL_CRD --> [policy-reporter]
    [policy-reporter] -->|"Dashboard"| UI["Policy Reporter UI"]
    [policy-reporter] -->|"Logs"| LOKI["Loki (Grafana)"]
end


Sources: <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/policy-reporter/app/helmrelease.yaml#L32-L40" min=32 max=40 file-path="kubernetes/apps/security/policy-reporter/app/helmrelease.yaml">Hii</FileRef> <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator-polr-adapter/app/helmrelease.yaml#L17-L37" min=17 max=37 file-path="kubernetes/apps/security/trivy-operator-polr-adapter/app/helmrelease.yaml">Hii</FileRef>
 
## Vulnerability Management: Trivy Operator
 
The `trivy-operator` provides continuous artifact scanning for the cluster. It automatically discovers all resources (including Pods, ConfigMaps, and Secrets) and scans them for security risks.
 
### Implementation Details
*   **Scanning Mode**: Configured in `ClientServer` mode with a built-in Trivy server <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/helmrelease.yaml#L40-L49" min=40 max=49 file-path="kubernetes/apps/security/trivy-operator/app/helmrelease.yaml">Hii</FileRef>
*   **Scope**: Specifically targets `HIGH` and `CRITICAL` severities while ignoring unfixed vulnerabilities to reduce noise <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/helmrelease.yaml#L48-L50" min=48 max=50 file-path="kubernetes/apps/security/trivy-operator/app/helmrelease.yaml">Hii</FileRef>
*   **Resource Management**: Concurrent scan jobs are limited to 1 to minimize node impact, with a TTL of 300s for completed jobs <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/helmrelease.yaml#L37-L38" min=37 max=38 file-path="kubernetes/apps/security/trivy-operator/app/helmrelease.yaml">Hii</FileRef>
*   **Exclusions**: Resources created by `volsync` are skipped to avoid scanning ephemeral backup jobs <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/helmrelease.yaml#L64-L64" min=64  file-path="kubernetes/apps/security/trivy-operator/app/helmrelease.yaml">Hii</FileRef>
 
### Trivy Operator Components
| Component | Function | File Reference |
| :--- | :--- | :--- |
| `vulnerabilityScanner` | Scans container images for known CVEs | <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/helmrelease.yaml#L41-L41" min=41  file-path="kubernetes/apps/security/trivy-operator/app/helmrelease.yaml">Hii</FileRef> |
| `configAuditScanner` | Checks Kubernetes manifests for misconfigurations | <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/helmrelease.yaml#L42-L42" min=42  file-path="kubernetes/apps/security/trivy-operator/app/helmrelease.yaml">Hii</FileRef> |
| `nodeCollector` | Scans node-level components like `etcd` and `kubelet` | <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/helmrelease.yaml#L74-L101" min=74 max=101 file-path="kubernetes/apps/security/trivy-operator/app/helmrelease.yaml">Hii</FileRef> |
 
Sources: <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/helmrelease.yaml#L17-L102" min=17 max=102 file-path="kubernetes/apps/security/trivy-operator/app/helmrelease.yaml">Hii</FileRef>
 
## Runtime Security: Tetragon
 
Tetragon leverages eBPF to provide deep visibility into and enforcement of system calls and network activity. It is integrated directly with the Cilium CNI for advanced security filtering.
 
### Tracing and Enforcement
Tetragon is deployed via Helm and exports metrics to Prometheus for observability <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/tetragon/app/helmrelease.yaml#L32-L37" min=32 max=37 file-path="kubernetes/apps/security/tetragon/app/helmrelease.yaml">Hii</FileRef> It monitors:
*   **Process Execution**: Tracking binary execution and lifecycle.
*   **Network Observability**: Correlating socket activity with Kubernetes identities.
*   **File Access**: Monitoring sensitive file paths.
 
### Integration with Cilium
The `cilium` agent is configured with extensive capabilities to support eBPF operations, including `BPF`, `PERFMON`, and `NET_ADMIN` <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/kube-system/cilium/app/helmrelease.yaml#L91-L106" min=91 max=106 file-path="kubernetes/apps/kube-system/cilium/app/helmrelease.yaml">Hii</FileRef>
 
Sources: <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/tetragon/app/helmrelease.yaml#L17-L37" min=17 max=37 file-path="kubernetes/apps/security/tetragon/app/helmrelease.yaml">Hii</FileRef> <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/kube-system/cilium/app/helmrelease.yaml#L91-L106" min=91 max=106 file-path="kubernetes/apps/kube-system/cilium/app/helmrelease.yaml">Hii</FileRef>
 
## Policy Engine: Kyverno
 
Kyverno is used as the primary policy engine for the cluster, managing admission control and background scans.
 
*   **Kyverno Policies**: A dedicated Kustomization `kyverno-policies` manages the deployment of ClusterPolicies <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/kyverno-policies/ks.yaml#L1-L34" min=1 max=34 file-path="kubernetes/apps/security/kyverno-policies/ks.yaml">Hii</FileRef>
*   **Reporting**: Kyverno generates standard `PolicyReport` objects when resources violate defined security standards.
 
Sources: <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/kyverno-policies/ks.yaml#L1-L34" min=1 max=34 file-path="kubernetes/apps/security/kyverno-policies/ks.yaml">Hii</FileRef> <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/kyverno/app/helmrelease.yaml#L1-L30" min=1 max=30 file-path="kubernetes/apps/security/kyverno/app/helmrelease.yaml">Hii</FileRef>
 
## Centralized Reporting: Policy Reporter
 
The `policy-reporter` acts as the visualization layer for all security findings. It collects `PolicyReport` and `ClusterPolicyReport` CRDs and provides a web UI and alerting.
 
### Reporting Pipeline Configuration
*   **Trivy Integration**: Enabled via the `trivy` plugin, which allows the UI to display vulnerability data alongside Kyverno policy violations <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/policy-reporter/app/helmrelease.yaml#L39-L40" min=39 max=40 file-path="kubernetes/apps/security/policy-reporter/app/helmrelease.yaml">Hii</FileRef>
*   **Loki Integration**: Violations are forwarded to Loki for long-term storage and correlation with application logs <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/policy-reporter/app/helmrelease.yaml#L66-L67" min=66 max=67 file-path="kubernetes/apps/security/policy-reporter/app/helmrelease.yaml">Hii</FileRef>
*   **Email Summaries**: Configured to send periodic summaries and violation alerts via SMTP <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/policy-reporter/app/helmrelease.yaml#L45-L63" min=45 max=63 file-path="kubernetes/apps/security/policy-reporter/app/helmrelease.yaml">Hii</FileRef>
 
### Security Entity Mapping
 
This diagram maps the logical security concepts to the specific Kubernetes resources and Helm releases used in the `home-ops` repository.
 
| Security Entity Mapping |
| :--- |
| ```mermaid
graph LR
    subgraph "Policy_Management"
        [Kyverno_HelmRelease] -->|"app/helmrelease.yaml"| KYV["kyverno"]
        [Kyverno_Policies] -->|"ks.yaml"| KYV_POL["kyverno-policies"]
    end
 
    subgraph "Vulnerability_Scanning"
        [Trivy_Operator] -->|"app/helmrelease.yaml"| TRIVY["trivy-operator"]
        [Trivy_Adapter] -->|"app/helmrelease.yaml"| TRIVY_ADAPT["trivy-operator-polr-adapter"]
    end
 
    subgraph "Runtime_Security"
        [Tetragon_HelmRelease] -->|"app/helmrelease.yaml"| TETRA["tetragon"]
        [Cilium_Security] -->|"app/helmrelease.yaml"| CIL["cilium"]
    end
 
    subgraph "Reporting_UI"
        [Policy_Reporter] -->|"app/helmrelease.yaml"| POL_REP["policy-reporter"]
    end
``` |
Sources: <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/policy-reporter/app/helmrelease.yaml#L19-L19" min=19  file-path="kubernetes/apps/security/policy-reporter/app/helmrelease.yaml">Hii</FileRef> <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/helmrelease.yaml#L19-L19" min=19  file-path="kubernetes/apps/security/trivy-operator/app/helmrelease.yaml">Hii</FileRef> <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/tetragon/app/helmrelease.yaml#L19-L19" min=19  file-path="kubernetes/apps/security/tetragon/app/helmrelease.yaml">Hii</FileRef> <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/kyverno/app/helmrelease.yaml#L17-L17" min=17  file-path="kubernetes/apps/security/kyverno/app/helmrelease.yaml">Hii</FileRef>
 
## Observability and Dashboards
 
Security metrics are integrated into the cluster's Grafana instance.
 
*   **Cilium/Hubble Dashboards**: Detailed dashboards for network security and Hubble flow visibility are deployed as `GrafanaDashboard` resources <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/kube-system/cilium/app/grafanadashboard.yaml#L1-L33" min=1 max=33 file-path="kubernetes/apps/kube-system/cilium/app/grafanadashboard.yaml">Hii</FileRef>
*   **Trivy Dashboards**: Vulnerability trends and scan results are visualized through custom dashboards <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/trivy-operator/app/grafanadashboard.yaml" undefined  file-path="kubernetes/apps/security/trivy-operator/app/grafanadashboard.yaml">Hii</FileRef>
*   **Policy Reporter UI**: Accessible via an internal ingress, providing a high-level overview of the cluster's security posture <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/policy-reporter/app/helmrelease.yaml#L32-L35" min=32 max=35 file-path="kubernetes/apps/security/policy-reporter/app/helmrelease.yaml">Hii</FileRef>
 
Sources: <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/kube-system/cilium/app/grafanadashboard.yaml#L1-L33" min=1 max=33 file-path="kubernetes/apps/kube-system/cilium/app/grafanadashboard.yaml">Hii</FileRef> <FileRef file-url="https://github.com/chaijunkin/home-ops/blob/b5f8d898/kubernetes/apps/security/policy-reporter/app/helmrelease.yaml#L32-L35" min=32 max=35 file-path="kubernetes/apps/security/policy-reporter/app/helmrelease.yaml">Hii</FileRef>

Graph View

Backlinks