Kubernetes Audit Observation

Kubernetes Audit Observation is the discipline of capturing, retaining, and analyzing the structured record of every request made to the Kubernetes API server, in order to reconstruct who did what, to which resource, when, and with what outcome. It is the accountability layer of cluster observability, distinct from metrics or application logs in that it records administrative and access activity rather than workload behavior.

The Kubernetes Audit Logging Mechanism

Audit Events

Every request that reaches the API server, whether from a human operator via kubectl, a controller, or an external integration, can generate an audit event. Each event captures:

• The request stage (RequestReceived, ResponseStarted, ResponseComplete, Panic)
• The identity of the requester, including username, UID, and group memberships
• The verb performed (get, list, create, update, patch, delete)
• The target resource, subresource, namespace, and object name
• The source IP address of the request
• The response status, including HTTP status code
• Optionally, the full request and response body, depending on the configured audit level

Audit Levels

The API server supports four audit levels, from least to most verbose:

• None — the event is dropped entirely.
• Metadata — only request metadata (user, verb, resource, timestamp) is logged, excluding request/response bodies.
• Request — metadata plus the request body is logged.
• RequestResponse — metadata, request body, and response body are all logged.

Higher levels provide more forensic detail but incur significantly higher storage and processing cost, so audit policies typically apply differentiated levels per resource type.

Audit Policy Configuration

Policy Structure

An audit policy is a list of rules evaluated in order; the first matching rule determines the audit level applied to a request.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: ""
        resources: ["endpoints", "services"]

  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]

  - level: Metadata
    resources:
      - group: ""
        resources: ["pods"]

  - level: Metadata

Enabling Auditing on the API Server

kube-apiserver \
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \
  --audit-log-path=/var/log/kubernetes/audit.log \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=10 \
  --audit-log-maxsize=100

Audit backends are pluggable: the log backend writes newline-delimited JSON to a file on the control-plane node, while the webhook backend forwards events to an external HTTP endpoint for centralized ingestion, commonly a SIEM or a log aggregation pipeline.

Observing and Interrogating Audit Data

Structure of a Single Audit Event

kind: Event
apiVersion: audit.k8s.io/v1
level: RequestResponse
auditID: 8b9e2f1a-1234-4abc-9def-0987654321ab
stage: ResponseComplete
requestURI: /api/v1/namespaces/production/secrets/db-credentials
verb: get
user:
  username: alice@example.com
  groups: ["system:authenticated"]
sourceIPs: ["10.0.4.22"]
objectRef:
  resource: secrets
  namespace: production
  name: db-credentials
responseStatus:
  code: 200
requestReceivedTimestamp: "2024-05-03T14:22:01.000000Z"
stageTimestamp: "2024-05-03T14:22:01.045000Z"

Common Observational Questions Audit Data Answers

• Which identity read or modified a specific Secret, and when.
• Whether a ClusterRoleBinding change was made by an expected automation account or an unexpected human actor.
• Whether repeated 403 Forbidden responses indicate reconnaissance or a misconfigured service account.
• The full sequence of API calls that led to an unexpected cluster state change, reconstructed from auditID correlation across RequestReceived and ResponseComplete stages.

Operational Integration

Forwarding to External Systems

Because control-plane nodes have limited local disk retention, audit logs are typically shipped off-node using a log-forwarding agent (Fluent Bit, Vector, or a sidecar tailing the audit log file) into a centralized store such as Elasticsearch, a SIEM, or object storage for long-term compliance retention.

Correlation with Other Observability Signals

Audit records lack the continuous cadence of metrics or traces; they are event-driven and sparse relative to telemetry volume, but they are uniquely authoritative for questions of authorization and administrative intent. Effective observation combines audit data with RBAC configuration snapshots to distinguish a legitimate privileged action from a policy violation.