OTel Logs Data Model and audit logs as a subsystem

A team passes a SOC 2 audit. The next year they switch log backends. They rewrite the log schema, redeploy the collector, and break the audit-log queries that the compliance team depends on. The log schema was vendor-proprietary — not OTel. The audit log separation was never formalized. Two months of remediation follows.

The OTel Logs Data Model

The OpenTelemetry Logs specification (API stable: late 2023; SDKs stable in most languages: late 2024) defines a log record as a typed structure with the following fields:

• Timestamp: when the event occurred (set by the application).
• ObservedTimestamp: when the collector observed the record. Under clock skew or collector backpressure these two diverge — senior teams alert on (ObservedTimestamp - Timestamp) p99 > 60s as a pipeline-health metric.
• SeverityNumber: a numeric ladder (TRACE=1-4, DEBUG=5-8, INFO=9-12, WARN=13-16, ERROR=17-20, FATAL=21-24) that lets backends compare severity across libraries that use different text labels (DEBUG vs Debug vs debug vs trace).
• SeverityText: the human-readable label (e.g., "INFO", "ERROR").
• Body: the human-readable log message.
• Resource: per-emitter attributes set once at service start — service.name, host.name, cloud.region, service.version, deployment.environment. These are the OTel Resource concept, not per-event fields.
• Attributes: the flat key-value map of event-specific data, using OTel Semantic Conventions where they apply — http.route, http.response.status_code, db.system, error.type, exception.type.
• TraceId, SpanId, TraceFlags: inherited from the active span at emit time (as covered in lesson 06).

The wire format is OTLP/Logs: protobuf over gRPC or HTTP/1.1. Compressed OTLP is 30-50% smaller than the equivalent JSON, which matters at fleet scale.

Why vendor portability matters

Adopting the OTel Logs Data Model now means backend swaps are a collector configuration change, not an instrumentation rewrite. A service that emits OTel-shaped logs today can point the OTel Collector at Honeycomb, ClickHouse, Loki, Datadog (via the OTLP receiver), or Splunk (via the OTel ingest path) by changing one exporter block. No application code changes.

The reverse — emitting to a vendor-proprietary schema — locks every log call site to that vendor’s field names and ingest format. Migration cost is effectively re-instrumenting every service.

The combination of cheap-to-adopt (every modern logger SDK is OTel-compatible or has a thin adapter) and high-cost-to-defer is the definition of a no-regret architecture choice.

Audit logs as a first-class subsystem

Operational logs answer “why is the service slow?” Audit logs answer “who did what, when, to which resource?”

They share the JSON schema and trace_id. But they differ on three axes that make mixing them a compliance failure:

Retention: operational logs are kept 7-90 days (hot/warm/cold tiers). Audit logs are kept 1-7 years (SOC 2: 1 year; HIPAA / PCI DSS: 7 years). A single index with a 30-day retention policy will silently delete audit events that compliance mandates be kept for years.

Immutability: operational logs can be deleted (retention policies, right-to-erasure, cost control). Audit logs must be append-only — hash-chained or cryptographically signed — so that tampering is detectable. Many compliance frameworks explicitly require that audit log integrity be verifiable.

Access: operational logs are readable by the development team for incident response. Audit logs are readable by a narrow role (platform security team + external auditors) with an audit-of-audits: access to the audit log index is itself logged.

The architectural pattern: emit audit events through the same logger SDK used for operational events, but with a category: "audit" attribute. The collector routes category=audit lines through a separate pipeline to a dedicated backend index with the stricter retention, immutability, and access properties. All other lines go to the operational pipeline. The routing is transparent to the application — it just sets the field.

What belongs in audit logs: authentication events (login, logout, failed login), authorization changes (role grants, permission changes), data access on regulated records (read of a customer’s health or financial data), configuration changes (secrets rotated, access control lists modified). The schema is narrower than operational logs: typically who, what, when, resource, outcome, trace_id.