Designing an Immutable Audit Log for ML Pipelines: Append-Only, Hash-Chained, and Regulator-Ready

The word "immutable" gets used loosely in software engineering — often to mean "we don't normally edit this" rather than "it is architecturally impossible to edit this." For compliance audit logs, that distinction matters. A history file that could theoretically be modified by a sufficiently privileged administrator does not provide the evidentiary value that regulated industries require from audit records.

This post describes the architectural decisions behind Cognify's audit log, why we made them, and the technical tradeoffs involved. We're writing this because the details of immutable log design are rarely discussed in ML infrastructure contexts — most writing on this topic comes from database systems or blockchain literature — and the constraints of an ML compliance audit log are specific enough that general-purpose solutions need careful adaptation.

What immutability actually means

Immutability in a compliance context means three distinct properties:

Write-once. Once a record is written, its content cannot be changed. The record representing "training run X started at time T with dataset version Y" is fixed at write time and cannot be altered by any user, administrator, or automated process.

Tamper-detectable. If any record is modified after writing — whether by a malicious actor, a software bug, or hardware corruption — the modification can be detected through verification. Tamper detection is not the same as tamper prevention; it means that if tampering occurred, it will be visible.

Append-only. The audit log grows by adding new records. It does not support deletion, update, or overwrite operations. If an incorrect record was written, the correction is itself a new record: "record ID 1234 was written in error; the correct value is X." The original incorrect record remains in the log alongside the correction.

Many "audit log" implementations in software systems satisfy none of these properties. A standard database table with an is_deleted flag and updated_at timestamp is not an audit log in any compliance-relevant sense — it's a table with change tracking, which is a different thing.

Append-only architecture fundamentals

The baseline architectural pattern for an append-only log is well-established: a sequential log file or table that permits INSERT but no UPDATE or DELETE. In a relational database, you can approximate this with row-level permissions that grant INSERT but not UPDATE or DELETE on the audit log table, combined with application-layer enforcement that never issues UPDATE or DELETE statements against the table.

The problem with database-level append-only enforcement is that it's enforced by the same system that could theoretically be modified by a database administrator with sufficient privileges. A DBA with SUPERUSER access on PostgreSQL can override row-level security. A cloud database service's customer-accessible administrative interface might expose a "truncate table" operation that bypasses normal access controls.

Stronger append-only guarantees come from storage systems with explicit immutability semantics. AWS S3 Object Lock with Compliance mode prevents deletion or modification of objects for a defined retention period, and the retention policy itself cannot be shortened by any IAM user including the root account during the lock period. Azure Blob Storage WORM (Write Once Read Many) policies have similar properties. These guarantees are stronger than database-level enforcement because they're implemented at the storage layer, below the application and database layers.

For ML compliance purposes, the practical approach is a combination: the application layer writes records to an append-only log structure, the underlying storage has object-level immutability enforcement at the storage service level, and the integrity of the log is verified by a separate cryptographic mechanism (hash chaining) that would detect tampering even if the storage-level immutability was somehow circumvented.

Hash chaining: making tampering detectable

Hash chaining is the mechanism that makes a log tamper-evident. Each log entry includes the hash of the previous entry as part of its content. When entry N+1 is written, it incorporates the hash of entry N. This creates a chain: if any entry in the chain is modified, the hash of that entry changes, which invalidates all entries after it in the chain — because they contain the (now incorrect) hash of the modified entry.

The mathematical structure here is straightforward:

entry_hash = sha256(
    previous_entry_hash +
    entry_timestamp +
    entry_content +
    entry_sequence_number
)

Where previous_entry_hash is the hash of the immediately preceding entry, and entry_content is the canonical serialization of the entry's data. The genesis entry (the first entry in the log) uses a known fixed value for previous_entry_hash.

To verify the integrity of the log, you recompute the hash chain from the genesis entry forward and compare each computed hash against the stored hash. If any entry has been modified, the recomputed hash of that entry will not match the stored hash in the following entry, and verification will fail at that point in the chain. You can identify exactly which entry was tampered with.

This approach is essentially the same structure used in blockchain systems, though without the distributed consensus mechanism — for a compliance audit log, distributed consensus isn't necessary because the chain is maintained by a single authoritative system. What matters is that the chain can be verified independently of the system that produced it, by anyone with a copy of the log.

Merkle tree structure for batch verification

For large logs, verifying the entire chain on each read is expensive. Merkle tree structure allows efficient partial verification: you can prove that a specific entry is present in the log and unmodified, without verifying every entry before it, by providing the Merkle path from the entry to the root.

Cognify's audit log uses a hybrid approach: entries within a "batch" (a fixed time window, typically one hour) are arranged as a Merkle tree, and batch roots are chained sequentially. This gives several practical benefits:

• Efficient spot verification. To verify that a specific training run record is authentic, you need the Merkle path within its batch, plus the chain of batch roots. This is O(log n) verification complexity rather than O(n).
• Parallel verification. Multiple verifiers can independently verify different portions of the log without reading the full chain.
• Efficient external audit. When a regulator requests verification of a specific model approval record, we can provide the Merkle proof for that specific record rather than exporting the entire audit log.

The SaaS immutability challenge

Implementing a genuinely immutable audit log in a SaaS product creates a specific challenge: your customers need to trust that you, as the SaaS vendor, haven't modified their audit records. The customer's compliance team relies on Cognify's audit log as evidence in their regulatory reviews — that reliance is only warranted if the customer can verify that the records are authentic.

We address this through several mechanisms:

Customer-verifiable chain roots. We publish the root hash of each completed batch to a customer-accessible endpoint. Customers can independently verify that their audit records match the published root, without relying on Cognify to perform the verification. If a record were modified in our storage, the published root would no longer match the recomputed root — the tampering would be detectable from the customer side.

On-premises deployment option. For customers with the highest trust requirements, Cognify's on-premises server runs within the customer's own infrastructure. The audit log storage is within the customer's control, and they can implement their own immutability guarantees at the storage layer. Cognify's software handles the hash chaining; the customer controls the storage.

Audit log export. The compliance package export includes a signed copy of the relevant audit log entries with their Merkle proofs. This export can be verified independently by the customer's compliance team or their external auditors, without requiring ongoing access to Cognify's systems.

The admin override problem

Every system has administrators with elevated privileges. The question of whether "even admins can't edit" is achievable in practice is important to address directly — because "no admin override" is a common claim that deserves scrutiny.

We're not claiming that Cognify's audit log is cryptographically protected against all possible attacks by a sufficiently determined insider with physical access to our infrastructure. What we're claiming is that any modification to audit log records would be detectable: it would break the hash chain, which means verification would fail, which means customers could detect the modification when they next verify their audit logs.

This is the appropriate standard for compliance purposes. The goal of an immutable audit log is not to make modification physically impossible — it's to make modification detectable and therefore discoverable. An audit log where modifications are immediately discoverable provides the evidentiary value that compliance frameworks require. An audit log where modifications are undetectable does not, regardless of the access control restrictions around it.

The practical implication is that our administrative tooling enforces separation: the systems that have write access to the audit log storage do not have the ability to regenerate valid hash chains after modification. An administrator who modified a stored record would produce an invalid chain that would be detected on the next verification — they cannot silently cover the modification.

Retention, archival, and the long-term read problem

ML compliance audit logs have long retention requirements. Healthcare AI documentation needs to be retained for the life of the system plus a specified period under state medical records laws and HIPAA. SR 11-7 for financial services model documentation typically requires retention for three to seven years. EU AI Act Article 18 requires ten years after the model is placed on the market.

Long-retention audit logs create a "long-term read problem": the data format, serialization format, hash algorithm, and verification tools need to remain accessible and usable over the retention period. A SHA-256 hash chain stored in a proprietary binary format might be technically valid for the full retention period, but practically unverifiable if the verification tools are no longer available.

Cognify's approach to this is: all audit log records are serialized as canonical JSON (UTF-8 encoded, sorted keys, no trailing whitespace), stored as line-delimited JSONL files, with SHA-256 hash chains using standard hex encoding. The verification algorithm is published and can be reimplemented independently of Cognify's software. Compliance package exports include the JSONL log entries and the verification algorithm documentation, so a customer can verify their audit records without requiring Cognify's software to be running.

This design ensures that a regulator examining a ten-year-old compliance package can verify its authenticity using the published algorithm and standard cryptographic tools — no dependency on Cognify's continued operation or software availability. For regulated industries where audit records are legal evidence, that kind of long-term verifiability is part of what makes the audit log genuinely useful rather than just technically compliant.