Deployment drift as a security finding

Source review creates a tempting assumption: the code being reviewed is the code running in production.

In a mature delivery process, that assumption may be supported by evidence. A deployment pipeline can start from a known commit, build an immutable artifact, record its digest, attach provenance metadata, and promote that exact artifact through staging and production.

In smaller environments, the relationship is often weaker.

An image may be built manually, pushed under a mutable tag, pulled onto a server, and restarted through Docker Compose. Infrastructure configuration may be edited directly on the host. Monitoring services may be added outside the repository. A production fix may exist live but not in source, while a control visible in source may never have reached production.

At that point, deployment drift is no longer merely a limitation attached to the assessment. It becomes a security-relevant condition in its own right.

The central problem is not simply that two systems differ. It is that the organization can no longer establish which system state is authoritative, reproducible, or covered by the review.

What deployment drift means

Deployment drift exists when the effective production system cannot be reliably reconstructed from the reviewed source, build configuration, and deployment records.

The drift may affect:

Application code
Container images
Runtime dependencies
Environment variables
Database migrations
Reverse-proxy configuration
Published ports
Host-level services
Security headers
Credentials and privileges
Logging and monitoring
Manual hotfixes

Drift does not automatically mean production is less secure.

Production may contain a hotfix that has not yet been committed. The repository may contain a security improvement that has not yet been deployed. A host operator may have added a defensive firewall rule that does not exist in infrastructure-as-code.

The problem is that neither direction can be trusted without additional evidence.

A reviewer cannot safely conclude that a source-level control protects production. An operator cannot safely conclude that the repository is sufficient to rebuild the running system after a host failure.

That uncertainty must be made explicit.

The attribution problem

Every source-based security claim depends on an artifact-attribution chain:

Reviewed commit
→ build process
→ generated artifact
→ registry digest
→ deployment configuration
→ running process

Confidence decreases when one of these links is missing.

Consider a review that confirms server-side authorization on every project-scoped endpoint. That is meaningful evidence about the reviewed commit.

It is not proof that the same authorization logic is running in production when:

The deployed commit is unknown
The application uses a mutable image tag
The runtime exposes no build identifier
The image was built on an unrecorded workstation
Production configuration differs from the repository
No deployment event maps the image to an approved change

The correct conclusion is not that authorization is broken.

The correct conclusion is:

Authorization is verified in the reviewed source. Its presence in the effective production deployment remains unverified.

That distinction is central to evidence-disciplined security work.

A review should distinguish between at least three kinds of statement:

Source-verified fact

The reviewed commit performs a project-scoped authorization check before loading the resource.

Runtime-verified fact

The deployed system rejected a cross-project request during authorized testing.

Inference

The production deployment is expected to contain the reviewed control, but the running artifact cannot be mapped to the reviewed commit.

These statements are related, but they are not interchangeable.

When drift becomes a finding

Not every discrepancy deserves a formal vulnerability entry.

I treat deployment drift as a reportable security weakness when it creates a concrete failure in attribution, reproducibility, incident response, recovery, or change control.

Production cannot be mapped to a reviewed commit

The running service exposes no commit SHA, build identifier, immutable image digest, or equivalent evidence.

A source review can still describe the reviewed code accurately, but it cannot validate the production revision.

Deployment uses mutable references

A reference such as latest can point to a different image tomorrow without any change to the deployment definition.

The configuration appears unchanged even though the executed code has changed.

Runtime observations contradict source

Examples include:

A security header exists in source but is absent live
A validation rule behaves differently in production
Additional ports or services appear during reconnaissance
Production uses a different TLS configuration
A worker or queue exists in production but not in the reviewed source
The database migration level does not match the repository

A contradiction does not prove which side is correct. It proves that the attribution chain is incomplete.

Reproduction depends on operator memory

The real deployment process lives in shell history, undocumented commands, local scripts, or direct server edits.

This makes the system difficult to reproduce consistently and difficult to review independently.

Rollback is not deterministic

The operator cannot identify and redeploy the exact previously known-good artifact.

“Roll back to the previous version” is not a meaningful procedure when the previous digest was never recorded.

Why this matters beyond audit confidence

Deployment provenance is often treated as a documentation concern. Its consequences are operational and security-critical.

Incident response

During an incident, the team needs to answer:

What changed?
Which artifact is running?
Who deployed it?
When was it deployed?
Was it rebuilt after the reviewed commit?
Is the same digest running elsewhere?
Which artifact is known to be clean?
Which revision can replace the suspected one?

Without provenance, containment becomes guesswork.

The team may redeploy an image that contains the same compromise, destroy useful evidence, or revert to an artifact whose contents are equally unknown.

Supply-chain security

A compromised registry account becomes considerably more dangerous when production automatically pulls mutable tags.

An attacker can replace the image behind a familiar tag without creating a corresponding source-code change.

The deployment system then retrieves malicious code through its normal operating process.

Recovery

A replacement host cannot be rebuilt confidently when production depends on:

Unknown images
Untracked host configuration
Undocumented environment variables
Manually created networks
Unrecorded migrations
Operator-specific commands

Backups protect data. They do not automatically preserve the application state required to use that data.

Finding closure

A developer may fix a vulnerability in source, but the finding cannot be considered closed until the corrected artifact is tied to the running production revision.

A defensible closure record should identify:

The fixing commit
The resulting artifact
The immutable digest
The deployment event
The running revision
The retest evidence

Without that chain, the report can say that a fix exists. It cannot say that production is fixed.

How to assess drift

A practical assessment should collect evidence from both directions.

From source and build configuration

Review:

Commit SHA
Working-tree state
Dockerfiles
Lockfiles
Build scripts
CI/CD configuration
Deployment manifests
Registry destination
Image tags
Image digests
Build metadata
Signing and attestation configuration
Migration definitions

From production

Collect:

Running image IDs and registry digests
Container labels
Effective orchestration configuration
Published ports
Host listeners
Reverse-proxy configuration
Environment-variable names
Database migration level
Health and build metadata
Installed host services
Relevant firewall rules
Deployment timestamps

The objective is not simply to compare two directories.

The objective is to determine whether a trustworthy chain exists between the reviewed source and the running system.

Manual deployment is not automatically insecure

A small team does not need an elaborate enterprise platform to establish provenance.

A disciplined manual process can still produce strong attribution:

Start from a clean checkout.
Record the full commit SHA.
Run the required tests.
Build the artifact.
Push it to the approved registry.
Capture the immutable registry digest.
Sign the digest.
Deploy using image@sha256:<digest>.
Record the operator and deployment time.
Verify the running digest.
Retain the previous known-good digest.

Automation reduces human error and improves consistency, but the underlying control is artifact identity, not the presence of a fashionable CI/CD product.

A manually deployed system can be attributable.

An automated system can still be opaque when it rebuilds mutable inputs, deploys tags rather than digests, or fails to record what actually reached production.

A useful production invariant

A defensible deployment process should enforce:

Every running production artifact maps to an approved repository commit, a recorded build, an immutable artifact digest, and a named deployment event.

This creates objective verification criteria:

The reviewed commit is recorded
The build starts from a clean source state
The generated digest is recorded
The deployment references that digest
The runtime exposes the expected build identity
Signature verification succeeds
The previous known-good digest is retained
Rollback has a defined target

This invariant gives both the assessor and the operator a stable object to reason about.

Scoring deployment drift

Deployment drift is unusual because it may not directly create an exploit path.

Its severity depends on the decisions that rely on the affected evidence.

A minor internal tool with no sensitive data may justify a low or moderate rating. The main consequence may be operational inconvenience.

A multi-tenant platform preparing to host customer data has more at stake. If a source review is being used to justify production readiness but the source cannot be tied to production, the uncertainty affects every source-derived conclusion.

The rating should therefore consider:

Sensitivity of the hosted data
Degree of contradiction between source and runtime
Use of mutable artifacts
Ability to reproduce production
Ability to identify a known-good rollback target
Importance of the assessment to a launch or risk-acceptance decision
Availability of independent runtime testing

The finding should not claim that every source-level control is absent.

It should state that the organization cannot prove which controls are present in the deployed system.

Closing thought

Security reviews evaluate evidence, not repository aesthetics.

A clean codebase, strong authorization model, and carefully designed configuration are meaningful only when the running system can be tied back to them.

When that link is absent, the uncertainty should not be hidden in a footnote. It should be documented, assigned an owner, given closure criteria, and remediated like any other security-relevant weakness.