Sigstore and Cosign: Verifying Container Images Before They Run

A container tag is a lie you’ve agreed to believe. nginx:latest today and nginx:latest next week can be entirely different bytes, and a tag tells you nothing about who built the image or whether it’s been swapped underneath you. The whole modern supply-chain panic — compromised build pipelines, typosquatted images, dependency confusion — comes down to that one weak link: we run images we can’t actually verify. Sigstore, and its CLI cosign, is the most practical fix I’ve adopted, mostly because it finally killed the part of signing that everyone hated: key management.

I put off image signing for years, and the reason was always the same. Every guide started with “generate a signing key”, and I knew exactly where that road led — a private key sitting in a CI secret somewhere, never rotated, forgotten until the day it leaked. The tool wasn’t the barrier. The lifetime custody of a secret was. Sigstore’s entire pitch is that it removes that secret, and once I tried it on a genuinely throwaway image I never went back to trusting bare tags.

Keyless signing, and why it’s the headline feature

Traditional code signing means generating a long-lived private key, guarding it forever, rotating it, and panicking if it leaks. Almost nobody does this well. Sigstore’s clever move is keyless signing: instead of a key you hold, you authenticate to an OIDC identity provider (your Google, GitHub, or corporate account), and Sigstore’s certificate authority, Fulcio, issues a short-lived certificate — valid for roughly ten minutes — bound to that identity. You sign with it, then it expires. There’s no key to steal because there’s no persistent key.

The word “keyless” is a slight fib, and it’s worth being precise about why. Cosign does generate a key pair — it just does so ephemerally, in memory, for the duration of one signing operation. The private half never touches disk and is discarded the moment the certificate expires. What you’re trusting instead of a stored secret is the binding between your OIDC identity and the certificate Fulcio issued, recorded permanently so it can be checked later. The threat model shifts from “protect this file forever” to “protect this login” — and you already have to protect that login for everything else you do.

The signature, certificate, and a record of the event are published to Rekor, a public append-only transparency log. That log is the trust anchor: anyone can later prove the signature existed and hasn’t been backdated. Because Rekor is append-only and independently monitored, a tampered or forged entry is detectable after the fact — the same property that makes Certificate Transparency work for the web PKI.

1
2
3
4
5

# sign an image keyless — a browser opens for OIDC auth
$ cosign sign registry.example.net/api@sha256:5d9f...
Generating ephemeral keys...
Retrieving signed certificate from Fulcio...
tlog entry created with index: 84213907

Note I signed the digest, not the tag. Always sign and verify by digest — signing a tag is signing a moving target. Cosign will actually warn you if you hand it a tag, because the tag can be repointed at different bytes tomorrow while the digest is the content-addressed truth. In CI, resolve the tag to a digest first (crane digest or docker buildx output) and sign that.

For a CI pipeline where no human is present to click through a browser, the identity comes from the workflow itself. GitHub Actions, GitLab, and most runners expose an OIDC token that Fulcio accepts, so the certificate is bound to the workflow’s identity rather than a person’s Google account. That’s the setup you actually want in production: the thing that signs is the pipeline, and the certificate says so.

Verifying before you trust

Verification is where the value lands. You assert who you expect to have signed it and which OIDC issuer vouched for that identity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ cosign verify \
    --certificate-identity '[email protected]' \
    --certificate-oidc-issuer 'https://accounts.google.com' \
    registry.example.net/api@sha256:5d9f... | jq '.[0].optional'

Verification for registry.example.net/api@sha256:5d9f... --
The following checks were performed:
  - The cosign claims were validated
  - Existence of the entry in the transparency log was verified
  - The signing certificate identity matches the expected one

Those two --certificate-* flags are not optional in spirit. If you skip them, cosign will refuse the verification outright in current versions — one of --certificate-identity or --certificate-identity-regexp must be set, and one of --certificate-oidc-issuer or the regexp form. This is a deliberate change of heart from early cosign, which would happily confirm that somebody signed the image. That answer is nearly useless: an attacker who can sign their own image with their own Google account also passes a check that doesn’t pin identity. The security comes entirely from pinning the expected identity and issuer.

For CI-signed images the identity is a URL, not an email, and it’s long. Rather than paste the exact workflow ref, the regexp flags earn their keep:

1
2
3
4

$ cosign verify \
    --certificate-identity-regexp '^https://github.com/myorg/.+' \
    --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
    registry.example.net/api@sha256:5d9f...

That says “any workflow in myorg, signed via GitHub’s OIDC issuer”. Tighten the regexp as far as your naming allows — pinning to a specific repository and branch is stronger than pinning to the whole org.

Going further: SBOMs and attestations

A signature says “this is the image I built”. An attestation says something about the image — most usefully an SBOM (software bill of materials) listing every package inside, so you can answer “is the new critical CVE in any image I’m running?” without guessing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# attach an SBOM as a signed attestation
$ cosign attest --type cyclonedx \
    --predicate sbom.cdx.json \
    registry.example.net/api@sha256:5d9f...

# later, verify and pull the SBOM back out
$ cosign verify-attestation --type cyclonedx \
    --certificate-identity '[email protected]' \
    --certificate-oidc-issuer 'https://accounts.google.com' \
    registry.example.net/api@sha256:5d9f... \
    | jq -r '.payload' | base64 -d | jq '.predicate.components | length'
142

Now a signature, an SBOM, and a transparency-log entry all travel with the image. That’s a real provenance trail rather than a hopeful tag. The SBOM itself you generate elsewhere — Syft is the usual choice, and it pairs naturally with a scanner. Once the bill of materials is signed and attached, the scanning step becomes trustworthy input rather than a guess about what’s inside; I lean on this directly in Trivy and container scanning, where the SBOM is what the vulnerability database gets matched against.

Attestations aren’t limited to SBOMs. SLSA provenance — a signed statement of how and where the image was built — is the other predicate worth attaching, because it’s the piece that defends against the compromised-pipeline attacks that signing alone doesn’t cover. A signature proves the image came from your CI; the provenance attestation proves what your CI actually did to produce it — which source commit, which builder, which parameters — so a tampered build step becomes detectable rather than invisible. This is the layer that turns “we sign our images” into a claim an auditor can actually check against the git history.

Enforcing it where it counts: admission control

Verifying by hand is good discipline; enforcing it in the cluster is what actually stops a bad image running. A policy controller — the Sigstore policy controller, or Kyverno’s image-verification rules — intercepts every pod and rejects images that don’t carry a valid signature from an approved identity.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-api-images
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "registry.example.net/*"
          attestors:
            - entries:
                - keyless:
                    issuer: "https://accounts.google.com"
                    subject: "[email protected]"

With Enforce, an unsigned or wrongly-signed image never schedules. That’s the line between “we sign things, mostly” and “unsigned images cannot run here”. Note the imageReferences glob scopes the rule to your registry — you almost never want to demand signatures from upstream public images you don’t control, because most of them aren’t signed the way your policy expects, or aren’t signed at all.

Troubleshooting: the failures you’ll actually hit

The first time you wire this up, expect friction. These are the errors I’ve hit repeatedly and what they actually mean.

no matching signatures on verify almost always means the identity or issuer doesn’t match — not that the image is unsigned. Run cosign verify without the identity flags first (in a lab only) to confirm a signature exists at all, then inspect the certificate’s subject and issuer to see what you should actually be pinning. Nine times out of ten the real subject is a slightly different string than you assumed — a CI ref with a branch suffix, or a Google account you didn’t expect.

Air-gapped or offline verification fails because cosign wants to reach Rekor and Fulcio’s roots. For disconnected environments you fetch the trust bundle ahead of time and point cosign at it, or run the check against a mirror. Plan for this early; discovering it at deploy time in a locked-down network is a bad afternoon.

Kyverno rejects everything, including images you signed. The usual cause is the policy reaching a registry it can’t authenticate to, or a clock skew problem where the ephemeral certificate’s validity window doesn’t line up. Check the Kyverno controller logs, not the pod events — the real error is upstream. And start every policy in Audit mode, watch the policy reports for a day, and only flip to Enforce once nothing legitimate is being flagged. Going straight to Enforce on a live cluster is how you page yourself at midnight.

A signed image is still vulnerable. This isn’t a bug, it’s the boundary of what signing does. Signatures verify provenance, not safety. Pinning versions and pruning stale images is a separate discipline — the housekeeping I cover in container image housekeeping matters just as much, because a signed latest is still a moving target you’ve merely proven the origin of.

The verdict

Worth it? If you pull images from anywhere you don’t fully control — and that’s nearly everyone — keyless signing removes the single biggest excuse for not signing, and verification by pinned identity genuinely raises the bar against supply-chain tampering. The honest caveats: keyless ties you to an OIDC provider and the public Rekor log, which some air-gapped or privacy-sensitive setups won’t accept (you can self-host both, at real operational cost). And signatures verify provenance, not safety — a signed image can still be vulnerable, which is why signing belongs alongside scanning, not instead of it.

This is for teams running their own registries and clusters who want to stop trusting tags on faith, and it pays off most when the signing happens in CI and the enforcement happens at admission — the two ends of the pipeline meeting in the middle. For a single hobby box pulling official images, it’s overkill; verifying the official signatures by hand is plenty, and you’ll get more security per hour spent on scanning and pinning than on standing up your own policy controller.