A Check That Reported Success Without Running

This host is due to get a minimal PHP CMS with its own auth. Hardening its config was the first part of that: before server-side code and credentials land on a host, what that host runs on has to be something a pipeline can check.

Twelve deliberately broken .htaccess files, run against the CI check meant to catch exactly that class of break: it caught six. One of the six it missed deleted the entire security-header block — HSTS, CSP, X-Frame-Options, all of it — and the check stayed green, because it was asking whether Apache started, not whether the response was correct. Fixing that took three tries, and two of the fixes introduced new defects of the same kind, one of them created by the recommendation the original exercise made. The check that ships now catches twelve of twelve, and still never looks at the server that answers visitors.

The check exists because of a production outage: an edit to .htaccess returned HTTP 500 on every URL. .htaccess fails differently from most config — one directive Apache won’t accept in that context, and it rejects the whole file, 500ing every request under that directory, with no reload step to catch it first. The directive was UseCanonicalName, which needs AllowOverride Options; the host grants FileInfo. The file had never been executed by any web server before production — next build doesn’t parse it, and the preview platform reads a separate _headers file and ignores Apache config entirely.

verify-htaccess.sh runs the built output under httpd:2.4 in CI and asserts the response — the natural next step is to write that check, run it once, and trust it. Instead it got mutation-tested: build fourteen variants of .htaccess, twelve of them real defects, and record what the check actually notices, rather than assume a check that compiles is a check that works.

The result the exercise was built to find: deleting the entire <IfModule mod_headers.c> block removes HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy, and the CSP in one edit. Apache is perfectly content with that, answers its normal redirect, and the check — which asserted a status code — passed. “Did the server start” was standing in for “is the site correct,” and that proxy holds for exactly one class of break: the one the outage happened to be. A second variant, an inverted HTTPS-redirect condition, passed the same way — it reads as a clean 200 in a plain-HTTP test container because no rule branching on %{HTTPS} being on can fire there.

The check’s own negative control had the same shape of problem. The script justified running under AllowOverride FileInfo by claiming a looser level would let a bad directive slip through undetected. Measured at every level, the directive it used for that control has no .htaccess context at all — it proves the file is being read, not that the level is right. The directives that actually distinguish an over-permissive level are Options-class ones, which is why a second control exists now.

The fix asserts the response instead of the status: six headers by name, HSTS and CSP by value, and the redirect target. Making that assertion mean anything took a single OVERRIDE variable feeding both the server and its controls, so relaxing one can’t leave the other quietly passing at the wrong level:

# What OVH grants. The assertions and the control that pins this level must read
# the SAME string, or the control pins nothing: with two independent literals,
# relaxing the assertions to All leaves the control still passing FileInfo, still
# returning 500, and still green — while an .htaccess that grew an Options line
# ships and breaks production.
OVERRIDE=FileInfo

Two more defects showed up building it. A missing docker binary in CI used to exit 0 — a runner that lost Docker would report success having tested nothing:

if ! command -v docker >/dev/null 2>&1; then
  # Skipping locally is a convenience. Skipping in CI would mean the one job that
  # executes this file silently stopped executing it.
  if [ -n "${CI:-}" ]; then
    echo "FAIL: docker unavailable in CI — this check cannot be skipped here"
    exit 1
  fi
  echo "SKIP: docker not available (this check runs in CI)"
  exit 0
fi

It now fails closed instead of skipping. And every header assertion ran against the 301 redirect, which a plain-HTTP container can reach, while real visitors receive their headers on the 200 that follows it — a third pass now serves a copy with the redirect stripped and checks the headers there, reporting “not checked” rather than passing vacuously if that fixture doesn’t return 200:

# 3. The 200 leg — real visitors get their headers here, not on the redirect.
NORD=$(mktemp -d)
cp -r out/. "$NORD"/
sed -i '/^RewriteCond/d; /^RewriteRule/d' "$NORD/.htaccess"
start_apache "$NORD"
code=$(status)
if [ "$code" = "200" ]; then
  note pass "serves the page itself with the redirect removed (200)"
  assert_headers "on the 200" "$(curl -s -D - -o /dev/null "http://localhost:$PORT/")"
else
  note FAIL "expected 200 with the redirect stripped, got $code — the 200-leg headers were NOT checked"
fi

Run for real in CI against the shipped build, that’s what actually passes — not a claim, the job’s own output:

pass  out/.htaccess: Apache parsed it and answered 301
pass  on the redirect: Strict-Transport-Security
pass  on the redirect: Content-Security-Policy-Report-Only
pass  HSTS max-age is 31536000
pass  CSP starts from default-src 'self'
pass  redirects to https://kmzawadzkie.pl/
pass  serves the page itself with the redirect removed (200)
pass  on the 200: Strict-Transport-Security
pass  negative control (file is read): UseCanonicalName On correctly produces 500
pass  negative control (override level): Options +Indexes correctly produces 500

HTACCESS CHECKS PASSED

One of those three was found by the mutation exercise itself, reading the script’s non-Apache paths. The other two were introduced implementing its own recommendations and only caught by review afterward — the exercise was never re-run against its own output. That’s the pattern, not an accident: a verification proves something about a snapshot, and the fix it prescribes is a new, unverified artifact. Mutation testing shows what a check catches across the inputs you thought to vary; it says nothing about whether the thing you varied is what the assertion actually reads.

Re-run against all fourteen variants, the fixed check alone now catches eleven of twelve. Paired with the separate build-output checks that already run in the same job, twelve of twelve. The one row the Apache check still misses on its own — a 404 page pointing at a missing file — goes unseen because every request the script makes is for /: the redirect-stripped copy in step 3 would reach the error handler, but nothing asks it for a missing path, so the build checks catch it instead, by static grep. One row still passes on purpose: an unterminated <IfModule> block that Apache 2.4.68 accepts silently, with every header confirmed present and nothing in the error log. It’s malformed, not defective, and staying green is the correct answer — the matrix keeps it in to separate checking for tidiness from checking for effect.

A thirteenth variant turned up after this was written. Changing the redirect from R=301 to R=302 leaves Location: byte-identical, so every response assertion passes and the check went green — a permanent redirect silently downgraded to temporary. The matrix varied the redirect’s target and its condition; it never varied its status. The check asserts the 301 now, and CI is red on expected 301, got 302 with pass redirects to https://kmzawadzkie.pl/ directly beneath it. Mutation testing shows what a check catches across the inputs you thought to vary — that sentence was in this document before the variant that proved it was.

What’s still true after all that: nothing automated ever looks at the actual production response. The checks run in a container and against the preview platform, which reads a different file entirely, so a header the live Apache config strips or rewrites would pass everywhere and fail only for visitors. The only evidence headers ever reached production at all is a one-time scan on securityheaders.com from an afternoon in August — evidence of a moment, not a check. GitHub itself can’t refuse a merge on red, either — required status checks need a paid plan or a public repo, and this is neither. What actually blocks a bad merge is a standing rule: a red run doesn’t get merged. That rule has held. But it’s enforced by remembering to look, not by a mechanism, so nothing records that it held, and nothing would catch the one time it’s forgotten, or the first time a second committer merges without knowing the convention exists. Nor is the merge the last gate: the release workflow force-pushes dev to prod and triggers the deploy, which consults no check at all. The rule guards the merge; nothing guards the release. The %{HTTPS} on branch is untested by construction, since the container only speaks plain HTTP. And header values beyond two spot checks — HSTS’s includeSubDomains, preload, everything in the CSP after its first directive — could change freely with nothing to notice.

Twelve of twelve detected is a real, mutation-tested number. “We don’t merge red” is also real, and has held so far — but it’s exactly the kind of claim this document spent its whole length refusing to take on faith from a check. Nobody has mutation-tested the rule itself: there’s no record of every red run being caught, and no test of what happens the one time it isn’t. A check that reports correctly and is held to by habit isn’t the same failure as a check that lies about passing — but it rests on the same kind of unverified trust this whole exercise was built to replace.