Askr 1.5.0: Failing Safely

There's a particular kind of bug I've come to dread more than crashes. A crash tells you something is wrong. This other kind logs a warning, carries on, and looks — from the outside, from your dashboard, from your uptime monitor — exactly like everything working.

Askr 1.5.0 is mostly about that kind. I didn't set out to write a release with a theme. It emerged from reading the code adversarially for a few days, and noticing how many findings had the same shape.

The bug that named the release

Start with the worst one, because it's also the clearest.

Askr has a shared-memory cache — a fixed-slot hash table in an anonymous mmap that every worker process sees. It's what lets you run Laravel sessions, cache, locks and counters without Redis. Writing to a slot takes a per-slot spinlock. Reading did too.

Two workers calling set() on the same key can pick different slots. The probe loop holds one slot lock at a time, so one worker can find a slot empty that the other has since filled, or the two can disagree about which entry is oldest because written_at moved underneath them. Fine — last writer wins, no corruption.

Except delete() looked like this:

if Self::matches(e, key, h) {
    ptr::write(ptr::addr_of_mut!((*e).state), 2);  // tombstone
    return true;                                    // ← and stop
}

It stopped at the first match. So it tombstoned one copy and left the other alive. And get() scans past tombstones by design, so the next lookup found the survivor.

For a cache entry that's a stale value. For a session key, that's a user who logged out and is logged back in.

What makes this my favourite finding is the comment that was sitting at the end of set():

// Re-validate under the lock: if a racing writer already put *our* key
// somewhere we'd now clobber, last-writer-wins is still correct; we just
// write our value. (Closes the evict/target race window.)

The comment describes exactly the fix that was missing. The code below it did no re-validation at all. Somebody — me, at some point — reasoned correctly about the race and then wrote something else.

The fix: delete() clears every match in the chain, and set() sweeps afterwards so duplicates converge instead of accumulating. The race isn't reproducible on demand, so the regression test plants the duplicate directly and asserts the consequence:

assert!(delete(key), "delete must report that it removed something");
assert_eq!(get(key), None, "a deleted session that reappears is the bug");

I ran that test against the old delete() before shipping it. It fails. A test that hasn't failed once is a decoration.

Warnings are not controls

Once you're looking for "logs and carries on", it's everywhere.

The sandbox. --sandbox installs a seccomp filter and, with write paths configured, a Landlock filesystem restriction. When a kernel is missing a feature:

Err(e) => tracing::warn!(error = %e, "seccomp: not applied"),

…and the worker serves traffic looking identical to one that hardened successfully. Nothing downstream can tell.

That default is staying — an upgrade that suddenly refuses to boot is worse than a warning — but you can now opt out of it:

[server]
sandbox = true
sandbox_write = ["/var/www/app/storage", "/tmp"]
sandbox_required = true    # refuse to serve unhardened

A worker that can't fully harden exits 78 instead of serving, and the crash-loop guard turns a fleet-wide failure into one clear "giving up".

There's a detail here I like. sandbox_required refuses to start without sandbox_write, and that's not pedantry. Seccomp blocks execve — but Askr interprets PHP in-process. A .php file written into your docroot needs no process creation at all. Seccomp-only "hardening" doesn't stop the thing most people are actually afraid of, so a required sandbox without Landlock write rules would be a promise the sandbox can't keep.

The canary. Askr can roll one worker first and compare it against the rest of the fleet before rolling the others. SIGHUP went through that gate. trigger_reload() — which is the admin API, an ACME renewal, and the cert-watcher — called roll_next() directly:

pub fn trigger_reload() {
    RELOAD_CURSOR.store(0, Ordering::SeqCst);
    roll_next();               // no gate, no health check, whole fleet
}

[reload] canary = true guarded the deploy you were watching and nothing else. The two reloads that happen with nobody watching were the two that skipped it.

The crash-loop guard. It counted a worker that exited non-zero shortly after boot. A worker killed by SIGSEGV fell into the healthy branch — and cleared the streak counter. So a segfaulting worker respawned forever, and a fleet mixing fatals and faults never accumulated a streak at all. The guard existed to stop exactly the loudest version of what it was ignoring.

Loopback is not a person

PURGE and BAN invalidate the response cache over HTTP. With no ASKR_ADMIN_TOKEN set, they were accepted from loopback peers — a sound rule for a server that's its own front door.

Behind nginx or Caddy on 127.0.0.1, every request is a loopback peer.

# from anywhere on the internet, through your proxy:
curl -X BAN -H 'X-Ban-Url: /*' https://example.com/

Now trusted_proxies — the operator stating in writing that loopback is where the proxy sits — makes the token mandatory for those methods. If you have trusted_proxies set and no token, cache invalidation will start answering 403 after you upgrade. That's the fix working; set the token.

The admin plane had two more of these. It never looked at Host, so a page on an attacker's domain could re-resolve its own hostname to 127.0.0.1 and read /api/status — PIDs, memory, error records — as same-origin. Nothing about that request looks cross-site, because to the browser it isn't. The only thing that gives it away is Host: evil.test naming a listener that isn't called that.

And POST /api/reload is a CORS "simple request": no custom headers, so no preflight, so CORS never got a say and any web page could roll your fleet. Both checks now apply whether or not a token is configured, because both attacks work fine against a plane that never had one.

Measure first, and let the benchmark surprise you

The one feature in this release started as a suspicion: get() took the slot lock exclusively, so every worker reading the same hot key — a session, a shared config blob, anything Laravel touches per request — queued behind the others.

Before touching the lock, I wrote a benchmark. Both read paths, measured in the same process, one hot key:

one hot key, 512 byte value:
  threads        locked      lock-free      speedup
        1      22085798       22860599         1.0x
        4       6018470       75000259        12.5x
       12       1547473       19119338        12.4x

The locked column is the finding. Throughput doesn't plateau as readers arrive — it falls, 22M/s to 1.5M/s, because a spinlock under contention burns cycles instead of waiting. Adding workers made reads slower in aggregate.

Reads are now lock-free, via a seqlock: sample a per-slot version counter, copy, sample again. A change means a writer overlapped the copy, so retry; after a bounded number of attempts, take the lock. Writes are unchanged.

Two things I'd have missed without measuring first.

The benchmark caught a regression it wasn't looking for. My first version allocated with vec![0u8; vlen], which zeroes the buffer before the copy. A 4 KB value read by a single thread came out at 0.8× of the locked path it replaced — purely from the extra pass over the page. I'd have shipped a slower cache for large values and called it an optimisation.

And the counter can't just be incremented. A writer killed mid-update leaves it odd forever; an increment would then make it even during the next write, and a reader would sample a stable-looking counter in the middle of a copy. So Writing::begin forces it odd and Drop forces it even-and-greater, which repairs the slot instead.

How do I know the seqlock works? I broke it and watched the test catch it. With the two counter samples removed: 584,533 of 2,145,847 reads came back half-written. With them, zero.

Two fixes that only make sense together

A panic unwinding out of an extern "C" function aborts the process. Askr had 35 of them and no catch_unwind anywhere, so a panic at the FFI boundary killed the worker mid-request. Each now runs inside a guard that answers the caller's failure value — a cache miss, a refused push, a 502 — and logs the entry point by name.

But: catching a panic mid-write() would have run Writing's destructor during unwinding, marking a half-updated slot as stable to the brand-new lock-free readers. The panic guard would have introduced torn reads.

fn drop(&mut self) {
    if std::thread::panicking() {
        return;   // leave it odd; readers take the lock, next writer repairs
    }
    unsafe { (*self.0).version.store(self.1, Ordering::Release) };
}

Neither fix is wrong alone. Together, in the wrong order, they'd have been a bug. That interaction is only visible if you're holding both.

"Verified self-update" now means something

askr upgrade runs as root and replaces the binary systemd starts. It fetched a tarball and its .sha256 — from the same release.

A checksum that travels with the file it describes proves the download arrived intact and nothing about who produced it. A compromised release, account or CI token serves a matching pair. That was the entire trust chain.

Releases are now signed with minisign, and upgrade verifies the signature against a public key compiled into the binary — streamed, so a 30 MB tarball is never held in memory to check it. A bad signature, a signature from another key, or no signature at all is a refusal.

$ askr upgrade
askr 1.4.14 → 1.5.0
↓ downloading askr-1.5.0-linux-x86_64.tar.gz …
· verifying sha256 …
· verifying signature …
✓ signed by the key this build trusts
· extracting …

Verify it yourself, if you'd rather not trust the updater:

minisign -V -p keys/release.pub -m askr-1.5.0-linux-x86_64.tar.gz
gh attestation verify askr-1.5.0-linux-x86_64.tar.gz --repo kwhorne/askr

The key lives in the repository and is include_str!'d into the binary, which is the point: changing what an install trusts means changing the source and getting it built and released. I don't hold that key and neither does any tool — it was generated by its owner, on their machine, and only its public half exists anywhere I can see.

Oh, and one more: this release was blocked for an afternoon because cargo audit was red. h2 0.4.15, RUSTSEC-2026-0258 — a peer can hold an HTTP/2 connection open with unbounded empty DATA frames. A denial of service against the transport Askr serves on by default. Now on 0.4.19. That one was found by asking why a red CI badge was red, which is a habit worth more than most tooling.

What isn't fixed

Six things are still open, and I'd rather name them than let you find them.

The queue's delete/release are unfenced against an expired lease — a delayed worker can ack a job another worker is running. The fix needs a lease token threaded through shim.c, the Rust bridge and the Laravel driver, and I'm not burying a three-layer signature change in a batch of small fixes.

The response cache refuses what it can't vary on rather than varying on it. If your app sends Vary: Accept-Language, that response is no longer cached at all. Doing it properly needs a two-level lookup in the cache, because the primary key can't be computed from the request alone once the response gets a say. Correctness costs hit rate on exactly the responses that were being served wrong.

And the Landlock ABI is still pinned to V1, below what current kernels offer. I could have written the negotiation code. I couldn't compile it — Landlock is a Linux-only dependency and the C shim needs a Linux cc to cross-check — and guessing at a crate API in a security path is how you ship a build break. It's waiting on a Linux build in the loop, and that's a better reason than a shipped guess.

Upgrading

Nothing is required. Two things are worth adopting deliberately, because upgrading alone won't turn them on: sandbox_required, and ASKR_ADMIN_TOKEN if you sit behind a local reverse proxy.

Three behaviour changes aren't configurable: responses carrying their own Vary aren't cached, scheme is part of the cache key (expect a one-time hit-rate dip if force_https is off), and underscored header names are dropped — X_Forwarded_For no longer becomes HTTP_X_FORWARDED_FOR in $_SERVER, because it collided with the dashed spelling and bypassed anything filtering it. Same default nginx ships.

130 unit tests, 21 end-to-end, cargo audit clean, and the full details in the changelog.

Thanks for reading. Go check whether your warnings are controls.