Askr 1.2 — learning to say no

Rate limiting, canaries that stop themselves, a cache that survives reboots, and a command that reads your app.

Askr 1.1 was about making things fast: ESI, PURGE/BAN, cache rules. Fun to build, immediately visible in a benchmark. This release is the other kind of work — the kind you only appreciate at 3am. Nearly every feature in 1.2 is about refusing something: refusing abusive traffic, refusing to finish a bad deploy, refusing to restore a cache that belongs to different code.

Saying no well turns out to be harder than saying yes.

The one client who ruins your afternoon

Askr had slowloris timeouts, body limits, a sandbox, and admin-token auth. What it didn't have was anything stopping a single client from hammering your most expensive route until the whole worker pool was busy serving them.

[server]
trusted_proxies = ["10.0.0.0/8"]
[[ratelimit]]
path = "/login"
limit = 5
window = 300           # 5 attempts per 5 minutes, per IP
[[ratelimit]]
path = "/api/*"
limit = 60
window = 60
by = "header:X-Api-Key"
burst = 20

The interesting part isn't the config, it's where it runs. The token buckets live in shared memory mapped before the fork, so the limit spans the whole worker fleet rather than each process keeping its own private count. With four workers and a limit of three:

200 200 200 429 429

Three, not twelve. That's the thing FPM + nginx can't do without an external store — and it's the same shape of win as replacing Redis: the data structure is already inside the process tree, so no network hop is needed to agree on a number.

A refused request costs no PHP cycle, no cache lookup, not even a disk stat. It's turned away in the same layer that serves cache hits.

The part that makes it real security instead of theatre

Rate limiting by IP is worthless if the client can choose their own IP. So X-Forwarded-For is believed only when the peer is in trusted_proxies, and then the rightmost hop that isn't itself a proxy wins. I tested both directions, because a security control you haven't tried to bypass is a guess:

trusted_proxies     six rotating fake X-Forwarded-For values


["127.0.0.1/32"]    200 ×6 — each address gets its own budget
(correct: we told Askr this really is a proxy)
[]                  200 200 200 429 429 429 — doesn't get past the limit

And if you configure limits without trusted proxies, Askr warns at startup — because behind a load balancer, every visitor would otherwise share a single bucket and you'd rate-limit your entire site down to five requests.

One deliberate choice worth stating: when the bucket table fills up, the limiter fails open. A client may get a fresh allowance rather than being wrongly refused. Turning away legitimate traffic to save a few kilobytes is the worse failure for a web server, and I'd rather write that down than have you discover it.

I was wrong in my own issue tracker

Here's the honest bit of this release.

I filed an issue saying canary reload existed but "the decision is manual", and planned to build automatic abort. Then I opened supervisor.rs to start — a habit from four review passes where verifying findings against the source repeatedly saved me from fixing non-bugs — and found this:

if alive && new_errors <= CANARY_ERR_THRESHOLD { roll_next() }
else { error!("canary UNHEALTHY — aborting reload; remaining workers keep old code") }

Auto-abort had existed since 0.2.0. My own issue was wrong about my own code.

So the work changed from "build a feature" to "fix three real weaknesses in one", which is usually more valuable anyway:

The error count was fleet-wide. It summed 5xx across all workers, so errors from the old workers counted against the canary. On any site with a normal error baseline, more than three 5xx in the five-second window would abort every single reload — while a canary that served no traffic at all sailed through. Two opposite failures in one mechanism, and you'd only discover either at the exact moment you were relying on it.

Askr now keeps per-worker counters in shared memory (the child records its slot right after fork) and compares the canary against the fleet over the same window, as a rate:

ERROR canary UNHEALTHY — aborting reload
reason=error rate 63.35% vs fleet 0.00% (allowed +2.00 points)

That sentence was impossible to say before. The old counter couldn't distinguish a bad new worker from a site that always serves a few errors.

There was no minimum. A quiet site's canary saw zero requests and passed — a health check that rubber-stamps. Now, below canary_min_requests the verdict is inconclusive, the rollout continues, and it says so loudly. A deploy shouldn't be blocked by an absence of evidence, but a silent pass looks exactly like a pass, and that's how you learn to trust something that never actually checked.

And then testing found the real hole. After a successful "abort" I measured the site:

healthy answers: 10/20

Half the requests still failed. Of course they did — the canary was still running. "The remaining workers keep old code" is worth very little when the failed deploy still answers one request in four. So a failed canary is now drained, and its slot quarantined (respawning it would just boot the same bad build). After the fix:

30/30 healthy, workers_alive: 3

You run one worker short on known-good code until you fix the deploy and reload, which clears the quarantine and refills the slot. Never below one worker, though — an empty fleet is worse than a bad one.

The nuance I had to write down rather than hide

In worker mode, the surviving workers hold the previous application in memory, so an abort genuinely keeps old code serving. In per-request mode, every worker reads the current files from disk — so a bad deploy affects all of them regardless. The gate still detects it and drains the canary, but it cannot roll you back to code that's no longer on disk.

That's in DEPLOYMENT.md now, along with the advice to deploy atomically. It would have been easy to leave the old, flattering log message in place.

Keeping the cache instead of guessing at it

The original idea here was predictive cache warming: after a deploy, replay your top-N URLs from the access log so nobody pays for the cold cache.

The more I looked at it, the less I liked it. It needs per-URL frequency data Askr doesn't collect. It needs synthetic requests that guess every key variant — Host, encoding, device class. And it risks a warm-up storm competing with real traffic right after a deploy, which is precisely when your system is most fragile.

Meanwhile the response cache is a flat array with everything stored inline. No pointers. You can just… write it to a file.

[cache]
persist = "/var/lib/askr/rcache.bin"
persist_key = "git-sha"    # optional, but this is the reliable signal

On graceful shutdown — after every worker is reaped, so no slot can be captured mid-lock — the region goes to disk, and comes back at boot. Verified through a full container stop/start:

before restart:  body=a28145
after restart:   body=a28145   ← identical, x-askr-cache: HIT

The subtle part, which I flagged in the issue before writing any code and then confirmed: tag generations live in a separate region, and each entry stores a snapshot of them. Dump only the entries and every restored page looks tag-invalidated. Both regions are saved, and askr_cache_forget_tag('posts') still busts a restored entry — tested, because "probably fine" isn't a test.

The guards matter more than the feature, since a wrong cache is worse than a cold one. The dump is refused unless the build, entry layout and cache size all match. It's refused when the application changed — appending a single line to index.php was enough to drop it in testing. Expired entries are removed on load, slot locks are zeroed so a boot can't inherit a held lock, and it's written via temp-file-and-rename so a crash mid-write can't leave a torn file. After a crash there's no dump at all, which is the right default.

A command that reads your app

The hard part of adopting Askr was never running it. It's knowing what to put in askr.toml when you've spent a decade with FPM's defaults.

askr tune --root public --requests 30

PHP boot              182.4 ms
Request (mean)        24.9 ms wall, 0.1 ms CPU
RSS after warm-up     94 MB

Suggested askr.toml:
[server]
workers = 64          # only 1% CPU-bound (waits on I/O) ⇒ more workers than cores
max_rss_mb = 220      # 2× observed peak; memory grew 0.31 MB/request

The measurement that earns its keep is wall time versus CPU time. That single ratio decides whether more workers than cores will help you: an app that computes is best served by one worker per core, while an app that spends its life waiting on a database can keep several times that many busy. Both paths are verified — a sqrt/usort page measures 100 % CPU-bound and gets one per core; a usleep() page measures 1 % and gets four times cores.

There's no HTTP load generator in here, on purpose. Askr's own benchmarks showed PHP is ~99.5 % of request time and I/O ~0.5 % — the same data that de-prioritised the io_uring work. If the interpreter is the whole story, measure the interpreter. That reasoning is printed in the output, not buried in the docs.

And the output ends by telling you what it didn't cover: one route, no cookies, no concurrency, and "run this against a copy of production data, not an empty database". A confidently wrong max_rss_mb buys you a recycling storm at peak traffic. A tool whose entire job is giving advice has to be honest about the limits of its evidence.

Which is why one detail bothered me more than it probably should: the first version pointed you at a metric called askr_worker_rss. That metric doesn't exist. It now points at rss_kb_total, which does.

What testing found that reading didn't

Three times this session my own test harness lied to me, and each time the mistake looked exactly like a result.

kill %1 killed nothing, because every shell invocation is a new shell. cargo test doesn't rebuild the server binary, so I "proved" a fix didn't work using an eighteen-minute-old build. And a leftover process from an earlier test held the port, so a "restored cache" was really just the original server still running.

But the best one was this: while testing cache persistence, the master process refused to die. No shutdown, no dump. The cause was a bug I had introduced in the previous feature — the "refill empty worker slots" pass added for canary quarantine happily respawned workers that were draining, fighting the shutdown forever.

Unit tests would never have caught it. It required sending SIGTERM to a running server and watching what happened. Reading code proves what it says; running it proves what it does.

The freeze held, again

1.0 froze Askr's surface under SemVer. Three releases later, that promise has cost nothing: 1.2.0 adds four user-visible features and breaks nothing. Every addition is a new default-off config key or a new subcommand — askr tune, duly recorded in STABILITY.md.

docker pull ghcr.io/kwhorne/askr:1.2.0        # or :latest
composer require kwhorne/askr-laravel

1.1 made Askr fast to serve. 1.2 makes it safer to run — which is a less exciting sentence, and a better one to read at 3am. 🌳