Running it

The three processes Félagi needs besides the web server, and what stops working without each.

The short version

php artisan serve                                    # or nginx, php-fpm
php artisan queue:listen --queue=broadcasts,mail,webhooks,imports,default
php artisan schedule:work                            # or a cron entry
php artisan reverb:start                             # websockets

composer dev runs all of them for local work. In production they are four supervised processes, or three plus a real web server.

Every one of them fails silently. Nothing throws, no page breaks, no log fills up — a feature simply stops happening, and the first person to notice is the one who wondered why their autopilot never ran.

Recommended setup

The whole configuration in one place, for an installation served by the application server this product is deployed on. Every line has a reason, and each reason is a thing that has gone wrong.

The application — docker/.env

SESSION_DRIVER=database      # or redis, memcached, file — anything but the app server
CACHE_STORE=database         # same reason, sharper consequences

Both tables ship with the application, so the database driver needs nothing set up. Why these two matter is below: a store held inside the application server is emptied by every release.

ADMIN_TOKEN is optional here and is normally left out — the release finds it by itself. See The admin token.

The application server — its own .env

ASKR_QUEUE="broadcasts,mail,webhooks,imports,default"
ASKR_ADMIN_TOKEN=<a long random string>
ASKR_APP_BASE=/var/www/your-site

broadcasts first, and present at all. The lanes are drained left to right, so a live update queued behind a batch import arrives minutes late and is worthless by then. Leaving it out of the list altogether is worse and quieter: the lane fills, nothing consumes it, and the only symptom is that boards stop moving on their own — a reload still shows the truth, so it does not read as a fault. Nobody reports it.

Check that all five are being drained rather than assuming it:

curl -sS -H "Authorization: Bearer $ASKR_ADMIN_TOKEN" \
    http://127.0.0.1:9000/api/status |
  python3 -c 'import json,sys
d = json.load(sys.stdin)
print("queue workers:", d["queue_workers"])
for q in d["queues"]:
    print("%(queue)s: %(pending)s pending, oldest %(oldest_pending_secs)ss" % q)'

A lane with pending jobs, no reserved jobs and an oldest that keeps growing is a lane nothing is listening to. That is not the same as a busy queue, and it is the failure worth watching for.

What each setting costs when it is wrong

The admin token

The application server's admin API — worker counts, memory, queue depth — requires a bearer token. It is ASKR_ADMIN_TOKEN in the application server's own .env, which is the file that owns it.

A release reads it from there. It knows where that directory is, because it asks Docker which compose project is serving the site, so nothing has to be copied and there is no second place to keep in step. Two copies of a credential means one of them goes stale, and it is always the copy nobody looks at.

Setting ADMIN_TOKEN in the application's own docker/.env overrides that, which is how a release run from somewhere that cannot read the server's directory can still be handed the token. It is not the normal case.

What it unlocks

The queue check before a release. The queue lives inside the application server, so recreating that process destroys anything still waiting:

▸ Preflight
    broadcasts: 14 pending, 0 reserved
  queued jobs will be LOST by the recreate below — wait for them, or accept it
  continue? [y/N]

Sessions and the response cache moved to the database and survive a release. The queue did not.

With no token, that check cannot run, and a release says so rather than skipping in silence:

▸ Preflight
  ! ADMIN_TOKEN is not set in .env, so the queue was not checked.
    The restart below destroys queued jobs: queue.default is 'askr', which keeps them in
    the process being replaced. Set ADMIN_TOKEN to have this checked.

That warning is the whole reason this is written down. The check had been skipped on every release for weeks — the token exists, and the release was looking for it under a different name in a different file — and a skipped check that prints nothing leaves a release looking identical to one that verified something.

What it does not unlock

Proving the release replaced the running process. That reads the container from Docker and needs no credential — see What a release verifies about itself. It used to read worker pids from the admin API, and when the token arrived it began answering 401 and the check went quiet. A verification step should not depend on a service being reachable and authenticated when the tool doing the work is already at hand.

/healthz on the same port stays open and should: it carries no information, and a release has to wait on it before any credential is in scope.

Sessions have to outlive a deploy

Deploying recreates the application server, so anything held inside that process is gone when it comes back. A session store there signs everybody out on every release; with releases a few days apart, that is most of the times anybody signs in at all.

SESSION_DRIVER=database      # or redis, memcached, file
CACHE_STORE=database

Both tables ship with the application, so the database driver needs nothing set up.

The cache matters for a sharper reason than speed. Inbound email and webhook deliveries write a short cache entry to record that they have already been handled. A store emptied by a deploy forgets those, and a retry arriving afterwards is processed a second time — a duplicate comment rather than a slower page.

felagi:check warns about either driver, because this is a thing only the running installation knows: the test suite cannot see a server's configuration, and the symptom reads as an application bug rather than as a setting.

It is the store, not the key. APP_KEY lives in a .env the deploy never touches, so the session cookie decrypts fine after an update — it points at a session the server no longer has. Worth knowing because a changing key is the other explanation for the same symptom, and it is the one that also breaks "remember me".

SESSION_LIFETIME is a separate question from this one: at 120 minutes people re-authenticate on a two-hour idle timer whatever the driver does.

A release compiles without deleting

php artisan view:cache clears before it compiles, and its clear removes every entry under the compiled path, directories included — including the one holding the compiled single-file components, which it does not rebuild.

The workers are serving throughout, so for as long as the compilation runs there are requests finding files that were there a moment ago. Three of them got an error page during the 0.70.16 release.

Recompiling the components afterwards was the first repair, and it could not work. The deletion is at the start of the caching, so the gap is the whole duration of it — measured at 2.1 seconds for 754 templates here. It narrowed the window from the restart to the compile and left it open.

So a release does not call view:cache. It calls:

php artisan felagi:compile-views        # every Blade template, deleting nothing
php artisan felagi:compile-components   # the single-file components, which Blade does not reach

Nothing is ever absent, so there is no window to be caught in.

Why dropping the clear is safe. It exists to sweep compiled files whose template is gone. That is housekeeping rather than correctness: a compiled name is derived from the template's path, Blade recompiles whenever the source is newer, and an orphan costs a few kilobytes of disk. php artisan view:clear is still there for whenever that is worth doing — at a moment somebody chose, rather than in the middle of every release.

Tip: A test compares the set of files felagi:compile-views produces against the set view:cache produces. The one thing it must not do is compile a different set, which would trade a visible fault — a few error pages during a release — for an invisible one, where some template is quietly never compiled.

Tip: The components need their own step because they are compiled by Livewire rather than by Blade and do not live under the view paths. Their names are md5 of the source path, not of its contents, so compiling again produces exactly the filenames a running worker is still asking for.

The same shape, already handled: a worker holding a Vite manifest from before the build serves a page referencing an asset that 404s. That is why the build runs before the restart and why a release asks for the stylesheet ten times cold afterwards.

What a release verifies about itself

The last step compares the serving container before and after the restart — its id and its process — and fails if they are the same. Same container means the files are new and the process serving them is not.

It used to read worker pids from Askr's admin API instead. That API is authenticated now, so it began answering 401, and the step printed "could not read worker pids, so the restart was not verified" and carried on. On every release for weeks: still running, still passing, checking nothing.

A verification step that degrades to a warning is worse than one that fails, because nobody reads the seventh warning. Docker is the tool the release already uses to do the restart, so asking docker closes the loop with no second service to be up and no credential to hold — and it fails rather than warns when it cannot answer.

The queue, and what a release cannot promise about it

Two checks read the application server's admin API. It is authenticated, and the release finds the token by itself — see The admin token.

The queue is the one thing a release destroys:

QUEUE_CONNECTION=askr

Sessions and the response cache moved to the database and survive a release. Queued jobs live in the process the release replaces, so anything still waiting is gone. A release lists what is pending and asks, and with no token it says that it could not look.

The scheduler

Without it, four things stop and nothing says so — and one thing grows without limit.

One process:

php artisan schedule:work

Or one crontab line, which is what a server usually wants:

* * * * * cd /path/to/felagi && php artisan schedule:run >> /dev/null 2>&1

felagi:reap runs twice a minute, which a plain schedule:run cannot do on its own — Laravel handles the sub-minute frequency internally, so the crontab entry above is still correct.

They run in-process, and that was not always true

Until 0.41.1 every one of these was scheduled by shelling out to a php command. On a server where PHP is compiled into the application server — Askr, for one — there is no php on the path, so each invocation exited immediately with "command not found" and all seven had never run in production. The failure was reported nowhere anybody looks.

They are dispatched in the application's own process now. Nothing about the schedule changed; what changed is that it happens.

Checking

php artisan schedule:list

Every command above should be listed with its next due time. If the list is empty, routes/console.php is not being loaded.

Listed is not the same as running. schedule:list describes intent; it says nothing about whether anything is executing it. Nothing in Félagi warns you when the scheduler has stopped — the check that would have noticed is felagi:check, which is itself one of the seven. So verify it from the outside:

# a cycle that is being rolled has a reading for today
php artisan tinker --execute 'echo \App\Models\CycleReading::whereDate("on", today())->count();'

Zero on a workspace with an open cycle means the scheduler is not running, whatever schedule:list says. Worth a line in whatever already watches your other processes.

The queue worker

Five lanes, and the order matters.

php artisan queue:listen --queue=broadcasts,mail,webhooks,imports,default

Listing them in that order is not decoration: a live update is worthless three minutes behind a batch import, and a worker drains lanes left to right.

The whiteboard is the exception. Its changes are sent immediately rather than queued, so a board keeps working with no worker at all — see Broadcasting for why that one is different.

Checking

The dashboard reads the backlog and warns when nothing is draining it. From a shell:

php artisan queue:monitor broadcasts,mail,webhooks,imports,default

Reverb

php artisan reverb:start

Without it: live updates and whiteboard cursors stop. Everything else is unaffected, because a failed broadcast is never allowed to fail the write that caused it — see App\Support\Announce.

The whiteboard needs accept_client_events_from on the Reverb app, which is members by default. Cursors are whispered between browsers and never reach the application.

Commands you run by hand

Everything else is scheduled.

PHP extensions the runtime needs

Composer resolves these at build time. What matters is the PHP that serves the page, which on a containerised deploy is not always the PHP that ran composer install — a difference that produced a 500 on every real two-factor enrolment while every test passed.

• ext-iconv — draws the two-factor QR code. Without it enrolment still works (the key is offered to type) but the square cannot be rendered. See two-factor.
• ext-gd or ext-imagick — image thumbnails on attachments. A thumbnail of a phone photo needs more than a stock memory_limit of 128 MB — GD holds a twelve-megapixel image at four bytes a pixel, twice over while it rotates — so the pipeline raises the limit for that request only, as far as the arithmetic says and never past 512 MB. A host that forbids ini_set('memory_limit') gets the original served instead of a thumbnail, which is the same answer every other refusal here gives. Nothing to configure; written down because running out is a fatal error rather than an exception, and the symptom is a broken image with nothing in the log.

php -m on the serving host is the check that counts.

What to watch

There is no metrics endpoint and no health check. What exists:

• The dashboard warns about a stalled queue and about runtimes that have gone quiet.
• storage/logs/laravel.log has every webhook attempt, every failed delivery and every reaped task.
• /admin/runtimes shows last-heartbeat times, which is the fastest way to tell whether the daemon side is alive.

Silent interactions

Administration → Silent interactions.

Every observability tool answers what crashed. This one answers the other question: what did not happen.

An interaction is recorded as silent when all five of these are true at once:

1. Nothing was written — no insert, update or delete
2. No error was shown
3. There was no redirect
4. Nothing was dispatched — no toast, no modal, no event to another component
5. No component property moved

Any four of those are ordinary. All five together mean somebody pressed a button and the application, as far as they could tell, ignored them.

It counts rather than alerts. Plenty of actions legitimately do nothing once — a filter set to what it already was, a guard that finds nothing to do. The same method silent four hundred times is a fault nobody has reported, because it does not look like one.

Having fixed something, forget it and watch whether it returns. A count still carrying yesterday's occurrences can never show that it stopped.

Why this exists

Nine faults reached production in this project before it was built, and every one was silent: no exception, no log line, no failing test. A title that would not save because the validation error had nowhere to be drawn. Three rate fields that had never once saved and a settings page that looked filled in. Forms that rendered perfectly and did nothing.

None of those would have appeared in any log. All of them would have appeared here.

Not there yet

• Only /up to point a monitor at. It answers 200 when the application boots, and nothing more — not whether the worker is draining or the scheduler ran.
• No metrics. No Prometheus, no queue depth over time.
• No supervisor or systemd units shipped. The commands above are the contract; how they stay running is yours.