Grove 1.9: because it runs your databases, not just points at them

Most local development tools point at your database. You install MySQL somewhere, and the tool is told a host and a port. From then on the database is a stranger to it. The tool can't tell whether it's busy or idle, it doesn't know which branch its tables belong to, and it can't safely touch what's inside it.

Grove has always been different in one respect: it installs and runs your databases itself. That's why grove env can write your .env without asking, and why grove db snapshot needs no connection string. 1.9.0 is the release that takes that difference seriously. It has two new features that only a tool that runs the database could offer, and three fixes where Grove had been reporting success for things that hadn't quite happened.

A database that costs nothing while you're not using it

Here's a number that surprised us when we measured it. An idle MySQL, one that no request has touched all afternoon, was sitting at 517 MB of memory. That's for every day you had it installed and weren't using it, times however many database servers you run.

grove service on-demand mysql on

Now Grove holds the port and the server isn't running at all. When the first connection arrives, Grove starts MySQL behind that connection and hands it over. When nothing has been connected for ten minutes (the default; --idle 30m if you want longer), Grove stops it cleanly.

The cost of that is small. The first query after an idle stop was answered in 0.35 seconds, including the server start, and every query after that ran exactly as before. Twenty clients hitting an idle server at the same moment got twenty answers from one start. 517 MB went to nothing.

A few details that are the difference between a clever trick and something you can leave on:

It's sound because every client goes through Grove. The daemon's count of open connections is the count of clients, so "nobody is connected" is a fact, not a guess. For the same reason, the server's Unix socket moves to a private name in this mode. A client connecting through the socket wouldn't be counted, and would be cut off when the server went idle. (None of the 90 projects on the machine this was built on connect that way, but a tool you can't reason about is one you'd turn off.)
It stops politely. An idle server gets SIGTERM and fifteen seconds to finish, not SIGKILL, so MySQL shuts down cleanly and doesn't replay its redo log the next time it starts.
Grove's own tools start it when they need it. Snapshots, restores and branch switches go through the same port, so none of them fail because the database happened to be asleep.
It works for MySQL, PostgreSQL, ElyraSQL and Redis, survives a daemon restart, and grove service list shows a stopped on-demand server as idle rather than stopped.

A database per git branch

This is the one we've wanted for years, and so have you, even if you've never put it into words.

You're on a feature branch. You write a migration that adds a column and backfills it. You run it. Then someone asks you to look at a bug on main, so you check out main, and now main is running against a schema it has never seen. The errors you get look like bugs in main. They aren't. They're your feature branch, still sitting in the database.

grove db branches on

Now each branch has its own data. Check out a branch and its database is swapped in. Go back to main and main's data comes back exactly as it was left. The first time you check out a new branch, it starts from the data you were on, so you never begin from an empty database.

Why the database keeps its name

The design decision worth noticing: the database keeps its name, and only its contents move.

The obvious implementation would be to point the app at myapp_feature_x when you're on feature-x. We didn't, because that would only reach the requests Grove proxies. Your terminal's php artisan migrate would still read DB_DATABASE=myapp from .env. So would your test run, and so would TablePlus. Half your tools would be on the branch and half wouldn't, and that's worse than no branching at all.

Keep the name and swap what's inside, and everything follows the checkout without being told: the app, artisan, the tests, the database client. On MySQL the swap is a single atomic RENAME TABLE that moves every table between the live schema and the branch's parked one. It was measured at 43–46 ms for a 70 MB schema, the same as for an empty one. The first visit to a new branch copies the data, which took 831 ms for 300,000 rows. On SQLite it's two renames of the database file and its -wal/-shm.

The edges, handled

During a switch the site answers 503 with Retry-After, and its grove dev processes restart. A queue worker comes back on the new branch's code and data, instead of running the new code against the old tables.
A detached HEAD (a rebase, a bisect) moves nothing, and a switch waits until a checkout has settled for a second. Rebasing twelve commits doesn't mean twelve database swaps.
A switch writes its intent before anything moves, so one interrupted by a crash is finished or undone the next time the daemon starts.
What it refuses, it names. MySQL on Grove's own server and SQLite are supported. A remote MySQL, PostgreSQL and ElyraSQL are refused with a message, and so is a MySQL database with views, triggers, stored routines or events, where "move every table" wouldn't move everything. Better a clear no than a swap that leaves a trigger pointing at the wrong schema.
Nothing is deleted behind your back. grove db branches shows what's live and what's parked, and flags copies whose git branch is gone. off keeps the copies and on picks them up again. drop is the only command that deletes one.

Three places that said "done" when it wasn't

Some of the most useful changes in this release aren't features. They're places where Grove had been telling you something reassuring that wasn't quite true.

grove service start said "started" for a server that had already died. A successful spawn means a process exists, not that a server is running. mysqld with no data directory, or Postgres with a stale lock file, exits within a second, and by then Grove had already reported success. Now start waits until the port actually accepts a connection, for up to 30 seconds. A server that exits before then fails the command and shows you its own error lines from the log. "Started" now means you can connect.

Two snapshots in the same second overwrote each other. Snapshot ids were to-the-second timestamps, and the file name was built from the id. A second dump in the same second replaced the first on disk, while the index kept both entries pointing at the same file. Two quick snapshots in a row were enough, and a sandboxed migration snapshots before it runs. A taken id now gets a -2, -3 suffix.

Restoring a MySQL snapshot left tables that were created after it. This is the one we're most glad to have caught. A mysqldump --databases file recreates the tables it contains and says nothing about any others. So you snapshot, run a migration that creates invoices, restore, and invoices is still there. That's precisely the case a snapshot before a migration exists for. Each database in the dump is now dropped and recreated on the way in, so it comes back exactly as it was. MySQL's own system schemas are never dropped, and databases the dump doesn't contain are left alone.

We had written "data restored exactly as it was" in our own course. It's true now. It's worth saying plainly that it wasn't quite true before.

One thing to do when you upgrade

1.8.0 moved the Grove daemon off root: launchd binds ports 53, 80 and 443 and hands the sockets to a daemon running as you. 1.9.0 finishes that job by deleting the code that let a root daemon run safely. With that code gone, a root daemon would start PHP-FPM and your databases as root, out of a directory you can write to. So 1.9.0 refuses to start as root.

If you updated to 1.8.0 and ran sudo grove install afterwards, nothing changes for you. If you didn't, including if you're coming straight from 1.7, the daemon stops at startup and says so:

Error: the Grove daemon will not run as root.
… Run `sudo grove install` to rewrite the service so it starts as you.

Run it once. Your sites, databases and certificates are where they were.

Why these, together

On-demand and branch databases share one precondition: Grove has to own the database. It has to know every connection, hold the port, and be able to move tables atomically and put them back after a crash. A tool that only knows your database's host and port can't offer either.

The three fixes share a principle. A tool you depend on should tell you what happened, not what it hoped would happen. "Started" should mean connectable, a snapshot shouldn't silently replace another, and a restore should restore.

Get it

Grove 1.9.0 is at elyracode.com/grove, for macOS on Apple silicon and Intel, and Linux as AppImage, deb and rpm. Both new features are opt-in: nothing about an existing service or database changes until you turn them on. And if you're coming from before 1.8, run sudo grove install once.

Then, in a project you work on across branches:

grove db branches on
grove service on-demand mysql on

Check out an old branch. Run its migrations. Go back to main and notice that nothing followed you there.