ElyraSQL 1.10.0 — the server was optional all along

Some features arrive after months of design. This one arrived after somebody ran grep.

We were mapping out what to build next when we went looking at our own dependency graph — not for anything in particular, just to see what the shape of the thing had become after a year of changes. And there it was, sitting in plain sight:

elyra-engine  ->  elyra-core  elyra-storage  elyra-vector  elyra-olap

That's the whole list. The SQL engine — all 44,000 lines of parser, planner and executor — depends on storage, on the vector index, on the analytics kernel, and on nothing above them. No elyra-wire. No elyra-server. No socket, no handshake, no MySQL protocol anywhere in its dependency tree.

Which means we hadn't been shipping a database server with a library hidden inside it. We'd been shipping a library with a server bolted on the front, and nobody had noticed hard enough to say so out loud.

1.10.0 says it out loud.

What you can do now

use elyra_embed::{Database, Value};
let db = Database::open("app.edb")?;
let conn = db.connect();
conn.execute("CREATE TABLE users (id INT PRIMARY KEY AUTO_INCREMENT, name TEXT)")?;
conn.execute("INSERT INTO users (name) VALUES ('Ada')")?;
let rows = conn.query("SELECT name FROM users")?;
assert_eq!(rows.get(0, "name"), Some(&Value::Text("Ada".into())));

No server process. No port. No connection string. No Docker Compose file that somebody has to remember to start. Just a file on disk and a function call.

And — this is the part we care most about — the SQL means exactly what it means on the server, because it is not a reimplementation, a subset, or a "compatible mode". It is the same executor, compiled into your binary instead of into ours.

Why this is a slightly unusual thing to exist

Embedded databases are embedding SQLite means your tests run against different semantics than your users do. Different type coercion. Different date handling. A different answer for ROUND(1.005, 2). You've traded a running server for a category of bug that only appears in production.

MySQL itself used to offer a way out. libmysqld, the embedded server library, let you link the real thing straight into your process. It was deprecated in 5.7 and removed in 8.0. The door closed.

So the combination in 1.10.0 is, as far as we can tell, not currently on offer anywhere else: MySQL semantics, in-process, over a single file, with vector search and full-text and exact DECIMAL in the box.

We'd love to be told we're wrong about that. Genuinely — if something else does this, we want to read its source.

We tested the claim rather than making it

"Same engine, same file" is easy to write in a changelog and annoying to actually verify. So we did the obvious thing.

First, we wrote a database entirely in-process, with no server involved anywhere:

$ cargo run --example basic /tmp/compat/embed.edb customer total deposit Ada 1250.00 Grace 875.50 218.88 Linus 399.95 99.99

3 orders, total 2525.45, average 841.816667

Then we took that file, mounted it into the released 1.9.9 Docker image, and queried it with a stock mysql client over the wire protocol:

+----------+---------+---------+
| customer | total   | deposit |
+----------+---------+---------+
| Ada      | 1250.00 |  312.50 |
| Grace    |  875.50 |  218.88 |
| Linus    |  399.95 |   99.99 |
+------+---------+------------+
| n    | sum     | avg

Look closely at 218.88. That's 875.50 × 0.25 = 218.875, rounded half away from zero — the way MySQL does it, not the way f64 does it. Route that number through a double and you get 218.87, because 218.875 has no exact binary representation and the rounding falls the wrong way. It's a rounding error of one cent, on a deposit, in a currency column.

That one cent is the entire argument for this release. If your tests run on semantics that round differently than production, your tests are lying to you politely.

The 841.816667 is the same story from the other end: MySQL's AVG gives the dividend's scale plus div_precision_increment, which is four. Not a float that happens to print similarly. The actual rule, because it's the actual

Where this actually helps

Test suites, first and loudest. This is the one we built it for.

let db = Database::temporary()?;   // real database, deleted when it drops
let conn = db.connect();

A fresh database per test, in a few milliseconds, with real MySQL semantics and real persistence for the duration of the test. No container to start. No port to allocate. No wait-for-it.sh. No CI job that's flaky one time in forty because MySQL hadn't finished booting.

Local development, where the file you develop against is byte-compatible with the one you deploy.

CLI tools and desktop apps that want SQL but have no business running a service.

Edge and single-tenant deployments where exactly one process owns its data and a network hop is pure overhead.

And keep the server for what a server is actually for: clients in other processes, replication, network access control, and the enormous ecosystem of MySQL drivers and ORMs that already work.

From C, and from everything that speaks C

elyra-embed-capi ships the same API over a C ABI, as a shared or static library with a header. Which means PHP, Python, Node, Ruby, Go — anything with an FFI:

#include "elyrasql.h"
ElyraDb *db = NULL;
if (elyra_db_open("app.edb", &db) != ELYRA_OK) {
fprintf(stderr, "%s\n", elyra_last_error());
return 1;
}
ElyraConn *conn = NULL;
elyra_db_connect(db, &conn);
elyra_conn_execute(conn, "INSERT INTO users (name) VALUES ('Ada')", NULL);
ElyraRows *rows = NULL;
elyra_conn_query(conn, "SELECT name FROM users", &rows);
for (size_t r = 0; r < elyra_rows_count(rows); r++) {
printf("%s\n", elyra_rows_value(rows, r, 0));
}
elyra_rows_free(rows);
elyra_conn_free(conn);
elyra_db_free(db);

Ownership is deliberately boring: every handle has a matching *_free, freeing NULL is a no-op so your cleanup path needs no guards, and strings borrow from the handle they came from — nothing is freed per value. Errors are a return code plus a thread-local message. A Rust panic cannot cross the boundary; the entry points abort instead.

For a Laravel shop, the shape this eventually takes is worth saying plainly: run your test suite against real MySQL semantics, in-process, with nothing to install.

The bug we found by using our own thing

Here's the part we enjoyed, in retrospect.

The first real test suite we wrote against the embedded library did the most ordinary thing a test suite does: open a database, use it, drop it, and do it again. It failed almost immediately.

storage error: Database already open. Cannot acquire lock.

Wait 250 milliseconds and retry, and it works fine. So the file lock was outliving the handle that owned it.

The cause turned out to be a decision made years ago that had never once mattered. Our storage writer runs on a dedicated OS thread — spawned detached, with no join handle kept, because why would you keep one:

thread::Builder::new()
.name("elyra-writer".into())
.spawn(move || writer_loop(writer_storage, rx, repl))

That thread holds a reference to the storage, and therefore the file lock. When the last handle drops, its channel sender drops, the loop sees the channel close and the thread exits — but on its own schedule, with nothing synchronising that back to whoever dropped the handle.

A server opens its file once when it starts and holds it until it stops. In that world this is invisible; it has been invisible for the entire life of the project. Open and close the same file in a loop, though — which is precisely what an embedded test suite does — and you hit it every single time.

We could have quietly papered over it. Instead: the embedded open now waits the window out, with a bounded budget, while a genuinely concurrent open still fails the way it should. And the real fix — a deterministic close at the storage layer — is written up as an issue with the two candidate designs, including a note that the workaround reintroduces the exact string-matching-on-error-messages pattern we removed from our catalog errors three releases ago. When someone fixes it properly, the code that carries the cost points straight at it.

We think that's the more useful kind of release note. Building the thing found a defect that four years of running it as a server never surfaced. That's not a mark against the design; that's what happens the first time you point a tool at itself.

Getting it

cargo add elyra-embed

Or, for the C ABI:

cargo build -p elyra-embed-capi --release
cc app.c -I crates/elyra-embed-capi/include -L target/release -lelyrasql -o app

No on-disk format change. A 1.9.x database opens in 1.10.0 unchanged, and a 1.10.0 database still opens in 1.9.x. There are no upgrade steps — the minor version bump is for the new surface, not for anything that moved underneath you.

471 tests green, clippy and rustfmt clean, and the whole thing round-tripped against the previous release's Docker image in both directions before we tagged it.

The embedded guide has the details worth knowing before you lean on it: the one-writer-per-file rule, why every call blocks and what happens if you make one from an async context, and an honest list of what a server still gives you that in-process does not.

Go put a database inside something.