Server

One process writes one file. Two processes over the same file corrupt it, which is why everything a server does — HTTP, replication, the graph keeper, the sweeper of rows past their time (@ttl), the --follow mirror — runs inside this one process rather than as programs of its own.

One thread per connection. Read-only statements run at the same time under a shared lock; writes take the exclusive one. HNSW search buffers are thread-local, so the same index can be searched in parallel — eight concurrent full scans on eight cores measured 4.3×.

A /batch is one block: its writes land in the file as one record, or none of them do, and a replica applies it whole. There are no interactive transactions held open across requests — the database has one writer, and a transaction a client held open would hold every other write with it. An Idempotency-Key makes a write, or a /batch, safe to retry (HTTP endpoint).

With --replica-of the process follows another fenec-server as a read replica: it serves every read and refuses every write with 403. The flags, promotion and what a failover loses are on Replication. With --dir <path> instead of --file it serves a directory of tenant files, one per tenant under /t/<tenant>/; that mode, its --admin-token and --idle-close, and the router in front of it are on Tenants and sharding.

--http-token is the server's own token: whoever sends it reads and writes everything. The comparison is constant time. Listening on a non-loopback address with no token is refused outright; it can be opened deliberately with --insecure.

With --jwt-secret (HS256, at least 32 bytes) or --jwt-keys (a JWKS file: oct keys for HS256, RSA keys for an identity provider's RS256, a token's kid picking the key, the file read again as it changes) the server also takes JSON Web Tokens, each held to the rules of --policy: which collections, to read, insert, update or delete, which rows (where owner = $jwt.sub, where room in $jwt.rooms), which fields of them a read shows (read(pickup, run_at) where run_at <= now(), Fields a read shows), and what an update may make of a row (where status = "ready" and new.status = "cancelled"). A token that may only read is the reporting login a dashboard needs, one that may only insert a client appending events, and an append-only collection takes no update or delete from any token but the server's. The rules are ANDed into every statement a token runs, and every document it writes is checked against them as well (Scoped access). --mint-token '{"sub": "ada"}' prints a token signed with the secret, for testing, which expires in an hour unless the claims name an exp. A token with no exp is refused, since it would be good for ever (--jwt-require-exp off takes it), and --jwt-max-age <s> refuses one whose exp lies further ahead than that. On a --dir node a token reaches only the tenant its tenant claim names (Tokens). Pass the secret with --jwt-secret-file or FENEC_JWT_SECRET — argv shows up in ps output.

A request refused for its token waits before it is answered — 100 ms, twice as long for each refusal from the same address within a minute, 5 s at most (--auth-delay; 0 turns it off). A token the server verifies clears its address's count; an answer that looked at no token, whatever Authorization it carried, clears nothing. --audit <path> appends a JSON line for each refused request, each change of the schema and each admin request (The audit log).

Writes accumulate in a 1 MB buffer. Under always every write waits for an fsync, about 3.7 ms on macOS for a write alone, and writes arriving together share one; 250 costs next to nothing; off nothing. A /batch lands with one fsync however many writes it holds.

Under always the fsync runs after the write has let go of the exclusive lock, and the bytes reach the file there too. Readers therefore do not wait on the disk, and writes that arrive while one fsync runs are made durable together by the next. One client still gets one write per fsync. The sync of 250 takes the same path, for a single database and for each tenant of a --dir node: held through the fsync, a tenant's lock had its reads and writes wait up to 27 ms once an interval.

SIGINT, SIGTERM and SIGHUP are caught and a final sync runs before exit. The exclusive lock is held until exit, so no write can be accepted between the sync and the exit and no write is left that looked successful to the client but never reached the disk. Under kill -9 you get exactly what the chosen policy promised, and no more.

When the disk refuses a write or a sync, the request fails with 500 — under always, before the client is told it succeeded — and every later write is refused until the process restarts; reads go on answering. The sync is not retried: after a failed fsync the kernel may already have dropped the pages, so a retry that succeeds would prove nothing.

The shutdown checkpoint. After the sync, if a vector index exists, a checkpoint is written: the HNSW graph lands in the file and the next open does not rebuild it — 9 ms instead of 3.4 s at 100 000 × 128. Vectors still waiting to be linked after a crash are written as waiting, and linked after the next open. The cost is that shutdown takes longer on a large database: the image is written into the new file as it is produced, so it is a copy on disk rather than one in memory. --no-checkpoint turns it off. The order is deliberate. The sync comes first, so even if the process is killed while the checkpoint is being written, the data is already on disk and only the graph is lost. The rewrite goes to a side file and is renamed, so a half-written checkpoint cannot corrupt anything.

Each index is built under the read lock on its own, so reads go on beside it, a read that needs the index being built waits for that build rather than starting its own, and a write waits out one index at most (a @text index of 100 000 products: 141 ms). The thread starts once the HTTP endpoint is up, whose start takes the write lock. Over 100 000 products with a text, two hash and an ordered index, on an M1: the first answer 16 ms after the start either way, the indexes built 208 ms after it, and the first match 1.4 ms warm against 166 ms cold; the process held 99 MB against 88 for a cold one that had only searched, and 25 before any read.

It is on for a file because an index is declared to be read, and a first query that builds one is the slowest a server answers: a shop's first search page took 2.68 s to its largest paint against 1.40 s warm. The memory is what the reads would have taken anyway; only an index nothing ever reads is held for nothing, and --warm names the rest. A node's tenants open on their first request and close when idle, and warming each would build every index of every tenant a request touches, so there it is off unless asked. The native library has the same call, fenec_warm, and the Rust API Database::warm and warm_index.

Left alone, a file of updated rows grows by a record a write, and its live records end up a few to a page among dead ones: under YCSB's updates, a million records of 1 KB took the file to 4.7 to 6.5 GB on an 8 GB machine, and one read in five then waited for the disk. With the default ratio the file stays under about twice what the last compact left it at, and each compact writes the live records once for as many bytes of updates.

What it costs, measured with make compact-bench: a million 1 KB records, five million updates of a field at keys drawn at random from one thread, buffered, while another reads whole records. With auto-compaction the file went from 1.03 GB to between 2.1 and 3.0 GB and back to 1.1 to 1.6 GB three or four times; each compact took 8 s at the median with no lock held and held the write lock 6.1 to 12.6 ms to put its file in place. The reads while a compact ran had a p99 of 1.3 to 5.3 ms. The updates ran at about 30 000 a second through a compact, against 110 000 to 170 000 while the file was smaller: on this 8 GB machine, beside its other programs, the page cache holds about 2 GB of the file, the size at which a compact comes due, and past it an update reads its row from the disk whether a compact runs or not -- 28 000 to 46 000 a second with auto-compaction off. At 250 000 records, whose file fits, the updates ran at 153 000 a second through a compact of 0.3 s, and at no less than 83 000 in its worst tenth of a second, against 180 000 with none. Once the updates were done the reads ran at 435 000 a second, p99 2.2 µs, from the first second. Without it the file ended at 6.2 GB, and the reads after the updates started at 19 000 a second and took 25 s to reach 400 000, their p99 0.11 ms over those 30 s. Durable, an fsync a write from 16 writers, 100 000 records updated 400 000 times at about 900 a second: the file went from 103 to 211 MB and back three times, each compact held the write lock 13.7 to 14.8 ms -- two F_FULLFSYNCs of this Mac's among it -- and ended at 190 MB against 515 without it; that file fits the page cache either way, and the reads ran alike, p99 2.2 µs.

Before the file outgrows the page cache. fenecdb keeps no cache of its own: a document is read where it lies in the mapped file, so what the reads need in memory is every page of the file that holds a live record -- and under updates at keys drawn at random that is every page until a compact (a million 1 KB records updated until the file stood at 2.1 GB left one on every 16 KB page and on 94% of 4 KB ones). Once the file outgrows what the system has for it, every read and update waits for the disk in turn. So a file is also compacted once a third of what the compact writes is dead, where its live records fit in that memory and the file has reached three quarters of it -- which Linux reports exactly, its available memory and a container's limit -- or the server takes 500 major faults a second, which is how macOS shows it: under pressure it compresses the other programs' memory and reports more room than it gives the file. That costs up to four bytes written for each byte updated, against two at half the file dead. A compact whose new file would not fit beside the old one writes it past the page cache and reads it in after the swap, so the old file, which the reads need until then, stays in memory. Measured with make compact-bench in a container held to 2 GB (docker run --memory 2g): at half the file dead the file stood over 2 GB for 69 to 74% of the run, the five million updates took 358 to 511 s, 3 500 to 4 100 a second at the median, and the process took 811 000 to 900 000 major faults; with the memory rule, 80 to 100 s and 34 000 to 39 000, the file at most 1.9 GB, the reads during a compact p99 0.44 to 0.49 ms against 4.6 to 6.5, for 3.0 to 3.2 bytes written a byte updated against 1.9 to 2.0. On this Mac beside 1 GB of memory held by another program, 191 to 260 s against 82 to 535: macOS gives a growing file more of the machine as it compresses the other programs' memory, which it did in the fastest run at half the file dead and not in the slowest, and a file kept smaller is not given it -- steadier, never the best case. A file that fits in memory is compacted as before.

The compact copies what it reads under the read lock, writes the live records into a side file with no lock held, and copies the writes made meanwhile in rounds -- a few thousand documents looked up under the read lock at a time and copied once it is let go -- until a round finds fewer than a thousand; it reads every page of the new file in before the swap, asking for them ahead in large reads, so no reader waits for one after it -- unless the two files would not fit in memory together, when it writes the new one past the page cache and reads it in once the old one is gone. The write lock is then held for the last of those writes, the side file's fsync and the rename. The server looks for a compact due every 5 s, and a look waits for a write under way rather than pass a compact over. A replica compacts its own file the same way: a compact changes no document, and an image its primary sends meanwhile fails the compact rather than be written over. /_metrics has the file's size, what a compact would give back of it and the compacts run (Monitoring), and each is logged with the file before and after and how long it held the lock.

One thread per connection means unlimited connections would leave the stack memory ceiling to the client. Past --max-connections a client gets 503; --idle-timeout closes a quiet connection. A subscription holds a thread for as long as it lasts, so subscriptions are counted apart as well, under --http-max-streams, a database's own: on a tenant node each tenant's, where --max-connections is what all of them share, and each tenant's streams and held writes take a quarter of it at most unless its quota says otherwise (quotas).

--max-memory puts a ceiling on the data footprint. Above it, requests that grow the data are refused with 507, while reads, del and compact keep working: hitting a ceiling with no way out would be no better than a cgroup OOM. The point is to bring the OOM forward: nobody warns you about a process felled by SIGKILL, and both the last sync and the checkpoint are lost with it. The measurement happens before the statement, so the overshoot is at most one statement.

--ping asks GET /_health on the --http address and exits 0 when it answers. The route needs no token and takes no lock: every query takes the database lock first, so during a long compact a probe that ran one would wait too and a healthy server would look dead. Because the listener binds after the file is opened, "healthy" also means "the documents are read".

What the server is doing is at /_metrics, in Prometheus's text format: --metrics <address> opens a listener for it alone, and --slow-ms <ms> logs every statement at least that slow with its text, a JSON line on stderr. The metrics, a Grafana dashboard, the log and the Datadog Agent's configuration are in Monitoring and Datadog and other monitoring systems.

The Dockerfile is two-stage: a static musl build copied into scratch. There are no dependencies and the binary runs its own health check (--ping), so the runtime image holds nothing but the binary — no shell, no package manager, no libc. The image is 2.99 MB on amd64 and 2.44 MB on arm64 — since no target is written in, both come out of the same file. make docker builds it, and docker compose up -d starts the compose file's.

The memory limit comes from the data. With no page cache there is no knob to tune: the file is mapped and what the process holds is the indexes, which is what --max-memory counts. Below that the cgroup sends SIGKILL, the clean shutdown never runs and up to one sync interval is lost. Set --max-memory to a third of the container limit to bring that forward.

The restart cost depends on how it stopped. docker stop sends SIGTERM and a checkpoint is written, so the next open takes 9 ms. After kill -9 or a cgroup OOM the graph of the last checkpoint is restored, and the vectors written since it go in unlinked: the server answers once the documents are read, and links them on a thread beside its queries, near measuring each of them until then. The only checkpoint the server writes on its own is the one at shutdown, but it appends its graphs to the file as they change — a graph that changed in 10 000 nodes, once the file has grown by three times its record, logged as 1 graph written into the file, with writes held off for the 5 ms that takes at 100 000 × 768 — so what waits is what came after the last of them: at 100 000 × 768 at most 10 000 vectors, linked in 2.6 s with near at 1.56 ms p50 meanwhile. With no graph in the file every vector waits: at 100 000 × 768 never checkpointed the first near was answered 1.23 s after the start, against 67.7 s when the open linked them first, and linking takes 19 s. fenec_vectors_unlinked counts down to 0.

Overriding CMD drops --file, and the container comes up with an in-memory database. There is no shell in the image, so docker exec ... sh does not work; what is left is docker logs and docker run --rm ghcr.io/fenecdb/fenec-server --help.

Two containers loading the same file and writing it corrupt it. On Kubernetes that means a StatefulSet rather than a Deployment, replicas: 1, strategy: Recreate and a ReadWriteOnce volume — a rolling update keeps two pods up together for a moment. There is a connection ceiling but no queue: an excess connection does not wait, it gets 503.