Tenants and sharding

A single fenecdb is bounded by one machine's memory and one writer. Past that, the data is split by tenant: each tenant is a file of its own, a node holds many of them, and fenec-shard knows which node holds which.

Below /t/<tenant> the surface is exactly the HTTP endpoint of a single database — routes, /query, /batch, subscriptions. A client changes its base URL and nothing else; the sync layer included:

const db = await sync({ url: 'http://127.0.0.1:8090/t/acme', shapes: [...] });

The tenant is picked by the request path, never by the query text, so nothing rewrites a query to add tenant = .... Two tenants sharing a collection would then read each other's rows. A file each keeps them apart by construction, and the same decision settles four things a shared file would have had to solve:

• Ids are counted per file, so they never collide and a moved tenant keeps its own.
• The change sequence behind /changes is one number per file, so a client's cursor stays a plain number.
• BM25 statistics — document count, average length, document frequency — are the tenant's own, which is what ranking inside a tenant should mean.
• lookup never crosses a file, so it stays one bucket probe per parent.

With a tenant whole on one node, every query runs where all of its data is: near, match, order, count and lookup keep their single-database meaning, and the router never parses a query. Spreading one collection across nodes would need a merge for every read shape — and global statistics before match could rank anything, because per-node IDF gives scores that do not compare and a merged order that is quietly wrong. That is a different feature, and the case for it is a single tenant that outgrows a node. Until then, a tenant has to fit one node's memory.

--dir serves a directory of <tenant>.fenec files instead of one file. It is still one process per directory — the single-writer rule is per file, and one process owning all of them is what keeps it. --http is required. The tenant is looked up for every request — so one created a moment ago is there, one closed as idle reopens, one deleted meanwhile answers 404, and one frozen for a move refuses writes while its reads go on.

• Opened on first use. Open cost is proportional to the file, so a node does not spend its start-up on tenants nobody asks for. A tenant that has not been created is 404 — a request never brings one into being, or a typo would make an empty database.
• Closed when idle. --idle-close <s> (default 300) syncs, checkpoints when there is a vector index, and closes a tenant nothing has touched for that long. One with an open request or subscription is never closed.
• --max-memory covers the open tenants together. Opening one more over the ceiling first closes idle tenants, least recently used first. With everything busy the open goes ahead anyway: refusing a read because other tenants are busy would turn a memory warning into an outage. A write that grows a tenant is refused once that tenant alone is past the ceiling, over either listener.
• Shutdown syncs every open tenant and checkpoints the ones with a graph, holding their write locks until exit.

The router drives the node through /_admin/, which answers only to --admin-token and is off (404) without it — a client allowed to read and write a tenant must not be able to delete it or take another tenant's image:

An image is replayed in memory before it is written, so a truncated transfer is refused rather than installed, and the file appears by rename, never half-written. Tenant names are [a-z0-9_-], 1 to 64 characters: they become file names, and that set means the same thing on every file system.

The router reads a request's head and body, looks the tenant up, and forwards it byte for byte — the original target, not a re-encoding of it. An answer with a length is read and passed back on a kept-alive connection; one without is a subscription, copied as it arrives until either side closes, so the stream format lives in one place, the node.

Connections to a node are kept and reused, and each is looked at before a request goes out on it: one the node closed for its silence (--idle-timeout) is let go of. A request that fails before it is wholly sent goes again on a fresh connection. One sent whole whose connection ends before an answer goes again only where running it twice is harmless — a GET, HEAD or OPTIONS, or a write with an Idempotency-Key, which the node makes once (Retrying a write). Any other write is answered 502, since the node may have run it: a client that sends it again should send it with a key.

Its only state is the directory: two collections in a fenecdb file of its own, nodes and tenants. Every change is written and synced before it takes effect, and a tenant's node and state change in one statement, so there is no moment on disk where a tenant belongs to both nodes or to neither. The node admin tokens stay in it; GET /_shard/nodes does not return them.

Placement goes by disk rather than memory: memory counts only the open tenants, and which ones are open depends on who asked last.

A page on another origin — an app, the sync layer, FenecHttp.live, the studio served by a node — reaches its tenants through the router once the router has --http-cors <origin>, read as fenec-server reads it (Security): * or one origin. The router answers a browser's preflight itself, before it looks the tenant up, and puts the CORS headers on every answer it gives. Its own — an unknown tenant's 404, a standby's 503, an unreachable node's 502, the 503 past --max-connections, the 401 of /_shard/ without its token — reach the page as the refusals they are; without the headers a browser hands the page a network error for each, and the sync layer backs off from what was an answer. On a node's answer the router's headers take the place of the node's, so the page is told the rule the router's preflight told it, and never two Access-Control-Allow-Origins, which a browser refuses whole. The 503s carry their wait as Retry-After and as retry_after in the body, as a quota's refusal does. Without the flag the router sends none of its own and passes a node's through as the node wrote them, and a preflight goes to the node.

What is bounded is what is known exactly and cheaply. A file's bytes are the engine's count of what it appended, read with no system call; rows are each store's count, a sum over the collections (those whose names start with _ are the server's: idempotency keys, consumers' places); streams and requests are counters a request takes and gives back; the rates are a token bucket each. A tenant with no quota touches none of it. Left out: the memory an open tenant holds (memory_bytes counts what has been built, and a read builds an index on first use — a quota on it would refuse a write because of a search, and the same data holds less after a restart; --max-memory closes idle tenants for the node as a whole), a statement's time (the engine cannot stop a running scan; requests in flight bound how many cores one tenant's long reads take), and rows per collection (a tenant's own design: @ttl on a log).

The block is refused before it lands. A quota asked before a write runs would let that write past it by whatever it adds, and a set over every row adds the whole collection again. The engine asks the quota as a block lands, with its record's length to the byte and its documents counted (Database::set_limit); one past it is put back as a failed one is. A block larger than 16 MB spills into the file before it lands, and each spill is asked first: let through and refused after, its bytes would stay in the file, dead and past the quota. The file passes its quota only by what a write that takes away adds — a delete's tombstones, a drop's record — and by what no client writes: the graphs a server keeps in its file's tail. The next compact gives both back. A rate is not asked at the landing: a write lands while its bucket holds a row's worth or any bytes, and its rows and bytes are charged as it lands -- the rows of the tenant's collections it wrote, not the server's own, such as an Idempotency-Key's -- so a write larger than a second's worth goes through and the next waits out the debt.

A refusal tells the quota, not what the tenant holds: a scoped token's user is told the tenant is full, not how many rows the other users make. Only the admin token reads usage.

The router enforces none of it. It holds no keys and parses no query, and a tenant lives whole on one node, so the node's count is the tenant's. A quota a node does not take — a node down when it was set — is logged, and sent again as that copy is promoted.

What it costs. A tenant with no quota touches none of it: against the code before quotas, in turns, a request to a tenant took 19.5–21.7 µs straight to the node and 43.7–46.0 µs through the router, against 19.9–21.8 and 43.6–45.7. A tenant with every quota answered as one with none over HTTP — reads 20.9 µs either way, writes 23.7 against 23.6 — and in process a put held to bytes and rows took 593 ns against 566, 618 with a bytes-a-second bucket besides (make quota-bench).

1. The directory marks the tenant moving; it is still routed to the source.
2. The source freezes it: writes in flight finish, new ones get 503 with Retry-After: 1. Reads go on being served.
3. The image is copied from the source and installed on the target.
4. The directory points the tenant at the target, in one statement.
5. The source copy is deleted. Its open subscriptions end with an error event, and the client reconnects through the router to the target. A client that had seen everything resumes from its cursor without a reseed: the change sequence travels in the image and the source was frozen, so nothing happened in between. One that was behind is reseeded, as after any restart — the target has the sequence but not the history before it.

The router refuses nothing itself during a move, so there is no window in which the tenant is nowhere — only one in which writes are asked to retry. If a step before the flip fails, the source is thawed and the directory put back. A router that stops mid-move comes back with the tenant still served from the source and marked moving; moving it again replaces whatever copy the interrupted attempt left on the target, which is stale by definition, since the directory never pointed at it.

Node and router in one process on loopback, one keep-alive client, 5 000 requests after 500 of warm-up (make shard-bench):

About 21 µs per request, which is one more loopback round trip; across a network the extra hop is whatever that network charges for one. Moving a tenant of 100 000 × 128 with an HNSW index — a 58.7 MB image — took 188–267 ms end to end over three runs, and the first query on the target about 70 ms, most of it opening the file: the graph travels in the image, so nothing is rebuilt.

• Failover is decided on a lease, not a guess. A router cannot tell a node that is gone from one it cannot reach, and guessing would make two primaries for one tenant. Without --auto-failover someone who knows the node is gone calls the failover; with it, the router promotes a node's tenants only once the node's own lease has certainly lapsed, and the node stops its writes as it does (Automatic failover).
• No query across tenants. Each request names one.
• No tenant larger than a node. A tenant has to fit one node's memory, with the same 2× open and 3× checkpoint peaks as any file.
• One router takes changes at a time. A standby follows its directory and forwards from the same placements, but only the primary places, moves and deletes; promoting the standby is a decision someone makes, not a vote the routers hold.

Every tenant file has a feed of its own, so a node's tenants are followed by another node's — the standby — tenant by tenant, through /t/<tenant>/_replication.

The router creates a tenant on both, so the replica has a file to follow into, and deletes it from both; a move takes the replica with it. Recording a pair gives the standby a following copy of every tenant already on the node. A replica node opens every tenant it holds at start and keeps them open — a follower that is not running is a replica falling behind — so --idle-close does not apply to them. No tenant is placed on a standby or moved to one: it would open there as a replica of a tenant the other node does not have.

A tenant's role is its file's, not the node's. On a node started with --replica-of, a replica's file follows and a new one follows, but a primary's file stays a primary — a tenant a failover promoted there included, whatever the node restarts with. Were every file on such a node made to follow, a restart with the same flags would undo the failover, and the old primary's copy would wipe the writes taken since.

When a node is gone, POST /_shard/nodes/<n>/failover promotes its tenants on the standby, one at a time, and points the directory at it. They are separate databases: there is nothing to make atomic between them, so the answer lists what was promoted and what was not.

curl -X POST localhost:8090/_shard/nodes/n1/failover
{"node":"n1","to":"n2","promoted":["acme","beta"],"failed":[]}

Measured with make shard-bench, node, standby and router in one process on loopback: a write through the router is answered in 2.93 ms (--sync always on the primary, which is an fsync a write) and is on the standby 0.089 ms later, 0.448 ms at p99 — timed by reading the standby's own copy. A failover of 20 tenants took 60 ms, 3.0 ms each, and the first write on the promoted node 0.18 ms.

A promoted tenant's history forks where the replica stood, so what the primary had not sent is gone — the same trade a database's failover makes, and --sync always on the primary is what makes it none. A failover that moved every tenant ends the pair; one that left a tenant behind keeps it, so a second call can take it. The old node rejoins as the standby of the new one: point its --replica-of there and record the pair the other way round. The router then has each of its tenants follow (POST /_admin/tenants/<t>/follow) — its own files are a primary's, and follow only when told to — and the writes it took that were never sent are gone.

fenec-server --dir /data/n1 --http 127.0.0.1:8081 --admin-token a1 \
         --replication-token r1 --replica-of http://127.0.0.1:8082
curl -X PUT localhost:8090/_shard/nodes/n2 \
  -d '{"addr":"127.0.0.1:8082","token":"a2","standby":"n1"}'
{"node":"n2","unreplicated":[]}

A tenant file that still follows on a node that follows nobody — a failover that left it behind, and the node restarted without --replica-of — refuses writes, says so in the log, and is promoted where it stands with POST /_admin/tenants/<t>/promote.

A standby waits idle, and takes every tenant of its node at once when the node goes. With --replicas the router gives each tenant a replica of its own instead, on another node in no pair: the one holding the fewest replicas of that tenant's node, then the fewest tenants and replicas altogether. Each node's tenants then follow on all the others, and every node holds primaries and replicas both.

The replica's node is told which node to follow (POST /_admin/tenants/acme/follow with {"from": "http://127.0.0.1:8081"}) and keeps it in acme.follows beside the file, so the tenant follows again when the node starts over. The replication token is the nodes', one for the cluster. A move takes the replica along: the copy goes on following the tenant on its new node, since the image carried its history, and when the tenant moves onto the node holding its replica, that copy goes and another is placed. A delete removes both.

A failover of a node in no pair promotes each of its tenants on its own replica, one at a time, and routes it there. The tenants whose replicas it held are named as unreplicated. POST /_shard/replicas gives every tenant without a replica one, on a node holding a copy of it first: a node back from a failover has its old primaries follow the nodes that took them over, and nothing of theirs is left lying there.

curl -X POST localhost:8090/_shard/nodes/n1/failover
{"node":"n1","promoted":[{"tenant":"acme","to":"n2"},{"tenant":"beta","to":"n3"}],"failed":[],"unreplicated":["gamma"]}
curl -X POST localhost:8090/_shard/replicas
{"replicated":[{"tenant":"acme","replica":"n1"},{"tenant":"beta","replica":"n1"},{"tenant":"gamma","replica":"n1"}],"failed":[]}

Measured with make shard-bench, three nodes and the router in one process, 30 tenants: each is created with its replica in 33 ms, the replicas 10 a node, and a write is on its replica 0.179 ms after it was answered, 0.421 ms at p99. A failover of one node's 10 tenants took 57 ms, 5 onto each of the other two, and the repair after it gave 20 tenants a replica in 176 ms, all on the returning node's copies. Placed by the fewest altogether alone, every tenant of one node followed on the same other: a tie went by name each time, and its failover put them all there.

With fenec-shard --auto-failover <s> the router fails a node over on its own. It leases each node the tenants it places there for that long, renews the lease every third of it while the node answers, and fails over a node that has gone a tenth past its lease unrenewed. A node started with --lease writes a tenant only while its lease names the tenant and has not lapsed, and starts with none.

The router still cannot tell a node that is gone from one cut off from it while its own clients reach it, and it does not have to. A node that hears from the router no more stops taking writes when its lease lapses, by its own clock, and the router promotes its tenants elsewhere only a tenth of a lease later, measured from the answer that came back rather than from the grant it sent — so no tenant has two primaries, whichever it was. What such a node took from clients of its own before its lease lapsed, and had not sent its replicas, goes with the failover as a crash takes it; a client going through the router loses nothing that way, since the router could not reach the node either. A write is refused as a replica's is: 503 with Retry-After.

The lease names tenants, not the node. A node failed over comes back with the files of tenants served elsewhere now, primaries' files still; a lease over the whole node would have them take writes again the moment it was renewed. Its new lease names none of them, and the router runs a repair as the node answers, which has those copies follow the tenants' new primaries. The list rides with a grant only when it changed, and a create or a move grants the node its new one at once, so a tenant takes writes as soon as it is placed. A router that starts leasing — a restart, or a standby promoted — counts every node as granted then: its predecessor may have granted a lease just before it stopped. A node started without --lease answers that it takes none, and is never failed over on its own.

Measured with make shard-bench, three nodes and the router in one process and a lease of a second: a node cut off stopped taking writes 697 ms after — its last renewal came a third of a lease before — and its 10 tenants took writes on their replicas again 1.29 to 1.33 s after, the lease and a tenth, the next round and 49 ms of promotions. With a lease of five seconds, a node lost costs its tenants' writes 5.5 to 7.2 s.

The directory is a fenecdb file, so it replicates the way any other does. A second router follows it and forwards from the same placements; the primary is the one that changes it.

Both forward /t/<tenant>/...; the standby's /_shard/ answers reads and refuses changes with 409, since its directory takes no write of its own. When the primary is gone, POST /_replication/promote on the standby forks its directory's history and it starts taking them — the same promotion a replica of a database gets, and the same rule after it: the old primary rejoins with --replica-of pointed at the new one, or its file is refused.

A standby is behind by whatever the last directory change has not reached it yet, which is one write and a sync: placements change when someone places, moves or deletes a tenant, not when clients read and write. Nothing of the tenants' own data goes through the router's directory, so a standby that is a second behind still forwards every request correctly. One whose directory has not arrived at all — just started, its primary not yet reached — answers 503 with Retry-After rather than 404: it knows of no tenant, which is not knowing there is none.

Each change the primary makes reaches the standby's maps through the change ring, as a subscription's rows do: the placements it put or deleted since the last one, 4 µs at 100 000 tenants, where reading every map again took 38 ms under the router's write lock. An image adopted, or more changes than the ring holds, and the maps are read again whole.

Three, each for a different job. Data requests carry whatever the nodes' --http-token expects, and the router passes Authorization through untouched. Each node has its own --admin-token, which only the router holds. The router's --token guards /_shard/; like the nodes, it refuses to listen on a non-loopback address without one unless given --insecure.

With --jwt-secret or --jwt-keys a node also takes JSON Web Tokens held to its --policy, and the policy is the node's, not a tenant's: notes read,write where owner = $jwt.sub matches alice's rows in every tenant's file alike. So on a --dir node a token reaches /t/<t>/ only when its tenant claim names <t>, a text or a list holding it, and is refused with 403 elsewhere: on every route under the prefix, the statements, REST, /batch, a subscription, /_changes, /_schema and the statement counts. The check comes before the tenant is looked up, so a refused token learns nothing of which tenants exist. A token naming no tenant is refused on a --dir node unless it is started with --jwt-unbound-tenants, which takes such a token for every tenant it serves; --jwt-tenant-claim <name> reads another claim than tenant. The node's own --http-token and --admin-token are not bound. The router forwards Authorization as it came and holds no keys, so the node is what enforces it, behind the router or reached directly. What a scoped token writes to a tenant is checked against its rules as on a single file (Scoped access).