Instead of Redis
The jobs an app adds Redis for — counters, rate limits, locks, sessions, queues — done by the database it already has, each a statement or two.
Four things make these work. A set can read the row it
writes ({n: n + 1}), and the read and the write happen under
the one writer's lock, so an increment is atomic. Every write answers how
many rows it changed, so a write that names what it expects says whether it
found it, and require makes that count a condition of its
/batch. @ttl lets a row expire, and a write take
its place the moment it does. And every write can be watched: a
subscription is sent the rows of a shape as they change, and
/_changes every write on disk. See Writes that read the row for the
rules.
Every FenecQL block on this page is run, statement by statement, by a
test (crates/fenec-core/tests/redis_docs.rs), and each
-- → x is the answer it checks: a write's count, or the first
value of the first row a read answered. A line -- 20 s later
moves the test's clock, and $1 is the rows the read before it
answered, as a client sends them back. The costs are make
counters-bench on an M1 Air, in process unless said otherwise; over
HTTP a request adds its round trip. Where another bench measured a recipe,
it is named.
Redis's patterns, and where each one is
| Pattern | Redis | Here |
|---|---|---|
| Counters | INCR, INCRBY, HINCRBY | set c {n: n + 1} |
| Rate limit, fixed window | INCR and EXPIRE | a row a window, @ttl |
| Rate limit, sliding window | ZADD, ZREMRANGEBYSCORE, ZCARD in MULTI | a row a request, @ttl, a read that must find none |
| Rate limit, token bucket | a Lua script | one set over the bucket's row |
| Lock with a lease | SET NX PX, a check-and-delete script | put … if absent, @ttl, del … where owner |
| Fencing token | none of its own: INCR beside the lock | counted by the write that takes the lock |
| Optimistic transaction | WATCH, MULTI, EXEC | a version, and require 1 in a /batch |
| Sessions | SET EX, GET, GETEX, EXPIRE | @ttl, set … returning |
| Cache with a TTL | SET EX, GET, DEL | @ttl and an upsert — no eviction |
| Leaderboard | ZADD, ZINCRBY, ZREVRANGE, ZRANGEBYSCORE, ZREVRANK | @sorted, order … limit, count |
| Sets, deduplication | SADD, SISMEMBER, SREM, SCARD, SINTER | @unique and if absent |
| Unique visitors | PFADD, PFCOUNT, PFMERGE | approx_count_distinct, sketches kept as bytes |
| Job queue | LPUSH, BRPOP, BLMOVE | a claim: set … order … limit … returning, Fenec-Wait |
| Delayed jobs | ZADD by time, ZRANGEBYSCORE polled | a run_at ahead, the claim woken at it |
| Streams, consumer groups | XADD, XRANGE, XREADGROUP, XACK | a collection, /_changes and its consumers |
| Pub/sub, keyspace notifications | PUBLISH, SUBSCRIBE, notify-keyspace-events | a row with a short @ttl, a subscription |
| Idempotency keys | SET NX EX on a request's id | Idempotency-Key, or @unique and if absent |
| Places near a point | GEOADD, GEOSEARCH, GEODIST | a geo field, @geo (Points and distances) |
| Counts by the minute | INCR on a key a minute | bucket(at, 1m) over the events (Analytics) |
| A cache that evicts under memory pressure | maxmemory, allkeys-lru, allkeys-lfu | not covered: a row lives until its @ttl |
| Scripts | EVAL, FUNCTION | not covered: a /batch with require |
| Bitmaps, Bloom filters, Top-K | SETBIT, BITCOUNT, BF.ADD, TOPK.ADD | not covered |
What every recipe can count on:
- Atomic. There is one writer: a statement finds its rows and
writes them under the write lock, and a
/batchis one block — every statement in it lands, or none does, with no other write between them. - Durable by
--sync. Under--sync alwaysa write is answered once it is on disk; under the default,250, a crash loses at most the last 250 ms of writes (Durability). - On a replica. A replica is sent each write as the primary
wrote it — a counter's new value, never
n + 1— once it is on the primary's disk, in the primary's order. It refuses writes (403), so increments, locks and claims go to the primary; a read sent there withFenec-Afterwaits for the write it names (Replication). A row past its@ttlis out of a replica's reads as it is out of the primary's, and deleted when the primary's sweep is sent.
Counter
An increment is a set that reads the field it writes.
create collection counters (name text @unique, n int)
put counters {name: "visits", n: 0}
set counters {n: n + 1} where name = "visits" -- → 1
get counters select n where name = "visits" -- → 1
set counters {n: n + 5} where name = "visits" returning n -- → 6: INCRBY's answer
put counters {name: "signups", n: 1} if absent else set {n: n + 1} -- → 1: made at 1
put counters {name: "signups", n: 1} if absent else set {n: n + 1} -- → 1: added to
get counters select n where name = "signups" -- → 2- None is lost. Sixteen HTTP clients each incrementing one key
10 000 times ended at 160 000, 102 000 increments a second against
fenec-server; sixteen threads in process, 320 000 a second. - An increment costs 1.36 us in process.
- From nothing:
coalesce(n, 0) + 1counts a row with nonyet;n + 1over anullisnull. A key that may not be there yet is made or added to by one statement, an upsert:if absent else set. returninganswers the row as written, so the new value comes back with no second read.
In JavaScript, inc() writes the coalesce for
you:
import { connect, inc } from '@fenecdb/web/client';
const db = connect('http://127.0.0.1:8080');
await db.from('counters').where('name', 'visits').update({ n: inc(1) });Python's inc(1), Go's fenecdb.Inc(1), .NET's
Computed.Inc(1), and Swift's, Kotlin's and Dart's builders
render the same text.
Rate limits
Three ways, each decided by what a write finds: a fixed window, the cheapest; a sliding window, exact to the moment; a token bucket, which lets a burst through up to its size and then a steady rate.
Fixed window
A row a client and window, made if absent and expiring after the window; the increment names the limit, so its answer is the decision.
create collection hits (key text @unique, n int, at timestamp @ttl(2m))
put hits {key: "ip1:29630000", n: 0, at: now()} if absent -- → 1: the window's row
put hits {key: "ip1:29630000", n: 0, at: now()} if absent -- → 0: there already
set hits {n: n + 1} where key = "ip1:29630000" and n < 2 -- → 1: allowed
set hits {n: n + 1} where key = "ip1:29630000" and n < 2 -- → 1: allowed
set hits {n: n + 1} where key = "ip1:29630000" and n < 2 -- → 0: over the limit- The window is in the key — here the minute,
floor(now / 60000), which the client works out — so a new window is a new row, and@ttllets the old ones go. - A request costs its two statements, 3.5 us; send them as
one
/batchfor one round trip. - A fixed window lets a burst through at its edge: the limit at the end of one minute and again at the start of the next. The two below do not.
Sliding window
A row a request, expiring after the window, and a read that must find
fewer than the limit: offset 2 limit 1 require 0 asks that no
third row is there, so it allows three a minute. Sent as one
/batch with the request's row, the read refuses the batch (412)
once three are, and the row is not written.
create collection calls (key text @hash, at timestamp @ttl(1m))
get calls where key = "ip1" offset 2 limit 1 require 0 -- → no row: allowed
put calls {key: "ip1", at: now()} -- → 1
get calls where key = "ip1" offset 2 limit 1 require 0 -- → no row: allowed
put calls {key: "ip1", at: now()} -- → 1
-- 30 s later
get calls where key = "ip1" offset 2 limit 1 require 0 -- → no row: allowed
put calls {key: "ip1", at: now()} -- → 1
get calls where key = "ip1" offset 2 limit 1 require 0 -- → refused: three in the last minute
-- 30 s later
get calls where key = "ip1" offset 2 limit 1 require 0 -- → no row: the first two are a minute old- Exact: the rows of the last minute, counted the moment the
request comes, since a row past its
@ttlis out of the read at once. The guard and the write are one block under the write lock, so two requests cannot both take the last place. - A row a request is what it keeps, as Redis's sorted set keeps a member a request; a limit of thousands a window is a fixed window or a bucket instead.
Token bucket
A bucket holds up to two tokens and gains one a second; a request takes
one. The row keeps the tokens and when they were counted, and one
set works out what the time since has added — the two
timestamps apart are milliseconds — and takes a token only where there is
one:
create collection buckets (key text @unique, tokens float, at timestamp @ttl(1h))
put buckets {key: "ip1", tokens: 2.0, at: now()} if absent -- → 1: a full bucket
set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} where key = "ip1" and tokens + (now() - at) / 1000.0 >= 1 -- → 1: allowed
set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} where key = "ip1" and tokens + (now() - at) / 1000.0 >= 1 -- → 1: allowed, empty now
set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} where key = "ip1" and tokens + (now() - at) / 1000.0 >= 1 -- → 0: refused
-- 1 s later
set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} where key = "ip1" and tokens + (now() - at) / 1000.0 >= 1 -- → 1: one token back- The script Redis needs for this is the one statement: the row is read and written under the write lock, so no two requests take the same token.
- A refused request writes nothing, so the time it waited still counts toward the next token.
- The
@ttlis longer than a refill: a bucket idle for an hour would be full again, so its row may go, and the next request makes it full.
From a client, the two statements go as one /batch, with
require 1 on the take so that a refusal is a 412:
import { connect, FenecError } from '@fenecdb/web/client';
const db = connect('http://127.0.0.1:8080');
// true: serve the request; false: answer it 429.
async function allow(key: string): Promise<boolean> {
try {
await db.batch([
['put buckets {key: $1, tokens: 2.0, at: now()} if absent', [key]],
['set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} ' +
'where key = $1 and tokens + (now() - at) / 1000.0 >= 1 require 1', [key]],
]);
return true;
} catch (e) {
if (e instanceof FenecError && e.status === 412) return false;
throw e;
}
}Lock with expiry
A lock is a row: taken by the write that makes it, held while it lives, and free again once its time passes, so a holder that dies cannot keep it.
create collection locks (name text @unique, owner text, at timestamp @ttl(30s))
put locks {name: "nightly", owner: "w1", at: now()} if absent -- → 1: w1 holds it
put locks {name: "nightly", owner: "w2", at: now()} if absent -- → 0: w2 does not
set locks {at: now()} where name = "nightly" and owner = "w1" -- → 1: renewed
set locks {at: now()} where name = "nightly" and owner = "w2" -- → 0: not w2's
del locks where name = "nightly" and owner = "w1" -- → 1: released- Exactly one wins. The write lock decides: of any number of
clients racing, one is answered 1 and the rest 0 — sixteen threads are
raced for it in
crates/fenec-core/tests/writes.rs, and again once its time has passed. - Thirty seconds after its last renewal the row is out of every
read, the renewal's answer is 0 — the holder has lost it — and the next
put … if absenttakes it, the old row deleted in the same statement. The sweeper need not have come by. - The check-and-delete is the
del: it names the owner, so a holder whose lock lapsed and was taken by another deletes nothing — the script Redis's documentation gives for the release. - Costs: refused while held 0.91 us, renewed 2.34, released and taken again 2.67.
Details
if absent passes over a document whose @unique
value or id a live row holds, and counts what it wrote: Redis's
SET NX. insert refuses the same document with a
409 instead, which serves too, but makes losing the race an
error to catch and aborts a /batch it is part of. A lock named
by a number can be its id:
put locks {id: 1, owner: $1, at: now()} if absent.
Time is the server's clock, and a lock lasts its ttl from its last renewal: renew well within it. As with any lock that expires, a holder paused past its ttl may still believe it holds the lock; give the work it guards a check of its own — a fencing token, below — where that matters.
Fencing tokens
A holder paused past its lease — a long garbage collection, a stalled
disk —
wakes believing it holds the lock, while another does. A fencing token
tells the two apart: a number the lock hands out, one more at every
acquisition, which the work it guards carries and the resource checks. Here
the lock's row keeps it, so the lock is a row that stays, and the lease is
its until:
create collection leases (name text @unique, owner text, until timestamp, fence int)
put leases {name: "report", fence: 0} if absent -- → 1: the lock's row, made once
set leases {owner: "w1", until: now() + 30000, fence: fence + 1} where name = "report" and (owner is null or until <= now()) returning fence -- → 1: w1 holds it, token 1
set leases {owner: "w2", until: now() + 30000, fence: fence + 1} where name = "report" and (owner is null or until <= now()) returning fence -- → no row: held
-- 40 s later
set leases {owner: "w2", until: now() + 30000, fence: fence + 1} where name = "report" and (owner is null or until <= now()) returning fence -- → 2: w1's lease lapsed
set leases {until: now() + 30000} where name = "report" and owner = "w2" and fence = 2 and until > now() require 1 -- → 1: renewedThe resource keeps the highest token it has been written with, and takes a write only from that token or a newer one:
create collection reports (day text @unique, body text, fence int)
put reports {day: "2026-05-03", fence: 0} if absent -- → 1
set reports {body: "by w2", fence: 2} where day = "2026-05-03" and fence <= 2 require 1 -- → 1: token 2 is the newest
set reports {body: "by w1", fence: 1} where day = "2026-05-03" and fence <= 1 require 1 -- → refused: w1 woke late, with token 1
set leases {owner: null, until: null} where name = "report" and owner = "w2" and fence = 2 require 1 -- → 1: released- The token is counted by the write that takes the lock, under
the write lock: no two acquisitions get one number, and none is
skipped. Redis's
SET NXhands out none: a counter beside the lock is a second key, kept in step with it by a script. - The row stays so the count does: under
@ttlit would be deleted when it lapsed, its count with it. A lapsed lease isuntil <= now(), which the next acquisition reads. - Waiting for it: sent over HTTP with
Fenec-Wait, an acquisition that takes nothing is held at the server until the holder releases it or its lease lapses, whichever comes first — both are a write or a time the server knows of (Holding a write until it can write). - A resource in fenecdb checks the token in the write itself, as above. One elsewhere — a file store, another service — needs a check of its own; the token is a number it can keep.
Optimistic concurrency
Read a row with its version; write it back only where the version is
still the one read, and move the version on in the same statement — what
Redis's WATCH and MULTI are for.
create collection docs (body text, version int)
put docs {id: 7, body: "draft", version: 1}
get docs select body, version where id = 7 -- → "draft"
set docs {body: "final", version: version + 1} where id = 7 and version = 1 -- → 1: written
set docs {body: "other", version: version + 1} where id = 7 and version = 1 -- → 0: read again
set docs {body: "other", version: version + 1} where id = 7 and version = 1 require 1 -- → refused: 412, and its batch put back- 0 means someone wrote first: read the row again and retry, or tell the user.
- In a
/batch, addrequire 1: a 0 would let the statements after it go on, where a lost race should stop them. With it the write that found another version is refused (412) and the batch is put back whole (Writes that must find their rows). Aget … require 1guards a batch on a value it does not write (A read that must find its rows). - Costs: a write that wins 1.13 us, one that loses 0.66.
Session store
A session is a row that lives while it is used: @ttl on the
time it was last seen, a @unique token to find it by.
create collection sessions (token text @unique, user text, data json, seen timestamp @ttl(30m))
put sessions {token: "s-81f2", user: "ada", data: {"cart": 2}, seen: now()}
get sessions select user where token = "s-81f2" -- → "ada"
set sessions {seen: now()} where token = "s-81f2" -- → 1: seen again
set sessions {seen: now()} where token = "s-81f2" returning user, data.cart -- → "ada": read and seen again, GETEX
del sessions where token = "s-81f2" -- → 1: signed out- An idle session is gone at once for every read, thirty minutes after it was last seen, and the server's sweeper deletes it one to two minutes later (Rows that expire).
- Read and renewed in one by
set … returning: the row as written comes back, and a session already gone answers no row. - Costs, over 10 000 sessions: a read 1.32 us, seen again 2.26.
Cache with expiry
A value kept for a while in front of something slow to work out: read it by its key, and on a miss work it out and keep it. The upsert writes a key that is there and one that is not alike, and starts its time again.
create collection cache (key text @unique, value json, at timestamp @ttl(5m))
get cache select value.name where key = "user:7" -- → no row: a miss
put cache {key: "user:7", value: {"name": "Ada"}, at: now()} if absent else set {value: new.value, at: now()} -- → 1: kept for five minutes
get cache select value.name where key = "user:7" -- → "Ada": a hit
put cache {key: "user:7", value: {"name": "Ada L."}, at: now()} if absent else set {value: new.value, at: now()} -- → 1: written over
del cache where key = "user:7" -- → 1: invalidated
put cache {key: "user:7", value: {"name": "Ada"}, at: now()} if absent else set {value: new.value, at: now()} -- → 1
-- 300 s later
get cache select value.name where key = "user:7" -- → no row: five minutes oldWhat it is, and what it is not:
- No eviction. There is no LRU or LFU and no
maxmemorypolicy: a row lives until its@ttlor adel, however rarely it is read. What the cache holds is what was written within its ttl. With--max-memory, a write past the ceiling is refused (507) rather than make room by dropping another row (Limits). A cache that must stay inside a memory budget by dropping what is used least wants Redis. - It survives a restart. The rows are in the file, durable by
--sync, so a server started again answers from a warm cache, and a replica holds it too. - The values are in the file, not the heap. The file is mapped: the process holds the indexes derived from the rows, and the rows themselves are pages the system keeps or lets go — a 1 GB file of 2.3 million rows with two indexes held 188 MB (The file is mapped).
- Expired rows go in the background. Out of every read at once,
deleted by the server's sweeper one to two minutes later, 1 000 a
block: 100 000 of 200 000 rows went in 0.56 s, each block holding the
write lock 1.06 ms at the median and 2.3 at most (
make ttl-bench). The space an overwrite leaves is taken back by the server's compaction once half the file is dead (Compaction). - Many clients missing one key at once each work it out; to have one do it, the first takes the key's lock and the rest read again after.
Leaderboard
A score field with @sorted: the top of the board is a walk
of the index from its end, and a player's rank is a count.
create collection scores (player text @unique, score int @sorted)
put scores [{player: "ada", score: 0}, {player: "bo", score: 90}, {player: "cy", score: 40}]
set scores {score: score + 120} where player = "ada" -- → 1: ZINCRBY
get scores select player, score order score desc limit 10 -- → "ada": ZREVRANGE 0 9
get scores select player, score order score desc limit 10 offset 1 -- → "bo": the next page
get scores select score where player = "ada" -- → 120: ZSCORE
get scores where score > 120 count -- → 0: ada is first, ZREVRANK
get scores select player where score >= 40 and score <= 100 order score desc -- → "bo": ZRANGEBYSCORE- A rank is a count of the scores above the player's: read the score, then count, two statements — or one, where the score is known.
- Ties come out in the order the players were first written, their ids', whichever way the board is read; Redis orders them by the members' names.
- Costs, over 100 000 players: a score added 1.79 us, the top ten 1.52. A player's rank is a count of the scores above it, the index's range gathered: 1.5 ms at the median of the board.
Sets and deduplication
A member is a row, and @unique over the set and the member
keeps it there once: if absent adds it and answers whether it
was new.
create collection members (list text @hash, member text, @unique(list, member))
put members {list: "tags:7", member: "rust"} if absent -- → 1: SADD, new
put members {list: "tags:7", member: "rust"} if absent -- → 0: there already
put members [{list: "tags:7", member: "db"}, {list: "tags:8", member: "rust"}] if absent -- → 2
get members where list = "tags:7" and member = "rust" count -- → 1: SISMEMBER
get members where list = "tags:7" count -- → 2: SCARD
get members select member where list = "tags:7" and member in (get members select member where list = "tags:8") -- → "rust": SINTER
del members where list = "tags:7" and member = "db" -- → 1: SREM- A set's members are its
@hashbucket, read without the others:SMEMBERSisget members select member where list = $1. - Seen-it sets — message ids, a feed's items already shown —
are a collection of one
@uniquekey and a@ttl, so an id is remembered for as long as a repeat can come. - Exact, at a row a member: a Bloom filter's fixed memory and its false positives are not here.
Unique visitors
approx_count_distinct is HyperLogLog with 214
registers — Redis's register count, so its standard error, 0.81% —
counted over the visits as they were written. count(distinct)
is the exact count, up to a million values a query.
create collection visits (user text, page text @hash, at timestamp @sorted)
put visits [{user: "ada", page: "/", at: "2026-05-03T09:00:00Z"}, {user: "bo", page: "/", at: "2026-05-03T09:05:00Z"}, {user: "ada", page: "/docs", at: "2026-05-03T10:00:00Z"}, {user: "cy", page: "/", at: "2026-05-04T08:00:00Z"}] -- → 4: PFADD, a row a visit
get visits select approx_count_distinct(user) as users -- → 3: PFCOUNT
get visits select approx_count_distinct(user) as users, bucket(at, 1d) as day group day -- → 2: on the 3rd
get visits select count(distinct user) as users where page = "/" -- → 3: exactA day's sketch kept in a row is Redis's key a day: read out as bytes
with hll_accumulate, put back as a client sends rows, and
merged — PFMERGE — by hll_combine, each visitor
of two days counted once:
create collection daily (day timestamp @unique, users bytes)
get visits select bucket(at, 1d) as day, hll_accumulate(user) as users group day -- → "2026-05-03T00:00:00Z"
put daily $1 -- → 2: the rows above
get daily select hll_estimate(hll_combine(users)) as users -- → 3: both days, ada once
get daily select hll_estimate(users) as users where day = "2026-05-03" -- → 2- Costs: per hour over a million events, 121 ms against
count(distinct)'s 128, the furthest of the 169 hours 1.3% off (make analytics-bench). - Sketches are the same bytes wherever they are made, the browser's or a server's, and a group of few values keeps few words rather than 16 KB (Distinct users).
Job queue
A job is a row. A worker claims a batch with one statement, which finds the oldest ready jobs and makes each its own for a lease, and acks a job by deleting it.
create collection jobs (payload json, run_at timestamp @sorted, owner text, attempts int)
put jobs [{payload: {"to": "ada"}, run_at: now(), attempts: 0}, {payload: {"to": "bo"}, run_at: now(), attempts: 0}]
set jobs {owner: "w1", run_at: now() + 30000, attempts: attempts + 1} where run_at <= now() order run_at limit 10 returning id -- → 1: both, w1's for 30 s
del jobs where id = 1 and owner = "w1" require 1 -- → 1: done- None is taken twice: the claim finds its jobs and writes them
under the write lock. A job whose worker died is ready again once its
lease lapses, and the next claim takes it — at least once, where
BRPOPloses a job its worker dies holding. - Costs, over a million jobs: a claim of ten 18.7 us; sixteen
threads claimed and acked them all in 5.4 s, 185 800 jobs a second,
and sixteen HTTP clients at 138 500 (
make queue-bench). BLPOP: sent over HTTP withFenec-Wait, a claim that finds no job is held at the server until one is enqueued, a delayed one comes due or a lease lapses, and handed to the worker that has waited longest: a job enqueued reached a waiting worker in 0.38 ms at the median (Waiting for a job).- Retries with a backoff, dead letters, delays and priorities are on their own page, Job queues.
Delayed jobs
A job delayed is one whose run_at is ahead: no claim takes
it before then, and the claim that would take it is the same one.
create collection reminders (payload json, run_at timestamp @sorted, owner text, attempts int)
put reminders {payload: {"to": "ada"}, run_at: now() + 60000, attempts: 0} -- → 1: due in a minute
set reminders {owner: "w1", run_at: now() + 30000, attempts: attempts + 1} where run_at <= now() order run_at limit 10 returning id -- → no row: not yet
-- 60 s later
set reminders {owner: "w1", run_at: now() + 30000, attempts: attempts + 1} where run_at <= now() order run_at limit 10 returning id -- → 1: due- No poll: a claim held with
Fenec-Waitis woken at the moment the next job comes due, which the server works out from the@sortedindex: 0.04 ms after its time at the median and 1.9 at p99, against 1.2 and 8.4 polling every 10 ms (Waiting for a job). - No ceiling on the delay, and a delayed job is read like any
row:
get reminders where run_at > now() count.
Streams
A stream is a collection written to and never written over: each entry a
row, its id counted up as it is added. A range is a walk of
the ids from the last one read, and old entries go by @ttl.
create collection activity (kind text @hash, user text, at timestamp @ttl(7d))
put activity {kind: "signup", user: "ada", at: now()} -- → 1: XADD
put activity [{kind: "order", user: "ada", at: now()}, {kind: "order", user: "bo", at: now()}] -- → 2
get activity select id, kind where id > 1 limit 100 -- → 2: XRANGE, after entry 1
get activity where kind = "order" count -- → 2A reader that must see every write once, and pick up where it left off,
reads the server's change stream: every write on disk, a JSON line each.
Started with --cdc, a named consumer keeps its place on the
server, as a consumer group's last delivered id does:
fenec-server --file data.fenec --http 127.0.0.1:8080 --cdc
curl -X POST http://127.0.0.1:8080/_changes/consumers/mailer # XGROUP CREATE … $
curl -i 'http://127.0.0.1:8080/_changes?consumer=mailer&wait=10000' # XREADGROUP … BLOCK 10000
# Fenec-Next: 42 -- handle the lines, then XACK up to there:
curl -X POST http://127.0.0.1:8080/_changes/consumers/mailer -d '{"since": 42}'- At least once where the server keeps the place: a consumer
that crashes between reading and committing is handed those writes
again (Consumers). One that keeps
Fenec-Nextbeside its own state, committed with it, has each write once (Change data capture). - Only what is on disk: a write is handed out once an fsync has covered it, so none a crash lost reaches a consumer.
- Every collection's writes are in it, each line naming its collection, puts, deletes and schema changes alike; a consumer passes over the rest.
- One consumer, every write. A consumer group whose consumers share the entries — each to one of them, the pending ones claimed again after a while — is a job queue: a claim and its lease.
- How far back: as many writes as
--replication-bufferholds, 64 MiB unless given; a cursor further back is answered 410 with the firstsinceit does serve, never with writes missing from the middle.
Pub/sub and change notification
A message is a row, kept for as long as a late subscriber should still see it, and a subscription is sent the rows of a shape — first those there are, then each change as it lands.
create collection messages (room text @hash, body text, at timestamp @ttl(1m))
put messages {room: "lobby", body: "hello", at: now()} -- → 1: PUBLISH# SUBSCRIBE: the rows of a shape, then each change to them as it lands
curl -N 'http://127.0.0.1:8080/messages/changes?room=eq.lobby'
# every write on disk, from where a named consumer left off
curl 'http://127.0.0.1:8080/_changes?consumer=mailer&wait=10000'import { connect } from '@fenecdb/web/client';
const db = connect('http://127.0.0.1:8080');
const stop = db.subscribe('messages', { room: 'eq.lobby' }, (ev) => {
if (ev.type === 'change') for (const m of ev.puts) console.log(m.body);
});
await db.run('put messages {room: $1, body: $2, at: now()}', ['lobby', 'hi']);
stop();- A subscription (Subscriptions) is woken by the write, not by polling: from a write sent to its event read by a subscriber took 0.037 ms at the median and 0.103 at p99 over HTTP on one machine.
- A late subscriber is sent the messages still inside their
@ttlfirst, where Redis's pub/sub hands it only what is published after it subscribed. - Fan-out costs the writer. Every write wakes every stream,
which reads what it changed: a write's p50 went from 0.26 ms with no
stream to 0.94 with 60 and 3.1 with 500, each stream a connection and
a thread, 64 at most unless
--http-max-streamssays otherwise. A page thousands view polls instead (Polling a read). - A live query keeps a query's answer rather than a shape's
rows:
db.live(query, cb)runs it again on the server after every write to what it reads (JavaScript). - The change stream (Change data
capture) hands out every write on disk exactly once to a consumer
that sends back where it is, which a subscription — a shape's rows kept
up to date — does not promise:
notify-keyspace-eventswith a cursor.
Idempotency keys
A client that sends a write again after a timeout, not knowing whether
the first landed, sends it with the same Idempotency-Key: the
server keeps the key and the answer in the same block as the write, and a
repeat is handed the first answer.
curl -X POST http://127.0.0.1:8080/query \
-H 'Idempotency-Key: 5f0c2d6e-order-1142' \
-d '{"query":"put orders {customer: $1, total: $2}","params":[7, 129.5]}'
# sent again with the key: the first answer, Idempotent-Replayed: true, nothing written- The key and the write land together, so a crash cannot leave
one without the other; a failed write keeps no key, and the key with
another request is refused (422). Keys are kept a day unless
--idempotency-ttlsays otherwise (Retrying a write). - Costs: 16 200 single puts a second over HTTP with a key, against 18 300 without.
A message that arrives more than once — a webhook redelivered, an event replayed — is told apart by its own id: a row of it, made if absent, and the work done only where it was.
create collection deliveries (event text @unique, at timestamp @ttl(1d))
put deliveries {event: "evt_1J8", at: now()} if absent -- → 1: first time, handle it
put deliveries {event: "evt_1J8", at: now()} if absent -- → 0: a repeat, already handled
put deliveries {event: "evt_1J8", at: now()} if absent require 1 -- → refused: in a /batch, its work put back with it- With
require 1in a/batchbeside the writes it guards, the mark and the work land together or not at all: a repeat is refused (412) and writes nothing, and a crash cannot leave work done with no mark.
Places near a point
GEOADD and GEOSEARCH are a geo
field with @geo: the metres between two points, the rows
within a radius or a box, the nearest first — see
Points and distances.
create collection sicily (name text, loc geo @geo)
put sicily [{name: "Palermo", loc: [13.361389, 38.115556]}, {name: "Catania", loc: [15.087269, 37.502669]}] -- → 2
get sicily select name where distance(loc, [15, 37]) <= 200000 near loc [15, 37] limit 10 -- → "Catania"- Costs, over a million points: every point within 1 km 12.6
us, against 34.3 us for Redis's
GEOSEARCH, and the ten nearest 33 us, against 198 ms — the server's own time for Redis, the network left out (make geo-bench, What it costs).
What fenecdb does not do
- Evict. No LRU or LFU, no
maxmemorypolicy: a row lives until its@ttlor adel, and--max-memoryrefuses a write past it (507) rather than drop another row (Cache with expiry). A cache that must stay inside a memory budget by forgetting what is used least wants Redis. - Keep its data in memory alone. A server always keeps a file,
and a write is as durable as
--syncsays: the default,250, answers before the fsync and loses at most the last 250 ms in a crash;alwaysanswers after it, a durable update 4.0 ms at the median on a Mac (F_FULLFSYNC) and 0.325 ms on Linux in Docker (One request's round trip);offsyncs at shutdown alone (Durability). A request is HTTP: a read by id took 0.022 ms at the median on one machine (A request over HTTP), and a network's round trip adds to it. Memory alone, a function call a statement, is the engine embedded in the app — in Rust, where an increment costs 1.36 us, or in the page (Embedded Rust, JavaScript). - Run scripts. No Lua, no
EVALorFUNCTION. What a script checks before it writes isrequireon the statements of a/batch, which lands whole or not at all, and a value worked out over the row is aset's expression orcase. A batch cannot read a value and branch on it: a decision between two reads is the client's, a round trip each. - Write from more than one writer. A file has one: writes run
one after another under its lock — 320 000 increments a second in
process and 102 000 over HTTP from sixteen clients (
make counters-bench), 185 800 jobs claimed and acked a second (make queue-bench) — and reads go beside them: four threads read 701 000 rows a second by id beside four writers (Writers and readers at once). Past one machine's writer the data is split by tenant, a file each (Tenants and sharding); there are no hash slots. - Fan out across tenants. A subscription and
/_changesare one file's: on a tenant node each tenant is a file of its own, and the router forwards a stream to the node holding it. There is no channel every tenant's subscribers hear; a message for all of them is a write to each. - Pub/sub without storage. Every message is a write that lands
in the file and moves the change counter; there is no fire-and-forget
channel. A row with a short
@ttland a subscription is the nearest (above). - Speak RESP. A Redis client cannot connect: the server speaks HTTP, and every language reaches it through it or an SDK (Your language).
- Bitmaps, Bloom filters and Top-K. No
SETBITorBITCOUNT, no Bloom or Cuckoo filter, Count-Min sketch or Top-K. Membership is exact, a row a member (Sets); HyperLogLog is there (Unique visitors). - Deliver exactly once. A queue's lease that lapses under a
slow worker hands its job to another, and a consumer that crashes
before it commits reads again: the work's own check — a
@uniquekey, anIdempotency-Key— makes the second run harmless.