Instead of Redis

The jobs an app adds Redis for — counters, rate limits, locks, sessions, queues — done by the database it already has, each a statement or two.

Four things make these work. A set can read the row it writes ({n: n + 1}), and the read and the write happen under the one writer's lock, so an increment is atomic. Every write answers how many rows it changed, so a write that names what it expects says whether it found it, and require makes that count a condition of its /batch. @ttl lets a row expire, and a write take its place the moment it does. And every write can be watched: a subscription is sent the rows of a shape as they change, and /_changes every write on disk. See Writes that read the row for the rules.

Every FenecQL block on this page is run, statement by statement, by a test (crates/fenec-core/tests/redis_docs.rs), and each -- → x is the answer it checks: a write's count, or the first value of the first row a read answered. A line -- 20 s later moves the test's clock, and $1 is the rows the read before it answered, as a client sends them back. The costs are make counters-bench on an M1 Air, in process unless said otherwise; over HTTP a request adds its round trip. Where another bench measured a recipe, it is named.

Redis's patterns, and where each one is

PatternRedisHere
CountersINCR, INCRBY, HINCRBYset c {n: n + 1}
Rate limit, fixed windowINCR and EXPIREa row a window, @ttl
Rate limit, sliding windowZADD, ZREMRANGEBYSCORE, ZCARD in MULTIa row a request, @ttl, a read that must find none
Rate limit, token bucketa Lua scriptone set over the bucket's row
Lock with a leaseSET NX PX, a check-and-delete scriptput … if absent, @ttl, del … where owner
Fencing tokennone of its own: INCR beside the lockcounted by the write that takes the lock
Optimistic transactionWATCH, MULTI, EXECa version, and require 1 in a /batch
SessionsSET EX, GET, GETEX, EXPIRE@ttl, set … returning
Cache with a TTLSET EX, GET, DEL@ttl and an upsert — no eviction
LeaderboardZADD, ZINCRBY, ZREVRANGE, ZRANGEBYSCORE, ZREVRANK@sorted, order … limit, count
Sets, deduplicationSADD, SISMEMBER, SREM, SCARD, SINTER@unique and if absent
Unique visitorsPFADD, PFCOUNT, PFMERGEapprox_count_distinct, sketches kept as bytes
Job queueLPUSH, BRPOP, BLMOVEa claim: set … order … limit … returning, Fenec-Wait
Delayed jobsZADD by time, ZRANGEBYSCORE polleda run_at ahead, the claim woken at it
Streams, consumer groupsXADD, XRANGE, XREADGROUP, XACKa collection, /_changes and its consumers
Pub/sub, keyspace notificationsPUBLISH, SUBSCRIBE, notify-keyspace-eventsa row with a short @ttl, a subscription
Idempotency keysSET NX EX on a request's idIdempotency-Key, or @unique and if absent
Places near a pointGEOADD, GEOSEARCH, GEODISTa geo field, @geo (Points and distances)
Counts by the minuteINCR on a key a minutebucket(at, 1m) over the events (Analytics)
A cache that evicts under memory pressuremaxmemory, allkeys-lru, allkeys-lfunot covered: a row lives until its @ttl
ScriptsEVAL, FUNCTIONnot covered: a /batch with require
Bitmaps, Bloom filters, Top-KSETBIT, BITCOUNT, BF.ADD, TOPK.ADDnot covered

What every recipe can count on:

  • Atomic. There is one writer: a statement finds its rows and writes them under the write lock, and a /batch is one block — every statement in it lands, or none does, with no other write between them.
  • Durable by --sync. Under --sync always a write is answered once it is on disk; under the default, 250, a crash loses at most the last 250 ms of writes (Durability).
  • On a replica. A replica is sent each write as the primary wrote it — a counter's new value, never n + 1 — once it is on the primary's disk, in the primary's order. It refuses writes (403), so increments, locks and claims go to the primary; a read sent there with Fenec-After waits for the write it names (Replication). A row past its @ttl is out of a replica's reads as it is out of the primary's, and deleted when the primary's sweep is sent.

Counter

An increment is a set that reads the field it writes.

create collection counters (name text @unique, n int)
put counters {name: "visits", n: 0}
set counters {n: n + 1} where name = "visits"     -- → 1
get counters select n where name = "visits"       -- → 1
set counters {n: n + 5} where name = "visits" returning n   -- → 6: INCRBY's answer
put counters {name: "signups", n: 1} if absent else set {n: n + 1}   -- → 1: made at 1
put counters {name: "signups", n: 1} if absent else set {n: n + 1}   -- → 1: added to
get counters select n where name = "signups"      -- → 2
  • None is lost. Sixteen HTTP clients each incrementing one key 10 000 times ended at 160 000, 102 000 increments a second against fenec-server; sixteen threads in process, 320 000 a second.
  • An increment costs 1.36 us in process.
  • From nothing: coalesce(n, 0) + 1 counts a row with no n yet; n + 1 over a null is null. A key that may not be there yet is made or added to by one statement, an upsert: if absent else set.
  • returning answers the row as written, so the new value comes back with no second read.

In JavaScript, inc() writes the coalesce for you:

import { connect, inc } from '@fenecdb/web/client';

const db = connect('http://127.0.0.1:8080');
await db.from('counters').where('name', 'visits').update({ n: inc(1) });

Python's inc(1), Go's fenecdb.Inc(1), .NET's Computed.Inc(1), and Swift's, Kotlin's and Dart's builders render the same text.

Rate limits

Three ways, each decided by what a write finds: a fixed window, the cheapest; a sliding window, exact to the moment; a token bucket, which lets a burst through up to its size and then a steady rate.

Fixed window

A row a client and window, made if absent and expiring after the window; the increment names the limit, so its answer is the decision.

create collection hits (key text @unique, n int, at timestamp @ttl(2m))
put hits {key: "ip1:29630000", n: 0, at: now()} if absent    -- → 1: the window's row
put hits {key: "ip1:29630000", n: 0, at: now()} if absent    -- → 0: there already
set hits {n: n + 1} where key = "ip1:29630000" and n < 2     -- → 1: allowed
set hits {n: n + 1} where key = "ip1:29630000" and n < 2     -- → 1: allowed
set hits {n: n + 1} where key = "ip1:29630000" and n < 2     -- → 0: over the limit
  • The window is in the key — here the minute, floor(now / 60000), which the client works out — so a new window is a new row, and @ttl lets the old ones go.
  • A request costs its two statements, 3.5 us; send them as one /batch for one round trip.
  • A fixed window lets a burst through at its edge: the limit at the end of one minute and again at the start of the next. The two below do not.

Sliding window

A row a request, expiring after the window, and a read that must find fewer than the limit: offset 2 limit 1 require 0 asks that no third row is there, so it allows three a minute. Sent as one /batch with the request's row, the read refuses the batch (412) once three are, and the row is not written.

create collection calls (key text @hash, at timestamp @ttl(1m))
get calls where key = "ip1" offset 2 limit 1 require 0    -- → no row: allowed
put calls {key: "ip1", at: now()}                          -- → 1
get calls where key = "ip1" offset 2 limit 1 require 0    -- → no row: allowed
put calls {key: "ip1", at: now()}                          -- → 1
-- 30 s later
get calls where key = "ip1" offset 2 limit 1 require 0    -- → no row: allowed
put calls {key: "ip1", at: now()}                          -- → 1
get calls where key = "ip1" offset 2 limit 1 require 0    -- → refused: three in the last minute
-- 30 s later
get calls where key = "ip1" offset 2 limit 1 require 0    -- → no row: the first two are a minute old
  • Exact: the rows of the last minute, counted the moment the request comes, since a row past its @ttl is out of the read at once. The guard and the write are one block under the write lock, so two requests cannot both take the last place.
  • A row a request is what it keeps, as Redis's sorted set keeps a member a request; a limit of thousands a window is a fixed window or a bucket instead.

Token bucket

A bucket holds up to two tokens and gains one a second; a request takes one. The row keeps the tokens and when they were counted, and one set works out what the time since has added — the two timestamps apart are milliseconds — and takes a token only where there is one:

create collection buckets (key text @unique, tokens float, at timestamp @ttl(1h))
put buckets {key: "ip1", tokens: 2.0, at: now()} if absent   -- → 1: a full bucket
set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} where key = "ip1" and tokens + (now() - at) / 1000.0 >= 1   -- → 1: allowed
set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} where key = "ip1" and tokens + (now() - at) / 1000.0 >= 1   -- → 1: allowed, empty now
set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} where key = "ip1" and tokens + (now() - at) / 1000.0 >= 1   -- → 0: refused
-- 1 s later
set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} where key = "ip1" and tokens + (now() - at) / 1000.0 >= 1   -- → 1: one token back
  • The script Redis needs for this is the one statement: the row is read and written under the write lock, so no two requests take the same token.
  • A refused request writes nothing, so the time it waited still counts toward the next token.
  • The @ttl is longer than a refill: a bucket idle for an hour would be full again, so its row may go, and the next request makes it full.

From a client, the two statements go as one /batch, with require 1 on the take so that a refusal is a 412:

import { connect, FenecError } from '@fenecdb/web/client';

const db = connect('http://127.0.0.1:8080');

// true: serve the request; false: answer it 429.
async function allow(key: string): Promise<boolean> {
  try {
    await db.batch([
      ['put buckets {key: $1, tokens: 2.0, at: now()} if absent', [key]],
      ['set buckets {tokens: least(2.0, tokens + (now() - at) / 1000.0) - 1, at: now()} ' +
        'where key = $1 and tokens + (now() - at) / 1000.0 >= 1 require 1', [key]],
    ]);
    return true;
  } catch (e) {
    if (e instanceof FenecError && e.status === 412) return false;
    throw e;
  }
}

Lock with expiry

A lock is a row: taken by the write that makes it, held while it lives, and free again once its time passes, so a holder that dies cannot keep it.

create collection locks (name text @unique, owner text, at timestamp @ttl(30s))
put locks {name: "nightly", owner: "w1", at: now()} if absent      -- → 1: w1 holds it
put locks {name: "nightly", owner: "w2", at: now()} if absent      -- → 0: w2 does not
set locks {at: now()} where name = "nightly" and owner = "w1"      -- → 1: renewed
set locks {at: now()} where name = "nightly" and owner = "w2"      -- → 0: not w2's
del locks where name = "nightly" and owner = "w1"                  -- → 1: released
  • Exactly one wins. The write lock decides: of any number of clients racing, one is answered 1 and the rest 0 — sixteen threads are raced for it in crates/fenec-core/tests/writes.rs, and again once its time has passed.
  • Thirty seconds after its last renewal the row is out of every read, the renewal's answer is 0 — the holder has lost it — and the next put … if absent takes it, the old row deleted in the same statement. The sweeper need not have come by.
  • The check-and-delete is the del: it names the owner, so a holder whose lock lapsed and was taken by another deletes nothing — the script Redis's documentation gives for the release.
  • Costs: refused while held 0.91 us, renewed 2.34, released and taken again 2.67.
Details

if absent passes over a document whose @unique value or id a live row holds, and counts what it wrote: Redis's SET NX. insert refuses the same document with a 409 instead, which serves too, but makes losing the race an error to catch and aborts a /batch it is part of. A lock named by a number can be its id: put locks {id: 1, owner: $1, at: now()} if absent.

Time is the server's clock, and a lock lasts its ttl from its last renewal: renew well within it. As with any lock that expires, a holder paused past its ttl may still believe it holds the lock; give the work it guards a check of its own — a fencing token, below — where that matters.

Fencing tokens

A holder paused past its lease — a long garbage collection, a stalled disk — wakes believing it holds the lock, while another does. A fencing token tells the two apart: a number the lock hands out, one more at every acquisition, which the work it guards carries and the resource checks. Here the lock's row keeps it, so the lock is a row that stays, and the lease is its until:

create collection leases (name text @unique, owner text, until timestamp, fence int)
put leases {name: "report", fence: 0} if absent        -- → 1: the lock's row, made once
set leases {owner: "w1", until: now() + 30000, fence: fence + 1} where name = "report" and (owner is null or until <= now()) returning fence   -- → 1: w1 holds it, token 1
set leases {owner: "w2", until: now() + 30000, fence: fence + 1} where name = "report" and (owner is null or until <= now()) returning fence   -- → no row: held
-- 40 s later
set leases {owner: "w2", until: now() + 30000, fence: fence + 1} where name = "report" and (owner is null or until <= now()) returning fence   -- → 2: w1's lease lapsed
set leases {until: now() + 30000} where name = "report" and owner = "w2" and fence = 2 and until > now() require 1   -- → 1: renewed

The resource keeps the highest token it has been written with, and takes a write only from that token or a newer one:

create collection reports (day text @unique, body text, fence int)
put reports {day: "2026-05-03", fence: 0} if absent    -- → 1
set reports {body: "by w2", fence: 2} where day = "2026-05-03" and fence <= 2 require 1   -- → 1: token 2 is the newest
set reports {body: "by w1", fence: 1} where day = "2026-05-03" and fence <= 1 require 1   -- → refused: w1 woke late, with token 1
set leases {owner: null, until: null} where name = "report" and owner = "w2" and fence = 2 require 1   -- → 1: released
  • The token is counted by the write that takes the lock, under the write lock: no two acquisitions get one number, and none is skipped. Redis's SET NX hands out none: a counter beside the lock is a second key, kept in step with it by a script.
  • The row stays so the count does: under @ttl it would be deleted when it lapsed, its count with it. A lapsed lease is until <= now(), which the next acquisition reads.
  • Waiting for it: sent over HTTP with Fenec-Wait, an acquisition that takes nothing is held at the server until the holder releases it or its lease lapses, whichever comes first — both are a write or a time the server knows of (Holding a write until it can write).
  • A resource in fenecdb checks the token in the write itself, as above. One elsewhere — a file store, another service — needs a check of its own; the token is a number it can keep.

Optimistic concurrency

Read a row with its version; write it back only where the version is still the one read, and move the version on in the same statement — what Redis's WATCH and MULTI are for.

create collection docs (body text, version int)
put docs {id: 7, body: "draft", version: 1}
get docs select body, version where id = 7                                   -- → "draft"
set docs {body: "final", version: version + 1} where id = 7 and version = 1   -- → 1: written
set docs {body: "other", version: version + 1} where id = 7 and version = 1   -- → 0: read again
set docs {body: "other", version: version + 1} where id = 7 and version = 1 require 1   -- → refused: 412, and its batch put back
  • 0 means someone wrote first: read the row again and retry, or tell the user.
  • In a /batch, add require 1: a 0 would let the statements after it go on, where a lost race should stop them. With it the write that found another version is refused (412) and the batch is put back whole (Writes that must find their rows). A get … require 1 guards a batch on a value it does not write (A read that must find its rows).
  • Costs: a write that wins 1.13 us, one that loses 0.66.

Session store

A session is a row that lives while it is used: @ttl on the time it was last seen, a @unique token to find it by.

create collection sessions (token text @unique, user text, data json, seen timestamp @ttl(30m))
put sessions {token: "s-81f2", user: "ada", data: {"cart": 2}, seen: now()}
get sessions select user where token = "s-81f2"            -- → "ada"
set sessions {seen: now()} where token = "s-81f2"           -- → 1: seen again
set sessions {seen: now()} where token = "s-81f2" returning user, data.cart   -- → "ada": read and seen again, GETEX
del sessions where token = "s-81f2"                         -- → 1: signed out
  • An idle session is gone at once for every read, thirty minutes after it was last seen, and the server's sweeper deletes it one to two minutes later (Rows that expire).
  • Read and renewed in one by set … returning: the row as written comes back, and a session already gone answers no row.
  • Costs, over 10 000 sessions: a read 1.32 us, seen again 2.26.

Cache with expiry

A value kept for a while in front of something slow to work out: read it by its key, and on a miss work it out and keep it. The upsert writes a key that is there and one that is not alike, and starts its time again.

create collection cache (key text @unique, value json, at timestamp @ttl(5m))
get cache select value.name where key = "user:7"      -- → no row: a miss
put cache {key: "user:7", value: {"name": "Ada"}, at: now()} if absent else set {value: new.value, at: now()}   -- → 1: kept for five minutes
get cache select value.name where key = "user:7"      -- → "Ada": a hit
put cache {key: "user:7", value: {"name": "Ada L."}, at: now()} if absent else set {value: new.value, at: now()}   -- → 1: written over
del cache where key = "user:7"                         -- → 1: invalidated
put cache {key: "user:7", value: {"name": "Ada"}, at: now()} if absent else set {value: new.value, at: now()}   -- → 1
-- 300 s later
get cache select value.name where key = "user:7"      -- → no row: five minutes old

What it is, and what it is not:

  • No eviction. There is no LRU or LFU and no maxmemory policy: a row lives until its @ttl or a del, however rarely it is read. What the cache holds is what was written within its ttl. With --max-memory, a write past the ceiling is refused (507) rather than make room by dropping another row (Limits). A cache that must stay inside a memory budget by dropping what is used least wants Redis.
  • It survives a restart. The rows are in the file, durable by --sync, so a server started again answers from a warm cache, and a replica holds it too.
  • The values are in the file, not the heap. The file is mapped: the process holds the indexes derived from the rows, and the rows themselves are pages the system keeps or lets go — a 1 GB file of 2.3 million rows with two indexes held 188 MB (The file is mapped).
  • Expired rows go in the background. Out of every read at once, deleted by the server's sweeper one to two minutes later, 1 000 a block: 100 000 of 200 000 rows went in 0.56 s, each block holding the write lock 1.06 ms at the median and 2.3 at most (make ttl-bench). The space an overwrite leaves is taken back by the server's compaction once half the file is dead (Compaction).
  • Many clients missing one key at once each work it out; to have one do it, the first takes the key's lock and the rest read again after.

Leaderboard

A score field with @sorted: the top of the board is a walk of the index from its end, and a player's rank is a count.

create collection scores (player text @unique, score int @sorted)
put scores [{player: "ada", score: 0}, {player: "bo", score: 90}, {player: "cy", score: 40}]
set scores {score: score + 120} where player = "ada"            -- → 1: ZINCRBY
get scores select player, score order score desc limit 10       -- → "ada": ZREVRANGE 0 9
get scores select player, score order score desc limit 10 offset 1   -- → "bo": the next page
get scores select score where player = "ada"                    -- → 120: ZSCORE
get scores where score > 120 count                              -- → 0: ada is first, ZREVRANK
get scores select player where score >= 40 and score <= 100 order score desc   -- → "bo": ZRANGEBYSCORE
  • A rank is a count of the scores above the player's: read the score, then count, two statements — or one, where the score is known.
  • Ties come out in the order the players were first written, their ids', whichever way the board is read; Redis orders them by the members' names.
  • Costs, over 100 000 players: a score added 1.79 us, the top ten 1.52. A player's rank is a count of the scores above it, the index's range gathered: 1.5 ms at the median of the board.

Sets and deduplication

A member is a row, and @unique over the set and the member keeps it there once: if absent adds it and answers whether it was new.

create collection members (list text @hash, member text, @unique(list, member))
put members {list: "tags:7", member: "rust"} if absent      -- → 1: SADD, new
put members {list: "tags:7", member: "rust"} if absent      -- → 0: there already
put members [{list: "tags:7", member: "db"}, {list: "tags:8", member: "rust"}] if absent   -- → 2
get members where list = "tags:7" and member = "rust" count   -- → 1: SISMEMBER
get members where list = "tags:7" count                       -- → 2: SCARD
get members select member where list = "tags:7" and member in (get members select member where list = "tags:8")   -- → "rust": SINTER
del members where list = "tags:7" and member = "db"           -- → 1: SREM
  • A set's members are its @hash bucket, read without the others: SMEMBERS is get members select member where list = $1.
  • Seen-it sets — message ids, a feed's items already shown — are a collection of one @unique key and a @ttl, so an id is remembered for as long as a repeat can come.
  • Exact, at a row a member: a Bloom filter's fixed memory and its false positives are not here.

Unique visitors

approx_count_distinct is HyperLogLog with 214 registers — Redis's register count, so its standard error, 0.81% — counted over the visits as they were written. count(distinct) is the exact count, up to a million values a query.

create collection visits (user text, page text @hash, at timestamp @sorted)
put visits [{user: "ada", page: "/", at: "2026-05-03T09:00:00Z"}, {user: "bo", page: "/", at: "2026-05-03T09:05:00Z"}, {user: "ada", page: "/docs", at: "2026-05-03T10:00:00Z"}, {user: "cy", page: "/", at: "2026-05-04T08:00:00Z"}]   -- → 4: PFADD, a row a visit
get visits select approx_count_distinct(user) as users                   -- → 3: PFCOUNT
get visits select approx_count_distinct(user) as users, bucket(at, 1d) as day group day   -- → 2: on the 3rd
get visits select count(distinct user) as users where page = "/"         -- → 3: exact

A day's sketch kept in a row is Redis's key a day: read out as bytes with hll_accumulate, put back as a client sends rows, and merged — PFMERGE — by hll_combine, each visitor of two days counted once:

create collection daily (day timestamp @unique, users bytes)
get visits select bucket(at, 1d) as day, hll_accumulate(user) as users group day   -- → "2026-05-03T00:00:00Z"
put daily $1                                                                       -- → 2: the rows above
get daily select hll_estimate(hll_combine(users)) as users                        -- → 3: both days, ada once
get daily select hll_estimate(users) as users where day = "2026-05-03"            -- → 2
  • Costs: per hour over a million events, 121 ms against count(distinct)'s 128, the furthest of the 169 hours 1.3% off (make analytics-bench).
  • Sketches are the same bytes wherever they are made, the browser's or a server's, and a group of few values keeps few words rather than 16 KB (Distinct users).

Job queue

A job is a row. A worker claims a batch with one statement, which finds the oldest ready jobs and makes each its own for a lease, and acks a job by deleting it.

create collection jobs (payload json, run_at timestamp @sorted, owner text, attempts int)
put jobs [{payload: {"to": "ada"}, run_at: now(), attempts: 0}, {payload: {"to": "bo"}, run_at: now(), attempts: 0}]
set jobs {owner: "w1", run_at: now() + 30000, attempts: attempts + 1} where run_at <= now() order run_at limit 10 returning id   -- → 1: both, w1's for 30 s
del jobs where id = 1 and owner = "w1" require 1                       -- → 1: done
  • None is taken twice: the claim finds its jobs and writes them under the write lock. A job whose worker died is ready again once its lease lapses, and the next claim takes it — at least once, where BRPOP loses a job its worker dies holding.
  • Costs, over a million jobs: a claim of ten 18.7 us; sixteen threads claimed and acked them all in 5.4 s, 185 800 jobs a second, and sixteen HTTP clients at 138 500 (make queue-bench).
  • BLPOP: sent over HTTP with Fenec-Wait, a claim that finds no job is held at the server until one is enqueued, a delayed one comes due or a lease lapses, and handed to the worker that has waited longest: a job enqueued reached a waiting worker in 0.38 ms at the median (Waiting for a job).
  • Retries with a backoff, dead letters, delays and priorities are on their own page, Job queues.

Delayed jobs

A job delayed is one whose run_at is ahead: no claim takes it before then, and the claim that would take it is the same one.

create collection reminders (payload json, run_at timestamp @sorted, owner text, attempts int)
put reminders {payload: {"to": "ada"}, run_at: now() + 60000, attempts: 0}   -- → 1: due in a minute
set reminders {owner: "w1", run_at: now() + 30000, attempts: attempts + 1} where run_at <= now() order run_at limit 10 returning id   -- → no row: not yet
-- 60 s later
set reminders {owner: "w1", run_at: now() + 30000, attempts: attempts + 1} where run_at <= now() order run_at limit 10 returning id   -- → 1: due
  • No poll: a claim held with Fenec-Wait is woken at the moment the next job comes due, which the server works out from the @sorted index: 0.04 ms after its time at the median and 1.9 at p99, against 1.2 and 8.4 polling every 10 ms (Waiting for a job).
  • No ceiling on the delay, and a delayed job is read like any row: get reminders where run_at > now() count.

Streams

A stream is a collection written to and never written over: each entry a row, its id counted up as it is added. A range is a walk of the ids from the last one read, and old entries go by @ttl.

create collection activity (kind text @hash, user text, at timestamp @ttl(7d))
put activity {kind: "signup", user: "ada", at: now()}                                         -- → 1: XADD
put activity [{kind: "order", user: "ada", at: now()}, {kind: "order", user: "bo", at: now()}]   -- → 2
get activity select id, kind where id > 1 limit 100      -- → 2: XRANGE, after entry 1
get activity where kind = "order" count                  -- → 2

A reader that must see every write once, and pick up where it left off, reads the server's change stream: every write on disk, a JSON line each. Started with --cdc, a named consumer keeps its place on the server, as a consumer group's last delivered id does:

fenec-server --file data.fenec --http 127.0.0.1:8080 --cdc

curl -X POST http://127.0.0.1:8080/_changes/consumers/mailer                  # XGROUP CREATE … $
curl -i 'http://127.0.0.1:8080/_changes?consumer=mailer&wait=10000'          # XREADGROUP … BLOCK 10000
# Fenec-Next: 42 -- handle the lines, then XACK up to there:
curl -X POST http://127.0.0.1:8080/_changes/consumers/mailer -d '{"since": 42}'
  • At least once where the server keeps the place: a consumer that crashes between reading and committing is handed those writes again (Consumers). One that keeps Fenec-Next beside its own state, committed with it, has each write once (Change data capture).
  • Only what is on disk: a write is handed out once an fsync has covered it, so none a crash lost reaches a consumer.
  • Every collection's writes are in it, each line naming its collection, puts, deletes and schema changes alike; a consumer passes over the rest.
  • One consumer, every write. A consumer group whose consumers share the entries — each to one of them, the pending ones claimed again after a while — is a job queue: a claim and its lease.
  • How far back: as many writes as --replication-buffer holds, 64 MiB unless given; a cursor further back is answered 410 with the first since it does serve, never with writes missing from the middle.

Pub/sub and change notification

A message is a row, kept for as long as a late subscriber should still see it, and a subscription is sent the rows of a shape — first those there are, then each change as it lands.

create collection messages (room text @hash, body text, at timestamp @ttl(1m))
put messages {room: "lobby", body: "hello", at: now()}     -- → 1: PUBLISH
# SUBSCRIBE: the rows of a shape, then each change to them as it lands
curl -N 'http://127.0.0.1:8080/messages/changes?room=eq.lobby'

# every write on disk, from where a named consumer left off
curl 'http://127.0.0.1:8080/_changes?consumer=mailer&wait=10000'
import { connect } from '@fenecdb/web/client';

const db = connect('http://127.0.0.1:8080');
const stop = db.subscribe('messages', { room: 'eq.lobby' }, (ev) => {
  if (ev.type === 'change') for (const m of ev.puts) console.log(m.body);
});
await db.run('put messages {room: $1, body: $2, at: now()}', ['lobby', 'hi']);
stop();
  • A subscription (Subscriptions) is woken by the write, not by polling: from a write sent to its event read by a subscriber took 0.037 ms at the median and 0.103 at p99 over HTTP on one machine.
  • A late subscriber is sent the messages still inside their @ttl first, where Redis's pub/sub hands it only what is published after it subscribed.
  • Fan-out costs the writer. Every write wakes every stream, which reads what it changed: a write's p50 went from 0.26 ms with no stream to 0.94 with 60 and 3.1 with 500, each stream a connection and a thread, 64 at most unless --http-max-streams says otherwise. A page thousands view polls instead (Polling a read).
  • A live query keeps a query's answer rather than a shape's rows: db.live(query, cb) runs it again on the server after every write to what it reads (JavaScript).
  • The change stream (Change data capture) hands out every write on disk exactly once to a consumer that sends back where it is, which a subscription — a shape's rows kept up to date — does not promise: notify-keyspace-events with a cursor.

Idempotency keys

A client that sends a write again after a timeout, not knowing whether the first landed, sends it with the same Idempotency-Key: the server keeps the key and the answer in the same block as the write, and a repeat is handed the first answer.

curl -X POST http://127.0.0.1:8080/query \
  -H 'Idempotency-Key: 5f0c2d6e-order-1142' \
  -d '{"query":"put orders {customer: $1, total: $2}","params":[7, 129.5]}'
# sent again with the key: the first answer, Idempotent-Replayed: true, nothing written
  • The key and the write land together, so a crash cannot leave one without the other; a failed write keeps no key, and the key with another request is refused (422). Keys are kept a day unless --idempotency-ttl says otherwise (Retrying a write).
  • Costs: 16 200 single puts a second over HTTP with a key, against 18 300 without.

A message that arrives more than once — a webhook redelivered, an event replayed — is told apart by its own id: a row of it, made if absent, and the work done only where it was.

create collection deliveries (event text @unique, at timestamp @ttl(1d))
put deliveries {event: "evt_1J8", at: now()} if absent             -- → 1: first time, handle it
put deliveries {event: "evt_1J8", at: now()} if absent             -- → 0: a repeat, already handled
put deliveries {event: "evt_1J8", at: now()} if absent require 1   -- → refused: in a /batch, its work put back with it
  • With require 1 in a /batch beside the writes it guards, the mark and the work land together or not at all: a repeat is refused (412) and writes nothing, and a crash cannot leave work done with no mark.

Places near a point

GEOADD and GEOSEARCH are a geo field with @geo: the metres between two points, the rows within a radius or a box, the nearest first — see Points and distances.

create collection sicily (name text, loc geo @geo)
put sicily [{name: "Palermo", loc: [13.361389, 38.115556]}, {name: "Catania", loc: [15.087269, 37.502669]}]   -- → 2
get sicily select name where distance(loc, [15, 37]) <= 200000 near loc [15, 37] limit 10   -- → "Catania"
  • Costs, over a million points: every point within 1 km 12.6 us, against 34.3 us for Redis's GEOSEARCH, and the ten nearest 33 us, against 198 ms — the server's own time for Redis, the network left out (make geo-bench, What it costs).

What fenecdb does not do

  • Evict. No LRU or LFU, no maxmemory policy: a row lives until its @ttl or a del, and --max-memory refuses a write past it (507) rather than drop another row (Cache with expiry). A cache that must stay inside a memory budget by forgetting what is used least wants Redis.
  • Keep its data in memory alone. A server always keeps a file, and a write is as durable as --sync says: the default, 250, answers before the fsync and loses at most the last 250 ms in a crash; always answers after it, a durable update 4.0 ms at the median on a Mac (F_FULLFSYNC) and 0.325 ms on Linux in Docker (One request's round trip); off syncs at shutdown alone (Durability). A request is HTTP: a read by id took 0.022 ms at the median on one machine (A request over HTTP), and a network's round trip adds to it. Memory alone, a function call a statement, is the engine embedded in the app — in Rust, where an increment costs 1.36 us, or in the page (Embedded Rust, JavaScript).
  • Run scripts. No Lua, no EVAL or FUNCTION. What a script checks before it writes is require on the statements of a /batch, which lands whole or not at all, and a value worked out over the row is a set's expression or case. A batch cannot read a value and branch on it: a decision between two reads is the client's, a round trip each.
  • Write from more than one writer. A file has one: writes run one after another under its lock — 320 000 increments a second in process and 102 000 over HTTP from sixteen clients (make counters-bench), 185 800 jobs claimed and acked a second (make queue-bench) — and reads go beside them: four threads read 701 000 rows a second by id beside four writers (Writers and readers at once). Past one machine's writer the data is split by tenant, a file each (Tenants and sharding); there are no hash slots.
  • Fan out across tenants. A subscription and /_changes are one file's: on a tenant node each tenant is a file of its own, and the router forwards a stream to the node holding it. There is no channel every tenant's subscribers hear; a message for all of them is a write to each.
  • Pub/sub without storage. Every message is a write that lands in the file and moves the change counter; there is no fire-and-forget channel. A row with a short @ttl and a subscription is the nearest (above).
  • Speak RESP. A Redis client cannot connect: the server speaks HTTP, and every language reaches it through it or an SDK (Your language).
  • Bitmaps, Bloom filters and Top-K. No SETBIT or BITCOUNT, no Bloom or Cuckoo filter, Count-Min sketch or Top-K. Membership is exact, a row a member (Sets); HyperLogLog is there (Unique visitors).
  • Deliver exactly once. A queue's lease that lapses under a slow worker hands its job to another, and a consumer that crashes before it commits reads again: the work's own check — a @unique key, an Idempotency-Key — makes the second run harmless.