Monitoring

With --http the HTTP listener serves /_metrics as well, and a --dir node serves it there. The path carries an underscore, as /_admin and /_replication do: a collection may well be called metrics, and GET /metrics is how its rows are read. Prometheus takes the path as metrics_path.

A scrape needs the server's token (--http-token) or the admin token (--admin-token); with neither and no JSON Web Tokens the server is open, and so are its metrics. A JWT is not taken: a user held to a policy must not learn the other collections' names from a scrape. A --metrics address outside loopback wants one of the two tokens, as the HTTP listener does, or --insecure.

A statement is timed from its arrival to its answer: the wait for the lock is in it, and under --sync always the wait for the disk, since that is the latency a client sees. A statement is a request, and /batch is one however many statements it carries. A statement that took the write lock is a write, so a get sent through POST /query is a read.

A tenant node counts its tenants and not what they hold: a tenant's collection names are not the node's to publish, and a label per tenant would be as many series as there are customers. The statement counters are one set per process, and start again at zero with it, which rate() takes in its stride.

monitoring/ in the repository holds a Grafana dashboard (grafana/fenecdb.json), a Prometheus configuration and a compose file that runs fenec-server with both, the dashboard already loaded. The same compose seeds a site's events and a week of trades and loads the data dashboards (grafana/fenecdb-data-plugin.json and grafana/fenecdb-data.json) beside it.

It opens on the figures that decide whether anything needs doing — the version and uptime, whether the disk has refused a write, statements and errors per second, the memory footprint — then statements by kind with their p50 and p99, errors, and slow statements a minute; documents, writes a second and each collection's dead share, the one that says when to compact; replication; tenants; and a router's requests, its latency beside its nodes', the nodes it could not reach, the tenants on each node and its moves. The instance variable picks one server or all of them. Imported on its own, the dashboard asks for a Prometheus data source.

# the p99 of reads, over five minutes
histogram_quantile(0.99, sum by (le) (rate(fenec_statement_duration_seconds_bucket{kind="read"}[5m])))

# worth an alert
fenec_storage_failed == 1
fenec_replica_behind > 10000
fenec_dead_bytes / (fenec_data_bytes + fenec_dead_bytes) > 0.5

The log line uses the same timing the histogram has. It is the audit log's shape — at, event, level, the peer and the request id — with the duration in milliseconds, whether the statement wrote (kind), whether it failed, the tenant on a tenant node, and the statement, so one parser reads both logs (Datadog and other monitoring systems). --slow-ms 0 logs every statement, as PostgreSQL's log_min_duration_statement = 0 does.

A parameter's value is not logged, only how many there were: a value bound as $1 stays out of the log, a literal written into the text does not. Over HTTP the line is the method and the target, and for /query and /batch the body that holds the statements. A line is cut at 1 000 bytes, so a bulk put of a megabyte shows what it was without being all of it.

Every statement is also counted by its shape — its text with each literal and parameter written $1, $2… and a list of literals alone, a vector's components, as one — so the same statement run with other values is one line: how often it ran, its total, mean, least and most time, the rows it returned or changed, and its failures. Over HTTP a /query or /batch is counted as its FenecQL; a call to a collection's path as its method and path. GET /_stats/statements answers JSON, most time first, and DELETE forgets it all.

The server keeps at most 5 000 shapes, as PostgreSQL's pg_stat_statements does by default, and forgets the least called when it is full. The counts need what reads the data without a JWT's scope, or the admin token: a shape names every collection. A tenant node keeps each tenant's apart — /t/<tenant>/_stats/statements — and lists every tenant's, named, to its admin alone.

{"at":"2026-10-01T09:30:12.310Z","event":"schema","level":"info","proto":"http","peer":"10.0.0.7:53112","request_id":"5d0e7c2a91f4b366","statement":"create collection notes (title text)","failed":false}
{"at":"2026-10-01T09:32:05.117Z","event":"refused","level":"warn","proto":"http","peer":"198.51.100.4:40031","request_id":"b81f03c6d2e94a57","method":"GET","path":"/notes"}
{"at":"2026-10-01T09:40:00.550Z","event":"admin","level":"info","proto":"http","peer":"10.0.0.2:61200","request_id":"deploy-7","method":"POST","path":"/_shard/tenants/acme/move","http_status":200}

--audit <path> appends one JSON line for each of these events:

• An HTTP request refused for its token (401).
• A statement that changes the schema: create, drop, alter or compact. The statement is logged by its shape, as /_stats/statements shows it, so no literal it held reaches the log.
• A request to /_admin/ or /_shard/ that changes something, with its status. fenec-shard takes --audit as well.

Each line has the time, a level (warn for a refusal, info otherwise), the protocol, the peer's address and the request id; an admin request's status is http_status, since a log collector takes a status for the line's level. Reads and writes are not logged, so the log costs nothing on their path. Lines are written as the events happen and are not synced: after a crash the log can lose its last lines, but the data loses nothing.

A refused token waits before it is answered, the way PostgreSQL's auth_delay works. The first failure from an address within a minute waits --auth-delay (100 ms by default). Each further failure from that address doubles the wait, up to 5 s. This applies to every 401, logged or not. A guessing client gets about one try every five seconds per address. A request whose token the server verifies — its --http-token, a JWT whose signature checks, the admin, replication or scrape token, a router's own --token — waits for nothing and clears its address's count, whatever the answer then says of the rest of the request. Nothing else clears it: an answer given without looking at a token — an unknown path, /_health, a preflight, the studio's files, a body over the ceiling, every answer of a server that asks for no token — leaves the count where it was, whatever Authorization the request carried. The count is the address's, not a token's: a client that holds one good token clears its count with it. Behind fenec-shard the router waits out a refusal itself, keyed by its client's address, and the node waits none for what the router forwards: counted at the node, every client had the router's address, four forged tokens in a row made a fifth client's expired one wait 1.6 s, and any good token from anyone cleared the attacker's count. The node knows a forwarded request by a mark the router derives from the node's --admin-token, sent beside the client's address (Fenec-Router: <address> <mark>); the router drops the header from what a client sends, and a node believes it from no one else. Its audit log names the client as peer and the router as via. The router holds no keys: the node tells it, in its answer, that it verified a forwarded request's token, and the router clears its client's count on that alone. Behind another reverse proxy every client has the proxy's address, so the proxy's own rate limit is the one that counts there. --auth-delay 0 turns the wait off.

fenec-shard serves /_metrics on its own listener, behind its --token as /_shard/ is: a scrape names the nodes. It counts what it answered and what the nodes took over it; what a tenant's statements did is its node's to count, and no label names a tenant. A request is timed from its arrival to its answer, the node's time included — a subscription to its head, since the stream lasts as long as its client stays; a standby router's replication stream is counted and not timed.

# what the router adds, at the median, over five minutes
histogram_quantile(0.5, sum by (le) (rate(fenec_router_request_duration_seconds_bucket{route="tenant"}[5m])))
  - histogram_quantile(0.5, sum by (le) (rate(fenec_router_upstream_duration_seconds_bucket[5m])))

# worth an alert
rate(fenec_router_upstream_errors_total[5m]) > 0
fenec_router_tenants_moving > 0
increase(fenec_router_auto_failovers_total[5m]) > 0
fenec_router_node_failed_over > 0

An id a client sends is kept when it is printable ASCII of at most 128 bytes; any other is replaced, as a header or a log line could not carry it whole. A made one is 16 hex characters. fenec-shard takes the client's id or makes one, sends it to the node, which takes it as its own, and answers with it: one request through the router has one id in the router's log, the node's and the answer. A text line on stderr written while a request was served — a sync error — ends with request_id=<id>. A subscription's and a replication stream's head carry the id too. Making one and writing it cost a request nothing measurable (make requests-bench, in turns).

The server span carries OpenTelemetry's names where one fits: http.request.method, http.route (a template: /t/{tenant}/{collection}/near), url.path, http.response.status_code, server.address and server.port, client.address, db.system (fenecdb), db.operation.name (get, put, create…) and, for FenecQL, db.query.text — the statement's shape, each literal and parameter written $n, as the statements' counts show it, so no value reaches a span. A REST request's target holds values in its query string, so it has its route alone. A --dir node's and the router's spans carry fenec.tenant, the router's fenec.node, and every server span fenec.request_id, the request id its log lines carry. A 5xx marks the server span an error; a 4xx is the client's and does not.

A request's traceparent is read as the W3C spec writes it; one that does not parse starts a trace of its own, and tracestate goes on unchanged. A trace starting at fenecdb is kept when its id's low 56 bits fall under the ratio, OpenTelemetry's TraceIdRatioBased, so a router and its nodes sampling at one ratio keep the same traces. --trace-sample is 1 with an endpoint; a busy server wants less. Health checks, scrapes, preflights and the studio's files are no spans, and a subscription or a replication stream, which lasts as long as its client, has none.

The endpoint is taken from --otlp-endpoint, then OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, both used as given unless they name no path, then OTEL_EXPORTER_OTLP_ENDPOINT, under which /v1/traces goes. --otlp-header k=v (and OTEL_EXPORTER_OTLP_HEADERS) adds a header to every post, an API key a collector asks for; OTEL_SERVICE_NAME names the service (fenec-server, fenec-shard) and OTEL_RESOURCE_ATTRIBUTES adds to it. Spans go as OTLP over HTTP with JSON, POST /v1/traces with application/json, the collector's port 4318: an OTEL_EXPORTER_OTLP_PROTOCOL of grpc is refused at the start rather than sent to a port that cannot read it, and an https:// endpoint likewise.

A finished request's spans go into a queue of OTEL_BSP_MAX_QUEUE_SIZE spans (2 048), and a thread of their own posts it every OTEL_BSP_SCHEDULE_DELAY (1 s), or as soon as OTEL_BSP_MAX_EXPORT_BATCH_SIZE (512) are in, a post at most that many, each bounded by OTEL_EXPORTER_OTLP_TIMEOUT (10 s) and kept alive between posts. A request never waits for the collector: with the queue full its spans are dropped and counted, and a post that fails drops its spans, logged once a minute at most. fenec-server posts the last second's spans as it shuts down.

Counting a statement is a handful of relaxed atomic adds, into counters a thread each: 6.7 ns a statement with eight threads counting at once. Into one shared set of counters the same eight cost 720 ns each, the cache line moving from core to core, which is why they are spread. The router counts a request the same way, 13.3 ns with eight threads counting, beside the 17 µs it adds to the request (make shard-bench). A scrape reads the gauges under the database's shared lock, beside the readers and ahead of a writer for as long as a count would be.

Tracing off is one relaxed atomic load at each hook: in make requests-bench, six runs in turns against the server without it, one client's round trip stayed 0.022, 0.033, 0.059 and 0.021 ms at the median for a row by id, a filter, a near and a put, against 0.022, 0.034, 0.059 and 0.022, and eight clients' requests a second inside the runs' own spread. On, every request sampled, to a collector on the same machine: 0.022, 0.034, 0.060 and 0.024 ms, and eight clients 2 to 9% fewer requests a second — the exporter's thread writing 250 000 spans a second as JSON took a core of the eight the clients and the server shared. A sampled request's spans cost it their timestamps and a few allocations; the JSON and the post are the exporter's.

Counting it by its shape is a pass over its text into a buffer the thread keeps, and an entry found in the thread's shard: 0.34 µs for a read of 87 bytes, 0.56 µs with eight threads counting at once — a sixth of the 2.2 µs parsing it takes. It grows with the text: 5.6 µs for a near with 768 components written into it, whose parse takes 85 (make statements-bench). A shape is made only as far as the 1 000 bytes kept of it, so two statements alike that far are one line: shaped whole, a put of 1 000 rows of 128-dim vectors spent 1.7 ms of its 12.6 there.