Datadog and other monitoring systems

A counter loses its _total and arrives as a count, fenecdb.<name>.count; a gauge arrives under its name. A histogram's buckets arrive, with histogram_buckets_as_distributions, as one distribution, fenecdb.<name>.bucket, whose percentiles Datadog takes over every server's buckets together — as Prometheus's histogram_quantile does — once they are turned on in the metric's summary; its .sum and .count give the mean. Every label is a tag: kind, collection, route, code, node, outcome, version.

No label names a tenant. A tenant node publishes its tenants' counts together (fenec_tenants, fenec_tenants_open, fenec_tenants_disk_bytes): a tenant's collection names are not the node's to publish, and a tag per tenant would be as many series as there are customers, which Datadog bills by. So a node's role goes into the instance's tags — fenec_role:tenant-node, fenec_node:n1 — and a tenant's own figures come from the logs, whose lines carry a tenant attribute, and from /t/<tenant>/_stats/statements.

The token goes in the Authorization header: kept out of the file with the Agent's secrets management ("Bearer ENC[fenec_metrics_token]"), or under Autodiscovery as "Bearer %%env_FENEC_METRICS_TOKEN%%". The metric names fenecdb publishes are in Monitoring.

Every line has at, event (slow, refused, schema or admin), level, proto, peer — with via, the router, where one forwarded the request — and the request's request_id. A slow statement adds duration_ms, kind (read or write), failed, tenant on a tenant node, and statement; a refusal method and path; a schema change statement, by its shape, tenant and failed; an admin request method, path and http_status — not status, which Datadog takes for the line's level before level.

stderr also holds the server's own text lines — that it listens, what it linked, an error. One written while a request was served ends with request_id=<id>, which the pipeline's first processor, a grok parser over the message, takes out. Under Docker, the container's log is stderr: type: docker with the same source and service, or the label com.datadoghq.ad.logs='[{"source":"fenecdb","service":"fenecdb"}]'.

--slow-ms 0 logs every statement, as PostgreSQL's log_min_duration_statement = 0 does: a pipeline is tried out on what a server answers now. A test starts fenec-server, makes it write each kind of line, and fails if the pipeline or the dashboard reads an attribute none of them has. The audit log and the slow statements are described in Monitoring.

Its groups, each on the metrics /_metrics sends:

• Requests: statements a second by kind, errors, p50, p95 and p99 by kind, the mean, slow statements, refusals (401) and connections.
• Writes and storage: writes a second — the rate of fenecdb.change_sequence — documents and each collection's dead share, the file and what a compact would give back, compactions --auto-compact ran, and vectors waiting to be linked.
• Replication: a replica's lag in writes, whether it is connected, since it last heard from its primary, a primary's replicas, reconnects and whole images.
• Tenants: on disk and open, their files, and the tenants placed on each node.
• Router: requests by route and status class, its p99 against its nodes', nodes it could not reach, moves and refusals.
• Logs: the slow statements, the refusals, and the schema changes and admin requests, each with its request id.

The host template variable picks one server or all of them. The percentiles need the distribution's percentiles turned on, once, in the metric's summary.

A load balancer or a gateway that sets X-Request-Id has its id carried through to fenecdb's logs; a client that sends none can read the one made for it off the answer — a subscription's and a replication stream's too — and report it. A made id is 64 bits of a mix of a key the process reads from the system once, the connection's thread and its count of requests: never the same twice within a process, and 15 ns to make, 12 to keep a client's. The id lives with the connection's thread, as the audit log's peer does, so nothing between the request's reading and a line's writing carries it. Making it, sending it and logging it cost a request nothing measurable: in make requests-bench, eight runs in turns, one client's round trip for a row by id stayed 0.016–0.022 ms either way.

Spans go as OTLP over HTTP with JSON, port 4318; the Agent's gRPC port, 4317, is not for them. A header a collector asks for goes with --otlp-header k=v or OTEL_EXPORTER_OTLP_HEADERS. With no endpoint, tracing is off and costs nothing; on, a request never waits for the Agent, and what the queue cannot hold is counted in fenec_trace_spans_dropped_total (Monitoring).