Mobile and native apps

Keep the database in a file on the device: no server, no network, the same engine and file format as the server and the browser.

An iOS, macOS, Android or Flutter app opens a fenecdb file, writes to it and asks it questions in FenecQL or through the query builder, with near, match and lookup as anywhere else. A live query hands the UI its rows again after every write to what it reads.

A todo list

The file in the app's own directory, a collection made once, a list on screen that is a live query, and writes from the buttons. The list redraws itself: nothing calls it after a write.

import FenecDB
import SwiftUI

struct Todo: Codable, Sendable, FenecValue {
    var id: Int?
    var title: String
    var done: Bool
}

@main struct TodosApp: App {
    @State var db: Fenec?

    var body: some Scene {
        WindowGroup {
            if let db { TodoList(db: db) } else { ProgressView().task { db = try? await open() } }
        }
    }

    func open() async throws -> Fenec {
        let dir = try FileManager.default.url(for: .applicationSupportDirectory, in: .userDomainMask,
                                              appropriateFor: nil, create: true)
        let db = try await Fenec.open(dir.appendingPathComponent("todos.fenec"))
        try await db.execute("create collection if not exists todos (title text, done bool @hash)")
        return db
    }
}

struct TodoList: View {
    let db: Fenec
    @StateObject var open: LiveQuery<Todo>

    init(db: Fenec) {
        self.db = db
        _open = StateObject(wrappedValue: LiveQuery(db, try! db.from("todos").where("done", false), as: Todo.self))
    }

    var body: some View {
        List(open.rows, id: \.id) { todo in
            Button(todo.title) {
                Task { try await db.from("todos").where("id", todo.id!).update(["done": true] as Value) }
            }
        }
        .toolbar {
            Button("Add") { Task { try await db.from("todos").insert(Todo(title: "milk", done: false)) } }
        }
    }
}

import com.fenecdb.*

class TodosModel(app: Application) : AndroidViewModel(app) {
    val db = Fenec.open(File(app.filesDir, "todos.fenec").path)   // in a background start-up, or openAsync

    init {
        db.executeBlocking("create collection if not exists todos (title text, done bool @hash)")
    }

    val open: Flow<List<Row>> = db.live(db.from("todos").where("done", false))

    fun add(title: String) = viewModelScope.launch { db.from("todos").insert(mapOf("title" to title, "done" to false)) }

    fun finish(id: Long) = viewModelScope.launch { db.from("todos").where("id", id).update(mapOf("done" to true)) }

    override fun onCleared() = db.close()
}

@Composable
fun Todos(model: TodosModel) {
    val open by model.open.collectAsState(emptyList())
    LazyColumn {
        items(open) { todo -> Text(todo.string("title") ?: "", Modifier.clickable { model.finish(todo.long("id")!!) }) }
    }
    Button(onClick = { model.add("milk") }) { Text("Add") }
}

import 'package:fenecdb_flutter/fenecdb_flutter.dart';
import 'package:flutter/material.dart';
import 'package:path_provider/path_provider.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  final dir = await getApplicationSupportDirectory();
  final db = await Fenec.open('${dir.path}/todos.fenec');
  await db.execute('create collection if not exists todos (title text, done bool @hash)');
  runApp(MaterialApp(home: Todos(db)));
}

class Todos extends StatelessWidget {
  final Fenec db;
  const Todos(this.db, {super.key});

  @override
  Widget build(BuildContext context) => Scaffold(
        body: StreamBuilder(
          stream: db.live(db.from('todos').where('done', false)),
          builder: (context, snap) => ListView(children: [
            for (final todo in snap.data ?? const <Map<String, Object?>>[])
              ListTile(
                title: Text(todo['title'] as String),
                onTap: () => db.from('todos').where('id', todo['id']).update({'done': true}),
              ),
          ]),
        ),
        floatingActionButton: FloatingActionButton(
          onPressed: () => db.from('todos').insert({'title': 'milk', 'done': false}),
          child: const Icon(Icons.add),
        ),
      );
}

A fuller app, Notes -- search with match and fuse, filters, a schema file, an add form -- is in the repository for each: SwiftUI (and a CLI), Jetpack Compose (and a JVM CLI) and Flutter, each built on every change and its headless part run.

The builder

The same query builder as JavaScript, Python, Go and .NET: from, select, where with every operator, near, match, fuse, rerank, highlight, snippet, facet, lookup, order, limit, and the writes, each value a parameter and each name checked. All seven make the same text of the same chain, to the byte, held to one golden file; each language's spelling is on Your language. A builder query also knows the collections it reads, which is what a live query runs again for. To see the text and parameters a chain builds, for logging or a test, toFenecQL() returns them and runs nothing.

let near = try await db.from("notes")
    .select("title")
    .where("lang", "en")
    .near("embed", embedding, ef: 64)   // a [Float]
    .limit(5)
    .rows(as: Note.self)

The schema in code

An app that makes its database declares it once, as create collection and create index statements -- a schema.fenecql in the app's resources -- and hands it to schema after each open. The engine compares the file with it: what only adds (a collection, a field, an index) is made, the migrations not yet recorded run first, in order, and are kept in _migrations, all one block; a field the text lacks, a type or an index changed, is refused with what resolves it, and nothing is written. It is the check the browser's Fenec.open({ schema }) and the server's /_schema run (Schema in code).

let plan = try await db.schema(schemaText, migrations: [
    "alter collection notes rename field title to name",
])

val plan = db.schema(schemaText, migrations = listOf(
    "alter collection notes rename field title to name",
))

final plan = await db.schema(schemaText, migrations: [
  'alter collection notes rename field title to name',
]);

fenec types --lang swift|kotlin|dart schema.fenecql writes a row of each collection as the language reads it: a Codable struct, a data class with from(row), a class with fromJson.

Live queries

A live query hands over its rows at once, and again after a write to a collection it reads. The writes of a burst -- several at once, a loop, a text of several statements -- run it once, a frame after the last of them lands. A write to another collection runs nothing, and a text that fails is put back whole and runs nothing either.

Sync with a server

The todo list kept in step with a server, every device seeing every change: open the file as a replica of the rows you want. That is the one line that changes -- the list is still a live query of the file, and a tap still writes to it.

let db = try await Fenec.sync(url: "https://api.example.com", token: jwt,
                              shapes: [Shape("todos", key: "key")],
                              path: dir.appendingPathComponent("todos.fenec").path)

val db = Fenec.sync(url = "https://api.example.com", token = jwt,
                    shapes = listOf(Shape("todos", key = "key")),
                    path = File(app.filesDir, "todos.fenec").path)

final db = await Fenec.openSynced(url: 'https://api.example.com', token: jwt,
                                  shapes: [Shape('todos', key: 'key')],
                                  path: '${dir.path}/todos.fenec');

The collection is made on the server, with a key the app's inserts are matched by -- create collection todos (key text @unique, title text, done bool @hash) -- and comes to the replica from there, so the create collection line goes. Dart's is openSynced: sync() is the fsync, and Dart has no static and instance member of one name.

• Reads are the file's. A query, a live query and the builder read the replica: no network, and the app works offline.
• A shape is a set. The rows of one collection a filter on the server picks, Shape("todos", where: ["owner": me]), in the JS sync layer's form; one shape a collection. Order and limits are a query's, not a shape's.
• A write shows at once. It is applied to the replica as it is made, and sent. An insert gets a temporary id and a key, and the server's copy takes its place when it arrives.
• A refused write is put back. When the server refuses it -- a 409 for a unique value held, a 403 from a token's rules, a 507 from a tenant's quota of bytes or rows (quota names it) -- the replica is put back as it was and the app is told: db.replica.refusals.
• Offline, writes wait. A write made while the server cannot be reached waits in the file, and goes once it can, in order, once each. An app closed meanwhile sends it when it opens again.
• It catches up from where it stopped, after a dropped connection, a server restart or a week in a drawer -- or is sent the whole shape again when the server's ring of changes no longer reaches back that far.

The status says online, offline or catchingUp (connecting, seeding, or sending the writes left from before), how many writes the server has not answered, and the last error.

TLS on devices

iOS (App Transport Security) and Android refuse plain http:// by default, and so should you: put TLS in front of fenec-server. Caddy does it in three lines, certificate included:

api.example.com {
    reverse_proxy 127.0.0.1:8080
}

Each binding makes its requests with the platform's client -- URLSession, HttpURLConnection, dart:io's HttpClient -- so the system's trust store, proxies and pinning apply. An Android app declares the INTERNET permission and a sandboxed macOS app the com.apple.security.network.client entitlement, as for any request. During development, an emulator reaches a server on the machine at plain http:// once allowed: iOS's NSAllowsLocalNetworking, Android's network_security_config with cleartextTrafficPermitted for 10.0.2.2.

Battery and the background

A replica holds one connection a shape open while the app runs, quiet but for the keepalive. iOS suspends an app soon after it leaves the screen, and Android stops one's network in Doze: the streams are cut, and nothing is lost -- a write waits in the file, a change waits on the server. As the app comes back, call resume() (on scenePhase == .active, onStart, AppLifecycleState.resumed): the streams open at once from their cursors, the waiting writes go first, and the live queries run again as the changes land.

Without a file: the server alone

An app that keeps nothing on the device asks the server directly: Fenec.connect makes every query a request, through the same builder, as connect does in JavaScript.

let db = Fenec.connect(url: "https://api.example.com", token: jwt)
let open = try await db.from("todos").where("done", false).rows()

Kotlin's is Fenec.connect(url, token = jwt), Dart's Fenec.connect(url, token: jwt); a replica's replica.remote is the same, for what the replica does not hold. Live queries are the file's alone.

Several writes that must land together are one batch (POST /batch): the builder's toInsert, toUpdate and toDelete under one write lock, all of them or none. One that stops it is an error naming its place (at, from 0) and the HTTP status -- 412, code unmet, for a write whose require was not met. A retry after a timeout cannot tell whether the write ran, so a write that must not run twice goes under an idempotency key: the second answer is the first one kept (replayed), and the key sent with another request is refused (422).

let accounts = try db.from("accounts")
let done = try await db.batch([
    try accounts.where("id", from).where("balance", ">=", 10).toUpdate(["balance": .inc(-10)] as Value, require: 1),
    try accounts.where("id", to).toUpdate(["balance": .inc(10)] as Value, require: 1),
], idempotencyKey: transferID)
// done.results, done.seq, done.replayed
try await db.withIdempotencyKey(orderID).from("orders").insert(order)

Kotlin's is db.batch(listOf(...), idempotencyKey = id) and runList(text, params, idempotencyKey = id), Dart's db.batch([...], idempotencyKey: id); all three have withIdempotencyKey(key), a copy whose every write -- the builder's too -- carries the key, and seq, the change the last write left the server at. withWait -- Swift's in seconds, Kotlin's in milliseconds, Dart's a Duration -- is a copy whose sets and dels the server holds while they write nothing, a worker's claim waiting there for a job (Waiting for a job); cancelling the task or the coroutine gives it up.

Where the file lives

Anywhere the app may write. On iOS the app's Application Support directory, which is backed up and never shown in the Files app; on Android context.filesDir, or noBackupFilesDir to keep it out of a backup; in Flutter path_provider's getApplicationSupportDirectory(). Beside the file the library keeps <name>.lock while it is open.

• Open a file once. A second open of the same file, in the app or in another process -- an app extension sharing the app group's container -- is refused (locked): two databases over one file corrupt it.
• iOS file protection. A file under the complete protection class cannot be read while the device is locked, and a page of a mapped file read then is the app's end, not an error. Keep the default class (complete until first user authentication), or open the file with .inMemory.
• The file compacts itself. An update writes the row again at the file's end; a thread of the library's looks every 5 s and compacts the file beside your calls once half of it is versions updates and deletes left behind, and at least 64 MB, so a file you keep updating stays under about twice what it holds -- sooner once it reaches three quarters of the memory the device has for its pages, so its reads stay in memory. Open with .noAutoCompact (NO_AUTO_COMPACT, noAutoCompact) to compact only when you run compact.

Durability

Every write is on disk before the call returns: written and fsynced, outside the database's lock, so reads go on meanwhile and writes arriving together share one fsync. On a Mac that fsync is about 4 ms. For a burst of writes, open the file without it and sync when they are done:

A phone ends an app in the background without a word, so an app opened with noSync calls flush() or sync() as it goes there -- scenePhase turning to .background, onStop, AppLifecycleState.paused.

Threads

One database is safe to use from every thread, task or coroutine at once: a read runs beside other reads, a write takes the database to itself, a long read — an aggregate, a scan, a text of several reads — pins what it reads and lets writes land beside it, as a server's does, and a lone create index or compact is built beside the database while reads and writes go on. Every binding makes its calls off the main thread, since any of them may wait -- for the lock, an fsync, a whole open.

Sizes and speed

The sync with a server is 165 to 198 KB of each. iOS's and Android's are measured by CI for every change, in the run's summary. An app ships the code it links: the XCFramework's static library is the objects before the app's linker takes what it uses.

The C ABI

Any language with a C FFI can call the library: the header is crates/fenec-ffi/include/fenec.h, and make ffi builds the library for this machine, or TARGET=... for another. Every call returns a code -- 0, an error's kind, or the boundary's own: FENEC_MISUSE, FENEC_LOCKED, FENEC_PANIC -- and writes its answer or its error as NUL-terminated JSON, freed with fenec_free_string.