Goodbye SQLite (for logs)

When we started Vibe Kanban, we picked SQLite as the primary datastore. It made sense: the app was fully local, and we had structured data that needed to be indexed - workspaces, repositories, session IDs.

But we also put the unstructured stuff in there too, eg raw stdout/stderr from coding agents, setup/cleanup scripts and dev servers.

Recently we started to regret that decision.

• "database is locked" kept showing up under real workloads.
• In SQLite, writes effectively serialize behind a database-level lock, so long writes can stall unrelated reads and writes.
• WAL mode can help readers, but it still doesn't solve parallel writer contention - and we couldn't use it anyway because it broke our SQLite pre-update-hook-based sync framework.
• When measured, ~98% of our DB size was logs, and lock incidents correlated with large logs being written in parallel.
• The logs table was basically a simple key-value store for append-only JSONL, not something benefiting from relational querying.

So we've decided to move logs out of SQLite and onto the filesystem: one file per execution, stored at:

/<first two chars of session id>/<session id>/<execution_process_id>.jsonl

After the migration:

• The DB is ~98% smaller (from gigabytes down to ~20 MiB).
• Lock errors have disappeared.
• Reads are faster because we're no longer fishing through a giant indexed logs table.

We thought it could be worth writing this as we see other open source projects dabbling with SQLite for logs. Don't do it!