One identity. A history of change.

A record should stay recognizable as its data changes. ShiverDB's proposed engine builds around that simple idea, with immutable versions and one ordered commit history underneath.

The identity stays. The bytes can move.

player:42 names a logical record, not a location on disk. A compact record directory maps that identity, or an internal record slot, to its current physical version.

This gives the engine room to reorganize storage without breaking references. Compaction, segment rewrites, and index rebuilding must preserve the record's identity.

The useful mental model: record X has state Y as of commit Z. Pages and offsets are the engine's concern.

Keep the change, not another full copy.

Committed versions are immutable. A base version contains a complete record. A delta contains changed fields and a link to the previous version. Changing a player's gold should not require copying their biography.

Base · 100name: 'sky', level: 1, gold: 100
Delta · 101gold: 75
Delta · 102level: 2
Conceptual versions for player:42. At snapshot 101, the player has level 1 and gold 75. Later changes are not yet visible.

Multi-version concurrency control, or MVCC, lets a reader observe a logical snapshot. The engine reconstructs the visible record from a base and the relevant deltas.

Delta chains cannot grow without bound. The design calls for new bases when chain length, delta size, or read amplification becomes costly. Exact thresholds remain implementation choices.

History first. Visibility after durability.

The commit log is the authoritative ordering and recovery record, not just a helper beside the tables. It must contain enough information to recover committed mutations even before background storage has caught up.

  1. 01Admit and prepare
  2. 02Evaluate and validate
  3. 03Append and flush
  4. 04Publish in order

Current state and change consumers follow the committed result. Background workers materialize immutable segments.

Group commit can share a durable flush across compatible transactions. A monotonic commit position ties together snapshot visibility and ordered change history.

Not application-level event sourcing. Applications still query and update ordinary current-state records. Append-only mutation history is an internal engine property.

Fast decisions. Room for colder data.

The decision plane

A small, synchronous, deterministic core evaluates domain changes and establishes transaction order over the state it needs.

The data plane

Parallel workers handle storage-heavy work: scans, compaction, historical lookup, blobs, backups, and specialized queries.

They expose one logical database and one transaction history. The entire database does not have to fit in memory.

Admission can prepare dependencies and prefetch cold records. If execution discovers an unexpected cold dependency, the preferred path is to yield or abort provisional work, load the data, and re-admit the command against a current snapshot. A cold fault is a performance event, not permission to change the result.

Access patterns and subscription interest can guide residency. Applications should not need to keep their own cache-pinning map.

Storage that keeps moving.

Row-oriented current state favors point reads and small updates. Immutable segments give compaction room to collapse deltas, reclaim obsolete versions, and improve physical layout. Columnar storage is a possible later optimization for scan-heavy data, not the default transactional representation.

Reclamation must respect active snapshots, replication and changefeed consumers, backups, and configured history retention. An old version is disposable only when nothing still needs it.

Recovery restores durable storage and replays committed log records. Incomplete transactions are discarded. A partial transaction must never become visible.

Prove the foundations first.

The initial proposal prioritizes a crash-safe commit log, correct transactions, a deterministic writer, record directories, B-tree indexes, delta MVCC, and background materialization.

Columnar encoding, specialized search indexes, replication transport, historical query support, and a subscription dependency engine are listed as optional later components. These are design directions, not available features or measured performance results.

Next: the language above the engine

From the specifications

This guide explains the design. The original documents carry the precise requirements and recommendations.