Storage Layout

S3WORM uses a configurable storage layout to organize system directories within an S3 bucket. The layout determines where operation logs, snapshots, and trash are stored relative to entity data.

Layout Modes

The storage.layout field in schema.json sets the default layout for all system directories. Three modes are available.

{
  "storage": {
    "layout": "root"   // "inline" | "collection" | "root"
  }
}

Inline

System directories live inside each entity folder. Every entity is self-contained with its own oplog, snapshots, and trash.

Best for small datasets and entity-level portability. Copy one entity folder and you get its full history.

my-bucket/
  .worm                                # bucket sentinel
  org/customers/
    .worm                              # collection sentinel
    a1b2c3d4-.../
      profile.json                     # entity document
      .oplog/                          # THIS entity's oplog
        0001.json
        0002.json
      .snapshots/                      # THIS entity's snapshots
        2026-02-24T12:00:00Z.json
    e5f6g7h8-.../
      profile.json
      .oplog/
        0001.json

Collection

System directories live at the collection level, grouped by entity ID. All entities in a collection share a single oplog directory, snapshot directory, and trash directory.

Best for medium datasets and easy per-collection audit scans.

my-bucket/
  .worm
  org/customers/
    .worm
    .oplog/                            # ALL customer oplogs here
      a1b2c3d4/
        0001.json
        0002.json
      e5f6g7h8/
        0001.json
    .snapshots/                        # ALL customer snapshots here
      a1b2c3d4/
        2026-02-24T12:00:00Z.json
    a1b2c3d4/
      profile.json                     # entity is clean — just data
    e5f6g7h8/
      profile.json

Root

System directories live at the bucket root. A single global oplog, snapshot directory, and trash directory serve all models. Entity paths remain pristine with no system directories alongside the data.

Best for large datasets, global audit logs, and keeping entity paths clean.

my-bucket/
  .worm                                # bucket sentinel
  .oplog/                              # GLOBAL oplog for all models
    customers/
      a1b2c3d4/
        0001.json
        0002.json
      e5f6g7h8/
        0001.json
    invoices/
      inv-001/
        0001.json
  .snapshots/                          # GLOBAL snapshots
    customers/
      a1b2c3d4/
        2026-02-24T12:00:00Z.json
  .worm.trash/                         # GLOBAL trash
    customers/
      x9y0z1/
        profile.json
        .worm.tombstone
  org/customers/                       # CLEAN entity paths
    .worm
    a1b2c3d4/
      profile.json
    e5f6g7h8/
      profile.json
  org/invoices/
    .worm
    inv-001/
      data.json

Mix and Match

You can override the layout for individual system directories. The top-level layout field sets the default, and per-directory overrides take precedence.

{
  "storage": {
    "layout": "root",            // default for everything
    "snapshots": "inline"        // override: snapshots inside entity folders
  }
}

This is useful when you want global audit (oplog at root) but entity-portable snapshots (snapshots inline).

Override Field	Controls
`oplog`	Operation log placement
`snapshots`	Snapshot storage placement
`trash`	Soft-delete trash placement

Oplog Storage

When oplog: true on a model, every mutation is recorded as a sequential JSON file. Each oplog entry is immutable and append-only.

Oplog Entry Schema

interface OplogEntry {
  seq: number;                            // monotonically increasing
  op: "create" | "update" | "delete" | "restore";
  ts: string;                            // ISO 8601 timestamp
  actor?: string;                         // who made the change
  snapshot?: Record<string, unknown>;     // full doc (create, delete, restore)
  diff?: Record<string, [before, after]>; // field diffs (update)
  metadata?: Record<string, unknown>;     // arbitrary context
}

Oplog File Layout

# layout: "inline"
org/customers/abc-123/.oplog/0001.json

# layout: "collection"
org/customers/.oplog/abc-123/0001.json

# layout: "root"
.oplog/customers/abc-123/0001.json

Oplog Entry Examples

// 0001.json — create
{ "seq": 1, "op": "create", "ts": "2026-02-24T10:00:00Z",
  "actor": "user:0xabc...",
  "snapshot": { "name": "Acme Corp", "email": "hi@acme.com", "status": "active" } }

// 0002.json — update
{ "seq": 2, "op": "update", "ts": "2026-02-24T11:00:00Z",
  "actor": "user:0xabc...",
  "diff": { "name": ["Acme Corp", "Acme Inc"] } }

// 0003.json — update
{ "seq": 3, "op": "update", "ts": "2026-02-24T12:00:00Z",
  "actor": "api:service-xyz",
  "diff": { "status": ["active", "inactive"] } }

// 0004.json — delete
{ "seq": 4, "op": "delete", "ts": "2026-02-24T13:00:00Z",
  "actor": "user:0xabc...",
  "snapshot": { "name": "Acme Inc", "email": "hi@acme.com", "status": "inactive" } }

What the Oplog Enables

Capability	Description
Change history	See every change to every entity
Rollback	Restore to any previous state by sequence number
Audit trail	Who changed what and when
CRDT-friendly	Immutable, append-only entries are safe for concurrent writers

SDK API

const customers = worm.model("Customer");

// Get full history for an entity
const history = await customers.getHistory("abc-123");
// [{ seq: 1, op: "create", ... }, { seq: 2, op: "update", ... }]

// Rollback to a specific sequence number
await customers.rollback("abc-123", { toSeq: 2 });

// Diff between two points in time
await customers.diff("abc-123", 1, 3);

Snapshots

Periodic full-document copies, independent of the oplog. While the oplog stores diffs, snapshots store complete documents at a point in time. This enables fast rollback without replaying the entire oplog.

Snapshot Configuration

{
  "models": {
    "Customer": {
      "snapshots": {
        "enabled": true,
        "every": 10,        // snapshot every 10 oplog entries
        "maxAge": "30d",    // auto-prune after 30 days
        "maxCount": 50      // keep at most 50 per entity
      }
    }
  }
}

Snapshot File Layout

Snapshots are named by ISO 8601 timestamp:

# layout: "root"
.snapshots/customers/abc-123/2026-02-24T12:00:00Z.json

# layout: "collection"
org/customers/.snapshots/abc-123/2026-02-24T12:00:00Z.json

# layout: "inline"
org/customers/abc-123/.snapshots/2026-02-24T12:00:00Z.json

SDK API

const customers = worm.model("Customer");

// Take a manual snapshot
await customers.snapshot("abc-123");

// List all snapshots for an entity
const snaps = await customers.listSnapshots("abc-123");
// [{ ts: "2026-02-24T12:00:00Z", size: 1234 }, ...]

// Restore from a specific snapshot
await customers.restoreSnapshot("abc-123", "2026-02-24T12:00:00Z");

Trash (Soft Delete)

When softDelete: true on a model, deleted entities are moved to trash instead of being permanently removed. A .worm.tombstone marker file records deletion metadata.

Tombstone Schema

interface Tombstone {
  deletedAt: string;     // ISO 8601
  deletedBy?: string;    // actor who deleted
  restorePath: string;   // original S3 key for restore
  ttl?: number;          // days until auto-purge (0 = keep forever)
}

Trash File Layout

# layout: "root"
.worm.trash/customers/x9y0z1/
  profile.json            # the deleted entity's data
  .worm.tombstone         # deletion metadata

# layout: "collection"
org/customers/.worm.trash/x9y0z1/
  profile.json
  .worm.tombstone

# layout: "inline"
org/customers/x9y0z1/.worm.trash/
  profile.json
  .worm.tombstone

SDK API

const customers = worm.model("Customer");

// Soft delete — moves to trash
await customers.delete("abc-123");

// List trashed entities
const trashed = await customers.listTrashed();

// Restore from trash
await customers.restore("abc-123");

// Permanently delete from trash
await customers.purge("abc-123");

// Purge all entities past their TTL
await customers.purgeExpired();

Sentinel Files

.worm sentinel files make buckets self-documenting. They are placed at the bucket root and at each collection directory. Any tool reading the bucket can immediately understand its structure.

Bucket-Level Sentinel

// my-bucket/.worm
{
  "schema": "1.0",
  "bucket": "my-bucket",
  "models": ["Customer", "Invoice", "OrgSettings"],
  "storage": { "layout": "root" }
}

Collection-Level Sentinel

// org/customers/.worm
{
  "model": "Customer",
  "idType": "uuid",
  "mode": "readwrite",
  "oplog": true,
  "softDelete": true,
  "count": 1423,
  "lastModified": "2026-02-24T14:30:00Z"
}

Key Points

Sentinel files are optional read metadata. The system works without them.
worm push and worm import write sentinels to make buckets self-documenting.
Sentinels are not the same as schema.json. The schema is the source of truth for definitions; sentinels are runtime markers.
The bucket-level sentinel includes storage.layout so any tool reading the bucket knows where to find system directories.

Access Modes

Each model declares an access mode that controls which operations are allowed.

Mode	Allowed Operations	Use Case
`"readonly"`	`findById`, `findAll`, `count`, `exists`	External data, shared read-only buckets
`"readwrite"`	Full CRUD (default)	Standard application data
`"append"`	`create` only, no update or delete	Immutable audit logs, write-once records

const customers = worm.model("Customer");       // mode: "readwrite"
customers.mode;                                   // "readwrite"

const externalData = worm.model("ExternalFeed"); // mode: "readonly"
await externalData.save(thing);                   // throws: "ExternalFeed is read-only"

Access modes are enforced at the SDK level. The "append" mode provides true write-once semantics -- entities can be created but never modified or deleted.

Path Resolution Summary

The SDK resolves system paths based on the active layout for each system directory type.

// StorageResolver API
const resolver = new StorageResolver(storageConfig);

resolver.oplogPath("Customer", "abc-123");
// "root"       -> ".oplog/customers/abc-123/"
// "collection" -> "org/customers/.oplog/abc-123/"
// "inline"     -> "org/customers/abc-123/.oplog/"

resolver.snapshotPath("Customer", "abc-123");
// "root"       -> ".snapshots/customers/abc-123/"
// "collection" -> "org/customers/.snapshots/abc-123/"
// "inline"     -> "org/customers/abc-123/.snapshots/"

resolver.trashPath("Customer", "abc-123");
// "root"       -> ".worm.trash/customers/abc-123/"
// "collection" -> "org/customers/.worm.trash/abc-123/"
// "inline"     -> "org/customers/abc-123/.worm.trash/"