Bucket Sync

@decoperations/bucket-sync provides local-first object storage with bidirectional S3 sync. Blobs are stored in OPFS (content-addressed), metadata lives in IndexedDB, and a sync engine handles pushing local changes to S3 and pulling remote updates.

Install

pnpm add @decoperations/bucket-sync

LocalBucket

The LocalBucket class is the main entry point. It provides an S3-like API backed by OPFS and IndexedDB:

import { LocalBucket } from "@decoperations/bucket-sync";

const bucket = new LocalBucket({
  name: "my-local-bucket",
  remote: {
    endpoint: "https://gateway.storjshare.io",
    region: "us-east-1",
    credentials: {
      accessKeyId: "AKIA...",
      secretAccessKey: "wJalr...",
    },
    bucket: "my-bucket",
  },
});

Object Operations

All operations are local-first. Data is written to OPFS immediately and synced to S3 in the background.

// Write a blob
await bucket.put("photos/cat.jpg", imageBlob, {
  contentType: "image/jpeg",
  metadata: { author: "jane" },
});

// Read a blob
const blob = await bucket.get("photos/cat.jpg");

// Check existence
const exists = await bucket.exists("photos/cat.jpg");

// Get metadata without fetching the blob
const entry = await bucket.head("photos/cat.jpg");
// -> { key, localHash, size, contentType, syncState, ... }

// List objects by prefix
const photos = await bucket.list("photos/");

// Delete an object
await bucket.delete("photos/cat.jpg");

Put Options

interface PutOptions {
  /** MIME content type (auto-detected from extension if omitted) */
  contentType?: string;
  /** Custom metadata key-value pairs */
  metadata?: Record<string, string>;
}

Get Options

interface GetOptions {
  /** Force fetch from S3 even if a local copy exists */
  remote?: boolean;
}

// Force a fresh copy from S3
const fresh = await bucket.get("data.json", { remote: true });

SyncEngine

The SyncEngine handles bidirectional sync between the local bucket and S3. It runs as part of LocalBucket -- you typically interact with it through the bucket's sync methods.

Manual Sync

// Full sync: push dirty objects, then pull remote changes
const result = await bucket.sync();
console.log(`Pushed: ${result.pushed}, Pulled: ${result.pulled}`);

// Push only (upload local changes)
await bucket.push();

// Pull only (download remote changes)
await bucket.pull();

Automatic Background Sync

// Start auto-sync on an interval (default: 30 seconds)
bucket.startSync();

// Stop auto-sync
bucket.stopSync();

Sync Result

Every sync operation returns a SyncResult:

interface SyncResult {
  pushed: number;       // Objects uploaded to S3
  pulled: number;       // Objects downloaded from S3
  conflicts: number;    // Conflicts detected
  errors: SyncError[];  // Failed operations
  duration: number;     // Duration in milliseconds
  timestamp: string;    // ISO 8601 completion time
}

Sync Status

const status = await bucket.status();
// {
//   total: 150,
//   synced: 140,
//   dirty: 5,         // Local changes not yet pushed
//   stale: 3,         // Remote changes not yet pulled
//   conflicts: 1,     // Both sides changed
//   pendingDelete: 1, // Awaiting remote delete
//   cacheSize: 52428800, // Bytes used in OPFS
//   isLeader: true,   // Tab leader for sync coordination
//   encrypted: false,
//   locked: false,
// }

Sync States

Each object in the manifest has a sync state:

Conflict Resolution

When both local and remote copies change between syncs, a conflict is flagged.

// List unresolved conflicts
const conflicts = await bucket.conflicts();

// Resolve a conflict
await bucket.resolve("data/report.json", "keep-local");

Resolution Strategies

Automatic Conflict Resolution

Configure a default strategy in the bucket config:

const bucket = new LocalBucket({
  name: "my-bucket",
  remote: { /* ... */ },
  sync: {
    conflictResolution: "last-writer-wins",  // or "keep-local" | "keep-remote" | "flag"
  },
});

When set to "flag" (default), conflicts require manual resolution.

Service Worker

Register a service worker to intercept fetch requests and serve bucket content offline:

import { registerBucketSW } from "@decoperations/bucket-sync/service-worker";

// Register the service worker
const registration = await registerBucketSW("/bucket-sw.js", {
  scope: "/",
});

// Unregister
import { unregisterBucketSW } from "@decoperations/bucket-sync/service-worker";
await unregisterBucketSW();

The service worker intercepts requests matching the configured route prefix (default: /__bucket/) and serves content from the local OPFS cache when offline.

Client-Side Encryption

LocalBucket supports AES-256-GCM encryption at rest. All blobs written to OPFS are encrypted with a key derived from a user passphrase.

const bucket = new LocalBucket({
  name: "encrypted-bucket",
  remote: { /* ... */ },
  encryption: {
    enabled: true,
  },
});

// Unlock with a passphrase (derives AES key via PBKDF2)
const success = await bucket.unlock("my-secret-passphrase");

if (success) {
  // All put/get operations now encrypt/decrypt transparently
  await bucket.put("secret.json", new Blob([JSON.stringify({ key: "value" })]));
  const blob = await bucket.get("secret.json"); // Decrypted automatically
}

// Lock the bucket (clears in-memory key)
bucket.lock();

// Check lock status
bucket.isLocked(); // true

On first unlock, a salt and key-check blob are generated and stored in IndexedDB. Subsequent unlocks verify the passphrase against the stored key-check.

Multi-Tab Coordination

When multiple browser tabs share the same bucket, a leader election protocol ensures only one tab runs the sync engine at a time:

const bucket = new LocalBucket({
  name: "shared-bucket",
  remote: { /* ... */ },
  tabs: {
    coordinate: true,           // Enable (default: true)
    heartbeatInterval: 5000,    // ms between heartbeats
    leaderTimeout: 10000,       // ms before promoting a new leader
  },
});

bucket.startSync();
// Only the leader tab performs sync. Other tabs observe via BroadcastChannel.

Events

const unsub = bucket.on((event) => {
  switch (event.type) {
    case "sync:start":        break;
    case "sync:complete":     console.log(event.result); break;
    case "sync:error":        console.error(event.error); break;
    case "object:pushed":     console.log("Pushed:", event.key); break;
    case "object:pulled":     console.log("Pulled:", event.key); break;
    case "object:conflict":   console.log("Conflict:", event.conflict.key); break;
    case "object:evicted":    console.log("Evicted:", event.key); break;
    case "online":            break;
    case "offline":           break;
    case "leader:promoted":   break;
    case "leader:demoted":    break;
    case "encryption:locked": break;
    case "encryption:unlocked": break;
  }
});

// Unsubscribe
unsub();

Scope Management

Limit sync to specific S3 key prefixes:

// Get current scope
bucket.getScope(); // [""]  (entire bucket by default)

// Add a prefix to sync
await bucket.addScope("photos/");
// Immediately triggers a pull for the new prefix

// Remove a prefix
await bucket.removeScope("photos/");
// Local data is preserved but no longer synced

Pin Management

Pin objects to prevent cache eviction:

// Pin objects for offline access
await bucket.pin(["important/report.pdf", "config/settings.json"]);

// Unpin
await bucket.unpin(["important/report.pdf"]);

Pinned objects are never evicted by the cache manager, even when the cache is full.

Cache Management

// Check cache usage
const usage = await bucket.cacheUsage();
// { used: 52428800, limit: 1073741824, percentage: 4.88 }

// Manually evict to free space
const freed = await bucket.evict(10 * 1024 * 1024); // Free 10 MB

Configuration Reference

interface LocalBucketConfig {
  /** Unique name (used as IDB database name + OPFS directory) */
  name: string;
  /** S3 connection */
  remote: {
    endpoint?: string;
    region?: string;
    credentials?: { accessKeyId: string; secretAccessKey: string };
    bucket: string;
  };
  /** Prefix scope */
  scope?: {
    prefixes: string[];  // Default: [""] (entire bucket)
  };
  /** Sync behavior */
  sync?: {
    interval?: number;                // Auto-sync interval in ms (default: 30000, 0 = manual)
    pullStrategy?: "eager" | "lazy" | "pinned-only";  // Default: "lazy"
    pushConcurrency?: number;         // Default: 4
    pullConcurrency?: number;         // Default: 4
    maxRetries?: number;              // Default: 3
    conflictResolution?: "last-writer-wins" | "keep-local" | "keep-remote" | "flag";
  };
  /** Storage limits */
  storage?: {
    maxCacheSize?: number;            // Default: 1 GB
    evictionStrategy?: "lru" | "lfu" | "oldest";
  };
  /** Encryption at rest */
  encryption?: {
    enabled: boolean;
  };
  /** Multi-tab coordination */
  tabs?: {
    coordinate: boolean;              // Default: true
    heartbeatInterval?: number;       // Default: 5000
    leaderTimeout?: number;           // Default: 10000
  };
}

Browser Requirements

OPFS is the minimum requirement. IndexedDB is available in all modern browsers. Multi-tab coordination and service worker features degrade gracefully if unavailable.

Lifecycle

// Create
const bucket = new LocalBucket(config, transport);

// Start auto-sync
bucket.startSync();

// ... use the bucket ...

// Stop sync and clean up
bucket.stopSync();

// Destroy (optionally clear all local data)
await bucket.destroy({ clearLocal: true });