Skip to content

Architecture

This page is for developers and contributors — it describes Vault's internal design. If you just want to use Vault, start with Getting Started.

Vault is a single Go binary that runs as a daemon on Unraid servers. It provides backup and restore for Docker containers, libvirt VMs, ZFS datasets, folders, and plugins.

Layered Design

CLI (Cobra) -> API Server (Chi + WebSocket Hub) -> Handlers -> DB / Storage / Engine / Runner
Layer Package Description
CLI internal/cli/ Cobra commands: vault daemon, vault replica, vault dedup, vault recover
API internal/api/ Chi router, REST handlers, WebSocket integration
MCP internal/mcp/ Model Context Protocol tools over streamable HTTP
Job Intake internal/jobs/ Validates, persists and activates every Job write; REST and MCP adapt to it
Database internal/db/ SQLite (WAL, pure-Go driver) with hybrid snapshot + USB shadow
Storage internal/storage/ Local, SFTP, SMB, NFS, WebDAV, and S3 adapters (factory-dispatched)
Engine internal/engine/ Per-type backup/restore handlers (container, VM, ZFS, folder, plugin)
Runner internal/runner/ Job orchestration, retention, verification, compression
Dedup internal/dedup/ Keyed-FastCDC chunker, per-destination dedup repo, GC
Crypto internal/crypto/ AES-256-GCM, server key, passphrase-derived data keys
Replication internal/replication/ Pull-mode replication client + syncer
Scheduler internal/scheduler/ Cron-based scheduling
WebSocket internal/ws/ Real-time event hub for backup progress and config changes
Notify internal/notify/ Unraid notifications + Discord webhooks
Diagnostics internal/diagnostics/ Redacted ZIP bundle (system info, schema, runs, scheduler, daemon log)
Logbuf internal/logbuf/ In-memory ring buffer that captures every log.* line for diagnostics

Key Interfaces

storage.Adapter

type Adapter interface {
    Write(path string, reader io.Reader) error
    WriteFrom(path string, open func() (io.ReadCloser, error)) error
    Read(path string) (io.ReadCloser, error)
    ReadRange(path string, offset, length int64) (io.ReadCloser, error)
    Delete(path string) error
    List(prefix string) ([]FileInfo, error)
    Stat(path string) (FileInfo, error)
    TestConnection() error
    GetCapacity(ctx context.Context) (Capacity, error)
    Usage() (free, total int64, err error)
}

WriteFrom lets the retry layer re-issue a failed upload from a fresh stream; ReadRange serves the dedup layer's partial pack reads; GetCapacity and Usage feed the capacity widgets and trajectory detector.

Adapters that hold persistent resources (SFTP, SMB) implement io.Closer and are released by storage.CloseAdapter. Bandwidth throttling is layered on top via a RateLimit config field for every remote type. Adapters are constructed via storage.NewAdapter(type, configJSON).

engine.Handler

type Handler interface {
    Backup(ctx context.Context, item BackupItem, dest string, progress ProgressFunc) (*BackupResult, error)
    Restore(ctx context.Context, item BackupItem, source string, progress ProgressFunc) error
    ListItems() ([]BackupItem, error)
}

Handlers that support content-defined chunking additionally implement engine.ChunkedHandler (BackupChunked / RestoreChunked) so the runner can route them through the dedup repo on dedup-enabled destinations.

Build Tags

  • vm.go, vm_restore.go, and other libvirt-touching files use //go:build linux — real libvirt RPC against /var/run/libvirt/libvirt-sock
  • vm_stub.go: //go:build !linux — stubs for macOS / Windows so the rest of the daemon still builds and tests run
  • ZFS (engine/zfs.go) calls out to the host zfs binary; the code itself is platform-neutral
  • Local builds and the full test suite work on macOS without libvirt installed

Database

SQLite with WAL mode and busy timeout. Pure Go driver via modernc.org/sqlite (no CGO).

sql.Open("sqlite", path+"?_txlock=immediate&_pragma=busy_timeout(30000)&_pragma=journal_mode(WAL)&_pragma=foreign_keys(ON)&_pragma=journal_size_limit(67108864)")

Schema is applied inline at open time via CREATE TABLE IF NOT EXISTS (no versioned migrations). Twelve tables: storage_destinations, jobs, job_items, job_runs, restore_points, settings, activity_log, verify_runs, replication_sources, dedup_packs, dedup_chunks, dedup_gc_runs.

Hybrid snapshot layout

To survive Unraid's USB-backed boot (where writing every commit to flash would wear the drive), the daemon runs in hybrid mode by default:

  1. Working DB — /var/local/vault/vault.db (tmpfs-backed; fast, never written to flash directly)
  2. Primary snapshot — <discovered cache pool>/.vault/vault.db (periodic copies; the authoritative on-disk source)
  3. USB shadow — /boot/config/plugins/vault/vault.db.backup (refreshed after config changes via the SetConfigChangeHook pathway, so the flash copy stays current without per-row writes)

On startup the daemon restores from the freshest integrity-passing source among the primary snapshot, the default cache snapshot, rotated snapshot copies, the USB-direct live DB (written when a boot ran without a mounted pool), and the USB shadow — falling back to a fresh DB only if none pass. Freshness is compared by modification time so a boot that ran in USB-direct mode can never be silently reverted by an older cache snapshot on the next hybrid boot (issue #241). An emhttp event script (event/stopping_svcs) stops the daemon gracefully on array stop/reboot so the final flush completes, and event/started ensures the daemon is running once the array is up.

Vault directories on Unraid storage

  • <pool>/.vault/ — the primary database snapshot (vault.db) plus timestamped copies under rotated/. Unraid exposes every top-level pool directory as a user share, so this same directory is also visible as /mnt/user/.vault. It is intentional persistent storage, not stray files; move it under Settings → General → Database Location if you prefer another location.
  • <pool>/.vault-stage/ — temporary staging for backups and restores, emptied as each run finishes.
  • /mnt/addons/vault-fuse/mount-<id>/ — read-only FUSE backup mounts. Each directory is created only when a mount starts and removed when it ends. Unraid reserves /mnt/addons for plugin and remote mounts; when it does not exist, Vault uses <tmp>/vault-fuse. fuse_mount_base_dir overrides the root. Releases up to v2026.09.01 used /mnt/vault-fuse, which Fix Common Problems flags as an invalid folder; Vault now removes that directory at startup once it is empty, and never deletes non-empty content there.

Do not delete .vault or .vault-stage while Vault is running. Pool discovery never treats addons or vault-fuse as pools.

Project Structure

├── cmd/vault/             # CLI entry point (main + version ldflags)
├── internal/
│   ├── api/               # HTTP server and REST handlers
│   │   ├── server.go      # Server struct, StartWithContext
│   │   ├── routes.go      # Route registration
│   │   └── handlers/      # Job, Storage, Replication, Settings, Browse, …
│   ├── cli/               # Cobra subcommands (daemon, replica, dedup, recover)
│   ├── config/            # Enum constants and shared types
│   ├── crypto/            # Server key, AES-256-GCM, passphrase derivation
│   ├── db/                # SQLite, repos, hybrid snapshot manager
│   ├── dedup/             # Keyed-FastCDC chunker, repo, index, GC
│   ├── diagnostics/       # Redacted ZIP bundle for support
│   ├── engine/            # Per-type handlers (container, VM, ZFS, folder, plugin)
│   │   ├── container.go   # Docker SDK
│   │   ├── vm.go          # libvirt (Linux only)
│   │   ├── vm_stub.go     # Non-Linux stub
│   │   └── zfs.go         # zfs send/receive
│   ├── fsstat/            # Portable filesystem capacity (statfs / GetDiskFreeSpaceEx)
│   ├── logbuf/            # In-memory ring buffer for daemon-log capture
│   ├── mcp/               # MCP tools over streamable HTTP
│   ├── notify/            # Unraid notifications + Discord webhook
│   ├── recovery/          # Server-less, read-only backup recovery (vault recover)
│   ├── replication/       # Pull-mode client + syncer
│   ├── runner/            # Job orchestration, compression, retention, verify
│   ├── scheduler/         # Cron scheduler
│   ├── storage/           # Adapter interface + factory
│   │   ├── local.go       # Local filesystem
│   │   ├── sftp.go        # SFTP
│   │   ├── smb.go         # SMB
│   │   ├── nfs.go         # NFS
│   │   ├── webdav.go      # WebDAV (with chunked upload + manifest)
│   │   └── s3.go          # AWS SDK Go v2 (S3-compatible safe)
│   ├── unraid/            # Pool discovery, /mnt scanning helpers
│   └── ws/                # WebSocket hub
├── plugin/                # Unraid plugin files (.plg, PHP, RC script)
├── ansible/               # Build / deploy / verify automation
├── scripts/               # Local dev helpers (stress, fixtures)
├── web/                   # Svelte 5 frontend (Vite build embedded via go:embed)
└── docs/                  # Documentation

Build Commands

Plugin Lifecycle (Ansible-driven)

make build               # Lint -> test -> web build -> cross-compile Linux/amd64
make deploy              # Deploy binary + assets to Unraid, start daemon
make verify              # Run endpoint checks plus smoke tests against Unraid
make redeploy            # Full lifecycle: uninstall -> build -> deploy -> verify

Local Development

make deps                # Download and tidy Go modules
make build-local         # Build for the current platform
make test                # Run all unit tests
make test-short          # Run short tests only
make test-coverage       # Generate coverage.out and coverage.html
make lint                # Run golangci-lint
make security-check      # Run gosec, govulncheck, and go mod verify
make pre-commit-install  # Install pre-commit hooks
make pre-commit-run      # Run the full local quality gate
make clean               # Remove build artifacts

Running the Daemon

./build/vault daemon --db=vault.db --addr=:24085

The production binary is built with CGO_ENABLED=0 using modernc.org/sqlite, keeping it pure Go.

Deployment

Unraid Plugin

Install via the Unraid Community Applications store, or paste the plugin URL into Plugins > Install Plugin:

https://raw.githubusercontent.com/tinkerologie/vault/main/plugin/vault.plg

The daemon runs at /usr/local/sbin/vault with the database at /boot/config/plugins/vault/vault.db.

Service management:

/etc/rc.d/rc.vault start
/etc/rc.d/rc.vault stop
/etc/rc.d/rc.vault restart
/etc/rc.d/rc.vault status

Uninstall removes runtime and configuration traces (including the database) while preserving backup data in configured storage destinations.

Ansible

cd ansible
cp inventory.yml.example inventory.yml   # Add server details
ansible-playbook -i inventory.yml ansible.yml --tags deploy
ansible-playbook -i inventory.yml ansible.yml --tags verify

See ansible/README.md for the full deployment workflow.

Key Dependencies

Package Purpose
github.com/go-chi/chi/v5 HTTP router
github.com/spf13/cobra CLI framework
github.com/robfig/cron/v3 Cron scheduler
modernc.org/sqlite Pure Go SQLite driver (no CGO)
github.com/docker/docker Docker Engine SDK
github.com/digitalocean/go-libvirt Pure Go libvirt RPC (Linux only)
github.com/vmware/go-nfs-client NFS storage adapter
github.com/cloudsoda/go-smb2 SMB storage adapter
github.com/pkg/sftp SFTP storage adapter
github.com/studio-b12/gowebdav WebDAV storage adapter
github.com/aws/aws-sdk-go-v2/... S3 / S3-compatible storage adapter
github.com/PlakarKorp/go-cdc-chunkers Keyed-FastCDC chunker for content-defined dedup
github.com/coder/websocket WebSocket server
github.com/modelcontextprotocol/go-sdk MCP streamable-HTTP / stdio transport
filippo.io/age Optional age-encrypted archive layer

Known Limitations

  • Dedup GC is delete-only. Garbage collection removes packs whose every chunk is unreferenced. A pack with even one live chunk is kept; its dead bytes are reported as Reclaimable ("rewritable bytes") on the Storage card and persisted per GC run in dedup_gc_runs, but they are not yet physically reclaimed. Compaction/repacking is tracked in #103. This matches early borg/restic behaviour and is a deliberate v1 choice — small (24 MiB) packs bound the stranded space.

Version

Vault uses date-based versioning in the VERSION file with the format YYYY.MM.PATCH. Version is injected via ldflags at build time (-X main.version, -X main.buildDate, -X main.commit).

plugin/pages/Backups.page is an optional top-nav entry (Menu="Tasks", Type="xmenu" — the same extension point Community Applications uses) gated by NAV_LINK in vault.cfg via its Cond header, toggled from the Vault settings page through control.php's navlink action. It embeds the SPA's proxy entrypoint (include/app.php) in an iframe sized to the viewport. Because app.php injects Unraid's CSRF token once per render and the SPA has no in-app refresh, the frame reloads itself when the tab regains visibility after >10 minutes hidden so write actions keep working across token rotations. In proxy mode the SPA polls; WebSocket-only live events do not stream here (matching the Utilities page).