feat: add floop vault sync for Lance-native S3 backup

[nvandessel]

Summary

• Implements floop vault feature for Lance-native S3 backup/sync to MinIO (or any S3-compatible backend)
• Adds bidirectional sync of LanceDB vectors (row-level) and SQLite graph data (V2 backup format)
• New internal/vault/ package with VaultConfig, VaultService, VectorSyncer, GraphSyncer, S3Client, sync state, and age encryption
• CLI commands: floop vault init, push, pull, sync, status
• Supports --dry-run, --scope (global/local/both), --from (cross-machine pull), --json output

Implementation (per plan)

Task	Description	Status
T0	lancedb-go fork tag v0.2.2 + go.mod	Done
T1	VaultConfig + validation	Done
T2	Sync state tracking	Done
T3	S3 file client (minio-go)	Done
T4	Vector sync (row-level, CGO build tags)	Done
T5	Graph data sync (V2 backup + corrections)	Done
T5b	Age encryption layer	Done
T6	Core VaultService orchestrator	Done
T7	CLI commands	Done
T8	MCP handlers	Deferred (per plan)
T9	Integration tests + docker-compose	Done
T10	Documentation	Done

Spec & Plan

• Spec: docs/team-up/specs/vault-sync.md
• Plan: docs/team-up/plans/vault-sync.md

Test plan

• go build ./... succeeds (CGO_ENABLED=0)
• go vet ./... clean
• go fmt ./... clean
• go test ./internal/vault/... passes (unit tests)
• go test ./cmd/floop/... passes
• Integration tests require MinIO: docker compose -f docker-compose.vault-test.yml up -d && go test -tags integration ./internal/vault/...
• Vector sync tests require CGO: go test -tags cgo ./internal/vault/...

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 30.09950% with 843 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.01%. Comparing base (7a1dad8) to head (86f2a8a).

Files with missing lines	Patch %	Lines
cmd/floop/cmd_vault.go	22.92%	311 Missing and 5 partials ⚠️
internal/vault/vault.go	3.98%	313 Missing ⚠️
internal/vault/graph_sync.go	49.46%	71 Missing and 23 partials ⚠️
internal/vault/s3client.go	36.04%	52 Missing and 3 partials ⚠️
internal/vault/encrypt.go	0.00%	24 Missing ⚠️
internal/vault/state.go	59.52%	12 Missing and 5 partials ⚠️
internal/vault/vector_sync_nocgo.go	0.00%	10 Missing ⚠️
internal/vault/config.go	92.10%	6 Missing and 3 partials ⚠️
internal/config/config.go	50.00%	2 Missing and 1 partial ⚠️
cmd/floop/main.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #278      +/-   ##
==========================================
- Coverage   78.76%   75.01%   -3.76%     
==========================================
  Files         189      197       +8     
  Lines       18779    24743    +5964     
==========================================
+ Hits        14792    18561    +3769     
- Misses       2754     4910    +2156     
- Partials     1233     1272      +39     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

Greptile Summary

This PR introduces the floop vault feature: bidirectional sync of LanceDB vectors and SQLite graph data to any S3-compatible backend (MinIO, AWS S3, R2) with optional age encryption, a new internal/vault/ package, and five CLI subcommands (init, push, pull, sync, status).

Multiple P0/P1 issues raised in prior review rounds have been addressed (nil-panic on corrupt state, silent delete-failure accumulation, dryRun scope blindness, ANN-only vector scan, corrections line-count cursor). Two items from previous threads have no developer reply and appear unresolved:

• scope=both silently overwrites global corrections with local corrections in S3 (same machines/{id}/graph/corrections.jsonl key used for both pushScope calls)
• Re-init with default --region us-east-1 unconditionally overwrites the existing region, breaking callers who only meant to rotate credentials

Confidence Score: 3/5

Not safe to merge — two P1 findings from previous review rounds have no developer acknowledgement or fix, and one causes silent data loss on scope=both push.

The current review pass finds only P2s, which would normally yield 5/5. However, two prior-thread P1 issues remain open with no developer reply: the scope=both corrections.jsonl S3 key collision (data loss on every push --scope=both) and the unconditional region/path-style overwrite on re-init. These are present in the submitted diff and lower the score below the P1 ceiling of 4.

internal/vault/vault.go (pushScope called twice with same machineID for scope=both), cmd/floop/cmd_vault.go (region flag default bypasses re-init guard), internal/vault/graph_sync.go (local scanner buffer error check)

Important Files Changed

Filename	Overview
internal/vault/vault.go	Core VaultService orchestrator — Push/Pull/Sync/Status/Verify; two issues from previous review remain open (scope=both corrections key collision, re-init region overwrite); two new P2s: Verify with conflicting flags, Sync 2× timeout
internal/vault/graph_sync.go	GraphSyncer for V2 backup + corrections.jsonl; content-based dedup now correct; local scanner in mergeCorrections silently drops lines > 64 KB (P2)
internal/vault/vector_sync.go	VectorSyncer using Select() for full-table scan; delete errors now propagated; extractVector helper retained for JSON-deserialized rows
cmd/floop/cmd_vault.go	CLI commands for vault; re-init region/path-style overwrite from previous review still unresolved; connectivity-first save ordering now correct
internal/vault/config.go	VaultConfig validation; endpoint now required in Validate(); machine ID regex; env var expansion support
internal/vault/state.go	SyncState load/save with atomic rename; nil guard on load; staleness classification looks correct
internal/vault/s3client.go	MinIO-backed S3 client; dead code removed; Exists() handles NoSuchKey correctly; compile-time interface check present
internal/vault/encrypt.go	Thin age binary wrapper; cleans up temp file on failure; clear error when binary is absent

Sequence Diagram

sequenceDiagram
    participant CLI as floop vault CLI
    participant VS as VaultService
    participant VSync as VectorSyncer (LanceDB)
    participant GSync as GraphSyncer
    participant S3 as S3Client (MinIO)
    participant LD as LanceDB Remote

    CLI->>VS: Push(ctx, graphStore, root, opts)
    VS->>VS: normalizeScope(opts.Scope)
    VS->>VS: pushScope(global)
    VS->>VSync: Push(ctx)
    VSync->>VSync: Select(ctx, QueryConfig{}) — full scan
    VSync->>LD: Delete + Add per row (upsert)
    VS->>GSync: Push(ctx, graphStore, correctionsPath)
    GSync->>GSync: BackupWithOptions (V2 gzip)
    GSync->>S3: Upload backup + corrections.jsonl
    VS->>VS: pushScope(local) — if scope=both
    Note over VS,S3: scope=both writes same S3 key for both scopes
    VS->>S3: PutJSON sync-state (best-effort)

    CLI->>VS: Pull(ctx, graphStore, opts)
    VS->>VSync: Pull(ctx)
    VSync->>LD: Read remote rows → upsert local
    VS->>GSync: Pull(ctx, graphStore, fromMachine, correctionsPath)
    GSync->>S3: Download backup + corrections
    GSync->>GSync: Restore (merge semantics)
    GSync->>GSync: mergeCorrections (content-based dedup)

Loading

_{Reviews (15): Last reviewed commit: "fix(lint): disable goconst linter" | Re-trigger Greptile}