docs: self-hosted prod/test/dev Azure VM guide with key rotation, Azure Key Vault workflows, and rsync data migration pattern

[benpsnyder]

Summary

This PR creates a comprehensive self-hosted JWT key rotation how-to for SpacetimeDB with reproducible defaults, identity-preserving rotation guidance, and host-scoped data migration runbooks. It is written for both operators and automation workflows, with explicit guardrails for production-like VM environments.

Key Changes

• Adds a top-level assumptions/risk section for a multi-host topology (prod, test, dev, optional local) including backup prerequisites before rotation or sync.
• Defines an opinionated, end-to-end path contract for keys:
  • host source: ./.generated/spacetimedb-keys
  • runtime mount: /etc/spacetimedb
  • startup args pinned to /etc/spacetimedb/id_ecdsa and /etc/spacetimedb/id_ecdsa.pub
• Includes reproducible quickstart scaffolding:
  • OpenSSL commands for compatible ES256/P-256 key generation
  • non-container and Docker startup examples
  • rotation/verify command sequence and marker checks
• Documents one unified tooling surface around spacetimedb-tooling.ts for:
  • rotate (--dry-run, --yes)
  • verify (--verify-only)
  • token continuity (--resign-token-only)
  • explicit token/key path overrides (--publisher-cli-toml-path, --private-key-path)
• Clarifies 401 vs 403 behavior and why identity drift causes publish failures after rotation.
• Covers all rotation strategies in one place:
  • clean-slate rotation (stateless/dev)
  • identity-preserving rotation (stateful)
  • OIDC-backed identity model for production
• Adds host-scoped migration runbooks for staged sync (prod -> test -> dev) with destination-side token re-signing semantics (run on destination host context, including SSH examples).
• Explicitly separates conceptual topology guidance from currently implemented sync primitives in reference tooling to avoid over-claiming.
• Adds operational guardrails for sync promotions:
  • rsync --delete is destructive
  • stop/start ordering around sync
  • required acceptance gates: key parity, destination re-sign, restart/redeploy, publish marker validation
• Expands verification and automation content:
  • PEM/parity/fingerprint/newline checks
  • AI/automation contract with inputs, outputs, required command order, and success/failure markers
  • troubleshooting flow for InvalidSignature and ownership mismatch errors

Why This Matters

The most common failure mode in self-hosted fleets is identity drift: signatures validate, but spacetime publish still fails with 403 Forbidden because destination ownership no longer matches the publishing identity after key rotation or data movement.

The updated how-to makes these operator requirements explicit:

1. preserve identity continuity across rotations and host promotions
2. re-sign destination tokens after rsync in destination host context
3. treat restart/redeploy and publish markers as promotion gates

Following this runbook prevents regressions where key material is correct but publish still fails due to stale signatures or owner mismatch state.

Why JWT Key Rotation Is Essential for Self-Hosters

In self-hosted deployments, operators are the signing authority. Rotation is both a security control and an operational correctness requirement.

Security value

• reduces exposure window for compromised private keys
• invalidates old/stolen tokens after rotation
• mitigates risk from snapshots, backups, clones, and long-lived VM access

Operational value

• keeps key material consistent across multi-VM environments
• reduces publish outages after promotion/sync operations
• prevents avoidable 401/403 failures in routine release workflows

Scope

• docs-only change
• public-safe examples only (no internal org names, private vault IDs, or private secret prefixes)
• includes non-container and Docker workflows, self-publish/versioning guidance, and marker-based validation (PUBLISH_SUCCESS, PUBLISH_FAILED)
• includes data migration workflow guidance (rsync + destination re-sign + restart/redeploy + publish checks)

[CLAassistant]

All committers have signed the CLA.

[benpsnyder]

Hi team - I just wrapped up getting this setup working on my first SpacetimeDB build. I am able to successfully add data to production, rsync to dev, and query and see that data on docker container restart. Pretty cool! Key rotation works too. I wanted to get some of these basic workflows before building a real app for users in our production environment.

Thanks @cloutiertyler for showing interest on X; this is all very new tech (and awesome) and I am willing to "bleed" a bit to learn and adopt. Hopefully other people that prefer self-hosting will find this guide useful when pointing their AI at it to help them get going with their setups.

Also we will try your paid service for sure! I just like to always know how to run stuff myself and have that option available. Thank you for the hard work you and the team did to make this possible.

[bfops]

Thank you for opening this PR & explaining everything so thoroughly!