Multi-fork architecture: roadmap and tracking issue

[leolara]

Multi-fork architecture: roadmap and tracking issue

Tracks the work to land the multi-fork architecture proposed in
Multi-Fork Architecture for leanSpec.

#638 introduces the skeleton (ForkProtocol, ForkRegistry, forks/devnet4/, forks/devnet5/). This issue scopes the follow-up work and lets us check items off as PRs land.

End state

• ForkProtocol is fully typed: no Any, every class-pointer bound by a structural protocol.
• Spec classes own the logic (state transition, fork choice, block production, signature verification). Containers (State, Block, Store) are pure Pydantic data.
• Store is generic over StateT / BlockT; swapping State or Block in a new fork is a typedef, not a copy.
• Subspecs (xmss/, validator/, storage/, sync/, chain/, node/) hold no imports from forks.devnetN.*. Fork-specific types reach them through the Spec or via structural protocols.
• A new devnet can override a single attribute (e.g. block_class = Devnet5Block) on a fork class and inherit 100% of the logic.
• Tests live under tests/lean_spec/forks/devnetN/ and can be mirrored per fork cheaply.
• Capability protocols (PQCapable, NetworkCapable, etc.) exist and drive test selection via @requires(...).

Design decision: ForkProtocol stays non-generic (Vision B)

Two strategies were considered for the class-pointer fields (state_class, block_class, store_class):

	Vision A: Generic protocol	Vision B: Plain narrowable attributes
Shape	class ForkProtocol(ABC, Generic[StateT, BlockT, StoreT])	class ForkProtocol(ABC)
Class-pointer fields	state_class: type[StateT]	state_class: type[StateProto]
Concrete fork	class Devnet4Spec(ForkProtocol[State, Block, Store])	class Devnet4Spec(ForkProtocol) with state_class: type[State] = State
class Devnet5Spec(Devnet4Spec) swapping a single field	Breaks — generics inherit fixed parameters and can't be re-bound on a concrete subclass	Works — Devnet5 overrides block_class: type[Devnet5Block] = Devnet5Block, inherits everything else

Vision B is the chosen direction. The architecture's central use case is "Devnet5 inherits Devnet4's logic and swaps one container type." Vision A makes that impossible. Vision B accepts looser type flow at method call sites in exchange for working subclass-and-narrow inheritance.

Implications:

• No Generic[StateT, BlockT, StoreT] on ForkProtocol.
• No ClassVar on state_class / block_class / store_class (it triggers pyright's invariance rule and blocks subclass narrowing). ClassVar stays correct for plain constants like NAME, VERSION, GOSSIP_DIGEST.
• Method return types in the base ForkProtocol are protocol-typed (StateProto, StoreProto, BlockProto).
• Store's genericity (Stage 5: class Store(Generic[StateT, BlockT])) is independent of ForkProtocol's shape.

Stage 1 — tighten the skeleton

• Add GOSSIP_DIGEST: ClassVar[str] to ForkProtocol; route consumers through fork.GOSSIP_DIGEST. (Stage 1: tighten ForkProtocol surface (trivial items) tcoratger/leanSpec#3)
• Use DEFAULT_REGISTRY in __main__.py instead of constructing a second ForkRegistry(FORK_SEQUENCE). (Stage 1: tighten ForkProtocol surface (trivial items) tcoratger/leanSpec#3)
• Make devnet5/spec.py use direct bindings, not module-level identity aliases. (Stage 1: tighten ForkProtocol surface (trivial items) tcoratger/leanSpec#3)
• Add previous: ClassVar[type[ForkProtocol] | None] linking each fork to its predecessor. (Stage 1: tighten ForkProtocol surface (trivial items) tcoratger/leanSpec#3)
• Rename SpecRunner → ForkRegistry (it's a registry, not a dispatcher). (Stage 1: tighten ForkProtocol surface (trivial items) tcoratger/leanSpec#3)
• Make upgrade_state abstract; remove the silent identity default. (Stage 1: tighten ForkProtocol surface (trivial items) tcoratger/leanSpec#3)
• Replace Any in ForkProtocol method signatures with fork-stable primitives (Uint64, Bytes32) and structural protocols.
• Add SpecBlockType Protocol so block_class stops being typed type (i.e. type[Any]).
• Declare structural attributes on SpecStateType / SpecStoreType (.slot, .config, .head, .blocks, etc.).
• Drop ClassVar from state_class / block_class / store_class. Stays non-Generic per Vision B.
• Delete the generate_pre_state fork fallback in consensus_testing/genesis.py. Require fork everywhere; thread it through the pre pytest fixture.

Stage 2 — move fork-stable types out of the fork package

Several types in forks/devnet4/containers/ are not genuinely fork-specific and should live in lean_spec.types:

• Move Slot.
• Move ValidatorIndex and SubnetId.
• Move Checkpoint.
• Evaluate Config (likely belongs with subspecs/chain/).
• Update ForkProtocol signatures to use these directly.

Stage 3 — decouple subspecs from forks.devnet4

The proposal claimed "subspecs/ contains only fork-agnostic shared libraries." Today most subspecs hard-import from forks.devnet4.containers.*. This stage makes the claim true.

Reference pattern: subspecs/observability/ (#667). Defines a SpecObserver Protocol, exposes a module-level singleton, lets the spec call hooks via context managers. Vendor-specific code lives in a separate adapter (metrics/spec_observer.py). Replicate this shape for the subspecs below.

• subspecs/xmss/ — uses Slot, ValidatorIndex, ValidatorIndices, AggregationBits.
• subspecs/storage/ — takes type[State] and stores Block, State by value.
• subspecs/sync/ — imports Slot, SignedBlock, Store.
• subspecs/chain/ — imports Slot, SignedAttestation.
• subspecs/validator/ — imports Block, BlockSignatures, Slot, ValidatorIndex.
• subspecs/node/node.py — centralize access through config.fork.*_class.
• CI guard: extend the AST-walk test to assert no file under src/lean_spec/subspecs/ imports from lean_spec.forks.devnetN.*.

Stage 4 — move logic from containers to Spec (staged migration)

Container methods (state_transition, process_block, on_block, on_tick, etc.) move from State/Store/SignedBlock to the fork's Spec class. Containers become pure Pydantic data. Inside Spec methods, every literal Block(...) / State(...) / Store(...) becomes self.block_class(...) / self.state_class(...) / self.store_class(...) so inherited methods construct the right fork's containers.

Split into four sub-PRs so each is independently reviewable and revertible:

• PR 4A: Add Spec method surface as delegators (no call-site changes; new surface is unused initially).
• PR 4B: Rewrite ~300 test call sites to go through Spec methods. Decide test ergonomics (parameter-passing vs. fluent wrapper) up front.
• PR 4C: Move method bodies from containers into Spec; replace literal class references with self.*_class(...). Carry observability hooks (observe_state_transition, observe_on_block, observe_on_attestation) along.
• PR 4D: Delete the now-trivial container forwarders.

Stage 5 — generic containers where it pays

• Make Store generic: class Store(StrictBaseModel, Generic[StateT, BlockT]).
• Make BlockLookup generic so Store[Devnet5State, Devnet5Block] composes.
• Document Devnet5Store = Store[Devnet5State, Devnet5Block] as the typedef pattern.

Stage 6 — test reorganization

• Move tests/lean_spec/subspecs/containers/test_state_*.py → tests/lean_spec/forks/devnet4/state/.
• Move tests/lean_spec/subspecs/forkchoice/*.py → tests/lean_spec/forks/devnet4/forkchoice/.
• Mirror structure for devnet5 (initially empty).

Stage 7 — capability protocols

Defer until 2+ forks have real capability divergence — speculative infrastructure otherwise.

• Define one real capability protocol first (e.g. SigScheme bound to current XMSS).
• Add @requires(capability) pytest marker on top of valid_from / valid_until.

Stage 8 — first real devnet divergence

Acceptance test for the architecture. When devnet5 actually changes something:

• Copy forks/devnet4/containers/ → forks/devnet5/containers/ and diverge the changed container(s).
• Override the diverged *_class attribute on Devnet5Spec.
• Implement a real upgrade_state migrating devnet4 state to devnet5 shape.

Non-goals

• Changing SSZ serialization or hash_tree_root behavior.
• Pydantic model inheritance for State (rejected in feat: add multi-fork architecture with ForkProtocol and SpecRunner #638; copy-then-diverge is the supported pattern).
• Runtime fork switching within a single process.
• Backwards-compatible shims for removed container methods.