Implement MVP: distributed P2P storage with E2E encryption and RS erasure coding#18
Open
keanu-thakalath wants to merge 1 commit intomainfrom
Open
Implement MVP: distributed P2P storage with E2E encryption and RS erasure coding#18keanu-thakalath wants to merge 1 commit intomainfrom
keanu-thakalath wants to merge 1 commit intomainfrom
Conversation
Shards are now distributed exclusively across peers — the uploading node stores nothing locally. Any peer failure is tolerated as long as DataShards (4) of the 6 RS shards remain reachable. Core additions - identity: persistent ed25519 keypair (identity.key, 0600) - crypto: AES-GCM file encryption + HKDF key wrapping - storage: content-addressed ShardStore with two-level fanout - manifest: state-based CRDT (G-set replicas, monotone tombstone, LWW) backed by BoltDB; merge is commutative, associative, idempotent - api/signed: SignedStoreRequest/Ack, SignedDeleteRequest, GetRequest/Response - p2p/data_ops: request/response correlation via per-hash waiter channels; pubkeyToPeerID resolver bridges manifest replica IDs (ed25519 hex) to connection peer IDs (IP:port); DataHandler callback interface - p2p/chunked: transparent app-layer fragmentation for datagrams >8 KB (chunkPayloadBytes=4096 stays safely under macOS net.inet.udp.maxdgram) - daemon/app: App singleton wiring identity, store, manifest, P2P client - daemon/operations: UploadFile / DownloadFile / DeleteFile with real logic - daemon/handlers: all 20 CLI commands implemented (replaces stubs) - daemon/server: rewritten to use App; MOSAIC_SOCKET env for multi-daemon Tests - TestFourNodeRedundancy: 4-node cluster (A uploads, B/C/D each hold 2 shards); killing any single peer still allows byte-identical reconstruction via RS repair from the remaining 4 shards - TestUploadDownload_TwoNodes: asserts owner.Store.UsedBytes()==0 (all shards live on the peer) - Race detector clean across all packages
keanu-thakalath
force-pushed
the
distribute-shards-to-peers
branch
from
e80c9cf to
6a6d8b4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Manifest.Replicas()returned ed25519 pubkey hex strings butc.peerswas keyed by IP:port) that was invisible before because download always short-circuited on local shard lookups.SignedStoreRequest/Ack,SignedDeleteRequest, unsignedGetRequest/Response; all signed messages verified with ed25519 + ±5 min clock-skew check.net.inet.udp.maxdgram(9216 bytes): 4 KB payload chunks, safely under limit after JSON/base64 encoding.MOSAIC_SOCKETenv var.Test plan
TestFourNodeRedundancy: 4-node cluster; owner uploads, B/C/D each hold 2 shards; killing any single peer → byte-identical reconstruction from remaining 4 shards (3 sub-tests: B_down, C_down, D_down)TestUploadDownload_TwoNodes: assertsowner.Store.UsedBytes() == 0(all shards on peer); download fetches everything over P2P; large-file chunked transport (~480 KB)go test ./... -race -timeout 180sgo build ./...andgo vet ./...clean