Skip to content

feat(sdk): canonical-object pipeline — SchemaHandler table, BufferAnchor, push_message_v2 ABI#86

Open
pabloinigoblasco wants to merge 3 commits into
mainfrom
feat/canonical-object-pipeline
Open

feat(sdk): canonical-object pipeline — SchemaHandler table, BufferAnchor, push_message_v2 ABI#86
pabloinigoblasco wants to merge 3 commits into
mainfrom
feat/canonical-object-pipeline

Conversation

@pabloinigoblasco
Copy link
Copy Markdown
Collaborator

Summary

A coherent set of changes across pj_base and pj_plugins that establishes the canonical-object pipeline — the contract by which MessageParser plugins produce decoded scene primitives (images, point clouds, …) and the host routes them through the ObjectStore.

This is a design sketch posted as a draft for review. It compiles cleanly with the companion parser_ros / runtime work and unit tests pass, but the end-to-end flow against real data sources is still pending validation.

What this adds

Canonical-object SDK (pj_base/sdk/canonical_object.hpp)

  • BufferAnchor + PayloadView: zero-copy payload sharing — non-owning Span<const uint8_t> plus a shared_ptr<const void> that keeps the underlying bytes alive.
  • sdk::Image, CompressedImage, PointCloud built around view + anchor; parsers return them without copying source bytes.
  • CanonicalObjectKind enum + SchemaClassification.
  • PixelFormat covers RGB/RGBA, BGR/BGRA, Mono8/Mono16.

MessageParser plugin base

  • SchemaHandler table: per-schema registration with parse_scalars and parse_object callables. The base's classifySchema / parseScalars / parseObject are now table lookups. Plugins call registerSchemaHandler(...) in their constructor or bindSchema to declare what they know.
  • parse() is no longer pure virtual: default implementation routes through parseScalars + writeHost.appendRecord, so plugins that register all their schemas inherit parse() for free.

C ABI extension

  • PJ_payload_t / PJ_payload_anchor_t / PJ_payload_fetcher_t cross-ABI types: idempotent byte-fetcher with a release callback.
  • New push_message_v2 tail slot on PJ_data_source_runtime_host_vtable_t: the DataSource hands the host an idempotent fetcher; the host applies the active ObjectIngestPolicy (kPureLazy / kLazyObjectsEagerScalars / kEager) to decide when (and whether) to invoke it.
  • abi_layout_sentinels_test vtable size updated deliberately: 80 → 104 (MIN_VTABLE_SIZE pinned at the v4.0 baseline of 80).

SDK C++ helpers

  • DataSourceRuntimeHostView::pushMessage template wraps a C++ closure (returning PayloadView or vector<uint8_t>) into a PJ_payload_fetcher_t and delegates to push_message_v2. Returns an explicit error when the host doesn't expose the slot — no silent fallback to the legacy raw-message path.
  • ObjectIngestPolicy + ObjectIngestPolicyResolver with hierarchical override cascade: topic > data_source > kind > default.
  • MessageParserHandle::classifySchema wrapper for the tail-slot call.

Tests

  • object_ingest_policy_test: cascade rules at all four levels + last-write-wins.
  • push_message_v2_test: mock host exercising the template's fetcher wrap (vector + PayloadView shapes), idempotency under repeated fetch, ctx lifetime via shared_ptr canary, anchor propagation past fetcher release, and the explicit error when the host predates the slot.

Test plan

  • Build clean (RelWithDebInfo).
  • Unit tests pass (object_ingest_policy_test, push_message_v2_test, abi_layout_sentinels_test).
  • End-to-end with the companion parser_ros / runtime-host work — pending.

Companion work

The parser-side (parser_ros), the file source (mcap_source) and the runtime host (in the consumer app) ship as separate PRs to keep review surfaces focused.

…hor, push_message_v2 ABI

A coherent set of changes across pj_base and pj_plugins that establishes
the canonical-object pipeline.

Canonical-object SDK (pj_base/sdk/canonical_object.hpp):
- BufferAnchor + PayloadView for zero-copy payload sharing (Span<const
  uint8_t> view + shared_ptr<const void> anchor).
- sdk::Image, CompressedImage, PointCloud canonical types built around
  the view+anchor pattern so parsers can return them without copying
  the source bytes.
- CanonicalObjectKind enum + SchemaClassification descriptor.
- PixelFormat with kRGB888/kRGBA8888/kBGR888/kBGRA8888/kMono8/kMono16.

MessageParser plugin base:
- SchemaHandler table: per-schema registration with parse_scalars and
  parse_object callables. The base's classifySchema / parseScalars /
  parseObject methods are now table lookups. Plugins call
  registerSchemaHandler() in their constructor (or in bindSchema) to
  declare what they know about each type name.
- parse() is no longer pure virtual: default implementation routes
  through parseScalars + writeHost.appendRecord, so plugins that
  register all their schemas via the table inherit parse() for free.

C ABI:
- PJ_payload_t / PJ_payload_anchor_t / PJ_payload_fetcher_t cross-ABI
  types: idempotent byte-fetcher with a release callback.
- New push_message_v2 tail slot on PJ_data_source_runtime_host_vtable_t:
  the DataSource hands the host an idempotent fetcher; the host applies
  the active ObjectIngestPolicy (kPureLazy / kLazyObjectsEagerScalars /
  kEager) to decide when (and whether) to invoke it.
- vtable size sentinel updated deliberately: 80 -> 104 bytes
  (MIN_VTABLE_SIZE pinned at the v4.0 baseline of 80).

SDK C++ helpers:
- DataSourceRuntimeHostView::pushMessage template: wraps a C++ closure
  (returning either PayloadView or vector<uint8_t>) into a
  PJ_payload_fetcher_t and delegates to push_message_v2. Returns an
  explicit error when the host doesn't expose the slot — no silent
  fallback to the legacy raw-message path.
- ObjectIngestPolicy + ObjectIngestPolicyResolver with hierarchical
  override cascade: topic > data_source > kind > default.
- MessageParserHandle::classifySchema wrapper for the tail-slot call.

Canonical-object blob serialization:
- Flat byte layout for Image / CompressedImage / PointCloud crossing
  the C ABI. Writer/reader pair under sdk/detail/.

Tests:
- object_ingest_policy_test: cascade rules at all four levels +
  last-write-wins.
- push_message_v2_test: mock host exercising the template's fetcher
  wrap (vector and PayloadView shapes), idempotency under repeated
  fetch, ctx lifetime via shared_ptr canary, anchor propagation past
  fetcher release, and the explicit error when the host predates the
  slot.

Status: design sketch posted as a draft. Compiles cleanly with the
companion parser/runtime work; not yet exercised end-to-end against
real data sources.
No behavior change. Reformats SDK headers, ABI headers, message
parser plumbing, and the new ingest policy / push_message_v2 tests
per .clang-format (Google style, 120-col, InsertBraces: true).
Output of running pre-commit's clang-format hook over the files
touched in this branch.
The file-level docstrings of canonical_object.hpp and
object_ingest_policy.hpp pointed at a private report path that
does not exist in this repository. Remove the dangling references;
the surrounding prose already explains the design.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant