feat: add 2-pass release build for Dynamic Dispatch table#119
Draft
bdero wants to merge 53 commits intoshorebird/devfrom
Draft
feat: add 2-pass release build for Dynamic Dispatch table#119bdero wants to merge 53 commits intoshorebird/devfrom
bdero wants to merge 53 commits intoshorebird/devfrom
Conversation
) * fix: stop using globals for patch data * chore: run et format * chore: add missing files * test: add unittest * chore: run et format * chore: move elf_cache down into runtime * chore: rename elf* to patch* * chore: clean up logs * chore: clean up comments * chore: use Shorebird dart * chore: small cleanup
…ASE_URL (#97) * fix: make dart/flutter work without FLUTTER_STORAGE_BASE_URL * feat: shorebird flutter should work without setting FLUTTER_STORAGE_BASE_URL * fix: flutter_tools test fixes * fix: enable running flutter_tools tests * chore: remove unnecessary workflow
* chore: move build_engine scripts into this repo * chore: fix path of content_aware_hash.sh
* chore: roll dart to 6a78a2deaee05bc74775fcfa2ff27aa53c96efca * wip * chore: run et format * chore: attempt to clean up shorebird.cc * chore: fix build * chore: remove FLUTTER_STORAGE_BASE_URL override
* feat: allow patch_verification_mode * test: update tests * chore: rename to patch_verification
…clude patch_verification option
* es/report_start_fix * fix: second callsite
* chore: add a C++ interface onto the updater * chore: centralize SHOREBIRD_PLATFORM_SUPPORTED * test: fix tests
Previously we stopped reporting start on android by accident. This fixes that. I also removed the once-per-process guard since it's not necessary. This should be correctly reporting once-per-shell and let the rust code only handle the first of the calls. Fixes shorebirdtech/shorebird#3488
As part of our previous fix for FlutterEngineGroup, we introduced a new bug whereby report_launch_start could be called more than once in a multi-engine scenerio. That would cause confusion about what the current boot patch is, since the current patch is updated as part of report_launch_start. report_launch_start should only be called once per processs, which this change fixes. We still need more end-to-end testing at this layer to prevent bugs like this from sneaking in.
…#108) - Create a template Flutter project once in setUpAll and copy it per test, avoiding repeated `flutter create` calls - Run a warm-up `flutter build apk` in setUpAll (outside per-test timeout) to prime the Gradle cache - Add actions/cache for ~/.gradle so subsequent CI runs start warm - Add VERBOSE env var and failure output logging from #107
* chore: split CI into parallel jobs Split the single CI job into three parallel jobs: 1. flutter-tools-tests: Runs on ubuntu + macOS (unchanged) 2. shorebird-android-tests: Runs on Ubuntu only (faster runners) 3. shorebird-ios-tests: Runs on macOS only (requires Xcode) This improves CI performance by: - Running all jobs in parallel instead of sequentially - Moving Android tests off macOS to faster Ubuntu runners - Removing Windows from the matrix (nothing was running there anyway) Expected speedup on macOS: ~5 minutes (no longer runs Android tests) * Add Android smoke test on macOS Run a single "can build an apk" test on macOS to catch any platform-specific issues with Android builds on macOS. * Add comment explaining why Windows is excluded
* feat: add sharded CI build runner Adds a Dart-based shard runner for parallel engine builds: - JSON configs for Linux, macOS, Windows shards - run_shard.dart: executes gn/ninja/rust builds - compose.dart: assembles iOS/macOS frameworks - GCS upload/download for artifact staging * refactor: parse compose.json into typed objects Move from reading compose.json directly as Map<String, dynamic> to using proper ComposeConfig and ComposeDef model classes. This follows a more idiomatic Dart pattern. * feat: add finalize.dart for manifest generation and uploads Implements the finalize job logic that: - Downloads artifacts from GCS staging - Generates artifacts_manifest.yaml - Uploads to production GCS bucket (download.shorebird.dev) Ports upload logic from linux_upload.sh, mac_upload.sh, and generate_manifest.sh into a single Dart script. * fix: correct cargo-ndk invocation for Android Rust builds - Use --target flag (not -t) with Rust target triples - Set ANDROID_NDK_HOME to engine's hermetic NDK - Build all Android targets in a single cargo ndk command - Remove incorrect _androidArch helper function This matches the behavior of linux_build.sh. * test: add unit tests for config parsing Tests cover: - PlatformConfig: single-step shards, multi-step shards, compose_input - BuildStep: gn_ninja and rust step types - ComposeConfig: compose definitions, requires, script, args - Error handling for unknown shards/compose names * feat: add artifacts field to shard configs Define artifacts declaratively in JSON configs instead of hardcoding upload paths in Dart. Each artifact specifies: - src: source path relative to out/ - dst: destination path with $engine placeholder - zip: whether to zip before upload (for directories like dart-sdk) - content_hash: whether to also upload to content-hash path (for Dart SDK) This makes the config self-describing and aligns with Flutter's ci/builders/*.json pattern of explicit artifact declarations. * refactor: read shard names from JSON configs instead of hardcoding Load PlatformConfig for each platform to get shard names dynamically, rather than maintaining a duplicate list in finalize.dart. * feat(ci): add manifest generation and bucket configuration - Extract generateManifest to lib/manifest.dart with tests - Refactor finalize.dart to use artifacts from JSON configs - Add --bucket flag for test uploads to alternate buckets - Add compare_buckets.dart for validating uploads against production * chore: add pubspec.lock for shard_runner * chore: allow shard_runner pubspec.lock in gitignore * chore: use local .gitignore for shard_runner pubspec.lock * refactor: load manifest from template file Move manifest content to artifacts_manifest.template.yaml and update generateManifest to load from template file with sync IO. * fix: fail finalize on download errors instead of continuing A missing shard download means incomplete artifacts. Better to fail loudly than silently upload an incomplete build. * fix: fail on gsutil/zip errors instead of warning Upload and zip failures should halt the build, not silently continue with missing artifacts. * refactor: clean up shard_runner CLI and config parsing - Add shared runChecked() helper to eliminate duplicated Process.run + exit code check patterns across config.dart, gcs.dart, finalize.dart, and compose.dart - Add @immutable annotations to all data classes (via package:meta) - Remove implicit single-step shard shorthand; all shards now use explicit steps arrays in JSON configs - Convert all async file IO to sync equivalents (existsSync, etc.) - Make --engine-src and --run-id mandatory CLI args, removing hidden defaults and GITHUB_RUN_ID env var fallback - Restructure compose.json to use explicit flags/path_args instead of a single args list that guessed flag vs path semantics - Collect outDirs from config upfront rather than accumulating during execution * ci: add shard_runner tests to shorebird_ci workflow - Add analysis_options.yaml (package:lints/recommended with strict mode) - Add shard-runner-tests job with format, analyze, and test steps - Fix stale await on sync PlatformConfig.load in compare_buckets.dart - Reformat all files to Dart standard (80 char width)
Each shard runs on a separate machine, so it needs its own Rust build step for the updater library. Previously only the host/android shards had Rust steps, but all shards that build libflutter need libupdater.a for their specific target triple.
On Windows, gcloud SDK tools like gsutil are installed as .cmd files. When Dart's Process.run is called without runInShell, it doesn't resolve these .cmd extensions. This adds a helper that explicitly checks for .cmd versions in PATH on Windows.
Simulators don't currently support Shorebird's Rust components. Remove the rust build steps from ios-sim-x64, ios-sim-x64-ext, ios-sim-arm64, and ios-sim-arm64-ext shards.
The x64 simulator shards need libupdater.a built for x86_64-apple-ios. The arm64 simulator shards don't require Rust (mac_build.sh doesn't build aarch64-apple-ios-sim either).
The upstream Flutter commit c0b808c changed the Android SDK CIPD package to an "unmodified" layout where the NDK lives at android_tools/sdk/ndk/<version> instead of android_tools/ndk. Update all build scripts to dynamically discover the NDK version directory.
The upstream Dart SDK renamed the dart_io_api GN target to common_embedder_dart_io. Update shell/testing/BUILD.gn to match.
The manifest template still referenced Flutter.dSYM.zip but mac_upload.sh uploads Flutter.framework.dSYM.zip (the new name as of Flutter 3.27.0). Update the template, generate script, and test to match what's actually uploaded. Relates to shorebirdtech/shorebird#3035
#117) * fix: filter non-stable version branches in shorebird version detection The shorebird version detection in GitTagVersion.determine() matches flutter_release/* branches to resolve the Flutter version. When non-stable branch names like flutter_release/3.41.4-rc2 are present, parse() fails to match them against the expected version pattern and returns GitTagVersion.unknown(), causing Flutter to report 0.0.0-unknown. Filter branch names to only match stable versions (X.Y.Z) so that release candidate branches are skipped and the correct version is resolved. Fixes shorebirdtech/shorebird#3662 * test: add tests for shorebird flutter_release branch version filtering Tests that: - A stable flutter_release branch (e.g. 3.41.4) resolves correctly - A non-stable branch (e.g. 3.41.4-rc2) is skipped, falling through to the upstream tag-based fallback - When both stable and rc branches match, the stable one is picked
* Integrate Rust updater build into GN/Ninja Add a GN action() that invokes cargo to build the Rust updater library, so that Ninja (and ET) can build the complete Shorebird engine without requiring a separate shell script prerequisite step. * Fix missing updater dep in runtime BUILD.gn dart_snapshot.cc calls shorebird::Updater::Instance().ReportLaunchStart() but the runtime source_set did not depend on shorebird:updater, causing link failures in test targets that depend on runtime without also pulling in the updater through other deps. * Fix relative NDK path resolution for Android cargo builds GN passes paths relative to the build output dir, but cargo's build scripts (e.g. ring's cc crate) run from a different directory. Resolve all paths to absolute before passing them to cargo/env vars. * Extract updater build config into .gni and glob Rust sources Move triple computation and variables into build_rust_updater.gni to keep BUILD.gn focused on target definitions. Add list_rust_files.py to automatically discover .rs source files at GN gen time via exec_script, so the inputs list stays in sync when the updater repo changes without manual updates.
When building for arm64 Apple platforms with the linker enabled, run gen_snapshot twice: first in ELF mode to produce a temporary snapshot for analyze_snapshot to compute the DD table manifest, caller links, and slot mapping, then in assembly mode with --dd_slot_mapping to produce the final snapshot with indirect calls wired up. The DD table files (App.dd.link, App.dd_callers.link) are copied into the shorebird supplement directory alongside the existing link files so they can be bundled with releases and used during patch builds.
The 2-pass DD table build runs gen_snapshot in ELF mode before the main assembly pass. Update tests to expect this additional command.
The arm64 DD analysis pass delays the arm64 assembly, so x86_64 (which skips DD) completes its build first when both run concurrently via Future.wait.
Move the analyze_snapshot existence check before the gen_snapshot ELF pass so the entire DD computation is a no-op on standard Flutter SDKs that don't ship analyze_snapshot. This fixes iOS smoke test failures where gen_snapshot was being invoked unnecessarily in ELF mode. Also reverts test changes that are no longer needed since DD commands won't appear when analyze_snapshot doesn't exist in the test filesystem.
The arm64 build's async _computeDDTable() check (even a no-op when analyze_snapshot is absent) introduces an await that lets x86_64 reach gen_snapshot first in Future.wait. Reorder test expectations to match the actual interleaving: x86_64 before arm64 at each step.
The DD slot mapping now uses kernel_offset-based function matching (instead of function names). The base build must export an identity side file during gen_snapshot pass 1 and pass it to analyze_snapshot --compute_dd_slot_mapping. Without this, the DDSlotMapping has empty kernel_offset_to_slot and FinalizeIndirectStaticCallTable can't assign any DD slots, resulting in an empty DD table in the base snapshot.
Read SHOREBIRD_DD_MAX_BYTES from the environment to allow overriding the cascade limiter threshold. Defaults to 10000 if not set. An environment variable is used (rather than a command-line flag) so that older Flutter builds without DD table support silently ignore it.
bdero
force-pushed
the
bdero/cascade-limiter
branch
4 times, most recently
from
a9427e1 to
a223f85
Compare
Updates dart_sdk_revision to include SIMARM64 simulator fixes (DoRedirectedCall, ClobberVolatileRegisters, Execute reason param) needed for ios_debug engine builds.
bdero
force-pushed
the
bdero/cascade-limiter
branch
from
a223f85 to
bcc42c8
Compare
The GN action for building the Rust updater via cargo declared only a stamp file as output. The library path referenced by libs[] was in the source tree, so Ninja couldn't map it to any build rule — causing "missing and no known rule to make it" for targets where the pre-built library didn't already exist (e.g. host_debug on macOS). Fix: the build_rust_updater.py script now copies the cargo-built library from the source tree to target_gen_dir, which is declared as an action output. This lets Ninja properly order the link step after the cargo build.
bdero
force-pushed
the
bdero/cascade-limiter
branch
from
bcc42c8 to
5572ff2
Compare
Snapshot of work in progress on the base build's DD pipeline: - debug print in AOTSnapshotter.build for usesDDTable diagnosis - BUILD.gn change to place analyze_snapshot in universal/ next to gen_snapshot so flutter_tools can resolve it by path substitution Committing to a clean base before restructuring the pipeline to support the pre-DD optimized pass added in the dart-sdk patch flow.
When shorebird_flutter ships with the DD table 2-pass release build
enabled but the underlying engine's gen_snapshot binary predates the
DD table work (e.g. a user has downgraded their engine cache, or is
running against a Shorebird release from before DD landed),
gen_snapshot hard-errors on the ELF pass with "Setting VM flags
failed: Unrecognized flags: print_dd_function_identity_to" and the
entire release build fails. Older engines simply don't know about
`--print_dd_function_identity_to`, `--dd_slot_mapping`, or any of the
DD table flag family.
Add a capability probe in _computeDDTable that runs gen_snapshot once
with the DD flag plus a bogus kernel input. Two possible failure
modes distinguish support:
- Flag recognized → flag parsing passes, kernel load fails:
"Can't load Kernel binary: File size is too small to be a valid
kernel file."
- Flag not recognized → VM init fails at flag parsing:
"Setting VM flags failed: Unrecognized flags:
print_dd_function_identity_to"
If stderr contains the "Unrecognized flags" token, skip the entire
DD pipeline and fall back to a plain single-pass Shorebird release
build. The cascade-limiter linker (in aot_tools) independently
handles the no-DD case by falling back to the CT pass's op.link for
final pass OP alignment (see dart-sdk commit 139dd8c864d), so the
patch side of the pipeline keeps working too.
Probe strategies that don't work:
- `gen_snapshot --help` doesn't list individual flags.
- `gen_snapshot --print_flags` exits early on "At least one input
is required" before dumping any flag info.
- Passing the flag without a snapshot kind and kernel silently
ignores it regardless of support.
The flag+bogus-kernel+snapshot-kind combination is the only
invocation that reliably distinguishes recognized-but-failed-later
from outright-rejected, on both DD-aware and pre-DD engines.
Result is cached per gen_snapshot path so multi-arch release builds
don't pay the probe cost more than once.
When SHOREBIRD_DD_MAX_BYTES is set, AOTSnapshotter now performs a 2-pass build: 1. Pass 1: gen_snapshot produces an ELF for analysis + DD identity file 2. analyze_snapshot computes DD table + caller links + slot mapping 3. Pass 2: gen_snapshot rebuilds with --dd_slot_mapping for DD-enabled code This produces DD-aware release snapshots where high-fanout cascade functions are routed through the indirect static call table, enabling the cascade limiter's link percentage benefit. Also adds DD supplement files (App.dd.link, App.dd_callers.link, App.dd_identity.link, App.dd_slots.link) to the LinkSupplement copy list so they're propagated to the Shorebird CLI's supplement directory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
analyze_snapshotto compute DD table manifest, caller links, and slot mapping--print_dd_function_identity_to) mapping each Code object's InstructionsTable index to its Function'skernel_offset, which is passed toanalyze_snapshot --compute_dd_slot_mappingvia--dd_function_identityfor kernel_offset-based function matching--dd_slot_mappingApp.dd.link,App.dd_callers.link) are copied into the supplement directory alongside existing link filesTest plan