Skip to content

193 gm sound for midi tracks#230

Merged
Sportinger merged 6 commits into
stagingfrom
193-gm-sound-for-midi-tracks
Jun 1, 2026
Merged

193 gm sound for midi tracks#230
Sportinger merged 6 commits into
stagingfrom
193-gm-sound-for-midi-tracks

Conversation

@kfxs

@kfxs kfxs commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

kfxs and others added 6 commits May 31, 2026 19:14
@kfxs @claude
GM sampler: wavetable instrument for MIDI tracks (#193 phases 1,3,4)
be2d848 
Add a General MIDI wavetable instrument alongside the simple oscillator
synth, selectable per MIDI track, behind a shared synth seam.

- Types: MidiInstrument is now a discriminated union (simple-synth | gm);
  createDefaultMidiInstrument(kind?) is kind-aware.
- Seam: IMidiSynth interface + createSynthForInstrument factory route all
  three note producers (live per-track bus, piano-roll preview, offline
  export renderer) through one place. MidiSynth implements IMidiSynth.
- GmSampleBank: shared HMR-persisted singleton storing raw Float32 PCM so a
  buffer builds synchronously for any context rate (live vs export) without
  decodeAudioData. Pure, unit-tested base64->PCM / zone-select / playbackRate
  helpers. Lazy-fetches public/instruments/gm/NNNN.json via BASE_URL; missing
  program degrades gracefully (silent, no crash).
- WavetableSynth: IMidiSynth over the bank (looped/one-shot sample + gain
  envelope from the zone).
- Preload orchestration: export preloads all GM programs once before the clip
  loop; the scheduler preloads on instrument-select (before play) and on
  start, so GM renders with sound and the first live note doesn't drop.
- UI/store: GM in the instrument picker; properties tab branches on kind;
  setTrackMidiInstrument handles kind changes cleanly.
- Asset: hand-built placeholder sine (program 0) proving the pipeline; the
  real FluidR3 .sf2 -> JSON converter is deferred (see GM-Sampler-Plan.md).

Browser-verified: GM audible live and in export, simple-synth regression
intact, graceful missing-asset handling. build + lint + 3315 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kfxs @claude
GM sampler: drums + full program set & picker UI (#193 phases 5,6)
e066390 
Phase 5 — drums:
- GmSampleBank is drum-aware: caches re-keyed by refId(program,isDrum) so a
  melodic program and a drum kit at the same program number are distinct assets;
  drum kits load from their own namespace (instruments/gm/drums/NNNN.json).
- New GmSoundRef type; ensureLoaded/isLoaded/preload are drum-aware.
- selectZoneIndex (pure, tested): covering zone wins; melodic falls back to the
  nearest zone by root key; drums get NO fallback, so an unmapped percussion note
  is silent rather than substituting a wrong drum.
- WavetableSynth plays drum hits as one-shots out to the sample's full length
  (the PCM encodes the decay), not gated by note duration.
- Percussion toggle in the properties tab; placeholder drum kit (0000.json).

Phase 6 — full program set + UI:
- gmPrograms.ts: 128 GM program names, the 16 families, 9 GM/GS drum kits, and
  name lookups. getMidiInstrumentLabel now reports the concrete program/kit name.
- MidiInstrumentTab: grouped program picker (16 optgroups / 128 programs) for
  melodic, drum-kit picker for percussion; toggling percussion resets to a valid
  program. The picker lists every program even though only built assets resolve to
  sound (the rest degrade gracefully to silence).
- Persistence test covers GM program/kit round-trip + clean kind-swap.

Browser-verified end to end (picker renders 16 optgroups/128 programs + 9 kits,
labels resolve, drums audible, unbuilt programs silent with no crash).
build + lint + 3320 tests green. Real FluidR3 assets (the .sf2 converter) remain
the only outstanding work.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kfxs @claude
GM sampler: FluidR3 .sf2 converter + per-zone rate + note-off fix (#193…
3252154 
… 2b)

The offline converter and the runtime/schema groundwork for real GM sounds.
Generated assets are NOT committed — they're large, produced locally from the
.sf2, and will move to a CDN later (GmSampleBank isolates the location). The
picker degrades gracefully when an asset is absent.

- scripts/build-gm-instruments.mjs: offline FluidR3 GM (.sf2) -> JSON converter.
  Resolves SF2 generators (instrument-absolute + preset-offset, range
  intersection), envelope unit conversions (timecents->seconds, centibels->level),
  loop/tune/root-key, and collapses velocity layers to one zone per key range.
  Verified: the grand piano converts to 20 keyboard zones and plays in tune across
  the keyboard with correct loop + note-off. Includes a `measure` dry-run mode for
  sizing the full set.
- gmAsset.ts / GmSampleBank.ts: per-zone `sampleRate` (SF2 samples have differing
  rates), falling back to the asset rate. Backward compatible.
- WavetableSynth: release a melodic note at the actual note-off via
  cancelAndHoldAtTime, instead of waiting for the (often tens-of-seconds) decay to
  finish — real GM decays are long, so a short note must not ring on.
- .gitignore: ignore the source `.sf2`/`soundfonts/` and the generated
  public/instruments/gm/**/*.json (README stays tracked). Untrack the prior
  placeholder assets.
- soundfont2 added as a dev dependency (offline use only).

build + lint green; tests reused (3320, src unchanged since the last run).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kfxs @claude
GM sampler: Int16 + lite/full profiles, committed lite set, sustain f…
5912a85 
…ix (#193)

Asset pipeline & committed lite set:
- Store PCM as Int16 (pcmFormat: 'i16'), lossless and half the size of Float32;
  GmSampleBank decodes via new decodeBase64ToInt16. Legacy/absent = 'f32'.
- Converter gains --profile lite|full. lite = curated 21-instrument set (20 melodic
  + Standard drum kit) downsampled to 22.05kHz (area-averaging decimation, loop points
  rescaled); full = native rate, any program set.
- Commit the lite set (~28MB, 22kHz) into public/instruments/gm so clones/deploys have
  sound with no hosting. .gitignore keeps the full set ignored except an explicit
  allowlist of exactly the 21 lite files. Full set stays reproducible from the .sf2.
- Fix converter drum-kit selection: prefer bank 128 preset 0 (real Standard kit) over
  the first bank-128 preset in file order.

UI:
- Rename the GM instrument kind label 'General MIDI' -> 'Wavetable Synth'.
- Restrict the program/drum pickers to the lite set without dropping the full GM data:
  GM_LITE_PROGRAMS / GM_PICKER_FAMILIES / GM_PICKER_DRUM_KITS. Widen to the full lists
  when the full set is hosted.

Bug fix (sustained instruments didn't sustain):
- WavetableSynth note-off used cancelAndHoldAtTime, which in Chrome failed to anchor the
  sustain, so the release ramp bled across the whole note (organ/pad/strings faded out).
  Anchor the level explicitly with cancelScheduledValues + setValueAtTime(holdLevel),
  matching MidiSynth. Verified via offline render: organ/pad/strings now hold then
  release; piano decays naturally.

Docs: GM-Sampler-Plan.md asset-profiles/sizing/git section. Tests: Int16 decode.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kfxs @claude
Fix: persist MIDI track instrument across reload (#193)
770911e 
The on-disk project save/load rebuilds tracks from an explicit field whitelist
(unlike the in-memory loadState spread), and midiInstrument was never in it — so the
synth + GM program were dropped when written to project.ln.json and not restored on a
hard refresh. Pre-existing gap from #182, surfaced by the GM instrument.

Add midiInstrument to both the projectSave and projectLoad track mappings (mirroring
audioState). Add a save-side regression test and correct the midiPersistence test's
comment that wrongly claimed the in-memory path was the on-disk path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kfxs @claude
MIDI: default new tracks to the Wavetable Synth (#193)
a26bd24 
New MIDI tracks now default to the GM Wavetable Synth (program 0, Acoustic Grand Piano,
in the committed lite set) instead of the simple oscillator synth. Changed the default
kind in createDefaultMidiInstrument; the kind-switch path passes kind explicitly so it
is unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kfxs

kfxs commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator Author

@Sportinger, this is a Lite version of the GM synth. We had to downsample the sounds and remove may instruments because of size. If we want to have better sound quality and more instruments in the synth we have to think in a way of CDN host the samples latter.

Sportinger
Sportinger approved these changes Jun 1, 2026
View reviewed changes

@Sportinger Sportinger left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — build + lint + GM unit tests verified locally, code reviewed. See review comment for details.

@Sportinger

Sportinger commented Jun 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Review:

Read through the full diff and ran the checks locally (CI here only runs the 3 security jobs — it never builds, lints, or tests — so the green checkmark says nothing about correctness).

Verified locally

  • npm run build — clean (~28s)
  • npm run lint — zero warnings
  • gmSampleBank.test.ts — 15/15 (PCM decode, zone selection, pitch math)
  • All three note producers (live scheduler, preview, offline export) route through createSynthForInstrument — no path bypasses the factory
  • Persistence is wired for the main timeline, not just nested comps (root timeline saves as the active composition via getSerializableState(), which spreads the whole track incl. midiInstrument)

Notes / things to know

  1. CI does not test correctness — only native-helper-security, secret-scanning, security-tests. No build/lint/test job exists, so "checks pass" was misleading. Ran them by hand instead.
  2. midiPersistence.test.ts couldn't run in my isolated worktree (known webp.wasm?url junction issue — environment, not a real failure). The pure-logic suite passed.
  3. Intentional behavior change: new MIDI tracks now default to the GM Wavetable Synth (Acoustic Grand Piano) instead of the oscillator synth.
  4. Branch is behind staging (branched at 2.0.7; staging is 2.1.0). Still MERGEABLE, but a rebase before merge would be cleaner.

Design quality

Clean IMidiSynth seam + factory; single HMR-persisted GmSampleBank storing raw Int16/Float32 PCM (not AudioBuffers) so one buffer serves both live 44.1/48k and the export OfflineAudioContext rate with no async decodeAudioData; graceful degradation (missing asset -> silent, never a crash); drums correctly get no pitch-shift and no wrong-sample fallback; full GM set gitignored except a curated 21-instrument "lite" set so clones have sound with no hosting.

Bottom line: solid implementation, approved on the merits. The only real gap is that CI gives a false sense of safety by never building/testing.

Generated with Claude Code

@Sportinger Sportinger merged commit acbedf2 into staging Jun 1, 2026
3 checks passed
@Sportinger Sportinger deleted the 193-gm-sound-for-midi-tracks branch June 1, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Sportinger Sportinger Sportinger approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kfxs @Sportinger