Async scheduling

[gty111]

TODO: refactor each first PP rank has its own scheduler

[Copilot]

Pull request overview

Adds an experimental “async scheduling” execution mode intended to overlap CPU-side scheduling/input preparation with GPU execution, including a GPU-side circular buffer (“FutureMap”) to avoid CPU token synchronization in the single-stage (PP=1) case.

Changes:

• Introduces async-scheduling control flow in Worker (driver + PP ranks) with deferred output finalization and overlapped step launch/resolve/collect.
• Adds FutureMap/AsyncSchedulerContext utilities and extends ModelRunner/Sampler with async-friendly APIs (GPU-returning sampler, async batch execution, async D2H copy).
• Exposes --async-scheduling through the API server and engine wiring.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
gllm/worker.py	Adds async scheduling loops for driver and PP ranks, including prefetch + deferred finalize pipeline.
gllm/scheduler.py	Adds deferred/finalize output processing for placeholder-token approach.
gllm/model_runner.py	Adds async execution primitives (step_launch/resolve/collect, run_batch_async, FutureMap init) and VL embedding handling for placeholders.
gllm/layers/sampler.py	Splits sampler into GPU-returning and CPU-returning paths to support async D2H.
gllm/dist_utils.py	Adds non-blocking PP recv helpers (irecv + wait).
gllm/async_utils.py	New module implementing FutureMap circular buffer + async stream context.
gllm/llm_engine.py	Threads async_scheduling config into worker startup.
gllm/entrypoints/api_server.py	Adds CLI flag and passes it into engine.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.