Handle orphan processes when running as PID 1#502
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces an orphan process reaper module (reaper.rs) in executor_manager to clean up zombie processes when running as PID 1 in a container. It adds the process feature to the nix dependency and spawns a background task that reaps zombies on SIGCHLD signals and periodic intervals. Feedback highlights a critical correctness risk where calling waitpid(-1) globally can intercept and steal exit statuses of child processes spawned by other parts of the application, potentially causing ECHILD errors. It is recommended to use a container init system (like tini) instead, or track spawned PIDs. Additionally, it is suggested to configure the periodic interval to skip missed ticks to avoid burst behaviors.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| loop { | ||
| match waitpid(Pid::from_raw(-1), Some(WaitPidFlag::WNOHANG)) { |
There was a problem hiding this comment.
Calling waitpid with Pid::from_raw(-1) (which reaps any child process) in a global loop poses a significant correctness risk for the application.
The Problem
If flame-executor-manager or any of its dependencies (such as Tokio's process module or standard library commands) spawns child processes (e.g., shims, executors, or sidecars) and expects to wait on them to retrieve their exit status, this global reaper loop will intercept and reap those processes first. When the spawning code subsequently attempts to wait on its specific child PID, it will receive an ECHILD error, leading to unexpected failures or lost exit statuses.
Recommended Solutions
- Use a Container Init System (Best Practice): Instead of implementing PID 1 reaping logic inside the application, run the container with a lightweight init system like tini or dumb-init as the entrypoint. These tools are specifically designed to reap orphans safely at the container level without interfering with the application's internal process management.
- Track Spawned PIDs: If you must reap inside the application, you need to maintain a registry of PIDs spawned by the application. When
waitpid(-1)reaps a process, check if its PID is in the registry. If it is, you must store its exit status in a shared map so that the code waiting for it can retrieve it from the map instead of callingwaitpiddirectly. However, this is complex and requires wrapping all process spawning in the application.
| } | ||
| }; | ||
|
|
||
| let mut reap_interval = interval(Duration::from_secs(5)); |
There was a problem hiding this comment.
By default, Tokio's interval uses MissedTickBehavior::Burst, which can cause multiple rapid ticks to fire if the event loop is delayed or busy. For a periodic background fallback like zombie reaping, it is safer to use MissedTickBehavior::Skip to avoid unnecessary bursts of reaping attempts.
| let mut reap_interval = interval(Duration::from_secs(5)); | |
| let mut reap_interval = interval(Duration::from_secs(5)); | |
| reap_interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); |
44f2710 to
0fa53ca
Compare
Install dumb-init in the executor-manager container image and use it as the entrypoint wrapper. This properly handles PID 1 responsibilities: - Reaps orphaned zombie processes - Forwards signals to child processes Fixes xflops#501
When flame-executor-manager runs as container entrypoint (PID 1), it now reaps orphaned child processes to prevent zombie accumulation.
Changes
Install dumb-init in the executor-manager container image and use it as
the entrypoint wrapper. This properly handles PID 1 responsibilities:
Fixes #501