Convergence feasibility of compressing training timesteps to 2 steps in video RL

[utenasama]

Hi @lhmd ,
Follow-up question regarding the sampling trajectory:
In the current implementation, we use 24 SDE steps for training (with the remaining steps running on deterministic ODE). If we aggressively modify this setup to use only 2 SDE steps (and run all the remaining steps on ODE via the ODE-to-SDE hybrid sampling framework), can the policy still successfully converge?
I am curious if you have ever tried this "2 SDE + remaining ODE" setup in video tasks, and whether it would suffer from severe reward sparsity. Thanks!

[lhmd]

Thanks for the follow-up question. In principle, if training stays stable, the "2 SDE + remaining ODE" setup could work for image tasks, but it may not transfer well to video, where reward sparsity tends to be more severe. We haven't tried this specific configuration ourselves, so we can't give a definitive answer, but it would be an interesting direction to explore.