Description
Currently, the Scheduler loads/reloads configuration on demand when schedule() is called. If this reload fails (e.g., due to temporary unreadability of ConfigMap, invalid YAML syntax), it raises an exception which propagates up.
- In Ray Serve (
router.py), this exception is caught and the router falls back to random routing.
- In vERL Hook (
verl_hook.py), the exception is NOT caught, causing the rollout worker and the entire Ray training job to crash.
Furthermore, if a reload fails but a previous valid configuration was already loaded, we should ideally fall back to the old configuration rather than failing the request entirely.
Proposed Changes
- Catch exceptions during config reload. If
self.profiles and self.profile_handler are already populated, log a warning and continue using the old configuration instead of raising an exception. Only raise if it's the initial load and we have no fallback.
- If any exception occurs , log the error and return
[] (empty list). Since both Ray Serve and vERL integrations fall back to their default routing (random / global LB) when the scheduler returns no endpoints, this will safely degrade instead of crashing.
Description
Currently, the
Schedulerloads/reloads configuration on demand whenschedule()is called. If this reload fails (e.g., due to temporary unreadability of ConfigMap, invalid YAML syntax), it raises an exception which propagates up.router.py), this exception is caught and the router falls back to random routing.verl_hook.py), the exception is NOT caught, causing the rollout worker and the entire Ray training job to crash.Furthermore, if a reload fails but a previous valid configuration was already loaded, we should ideally fall back to the old configuration rather than failing the request entirely.
Proposed Changes
self.profilesandself.profile_handlerare already populated, log a warning and continue using the old configuration instead of raising an exception. Only raise if it's the initial load and we have no fallback.[](empty list). Since both Ray Serve and vERL integrations fall back to their default routing (random / global LB) when the scheduler returns no endpoints, this will safely degrade instead of crashing.