Camera recovery blocks the dora event loop #13

New Issue

2026-09-04T20:35:45Z

cristhian commented

2026-09-04 20:35:45 +00:00

Bug: Camera recovery blocks the dora event loop

Current behaviour

zed::Camera::open() in src/zed/camera.cc retries in an unbounded while (true) loop with a fixed sleep of ZED_RETRY_INTERVAL_SECONDS.
On a failed grab, src/main.cc calls camera.close() then camera.open(). If the camera does not come back, the node stays inside open() indefinitely, stops reading dora events, and no longer honours Stop or AllInputsClosed.
sl::InitParameters::async_grab_camera_recovery is not set, so a link interruption blocks grab() inside the SDK for up to roughly 17 seconds.
There is no blank-frame detection: an all-black frame is published as valid data.
There is no stall detection: a grab that never returns is indistinguishable from a slow one.
There is no escalation to sl::Camera::reboot() for a wedged USB endpoint.

Expected behaviour

The node stays responsive to dora events while the camera is unavailable, recovers when
the camera returns, and does not publish invalid frames.

Requirements

Set sl::InitParameters::async_grab_camera_recovery = true.
Make reconnection non-blocking: return to the dora event loop between attempts so Stop is honoured while the camera is down.
Bound the reopen attempts with a backoff rather than a fixed interval.
Detect all-black frames and treat them as a failed grab rather than publishing them.
Detect a stalled grab and trigger recovery.
Escalate from close()/open() to sl::Camera::reboot() after repeated failures. Guard the call: it applies only to ZED 2 and newer.
Publish camera state so downstream nodes can distinguish "no frame yet" from "camera lost".

Acceptance Criteria

Unplugging the camera during a run does not hang the node; dora stop still terminates the dataflow.
Replugging the camera resumes publishing without restarting the dataflow.
Blank frames are not published.
Backoff intervals and recovery escalation are visible in the node log.
Camera state is observable by a downstream node.

Notes

Reference implementation littlehand/cameras/service.py: STALL_TIMEOUT_S = 5.0,
GRAB_RETRY_S = 0.5, REOPEN_BACKOFF_S = (1, 2, 5, 10),
DRIVER_RECOVERY_TIMEOUT_S = 30.0, and is_blank() which samples a strided subset of
the frame. The escalation there is close() on the first attempt and reset() on
attempt two and later.

The reboot guard is in littlehand/cameras/zed.py: sl.Camera.reboot() reaches ZED 2 and
newer only; on first-generation ZED and ZED_M it costs time and fills the log without
resetting the device.

# Bug: Camera recovery blocks the dora event loop ## Current behaviour - `zed::Camera::open()` in `src/zed/camera.cc` retries in an unbounded `while (true)` loop with a fixed sleep of `ZED_RETRY_INTERVAL_SECONDS`. - On a failed grab, `src/main.cc` calls `camera.close()` then `camera.open()`. If the camera does not come back, the node stays inside `open()` indefinitely, stops reading dora events, and no longer honours `Stop` or `AllInputsClosed`. - `sl::InitParameters::async_grab_camera_recovery` is not set, so a link interruption blocks `grab()` inside the SDK for up to roughly 17 seconds. - There is no blank-frame detection: an all-black frame is published as valid data. - There is no stall detection: a grab that never returns is indistinguishable from a slow one. - There is no escalation to `sl::Camera::reboot()` for a wedged USB endpoint. ## Expected behaviour The node stays responsive to dora events while the camera is unavailable, recovers when the camera returns, and does not publish invalid frames. ## Requirements - Set `sl::InitParameters::async_grab_camera_recovery = true`. - Make reconnection non-blocking: return to the dora event loop between attempts so `Stop` is honoured while the camera is down. - Bound the reopen attempts with a backoff rather than a fixed interval. - Detect all-black frames and treat them as a failed grab rather than publishing them. - Detect a stalled grab and trigger recovery. - Escalate from `close()`/`open()` to `sl::Camera::reboot()` after repeated failures. Guard the call: it applies only to ZED 2 and newer. - Publish camera state so downstream nodes can distinguish "no frame yet" from "camera lost". ## Acceptance Criteria - [ ] Unplugging the camera during a run does not hang the node; `dora stop` still terminates the dataflow. - [ ] Replugging the camera resumes publishing without restarting the dataflow. - [ ] Blank frames are not published. - [ ] Backoff intervals and recovery escalation are visible in the node log. - [ ] Camera state is observable by a downstream node. ## Notes Reference implementation `littlehand/cameras/service.py`: `STALL_TIMEOUT_S = 5.0`, `GRAB_RETRY_S = 0.5`, `REOPEN_BACKOFF_S = (1, 2, 5, 10)`, `DRIVER_RECOVERY_TIMEOUT_S = 30.0`, and `is_blank()` which samples a strided subset of the frame. The escalation there is `close()` on the first attempt and `reset()` on attempt two and later. The reboot guard is in `littlehand/cameras/zed.py`: `sl.Camera.reboot()` reaches ZED 2 and newer only; on first-generation `ZED` and `ZED_M` it costs time and fills the log without resetting the device.

cristhian added this to the v0.2.0 Pure camera source on dora 1.0 milestone 2026-09-04 20:35:45 +00:00

cristhian added the bug priority:high labels 2026-09-04 20:35:45 +00:00

cristhian referenced a pull request that will close this issue

2026-09-08 12:39:27 +00:00

v0.2.0: pure camera source on dora 1.0 #18

cristhian closed this issue

2026-09-08 15:06:41 +00:00

Sign in to join this conversation.