fix the camera blocking the dora event loop by moving it to its own thread
validate / validate (push) Successful in 59m1s
validate / validate (pull_request) Successful in 59m2s

Hardware testing showed the tick-driven recovery state machine did not work.
async_grab_camera_recovery does not make grab() prompt on a disconnected ZED-M
under SDK 5.3: grabs took ~3 s each while the link was down and then blocked
indefinitely. The event loop stopped turning, so Stop went unanswered, the
30 s patience timer and the backoff were never reached, and replugging the
camera did nothing. Measured: 122 s frozen, then dora stop needed 10 s and a
forced kill.

Move everything that can block onto a capture thread. zed::Capture owns the
camera and runs open, warmup, grab, blank and stall detection, backoff and
escalation; it packs each frame into owned buffers and hands the newest one to
the reader by swapping under a mutex. The dora event loop now only drains that
buffer and publishes, so it never calls into the SDK and stays responsive
however long the SDK blocks.

The newest frame wins and uncollected frames are dropped, which is what a
consumer wants from a camera source. Swapping rather than assigning returns the
previous frame's buffers to the capture thread, so steady state does not
reallocate. The point cloud is now copied out of the SDK buffer rather than
published in place, since the reader is another thread.

On exit the loop waits 2 s for the thread and then _Exit()s if it is still
inside the SDK, so Stop is always answered promptly.

This also removes the blocking warmup from the event loop: it stays a blocking
burst, which is harmless on the capture thread.

Verified on a ZED-M: unplug leaves the node responsive and logs the backoff
1, 2, 5 then 10 s; replug gives "camera recovered after 13 attempt(s)" and
publishing resumes without restarting the dataflow; dora stop with the camera
unplugged returns in 0.70 s with the thread joining gracefully.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7g2cBJ5e3r15ZbJ9WZn8c
This commit is contained in:
cesar
2026-09-08 12:04:35 -03:00
co-authored by Claude Opus 5
parent 2f55ca3f74
commit 5822a870bb
8 changed files with 498 additions and 287 deletions
+14 -4
View File
@@ -152,6 +152,10 @@ Consumers that need 3D back-project `depth` with the published intrinsics instea
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 1080p) |
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
The cloud is copied out of the SDK buffer, since the capture thread and the event loop are
different threads. That copy is the main reason to leave `ZED_POINT_CLOUD` off unless a
consumer needs it.
## Inputs
| Input | Description |
@@ -179,10 +183,16 @@ frame yet from one that has been lost:
| `streaming` | Frames are being published |
| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back |
Recovery runs inside the dora event loop rather than in a retry loop of its own, so the
node keeps reading events — and honours `Stop` — while the camera is down. Reopening
never blocks the loop between attempts: the tick timer paces them, and a backoff deadline
of 1, 2, 5 then 10 seconds decides which ticks actually try.
The camera runs on its own thread. The ZED SDK can block for an unbounded time inside
`grab()` and `open()` when the camera is disconnected — measured blocking indefinitely on
a ZED-M even with `async_grab_camera_recovery` set — so driving it from the dora event
loop would freeze the node: it would stop reading events, leaving `Stop` unanswered and
the recovery logic itself unreachable. The event loop only reads the newest frame the
capture thread has already produced, and therefore never calls into the SDK.
Reopening is paced by a backoff of 1, 2, 5 then 10 seconds, the last repeated. Frames that
are not collected before the next one arrives are dropped: a consumer always gets the
newest frame rather than a queue of stale ones.
The node treats an all-black frame as a failed grab: a camera whose link has dropped does
not necessarily fail its grabs, and can keep reporting success while handing back a