fix camera recovery blocking the dora event loop

Camera::open() retried forever inside a fixed sleep, so a camera that did not
come back left the node stuck there: it stopped reading dora events and no
longer honoured Stop or AllInputsClosed.

Recovery is now a state machine over ticks. open() makes one attempt and
returns; the loop decides when to try again from a 1/2/5/10 s backoff deadline,
and nothing sleeps, so events keep being read while the camera is down. The
first attempt closes the handle; past that the device is wedged and the node
escalates to sl::Camera::reboot(), guarded to the ZED 2 and newer since it does
nothing on the ZED and ZED Mini. Reopening pins the serial so recovery gets the
same physical camera back.

Set async_grab_camera_recovery so a link hiccup returns CAMERA_REBOOTING
immediately instead of blocking the grab for ~17 s, and enable_image_validity_
check so the SDK flags frames a stalled camera hands back looking valid. Those
transient errors no longer drop the handle, since the SDK is already
reconnecting it.

An all-black frame now counts as a failed grab rather than being published, and
a camera delivering nothing usable for 5 s (30 s while the SDK reports it is
reconnecting) triggers recovery. A new camera_state output publishes
opening/streaming/recovering so a downstream node can tell "no frame yet" from
"camera lost".

ZED_RETRY_INTERVAL_SECONDS is replaced by the backoff.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7g2cBJ5e3r15ZbJ9WZn8c

README.md

@@ -20,6 +20,7 @@ nodes:
       - depth
       - point_cloud
       - camera_info
+      - camera_state
 ```
 
 See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.
@@ -103,7 +104,6 @@ All behavior is controlled via environment variables.
 | `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. |
 | `ZED_FLIP` | `ON` | `ON` or `OFF` |
 | `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
-| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |
 
 The resolved depth mode, both confidence thresholds and the depth range are logged at
 startup. The defaults match the reference camera driver the perception stack was tuned
@@ -140,6 +140,36 @@ Consumers that need 3D back-project `depth` with the published intrinsics instea
 | `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. |
 | `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud, only when `ZED_POINT_CLOUD` is on. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. |
 | `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. Published **once per camera session** — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution. |
+| `camera_state` | `string` | UTF-8 state name, published on each change. Metadata: `dtype=string`, `encoding=utf8`, `host_timestamp_ns`. |
+
+### Camera state and recovery
+
+`camera_state` lets a downstream node tell a camera that has not delivered its first
+frame yet from one that has been lost:
+
+| State | Meaning |
+|---|---|
+| `opening` | The camera has not delivered a usable frame yet in this run |
+| `streaming` | Frames are being published |
+| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back |
+
+Recovery runs inside the dora event loop rather than in a retry loop of its own, so the
+node keeps reading events — and honours `Stop` — while the camera is down. Reopening
+never blocks the loop between attempts: the tick timer paces them, and a backoff deadline
+of 1, 2, 5 then 10 seconds decides which ticks actually try.
+
+The node treats an all-black frame as a failed grab: a camera whose link has dropped does
+not necessarily fail its grabs, and can keep reporting success while handing back a
+uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30
+seconds if the SDK reports it is reconnecting the link itself, since taking the handle
+away mid-reconnect aborts a recovery that was already under way.
+
+The first recovery attempt closes the handle. Past that the device itself is wedged, and
+the node escalates to `sl::Camera::reboot()`, which resets it over USB without unplugging
+it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED
+Mini the node closes the handle and lets the backoff give the endpoint time to clear.
+Every attempt and every escalation is logged.
+
 
 Consumers that need 3D back-project `depth` with the published intrinsics rather than
 consuming `point_cloud`; the point cloud is not a prerequisite for 3D.