Merge pull request 'Split the opencv-plot viewer into its own dataflow' (#19) from split-plot-dataflow into main

Reviewed-on: #19

.gitea/workflows/validate.yml

@@ -0,0 +1,36 @@
+# Checks both dataflows against the typed contract in dora-node.yml. This runs
+# no node, so it needs neither a camera nor the ZED SDK.
+name: validate
+
+on: [push, pull_request]
+
+env:
+  DORA_VERSION: v1.0.1
+  DORA_TARGET: x86_64-unknown-linux-gnu
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install the dora CLI
+        run: |
+          curl -fsSL -o dora-cli.tar.gz \
+            "https://github.com/dora-rs/dora/releases/download/${DORA_VERSION}/dora-cli-${DORA_TARGET}.tar.gz"
+          tar xzf dora-cli.tar.gz
+          mkdir -p "$HOME/.local/bin"
+          install -m 755 "dora-cli-${DORA_TARGET}/dora" "$HOME/.local/bin/dora"
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+          rm -rf dora-cli.tar.gz "dora-cli-${DORA_TARGET}"
+
+      - run: dora --version
+
+      # strict_types: true in each dataflow turns a type warning into a
+      # non-zero exit, so a mismatched annotation fails here rather than at
+      # runtime. The plot file is checked too: it wires the same outputs, and a
+      # rename that breaks only the viewer should still fail the build.
+      - name: Validate the dataflows
+        run: |
+          dora validate dataflow.yml
+          dora validate dataflow_plot.yml

.gitignore

@@ -1,6 +1,7 @@
 build/
 bin/
 tmp/
-test/assets/*.png
-test/assets/*.jpg
-test/assets/*_result.*
+
+# dora build working directories
+git/
+out/

AGENTS.md

@@ -4,17 +4,21 @@ This file provides guidance to AI coding assistants when working with code in th
 
 ## What this is
 
-A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame, runs object detection, and publishes four outputs over the dora dataflow: a BGR image, an XYZRGBA point cloud, camera intrinsics, and detected objects with 3D positions. Camera management lives in `src/zed/`, detection engines in `src/engines/`, and the dora event loop in `src/main.cc`.
+A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame and publishes it over the dora dataflow: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Camera management lives in `src/zed/` and the dora event loop in `src/main.cc`.
+
+The node is a camera source only. It ran inference in-process through `v0.1.0`, under `src/engines/`; that was removed so the node has no inference dependency and builds on any platform the ZED SDK supports. The removed code remains reachable at tag `v0.1.0`.
 
 ## Build
 
 ```bash
-mkdir -p build && cd build && cmake .. && make install
+scripts/build.sh
 ```
 
-The binary is installed to `bin/dora_zed` relative to the repo root.
+The binary is installed to `bin/dora_zed` relative to the repo root. The script is also
+the dataflow's `build:` command: dora 1.0 executes build commands directly rather than
+through a shell, so `cd`, `&&` and command substitution do not work there.
 
-**Requirements:** CMake 3.21+, ZED SDK 5.x (at `/usr/local/zed`), CUDA Toolkit, ONNX Runtime 1.20+ (system install), Rust stable toolchain (for dora-rs C++ bindings).
+**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x (at `/usr/local/zed`), CUDA Toolkit, Rust stable toolchain (for dora-rs C++ bindings).
 
 The first build clones dora-rs from GitHub and compiles `dora-node-api-cxx` via Cargo. This takes a while. Subsequent builds are incremental. If `DORA_ROOT_DIR` is set to a local dora checkout, it builds against that instead.
 
@@ -24,32 +28,33 @@ For a build without installing the SDK stack on the host, see the `Dockerfile` (
 
 ```bash
 dora up
-dora start dataflow.yml
+dora start dataflow.yml       # camera alone
+dora start dataflow_plot.yml  # camera plus the opencv-plot viewer
 ```
 
-Camera and detection behavior is configured entirely through environment variables (see README.md for the full list: `ZED_RESOLUTION`, `ZED_FPS`, `ZED_DEPTH_MODE`, `ZED_DETECTION_MODEL`, etc.).
+The two dataflows both declare the camera node, so run one at a time.
+
+Camera behavior is configured entirely through environment variables (see README.md for the full list: `ZED_RESOLUTION`, `ZED_FPS`, `ZED_DEPTH_MODE`, etc.).
 
 ## Architecture
 
 - **`src/util/env.h`** — Shared inline env-variable parsing helpers (`get_env_int`, `get_env_str`, `get_env_bool`, `get_env_float`).
-- **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: open (with retry loop), warmup, grab (image + point cloud), calibration, and close. Exposes `handle()` for engine access.
-- **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace.
-- **`src/engines/engine.h`** — `engines::DetectedObject` struct and `engines::Engine` abstract base class (`enable`, `detect`, `objects`, `label_names`).
-- **`src/engines/zed_native.h/cc`** — `engines::ZedNative`: wraps ZED SDK's TensorRT-optimized object detection module.
-- **`src/engines/yolo_helpers.h`** — Header-only shared helpers used by the ONNX engine: `RawDetection`, `nms()`, `letterbox()`, `fuse_bbox()`, `backproject()`. Lives in `engines::detail` namespace.
-- **`src/engines/yolo_onnx.h/cc`** — `engines::YoloOnnx` / `YoloOnnxConfig`: closed-vocabulary detection for any single-input YOLO ONNX model (YOLOv8, YOLOv11, YOLOv5, etc.). Auto-detects output layout (`NC_FIRST` / `NC_LAST` / `NC_LAST_OBJ`) from tensor shape; override with `ZED_ONNX_OUTPUT_FORMAT`. Labels loaded from `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE`.
-- **`src/main.cc`** — Dora event loop. Selects engine via `ZED_ENGINE` env var (`yolo_onnx`/`onnx` or `native`), holds a `std::unique_ptr<engines::Engine>`. Parses `ZED_PC_MODE` (`FULL`/`HALF`/`ROI`) once before the loop. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()` and `detector->detect()`, then sends `image_bgr`, `point_cloud`, `camera_info`, and `objects` outputs. Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension; `ROI` emits only valid points within detected bounding boxes as a flat `[N, 4]` array. Handles reconnection on grab failure.
+- **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: `open()` (one attempt, returns success), warmup, grab (image + depth + point cloud), calibration, device timestamp, blank-frame check, `close()` and `reset()` (close plus `sl::Camera::reboot()` where the model allows it). Exposes `handle()` for engine access.
+- **`src/zed/capture.h/cc`** — `zed::Capture` and `zed::Frame`. Owns the camera on a dedicated thread: open, warmup, grab, blank/stall detection, backoff and escalation all live here, along with the packing of every payload into owned buffers. The newest frame is handed to the reader by swapping under a mutex (`take_frame`), which also recycles the previous frame's buffers so steady-state capture does not reallocate. **Nothing outside this file may call the ZED SDK** — it blocks unboundedly on a disconnected camera, which is what froze the node when the loop drove the camera directly.
+- **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace. `parse_depth_mode` runs in `Config::from_env()`, not in the constructor, so an unrecognised `ZED_DEPTH_MODE` exits before the node joins the dataflow rather than falling back to a default.
+- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()`, then sends `image_bgr` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `pack_depth` copies the depth map into a tightly packed `[H, W]` float buffer (sl::Mat rows can be padded) and clears `inf`/`nan` to `0`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds). Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension. The loop never touches the SDK: it drains `take_state_change` and `take_frame` from the capture thread and publishes. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node.
 - **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue.
-- **`dataflow.yml`** — Example dora dataflow definition using the `git:` source format.
-- **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, ONNX Runtime, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. ONNX Runtime's CMake package files are relocated to `/usr/local/lib/cmake` because Ubuntu's multiarch CMake does not search `lib64`. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`.
-- **`scripts/export_yolo.py`** — PEP 723 uv script. Exports a standard YOLOv11 model to ONNX for closed-vocabulary detection. Run: `uv run scripts/export_yolo.py --model n --output-dir models/`.
-- **`src/test_yolo_onnx.cc`** — Standalone C++ test binary (`bin/test_yolo_onnx`). Exercises `YoloOnnx` with an image (default `test/assets/yolo.png`). No dora and no live camera required. Uses the same env vars as the main binary. Saves an annotated result image alongside the input.
+- **`scripts/build.sh`** — Configure/compile/install in one shell-free-callable step. Referenced by both dataflows, the `Dockerfile` and the README.
+- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why both dataflows build from the checkout. Type URNs name the element type (`std/core/v1/UInt8`, `.../Float32`, `.../String`) because every payload is a raw buffer described by its metadata, not an Arrow struct.
+- **`dataflow.yml`** — Example dora dataflow: the camera alone, no viewer and no Python dependency. Built from this checkout with `path:` + `build:` and `strict_types: true`. A consumer wires the node from a tag instead (see README) — dora 1.0 pins a source with the mutually exclusive `tag:`, `rev:` or `branch:`.
+- **`dataflow_plot.yml`** — The same camera node plus an `opencv-plot` viewer on `image_bgr`. Split out because the viewer pulls dora-hub over `pip` and needs a display. dora 1.0 has no cross-dataflow wiring (an input's `source` must resolve to a node in the same descriptor), so the camera node is duplicated here rather than referenced, and the two files are run one at a time.
+- **`.gitea/workflows/validate.yml`** — CI gate running `dora validate` on both dataflows on push and pull request. Needs no camera and no ZED SDK.
+- **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`.
 
 ## Key APIs
 
 - **dora C++ API** (`<dora-node-api.h>`): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`.
-- **ZED SDK** (`<sl/Camera.hpp>`): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`, `sl::ObjectDetectionParameters`, `sl::Objects`.
-- **ONNX Runtime** (`<onnxruntime_cxx_api.h>`): `Ort::Env`, `Ort::Session`, `Ort::SessionOptions`, `Ort::Value`. Used by the `yolo_onnx` engine.
+- **ZED SDK** (`<sl/Camera.hpp>`): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`.
 - Output data is sent as raw byte slices (`rust::Slice<const uint8_t>`) with typed metadata (shape, dtype, encoding).
 
 ## Tools

CMakeLists.txt

@@ -1,9 +1,15 @@
 cmake_minimum_required(VERSION 3.21)
-project(dora_zed VERSION 0.1.0 LANGUAGES C CXX)
+project(dora_zed VERSION 0.2.0 LANGUAGES C CXX)
 
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_FLAGS "-fPIC")
 
+# DoraTargets.cmake picks the cargo profile from this, so an unset build type
+# would link debug dora bindings into an otherwise unoptimized binary.
+if(NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
+    set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type" FORCE)
+endif()
+
 include(DoraTargets.cmake)
 
 set(ZED_DIR "/usr/local/zed" CACHE PATH "Path to the ZED SDK")
@@ -13,18 +19,13 @@ find_package(zed REQUIRED)
 
 find_package(OpenCV REQUIRED)
 
-find_package(onnxruntime REQUIRED)
-set(ONNXRUNTIME_INCLUDE_DIRS "")  # propagated via imported target
-set(ONNXRUNTIME_LIBRARIES onnxruntime::onnxruntime)
-
 link_directories(${dora_link_dirs})
 link_directories(${ZED_LIBRARY_DIR})
 
 add_executable(dora_zed
     src/main.cc
     src/zed/camera.cc
-    src/engines/zed_native.cc
-    src/engines/yolo_onnx.cc
+    src/zed/capture.cc
     ${node_bridge}
 )
 add_dependencies(dora_zed Dora_cxx)
@@ -38,29 +39,8 @@ target_include_directories(
         ${ZED_INCLUDE_DIRS}
         ${OpenCV_INCLUDE_DIRS}
         ${CUDAToolkit_INCLUDE_DIRS}
-        ${ONNXRUNTIME_INCLUDE_DIRS}
 )
 
-target_link_libraries(dora_zed dora_node_api_cxx ${ZED_LIBRARIES} ${OpenCV_LIBS} ${ONNXRUNTIME_LIBRARIES})
+target_link_libraries(dora_zed dora_node_api_cxx ${ZED_LIBRARIES} ${OpenCV_LIBS})
 
 install(TARGETS dora_zed DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/bin)
-
-# Standalone detector tests — no dora, no live camera required
-
-add_executable(test_yolo_onnx
-    src/test_yolo_onnx.cc
-    src/engines/yolo_onnx.cc
-)
-target_include_directories(test_yolo_onnx PRIVATE
-    ${CMAKE_CURRENT_SOURCE_DIR}/src
-    ${ZED_INCLUDE_DIRS}
-    ${OpenCV_INCLUDE_DIRS}
-    ${CUDAToolkit_INCLUDE_DIRS}
-    ${ONNXRUNTIME_INCLUDE_DIRS}
-)
-target_link_libraries(test_yolo_onnx PRIVATE
-    ${ZED_LIBRARIES}
-    ${OpenCV_LIBS}
-    ${ONNXRUNTIME_LIBRARIES}
-)
-install(TARGETS test_yolo_onnx DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/bin)

Dockerfile

@@ -21,7 +21,6 @@
 FROM nvidia/cuda:12.8.1-devel-ubuntu24.04
 
 ARG ZED_VERSION=5.2
-ARG ORT_VERSION=1.24.4
 
 ENV DEBIAN_FRONTEND=noninteractive
 
@@ -51,25 +50,7 @@ RUN curl -fSL -o /tmp/ZED_SDK.run \
     && chmod -R a+rX /usr/local/zed \
     && rm -f /tmp/ZED_SDK.run
 
-# The tarball puts its CMake package files under lib/, but onnxruntimeTargets.cmake
-# resolves the library as <prefix>/lib64. Ubuntu is multiarch, so CMake does not
-# search lib64 for packages and ld.so does not search /usr/local/lib64 at all:
-# keep the libraries in lib64, move only the package files to lib/cmake, and add
-# lib64 to the linker path.
-RUN curl -fSL -o /tmp/ort.tgz \
-        "https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VERSION}/onnxruntime-linux-x64-gpu-${ORT_VERSION}.tgz" \
-    && tar -xzf /tmp/ort.tgz -C /tmp \
-    && mkdir -p /usr/local/include/onnxruntime /usr/local/lib64 \
-    && cp -r /tmp/onnxruntime-linux-x64-gpu-${ORT_VERSION}/include/* /usr/local/include/onnxruntime/ \
-    && cp -r /tmp/onnxruntime-linux-x64-gpu-${ORT_VERSION}/lib/* /usr/local/lib64/ \
-    && mkdir -p /usr/local/lib/cmake \
-    && mv /usr/local/lib64/cmake/onnxruntime /usr/local/lib/cmake/ \
-    && rmdir /usr/local/lib64/cmake \
-    && echo /usr/local/lib64 > /etc/ld.so.conf.d/onnxruntime.conf \
-    && ldconfig \
-    && rm -rf /tmp/ort.tgz /tmp/onnxruntime-linux-x64-gpu-${ORT_VERSION}
-
 ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
 
 WORKDIR /src
-CMD ["bash", "-lc", "mkdir -p build && cd build && cmake .. && make -j$(nproc) install"]
+CMD ["scripts/build.sh"]

README.md

@@ -1,6 +1,6 @@
 # dora-zed
 
-Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.
+Dora node for ZED stereo cameras. Publishes a BGR image and a depth map on every frame, plus camera intrinsics, a camera state, and an optional point cloud.
 
 ## Usage
 
@@ -10,29 +10,69 @@ Add to your dataflow:
 nodes:
   - id: zed-camera
     git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
-    git_checkout: v0.1.0
-    build: mkdir -p build && cd build && cmake .. && make install
+    tag: v0.2.0
+    build: scripts/build.sh
     path: bin/dora_zed
     inputs:
       tick: dora/timer/millis/100
     outputs:
       - image_bgr
+      - depth
       - point_cloud
       - camera_info
-      - objects
+      - camera_state
 ```
 
-See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.
+### Runnable examples
+
+Two dataflows live here, and they are run one at a time — dora has no cross-dataflow
+wiring, so each declares the camera node itself:
+
+| File | Contents | Run |
+|---|---|---|
+| [`dataflow.yml`](dataflow.yml) | The camera alone | `dora run dataflow.yml` |
+| [`dataflow_plot.yml`](dataflow_plot.yml) | The camera plus an `opencv-plot` viewer on `image_bgr` | `dora run dataflow_plot.yml` |
+
+The viewer is a separate file because it pulls [dora-hub](https://github.com/dora-rs/dora-hub)
+and installs it with `pip`, and it needs a display. Neither is a cost the camera should
+carry on a headless robot, so `dataflow.yml` has no Python dependency at all.
+
+Both build the node from the checkout (`path:` + `build:`) rather than from a tag, which
+is what lets `dora validate` pick up the manifest described below.
+
+### Node manifest
+
+[`dora-node.yml`](dora-node.yml) declares the node's inputs and outputs with type URNs.
+No source package is shared between node repositories, so it is the only contract a
+consumer has, and it is what makes a wiring mistake fail at build time instead of at
+runtime:
+
+```
+$ dora validate dataflow.yml
+  node "zed-camera": injected 6 contract type(s) from .../dora-node.yml
+All type annotations OK.
+```
+
+Types are `std/core/v1/UInt8` for `image_bgr`, `std/core/v1/Float32` for `depth`,
+`camera_info` and `point_cloud`, and `std/core/v1/String` for `camera_state`; every
+payload is a raw buffer whose shape and dtype are in the metadata, so the URN names the
+element type rather than a struct. The `tick` input is `std/core/v1/UInt64`, matching how
+dora types timer inputs.
+
+`strict_types: true` in both dataflows makes a type warning a non-zero exit.
+[`.gitea/workflows/validate.yml`](.gitea/workflows/validate.yml) validates both on every
+push and pull request; it needs no camera and no ZED SDK, since validation never runs the
+node.
 
 ## Dependencies
 
 | Dependency | Version | Notes |
 |---|---|---|
+| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time |
 | [CMake](https://cmake.org/download/) | 3.21+ | |
 | [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
 | [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
 | [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
-| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required at configure time; used at runtime only by `yolo_onnx` |
 
 ### Installation scripts
 
@@ -42,23 +82,27 @@ See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera
 | NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
 | Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
 
-> **Jetson:** No official ONNX Runtime GPU binary for ARM64. It is still needed to configure the build; only the `yolo_onnx` engine uses it at runtime. Requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
->
-> **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
+> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release).
 
 ## Local build
 
 ```bash
 git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
 cd dora-zed
-mkdir -p build && cd build && cmake .. && make install
+scripts/build.sh
 ```
 
 Binary installed to `bin/dora_zed`.
 
+`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build
+commands directly rather than through a shell, so `cd`, `&&` and command substitution are
+not available there; keeping the sequence in one script is what makes the dataflow build
+work. It configures a `Release` build, which is also what links the optimized dora
+bindings.
+
 ### Containerized build
 
-[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK, ONNX Runtime and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.
+[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.
 
 ```bash
 docker build -t dora-zed-build .
@@ -66,6 +110,8 @@ docker run --rm --gpus all --user "$(id -u):$(id -g)" \
     -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
 ```
 
+The container runs the same `scripts/build.sh`.
+
 The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs.
 `--user` keeps `build/` and `bin/` owned by you rather than root.
 
@@ -87,105 +133,111 @@ All behavior is controlled via environment variables.
 
 | Variable | Default | Description |
 |---|---|---|
-| `ZED_RESOLUTION` | `720` | `720`, `1080`, or `2000` |
+| `ZED_RESOLUTION` | `1080` | `720`, `1080`, or `2000` |
 | `ZED_FPS` | `15` | Frame rate |
-| `ZED_DEPTH_MODE` | `NEURAL` | `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
-| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth (mm) |
-| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth (mm) |
-| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
+| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY`, or `NONE`. An unrecognised value is a startup error, not a fallback. |
+| `ZED_DEPTH_MIN_MM` | `200` | Minimum depth (mm) |
+| `ZED_DEPTH_MAX_MM` | `600` | Maximum depth (mm) |
+| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) |
+| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) |
+| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. |
 | `ZED_FLIP` | `ON` | `ON` or `OFF` |
 | `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
-| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |
 
-### Detection engine
-
-Set `ZED_ENGINE` to select the detection backend:
-
-| `ZED_ENGINE` | Custom model? | Classes |
-|---|---|---|
-| `native` (default) | No — ZED SDK built-in TensorRT | Fixed: 7 categories |
-| `yolo_onnx` (alias: `onnx`) | Yes — any single-input YOLO ONNX (v5/v8/v11) | Fixed labels from file or env var |
-
-#### `native` engine
-
-Uses ZED SDK's built-in TensorRT models — no model files to download or configure. `ZED_DETECTION_MODEL` selects the speed/accuracy variant, not a file path.
-
-Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.
-
-| Variable | Default | Description |
-|---|---|---|
-| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
-| `ZED_DETECTION_CONFIDENCE` | `40` | Confidence threshold (0–100) |
-| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking |
-
-#### `yolo_onnx` engine (closed-vocabulary)
-
-For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).
-
-Export a YOLOv11 model once:
-
-```bash
-uv run scripts/export_yolo.py --model n --output-dir models/
-```
-
-Then run:
-
-```bash
-ZED_ENGINE=yolo_onnx \
-ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
-ZED_ONNX_LABELS_FILE=models/coco80.txt \
-dora start dataflow.yml
-```
-
-**Required:**
-
-| Variable | Description |
-|---|---|
-| `ZED_ONNX_MODEL_PATH` | Path to YOLO ONNX model |
-| `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE` | Class names (inline CSV or one-per-line file) |
-
-**Optional:**
-
-| Variable | Default | Description |
-|---|---|---|
-| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
-| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
-| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
-| `ZED_ONNX_OUTPUT_FORMAT` | `auto` | Output layout: `auto`, `nc_first`/`yolov8`/`yolov11`, `nc_last`, `nc_last_obj`/`yolov5`. `auto` detects from tensor shape. |
+The resolved depth mode, both confidence thresholds and the depth range are logged at
+startup. The defaults match the reference camera driver the perception stack was tuned
+against, so depth quality on the robot matches what it expects.
 
 ### Point cloud
 
+Off by default. With `ZED_POINT_CLOUD` unset the XYZRGBA measure is not retrieved at all
+and no `point_cloud` output is published — at 1080p a full cloud is roughly 14.7 MB per
+frame, and retrieving it costs capture time whether or not anything consumes it.
+Consumers that need 3D back-project `depth` with the published intrinsics instead.
+
+| Variable | Default | Description |
+|---|---|---|
+| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud |
+| `ZED_PC_MODE` | `FULL` | Only applies when `ZED_POINT_CLOUD` is on |
+
 | `ZED_PC_MODE` | Shape | Description |
 |---|---|---|
-| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) |
+| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 1080p) |
 | `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
-| `ROI` | `[N, 4]` | Valid points within detected bounding boxes only |
+
+The cloud is copied out of the SDK buffer, since the capture thread and the event loop are
+different threads. That copy is the main reason to leave `ZED_POINT_CLOUD` off unless a
+consumer needs it.
 
 ## Inputs
 
 | Input | Description |
 |---|---|
-| `tick` | Triggers a camera grab and detection. Connect to `dora/timer/millis/N`. |
+| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. |
 
 ## Outputs
 
 | Output | Type | Description |
 |---|---|---|
-| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, `timestamp_ns`. |
-| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics. |
-| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients. |
-| `objects` | `float32 [N, 37]` | Detected objects. See below. |
+| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, timestamps. |
+| `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. |
+| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud, only when `ZED_POINT_CLOUD` is on. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. |
+| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. Published **once per camera session** — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution. |
+| `camera_state` | `string` | UTF-8 state name, published on each change. Metadata: `dtype=string`, `encoding=utf8`, `host_timestamp_ns`. |
 
-### `objects` layout
+### Camera state and recovery
 
-Each of the N rows contains 37 floats:
+`camera_state` lets a downstream node tell a camera that has not delivered its first
+frame yet from one that has been lost:
 
-| Index | Field | Description |
-|---|---|---|
-| 0 | `label_id` | Class index |
-| 1 | `confidence` | Score (0–100) |
-| 2–4 | `x, y, z` | 3D center (mm) |
-| 5–12 | `bb2d` | 2D bounding box: 4 corners × (x, y) px |
-| 13–36 | `bb3d` | 3D bounding box: 8 corners × (x, y, z) mm |
+| State | Meaning |
+|---|---|
+| `opening` | The camera has not delivered a usable frame yet in this run |
+| `streaming` | Frames are being published |
+| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back |
 
-Metadata includes `engine`, `label_names`, `image_width`, `image_height`, and `timestamp_ns`. An empty array `[0, 37]` is sent when no objects are detected.
+The camera runs on its own thread. The ZED SDK can block for an unbounded time inside
+`grab()` and `open()` when the camera is disconnected — measured blocking indefinitely on
+a ZED-M even with `async_grab_camera_recovery` set — so driving it from the dora event
+loop would freeze the node: it would stop reading events, leaving `Stop` unanswered and
+the recovery logic itself unreachable. The event loop only reads the newest frame the
+capture thread has already produced, and therefore never calls into the SDK.
+
+Reopening is paced by a backoff of 1, 2, 5 then 10 seconds, the last repeated. Frames that
+are not collected before the next one arrives are dropped: a consumer always gets the
+newest frame rather than a queue of stale ones.
+
+The node treats an all-black frame as a failed grab: a camera whose link has dropped does
+not necessarily fail its grabs, and can keep reporting success while handing back a
+uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30
+seconds if the SDK reports it is reconnecting the link itself, since taking the handle
+away mid-reconnect aborts a recovery that was already under way.
+
+The first recovery attempt closes the handle. Past that the device itself is wedged, and
+the node escalates to `sl::Camera::reboot()`, which resets it over USB without unplugging
+it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED
+Mini the node closes the handle and lets the backoff give the endpoint time to clear.
+Every attempt and every escalation is logged.
+
+### Timestamps
+
+Every frame output — `image_bgr`, `depth`, `point_cloud` and `camera_info` — carries both
+clocks, as dora `timestamp` metadata parameters. `camera_state` carries
+`host_timestamp_ns` only, since it is not tied to a frame.
+
+| Key | Epoch | Unit | Source |
+|---|---|---|---|
+| `device_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE)` — when the camera captured the frame |
+| `host_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `std::chrono::system_clock` on the host — when this node published the frame |
+
+Both are needed. The device timestamp is the capture instant and is the value two
+nodes reading the same frame agree on; the host timestamp is the one the rest of the
+dataflow and the session recorder share. On a Jetson the wall clock can step when NTP
+synchronises after boot, so the device timestamp is recorded alongside the host one
+rather than in place of it.
+
+The node calls `sl::setTimestampClock(SYSTEM_CLOCK)` before opening the camera, which
+is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock.
+
+Every output for a frame also carries the same `frame_id`, an integer counter starting
+at 0.

dataflow.yml

@@ -1,18 +1,20 @@
+# The repository's own example: builds and runs the node from this checkout, so
+# `dora validate` picks up the typed contract in dora-node.yml next to it.
+# A consumer wires the node from a tag instead — see README.md.
+#
+# The camera alone, with no viewer and no Python dependency. For a run that
+# also displays the stream, use dataflow_plot.yml.
+strict_types: true
+
 nodes:
   - id: zed-camera
-    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
-    git_checkout: v0.1.0
-    build: mkdir -p build && cd build && cmake .. && make install
     path: bin/dora_zed
+    build: scripts/build.sh
     inputs:
       tick: dora/timer/millis/100
     outputs:
       - image_bgr
-      - objects
-
-  - id: plot
-    git: https://github.com/dora-rs/dora-hub.git
-    build: pip install ./node-hub/opencv-plot
-    path: opencv-plot
-    inputs:
-      image: zed-camera/image_bgr
+      - depth
+      - point_cloud
+      - camera_info
+      - camera_state

dataflow_plot.yml

@@ -0,0 +1,27 @@
+# The camera plus an opencv-plot viewer on image_bgr.
+#
+# Separate from dataflow.yml because the viewer pulls dora-hub and installs it
+# with pip, and needs a display: the camera on its own should cost neither.
+# dora has no cross-dataflow wiring, so the camera node is declared here too
+# rather than referenced — run this file or dataflow.yml, not both.
+strict_types: true
+
+nodes:
+  - id: zed-camera
+    path: bin/dora_zed
+    build: scripts/build.sh
+    inputs:
+      tick: dora/timer/millis/100
+    outputs:
+      - image_bgr
+      - depth
+      - point_cloud
+      - camera_info
+      - camera_state
+
+  - id: plot
+    git: https://github.com/dora-rs/dora-hub.git
+    build: pip install ./node-hub/opencv-plot
+    path: opencv-plot
+    inputs:
+      image: zed-camera/image_bgr

dora-node.yml

@@ -0,0 +1,39 @@
+# Node manifest: the typed contract a consumer wires against.
+#
+# No source package is shared between node repositories, so this file and its
+# type URNs are the only contract a consumer has. `dora validate` checks a
+# dataflow against it without running the node, so CI needs no camera and no
+# ZED SDK.
+apiVersion: 1
+name: dora-zed
+namespace: liria
+description: ZED stereo camera source publishing a BGR image, a depth map, intrinsics and an optional point cloud
+categories: [sensor]
+keywords: [zed, stereolabs, camera, depth, stereo, jetson]
+
+runtime: cpp
+entrypoint: bin/dora_zed
+build: scripts/build.sh
+
+inputs:
+  tick:
+    type: std/core/v1/UInt64
+    required: true
+    description: Each tick triggers one camera grab. Connect to dora/timer/millis/N.
+
+outputs:
+  image_bgr:
+    type: std/core/v1/UInt8
+    description: BGR image bytes, [H, W, 3]. Shape and encoding are in the metadata.
+  depth:
+    type: std/core/v1/Float32
+    description: Per-pixel depth in millimetres, [H, W]. Missing depth is 0, never inf or nan.
+  camera_info:
+    type: std/core/v1/Float32
+    description: Intrinsics [fx, fy, cx, cy], distortion in the metadata. Published once per camera session.
+  point_cloud:
+    type: std/core/v1/Float32
+    description: XYZRGBA point cloud. Published only when ZED_POINT_CLOUD is on.
+  camera_state:
+    type: std/core/v1/String
+    description: One of opening, streaming, recovering. Published on every change.

scripts/build.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Configure, compile and install the node into bin/.
+#
+# dora 1.0 runs a node's `build:` command directly rather than through a shell,
+# so `cd`, `&&` and command substitution are not available there. The whole
+# sequence lives here instead, and the dataflow only invokes this script.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_DIR="$(dirname "$SCRIPT_DIR")"
+BUILD_DIR="${REPO_DIR}/build"
+
+cmake -S "$REPO_DIR" -B "$BUILD_DIR" -DCMAKE_BUILD_TYPE=Release
+cmake --build "$BUILD_DIR" --target install --parallel "$(nproc)"

scripts/export_yolo.py

@@ -1,111 +0,0 @@
-# /// script
-# requires-python = ">=3.10"
-# dependencies = ["ultralytics>=8.3"]
-# ///
-"""Export a standard YOLOv11 model to ONNX for closed-vocabulary detection.
-
-The exported model has one input:
-  images  (1, 3, imgsz, imgsz)  float32
-
-Class names are fixed at training time. Provide them at runtime via
-ZED_ONNX_LABELS or ZED_ONNX_LABELS_FILE.
-
-Usage:
-    uv run scripts/export_yolo.py
-    uv run scripts/export_yolo.py --model s --output-dir models/
-    uv run scripts/export_yolo.py --model m --imgsz 1280
-"""
-
-import argparse
-import shutil
-import sys
-from pathlib import Path
-
-MODELS = {
-    "n": "yolo11n",
-    "s": "yolo11s",
-    "m": "yolo11m",
-    "l": "yolo11l",
-    "x": "yolo11x",
-}
-
-
-def parse_args():
-    p = argparse.ArgumentParser(
-        description="Export YOLOv11 to ONNX for closed-vocabulary detection"
-    )
-    p.add_argument(
-        "--model", "-m",
-        choices=list(MODELS),
-        default="n",
-        help="Model size: n (~6 MB), s (~18 MB), m (~38 MB), l (~49 MB), x (~109 MB). Default: n",
-    )
-    p.add_argument(
-        "--imgsz",
-        type=int,
-        default=640,
-        help="Input image size in pixels (square). Default: 640",
-    )
-    p.add_argument(
-        "--output-dir", "-o",
-        default=".",
-        help="Output directory for the .onnx file. Default: current directory",
-    )
-    return p.parse_args()
-
-
-def verify_output(onnx_path: Path):
-    try:
-        import onnx
-        model = onnx.load(str(onnx_path))
-        inputs  = [(i.name, [d.dim_value for d in i.type.tensor_type.shape.dim])
-                   for i in model.graph.input]
-        outputs = [(o.name, [d.dim_value for d in o.type.tensor_type.shape.dim])
-                   for o in model.graph.output]
-        print(f"  Inputs:  {inputs}")
-        print(f"  Outputs: {outputs}")
-    except ImportError:
-        print("  (install 'onnx' to verify model structure)")
-
-
-def main():
-    args = parse_args()
-
-    try:
-        from ultralytics import YOLO
-    except ImportError:
-        print("Error: ultralytics not installed. Run: uv run scripts/export_yolo.py", file=sys.stderr)
-        sys.exit(1)
-
-    model_name = MODELS[args.model]
-    out_dir = Path(args.output_dir)
-    out_dir.mkdir(parents=True, exist_ok=True)
-    out_name = f"{model_name}_i{args.imgsz}.onnx"
-    dst = out_dir / out_name
-
-    print(f"Model:      {model_name}.pt")
-    print(f"Input size: {args.imgsz}")
-    print(f"Output:     {dst}")
-    print()
-
-    model = YOLO(f"{model_name}.pt")
-    exported = Path(model.export(format="onnx", imgsz=args.imgsz))
-
-    if exported.resolve() != dst.resolve():
-        shutil.move(str(exported), str(dst))
-
-    print(f"\nVerifying exported model...")
-    verify_output(dst)
-
-    print(f"\nExported: {dst.resolve()}")
-    print(f"\nRun with (COCO labels example):")
-    print(f"  ZED_ENGINE=onnx \\")
-    print(f"  ZED_ONNX_MODEL_PATH={dst.resolve()} \\")
-    print(f"  ZED_ONNX_LABELS_FILE=models/coco80.txt \\")
-    print(f"  dora start dataflow.yml")
-    print(f"\nOr with inline labels:")
-    print(f"  ZED_ONNX_LABELS=\"person,car,chair\" \\")
-
-
-if __name__ == "__main__":
-    main()

scripts/install_linux_arm.sh

@@ -9,11 +9,6 @@
 #   JetPack 6.2.2   (L4T 36.5)
 #   JetPack 7.0     (L4T 38.2) — Jetson Thor
 #   JetPack 7.1     (L4T 38.4)
-#
-# ONNX Runtime: Microsoft does not publish official GPU binaries for ARM64.
-#   ONNX Runtime is still needed to configure the build; only the yolo_onnx
-#   engine uses it at runtime. Build it from source:
-#   https://onnxruntime.ai/docs/build/eps.html#tensorrt
 set -euo pipefail
 
 ZED_VERSION="5.2"
@@ -63,13 +58,5 @@ else
     sudo "${TMP_DIR}/ZED_SDK.run" -- silent skip_python skip_tools
 fi
 
-# --- ONNX Runtime ---
-echo ""
-echo ">>> ONNX Runtime (ARM64 GPU): no official pre-built binary available."
-echo "    Needed to configure the build; only yolo_onnx uses it at runtime."
-echo "    Build from source:"
-echo "      https://onnxruntime.ai/docs/build/eps.html#tensorrt"
-echo ""
-
-echo ">>> Done. You can now run: mkdir -p build && cd build && cmake .. && make install"
+echo ">>> Done. You can now run: scripts/build.sh"
 echo ">>> Downloaded files are in ${TMP_DIR}"

scripts/install_linux_x64.sh

@@ -6,7 +6,6 @@
 set -euo pipefail
 
 ZED_VERSION="5.2"
-ORT_VERSION="1.24.4"
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 TMP_DIR="${SCRIPT_DIR}/../tmp"
@@ -34,32 +33,5 @@ else
     sudo "${TMP_DIR}/ZED_SDK.run" -- silent skip_python skip_tools
 fi
 
-# --- ONNX Runtime (C++ GPU, installs to /usr/local) ---
-if [ -f /usr/local/lib64/libonnxruntime.so ] && [ -d /usr/local/lib/cmake/onnxruntime ]; then
-    echo ">>> ONNX Runtime already installed — skipping."
-else
-    echo ">>> Downloading ONNX Runtime ${ORT_VERSION} GPU..."
-    download \
-        "https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VERSION}/onnxruntime-linux-x64-gpu-${ORT_VERSION}.tgz" \
-        "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}.tgz"
-    echo ">>> Extracting..."
-    tar -xzf "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}.tgz" -C "${TMP_DIR}"
-    echo ">>> Installing ONNX Runtime to /usr/local..."
-    sudo mkdir -p /usr/local/include/onnxruntime
-    sudo cp -r "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}/include/"* /usr/local/include/onnxruntime/
-    sudo mkdir -p /usr/local/lib64
-    sudo cp -r "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}/lib/"* /usr/local/lib64/
-    # onnxruntimeTargets.cmake resolves the library as <prefix>/lib64, but Ubuntu
-    # is multiarch: CMake does not search lib64 for packages and ld.so does not
-    # search /usr/local/lib64 at all. Move only the package files to lib/cmake
-    # and put lib64 on the linker path.
-    sudo mkdir -p /usr/local/lib/cmake
-    sudo rm -rf /usr/local/lib/cmake/onnxruntime
-    sudo mv /usr/local/lib64/cmake/onnxruntime /usr/local/lib/cmake/
-    sudo rmdir /usr/local/lib64/cmake
-    echo /usr/local/lib64 | sudo tee /etc/ld.so.conf.d/onnxruntime.conf > /dev/null
-    sudo ldconfig
-fi
-
-echo ">>> Done. You can now run: mkdir -p build && cd build && cmake .. && make install"
+echo ">>> Done. You can now run: scripts/build.sh"
 echo ">>> Downloaded files are in ${TMP_DIR}"

scripts/install_windows.ps1

@@ -10,10 +10,6 @@
 $ErrorActionPreference = "Stop"
 
 $ZED_INSTALL_DIR = "C:\Program Files (x86)\ZED SDK"
-$ORT_VERSION     = "1.24.4"
-$ORT_NAME        = "onnxruntime-win-x64-gpu_cuda13-$ORT_VERSION"
-$ORT_URL         = "https://github.com/microsoft/onnxruntime/releases/download/v$ORT_VERSION/$ORT_NAME.zip"
-$ORT_DEST        = "C:\onnxruntime"
 
 # --- ZED SDK ---
 if (Test-Path $ZED_INSTALL_DIR) {
@@ -25,23 +21,5 @@ if (Test-Path $ZED_INSTALL_DIR) {
     Write-Host ""
 }
 
-# --- ONNX Runtime ---
-if (Test-Path "$ORT_DEST\lib\onnxruntime.dll") {
-    Write-Host ">>> ONNX Runtime already installed at '$ORT_DEST' — skipping."
-} else {
-    Write-Host ">>> Downloading ONNX Runtime $ORT_VERSION GPU (CUDA 13)..."
-    $tmp = "$env:TEMP\$ORT_NAME.zip"
-    Invoke-WebRequest -Uri $ORT_URL -OutFile $tmp
-
-    Write-Host ">>> Extracting to $ORT_DEST..."
-    Expand-Archive -Path $tmp -DestinationPath $env:TEMP -Force
-    if (Test-Path $ORT_DEST) { Remove-Item $ORT_DEST -Recurse -Force }
-    Move-Item "$env:TEMP\$ORT_NAME" $ORT_DEST
-    Remove-Item $tmp
-
-    Write-Host ">>> ONNX Runtime installed to $ORT_DEST"
-    Write-Host "    Pass it to CMake: cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime"
-}
-
 Write-Host ""
 Write-Host ">>> Done."

src/engines/engine.h

@@ -1,27 +0,0 @@
-#pragma once
-
-#include <sl/Camera.hpp>
-#include <string>
-#include <vector>
-
-namespace engines {
-
-struct DetectedObject {
-    int label_id;
-    float confidence;
-    float position[3];         // center x, y, z (mm)
-    float bounding_box_2d[8];  // 4 corners × 2 coords (pixels)
-    float bounding_box_3d[24]; // 8 corners × 3 coords (mm)
-};
-
-class Engine {
-public:
-    virtual ~Engine() = default;
-
-    virtual void enable(sl::Camera& camera) = 0;
-    virtual void detect(sl::Camera& camera) = 0;
-    virtual const std::vector<DetectedObject>& objects() const = 0;
-    virtual std::string label_names() const = 0;
-};
-
-} // namespace engines

src/engines/yolo_helpers.h

@@ -1,136 +0,0 @@
-#pragma once
-
-#include <opencv2/opencv.hpp>
-#include <sl/Camera.hpp>
-
-#include <algorithm>
-#include <cmath>
-#include <vector>
-
-namespace engines::detail {
-
-// --- NMS ---
-
-struct RawDetection {
-    float x1, y1, x2, y2;
-    float score;
-    int class_id;
-};
-
-inline float iou(const RawDetection& a, const RawDetection& b) {
-    float inter_x1 = std::max(a.x1, b.x1);
-    float inter_y1 = std::max(a.y1, b.y1);
-    float inter_x2 = std::min(a.x2, b.x2);
-    float inter_y2 = std::min(a.y2, b.y2);
-    float inter_w = std::max(0.0f, inter_x2 - inter_x1);
-    float inter_h = std::max(0.0f, inter_y2 - inter_y1);
-    float inter_area = inter_w * inter_h;
-    float area_a = (a.x2 - a.x1) * (a.y2 - a.y1);
-    float area_b = (b.x2 - b.x1) * (b.y2 - b.y1);
-    float union_area = area_a + area_b - inter_area;
-    return union_area > 0.0f ? inter_area / union_area : 0.0f;
-}
-
-inline std::vector<RawDetection> nms(std::vector<RawDetection> dets, float threshold) {
-    std::sort(dets.begin(), dets.end(),
-              [](const RawDetection& a, const RawDetection& b) {
-                  return a.score > b.score;
-              });
-    std::vector<bool> suppressed(dets.size(), false);
-    std::vector<RawDetection> result;
-    for (size_t i = 0; i < dets.size(); i++) {
-        if (suppressed[i]) continue;
-        result.push_back(dets[i]);
-        for (size_t j = i + 1; j < dets.size(); j++) {
-            if (!suppressed[j] && iou(dets[i], dets[j]) > threshold)
-                suppressed[j] = true;
-        }
-    }
-    return result;
-}
-
-// --- Letterbox resize ---
-
-inline cv::Mat letterbox(const cv::Mat& src, int size,
-                         float& scale, float& pad_x, float& pad_y) {
-    float scale_w = static_cast<float>(size) / src.cols;
-    float scale_h = static_cast<float>(size) / src.rows;
-    scale = std::min(scale_w, scale_h);
-    int new_w = static_cast<int>(std::round(src.cols * scale));
-    int new_h = static_cast<int>(std::round(src.rows * scale));
-    pad_x = (size - new_w) / 2.0f;
-    pad_y = (size - new_h) / 2.0f;
-
-    cv::Mat resized;
-    cv::resize(src, resized, {new_w, new_h});
-
-    cv::Mat canvas(size, size, CV_8UC3, cv::Scalar(114, 114, 114));
-    resized.copyTo(canvas(cv::Rect(
-        static_cast<int>(pad_x), static_cast<int>(pad_y), new_w, new_h)));
-    return canvas;
-}
-
-// --- Point cloud fusion ---
-
-struct FusionResult {
-    float cx, cy, cz;
-    float depth_near, depth_far;
-    bool valid = false;
-};
-
-inline FusionResult fuse_bbox(const sl::Mat& pc, int x1, int y1, int x2, int y2) {
-    FusionResult result{};
-    int w = static_cast<int>(pc.getWidth());
-    int h = static_cast<int>(pc.getHeight());
-    x1 = std::max(x1, 0);
-    y1 = std::max(y1, 0);
-    x2 = std::min(x2, w - 1);
-    y2 = std::min(y2, h - 1);
-
-    std::vector<float> xs, ys, zs;
-    xs.reserve((x2 - x1) * (y2 - y1));
-    ys.reserve(xs.capacity());
-    zs.reserve(xs.capacity());
-
-    for (int y = y1; y <= y2; y++) {
-        for (int x = x1; x <= x2; x++) {
-            sl::float4 pt;
-            pc.getValue(x, y, &pt);
-            if (!std::isnan(pt.x) && !std::isnan(pt.y) && !std::isnan(pt.z) &&
-                !std::isinf(pt.x) && !std::isinf(pt.y) && !std::isinf(pt.z)) {
-                xs.push_back(pt.x);
-                ys.push_back(pt.y);
-                zs.push_back(pt.z);
-            }
-        }
-    }
-
-    if (xs.empty()) return result;
-
-    std::sort(xs.begin(), xs.end());
-    std::sort(ys.begin(), ys.end());
-    std::sort(zs.begin(), zs.end());
-
-    size_t n   = xs.size();
-    size_t mid = n / 2;
-    size_t p25 = n / 4;
-    size_t p75 = 3 * n / 4;
-
-    result.cx         = xs[mid];
-    result.cy         = ys[mid];
-    result.cz         = zs[mid];
-    result.depth_near = zs[p25];
-    result.depth_far  = zs[p75];
-    result.valid      = true;
-    return result;
-}
-
-inline void backproject(float u, float v, float z,
-                        float fx, float fy, float cx, float cy,
-                        float& X, float& Y, float& Z) {
-    Z = z;
-    X = (u - cx) * z / fx;
-    Y = (v - cy) * z / fy;
-}
-
-} // namespace engines::detail

src/engines/yolo_onnx.cc

@@ -1,369 +0,0 @@
-#include "engines/yolo_onnx.h"
-#include "engines/yolo_helpers.h"
-#include "util/env.h"
-
-#include <onnxruntime_cxx_api.h>
-#include <opencv2/opencv.hpp>
-#include <sl/Camera.hpp>
-
-#include <cmath>
-#include <fstream>
-#include <iostream>
-#include <string>
-#include <vector>
-
-namespace engines {
-
-using namespace detail;
-
-// --- Impl ---
-
-struct YoloOnnx::Impl {
-    Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "yolo_onnx"};
-    Ort::SessionOptions session_options;
-    std::unique_ptr<Ort::Session> session;
-    Ort::AllocatorWithDefaultOptions allocator;
-
-    std::string input_name;
-    std::string output_name;
-    int num_classes = 0;
-    YoloOutputLayout resolved_layout = YoloOutputLayout::NC_FIRST;
-
-    std::vector<RawDetection> run_inference(const YoloOnnxConfig& config,
-                                            const cv::Mat& image_bgr);
-};
-
-// --- Impl::run_inference ---
-
-std::vector<RawDetection> YoloOnnx::Impl::run_inference(
-    const YoloOnnxConfig& config, const cv::Mat& image_bgr)
-{
-    if (!session) return {};
-
-    int orig_w = image_bgr.cols;
-    int orig_h = image_bgr.rows;
-
-    cv::Mat rgb;
-    cv::cvtColor(image_bgr, rgb, cv::COLOR_BGR2RGB);
-
-    float scale, pad_x, pad_y;
-    cv::Mat lb = letterbox(rgb, config.input_size, scale, pad_x, pad_y);
-
-    cv::Mat lb_float;
-    lb.convertTo(lb_float, CV_32F, 1.0f / 255.0f);
-
-    int sz = config.input_size;
-    std::vector<float> input_data(3 * sz * sz);
-    for (int c = 0; c < 3; c++) {
-        for (int h = 0; h < sz; h++) {
-            for (int w = 0; w < sz; w++) {
-                input_data[c * sz * sz + h * sz + w] =
-                    lb_float.at<cv::Vec3f>(h, w)[c];
-            }
-        }
-    }
-
-    auto mem_info = Ort::MemoryInfo::CreateCpu(
-        OrtAllocatorType::OrtArenaAllocator, OrtMemTypeDefault);
-
-    std::vector<int64_t> image_shape{1, 3, sz, sz};
-    Ort::Value image_tensor = Ort::Value::CreateTensor<float>(
-        mem_info, input_data.data(), input_data.size(),
-        image_shape.data(), image_shape.size());
-
-    const char* input_name_cstr  = input_name.c_str();
-    const char* output_name_cstr = output_name.c_str();
-
-    std::vector<Ort::Value> outputs;
-    try {
-        outputs = session->Run(
-            Ort::RunOptions{nullptr},
-            &input_name_cstr, &image_tensor, 1,
-            &output_name_cstr, 1);
-    } catch (const Ort::Exception& e) {
-        std::cerr << "YoloOnnx: inference failed: " << e.what() << std::endl;
-        return {};
-    }
-
-    auto out_shape = outputs[0].GetTensorTypeAndShapeInfo().GetShape();
-    if (out_shape.size() < 3) return {};
-
-    const float* out = outputs[0].GetTensorData<float>();
-    int nc = num_classes;
-    if (nc <= 0) return {};
-
-    std::vector<RawDetection> candidates;
-
-    if (resolved_layout == YoloOutputLayout::NC_FIRST) {
-        // (1, 4+N, anchors) — YOLOv8, YOLOv11
-        int num_anchors = static_cast<int>(out_shape[2]);
-        for (int a = 0; a < num_anchors; a++) {
-            float cx_lb = out[0 * num_anchors + a];
-            float cy_lb = out[1 * num_anchors + a];
-            float w_lb  = out[2 * num_anchors + a];
-            float h_lb  = out[3 * num_anchors + a];
-
-            float best_score = 0.0f;
-            int   best_class = 0;
-            for (int c = 0; c < nc; c++) {
-                float s = out[(4 + c) * num_anchors + a];
-                if (s > best_score) { best_score = s; best_class = c; }
-            }
-            if (best_score < config.confidence_threshold) continue;
-
-            float x1_lb = cx_lb - w_lb / 2.0f;
-            float y1_lb = cy_lb - h_lb / 2.0f;
-            float x2_lb = cx_lb + w_lb / 2.0f;
-            float y2_lb = cy_lb + h_lb / 2.0f;
-
-            float x1 = std::max(0.0f, std::min((x1_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
-            float y1 = std::max(0.0f, std::min((y1_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
-            float x2 = std::max(0.0f, std::min((x2_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
-            float y2 = std::max(0.0f, std::min((y2_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
-            candidates.push_back({x1, y1, x2, y2, best_score, best_class});
-        }
-    } else {
-        // (1, anchors, 4+N) or (1, anchors, 5+N)
-        int num_anchors = static_cast<int>(out_shape[1]);
-        int row_stride  = static_cast<int>(out_shape[2]);
-        int cls_offset  = (resolved_layout == YoloOutputLayout::NC_LAST_OBJ) ? 5 : 4;
-
-        for (int a = 0; a < num_anchors; a++) {
-            const float* row = out + a * row_stride;
-            float cx_lb = row[0], cy_lb = row[1], w_lb = row[2], h_lb = row[3];
-            float obj_conf = (resolved_layout == YoloOutputLayout::NC_LAST_OBJ) ? row[4] : 1.0f;
-
-            float best_score = 0.0f;
-            int   best_class = 0;
-            for (int c = 0; c < nc; c++) {
-                float s = obj_conf * row[cls_offset + c];
-                if (s > best_score) { best_score = s; best_class = c; }
-            }
-            if (best_score < config.confidence_threshold) continue;
-
-            float x1_lb = cx_lb - w_lb / 2.0f;
-            float y1_lb = cy_lb - h_lb / 2.0f;
-            float x2_lb = cx_lb + w_lb / 2.0f;
-            float y2_lb = cy_lb + h_lb / 2.0f;
-
-            float x1 = std::max(0.0f, std::min((x1_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
-            float y1 = std::max(0.0f, std::min((y1_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
-            float x2 = std::max(0.0f, std::min((x2_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
-            float y2 = std::max(0.0f, std::min((y2_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
-            candidates.push_back({x1, y1, x2, y2, best_score, best_class});
-        }
-    }
-
-    return nms(std::move(candidates), config.nms_threshold);
-}
-
-// --- Config ---
-
-YoloOnnxConfig YoloOnnxConfig::from_env() {
-    YoloOnnxConfig c;
-    c.model_path           = util::get_env_str("ZED_ONNX_MODEL_PATH",      c.model_path);
-    c.confidence_threshold = util::get_env_float("ZED_ONNX_CONFIDENCE",    c.confidence_threshold);
-    c.nms_threshold        = util::get_env_float("ZED_ONNX_NMS_THRESHOLD", c.nms_threshold);
-    c.input_size           = util::get_env_int("ZED_ONNX_INPUT_SIZE",      c.input_size);
-    c.labels               = util::get_env_str("ZED_ONNX_LABELS",          c.labels);
-    c.labels_file          = util::get_env_str("ZED_ONNX_LABELS_FILE",     c.labels_file);
-
-    std::string fmt = util::get_env_str("ZED_ONNX_OUTPUT_FORMAT", "");
-    if      (fmt == "nc_first"    || fmt == "yolov8" || fmt == "yolov11")
-        c.output_layout = YoloOutputLayout::NC_FIRST;
-    else if (fmt == "nc_last")
-        c.output_layout = YoloOutputLayout::NC_LAST;
-    else if (fmt == "nc_last_obj" || fmt == "yolov5")
-        c.output_layout = YoloOutputLayout::NC_LAST_OBJ;
-
-    return c;
-}
-
-// --- YoloOnnx ---
-
-YoloOnnx::YoloOnnx(const YoloOnnxConfig& config)
-    : config_(config), impl_(std::make_unique<Impl>()) {}
-
-YoloOnnx::~YoloOnnx() = default;
-
-void YoloOnnx::enable(sl::Camera& /* camera */) {
-    if (config_.model_path.empty()) {
-        std::cerr << "YoloOnnx: ZED_ONNX_MODEL_PATH is not set." << std::endl;
-        return;
-    }
-
-    OrtCUDAProviderOptions cuda_opts{};
-    cuda_opts.device_id = 0;
-    try {
-        impl_->session_options.AppendExecutionProvider_CUDA(cuda_opts);
-    } catch (const Ort::Exception& e) {
-        std::cerr << "YoloOnnx: CUDA unavailable (" << e.what() << "), using CPU." << std::endl;
-    }
-    impl_->session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);
-
-    std::cout << "YoloOnnx: loading model (" << config_.model_path << ")..." << std::endl;
-    try {
-        impl_->session = std::make_unique<Ort::Session>(
-            impl_->env, config_.model_path.c_str(), impl_->session_options);
-    } catch (const Ort::Exception& e) {
-        std::cerr << "YoloOnnx: failed to load model: " << e.what() << std::endl;
-        return;
-    }
-
-    impl_->input_name  = impl_->session->GetInputNameAllocated(0, impl_->allocator).get();
-    impl_->output_name = impl_->session->GetOutputNameAllocated(0, impl_->allocator).get();
-
-    size_t num_inputs = impl_->session->GetInputCount();
-    if (num_inputs >= 2) {
-        std::cerr << "YoloOnnx: model has " << num_inputs << " inputs — "
-                     "this engine only supports single-input models."
-                  << std::endl;
-    }
-
-    auto out_info = impl_->session->GetOutputTypeInfo(0).GetTensorTypeAndShapeInfo();
-    auto shape    = out_info.GetShape();
-    if (shape.size() == 3) {
-        int64_t dim1 = shape[1];
-        int64_t dim2 = shape[2];
-
-        YoloOutputLayout resolved;
-        if (config_.output_layout == YoloOutputLayout::AUTO) {
-            // dim1 < dim2 → NC_FIRST (e.g. 84 vs 8400 for YOLOv8/11)
-            // dim1 > dim2 → NC_LAST_OBJ (e.g. 25200 vs 85 for YOLOv5)
-            resolved = (dim1 < dim2) ? YoloOutputLayout::NC_FIRST
-                                     : YoloOutputLayout::NC_LAST_OBJ;
-        } else {
-            resolved = config_.output_layout;
-        }
-        impl_->resolved_layout = resolved;
-
-        if (resolved == YoloOutputLayout::NC_FIRST) {
-            impl_->num_classes = static_cast<int>(dim1) - 4;
-        } else if (resolved == YoloOutputLayout::NC_LAST) {
-            impl_->num_classes = static_cast<int>(dim2) - 4;
-        } else { // NC_LAST_OBJ
-            impl_->num_classes = static_cast<int>(dim2) - 5;
-        }
-    }
-
-    if (!config_.labels_file.empty() && config_.labels.empty()) {
-        std::ifstream lf(config_.labels_file);
-        if (!lf) {
-            std::cerr << "YoloOnnx: cannot open ZED_ONNX_LABELS_FILE: "
-                      << config_.labels_file << std::endl;
-        } else {
-            std::string line, joined;
-            while (std::getline(lf, line)) {
-                size_t l = line.find_first_not_of(" \t\r");
-                size_t r = line.find_last_not_of(" \t\r");
-                if (l == std::string::npos) continue;
-                if (!joined.empty()) joined += ',';
-                joined += line.substr(l, r - l + 1);
-            }
-            if (!joined.empty()) {
-                config_.labels = joined;
-                std::cout << "YoloOnnx: loaded labels from file ("
-                          << config_.labels_file << ")" << std::endl;
-            }
-        }
-    }
-
-    std::cout << "YoloOnnx: model loaded."
-              << " Classes=" << impl_->num_classes
-              << " Layout=" << (impl_->resolved_layout == YoloOutputLayout::NC_FIRST ? "NC_FIRST"
-                              : impl_->resolved_layout == YoloOutputLayout::NC_LAST  ? "NC_LAST"
-                                                                                      : "NC_LAST_OBJ")
-              << std::endl;
-}
-
-void YoloOnnx::detect(sl::Camera& camera) {
-    detected_.clear();
-    if (!impl_->session) return;
-
-    sl::Mat img_mat;
-    camera.retrieveImage(img_mat, sl::VIEW::LEFT);
-
-    sl::Mat pc_mat;
-    camera.retrieveMeasure(pc_mat, sl::MEASURE::XYZRGBA);
-
-    sl::CameraInformation info = camera.getCameraInformation();
-    auto calib = info.camera_configuration.calibration_parameters.left_cam;
-    float fx = calib.fx, fy = calib.fy, cx = calib.cx, cy = calib.cy;
-
-    int orig_w = static_cast<int>(img_mat.getWidth());
-    int orig_h = static_cast<int>(img_mat.getHeight());
-
-    cv::Mat rgba(orig_h, orig_w, CV_8UC4, img_mat.getPtr<sl::uchar1>());
-    cv::Mat bgr;
-    cv::cvtColor(rgba, bgr, cv::COLOR_RGBA2BGR);
-
-    auto kept = impl_->run_inference(config_, bgr);
-
-    for (const auto& det : kept) {
-        FusionResult fusion = fuse_bbox(pc_mat,
-            static_cast<int>(det.x1), static_cast<int>(det.y1),
-            static_cast<int>(det.x2), static_cast<int>(det.y2));
-
-        DetectedObject obj{};
-        obj.label_id   = det.class_id;
-        obj.confidence = det.score * 100.0f;
-
-        if (fusion.valid) {
-            obj.position[0] = fusion.cx;
-            obj.position[1] = fusion.cy;
-            obj.position[2] = fusion.cz;
-        }
-
-        obj.bounding_box_2d[0] = det.x1; obj.bounding_box_2d[1] = det.y1;
-        obj.bounding_box_2d[2] = det.x2; obj.bounding_box_2d[3] = det.y1;
-        obj.bounding_box_2d[4] = det.x2; obj.bounding_box_2d[5] = det.y2;
-        obj.bounding_box_2d[6] = det.x1; obj.bounding_box_2d[7] = det.y2;
-
-        if (fusion.valid) {
-            float corners_u[4] = {det.x1, det.x2, det.x2, det.x1};
-            float corners_v[4] = {det.y1, det.y1, det.y2, det.y2};
-            float depths[2]    = {fusion.depth_near, fusion.depth_far};
-            for (int face = 0; face < 2; face++) {
-                for (int c = 0; c < 4; c++) {
-                    float X, Y, Z;
-                    backproject(corners_u[c], corners_v[c], depths[face],
-                                fx, fy, cx, cy, X, Y, Z);
-                    int idx = (face * 4 + c) * 3;
-                    obj.bounding_box_3d[idx + 0] = X;
-                    obj.bounding_box_3d[idx + 1] = Y;
-                    obj.bounding_box_3d[idx + 2] = Z;
-                }
-            }
-        }
-
-        detected_.push_back(obj);
-    }
-}
-
-void YoloOnnx::detect(const cv::Mat& image_bgr) {
-    detected_.clear();
-    if (!impl_->session) return;
-
-    auto kept = impl_->run_inference(config_, image_bgr);
-
-    for (const auto& det : kept) {
-        DetectedObject obj{};
-        obj.label_id   = det.class_id;
-        obj.confidence = det.score * 100.0f;
-        obj.bounding_box_2d[0] = det.x1; obj.bounding_box_2d[1] = det.y1;
-        obj.bounding_box_2d[2] = det.x2; obj.bounding_box_2d[3] = det.y1;
-        obj.bounding_box_2d[4] = det.x2; obj.bounding_box_2d[5] = det.y2;
-        obj.bounding_box_2d[6] = det.x1; obj.bounding_box_2d[7] = det.y2;
-        detected_.push_back(obj);
-    }
-}
-
-const std::vector<DetectedObject>& YoloOnnx::objects() const {
-    return detected_;
-}
-
-std::string YoloOnnx::label_names() const {
-    return config_.labels;
-}
-
-} // namespace engines

src/engines/yolo_onnx.h

@@ -1,48 +0,0 @@
-#pragma once
-
-#include "engines/engine.h"
-
-#include <opencv2/core.hpp>
-#include <memory>
-#include <string>
-
-namespace engines {
-
-enum class YoloOutputLayout {
-    AUTO,          // detect from output tensor shape at load time
-    NC_FIRST,      // (1, 4+N, anchors) — YOLOv8, YOLOv11
-    NC_LAST,       // (1, anchors, 4+N) — rows-first, no objectness
-    NC_LAST_OBJ    // (1, anchors, 5+N) — YOLOv5, rows-first with objectness
-};
-
-struct YoloOnnxConfig {
-    std::string model_path;              // ZED_ONNX_MODEL_PATH (required)
-    float confidence_threshold = 0.25f;  // ZED_ONNX_CONFIDENCE
-    float nms_threshold = 0.45f;         // ZED_ONNX_NMS_THRESHOLD
-    int input_size = 640;                // ZED_ONNX_INPUT_SIZE
-    std::string labels;                  // ZED_ONNX_LABELS (comma-separated)
-    std::string labels_file;             // ZED_ONNX_LABELS_FILE (one class name per line)
-    YoloOutputLayout output_layout = YoloOutputLayout::AUTO;
-
-    static YoloOnnxConfig from_env();
-};
-
-class YoloOnnx : public Engine {
-public:
-    explicit YoloOnnx(const YoloOnnxConfig& config);
-    ~YoloOnnx() override;
-
-    void enable(sl::Camera& camera) override;
-    void detect(sl::Camera& camera) override;
-    void detect(const cv::Mat& image_bgr);      // image-only; no 3D fusion
-    const std::vector<DetectedObject>& objects() const override;
-    std::string label_names() const override;
-
-private:
-    YoloOnnxConfig config_;
-    struct Impl;
-    std::unique_ptr<Impl> impl_;
-    std::vector<DetectedObject> detected_;
-};
-
-} // namespace engines

src/engines/zed_native.cc

@@ -1,99 +0,0 @@
-#include "engines/zed_native.h"
-#include "util/env.h"
-
-#include <iostream>
-#include <string>
-
-namespace {
-
-sl::OBJECT_DETECTION_MODEL parse_detection_model(const std::string& model) {
-    if (model == "MULTI_CLASS_BOX_FAST") {
-        return sl::OBJECT_DETECTION_MODEL::MULTI_CLASS_BOX_FAST;
-    }
-    if (model == "MULTI_CLASS_BOX_ACCURATE") {
-        return sl::OBJECT_DETECTION_MODEL::MULTI_CLASS_BOX_ACCURATE;
-    }
-    return sl::OBJECT_DETECTION_MODEL::MULTI_CLASS_BOX_MEDIUM;
-}
-
-} // namespace
-
-namespace engines {
-
-DetectionConfig DetectionConfig::from_env() {
-    DetectionConfig c;
-    c.model = util::get_env_str("ZED_DETECTION_MODEL", c.model);
-    c.confidence_threshold = util::get_env_float("ZED_DETECTION_CONFIDENCE", c.confidence_threshold);
-    c.tracking = util::get_env_bool("ZED_DETECTION_TRACKING", c.tracking);
-    return c;
-}
-
-ZedNative::ZedNative(const DetectionConfig& config) : config_(config) {
-    det_params_.detection_model = parse_detection_model(config_.model);
-    det_params_.enable_tracking = config_.tracking;
-    det_params_.enable_segmentation = false;
-
-    det_rt_params_.detection_confidence_threshold = config_.confidence_threshold;
-}
-
-void ZedNative::enable(sl::Camera& camera) {
-    if (config_.tracking) {
-        sl::PositionalTrackingParameters tracking_params;
-        sl::ERROR_CODE err = camera.enablePositionalTracking(tracking_params);
-        if (err != sl::ERROR_CODE::SUCCESS) {
-            std::cerr << "Failed to enable positional tracking: "
-                      << sl::toString(err) << std::endl;
-        }
-    }
-
-    std::cout << "Object Detection: loading model (" << config_.model << ")..." << std::endl;
-    sl::ERROR_CODE err = camera.enableObjectDetection(det_params_);
-    if (err != sl::ERROR_CODE::SUCCESS) {
-        std::cerr << "Failed to enable object detection: "
-                  << sl::toString(err) << std::endl;
-        return;
-    }
-    std::cout << "Object Detection: model loaded." << std::endl;
-}
-
-void ZedNative::detect(sl::Camera& camera) {
-    detected_.clear();
-
-    sl::ERROR_CODE err = camera.retrieveObjects(raw_objects_, det_rt_params_);
-    if (err != sl::ERROR_CODE::SUCCESS) {
-        return;
-    }
-
-    for (const auto& obj : raw_objects_.object_list) {
-        DetectedObject det{};
-        det.label_id = static_cast<int>(obj.label);
-        det.confidence = obj.confidence;
-
-        det.position[0] = obj.position.x;
-        det.position[1] = obj.position.y;
-        det.position[2] = obj.position.z;
-
-        for (int i = 0; i < 4 && i < static_cast<int>(obj.bounding_box_2d.size()); i++) {
-            det.bounding_box_2d[i * 2 + 0] = obj.bounding_box_2d[i].x;
-            det.bounding_box_2d[i * 2 + 1] = obj.bounding_box_2d[i].y;
-        }
-
-        for (int i = 0; i < 8 && i < static_cast<int>(obj.bounding_box.size()); i++) {
-            det.bounding_box_3d[i * 3 + 0] = obj.bounding_box[i].x;
-            det.bounding_box_3d[i * 3 + 1] = obj.bounding_box[i].y;
-            det.bounding_box_3d[i * 3 + 2] = obj.bounding_box[i].z;
-        }
-
-        detected_.push_back(det);
-    }
-}
-
-const std::vector<DetectedObject>& ZedNative::objects() const {
-    return detected_;
-}
-
-std::string ZedNative::label_names() const {
-    return "PERSON,VEHICLE,BAG,ANIMAL,ELECTRONICS,FRUIT_VEGETABLE,SPORT";
-}
-
-} // namespace engines

src/engines/zed_native.h

@@ -1,34 +0,0 @@
-#pragma once
-
-#include "engines/engine.h"
-
-#include <string>
-
-namespace engines {
-
-struct DetectionConfig {
-    std::string model = "MULTI_CLASS_BOX_MEDIUM";
-    float confidence_threshold = 40.0f;
-    bool tracking = false;
-
-    static DetectionConfig from_env();
-};
-
-class ZedNative : public Engine {
-public:
-    explicit ZedNative(const DetectionConfig& config);
-
-    void enable(sl::Camera& camera) override;
-    void detect(sl::Camera& camera) override;
-    const std::vector<DetectedObject>& objects() const override;
-    std::string label_names() const override;
-
-private:
-    DetectionConfig config_;
-    sl::ObjectDetectionParameters det_params_;
-    sl::ObjectDetectionRuntimeParameters det_rt_params_;
-    sl::Objects raw_objects_;
-    std::vector<DetectedObject> detected_;
-};
-
-} // namespace engines

src/main.cc

@@ -1,32 +1,22 @@
-#include "engines/engine.h"
-#include "engines/yolo_onnx.h"
-#include "engines/zed_native.h"
-#include "util/env.h"
-#include "zed/camera.h"
+#include "zed/capture.h"
 
 #include <dora-node-api.h>
-#include <opencv2/opencv.hpp>
 
 #include <chrono>
-#include <cstring>
+#include <cstdlib>
 #include <initializer_list>
 #include <iostream>
-#include <memory>
 #include <string>
 #include <vector>
 
 namespace {
 
-constexpr int FIELDS_PER_OBJECT = 37;
-
-enum class PcMode { FULL, HALF, ROI };
-
-rust::Vec<int64_t> make_shape(int height, int width, int channels) {
-    rust::Vec<int64_t> shape;
-    shape.push_back(height);
-    shape.push_back(width);
-    shape.push_back(channels);
-    return shape;
+rust::Vec<int64_t> make_vector_int(std::initializer_list<int64_t> values) {
+    rust::Vec<int64_t> out;
+    for (int64_t v : values) {
+        out.push_back(v);
+    }
+    return out;
 }
 
 rust::Vec<double> make_distortion(const std::vector<double>& distortion) {
@@ -37,58 +27,55 @@ rust::Vec<double> make_distortion(const std::vector<double>& distortion) {
     return values;
 }
 
-rust::Vec<int64_t> make_vector_int(std::initializer_list<int64_t> values) {
-    rust::Vec<int64_t> out;
-    for (int64_t v : values) {
-        out.push_back(v);
-    }
-    return out;
+void set_intrinsics(Metadata& metadata, const zed::Calibration& calib) {
+    metadata.set_float("fx", calib.fx);
+    metadata.set_float("fy", calib.fy);
+    metadata.set_float("cx", calib.cx);
+    metadata.set_float("cy", calib.cy);
+    metadata.set_list_float("distortion", make_distortion(calib.distortion));
 }
 
-std::vector<float> serialize_objects(const std::vector<engines::DetectedObject>& objects) {
-    std::vector<float> buffer(objects.size() * FIELDS_PER_OBJECT);
-    for (size_t i = 0; i < objects.size(); i++) {
-        float* row = buffer.data() + i * FIELDS_PER_OBJECT;
-        const auto& obj = objects[i];
-        row[0] = static_cast<float>(obj.label_id);
-        row[1] = obj.confidence;
-        row[2] = obj.position[0];
-        row[3] = obj.position[1];
-        row[4] = obj.position[2];
-        std::memcpy(row + 5, obj.bounding_box_2d, sizeof(obj.bounding_box_2d));
-        std::memcpy(row + 13, obj.bounding_box_3d, sizeof(obj.bounding_box_3d));
+// Both clocks are needed. The device timestamp is the instant the frame was
+// captured and is the only value two nodes reading the same frame can agree on.
+// The host timestamp is what the rest of the dataflow and the session recorder
+// share, and on a Jetson it can step when NTP synchronises after boot - which is
+// exactly why the device timestamp is recorded alongside it rather than instead
+// of it.
+void set_frame_stamps(Metadata& metadata, const zed::Frame& frame) {
+    metadata.set_int("frame_id", frame.frame_id);
+    metadata.set_timestamp("device_timestamp_ns", frame.device_ns);
+    metadata.set_timestamp("host_timestamp_ns", frame.host_ns);
+}
+
+void report(const char* output, const DoraResult& result) {
+    if (!std::string(result.error).empty()) {
+        std::cerr << "Error sending " << output << ": " << std::string(result.error) << std::endl;
     }
-    return buffer;
+}
+
+int64_t host_timestamp_ns() {
+    return std::chrono::duration_cast<std::chrono::nanoseconds>(
+               std::chrono::system_clock::now().time_since_epoch())
+        .count();
 }
 
 } // namespace
 
 int main() {
+    // Read the environment before joining the dataflow, so a bad configuration
+    // fails with its own message rather than after the node has registered.
+    auto config = zed::Config::from_env();
+
     auto dora_node = init_dora_node();
 
-    auto config = zed::Config::from_env();
-    zed::Camera camera(config);
-    camera.open();
-    camera.warmup();
-    auto calib = camera.calibration();
+    // The camera runs on its own thread. The SDK can block for an unbounded time
+    // inside grab() and open() when the camera is disconnected, so nothing below
+    // may call into it: this loop only reads what the capture thread has already
+    // produced, which is what keeps Stop answerable while the camera is down.
+    zed::Capture capture(config);
+    capture.start();
 
-    std::string engine_name = util::get_env_str("ZED_ENGINE", "native");
-    std::unique_ptr<engines::Engine> detector;
-    if (engine_name == "yolo_onnx" || engine_name == "onnx") {
-        auto cfg = engines::YoloOnnxConfig::from_env();
-        detector = std::make_unique<engines::YoloOnnx>(cfg);
-    } else {
-        auto cfg = engines::DetectionConfig::from_env();
-        detector = std::make_unique<engines::ZedNative>(cfg);
-    }
-    detector->enable(camera.handle());
-
-    std::string pc_mode_str = util::get_env_str("ZED_PC_MODE", "FULL");
-    PcMode pc_mode = (pc_mode_str == "HALF") ? PcMode::HALF
-                   : (pc_mode_str == "ROI")  ? PcMode::ROI
-                                             : PcMode::FULL;
-
-    int64_t frame_id = 0;
+    zed::Frame frame;
     while (true) {
         auto event = dora_node.events->next();
         auto type = event_type(event);
@@ -101,180 +88,104 @@ int main() {
             continue;
         }
 
-        if (!camera.grab()) {
-            std::cerr << "Grab failed; attempting to reconnect..." << std::endl;
-            camera.close();
-            camera.open();
-            calib = camera.calibration();
-            detector->enable(camera.handle());
+        std::string state;
+        if (capture.take_state_change(state)) {
+            rust::Slice<const uint8_t> state_slice{
+                reinterpret_cast<const uint8_t*>(state.data()), state.size()};
+
+            auto state_metadata = new_metadata();
+            state_metadata->set_string("dtype", "string");
+            state_metadata->set_string("encoding", "utf8");
+            state_metadata->set_timestamp("host_timestamp_ns", host_timestamp_ns());
+
+            report("camera_state",
+                   send_output_with_metadata(dora_node.send_output, "camera_state", state_slice,
+                                             std::move(state_metadata)));
+        }
+
+        // No new frame since the last tick: the camera is slower than the timer,
+        // or it is down and the capture thread is recovering it.
+        if (!capture.take_frame(frame)) {
             continue;
         }
 
-        detector->detect(camera.handle());
-
-        sl::Mat& image_mat = camera.image();
-        sl::Mat& point_cloud_mat = camera.point_cloud();
-
-        int width = image_mat.getWidth();
-        int height = image_mat.getHeight();
-
-        int64_t current_frame_id = frame_id++;
-        int64_t timestamp_ns = static_cast<int64_t>(
-            std::chrono::duration_cast<std::chrono::nanoseconds>(
-                std::chrono::steady_clock::now().time_since_epoch()
-            ).count()
-        );
-
         // --- image_bgr ---
-        cv::Mat image_rgba(height, width, CV_8UC4, image_mat.getPtr<sl::uchar1>());
-        cv::Mat image_bgr;
-        cv::cvtColor(image_rgba, image_bgr, cv::COLOR_BGRA2BGR);
-
-        const auto image_size = static_cast<size_t>(image_bgr.total() * image_bgr.elemSize());
-        rust::Slice<const uint8_t> image_slice{
-            reinterpret_cast<const uint8_t*>(image_bgr.data), image_size};
+        rust::Slice<const uint8_t> image_slice{frame.image_bgr.data(), frame.image_bgr.size()};
 
         auto image_metadata = new_metadata();
         image_metadata->set_string("dtype", "uint8");
         image_metadata->set_string("encoding", "bgr8");
         image_metadata->set_string("layout", "HWC");
-        image_metadata->set_list_int("shape", make_shape(height, width, 3));
-        image_metadata->set_float("fx", calib.fx);
-        image_metadata->set_float("fy", calib.fy);
-        image_metadata->set_float("cx", calib.cx);
-        image_metadata->set_float("cy", calib.cy);
-        image_metadata->set_list_float("distortion", make_distortion(calib.distortion));
-        image_metadata->set_int("frame_id", current_frame_id);
-        image_metadata->set_int("timestamp_ns", timestamp_ns);
+        image_metadata->set_list_int("shape", make_vector_int({frame.height, frame.width, 3}));
+        set_intrinsics(*image_metadata, frame.calib);
+        set_frame_stamps(*image_metadata, frame);
 
-        auto image_result = send_output_with_metadata(
-            dora_node.send_output, "image_bgr", image_slice, std::move(image_metadata));
-        if (!std::string(image_result.error).empty()) {
-            std::cerr << "Error sending image_bgr: " << std::string(image_result.error) << std::endl;
-        }
+        report("image_bgr", send_output_with_metadata(dora_node.send_output, "image_bgr",
+                                                      image_slice, std::move(image_metadata)));
+
+        // --- depth ---
+        rust::Slice<const uint8_t> depth_slice{
+            reinterpret_cast<const uint8_t*>(frame.depth.data()),
+            frame.depth.size() * sizeof(float)};
+
+        auto depth_metadata = new_metadata();
+        depth_metadata->set_string("dtype", "float32");
+        depth_metadata->set_string("layout", "HW");
+        depth_metadata->set_string("units", "mm");
+        depth_metadata->set_list_int("shape", make_vector_int({frame.height, frame.width}));
+        set_intrinsics(*depth_metadata, frame.calib);
+        set_frame_stamps(*depth_metadata, frame);
+
+        report("depth", send_output_with_metadata(dora_node.send_output, "depth", depth_slice,
+                                                  std::move(depth_metadata)));
 
         // --- point_cloud ---
-        int pc_width  = point_cloud_mat.getWidth();
-        int pc_height = point_cloud_mat.getHeight();
+        if (frame.has_point_cloud) {
+            rust::Slice<const uint8_t> pc_slice{
+                reinterpret_cast<const uint8_t*>(frame.point_cloud.data()),
+                frame.point_cloud.size() * sizeof(float)};
 
-        std::vector<float> pc_buf;
-        const uint8_t* pc_data_ptr = nullptr;
-        size_t         pc_data_size = 0;
-        auto pc_metadata = new_metadata();
-        pc_metadata->set_string("dtype", "float32");
-        pc_metadata->set_string("channels", "XYZRGBA");
-        pc_metadata->set_string("units", "mm");
-        pc_metadata->set_float("fx", calib.fx);
-        pc_metadata->set_float("fy", calib.fy);
-        pc_metadata->set_float("cx", calib.cx);
-        pc_metadata->set_float("cy", calib.cy);
-        pc_metadata->set_list_float("distortion", make_distortion(calib.distortion));
-
-        if (pc_mode == PcMode::HALF) {
-            int hw = pc_width / 2, hh = pc_height / 2;
-            pc_buf.resize(static_cast<size_t>(hh * hw * 4));
-            const sl::float4* src = point_cloud_mat.getPtr<sl::float4>();
-            for (int y = 0; y < hh; y++) {
-                for (int x = 0; x < hw; x++) {
-                    const sl::float4& pt = src[(y * 2) * pc_width + (x * 2)];
-                    float* dst = pc_buf.data() + (y * hw + x) * 4;
-                    dst[0] = pt.x; dst[1] = pt.y; dst[2] = pt.z; dst[3] = pt.w;
-                }
-            }
+            auto pc_metadata = new_metadata();
+            pc_metadata->set_string("dtype", "float32");
+            pc_metadata->set_string("channels", "XYZRGBA");
+            pc_metadata->set_string("units", "mm");
             pc_metadata->set_string("layout", "HWC");
-            pc_metadata->set_list_int("shape", make_shape(hh, hw, 4));
-            pc_data_ptr  = reinterpret_cast<const uint8_t*>(pc_buf.data());
-            pc_data_size = pc_buf.size() * sizeof(float);
-        } else if (pc_mode == PcMode::ROI) {
-            const auto& objs = detector->objects();
-            for (const auto& obj : objs) {
-                int x1 = std::max(0, static_cast<int>(obj.bounding_box_2d[0]));
-                int y1 = std::max(0, static_cast<int>(obj.bounding_box_2d[1]));
-                int x2 = std::min(pc_width  - 1, static_cast<int>(obj.bounding_box_2d[4]));
-                int y2 = std::min(pc_height - 1, static_cast<int>(obj.bounding_box_2d[5]));
-                for (int y = y1; y <= y2; y++) {
-                    for (int x = x1; x <= x2; x++) {
-                        sl::float4 pt;
-                        point_cloud_mat.getValue(x, y, &pt);
-                        if (!std::isnan(pt.x) && !std::isinf(pt.x)) {
-                            pc_buf.push_back(pt.x); pc_buf.push_back(pt.y);
-                            pc_buf.push_back(pt.z); pc_buf.push_back(pt.w);
-                        }
-                    }
-                }
-            }
-            int N = static_cast<int>(pc_buf.size() / 4);
-            pc_metadata->set_string("layout", "NC");
-            pc_metadata->set_string("mode", "roi");
-            pc_metadata->set_list_int("shape", make_vector_int({N, 4}));
-            pc_data_ptr  = reinterpret_cast<const uint8_t*>(pc_buf.data());
-            pc_data_size = pc_buf.size() * sizeof(float);
-        } else {
-            // FULL — send raw sl::Mat pointer directly (no copy)
-            pc_data_size = static_cast<size_t>(pc_width * pc_height * sizeof(sl::float4));
-            pc_data_ptr  = reinterpret_cast<const uint8_t*>(point_cloud_mat.getPtr<sl::float4>());
-            pc_metadata->set_string("layout", "HWC");
-            pc_metadata->set_list_int("shape", make_shape(pc_height, pc_width, 4));
-        }
+            pc_metadata->set_list_int("shape",
+                                      make_vector_int({frame.pc_height, frame.pc_width, 4}));
+            set_intrinsics(*pc_metadata, frame.calib);
+            set_frame_stamps(*pc_metadata, frame);
 
-        rust::Slice<const uint8_t> pc_slice{pc_data_ptr, pc_data_size};
-        auto pc_result = send_output_with_metadata(
-            dora_node.send_output, "point_cloud", pc_slice, std::move(pc_metadata));
-        if (!std::string(pc_result.error).empty()) {
-            std::cerr << "Error sending point_cloud: " << std::string(pc_result.error) << std::endl;
+            report("point_cloud", send_output_with_metadata(dora_node.send_output, "point_cloud",
+                                                            pc_slice, std::move(pc_metadata)));
         }
 
         // --- camera_info ---
-        float camera_data[4] = {calib.fx, calib.fy, calib.cx, calib.cy};
-        rust::Slice<const uint8_t> cam_slice{
-            reinterpret_cast<const uint8_t*>(camera_data), sizeof(camera_data)};
+        if (frame.camera_info_first) {
+            float camera_data[4] = {frame.calib.fx, frame.calib.fy, frame.calib.cx,
+                                    frame.calib.cy};
+            rust::Slice<const uint8_t> cam_slice{
+                reinterpret_cast<const uint8_t*>(camera_data), sizeof(camera_data)};
 
-        auto cam_metadata = new_metadata();
-        cam_metadata->set_string("dtype", "float32");
-        cam_metadata->set_string("layout", "C");
-        cam_metadata->set_list_int("shape", make_vector_int({4}));
-        cam_metadata->set_list_float("distortion", make_distortion(calib.distortion));
+            auto cam_metadata = new_metadata();
+            cam_metadata->set_string("dtype", "float32");
+            cam_metadata->set_string("layout", "C");
+            cam_metadata->set_list_int("shape", make_vector_int({4}));
+            cam_metadata->set_list_float("distortion", make_distortion(frame.calib.distortion));
+            set_frame_stamps(*cam_metadata, frame);
 
-        auto cam_result = send_output_with_metadata(
-            dora_node.send_output, "camera_info", cam_slice, std::move(cam_metadata));
-        if (!std::string(cam_result.error).empty()) {
-            std::cerr << "Error sending camera_info: " << std::string(cam_result.error) << std::endl;
-        }
-
-        // --- objects ---
-        const auto& objects = detector->objects();
-        auto obj_buffer = serialize_objects(objects);
-        int num_objects = static_cast<int>(objects.size());
-
-        const size_t obj_size = obj_buffer.size() * sizeof(float);
-        rust::Slice<const uint8_t> obj_slice{
-            reinterpret_cast<const uint8_t*>(obj_buffer.data()), obj_size};
-
-        auto obj_metadata = new_metadata();
-        obj_metadata->set_string("dtype", "float32");
-        obj_metadata->set_string("layout", "RC");
-        obj_metadata->set_string("units", "mm");
-        obj_metadata->set_list_int("shape", make_vector_int({num_objects, FIELDS_PER_OBJECT}));
-        obj_metadata->set_string("fields",
-            "label_id,confidence,x,y,z,"
-            "bb2d_x0,bb2d_y0,bb2d_x1,bb2d_y1,bb2d_x2,bb2d_y2,bb2d_x3,bb2d_y3,"
-            "bb3d_x0,bb3d_y0,bb3d_z0,bb3d_x1,bb3d_y1,bb3d_z1,"
-            "bb3d_x2,bb3d_y2,bb3d_z2,bb3d_x3,bb3d_y3,bb3d_z3,"
-            "bb3d_x4,bb3d_y4,bb3d_z4,bb3d_x5,bb3d_y5,bb3d_z5,"
-            "bb3d_x6,bb3d_y6,bb3d_z6,bb3d_x7,bb3d_y7,bb3d_z7");
-        obj_metadata->set_string("label_names", detector->label_names());
-        obj_metadata->set_string("engine", engine_name);
-        obj_metadata->set_int("frame_id", current_frame_id);
-        obj_metadata->set_int("timestamp_ns", timestamp_ns);
-        obj_metadata->set_int("image_width", width);
-        obj_metadata->set_int("image_height", height);
-
-        auto obj_result = send_output_with_metadata(
-            dora_node.send_output, "objects", obj_slice, std::move(obj_metadata));
-        if (!std::string(obj_result.error).empty()) {
-            std::cerr << "Error sending objects: " << std::string(obj_result.error) << std::endl;
+            report("camera_info", send_output_with_metadata(dora_node.send_output, "camera_info",
+                                                            cam_slice, std::move(cam_metadata)));
         }
     }
 
+    capture.stop();
+    if (!capture.wait_exit(std::chrono::seconds(2))) {
+        // The capture thread is inside a blocking SDK call, where nothing can
+        // reach it. Leave rather than let dora wait out its grace period and kill
+        // the node: the frames are gone either way, and Stop has been answered.
+        std::cerr << "capture thread still in the SDK; exiting anyway" << std::endl;
+        std::cout.flush();
+        std::_Exit(0);
+    }
     return 0;
 }

src/test_yolo_onnx.cc

@@ -1,113 +0,0 @@
-// Standalone test for the YoloOnnx (closed-vocabulary) detection engine.
-// No dora framework, no live ZED camera — loads an image from disk.
-//
-// Usage:
-//   bin/test_yolo_onnx [image_path]
-//   image_path   default: test/assets/yolo.png
-//
-// Required env vars:
-//   ZED_ONNX_MODEL_PATH        path to YOLO ONNX model
-//
-// Optional env vars:
-//   ZED_ONNX_LABELS            comma-separated class names
-//   ZED_ONNX_LABELS_FILE       path to newline-separated class names file
-//   ZED_ONNX_CONFIDENCE        detection threshold (default 0.25)
-//   ZED_ONNX_NMS_THRESHOLD     NMS IoU threshold (default 0.45)
-//   ZED_ONNX_INPUT_SIZE        model input size (default 640)
-//   ZED_ONNX_OUTPUT_FORMAT     nc_first|nc_last|nc_last_obj|yolov5|yolov8|yolov11|auto
-
-#include "engines/yolo_onnx.h"
-#include "util/env.h"
-
-#include <opencv2/opencv.hpp>
-#include <sl/Camera.hpp>
-
-#include <filesystem>
-#include <iostream>
-#include <sstream>
-#include <string>
-#include <vector>
-
-namespace fs = std::filesystem;
-
-namespace {
-
-std::vector<std::string> parse_csv(const std::string& s) {
-    std::vector<std::string> out;
-    std::istringstream ss(s);
-    std::string tok;
-    while (std::getline(ss, tok, ',')) {
-        size_t l = tok.find_first_not_of(" \t");
-        size_t r = tok.find_last_not_of(" \t");
-        if (l != std::string::npos)
-            out.push_back(tok.substr(l, r - l + 1));
-    }
-    return out;
-}
-
-void draw_detections(cv::Mat& img,
-                     const std::vector<engines::DetectedObject>& objects,
-                     const std::vector<std::string>& labels) {
-    for (const auto& obj : objects) {
-        int x1 = static_cast<int>(obj.bounding_box_2d[0]);
-        int y1 = static_cast<int>(obj.bounding_box_2d[1]);
-        int x2 = static_cast<int>(obj.bounding_box_2d[4]);
-        int y2 = static_cast<int>(obj.bounding_box_2d[5]);
-
-        std::string name = (obj.label_id >= 0 && obj.label_id < static_cast<int>(labels.size()))
-                           ? labels[obj.label_id] : std::to_string(obj.label_id);
-
-        char buf[64];
-        std::snprintf(buf, sizeof(buf), " %.2f", obj.confidence / 100.0f);
-
-        cv::rectangle(img, {x1, y1}, {x2, y2}, {0, 200, 0}, 2);
-        cv::putText(img, name + buf, {x1, std::max(y1 - 6, 14)},
-                    cv::FONT_HERSHEY_SIMPLEX, 0.55, {0, 200, 0}, 2, cv::LINE_AA);
-    }
-}
-
-} // namespace
-
-int main(int argc, char* argv[]) {
-    std::string image_path = (argc > 1) ? argv[1] : "test/assets/yolo.png";
-
-    auto config = engines::YoloOnnxConfig::from_env();
-
-    sl::Camera dummy_cam;
-    engines::YoloOnnx engine(config);
-    engine.enable(dummy_cam);
-
-    cv::Mat img = cv::imread(image_path);
-    if (img.empty()) {
-        std::cerr << "Error: cannot read image: " << image_path << std::endl;
-        return 1;
-    }
-
-    engine.detect(img);
-    const auto& objects = engine.objects();
-    auto labels = parse_csv(engine.label_names());
-
-    std::cout << "[" << image_path << "] " << objects.size() << " detection(s):" << std::endl;
-    for (const auto& obj : objects) {
-        std::string name = (obj.label_id >= 0 && obj.label_id < static_cast<int>(labels.size()))
-                           ? labels[obj.label_id] : std::to_string(obj.label_id);
-        std::cout << "  " << name
-                  << "  conf=" << obj.confidence / 100.0f
-                  << "  bbox=["
-                  << obj.bounding_box_2d[0] << "," << obj.bounding_box_2d[1]
-                  << " -> "
-                  << obj.bounding_box_2d[4] << "," << obj.bounding_box_2d[5]
-                  << "]" << std::endl;
-    }
-
-    cv::Mat annotated = img.clone();
-    draw_detections(annotated, objects, labels);
-
-    fs::path stem   = fs::path(image_path).stem();
-    fs::path parent = fs::path(image_path).parent_path();
-    std::string result_path = (parent / (stem.string() + "_result.png")).string();
-    cv::imwrite(result_path, annotated);
-    std::cout << "Saved: " << result_path << std::endl;
-
-    return 0;
-}

src/zed/camera.cc

@@ -1,11 +1,12 @@
 #include "zed/camera.h"
 #include "util/env.h"
 
-#include <chrono>
+#include <algorithm>
+#include <cstdlib>
 #include <iostream>
 #include <iterator>
 #include <string>
-#include <thread>
+#include <utility>
 
 namespace {
 
@@ -21,17 +22,30 @@ sl::RESOLUTION parse_resolution(int resolution) {
     }
 }
 
+const std::pair<const char*, sl::DEPTH_MODE> DEPTH_MODES[] = {
+    {"NEURAL", sl::DEPTH_MODE::NEURAL},
+    {"NEURAL_PLUS", sl::DEPTH_MODE::NEURAL_PLUS},
+    {"NEURAL_LIGHT", sl::DEPTH_MODE::NEURAL_LIGHT},
+    {"PERFORMANCE", sl::DEPTH_MODE::PERFORMANCE},
+    {"QUALITY", sl::DEPTH_MODE::QUALITY},
+    {"NONE", sl::DEPTH_MODE::NONE},
+};
+
+// A typo here silently changes depth quality if it falls back to a default, and
+// the resulting depth is wrong in a way that only shows up downstream, so an
+// unrecognised name stops the node instead.
 sl::DEPTH_MODE parse_depth_mode(const std::string& mode) {
-    if (mode == "PERFORMANCE") {
-        return sl::DEPTH_MODE::PERFORMANCE;
+    for (const auto& entry : DEPTH_MODES) {
+        if (mode == entry.first) {
+            return entry.second;
+        }
     }
-    if (mode == "QUALITY") {
-        return sl::DEPTH_MODE::QUALITY;
+    std::cerr << "Unknown ZED_DEPTH_MODE '" << mode << "'. Accepted values:";
+    for (const auto& entry : DEPTH_MODES) {
+        std::cerr << " " << entry.first;
     }
-    if (mode == "NONE") {
-        return sl::DEPTH_MODE::NONE;
-    }
-    return sl::DEPTH_MODE::NEURAL;
+    std::cerr << std::endl;
+    std::exit(1);
 }
 
 sl::FLIP_MODE parse_flip(const std::string& mode) {
@@ -41,6 +55,24 @@ sl::FLIP_MODE parse_flip(const std::string& mode) {
     return sl::FLIP_MODE::ON;
 }
 
+// A blank frame is checked on a strided sample rather than the full image, which
+// is just as conclusive and costs nothing at frame rate.
+constexpr int BLANK_FRAME_STRIDE = 16;
+
+// sl::Camera::reboot() reaches the ZED 2 and newer. On the first-generation ZED
+// and the ZED Mini it returns without touching the device, so calling it there
+// only costs time and fills the log.
+bool model_supports_reboot(sl::MODEL model) {
+    return model != sl::MODEL::ZED && model != sl::MODEL::ZED_M;
+}
+
+// The SDK reconnects a camera whose link hiccuped by itself. Dropping the handle
+// over one of these takes the recovery away from the SDK and hands it to us,
+// which is slower and, on a device the kernel is about to suspend, often fatal.
+bool is_transient_grab_error(sl::ERROR_CODE err) {
+    return err == sl::ERROR_CODE::CAMERA_REBOOTING || err == sl::ERROR_CODE::CORRUPTED_FRAME;
+}
+
 void list_available_cameras() {
     auto devices = sl::Camera::getDeviceList();
     std::cout << "Detected ZED cameras: " << devices.size() << std::endl;
@@ -59,50 +91,112 @@ Config Config::from_env() {
     Config c;
     c.resolution = util::get_env_int("ZED_RESOLUTION", c.resolution);
     c.fps = util::get_env_int("ZED_FPS", c.fps);
-    c.depth_mode = util::get_env_str("ZED_DEPTH_MODE", c.depth_mode);
+    c.depth_mode = parse_depth_mode(util::get_env_str("ZED_DEPTH_MODE", "NEURAL"));
     c.depth_min_mm = util::get_env_int("ZED_DEPTH_MIN_MM", c.depth_min_mm);
     c.depth_max_mm = util::get_env_int("ZED_DEPTH_MAX_MM", c.depth_max_mm);
     c.depth_fill = util::get_env_bool("ZED_DEPTH_FILL", c.depth_fill);
+    c.point_cloud = util::get_env_bool("ZED_POINT_CLOUD", c.point_cloud);
+    c.pc_mode = util::get_env_str("ZED_PC_MODE", "FULL") == "HALF" ? PcMode::HALF : PcMode::FULL;
+    c.confidence_threshold = util::get_env_int("ZED_CONFIDENCE_THRESHOLD", c.confidence_threshold);
+    c.texture_confidence_threshold =
+        util::get_env_int("ZED_TEXTURE_CONFIDENCE_THRESHOLD", c.texture_confidence_threshold);
     c.flip = util::get_env_str("ZED_FLIP", c.flip);
     c.warmup_frames = util::get_env_int("ZED_WARMUP_FRAMES", c.warmup_frames);
-    int retry_seconds = util::get_env_int("ZED_RETRY_INTERVAL_SECONDS", 5);
-    c.retry_interval_ms = retry_seconds * 1000;
     return c;
 }
 
 Camera::Camera(const Config& config) : config_(config) {
+    // Process-wide, and must be set before any camera is opened. Without it the
+    // epoch of the timestamps published alongside every frame depends on the SDK
+    // default rather than on anything this node states.
+    sl::setTimestampClock(sl::TIMESTAMP_CLOCK::SYSTEM_CLOCK);
+
     init_params_.camera_resolution = parse_resolution(config_.resolution);
     init_params_.camera_fps = config_.fps;
-    init_params_.depth_mode = parse_depth_mode(config_.depth_mode);
+    init_params_.depth_mode = config_.depth_mode;
     init_params_.coordinate_units = sl::UNIT::MILLIMETER;
     init_params_.camera_image_flip = parse_flip(config_.flip);
     init_params_.depth_minimum_distance = config_.depth_min_mm;
     init_params_.depth_maximum_distance = config_.depth_max_mm;
+    // Recover a hiccuping link inside the SDK rather than out here. Without this
+    // a grab blocks for the roughly 17 s the SDK spends reconnecting, freezing the
+    // dora event loop; with it the grab returns CAMERA_REBOOTING straight away and
+    // the SDK reconnects in the background, holding the device open meanwhile.
+    init_params_.async_grab_camera_recovery = true;
+    // Let the SDK flag the frames a stalled camera hands back looking valid.
+    init_params_.enable_image_validity_check = 1;
 
     runtime_params_.enable_depth = true;
     runtime_params_.enable_fill_mode = config_.depth_fill;
+    // The two thresholds discard the depth the SDK is least certain of, which is
+    // the depth at object edges: those pixels straddle the object and whatever is
+    // behind it, and belong to neither.
+    runtime_params_.confidence_threshold = config_.confidence_threshold;
+    runtime_params_.texture_confidence_threshold = config_.texture_confidence_threshold;
+
+    std::cout << "ZED depth: mode=" << sl::toString(init_params_.depth_mode)
+              << ", range=" << config_.depth_min_mm << "-" << config_.depth_max_mm << " mm"
+              << ", confidence=" << config_.confidence_threshold
+              << ", texture_confidence=" << config_.texture_confidence_threshold
+              << ", fill=" << (config_.depth_fill ? "on" : "off") << std::endl;
+
+    list_available_cameras();
 }
 
 Camera::~Camera() {
     close();
 }
 
-void Camera::open() {
-    list_available_cameras();
-    while (true) {
-        sl::ERROR_CODE err = zed_.open(init_params_);
-        if (err == sl::ERROR_CODE::SUCCESS) {
-            return;
-        }
+bool Camera::open() {
+    sl::ERROR_CODE err = zed_.open(init_params_);
+    if (err != sl::ERROR_CODE::SUCCESS) {
         std::cerr << "Failed to open ZED camera: " << sl::toString(err) << std::endl;
-        std::this_thread::sleep_for(std::chrono::milliseconds(config_.retry_interval_ms));
+        return false;
     }
+    recovering_ = false;
+
+    sl::CameraInformation info = zed_.getCameraInformation();
+    serial_number_ = info.serial_number;
+    model_ = info.camera_model;
+    // After the first open, reopen the same physical camera. Recovery runs while
+    // the device may be mid-re-enumeration, and picking whichever ZED answers
+    // first is not the same thing as picking ours.
+    init_params_.input.setFromSerialNumber(serial_number_);
+
+    std::cout << "ZED camera opened (" << sl::toString(model_) << ", SN " << serial_number_ << ")"
+              << std::endl;
+    return true;
 }
 
 void Camera::close() {
+    recovering_ = false;
     zed_.close();
 }
 
+void Camera::reset() {
+    close();
+    if (serial_number_ == 0 || !model_supports_reboot(model_)) {
+        return;
+    }
+    // The ZED can stop delivering frames while staying enumerated: the video
+    // endpoint halts, grabs fail, and reopening returns CAMERA FAILED TO SETUP
+    // because the endpoint is still halted. A device reset clears that, and
+    // reboot() is the SDK's way to ask for one without unplugging the camera.
+    // The device re-enumerates afterwards, so the caller's backoff before the
+    // next open is what lets it settle.
+    sl::ERROR_CODE err = sl::Camera::reboot(static_cast<int>(serial_number_), true);
+    std::cout << "reset ZED camera (SN " << serial_number_ << "): " << sl::toString(err)
+              << std::endl;
+}
+
+bool Camera::is_open() const {
+    return zed_.isOpened();
+}
+
+bool Camera::recovering() const {
+    return recovering_;
+}
+
 void Camera::warmup() {
     for (int i = 0; i < config_.warmup_frames; i++) {
         zed_.grab(runtime_params_);
@@ -110,11 +204,34 @@ void Camera::warmup() {
 }
 
 bool Camera::grab() {
-    if (zed_.grab(runtime_params_) != sl::ERROR_CODE::SUCCESS) {
+    sl::ERROR_CODE err = zed_.grab(runtime_params_);
+    if (err != sl::ERROR_CODE::SUCCESS) {
+        if (is_transient_grab_error(err)) {
+            // Every grab repeats this for as long as the reconnect takes, so only
+            // the first one is worth a line in the log.
+            if (!recovering_) {
+                recovering_ = true;
+                std::cerr << "ZED camera link dropped (" << sl::toString(err)
+                          << "); the SDK is reconnecting it" << std::endl;
+            }
+        } else {
+            recovering_ = false;
+            std::cerr << "Failed to grab frame: " << sl::toString(err) << std::endl;
+        }
         return false;
     }
+    if (recovering_) {
+        recovering_ = false;
+        std::cout << "ZED camera link restored" << std::endl;
+    }
+
     zed_.retrieveImage(image_mat_, sl::VIEW::LEFT);
-    zed_.retrieveMeasure(point_cloud_mat_, sl::MEASURE::XYZRGBA);
+    zed_.retrieveMeasure(depth_mat_, sl::MEASURE::DEPTH);
+    // At 1080p a full XYZRGBA cloud is ~14.7 MB per frame, and retrieving it
+    // costs capture time whether or not anything consumes it.
+    if (config_.point_cloud) {
+        zed_.retrieveMeasure(point_cloud_mat_, sl::MEASURE::XYZRGBA);
+    }
     return true;
 }
 
@@ -135,16 +252,48 @@ Calibration Camera::calibration() const {
     return result;
 }
 
+bool Camera::image_is_blank() {
+    const int height = image_mat_.getHeight();
+    const int width = image_mat_.getWidth();
+    if (height == 0 || width == 0) {
+        return true;
+    }
+
+    const size_t step = image_mat_.getStep(sl::MEM::CPU);
+    const sl::uchar4* src = image_mat_.getPtr<sl::uchar4>();
+
+    unsigned char lowest[3] = {255, 255, 255};
+    unsigned char highest[3] = {0, 0, 0};
+    for (int y = 0; y < height; y += BLANK_FRAME_STRIDE) {
+        const sl::uchar4* row = src + static_cast<size_t>(y) * step;
+        for (int x = 0; x < width; x += BLANK_FRAME_STRIDE) {
+            const unsigned char channels[3] = {row[x].x, row[x].y, row[x].z};
+            for (int c = 0; c < 3; c++) {
+                lowest[c] = std::min(lowest[c], channels[c]);
+                highest[c] = std::max(highest[c], channels[c]);
+            }
+        }
+    }
+    // Sensor noise makes a single-colour frame impossible for a live camera, so
+    // this tells a dead device apart from any real scene.
+    return lowest[0] == highest[0] && lowest[1] == highest[1] && lowest[2] == highest[2];
+}
+
+int64_t Camera::device_timestamp_ns() {
+    return static_cast<int64_t>(
+        zed_.getTimestamp(sl::TIME_REFERENCE::IMAGE).getNanoseconds());
+}
+
 sl::Mat& Camera::image() {
     return image_mat_;
 }
 
+sl::Mat& Camera::depth() {
+    return depth_mat_;
+}
+
 sl::Mat& Camera::point_cloud() {
     return point_cloud_mat_;
 }
 
-sl::Camera& Camera::handle() {
-    return zed_;
-}
-
 } // namespace zed

src/zed/camera.h

@@ -1,27 +1,33 @@
 #pragma once
 
 #include <sl/Camera.hpp>
+#include <cstdint>
 #include <string>
 #include <vector>
 
 namespace zed {
 
+enum class PcMode { FULL, HALF };
+
 struct Config {
-    int resolution = 720;
+    int resolution = 1080;
     int fps = 15;
-    std::string depth_mode = "NEURAL";
-    int depth_min_mm = 10;
-    int depth_max_mm = 500;
+    sl::DEPTH_MODE depth_mode = sl::DEPTH_MODE::NEURAL;
+    int depth_min_mm = 200;
+    int depth_max_mm = 600;
     bool depth_fill = false;
+    bool point_cloud = false;
+    PcMode pc_mode = PcMode::FULL;
+    int confidence_threshold = 60;
+    int texture_confidence_threshold = 100;
     std::string flip = "ON";
     int warmup_frames = 30;
-    int retry_interval_ms = 5000;
 
     static Config from_env();
 };
 
 struct Calibration {
-    float fx, fy, cx, cy;
+    float fx = 0, fy = 0, cx = 0, cy = 0;
     std::vector<double> distortion;
 };
 
@@ -30,14 +36,26 @@ public:
     explicit Camera(const Config& config);
     ~Camera();
 
-    void open();
+    // One attempt, returning whether it succeeded. The caller decides when to try
+    // again, so a camera that is down never blocks the dora event loop.
+    bool open();
     void close();
+    // Close the handle and reset the device over USB, where the model allows it.
+    void reset();
     void warmup();
     bool grab();
+    bool is_open() const;
+    // True while the SDK is reconnecting the camera after a link hiccup.
+    bool recovering() const;
+    // True when the last grabbed frame carries no picture at all. A camera whose
+    // link has dropped does not necessarily fail its grabs: it can keep reporting
+    // success and hand back a uniform frame.
+    bool image_is_blank();
     Calibration calibration() const;
+    int64_t device_timestamp_ns();
     sl::Mat& image();
+    sl::Mat& depth();
     sl::Mat& point_cloud();
-    sl::Camera& handle();
 
 private:
     Config config_;
@@ -45,7 +63,13 @@ private:
     sl::InitParameters init_params_;
     sl::RuntimeParameters runtime_params_;
     sl::Mat image_mat_;
+    sl::Mat depth_mat_;
     sl::Mat point_cloud_mat_;
+    // Read off the device once it is open and kept afterwards: resetting a wedged
+    // camera needs the serial and the model, and by then the handle is gone.
+    unsigned int serial_number_ = 0;
+    sl::MODEL model_ = sl::MODEL::LAST;
+    bool recovering_ = false;
 };
 
 } // namespace zed

src/zed/capture.cc

@@ -0,0 +1,302 @@
+#include "zed/capture.h"
+
+#include <opencv2/opencv.hpp>
+
+#include <algorithm>
+#include <chrono>
+#include <cmath>
+#include <iostream>
+
+namespace {
+
+using Clock = std::chrono::steady_clock;
+
+// Going this long without a usable frame means the camera is wedged rather than
+// dropping the odd frame. Measured in elapsed time rather than in failed grabs,
+// because a stalling camera can spend seconds inside a single grab.
+constexpr double STALL_TIMEOUT_S = 5.0;
+
+// How long the SDK is left to reconnect the camera on its own before the loop
+// stops waiting on it and rebuilds the camera anyway. A driver that says it is
+// recovering is usually right, but it has no way to tell us it has given up.
+constexpr double DRIVER_RECOVERY_TIMEOUT_S = 30.0;
+
+// How long to wait between grabs while the camera is not delivering.
+constexpr std::chrono::milliseconds GRAB_RETRY{500};
+
+// How long to wait before each reopen: one entry per attempt, the last repeated
+// for every attempt beyond it.
+constexpr int REOPEN_BACKOFF_S[] = {1, 2, 5, 10};
+
+std::chrono::seconds reopen_backoff(int attempt) {
+    const size_t index = std::min(static_cast<size_t>(attempt), std::size(REOPEN_BACKOFF_S)) - 1;
+    return std::chrono::seconds(REOPEN_BACKOFF_S[index]);
+}
+
+double seconds_since(Clock::time_point since) {
+    return std::chrono::duration<double>(Clock::now() - since).count();
+}
+
+int64_t host_timestamp_ns() {
+    return std::chrono::duration_cast<std::chrono::nanoseconds>(
+               std::chrono::system_clock::now().time_since_epoch())
+        .count();
+}
+
+// Copy the depth map into a tightly packed [H, W] buffer of millimetres.
+//
+// The rows of an sl::Mat can be padded, so the buffer is not necessarily H * W
+// floats. inf and nan mark depth the SDK could not measure; they are cleared to
+// 0 here, as the reference driver does, because a hole is meaningful and a
+// consumer cannot back-project a non-finite value.
+void pack_depth(sl::Mat& depth_mat, std::vector<float>& out) {
+    const int height = depth_mat.getHeight();
+    const int width = depth_mat.getWidth();
+    const size_t step = depth_mat.getStep(sl::MEM::CPU);
+    const float* src = depth_mat.getPtr<sl::float1>();
+
+    out.resize(static_cast<size_t>(height) * static_cast<size_t>(width));
+    for (int y = 0; y < height; y++) {
+        const float* row = src + static_cast<size_t>(y) * step;
+        float* dst = out.data() + static_cast<size_t>(y) * static_cast<size_t>(width);
+        for (int x = 0; x < width; x++) {
+            dst[x] = std::isfinite(row[x]) ? row[x] : 0.0f;
+        }
+    }
+}
+
+} // namespace
+
+namespace zed {
+
+Capture::Capture(const Config& config) : config_(config), camera_(config) {}
+
+Capture::~Capture() {
+    stop();
+    if (thread_.joinable()) {
+        // The thread may still be inside the SDK. Detaching leaks it for the few
+        // moments the process has left, which is preferable to blocking here.
+        if (wait_exit(std::chrono::seconds(1))) {
+            thread_.join();
+        } else {
+            thread_.detach();
+        }
+    }
+}
+
+void Capture::start() {
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        running_ = true;
+    }
+    thread_ = std::thread(&Capture::run, this);
+}
+
+void Capture::stop() {
+    std::lock_guard<std::mutex> lock(mutex_);
+    running_ = false;
+    wake_.notify_all();
+}
+
+bool Capture::wait_exit(std::chrono::milliseconds timeout) {
+    std::unique_lock<std::mutex> lock(mutex_);
+    return exited_.wait_for(lock, timeout, [this] { return finished_; });
+}
+
+bool Capture::take_frame(Frame& out) {
+    std::lock_guard<std::mutex> lock(mutex_);
+    if (!has_frame_) {
+        return false;
+    }
+    std::swap(out, latest_);
+    has_frame_ = false;
+    return true;
+}
+
+bool Capture::take_state_change(std::string& out) {
+    std::lock_guard<std::mutex> lock(mutex_);
+    if (!state_changed_) {
+        return false;
+    }
+    out = state_;
+    state_changed_ = false;
+    return true;
+}
+
+void Capture::set_state(const char* state) {
+    std::lock_guard<std::mutex> lock(mutex_);
+    if (state_ == state) {
+        return;
+    }
+    state_ = state;
+    state_changed_ = true;
+    std::cout << "camera state: " << state_ << std::endl;
+}
+
+void Capture::sleep_interruptible(std::chrono::milliseconds duration) {
+    std::unique_lock<std::mutex> lock(mutex_);
+    wake_.wait_for(lock, duration, [this] { return !running_; });
+}
+
+void Capture::begin_recovery(int attempt) {
+    // The first attempt only closes the handle, which is enough when the driver
+    // rather than the device lost its way; past that the device itself is wedged
+    // and nothing short of resetting it will bring the frames back.
+    if (attempt <= 1) {
+        camera_.close();
+    } else {
+        camera_.reset();
+    }
+    const auto wait = reopen_backoff(attempt);
+    std::cerr << "recovering camera (attempt " << attempt << "); next open in " << wait.count()
+              << "s" << std::endl;
+    sleep_interruptible(std::chrono::duration_cast<std::chrono::milliseconds>(wait));
+}
+
+void Capture::pack_frame(Frame& frame, int64_t frame_id, bool camera_info_first) {
+    sl::Mat& image_mat = camera_.image();
+    const int width = image_mat.getWidth();
+    const int height = image_mat.getHeight();
+
+    frame.frame_id = frame_id;
+    frame.device_ns = camera_.device_timestamp_ns();
+    frame.host_ns = host_timestamp_ns();
+    frame.width = width;
+    frame.height = height;
+    frame.calib = camera_.calibration();
+    frame.camera_info_first = camera_info_first;
+
+    cv::Mat image_rgba(height, width, CV_8UC4, image_mat.getPtr<sl::uchar1>(),
+                       image_mat.getStep(sl::MEM::CPU) * sizeof(sl::uchar4));
+    frame.image_bgr.resize(static_cast<size_t>(height) * width * 3);
+    cv::Mat image_bgr(height, width, CV_8UC3, frame.image_bgr.data());
+    cv::cvtColor(image_rgba, image_bgr, cv::COLOR_BGRA2BGR);
+
+    pack_depth(camera_.depth(), frame.depth);
+
+    frame.has_point_cloud = config_.point_cloud;
+    if (!frame.has_point_cloud) {
+        return;
+    }
+
+    // Unlike the image and depth, the cloud used to be published straight from
+    // the SDK buffer. It has to be copied now that the reader is another thread.
+    sl::Mat& pc_mat = camera_.point_cloud();
+    const int pc_width = pc_mat.getWidth();
+    const int pc_height = pc_mat.getHeight();
+    const size_t pc_step = pc_mat.getStep(sl::MEM::CPU);
+    const sl::float4* src = pc_mat.getPtr<sl::float4>();
+
+    const int stride = config_.pc_mode == PcMode::HALF ? 2 : 1;
+    frame.pc_width = pc_width / stride;
+    frame.pc_height = pc_height / stride;
+    frame.point_cloud.resize(static_cast<size_t>(frame.pc_height) * frame.pc_width * 4);
+
+    for (int y = 0; y < frame.pc_height; y++) {
+        const sl::float4* row = src + static_cast<size_t>(y * stride) * pc_step;
+        float* dst = frame.point_cloud.data() + static_cast<size_t>(y) * frame.pc_width * 4;
+        for (int x = 0; x < frame.pc_width; x++) {
+            const sl::float4& point = row[x * stride];
+            dst[x * 4 + 0] = point.x;
+            dst[x * 4 + 1] = point.y;
+            dst[x * 4 + 2] = point.z;
+            dst[x * 4 + 3] = point.w;
+        }
+    }
+}
+
+void Capture::run() {
+    int recovery_attempts = 0;
+    bool streamed_once = false;
+    bool stall_reported = false;
+    bool camera_info_pending = false;
+    auto last_good_frame_at = Clock::now();
+    int64_t frame_id = 0;
+
+    set_state("opening");
+
+    while (true) {
+        {
+            std::lock_guard<std::mutex> lock(mutex_);
+            if (!running_) {
+                break;
+            }
+        }
+
+        const char* down_state = streamed_once ? "recovering" : "opening";
+
+        if (!camera_.is_open()) {
+            set_state(down_state);
+            if (!camera_.open()) {
+                recovery_attempts++;
+                begin_recovery(recovery_attempts);
+                continue;
+            }
+            // Discard the first frames so auto-exposure settles. This blocks, and
+            // on this thread that is fine.
+            camera_.warmup();
+            camera_info_pending = true;
+            // Opening the ZED takes seconds. Start the stall clock from here so
+            // that cost is never mistaken for a stalled stream.
+            last_good_frame_at = Clock::now();
+            // recovery_attempts is deliberately not cleared here. A wedged device
+            // can open and still deliver nothing; clearing on open would restart
+            // the escalation ladder every cycle and never reach the reset.
+            continue;
+        }
+
+        // A blank frame counts as a failed grab: a camera whose link has dropped
+        // does not necessarily fail its grabs, it can keep reporting success and
+        // hand back a uniform frame with an empty depth map.
+        if (!camera_.grab() || camera_.image_is_blank()) {
+            set_state(down_state);
+            const double stalled_for = seconds_since(last_good_frame_at);
+            // The SDK holds the camera open while it reconnects a dropped link.
+            // Taking the handle away mid-reconnect aborts that and leaves us to
+            // redo it the slow way, so it gets longer than a plain stall.
+            const double patience =
+                camera_.recovering() ? DRIVER_RECOVERY_TIMEOUT_S : STALL_TIMEOUT_S;
+            if (stalled_for < patience) {
+                sleep_interruptible(GRAB_RETRY);
+                continue;
+            }
+            if (!stall_reported) {
+                stall_reported = true;
+                std::cerr << "camera delivered no usable frame for " << stalled_for << "s"
+                          << std::endl;
+            }
+            recovery_attempts++;
+            begin_recovery(recovery_attempts);
+            continue;
+        }
+
+        if (stall_reported) {
+            std::cout << "camera recovered after " << recovery_attempts << " attempt(s)"
+                      << std::endl;
+            stall_reported = false;
+        }
+        last_good_frame_at = Clock::now();
+        recovery_attempts = 0;
+        streamed_once = true;
+
+        pack_frame(scratch_, frame_id++, camera_info_pending);
+        camera_info_pending = false;
+        {
+            std::lock_guard<std::mutex> lock(mutex_);
+            // Swapping rather than assigning hands the previous frame's buffers
+            // back, so steady-state capture does not reallocate.
+            std::swap(latest_, scratch_);
+            has_frame_ = true;
+        }
+        set_state("streaming");
+    }
+
+    camera_.close();
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        finished_ = true;
+        exited_.notify_all();
+    }
+}
+
+} // namespace zed

src/zed/capture.h

@@ -0,0 +1,94 @@
+#pragma once
+
+#include "zed/camera.h"
+
+#include <condition_variable>
+#include <cstdint>
+#include <mutex>
+#include <string>
+#include <thread>
+#include <vector>
+
+namespace zed {
+
+// One captured frame, owning its buffers.
+//
+// The SDK reuses its own buffers on the next grab, so nothing here can point
+// into an sl::Mat: the capture thread copies every payload out before handing
+// the frame over.
+struct Frame {
+    int64_t frame_id = 0;
+    int64_t device_ns = 0;
+    int64_t host_ns = 0;
+    int width = 0;
+    int height = 0;
+    std::vector<uint8_t> image_bgr; // height * width * 3
+    std::vector<float> depth;       // height * width, millimetres
+    bool has_point_cloud = false;
+    int pc_width = 0;
+    int pc_height = 0;
+    std::vector<float> point_cloud; // pc_height * pc_width * 4
+    Calibration calib;
+    // Set on the first frame of a camera session, so camera_info is published
+    // once per session rather than per frame.
+    bool camera_info_first = false;
+};
+
+// Owns the camera on its own thread and hands the newest frame to the reader.
+//
+// The ZED SDK can block for an unbounded time inside grab() and open() — a
+// disconnected camera has been measured blocking indefinitely, even with
+// InitParameters::async_grab_camera_recovery set. Driving the camera from the
+// dora event loop therefore froze the node: it stopped reading events, so Stop
+// went unanswered and the recovery logic itself was never reached. Everything
+// that can block lives on this thread instead, and the event loop only ever
+// touches the buffer below.
+class Capture {
+public:
+    explicit Capture(const Config& config);
+    ~Capture();
+
+    void start();
+    // Ask the thread to finish. Returns immediately: the thread may be inside a
+    // blocking SDK call and cannot be interrupted there.
+    void stop();
+    // Wait for the thread to actually leave, up to timeout. False means it is
+    // still stuck in the SDK.
+    bool wait_exit(std::chrono::milliseconds timeout);
+
+    // Move the newest frame out, if one arrived since the last call. Frames that
+    // are not collected are overwritten: the reader always wants the latest.
+    bool take_frame(Frame& out);
+    // Report the camera state, if it changed since the last call.
+    bool take_state_change(std::string& out);
+
+private:
+    void run();
+    void set_state(const char* state);
+    // Release the camera and wait out the backoff for this attempt. The wait is
+    // interruptible, so stop() is honoured while the camera is down.
+    void begin_recovery(int attempt);
+    void sleep_interruptible(std::chrono::milliseconds duration);
+    void pack_frame(Frame& frame, int64_t frame_id, bool camera_info_first);
+
+    Config config_;
+    Camera camera_;
+    std::thread thread_;
+
+    std::mutex mutex_;
+    std::condition_variable wake_;
+    std::condition_variable exited_;
+    bool running_ = false;
+    bool finished_ = false;
+
+    Frame latest_;
+    // Holds the previously published frame once the reader has taken the newest
+    // one, so its buffers are reused instead of reallocated every frame.
+    Frame scratch_;
+    bool has_frame_ = false;
+
+    std::string state_;
+    bool state_changed_ = false;
+};
+
+} // namespace zed