diff --git a/AGENTS.md b/AGENTS.md index 8f7ca9f..dfc0fe8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,10 +4,18 @@ This file provides guidance to AI coding assistants when working with code in th ## What this is -A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame and publishes it over the dora dataflow: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Camera management lives in `src/zed/` and the dora event loop in `src/main.cc`. +A dora-rs C++ node that interfaces with ZED stereo cameras. Capture is free-running on its own thread at `ZED_FPS`; on each tick the event loop publishes the newest frame it has produced, or nothing if none arrived since the last tick. What goes out: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Every payload is a typed Arrow array — the two frames as `std/media/v1/Image`, the rest as `std/core/v1/Float32` and `.../String` — which is what lets the node feed [dora-image-view](https://gitea.liria.cl/liria/dora-image-view) and anything else built against those types. Camera management lives in `src/zed/`, payload packing in `src/arrow/`, and the dora event loop in `src/main.cc`. The node is a camera source only. It ran inference in-process through `v0.1.0`, under `src/engines/`; that was removed so the node has no inference dependency and builds on any platform the ZED SDK supports. The removed code remains reachable at tag `v0.1.0`. +## Test + +```bash +tests/run.sh +``` + +Needs `g++` and `uv`; no camera, no ZED SDK, no CUDA. + ## Build ```bash @@ -28,8 +36,8 @@ For a build without installing the SDK stack on the host, see the `Dockerfile` ( ```bash dora up -dora start dataflow.yml # camera alone -dora start dataflow_plot.yml # camera plus the opencv-plot viewer +dora start dataflow.yml # camera alone +dora start dataflow_view.yml --uv # camera plus dora-image-view ``` The two dataflows both declare the camera node, so run one at a time. @@ -42,20 +50,22 @@ Camera behavior is configured entirely through environment variables (see README - **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: `open()` (one attempt, returns success), warmup, grab (image + depth + point cloud), calibration, device timestamp, blank-frame check, `close()` and `reset()` (close plus `sl::Camera::reboot()` where the model allows it). Exposes `handle()` for engine access. - **`src/zed/capture.h/cc`** — `zed::Capture` and `zed::Frame`. Owns the camera on a dedicated thread: open, warmup, grab, blank/stall detection, backoff and escalation all live here, along with the packing of every payload into owned buffers. The newest frame is handed to the reader by swapping under a mutex (`take_frame`), which also recycles the previous frame's buffers so steady-state capture does not reallocate. **Nothing outside this file may call the ZED SDK** — it blocks unboundedly on a disconnected camera, which is what froze the node when the loop drove the camera directly. - **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace. `parse_depth_mode` runs in `Config::from_env()`, not in the constructor, so an unrecognised `ZED_DEPTH_MODE` exits before the node joins the dataflow rather than falling back to a default. -- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()`, then sends `image_bgr` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `pack_depth` copies the depth map into a tightly packed `[H, W]` float buffer (sl::Mat rows can be padded) and clears `inf`/`nan` to `0`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds). Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension. The loop never touches the SDK: it drains `take_state_change` and `take_frame` from the capture thread and publishes. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node. +- **`src/arrow/array.h/cc`** — Builds the Arrow arrays the node publishes, against the Arrow C data interface: the two ABI structs are defined in the header rather than pulled from Arrow C++, which would be a heavy dependency for a producer that exports one array per output and never reads one. `image()` packs an `std/media/v1/Image` row, `float32()` and `string()` the primitive payloads. The payload bytes are **borrowed**, not copied — they only have to outlive the send, since dora IPC-encodes the array into shared memory inside `send_arrow_output` and then calls the release callbacks. Each node's release frees its children before itself, so releasing the root frees the tree. +- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected) it drains `take_state_change` and `take_frame` from the capture thread and publishes `image` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `send()` hands one `arrow_c::Exported` to `send_arrow_output`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds); nothing about a frame's shape or encoding travels as metadata any more, since the Image struct carries it. The loop never touches the SDK. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node. - **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue. - **`scripts/build.sh`** — Configure/compile/install in one shell-free-callable step. Referenced by both dataflows, the `Dockerfile` and the README. -- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why both dataflows build from the checkout. Type URNs name the element type (`std/core/v1/UInt8`, `.../Float32`, `.../String`) because every payload is a raw buffer described by its metadata, not an Arrow struct. +- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why both dataflows build from the checkout. `image` and `depth` are `std/media/v1/Image`; dora's compatibility graph has no edge from `std/core/v1/UInt8` to it, so a consumer declaring the struct rejects a producer declaring the buffer whatever it actually sends — the manifest has to move with the payload. - **`dataflow.yml`** — Example dora dataflow: the camera alone, no viewer and no Python dependency. Built from this checkout with `path:` + `build:` and `strict_types: true`. A consumer wires the node from a tag instead (see README) — dora 1.0 pins a source with the mutually exclusive `tag:`, `rev:` or `branch:`. -- **`dataflow_plot.yml`** — The same camera node plus an `opencv-plot` viewer on `image_bgr`. Split out because the viewer pulls dora-hub over `pip` and needs a display. dora 1.0 has no cross-dataflow wiring (an input's `source` must resolve to a node in the same descriptor), so the camera node is duplicated here rather than referenced, and the two files are run one at a time. -- **`.gitea/workflows/validate.yml`** — CI gate running `dora validate` on both dataflows on push and pull request. Needs no camera and no ZED SDK. +- **`dataflow_view.yml`** — The same camera node plus [dora-image-view](https://gitea.liria.cl/liria/dora-image-view) on `image`, `depth` and `camera_info`, fetched from a tag. Run it with `--uv`, which gives the viewer its own managed environment. The viewer's manifest is not read at validate time (only a `path:` node's is), so the file repeats its contract in `input_types:` to get the two checked against each other. dora 1.0 has no cross-dataflow wiring (an input's `source` must resolve to a node in the same descriptor), so the camera node is duplicated here rather than referenced, and the two files are run one at a time. +- **`tests/run.sh`** — Builds and runs the payload tests: `tests/arrow_test.cc` under `-fsanitize=address,undefined` (the release callbacks have to free the whole tree), and `tests/test_arrow.py`, which imports the exported arrays through pyarrow's C data interface and checks the values. `tests/arrow_export.cc` is the C shim the Python side calls. Covers `src/arrow` only, so it needs no camera, no ZED SDK and no dora. +- **`.gitea/workflows/validate.yml`** — CI gates on push and pull request: `dora validate` on both dataflows, and `tests/run.sh`. Neither needs a camera, the ZED SDK or CUDA. - **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`. ## Key APIs -- **dora C++ API** (``): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`. +- **dora C++ API** (``): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_arrow_output()`, `new_metadata()`. - **ZED SDK** (``): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`. -- Output data is sent as raw byte slices (`rust::Slice`) with typed metadata (shape, dtype, encoding). +- Output data is sent as Arrow arrays: `send_arrow_output` takes an `ArrowArray`/`ArrowSchema` pair by pointer and moves both out, so a payload is built once with `src/arrow` and handed over. ## Tools diff --git a/README.md b/README.md index 4380ebe..b577a48 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # dora-zed -Dora node for ZED stereo cameras. Publishes a BGR image and a depth map on every frame, plus camera intrinsics, a camera state, and an optional point cloud. +Dora node for ZED stereo cameras. On every tick it publishes a colour image and a depth +map as `std/media/v1/Image`, plus camera intrinsics, a camera state, and an optional +point cloud. A source for a vision pipeline: it takes a tick and produces frames. ## Usage @@ -9,100 +11,169 @@ Add to your dataflow: ```yaml nodes: - id: zed-camera - git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git - tag: v0.2.0 + git: https://gitea.liria.cl/liria/dora-zed.git + tag: v0.3.0 build: scripts/build.sh path: bin/dora_zed inputs: tick: dora/timer/millis/100 outputs: - - image_bgr + - image - depth - point_cloud - camera_info - camera_state ``` -### Runnable examples +The node is C++: building it needs the ZED SDK and CUDA on the machine that runs the +dataflow — see [Build](#build). An output left out of `outputs:` is simply not published, +so a dataflow that never turns the point cloud on can drop that line. -Two dataflows live here, and they are run one at a time — dora has no cross-dataflow -wiring, so each declares the camera node itself: +### Runnable dataflows | File | Contents | Run | |---|---|---| -| [`dataflow.yml`](dataflow.yml) | The camera alone | `dora run dataflow.yml` | -| [`dataflow_plot.yml`](dataflow_plot.yml) | The camera plus an `opencv-plot` viewer on `image_bgr` | `dora run dataflow_plot.yml` | +| [`dataflow.yml`](dataflow.yml) | The camera alone, no viewer and no Python dependency | `dora run dataflow.yml` | +| [`dataflow_view.yml`](dataflow_view.yml) | The camera plus [dora-image-view][view] | `dora run dataflow_view.yml --uv` | -The viewer is a separate file because it pulls [dora-hub](https://github.com/dora-rs/dora-hub) -and installs it with `pip`, and it needs a display. Neither is a cost the camera should -carry on a headless robot, so `dataflow.yml` has no Python dependency at all. +[view]: https://gitea.liria.cl/liria/dora-image-view -Both build the node from the checkout (`path:` + `build:`) rather than from a tag, which -is what lets `dora validate` pick up the manifest described below. +`dataflow_view.yml` is the worked example of consuming this node: a window opens with +the colour image and the depth map side by side, the camera model captioned across them. +Press `q` to quit. It needs a display, and `--uv` so the viewer gets its own managed +environment. -### Node manifest +dora has no cross-dataflow wiring, so both files declare the camera node themselves: run +one at a time. Both build it from the checkout, which is what lets `dora validate` check +them against [`dora-node.yml`](dora-node.yml) — the typed contract a consumer wires +against, and what [CI](.gitea/workflows/validate.yml) validates on every push. -[`dora-node.yml`](dora-node.yml) declares the node's inputs and outputs with type URNs. -No source package is shared between node repositories, so it is the only contract a -consumer has, and it is what makes a wiring mistake fail at build time instead of at -runtime: +## Inputs -``` -$ dora validate dataflow.yml - node "zed-camera": injected 6 contract type(s) from .../dora-node.yml -All type annotations OK. -``` - -Types are `std/core/v1/UInt8` for `image_bgr`, `std/core/v1/Float32` for `depth`, -`camera_info` and `point_cloud`, and `std/core/v1/String` for `camera_state`; every -payload is a raw buffer whose shape and dtype are in the metadata, so the URN names the -element type rather than a struct. The `tick` input is `std/core/v1/UInt64`, matching how -dora types timer inputs. - -`strict_types: true` in both dataflows makes a type warning a non-zero exit. -[`.gitea/workflows/validate.yml`](.gitea/workflows/validate.yml) validates both on every -push and pull request; it needs no camera and no ZED SDK, since validation never runs the -node. - -## Dependencies - -| Dependency | Version | Notes | -|---|---|---| -| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time | -| [CMake](https://cmake.org/download/) | 3.21+ | | -| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` | -| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK | -| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings | - -### Installation scripts - -| Platform | Script | +| Input | Description | |---|---| -| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` | -| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` | -| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` | +| `tick` | Publishes the newest frame captured since the last tick, or nothing. Wire to `dora/timer/millis/N`. | -> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release). +The tick does not drive the camera. Capture is free-running on its own thread, paced by +the SDK to `ZED_FPS`, and each frame overwrites whatever the previous one left +uncollected; a tick forwards the newest there is, and publishes nothing when none arrived +since the last one. So the tick sets the publish rate, never the capture rate: below +`ZED_FPS` it drops frames — the default 100 ms tick against the default 15 fps keeps two +out of every three — and above it, the ticks that find nothing publish nothing. +`dora/timer/millis/66` at 15 fps forwards every frame, each at most one capture period +old and none of them twice. -## Local build +## Outputs + +| Output | Payload | Metadata | +|---|---|---| +| `image` | `std/media/v1/Image` struct, `bgr8` | intrinsics, `frame_id`, timestamps | +| `depth` | `std/media/v1/Image` struct, `32FC1` | `units=mm`, intrinsics, `frame_id`, timestamps | +| `camera_info` | `float32` `[fx, fy, cx, cy]` | `distortion`, `frame_id`, timestamps | +| `point_cloud` | `float32` XYZRGBA, off by default | `shape`, `channels`, `units=mm`, intrinsics, timestamps | +| `camera_state` | `string` | `host_timestamp_ns` | + +`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`, +`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself, +so nothing about its shape travels as message metadata. A depth map is an image whose +pixels are distances, as ROS also treats it: `32FC1` is one float32 per pixel, `units` +says what those floats measure, and `0` means missing depth — the payload contains no +`inf` and no `nan`. `depth` is published on every frame `image` is, at the same size and +`frame_id`. + +`camera_info` is published **once per camera session** — with the first frame after the +camera opens, and again after a reconnection — since the intrinsics are fixed for a given +camera and resolution. Every frame also carries `fx`, `fy`, `cx` and `cy` in its +metadata, so a consumer that started after the camera did still ends up with them. + +Every output of a frame carries the same `frame_id` and both clocks as dora `timestamp` +metadata: `device_timestamp_ns`, when the camera captured the frame, and +`host_timestamp_ns`, when this node published it — both Unix-epoch nanoseconds. Two nodes +reading the same frame agree on the device clock; the rest of the dataflow shares the +host one. + +### Camera state + +`camera_state` lets a consumer tell a camera that has not delivered its first frame yet +from one that has been lost: + +| State | Meaning | +|---|---| +| `opening` | The camera has not delivered a usable frame yet in this run | +| `streaming` | Frames are being published | +| `recovering` | The camera was streaming and has stopped; the node is bringing it back | + +The camera runs on its own thread, since the SDK can block indefinitely inside `grab()` +and `open()` on a disconnected camera and would otherwise freeze the event loop. A camera +that delivers no usable frame for 5 seconds is recovered, reopening on a 1, 2, 5 then 10 +second backoff and escalating to `sl::Camera::reboot()` past the first attempt; an +all-black frame counts as a failed grab, since a camera whose link has dropped can keep +reporting success while handing back a uniform one. Uncollected frames are dropped, so a +consumer always gets the newest one. + +## Configuration + +Settings are environment variables, which a dataflow can carry per node, keeping them +with the wiring rather than in whatever shell starts it: + +```yaml + env: + ZED_RESOLUTION: 720 + ZED_FPS: 30 +``` + +A node's `env:` wins over the environment the dataflow was started in; variables it does +not set still pass through. + +| Variable | Default | Meaning | +|---|---|---| +| `ZED_RESOLUTION` | `1080` | `720`, `1080` or `2000` | +| `ZED_FPS` | `15` | Frame rate | +| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY` or `NONE`. An unrecognised value is a startup error, not a fallback. | +| `ZED_DEPTH_MIN_MM` | `200` | Near end of the depth range, in millimetres | +| `ZED_DEPTH_MAX_MM` | `600` | Far end of the depth range, in millimetres | +| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) | +| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) | +| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. | +| `ZED_FLIP` | `ON` | `ON` or `OFF` | +| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup | +| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud | +| `ZED_PC_MODE` | `FULL` | `FULL` is a `[H, W, 4]` grid; `HALF` is a stride-2 downsample. Only applies with `ZED_POINT_CLOUD` on. | + +The defaults match the reference camera driver the perception stack was tuned against, and +the resolved depth settings are logged at startup. + +The point cloud is off because it is expensive: roughly 14.7 MB per frame at 1080p, paid +in retrieval and in a copy whether or not anything consumes it. Consumers that need 3D +back-project `depth` with the published intrinsics instead. + +## Build ```bash -git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git -cd dora-zed scripts/build.sh ``` -Binary installed to `bin/dora_zed`. +The binary is installed to `bin/dora_zed`. The script is also the dataflow's `build:` +command, since dora 1.0 runs build commands directly rather than through a shell — `cd` +and `&&` are not available there. The first build clones dora-rs and compiles its C++ +bindings with Cargo, which takes a while; later builds are incremental, and +`DORA_ROOT_DIR` builds against a local dora checkout instead. -`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build -commands directly rather than through a shell, so `cd`, `&&` and command substitution are -not available there; keeping the sequence in one script is what makes the dataflow build -work. It configures a `Release` build, which is also what links the optimized dora -bindings. +**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x at +`/usr/local/zed` (it bundles OpenCV), the CUDA Toolkit it needs, and a stable Rust +toolchain for the dora bindings. -### Containerized build +| Platform | Install the toolchain with | +|---|---| +| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` | +| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` | +| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1`, plus the [ZED SDK][sdk] by hand | -[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed. +[sdk]: https://www.stereolabs.com/developers/release + +[`Dockerfile`](Dockerfile) carries the same toolchain on Ubuntu 24.04 — the newest release +Stereolabs ships a ZED SDK for — so the host needs none of it, only an NVIDIA driver and +the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html): ```bash docker build -t dora-zed-build . @@ -110,134 +181,18 @@ docker run --rm --gpus all --user "$(id -u):$(id -g)" \ -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build ``` -The container runs the same `scripts/build.sh`. +It runs the same `scripts/build.sh`. The named volume keeps the Cargo cache across runs +and `--user` keeps `build/` and `bin/` owned by you. x86-64 only: on Jetson, run +[`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) and build natively. -The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs. -`--user` keeps `build/` and `bin/` owned by you rather than root. +## Test -x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the -device and build natively — there is no ARM64 image here. +```bash +tests/run.sh +``` -Needs an NVIDIA driver and the -[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) -on the host, as the ZED SDK does anyway. The image sets -`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC -(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host -only when `video` is in the capability list. - -## Configuration - -All behavior is controlled via environment variables. - -### Camera - -| Variable | Default | Description | -|---|---|---| -| `ZED_RESOLUTION` | `1080` | `720`, `1080`, or `2000` | -| `ZED_FPS` | `15` | Frame rate | -| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY`, or `NONE`. An unrecognised value is a startup error, not a fallback. | -| `ZED_DEPTH_MIN_MM` | `200` | Minimum depth (mm) | -| `ZED_DEPTH_MAX_MM` | `600` | Maximum depth (mm) | -| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) | -| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) | -| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. | -| `ZED_FLIP` | `ON` | `ON` or `OFF` | -| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup | - -The resolved depth mode, both confidence thresholds and the depth range are logged at -startup. The defaults match the reference camera driver the perception stack was tuned -against, so depth quality on the robot matches what it expects. - -### Point cloud - -Off by default. With `ZED_POINT_CLOUD` unset the XYZRGBA measure is not retrieved at all -and no `point_cloud` output is published — at 1080p a full cloud is roughly 14.7 MB per -frame, and retrieving it costs capture time whether or not anything consumes it. -Consumers that need 3D back-project `depth` with the published intrinsics instead. - -| Variable | Default | Description | -|---|---|---| -| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud | -| `ZED_PC_MODE` | `FULL` | Only applies when `ZED_POINT_CLOUD` is on | - -| `ZED_PC_MODE` | Shape | Description | -|---|---|---| -| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 1080p) | -| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) | - -The cloud is copied out of the SDK buffer, since the capture thread and the event loop are -different threads. That copy is the main reason to leave `ZED_POINT_CLOUD` off unless a -consumer needs it. - -## Inputs - -| Input | Description | -|---|---| -| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. | - -## Outputs - -| Output | Type | Description | -|---|---|---| -| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, timestamps. | -| `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. | -| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud, only when `ZED_POINT_CLOUD` is on. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. | -| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. Published **once per camera session** — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution. | -| `camera_state` | `string` | UTF-8 state name, published on each change. Metadata: `dtype=string`, `encoding=utf8`, `host_timestamp_ns`. | - -### Camera state and recovery - -`camera_state` lets a downstream node tell a camera that has not delivered its first -frame yet from one that has been lost: - -| State | Meaning | -|---|---| -| `opening` | The camera has not delivered a usable frame yet in this run | -| `streaming` | Frames are being published | -| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back | - -The camera runs on its own thread. The ZED SDK can block for an unbounded time inside -`grab()` and `open()` when the camera is disconnected — measured blocking indefinitely on -a ZED-M even with `async_grab_camera_recovery` set — so driving it from the dora event -loop would freeze the node: it would stop reading events, leaving `Stop` unanswered and -the recovery logic itself unreachable. The event loop only reads the newest frame the -capture thread has already produced, and therefore never calls into the SDK. - -Reopening is paced by a backoff of 1, 2, 5 then 10 seconds, the last repeated. Frames that -are not collected before the next one arrives are dropped: a consumer always gets the -newest frame rather than a queue of stale ones. - -The node treats an all-black frame as a failed grab: a camera whose link has dropped does -not necessarily fail its grabs, and can keep reporting success while handing back a -uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30 -seconds if the SDK reports it is reconnecting the link itself, since taking the handle -away mid-reconnect aborts a recovery that was already under way. - -The first recovery attempt closes the handle. Past that the device itself is wedged, and -the node escalates to `sl::Camera::reboot()`, which resets it over USB without unplugging -it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED -Mini the node closes the handle and lets the backoff give the endpoint time to clear. -Every attempt and every escalation is logged. - -### Timestamps - -Every frame output — `image_bgr`, `depth`, `point_cloud` and `camera_info` — carries both -clocks, as dora `timestamp` metadata parameters. `camera_state` carries -`host_timestamp_ns` only, since it is not tied to a frame. - -| Key | Epoch | Unit | Source | -|---|---|---|---| -| `device_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE)` — when the camera captured the frame | -| `host_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `std::chrono::system_clock` on the host — when this node published the frame | - -Both are needed. The device timestamp is the capture instant and is the value two -nodes reading the same frame agree on; the host timestamp is the one the rest of the -dataflow and the session recorder share. On a Jetson the wall clock can step when NTP -synchronises after boot, so the device timestamp is recorded alongside the host one -rather than in place of it. - -The node calls `sl::setTimestampClock(SYSTEM_CLOCK)` before opening the camera, which -is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock. - -Every output for a frame also carries the same `frame_id`, an integer counter starting -at 0. +Unit tests over `src/arrow`, which packs every payload this node sends: each exported +Arrow array is round-tripped through pyarrow to check what a consumer reads back, and +built again under the sanitizers to check that releasing one frees it. They need neither a +camera, the ZED SDK nor dora, which is what lets CI run them; everything past the packing +is covered by running a dataflow against a real camera.