liria/dora-zed
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Publish typed Arrow payloads and wire dora-image-view #20

Merged
cristhian merged 2 commits from typed-image-payload into main 2026-09-15 15:13:34 +00:00
Conversation 2 Commits 2 Files Changed 15
+166 -201
2 changed files with 166 additions and 201 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit a4cd493330 - Show all commits
AGENTS.md
+19 -9
View File
@@ -4,10 +4,18 @@ This file provides guidance to AI coding assistants when working with code in th
## What this is
A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame and publishes it over the dora dataflow: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Camera management lives in `src/zed/` and the dora event loop in `src/main.cc`.
A dora-rs C++ node that interfaces with ZED stereo cameras. Capture is free-running on its own thread at `ZED_FPS`; on each tick the event loop publishes the newest frame it has produced, or nothing if none arrived since the last tick. What goes out: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Every payload is a typed Arrow array — the two frames as `std/media/v1/Image`, the rest as `std/core/v1/Float32` and `.../String` — which is what lets the node feed [dora-image-view](https://gitea.liria.cl/liria/dora-image-view) and anything else built against those types. Camera management lives in `src/zed/`, payload packing in `src/arrow/`, and the dora event loop in `src/main.cc`.
The node is a camera source only. It ran inference in-process through `v0.1.0`, under `src/engines/`; that was removed so the node has no inference dependency and builds on any platform the ZED SDK supports. The removed code remains reachable at tag `v0.1.0`.
## Test
```bash
tests/run.sh
```
Needs `g++` and `uv`; no camera, no ZED SDK, no CUDA.
## Build
```bash
@@ -28,8 +36,8 @@ For a build without installing the SDK stack on the host, see the `Dockerfile` (
```bash
dora up
dora start dataflow.yml # camera alone
dora start dataflow_plot.yml # camera plus the opencv-plot viewer
dora start dataflow.yml # camera alone
dora start dataflow_view.yml --uv # camera plus dora-image-view
```
The two dataflows both declare the camera node, so run one at a time.
@@ -42,20 +50,22 @@ Camera behavior is configured entirely through environment variables (see README
- **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: `open()` (one attempt, returns success), warmup, grab (image + depth + point cloud), calibration, device timestamp, blank-frame check, `close()` and `reset()` (close plus `sl::Camera::reboot()` where the model allows it). Exposes `handle()` for engine access.
- **`src/zed/capture.h/cc`** — `zed::Capture` and `zed::Frame`. Owns the camera on a dedicated thread: open, warmup, grab, blank/stall detection, backoff and escalation all live here, along with the packing of every payload into owned buffers. The newest frame is handed to the reader by swapping under a mutex (`take_frame`), which also recycles the previous frame's buffers so steady-state capture does not reallocate. **Nothing outside this file may call the ZED SDK** — it blocks unboundedly on a disconnected camera, which is what froze the node when the loop drove the camera directly.
- **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace. `parse_depth_mode` runs in `Config::from_env()`, not in the constructor, so an unrecognised `ZED_DEPTH_MODE` exits before the node joins the dataflow rather than falling back to a default.
- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()`, then sends `image_bgr` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `pack_depth` copies the depth map into a tightly packed `[H, W]` float buffer (sl::Mat rows can be padded) and clears `inf`/`nan` to `0`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds). Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension. The loop never touches the SDK: it drains `take_state_change` and `take_frame` from the capture thread and publishes. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node.
- **`src/arrow/array.h/cc`** — Builds the Arrow arrays the node publishes, against the Arrow C data interface: the two ABI structs are defined in the header rather than pulled from Arrow C++, which would be a heavy dependency for a producer that exports one array per output and never reads one. `image()` packs an `std/media/v1/Image` row, `float32()` and `string()` the primitive payloads. The payload bytes are **borrowed**, not copied — they only have to outlive the send, since dora IPC-encodes the array into shared memory inside `send_arrow_output` and then calls the release callbacks. Each node's release frees its children before itself, so releasing the root frees the tree.
- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected) it drains `take_state_change` and `take_frame` from the capture thread and publishes `image` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `send()` hands one `arrow_c::Exported` to `send_arrow_output`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds); nothing about a frame's shape or encoding travels as metadata any more, since the Image struct carries it. The loop never touches the SDK. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node.
- **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue.
- **`scripts/build.sh`** — Configure/compile/install in one shell-free-callable step. Referenced by both dataflows, the `Dockerfile` and the README.
- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why both dataflows build from the checkout. Type URNs name the element type (`std/core/v1/UInt8`, `.../Float32`, `.../String`) because every payload is a raw buffer described by its metadata, not an Arrow struct.
- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why both dataflows build from the checkout. `image` and `depth` are `std/media/v1/Image`; dora's compatibility graph has no edge from `std/core/v1/UInt8` to it, so a consumer declaring the struct rejects a producer declaring the buffer whatever it actually sends — the manifest has to move with the payload.
- **`dataflow.yml`** — Example dora dataflow: the camera alone, no viewer and no Python dependency. Built from this checkout with `path:` + `build:` and `strict_types: true`. A consumer wires the node from a tag instead (see README) — dora 1.0 pins a source with the mutually exclusive `tag:`, `rev:` or `branch:`.
- **`dataflow_plot.yml`** — The same camera node plus an `opencv-plot` viewer on `image_bgr`. Split out because the viewer pulls dora-hub over `pip` and needs a display. dora 1.0 has no cross-dataflow wiring (an input's `source` must resolve to a node in the same descriptor), so the camera node is duplicated here rather than referenced, and the two files are run one at a time.
- **`.gitea/workflows/validate.yml`** — CI gate running `dora validate` on both dataflows on push and pull request. Needs no camera and no ZED SDK.
- **`dataflow_view.yml`** — The same camera node plus [dora-image-view](https://gitea.liria.cl/liria/dora-image-view) on `image`, `depth` and `camera_info`, fetched from a tag. Run it with `--uv`, which gives the viewer its own managed environment. The viewer's manifest is not read at validate time (only a `path:` node's is), so the file repeats its contract in `input_types:` to get the two checked against each other. dora 1.0 has no cross-dataflow wiring (an input's `source` must resolve to a node in the same descriptor), so the camera node is duplicated here rather than referenced, and the two files are run one at a time.
- **`tests/run.sh`** — Builds and runs the payload tests: `tests/arrow_test.cc` under `-fsanitize=address,undefined` (the release callbacks have to free the whole tree), and `tests/test_arrow.py`, which imports the exported arrays through pyarrow's C data interface and checks the values. `tests/arrow_export.cc` is the C shim the Python side calls. Covers `src/arrow` only, so it needs no camera, no ZED SDK and no dora.
- **`.gitea/workflows/validate.yml`** — CI gates on push and pull request: `dora validate` on both dataflows, and `tests/run.sh`. Neither needs a camera, the ZED SDK or CUDA.
- **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`.
## Key APIs
- **dora C++ API** (`<dora-node-api.h>`): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`.
- **dora C++ API** (`<dora-node-api.h>`): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_arrow_output()`, `new_metadata()`.
- **ZED SDK** (`<sl/Camera.hpp>`): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`.
- Output data is sent as raw byte slices (`rust::Slice<const uint8_t>`) with typed metadata (shape, dtype, encoding).
- Output data is sent as Arrow arrays: `send_arrow_output` takes an `ArrowArray`/`ArrowSchema` pair by pointer and moves both out, so a payload is built once with `src/arrow` and handed over.
## Tools
README.md
+147 -192
View File
@@ -1,6 +1,8 @@
# dora-zed
Dora node for ZED stereo cameras. Publishes a BGR image and a depth map on every frame, plus camera intrinsics, a camera state, and an optional point cloud.
Dora node for ZED stereo cameras. On every tick it publishes a colour image and a depth
map as `std/media/v1/Image`, plus camera intrinsics, a camera state, and an optional
point cloud. A source for a vision pipeline: it takes a tick and produces frames.
## Usage
@@ -9,100 +11,169 @@ Add to your dataflow:
```yaml
nodes:
- id: zed-camera
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
tag: v0.2.0
git: https://gitea.liria.cl/liria/dora-zed.git
tag: v0.3.0
build: scripts/build.sh
path: bin/dora_zed
inputs:
tick: dora/timer/millis/100
outputs:
- image_bgr
- image
- depth
- point_cloud
- camera_info
- camera_state
```
### Runnable examples
The node is C++: building it needs the ZED SDK and CUDA on the machine that runs the
dataflow — see [Build](#build). An output left out of `outputs:` is simply not published,
so a dataflow that never turns the point cloud on can drop that line.
Two dataflows live here, and they are run one at a time — dora has no cross-dataflow
wiring, so each declares the camera node itself:
### Runnable dataflows
| File | Contents | Run |
|---|---|---|
| [`dataflow.yml`](dataflow.yml) | The camera alone | `dora run dataflow.yml` |
| [`dataflow_plot.yml`](dataflow_plot.yml) | The camera plus an `opencv-plot` viewer on `image_bgr` | `dora run dataflow_plot.yml` |
| [`dataflow.yml`](dataflow.yml) | The camera alone, no viewer and no Python dependency | `dora run dataflow.yml` |
| [`dataflow_view.yml`](dataflow_view.yml) | The camera plus [dora-image-view][view] | `dora run dataflow_view.yml --uv` |
The viewer is a separate file because it pulls [dora-hub](https://github.com/dora-rs/dora-hub)
and installs it with `pip`, and it needs a display. Neither is a cost the camera should
carry on a headless robot, so `dataflow.yml` has no Python dependency at all.
[view]: https://gitea.liria.cl/liria/dora-image-view
Both build the node from the checkout (`path:` + `build:`) rather than from a tag, which
is what lets `dora validate` pick up the manifest described below.
`dataflow_view.yml` is the worked example of consuming this node: a window opens with
the colour image and the depth map side by side, the camera model captioned across them.
Press `q` to quit. It needs a display, and `--uv` so the viewer gets its own managed
environment.
### Node manifest
dora has no cross-dataflow wiring, so both files declare the camera node themselves: run
one at a time. Both build it from the checkout, which is what lets `dora validate` check
them against [`dora-node.yml`](dora-node.yml) — the typed contract a consumer wires
against, and what [CI](.gitea/workflows/validate.yml) validates on every push.
[`dora-node.yml`](dora-node.yml) declares the node's inputs and outputs with type URNs.
No source package is shared between node repositories, so it is the only contract a
consumer has, and it is what makes a wiring mistake fail at build time instead of at
runtime:
## Inputs
```
$ dora validate dataflow.yml
node "zed-camera": injected 6 contract type(s) from .../dora-node.yml
All type annotations OK.
```
Types are `std/core/v1/UInt8` for `image_bgr`, `std/core/v1/Float32` for `depth`,
`camera_info` and `point_cloud`, and `std/core/v1/String` for `camera_state`; every
payload is a raw buffer whose shape and dtype are in the metadata, so the URN names the
element type rather than a struct. The `tick` input is `std/core/v1/UInt64`, matching how
dora types timer inputs.
`strict_types: true` in both dataflows makes a type warning a non-zero exit.
[`.gitea/workflows/validate.yml`](.gitea/workflows/validate.yml) validates both on every
push and pull request; it needs no camera and no ZED SDK, since validation never runs the
node.
## Dependencies
| Dependency | Version | Notes |
|---|---|---|
| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time |
| [CMake](https://cmake.org/download/) | 3.21+ | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
### Installation scripts
| Platform | Script |
| Input | Description |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
| `tick` | Publishes the newest frame captured since the last tick, or nothing. Wire to `dora/timer/millis/N`. |
> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release).
The tick does not drive the camera. Capture is free-running on its own thread, paced by
the SDK to `ZED_FPS`, and each frame overwrites whatever the previous one left
uncollected; a tick forwards the newest there is, and publishes nothing when none arrived
since the last one. So the tick sets the publish rate, never the capture rate: below
`ZED_FPS` it drops frames — the default 100 ms tick against the default 15 fps keeps two
out of every three — and above it, the ticks that find nothing publish nothing.
`dora/timer/millis/66` at 15 fps forwards every frame, each at most one capture period
old and none of them twice.
## Local build
## Outputs
| Output | Payload | Metadata |
|---|---|---|
| `image` | `std/media/v1/Image` struct, `bgr8` | intrinsics, `frame_id`, timestamps |
| `depth` | `std/media/v1/Image` struct, `32FC1` | `units=mm`, intrinsics, `frame_id`, timestamps |
| `camera_info` | `float32` `[fx, fy, cx, cy]` | `distortion`, `frame_id`, timestamps |
| `point_cloud` | `float32` XYZRGBA, off by default | `shape`, `channels`, `units=mm`, intrinsics, timestamps |
| `camera_state` | `string` | `host_timestamp_ns` |
`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`,
`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself,
so nothing about its shape travels as message metadata. A depth map is an image whose
pixels are distances, as ROS also treats it: `32FC1` is one float32 per pixel, `units`
says what those floats measure, and `0` means missing depth — the payload contains no
`inf` and no `nan`. `depth` is published on every frame `image` is, at the same size and
`frame_id`.
`camera_info` is published **once per camera session** — with the first frame after the
camera opens, and again after a reconnection — since the intrinsics are fixed for a given
camera and resolution. Every frame also carries `fx`, `fy`, `cx` and `cy` in its
metadata, so a consumer that started after the camera did still ends up with them.
Every output of a frame carries the same `frame_id` and both clocks as dora `timestamp`
metadata: `device_timestamp_ns`, when the camera captured the frame, and
`host_timestamp_ns`, when this node published it — both Unix-epoch nanoseconds. Two nodes
reading the same frame agree on the device clock; the rest of the dataflow shares the
host one.
### Camera state
`camera_state` lets a consumer tell a camera that has not delivered its first frame yet
from one that has been lost:
| State | Meaning |
|---|---|
| `opening` | The camera has not delivered a usable frame yet in this run |
| `streaming` | Frames are being published |
| `recovering` | The camera was streaming and has stopped; the node is bringing it back |
The camera runs on its own thread, since the SDK can block indefinitely inside `grab()`
and `open()` on a disconnected camera and would otherwise freeze the event loop. A camera
that delivers no usable frame for 5 seconds is recovered, reopening on a 1, 2, 5 then 10
second backoff and escalating to `sl::Camera::reboot()` past the first attempt; an
all-black frame counts as a failed grab, since a camera whose link has dropped can keep
reporting success while handing back a uniform one. Uncollected frames are dropped, so a
consumer always gets the newest one.
## Configuration
Settings are environment variables, which a dataflow can carry per node, keeping them
with the wiring rather than in whatever shell starts it:
```yaml
env:
ZED_RESOLUTION: 720
ZED_FPS: 30
```
A node's `env:` wins over the environment the dataflow was started in; variables it does
not set still pass through.
| Variable | Default | Meaning |
|---|---|---|
| `ZED_RESOLUTION` | `1080` | `720`, `1080` or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY` or `NONE`. An unrecognised value is a startup error, not a fallback. |
| `ZED_DEPTH_MIN_MM` | `200` | Near end of the depth range, in millimetres |
| `ZED_DEPTH_MAX_MM` | `600` | Far end of the depth range, in millimetres |
| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) |
| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) |
| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud |
| `ZED_PC_MODE` | `FULL` | `FULL` is a `[H, W, 4]` grid; `HALF` is a stride-2 downsample. Only applies with `ZED_POINT_CLOUD` on. |
The defaults match the reference camera driver the perception stack was tuned against, and
the resolved depth settings are logged at startup.
The point cloud is off because it is expensive: roughly 14.7 MB per frame at 1080p, paid
in retrieval and in a copy whether or not anything consumes it. Consumers that need 3D
back-project `depth` with the published intrinsics instead.
## Build
```bash
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
scripts/build.sh
```
Binary installed to `bin/dora_zed`.
The binary is installed to `bin/dora_zed`. The script is also the dataflow's `build:`
command, since dora 1.0 runs build commands directly rather than through a shell — `cd`
and `&&` are not available there. The first build clones dora-rs and compiles its C++
bindings with Cargo, which takes a while; later builds are incremental, and
`DORA_ROOT_DIR` builds against a local dora checkout instead.
`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build
commands directly rather than through a shell, so `cd`, `&&` and command substitution are
not available there; keeping the sequence in one script is what makes the dataflow build
work. It configures a `Release` build, which is also what links the optimized dora
bindings.
**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x at
`/usr/local/zed` (it bundles OpenCV), the CUDA Toolkit it needs, and a stable Rust
toolchain for the dora bindings.
### Containerized build
| Platform | Install the toolchain with |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1`, plus the [ZED SDK][sdk] by hand |
[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.
[sdk]: https://www.stereolabs.com/developers/release
[`Dockerfile`](Dockerfile) carries the same toolchain on Ubuntu 24.04 — the newest release
Stereolabs ships a ZED SDK for — so the host needs none of it, only an NVIDIA driver and
the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html):
```bash
docker build -t dora-zed-build .
@@ -110,134 +181,18 @@ docker run --rm --gpus all --user "$(id -u):$(id -g)" \
-v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
```
The container runs the same `scripts/build.sh`.
It runs the same `scripts/build.sh`. The named volume keeps the Cargo cache across runs
and `--user` keeps `build/` and `bin/` owned by you. x86-64 only: on Jetson, run
[`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) and build natively.
The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs.
`--user` keeps `build/` and `bin/` owned by you rather than root.
## Test
x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the
device and build natively — there is no ARM64 image here.
```bash
tests/run.sh
```
Needs an NVIDIA driver and the
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
on the host, as the ZED SDK does anyway. The image sets
`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC
(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host
only when `video` is in the capability list.
## Configuration
All behavior is controlled via environment variables.
### Camera
| Variable | Default | Description |
|---|---|---|
| `ZED_RESOLUTION` | `1080` | `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY`, or `NONE`. An unrecognised value is a startup error, not a fallback. |
| `ZED_DEPTH_MIN_MM` | `200` | Minimum depth (mm) |
| `ZED_DEPTH_MAX_MM` | `600` | Maximum depth (mm) |
| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) |
| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) |
| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
The resolved depth mode, both confidence thresholds and the depth range are logged at
startup. The defaults match the reference camera driver the perception stack was tuned
against, so depth quality on the robot matches what it expects.
### Point cloud
Off by default. With `ZED_POINT_CLOUD` unset the XYZRGBA measure is not retrieved at all
and no `point_cloud` output is published — at 1080p a full cloud is roughly 14.7 MB per
frame, and retrieving it costs capture time whether or not anything consumes it.
Consumers that need 3D back-project `depth` with the published intrinsics instead.
| Variable | Default | Description |
|---|---|---|
| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud |
| `ZED_PC_MODE` | `FULL` | Only applies when `ZED_POINT_CLOUD` is on |
| `ZED_PC_MODE` | Shape | Description |
|---|---|---|
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 1080p) |
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
The cloud is copied out of the SDK buffer, since the capture thread and the event loop are
different threads. That copy is the main reason to leave `ZED_POINT_CLOUD` off unless a
consumer needs it.
## Inputs
| Input | Description |
|---|---|
| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. |
## Outputs
| Output | Type | Description |
|---|---|---|
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, timestamps. |
| `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. |
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud, only when `ZED_POINT_CLOUD` is on. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. |
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. Published **once per camera session** — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution. |
| `camera_state` | `string` | UTF-8 state name, published on each change. Metadata: `dtype=string`, `encoding=utf8`, `host_timestamp_ns`. |
### Camera state and recovery
`camera_state` lets a downstream node tell a camera that has not delivered its first
frame yet from one that has been lost:
| State | Meaning |
|---|---|
| `opening` | The camera has not delivered a usable frame yet in this run |
| `streaming` | Frames are being published |
| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back |
The camera runs on its own thread. The ZED SDK can block for an unbounded time inside
`grab()` and `open()` when the camera is disconnected — measured blocking indefinitely on
a ZED-M even with `async_grab_camera_recovery` set — so driving it from the dora event
loop would freeze the node: it would stop reading events, leaving `Stop` unanswered and
the recovery logic itself unreachable. The event loop only reads the newest frame the
capture thread has already produced, and therefore never calls into the SDK.
Reopening is paced by a backoff of 1, 2, 5 then 10 seconds, the last repeated. Frames that
are not collected before the next one arrives are dropped: a consumer always gets the
newest frame rather than a queue of stale ones.
The node treats an all-black frame as a failed grab: a camera whose link has dropped does
not necessarily fail its grabs, and can keep reporting success while handing back a
uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30
seconds if the SDK reports it is reconnecting the link itself, since taking the handle
away mid-reconnect aborts a recovery that was already under way.
The first recovery attempt closes the handle. Past that the device itself is wedged, and
the node escalates to `sl::Camera::reboot()`, which resets it over USB without unplugging
it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED
Mini the node closes the handle and lets the backoff give the endpoint time to clear.
Every attempt and every escalation is logged.
### Timestamps
Every frame output — `image_bgr`, `depth`, `point_cloud` and `camera_info` — carries both
clocks, as dora `timestamp` metadata parameters. `camera_state` carries
`host_timestamp_ns` only, since it is not tied to a frame.
| Key | Epoch | Unit | Source |
|---|---|---|---|
| `device_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE)` — when the camera captured the frame |
| `host_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `std::chrono::system_clock` on the host — when this node published the frame |
Both are needed. The device timestamp is the capture instant and is the value two
nodes reading the same frame agree on; the host timestamp is the one the rest of the
dataflow and the session recorder share. On a Jetson the wall clock can step when NTP
synchronises after boot, so the device timestamp is recorded alongside the host one
rather than in place of it.
The node calls `sl::setTimestampClock(SYSTEM_CLOCK)` before opening the camera, which
is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock.
Every output for a frame also carries the same `frame_id`, an integer counter starting
at 0.
Unit tests over `src/arrow`, which packs every payload this node sends: each exported
Arrow array is round-tripped through pyarrow to check what a consumer reads back, and
built again under the sanitizers to check that releasing one frees it. They need neither a
camera, the ZED SDK nor dora, which is what lets CI run them; everything past the packing
is covered by running a dataflow against a real camera.