liria/dora-zed
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

docs: shorten the README and correct how the tick is described
validate / validate (push) Successful in 58m47s
Details
validate / test (push) Successful in 58m52s
Details
validate / test (pull_request) Successful in 58m49s
Details
validate / validate (pull_request) Successful in 58m48s
Details

Browse Source 
The README carried explanations that belong in AGENTS.md; it now follows
the shape of the sibling node repositories and is a quarter shorter.

The tick description was also wrong: it does not trigger a grab. Capture
is free-running at ZED_FPS and a tick forwards the newest frame there is,
so the tick sets the publish rate, and frames captured between two ticks
are dropped. The usage snippet listed four of the five outputs, which
read as an omission rather than the choice it is.
This commit is contained in:
cristhian
2026-09-15 10:52:12 -05:00
parent c1a7d70ba6
commit a4cd493330
2 changed files with 166 additions and 201 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+147 -192
View File
@@ -1,6 +1,8 @@
# dora-zed
Dora node for ZED stereo cameras. Publishes a BGR image and a depth map on every frame, plus camera intrinsics, a camera state, and an optional point cloud.
Dora node for ZED stereo cameras. On every tick it publishes a colour image and a depth
map as `std/media/v1/Image`, plus camera intrinsics, a camera state, and an optional
point cloud. A source for a vision pipeline: it takes a tick and produces frames.
## Usage
@@ -9,100 +11,169 @@ Add to your dataflow:
```yaml
nodes:
- id: zed-camera
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
tag: v0.2.0
git: https://gitea.liria.cl/liria/dora-zed.git
tag: v0.3.0
build: scripts/build.sh
path: bin/dora_zed
inputs:
tick: dora/timer/millis/100
outputs:
- image_bgr
- image
- depth
- point_cloud
- camera_info
- camera_state
```
### Runnable examples
The node is C++: building it needs the ZED SDK and CUDA on the machine that runs the
dataflow — see [Build](#build). An output left out of `outputs:` is simply not published,
so a dataflow that never turns the point cloud on can drop that line.
Two dataflows live here, and they are run one at a time — dora has no cross-dataflow
wiring, so each declares the camera node itself:
### Runnable dataflows
| File | Contents | Run |
|---|---|---|
| [`dataflow.yml`](dataflow.yml) | The camera alone | `dora run dataflow.yml` |
| [`dataflow_plot.yml`](dataflow_plot.yml) | The camera plus an `opencv-plot` viewer on `image_bgr` | `dora run dataflow_plot.yml` |
| [`dataflow.yml`](dataflow.yml) | The camera alone, no viewer and no Python dependency | `dora run dataflow.yml` |
| [`dataflow_view.yml`](dataflow_view.yml) | The camera plus [dora-image-view][view] | `dora run dataflow_view.yml --uv` |
The viewer is a separate file because it pulls [dora-hub](https://github.com/dora-rs/dora-hub)
and installs it with `pip`, and it needs a display. Neither is a cost the camera should
carry on a headless robot, so `dataflow.yml` has no Python dependency at all.
[view]: https://gitea.liria.cl/liria/dora-image-view
Both build the node from the checkout (`path:` + `build:`) rather than from a tag, which
is what lets `dora validate` pick up the manifest described below.
`dataflow_view.yml` is the worked example of consuming this node: a window opens with
the colour image and the depth map side by side, the camera model captioned across them.
Press `q` to quit. It needs a display, and `--uv` so the viewer gets its own managed
environment.
### Node manifest
dora has no cross-dataflow wiring, so both files declare the camera node themselves: run
one at a time. Both build it from the checkout, which is what lets `dora validate` check
them against [`dora-node.yml`](dora-node.yml) — the typed contract a consumer wires
against, and what [CI](.gitea/workflows/validate.yml) validates on every push.
[`dora-node.yml`](dora-node.yml) declares the node's inputs and outputs with type URNs.
No source package is shared between node repositories, so it is the only contract a
consumer has, and it is what makes a wiring mistake fail at build time instead of at
runtime:
## Inputs
```
$ dora validate dataflow.yml
node "zed-camera": injected 6 contract type(s) from .../dora-node.yml
All type annotations OK.
```
Types are `std/core/v1/UInt8` for `image_bgr`, `std/core/v1/Float32` for `depth`,
`camera_info` and `point_cloud`, and `std/core/v1/String` for `camera_state`; every
payload is a raw buffer whose shape and dtype are in the metadata, so the URN names the
element type rather than a struct. The `tick` input is `std/core/v1/UInt64`, matching how
dora types timer inputs.
`strict_types: true` in both dataflows makes a type warning a non-zero exit.
[`.gitea/workflows/validate.yml`](.gitea/workflows/validate.yml) validates both on every
push and pull request; it needs no camera and no ZED SDK, since validation never runs the
node.
## Dependencies
| Dependency | Version | Notes |
|---|---|---|
| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time |
| [CMake](https://cmake.org/download/) | 3.21+ | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
### Installation scripts
| Platform | Script |
| Input | Description |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
| `tick` | Publishes the newest frame captured since the last tick, or nothing. Wire to `dora/timer/millis/N`. |
> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release).
The tick does not drive the camera. Capture is free-running on its own thread, paced by
the SDK to `ZED_FPS`, and each frame overwrites whatever the previous one left
uncollected; a tick forwards the newest there is, and publishes nothing when none arrived
since the last one. So the tick sets the publish rate, never the capture rate: below
`ZED_FPS` it drops frames — the default 100 ms tick against the default 15 fps keeps two
out of every three — and above it, the ticks that find nothing publish nothing.
`dora/timer/millis/66` at 15 fps forwards every frame, each at most one capture period
old and none of them twice.
## Local build
## Outputs
| Output | Payload | Metadata |
|---|---|---|
| `image` | `std/media/v1/Image` struct, `bgr8` | intrinsics, `frame_id`, timestamps |
| `depth` | `std/media/v1/Image` struct, `32FC1` | `units=mm`, intrinsics, `frame_id`, timestamps |
| `camera_info` | `float32` `[fx, fy, cx, cy]` | `distortion`, `frame_id`, timestamps |
| `point_cloud` | `float32` XYZRGBA, off by default | `shape`, `channels`, `units=mm`, intrinsics, timestamps |
| `camera_state` | `string` | `host_timestamp_ns` |
`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`,
`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself,
so nothing about its shape travels as message metadata. A depth map is an image whose
pixels are distances, as ROS also treats it: `32FC1` is one float32 per pixel, `units`
says what those floats measure, and `0` means missing depth — the payload contains no
`inf` and no `nan`. `depth` is published on every frame `image` is, at the same size and
`frame_id`.
`camera_info` is published **once per camera session** — with the first frame after the
camera opens, and again after a reconnection — since the intrinsics are fixed for a given
camera and resolution. Every frame also carries `fx`, `fy`, `cx` and `cy` in its
metadata, so a consumer that started after the camera did still ends up with them.
Every output of a frame carries the same `frame_id` and both clocks as dora `timestamp`
metadata: `device_timestamp_ns`, when the camera captured the frame, and
`host_timestamp_ns`, when this node published it — both Unix-epoch nanoseconds. Two nodes
reading the same frame agree on the device clock; the rest of the dataflow shares the
host one.
### Camera state
`camera_state` lets a consumer tell a camera that has not delivered its first frame yet
from one that has been lost:
| State | Meaning |
|---|---|
| `opening` | The camera has not delivered a usable frame yet in this run |
| `streaming` | Frames are being published |
| `recovering` | The camera was streaming and has stopped; the node is bringing it back |
The camera runs on its own thread, since the SDK can block indefinitely inside `grab()`
and `open()` on a disconnected camera and would otherwise freeze the event loop. A camera
that delivers no usable frame for 5 seconds is recovered, reopening on a 1, 2, 5 then 10
second backoff and escalating to `sl::Camera::reboot()` past the first attempt; an
all-black frame counts as a failed grab, since a camera whose link has dropped can keep
reporting success while handing back a uniform one. Uncollected frames are dropped, so a
consumer always gets the newest one.
## Configuration
Settings are environment variables, which a dataflow can carry per node, keeping them
with the wiring rather than in whatever shell starts it:
```yaml
env:
ZED_RESOLUTION: 720
ZED_FPS: 30
```
A node's `env:` wins over the environment the dataflow was started in; variables it does
not set still pass through.
| Variable | Default | Meaning |
|---|---|---|
| `ZED_RESOLUTION` | `1080` | `720`, `1080` or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY` or `NONE`. An unrecognised value is a startup error, not a fallback. |
| `ZED_DEPTH_MIN_MM` | `200` | Near end of the depth range, in millimetres |
| `ZED_DEPTH_MAX_MM` | `600` | Far end of the depth range, in millimetres |
| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) |
| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) |
| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud |
| `ZED_PC_MODE` | `FULL` | `FULL` is a `[H, W, 4]` grid; `HALF` is a stride-2 downsample. Only applies with `ZED_POINT_CLOUD` on. |
The defaults match the reference camera driver the perception stack was tuned against, and
the resolved depth settings are logged at startup.
The point cloud is off because it is expensive: roughly 14.7 MB per frame at 1080p, paid
in retrieval and in a copy whether or not anything consumes it. Consumers that need 3D
back-project `depth` with the published intrinsics instead.
## Build
```bash
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
scripts/build.sh
```
Binary installed to `bin/dora_zed`.
The binary is installed to `bin/dora_zed`. The script is also the dataflow's `build:`
command, since dora 1.0 runs build commands directly rather than through a shell — `cd`
and `&&` are not available there. The first build clones dora-rs and compiles its C++
bindings with Cargo, which takes a while; later builds are incremental, and
`DORA_ROOT_DIR` builds against a local dora checkout instead.
`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build
commands directly rather than through a shell, so `cd`, `&&` and command substitution are
not available there; keeping the sequence in one script is what makes the dataflow build
work. It configures a `Release` build, which is also what links the optimized dora
bindings.
**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x at
`/usr/local/zed` (it bundles OpenCV), the CUDA Toolkit it needs, and a stable Rust
toolchain for the dora bindings.
### Containerized build
| Platform | Install the toolchain with |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1`, plus the [ZED SDK][sdk] by hand |
[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.
[sdk]: https://www.stereolabs.com/developers/release
[`Dockerfile`](Dockerfile) carries the same toolchain on Ubuntu 24.04 — the newest release
Stereolabs ships a ZED SDK for — so the host needs none of it, only an NVIDIA driver and
the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html):
```bash
docker build -t dora-zed-build .
@@ -110,134 +181,18 @@ docker run --rm --gpus all --user "$(id -u):$(id -g)" \
-v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
```
The container runs the same `scripts/build.sh`.
It runs the same `scripts/build.sh`. The named volume keeps the Cargo cache across runs
and `--user` keeps `build/` and `bin/` owned by you. x86-64 only: on Jetson, run
[`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) and build natively.
The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs.
`--user` keeps `build/` and `bin/` owned by you rather than root.
## Test
x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the
device and build natively — there is no ARM64 image here.
```bash
tests/run.sh
```
Needs an NVIDIA driver and the
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
on the host, as the ZED SDK does anyway. The image sets
`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC
(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host
only when `video` is in the capability list.
## Configuration
All behavior is controlled via environment variables.
### Camera
| Variable | Default | Description |
|---|---|---|
| `ZED_RESOLUTION` | `1080` | `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY`, or `NONE`. An unrecognised value is a startup error, not a fallback. |
| `ZED_DEPTH_MIN_MM` | `200` | Minimum depth (mm) |
| `ZED_DEPTH_MAX_MM` | `600` | Maximum depth (mm) |
| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) |
| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) |
| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
The resolved depth mode, both confidence thresholds and the depth range are logged at
startup. The defaults match the reference camera driver the perception stack was tuned
against, so depth quality on the robot matches what it expects.
### Point cloud
Off by default. With `ZED_POINT_CLOUD` unset the XYZRGBA measure is not retrieved at all
and no `point_cloud` output is published — at 1080p a full cloud is roughly 14.7 MB per
frame, and retrieving it costs capture time whether or not anything consumes it.
Consumers that need 3D back-project `depth` with the published intrinsics instead.
| Variable | Default | Description |
|---|---|---|
| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud |
| `ZED_PC_MODE` | `FULL` | Only applies when `ZED_POINT_CLOUD` is on |
| `ZED_PC_MODE` | Shape | Description |
|---|---|---|
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 1080p) |
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
The cloud is copied out of the SDK buffer, since the capture thread and the event loop are
different threads. That copy is the main reason to leave `ZED_POINT_CLOUD` off unless a
consumer needs it.
## Inputs
| Input | Description |
|---|---|
| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. |
## Outputs
| Output | Type | Description |
|---|---|---|
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, timestamps. |
| `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. |
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud, only when `ZED_POINT_CLOUD` is on. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. |
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. Published **once per camera session** — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution. |
| `camera_state` | `string` | UTF-8 state name, published on each change. Metadata: `dtype=string`, `encoding=utf8`, `host_timestamp_ns`. |
### Camera state and recovery
`camera_state` lets a downstream node tell a camera that has not delivered its first
frame yet from one that has been lost:
| State | Meaning |
|---|---|
| `opening` | The camera has not delivered a usable frame yet in this run |
| `streaming` | Frames are being published |
| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back |
The camera runs on its own thread. The ZED SDK can block for an unbounded time inside
`grab()` and `open()` when the camera is disconnected — measured blocking indefinitely on
a ZED-M even with `async_grab_camera_recovery` set — so driving it from the dora event
loop would freeze the node: it would stop reading events, leaving `Stop` unanswered and
the recovery logic itself unreachable. The event loop only reads the newest frame the
capture thread has already produced, and therefore never calls into the SDK.
Reopening is paced by a backoff of 1, 2, 5 then 10 seconds, the last repeated. Frames that
are not collected before the next one arrives are dropped: a consumer always gets the
newest frame rather than a queue of stale ones.
The node treats an all-black frame as a failed grab: a camera whose link has dropped does
not necessarily fail its grabs, and can keep reporting success while handing back a
uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30
seconds if the SDK reports it is reconnecting the link itself, since taking the handle
away mid-reconnect aborts a recovery that was already under way.
The first recovery attempt closes the handle. Past that the device itself is wedged, and
the node escalates to `sl::Camera::reboot()`, which resets it over USB without unplugging
it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED
Mini the node closes the handle and lets the backoff give the endpoint time to clear.
Every attempt and every escalation is logged.
### Timestamps
Every frame output — `image_bgr`, `depth`, `point_cloud` and `camera_info` — carries both
clocks, as dora `timestamp` metadata parameters. `camera_state` carries
`host_timestamp_ns` only, since it is not tied to a frame.
| Key | Epoch | Unit | Source |
|---|---|---|---|
| `device_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE)` — when the camera captured the frame |
| `host_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `std::chrono::system_clock` on the host — when this node published the frame |
Both are needed. The device timestamp is the capture instant and is the value two
nodes reading the same frame agree on; the host timestamp is the one the rest of the
dataflow and the session recorder share. On a Jetson the wall clock can step when NTP
synchronises after boot, so the device timestamp is recorded alongside the host one
rather than in place of it.
The node calls `sl::setTimestampClock(SYSTEM_CLOCK)` before opening the camera, which
is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock.
Every output for a frame also carries the same `frame_id`, an integer counter starting
at 0.
Unit tests over `src/arrow`, which packs every payload this node sends: each exported
Arrow array is round-tripped through pyarrow to check what a consumer reads back, and
built again under the sanitizers to check that releasing one frees it. They need neither a
camera, the ZED SDK nor dora, which is what lets CI run them; everything past the packing
is covered by running a dataflow against a real camera.