dora-zed/README.md

dora-zed

Dora node for ZED stereo cameras. On every tick it publishes a colour image and a depth map as std/media/v1/Image, plus camera intrinsics, a camera state, and an optional point cloud. A source for a vision pipeline: it takes a tick and produces frames.

Usage

Add to your dataflow:

nodes:
  - id: zed-camera
    git: https://gitea.liria.cl/liria/dora-zed.git
    tag: v0.3.0
    build: scripts/build.sh
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
    outputs:
      - image
      - depth
      - point_cloud
      - camera_info
      - camera_state

The node is C++: building it needs the ZED SDK and CUDA on the machine that runs the dataflow — see Build. An output left out of outputs: is simply not published, so a dataflow that never turns the point cloud on can drop that line.

Runnable dataflows

File	Contents	Run
dataflow.yml	The camera alone, no viewer and no Python dependency	dora run dataflow.yml
dataflow_view.yml	The camera plus dora-image-view	dora run dataflow_view.yml --uv

dataflow_view.yml is the worked example of consuming this node: a window opens with the colour image and the depth map side by side, the camera model captioned across them. Press q to quit. It needs a display, and --uv so the viewer gets its own managed environment.

dora has no cross-dataflow wiring, so both files declare the camera node themselves: run one at a time. Both build it from the checkout, which is what lets dora validate check them against dora-node.yml — the typed contract a consumer wires against, and what CI validates on every push.

Inputs

Input	Description
tick	Publishes the newest frame captured since the last tick, or nothing. Wire to dora/timer/millis/N.

The tick does not drive the camera. Capture is free-running on its own thread, paced by the SDK to ZED_FPS, and each frame overwrites whatever the previous one left uncollected; a tick forwards the newest there is, and publishes nothing when none arrived since the last one. So the tick sets the publish rate, never the capture rate: below ZED_FPS it drops frames — the default 100 ms tick against the default 15 fps keeps two out of every three — and above it, the ticks that find nothing publish nothing. dora/timer/millis/66 at 15 fps forwards every frame, each at most one capture period old and none of them twice.

Outputs

Output	Payload	Metadata
image	std/media/v1/Image struct, bgr8	intrinsics, frame_id, timestamps
depth	std/media/v1/Image struct, 32FC1	units=mm, intrinsics, frame_id, timestamps
camera_info	float32 [fx, fy, cx, cy]	distortion, frame_id, timestamps
point_cloud	float32 XYZRGBA, off by default	shape, channels, units=mm, intrinsics, timestamps
camera_state	string	host_timestamp_ns

image and depth are both std/media/v1/Image: an Arrow struct of width: UInt32, height: UInt32, encoding: Utf8 and data: LargeBinary. The payload describes itself, so nothing about its shape travels as message metadata. A depth map is an image whose pixels are distances, as ROS also treats it: 32FC1 is one float32 per pixel, units says what those floats measure, and 0 means missing depth — the payload contains no inf and no nan. depth is published on every frame image is, at the same size and frame_id.

camera_info is published once per camera session — with the first frame after the camera opens, and again after a reconnection — since the intrinsics are fixed for a given camera and resolution. Every frame also carries fx, fy, cx and cy in its metadata, so a consumer that started after the camera did still ends up with them.

Every output of a frame carries the same frame_id and both clocks as dora timestamp metadata: device_timestamp_ns, when the camera captured the frame, and host_timestamp_ns, when this node published it — both Unix-epoch nanoseconds. Two nodes reading the same frame agree on the device clock; the rest of the dataflow shares the host one.

Camera state

camera_state lets a consumer tell a camera that has not delivered its first frame yet from one that has been lost:

State	Meaning
opening	The camera has not delivered a usable frame yet in this run
streaming	Frames are being published
recovering	The camera was streaming and has stopped; the node is bringing it back

The camera runs on its own thread, since the SDK can block indefinitely inside grab() and open() on a disconnected camera and would otherwise freeze the event loop. A camera that delivers no usable frame for 5 seconds is recovered, reopening on a 1, 2, 5 then 10 second backoff and escalating to sl::Camera::reboot() past the first attempt; an all-black frame counts as a failed grab, since a camera whose link has dropped can keep reporting success while handing back a uniform one. Uncollected frames are dropped, so a consumer always gets the newest one.

Configuration

Settings are environment variables, which a dataflow can carry per node, keeping them with the wiring rather than in whatever shell starts it:

    env:
      ZED_RESOLUTION: 720
      ZED_FPS: 30

A node's env: wins over the environment the dataflow was started in; variables it does not set still pass through.

Variable	Default	Meaning
ZED_RESOLUTION	1080	720, 1080 or 2000
ZED_FPS	15	Frame rate
ZED_DEPTH_MODE	NEURAL	NEURAL, NEURAL_PLUS, NEURAL_LIGHT, PERFORMANCE, QUALITY or NONE. An unrecognised value is a startup error, not a fallback.
ZED_DEPTH_MIN_MM	200	Near end of the depth range, in millimetres
ZED_DEPTH_MAX_MM	600	Far end of the depth range, in millimetres
ZED_CONFIDENCE_THRESHOLD	60	Reject depth less certain than this (1–100, lower is stricter)
ZED_TEXTURE_CONFIDENCE_THRESHOLD	100	Reject depth on surfaces less textured than this (1–100)
ZED_DEPTH_FILL	false	Interpolate depth over holes. Overrides both confidence thresholds.
ZED_FLIP	ON	ON or OFF
ZED_WARMUP_FRAMES	30	Frames to discard on startup
ZED_POINT_CLOUD	false	Retrieve and publish the point cloud
ZED_PC_MODE	FULL	FULL is a [H, W, 4] grid; HALF is a stride-2 downsample. Only applies with ZED_POINT_CLOUD on.

The defaults match the reference camera driver the perception stack was tuned against, and the resolved depth settings are logged at startup.

The point cloud is off because it is expensive: roughly 14.7 MB per frame at 1080p, paid in retrieval and in a copy whether or not anything consumes it. Consumers that need 3D back-project depth with the published intrinsics instead.

Build

scripts/build.sh

The binary is installed to bin/dora_zed. The script is also the dataflow's build: command, since dora 1.0 runs build commands directly rather than through a shell — cd and && are not available there. The first build clones dora-rs and compiles its C++ bindings with Cargo, which takes a while; later builds are incremental, and DORA_ROOT_DIR builds against a local dora checkout instead.

Requirements: dora 1.0+ (verified with dora-cli 1.0.1), CMake 3.21+, ZED SDK 5.x at /usr/local/zed (it bundles OpenCV), the CUDA Toolkit it needs, and a stable Rust toolchain for the dora bindings.

Platform	Install the toolchain with
Ubuntu 24.04 x86_64	sudo scripts/install_linux_x64.sh
NVIDIA Jetson (ARM64)	sudo scripts/install_linux_arm.sh
Windows x64 (PowerShell admin)	scripts/install_windows.ps1, plus the ZED SDK by hand

Dockerfile carries the same toolchain on Ubuntu 24.04 — the newest release Stereolabs ships a ZED SDK for — so the host needs none of it, only an NVIDIA driver and the NVIDIA Container Toolkit:

docker build -t dora-zed-build .
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
    -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build

It runs the same scripts/build.sh. The named volume keeps the Cargo cache across runs and --user keeps build/ and bin/ owned by you. x86-64 only: on Jetson, run scripts/install_linux_arm.sh and build natively.

Test

tests/run.sh

Unit tests over src/arrow, which packs every payload this node sends: each exported Arrow array is round-tripped through pyarrow to check what a consumer reads back, and built again under the sanitizers to check that releasing one frees it. They need neither a camera, the ZED SDK nor dora, which is what lets CI run them; everything past the packing is covered by running a dataflow against a real camera.