dora-zed/README.md

dora-zed

Dora node for ZED stereo cameras. Publishes a BGR image and a depth map on every frame, plus camera intrinsics, a camera state, and an optional point cloud.

Usage

Add to your dataflow:

nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    tag: v0.2.0
    build: scripts/build.sh
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
    outputs:
      - image_bgr
      - depth
      - point_cloud
      - camera_info
      - camera_state

See dataflow.yml for a complete runnable example with a camera display node. That file builds the node from the checkout (path: + build:) rather than from a tag, which is what lets dora validate pick up the manifest described below.

Node manifest

dora-node.yml declares the node's inputs and outputs with type URNs. No source package is shared between node repositories, so it is the only contract a consumer has, and it is what makes a wiring mistake fail at build time instead of at runtime:

$ dora validate dataflow.yml
  node "zed-camera": injected 6 contract type(s) from .../dora-node.yml
All type annotations OK.

Types are std/core/v1/UInt8 for image_bgr, std/core/v1/Float32 for depth, camera_info and point_cloud, and std/core/v1/String for camera_state; every payload is a raw buffer whose shape and dtype are in the metadata, so the URN names the element type rather than a struct. The tick input is std/core/v1/UInt64, matching how dora types timer inputs.

strict_types: true in dataflow.yml makes a type warning a non-zero exit. .gitea/workflows/validate.yml runs dora validate dataflow.yml on every push and pull request; it needs no camera and no ZED SDK, since validation never runs the node.

Dependencies

Dependency	Version	Notes
dora-rs	1.0+	Verified against dora-cli 1.0.1; the C++ bindings are built from source at configure time
CMake	3.21+
ZED SDK	5.x	Bundles OpenCV; default path /usr/local/zed
CUDA Toolkit	—	Required by ZED SDK
Rust toolchain	stable	Builds dora-rs C++ bindings

Installation scripts

Platform	Script
Ubuntu 24.04 x86_64	sudo scripts/install_linux_x64.sh
NVIDIA Jetson (ARM64)	sudo scripts/install_linux_arm.sh
Windows x64 (PowerShell admin)	scripts/install_windows.ps1

Windows: Install the ZED SDK manually from stereolabs.com.

Local build

git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
scripts/build.sh

Binary installed to bin/dora_zed.

scripts/build.sh is also the build: command in the dataflow. dora 1.0 runs build commands directly rather than through a shell, so cd, && and command substitution are not available there; keeping the sequence in one script is what makes the dataflow build work. It configures a Release build, which is also what links the optimized dora bindings.

Containerized build

Dockerfile provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.

docker build -t dora-zed-build .
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
    -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build

The container runs the same scripts/build.sh.

The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs. --user keeps build/ and bin/ owned by you rather than root.

x86-64 only. On Jetson, run scripts/install_linux_arm.sh on the device and build natively — there is no ARM64 image here.

Needs an NVIDIA driver and the NVIDIA Container Toolkit on the host, as the ZED SDK does anyway. The image sets NVIDIA_DRIVER_CAPABILITIES=compute,utility,video: libsl_zed.so links against NVDEC (libnvcuvid) and NVENC (libnvidia-encode), which the container runtime mounts from the host only when video is in the capability list.

Configuration

All behavior is controlled via environment variables.

Camera

Variable	Default	Description
ZED_RESOLUTION	1080	720, 1080, or 2000
ZED_FPS	15	Frame rate
ZED_DEPTH_MODE	NEURAL	NEURAL, NEURAL_PLUS, NEURAL_LIGHT, PERFORMANCE, QUALITY, or NONE. An unrecognised value is a startup error, not a fallback.
ZED_DEPTH_MIN_MM	200	Minimum depth (mm)
ZED_DEPTH_MAX_MM	600	Maximum depth (mm)
ZED_CONFIDENCE_THRESHOLD	60	Reject depth less certain than this (1–100, lower is stricter)
ZED_TEXTURE_CONFIDENCE_THRESHOLD	100	Reject depth on surfaces less textured than this (1–100)
ZED_DEPTH_FILL	false	Interpolate depth over holes. Overrides both confidence thresholds.
ZED_FLIP	ON	ON or OFF
ZED_WARMUP_FRAMES	30	Frames to discard on startup

The resolved depth mode, both confidence thresholds and the depth range are logged at startup. The defaults match the reference camera driver the perception stack was tuned against, so depth quality on the robot matches what it expects.

Point cloud

Off by default. With ZED_POINT_CLOUD unset the XYZRGBA measure is not retrieved at all and no point_cloud output is published — at 1080p a full cloud is roughly 14.7 MB per frame, and retrieving it costs capture time whether or not anything consumes it. Consumers that need 3D back-project depth with the published intrinsics instead.

Variable	Default	Description
ZED_POINT_CLOUD	false	Retrieve and publish the point cloud
ZED_PC_MODE	FULL	Only applies when ZED_POINT_CLOUD is on

ZED_PC_MODE	Shape	Description
FULL (default)	[H, W, 4]	Full resolution grid (~14.7 MB/frame at 1080p)
HALF	[H/2, W/2, 4]	Stride-2 downsample (~4× smaller)

The cloud is copied out of the SDK buffer, since the capture thread and the event loop are different threads. That copy is the main reason to leave ZED_POINT_CLOUD off unless a consumer needs it.

Inputs

Input	Description
tick	Triggers a camera grab. Connect to dora/timer/millis/N.

Outputs

Output	Type	Description
image_bgr	uint8 [H, W, 3]	BGR image. Metadata: shape, encoding=bgr8, intrinsics, timestamps.
depth	float32 [H, W]	Per-pixel depth in millimetres, published on every frame image_bgr is. Metadata: dtype, layout=HW, shape, units=mm, intrinsics, timestamps. H and W match image_bgr for the same frame_id. Missing depth is 0 — the payload contains no inf and no nan. Holes are not interpolated unless ZED_DEPTH_FILL is set.
point_cloud	float32 [*, 4]	XYZRGBA point cloud, only when ZED_POINT_CLOUD is on. Shape depends on ZED_PC_MODE. Metadata: units=mm, intrinsics, timestamps.
camera_info	float32 [4]	[fx, fy, cx, cy]. Metadata: distortion coefficients, timestamps. Published once per camera session — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution.
camera_state	string	UTF-8 state name, published on each change. Metadata: dtype=string, encoding=utf8, host_timestamp_ns.

Camera state and recovery

camera_state lets a downstream node tell a camera that has not delivered its first frame yet from one that has been lost:

State	Meaning
opening	The camera has not delivered a usable frame yet in this run
streaming	Frames are being published
recovering	The camera was streaming and has stopped; the node is trying to bring it back

The camera runs on its own thread. The ZED SDK can block for an unbounded time inside grab() and open() when the camera is disconnected — measured blocking indefinitely on a ZED-M even with async_grab_camera_recovery set — so driving it from the dora event loop would freeze the node: it would stop reading events, leaving Stop unanswered and the recovery logic itself unreachable. The event loop only reads the newest frame the capture thread has already produced, and therefore never calls into the SDK.

Reopening is paced by a backoff of 1, 2, 5 then 10 seconds, the last repeated. Frames that are not collected before the next one arrives are dropped: a consumer always gets the newest frame rather than a queue of stale ones.

The node treats an all-black frame as a failed grab: a camera whose link has dropped does not necessarily fail its grabs, and can keep reporting success while handing back a uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30 seconds if the SDK reports it is reconnecting the link itself, since taking the handle away mid-reconnect aborts a recovery that was already under way.

The first recovery attempt closes the handle. Past that the device itself is wedged, and the node escalates to sl::Camera::reboot(), which resets it over USB without unplugging it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED Mini the node closes the handle and lets the backoff give the endpoint time to clear. Every attempt and every escalation is logged.

Timestamps

Every frame output — image_bgr, depth, point_cloud and camera_info — carries both clocks, as dora timestamp metadata parameters. camera_state carries host_timestamp_ns only, since it is not tied to a frame.

Key	Epoch	Unit	Source
device_timestamp_ns	Unix epoch (UTC)	nanoseconds	sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE) — when the camera captured the frame
host_timestamp_ns	Unix epoch (UTC)	nanoseconds	std::chrono::system_clock on the host — when this node published the frame

Both are needed. The device timestamp is the capture instant and is the value two nodes reading the same frame agree on; the host timestamp is the one the rest of the dataflow and the session recorder share. On a Jetson the wall clock can step when NTP synchronises after boot, so the device timestamp is recorded alongside the host one rather than in place of it.

The node calls sl::setTimestampClock(SYSTEM_CLOCK) before opening the camera, which is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock.

Every output for a frame also carries the same frame_id, an integer counter starting at 0.