dora-zed/README.md

dora-zed

Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.

Usage

Add to your dataflow:

nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    git_checkout: v0.1.0
    build: mkdir -p build && cd build && cmake .. && make install
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
    outputs:
      - image_bgr
      - point_cloud
      - camera_info
      - objects

See dataflow.yml for a complete runnable example with a camera display node.

Dependencies

Dependency	Version	Notes
CMake	3.21+
ZED SDK	5.x	Bundles OpenCV; default path /usr/local/zed
CUDA Toolkit	—	Required by ZED SDK
Rust toolchain	stable	Builds dora-rs C++ bindings
ONNX Runtime	1.20+	Required at configure time; used at runtime only by yolo_onnx

Installation scripts

Platform	Script
Ubuntu 24.04 x86_64	sudo scripts/install_linux_x64.sh
NVIDIA Jetson (ARM64)	sudo scripts/install_linux_arm.sh
Windows x64 (PowerShell admin)	scripts/install_windows.ps1

Jetson: No official ONNX Runtime GPU binary for ARM64. It is still needed to configure the build; only the yolo_onnx engine uses it at runtime. Requires building from source.

Windows: Install ZED SDK manually from stereolabs.com, then pass the ONNX Runtime path: cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime

Local build

git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
mkdir -p build && cd build && cmake .. && make install

Binary installed to bin/dora_zed.

Containerized build

Dockerfile provides the full toolchain — CUDA, ZED SDK, ONNX Runtime and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.

docker build -t dora-zed-build .
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
    -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build

The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs. --user keeps build/ and bin/ owned by you rather than root.

x86-64 only. On Jetson, run scripts/install_linux_arm.sh on the device and build natively — there is no ARM64 image here.

Needs an NVIDIA driver and the NVIDIA Container Toolkit on the host, as the ZED SDK does anyway. The image sets NVIDIA_DRIVER_CAPABILITIES=compute,utility,video: libsl_zed.so links against NVDEC (libnvcuvid) and NVENC (libnvidia-encode), which the container runtime mounts from the host only when video is in the capability list.

Configuration

All behavior is controlled via environment variables.

Camera

Variable	Default	Description
ZED_RESOLUTION	720	720, 1080, or 2000
ZED_FPS	15	Frame rate
ZED_DEPTH_MODE	NEURAL	PERFORMANCE, QUALITY, NEURAL, or NONE
ZED_DEPTH_MIN_MM	10	Minimum depth (mm)
ZED_DEPTH_MAX_MM	500	Maximum depth (mm)
ZED_DEPTH_FILL	false	Enable depth fill mode
ZED_FLIP	ON	ON or OFF
ZED_WARMUP_FRAMES	30	Frames to discard on startup
ZED_RETRY_INTERVAL_SECONDS	5	Reconnection interval (s)

Detection engine

Set ZED_ENGINE to select the detection backend:

ZED_ENGINE	Custom model?	Classes
native (default)	No — ZED SDK built-in TensorRT	Fixed: 7 categories
yolo_onnx (alias: onnx)	Yes — any single-input YOLO ONNX (v5/v8/v11)	Fixed labels from file or env var

native engine

Uses ZED SDK's built-in TensorRT models — no model files to download or configure. ZED_DETECTION_MODEL selects the speed/accuracy variant, not a file path.

Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.

Variable	Default	Description
ZED_DETECTION_MODEL	MULTI_CLASS_BOX_MEDIUM	MULTI_CLASS_BOX_FAST, MULTI_CLASS_BOX_MEDIUM, or MULTI_CLASS_BOX_ACCURATE
ZED_DETECTION_CONFIDENCE	40	Confidence threshold (0–100)
ZED_DETECTION_TRACKING	false	Enable object tracking

yolo_onnx engine (closed-vocabulary)

For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).

Export a YOLOv11 model once:

uv run scripts/export_yolo.py --model n --output-dir models/

Then run:

ZED_ENGINE=yolo_onnx \
ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
ZED_ONNX_LABELS_FILE=models/coco80.txt \
dora start dataflow.yml

Required:

Variable	Description
ZED_ONNX_MODEL_PATH	Path to YOLO ONNX model
ZED_ONNX_LABELS or ZED_ONNX_LABELS_FILE	Class names (inline CSV or one-per-line file)

Optional:

Variable	Default	Description
ZED_ONNX_CONFIDENCE	0.25	Confidence threshold (0.0–1.0)
ZED_ONNX_NMS_THRESHOLD	0.45	NMS IoU threshold
ZED_ONNX_INPUT_SIZE	640	Model input size (px)
ZED_ONNX_OUTPUT_FORMAT	auto	Output layout: auto, nc_first/yolov8/yolov11, nc_last, nc_last_obj/yolov5. auto detects from tensor shape.

Point cloud

ZED_PC_MODE	Shape	Description
FULL (default)	[H, W, 4]	Full resolution grid (~14.7 MB/frame at 720p)
HALF	[H/2, W/2, 4]	Stride-2 downsample (~4× smaller)
ROI	[N, 4]	Valid points within detected bounding boxes only

Inputs

Input	Description
tick	Triggers a camera grab and detection. Connect to dora/timer/millis/N.

Outputs

Output	Type	Description
image_bgr	uint8 [H, W, 3]	BGR image. Metadata: shape, encoding=bgr8, intrinsics, timestamp_ns.
point_cloud	float32 [*, 4]	XYZRGBA point cloud. Shape depends on ZED_PC_MODE. Metadata: units=mm, intrinsics.
camera_info	float32 [4]	[fx, fy, cx, cy]. Metadata: distortion coefficients.
objects	float32 [N, 37]	Detected objects. See below.

objects layout

Each of the N rows contains 37 floats:

Index	Field	Description
0	label_id	Class index
1	confidence	Score (0–100)
2–4	x, y, z	3D center (mm)
5–12	bb2d	2D bounding box: 4 corners × (x, y) px
13–36	bb3d	3D bounding box: 8 corners × (x, y, z) mm

Metadata includes engine, label_names, image_width, image_height, and timestamp_ns. An empty array [0, 37] is sent when no objects are detected.