dora-zed/README.md

dora-zed

Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.

Usage

Add to your dataflow:

nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    git_checkout: v0.1.0
    build: mkdir -p build && cd build && cmake .. && make install
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
    outputs:
      - image_bgr
      - point_cloud
      - camera_info
      - objects

See dataflow.yml for a complete runnable example with a camera display node.

Dependencies

Dependency	Version	Notes
CMake	3.21+
ZED SDK	5.x	Bundles OpenCV; default path /usr/local/zed
CUDA Toolkit	—	Required by ZED SDK
Rust toolchain	stable	Builds dora-rs C++ bindings
ONNX Runtime	1.20+	Required only for open_vocab engine

Installation scripts

Platform	Script
Ubuntu 24.04 x86_64	sudo scripts/install_linux_x64.sh
NVIDIA Jetson (ARM64)	sudo scripts/install_linux_arm.sh
Windows x64 (PowerShell admin)	scripts/install_windows.ps1

Jetson: No official ONNX Runtime GPU binary for ARM64. The native engine works without it. The open_vocab engine requires building from source.

Windows: Install ZED SDK manually from stereolabs.com, then pass the ONNX Runtime path: cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime

Local build

git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
mkdir -p build && cd build && cmake .. && make install

Binary installed to bin/dora_zed.

Configuration

All behavior is controlled via environment variables.

Camera

Variable	Default	Description
ZED_RESOLUTION	720	720, 1080, or 2000
ZED_FPS	15	Frame rate
ZED_DEPTH_MODE	NEURAL	PERFORMANCE, QUALITY, NEURAL, or NONE
ZED_DEPTH_MIN_MM	10	Minimum depth (mm)
ZED_DEPTH_MAX_MM	500	Maximum depth (mm)
ZED_DEPTH_FILL	false	Enable depth fill mode
ZED_FLIP	ON	ON or OFF
ZED_WARMUP_FRAMES	30	Frames to discard on startup
ZED_RETRY_INTERVAL_SECONDS	5	Reconnection interval (s)

Detection engine

Set ZED_ENGINE to select the detection backend:

ZED_ENGINE	Custom model?	Classes
native (default)	No — ZED SDK built-in TensorRT	Fixed: 7 categories
yolo_onnx (alias: onnx)	Yes — any single-input YOLO ONNX (v5/v8/v11)	Fixed labels from file or env var
yoloworld_onnx (alias: open_vocab)	Yes — YOLO-World two-input ONNX	Defined at runtime via CLIP

native engine

Uses ZED SDK's built-in TensorRT models — no model files to download or configure. ZED_DETECTION_MODEL selects the speed/accuracy variant, not a file path.

Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.

Variable	Default	Description
ZED_DETECTION_MODEL	MULTI_CLASS_BOX_MEDIUM	MULTI_CLASS_BOX_FAST, MULTI_CLASS_BOX_MEDIUM, or MULTI_CLASS_BOX_ACCURATE
ZED_DETECTION_CONFIDENCE	40	Confidence threshold (0–100)
ZED_DETECTION_TRACKING	false	Enable object tracking

yolo_onnx engine (closed-vocabulary)

For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).

Export a YOLOv11 model once:

uv run scripts/export_yolo.py --model n --output-dir models/

Then run:

ZED_ENGINE=yolo_onnx \
ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
ZED_ONNX_LABELS_FILE=models/coco80.txt \
dora start dataflow.yml

Required:

Variable	Description
ZED_ONNX_MODEL_PATH	Path to YOLO ONNX model
ZED_ONNX_LABELS or ZED_ONNX_LABELS_FILE	Class names (inline CSV or one-per-line file)

Optional:

Variable	Default	Description
ZED_ONNX_CONFIDENCE	0.25	Confidence threshold (0.0–1.0)
ZED_ONNX_NMS_THRESHOLD	0.45	NMS IoU threshold
ZED_ONNX_INPUT_SIZE	640	Model input size (px)
ZED_ONNX_OUTPUT_FORMAT	auto	Output layout: auto, nc_first/yolov8/yolov11, nc_last, nc_last_obj/yolov5. auto detects from tensor shape.

yoloworld_onnx engine (open-vocabulary)

For YOLO-World two-input ONNX models with a CLIP text encoder. Labels are defined at runtime — no recompilation required. Use ZED_ENGINE=open_vocab as a legacy alias.

Export models once:

uv run scripts/export_yoloworld.py --model s --num-classes 10 --output-dir models/
uv run scripts/export_clip_text_encoder.py --output-dir models/

Then run:

ZED_ENGINE=yoloworld_onnx \
ZED_ONNX_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,chair" \
dora start dataflow.yml

Required:

Variable	Description
ZED_ONNX_MODEL_PATH	Path to YOLO-World ONNX model
ZED_ONNX_TEXT_ENCODER_PATH	Path to clip_text_encoder.onnx
ZED_ONNX_CLIP_VOCAB	Path to clip_vocab.txt
ZED_ONNX_CLIP_MERGES	Path to clip_merges.txt
ZED_ONNX_LABELS	Comma-separated labels, e.g. person,car,chair

Optional:

Variable	Default	Description
ZED_ONNX_CONFIDENCE	0.25	Confidence threshold (0.0–1.0)
ZED_ONNX_NMS_THRESHOLD	0.45	NMS IoU threshold
ZED_ONNX_INPUT_SIZE	640	Model input size (px)

Labels can be updated at runtime without restarting — send a new comma-separated string to the update_vocabulary input.

Point cloud

ZED_PC_MODE	Shape	Description
FULL (default)	[H, W, 4]	Full resolution grid (~14.7 MB/frame at 720p)
HALF	[H/2, W/2, 4]	Stride-2 downsample (~4× smaller)
ROI	[N, 4]	Valid points within detected bounding boxes only

Inputs

Input	Description
tick	Triggers a camera grab and detection. Connect to dora/timer/millis/N.
update_vocabulary	(open_vocab only) UTF-8 comma-separated label string. Re-encodes CLIP embeddings immediately.

Outputs

Output	Type	Description
image_bgr	uint8 [H, W, 3]	BGR image. Metadata: shape, encoding=bgr8, intrinsics, timestamp_ns.
point_cloud	float32 [*, 4]	XYZRGBA point cloud. Shape depends on ZED_PC_MODE. Metadata: units=mm, intrinsics.
camera_info	float32 [4]	[fx, fy, cx, cy]. Metadata: distortion coefficients.
objects	float32 [N, 37]	Detected objects. See below.

objects layout

Each of the N rows contains 37 floats:

Index	Field	Description
0	label_id	Class index
1	confidence	Score (0–100)
2–4	x, y, z	3D center (mm)
5–12	bb2d	2D bounding box: 4 corners × (x, y) px
13–36	bb3d	3D bounding box: 8 corners × (x, y, z) mm

Metadata includes engine, label_names, image_width, image_height, and timestamp_ns. An empty array [0, 37] is sent when no objects are detected.