dora-zed/README.md

dora-zed

Dora C++ node for ZED stereo cameras. Captures frames and publishes BGR images, XYZRGBA point clouds, camera intrinsics, and 3D object detections.

Usage via git: in a dataflow

nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    build: mkdir -p build && cd build && cmake .. && make install
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
    outputs:
      - image_bgr
      - point_cloud
      - camera_info
      - objects

Then run:

dora build dataflow.yml
dora up
dora start dataflow.yml

Local build

git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
mkdir -p build && cd build
cmake ..
make install

The binary is installed to bin/dora_zed relative to the repository root.

Dependencies

Dependency	Minimum version	Notes
CMake	3.21
ZED SDK	5.x	Default path: /usr/local/zed; bundles OpenCV
CUDA Toolkit	Required by ZED SDK
Rust toolchain	stable	Used to build dora-rs C++ bindings
ONNX Runtime	1.20+	Must be installed before building; required only for open_vocab engine

Installation scripts

Helper scripts are in scripts/:

Platform	Script	Notes
Ubuntu 24.04 x86_64	scripts/install_linux_x64.sh	Installs ZED SDK + ONNX Runtime
NVIDIA Jetson (ARM64)	scripts/install_linux_arm.sh	Installs ZED SDK; auto-detects L4T version
Windows x64	scripts/install_windows.ps1	Installs ONNX Runtime; ZED SDK downloaded manually

Linux / Jetson:

chmod +x scripts/install_linux_x64.sh   # or install_linux_arm.sh
sudo scripts/install_linux_x64.sh

Windows (elevated PowerShell):

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\scripts\install_windows.ps1

Jetson note: No official ONNX Runtime GPU binary exists for ARM64. The native engine (ZED SDK built-in TensorRT) works without it. The open_vocab engine requires building ONNX Runtime from source.

Windows note: ZED SDK must be downloaded and installed manually from stereolabs.com/developers/release. Pass the ONNX Runtime path to CMake: cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime

Environment variables

Variable	Default	Description
ZED_RESOLUTION	720	Camera resolution: 720, 1080, or 2000
ZED_FPS	15	Camera frame rate
ZED_DEPTH_MODE	NEURAL	Depth mode: PERFORMANCE, QUALITY, NEURAL, or NONE
ZED_DEPTH_MIN_MM	10	Minimum depth distance in millimeters
ZED_DEPTH_MAX_MM	500	Maximum depth distance in millimeters
ZED_DEPTH_FILL	false	Enable depth fill mode
ZED_FLIP	ON	Image flip: ON or OFF
ZED_WARMUP_FRAMES	30	Number of warmup frames to discard
ZED_RETRY_INTERVAL_SECONDS	5	Seconds between reconnection attempts
ZED_ENGINE	native	Detection engine: native (ZED SDK) or open_vocab (YOLO-World via ONNX Runtime)
ZED_DETECTION_MODEL	MULTI_CLASS_BOX_MEDIUM	(native) Detection model: MULTI_CLASS_BOX_FAST, MULTI_CLASS_BOX_MEDIUM, or MULTI_CLASS_BOX_ACCURATE
ZED_DETECTION_CONFIDENCE	40	(native) Detection confidence threshold (0-100)
ZED_DETECTION_TRACKING	false	(native) Enable object tracking across frames
ZED_ONNX_MODEL_PATH	(required)	(open_vocab) Path to YOLO-World ONNX model file
ZED_ONNX_LABELS	(required)	(open_vocab) Comma-separated label names (e.g. person,cup,chair). Can be changed between ticks — the node re-encodes embeddings automatically
ZED_ONNX_CONFIDENCE	0.25	(open_vocab) Detection confidence threshold (0.0-1.0)
ZED_ONNX_NMS_THRESHOLD	0.45	(open_vocab) NMS IoU threshold
ZED_ONNX_INPUT_SIZE	640	(open_vocab) Model input image size in pixels
ZED_ONNX_TEXT_ENCODER_PATH	(required for open-vocab)	(open_vocab) Path to clip_text_encoder.onnx. Required when the visual model has two inputs (images + text_feats)
ZED_ONNX_CLIP_VOCAB	(required for open-vocab)	(open_vocab) Path to clip_vocab.txt BPE vocabulary file
ZED_ONNX_CLIP_MERGES	(required for open-vocab)	(open_vocab) Path to clip_merges.txt BPE merge rules file

Inputs

Input	Description
tick	Required. Triggers a camera grab and detection on each message. Connect to dora/timer/millis/N.
update_vocabulary	Optional (open_vocab engine only). Payload: raw UTF-8 bytes of a comma-separated label string (e.g. "person,cup,chair"). Immediately re-encodes CLIP embeddings; the next tick uses the new labels.

Outputs

image_bgr

BGR image as raw bytes. Metadata includes shape (HxWx3), encoding (bgr8), camera intrinsics (fx, fy, cx, cy), distortion, frame_id, and timestamp_ns.

point_cloud

XYZRGBA point cloud as float32 bytes. Metadata includes shape (HxWx4), units (mm), and camera intrinsics.

camera_info

Four float32 values: [fx, fy, cx, cy]. Metadata includes distortion coefficients.

objects

Detected objects as a flat float32 array with shape [num_objects, 37]. Each row contains 37 floats:

Index	Field	Description
0	label_id	Object class (0=PERSON, 1=VEHICLE, 2=BAG, 3=ANIMAL, 4=ELECTRONICS, 5=FRUIT_VEGETABLE, 6=SPORT)
1	confidence	Detection confidence (0-100)
2-4	x, y, z	3D center position in mm
5-12	bb2d_*	2D bounding box: 4 corners × 2 pixel coords
13-36	bb3d_*	3D bounding box: 8 corners × 3 world coords (mm)

Metadata includes frame_id and timestamp_ns matching the corresponding image_bgr, image_width and image_height for interpreting 2D coordinates, engine (native or open_vocab), and label_names (comma-separated class names for the active engine). When no objects are detected, an empty array with shape [0, 37] is sent.

open_vocab engine: Uses a YOLO-World model exported with open-vocabulary support (two ONNX inputs: images and text_feats). Labels are set at runtime via ZED_ONNX_LABELS and re-encoded automatically on each tick when they change — no node restart required. Export the model and CLIP text encoder with the helper scripts:

# Export the visual model (N=10 class slots, can be changed)
uv run scripts/export_yoloworld.py --model s --num-classes 10 --output-dir models/

# Export the CLIP text encoder + vocab files (once, reused for all model sizes)
uv run scripts/export_clip_text_encoder.py --output-dir models/

# Run
ZED_ENGINE=open_vocab \
ZED_ONNX_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,bag,chair,laptop" \
dora start dataflow.yml

To verify the model works before connecting the camera, build and run the standalone C++ test binary (bin/test_open_vocab). It requires no live camera — images are loaded from disk:

cd build && cmake .. && make test_open_vocab

# Place test images:
cp street.jpg test/assets/yolo.png
cp custom.jpg test/assets/yoloworld.png

# Fixed-vocab YOLO (both images use the same model):
ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
ZED_ONNX_LABELS="person,car,bus,bicycle" \
bin/test_open_vocab

# Open-vocab YOLO-World (separate model for second image via ZED_ONNX_WORLD_MODEL_PATH):
ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
ZED_ONNX_WORLD_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,bus,bicycle" \
bin/test_open_vocab

# Override image paths:
bin/test_open_vocab /path/to/img1.jpg /path/to/img2.jpg

Annotated results are saved alongside the input images as yolo_result.png and yoloworld_result.png.

3D positions are computed by fusing the 2D detections with the ZED point cloud (median of valid depth samples within each bounding box). 3D bounding box corners are back-projected at the p25/p75 depth percentiles.