liria/dora-zed
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

refactor open_vocab into yolo_onnx and yoloworld_onnx engines

Browse Source 
Split the monolithic open_vocab engine into two focused engines:
yolo_onnx (closed-vocabulary, any single-input YOLO ONNX) and
yoloworld_onnx (open-vocabulary YOLO-World with CLIP encoder).
Shared NMS/letterbox/fuse helpers extracted to yolo_helpers.h.
Add export_yolo.py script and separate test binaries for each engine.
This commit is contained in:
cristhian aguilera
2026-03-30 04:44:25 -03:00
parent 748b0c3665
commit 44eec29b83
14 changed files with 1467 additions and 825 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+164 -149
View File
@@ -1,13 +1,16 @@
# dora-zed
Dora C++ node for ZED stereo cameras. Captures frames and publishes BGR images, XYZRGBA point clouds, camera intrinsics, and 3D object detections.
Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.
## Usage via `git:` in a dataflow
## Usage
Add to your dataflow:
```yaml
nodes:
- id: zed-camera
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
git_checkout: v0.1.0
build: mkdir -p build && cd build && cmake .. && make install
path: bin/dora_zed
inputs:
@@ -19,180 +22,192 @@ nodes:
- objects
```
Then run:
See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.
```bash
dora build dataflow.yml
dora up
dora start dataflow.yml
```
## Dependencies
| Dependency | Version | Notes |
|---|---|---|
| [CMake](https://cmake.org/download/) | 3.21+ | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required only for `open_vocab` engine |
### Installation scripts
| Platform | Script |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
> **Jetson:** No official ONNX Runtime GPU binary for ARM64. The `native` engine works without it. The `open_vocab` engine requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
>
> **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
## Local build
```bash
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
mkdir -p build && cd build
cmake ..
make install
mkdir -p build && cd build && cmake .. && make install
```
The binary is installed to `bin/dora_zed` relative to the repository root.
Binary installed to `bin/dora_zed`.
## Dependencies
## Configuration
| Dependency | Minimum version | Notes |
|---|---|---|
| [CMake](https://cmake.org/download/) | 3.21 | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Default path: `/usr/local/zed`; bundles OpenCV |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | Required by ZED SDK | |
| [Rust toolchain](https://rustup.rs) | stable | Used to build dora-rs C++ bindings |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Must be installed before building; required only for `open_vocab` engine |
All behavior is controlled via environment variables.
## Installation scripts
Helper scripts are in [`scripts/`](scripts/):
| Platform | Script | Notes |
|---|---|---|
| Ubuntu 24.04 x86_64 | [`scripts/install_linux_x64.sh`](scripts/install_linux_x64.sh) | Installs ZED SDK + ONNX Runtime |
| NVIDIA Jetson (ARM64) | [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) | Installs ZED SDK; auto-detects L4T version |
| Windows x64 | [`scripts/install_windows.ps1`](scripts/install_windows.ps1) | Installs ONNX Runtime; ZED SDK downloaded manually |
**Linux / Jetson:**
```bash
chmod +x scripts/install_linux_x64.sh # or install_linux_arm.sh
sudo scripts/install_linux_x64.sh
```
**Windows** (elevated PowerShell):
```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\scripts\install_windows.ps1
```
> **Jetson note:** No official ONNX Runtime GPU binary exists for ARM64. The `native` engine (ZED SDK built-in TensorRT) works without it. The `open_vocab` engine requires [building ONNX Runtime from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
> **Windows note:** ZED SDK must be downloaded and installed manually from [stereolabs.com/developers/release](https://www.stereolabs.com/developers/release). Pass the ONNX Runtime path to CMake: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
## Environment variables
### Camera
| Variable | Default | Description |
|---|---|---|
| `ZED_RESOLUTION` | `720` | Camera resolution: `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Camera frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | Depth mode: `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth distance in millimeters |
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth distance in millimeters |
| `ZED_RESOLUTION` | `720` | `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth (mm) |
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth (mm) |
| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
| `ZED_FLIP` | `ON` | Image flip: `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Number of warmup frames to discard |
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Seconds between reconnection attempts |
| `ZED_ENGINE` | `native` | Detection engine: `native` (ZED SDK) or `open_vocab` (YOLO-World via ONNX Runtime) |
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | (`native`) Detection model: `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
| `ZED_DETECTION_CONFIDENCE` | `40` | (`native`) Detection confidence threshold (0-100) |
| `ZED_DETECTION_TRACKING` | `false` | (`native`) Enable object tracking across frames |
| `ZED_ONNX_MODEL_PATH` | _(required)_ | (`open_vocab`) Path to YOLO-World ONNX model file |
| `ZED_ONNX_LABELS` | _(required)_ | (`open_vocab`) Comma-separated label names (e.g. `person,cup,chair`). Can be changed between ticks — the node re-encodes embeddings automatically |
| `ZED_ONNX_CONFIDENCE` | `0.25` | (`open_vocab`) Detection confidence threshold (0.0-1.0) |
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | (`open_vocab`) NMS IoU threshold |
| `ZED_ONNX_INPUT_SIZE` | `640` | (`open_vocab`) Model input image size in pixels |
| `ZED_ONNX_TEXT_ENCODER_PATH` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_text_encoder.onnx`. Required when the visual model has two inputs (`images` + `text_feats`) |
| `ZED_ONNX_CLIP_VOCAB` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_vocab.txt` BPE vocabulary file |
| `ZED_ONNX_CLIP_MERGES` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_merges.txt` BPE merge rules file |
| `ZED_PC_MODE` | `FULL` | Point cloud output mode: `FULL` (full HxW grid), `HALF` (stride-2 downsample, ~4× smaller), or `ROI` (valid points within detected bounding boxes only, flat list) |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |
### Detection engine
Set `ZED_ENGINE` to select the detection backend:
| `ZED_ENGINE` | Custom model? | Classes |
|---|---|---|
| `native` (default) | No — ZED SDK built-in TensorRT | Fixed: 7 categories |
| `yolo_onnx` (alias: `onnx`) | Yes — any single-input YOLO ONNX (v5/v8/v11) | Fixed labels from file or env var |
| `yoloworld_onnx` (alias: `open_vocab`) | Yes — YOLO-World two-input ONNX | Defined at runtime via CLIP |
#### `native` engine
Uses ZED SDK's built-in TensorRT models — no model files to download or configure. `ZED_DETECTION_MODEL` selects the speed/accuracy variant, not a file path.
Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.
| Variable | Default | Description |
|---|---|---|
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
| `ZED_DETECTION_CONFIDENCE` | `40` | Confidence threshold (0–100) |
| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking |
#### `yolo_onnx` engine (closed-vocabulary)
For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).
Export a YOLOv11 model once:
```bash
uv run scripts/export_yolo.py --model n --output-dir models/
```
Then run:
```bash
ZED_ENGINE=yolo_onnx \
ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
ZED_ONNX_LABELS_FILE=models/coco80.txt \
dora start dataflow.yml
```
**Required:**
| Variable | Description |
|---|---|
| `ZED_ONNX_MODEL_PATH` | Path to YOLO ONNX model |
| `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE` | Class names (inline CSV or one-per-line file) |
**Optional:**
| Variable | Default | Description |
|---|---|---|
| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
| `ZED_ONNX_OUTPUT_FORMAT` | `auto` | Output layout: `auto`, `nc_first`/`yolov8`/`yolov11`, `nc_last`, `nc_last_obj`/`yolov5`. `auto` detects from tensor shape. |
#### `yoloworld_onnx` engine (open-vocabulary)
For YOLO-World two-input ONNX models with a CLIP text encoder. Labels are defined at runtime — no recompilation required. Use `ZED_ENGINE=open_vocab` as a legacy alias.
Export models once:
```bash
uv run scripts/export_yoloworld.py --model s --num-classes 10 --output-dir models/
uv run scripts/export_clip_text_encoder.py --output-dir models/
```
Then run:
```bash
ZED_ENGINE=yoloworld_onnx \
ZED_ONNX_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,chair" \
dora start dataflow.yml
```
**Required:**
| Variable | Description |
|---|---|
| `ZED_ONNX_MODEL_PATH` | Path to YOLO-World ONNX model |
| `ZED_ONNX_TEXT_ENCODER_PATH` | Path to `clip_text_encoder.onnx` |
| `ZED_ONNX_CLIP_VOCAB` | Path to `clip_vocab.txt` |
| `ZED_ONNX_CLIP_MERGES` | Path to `clip_merges.txt` |
| `ZED_ONNX_LABELS` | Comma-separated labels, e.g. `person,car,chair` |
**Optional:**
| Variable | Default | Description |
|---|---|---|
| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
Labels can be updated at runtime without restarting — send a new comma-separated string to the `update_vocabulary` input.
### Point cloud
| `ZED_PC_MODE` | Shape | Description |
|---|---|---|
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) |
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
| `ROI` | `[N, 4]` | Valid points within detected bounding boxes only |
## Inputs
| Input | Description |
|---|---|
| `tick` | Required. Triggers a camera grab and detection on each message. Connect to `dora/timer/millis/N`. |
| `update_vocabulary` | Optional (`open_vocab` engine only). Payload: raw UTF-8 bytes of a comma-separated label string (e.g. `"person,cup,chair"`). Immediately re-encodes CLIP embeddings; the next `tick` uses the new labels. |
| `tick` | Triggers a camera grab and detection. Connect to `dora/timer/millis/N`. |
| `update_vocabulary` | (`open_vocab` only) UTF-8 comma-separated label string. Re-encodes CLIP embeddings immediately. |
## Outputs
### `image_bgr`
| Output | Type | Description |
|---|---|---|
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, `timestamp_ns`. |
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics. |
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients. |
| `objects` | `float32 [N, 37]` | Detected objects. See below. |
BGR image as raw bytes. Metadata includes `shape` (HxWx3), `encoding` (`bgr8`), camera intrinsics (`fx`, `fy`, `cx`, `cy`), `distortion`, `frame_id`, and `timestamp_ns`.
### `objects` layout
### `point_cloud`
XYZRGBA point cloud as float32 bytes. Controlled by `ZED_PC_MODE`:
| Mode | Shape | Layout | Description |
|---|---|---|---|
| `FULL` | `[H, W, 4]` | `HWC` | Full resolution grid (default). ~14.7 MB/frame at 720p |
| `HALF` | `[H/2, W/2, 4]` | `HWC` | Stride-2 downsample (~4× smaller) |
| `ROI` | `[N, 4]` | `NC` | Valid points within detected object bounding boxes only. `N` varies by scene. Metadata includes `mode=roi` |
Metadata includes `units` (`mm`) and camera intrinsics.
### `camera_info`
Four float32 values: `[fx, fy, cx, cy]`. Metadata includes `distortion` coefficients.
### `objects`
Detected objects as a flat float32 array with shape `[num_objects, 37]`. Each row contains 37 floats:
Each of the N rows contains 37 floats:
| Index | Field | Description |
|---|---|---|
| 0 | `label_id` | Object class (0=PERSON, 1=VEHICLE, 2=BAG, 3=ANIMAL, 4=ELECTRONICS, 5=FRUIT_VEGETABLE, 6=SPORT) |
| 1 | `confidence` | Detection confidence (0-100) |
| 2-4 | `x, y, z` | 3D center position in mm |
| 5-12 | `bb2d_*` | 2D bounding box: 4 corners × 2 pixel coords |
| 13-36 | `bb3d_*` | 3D bounding box: 8 corners × 3 world coords (mm) |
| 0 | `label_id` | Class index |
| 1 | `confidence` | Score (0–100) |
| 2–4 | `x, y, z` | 3D center (mm) |
| 5–12 | `bb2d` | 2D bounding box: 4 corners × (x, y) px |
| 13–36 | `bb3d` | 3D bounding box: 8 corners × (x, y, z) mm |
Metadata includes `frame_id` and `timestamp_ns` matching the corresponding `image_bgr`, `image_width` and `image_height` for interpreting 2D coordinates, `engine` (`native` or `open_vocab`), and `label_names` (comma-separated class names for the active engine). When no objects are detected, an empty array with shape `[0, 37]` is sent.
**`open_vocab` engine:** Uses a YOLO-World model exported with open-vocabulary support (two ONNX inputs: `images` and `text_feats`). Labels are set at runtime via `ZED_ONNX_LABELS` and re-encoded automatically on each tick when they change — no node restart required. Export the model and CLIP text encoder with the helper scripts:
```bash
# Export the visual model (N=10 class slots, can be changed)
uv run scripts/export_yoloworld.py --model s --num-classes 10 --output-dir models/
# Export the CLIP text encoder + vocab files (once, reused for all model sizes)
uv run scripts/export_clip_text_encoder.py --output-dir models/
# Run
ZED_ENGINE=open_vocab \
ZED_ONNX_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,bag,chair,laptop" \
dora start dataflow.yml
```
To verify the model works before connecting the camera, build and run the standalone C++ test binary (`bin/test_open_vocab`). It requires no live camera — images are loaded from disk:
```bash
cd build && cmake .. && make test_open_vocab
# Place test images:
cp street.jpg test/assets/yolo.png
cp custom.jpg test/assets/yoloworld.png
# Fixed-vocab YOLO (both images use the same model):
ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
ZED_ONNX_LABELS="person,car,bus,bicycle" \
bin/test_open_vocab
# Open-vocab YOLO-World (separate model for second image via ZED_ONNX_WORLD_MODEL_PATH):
ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
ZED_ONNX_WORLD_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,bus,bicycle" \
bin/test_open_vocab
# Override image paths:
bin/test_open_vocab /path/to/img1.jpg /path/to/img2.jpg
```
Annotated results are saved alongside the input images as `yolo_result.png` and `yoloworld_result.png`.
3D positions are computed by fusing the 2D detections with the ZED point cloud (median of valid depth samples within each bounding box). 3D bounding box corners are back-projected at the p25/p75 depth percentiles.
Metadata includes `engine`, `label_names`, `image_width`, `image_height`, and `timestamp_ns`. An empty array `[0, 37]` is sent when no objects are detected.