liria/dora-zed
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

add open-vocab YOLO-World engine with CLIP text encoder

Browse Source 
- Add OpenVocab engine: YOLO-World ONNX detection with runtime
  vocabulary via CLIP text embeddings (two-input model support)
- Add header-only C++ CLIP BPE tokenizer (no external deps)
- Add detect(cv::Mat) overload to OpenVocab for camera-free testing
- Add standalone test binary (test_open_vocab) using test/assets images
- Add export scripts for YOLO-World and CLIP text encoder (uv/PEP 723)
- Add installation scripts for Linux x64, ARM (Jetson), and Windows
- Update CMakeLists to use find_package(onnxruntime) instead of FetchContent
- Update README and AGENTS with env vars, setup workflow, and test usage
This commit is contained in:
cristhian aguilera
2026-03-28 10:41:11 -03:00
parent 6e6dc8ef27
commit fd6993e294
18 changed files with 1611 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+100 -9
View File
@@ -43,11 +43,39 @@ The binary is installed to `bin/dora_zed` relative to the repository root.
| Dependency | Minimum version | Notes |
|---|---|---|
| CMake | 3.21 | |
| ZED SDK | 4.x | Default path: `/usr/local/zed` |
| CUDA Toolkit | Required by ZED SDK | |
| OpenCV | 4.x | |
| Rust toolchain | stable | Used to build dora-rs C++ bindings |
| [CMake](https://cmake.org/download/) | 3.21 | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Default path: `/usr/local/zed`; bundles OpenCV |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | Required by ZED SDK | |
| [Rust toolchain](https://rustup.rs) | stable | Used to build dora-rs C++ bindings |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Must be installed before building; required only for `open_vocab` engine |
## Installation scripts
Helper scripts are in [`scripts/`](scripts/):
| Platform | Script | Notes |
|---|---|---|
| Ubuntu 24.04 x86_64 | [`scripts/install_linux_x64.sh`](scripts/install_linux_x64.sh) | Installs ZED SDK + ONNX Runtime |
| NVIDIA Jetson (ARM64) | [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) | Installs ZED SDK; auto-detects L4T version |
| Windows x64 | [`scripts/install_windows.ps1`](scripts/install_windows.ps1) | Installs ONNX Runtime; ZED SDK downloaded manually |
**Linux / Jetson:**
```bash
chmod +x scripts/install_linux_x64.sh # or install_linux_arm.sh
sudo scripts/install_linux_x64.sh
```
**Windows** (elevated PowerShell):
```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\scripts\install_windows.ps1
```
> **Jetson note:** No official ONNX Runtime GPU binary exists for ARM64. The `native` engine (ZED SDK built-in TensorRT) works without it. The `open_vocab` engine requires [building ONNX Runtime from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
> **Windows note:** ZED SDK must be downloaded and installed manually from [stereolabs.com/developers/release](https://www.stereolabs.com/developers/release). Pass the ONNX Runtime path to CMake: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
## Environment variables
@@ -62,22 +90,35 @@ The binary is installed to `bin/dora_zed` relative to the repository root.
| `ZED_FLIP` | `ON` | Image flip: `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Number of warmup frames to discard |
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Seconds between reconnection attempts |
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | Detection model: `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
| `ZED_DETECTION_CONFIDENCE` | `40` | Detection confidence threshold (0-100) |
| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking across frames (enables positional tracking) |
| `ZED_ENGINE` | `native` | Detection engine: `native` (ZED SDK) or `open_vocab` (YOLO-World via ONNX Runtime) |
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | (`native`) Detection model: `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
| `ZED_DETECTION_CONFIDENCE` | `40` | (`native`) Detection confidence threshold (0-100) |
| `ZED_DETECTION_TRACKING` | `false` | (`native`) Enable object tracking across frames |
| `ZED_ONNX_MODEL_PATH` | _(required)_ | (`open_vocab`) Path to YOLO-World ONNX model file |
| `ZED_ONNX_LABELS` | _(required)_ | (`open_vocab`) Comma-separated label names (e.g. `person,cup,chair`). Can be changed between ticks — the node re-encodes embeddings automatically |
| `ZED_ONNX_CONFIDENCE` | `0.25` | (`open_vocab`) Detection confidence threshold (0.0-1.0) |
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | (`open_vocab`) NMS IoU threshold |
| `ZED_ONNX_INPUT_SIZE` | `640` | (`open_vocab`) Model input image size in pixels |
| `ZED_ONNX_TEXT_ENCODER_PATH` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_text_encoder.onnx`. Required when the visual model has two inputs (`images` + `text_feats`) |
| `ZED_ONNX_CLIP_VOCAB` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_vocab.txt` BPE vocabulary file |
| `ZED_ONNX_CLIP_MERGES` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_merges.txt` BPE merge rules file |
## Outputs
### `image_bgr`
BGR image as raw bytes. Metadata includes `shape` (HxWx3), `encoding` (`bgr8`), camera intrinsics (`fx`, `fy`, `cx`, `cy`), `distortion`, `frame_id`, and `timestamp_ns`.
### `point_cloud`
XYZRGBA point cloud as float32 bytes. Metadata includes `shape` (HxWx4), `units` (`mm`), and camera intrinsics.
### `camera_info`
Four float32 values: `[fx, fy, cx, cy]`. Metadata includes `distortion` coefficients.
### `objects`
Detected objects as a flat float32 array with shape `[num_objects, 37]`. Each row contains 37 floats:
| Index | Field | Description |
@@ -88,4 +129,54 @@ Detected objects as a flat float32 array with shape `[num_objects, 37]`. Each ro
| 5-12 | `bb2d_*` | 2D bounding box: 4 corners × 2 pixel coords |
| 13-36 | `bb3d_*` | 3D bounding box: 8 corners × 3 world coords (mm) |
Metadata includes `frame_id` and `timestamp_ns` matching the corresponding `image_bgr`, plus `image_width` and `image_height` for interpreting 2D bounding box pixel coordinates. When no objects are detected, an empty array with shape `[0, 37]` is sent.
Metadata includes `frame_id` and `timestamp_ns` matching the corresponding `image_bgr`, `image_width` and `image_height` for interpreting 2D coordinates, `engine` (`native` or `open_vocab`), and `label_names` (comma-separated class names for the active engine). When no objects are detected, an empty array with shape `[0, 37]` is sent.
**`open_vocab` engine:** Uses a YOLO-World model exported with open-vocabulary support (two ONNX inputs: `images` and `text_feats`). Labels are set at runtime via `ZED_ONNX_LABELS` and re-encoded automatically on each tick when they change — no node restart required. Export the model and CLIP text encoder with the helper scripts:
```bash
# Export the visual model (N=10 class slots, can be changed)
uv run scripts/export_yoloworld.py --model s --num-classes 10 --output-dir models/
# Export the CLIP text encoder + vocab files (once, reused for all model sizes)
uv run scripts/export_clip_text_encoder.py --output-dir models/
# Run
ZED_ENGINE=open_vocab \
ZED_ONNX_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,bag,chair,laptop" \
dora start dataflow.yml
```
To verify the model works before connecting the camera, build and run the standalone C++ test binary (`bin/test_open_vocab`). It requires no live camera — images are loaded from disk:
```bash
cd build && cmake .. && make test_open_vocab
# Place test images:
cp street.jpg test/assets/yolo.png
cp custom.jpg test/assets/yoloworld.png
# Fixed-vocab YOLO (both images use the same model):
ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
ZED_ONNX_LABELS="person,car,bus,bicycle" \
bin/test_open_vocab
# Open-vocab YOLO-World (separate model for second image via ZED_ONNX_WORLD_MODEL_PATH):
ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
ZED_ONNX_WORLD_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,bus,bicycle" \
bin/test_open_vocab
# Override image paths:
bin/test_open_vocab /path/to/img1.jpg /path/to/img2.jpg
```
Annotated results are saved alongside the input images as `yolo_result.png` and `yoloworld_result.png`.
3D positions are computed by fusing the 2D detections with the ZED point cloud (median of valid depth samples within each bounding box). 3D bounding box corners are back-projected at the p25/p75 depth percentiles.