liria/dora-zed
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity
Files
2d10a685274cb291faaae85d9e920be61d92b827
dora-zed/README.md
T
cristhian aguilera 2d10a68527 add update_vocabulary dora input for runtime label changes 
The open_vocab engine now accepts an update_vocabulary input event
carrying a UTF-8 comma-separated label string. Labels are re-encoded
immediately via the CLIP text encoder; the next tick uses the new
vocabulary. ZedNative ignores the input via a no-op base class method.
2026-03-28 11:03:18 -03:00

190 lines
8.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# dora-zed
Dora C++ node for ZED stereo cameras. Captures frames and publishes BGR images, XYZRGBA point clouds, camera intrinsics, and 3D object detections.
## Usage via `git:` in a dataflow
```yaml
nodes:
- id: zed-camera
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
build: mkdir -p build && cd build && cmake .. && make install
path: bin/dora_zed
inputs:
tick: dora/timer/millis/100
outputs:
- image_bgr
- point_cloud
- camera_info
- objects
```
Then run:
```bash
dora build dataflow.yml
dora up
dora start dataflow.yml
```
## Local build
```bash
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
mkdir -p build && cd build
cmake ..
make install
```
The binary is installed to `bin/dora_zed` relative to the repository root.
## Dependencies
| Dependency | Minimum version | Notes |
|---|---|---|
| [CMake](https://cmake.org/download/) | 3.21 | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Default path: `/usr/local/zed`; bundles OpenCV |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | Required by ZED SDK | |
| [Rust toolchain](https://rustup.rs) | stable | Used to build dora-rs C++ bindings |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Must be installed before building; required only for `open_vocab` engine |
## Installation scripts
Helper scripts are in [`scripts/`](scripts/):
| Platform | Script | Notes |
|---|---|---|
| Ubuntu 24.04 x86_64 | [`scripts/install_linux_x64.sh`](scripts/install_linux_x64.sh) | Installs ZED SDK + ONNX Runtime |
| NVIDIA Jetson (ARM64) | [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) | Installs ZED SDK; auto-detects L4T version |
| Windows x64 | [`scripts/install_windows.ps1`](scripts/install_windows.ps1) | Installs ONNX Runtime; ZED SDK downloaded manually |
**Linux / Jetson:**
```bash
chmod +x scripts/install_linux_x64.sh # or install_linux_arm.sh
sudo scripts/install_linux_x64.sh
```
**Windows** (elevated PowerShell):
```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\scripts\install_windows.ps1
```
> **Jetson note:** No official ONNX Runtime GPU binary exists for ARM64. The `native` engine (ZED SDK built-in TensorRT) works without it. The `open_vocab` engine requires [building ONNX Runtime from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
> **Windows note:** ZED SDK must be downloaded and installed manually from [stereolabs.com/developers/release](https://www.stereolabs.com/developers/release). Pass the ONNX Runtime path to CMake: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
## Environment variables
| Variable | Default | Description |
|---|---|---|
| `ZED_RESOLUTION` | `720` | Camera resolution: `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Camera frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | Depth mode: `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth distance in millimeters |
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth distance in millimeters |
| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
| `ZED_FLIP` | `ON` | Image flip: `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Number of warmup frames to discard |
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Seconds between reconnection attempts |
| `ZED_ENGINE` | `native` | Detection engine: `native` (ZED SDK) or `open_vocab` (YOLO-World via ONNX Runtime) |
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | (`native`) Detection model: `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
| `ZED_DETECTION_CONFIDENCE` | `40` | (`native`) Detection confidence threshold (0-100) |
| `ZED_DETECTION_TRACKING` | `false` | (`native`) Enable object tracking across frames |
| `ZED_ONNX_MODEL_PATH` | _(required)_ | (`open_vocab`) Path to YOLO-World ONNX model file |
| `ZED_ONNX_LABELS` | _(required)_ | (`open_vocab`) Comma-separated label names (e.g. `person,cup,chair`). Can be changed between ticks — the node re-encodes embeddings automatically |
| `ZED_ONNX_CONFIDENCE` | `0.25` | (`open_vocab`) Detection confidence threshold (0.0-1.0) |
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | (`open_vocab`) NMS IoU threshold |
| `ZED_ONNX_INPUT_SIZE` | `640` | (`open_vocab`) Model input image size in pixels |
| `ZED_ONNX_TEXT_ENCODER_PATH` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_text_encoder.onnx`. Required when the visual model has two inputs (`images` + `text_feats`) |
| `ZED_ONNX_CLIP_VOCAB` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_vocab.txt` BPE vocabulary file |
| `ZED_ONNX_CLIP_MERGES` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_merges.txt` BPE merge rules file |
## Inputs
| Input | Description |
|---|---|
| `tick` | Required. Triggers a camera grab and detection on each message. Connect to `dora/timer/millis/N`. |
| `update_vocabulary` | Optional (`open_vocab` engine only). Payload: raw UTF-8 bytes of a comma-separated label string (e.g. `"person,cup,chair"`). Immediately re-encodes CLIP embeddings; the next `tick` uses the new labels. |
## Outputs
### `image_bgr`
BGR image as raw bytes. Metadata includes `shape` (HxWx3), `encoding` (`bgr8`), camera intrinsics (`fx`, `fy`, `cx`, `cy`), `distortion`, `frame_id`, and `timestamp_ns`.
### `point_cloud`
XYZRGBA point cloud as float32 bytes. Metadata includes `shape` (HxWx4), `units` (`mm`), and camera intrinsics.
### `camera_info`
Four float32 values: `[fx, fy, cx, cy]`. Metadata includes `distortion` coefficients.
### `objects`
Detected objects as a flat float32 array with shape `[num_objects, 37]`. Each row contains 37 floats:
| Index | Field | Description |
|---|---|---|
| 0 | `label_id` | Object class (0=PERSON, 1=VEHICLE, 2=BAG, 3=ANIMAL, 4=ELECTRONICS, 5=FRUIT_VEGETABLE, 6=SPORT) |
| 1 | `confidence` | Detection confidence (0-100) |
| 2-4 | `x, y, z` | 3D center position in mm |
| 5-12 | `bb2d_*` | 2D bounding box: 4 corners × 2 pixel coords |
| 13-36 | `bb3d_*` | 3D bounding box: 8 corners × 3 world coords (mm) |
Metadata includes `frame_id` and `timestamp_ns` matching the corresponding `image_bgr`, `image_width` and `image_height` for interpreting 2D coordinates, `engine` (`native` or `open_vocab`), and `label_names` (comma-separated class names for the active engine). When no objects are detected, an empty array with shape `[0, 37]` is sent.
**`open_vocab` engine:** Uses a YOLO-World model exported with open-vocabulary support (two ONNX inputs: `images` and `text_feats`). Labels are set at runtime via `ZED_ONNX_LABELS` and re-encoded automatically on each tick when they change — no node restart required. Export the model and CLIP text encoder with the helper scripts:
```bash
# Export the visual model (N=10 class slots, can be changed)
uv run scripts/export_yoloworld.py --model s --num-classes 10 --output-dir models/
# Export the CLIP text encoder + vocab files (once, reused for all model sizes)
uv run scripts/export_clip_text_encoder.py --output-dir models/
# Run
ZED_ENGINE=open_vocab \
ZED_ONNX_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,bag,chair,laptop" \
dora start dataflow.yml
```
To verify the model works before connecting the camera, build and run the standalone C++ test binary (`bin/test_open_vocab`). It requires no live camera — images are loaded from disk:
```bash
cd build && cmake .. && make test_open_vocab
# Place test images:
cp street.jpg test/assets/yolo.png
cp custom.jpg test/assets/yoloworld.png
# Fixed-vocab YOLO (both images use the same model):
ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
ZED_ONNX_LABELS="person,car,bus,bicycle" \
bin/test_open_vocab
# Open-vocab YOLO-World (separate model for second image via ZED_ONNX_WORLD_MODEL_PATH):
ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
ZED_ONNX_WORLD_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,bus,bicycle" \
bin/test_open_vocab
# Override image paths:
bin/test_open_vocab /path/to/img1.jpg /path/to/img2.jpg
```
Annotated results are saved alongside the input images as `yolo_result.png` and `yoloworld_result.png`.
3D positions are computed by fusing the 2D detections with the ZED point cloud (median of valid depth samples within each bounding box). 3D bounding box corners are back-projected at the p25/p75 depth percentiles.
Reference in New Issue View Git Blame Copy Permalink