liria/dora-zed
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity
Files
44eec29b8313fe2fda7f8b97cdc125742b8c1aae
dora-zed/README.md
T
cristhian aguilera 44eec29b83 refactor open_vocab into yolo_onnx and yoloworld_onnx engines 
Split the monolithic open_vocab engine into two focused engines:
yolo_onnx (closed-vocabulary, any single-input YOLO ONNX) and
yoloworld_onnx (open-vocabulary YOLO-World with CLIP encoder).
Shared NMS/letterbox/fuse helpers extracted to yolo_helpers.h.
Add export_yolo.py script and separate test binaries for each engine.
2026-03-30 04:44:25 -03:00

214 lines
7.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# dora-zed
Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.
## Usage
Add to your dataflow:
```yaml
nodes:
- id: zed-camera
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
git_checkout: v0.1.0
build: mkdir -p build && cd build && cmake .. && make install
path: bin/dora_zed
inputs:
tick: dora/timer/millis/100
outputs:
- image_bgr
- point_cloud
- camera_info
- objects
```
See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.
## Dependencies
| Dependency | Version | Notes |
|---|---|---|
| [CMake](https://cmake.org/download/) | 3.21+ | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required only for `open_vocab` engine |
### Installation scripts
| Platform | Script |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
> **Jetson:** No official ONNX Runtime GPU binary for ARM64. The `native` engine works without it. The `open_vocab` engine requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
>
> **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
## Local build
```bash
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
mkdir -p build && cd build && cmake .. && make install
```
Binary installed to `bin/dora_zed`.
## Configuration
All behavior is controlled via environment variables.
### Camera
| Variable | Default | Description |
|---|---|---|
| `ZED_RESOLUTION` | `720` | `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth (mm) |
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth (mm) |
| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |
### Detection engine
Set `ZED_ENGINE` to select the detection backend:
| `ZED_ENGINE` | Custom model? | Classes |
|---|---|---|
| `native` (default) | No — ZED SDK built-in TensorRT | Fixed: 7 categories |
| `yolo_onnx` (alias: `onnx`) | Yes — any single-input YOLO ONNX (v5/v8/v11) | Fixed labels from file or env var |
| `yoloworld_onnx` (alias: `open_vocab`) | Yes — YOLO-World two-input ONNX | Defined at runtime via CLIP |
#### `native` engine
Uses ZED SDK's built-in TensorRT models — no model files to download or configure. `ZED_DETECTION_MODEL` selects the speed/accuracy variant, not a file path.
Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.
| Variable | Default | Description |
|---|---|---|
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
| `ZED_DETECTION_CONFIDENCE` | `40` | Confidence threshold (0–100) |
| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking |
#### `yolo_onnx` engine (closed-vocabulary)
For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).
Export a YOLOv11 model once:
```bash
uv run scripts/export_yolo.py --model n --output-dir models/
```
Then run:
```bash
ZED_ENGINE=yolo_onnx \
ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
ZED_ONNX_LABELS_FILE=models/coco80.txt \
dora start dataflow.yml
```
**Required:**
| Variable | Description |
|---|---|
| `ZED_ONNX_MODEL_PATH` | Path to YOLO ONNX model |
| `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE` | Class names (inline CSV or one-per-line file) |
**Optional:**
| Variable | Default | Description |
|---|---|---|
| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
| `ZED_ONNX_OUTPUT_FORMAT` | `auto` | Output layout: `auto`, `nc_first`/`yolov8`/`yolov11`, `nc_last`, `nc_last_obj`/`yolov5`. `auto` detects from tensor shape. |
#### `yoloworld_onnx` engine (open-vocabulary)
For YOLO-World two-input ONNX models with a CLIP text encoder. Labels are defined at runtime — no recompilation required. Use `ZED_ENGINE=open_vocab` as a legacy alias.
Export models once:
```bash
uv run scripts/export_yoloworld.py --model s --num-classes 10 --output-dir models/
uv run scripts/export_clip_text_encoder.py --output-dir models/
```
Then run:
```bash
ZED_ENGINE=yoloworld_onnx \
ZED_ONNX_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
ZED_ONNX_LABELS="person,car,chair" \
dora start dataflow.yml
```
**Required:**
| Variable | Description |
|---|---|
| `ZED_ONNX_MODEL_PATH` | Path to YOLO-World ONNX model |
| `ZED_ONNX_TEXT_ENCODER_PATH` | Path to `clip_text_encoder.onnx` |
| `ZED_ONNX_CLIP_VOCAB` | Path to `clip_vocab.txt` |
| `ZED_ONNX_CLIP_MERGES` | Path to `clip_merges.txt` |
| `ZED_ONNX_LABELS` | Comma-separated labels, e.g. `person,car,chair` |
**Optional:**
| Variable | Default | Description |
|---|---|---|
| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
Labels can be updated at runtime without restarting — send a new comma-separated string to the `update_vocabulary` input.
### Point cloud
| `ZED_PC_MODE` | Shape | Description |
|---|---|---|
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) |
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
| `ROI` | `[N, 4]` | Valid points within detected bounding boxes only |
## Inputs
| Input | Description |
|---|---|
| `tick` | Triggers a camera grab and detection. Connect to `dora/timer/millis/N`. |
| `update_vocabulary` | (`open_vocab` only) UTF-8 comma-separated label string. Re-encodes CLIP embeddings immediately. |
## Outputs
| Output | Type | Description |
|---|---|---|
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, `timestamp_ns`. |
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics. |
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients. |
| `objects` | `float32 [N, 37]` | Detected objects. See below. |
### `objects` layout
Each of the N rows contains 37 floats:
| Index | Field | Description |
|---|---|---|
| 0 | `label_id` | Class index |
| 1 | `confidence` | Score (0–100) |
| 2–4 | `x, y, z` | 3D center (mm) |
| 5–12 | `bb2d` | 2D bounding box: 4 corners × (x, y) px |
| 13–36 | `bb3d` | 3D bounding box: 8 corners × (x, y, z) mm |
Metadata includes `engine`, `label_names`, `image_width`, `image_height`, and `timestamp_ns`. An empty array `[0, 37]` is sent when no objects are detected.
Reference in New Issue View Git Blame Copy Permalink