cristhian
f1934c9b4a
Deletes the yoloworld_onnx engine, the CLIP BPE tokenizer, the standalone test binary and both export scripts. Drops the update_vocabulary input and the yoloworld_onnx/open_vocab dispatch from main.cc, along with the now unused Engine::set_labels. Documentation follows.
169 lines
5.8 KiB
Markdown
169 lines
5.8 KiB
Markdown
# dora-zed
|
||
|
||
Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.
|
||
|
||
## Usage
|
||
|
||
Add to your dataflow:
|
||
|
||
```yaml
|
||
nodes:
|
||
- id: zed-camera
|
||
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
|
||
git_checkout: v0.1.0
|
||
build: mkdir -p build && cd build && cmake .. && make install
|
||
path: bin/dora_zed
|
||
inputs:
|
||
tick: dora/timer/millis/100
|
||
outputs:
|
||
- image_bgr
|
||
- point_cloud
|
||
- camera_info
|
||
- objects
|
||
```
|
||
|
||
See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.
|
||
|
||
## Dependencies
|
||
|
||
| Dependency | Version | Notes |
|
||
|---|---|---|
|
||
| [CMake](https://cmake.org/download/) | 3.21+ | |
|
||
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
|
||
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
|
||
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
|
||
| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required only for the `yolo_onnx` engine |
|
||
|
||
### Installation scripts
|
||
|
||
| Platform | Script |
|
||
|---|---|
|
||
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
|
||
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
|
||
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
|
||
|
||
> **Jetson:** No official ONNX Runtime GPU binary for ARM64. The `native` engine works without it. The `yolo_onnx` engine requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
|
||
>
|
||
> **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
|
||
|
||
## Local build
|
||
|
||
```bash
|
||
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
|
||
cd dora-zed
|
||
mkdir -p build && cd build && cmake .. && make install
|
||
```
|
||
|
||
Binary installed to `bin/dora_zed`.
|
||
|
||
## Configuration
|
||
|
||
All behavior is controlled via environment variables.
|
||
|
||
### Camera
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `ZED_RESOLUTION` | `720` | `720`, `1080`, or `2000` |
|
||
| `ZED_FPS` | `15` | Frame rate |
|
||
| `ZED_DEPTH_MODE` | `NEURAL` | `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
|
||
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth (mm) |
|
||
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth (mm) |
|
||
| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
|
||
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
|
||
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
|
||
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |
|
||
|
||
### Detection engine
|
||
|
||
Set `ZED_ENGINE` to select the detection backend:
|
||
|
||
| `ZED_ENGINE` | Custom model? | Classes |
|
||
|---|---|---|
|
||
| `native` (default) | No — ZED SDK built-in TensorRT | Fixed: 7 categories |
|
||
| `yolo_onnx` (alias: `onnx`) | Yes — any single-input YOLO ONNX (v5/v8/v11) | Fixed labels from file or env var |
|
||
|
||
#### `native` engine
|
||
|
||
Uses ZED SDK's built-in TensorRT models — no model files to download or configure. `ZED_DETECTION_MODEL` selects the speed/accuracy variant, not a file path.
|
||
|
||
Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
|
||
| `ZED_DETECTION_CONFIDENCE` | `40` | Confidence threshold (0–100) |
|
||
| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking |
|
||
|
||
#### `yolo_onnx` engine (closed-vocabulary)
|
||
|
||
For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).
|
||
|
||
Export a YOLOv11 model once:
|
||
|
||
```bash
|
||
uv run scripts/export_yolo.py --model n --output-dir models/
|
||
```
|
||
|
||
Then run:
|
||
|
||
```bash
|
||
ZED_ENGINE=yolo_onnx \
|
||
ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
|
||
ZED_ONNX_LABELS_FILE=models/coco80.txt \
|
||
dora start dataflow.yml
|
||
```
|
||
|
||
**Required:**
|
||
|
||
| Variable | Description |
|
||
|---|---|
|
||
| `ZED_ONNX_MODEL_PATH` | Path to YOLO ONNX model |
|
||
| `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE` | Class names (inline CSV or one-per-line file) |
|
||
|
||
**Optional:**
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
|
||
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
|
||
| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
|
||
| `ZED_ONNX_OUTPUT_FORMAT` | `auto` | Output layout: `auto`, `nc_first`/`yolov8`/`yolov11`, `nc_last`, `nc_last_obj`/`yolov5`. `auto` detects from tensor shape. |
|
||
|
||
### Point cloud
|
||
|
||
| `ZED_PC_MODE` | Shape | Description |
|
||
|---|---|---|
|
||
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) |
|
||
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
|
||
| `ROI` | `[N, 4]` | Valid points within detected bounding boxes only |
|
||
|
||
## Inputs
|
||
|
||
| Input | Description |
|
||
|---|---|
|
||
| `tick` | Triggers a camera grab and detection. Connect to `dora/timer/millis/N`. |
|
||
|
||
## Outputs
|
||
|
||
| Output | Type | Description |
|
||
|---|---|---|
|
||
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, `timestamp_ns`. |
|
||
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics. |
|
||
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients. |
|
||
| `objects` | `float32 [N, 37]` | Detected objects. See below. |
|
||
|
||
### `objects` layout
|
||
|
||
Each of the N rows contains 37 floats:
|
||
|
||
| Index | Field | Description |
|
||
|---|---|---|
|
||
| 0 | `label_id` | Class index |
|
||
| 1 | `confidence` | Score (0–100) |
|
||
| 2–4 | `x, y, z` | 3D center (mm) |
|
||
| 5–12 | `bb2d` | 2D bounding box: 4 corners × (x, y) px |
|
||
| 13–36 | `bb3d` | 3D bounding box: 8 corners × (x, y, z) mm |
|
||
|
||
Metadata includes `engine`, `label_names`, `image_width`, `image_height`, and `timestamp_ns`. An empty array `[0, 37]` is sent when no objects are detected.
|