cristhian
b16dfcd965
ONNX Runtime is a configure-time dependency for every build, not only for the yolo_onnx engine. The event loop no longer inspects the input id, so it grabs on any input rather than specifically on tick.
169 lines
5.9 KiB
Markdown
169 lines
5.9 KiB
Markdown
# dora-zed
|
||
|
||
Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.
|
||
|
||
## Usage
|
||
|
||
Add to your dataflow:
|
||
|
||
```yaml
|
||
nodes:
|
||
- id: zed-camera
|
||
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
|
||
git_checkout: v0.1.0
|
||
build: mkdir -p build && cd build && cmake .. && make install
|
||
path: bin/dora_zed
|
||
inputs:
|
||
tick: dora/timer/millis/100
|
||
outputs:
|
||
- image_bgr
|
||
- point_cloud
|
||
- camera_info
|
||
- objects
|
||
```
|
||
|
||
See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.
|
||
|
||
## Dependencies
|
||
|
||
| Dependency | Version | Notes |
|
||
|---|---|---|
|
||
| [CMake](https://cmake.org/download/) | 3.21+ | |
|
||
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
|
||
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
|
||
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
|
||
| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required at configure time; used at runtime only by `yolo_onnx` |
|
||
|
||
### Installation scripts
|
||
|
||
| Platform | Script |
|
||
|---|---|
|
||
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
|
||
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
|
||
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
|
||
|
||
> **Jetson:** No official ONNX Runtime GPU binary for ARM64. It is still needed to configure the build; only the `yolo_onnx` engine uses it at runtime. Requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
|
||
>
|
||
> **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
|
||
|
||
## Local build
|
||
|
||
```bash
|
||
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
|
||
cd dora-zed
|
||
mkdir -p build && cd build && cmake .. && make install
|
||
```
|
||
|
||
Binary installed to `bin/dora_zed`.
|
||
|
||
## Configuration
|
||
|
||
All behavior is controlled via environment variables.
|
||
|
||
### Camera
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `ZED_RESOLUTION` | `720` | `720`, `1080`, or `2000` |
|
||
| `ZED_FPS` | `15` | Frame rate |
|
||
| `ZED_DEPTH_MODE` | `NEURAL` | `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
|
||
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth (mm) |
|
||
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth (mm) |
|
||
| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
|
||
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
|
||
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
|
||
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |
|
||
|
||
### Detection engine
|
||
|
||
Set `ZED_ENGINE` to select the detection backend:
|
||
|
||
| `ZED_ENGINE` | Custom model? | Classes |
|
||
|---|---|---|
|
||
| `native` (default) | No — ZED SDK built-in TensorRT | Fixed: 7 categories |
|
||
| `yolo_onnx` (alias: `onnx`) | Yes — any single-input YOLO ONNX (v5/v8/v11) | Fixed labels from file or env var |
|
||
|
||
#### `native` engine
|
||
|
||
Uses ZED SDK's built-in TensorRT models — no model files to download or configure. `ZED_DETECTION_MODEL` selects the speed/accuracy variant, not a file path.
|
||
|
||
Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
|
||
| `ZED_DETECTION_CONFIDENCE` | `40` | Confidence threshold (0–100) |
|
||
| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking |
|
||
|
||
#### `yolo_onnx` engine (closed-vocabulary)
|
||
|
||
For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).
|
||
|
||
Export a YOLOv11 model once:
|
||
|
||
```bash
|
||
uv run scripts/export_yolo.py --model n --output-dir models/
|
||
```
|
||
|
||
Then run:
|
||
|
||
```bash
|
||
ZED_ENGINE=yolo_onnx \
|
||
ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
|
||
ZED_ONNX_LABELS_FILE=models/coco80.txt \
|
||
dora start dataflow.yml
|
||
```
|
||
|
||
**Required:**
|
||
|
||
| Variable | Description |
|
||
|---|---|
|
||
| `ZED_ONNX_MODEL_PATH` | Path to YOLO ONNX model |
|
||
| `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE` | Class names (inline CSV or one-per-line file) |
|
||
|
||
**Optional:**
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
|
||
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
|
||
| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
|
||
| `ZED_ONNX_OUTPUT_FORMAT` | `auto` | Output layout: `auto`, `nc_first`/`yolov8`/`yolov11`, `nc_last`, `nc_last_obj`/`yolov5`. `auto` detects from tensor shape. |
|
||
|
||
### Point cloud
|
||
|
||
| `ZED_PC_MODE` | Shape | Description |
|
||
|---|---|---|
|
||
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) |
|
||
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
|
||
| `ROI` | `[N, 4]` | Valid points within detected bounding boxes only |
|
||
|
||
## Inputs
|
||
|
||
| Input | Description |
|
||
|---|---|
|
||
| `tick` | Triggers a camera grab and detection. Connect to `dora/timer/millis/N`. |
|
||
|
||
## Outputs
|
||
|
||
| Output | Type | Description |
|
||
|---|---|---|
|
||
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, `timestamp_ns`. |
|
||
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics. |
|
||
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients. |
|
||
| `objects` | `float32 [N, 37]` | Detected objects. See below. |
|
||
|
||
### `objects` layout
|
||
|
||
Each of the N rows contains 37 floats:
|
||
|
||
| Index | Field | Description |
|
||
|---|---|---|
|
||
| 0 | `label_id` | Class index |
|
||
| 1 | `confidence` | Score (0–100) |
|
||
| 2–4 | `x, y, z` | 3D center (mm) |
|
||
| 5–12 | `bb2d` | 2D bounding box: 4 corners × (x, y) px |
|
||
| 13–36 | `bb3d` | 3D bounding box: 8 corners × (x, y, z) mm |
|
||
|
||
Metadata includes `engine`, `label_names`, `image_width`, `image_height`, and `timestamp_ns`. An empty array `[0, 37]` is sent when no objects are detected.
|