dora-zed/README.md

# dora-zed

Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.

## Usage

Add to your dataflow:

```yaml
nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    tag: v0.2.0
    build: scripts/build.sh
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
    outputs:
      - image_bgr
      - point_cloud
      - camera_info
      - objects
```

See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.

## Dependencies

| Dependency | Version | Notes |
|---|---|---|
| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time |
| [CMake](https://cmake.org/download/) | 3.21+ | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required at configure time; used at runtime only by `yolo_onnx` |

### Installation scripts

| Platform | Script |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |

> **Jetson:** No official ONNX Runtime GPU binary for ARM64. It is still needed to configure the build; only the `yolo_onnx` engine uses it at runtime. Requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
>
> **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`

## Local build

```bash
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
scripts/build.sh
```

Binary installed to `bin/dora_zed`.

`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build
commands directly rather than through a shell, so `cd`, `&&` and command substitution are
not available there; keeping the sequence in one script is what makes the dataflow build
work. It configures a `Release` build, which is also what links the optimized dora
bindings.

### Containerized build

[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK, ONNX Runtime and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.

```bash
docker build -t dora-zed-build .
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
    -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
```

The container runs the same `scripts/build.sh`.

The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs.
`--user` keeps `build/` and `bin/` owned by you rather than root.

x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the
device and build natively — there is no ARM64 image here.

Needs an NVIDIA driver and the
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
on the host, as the ZED SDK does anyway. The image sets
`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC
(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host
only when `video` is in the capability list.

## Configuration

All behavior is controlled via environment variables.

### Camera

| Variable | Default | Description |
|---|---|---|
| `ZED_RESOLUTION` | `720` | `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth (mm) |
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth (mm) |
| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |

### Detection engine

Set `ZED_ENGINE` to select the detection backend:

| `ZED_ENGINE` | Custom model? | Classes |
|---|---|---|
| `native` (default) | No — ZED SDK built-in TensorRT | Fixed: 7 categories |
| `yolo_onnx` (alias: `onnx`) | Yes — any single-input YOLO ONNX (v5/v8/v11) | Fixed labels from file or env var |

#### `native` engine

Uses ZED SDK's built-in TensorRT models — no model files to download or configure. `ZED_DETECTION_MODEL` selects the speed/accuracy variant, not a file path.

Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.

| Variable | Default | Description |
|---|---|---|
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
| `ZED_DETECTION_CONFIDENCE` | `40` | Confidence threshold (0–100) |
| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking |

#### `yolo_onnx` engine (closed-vocabulary)

For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).

Export a YOLOv11 model once:

```bash
uv run scripts/export_yolo.py --model n --output-dir models/
```

Then run:

```bash
ZED_ENGINE=yolo_onnx \
ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
ZED_ONNX_LABELS_FILE=models/coco80.txt \
dora start dataflow.yml
```

**Required:**

| Variable | Description |
|---|---|
| `ZED_ONNX_MODEL_PATH` | Path to YOLO ONNX model |
| `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE` | Class names (inline CSV or one-per-line file) |

**Optional:**

| Variable | Default | Description |
|---|---|---|
| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
| `ZED_ONNX_OUTPUT_FORMAT` | `auto` | Output layout: `auto`, `nc_first`/`yolov8`/`yolov11`, `nc_last`, `nc_last_obj`/`yolov5`. `auto` detects from tensor shape. |

### Point cloud

| `ZED_PC_MODE` | Shape | Description |
|---|---|---|
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) |
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
| `ROI` | `[N, 4]` | Valid points within detected bounding boxes only |

## Inputs

| Input | Description |
|---|---|
| `tick` | Triggers a camera grab and detection. Connect to `dora/timer/millis/N`. |

## Outputs

| Output | Type | Description |
|---|---|---|
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, `timestamp_ns`. |
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics. |
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients. |
| `objects` | `float32 [N, 37]` | Detected objects. See below. |

### `objects` layout

Each of the N rows contains 37 floats:

| Index | Field | Description |
|---|---|---|
| 0 | `label_id` | Class index |
| 1 | `confidence` | Score (0–100) |
| 2–4 | `x, y, z` | 3D center (mm) |
| 5–12 | `bb2d` | 2D bounding box: 4 corners × (x, y) px |
| 13–36 | `bb3d` | 3D bounding box: 8 corners × (x, y, z) mm |

Metadata includes `engine`, `label_names`, `image_width`, `image_height`, and `timestamp_ns`. An empty array `[0, 37]` is sent when no objects are detected.