liria/dora-zed
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity
Files
f40e697b2a9f905792dfc2c463ace04a1768bf2c
dora-zed/README.md
T
cristhian f40e697b2a add a container build environment 
Dockerfile provisions CUDA 12.8, ZED SDK 5.2, ONNX Runtime and Rust on
Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for, so the
node can be compiled and the build verified without installing the SDK
stack on the host.

The compile needs --gpus all: libsl_zed.so links against the CUDA driver
API, NVDEC and NVENC, which ship with the display driver rather than the
CUDA toolkit and so exist in no image. The container runtime mounts them
from the host, and only includes NVDEC/NVENC when the video capability
is requested.

Runs as the invoking user so build/ and bin/ are not left root-owned,
which needs the toolchain outside /root and the ZED SDK directory
readable to an arbitrary uid.
2026-09-07 14:33:58 -03:00

192 lines
7.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# dora-zed
Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.
## Usage
Add to your dataflow:
```yaml
nodes:
- id: zed-camera
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
git_checkout: v0.1.0
build: mkdir -p build && cd build && cmake .. && make install
path: bin/dora_zed
inputs:
tick: dora/timer/millis/100
outputs:
- image_bgr
- point_cloud
- camera_info
- objects
```
See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.
## Dependencies
| Dependency | Version | Notes |
|---|---|---|
| [CMake](https://cmake.org/download/) | 3.21+ | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required at configure time; used at runtime only by `yolo_onnx` |
### Installation scripts
| Platform | Script |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
> **Jetson:** No official ONNX Runtime GPU binary for ARM64. It is still needed to configure the build; only the `yolo_onnx` engine uses it at runtime. Requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
>
> **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
## Local build
```bash
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
mkdir -p build && cd build && cmake .. && make install
```
Binary installed to `bin/dora_zed`.
### Containerized build
[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK, ONNX Runtime and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.
```bash
docker build -t dora-zed-build .
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
-v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
```
The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs.
`--user` keeps `build/` and `bin/` owned by you rather than root.
x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the
device and build natively — there is no ARM64 image here.
Needs an NVIDIA driver and the
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
on the host, as the ZED SDK does anyway. The image sets
`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC
(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host
only when `video` is in the capability list.
## Configuration
All behavior is controlled via environment variables.
### Camera
| Variable | Default | Description |
|---|---|---|
| `ZED_RESOLUTION` | `720` | `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth (mm) |
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth (mm) |
| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |
### Detection engine
Set `ZED_ENGINE` to select the detection backend:
| `ZED_ENGINE` | Custom model? | Classes |
|---|---|---|
| `native` (default) | No — ZED SDK built-in TensorRT | Fixed: 7 categories |
| `yolo_onnx` (alias: `onnx`) | Yes — any single-input YOLO ONNX (v5/v8/v11) | Fixed labels from file or env var |
#### `native` engine
Uses ZED SDK's built-in TensorRT models — no model files to download or configure. `ZED_DETECTION_MODEL` selects the speed/accuracy variant, not a file path.
Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.
| Variable | Default | Description |
|---|---|---|
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
| `ZED_DETECTION_CONFIDENCE` | `40` | Confidence threshold (0–100) |
| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking |
#### `yolo_onnx` engine (closed-vocabulary)
For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).
Export a YOLOv11 model once:
```bash
uv run scripts/export_yolo.py --model n --output-dir models/
```
Then run:
```bash
ZED_ENGINE=yolo_onnx \
ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
ZED_ONNX_LABELS_FILE=models/coco80.txt \
dora start dataflow.yml
```
**Required:**
| Variable | Description |
|---|---|
| `ZED_ONNX_MODEL_PATH` | Path to YOLO ONNX model |
| `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE` | Class names (inline CSV or one-per-line file) |
**Optional:**
| Variable | Default | Description |
|---|---|---|
| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
| `ZED_ONNX_OUTPUT_FORMAT` | `auto` | Output layout: `auto`, `nc_first`/`yolov8`/`yolov11`, `nc_last`, `nc_last_obj`/`yolov5`. `auto` detects from tensor shape. |
### Point cloud
| `ZED_PC_MODE` | Shape | Description |
|---|---|---|
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) |
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
| `ROI` | `[N, 4]` | Valid points within detected bounding boxes only |
## Inputs
| Input | Description |
|---|---|
| `tick` | Triggers a camera grab and detection. Connect to `dora/timer/millis/N`. |
## Outputs
| Output | Type | Description |
|---|---|---|
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, `timestamp_ns`. |
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics. |
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients. |
| `objects` | `float32 [N, 37]` | Detected objects. See below. |
### `objects` layout
Each of the N rows contains 37 floats:
| Index | Field | Description |
|---|---|---|
| 0 | `label_id` | Class index |
| 1 | `confidence` | Score (0–100) |
| 2–4 | `x, y, z` | 3D center (mm) |
| 5–12 | `bb2d` | 2D bounding box: 4 corners × (x, y) px |
| 13–36 | `bb3d` | 3D bounding box: 8 corners × (x, y, z) mm |
Metadata includes `engine`, `label_names`, `image_width`, `image_height`, and `timestamp_ns`. An empty array `[0, 37]` is sent when no objects are detected.
Reference in New Issue View Git Blame Copy Permalink