Files

T

cristhian aguilera 6e6dc8ef27 add native 3D object detection using ZED SDK optimized models

Add engines::ZedNative class wrapping ZED SDK's TensorRT-optimized
object detection. On every frame grab, detected objects (label,
confidence, 3D position, 2D/3D bounding boxes) are serialized as a
flat float32 array and sent as a new "objects" dora output with
frame_id correlation to image_bgr.

Extract shared env-var helpers into src/util/env.h and expose
sl::Camera& via Camera::handle() for engine access.

Closes #3

2026-03-26 10:53:38 -03:00

3.2 KiB

Raw Blame History

dora-zed

Dora C++ node for ZED stereo cameras. Captures frames and publishes BGR images, XYZRGBA point clouds, camera intrinsics, and 3D object detections.

Usage via `git:` in a dataflow

nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    build: mkdir -p build && cd build && cmake .. && make install
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
    outputs:
      - image_bgr
      - point_cloud
      - camera_info
      - objects

Then run:

dora build dataflow.yml
dora up
dora start dataflow.yml

Local build

git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
mkdir -p build && cd build
cmake ..
make install

The binary is installed to bin/dora_zed relative to the repository root.

Dependencies

Dependency	Minimum version	Notes
CMake	3.21
ZED SDK	4.x	Default path: `/usr/local/zed`
CUDA Toolkit	Required by ZED SDK
OpenCV	4.x
Rust toolchain	stable	Used to build dora-rs C++ bindings

Environment variables

Variable	Default	Description
`ZED_RESOLUTION`	`720`	Camera resolution: `720`, `1080`, or `2000`
`ZED_FPS`	`15`	Camera frame rate
`ZED_DEPTH_MODE`	`NEURAL`	Depth mode: `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE`
`ZED_DEPTH_MIN_MM`	`10`	Minimum depth distance in millimeters
`ZED_DEPTH_MAX_MM`	`500`	Maximum depth distance in millimeters
`ZED_DEPTH_FILL`	`false`	Enable depth fill mode
`ZED_FLIP`	`ON`	Image flip: `ON` or `OFF`
`ZED_WARMUP_FRAMES`	`30`	Number of warmup frames to discard
`ZED_RETRY_INTERVAL_SECONDS`	`5`	Seconds between reconnection attempts
`ZED_DETECTION_MODEL`	`MULTI_CLASS_BOX_MEDIUM`	Detection model: `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE`
`ZED_DETECTION_CONFIDENCE`	`40`	Detection confidence threshold (0-100)
`ZED_DETECTION_TRACKING`	`false`	Enable object tracking across frames (enables positional tracking)

Outputs

`image_bgr`

BGR image as raw bytes. Metadata includes shape (HxWx3), encoding (bgr8), camera intrinsics (fx, fy, cx, cy), distortion, frame_id, and timestamp_ns.

`point_cloud`

XYZRGBA point cloud as float32 bytes. Metadata includes shape (HxWx4), units (mm), and camera intrinsics.

`camera_info`

Four float32 values: [fx, fy, cx, cy]. Metadata includes distortion coefficients.

`objects`

Detected objects as a flat float32 array with shape [num_objects, 37]. Each row contains 37 floats:

Index	Field	Description
0	`label_id`	Object class (0=PERSON, 1=VEHICLE, 2=BAG, 3=ANIMAL, 4=ELECTRONICS, 5=FRUIT_VEGETABLE, 6=SPORT)
1	`confidence`	Detection confidence (0-100)
2-4	`x, y, z`	3D center position in mm
5-12	`bb2d_*`	2D bounding box: 4 corners × 2 pixel coords
13-36	`bb3d_*`	3D bounding box: 8 corners × 3 world coords (mm)

Metadata includes frame_id and timestamp_ns matching the corresponding image_bgr, plus image_width and image_height for interpreting 2D bounding box pixel coordinates. When no objects are detected, an empty array with shape [0, 37] is sent.

3.2 KiB Raw Blame History Unescape Escape

dora-zed

Usage via git: in a dataflow