liria/dora-zed
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

add native 3D object detection using ZED SDK optimized models

Browse Source 
Add engines::ZedNative class wrapping ZED SDK's TensorRT-optimized
object detection. On every frame grab, detected objects (label,
confidence, 3D position, 2D/3D bounding boxes) are serialized as a
flat float32 array and sent as a new "objects" dora output with
frame_id correlation to image_bgr.

Extract shared env-var helpers into src/util/env.h and expose
sl::Camera& via Camera::handle() for engine access.

Closes #3
This commit is contained in:
cristhian aguilera
2026-03-26 10:53:38 -03:00
parent 55a87d3a14
commit 6e6dc8ef27
10 changed files with 303 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+13 -6
View File
@@ -4,7 +4,7 @@ This file provides guidance to AI coding assistants when working with code in th
## What this is
A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame and publishes three outputs over the dora dataflow: a BGR image, an XYZRGBA point cloud, and camera intrinsics. Camera management lives in `src/zed/` and the dora event loop in `src/main.cc`.
A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame, runs object detection, and publishes four outputs over the dora dataflow: a BGR image, an XYZRGBA point cloud, camera intrinsics, and detected objects with 3D positions. Camera management lives in `src/zed/`, detection engines in `src/engines/`, and the dora event loop in `src/main.cc`.
## Build
@@ -25,22 +25,29 @@ dora up
dora start dataflow.yml
```
Camera behavior is configured entirely through environment variables (see README.md for the full list: `ZED_RESOLUTION`, `ZED_FPS`, `ZED_DEPTH_MODE`, etc.).
Camera and detection behavior is configured entirely through environment variables (see README.md for the full list: `ZED_RESOLUTION`, `ZED_FPS`, `ZED_DEPTH_MODE`, `ZED_DETECTION_MODEL`, etc.).
## Architecture
- **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: open (with retry loop), warmup, grab (image + point cloud), calibration, and close.
- **`src/zed/camera.cc`** — Implementation. Env-variable parsing helpers and SDK enum parsers live in an anonymous namespace.
- **`src/main.cc`** — Dora event loop. Receives `tick` inputs, calls `camera.grab()`, and sends `image_bgr`, `point_cloud`, and `camera_info` outputs with metadata. Handles reconnection on grab failure.
- **`src/util/env.h`** — Shared inline env-variable parsing helpers (`get_env_int`, `get_env_str`, `get_env_bool`, `get_env_float`).
- **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: open (with retry loop), warmup, grab (image + point cloud), calibration, and close. Exposes `handle()` for engine access.
- **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace.
- **`src/engines/zed_native.h`** — `engines::DetectionConfig`, `engines::DetectedObject`, and `engines::ZedNative` class. Wraps ZED SDK's TensorRT-optimized object detection module.
- **`src/engines/zed_native.cc`** — Implementation. Enables detection on the camera, retrieves objects per frame, converts to `DetectedObject` structs.
- **`src/main.cc`** — Dora event loop. Receives `tick` inputs, calls `camera.grab()` and `detector.detect()`, then sends `image_bgr`, `point_cloud`, `camera_info`, and `objects` outputs with shared `frame_id`. Handles reconnection on grab failure.
- **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue.
- **`dataflow.yml`** — Example dora dataflow definition using the `git:` source format.
## Key APIs
- **dora C++ API** (`<dora-node-api.h>`): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`.
- **ZED SDK** (`<sl/Camera.hpp>`): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`.
- **ZED SDK** (`<sl/Camera.hpp>`): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`, `sl::ObjectDetectionParameters`, `sl::Objects`.
- Output data is sent as raw byte slices (`rust::Slice<const uint8_t>`) with typed metadata (shape, dtype, encoding).
## Tools
- **Issue tracker:** Gitea. Use the Gitea MCP tools (`mcp__gitea__*`) to read and manage issues, pull requests, and other repository operations.
## Rules
- Do not mention AI tools in commit messages or documentation.