Files

T

cristhian aguilera 6e6dc8ef27 add native 3D object detection using ZED SDK optimized models

Add engines::ZedNative class wrapping ZED SDK's TensorRT-optimized
object detection. On every frame grab, detected objects (label,
confidence, 3D position, 2D/3D bounding boxes) are serialized as a
flat float32 array and sent as a new "objects" dora output with
frame_id correlation to image_bgr.

Extract shared env-var helpers into src/util/env.h and expose
sl::Camera& via Camera::handle() for engine access.

Closes #3

2026-03-26 10:53:38 -03:00

3.3 KiB

Raw Blame History

AGENTS.md

This file provides guidance to AI coding assistants when working with code in this repository.

What this is

A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame, runs object detection, and publishes four outputs over the dora dataflow: a BGR image, an XYZRGBA point cloud, camera intrinsics, and detected objects with 3D positions. Camera management lives in src/zed/, detection engines in src/engines/, and the dora event loop in src/main.cc.

Build

mkdir -p build && cd build && cmake .. && make install

The binary is installed to bin/dora_zed relative to the repo root.

Requirements: CMake 3.21+, ZED SDK 4.x (at /usr/local/zed), CUDA Toolkit, OpenCV 4.x, Rust stable toolchain (for dora-rs C++ bindings).

The first build clones dora-rs from GitHub and compiles dora-node-api-cxx via Cargo. This takes a while. Subsequent builds are incremental. If DORA_ROOT_DIR is set to a local dora checkout, it builds against that instead.

Run

dora up
dora start dataflow.yml

Camera and detection behavior is configured entirely through environment variables (see README.md for the full list: ZED_RESOLUTION, ZED_FPS, ZED_DEPTH_MODE, ZED_DETECTION_MODEL, etc.).

Architecture

src/util/env.h — Shared inline env-variable parsing helpers (get_env_int, get_env_str, get_env_bool, get_env_float).
src/zed/camera.h — zed::Config, zed::Calibration, and zed::Camera class. Camera wraps the ZED SDK lifecycle: open (with retry loop), warmup, grab (image + point cloud), calibration, and close. Exposes handle() for engine access.
src/zed/camera.cc — Implementation. SDK enum parsers live in an anonymous namespace.
src/engines/zed_native.h — engines::DetectionConfig, engines::DetectedObject, and engines::ZedNative class. Wraps ZED SDK's TensorRT-optimized object detection module.
src/engines/zed_native.cc — Implementation. Enables detection on the camera, retrieves objects per frame, converts to DetectedObject structs.
src/main.cc — Dora event loop. Receives tick inputs, calls camera.grab() and detector.detect(), then sends image_bgr, point_cloud, camera_info, and objects outputs with shared frame_id. Handles reconnection on grab failure.
DoraTargets.cmake — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (DORA_ROOT_DIR) or git clone. Produces dora_node_api_cxx library and the node_bridge.cc glue.
dataflow.yml — Example dora dataflow definition using the git: source format.

Key APIs

dora C++ API (<dora-node-api.h>): init_dora_node(), dora_node.events->next(), event_type(), send_output_with_metadata(), new_metadata().
ZED SDK (<sl/Camera.hpp>): sl::Camera, sl::InitParameters, sl::RuntimeParameters, sl::Mat, sl::ObjectDetectionParameters, sl::Objects.
Output data is sent as raw byte slices (rust::Slice<const uint8_t>) with typed metadata (shape, dtype, encoding).

Tools

Issue tracker: Gitea. Use the Gitea MCP tools (mcp__gitea__*) to read and manage issues, pull requests, and other repository operations.

Rules

Do not mention AI tools in commit messages or documentation.

3.3 KiB Raw Blame History