cristhian aguilera
6e6dc8ef27
Add engines::ZedNative class wrapping ZED SDK's TensorRT-optimized object detection. On every frame grab, detected objects (label, confidence, 3D position, 2D/3D bounding boxes) are serialized as a flat float32 array and sent as a new "objects" dora output with frame_id correlation to image_bgr. Extract shared env-var helpers into src/util/env.h and expose sl::Camera& via Camera::handle() for engine access. Closes #3
54 lines
3.3 KiB
Markdown
54 lines
3.3 KiB
Markdown
# AGENTS.md
|
|
|
|
This file provides guidance to AI coding assistants when working with code in this repository.
|
|
|
|
## What this is
|
|
|
|
A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame, runs object detection, and publishes four outputs over the dora dataflow: a BGR image, an XYZRGBA point cloud, camera intrinsics, and detected objects with 3D positions. Camera management lives in `src/zed/`, detection engines in `src/engines/`, and the dora event loop in `src/main.cc`.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
mkdir -p build && cd build && cmake .. && make install
|
|
```
|
|
|
|
The binary is installed to `bin/dora_zed` relative to the repo root.
|
|
|
|
**Requirements:** CMake 3.21+, ZED SDK 4.x (at `/usr/local/zed`), CUDA Toolkit, OpenCV 4.x, Rust stable toolchain (for dora-rs C++ bindings).
|
|
|
|
The first build clones dora-rs from GitHub and compiles `dora-node-api-cxx` via Cargo. This takes a while. Subsequent builds are incremental. If `DORA_ROOT_DIR` is set to a local dora checkout, it builds against that instead.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
dora up
|
|
dora start dataflow.yml
|
|
```
|
|
|
|
Camera and detection behavior is configured entirely through environment variables (see README.md for the full list: `ZED_RESOLUTION`, `ZED_FPS`, `ZED_DEPTH_MODE`, `ZED_DETECTION_MODEL`, etc.).
|
|
|
|
## Architecture
|
|
|
|
- **`src/util/env.h`** — Shared inline env-variable parsing helpers (`get_env_int`, `get_env_str`, `get_env_bool`, `get_env_float`).
|
|
- **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: open (with retry loop), warmup, grab (image + point cloud), calibration, and close. Exposes `handle()` for engine access.
|
|
- **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace.
|
|
- **`src/engines/zed_native.h`** — `engines::DetectionConfig`, `engines::DetectedObject`, and `engines::ZedNative` class. Wraps ZED SDK's TensorRT-optimized object detection module.
|
|
- **`src/engines/zed_native.cc`** — Implementation. Enables detection on the camera, retrieves objects per frame, converts to `DetectedObject` structs.
|
|
- **`src/main.cc`** — Dora event loop. Receives `tick` inputs, calls `camera.grab()` and `detector.detect()`, then sends `image_bgr`, `point_cloud`, `camera_info`, and `objects` outputs with shared `frame_id`. Handles reconnection on grab failure.
|
|
- **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue.
|
|
- **`dataflow.yml`** — Example dora dataflow definition using the `git:` source format.
|
|
|
|
## Key APIs
|
|
|
|
- **dora C++ API** (`<dora-node-api.h>`): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`.
|
|
- **ZED SDK** (`<sl/Camera.hpp>`): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`, `sl::ObjectDetectionParameters`, `sl::Objects`.
|
|
- Output data is sent as raw byte slices (`rust::Slice<const uint8_t>`) with typed metadata (shape, dtype, encoding).
|
|
|
|
## Tools
|
|
|
|
- **Issue tracker:** Gitea. Use the Gitea MCP tools (`mcp__gitea__*`) to read and manage issues, pull requests, and other repository operations.
|
|
|
|
## Rules
|
|
|
|
- Do not mention AI tools in commit messages or documentation.
|