dora-zed/AGENTS.md

AGENTS.md

This file provides guidance to AI coding assistants when working with code in this repository.

What this is

A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame, runs object detection, and publishes four outputs over the dora dataflow: a BGR image, an XYZRGBA point cloud, camera intrinsics, and detected objects with 3D positions. Camera management lives in src/zed/, detection engines in src/engines/, and the dora event loop in src/main.cc.

Build

mkdir -p build && cd build && cmake .. && make install

The binary is installed to bin/dora_zed relative to the repo root.

Requirements: CMake 3.21+, ZED SDK 5.x (at /usr/local/zed), CUDA Toolkit, ONNX Runtime 1.20+ (system install), Rust stable toolchain (for dora-rs C++ bindings).

The first build clones dora-rs from GitHub and compiles dora-node-api-cxx via Cargo. This takes a while. Subsequent builds are incremental. If DORA_ROOT_DIR is set to a local dora checkout, it builds against that instead.

Run

dora up
dora start dataflow.yml

Camera and detection behavior is configured entirely through environment variables (see README.md for the full list: ZED_RESOLUTION, ZED_FPS, ZED_DEPTH_MODE, ZED_DETECTION_MODEL, etc.).

Architecture

• src/util/env.h — Shared inline env-variable parsing helpers (get_env_int, get_env_str, get_env_bool, get_env_float).
• src/zed/camera.h — zed::Config, zed::Calibration, and zed::Camera class. Camera wraps the ZED SDK lifecycle: open (with retry loop), warmup, grab (image + point cloud), calibration, and close. Exposes handle() for engine access.
• src/zed/camera.cc — Implementation. SDK enum parsers live in an anonymous namespace.
• src/engines/engine.h — engines::DetectedObject struct and engines::Engine abstract base class (enable, detect, objects, label_names).
• src/engines/zed_native.h/cc — engines::ZedNative: wraps ZED SDK's TensorRT-optimized object detection module.
• src/engines/yolo_helpers.h — Header-only shared helpers used by both ONNX engines: RawDetection, nms(), letterbox(), fuse_bbox(), backproject(), split_labels(). Lives in engines::detail namespace.
• src/engines/yolo_onnx.h/cc — engines::YoloOnnx / YoloOnnxConfig: closed-vocabulary detection for any single-input YOLO ONNX model (YOLOv8, YOLOv11, YOLOv5, etc.). Auto-detects output layout (NC_FIRST / NC_LAST / NC_LAST_OBJ) from tensor shape; override with ZED_ONNX_OUTPUT_FORMAT. Labels loaded from ZED_ONNX_LABELS or ZED_ONNX_LABELS_FILE.
• src/engines/yoloworld_onnx.h/cc — engines::YoloWorldOnnx / YoloWorldOnnxConfig: open-vocabulary detection via YOLO-World ONNX (two inputs: images + text_feats). Reads ZED_ONNX_LABELS on every tick; when labels change, re-tokenizes and re-encodes via the CLIP text encoder ONNX session, passing (1, N, 512) L2-normalized embeddings to the visual model. Output is always NC_FIRST (1, 4+N, anchors).
• src/util/clip_tokenizer.h — Header-only C++ CLIP BPE tokenizer (no external deps). Implements the GPT-2 bytes_to_unicode() byte encoder + iterative BPE merge algorithm. Loads clip_vocab.txt and clip_merges.txt generated by scripts/export_clip_text_encoder.py. Returns 77 int64 token IDs (BOS=49406, EOS=49407, padded with 0).
• src/main.cc — Dora event loop. Selects engine via ZED_ENGINE env var (yolo_onnx/onnx, yoloworld_onnx/open_vocab, or native), holds a std::unique_ptr<engines::Engine>. Parses ZED_PC_MODE (FULL/HALF/ROI) once before the loop. On tick input: calls camera.grab() and detector->detect(), then sends image_bgr, point_cloud, camera_info, and objects outputs. Point cloud output varies by mode: FULL sends the raw sl::Mat pointer (no copy); HALF downsamples by 2 in each dimension; ROI emits only valid points within detected bounding boxes as a flat [N, 4] array. On update_vocabulary input: decodes payload as a UTF-8 label string and calls detector->set_labels() for immediate re-encoding. Handles reconnection on grab failure.
• DoraTargets.cmake — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (DORA_ROOT_DIR) or git clone. Produces dora_node_api_cxx library and the node_bridge.cc glue.
• dataflow.yml — Example dora dataflow definition using the git: source format.
• scripts/export_yolo.py — PEP 723 uv script. Exports a standard YOLOv11 model to ONNX for closed-vocabulary detection. Run: uv run scripts/export_yolo.py --model n --output-dir models/.
• scripts/export_yoloworld.py — PEP 723 uv script. Exports a YOLOv8-worldv2 model to ONNX with N class slots (labels arbitrary at runtime). Run: uv run scripts/export_yoloworld.py --model s --num-classes 10.
• scripts/export_clip_text_encoder.py — PEP 723 uv script. Exports CLIP ViT-B/32 text encoder to ONNX and saves clip_vocab.txt / clip_merges.txt. Run once; reused across all YOLO-World model sizes. Run: uv run scripts/export_clip_text_encoder.py --output-dir models/.
• src/test_yolo_onnx.cc — Standalone C++ test binary (bin/test_yolo_onnx). Exercises YoloOnnx with an image (default test/assets/yolo.png). No dora and no live camera required. Uses the same env vars as the main binary. Saves an annotated result image alongside the input.
• src/test_yoloworld_onnx.cc — Standalone C++ test binary (bin/test_yoloworld_onnx). Exercises YoloWorldOnnx with an image (default test/assets/yoloworld.png). Same interface as test_yolo_onnx.

Key APIs

• dora C++ API (<dora-node-api.h>): init_dora_node(), dora_node.events->next(), event_type(), send_output_with_metadata(), new_metadata().
• ZED SDK (<sl/Camera.hpp>): sl::Camera, sl::InitParameters, sl::RuntimeParameters, sl::Mat, sl::ObjectDetectionParameters, sl::Objects.
• ONNX Runtime (<onnxruntime_cxx_api.h>): Ort::Env, Ort::Session, Ort::SessionOptions, Ort::Value. Used by the open_vocab engine.
• Output data is sent as raw byte slices (rust::Slice<const uint8_t>) with typed metadata (shape, dtype, encoding).

Tools

• Issue tracker: Gitea. Use the Gitea MCP tools (mcp__gitea__*) to read and manage issues, pull requests, and other repository operations.

Rules

• Do not mention AI tools in commit messages or documentation.