dora-zed/AGENTS.md

AGENTS.md

This file provides guidance to AI coding assistants when working with code in this repository.

What this is

A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame, runs object detection, and publishes four outputs over the dora dataflow: a BGR image, an XYZRGBA point cloud, camera intrinsics, and detected objects with 3D positions. Camera management lives in src/zed/, detection engines in src/engines/, and the dora event loop in src/main.cc.

Build

mkdir -p build && cd build && cmake .. && make install

The binary is installed to bin/dora_zed relative to the repo root.

Requirements: CMake 3.21+, ZED SDK 5.x (at /usr/local/zed), CUDA Toolkit, ONNX Runtime 1.20+ (system install), Rust stable toolchain (for dora-rs C++ bindings).

The first build clones dora-rs from GitHub and compiles dora-node-api-cxx via Cargo. This takes a while. Subsequent builds are incremental. If DORA_ROOT_DIR is set to a local dora checkout, it builds against that instead.

Run

dora up
dora start dataflow.yml

Camera and detection behavior is configured entirely through environment variables (see README.md for the full list: ZED_RESOLUTION, ZED_FPS, ZED_DEPTH_MODE, ZED_DETECTION_MODEL, etc.).

Architecture

• src/util/env.h — Shared inline env-variable parsing helpers (get_env_int, get_env_str, get_env_bool, get_env_float).
• src/zed/camera.h — zed::Config, zed::Calibration, and zed::Camera class. Camera wraps the ZED SDK lifecycle: open (with retry loop), warmup, grab (image + point cloud), calibration, and close. Exposes handle() for engine access.
• src/zed/camera.cc — Implementation. SDK enum parsers live in an anonymous namespace.
• src/engines/engine.h — engines::DetectedObject struct and engines::Engine abstract base class (enable, detect, objects, label_names).
• src/engines/zed_native.h/cc — engines::ZedNative: wraps ZED SDK's TensorRT-optimized object detection module.
• src/engines/open_vocab.h/cc — engines::OpenVocab: YOLO-World open-vocabulary detection via ONNX Runtime. Supports two modes: single-input (fixed vocab, legacy) and two-input (open-vocab, images + text_feats). In two-input mode, reads ZED_ONNX_LABELS on every tick; when the labels change, re-tokenizes and re-encodes via the CLIP text encoder ONNX session, then passes the resulting (1, N, 512) L2-normalized embeddings to the visual model. Preprocesses images (letterbox, normalize, CHW), runs inference, applies NMS, then fuses 2D detections with the ZED point cloud to compute 3D positions and bounding boxes.
• src/util/clip_tokenizer.h — Header-only C++ CLIP BPE tokenizer (no external deps). Implements the GPT-2 bytes_to_unicode() byte encoder + iterative BPE merge algorithm. Loads clip_vocab.txt and clip_merges.txt generated by scripts/export_clip_text_encoder.py. Returns 77 int64 token IDs (BOS=49406, EOS=49407, padded with 0).
• src/main.cc — Dora event loop. Selects engine via ZED_ENGINE env var, holds a std::unique_ptr<engines::Engine>. Parses ZED_PC_MODE (FULL/HALF/ROI) once before the loop. On tick input: calls camera.grab() and detector->detect(), then sends image_bgr, point_cloud, camera_info, and objects outputs. Point cloud output varies by mode: FULL sends the raw sl::Mat pointer (no copy); HALF downsamples by 2 in each dimension; ROI emits only valid points within detected bounding boxes as a flat [N, 4] array. On update_vocabulary input: decodes payload as a UTF-8 label string and calls detector->set_labels() for immediate re-encoding. Handles reconnection on grab failure.
• DoraTargets.cmake — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (DORA_ROOT_DIR) or git clone. Produces dora_node_api_cxx library and the node_bridge.cc glue.
• dataflow.yml — Example dora dataflow definition using the git: source format.
• scripts/export_yoloworld.py — PEP 723 uv script. Exports a YOLOv8-worldv2 model to ONNX with N class slots (labels arbitrary at runtime). Run: uv run scripts/export_yoloworld.py --model s --num-classes 10.
• scripts/export_clip_text_encoder.py — PEP 723 uv script. Exports CLIP ViT-B/32 text encoder to ONNX and saves clip_vocab.txt / clip_merges.txt. Run once; reused across all YOLO-World model sizes. Run: uv run scripts/export_clip_text_encoder.py --output-dir models/.
• src/test_open_vocab.cc — Standalone C++ test binary (bin/test_open_vocab). Exercises the OpenVocab engine with images from test/assets/yolo.png and test/assets/yoloworld.png (paths overridable via argv). No dora and no live camera required — enable() ignores the camera for this engine. Uses the same env vars as the main binary. Saves annotated result images alongside inputs.

Key APIs

• dora C++ API (<dora-node-api.h>): init_dora_node(), dora_node.events->next(), event_type(), send_output_with_metadata(), new_metadata().
• ZED SDK (<sl/Camera.hpp>): sl::Camera, sl::InitParameters, sl::RuntimeParameters, sl::Mat, sl::ObjectDetectionParameters, sl::Objects.
• ONNX Runtime (<onnxruntime_cxx_api.h>): Ort::Env, Ort::Session, Ort::SessionOptions, Ort::Value. Used by the open_vocab engine.
• Output data is sent as raw byte slices (rust::Slice<const uint8_t>) with typed metadata (shape, dtype, encoding).

Tools

• Issue tracker: Gitea. Use the Gitea MCP tools (mcp__gitea__*) to read and manage issues, pull requests, and other repository operations.

Rules

• Do not mention AI tools in commit messages or documentation.