diff --git a/AGENTS.md b/AGENTS.md index 298b6e6..b82454d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,7 @@ Camera and detection behavior is configured entirely through environment variabl - **`src/engines/zed_native.h/cc`** — `engines::ZedNative`: wraps ZED SDK's TensorRT-optimized object detection module. - **`src/engines/yolo_helpers.h`** — Header-only shared helpers used by the ONNX engine: `RawDetection`, `nms()`, `letterbox()`, `fuse_bbox()`, `backproject()`. Lives in `engines::detail` namespace. - **`src/engines/yolo_onnx.h/cc`** — `engines::YoloOnnx` / `YoloOnnxConfig`: closed-vocabulary detection for any single-input YOLO ONNX model (YOLOv8, YOLOv11, YOLOv5, etc.). Auto-detects output layout (`NC_FIRST` / `NC_LAST` / `NC_LAST_OBJ`) from tensor shape; override with `ZED_ONNX_OUTPUT_FORMAT`. Labels loaded from `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE`. -- **`src/main.cc`** — Dora event loop. Selects engine via `ZED_ENGINE` env var (`yolo_onnx`/`onnx` or `native`), holds a `std::unique_ptr`. Parses `ZED_PC_MODE` (`FULL`/`HALF`/`ROI`) once before the loop. On `tick` input: calls `camera.grab()` and `detector->detect()`, then sends `image_bgr`, `point_cloud`, `camera_info`, and `objects` outputs. Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension; `ROI` emits only valid points within detected bounding boxes as a flat `[N, 4]` array. Handles reconnection on grab failure. +- **`src/main.cc`** — Dora event loop. Selects engine via `ZED_ENGINE` env var (`yolo_onnx`/`onnx` or `native`), holds a `std::unique_ptr`. Parses `ZED_PC_MODE` (`FULL`/`HALF`/`ROI`) once before the loop. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()` and `detector->detect()`, then sends `image_bgr`, `point_cloud`, `camera_info`, and `objects` outputs. Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension; `ROI` emits only valid points within detected bounding boxes as a flat `[N, 4]` array. Handles reconnection on grab failure. - **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue. - **`dataflow.yml`** — Example dora dataflow definition using the `git:` source format. - **`scripts/export_yolo.py`** — PEP 723 uv script. Exports a standard YOLOv11 model to ONNX for closed-vocabulary detection. Run: `uv run scripts/export_yolo.py --model n --output-dir models/`. diff --git a/README.md b/README.md index 26d97c3..dce948c 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera | [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` | | [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK | | [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings | -| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required only for the `yolo_onnx` engine | +| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required at configure time; used at runtime only by `yolo_onnx` | ### Installation scripts @@ -42,7 +42,7 @@ See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera | NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` | | Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` | -> **Jetson:** No official ONNX Runtime GPU binary for ARM64. The `native` engine works without it. The `yolo_onnx` engine requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt). +> **Jetson:** No official ONNX Runtime GPU binary for ARM64. It is still needed to configure the build; only the `yolo_onnx` engine uses it at runtime. Requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt). > > **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime` diff --git a/scripts/install_linux_arm.sh b/scripts/install_linux_arm.sh index 195d881..f6d6afd 100755 --- a/scripts/install_linux_arm.sh +++ b/scripts/install_linux_arm.sh @@ -11,8 +11,8 @@ # JetPack 7.1 (L4T 38.4) # # ONNX Runtime: Microsoft does not publish official GPU binaries for ARM64. -# The native engine (ZED SDK built-in TensorRT) works without ONNX Runtime. -# To use the yolo_onnx engine, build ONNX Runtime from source: +# ONNX Runtime is still needed to configure the build; only the yolo_onnx +# engine uses it at runtime. Build it from source: # https://onnxruntime.ai/docs/build/eps.html#tensorrt set -euo pipefail @@ -66,8 +66,8 @@ fi # --- ONNX Runtime --- echo "" echo ">>> ONNX Runtime (ARM64 GPU): no official pre-built binary available." -echo " The native ZED engine works without ONNX Runtime." -echo " To use the yolo_onnx engine, build from source:" +echo " Needed to configure the build; only yolo_onnx uses it at runtime." +echo " Build from source:" echo " https://onnxruntime.ai/docs/build/eps.html#tensorrt" echo ""