diff --git a/AGENTS.md b/AGENTS.md index b5c9da8..f37926f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,9 @@ This file provides guidance to AI coding assistants when working with code in th ## What this is -A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame, runs object detection, and publishes four outputs over the dora dataflow: a BGR image, an XYZRGBA point cloud, camera intrinsics, and detected objects with 3D positions. Camera management lives in `src/zed/`, detection engines in `src/engines/`, and the dora event loop in `src/main.cc`. +A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame and publishes it over the dora dataflow: a BGR image, an XYZRGBA point cloud and camera intrinsics. Camera management lives in `src/zed/` and the dora event loop in `src/main.cc`. + +The node is a camera source only. It ran inference in-process through `v0.1.0`, under `src/engines/`; that was removed so the node has no inference dependency and builds on any platform the ZED SDK supports. The removed code remains reachable at tag `v0.1.0`. ## Build @@ -16,7 +18,7 @@ The binary is installed to `bin/dora_zed` relative to the repo root. The script the dataflow's `build:` command: dora 1.0 executes build commands directly rather than through a shell, so `cd`, `&&` and command substitution do not work there. -**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x (at `/usr/local/zed`), CUDA Toolkit, ONNX Runtime 1.20+ (system install), Rust stable toolchain (for dora-rs C++ bindings). +**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x (at `/usr/local/zed`), CUDA Toolkit, Rust stable toolchain (for dora-rs C++ bindings). The first build clones dora-rs from GitHub and compiles `dora-node-api-cxx` via Cargo. This takes a while. Subsequent builds are incremental. If `DORA_ROOT_DIR` is set to a local dora checkout, it builds against that instead. @@ -29,30 +31,23 @@ dora up dora start dataflow.yml ``` -Camera and detection behavior is configured entirely through environment variables (see README.md for the full list: `ZED_RESOLUTION`, `ZED_FPS`, `ZED_DEPTH_MODE`, `ZED_DETECTION_MODEL`, etc.). +Camera behavior is configured entirely through environment variables (see README.md for the full list: `ZED_RESOLUTION`, `ZED_FPS`, `ZED_DEPTH_MODE`, etc.). ## Architecture - **`src/util/env.h`** — Shared inline env-variable parsing helpers (`get_env_int`, `get_env_str`, `get_env_bool`, `get_env_float`). - **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: open (with retry loop), warmup, grab (image + point cloud), calibration, and close. Exposes `handle()` for engine access. - **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace. -- **`src/engines/engine.h`** — `engines::DetectedObject` struct and `engines::Engine` abstract base class (`enable`, `detect`, `objects`, `label_names`). -- **`src/engines/zed_native.h/cc`** — `engines::ZedNative`: wraps ZED SDK's TensorRT-optimized object detection module. -- **`src/engines/yolo_helpers.h`** — Header-only shared helpers used by the ONNX engine: `RawDetection`, `nms()`, `letterbox()`, `fuse_bbox()`, `backproject()`. Lives in `engines::detail` namespace. -- **`src/engines/yolo_onnx.h/cc`** — `engines::YoloOnnx` / `YoloOnnxConfig`: closed-vocabulary detection for any single-input YOLO ONNX model (YOLOv8, YOLOv11, YOLOv5, etc.). Auto-detects output layout (`NC_FIRST` / `NC_LAST` / `NC_LAST_OBJ`) from tensor shape; override with `ZED_ONNX_OUTPUT_FORMAT`. Labels loaded from `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE`. -- **`src/main.cc`** — Dora event loop. Selects engine via `ZED_ENGINE` env var (`yolo_onnx`/`onnx` or `native`), holds a `std::unique_ptr`. Parses `ZED_PC_MODE` (`FULL`/`HALF`/`ROI`) once before the loop. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()` and `detector->detect()`, then sends `image_bgr`, `point_cloud`, `camera_info`, and `objects` outputs. Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension; `ROI` emits only valid points within detected bounding boxes as a flat `[N, 4]` array. Handles reconnection on grab failure. +- **`src/main.cc`** — Dora event loop. Parses `ZED_PC_MODE` (`FULL`/`HALF`) once before the loop. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()`, then sends `image_bgr`, `point_cloud` and `camera_info`. Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension. Handles reconnection on grab failure. - **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue. - **`scripts/build.sh`** — Configure/compile/install in one shell-free-callable step. Referenced by `dataflow.yml`, the `Dockerfile` and the README. - **`dataflow.yml`** — Example dora dataflow definition using the `git:` source format, pinned with `tag:` — dora 1.0 pins a source with the mutually exclusive `tag:`, `rev:` or `branch:`. -- **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, ONNX Runtime, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. ONNX Runtime's CMake package files are relocated to `/usr/local/lib/cmake` because Ubuntu's multiarch CMake does not search `lib64`. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`. -- **`scripts/export_yolo.py`** — PEP 723 uv script. Exports a standard YOLOv11 model to ONNX for closed-vocabulary detection. Run: `uv run scripts/export_yolo.py --model n --output-dir models/`. -- **`src/test_yolo_onnx.cc`** — Standalone C++ test binary (`bin/test_yolo_onnx`). Exercises `YoloOnnx` with an image (default `test/assets/yolo.png`). No dora and no live camera required. Uses the same env vars as the main binary. Saves an annotated result image alongside the input. +- **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`. ## Key APIs - **dora C++ API** (``): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`. -- **ZED SDK** (``): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`, `sl::ObjectDetectionParameters`, `sl::Objects`. -- **ONNX Runtime** (``): `Ort::Env`, `Ort::Session`, `Ort::SessionOptions`, `Ort::Value`. Used by the `yolo_onnx` engine. +- **ZED SDK** (``): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`. - Output data is sent as raw byte slices (`rust::Slice`) with typed metadata (shape, dtype, encoding). ## Tools diff --git a/CMakeLists.txt b/CMakeLists.txt index 3baf07b..77fa223 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,5 +1,5 @@ cmake_minimum_required(VERSION 3.21) -project(dora_zed VERSION 0.1.0 LANGUAGES C CXX) +project(dora_zed VERSION 0.2.0 LANGUAGES C CXX) set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_FLAGS "-fPIC") @@ -19,18 +19,12 @@ find_package(zed REQUIRED) find_package(OpenCV REQUIRED) -find_package(onnxruntime REQUIRED) -set(ONNXRUNTIME_INCLUDE_DIRS "") # propagated via imported target -set(ONNXRUNTIME_LIBRARIES onnxruntime::onnxruntime) - link_directories(${dora_link_dirs}) link_directories(${ZED_LIBRARY_DIR}) add_executable(dora_zed src/main.cc src/zed/camera.cc - src/engines/zed_native.cc - src/engines/yolo_onnx.cc ${node_bridge} ) add_dependencies(dora_zed Dora_cxx) @@ -44,29 +38,8 @@ target_include_directories( ${ZED_INCLUDE_DIRS} ${OpenCV_INCLUDE_DIRS} ${CUDAToolkit_INCLUDE_DIRS} - ${ONNXRUNTIME_INCLUDE_DIRS} ) -target_link_libraries(dora_zed dora_node_api_cxx ${ZED_LIBRARIES} ${OpenCV_LIBS} ${ONNXRUNTIME_LIBRARIES}) +target_link_libraries(dora_zed dora_node_api_cxx ${ZED_LIBRARIES} ${OpenCV_LIBS}) install(TARGETS dora_zed DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/bin) - -# Standalone detector tests — no dora, no live camera required - -add_executable(test_yolo_onnx - src/test_yolo_onnx.cc - src/engines/yolo_onnx.cc -) -target_include_directories(test_yolo_onnx PRIVATE - ${CMAKE_CURRENT_SOURCE_DIR}/src - ${ZED_INCLUDE_DIRS} - ${OpenCV_INCLUDE_DIRS} - ${CUDAToolkit_INCLUDE_DIRS} - ${ONNXRUNTIME_INCLUDE_DIRS} -) -target_link_libraries(test_yolo_onnx PRIVATE - ${ZED_LIBRARIES} - ${OpenCV_LIBS} - ${ONNXRUNTIME_LIBRARIES} -) -install(TARGETS test_yolo_onnx DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/bin) diff --git a/Dockerfile b/Dockerfile index b7c0016..437d4bc 100644 --- a/Dockerfile +++ b/Dockerfile @@ -21,7 +21,6 @@ FROM nvidia/cuda:12.8.1-devel-ubuntu24.04 ARG ZED_VERSION=5.2 -ARG ORT_VERSION=1.24.4 ENV DEBIAN_FRONTEND=noninteractive @@ -51,24 +50,6 @@ RUN curl -fSL -o /tmp/ZED_SDK.run \ && chmod -R a+rX /usr/local/zed \ && rm -f /tmp/ZED_SDK.run -# The tarball puts its CMake package files under lib/, but onnxruntimeTargets.cmake -# resolves the library as /lib64. Ubuntu is multiarch, so CMake does not -# search lib64 for packages and ld.so does not search /usr/local/lib64 at all: -# keep the libraries in lib64, move only the package files to lib/cmake, and add -# lib64 to the linker path. -RUN curl -fSL -o /tmp/ort.tgz \ - "https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VERSION}/onnxruntime-linux-x64-gpu-${ORT_VERSION}.tgz" \ - && tar -xzf /tmp/ort.tgz -C /tmp \ - && mkdir -p /usr/local/include/onnxruntime /usr/local/lib64 \ - && cp -r /tmp/onnxruntime-linux-x64-gpu-${ORT_VERSION}/include/* /usr/local/include/onnxruntime/ \ - && cp -r /tmp/onnxruntime-linux-x64-gpu-${ORT_VERSION}/lib/* /usr/local/lib64/ \ - && mkdir -p /usr/local/lib/cmake \ - && mv /usr/local/lib64/cmake/onnxruntime /usr/local/lib/cmake/ \ - && rmdir /usr/local/lib64/cmake \ - && echo /usr/local/lib64 > /etc/ld.so.conf.d/onnxruntime.conf \ - && ldconfig \ - && rm -rf /tmp/ort.tgz /tmp/onnxruntime-linux-x64-gpu-${ORT_VERSION} - ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility,video WORKDIR /src diff --git a/README.md b/README.md index 8a5c825..80a5c08 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # dora-zed -Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections. +Dora node for ZED stereo cameras. Publishes BGR images, point clouds and camera intrinsics. ## Usage @@ -19,7 +19,6 @@ nodes: - image_bgr - point_cloud - camera_info - - objects ``` See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node. @@ -33,7 +32,6 @@ See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera | [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` | | [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK | | [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings | -| [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required at configure time; used at runtime only by `yolo_onnx` | ### Installation scripts @@ -43,9 +41,7 @@ See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera | NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` | | Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` | -> **Jetson:** No official ONNX Runtime GPU binary for ARM64. It is still needed to configure the build; only the `yolo_onnx` engine uses it at runtime. Requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt). -> -> **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime` +> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release). ## Local build @@ -65,7 +61,7 @@ bindings. ### Containerized build -[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK, ONNX Runtime and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed. +[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed. ```bash docker build -t dora-zed-build . @@ -106,75 +102,18 @@ All behavior is controlled via environment variables. | `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup | | `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) | -### Detection engine - -Set `ZED_ENGINE` to select the detection backend: - -| `ZED_ENGINE` | Custom model? | Classes | -|---|---|---| -| `native` (default) | No — ZED SDK built-in TensorRT | Fixed: 7 categories | -| `yolo_onnx` (alias: `onnx`) | Yes — any single-input YOLO ONNX (v5/v8/v11) | Fixed labels from file or env var | - -#### `native` engine - -Uses ZED SDK's built-in TensorRT models — no model files to download or configure. `ZED_DETECTION_MODEL` selects the speed/accuracy variant, not a file path. - -Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT. - -| Variable | Default | Description | -|---|---|---| -| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` | -| `ZED_DETECTION_CONFIDENCE` | `40` | Confidence threshold (0–100) | -| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking | - -#### `yolo_onnx` engine (closed-vocabulary) - -For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained). - -Export a YOLOv11 model once: - -```bash -uv run scripts/export_yolo.py --model n --output-dir models/ -``` - -Then run: - -```bash -ZED_ENGINE=yolo_onnx \ -ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \ -ZED_ONNX_LABELS_FILE=models/coco80.txt \ -dora start dataflow.yml -``` - -**Required:** - -| Variable | Description | -|---|---| -| `ZED_ONNX_MODEL_PATH` | Path to YOLO ONNX model | -| `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE` | Class names (inline CSV or one-per-line file) | - -**Optional:** - -| Variable | Default | Description | -|---|---|---| -| `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) | -| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold | -| `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) | -| `ZED_ONNX_OUTPUT_FORMAT` | `auto` | Output layout: `auto`, `nc_first`/`yolov8`/`yolov11`, `nc_last`, `nc_last_obj`/`yolov5`. `auto` detects from tensor shape. | - ### Point cloud | `ZED_PC_MODE` | Shape | Description | |---|---|---| | `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) | | `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) | -| `ROI` | `[N, 4]` | Valid points within detected bounding boxes only | ## Inputs | Input | Description | |---|---| -| `tick` | Triggers a camera grab and detection. Connect to `dora/timer/millis/N`. | +| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. | ## Outputs @@ -183,18 +122,3 @@ dora start dataflow.yml | `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, `timestamp_ns`. | | `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics. | | `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients. | -| `objects` | `float32 [N, 37]` | Detected objects. See below. | - -### `objects` layout - -Each of the N rows contains 37 floats: - -| Index | Field | Description | -|---|---|---| -| 0 | `label_id` | Class index | -| 1 | `confidence` | Score (0–100) | -| 2–4 | `x, y, z` | 3D center (mm) | -| 5–12 | `bb2d` | 2D bounding box: 4 corners × (x, y) px | -| 13–36 | `bb3d` | 3D bounding box: 8 corners × (x, y, z) mm | - -Metadata includes `engine`, `label_names`, `image_width`, `image_height`, and `timestamp_ns`. An empty array `[0, 37]` is sent when no objects are detected. diff --git a/scripts/export_yolo.py b/scripts/export_yolo.py deleted file mode 100644 index 8e1c0b3..0000000 --- a/scripts/export_yolo.py +++ /dev/null @@ -1,111 +0,0 @@ -# /// script -# requires-python = ">=3.10" -# dependencies = ["ultralytics>=8.3"] -# /// -"""Export a standard YOLOv11 model to ONNX for closed-vocabulary detection. - -The exported model has one input: - images (1, 3, imgsz, imgsz) float32 - -Class names are fixed at training time. Provide them at runtime via -ZED_ONNX_LABELS or ZED_ONNX_LABELS_FILE. - -Usage: - uv run scripts/export_yolo.py - uv run scripts/export_yolo.py --model s --output-dir models/ - uv run scripts/export_yolo.py --model m --imgsz 1280 -""" - -import argparse -import shutil -import sys -from pathlib import Path - -MODELS = { - "n": "yolo11n", - "s": "yolo11s", - "m": "yolo11m", - "l": "yolo11l", - "x": "yolo11x", -} - - -def parse_args(): - p = argparse.ArgumentParser( - description="Export YOLOv11 to ONNX for closed-vocabulary detection" - ) - p.add_argument( - "--model", "-m", - choices=list(MODELS), - default="n", - help="Model size: n (~6 MB), s (~18 MB), m (~38 MB), l (~49 MB), x (~109 MB). Default: n", - ) - p.add_argument( - "--imgsz", - type=int, - default=640, - help="Input image size in pixels (square). Default: 640", - ) - p.add_argument( - "--output-dir", "-o", - default=".", - help="Output directory for the .onnx file. Default: current directory", - ) - return p.parse_args() - - -def verify_output(onnx_path: Path): - try: - import onnx - model = onnx.load(str(onnx_path)) - inputs = [(i.name, [d.dim_value for d in i.type.tensor_type.shape.dim]) - for i in model.graph.input] - outputs = [(o.name, [d.dim_value for d in o.type.tensor_type.shape.dim]) - for o in model.graph.output] - print(f" Inputs: {inputs}") - print(f" Outputs: {outputs}") - except ImportError: - print(" (install 'onnx' to verify model structure)") - - -def main(): - args = parse_args() - - try: - from ultralytics import YOLO - except ImportError: - print("Error: ultralytics not installed. Run: uv run scripts/export_yolo.py", file=sys.stderr) - sys.exit(1) - - model_name = MODELS[args.model] - out_dir = Path(args.output_dir) - out_dir.mkdir(parents=True, exist_ok=True) - out_name = f"{model_name}_i{args.imgsz}.onnx" - dst = out_dir / out_name - - print(f"Model: {model_name}.pt") - print(f"Input size: {args.imgsz}") - print(f"Output: {dst}") - print() - - model = YOLO(f"{model_name}.pt") - exported = Path(model.export(format="onnx", imgsz=args.imgsz)) - - if exported.resolve() != dst.resolve(): - shutil.move(str(exported), str(dst)) - - print(f"\nVerifying exported model...") - verify_output(dst) - - print(f"\nExported: {dst.resolve()}") - print(f"\nRun with (COCO labels example):") - print(f" ZED_ENGINE=onnx \\") - print(f" ZED_ONNX_MODEL_PATH={dst.resolve()} \\") - print(f" ZED_ONNX_LABELS_FILE=models/coco80.txt \\") - print(f" dora start dataflow.yml") - print(f"\nOr with inline labels:") - print(f" ZED_ONNX_LABELS=\"person,car,chair\" \\") - - -if __name__ == "__main__": - main() diff --git a/scripts/install_linux_arm.sh b/scripts/install_linux_arm.sh index f6d6afd..14a9016 100755 --- a/scripts/install_linux_arm.sh +++ b/scripts/install_linux_arm.sh @@ -9,11 +9,6 @@ # JetPack 6.2.2 (L4T 36.5) # JetPack 7.0 (L4T 38.2) — Jetson Thor # JetPack 7.1 (L4T 38.4) -# -# ONNX Runtime: Microsoft does not publish official GPU binaries for ARM64. -# ONNX Runtime is still needed to configure the build; only the yolo_onnx -# engine uses it at runtime. Build it from source: -# https://onnxruntime.ai/docs/build/eps.html#tensorrt set -euo pipefail ZED_VERSION="5.2" @@ -63,13 +58,5 @@ else sudo "${TMP_DIR}/ZED_SDK.run" -- silent skip_python skip_tools fi -# --- ONNX Runtime --- -echo "" -echo ">>> ONNX Runtime (ARM64 GPU): no official pre-built binary available." -echo " Needed to configure the build; only yolo_onnx uses it at runtime." -echo " Build from source:" -echo " https://onnxruntime.ai/docs/build/eps.html#tensorrt" -echo "" - -echo ">>> Done. You can now run: mkdir -p build && cd build && cmake .. && make install" +echo ">>> Done. You can now run: scripts/build.sh" echo ">>> Downloaded files are in ${TMP_DIR}" diff --git a/scripts/install_linux_x64.sh b/scripts/install_linux_x64.sh index ff05d0a..ed1e03d 100755 --- a/scripts/install_linux_x64.sh +++ b/scripts/install_linux_x64.sh @@ -6,7 +6,6 @@ set -euo pipefail ZED_VERSION="5.2" -ORT_VERSION="1.24.4" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="${SCRIPT_DIR}/../tmp" @@ -34,32 +33,5 @@ else sudo "${TMP_DIR}/ZED_SDK.run" -- silent skip_python skip_tools fi -# --- ONNX Runtime (C++ GPU, installs to /usr/local) --- -if [ -f /usr/local/lib64/libonnxruntime.so ] && [ -d /usr/local/lib/cmake/onnxruntime ]; then - echo ">>> ONNX Runtime already installed — skipping." -else - echo ">>> Downloading ONNX Runtime ${ORT_VERSION} GPU..." - download \ - "https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VERSION}/onnxruntime-linux-x64-gpu-${ORT_VERSION}.tgz" \ - "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}.tgz" - echo ">>> Extracting..." - tar -xzf "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}.tgz" -C "${TMP_DIR}" - echo ">>> Installing ONNX Runtime to /usr/local..." - sudo mkdir -p /usr/local/include/onnxruntime - sudo cp -r "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}/include/"* /usr/local/include/onnxruntime/ - sudo mkdir -p /usr/local/lib64 - sudo cp -r "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}/lib/"* /usr/local/lib64/ - # onnxruntimeTargets.cmake resolves the library as /lib64, but Ubuntu - # is multiarch: CMake does not search lib64 for packages and ld.so does not - # search /usr/local/lib64 at all. Move only the package files to lib/cmake - # and put lib64 on the linker path. - sudo mkdir -p /usr/local/lib/cmake - sudo rm -rf /usr/local/lib/cmake/onnxruntime - sudo mv /usr/local/lib64/cmake/onnxruntime /usr/local/lib/cmake/ - sudo rmdir /usr/local/lib64/cmake - echo /usr/local/lib64 | sudo tee /etc/ld.so.conf.d/onnxruntime.conf > /dev/null - sudo ldconfig -fi - -echo ">>> Done. You can now run: mkdir -p build && cd build && cmake .. && make install" +echo ">>> Done. You can now run: scripts/build.sh" echo ">>> Downloaded files are in ${TMP_DIR}" diff --git a/scripts/install_windows.ps1 b/scripts/install_windows.ps1 index 8f8979e..5a10cec 100755 --- a/scripts/install_windows.ps1 +++ b/scripts/install_windows.ps1 @@ -10,10 +10,6 @@ $ErrorActionPreference = "Stop" $ZED_INSTALL_DIR = "C:\Program Files (x86)\ZED SDK" -$ORT_VERSION = "1.24.4" -$ORT_NAME = "onnxruntime-win-x64-gpu_cuda13-$ORT_VERSION" -$ORT_URL = "https://github.com/microsoft/onnxruntime/releases/download/v$ORT_VERSION/$ORT_NAME.zip" -$ORT_DEST = "C:\onnxruntime" # --- ZED SDK --- if (Test-Path $ZED_INSTALL_DIR) { @@ -25,23 +21,5 @@ if (Test-Path $ZED_INSTALL_DIR) { Write-Host "" } -# --- ONNX Runtime --- -if (Test-Path "$ORT_DEST\lib\onnxruntime.dll") { - Write-Host ">>> ONNX Runtime already installed at '$ORT_DEST' — skipping." -} else { - Write-Host ">>> Downloading ONNX Runtime $ORT_VERSION GPU (CUDA 13)..." - $tmp = "$env:TEMP\$ORT_NAME.zip" - Invoke-WebRequest -Uri $ORT_URL -OutFile $tmp - - Write-Host ">>> Extracting to $ORT_DEST..." - Expand-Archive -Path $tmp -DestinationPath $env:TEMP -Force - if (Test-Path $ORT_DEST) { Remove-Item $ORT_DEST -Recurse -Force } - Move-Item "$env:TEMP\$ORT_NAME" $ORT_DEST - Remove-Item $tmp - - Write-Host ">>> ONNX Runtime installed to $ORT_DEST" - Write-Host " Pass it to CMake: cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime" -} - Write-Host "" Write-Host ">>> Done." diff --git a/src/engines/engine.h b/src/engines/engine.h deleted file mode 100644 index eb3f07a..0000000 --- a/src/engines/engine.h +++ /dev/null @@ -1,27 +0,0 @@ -#pragma once - -#include -#include -#include - -namespace engines { - -struct DetectedObject { - int label_id; - float confidence; - float position[3]; // center x, y, z (mm) - float bounding_box_2d[8]; // 4 corners × 2 coords (pixels) - float bounding_box_3d[24]; // 8 corners × 3 coords (mm) -}; - -class Engine { -public: - virtual ~Engine() = default; - - virtual void enable(sl::Camera& camera) = 0; - virtual void detect(sl::Camera& camera) = 0; - virtual const std::vector& objects() const = 0; - virtual std::string label_names() const = 0; -}; - -} // namespace engines diff --git a/src/engines/yolo_helpers.h b/src/engines/yolo_helpers.h deleted file mode 100644 index 1f4645e..0000000 --- a/src/engines/yolo_helpers.h +++ /dev/null @@ -1,136 +0,0 @@ -#pragma once - -#include -#include - -#include -#include -#include - -namespace engines::detail { - -// --- NMS --- - -struct RawDetection { - float x1, y1, x2, y2; - float score; - int class_id; -}; - -inline float iou(const RawDetection& a, const RawDetection& b) { - float inter_x1 = std::max(a.x1, b.x1); - float inter_y1 = std::max(a.y1, b.y1); - float inter_x2 = std::min(a.x2, b.x2); - float inter_y2 = std::min(a.y2, b.y2); - float inter_w = std::max(0.0f, inter_x2 - inter_x1); - float inter_h = std::max(0.0f, inter_y2 - inter_y1); - float inter_area = inter_w * inter_h; - float area_a = (a.x2 - a.x1) * (a.y2 - a.y1); - float area_b = (b.x2 - b.x1) * (b.y2 - b.y1); - float union_area = area_a + area_b - inter_area; - return union_area > 0.0f ? inter_area / union_area : 0.0f; -} - -inline std::vector nms(std::vector dets, float threshold) { - std::sort(dets.begin(), dets.end(), - [](const RawDetection& a, const RawDetection& b) { - return a.score > b.score; - }); - std::vector suppressed(dets.size(), false); - std::vector result; - for (size_t i = 0; i < dets.size(); i++) { - if (suppressed[i]) continue; - result.push_back(dets[i]); - for (size_t j = i + 1; j < dets.size(); j++) { - if (!suppressed[j] && iou(dets[i], dets[j]) > threshold) - suppressed[j] = true; - } - } - return result; -} - -// --- Letterbox resize --- - -inline cv::Mat letterbox(const cv::Mat& src, int size, - float& scale, float& pad_x, float& pad_y) { - float scale_w = static_cast(size) / src.cols; - float scale_h = static_cast(size) / src.rows; - scale = std::min(scale_w, scale_h); - int new_w = static_cast(std::round(src.cols * scale)); - int new_h = static_cast(std::round(src.rows * scale)); - pad_x = (size - new_w) / 2.0f; - pad_y = (size - new_h) / 2.0f; - - cv::Mat resized; - cv::resize(src, resized, {new_w, new_h}); - - cv::Mat canvas(size, size, CV_8UC3, cv::Scalar(114, 114, 114)); - resized.copyTo(canvas(cv::Rect( - static_cast(pad_x), static_cast(pad_y), new_w, new_h))); - return canvas; -} - -// --- Point cloud fusion --- - -struct FusionResult { - float cx, cy, cz; - float depth_near, depth_far; - bool valid = false; -}; - -inline FusionResult fuse_bbox(const sl::Mat& pc, int x1, int y1, int x2, int y2) { - FusionResult result{}; - int w = static_cast(pc.getWidth()); - int h = static_cast(pc.getHeight()); - x1 = std::max(x1, 0); - y1 = std::max(y1, 0); - x2 = std::min(x2, w - 1); - y2 = std::min(y2, h - 1); - - std::vector xs, ys, zs; - xs.reserve((x2 - x1) * (y2 - y1)); - ys.reserve(xs.capacity()); - zs.reserve(xs.capacity()); - - for (int y = y1; y <= y2; y++) { - for (int x = x1; x <= x2; x++) { - sl::float4 pt; - pc.getValue(x, y, &pt); - if (!std::isnan(pt.x) && !std::isnan(pt.y) && !std::isnan(pt.z) && - !std::isinf(pt.x) && !std::isinf(pt.y) && !std::isinf(pt.z)) { - xs.push_back(pt.x); - ys.push_back(pt.y); - zs.push_back(pt.z); - } - } - } - - if (xs.empty()) return result; - - std::sort(xs.begin(), xs.end()); - std::sort(ys.begin(), ys.end()); - std::sort(zs.begin(), zs.end()); - - size_t n = xs.size(); - size_t mid = n / 2; - size_t p25 = n / 4; - size_t p75 = 3 * n / 4; - - result.cx = xs[mid]; - result.cy = ys[mid]; - result.cz = zs[mid]; - result.depth_near = zs[p25]; - result.depth_far = zs[p75]; - result.valid = true; - return result; -} - -inline void backproject(float u, float v, float z, - float fx, float fy, float cx, float cy, - float& X, float& Y, float& Z) { - Z = z; - X = (u - cx) * z / fx; - Y = (v - cy) * z / fy; -} - -} // namespace engines::detail diff --git a/src/engines/yolo_onnx.cc b/src/engines/yolo_onnx.cc deleted file mode 100644 index f170edd..0000000 --- a/src/engines/yolo_onnx.cc +++ /dev/null @@ -1,369 +0,0 @@ -#include "engines/yolo_onnx.h" -#include "engines/yolo_helpers.h" -#include "util/env.h" - -#include -#include -#include - -#include -#include -#include -#include -#include - -namespace engines { - -using namespace detail; - -// --- Impl --- - -struct YoloOnnx::Impl { - Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "yolo_onnx"}; - Ort::SessionOptions session_options; - std::unique_ptr session; - Ort::AllocatorWithDefaultOptions allocator; - - std::string input_name; - std::string output_name; - int num_classes = 0; - YoloOutputLayout resolved_layout = YoloOutputLayout::NC_FIRST; - - std::vector run_inference(const YoloOnnxConfig& config, - const cv::Mat& image_bgr); -}; - -// --- Impl::run_inference --- - -std::vector YoloOnnx::Impl::run_inference( - const YoloOnnxConfig& config, const cv::Mat& image_bgr) -{ - if (!session) return {}; - - int orig_w = image_bgr.cols; - int orig_h = image_bgr.rows; - - cv::Mat rgb; - cv::cvtColor(image_bgr, rgb, cv::COLOR_BGR2RGB); - - float scale, pad_x, pad_y; - cv::Mat lb = letterbox(rgb, config.input_size, scale, pad_x, pad_y); - - cv::Mat lb_float; - lb.convertTo(lb_float, CV_32F, 1.0f / 255.0f); - - int sz = config.input_size; - std::vector input_data(3 * sz * sz); - for (int c = 0; c < 3; c++) { - for (int h = 0; h < sz; h++) { - for (int w = 0; w < sz; w++) { - input_data[c * sz * sz + h * sz + w] = - lb_float.at(h, w)[c]; - } - } - } - - auto mem_info = Ort::MemoryInfo::CreateCpu( - OrtAllocatorType::OrtArenaAllocator, OrtMemTypeDefault); - - std::vector image_shape{1, 3, sz, sz}; - Ort::Value image_tensor = Ort::Value::CreateTensor( - mem_info, input_data.data(), input_data.size(), - image_shape.data(), image_shape.size()); - - const char* input_name_cstr = input_name.c_str(); - const char* output_name_cstr = output_name.c_str(); - - std::vector outputs; - try { - outputs = session->Run( - Ort::RunOptions{nullptr}, - &input_name_cstr, &image_tensor, 1, - &output_name_cstr, 1); - } catch (const Ort::Exception& e) { - std::cerr << "YoloOnnx: inference failed: " << e.what() << std::endl; - return {}; - } - - auto out_shape = outputs[0].GetTensorTypeAndShapeInfo().GetShape(); - if (out_shape.size() < 3) return {}; - - const float* out = outputs[0].GetTensorData(); - int nc = num_classes; - if (nc <= 0) return {}; - - std::vector candidates; - - if (resolved_layout == YoloOutputLayout::NC_FIRST) { - // (1, 4+N, anchors) — YOLOv8, YOLOv11 - int num_anchors = static_cast(out_shape[2]); - for (int a = 0; a < num_anchors; a++) { - float cx_lb = out[0 * num_anchors + a]; - float cy_lb = out[1 * num_anchors + a]; - float w_lb = out[2 * num_anchors + a]; - float h_lb = out[3 * num_anchors + a]; - - float best_score = 0.0f; - int best_class = 0; - for (int c = 0; c < nc; c++) { - float s = out[(4 + c) * num_anchors + a]; - if (s > best_score) { best_score = s; best_class = c; } - } - if (best_score < config.confidence_threshold) continue; - - float x1_lb = cx_lb - w_lb / 2.0f; - float y1_lb = cy_lb - h_lb / 2.0f; - float x2_lb = cx_lb + w_lb / 2.0f; - float y2_lb = cy_lb + h_lb / 2.0f; - - float x1 = std::max(0.0f, std::min((x1_lb - pad_x) / scale, static_cast(orig_w - 1))); - float y1 = std::max(0.0f, std::min((y1_lb - pad_y) / scale, static_cast(orig_h - 1))); - float x2 = std::max(0.0f, std::min((x2_lb - pad_x) / scale, static_cast(orig_w - 1))); - float y2 = std::max(0.0f, std::min((y2_lb - pad_y) / scale, static_cast(orig_h - 1))); - candidates.push_back({x1, y1, x2, y2, best_score, best_class}); - } - } else { - // (1, anchors, 4+N) or (1, anchors, 5+N) - int num_anchors = static_cast(out_shape[1]); - int row_stride = static_cast(out_shape[2]); - int cls_offset = (resolved_layout == YoloOutputLayout::NC_LAST_OBJ) ? 5 : 4; - - for (int a = 0; a < num_anchors; a++) { - const float* row = out + a * row_stride; - float cx_lb = row[0], cy_lb = row[1], w_lb = row[2], h_lb = row[3]; - float obj_conf = (resolved_layout == YoloOutputLayout::NC_LAST_OBJ) ? row[4] : 1.0f; - - float best_score = 0.0f; - int best_class = 0; - for (int c = 0; c < nc; c++) { - float s = obj_conf * row[cls_offset + c]; - if (s > best_score) { best_score = s; best_class = c; } - } - if (best_score < config.confidence_threshold) continue; - - float x1_lb = cx_lb - w_lb / 2.0f; - float y1_lb = cy_lb - h_lb / 2.0f; - float x2_lb = cx_lb + w_lb / 2.0f; - float y2_lb = cy_lb + h_lb / 2.0f; - - float x1 = std::max(0.0f, std::min((x1_lb - pad_x) / scale, static_cast(orig_w - 1))); - float y1 = std::max(0.0f, std::min((y1_lb - pad_y) / scale, static_cast(orig_h - 1))); - float x2 = std::max(0.0f, std::min((x2_lb - pad_x) / scale, static_cast(orig_w - 1))); - float y2 = std::max(0.0f, std::min((y2_lb - pad_y) / scale, static_cast(orig_h - 1))); - candidates.push_back({x1, y1, x2, y2, best_score, best_class}); - } - } - - return nms(std::move(candidates), config.nms_threshold); -} - -// --- Config --- - -YoloOnnxConfig YoloOnnxConfig::from_env() { - YoloOnnxConfig c; - c.model_path = util::get_env_str("ZED_ONNX_MODEL_PATH", c.model_path); - c.confidence_threshold = util::get_env_float("ZED_ONNX_CONFIDENCE", c.confidence_threshold); - c.nms_threshold = util::get_env_float("ZED_ONNX_NMS_THRESHOLD", c.nms_threshold); - c.input_size = util::get_env_int("ZED_ONNX_INPUT_SIZE", c.input_size); - c.labels = util::get_env_str("ZED_ONNX_LABELS", c.labels); - c.labels_file = util::get_env_str("ZED_ONNX_LABELS_FILE", c.labels_file); - - std::string fmt = util::get_env_str("ZED_ONNX_OUTPUT_FORMAT", ""); - if (fmt == "nc_first" || fmt == "yolov8" || fmt == "yolov11") - c.output_layout = YoloOutputLayout::NC_FIRST; - else if (fmt == "nc_last") - c.output_layout = YoloOutputLayout::NC_LAST; - else if (fmt == "nc_last_obj" || fmt == "yolov5") - c.output_layout = YoloOutputLayout::NC_LAST_OBJ; - - return c; -} - -// --- YoloOnnx --- - -YoloOnnx::YoloOnnx(const YoloOnnxConfig& config) - : config_(config), impl_(std::make_unique()) {} - -YoloOnnx::~YoloOnnx() = default; - -void YoloOnnx::enable(sl::Camera& /* camera */) { - if (config_.model_path.empty()) { - std::cerr << "YoloOnnx: ZED_ONNX_MODEL_PATH is not set." << std::endl; - return; - } - - OrtCUDAProviderOptions cuda_opts{}; - cuda_opts.device_id = 0; - try { - impl_->session_options.AppendExecutionProvider_CUDA(cuda_opts); - } catch (const Ort::Exception& e) { - std::cerr << "YoloOnnx: CUDA unavailable (" << e.what() << "), using CPU." << std::endl; - } - impl_->session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL); - - std::cout << "YoloOnnx: loading model (" << config_.model_path << ")..." << std::endl; - try { - impl_->session = std::make_unique( - impl_->env, config_.model_path.c_str(), impl_->session_options); - } catch (const Ort::Exception& e) { - std::cerr << "YoloOnnx: failed to load model: " << e.what() << std::endl; - return; - } - - impl_->input_name = impl_->session->GetInputNameAllocated(0, impl_->allocator).get(); - impl_->output_name = impl_->session->GetOutputNameAllocated(0, impl_->allocator).get(); - - size_t num_inputs = impl_->session->GetInputCount(); - if (num_inputs >= 2) { - std::cerr << "YoloOnnx: model has " << num_inputs << " inputs — " - "this engine only supports single-input models." - << std::endl; - } - - auto out_info = impl_->session->GetOutputTypeInfo(0).GetTensorTypeAndShapeInfo(); - auto shape = out_info.GetShape(); - if (shape.size() == 3) { - int64_t dim1 = shape[1]; - int64_t dim2 = shape[2]; - - YoloOutputLayout resolved; - if (config_.output_layout == YoloOutputLayout::AUTO) { - // dim1 < dim2 → NC_FIRST (e.g. 84 vs 8400 for YOLOv8/11) - // dim1 > dim2 → NC_LAST_OBJ (e.g. 25200 vs 85 for YOLOv5) - resolved = (dim1 < dim2) ? YoloOutputLayout::NC_FIRST - : YoloOutputLayout::NC_LAST_OBJ; - } else { - resolved = config_.output_layout; - } - impl_->resolved_layout = resolved; - - if (resolved == YoloOutputLayout::NC_FIRST) { - impl_->num_classes = static_cast(dim1) - 4; - } else if (resolved == YoloOutputLayout::NC_LAST) { - impl_->num_classes = static_cast(dim2) - 4; - } else { // NC_LAST_OBJ - impl_->num_classes = static_cast(dim2) - 5; - } - } - - if (!config_.labels_file.empty() && config_.labels.empty()) { - std::ifstream lf(config_.labels_file); - if (!lf) { - std::cerr << "YoloOnnx: cannot open ZED_ONNX_LABELS_FILE: " - << config_.labels_file << std::endl; - } else { - std::string line, joined; - while (std::getline(lf, line)) { - size_t l = line.find_first_not_of(" \t\r"); - size_t r = line.find_last_not_of(" \t\r"); - if (l == std::string::npos) continue; - if (!joined.empty()) joined += ','; - joined += line.substr(l, r - l + 1); - } - if (!joined.empty()) { - config_.labels = joined; - std::cout << "YoloOnnx: loaded labels from file (" - << config_.labels_file << ")" << std::endl; - } - } - } - - std::cout << "YoloOnnx: model loaded." - << " Classes=" << impl_->num_classes - << " Layout=" << (impl_->resolved_layout == YoloOutputLayout::NC_FIRST ? "NC_FIRST" - : impl_->resolved_layout == YoloOutputLayout::NC_LAST ? "NC_LAST" - : "NC_LAST_OBJ") - << std::endl; -} - -void YoloOnnx::detect(sl::Camera& camera) { - detected_.clear(); - if (!impl_->session) return; - - sl::Mat img_mat; - camera.retrieveImage(img_mat, sl::VIEW::LEFT); - - sl::Mat pc_mat; - camera.retrieveMeasure(pc_mat, sl::MEASURE::XYZRGBA); - - sl::CameraInformation info = camera.getCameraInformation(); - auto calib = info.camera_configuration.calibration_parameters.left_cam; - float fx = calib.fx, fy = calib.fy, cx = calib.cx, cy = calib.cy; - - int orig_w = static_cast(img_mat.getWidth()); - int orig_h = static_cast(img_mat.getHeight()); - - cv::Mat rgba(orig_h, orig_w, CV_8UC4, img_mat.getPtr()); - cv::Mat bgr; - cv::cvtColor(rgba, bgr, cv::COLOR_RGBA2BGR); - - auto kept = impl_->run_inference(config_, bgr); - - for (const auto& det : kept) { - FusionResult fusion = fuse_bbox(pc_mat, - static_cast(det.x1), static_cast(det.y1), - static_cast(det.x2), static_cast(det.y2)); - - DetectedObject obj{}; - obj.label_id = det.class_id; - obj.confidence = det.score * 100.0f; - - if (fusion.valid) { - obj.position[0] = fusion.cx; - obj.position[1] = fusion.cy; - obj.position[2] = fusion.cz; - } - - obj.bounding_box_2d[0] = det.x1; obj.bounding_box_2d[1] = det.y1; - obj.bounding_box_2d[2] = det.x2; obj.bounding_box_2d[3] = det.y1; - obj.bounding_box_2d[4] = det.x2; obj.bounding_box_2d[5] = det.y2; - obj.bounding_box_2d[6] = det.x1; obj.bounding_box_2d[7] = det.y2; - - if (fusion.valid) { - float corners_u[4] = {det.x1, det.x2, det.x2, det.x1}; - float corners_v[4] = {det.y1, det.y1, det.y2, det.y2}; - float depths[2] = {fusion.depth_near, fusion.depth_far}; - for (int face = 0; face < 2; face++) { - for (int c = 0; c < 4; c++) { - float X, Y, Z; - backproject(corners_u[c], corners_v[c], depths[face], - fx, fy, cx, cy, X, Y, Z); - int idx = (face * 4 + c) * 3; - obj.bounding_box_3d[idx + 0] = X; - obj.bounding_box_3d[idx + 1] = Y; - obj.bounding_box_3d[idx + 2] = Z; - } - } - } - - detected_.push_back(obj); - } -} - -void YoloOnnx::detect(const cv::Mat& image_bgr) { - detected_.clear(); - if (!impl_->session) return; - - auto kept = impl_->run_inference(config_, image_bgr); - - for (const auto& det : kept) { - DetectedObject obj{}; - obj.label_id = det.class_id; - obj.confidence = det.score * 100.0f; - obj.bounding_box_2d[0] = det.x1; obj.bounding_box_2d[1] = det.y1; - obj.bounding_box_2d[2] = det.x2; obj.bounding_box_2d[3] = det.y1; - obj.bounding_box_2d[4] = det.x2; obj.bounding_box_2d[5] = det.y2; - obj.bounding_box_2d[6] = det.x1; obj.bounding_box_2d[7] = det.y2; - detected_.push_back(obj); - } -} - -const std::vector& YoloOnnx::objects() const { - return detected_; -} - -std::string YoloOnnx::label_names() const { - return config_.labels; -} - -} // namespace engines diff --git a/src/engines/yolo_onnx.h b/src/engines/yolo_onnx.h deleted file mode 100644 index 8dec6b4..0000000 --- a/src/engines/yolo_onnx.h +++ /dev/null @@ -1,48 +0,0 @@ -#pragma once - -#include "engines/engine.h" - -#include -#include -#include - -namespace engines { - -enum class YoloOutputLayout { - AUTO, // detect from output tensor shape at load time - NC_FIRST, // (1, 4+N, anchors) — YOLOv8, YOLOv11 - NC_LAST, // (1, anchors, 4+N) — rows-first, no objectness - NC_LAST_OBJ // (1, anchors, 5+N) — YOLOv5, rows-first with objectness -}; - -struct YoloOnnxConfig { - std::string model_path; // ZED_ONNX_MODEL_PATH (required) - float confidence_threshold = 0.25f; // ZED_ONNX_CONFIDENCE - float nms_threshold = 0.45f; // ZED_ONNX_NMS_THRESHOLD - int input_size = 640; // ZED_ONNX_INPUT_SIZE - std::string labels; // ZED_ONNX_LABELS (comma-separated) - std::string labels_file; // ZED_ONNX_LABELS_FILE (one class name per line) - YoloOutputLayout output_layout = YoloOutputLayout::AUTO; - - static YoloOnnxConfig from_env(); -}; - -class YoloOnnx : public Engine { -public: - explicit YoloOnnx(const YoloOnnxConfig& config); - ~YoloOnnx() override; - - void enable(sl::Camera& camera) override; - void detect(sl::Camera& camera) override; - void detect(const cv::Mat& image_bgr); // image-only; no 3D fusion - const std::vector& objects() const override; - std::string label_names() const override; - -private: - YoloOnnxConfig config_; - struct Impl; - std::unique_ptr impl_; - std::vector detected_; -}; - -} // namespace engines diff --git a/src/engines/zed_native.cc b/src/engines/zed_native.cc deleted file mode 100644 index efe6600..0000000 --- a/src/engines/zed_native.cc +++ /dev/null @@ -1,99 +0,0 @@ -#include "engines/zed_native.h" -#include "util/env.h" - -#include -#include - -namespace { - -sl::OBJECT_DETECTION_MODEL parse_detection_model(const std::string& model) { - if (model == "MULTI_CLASS_BOX_FAST") { - return sl::OBJECT_DETECTION_MODEL::MULTI_CLASS_BOX_FAST; - } - if (model == "MULTI_CLASS_BOX_ACCURATE") { - return sl::OBJECT_DETECTION_MODEL::MULTI_CLASS_BOX_ACCURATE; - } - return sl::OBJECT_DETECTION_MODEL::MULTI_CLASS_BOX_MEDIUM; -} - -} // namespace - -namespace engines { - -DetectionConfig DetectionConfig::from_env() { - DetectionConfig c; - c.model = util::get_env_str("ZED_DETECTION_MODEL", c.model); - c.confidence_threshold = util::get_env_float("ZED_DETECTION_CONFIDENCE", c.confidence_threshold); - c.tracking = util::get_env_bool("ZED_DETECTION_TRACKING", c.tracking); - return c; -} - -ZedNative::ZedNative(const DetectionConfig& config) : config_(config) { - det_params_.detection_model = parse_detection_model(config_.model); - det_params_.enable_tracking = config_.tracking; - det_params_.enable_segmentation = false; - - det_rt_params_.detection_confidence_threshold = config_.confidence_threshold; -} - -void ZedNative::enable(sl::Camera& camera) { - if (config_.tracking) { - sl::PositionalTrackingParameters tracking_params; - sl::ERROR_CODE err = camera.enablePositionalTracking(tracking_params); - if (err != sl::ERROR_CODE::SUCCESS) { - std::cerr << "Failed to enable positional tracking: " - << sl::toString(err) << std::endl; - } - } - - std::cout << "Object Detection: loading model (" << config_.model << ")..." << std::endl; - sl::ERROR_CODE err = camera.enableObjectDetection(det_params_); - if (err != sl::ERROR_CODE::SUCCESS) { - std::cerr << "Failed to enable object detection: " - << sl::toString(err) << std::endl; - return; - } - std::cout << "Object Detection: model loaded." << std::endl; -} - -void ZedNative::detect(sl::Camera& camera) { - detected_.clear(); - - sl::ERROR_CODE err = camera.retrieveObjects(raw_objects_, det_rt_params_); - if (err != sl::ERROR_CODE::SUCCESS) { - return; - } - - for (const auto& obj : raw_objects_.object_list) { - DetectedObject det{}; - det.label_id = static_cast(obj.label); - det.confidence = obj.confidence; - - det.position[0] = obj.position.x; - det.position[1] = obj.position.y; - det.position[2] = obj.position.z; - - for (int i = 0; i < 4 && i < static_cast(obj.bounding_box_2d.size()); i++) { - det.bounding_box_2d[i * 2 + 0] = obj.bounding_box_2d[i].x; - det.bounding_box_2d[i * 2 + 1] = obj.bounding_box_2d[i].y; - } - - for (int i = 0; i < 8 && i < static_cast(obj.bounding_box.size()); i++) { - det.bounding_box_3d[i * 3 + 0] = obj.bounding_box[i].x; - det.bounding_box_3d[i * 3 + 1] = obj.bounding_box[i].y; - det.bounding_box_3d[i * 3 + 2] = obj.bounding_box[i].z; - } - - detected_.push_back(det); - } -} - -const std::vector& ZedNative::objects() const { - return detected_; -} - -std::string ZedNative::label_names() const { - return "PERSON,VEHICLE,BAG,ANIMAL,ELECTRONICS,FRUIT_VEGETABLE,SPORT"; -} - -} // namespace engines diff --git a/src/engines/zed_native.h b/src/engines/zed_native.h deleted file mode 100644 index 45df940..0000000 --- a/src/engines/zed_native.h +++ /dev/null @@ -1,34 +0,0 @@ -#pragma once - -#include "engines/engine.h" - -#include - -namespace engines { - -struct DetectionConfig { - std::string model = "MULTI_CLASS_BOX_MEDIUM"; - float confidence_threshold = 40.0f; - bool tracking = false; - - static DetectionConfig from_env(); -}; - -class ZedNative : public Engine { -public: - explicit ZedNative(const DetectionConfig& config); - - void enable(sl::Camera& camera) override; - void detect(sl::Camera& camera) override; - const std::vector& objects() const override; - std::string label_names() const override; - -private: - DetectionConfig config_; - sl::ObjectDetectionParameters det_params_; - sl::ObjectDetectionRuntimeParameters det_rt_params_; - sl::Objects raw_objects_; - std::vector detected_; -}; - -} // namespace engines diff --git a/src/main.cc b/src/main.cc index c06c814..8bb8600 100644 --- a/src/main.cc +++ b/src/main.cc @@ -1,6 +1,3 @@ -#include "engines/engine.h" -#include "engines/yolo_onnx.h" -#include "engines/zed_native.h" #include "util/env.h" #include "zed/camera.h" @@ -8,18 +5,14 @@ #include #include -#include #include #include -#include #include #include namespace { -constexpr int FIELDS_PER_OBJECT = 37; - -enum class PcMode { FULL, HALF, ROI }; +enum class PcMode { FULL, HALF }; rust::Vec make_shape(int height, int width, int channels) { rust::Vec shape; @@ -45,22 +38,6 @@ rust::Vec make_vector_int(std::initializer_list values) { return out; } -std::vector serialize_objects(const std::vector& objects) { - std::vector buffer(objects.size() * FIELDS_PER_OBJECT); - for (size_t i = 0; i < objects.size(); i++) { - float* row = buffer.data() + i * FIELDS_PER_OBJECT; - const auto& obj = objects[i]; - row[0] = static_cast(obj.label_id); - row[1] = obj.confidence; - row[2] = obj.position[0]; - row[3] = obj.position[1]; - row[4] = obj.position[2]; - std::memcpy(row + 5, obj.bounding_box_2d, sizeof(obj.bounding_box_2d)); - std::memcpy(row + 13, obj.bounding_box_3d, sizeof(obj.bounding_box_3d)); - } - return buffer; -} - } // namespace int main() { @@ -72,21 +49,8 @@ int main() { camera.warmup(); auto calib = camera.calibration(); - std::string engine_name = util::get_env_str("ZED_ENGINE", "native"); - std::unique_ptr detector; - if (engine_name == "yolo_onnx" || engine_name == "onnx") { - auto cfg = engines::YoloOnnxConfig::from_env(); - detector = std::make_unique(cfg); - } else { - auto cfg = engines::DetectionConfig::from_env(); - detector = std::make_unique(cfg); - } - detector->enable(camera.handle()); - std::string pc_mode_str = util::get_env_str("ZED_PC_MODE", "FULL"); - PcMode pc_mode = (pc_mode_str == "HALF") ? PcMode::HALF - : (pc_mode_str == "ROI") ? PcMode::ROI - : PcMode::FULL; + PcMode pc_mode = (pc_mode_str == "HALF") ? PcMode::HALF : PcMode::FULL; int64_t frame_id = 0; while (true) { @@ -106,12 +70,9 @@ int main() { camera.close(); camera.open(); calib = camera.calibration(); - detector->enable(camera.handle()); continue; } - detector->detect(camera.handle()); - sl::Mat& image_mat = camera.image(); sl::Mat& point_cloud_mat = camera.point_cloud(); @@ -185,30 +146,6 @@ int main() { pc_metadata->set_list_int("shape", make_shape(hh, hw, 4)); pc_data_ptr = reinterpret_cast(pc_buf.data()); pc_data_size = pc_buf.size() * sizeof(float); - } else if (pc_mode == PcMode::ROI) { - const auto& objs = detector->objects(); - for (const auto& obj : objs) { - int x1 = std::max(0, static_cast(obj.bounding_box_2d[0])); - int y1 = std::max(0, static_cast(obj.bounding_box_2d[1])); - int x2 = std::min(pc_width - 1, static_cast(obj.bounding_box_2d[4])); - int y2 = std::min(pc_height - 1, static_cast(obj.bounding_box_2d[5])); - for (int y = y1; y <= y2; y++) { - for (int x = x1; x <= x2; x++) { - sl::float4 pt; - point_cloud_mat.getValue(x, y, &pt); - if (!std::isnan(pt.x) && !std::isinf(pt.x)) { - pc_buf.push_back(pt.x); pc_buf.push_back(pt.y); - pc_buf.push_back(pt.z); pc_buf.push_back(pt.w); - } - } - } - } - int N = static_cast(pc_buf.size() / 4); - pc_metadata->set_string("layout", "NC"); - pc_metadata->set_string("mode", "roi"); - pc_metadata->set_list_int("shape", make_vector_int({N, 4})); - pc_data_ptr = reinterpret_cast(pc_buf.data()); - pc_data_size = pc_buf.size() * sizeof(float); } else { // FULL — send raw sl::Mat pointer directly (no copy) pc_data_size = static_cast(pc_width * pc_height * sizeof(sl::float4)); @@ -240,40 +177,6 @@ int main() { if (!std::string(cam_result.error).empty()) { std::cerr << "Error sending camera_info: " << std::string(cam_result.error) << std::endl; } - - // --- objects --- - const auto& objects = detector->objects(); - auto obj_buffer = serialize_objects(objects); - int num_objects = static_cast(objects.size()); - - const size_t obj_size = obj_buffer.size() * sizeof(float); - rust::Slice obj_slice{ - reinterpret_cast(obj_buffer.data()), obj_size}; - - auto obj_metadata = new_metadata(); - obj_metadata->set_string("dtype", "float32"); - obj_metadata->set_string("layout", "RC"); - obj_metadata->set_string("units", "mm"); - obj_metadata->set_list_int("shape", make_vector_int({num_objects, FIELDS_PER_OBJECT})); - obj_metadata->set_string("fields", - "label_id,confidence,x,y,z," - "bb2d_x0,bb2d_y0,bb2d_x1,bb2d_y1,bb2d_x2,bb2d_y2,bb2d_x3,bb2d_y3," - "bb3d_x0,bb3d_y0,bb3d_z0,bb3d_x1,bb3d_y1,bb3d_z1," - "bb3d_x2,bb3d_y2,bb3d_z2,bb3d_x3,bb3d_y3,bb3d_z3," - "bb3d_x4,bb3d_y4,bb3d_z4,bb3d_x5,bb3d_y5,bb3d_z5," - "bb3d_x6,bb3d_y6,bb3d_z6,bb3d_x7,bb3d_y7,bb3d_z7"); - obj_metadata->set_string("label_names", detector->label_names()); - obj_metadata->set_string("engine", engine_name); - obj_metadata->set_int("frame_id", current_frame_id); - obj_metadata->set_int("timestamp_ns", timestamp_ns); - obj_metadata->set_int("image_width", width); - obj_metadata->set_int("image_height", height); - - auto obj_result = send_output_with_metadata( - dora_node.send_output, "objects", obj_slice, std::move(obj_metadata)); - if (!std::string(obj_result.error).empty()) { - std::cerr << "Error sending objects: " << std::string(obj_result.error) << std::endl; - } } return 0; diff --git a/src/test_yolo_onnx.cc b/src/test_yolo_onnx.cc deleted file mode 100644 index e1439d1..0000000 --- a/src/test_yolo_onnx.cc +++ /dev/null @@ -1,113 +0,0 @@ -// Standalone test for the YoloOnnx (closed-vocabulary) detection engine. -// No dora framework, no live ZED camera — loads an image from disk. -// -// Usage: -// bin/test_yolo_onnx [image_path] -// image_path default: test/assets/yolo.png -// -// Required env vars: -// ZED_ONNX_MODEL_PATH path to YOLO ONNX model -// -// Optional env vars: -// ZED_ONNX_LABELS comma-separated class names -// ZED_ONNX_LABELS_FILE path to newline-separated class names file -// ZED_ONNX_CONFIDENCE detection threshold (default 0.25) -// ZED_ONNX_NMS_THRESHOLD NMS IoU threshold (default 0.45) -// ZED_ONNX_INPUT_SIZE model input size (default 640) -// ZED_ONNX_OUTPUT_FORMAT nc_first|nc_last|nc_last_obj|yolov5|yolov8|yolov11|auto - -#include "engines/yolo_onnx.h" -#include "util/env.h" - -#include -#include - -#include -#include -#include -#include -#include - -namespace fs = std::filesystem; - -namespace { - -std::vector parse_csv(const std::string& s) { - std::vector out; - std::istringstream ss(s); - std::string tok; - while (std::getline(ss, tok, ',')) { - size_t l = tok.find_first_not_of(" \t"); - size_t r = tok.find_last_not_of(" \t"); - if (l != std::string::npos) - out.push_back(tok.substr(l, r - l + 1)); - } - return out; -} - -void draw_detections(cv::Mat& img, - const std::vector& objects, - const std::vector& labels) { - for (const auto& obj : objects) { - int x1 = static_cast(obj.bounding_box_2d[0]); - int y1 = static_cast(obj.bounding_box_2d[1]); - int x2 = static_cast(obj.bounding_box_2d[4]); - int y2 = static_cast(obj.bounding_box_2d[5]); - - std::string name = (obj.label_id >= 0 && obj.label_id < static_cast(labels.size())) - ? labels[obj.label_id] : std::to_string(obj.label_id); - - char buf[64]; - std::snprintf(buf, sizeof(buf), " %.2f", obj.confidence / 100.0f); - - cv::rectangle(img, {x1, y1}, {x2, y2}, {0, 200, 0}, 2); - cv::putText(img, name + buf, {x1, std::max(y1 - 6, 14)}, - cv::FONT_HERSHEY_SIMPLEX, 0.55, {0, 200, 0}, 2, cv::LINE_AA); - } -} - -} // namespace - -int main(int argc, char* argv[]) { - std::string image_path = (argc > 1) ? argv[1] : "test/assets/yolo.png"; - - auto config = engines::YoloOnnxConfig::from_env(); - - sl::Camera dummy_cam; - engines::YoloOnnx engine(config); - engine.enable(dummy_cam); - - cv::Mat img = cv::imread(image_path); - if (img.empty()) { - std::cerr << "Error: cannot read image: " << image_path << std::endl; - return 1; - } - - engine.detect(img); - const auto& objects = engine.objects(); - auto labels = parse_csv(engine.label_names()); - - std::cout << "[" << image_path << "] " << objects.size() << " detection(s):" << std::endl; - for (const auto& obj : objects) { - std::string name = (obj.label_id >= 0 && obj.label_id < static_cast(labels.size())) - ? labels[obj.label_id] : std::to_string(obj.label_id); - std::cout << " " << name - << " conf=" << obj.confidence / 100.0f - << " bbox=[" - << obj.bounding_box_2d[0] << "," << obj.bounding_box_2d[1] - << " -> " - << obj.bounding_box_2d[4] << "," << obj.bounding_box_2d[5] - << "]" << std::endl; - } - - cv::Mat annotated = img.clone(); - draw_detections(annotated, objects, labels); - - fs::path stem = fs::path(image_path).stem(); - fs::path parent = fs::path(image_path).parent_path(); - std::string result_path = (parent / (stem.string() + "_result.png")).string(); - cv::imwrite(result_path, annotated); - std::cout << "Saved: " << result_path << std::endl; - - return 0; -} diff --git a/test/assets/.gitkeep b/test/assets/.gitkeep deleted file mode 100644 index e69de29..0000000