add a container build environment

Dockerfile provisions CUDA 12.8, ZED SDK 5.2, ONNX Runtime and Rust on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for, so the node can be compiled and the build verified without installing the SDK stack on the host. The compile needs --gpus all: libsl_zed.so links against the CUDA driver API, NVDEC and NVENC, which ship with the display driver rather than the CUDA toolkit and so exist in no image. The container runtime mounts them from the host, and only includes NVDEC/NVENC when the video capability is requested. Runs as the invoking user so build/ and bin/ are not left root-owned, which needs the toolchain outside /root and the ZED SDK directory readable to an arbitrary uid.
match the cargo profile to CMAKE_BUILD_TYPE
2026-09-07 14:33:58 -03:00 · 2026-09-07 14:33:50 -03:00 · 2026-09-07 14:33:50 -03:00 · 2026-09-07 14:33:33 -03:00 · 2026-09-07 14:30:01 -03:00 · 2026-09-07 14:29:52 -03:00
20 changed files with 965 additions and 1325 deletions
@@ -0,0 +1,5 @@
 .git/
 build/
 bin/
 tmp/
 models/
@@ -18,6 +18,8 @@ The binary is installed to `bin/dora_zed` relative to the repo root.
 The first build clones dora-rs from GitHub and compiles `dora-node-api-cxx` via Cargo. This takes a while. Subsequent builds are incremental. If `DORA_ROOT_DIR` is set to a local dora checkout, it builds against that instead.
 For a build without installing the SDK stack on the host, see the `Dockerfile` (Ubuntu 24.04 — the newest release with a ZED SDK; there is no Stereolabs build for 26.04). Run the compile with `--gpus all`.
 ## Run
 ```bash
@@ -34,20 +36,20 @@ Camera and detection behavior is configured entirely through environment variabl
 - **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace.
 - **`src/engines/engine.h`** — `engines::DetectedObject` struct and `engines::Engine` abstract base class (`enable`, `detect`, `objects`, `label_names`).
 - **`src/engines/zed_native.h/cc`** — `engines::ZedNative`: wraps ZED SDK's TensorRT-optimized object detection module.
- **`src/engines/open_vocab.h/cc`** — `engines::OpenVocab`: YOLO-World open-vocabulary detection via ONNX Runtime. Supports two modes: single-input (fixed vocab, legacy) and two-input (open-vocab, `images` + `text_feats`). In two-input mode, reads `ZED_ONNX_LABELS` on every tick; when the labels change, re-tokenizes and re-encodes via the CLIP text encoder ONNX session, then passes the resulting `(1, N, 512)` L2-normalized embeddings to the visual model. Preprocesses images (letterbox, normalize, CHW), runs inference, applies NMS, then fuses 2D detections with the ZED point cloud to compute 3D positions and bounding boxes.
+- **`src/engines/yolo_helpers.h`** — Header-only shared helpers used by the ONNX engine: `RawDetection`, `nms()`, `letterbox()`, `fuse_bbox()`, `backproject()`. Lives in `engines::detail` namespace.
- **`src/util/clip_tokenizer.h`** — Header-only C++ CLIP BPE tokenizer (no external deps). Implements the GPT-2 `bytes_to_unicode()` byte encoder + iterative BPE merge algorithm. Loads `clip_vocab.txt` and `clip_merges.txt` generated by `scripts/export_clip_text_encoder.py`. Returns 77 `int64` token IDs (BOS=49406, EOS=49407, padded with 0).
+- **`src/engines/yolo_onnx.h/cc`** — `engines::YoloOnnx` / `YoloOnnxConfig`: closed-vocabulary detection for any single-input YOLO ONNX model (YOLOv8, YOLOv11, YOLOv5, etc.). Auto-detects output layout (`NC_FIRST` / `NC_LAST` / `NC_LAST_OBJ`) from tensor shape; override with `ZED_ONNX_OUTPUT_FORMAT`. Labels loaded from `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE`.
- **`src/main.cc`** — Dora event loop. Selects engine via `ZED_ENGINE` env var, holds a `std::unique_ptr<engines::Engine>`. Parses `ZED_PC_MODE` (`FULL`/`HALF`/`ROI`) once before the loop. On `tick` input: calls `camera.grab()` and `detector->detect()`, then sends `image_bgr`, `point_cloud`, `camera_info`, and `objects` outputs. Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension; `ROI` emits only valid points within detected bounding boxes as a flat `[N, 4]` array. On `update_vocabulary` input: decodes payload as a UTF-8 label string and calls `detector->set_labels()` for immediate re-encoding. Handles reconnection on grab failure.
+- **`src/main.cc`** — Dora event loop. Selects engine via `ZED_ENGINE` env var (`yolo_onnx`/`onnx` or `native`), holds a `std::unique_ptr<engines::Engine>`. Parses `ZED_PC_MODE` (`FULL`/`HALF`/`ROI`) once before the loop. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()` and `detector->detect()`, then sends `image_bgr`, `point_cloud`, `camera_info`, and `objects` outputs. Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension; `ROI` emits only valid points within detected bounding boxes as a flat `[N, 4]` array. Handles reconnection on grab failure.
 - **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue.
 - **`dataflow.yml`** — Example dora dataflow definition using the `git:` source format.
- **`scripts/export_yoloworld.py`** — PEP 723 uv script. Exports a YOLOv8-worldv2 model to ONNX with N class slots (labels arbitrary at runtime). Run: `uv run scripts/export_yoloworld.py --model s --num-classes 10`.
+- **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, ONNX Runtime, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. ONNX Runtime's CMake package files are relocated to `/usr/local/lib/cmake` because Ubuntu's multiarch CMake does not search `lib64`. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`.
- **`scripts/export_clip_text_encoder.py`** — PEP 723 uv script. Exports CLIP ViT-B/32 text encoder to ONNX and saves `clip_vocab.txt` / `clip_merges.txt`. Run once; reused across all YOLO-World model sizes. Run: `uv run scripts/export_clip_text_encoder.py --output-dir models/`.
+- **`scripts/export_yolo.py`** — PEP 723 uv script. Exports a standard YOLOv11 model to ONNX for closed-vocabulary detection. Run: `uv run scripts/export_yolo.py --model n --output-dir models/`.
- **`src/test_open_vocab.cc`** — Standalone C++ test binary (`bin/test_open_vocab`). Exercises the `OpenVocab` engine with images from `test/assets/yolo.png` and `test/assets/yoloworld.png` (paths overridable via argv). No dora and no live camera required — `enable()` ignores the camera for this engine. Uses the same env vars as the main binary. Saves annotated result images alongside inputs.
+- **`src/test_yolo_onnx.cc`** — Standalone C++ test binary (`bin/test_yolo_onnx`). Exercises `YoloOnnx` with an image (default `test/assets/yolo.png`). No dora and no live camera required. Uses the same env vars as the main binary. Saves an annotated result image alongside the input.
 ## Key APIs
 - **dora C++ API** (`<dora-node-api.h>`): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`.
 - **ZED SDK** (`<sl/Camera.hpp>`): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`, `sl::ObjectDetectionParameters`, `sl::Objects`.
- **ONNX Runtime** (`<onnxruntime_cxx_api.h>`): `Ort::Env`, `Ort::Session`, `Ort::SessionOptions`, `Ort::Value`. Used by the `open_vocab` engine.
+- **ONNX Runtime** (`<onnxruntime_cxx_api.h>`): `Ort::Env`, `Ort::Session`, `Ort::SessionOptions`, `Ort::Value`. Used by the `yolo_onnx` engine.
 - Output data is sent as raw byte slices (`rust::Slice<const uint8_t>`) with typed metadata (shape, dtype, encoding).
 ## Tools
@@ -24,7 +24,7 @@ add_executable(dora_zed
    src/main.cc
    src/zed/camera.cc
    src/engines/zed_native.cc
-    src/engines/open_vocab.cc
+    src/engines/yolo_onnx.cc
    ${node_bridge}
 )
 add_dependencies(dora_zed Dora_cxx)
@@ -45,21 +45,22 @@ target_link_libraries(dora_zed dora_node_api_cxx ${ZED_LIBRARIES} ${OpenCV_LIBS}
 install(TARGETS dora_zed DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/bin)
-# Standalone detector test — no dora, no live camera required
+# Standalone detector tests — no dora, no live camera required
-add_executable(test_open_vocab
+
-    src/test_open_vocab.cc
+add_executable(test_yolo_onnx
-    src/engines/open_vocab.cc
+    src/test_yolo_onnx.cc
    src/engines/yolo_onnx.cc
 )
-target_include_directories(test_open_vocab PRIVATE
+target_include_directories(test_yolo_onnx PRIVATE
    ${CMAKE_CURRENT_SOURCE_DIR}/src
    ${ZED_INCLUDE_DIRS}
    ${OpenCV_INCLUDE_DIRS}
    ${CUDAToolkit_INCLUDE_DIRS}
    ${ONNXRUNTIME_INCLUDE_DIRS}
 )
-target_link_libraries(test_open_vocab PRIVATE
+target_link_libraries(test_yolo_onnx PRIVATE
    ${ZED_LIBRARIES}
    ${OpenCV_LIBS}
    ${ONNXRUNTIME_LIBRARIES}
 )
-install(TARGETS test_open_vocab DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/bin)
+install(TARGETS test_yolo_onnx DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/bin)
@@ -0,0 +1,75 @@
 # Reproducible build environment for dora-zed.
 #
 # Compiles only — no camera is needed. Requires an NVIDIA driver and the NVIDIA
 # Container Toolkit on the host, as the ZED SDK does anyway.
 #
 # NVIDIA_DRIVER_CAPABILITIES below includes `video` because libsl_zed.so links
 # against NVDEC (libnvcuvid) and NVENC (libnvidia-encode); the container runtime
 # mounts those from the host only when `video` is in the list.
 #
 # Ubuntu 24.04 is the newest release Stereolabs ships a ZED SDK for.
 #
 #   docker build -t dora-zed-build .
 #   docker run --rm --gpus all --user "$(id -u):$(id -g)" \
 #     -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
 #
 # --user keeps build/ and bin/ owned by you instead of root.
 #
 # The first run clones dora-rs and compiles it with Cargo, which takes a while;
 # the named volume keeps the registry cache across runs.
 FROM nvidia/cuda:12.8.1-devel-ubuntu24.04
 ARG ZED_VERSION=5.2
 ARG ORT_VERSION=1.24.4
 ENV DEBIAN_FRONTEND=noninteractive
 RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential cmake git curl ca-certificates pkg-config \
        libopencv-dev zstd udev sudo \
    && rm -rf /var/lib/apt/lists/*
 # The toolchain lives outside /root so the container can run as the invoking
 # user (--user) and still write the cargo caches. registry/ is created here so
 # that a named volume mounted over it inherits these permissions rather than
 # being created root-owned.
 ENV RUSTUP_HOME=/usr/local/rustup \
    CARGO_HOME=/usr/local/cargo \
    PATH=/usr/local/cargo/bin:${PATH}
 RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
        | sh -s -- -y --profile minimal --default-toolchain stable \
    && mkdir -p ${CARGO_HOME}/registry \
    && chmod -R a+w ${RUSTUP_HOME} ${CARGO_HOME}
 # The installer leaves /usr/local/zed as drwxrwx--- root:zed, which an arbitrary
 # --user uid cannot read, so relax it to make find_package(zed) work for one.
 RUN curl -fSL -o /tmp/ZED_SDK.run \
        "https://download.stereolabs.com/zedsdk/${ZED_VERSION}/cu12/ubuntu24" \
    && chmod +x /tmp/ZED_SDK.run \
    && /tmp/ZED_SDK.run -- silent skip_python skip_tools skip_drivers skip_cuda \
    && chmod -R a+rX /usr/local/zed \
    && rm -f /tmp/ZED_SDK.run
 # The tarball puts its CMake package files under lib/, but onnxruntimeTargets.cmake
 # resolves the library as <prefix>/lib64. Ubuntu is multiarch, so CMake does not
 # search lib64 for packages and ld.so does not search /usr/local/lib64 at all:
 # keep the libraries in lib64, move only the package files to lib/cmake, and add
 # lib64 to the linker path.
 RUN curl -fSL -o /tmp/ort.tgz \
        "https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VERSION}/onnxruntime-linux-x64-gpu-${ORT_VERSION}.tgz" \
    && tar -xzf /tmp/ort.tgz -C /tmp \
    && mkdir -p /usr/local/include/onnxruntime /usr/local/lib64 \
    && cp -r /tmp/onnxruntime-linux-x64-gpu-${ORT_VERSION}/include/* /usr/local/include/onnxruntime/ \
    && cp -r /tmp/onnxruntime-linux-x64-gpu-${ORT_VERSION}/lib/* /usr/local/lib64/ \
    && mkdir -p /usr/local/lib/cmake \
    && mv /usr/local/lib64/cmake/onnxruntime /usr/local/lib/cmake/ \
    && rmdir /usr/local/lib64/cmake \
    && echo /usr/local/lib64 > /etc/ld.so.conf.d/onnxruntime.conf \
    && ldconfig \
    && rm -rf /tmp/ort.tgz /tmp/onnxruntime-linux-x64-gpu-${ORT_VERSION}
 ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
 WORKDIR /src
 CMD ["bash", "-lc", "mkdir -p build && cd build && cmake .. && make -j$(nproc) install"]
@@ -5,6 +5,16 @@ set(dora_c_include_dir "${CMAKE_CURRENT_BINARY_DIR}/include/c")
 set(dora_cxx_include_dir "${CMAKE_CURRENT_BINARY_DIR}/include/cxx")
 set(node_bridge "${CMAKE_CURRENT_BINARY_DIR}/node_bridge.cc")
 # Match the cargo profile to the CMake build type, so a release build does not
 # link unoptimized dora bindings.
 if(CMAKE_BUILD_TYPE MATCHES "^(Release|RelWithDebInfo|MinSizeRel)$")
    set(_cargo_profile_flag --release)
    set(_cargo_target_subdir release)
 else()
    set(_cargo_profile_flag "")
    set(_cargo_target_subdir debug)
 endif()
 if(DORA_ROOT_DIR AND EXISTS "${DORA_ROOT_DIR}/Cargo.toml")
    include(ExternalProject)
    ExternalProject_Add(Dora
@@ -12,10 +22,10 @@ if(DORA_ROOT_DIR AND EXISTS "${DORA_ROOT_DIR}/Cargo.toml")
        BUILD_IN_SOURCE True
        CONFIGURE_COMMAND ""
        BUILD_COMMAND
-            cargo build
+            cargo build ${_cargo_profile_flag}
            --package dora-node-api-c
            &&
-            cargo build
+            cargo build ${_cargo_profile_flag}
            --package dora-node-api-cxx
        INSTALL_COMMAND ""
    )
@@ -37,39 +47,53 @@ if(DORA_ROOT_DIR AND EXISTS "${DORA_ROOT_DIR}/Cargo.toml")
    add_custom_target(Dora_c DEPENDS ${dora_c_include_dir})
    add_custom_target(Dora_cxx DEPENDS ${node_bridge} ${dora_cxx_include_dir})
-    set(dora_link_dirs ${DORA_ROOT_DIR}/target/debug)
+    set(dora_link_dirs ${DORA_ROOT_DIR}/target/${_cargo_target_subdir})
 else()
    include(ExternalProject)
-    set(DORA_GIT_TAG "main")
+
-    find_program(DORA_CLI dora)
+    # This node targets dora 1.0+. Set DORA_GIT_TAG to pin a tag explicitly;
-    if(DORA_CLI)
+    # left empty it follows the installed dora CLI so the bindings match the
-        execute_process(
+    # runtime, and falls back to the default below when no CLI is present.
-            COMMAND ${DORA_CLI} --version
+    set(DORA_GIT_TAG "" CACHE STRING "dora-rs git tag to build against (empty: follow the dora CLI)")
-            RESULT_VARIABLE _dora_version_result
+    set(_dora_tag "${DORA_GIT_TAG}")
-            OUTPUT_VARIABLE _dora_version_out
+
-            ERROR_VARIABLE _dora_version_err
+    if(NOT _dora_tag)
-            OUTPUT_STRIP_TRAILING_WHITESPACE
+        set(_dora_tag "v1.0.1")
-        )
+        find_program(DORA_CLI dora)
-        if(_dora_version_result EQUAL 0)
+        if(DORA_CLI)
-            string(REGEX MATCH "([0-9]+\\.[0-9]+\\.[0-9]+)" _dora_version_match "${_dora_version_out}")
+            execute_process(
-            if(_dora_version_match)
+                COMMAND ${DORA_CLI} --version
-                set(DORA_GIT_TAG "v${CMAKE_MATCH_1}")
+                RESULT_VARIABLE _dora_version_result
                OUTPUT_VARIABLE _dora_version_out
                ERROR_VARIABLE _dora_version_err
                OUTPUT_STRIP_TRAILING_WHITESPACE
            )
            if(_dora_version_result EQUAL 0)
                string(REGEX MATCH "([0-9]+\\.[0-9]+\\.[0-9]+)" _dora_version_match "${_dora_version_out}")
                if(_dora_version_match)
                    set(_dora_tag "v${CMAKE_MATCH_1}")
                    if(CMAKE_MATCH_1 VERSION_LESS 1.0)
                        message(WARNING "dora CLI ${CMAKE_MATCH_1} is older than the 1.0 this node targets")
                    endif()
                endif()
            endif()
        endif()
    endif()
    message(STATUS "dora-rs: building C++ bindings from ${_dora_tag}")
    ExternalProject_Add(Dora
        PREFIX ${CMAKE_CURRENT_BINARY_DIR}/dora
        GIT_REPOSITORY https://github.com/dora-rs/dora.git
-        GIT_TAG ${DORA_GIT_TAG}
+        GIT_TAG ${_dora_tag}
        BUILD_IN_SOURCE True
        CONFIGURE_COMMAND ""
        BUILD_COMMAND
-            cargo build
+            cargo build ${_cargo_profile_flag}
            --package dora-node-api-c
            --target-dir ${CMAKE_CURRENT_BINARY_DIR}/dora/src/Dora/target
            &&
-            cargo build
+            cargo build ${_cargo_profile_flag}
            --package dora-node-api-cxx
            --target-dir ${CMAKE_CURRENT_BINARY_DIR}/dora/src/Dora/target
        INSTALL_COMMAND ""
@@ -90,7 +114,7 @@ else()
            cp ../apis/c/node ${CMAKE_CURRENT_BINARY_DIR}/include/c -r
    )
-    set(dora_link_dirs ${CMAKE_CURRENT_BINARY_DIR}/dora/src/Dora/target/debug)
+    set(dora_link_dirs ${CMAKE_CURRENT_BINARY_DIR}/dora/src/Dora/target/${_cargo_target_subdir})
    add_custom_target(Dora_c DEPENDS ${dora_c_include_dir})
    add_custom_target(Dora_cxx DEPENDS ${node_bridge} ${dora_cxx_include_dir})
@@ -1,13 +1,16 @@
 # dora-zed
-Dora C++ node for ZED stereo cameras. Captures frames and publishes BGR images, XYZRGBA point clouds, camera intrinsics, and 3D object detections.
+Dora node for ZED stereo cameras. Publishes BGR images, point clouds, camera intrinsics, and 3D object detections.
-## Usage via `git:` in a dataflow
+## Usage
 Add to your dataflow:
 ```yaml
 nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    git_checkout: v0.1.0
    build: mkdir -p build && cd build && cmake .. && make install
    path: bin/dora_zed
    inputs:
@@ -19,180 +22,170 @@ nodes:
      - objects
 ```
-Then run:
+See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.
-```bash
+## Dependencies
-dora build dataflow.yml
+
-dora up
+| Dependency | Version | Notes |
-dora start dataflow.yml
+|---|---|---|
-```
+| [CMake](https://cmake.org/download/) | 3.21+ | |
 | [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
 | [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
 | [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
 | [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Required at configure time; used at runtime only by `yolo_onnx` |
 ### Installation scripts
 | Platform | Script |
 |---|---|
 | Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
 | NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
 | Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
 > **Jetson:** No official ONNX Runtime GPU binary for ARM64. It is still needed to configure the build; only the `yolo_onnx` engine uses it at runtime. Requires [building from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
 >
 > **Windows:** Install ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release), then pass the ONNX Runtime path: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
 ## Local build
 ```bash
 git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
 cd dora-zed
-mkdir -p build && cd build
+mkdir -p build && cd build && cmake .. && make install
 cmake ..
 make install
 ```
-The binary is installed to `bin/dora_zed` relative to the repository root.
+Binary installed to `bin/dora_zed`.
-## Dependencies
+### Containerized build
-| Dependency | Minimum version | Notes |
+[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK, ONNX Runtime and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.
 |---|---|---|
 | [CMake](https://cmake.org/download/) | 3.21 | |
 | [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Default path: `/usr/local/zed`; bundles OpenCV |
 | [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | Required by ZED SDK | |
 | [Rust toolchain](https://rustup.rs) | stable | Used to build dora-rs C++ bindings |
 | [ONNX Runtime](https://github.com/microsoft/onnxruntime/releases) | 1.20+ | Must be installed before building; required only for `open_vocab` engine |
 ## Installation scripts
 Helper scripts are in [`scripts/`](scripts/):
 | Platform | Script | Notes |
 |---|---|---|
 | Ubuntu 24.04 x86_64 | [`scripts/install_linux_x64.sh`](scripts/install_linux_x64.sh) | Installs ZED SDK + ONNX Runtime |
 | NVIDIA Jetson (ARM64) | [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) | Installs ZED SDK; auto-detects L4T version |
 | Windows x64 | [`scripts/install_windows.ps1`](scripts/install_windows.ps1) | Installs ONNX Runtime; ZED SDK downloaded manually |
 **Linux / Jetson:**
 ```bash
-chmod +x scripts/install_linux_x64.sh   # or install_linux_arm.sh
+docker build -t dora-zed-build .
-sudo scripts/install_linux_x64.sh
+docker run --rm --gpus all --user "$(id -u):$(id -g)" \
    -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
 ```
-**Windows** (elevated PowerShell):
+The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs.
 `--user` keeps `build/` and `bin/` owned by you rather than root.
-```powershell
+x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the
-Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
+device and build natively — there is no ARM64 image here.
 .\scripts\install_windows.ps1
 ```
-> **Jetson note:** No official ONNX Runtime GPU binary exists for ARM64. The `native` engine (ZED SDK built-in TensorRT) works without it. The `open_vocab` engine requires [building ONNX Runtime from source](https://onnxruntime.ai/docs/build/eps.html#tensorrt).
+Needs an NVIDIA driver and the
 [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
 on the host, as the ZED SDK does anyway. The image sets
 `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC
 (`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host
 only when `video` is in the capability list.
-> **Windows note:** ZED SDK must be downloaded and installed manually from [stereolabs.com/developers/release](https://www.stereolabs.com/developers/release). Pass the ONNX Runtime path to CMake: `cmake .. -DCMAKE_PREFIX_PATH=C:\onnxruntime`
+## Configuration
-## Environment variables
+All behavior is controlled via environment variables.
 ### Camera
 | Variable | Default | Description |
 |---|---|---|
-| `ZED_RESOLUTION` | `720` | Camera resolution: `720`, `1080`, or `2000` |
+| `ZED_RESOLUTION` | `720` | `720`, `1080`, or `2000` |
-| `ZED_FPS` | `15` | Camera frame rate |
+| `ZED_FPS` | `15` | Frame rate |
-| `ZED_DEPTH_MODE` | `NEURAL` | Depth mode: `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
+| `ZED_DEPTH_MODE` | `NEURAL` | `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
-| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth distance in millimeters |
+| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth (mm) |
-| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth distance in millimeters |
+| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth (mm) |
 | `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
-| `ZED_FLIP` | `ON` | Image flip: `ON` or `OFF` |
+| `ZED_FLIP` | `ON` | `ON` or `OFF` |
-| `ZED_WARMUP_FRAMES` | `30` | Number of warmup frames to discard |
+| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
-| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Seconds between reconnection attempts |
+| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |
-| `ZED_ENGINE` | `native` | Detection engine: `native` (ZED SDK) or `open_vocab` (YOLO-World via ONNX Runtime) |
+
-| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | (`native`) Detection model: `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
+### Detection engine
-| `ZED_DETECTION_CONFIDENCE` | `40` | (`native`) Detection confidence threshold (0-100) |
+
-| `ZED_DETECTION_TRACKING` | `false` | (`native`) Enable object tracking across frames |
+Set `ZED_ENGINE` to select the detection backend:
-| `ZED_ONNX_MODEL_PATH` | _(required)_ | (`open_vocab`) Path to YOLO-World ONNX model file |
+
-| `ZED_ONNX_LABELS` | _(required)_ | (`open_vocab`) Comma-separated label names (e.g. `person,cup,chair`). Can be changed between ticks — the node re-encodes embeddings automatically |
+| `ZED_ENGINE` | Custom model? | Classes |
-| `ZED_ONNX_CONFIDENCE` | `0.25` | (`open_vocab`) Detection confidence threshold (0.0-1.0) |
+|---|---|---|
-| `ZED_ONNX_NMS_THRESHOLD` | `0.45` | (`open_vocab`) NMS IoU threshold |
+| `native` (default) | No — ZED SDK built-in TensorRT | Fixed: 7 categories |
-| `ZED_ONNX_INPUT_SIZE` | `640` | (`open_vocab`) Model input image size in pixels |
+| `yolo_onnx` (alias: `onnx`) | Yes — any single-input YOLO ONNX (v5/v8/v11) | Fixed labels from file or env var |
-| `ZED_ONNX_TEXT_ENCODER_PATH` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_text_encoder.onnx`. Required when the visual model has two inputs (`images` + `text_feats`) |
+
-| `ZED_ONNX_CLIP_VOCAB` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_vocab.txt` BPE vocabulary file |
+#### `native` engine
-| `ZED_ONNX_CLIP_MERGES` | _(required for open-vocab)_ | (`open_vocab`) Path to `clip_merges.txt` BPE merge rules file |
+
-| `ZED_PC_MODE` | `FULL` | Point cloud output mode: `FULL` (full HxW grid), `HALF` (stride-2 downsample, ~4× smaller), or `ROI` (valid points within detected bounding boxes only, flat list) |
+Uses ZED SDK's built-in TensorRT models — no model files to download or configure. `ZED_DETECTION_MODEL` selects the speed/accuracy variant, not a file path.
 Detects 7 fixed categories: PERSON, VEHICLE, BAG, ANIMAL, ELECTRONICS, FRUIT_VEGETABLE, SPORT.
 | Variable | Default | Description |
 |---|---|---|
 | `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
 | `ZED_DETECTION_CONFIDENCE` | `40` | Confidence threshold (0–100) |
 | `ZED_DETECTION_TRACKING` | `false` | Enable object tracking |
 #### `yolo_onnx` engine (closed-vocabulary)
 For any single-input YOLO ONNX model with fixed class names (YOLOv8, YOLOv11, YOLOv5, custom-trained).
 Export a YOLOv11 model once:
 ```bash
 uv run scripts/export_yolo.py --model n --output-dir models/
 ```
 Then run:
 ```bash
 ZED_ENGINE=yolo_onnx \
 ZED_ONNX_MODEL_PATH=models/yolo11n_i640.onnx \
 ZED_ONNX_LABELS_FILE=models/coco80.txt \
 dora start dataflow.yml
 ```
 **Required:**
 | Variable | Description |
 |---|---|
 | `ZED_ONNX_MODEL_PATH` | Path to YOLO ONNX model |
 | `ZED_ONNX_LABELS` or `ZED_ONNX_LABELS_FILE` | Class names (inline CSV or one-per-line file) |
 **Optional:**
 | Variable | Default | Description |
 |---|---|---|
 | `ZED_ONNX_CONFIDENCE` | `0.25` | Confidence threshold (0.0–1.0) |
 | `ZED_ONNX_NMS_THRESHOLD` | `0.45` | NMS IoU threshold |
 | `ZED_ONNX_INPUT_SIZE` | `640` | Model input size (px) |
 | `ZED_ONNX_OUTPUT_FORMAT` | `auto` | Output layout: `auto`, `nc_first`/`yolov8`/`yolov11`, `nc_last`, `nc_last_obj`/`yolov5`. `auto` detects from tensor shape. |
 ### Point cloud
 | `ZED_PC_MODE` | Shape | Description |
 |---|---|---|
 | `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) |
 | `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
 | `ROI` | `[N, 4]` | Valid points within detected bounding boxes only |
 ## Inputs
 | Input | Description |
 |---|---|
-| `tick` | Required. Triggers a camera grab and detection on each message. Connect to `dora/timer/millis/N`. |
+| `tick` | Triggers a camera grab and detection. Connect to `dora/timer/millis/N`. |
 | `update_vocabulary` | Optional (`open_vocab` engine only). Payload: raw UTF-8 bytes of a comma-separated label string (e.g. `"person,cup,chair"`). Immediately re-encodes CLIP embeddings; the next `tick` uses the new labels. |
 ## Outputs
-### `image_bgr`
+| Output | Type | Description |
 |---|---|---|
 | `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, `timestamp_ns`. |
 | `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics. |
 | `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients. |
 | `objects` | `float32 [N, 37]` | Detected objects. See below. |
-BGR image as raw bytes. Metadata includes `shape` (HxWx3), `encoding` (`bgr8`), camera intrinsics (`fx`, `fy`, `cx`, `cy`), `distortion`, `frame_id`, and `timestamp_ns`.
+### `objects` layout
-### `point_cloud`
+Each of the N rows contains 37 floats:
 XYZRGBA point cloud as float32 bytes. Controlled by `ZED_PC_MODE`:
 | Mode | Shape | Layout | Description |
 |---|---|---|---|
 | `FULL` | `[H, W, 4]` | `HWC` | Full resolution grid (default). ~14.7 MB/frame at 720p |
 | `HALF` | `[H/2, W/2, 4]` | `HWC` | Stride-2 downsample (~4× smaller) |
 | `ROI` | `[N, 4]` | `NC` | Valid points within detected object bounding boxes only. `N` varies by scene. Metadata includes `mode=roi` |
 Metadata includes `units` (`mm`) and camera intrinsics.
 ### `camera_info`
 Four float32 values: `[fx, fy, cx, cy]`. Metadata includes `distortion` coefficients.
 ### `objects`
 Detected objects as a flat float32 array with shape `[num_objects, 37]`. Each row contains 37 floats:
 | Index | Field | Description |
 |---|---|---|
-| 0 | `label_id` | Object class (0=PERSON, 1=VEHICLE, 2=BAG, 3=ANIMAL, 4=ELECTRONICS, 5=FRUIT_VEGETABLE, 6=SPORT) |
+| 0 | `label_id` | Class index |
-| 1 | `confidence` | Detection confidence (0-100) |
+| 1 | `confidence` | Score (0–100) |
-| 2-4 | `x, y, z` | 3D center position in mm |
+| 2–4 | `x, y, z` | 3D center (mm) |
-| 5-12 | `bb2d_*` | 2D bounding box: 4 corners × 2 pixel coords |
+| 5–12 | `bb2d` | 2D bounding box: 4 corners × (x, y) px |
-| 13-36 | `bb3d_*` | 3D bounding box: 8 corners × 3 world coords (mm) |
+| 13–36 | `bb3d` | 3D bounding box: 8 corners × (x, y, z) mm |
-Metadata includes `frame_id` and `timestamp_ns` matching the corresponding `image_bgr`, `image_width` and `image_height` for interpreting 2D coordinates, `engine` (`native` or `open_vocab`), and `label_names` (comma-separated class names for the active engine). When no objects are detected, an empty array with shape `[0, 37]` is sent.
+Metadata includes `engine`, `label_names`, `image_width`, `image_height`, and `timestamp_ns`. An empty array `[0, 37]` is sent when no objects are detected.
 **`open_vocab` engine:** Uses a YOLO-World model exported with open-vocabulary support (two ONNX inputs: `images` and `text_feats`). Labels are set at runtime via `ZED_ONNX_LABELS` and re-encoded automatically on each tick when they change — no node restart required. Export the model and CLIP text encoder with the helper scripts:
 ```bash
 # Export the visual model (N=10 class slots, can be changed)
 uv run scripts/export_yoloworld.py --model s --num-classes 10 --output-dir models/
 # Export the CLIP text encoder + vocab files (once, reused for all model sizes)
 uv run scripts/export_clip_text_encoder.py --output-dir models/
 # Run
 ZED_ENGINE=open_vocab \
 ZED_ONNX_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
 ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
 ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
 ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
 ZED_ONNX_LABELS="person,car,bag,chair,laptop" \
 dora start dataflow.yml
 ```
 To verify the model works before connecting the camera, build and run the standalone C++ test binary (`bin/test_open_vocab`). It requires no live camera — images are loaded from disk:
 ```bash
 cd build && cmake .. && make test_open_vocab
 # Place test images:
 cp street.jpg test/assets/yolo.png
 cp custom.jpg test/assets/yoloworld.png
 # Fixed-vocab YOLO (both images use the same model):
 ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
 ZED_ONNX_LABELS="person,car,bus,bicycle" \
 bin/test_open_vocab
 # Open-vocab YOLO-World (separate model for second image via ZED_ONNX_WORLD_MODEL_PATH):
 ZED_ONNX_MODEL_PATH=models/yolov8s_n80_i640.onnx \
 ZED_ONNX_WORLD_MODEL_PATH=models/yolov8s-worldv2_n10_i640.onnx \
 ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \
 ZED_ONNX_CLIP_VOCAB=models/clip_vocab.txt \
 ZED_ONNX_CLIP_MERGES=models/clip_merges.txt \
 ZED_ONNX_LABELS="person,car,bus,bicycle" \
 bin/test_open_vocab
 # Override image paths:
 bin/test_open_vocab /path/to/img1.jpg /path/to/img2.jpg
 ```
 Annotated results are saved alongside the input images as `yolo_result.png` and `yoloworld_result.png`.
 3D positions are computed by fusing the 2D detections with the ZED point cloud (median of valid depth samples within each bounding box). 3D bounding box corners are back-projected at the p25/p75 depth percentiles.
@@ -1,13 +1,18 @@
 nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    git_checkout: v0.1.0
    build: mkdir -p build && cd build && cmake .. && make install
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
      # update_vocabulary: vocabulary-node/labels  # optional: send new comma-separated labels at runtime
    outputs:
      - image_bgr
      - point_cloud
      - camera_info
      - objects
  - id: plot
    git: https://github.com/dora-rs/dora-hub.git
    build: pip install ./node-hub/opencv-plot
    path: opencv-plot
    inputs:
      image: zed-camera/image_bgr
@@ -1,105 +0,0 @@
 # /// script
 # requires-python = ">=3.10"
 # dependencies = ["transformers>=4.30", "torch", "onnx"]
 # ///
 """Export the CLIP ViT-B/32 text encoder to ONNX and save vocab files.
 Compatible with all YOLOv8-worldv2 sizes (s/m/l/x) — run once and reuse.
 The C++ node uses these three files to compute text embeddings at runtime:
  clip_text_encoder.onnx  input:  input_ids  (1, 77) int64
                          output: text_embeds (1, 512) float32  (L2-normalize in C++)
  clip_vocab.txt          token → id mapping (tab-separated, one per line)
  clip_merges.txt         BPE merge rules (space-separated pairs, one per line)
 Usage:
    uv run scripts/export_clip_text_encoder.py
    uv run scripts/export_clip_text_encoder.py --output-dir models/
 """
 import argparse
 import sys
 from pathlib import Path
 CLIP_MODEL = "openai/clip-vit-base-patch32"
 def parse_args():
    p = argparse.ArgumentParser(description="Export CLIP ViT-B/32 text encoder to ONNX")
    p.add_argument(
        "--output-dir", "-o",
        default="models",
        help="Directory for output files. Default: models/",
    )
    return p.parse_args()
 def main():
    args = parse_args()
    try:
        import torch
        from transformers import CLIPTextModelWithProjection, CLIPTokenizer
    except ImportError as e:
        print(f"Error: {e}\nRun: uv run scripts/export_clip_text_encoder.py", file=sys.stderr)
        sys.exit(1)
    out = Path(args.output_dir)
    out.mkdir(parents=True, exist_ok=True)
    print(f"Loading {CLIP_MODEL} ...")
    model = CLIPTextModelWithProjection.from_pretrained(CLIP_MODEL)
    tokenizer = CLIPTokenizer.from_pretrained(CLIP_MODEL)
    model.eval()
    # --- Export ONNX text encoder ---
    # Input:  input_ids   (1, 77) int64   — BPE token IDs, padded/truncated to 77
    # Output: text_embeds (1, 512) float32 — projected features (NOT normalized; C++ normalizes)
    onnx_path = out / "clip_text_encoder.onnx"
    print(f"Exporting ONNX → {onnx_path} ...")
    dummy = torch.zeros(1, 77, dtype=torch.long)
    torch.onnx.export(
        model,
        dummy,
        str(onnx_path),
        input_names=["input_ids"],
        output_names=["text_embeds"],
        dynamic_axes={
            "input_ids":   {0: "batch_size"},
            "text_embeds": {0: "batch_size"},
        },
        opset_version=14,
    )
    # --- Save vocab as tab-separated text (token\tid) ---
    # Avoids JSON parsing in C++. Tokens with special chars are escaped.
    vocab_path = out / "clip_vocab.txt"
    print(f"Saving vocab  → {vocab_path} ...")
    vocab = tokenizer.get_vocab()
    with open(vocab_path, "w", encoding="utf-8") as f:
        for token, idx in sorted(vocab.items(), key=lambda x: x[1]):
            token_esc = (token
                         .replace("\\", "\\\\")
                         .replace("\t", "\\t")
                         .replace("\n", "\\n"))
            f.write(f"{token_esc}\t{idx}\n")
    # --- Save BPE merge rules ---
    merges_path = out / "clip_merges.txt"
    print(f"Saving merges → {merges_path} ...")
    with open(merges_path, "w", encoding="utf-8") as f:
        for (a, b) in tokenizer.bpe_ranks.keys():
            f.write(f"{a} {b}\n")
    print(f"\nDone. Files written to {out.resolve()}/")
    print(f"  clip_text_encoder.onnx  — ONNX model")
    print(f"  clip_vocab.txt          — {len(vocab)} tokens")
    print(f"  clip_merges.txt         — {len(tokenizer.bpe_ranks)} merge rules")
    print(f"\nSet env vars:")
    print(f"  ZED_ONNX_TEXT_ENCODER_PATH={out.resolve()}/clip_text_encoder.onnx")
    print(f"  ZED_ONNX_CLIP_VOCAB={out.resolve()}/clip_vocab.txt")
    print(f"  ZED_ONNX_CLIP_MERGES={out.resolve()}/clip_merges.txt")
 if __name__ == "__main__":
    main()
@@ -0,0 +1,111 @@
 # /// script
 # requires-python = ">=3.10"
 # dependencies = ["ultralytics>=8.3"]
 # ///
 """Export a standard YOLOv11 model to ONNX for closed-vocabulary detection.
 The exported model has one input:
  images  (1, 3, imgsz, imgsz)  float32
 Class names are fixed at training time. Provide them at runtime via
 ZED_ONNX_LABELS or ZED_ONNX_LABELS_FILE.
 Usage:
    uv run scripts/export_yolo.py
    uv run scripts/export_yolo.py --model s --output-dir models/
    uv run scripts/export_yolo.py --model m --imgsz 1280
 """
 import argparse
 import shutil
 import sys
 from pathlib import Path
 MODELS = {
    "n": "yolo11n",
    "s": "yolo11s",
    "m": "yolo11m",
    "l": "yolo11l",
    "x": "yolo11x",
 }
 def parse_args():
    p = argparse.ArgumentParser(
        description="Export YOLOv11 to ONNX for closed-vocabulary detection"
    )
    p.add_argument(
        "--model", "-m",
        choices=list(MODELS),
        default="n",
        help="Model size: n (~6 MB), s (~18 MB), m (~38 MB), l (~49 MB), x (~109 MB). Default: n",
    )
    p.add_argument(
        "--imgsz",
        type=int,
        default=640,
        help="Input image size in pixels (square). Default: 640",
    )
    p.add_argument(
        "--output-dir", "-o",
        default=".",
        help="Output directory for the .onnx file. Default: current directory",
    )
    return p.parse_args()
 def verify_output(onnx_path: Path):
    try:
        import onnx
        model = onnx.load(str(onnx_path))
        inputs  = [(i.name, [d.dim_value for d in i.type.tensor_type.shape.dim])
                   for i in model.graph.input]
        outputs = [(o.name, [d.dim_value for d in o.type.tensor_type.shape.dim])
                   for o in model.graph.output]
        print(f"  Inputs:  {inputs}")
        print(f"  Outputs: {outputs}")
    except ImportError:
        print("  (install 'onnx' to verify model structure)")
 def main():
    args = parse_args()
    try:
        from ultralytics import YOLO
    except ImportError:
        print("Error: ultralytics not installed. Run: uv run scripts/export_yolo.py", file=sys.stderr)
        sys.exit(1)
    model_name = MODELS[args.model]
    out_dir = Path(args.output_dir)
    out_dir.mkdir(parents=True, exist_ok=True)
    out_name = f"{model_name}_i{args.imgsz}.onnx"
    dst = out_dir / out_name
    print(f"Model:      {model_name}.pt")
    print(f"Input size: {args.imgsz}")
    print(f"Output:     {dst}")
    print()
    model = YOLO(f"{model_name}.pt")
    exported = Path(model.export(format="onnx", imgsz=args.imgsz))
    if exported.resolve() != dst.resolve():
        shutil.move(str(exported), str(dst))
    print(f"\nVerifying exported model...")
    verify_output(dst)
    print(f"\nExported: {dst.resolve()}")
    print(f"\nRun with (COCO labels example):")
    print(f"  ZED_ENGINE=onnx \\")
    print(f"  ZED_ONNX_MODEL_PATH={dst.resolve()} \\")
    print(f"  ZED_ONNX_LABELS_FILE=models/coco80.txt \\")
    print(f"  dora start dataflow.yml")
    print(f"\nOr with inline labels:")
    print(f"  ZED_ONNX_LABELS=\"person,car,chair\" \\")
 if __name__ == "__main__":
    main()
@@ -1,124 +0,0 @@
 # /// script
 # requires-python = ">=3.10"
 # dependencies = ["ultralytics>=8.0"]
 # ///
 """Export a YOLOv8-worldv2 model to ONNX with N open-vocabulary class slots.
 The exported model has two inputs:
  images     (1, 3, imgsz, imgsz)  float32
  text_feats (1, N, 512)           float32  ← CLIP embeddings, set at runtime
 N is fixed per export. Use export_clip_text_encoder.py to generate the text
 encoder and vocab files used to compute embeddings at runtime in the C++ node.
 Usage:
    uv run scripts/export_yoloworld.py
    uv run scripts/export_yoloworld.py --model m --num-classes 20
    uv run scripts/export_yoloworld.py --model l --num-classes 5 --output-dir models/
 """
 import argparse
 import shutil
 import sys
 from pathlib import Path
 MODELS = {
    "s": "yolov8s-worldv2",
    "m": "yolov8m-worldv2",
    "l": "yolov8l-worldv2",
    "x": "yolov8x-worldv2",
 }
 def parse_args():
    p = argparse.ArgumentParser(
        description="Export YOLOv8-worldv2 to ONNX with open-vocabulary text inputs"
    )
    p.add_argument(
        "--model", "-m",
        choices=list(MODELS),
        default="s",
        help="Model size: s (~14 MB), m (~30 MB), l (~50 MB), x (~100 MB). Default: s",
    )
    p.add_argument(
        "--num-classes", "-n",
        type=int,
        default=10,
        help="Number of concurrent class slots (must match number of labels at runtime). Default: 10",
    )
    p.add_argument(
        "--imgsz",
        type=int,
        default=640,
        help="Input image size in pixels (square). Default: 640",
    )
    p.add_argument(
        "--output-dir", "-o",
        default=".",
        help="Output directory for the .onnx file. Default: current directory",
    )
    return p.parse_args()
 def verify_inputs(onnx_path: Path, expected_inputs: int):
    try:
        import onnx
        model = onnx.load(str(onnx_path))
        actual = len(model.graph.input)
        if actual != expected_inputs:
            print(f"\nWarning: expected {expected_inputs} inputs, got {actual}.")
            print("The visual model may have baked text embeddings (ultralytics version dependent).")
            print("In that case, use a fixed vocabulary instead of the text encoder.")
        else:
            inputs = [(i.name, [d.dim_value for d in i.type.tensor_type.shape.dim])
                      for i in model.graph.input]
            print(f"  Inputs: {inputs}")
    except ImportError:
        print("  (install 'onnx' to verify model inputs)")
 def main():
    args = parse_args()
    try:
        from ultralytics import YOLO
    except ImportError:
        print("Error: ultralytics not installed. Run: uv run scripts/export_yoloworld.py", file=sys.stderr)
        sys.exit(1)
    model_name = MODELS[args.model]
    out_dir = Path(args.output_dir)
    out_dir.mkdir(parents=True, exist_ok=True)
    out_name = f"{model_name}_n{args.num_classes}_i{args.imgsz}.onnx"
    dst = out_dir / out_name
    print(f"Model:       {model_name}.pt")
    print(f"Class slots: {args.num_classes}")
    print(f"Input size:  {args.imgsz}")
    print(f"Output:      {dst}")
    print()
    model = YOLO(f"{model_name}.pt")
    # Set N dummy class slots — actual label meanings are provided at runtime via text_feats
    model.set_classes(["object"] * args.num_classes)
    exported = Path(model.export(format="onnx", imgsz=args.imgsz))
    if exported.resolve() != dst.resolve():
        shutil.move(str(exported), str(dst))
    print(f"\nVerifying exported model...")
    verify_inputs(dst, expected_inputs=2)
    print(f"\nExported: {dst.resolve()}")
    print(f"\nNext step — export the CLIP text encoder (once, reused for all model sizes):")
    print(f"  uv run scripts/export_clip_text_encoder.py --output-dir models/")
    print(f"\nRun with:")
    print(f"  ZED_ENGINE=open_vocab \\")
    print(f"  ZED_ONNX_MODEL_PATH={dst.resolve()} \\")
    print(f"  ZED_ONNX_TEXT_ENCODER_PATH=models/clip_text_encoder.onnx \\")
    print(f'  ZED_ONNX_LABELS="person,car,bag,...  (exactly {args.num_classes} labels)" \\')
    print(f"  dora start dataflow.yml")
 if __name__ == "__main__":
    main()
@@ -11,8 +11,8 @@
 #   JetPack 7.1     (L4T 38.4)
 #
 # ONNX Runtime: Microsoft does not publish official GPU binaries for ARM64.
-#   The native engine (ZED SDK built-in TensorRT) works without ONNX Runtime.
+#   ONNX Runtime is still needed to configure the build; only the yolo_onnx
-#   To use the open_vocab engine, build ONNX Runtime from source:
+#   engine uses it at runtime. Build it from source:
 #   https://onnxruntime.ai/docs/build/eps.html#tensorrt
 set -euo pipefail
@@ -66,8 +66,8 @@ fi
 # --- ONNX Runtime ---
 echo ""
 echo ">>> ONNX Runtime (ARM64 GPU): no official pre-built binary available."
-echo "    The native ZED engine works without ONNX Runtime."
+echo "    Needed to configure the build; only yolo_onnx uses it at runtime."
-echo "    To use the open_vocab engine, build from source:"
+echo "    Build from source:"
 echo "      https://onnxruntime.ai/docs/build/eps.html#tensorrt"
 echo ""
@@ -35,7 +35,7 @@ else
 fi
 # --- ONNX Runtime (C++ GPU, installs to /usr/local) ---
-if [ -f /usr/local/lib64/libonnxruntime.so ] && [ -d /usr/local/include/onnxruntime ]; then
+if [ -f /usr/local/lib64/libonnxruntime.so ] && [ -d /usr/local/lib/cmake/onnxruntime ]; then
    echo ">>> ONNX Runtime already installed — skipping."
 else
    echo ">>> Downloading ONNX Runtime ${ORT_VERSION} GPU..."
@@ -49,6 +49,15 @@ else
    sudo cp -r "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}/include/"* /usr/local/include/onnxruntime/
    sudo mkdir -p /usr/local/lib64
    sudo cp -r "${TMP_DIR}/onnxruntime-linux-x64-gpu-${ORT_VERSION}/lib/"* /usr/local/lib64/
    # onnxruntimeTargets.cmake resolves the library as <prefix>/lib64, but Ubuntu
    # is multiarch: CMake does not search lib64 for packages and ld.so does not
    # search /usr/local/lib64 at all. Move only the package files to lib/cmake
    # and put lib64 on the linker path.
    sudo mkdir -p /usr/local/lib/cmake
    sudo rm -rf /usr/local/lib/cmake/onnxruntime
    sudo mv /usr/local/lib64/cmake/onnxruntime /usr/local/lib/cmake/
    sudo rmdir /usr/local/lib64/cmake
    echo /usr/local/lib64 | sudo tee /etc/ld.so.conf.d/onnxruntime.conf > /dev/null
    sudo ldconfig
 fi
@@ -20,7 +20,6 @@ public:
    virtual void enable(sl::Camera& camera) = 0;
    virtual void detect(sl::Camera& camera) = 0;
    virtual void set_labels(const std::string& /*labels*/) {}
    virtual const std::vector<DetectedObject>& objects() const = 0;
    virtual std::string label_names() const = 0;
 };
@@ -1,592 +0,0 @@
 #include "engines/open_vocab.h"
 #include "util/clip_tokenizer.h"
 #include "util/env.h"
 #include <onnxruntime_cxx_api.h>
 #include <opencv2/opencv.hpp>
 #include <sl/Camera.hpp>
 #include <algorithm>
 #include <cmath>
 #include <iostream>
 #include <numeric>
 #include <sstream>
 #include <string>
 #include <vector>
 namespace {
 // --- NMS helpers ---
 struct RawDetection {
    float x1, y1, x2, y2;
    float score;
    int class_id;
 };
 float iou(const RawDetection& a, const RawDetection& b) {
    float inter_x1 = std::max(a.x1, b.x1);
    float inter_y1 = std::max(a.y1, b.y1);
    float inter_x2 = std::min(a.x2, b.x2);
    float inter_y2 = std::min(a.y2, b.y2);
    float inter_w = std::max(0.0f, inter_x2 - inter_x1);
    float inter_h = std::max(0.0f, inter_y2 - inter_y1);
    float inter_area = inter_w * inter_h;
    float area_a = (a.x2 - a.x1) * (a.y2 - a.y1);
    float area_b = (b.x2 - b.x1) * (b.y2 - b.y1);
    float union_area = area_a + area_b - inter_area;
    return union_area > 0.0f ? inter_area / union_area : 0.0f;
 }
 std::vector<RawDetection> nms(std::vector<RawDetection> dets, float threshold) {
    std::sort(dets.begin(), dets.end(),
              [](const RawDetection& a, const RawDetection& b) {
                  return a.score > b.score;
              });
    std::vector<bool> suppressed(dets.size(), false);
    std::vector<RawDetection> result;
    for (size_t i = 0; i < dets.size(); i++) {
        if (suppressed[i]) continue;
        result.push_back(dets[i]);
        for (size_t j = i + 1; j < dets.size(); j++) {
            if (!suppressed[j] && iou(dets[i], dets[j]) > threshold) {
                suppressed[j] = true;
            }
        }
    }
    return result;
 }
 // --- Letterbox resize ---
 cv::Mat letterbox(const cv::Mat& src, int size, float& scale, float& pad_x, float& pad_y) {
    float scale_w = static_cast<float>(size) / src.cols;
    float scale_h = static_cast<float>(size) / src.rows;
    scale = std::min(scale_w, scale_h);
    int new_w = static_cast<int>(std::round(src.cols * scale));
    int new_h = static_cast<int>(std::round(src.rows * scale));
    pad_x = (size - new_w) / 2.0f;
    pad_y = (size - new_h) / 2.0f;
    cv::Mat resized;
    cv::resize(src, resized, {new_w, new_h});
    cv::Mat canvas(size, size, CV_8UC3, cv::Scalar(114, 114, 114));
    resized.copyTo(canvas(cv::Rect(
        static_cast<int>(pad_x), static_cast<int>(pad_y), new_w, new_h)));
    return canvas;
 }
 // --- Point cloud fusion ---
 struct FusionResult {
    float cx, cy, cz;
    float depth_near, depth_far;
    bool valid = false;
 };
 FusionResult fuse_bbox(const sl::Mat& pc, int x1, int y1, int x2, int y2) {
    FusionResult result{};
    int w = static_cast<int>(pc.getWidth());
    int h = static_cast<int>(pc.getHeight());
    x1 = std::max(x1, 0);
    y1 = std::max(y1, 0);
    x2 = std::min(x2, w - 1);
    y2 = std::min(y2, h - 1);
    std::vector<float> xs, ys, zs;
    xs.reserve((x2 - x1) * (y2 - y1));
    ys.reserve(xs.capacity());
    zs.reserve(xs.capacity());
    for (int y = y1; y <= y2; y++) {
        for (int x = x1; x <= x2; x++) {
            sl::float4 pt;
            pc.getValue(x, y, &pt);
            if (!std::isnan(pt.x) && !std::isnan(pt.y) && !std::isnan(pt.z) &&
                !std::isinf(pt.x) && !std::isinf(pt.y) && !std::isinf(pt.z)) {
                xs.push_back(pt.x);
                ys.push_back(pt.y);
                zs.push_back(pt.z);
            }
        }
    }
    if (xs.empty()) return result;
    std::sort(xs.begin(), xs.end());
    std::sort(ys.begin(), ys.end());
    std::sort(zs.begin(), zs.end());
    size_t n   = xs.size();
    size_t mid = n / 2;
    size_t p25 = n / 4;
    size_t p75 = 3 * n / 4;
    result.cx         = xs[mid];
    result.cy         = ys[mid];
    result.cz         = zs[mid];
    result.depth_near = zs[p25];
    result.depth_far  = zs[p75];
    result.valid      = true;
    return result;
 }
 void backproject(float u, float v, float z, float fx, float fy, float cx, float cy,
                 float& X, float& Y, float& Z) {
    Z = z;
    X = (u - cx) * z / fx;
    Y = (v - cy) * z / fy;
 }
 // Split a comma-separated string into a vector of trimmed tokens
 std::vector<std::string> split_labels(const std::string& s) {
    std::vector<std::string> out;
    std::stringstream ss(s);
    std::string tok;
    while (std::getline(ss, tok, ',')) {
        size_t l = tok.find_first_not_of(" \t");
        size_t r = tok.find_last_not_of(" \t");
        if (l != std::string::npos)
            out.push_back(tok.substr(l, r - l + 1));
    }
    return out;
 }
 } // namespace
 namespace engines {
 // --- Impl ---
 struct OpenVocab::Impl {
    // Visual model
    Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "open_vocab"};
    Ort::SessionOptions session_options;
    std::unique_ptr<Ort::Session> session;
    Ort::AllocatorWithDefaultOptions allocator;
    std::string input_name;       // visual model: image input
    std::string output_name;      // visual model: detection output
    std::string text_input_name;  // visual model: text_feats input (empty if single-input model)
    int num_classes = 0;
    bool has_text_input = false;
    // Text encoder (optional; used when has_text_input == true)
    Ort::SessionOptions text_session_options;
    std::unique_ptr<Ort::Session> text_session;
    std::string text_enc_input_name;   // "input_ids"
    std::string text_enc_output_name;  // "text_embeds"
    // Tokenizer + embedding cache
    util::ClipTokenizer tokenizer;
    std::vector<float> cached_text_feats;  // (num_classes * 512) flat, row-major
    std::string cached_labels;
    // Preprocessing + ONNX inference + NMS.
    // Takes a BGR image (cv::imread format); returns 2D detections in original image coords.
    std::vector<RawDetection> run_inference(const OpenVocabConfig& config,
                                             const cv::Mat& image_bgr);
 };
 // --- Impl::run_inference ---
 std::vector<RawDetection> OpenVocab::Impl::run_inference(
    const OpenVocabConfig& config, const cv::Mat& image_bgr)
 {
    if (!session) return {};
    if (has_text_input && cached_text_feats.empty()) return {};
    int orig_w = image_bgr.cols;
    int orig_h = image_bgr.rows;
    // BGR → RGB (YOLO expects RGB)
    cv::Mat rgb;
    cv::cvtColor(image_bgr, rgb, cv::COLOR_BGR2RGB);
    float scale, pad_x, pad_y;
    cv::Mat lb = letterbox(rgb, config.input_size, scale, pad_x, pad_y);
    cv::Mat lb_float;
    lb.convertTo(lb_float, CV_32F, 1.0f / 255.0f);
    int sz = config.input_size;
    std::vector<float> input_data(3 * sz * sz);
    for (int c = 0; c < 3; c++) {
        for (int h = 0; h < sz; h++) {
            for (int w = 0; w < sz; w++) {
                input_data[c * sz * sz + h * sz + w] =
                    lb_float.at<cv::Vec3f>(h, w)[c];
            }
        }
    }
    auto mem_info = Ort::MemoryInfo::CreateCpu(
        OrtAllocatorType::OrtArenaAllocator, OrtMemTypeDefault);
    std::vector<int64_t> image_shape{1, 3, sz, sz};
    Ort::Value image_tensor = Ort::Value::CreateTensor<float>(
        mem_info, input_data.data(), input_data.size(),
        image_shape.data(), image_shape.size());
    const char* output_name_cstr = output_name.c_str();
    std::vector<Ort::Value> outputs;
    try {
        if (has_text_input) {
            std::vector<int64_t> text_shape{1, num_classes, 512};
            Ort::Value text_tensor = Ort::Value::CreateTensor<float>(
                mem_info,
                cached_text_feats.data(),
                cached_text_feats.size(),
                text_shape.data(), text_shape.size());
            const char* input_names[2] = {
                input_name.c_str(),
                text_input_name.c_str()
            };
            Ort::Value input_tensors[2] = {
                std::move(image_tensor),
                std::move(text_tensor)
            };
            outputs = session->Run(
                Ort::RunOptions{nullptr},
                input_names, input_tensors, 2,
                &output_name_cstr, 1);
        } else {
            const char* input_name_cstr = input_name.c_str();
            outputs = session->Run(
                Ort::RunOptions{nullptr},
                &input_name_cstr, &image_tensor, 1,
                &output_name_cstr, 1);
        }
    } catch (const Ort::Exception& e) {
        std::cerr << "OpenVocab: inference failed: " << e.what() << std::endl;
        return {};
    }
    // Post-process: output shape (1, 4+num_classes, num_anchors)
    auto out_shape = outputs[0].GetTensorTypeAndShapeInfo().GetShape();
    if (out_shape.size() < 3) return {};
    int num_anchors = static_cast<int>(out_shape[2]);
    int nc = num_classes;
    if (nc <= 0) return {};
    const float* out = outputs[0].GetTensorData<float>();
    std::vector<RawDetection> candidates;
    for (int a = 0; a < num_anchors; a++) {
        float cx_lb = out[0 * num_anchors + a];
        float cy_lb = out[1 * num_anchors + a];
        float w_lb  = out[2 * num_anchors + a];
        float h_lb  = out[3 * num_anchors + a];
        float best_score = 0.0f;
        int   best_class = 0;
        for (int c = 0; c < nc; c++) {
            float s = out[(4 + c) * num_anchors + a];
            if (s > best_score) { best_score = s; best_class = c; }
        }
        if (best_score < config.confidence_threshold) continue;
        float x1_lb = cx_lb - w_lb / 2.0f;
        float y1_lb = cy_lb - h_lb / 2.0f;
        float x2_lb = cx_lb + w_lb / 2.0f;
        float y2_lb = cy_lb + h_lb / 2.0f;
        float x1 = std::max(0.0f, std::min((x1_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
        float y1 = std::max(0.0f, std::min((y1_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
        float x2 = std::max(0.0f, std::min((x2_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
        float y2 = std::max(0.0f, std::min((y2_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
        candidates.push_back({x1, y1, x2, y2, best_score, best_class});
    }
    return nms(std::move(candidates), config.nms_threshold);
 }
 // --- Config ---
 OpenVocabConfig OpenVocabConfig::from_env() {
    OpenVocabConfig c;
    c.model_path           = util::get_env_str("ZED_ONNX_MODEL_PATH",        c.model_path);
    c.confidence_threshold = util::get_env_float("ZED_ONNX_CONFIDENCE",      c.confidence_threshold);
    c.nms_threshold        = util::get_env_float("ZED_ONNX_NMS_THRESHOLD",   c.nms_threshold);
    c.input_size           = util::get_env_int("ZED_ONNX_INPUT_SIZE",        c.input_size);
    c.labels               = util::get_env_str("ZED_ONNX_LABELS",            c.labels);
    c.text_encoder_path    = util::get_env_str("ZED_ONNX_TEXT_ENCODER_PATH", c.text_encoder_path);
    c.clip_vocab_path      = util::get_env_str("ZED_ONNX_CLIP_VOCAB",        c.clip_vocab_path);
    c.clip_merges_path     = util::get_env_str("ZED_ONNX_CLIP_MERGES",       c.clip_merges_path);
    return c;
 }
 // --- OpenVocab ---
 OpenVocab::OpenVocab(const OpenVocabConfig& config)
    : config_(config), impl_(std::make_unique<Impl>()) {}
 OpenVocab::~OpenVocab() = default;
 void OpenVocab::enable(sl::Camera& /* camera */) {
    if (config_.model_path.empty()) {
        std::cerr << "OpenVocab: ZED_ONNX_MODEL_PATH is not set." << std::endl;
        return;
    }
    // --- Load visual model ---
    OrtCUDAProviderOptions cuda_opts{};
    cuda_opts.device_id = 0;
    try {
        impl_->session_options.AppendExecutionProvider_CUDA(cuda_opts);
    } catch (const Ort::Exception& e) {
        std::cerr << "OpenVocab: CUDA unavailable (" << e.what() << "), using CPU." << std::endl;
    }
    impl_->session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);
    std::cout << "OpenVocab: loading visual model (" << config_.model_path << ")..." << std::endl;
    try {
        impl_->session = std::make_unique<Ort::Session>(
            impl_->env, config_.model_path.c_str(), impl_->session_options);
    } catch (const Ort::Exception& e) {
        std::cerr << "OpenVocab: failed to load visual model: " << e.what() << std::endl;
        return;
    }
    impl_->input_name  = impl_->session->GetInputNameAllocated(0, impl_->allocator).get();
    impl_->output_name = impl_->session->GetOutputNameAllocated(0, impl_->allocator).get();
    size_t num_inputs = impl_->session->GetInputCount();
    impl_->has_text_input = (num_inputs >= 2);
    if (impl_->has_text_input) {
        impl_->text_input_name = impl_->session->GetInputNameAllocated(1, impl_->allocator).get();
    }
    auto out_info = impl_->session->GetOutputTypeInfo(0).GetTensorTypeAndShapeInfo();
    auto shape    = out_info.GetShape();
    if (shape.size() == 3 && shape[1] > 4) {
        impl_->num_classes = static_cast<int>(shape[1]) - 4;
    }
    std::cout << "OpenVocab: visual model loaded."
              << " Inputs=" << num_inputs
              << " Classes=" << impl_->num_classes
              << (impl_->has_text_input ? " (open-vocab)" : " (fixed-vocab)")
              << std::endl;
    // --- Load text encoder + tokenizer (only needed for open-vocab model) ---
    if (impl_->has_text_input) {
        if (config_.text_encoder_path.empty()) {
            std::cerr << "OpenVocab: ZED_ONNX_TEXT_ENCODER_PATH is not set "
                         "(required for open-vocab model)." << std::endl;
            return;
        }
        impl_->text_session_options.SetGraphOptimizationLevel(
            GraphOptimizationLevel::ORT_ENABLE_ALL);
        std::cout << "OpenVocab: loading text encoder (" << config_.text_encoder_path
                  << ")..." << std::endl;
        try {
            impl_->text_session = std::make_unique<Ort::Session>(
                impl_->env, config_.text_encoder_path.c_str(), impl_->text_session_options);
        } catch (const Ort::Exception& e) {
            std::cerr << "OpenVocab: failed to load text encoder: " << e.what() << std::endl;
            return;
        }
        impl_->text_enc_input_name  =
            impl_->text_session->GetInputNameAllocated(0, impl_->allocator).get();
        impl_->text_enc_output_name =
            impl_->text_session->GetOutputNameAllocated(0, impl_->allocator).get();
        if (!config_.clip_vocab_path.empty() && !config_.clip_merges_path.empty()) {
            try {
                impl_->tokenizer.load(config_.clip_vocab_path, config_.clip_merges_path);
                std::cout << "OpenVocab: tokenizer loaded." << std::endl;
            } catch (const std::exception& e) {
                std::cerr << "OpenVocab: failed to load tokenizer: " << e.what() << std::endl;
                return;
            }
        } else {
            std::cerr << "OpenVocab: ZED_ONNX_CLIP_VOCAB / ZED_ONNX_CLIP_MERGES not set." << std::endl;
            return;
        }
        if (!config_.labels.empty()) {
            encode_labels(config_.labels);
        }
    }
 }
 // Encode a comma-separated label string into cached_text_feats.
 // Always fills exactly num_classes * 512 floats (pads with zeros if fewer labels).
 void OpenVocab::encode_labels(const std::string& labels_str) {
    if (!impl_->text_session || !impl_->tokenizer.is_loaded()) return;
    auto label_vec = split_labels(labels_str);
    int N = impl_->num_classes;
    impl_->cached_text_feats.assign(static_cast<size_t>(N) * 512, 0.0f);
    auto mem_info = Ort::MemoryInfo::CreateCpu(
        OrtAllocatorType::OrtArenaAllocator, OrtMemTypeDefault);
    int encode_n = std::min(static_cast<int>(label_vec.size()), N);
    for (int i = 0; i < encode_n; i++) {
        auto tokens = impl_->tokenizer.tokenize(label_vec[i]);
        std::vector<int64_t> input_shape{1, util::ClipTokenizer::SEQ_LEN};
        Ort::Value token_tensor = Ort::Value::CreateTensor<int64_t>(
            mem_info, tokens.data(), tokens.size(),
            input_shape.data(), input_shape.size());
        const char* in  = impl_->text_enc_input_name.c_str();
        const char* out = impl_->text_enc_output_name.c_str();
        std::vector<Ort::Value> outputs;
        try {
            outputs = impl_->text_session->Run(
                Ort::RunOptions{nullptr}, &in, &token_tensor, 1, &out, 1);
        } catch (const Ort::Exception& e) {
            std::cerr << "OpenVocab: text encoder failed for \"" << label_vec[i]
                      << "\": " << e.what() << std::endl;
            continue;
        }
        const float* emb = outputs[0].GetTensorData<float>();
        // L2 normalize
        float norm = 0.0f;
        for (int j = 0; j < 512; j++) norm += emb[j] * emb[j];
        norm = std::sqrt(norm);
        float inv_norm = (norm > 1e-8f) ? 1.0f / norm : 0.0f;
        for (int j = 0; j < 512; j++)
            impl_->cached_text_feats[i * 512 + j] = emb[j] * inv_norm;
    }
    impl_->cached_labels = labels_str;
    std::cout << "OpenVocab: encoded " << encode_n << " label(s): " << labels_str << std::endl;
 }
 // --- detect(sl::Camera&) — production path with 3D fusion ---
 void OpenVocab::detect(sl::Camera& camera) {
    detected_.clear();
    if (!impl_->session) return;
    // Re-encode if labels changed (on-the-fly vocabulary switching)
    if (impl_->has_text_input) {
        const char* env_labels = std::getenv("ZED_ONNX_LABELS");
        std::string current_labels = env_labels ? env_labels : config_.labels;
        if (current_labels != impl_->cached_labels) {
            encode_labels(current_labels);
        }
        if (impl_->cached_text_feats.empty()) return;
    }
    // Retrieve image and point cloud from camera
    sl::Mat img_mat;
    camera.retrieveImage(img_mat, sl::VIEW::LEFT);
    sl::Mat pc_mat;
    camera.retrieveMeasure(pc_mat, sl::MEASURE::XYZRGBA);
    sl::CameraInformation info = camera.getCameraInformation();
    auto calib = info.camera_configuration.calibration_parameters.left_cam;
    float fx = calib.fx, fy = calib.fy, cx = calib.cx, cy = calib.cy;
    int orig_w = static_cast<int>(img_mat.getWidth());
    int orig_h = static_cast<int>(img_mat.getHeight());
    // RGBA → BGR for run_inference
    cv::Mat rgba(orig_h, orig_w, CV_8UC4, img_mat.getPtr<sl::uchar1>());
    cv::Mat bgr;
    cv::cvtColor(rgba, bgr, cv::COLOR_RGBA2BGR);
    auto kept = impl_->run_inference(config_, bgr);
    // 3D fusion
    for (const auto& det : kept) {
        FusionResult fusion = fuse_bbox(pc_mat,
            static_cast<int>(det.x1), static_cast<int>(det.y1),
            static_cast<int>(det.x2), static_cast<int>(det.y2));
        DetectedObject obj{};
        obj.label_id   = det.class_id;
        obj.confidence = det.score * 100.0f;
        if (fusion.valid) {
            obj.position[0] = fusion.cx;
            obj.position[1] = fusion.cy;
            obj.position[2] = fusion.cz;
        }
        obj.bounding_box_2d[0] = det.x1; obj.bounding_box_2d[1] = det.y1;
        obj.bounding_box_2d[2] = det.x2; obj.bounding_box_2d[3] = det.y1;
        obj.bounding_box_2d[4] = det.x2; obj.bounding_box_2d[5] = det.y2;
        obj.bounding_box_2d[6] = det.x1; obj.bounding_box_2d[7] = det.y2;
        if (fusion.valid) {
            float corners_u[4] = {det.x1, det.x2, det.x2, det.x1};
            float corners_v[4] = {det.y1, det.y1, det.y2, det.y2};
            float depths[2]    = {fusion.depth_near, fusion.depth_far};
            for (int face = 0; face < 2; face++) {
                for (int c = 0; c < 4; c++) {
                    float X, Y, Z;
                    backproject(corners_u[c], corners_v[c], depths[face],
                                fx, fy, cx, cy, X, Y, Z);
                    int idx = (face * 4 + c) * 3;
                    obj.bounding_box_3d[idx + 0] = X;
                    obj.bounding_box_3d[idx + 1] = Y;
                    obj.bounding_box_3d[idx + 2] = Z;
                }
            }
        }
        detected_.push_back(obj);
    }
 }
 // --- detect(const cv::Mat&) — image-only path, no 3D fusion ---
 void OpenVocab::detect(const cv::Mat& image_bgr) {
    detected_.clear();
    if (!impl_->session) return;
    // Re-encode if labels changed
    if (impl_->has_text_input) {
        const char* env_labels = std::getenv("ZED_ONNX_LABELS");
        std::string current_labels = env_labels ? env_labels : config_.labels;
        if (current_labels != impl_->cached_labels) {
            encode_labels(current_labels);
        }
        if (impl_->cached_text_feats.empty()) return;
    }
    auto kept = impl_->run_inference(config_, image_bgr);
    for (const auto& det : kept) {
        DetectedObject obj{};
        obj.label_id   = det.class_id;
        obj.confidence = det.score * 100.0f;
        // position stays zero (no point cloud available)
        obj.bounding_box_2d[0] = det.x1; obj.bounding_box_2d[1] = det.y1;
        obj.bounding_box_2d[2] = det.x2; obj.bounding_box_2d[3] = det.y1;
        obj.bounding_box_2d[4] = det.x2; obj.bounding_box_2d[5] = det.y2;
        obj.bounding_box_2d[6] = det.x1; obj.bounding_box_2d[7] = det.y2;
        // bounding_box_3d stays zero
        detected_.push_back(obj);
    }
 }
 void OpenVocab::set_labels(const std::string& labels) {
    config_.labels = labels;
    encode_labels(labels);
 }
 const std::vector<DetectedObject>& OpenVocab::objects() const {
    return detected_;
 }
 std::string OpenVocab::label_names() const {
    return impl_->cached_labels.empty() ? config_.labels : impl_->cached_labels;
 }
 } // namespace engines
@@ -0,0 +1,136 @@
 #pragma once
 #include <opencv2/opencv.hpp>
 #include <sl/Camera.hpp>
 #include <algorithm>
 #include <cmath>
 #include <vector>
 namespace engines::detail {
 // --- NMS ---
 struct RawDetection {
    float x1, y1, x2, y2;
    float score;
    int class_id;
 };
 inline float iou(const RawDetection& a, const RawDetection& b) {
    float inter_x1 = std::max(a.x1, b.x1);
    float inter_y1 = std::max(a.y1, b.y1);
    float inter_x2 = std::min(a.x2, b.x2);
    float inter_y2 = std::min(a.y2, b.y2);
    float inter_w = std::max(0.0f, inter_x2 - inter_x1);
    float inter_h = std::max(0.0f, inter_y2 - inter_y1);
    float inter_area = inter_w * inter_h;
    float area_a = (a.x2 - a.x1) * (a.y2 - a.y1);
    float area_b = (b.x2 - b.x1) * (b.y2 - b.y1);
    float union_area = area_a + area_b - inter_area;
    return union_area > 0.0f ? inter_area / union_area : 0.0f;
 }
 inline std::vector<RawDetection> nms(std::vector<RawDetection> dets, float threshold) {
    std::sort(dets.begin(), dets.end(),
              [](const RawDetection& a, const RawDetection& b) {
                  return a.score > b.score;
              });
    std::vector<bool> suppressed(dets.size(), false);
    std::vector<RawDetection> result;
    for (size_t i = 0; i < dets.size(); i++) {
        if (suppressed[i]) continue;
        result.push_back(dets[i]);
        for (size_t j = i + 1; j < dets.size(); j++) {
            if (!suppressed[j] && iou(dets[i], dets[j]) > threshold)
                suppressed[j] = true;
        }
    }
    return result;
 }
 // --- Letterbox resize ---
 inline cv::Mat letterbox(const cv::Mat& src, int size,
                         float& scale, float& pad_x, float& pad_y) {
    float scale_w = static_cast<float>(size) / src.cols;
    float scale_h = static_cast<float>(size) / src.rows;
    scale = std::min(scale_w, scale_h);
    int new_w = static_cast<int>(std::round(src.cols * scale));
    int new_h = static_cast<int>(std::round(src.rows * scale));
    pad_x = (size - new_w) / 2.0f;
    pad_y = (size - new_h) / 2.0f;
    cv::Mat resized;
    cv::resize(src, resized, {new_w, new_h});
    cv::Mat canvas(size, size, CV_8UC3, cv::Scalar(114, 114, 114));
    resized.copyTo(canvas(cv::Rect(
        static_cast<int>(pad_x), static_cast<int>(pad_y), new_w, new_h)));
    return canvas;
 }
 // --- Point cloud fusion ---
 struct FusionResult {
    float cx, cy, cz;
    float depth_near, depth_far;
    bool valid = false;
 };
 inline FusionResult fuse_bbox(const sl::Mat& pc, int x1, int y1, int x2, int y2) {
    FusionResult result{};
    int w = static_cast<int>(pc.getWidth());
    int h = static_cast<int>(pc.getHeight());
    x1 = std::max(x1, 0);
    y1 = std::max(y1, 0);
    x2 = std::min(x2, w - 1);
    y2 = std::min(y2, h - 1);
    std::vector<float> xs, ys, zs;
    xs.reserve((x2 - x1) * (y2 - y1));
    ys.reserve(xs.capacity());
    zs.reserve(xs.capacity());
    for (int y = y1; y <= y2; y++) {
        for (int x = x1; x <= x2; x++) {
            sl::float4 pt;
            pc.getValue(x, y, &pt);
            if (!std::isnan(pt.x) && !std::isnan(pt.y) && !std::isnan(pt.z) &&
                !std::isinf(pt.x) && !std::isinf(pt.y) && !std::isinf(pt.z)) {
                xs.push_back(pt.x);
                ys.push_back(pt.y);
                zs.push_back(pt.z);
            }
        }
    }
    if (xs.empty()) return result;
    std::sort(xs.begin(), xs.end());
    std::sort(ys.begin(), ys.end());
    std::sort(zs.begin(), zs.end());
    size_t n   = xs.size();
    size_t mid = n / 2;
    size_t p25 = n / 4;
    size_t p75 = 3 * n / 4;
    result.cx         = xs[mid];
    result.cy         = ys[mid];
    result.cz         = zs[mid];
    result.depth_near = zs[p25];
    result.depth_far  = zs[p75];
    result.valid      = true;
    return result;
 }
 inline void backproject(float u, float v, float z,
                        float fx, float fy, float cx, float cy,
                        float& X, float& Y, float& Z) {
    Z = z;
    X = (u - cx) * z / fx;
    Y = (v - cy) * z / fy;
 }
 } // namespace engines::detail
@@ -0,0 +1,369 @@
 #include "engines/yolo_onnx.h"
 #include "engines/yolo_helpers.h"
 #include "util/env.h"
 #include <onnxruntime_cxx_api.h>
 #include <opencv2/opencv.hpp>
 #include <sl/Camera.hpp>
 #include <cmath>
 #include <fstream>
 #include <iostream>
 #include <string>
 #include <vector>
 namespace engines {
 using namespace detail;
 // --- Impl ---
 struct YoloOnnx::Impl {
    Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "yolo_onnx"};
    Ort::SessionOptions session_options;
    std::unique_ptr<Ort::Session> session;
    Ort::AllocatorWithDefaultOptions allocator;
    std::string input_name;
    std::string output_name;
    int num_classes = 0;
    YoloOutputLayout resolved_layout = YoloOutputLayout::NC_FIRST;
    std::vector<RawDetection> run_inference(const YoloOnnxConfig& config,
                                            const cv::Mat& image_bgr);
 };
 // --- Impl::run_inference ---
 std::vector<RawDetection> YoloOnnx::Impl::run_inference(
    const YoloOnnxConfig& config, const cv::Mat& image_bgr)
 {
    if (!session) return {};
    int orig_w = image_bgr.cols;
    int orig_h = image_bgr.rows;
    cv::Mat rgb;
    cv::cvtColor(image_bgr, rgb, cv::COLOR_BGR2RGB);
    float scale, pad_x, pad_y;
    cv::Mat lb = letterbox(rgb, config.input_size, scale, pad_x, pad_y);
    cv::Mat lb_float;
    lb.convertTo(lb_float, CV_32F, 1.0f / 255.0f);
    int sz = config.input_size;
    std::vector<float> input_data(3 * sz * sz);
    for (int c = 0; c < 3; c++) {
        for (int h = 0; h < sz; h++) {
            for (int w = 0; w < sz; w++) {
                input_data[c * sz * sz + h * sz + w] =
                    lb_float.at<cv::Vec3f>(h, w)[c];
            }
        }
    }
    auto mem_info = Ort::MemoryInfo::CreateCpu(
        OrtAllocatorType::OrtArenaAllocator, OrtMemTypeDefault);
    std::vector<int64_t> image_shape{1, 3, sz, sz};
    Ort::Value image_tensor = Ort::Value::CreateTensor<float>(
        mem_info, input_data.data(), input_data.size(),
        image_shape.data(), image_shape.size());
    const char* input_name_cstr  = input_name.c_str();
    const char* output_name_cstr = output_name.c_str();
    std::vector<Ort::Value> outputs;
    try {
        outputs = session->Run(
            Ort::RunOptions{nullptr},
            &input_name_cstr, &image_tensor, 1,
            &output_name_cstr, 1);
    } catch (const Ort::Exception& e) {
        std::cerr << "YoloOnnx: inference failed: " << e.what() << std::endl;
        return {};
    }
    auto out_shape = outputs[0].GetTensorTypeAndShapeInfo().GetShape();
    if (out_shape.size() < 3) return {};
    const float* out = outputs[0].GetTensorData<float>();
    int nc = num_classes;
    if (nc <= 0) return {};
    std::vector<RawDetection> candidates;
    if (resolved_layout == YoloOutputLayout::NC_FIRST) {
        // (1, 4+N, anchors) — YOLOv8, YOLOv11
        int num_anchors = static_cast<int>(out_shape[2]);
        for (int a = 0; a < num_anchors; a++) {
            float cx_lb = out[0 * num_anchors + a];
            float cy_lb = out[1 * num_anchors + a];
            float w_lb  = out[2 * num_anchors + a];
            float h_lb  = out[3 * num_anchors + a];
            float best_score = 0.0f;
            int   best_class = 0;
            for (int c = 0; c < nc; c++) {
                float s = out[(4 + c) * num_anchors + a];
                if (s > best_score) { best_score = s; best_class = c; }
            }
            if (best_score < config.confidence_threshold) continue;
            float x1_lb = cx_lb - w_lb / 2.0f;
            float y1_lb = cy_lb - h_lb / 2.0f;
            float x2_lb = cx_lb + w_lb / 2.0f;
            float y2_lb = cy_lb + h_lb / 2.0f;
            float x1 = std::max(0.0f, std::min((x1_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
            float y1 = std::max(0.0f, std::min((y1_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
            float x2 = std::max(0.0f, std::min((x2_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
            float y2 = std::max(0.0f, std::min((y2_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
            candidates.push_back({x1, y1, x2, y2, best_score, best_class});
        }
    } else {
        // (1, anchors, 4+N) or (1, anchors, 5+N)
        int num_anchors = static_cast<int>(out_shape[1]);
        int row_stride  = static_cast<int>(out_shape[2]);
        int cls_offset  = (resolved_layout == YoloOutputLayout::NC_LAST_OBJ) ? 5 : 4;
        for (int a = 0; a < num_anchors; a++) {
            const float* row = out + a * row_stride;
            float cx_lb = row[0], cy_lb = row[1], w_lb = row[2], h_lb = row[3];
            float obj_conf = (resolved_layout == YoloOutputLayout::NC_LAST_OBJ) ? row[4] : 1.0f;
            float best_score = 0.0f;
            int   best_class = 0;
            for (int c = 0; c < nc; c++) {
                float s = obj_conf * row[cls_offset + c];
                if (s > best_score) { best_score = s; best_class = c; }
            }
            if (best_score < config.confidence_threshold) continue;
            float x1_lb = cx_lb - w_lb / 2.0f;
            float y1_lb = cy_lb - h_lb / 2.0f;
            float x2_lb = cx_lb + w_lb / 2.0f;
            float y2_lb = cy_lb + h_lb / 2.0f;
            float x1 = std::max(0.0f, std::min((x1_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
            float y1 = std::max(0.0f, std::min((y1_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
            float x2 = std::max(0.0f, std::min((x2_lb - pad_x) / scale, static_cast<float>(orig_w - 1)));
            float y2 = std::max(0.0f, std::min((y2_lb - pad_y) / scale, static_cast<float>(orig_h - 1)));
            candidates.push_back({x1, y1, x2, y2, best_score, best_class});
        }
    }
    return nms(std::move(candidates), config.nms_threshold);
 }
 // --- Config ---
 YoloOnnxConfig YoloOnnxConfig::from_env() {
    YoloOnnxConfig c;
    c.model_path           = util::get_env_str("ZED_ONNX_MODEL_PATH",      c.model_path);
    c.confidence_threshold = util::get_env_float("ZED_ONNX_CONFIDENCE",    c.confidence_threshold);
    c.nms_threshold        = util::get_env_float("ZED_ONNX_NMS_THRESHOLD", c.nms_threshold);
    c.input_size           = util::get_env_int("ZED_ONNX_INPUT_SIZE",      c.input_size);
    c.labels               = util::get_env_str("ZED_ONNX_LABELS",          c.labels);
    c.labels_file          = util::get_env_str("ZED_ONNX_LABELS_FILE",     c.labels_file);
    std::string fmt = util::get_env_str("ZED_ONNX_OUTPUT_FORMAT", "");
    if      (fmt == "nc_first"    || fmt == "yolov8" || fmt == "yolov11")
        c.output_layout = YoloOutputLayout::NC_FIRST;
    else if (fmt == "nc_last")
        c.output_layout = YoloOutputLayout::NC_LAST;
    else if (fmt == "nc_last_obj" || fmt == "yolov5")
        c.output_layout = YoloOutputLayout::NC_LAST_OBJ;
    return c;
 }
 // --- YoloOnnx ---
 YoloOnnx::YoloOnnx(const YoloOnnxConfig& config)
    : config_(config), impl_(std::make_unique<Impl>()) {}
 YoloOnnx::~YoloOnnx() = default;
 void YoloOnnx::enable(sl::Camera& /* camera */) {
    if (config_.model_path.empty()) {
        std::cerr << "YoloOnnx: ZED_ONNX_MODEL_PATH is not set." << std::endl;
        return;
    }
    OrtCUDAProviderOptions cuda_opts{};
    cuda_opts.device_id = 0;
    try {
        impl_->session_options.AppendExecutionProvider_CUDA(cuda_opts);
    } catch (const Ort::Exception& e) {
        std::cerr << "YoloOnnx: CUDA unavailable (" << e.what() << "), using CPU." << std::endl;
    }
    impl_->session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);
    std::cout << "YoloOnnx: loading model (" << config_.model_path << ")..." << std::endl;
    try {
        impl_->session = std::make_unique<Ort::Session>(
            impl_->env, config_.model_path.c_str(), impl_->session_options);
    } catch (const Ort::Exception& e) {
        std::cerr << "YoloOnnx: failed to load model: " << e.what() << std::endl;
        return;
    }
    impl_->input_name  = impl_->session->GetInputNameAllocated(0, impl_->allocator).get();
    impl_->output_name = impl_->session->GetOutputNameAllocated(0, impl_->allocator).get();
    size_t num_inputs = impl_->session->GetInputCount();
    if (num_inputs >= 2) {
        std::cerr << "YoloOnnx: model has " << num_inputs << " inputs — "
                     "this engine only supports single-input models."
                  << std::endl;
    }
    auto out_info = impl_->session->GetOutputTypeInfo(0).GetTensorTypeAndShapeInfo();
    auto shape    = out_info.GetShape();
    if (shape.size() == 3) {
        int64_t dim1 = shape[1];
        int64_t dim2 = shape[2];
        YoloOutputLayout resolved;
        if (config_.output_layout == YoloOutputLayout::AUTO) {
            // dim1 < dim2 → NC_FIRST (e.g. 84 vs 8400 for YOLOv8/11)
            // dim1 > dim2 → NC_LAST_OBJ (e.g. 25200 vs 85 for YOLOv5)
            resolved = (dim1 < dim2) ? YoloOutputLayout::NC_FIRST
                                     : YoloOutputLayout::NC_LAST_OBJ;
        } else {
            resolved = config_.output_layout;
        }
        impl_->resolved_layout = resolved;
        if (resolved == YoloOutputLayout::NC_FIRST) {
            impl_->num_classes = static_cast<int>(dim1) - 4;
        } else if (resolved == YoloOutputLayout::NC_LAST) {
            impl_->num_classes = static_cast<int>(dim2) - 4;
        } else { // NC_LAST_OBJ
            impl_->num_classes = static_cast<int>(dim2) - 5;
        }
    }
    if (!config_.labels_file.empty() && config_.labels.empty()) {
        std::ifstream lf(config_.labels_file);
        if (!lf) {
            std::cerr << "YoloOnnx: cannot open ZED_ONNX_LABELS_FILE: "
                      << config_.labels_file << std::endl;
        } else {
            std::string line, joined;
            while (std::getline(lf, line)) {
                size_t l = line.find_first_not_of(" \t\r");
                size_t r = line.find_last_not_of(" \t\r");
                if (l == std::string::npos) continue;
                if (!joined.empty()) joined += ',';
                joined += line.substr(l, r - l + 1);
            }
            if (!joined.empty()) {
                config_.labels = joined;
                std::cout << "YoloOnnx: loaded labels from file ("
                          << config_.labels_file << ")" << std::endl;
            }
        }
    }
    std::cout << "YoloOnnx: model loaded."
              << " Classes=" << impl_->num_classes
              << " Layout=" << (impl_->resolved_layout == YoloOutputLayout::NC_FIRST ? "NC_FIRST"
                              : impl_->resolved_layout == YoloOutputLayout::NC_LAST  ? "NC_LAST"
                                                                                      : "NC_LAST_OBJ")
              << std::endl;
 }
 void YoloOnnx::detect(sl::Camera& camera) {
    detected_.clear();
    if (!impl_->session) return;
    sl::Mat img_mat;
    camera.retrieveImage(img_mat, sl::VIEW::LEFT);
    sl::Mat pc_mat;
    camera.retrieveMeasure(pc_mat, sl::MEASURE::XYZRGBA);
    sl::CameraInformation info = camera.getCameraInformation();
    auto calib = info.camera_configuration.calibration_parameters.left_cam;
    float fx = calib.fx, fy = calib.fy, cx = calib.cx, cy = calib.cy;
    int orig_w = static_cast<int>(img_mat.getWidth());
    int orig_h = static_cast<int>(img_mat.getHeight());
    cv::Mat rgba(orig_h, orig_w, CV_8UC4, img_mat.getPtr<sl::uchar1>());
    cv::Mat bgr;
    cv::cvtColor(rgba, bgr, cv::COLOR_RGBA2BGR);
    auto kept = impl_->run_inference(config_, bgr);
    for (const auto& det : kept) {
        FusionResult fusion = fuse_bbox(pc_mat,
            static_cast<int>(det.x1), static_cast<int>(det.y1),
            static_cast<int>(det.x2), static_cast<int>(det.y2));
        DetectedObject obj{};
        obj.label_id   = det.class_id;
        obj.confidence = det.score * 100.0f;
        if (fusion.valid) {
            obj.position[0] = fusion.cx;
            obj.position[1] = fusion.cy;
            obj.position[2] = fusion.cz;
        }
        obj.bounding_box_2d[0] = det.x1; obj.bounding_box_2d[1] = det.y1;
        obj.bounding_box_2d[2] = det.x2; obj.bounding_box_2d[3] = det.y1;
        obj.bounding_box_2d[4] = det.x2; obj.bounding_box_2d[5] = det.y2;
        obj.bounding_box_2d[6] = det.x1; obj.bounding_box_2d[7] = det.y2;
        if (fusion.valid) {
            float corners_u[4] = {det.x1, det.x2, det.x2, det.x1};
            float corners_v[4] = {det.y1, det.y1, det.y2, det.y2};
            float depths[2]    = {fusion.depth_near, fusion.depth_far};
            for (int face = 0; face < 2; face++) {
                for (int c = 0; c < 4; c++) {
                    float X, Y, Z;
                    backproject(corners_u[c], corners_v[c], depths[face],
                                fx, fy, cx, cy, X, Y, Z);
                    int idx = (face * 4 + c) * 3;
                    obj.bounding_box_3d[idx + 0] = X;
                    obj.bounding_box_3d[idx + 1] = Y;
                    obj.bounding_box_3d[idx + 2] = Z;
                }
            }
        }
        detected_.push_back(obj);
    }
 }
 void YoloOnnx::detect(const cv::Mat& image_bgr) {
    detected_.clear();
    if (!impl_->session) return;
    auto kept = impl_->run_inference(config_, image_bgr);
    for (const auto& det : kept) {
        DetectedObject obj{};
        obj.label_id   = det.class_id;
        obj.confidence = det.score * 100.0f;
        obj.bounding_box_2d[0] = det.x1; obj.bounding_box_2d[1] = det.y1;
        obj.bounding_box_2d[2] = det.x2; obj.bounding_box_2d[3] = det.y1;
        obj.bounding_box_2d[4] = det.x2; obj.bounding_box_2d[5] = det.y2;
        obj.bounding_box_2d[6] = det.x1; obj.bounding_box_2d[7] = det.y2;
        detected_.push_back(obj);
    }
 }
 const std::vector<DetectedObject>& YoloOnnx::objects() const {
    return detected_;
 }
 std::string YoloOnnx::label_names() const {
    return config_.labels;
 }
 } // namespace engines
@@ -8,40 +8,41 @@
 namespace engines {
-struct OpenVocabConfig {
+enum class YoloOutputLayout {
    AUTO,          // detect from output tensor shape at load time
    NC_FIRST,      // (1, 4+N, anchors) — YOLOv8, YOLOv11
    NC_LAST,       // (1, anchors, 4+N) — rows-first, no objectness
    NC_LAST_OBJ    // (1, anchors, 5+N) — YOLOv5, rows-first with objectness
 };
 struct YoloOnnxConfig {
    std::string model_path;              // ZED_ONNX_MODEL_PATH (required)
    float confidence_threshold = 0.25f;  // ZED_ONNX_CONFIDENCE
    float nms_threshold = 0.45f;         // ZED_ONNX_NMS_THRESHOLD
    int input_size = 640;                // ZED_ONNX_INPUT_SIZE
    std::string labels;                  // ZED_ONNX_LABELS (comma-separated)
    std::string labels_file;             // ZED_ONNX_LABELS_FILE (one class name per line)
    YoloOutputLayout output_layout = YoloOutputLayout::AUTO;
-    // Open-vocabulary text encoder (required when model has two inputs)
+    static YoloOnnxConfig from_env();
    std::string text_encoder_path;       // ZED_ONNX_TEXT_ENCODER_PATH
    std::string clip_vocab_path;         // ZED_ONNX_CLIP_VOCAB
    std::string clip_merges_path;        // ZED_ONNX_CLIP_MERGES
    static OpenVocabConfig from_env();
 };
-class OpenVocab : public Engine {
+class YoloOnnx : public Engine {
 public:
-    explicit OpenVocab(const OpenVocabConfig& config);
+    explicit YoloOnnx(const YoloOnnxConfig& config);
-    ~OpenVocab() override;
+    ~YoloOnnx() override;
    void enable(sl::Camera& camera) override;
    void detect(sl::Camera& camera) override;
-    void detect(const cv::Mat& image_bgr);       // image-only; no 3D fusion
+    void detect(const cv::Mat& image_bgr);      // image-only; no 3D fusion
    void set_labels(const std::string& labels) override;
    const std::vector<DetectedObject>& objects() const override;
    std::string label_names() const override;
 private:
-    OpenVocabConfig config_;
+    YoloOnnxConfig config_;
    struct Impl;
    std::unique_ptr<Impl> impl_;
    std::vector<DetectedObject> detected_;
    void encode_labels(const std::string& labels_str);
 };
 } // namespace engines
@@ -1,5 +1,5 @@
 #include "engines/engine.h"
-#include "engines/open_vocab.h"
+#include "engines/yolo_onnx.h"
 #include "engines/zed_native.h"
 #include "util/env.h"
 #include "zed/camera.h"
@@ -74,9 +74,9 @@ int main() {
    std::string engine_name = util::get_env_str("ZED_ENGINE", "native");
    std::unique_ptr<engines::Engine> detector;
-    if (engine_name == "open_vocab") {
+    if (engine_name == "yolo_onnx" || engine_name == "onnx") {
-        auto cfg = engines::OpenVocabConfig::from_env();
+        auto cfg = engines::YoloOnnxConfig::from_env();
-        detector = std::make_unique<engines::OpenVocab>(cfg);
+        detector = std::make_unique<engines::YoloOnnx>(cfg);
    } else {
        auto cfg = engines::DetectionConfig::from_env();
        detector = std::make_unique<engines::ZedNative>(cfg);
@@ -101,20 +101,6 @@ int main() {
            continue;
        }
        auto input = event_as_input(std::move(event));
        std::string input_id = std::string(input.id);
        if (input_id == "update_vocabulary") {
            if (!input.data.empty()) {
                std::string labels(
                    reinterpret_cast<const char*>(input.data.data()),
                    input.data.size());
                detector->set_labels(labels);
                std::cout << "Vocabulary updated: " << labels << std::endl;
            }
            continue;
        }
        if (!camera.grab()) {
            std::cerr << "Grab failed; attempting to reconnect..." << std::endl;
            camera.close();
@@ -1,24 +1,22 @@
-// Standalone test for the OpenVocab detection engine.
+// Standalone test for the YoloOnnx (closed-vocabulary) detection engine.
-// No dora framework, no live ZED camera — loads images from disk.
+// No dora framework, no live ZED camera — loads an image from disk.
 //
 // Usage:
-//   bin/test_open_vocab [yolo_image] [yoloworld_image]
+//   bin/test_yolo_onnx [image_path]
-//   yolo_image       default: test/assets/yolo.png
+//   image_path   default: test/assets/yolo.png
 //   yoloworld_image  default: test/assets/yoloworld.png
 //
-// Uses the same env vars as the main binary:
+// Required env vars:
-//   ZED_ONNX_MODEL_PATH           path to YOLO ONNX model (required)
+//   ZED_ONNX_MODEL_PATH        path to YOLO ONNX model
-//   ZED_ONNX_LABELS               comma-separated labels (required)
+//
-//   ZED_ONNX_CONFIDENCE           detection threshold (default 0.25)
+// Optional env vars:
-//   ZED_ONNX_NMS_THRESHOLD        NMS IoU threshold (default 0.45)
+//   ZED_ONNX_LABELS            comma-separated class names
-//   ZED_ONNX_INPUT_SIZE           model input size (default 640)
+//   ZED_ONNX_LABELS_FILE       path to newline-separated class names file
-//   ZED_ONNX_TEXT_ENCODER_PATH    clip text encoder ONNX (open-vocab only)
+//   ZED_ONNX_CONFIDENCE        detection threshold (default 0.25)
-//   ZED_ONNX_CLIP_VOCAB           clip_vocab.txt path (open-vocab only)
+//   ZED_ONNX_NMS_THRESHOLD     NMS IoU threshold (default 0.45)
-//   ZED_ONNX_CLIP_MERGES          clip_merges.txt path (open-vocab only)
+//   ZED_ONNX_INPUT_SIZE        model input size (default 640)
-//   ZED_ONNX_WORLD_MODEL_PATH     alternative ONNX model for the second image;
+//   ZED_ONNX_OUTPUT_FORMAT     nc_first|nc_last|nc_last_obj|yolov5|yolov8|yolov11|auto
 //                                 falls back to ZED_ONNX_MODEL_PATH if not set
-#include "engines/open_vocab.h"
+#include "engines/yolo_onnx.h"
 #include "util/env.h"
 #include <opencv2/opencv.hpp>
@@ -61,19 +59,28 @@ void draw_detections(cv::Mat& img,
        char buf[64];
        std::snprintf(buf, sizeof(buf), " %.2f", obj.confidence / 100.0f);
        std::string text = name + buf;
        cv::rectangle(img, {x1, y1}, {x2, y2}, {0, 200, 0}, 2);
-        cv::putText(img, text, {x1, std::max(y1 - 6, 14)},
+        cv::putText(img, name + buf, {x1, std::max(y1 - 6, 14)},
                    cv::FONT_HERSHEY_SIMPLEX, 0.55, {0, 200, 0}, 2, cv::LINE_AA);
    }
 }
-bool run_test(engines::OpenVocab& engine, const std::string& image_path) {
+} // namespace
 int main(int argc, char* argv[]) {
    std::string image_path = (argc > 1) ? argv[1] : "test/assets/yolo.png";
    auto config = engines::YoloOnnxConfig::from_env();
    sl::Camera dummy_cam;
    engines::YoloOnnx engine(config);
    engine.enable(dummy_cam);
    cv::Mat img = cv::imread(image_path);
    if (img.empty()) {
        std::cerr << "Error: cannot read image: " << image_path << std::endl;
-        return false;
+        return 1;
    }
    engine.detect(img);
@@ -101,42 +108,6 @@ bool run_test(engines::OpenVocab& engine, const std::string& image_path) {
    std::string result_path = (parent / (stem.string() + "_result.png")).string();
    cv::imwrite(result_path, annotated);
    std::cout << "Saved: " << result_path << std::endl;
    return true;
 }
-} // namespace
+    return 0;
 int main(int argc, char* argv[]) {
    std::string yolo_path      = (argc > 1) ? argv[1] : "test/assets/yolo.png";
    std::string yoloworld_path = (argc > 2) ? argv[2] : "test/assets/yoloworld.png";
    auto config = engines::OpenVocabConfig::from_env();
    // enable() ignores this parameter for OpenVocab
    sl::Camera dummy_cam;
    int failures = 0;
    // --- Test 1: yolo image ---
    {
        std::cout << "\n=== Test 1: " << yolo_path << " ===" << std::endl;
        engines::OpenVocab engine(config);
        engine.enable(dummy_cam);
        if (!run_test(engine, yolo_path)) ++failures;
    }
    // --- Test 2: yoloworld image (optionally with a different model) ---
    {
        std::cout << "\n=== Test 2: " << yoloworld_path << " ===" << std::endl;
        engines::OpenVocabConfig config2 = config;
        const char* world_model = std::getenv("ZED_ONNX_WORLD_MODEL_PATH");
        if (world_model && *world_model)
            config2.model_path = world_model;
        engines::OpenVocab engine(config2);
        engine.enable(dummy_cam);
        if (!run_test(engine, yoloworld_path)) ++failures;
    }
    return failures > 0 ? 1 : 0;
 }
@@ -1,226 +0,0 @@
 #pragma once
 // Minimal CLIP ViT-B/32 BPE tokenizer — header-only, C++17, no external deps.
 //
 // Loads clip_vocab.txt (token\tid, tab-separated) and clip_merges.txt
 // (a b, space-separated BPE pairs) produced by export_clip_text_encoder.py.
 // Returns 77 int64 token IDs compatible with clip_text_encoder.onnx.
 #include <algorithm>
 #include <climits>
 #include <fstream>
 #include <sstream>
 #include <stdexcept>
 #include <string>
 #include <unordered_map>
 #include <vector>
 namespace util {
 class ClipTokenizer {
 public:
    static constexpr int     SEQ_LEN = 77;
    static constexpr int64_t BOS     = 49406;
    static constexpr int64_t EOS     = 49407;
    static constexpr int64_t PAD     = 0;
    void load(const std::string& vocab_path, const std::string& merges_path) {
        load_vocab(vocab_path);
        load_merges(merges_path);
        init_byte_encoder();
    }
    bool is_loaded() const { return !vocab_.empty(); }
    // Tokenize a label string. Returns exactly SEQ_LEN int64 token IDs.
    std::vector<int64_t> tokenize(const std::string& text) const {
        std::vector<int64_t> ids;
        ids.reserve(SEQ_LEN);
        ids.push_back(BOS);
        std::string lower = text;
        std::transform(lower.begin(), lower.end(), lower.begin(),
                       [](unsigned char c) { return static_cast<char>(std::tolower(c)); });
        // Split on whitespace — sufficient for typical detection label names
        std::istringstream ss(lower);
        std::string word;
        while (ss >> word) {
            if (static_cast<int>(ids.size()) >= SEQ_LEN - 1) break;
            // Byte-encode the word via the GPT-2 byte encoder
            std::string encoded;
            encoded.reserve(word.size() * 3);
            for (unsigned char c : word) {
                auto it = byte_encoder_.find(c);
                if (it != byte_encoder_.end())
                    encoded += utf8_of(it->second);
            }
            if (encoded.empty()) continue;
            // Apply BPE
            auto tokens = bpe_encode(encoded);
            for (const auto& tok : tokens) {
                if (static_cast<int>(ids.size()) >= SEQ_LEN - 1) break;
                auto vit = vocab_.find(tok);
                if (vit != vocab_.end())
                    ids.push_back(static_cast<int64_t>(vit->second));
            }
        }
        ids.push_back(EOS);
        ids.resize(SEQ_LEN, PAD);
        return ids;
    }
 private:
    std::unordered_map<std::string, int>    vocab_;          // token string → id
    std::unordered_map<std::string, int>    merge_ranks_;    // "a b" → rank
    std::unordered_map<unsigned char, uint32_t> byte_encoder_; // byte → unicode codepoint
    // -----------------------------------------------------------------
    // Byte encoder — matches Python's bytes_to_unicode() in OpenAI CLIP
    // -----------------------------------------------------------------
    void init_byte_encoder() {
        // Printable ASCII + Latin-1 supplement ranges that map to themselves
        std::vector<int> bs;
        for (int i = '!';  i <= '~';  i++) bs.push_back(i);
        for (int i = 0xA1; i <= 0xAC; i++) bs.push_back(i);
        for (int i = 0xAE; i <= 0xFF; i++) bs.push_back(i);
        std::vector<uint32_t> cs;
        cs.reserve(bs.size());
        for (int b : bs) cs.push_back(static_cast<uint32_t>(b));
        // Remaining bytes map to codepoints starting at U+0100
        int n = 0;
        for (int b = 0; b < 256; b++) {
            if (std::find(bs.begin(), bs.end(), b) == bs.end()) {
                bs.push_back(b);
                cs.push_back(256u + static_cast<uint32_t>(n));
                ++n;
            }
        }
        byte_encoder_.clear();
        for (size_t i = 0; i < bs.size(); i++)
            byte_encoder_[static_cast<unsigned char>(bs[i])] = cs[i];
    }
    // Encode a unicode codepoint as a UTF-8 std::string
    static std::string utf8_of(uint32_t cp) {
        std::string s;
        if (cp < 0x80u) {
            s += static_cast<char>(cp);
        } else if (cp < 0x800u) {
            s += static_cast<char>(0xC0u | (cp >> 6));
            s += static_cast<char>(0x80u | (cp & 0x3Fu));
        } else if (cp < 0x10000u) {
            s += static_cast<char>(0xE0u | (cp >> 12));
            s += static_cast<char>(0x80u | ((cp >> 6) & 0x3Fu));
            s += static_cast<char>(0x80u | (cp & 0x3Fu));
        } else {
            s += static_cast<char>(0xF0u | (cp >> 18));
            s += static_cast<char>(0x80u | ((cp >> 12) & 0x3Fu));
            s += static_cast<char>(0x80u | ((cp >> 6)  & 0x3Fu));
            s += static_cast<char>(0x80u | (cp & 0x3Fu));
        }
        return s;
    }
    // -----------------------------------------------------------------
    // BPE
    // -----------------------------------------------------------------
    // Split a UTF-8 string into individual Unicode character substrings
    static std::vector<std::string> split_utf8_chars(const std::string& s) {
        std::vector<std::string> chars;
        for (size_t i = 0; i < s.size(); ) {
            unsigned char c = static_cast<unsigned char>(s[i]);
            int len = (c < 0x80) ? 1 : (c < 0xE0) ? 2 : (c < 0xF0) ? 3 : 4;
            chars.push_back(s.substr(i, len));
            i += len;
        }
        return chars;
    }
    // BPE-encode a single byte-encoded word, returning the token strings
    std::vector<std::string> bpe_encode(const std::string& word_str) const {
        std::vector<std::string> word = split_utf8_chars(word_str);
        if (word.size() <= 1) return word;
        while (true) {
            // Find the pair with the lowest merge rank
            int best_rank = INT_MAX;
            int best_i    = -1;
            for (int i = 0; i + 1 < static_cast<int>(word.size()); i++) {
                auto it = merge_ranks_.find(word[i] + " " + word[i + 1]);
                if (it != merge_ranks_.end() && it->second < best_rank) {
                    best_rank = it->second;
                    best_i    = i;
                }
            }
            if (best_i < 0) break;
            // Apply merge at best_i
            std::string merged = word[best_i] + word[best_i + 1];
            word.erase(word.begin() + best_i, word.begin() + best_i + 2);
            word.insert(word.begin() + best_i, std::move(merged));
        }
        return word;
    }
    // -----------------------------------------------------------------
    // File loading
    // -----------------------------------------------------------------
    void load_vocab(const std::string& path) {
        std::ifstream f(path);
        if (!f.is_open())
            throw std::runtime_error("ClipTokenizer: cannot open vocab file: " + path);
        std::string line;
        while (std::getline(f, line)) {
            auto tab = line.rfind('\t');
            if (tab == std::string::npos) continue;
            std::string token = unescape(line.substr(0, tab));
            int id = std::stoi(line.substr(tab + 1));
            vocab_[token] = id;
        }
    }
    void load_merges(const std::string& path) {
        std::ifstream f(path);
        if (!f.is_open())
            throw std::runtime_error("ClipTokenizer: cannot open merges file: " + path);
        std::string line;
        int rank = 0;
        while (std::getline(f, line)) {
            if (line.empty() || line[0] == '#') continue;
            auto sp = line.find(' ');
            if (sp == std::string::npos) continue;
            merge_ranks_[line] = rank++;  // key is "a b" as written in the file
        }
    }
    static std::string unescape(const std::string& s) {
        std::string r;
        r.reserve(s.size());
        for (size_t i = 0; i < s.size(); i++) {
            if (s[i] == '\\' && i + 1 < s.size()) {
                switch (s[++i]) {
                    case '\\': r += '\\'; break;
                    case 'n':  r += '\n'; break;
                    case 't':  r += '\t'; break;
                    default:   r += '\\'; r += s[i]; break;
                }
            } else {
                r += s[i];
            }
        }
        return r;
    }
 };
 } // namespace util
Author	SHA1	Message	Date
cristhian	f40e697b2a	add a container build environment Dockerfile provisions CUDA 12.8, ZED SDK 5.2, ONNX Runtime and Rust on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for, so the node can be compiled and the build verified without installing the SDK stack on the host. The compile needs --gpus all: libsl_zed.so links against the CUDA driver API, NVDEC and NVENC, which ship with the display driver rather than the CUDA toolkit and so exist in no image. The container runtime mounts them from the host, and only includes NVDEC/NVENC when the video capability is requested. Runs as the invoking user so build/ and bin/ are not left root-owned, which needs the toolchain outside /root and the ZED SDK directory readable to an arbitrary uid.	2026-09-07 14:33:58 -03:00
cristhian	b4144dfba5	match the cargo profile to CMAKE_BUILD_TYPE The dora bindings were always built with the debug profile and linked from target/debug, so a release build still pulled in unoptimized arrow, zenoh and opentelemetry code -- a 640 MB dora_zed against 48 MB once the profile follows the build type.	2026-09-07 14:33:50 -03:00
cristhian	d62f4fbb29	pin dora-rs to v1.0.1 instead of tracking main Cloning main made the build non-reproducible and free to drift from the dora runtime that launches the node. DORA_GIT_TAG can still pin a tag explicitly, and an installed dora CLI continues to take precedence so the bindings match the runtime, now with a warning when that CLI predates the 1.0 this node targets.	2026-09-07 14:33:50 -03:00
cristhian	647be61b7b	fix ONNX Runtime install layout so CMake can find it The release tarball ships its CMake package files under lib/, but onnxruntimeTargets.cmake resolves the library as <prefix>/lib64. Ubuntu is multiarch, so CMake does not search lib64 for packages and ld.so does not search /usr/local/lib64 at all, leaving find_package(onnxruntime) failing on a stock install. Keep the libraries in lib64, move only the package files to lib/cmake, and add lib64 to the linker path.	2026-09-07 14:33:33 -03:00
cristhian	b16dfcd965	docs: clarify ONNX Runtime dependency and input handling ONNX Runtime is a configure-time dependency for every build, not only for the yolo_onnx engine. The event loop no longer inspects the input id, so it grabs on any input rather than specifically on tick.	2026-09-07 14:30:01 -03:00
cristhian	d6ae0f7fef	remove dead split_labels helper Its only caller was the deleted yoloworld_onnx.cc.	2026-09-07 14:29:52 -03:00
cristhian	f1934c9b4a	remove the YOLO-World open-vocabulary engine Deletes the yoloworld_onnx engine, the CLIP BPE tokenizer, the standalone test binary and both export scripts. Drops the update_vocabulary input and the yoloworld_onnx/open_vocab dispatch from main.cc, along with the now unused Engine::set_labels. Documentation follows.	2026-09-07 05:10:59 -03:00
cristhian aguilera	44eec29b83	refactor open_vocab into yolo_onnx and yoloworld_onnx engines Split the monolithic open_vocab engine into two focused engines: yolo_onnx (closed-vocabulary, any single-input YOLO ONNX) and yoloworld_onnx (open-vocabulary YOLO-World with CLIP encoder). Shared NMS/letterbox/fuse helpers extracted to yolo_helpers.h. Add export_yolo.py script and separate test binaries for each engine.	2026-03-30 04:44:25 -03:00