remove the YOLO-World engine and add a container build to verify it #17

Merged
cristhian merged 7 commits from remove-yoloworld-engine into main 2026-09-07 20:07:26 +00:00
Owner

Summary

Removes the open-vocabulary detection engine so the node carries no CLIP tokenizer, text encoder or runtime vocabulary machinery. The remaining engines are native (ZED SDK TensorRT) and yolo_onnx (closed-vocabulary ONNX).

The review of the first revision blocked on the change never having been compiled. It now ships with the build environment that compiles it, so a reviewer can reproduce the result rather than take it on trust:

docker build -t dora-zed-build .
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
    -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build

Getting that to work surfaced a pre-existing bug: scripts/install_linux_x64.sh installed ONNX Runtime where CMake cannot find it, so find_package(onnxruntime REQUIRED) had never succeeded on a stock host install either. That fix is included.

The removed code remains reachable at tag v0.1.0.

Related Issues

Closes #7

Changes

Removing the engine

  • Delete src/engines/yoloworld_onnx.{cc,h}, src/util/clip_tokenizer.h, src/test_yoloworld_onnx.cc, scripts/export_yoloworld.py and scripts/export_clip_text_encoder.py.
  • CMakeLists.txt: drop yoloworld_onnx.cc from the dora_zed target and remove the test_yoloworld_onnx target entirely.
  • src/main.cc: remove the yoloworld_onnx/open_vocab branch from the ZED_ENGINE dispatch and the update_vocabulary input handler. That handler was the only consumer of event_as_input/input_id, so those go too — every Input event now falls straight through to the grab, which is what it already did for anything other than update_vocabulary.
  • src/engines/engine.h: remove virtual set_labels. label_names stays: main.cc, test_yolo_onnx.cc, yolo_onnx and zed_native all use it.
  • src/engines/yolo_onnx.{h,cc}: remove the now-orphaned set_labels override, and reword the multi-input warning that named YOLO-World.
  • src/engines/yolo_helpers.h: remove split_labels(), dead once its only caller yoloworld_onnx.cc was deleted.
  • README.md / AGENTS.md: drop the engine-table row, the yoloworld_onnx section (ZED_ONNX_TEXT_ENCODER_PATH, ZED_ONNX_CLIP_VOCAB, ZED_ONNX_CLIP_MERGES), the update_vocabulary input row and the entries for the deleted files.

Making it verifiable

  • scripts/install_linux_x64.sh: fix the ONNX Runtime install layout. The tarball ships its CMake package files under lib/, but onnxruntimeTargets.cmake resolves the library as <prefix>/lib64. Ubuntu is multiarch, so CMake does not search lib64 for packages and /usr/local/lib64 is not on the linker path — find_package(onnxruntime) could not succeed. Libraries stay in lib64; only the package files move to lib/cmake, and lib64 goes on the linker path.
  • Dockerfile / .dockerignore: CUDA 12.8, ZED SDK 5.2, ONNX Runtime 1.24.4 and Rust on Ubuntu 24.04 — the newest release Stereolabs ships a ZED SDK for. x86-64 only; Jetson still builds natively via scripts/install_linux_arm.sh.
  • DoraTargets.cmake: pin dora-rs to v1.0.1 rather than tracking main. DORA_GIT_TAG still pins explicitly, and an installed dora CLI takes precedence so the bindings match the runtime, now warning if that CLI predates 1.0.
  • DoraTargets.cmake: match the cargo profile to CMAKE_BUILD_TYPE. The bindings were always built debug and linked from target/debug, so even a release build pulled in unoptimized arrow/zenoh/opentelemetry.

Notes for the reviewer

The container needs --gpus all, and the host an NVIDIA driver plus the NVIDIA Container Toolkit — as the ZED SDK does anyway. libsl_zed.so links against the CUDA driver API, NVDEC (libnvcuvid) and NVENC (libnvidia-encode), which ship with the display driver rather than the CUDA toolkit and therefore exist in no image; the container runtime mounts them from the host, and only includes NVDEC/NVENC when the video capability is requested. Hence NVIDIA_DRIVER_CAPABILITIES=compute,utility,video in the image. Building the image itself needs no GPU.

--user keeps build/ and bin/ owned by the caller. That required moving the Rust toolchain out of /root and relaxing the ZED SDK directory, which the installer leaves as drwxrwx--- root:zed and an arbitrary uid cannot read.

Review feedback

All four findings from the previous review are addressed:

Finding Resolution
issue (blocking) — never configured or built Builds; environment included in this PR
split_labels() left as dead code Removed in d6ae0f7
ONNX Runtime described as yolo_onnx-only, contradicting find_package(... REQUIRED) Reworded in b16dfcd; the underlying install bug fixed in 647be61
AGENTS.md implies a tick id check that no longer exists Reworded in b16dfcd

Validation

  • git grep -i -e yoloworld -e open_vocab -e clip returns no matches
  • README.md documents no open-vocabulary engine and no update_vocabulary input
  • Configures and builds from a clean checkout of this branch, via the two commands above. Produces bin/dora_zed and bin/test_yolo_onnx; ldd resolves every dependency of the result. Verified in both the default and -DCMAKE_BUILD_TYPE=Release configurations, the latter linking dora v1.0.1 and giving a 48 MB dora_zed against 640 MB before the profile fix.

One limit worth stating: this is compile-and-link verification only. The node has not been run against a camera, so the runtime behaviour of the ZED_ENGINE dispatch after the open_vocab branch was removed is unexercised.

# Summary Removes the open-vocabulary detection engine so the node carries no CLIP tokenizer, text encoder or runtime vocabulary machinery. The remaining engines are `native` (ZED SDK TensorRT) and `yolo_onnx` (closed-vocabulary ONNX). The review of the first revision blocked on the change never having been compiled. It now ships with the build environment that compiles it, so a reviewer can reproduce the result rather than take it on trust: ```bash docker build -t dora-zed-build . docker run --rm --gpus all --user "$(id -u):$(id -g)" \ -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build ``` Getting that to work surfaced a pre-existing bug: `scripts/install_linux_x64.sh` installed ONNX Runtime where CMake cannot find it, so `find_package(onnxruntime REQUIRED)` had never succeeded on a stock host install either. That fix is included. The removed code remains reachable at tag `v0.1.0`. ## Related Issues Closes #7 ## Changes ### Removing the engine - Delete `src/engines/yoloworld_onnx.{cc,h}`, `src/util/clip_tokenizer.h`, `src/test_yoloworld_onnx.cc`, `scripts/export_yoloworld.py` and `scripts/export_clip_text_encoder.py`. - `CMakeLists.txt`: drop `yoloworld_onnx.cc` from the `dora_zed` target and remove the `test_yoloworld_onnx` target entirely. - `src/main.cc`: remove the `yoloworld_onnx`/`open_vocab` branch from the `ZED_ENGINE` dispatch and the `update_vocabulary` input handler. That handler was the only consumer of `event_as_input`/`input_id`, so those go too — every `Input` event now falls straight through to the grab, which is what it already did for anything other than `update_vocabulary`. - `src/engines/engine.h`: remove `virtual set_labels`. `label_names` stays: `main.cc`, `test_yolo_onnx.cc`, `yolo_onnx` and `zed_native` all use it. - `src/engines/yolo_onnx.{h,cc}`: remove the now-orphaned `set_labels` override, and reword the multi-input warning that named YOLO-World. - `src/engines/yolo_helpers.h`: remove `split_labels()`, dead once its only caller `yoloworld_onnx.cc` was deleted. - `README.md` / `AGENTS.md`: drop the engine-table row, the `yoloworld_onnx` section (`ZED_ONNX_TEXT_ENCODER_PATH`, `ZED_ONNX_CLIP_VOCAB`, `ZED_ONNX_CLIP_MERGES`), the `update_vocabulary` input row and the entries for the deleted files. ### Making it verifiable - `scripts/install_linux_x64.sh`: fix the ONNX Runtime install layout. The tarball ships its CMake package files under `lib/`, but `onnxruntimeTargets.cmake` resolves the library as `<prefix>/lib64`. Ubuntu is multiarch, so CMake does not search `lib64` for packages and `/usr/local/lib64` is not on the linker path — `find_package(onnxruntime)` could not succeed. Libraries stay in `lib64`; only the package files move to `lib/cmake`, and `lib64` goes on the linker path. - `Dockerfile` / `.dockerignore`: CUDA 12.8, ZED SDK 5.2, ONNX Runtime 1.24.4 and Rust on Ubuntu 24.04 — the newest release Stereolabs ships a ZED SDK for. x86-64 only; Jetson still builds natively via `scripts/install_linux_arm.sh`. - `DoraTargets.cmake`: pin dora-rs to `v1.0.1` rather than tracking `main`. `DORA_GIT_TAG` still pins explicitly, and an installed `dora` CLI takes precedence so the bindings match the runtime, now warning if that CLI predates 1.0. - `DoraTargets.cmake`: match the cargo profile to `CMAKE_BUILD_TYPE`. The bindings were always built debug and linked from `target/debug`, so even a release build pulled in unoptimized arrow/zenoh/opentelemetry. ## Notes for the reviewer The container needs `--gpus all`, and the host an NVIDIA driver plus the NVIDIA Container Toolkit — as the ZED SDK does anyway. `libsl_zed.so` links against the CUDA driver API, NVDEC (`libnvcuvid`) and NVENC (`libnvidia-encode`), which ship with the display driver rather than the CUDA toolkit and therefore exist in no image; the container runtime mounts them from the host, and only includes NVDEC/NVENC when the `video` capability is requested. Hence `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` in the image. Building the image itself needs no GPU. `--user` keeps `build/` and `bin/` owned by the caller. That required moving the Rust toolchain out of `/root` and relaxing the ZED SDK directory, which the installer leaves as `drwxrwx--- root:zed` and an arbitrary uid cannot read. ## Review feedback All four findings from the previous review are addressed: | Finding | Resolution | | --- | --- | | **issue (blocking)** — never configured or built | Builds; environment included in this PR | | `split_labels()` left as dead code | Removed in d6ae0f7 | | ONNX Runtime described as `yolo_onnx`-only, contradicting `find_package(... REQUIRED)` | Reworded in b16dfcd; the underlying install bug fixed in 647be61 | | `AGENTS.md` implies a `tick` id check that no longer exists | Reworded in b16dfcd | ## Validation - [x] `git grep -i -e yoloworld -e open_vocab -e clip` returns no matches - [x] `README.md` documents no open-vocabulary engine and no `update_vocabulary` input - [x] **Configures and builds from a clean checkout of this branch**, via the two commands above. Produces `bin/dora_zed` and `bin/test_yolo_onnx`; `ldd` resolves every dependency of the result. Verified in both the default and `-DCMAKE_BUILD_TYPE=Release` configurations, the latter linking dora `v1.0.1` and giving a 48 MB `dora_zed` against 640 MB before the profile fix. One limit worth stating: this is compile-and-link verification only. The node has not been run against a camera, so the runtime behaviour of the `ZED_ENGINE` dispatch after the `open_vocab` branch was removed is unexercised.
cristhian added 1 commit 2026-09-07 08:08:07 +00:00
Deletes the yoloworld_onnx engine, the CLIP BPE tokenizer, the standalone
test binary and both export scripts. Drops the update_vocabulary input and
the yoloworld_onnx/open_vocab dispatch from main.cc, along with the now
unused Engine::set_labels. Documentation follows.
Author
Owner

/bot review

/bot review
bot reviewed 2026-09-07 08:11:38 +00:00
bot left a comment
Owner

Summary

Deletes the YOLO-World open-vocabulary engine, its CLIP tokenizer, export scripts, test target, and the update_vocabulary runtime-labels mechanism, leaving native and yolo_onnx. The deletion is thorough and internally consistent (no dangling set_labels overrides, YoloWorldOnnx references, or dataflow.yml usages remain at head), but the PR has never been configured or built and leaves a few doc/code loose ends.

Findings

issue (blocking) -- CMakeLists.txt:28
The PR's own validation checklist has "configure and build" unchecked, and there is no CI that builds this branch. Since this change edits two build targets and deletes a translation unit, it must get at least one successful cmake .. && make on a machine with the ZED SDK before merging; static inspection here shows no dangling references, but the author-stated risk of an unbuilt merge stands.

suggestion (non-blocking) -- AGENTS.md:37
The updated bullet still lists split_labels() as a helper "used by the ONNX engine", but its only caller was the deleted yoloworld_onnx.cc (auto label_vec = split_labels(labels_str);); after this PR it is dead code in src/engines/yolo_helpers.h. Drop the function and remove it from the bullet.

suggestion (non-blocking) -- README.md:35
The reworded row "Required only for the yolo_onnx engine" (and the matching note in scripts/install_linux_arm.sh:15) contradicts CMakeLists.txt:16, where find_package(onnxruntime REQUIRED) makes ONNX Runtime a hard configure-time dependency even for native-only builds. Either make the dependency optional (guard find_package and the yolo-dependent sources/targets) or reword to "required at configure time; used at runtime only by yolo_onnx".

nitpick (non-blocking) -- AGENTS.md:39
The rewritten bullet says "On tick input: calls camera.grab()", but with input_id parsing removed the loop now grabs and publishes on any Input event without inspecting the id. Worth rewording to "on any input event (intended to be driven by a tick timer)" so the doc doesn't imply an id check that no longer exists anywhere.

Recommendation

Request changes -- the deletion itself is clean, but the PR must be built before merge, which its own checklist flags as pending.

## Summary Deletes the YOLO-World open-vocabulary engine, its CLIP tokenizer, export scripts, test target, and the `update_vocabulary` runtime-labels mechanism, leaving `native` and `yolo_onnx`. The deletion is thorough and internally consistent (no dangling `set_labels` overrides, `YoloWorldOnnx` references, or `dataflow.yml` usages remain at head), but the PR has never been configured or built and leaves a few doc/code loose ends. ## Findings **issue (blocking)** -- `CMakeLists.txt:28` The PR's own validation checklist has "configure and build" unchecked, and there is no CI that builds this branch. Since this change edits two build targets and deletes a translation unit, it must get at least one successful `cmake .. && make` on a machine with the ZED SDK before merging; static inspection here shows no dangling references, but the author-stated risk of an unbuilt merge stands. **suggestion (non-blocking)** -- `AGENTS.md:37` The updated bullet still lists `split_labels()` as a helper "used by the ONNX engine", but its only caller was the deleted `yoloworld_onnx.cc` (`auto label_vec = split_labels(labels_str);`); after this PR it is dead code in `src/engines/yolo_helpers.h`. Drop the function and remove it from the bullet. **suggestion (non-blocking)** -- `README.md:35` The reworded row "Required only for the `yolo_onnx` engine" (and the matching note in `scripts/install_linux_arm.sh:15`) contradicts `CMakeLists.txt:16`, where `find_package(onnxruntime REQUIRED)` makes ONNX Runtime a hard configure-time dependency even for `native`-only builds. Either make the dependency optional (guard `find_package` and the yolo-dependent sources/targets) or reword to "required at configure time; used at runtime only by yolo_onnx". **nitpick (non-blocking)** -- `AGENTS.md:39` The rewritten bullet says "On `tick` input: calls `camera.grab()`", but with `input_id` parsing removed the loop now grabs and publishes on *any* `Input` event without inspecting the id. Worth rewording to "on any input event (intended to be driven by a tick timer)" so the doc doesn't imply an id check that no longer exists anywhere. ## Recommendation **Request changes** -- the deletion itself is clean, but the PR must be built before merge, which its own checklist flags as pending.
cristhian added 2 commits 2026-09-07 17:27:07 +00:00
Its only caller was the deleted yoloworld_onnx.cc.
ONNX Runtime is a configure-time dependency for every build, not only
for the yolo_onnx engine. The event loop no longer inspects the input
id, so it grabs on any input rather than specifically on tick.
cristhian added 4 commits 2026-09-07 17:31:02 +00:00
The release tarball ships its CMake package files under lib/, but
onnxruntimeTargets.cmake resolves the library as <prefix>/lib64. Ubuntu
is multiarch, so CMake does not search lib64 for packages and ld.so does
not search /usr/local/lib64 at all, leaving find_package(onnxruntime)
failing on a stock install. Keep the libraries in lib64, move only the
package files to lib/cmake, and add lib64 to the linker path.
Cloning main made the build non-reproducible and free to drift from the
dora runtime that launches the node. DORA_GIT_TAG can still pin a tag
explicitly, and an installed dora CLI continues to take precedence so
the bindings match the runtime, now with a warning when that CLI
predates the 1.0 this node targets.
The dora bindings were always built with the debug profile and linked
from target/debug, so a release build still pulled in unoptimized arrow,
zenoh and opentelemetry code -- a 640 MB dora_zed against 48 MB once the
profile follows the build type.
Dockerfile provisions CUDA 12.8, ZED SDK 5.2, ONNX Runtime and Rust on
Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for, so the
node can be compiled and the build verified without installing the SDK
stack on the host.

The compile needs --gpus all: libsl_zed.so links against the CUDA driver
API, NVDEC and NVENC, which ship with the display driver rather than the
CUDA toolkit and so exist in no image. The container runtime mounts them
from the host, and only includes NVDEC/NVENC when the video capability
is requested.

Runs as the invoking user so build/ and bin/ are not left root-owned,
which needs the toolchain outside /root and the ZED SDK directory
readable to an arbitrary uid.
cristhian changed title from remove the YOLO-World open-vocabulary engine to remove the YOLO-World engine and add a container build to verify it 2026-09-07 17:31:43 +00:00
Author
Owner

/bot verify

/bot verify
Author
Owner

/bot review

/bot review
bot reviewed 2026-09-07 20:06:49 +00:00
bot left a comment
Owner

Summary

This PR removes the YOLO-World open-vocabulary engine (engine class, CLIP tokenizer, export scripts, test target, and all doc references) while keeping native and yolo_onnx, and adds a reproducible Ubuntu 24.04 build container plus dora pinning to v1.0.1 with cargo profile matching, along with a fix for the ONNX Runtime CMake package layout. The removal is complete and coherent — no orphaned set_labels/split_labels/event_as_input references, and the README, AGENTS.md and install scripts all match the remaining code. Validation is compile-and-link only, as the author states; runtime behavior after the open_vocab dispatch removal is unexercised.

Findings

None.

Recommendation

Approve — the removal is thorough, the build/verify environment is self-consistent with the documented layout fixes, and no correctness or design defects block merge.

## Summary This PR removes the YOLO-World open-vocabulary engine (engine class, CLIP tokenizer, export scripts, test target, and all doc references) while keeping `native` and `yolo_onnx`, and adds a reproducible Ubuntu 24.04 build container plus dora pinning to `v1.0.1` with cargo profile matching, along with a fix for the ONNX Runtime CMake package layout. The removal is complete and coherent — no orphaned `set_labels`/`split_labels`/`event_as_input` references, and the README, AGENTS.md and install scripts all match the remaining code. Validation is compile-and-link only, as the author states; runtime behavior after the `open_vocab` dispatch removal is unexercised. ## Findings None. ## Recommendation **Approve** — the removal is thorough, the build/verify environment is self-consistent with the documented layout fixes, and no correctness or design defects block merge.
cristhian merged commit 1dd1cefcc9 into main 2026-09-07 20:07:26 +00:00
Sign in to join this conversation.