diff --git a/.gitea/workflows/validate.yml b/.gitea/workflows/validate.yml index e7e8878..24ed5ef 100644 --- a/.gitea/workflows/validate.yml +++ b/.gitea/workflows/validate.yml @@ -1,5 +1,6 @@ -# Checks both dataflows against the typed contract in dora-node.yml. This runs -# no node, so it needs neither a camera nor the ZED SDK. +# Two gates, neither of which needs a camera, the ZED SDK or CUDA: the dataflows +# are checked against the typed contract in dora-node.yml without running the +# node, and the payload packing is compiled and tested on its own. name: validate on: [push, pull_request] @@ -28,9 +29,22 @@ jobs: # strict_types: true in each dataflow turns a type warning into a # non-zero exit, so a mismatched annotation fails here rather than at - # runtime. The plot file is checked too: it wires the same outputs, and a - # rename that breaks only the viewer should still fail the build. + # runtime. The viewer file is checked too: it wires the same outputs, and + # a rename that breaks only the consumer should still fail the build. - name: Validate the dataflows run: | dora validate dataflow.yml - dora validate dataflow_plot.yml + dora validate dataflow_view.yml + + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install uv + run: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.local/bin" >> "$GITHUB_PATH" + + - name: Test the payload packing + run: tests/run.sh diff --git a/AGENTS.md b/AGENTS.md index 8f7ca9f..dfc0fe8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,10 +4,18 @@ This file provides guidance to AI coding assistants when working with code in th ## What this is -A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame and publishes it over the dora dataflow: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Camera management lives in `src/zed/` and the dora event loop in `src/main.cc`. +A dora-rs C++ node that interfaces with ZED stereo cameras. Capture is free-running on its own thread at `ZED_FPS`; on each tick the event loop publishes the newest frame it has produced, or nothing if none arrived since the last tick. What goes out: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Every payload is a typed Arrow array — the two frames as `std/media/v1/Image`, the rest as `std/core/v1/Float32` and `.../String` — which is what lets the node feed [dora-image-view](https://gitea.liria.cl/liria/dora-image-view) and anything else built against those types. Camera management lives in `src/zed/`, payload packing in `src/arrow/`, and the dora event loop in `src/main.cc`. The node is a camera source only. It ran inference in-process through `v0.1.0`, under `src/engines/`; that was removed so the node has no inference dependency and builds on any platform the ZED SDK supports. The removed code remains reachable at tag `v0.1.0`. +## Test + +```bash +tests/run.sh +``` + +Needs `g++` and `uv`; no camera, no ZED SDK, no CUDA. + ## Build ```bash @@ -28,8 +36,8 @@ For a build without installing the SDK stack on the host, see the `Dockerfile` ( ```bash dora up -dora start dataflow.yml # camera alone -dora start dataflow_plot.yml # camera plus the opencv-plot viewer +dora start dataflow.yml # camera alone +dora start dataflow_view.yml --uv # camera plus dora-image-view ``` The two dataflows both declare the camera node, so run one at a time. @@ -42,20 +50,22 @@ Camera behavior is configured entirely through environment variables (see README - **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: `open()` (one attempt, returns success), warmup, grab (image + depth + point cloud), calibration, device timestamp, blank-frame check, `close()` and `reset()` (close plus `sl::Camera::reboot()` where the model allows it). Exposes `handle()` for engine access. - **`src/zed/capture.h/cc`** — `zed::Capture` and `zed::Frame`. Owns the camera on a dedicated thread: open, warmup, grab, blank/stall detection, backoff and escalation all live here, along with the packing of every payload into owned buffers. The newest frame is handed to the reader by swapping under a mutex (`take_frame`), which also recycles the previous frame's buffers so steady-state capture does not reallocate. **Nothing outside this file may call the ZED SDK** — it blocks unboundedly on a disconnected camera, which is what froze the node when the loop drove the camera directly. - **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace. `parse_depth_mode` runs in `Config::from_env()`, not in the constructor, so an unrecognised `ZED_DEPTH_MODE` exits before the node joins the dataflow rather than falling back to a default. -- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()`, then sends `image_bgr` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `pack_depth` copies the depth map into a tightly packed `[H, W]` float buffer (sl::Mat rows can be padded) and clears `inf`/`nan` to `0`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds). Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension. The loop never touches the SDK: it drains `take_state_change` and `take_frame` from the capture thread and publishes. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node. +- **`src/arrow/array.h/cc`** — Builds the Arrow arrays the node publishes, against the Arrow C data interface: the two ABI structs are defined in the header rather than pulled from Arrow C++, which would be a heavy dependency for a producer that exports one array per output and never reads one. `image()` packs an `std/media/v1/Image` row, `float32()` and `string()` the primitive payloads. The payload bytes are **borrowed**, not copied — they only have to outlive the send, since dora IPC-encodes the array into shared memory inside `send_arrow_output` and then calls the release callbacks. Each node's release frees its children before itself, so releasing the root frees the tree. +- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected) it drains `take_state_change` and `take_frame` from the capture thread and publishes `image` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `send()` hands one `arrow_c::Exported` to `send_arrow_output`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds); nothing about a frame's shape or encoding travels as metadata any more, since the Image struct carries it. The loop never touches the SDK. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node. - **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue. - **`scripts/build.sh`** — Configure/compile/install in one shell-free-callable step. Referenced by both dataflows, the `Dockerfile` and the README. -- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why both dataflows build from the checkout. Type URNs name the element type (`std/core/v1/UInt8`, `.../Float32`, `.../String`) because every payload is a raw buffer described by its metadata, not an Arrow struct. +- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why both dataflows build from the checkout. `image` and `depth` are `std/media/v1/Image`; dora's compatibility graph has no edge from `std/core/v1/UInt8` to it, so a consumer declaring the struct rejects a producer declaring the buffer whatever it actually sends — the manifest has to move with the payload. - **`dataflow.yml`** — Example dora dataflow: the camera alone, no viewer and no Python dependency. Built from this checkout with `path:` + `build:` and `strict_types: true`. A consumer wires the node from a tag instead (see README) — dora 1.0 pins a source with the mutually exclusive `tag:`, `rev:` or `branch:`. -- **`dataflow_plot.yml`** — The same camera node plus an `opencv-plot` viewer on `image_bgr`. Split out because the viewer pulls dora-hub over `pip` and needs a display. dora 1.0 has no cross-dataflow wiring (an input's `source` must resolve to a node in the same descriptor), so the camera node is duplicated here rather than referenced, and the two files are run one at a time. -- **`.gitea/workflows/validate.yml`** — CI gate running `dora validate` on both dataflows on push and pull request. Needs no camera and no ZED SDK. +- **`dataflow_view.yml`** — The same camera node plus [dora-image-view](https://gitea.liria.cl/liria/dora-image-view) on `image`, `depth` and `camera_info`, fetched from a tag. Run it with `--uv`, which gives the viewer its own managed environment. The viewer's manifest is not read at validate time (only a `path:` node's is), so the file repeats its contract in `input_types:` to get the two checked against each other. dora 1.0 has no cross-dataflow wiring (an input's `source` must resolve to a node in the same descriptor), so the camera node is duplicated here rather than referenced, and the two files are run one at a time. +- **`tests/run.sh`** — Builds and runs the payload tests: `tests/arrow_test.cc` under `-fsanitize=address,undefined` (the release callbacks have to free the whole tree), and `tests/test_arrow.py`, which imports the exported arrays through pyarrow's C data interface and checks the values. `tests/arrow_export.cc` is the C shim the Python side calls. Covers `src/arrow` only, so it needs no camera, no ZED SDK and no dora. +- **`.gitea/workflows/validate.yml`** — CI gates on push and pull request: `dora validate` on both dataflows, and `tests/run.sh`. Neither needs a camera, the ZED SDK or CUDA. - **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`. ## Key APIs -- **dora C++ API** (``): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`. +- **dora C++ API** (``): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_arrow_output()`, `new_metadata()`. - **ZED SDK** (``): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`. -- Output data is sent as raw byte slices (`rust::Slice`) with typed metadata (shape, dtype, encoding). +- Output data is sent as Arrow arrays: `send_arrow_output` takes an `ArrowArray`/`ArrowSchema` pair by pointer and moves both out, so a payload is built once with `src/arrow` and handed over. ## Tools diff --git a/CMakeLists.txt b/CMakeLists.txt index 0fbd2c8..e2dc86f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,5 +1,5 @@ cmake_minimum_required(VERSION 3.21) -project(dora_zed VERSION 0.2.0 LANGUAGES C CXX) +project(dora_zed VERSION 0.3.0 LANGUAGES C CXX) set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_FLAGS "-fPIC") @@ -24,6 +24,7 @@ link_directories(${ZED_LIBRARY_DIR}) add_executable(dora_zed src/main.cc + src/arrow/array.cc src/zed/camera.cc src/zed/capture.cc ${node_bridge} diff --git a/README.md b/README.md index 4380ebe..b577a48 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # dora-zed -Dora node for ZED stereo cameras. Publishes a BGR image and a depth map on every frame, plus camera intrinsics, a camera state, and an optional point cloud. +Dora node for ZED stereo cameras. On every tick it publishes a colour image and a depth +map as `std/media/v1/Image`, plus camera intrinsics, a camera state, and an optional +point cloud. A source for a vision pipeline: it takes a tick and produces frames. ## Usage @@ -9,100 +11,169 @@ Add to your dataflow: ```yaml nodes: - id: zed-camera - git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git - tag: v0.2.0 + git: https://gitea.liria.cl/liria/dora-zed.git + tag: v0.3.0 build: scripts/build.sh path: bin/dora_zed inputs: tick: dora/timer/millis/100 outputs: - - image_bgr + - image - depth - point_cloud - camera_info - camera_state ``` -### Runnable examples +The node is C++: building it needs the ZED SDK and CUDA on the machine that runs the +dataflow — see [Build](#build). An output left out of `outputs:` is simply not published, +so a dataflow that never turns the point cloud on can drop that line. -Two dataflows live here, and they are run one at a time — dora has no cross-dataflow -wiring, so each declares the camera node itself: +### Runnable dataflows | File | Contents | Run | |---|---|---| -| [`dataflow.yml`](dataflow.yml) | The camera alone | `dora run dataflow.yml` | -| [`dataflow_plot.yml`](dataflow_plot.yml) | The camera plus an `opencv-plot` viewer on `image_bgr` | `dora run dataflow_plot.yml` | +| [`dataflow.yml`](dataflow.yml) | The camera alone, no viewer and no Python dependency | `dora run dataflow.yml` | +| [`dataflow_view.yml`](dataflow_view.yml) | The camera plus [dora-image-view][view] | `dora run dataflow_view.yml --uv` | -The viewer is a separate file because it pulls [dora-hub](https://github.com/dora-rs/dora-hub) -and installs it with `pip`, and it needs a display. Neither is a cost the camera should -carry on a headless robot, so `dataflow.yml` has no Python dependency at all. +[view]: https://gitea.liria.cl/liria/dora-image-view -Both build the node from the checkout (`path:` + `build:`) rather than from a tag, which -is what lets `dora validate` pick up the manifest described below. +`dataflow_view.yml` is the worked example of consuming this node: a window opens with +the colour image and the depth map side by side, the camera model captioned across them. +Press `q` to quit. It needs a display, and `--uv` so the viewer gets its own managed +environment. -### Node manifest +dora has no cross-dataflow wiring, so both files declare the camera node themselves: run +one at a time. Both build it from the checkout, which is what lets `dora validate` check +them against [`dora-node.yml`](dora-node.yml) — the typed contract a consumer wires +against, and what [CI](.gitea/workflows/validate.yml) validates on every push. -[`dora-node.yml`](dora-node.yml) declares the node's inputs and outputs with type URNs. -No source package is shared between node repositories, so it is the only contract a -consumer has, and it is what makes a wiring mistake fail at build time instead of at -runtime: +## Inputs -``` -$ dora validate dataflow.yml - node "zed-camera": injected 6 contract type(s) from .../dora-node.yml -All type annotations OK. -``` - -Types are `std/core/v1/UInt8` for `image_bgr`, `std/core/v1/Float32` for `depth`, -`camera_info` and `point_cloud`, and `std/core/v1/String` for `camera_state`; every -payload is a raw buffer whose shape and dtype are in the metadata, so the URN names the -element type rather than a struct. The `tick` input is `std/core/v1/UInt64`, matching how -dora types timer inputs. - -`strict_types: true` in both dataflows makes a type warning a non-zero exit. -[`.gitea/workflows/validate.yml`](.gitea/workflows/validate.yml) validates both on every -push and pull request; it needs no camera and no ZED SDK, since validation never runs the -node. - -## Dependencies - -| Dependency | Version | Notes | -|---|---|---| -| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time | -| [CMake](https://cmake.org/download/) | 3.21+ | | -| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` | -| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK | -| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings | - -### Installation scripts - -| Platform | Script | +| Input | Description | |---|---| -| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` | -| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` | -| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` | +| `tick` | Publishes the newest frame captured since the last tick, or nothing. Wire to `dora/timer/millis/N`. | -> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release). +The tick does not drive the camera. Capture is free-running on its own thread, paced by +the SDK to `ZED_FPS`, and each frame overwrites whatever the previous one left +uncollected; a tick forwards the newest there is, and publishes nothing when none arrived +since the last one. So the tick sets the publish rate, never the capture rate: below +`ZED_FPS` it drops frames — the default 100 ms tick against the default 15 fps keeps two +out of every three — and above it, the ticks that find nothing publish nothing. +`dora/timer/millis/66` at 15 fps forwards every frame, each at most one capture period +old and none of them twice. -## Local build +## Outputs + +| Output | Payload | Metadata | +|---|---|---| +| `image` | `std/media/v1/Image` struct, `bgr8` | intrinsics, `frame_id`, timestamps | +| `depth` | `std/media/v1/Image` struct, `32FC1` | `units=mm`, intrinsics, `frame_id`, timestamps | +| `camera_info` | `float32` `[fx, fy, cx, cy]` | `distortion`, `frame_id`, timestamps | +| `point_cloud` | `float32` XYZRGBA, off by default | `shape`, `channels`, `units=mm`, intrinsics, timestamps | +| `camera_state` | `string` | `host_timestamp_ns` | + +`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`, +`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself, +so nothing about its shape travels as message metadata. A depth map is an image whose +pixels are distances, as ROS also treats it: `32FC1` is one float32 per pixel, `units` +says what those floats measure, and `0` means missing depth — the payload contains no +`inf` and no `nan`. `depth` is published on every frame `image` is, at the same size and +`frame_id`. + +`camera_info` is published **once per camera session** — with the first frame after the +camera opens, and again after a reconnection — since the intrinsics are fixed for a given +camera and resolution. Every frame also carries `fx`, `fy`, `cx` and `cy` in its +metadata, so a consumer that started after the camera did still ends up with them. + +Every output of a frame carries the same `frame_id` and both clocks as dora `timestamp` +metadata: `device_timestamp_ns`, when the camera captured the frame, and +`host_timestamp_ns`, when this node published it — both Unix-epoch nanoseconds. Two nodes +reading the same frame agree on the device clock; the rest of the dataflow shares the +host one. + +### Camera state + +`camera_state` lets a consumer tell a camera that has not delivered its first frame yet +from one that has been lost: + +| State | Meaning | +|---|---| +| `opening` | The camera has not delivered a usable frame yet in this run | +| `streaming` | Frames are being published | +| `recovering` | The camera was streaming and has stopped; the node is bringing it back | + +The camera runs on its own thread, since the SDK can block indefinitely inside `grab()` +and `open()` on a disconnected camera and would otherwise freeze the event loop. A camera +that delivers no usable frame for 5 seconds is recovered, reopening on a 1, 2, 5 then 10 +second backoff and escalating to `sl::Camera::reboot()` past the first attempt; an +all-black frame counts as a failed grab, since a camera whose link has dropped can keep +reporting success while handing back a uniform one. Uncollected frames are dropped, so a +consumer always gets the newest one. + +## Configuration + +Settings are environment variables, which a dataflow can carry per node, keeping them +with the wiring rather than in whatever shell starts it: + +```yaml + env: + ZED_RESOLUTION: 720 + ZED_FPS: 30 +``` + +A node's `env:` wins over the environment the dataflow was started in; variables it does +not set still pass through. + +| Variable | Default | Meaning | +|---|---|---| +| `ZED_RESOLUTION` | `1080` | `720`, `1080` or `2000` | +| `ZED_FPS` | `15` | Frame rate | +| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY` or `NONE`. An unrecognised value is a startup error, not a fallback. | +| `ZED_DEPTH_MIN_MM` | `200` | Near end of the depth range, in millimetres | +| `ZED_DEPTH_MAX_MM` | `600` | Far end of the depth range, in millimetres | +| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) | +| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) | +| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. | +| `ZED_FLIP` | `ON` | `ON` or `OFF` | +| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup | +| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud | +| `ZED_PC_MODE` | `FULL` | `FULL` is a `[H, W, 4]` grid; `HALF` is a stride-2 downsample. Only applies with `ZED_POINT_CLOUD` on. | + +The defaults match the reference camera driver the perception stack was tuned against, and +the resolved depth settings are logged at startup. + +The point cloud is off because it is expensive: roughly 14.7 MB per frame at 1080p, paid +in retrieval and in a copy whether or not anything consumes it. Consumers that need 3D +back-project `depth` with the published intrinsics instead. + +## Build ```bash -git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git -cd dora-zed scripts/build.sh ``` -Binary installed to `bin/dora_zed`. +The binary is installed to `bin/dora_zed`. The script is also the dataflow's `build:` +command, since dora 1.0 runs build commands directly rather than through a shell — `cd` +and `&&` are not available there. The first build clones dora-rs and compiles its C++ +bindings with Cargo, which takes a while; later builds are incremental, and +`DORA_ROOT_DIR` builds against a local dora checkout instead. -`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build -commands directly rather than through a shell, so `cd`, `&&` and command substitution are -not available there; keeping the sequence in one script is what makes the dataflow build -work. It configures a `Release` build, which is also what links the optimized dora -bindings. +**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x at +`/usr/local/zed` (it bundles OpenCV), the CUDA Toolkit it needs, and a stable Rust +toolchain for the dora bindings. -### Containerized build +| Platform | Install the toolchain with | +|---|---| +| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` | +| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` | +| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1`, plus the [ZED SDK][sdk] by hand | -[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed. +[sdk]: https://www.stereolabs.com/developers/release + +[`Dockerfile`](Dockerfile) carries the same toolchain on Ubuntu 24.04 — the newest release +Stereolabs ships a ZED SDK for — so the host needs none of it, only an NVIDIA driver and +the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html): ```bash docker build -t dora-zed-build . @@ -110,134 +181,18 @@ docker run --rm --gpus all --user "$(id -u):$(id -g)" \ -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build ``` -The container runs the same `scripts/build.sh`. +It runs the same `scripts/build.sh`. The named volume keeps the Cargo cache across runs +and `--user` keeps `build/` and `bin/` owned by you. x86-64 only: on Jetson, run +[`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) and build natively. -The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs. -`--user` keeps `build/` and `bin/` owned by you rather than root. +## Test -x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the -device and build natively — there is no ARM64 image here. +```bash +tests/run.sh +``` -Needs an NVIDIA driver and the -[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) -on the host, as the ZED SDK does anyway. The image sets -`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC -(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host -only when `video` is in the capability list. - -## Configuration - -All behavior is controlled via environment variables. - -### Camera - -| Variable | Default | Description | -|---|---|---| -| `ZED_RESOLUTION` | `1080` | `720`, `1080`, or `2000` | -| `ZED_FPS` | `15` | Frame rate | -| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY`, or `NONE`. An unrecognised value is a startup error, not a fallback. | -| `ZED_DEPTH_MIN_MM` | `200` | Minimum depth (mm) | -| `ZED_DEPTH_MAX_MM` | `600` | Maximum depth (mm) | -| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) | -| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) | -| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. | -| `ZED_FLIP` | `ON` | `ON` or `OFF` | -| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup | - -The resolved depth mode, both confidence thresholds and the depth range are logged at -startup. The defaults match the reference camera driver the perception stack was tuned -against, so depth quality on the robot matches what it expects. - -### Point cloud - -Off by default. With `ZED_POINT_CLOUD` unset the XYZRGBA measure is not retrieved at all -and no `point_cloud` output is published — at 1080p a full cloud is roughly 14.7 MB per -frame, and retrieving it costs capture time whether or not anything consumes it. -Consumers that need 3D back-project `depth` with the published intrinsics instead. - -| Variable | Default | Description | -|---|---|---| -| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud | -| `ZED_PC_MODE` | `FULL` | Only applies when `ZED_POINT_CLOUD` is on | - -| `ZED_PC_MODE` | Shape | Description | -|---|---|---| -| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 1080p) | -| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) | - -The cloud is copied out of the SDK buffer, since the capture thread and the event loop are -different threads. That copy is the main reason to leave `ZED_POINT_CLOUD` off unless a -consumer needs it. - -## Inputs - -| Input | Description | -|---|---| -| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. | - -## Outputs - -| Output | Type | Description | -|---|---|---| -| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, timestamps. | -| `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. | -| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud, only when `ZED_POINT_CLOUD` is on. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. | -| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. Published **once per camera session** — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution. | -| `camera_state` | `string` | UTF-8 state name, published on each change. Metadata: `dtype=string`, `encoding=utf8`, `host_timestamp_ns`. | - -### Camera state and recovery - -`camera_state` lets a downstream node tell a camera that has not delivered its first -frame yet from one that has been lost: - -| State | Meaning | -|---|---| -| `opening` | The camera has not delivered a usable frame yet in this run | -| `streaming` | Frames are being published | -| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back | - -The camera runs on its own thread. The ZED SDK can block for an unbounded time inside -`grab()` and `open()` when the camera is disconnected — measured blocking indefinitely on -a ZED-M even with `async_grab_camera_recovery` set — so driving it from the dora event -loop would freeze the node: it would stop reading events, leaving `Stop` unanswered and -the recovery logic itself unreachable. The event loop only reads the newest frame the -capture thread has already produced, and therefore never calls into the SDK. - -Reopening is paced by a backoff of 1, 2, 5 then 10 seconds, the last repeated. Frames that -are not collected before the next one arrives are dropped: a consumer always gets the -newest frame rather than a queue of stale ones. - -The node treats an all-black frame as a failed grab: a camera whose link has dropped does -not necessarily fail its grabs, and can keep reporting success while handing back a -uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30 -seconds if the SDK reports it is reconnecting the link itself, since taking the handle -away mid-reconnect aborts a recovery that was already under way. - -The first recovery attempt closes the handle. Past that the device itself is wedged, and -the node escalates to `sl::Camera::reboot()`, which resets it over USB without unplugging -it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED -Mini the node closes the handle and lets the backoff give the endpoint time to clear. -Every attempt and every escalation is logged. - -### Timestamps - -Every frame output — `image_bgr`, `depth`, `point_cloud` and `camera_info` — carries both -clocks, as dora `timestamp` metadata parameters. `camera_state` carries -`host_timestamp_ns` only, since it is not tied to a frame. - -| Key | Epoch | Unit | Source | -|---|---|---|---| -| `device_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE)` — when the camera captured the frame | -| `host_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `std::chrono::system_clock` on the host — when this node published the frame | - -Both are needed. The device timestamp is the capture instant and is the value two -nodes reading the same frame agree on; the host timestamp is the one the rest of the -dataflow and the session recorder share. On a Jetson the wall clock can step when NTP -synchronises after boot, so the device timestamp is recorded alongside the host one -rather than in place of it. - -The node calls `sl::setTimestampClock(SYSTEM_CLOCK)` before opening the camera, which -is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock. - -Every output for a frame also carries the same `frame_id`, an integer counter starting -at 0. +Unit tests over `src/arrow`, which packs every payload this node sends: each exported +Arrow array is round-tripped through pyarrow to check what a consumer reads back, and +built again under the sanitizers to check that releasing one frees it. They need neither a +camera, the ZED SDK nor dora, which is what lets CI run them; everything past the packing +is covered by running a dataflow against a real camera. diff --git a/dataflow.yml b/dataflow.yml index a1001e2..fd64efa 100644 --- a/dataflow.yml +++ b/dataflow.yml @@ -3,7 +3,7 @@ # A consumer wires the node from a tag instead — see README.md. # # The camera alone, with no viewer and no Python dependency. For a run that -# also displays the stream, use dataflow_plot.yml. +# also displays the stream, use dataflow_view.yml. strict_types: true nodes: @@ -13,7 +13,7 @@ nodes: inputs: tick: dora/timer/millis/100 outputs: - - image_bgr + - image - depth - point_cloud - camera_info diff --git a/dataflow_plot.yml b/dataflow_plot.yml deleted file mode 100644 index 570cada..0000000 --- a/dataflow_plot.yml +++ /dev/null @@ -1,27 +0,0 @@ -# The camera plus an opencv-plot viewer on image_bgr. -# -# Separate from dataflow.yml because the viewer pulls dora-hub and installs it -# with pip, and needs a display: the camera on its own should cost neither. -# dora has no cross-dataflow wiring, so the camera node is declared here too -# rather than referenced — run this file or dataflow.yml, not both. -strict_types: true - -nodes: - - id: zed-camera - path: bin/dora_zed - build: scripts/build.sh - inputs: - tick: dora/timer/millis/100 - outputs: - - image_bgr - - depth - - point_cloud - - camera_info - - camera_state - - - id: plot - git: https://github.com/dora-rs/dora-hub.git - build: pip install ./node-hub/opencv-plot - path: opencv-plot - inputs: - image: zed-camera/image_bgr diff --git a/dataflow_view.yml b/dataflow_view.yml new file mode 100644 index 0000000..11d44df --- /dev/null +++ b/dataflow_view.yml @@ -0,0 +1,46 @@ +# The camera into dora-image-view, and the worked example of wiring this node to +# a consumer. +# +# dora has no cross-dataflow wiring, so the camera node is declared here too +# rather than referenced — run this file or dataflow.yml, not both. +# +# Run it with `--uv`, which gives the viewer its own managed environment: +# +# dora run dataflow_view.yml --uv +# +# Fetching the viewer needs anonymous read access to its repository. dora-cli +# 1.0.1 fetches through libgit2 with no credential callback, so ssh:// cannot +# authenticate at all; on Gitea, marking the repository public is not enough +# either, since a repository inside a private organization answers an anonymous +# clone with 401. +strict_types: true + +nodes: + - id: zed-camera + path: bin/dora_zed + build: scripts/build.sh + inputs: + tick: dora/timer/millis/100 + outputs: + - image + - depth + - point_cloud + - camera_info + - camera_state + + - id: view + git: https://gitea.liria.cl/liria/dora-image-view.git + tag: v0.1.0 + build: uv pip install . + path: dora-image-view + inputs: + image: zed-camera/image + depth: zed-camera/depth + camera_info: zed-camera/camera_info + # The viewer's own manifest says this, but a `git:` node's manifest is not + # read at validate time — only at build. Repeating it here is what makes the + # two contracts checked against each other now rather than at runtime. + input_types: + image: std/media/v1/Image + depth: std/media/v1/Image + camera_info: std/core/v1/Float32 diff --git a/dora-node.yml b/dora-node.yml index 336e30a..ccf2916 100644 --- a/dora-node.yml +++ b/dora-node.yml @@ -7,33 +7,60 @@ apiVersion: 1 name: dora-zed namespace: liria -description: ZED stereo camera source publishing a BGR image, a depth map, intrinsics and an optional point cloud +description: ZED stereo camera source publishing an image, a depth map, intrinsics and an optional point cloud categories: [sensor] keywords: [zed, stereolabs, camera, depth, stereo, jetson] runtime: cpp entrypoint: bin/dora_zed build: scripts/build.sh +dora: ">=1.0" inputs: tick: type: std/core/v1/UInt64 required: true - description: Each tick triggers one camera grab. Connect to dora/timer/millis/N. + description: >- + Publishes the newest frame the capture thread has grabbed, or nothing if + none arrived since the last tick. It does not drive the camera: capture is + free-running at ZED_FPS, so this sets the publish rate and anything + captured between two ticks is dropped. Connect to dora/timer/millis/N. outputs: - image_bgr: - type: std/core/v1/UInt8 - description: BGR image bytes, [H, W, 3]. Shape and encoding are in the metadata. + image: + type: std/media/v1/Image + description: >- + Captured frame as an Arrow struct {width, height, encoding, data}. The + encoding is always bgr8, three bytes per pixel. depth: - type: std/core/v1/Float32 - description: Per-pixel depth in millimetres, [H, W]. Missing depth is 0, never inf or nan. + type: std/media/v1/Image + description: >- + Depth map as an Arrow struct {width, height, encoding, data}, one channel + per pixel -- dora has no depth type, and a depth map is an image whose + pixels are distances, which is how ROS spells it too. The encoding is + 32FC1 and the `units` metadata is mm. Missing depth is 0, never inf or + nan. camera_info: type: std/core/v1/Float32 description: Intrinsics [fx, fy, cx, cy], distortion in the metadata. Published once per camera session. point_cloud: type: std/core/v1/Float32 - description: XYZRGBA point cloud. Published only when ZED_POINT_CLOUD is on. + description: XYZRGBA point cloud, shape in the metadata. Published only when ZED_POINT_CLOUD is on. camera_state: type: std/core/v1/String description: One of opening, streaming, recovering. Published on every change. + +example: | + - id: zed-camera + git: https://gitea.liria.cl/liria/dora-zed.git + tag: v0.3.0 + build: scripts/build.sh + path: bin/dora_zed + inputs: + tick: dora/timer/millis/100 + outputs: + - image + - depth + - point_cloud + - camera_info + - camera_state diff --git a/src/arrow/array.cc b/src/arrow/array.cc new file mode 100644 index 0000000..3336f90 --- /dev/null +++ b/src/arrow/array.cc @@ -0,0 +1,179 @@ +#include "arrow/array.h" + +#include +#include +#include +#include + +namespace arrow_c { +namespace { + +// Producer-owned storage for one schema node. The struct's `format`, `name` and +// `children` pointers point in here, which is why it outlives the call that +// built it and is freed by the release callback rather than by the builder. +struct SchemaData { + std::string format; + std::string name; + std::vector children; +}; + +// Producer-owned storage for one array node. `owned` holds the bytes this +// node's own buffers point into - offset arrays, and scalars that have no +// storage elsewhere. The payload itself is borrowed from the caller. +struct ArrayData { + std::vector owned; + std::vector buffers; + std::vector children; +}; + +// A node's release callback releases its children before freeing itself, so a +// consumer that only releases the root - which is all the interface requires - +// still frees the whole tree. +void release_schema(ArrowSchema* schema) { + auto* data = static_cast(schema->private_data); + + for (ArrowSchema* child : data->children) { + if (child->release != nullptr) { + child->release(child); + } + delete child; + } + + delete data; + schema->release = nullptr; +} + +void release_array(ArrowArray* array) { + auto* data = static_cast(array->private_data); + + for (ArrowArray* child : data->children) { + if (child->release != nullptr) { + child->release(child); + } + delete child; + } + + delete data; + array->release = nullptr; +} + +void set_schema(ArrowSchema& schema, const char* format, const char* name, + std::vector children) { + auto* data = new SchemaData{format, name, std::move(children)}; + + schema.format = data->format.c_str(); + schema.name = data->name.c_str(); + schema.metadata = nullptr; + // Nullable everywhere, which is what pyarrow's defaults produce: the same + // struct built by a Python node has to compare equal to this one. + schema.flags = ARROW_FLAG_NULLABLE; + schema.n_children = static_cast(data->children.size()); + schema.children = data->children.empty() ? nullptr : data->children.data(); + schema.dictionary = nullptr; + schema.release = release_schema; + schema.private_data = data; +} + +// `buffers` may point into `owned`: moving a vector keeps its heap block, so +// pointers taken before the move stay valid after it. +void set_array(ArrowArray& array, int64_t length, std::vector owned, + std::vector buffers, std::vector children) { + auto* data = new ArrayData{std::move(owned), std::move(buffers), std::move(children)}; + + array.length = length; + // Nothing this node sends is null, so the validity bitmap - the first + // buffer of every node below - is left out entirely. + array.null_count = 0; + array.offset = 0; + array.n_buffers = static_cast(data->buffers.size()); + array.n_children = static_cast(data->children.size()); + array.buffers = data->buffers.data(); + array.children = data->children.empty() ? nullptr : data->children.data(); + array.dictionary = nullptr; + array.release = release_array; + array.private_data = data; +} + +// One field of a struct, owned by its parent. +struct Child { + ArrowSchema* schema; + ArrowArray* array; +}; + +Child new_child() { + return {new ArrowSchema{}, new ArrowArray{}}; +} + +void fill_uint32(ArrowSchema& schema, ArrowArray& array, const char* name, uint32_t value) { + std::vector owned(sizeof(value)); + std::memcpy(owned.data(), &value, sizeof(value)); + const void* values = owned.data(); + + set_schema(schema, "I", name, {}); + set_array(array, 1, std::move(owned), {nullptr, values}, {}); +} + +// One string, offsets and characters in a single allocation. +void fill_utf8(ArrowSchema& schema, ArrowArray& array, const char* name, const char* text, + size_t size) { + const int32_t offsets[2] = {0, static_cast(size)}; + + std::vector owned(sizeof(offsets) + size); + std::memcpy(owned.data(), offsets, sizeof(offsets)); + std::memcpy(owned.data() + sizeof(offsets), text, size); + const void* offsets_buffer = owned.data(); + const void* values_buffer = owned.data() + sizeof(offsets); + + set_schema(schema, "u", name, {}); + set_array(array, 1, std::move(owned), {nullptr, offsets_buffer, values_buffer}, {}); +} + +// One byte string. Only the offsets are owned; the bytes are the caller's. +void fill_large_binary(ArrowSchema& schema, ArrowArray& array, const char* name, + const uint8_t* data, size_t size) { + const int64_t offsets[2] = {0, static_cast(size)}; + + std::vector owned(sizeof(offsets)); + std::memcpy(owned.data(), offsets, sizeof(offsets)); + const void* offsets_buffer = owned.data(); + + set_schema(schema, "Z", name, {}); + set_array(array, 1, std::move(owned), {nullptr, offsets_buffer, data}, {}); +} + +} // namespace + +Exported image(uint32_t width, uint32_t height, const char* encoding, const uint8_t* data, + size_t size) { + Child w = new_child(); + fill_uint32(*w.schema, *w.array, "width", width); + + Child h = new_child(); + fill_uint32(*h.schema, *h.array, "height", height); + + Child e = new_child(); + fill_utf8(*e.schema, *e.array, "encoding", encoding, std::strlen(encoding)); + + Child d = new_child(); + fill_large_binary(*d.schema, *d.array, "data", data, size); + + Exported out; + set_schema(out.schema, "+s", "", {w.schema, h.schema, e.schema, d.schema}); + set_array(out.array, 1, {}, {nullptr}, {w.array, h.array, e.array, d.array}); + return out; +} + +Exported float32(const float* values, size_t count) { + Exported out; + set_schema(out.schema, "f", "", {}); + set_array(out.array, static_cast(count), {}, {nullptr, values}, {}); + return out; +} + +Exported string(const char* text, size_t size) { + Exported out; + fill_utf8(out.schema, out.array, "", text, size); + return out; +} + +} // namespace arrow_c diff --git a/src/arrow/array.h b/src/arrow/array.h new file mode 100644 index 0000000..e3502f0 --- /dev/null +++ b/src/arrow/array.h @@ -0,0 +1,75 @@ +#pragma once + +#include +#include + +// The Arrow C data interface, verbatim from the Arrow specification. +// +// These two structs are the whole ABI dora's send_arrow_output takes, so +// defining them here is what lets this node publish typed Arrow arrays without +// linking Arrow C++ - a heavy dependency for a producer that exports one array +// per output and never reads one. +#ifndef ARROW_C_DATA_INTERFACE +#define ARROW_C_DATA_INTERFACE + +#define ARROW_FLAG_DICTIONARY_ORDERED 1 +#define ARROW_FLAG_NULLABLE 2 +#define ARROW_FLAG_MAP_KEYS_SORTED 4 + +struct ArrowSchema { + const char* format; + const char* name; + const char* metadata; + int64_t flags; + int64_t n_children; + struct ArrowSchema** children; + struct ArrowSchema* dictionary; + + void (*release)(struct ArrowSchema*); + void* private_data; +}; + +struct ArrowArray { + int64_t length; + int64_t null_count; + int64_t offset; + int64_t n_buffers; + int64_t n_children; + const void** buffers; + struct ArrowArray** children; + struct ArrowArray* dictionary; + + void (*release)(struct ArrowArray*); + void* private_data; +}; + +#endif // ARROW_C_DATA_INTERFACE + +namespace arrow_c { + +// One array with the schema describing it. +// +// dora moves both structs out of here and calls their release callbacks once it +// has copied the payload into shared memory, so an Exported is built, sent +// once, and never touched again. +struct Exported { + ArrowArray array{}; + ArrowSchema schema{}; +}; + +// std/media/v1/Image: one row of {width, height, encoding, data}. +// +// The payload is borrowed rather than copied, so it has to stay valid until the +// send returns. A capture-thread frame buffer does: the event loop holds the +// frame across the call. +Exported image(uint32_t width, uint32_t height, const char* encoding, const uint8_t* data, + size_t size); + +// std/core/v1/Float32: `count` values, borrowed on the same terms as image(). +Exported float32(const float* values, size_t count); + +// std/core/v1/String: one UTF-8 string. Copied, since every string this node +// sends is a short state name. +Exported string(const char* text, size_t size); + +} // namespace arrow_c diff --git a/src/main.cc b/src/main.cc index e9cd256..1ae38b0 100644 --- a/src/main.cc +++ b/src/main.cc @@ -1,12 +1,15 @@ +#include "arrow/array.h" #include "zed/capture.h" #include #include +#include #include #include #include #include +#include #include namespace { @@ -47,9 +50,18 @@ void set_frame_stamps(Metadata& metadata, const zed::Frame& frame) { metadata.set_timestamp("host_timestamp_ns", frame.host_ns); } -void report(const char* output, const DoraResult& result) { +// Publish one Arrow array. dora moves both structs out and releases them once +// the payload is in shared memory, so the buffers they borrow only have to +// outlive this call - which the frame the caller is holding does. +void send(DoraNode& dora_node, const char* id, arrow_c::Exported payload, + rust::Box metadata) { + auto result = send_arrow_output(dora_node.send_output, id, + reinterpret_cast(&payload.array), + reinterpret_cast(&payload.schema), + std::move(metadata)); + if (!std::string(result.error).empty()) { - std::cerr << "Error sending " << output << ": " << std::string(result.error) << std::endl; + std::cerr << "Error sending " << id << ": " << std::string(result.error) << std::endl; } } @@ -90,17 +102,11 @@ int main() { std::string state; if (capture.take_state_change(state)) { - rust::Slice state_slice{ - reinterpret_cast(state.data()), state.size()}; - auto state_metadata = new_metadata(); - state_metadata->set_string("dtype", "string"); - state_metadata->set_string("encoding", "utf8"); state_metadata->set_timestamp("host_timestamp_ns", host_timestamp_ns()); - report("camera_state", - send_output_with_metadata(dora_node.send_output, "camera_state", state_slice, - std::move(state_metadata))); + send(dora_node, "camera_state", arrow_c::string(state.data(), state.size()), + std::move(state_metadata)); } // No new frame since the last tick: the camera is slower than the timer, @@ -109,44 +115,41 @@ int main() { continue; } - // --- image_bgr --- - rust::Slice image_slice{frame.image_bgr.data(), frame.image_bgr.size()}; + auto width = static_cast(frame.width); + auto height = static_cast(frame.height); + // --- image --- + // The payload carries its own width, height and encoding, so none of + // that travels as metadata: what is left is what describes the frame + // rather than the picture. auto image_metadata = new_metadata(); - image_metadata->set_string("dtype", "uint8"); - image_metadata->set_string("encoding", "bgr8"); - image_metadata->set_string("layout", "HWC"); - image_metadata->set_list_int("shape", make_vector_int({frame.height, frame.width, 3})); set_intrinsics(*image_metadata, frame.calib); set_frame_stamps(*image_metadata, frame); - report("image_bgr", send_output_with_metadata(dora_node.send_output, "image_bgr", - image_slice, std::move(image_metadata))); + send(dora_node, "image", + arrow_c::image(width, height, "bgr8", frame.image_bgr.data(), frame.image_bgr.size()), + std::move(image_metadata)); // --- depth --- - rust::Slice depth_slice{ - reinterpret_cast(frame.depth.data()), - frame.depth.size() * sizeof(float)}; - + // A depth map is an image whose pixels are distances, which is how ROS + // spells it too: 32FC1 is one float32 per pixel, and `units` says what + // those floats measure. auto depth_metadata = new_metadata(); - depth_metadata->set_string("dtype", "float32"); - depth_metadata->set_string("layout", "HW"); depth_metadata->set_string("units", "mm"); - depth_metadata->set_list_int("shape", make_vector_int({frame.height, frame.width})); set_intrinsics(*depth_metadata, frame.calib); set_frame_stamps(*depth_metadata, frame); - report("depth", send_output_with_metadata(dora_node.send_output, "depth", depth_slice, - std::move(depth_metadata))); + send(dora_node, "depth", + arrow_c::image(width, height, "32FC1", + reinterpret_cast(frame.depth.data()), + frame.depth.size() * sizeof(float)), + std::move(depth_metadata)); // --- point_cloud --- if (frame.has_point_cloud) { - rust::Slice pc_slice{ - reinterpret_cast(frame.point_cloud.data()), - frame.point_cloud.size() * sizeof(float)}; - + // Unlike the two images, a cloud is a flat array of values: its + // grid shape is only in the metadata. auto pc_metadata = new_metadata(); - pc_metadata->set_string("dtype", "float32"); pc_metadata->set_string("channels", "XYZRGBA"); pc_metadata->set_string("units", "mm"); pc_metadata->set_string("layout", "HWC"); @@ -155,26 +158,22 @@ int main() { set_intrinsics(*pc_metadata, frame.calib); set_frame_stamps(*pc_metadata, frame); - report("point_cloud", send_output_with_metadata(dora_node.send_output, "point_cloud", - pc_slice, std::move(pc_metadata))); + send(dora_node, "point_cloud", + arrow_c::float32(frame.point_cloud.data(), frame.point_cloud.size()), + std::move(pc_metadata)); } // --- camera_info --- if (frame.camera_info_first) { - float camera_data[4] = {frame.calib.fx, frame.calib.fy, frame.calib.cx, - frame.calib.cy}; - rust::Slice cam_slice{ - reinterpret_cast(camera_data), sizeof(camera_data)}; + const float camera_data[4] = {frame.calib.fx, frame.calib.fy, frame.calib.cx, + frame.calib.cy}; auto cam_metadata = new_metadata(); - cam_metadata->set_string("dtype", "float32"); - cam_metadata->set_string("layout", "C"); - cam_metadata->set_list_int("shape", make_vector_int({4})); cam_metadata->set_list_float("distortion", make_distortion(frame.calib.distortion)); set_frame_stamps(*cam_metadata, frame); - report("camera_info", send_output_with_metadata(dora_node.send_output, "camera_info", - cam_slice, std::move(cam_metadata))); + send(dora_node, "camera_info", arrow_c::float32(camera_data, 4), + std::move(cam_metadata)); } } diff --git a/tests/arrow_export.cc b/tests/arrow_export.cc new file mode 100644 index 0000000..9c594f8 --- /dev/null +++ b/tests/arrow_export.cc @@ -0,0 +1,36 @@ +// C entry points for tests/test_arrow.py. +// +// The Python side cannot call the C++ builders, so each one is wrapped here and +// the payload is passed in from the test rather than fixed in this file. + +#include "arrow/array.h" + +#include + +namespace { + +void hand_over(void* array, void* schema, const arrow_c::Exported& exported) { + // The caller owns the two structs, exactly as dora does: it reads them and + // is then responsible for releasing what they point at. + std::memcpy(array, &exported.array, sizeof(exported.array)); + std::memcpy(schema, &exported.schema, sizeof(exported.schema)); +} + +} // namespace + +extern "C" { + +void zed_export_image(void* array, void* schema, uint32_t width, uint32_t height, + const char* encoding, const uint8_t* data, size_t size) { + hand_over(array, schema, arrow_c::image(width, height, encoding, data, size)); +} + +void zed_export_float32(void* array, void* schema, const float* values, size_t count) { + hand_over(array, schema, arrow_c::float32(values, count)); +} + +void zed_export_string(void* array, void* schema, const char* text, size_t size) { + hand_over(array, schema, arrow_c::string(text, size)); +} + +} // extern "C" diff --git a/tests/arrow_test.cc b/tests/arrow_test.cc new file mode 100644 index 0000000..e5a2b92 --- /dev/null +++ b/tests/arrow_test.cc @@ -0,0 +1,54 @@ +// Release has to free the whole tree: the array nodes, the schema nodes, and +// the storage their buffers point into. Nothing here asserts that directly - +// the sanitizers tests/run.sh builds this with are the assertion, and a node +// that leaks per frame is a node that dies on a long run. + +#include "arrow/array.h" + +#include +#include +#include +#include + +namespace { + +void release(arrow_c::Exported& exported) { + exported.array.release(&exported.array); + exported.schema.release(&exported.schema); + + // A released struct marks itself, which is how a consumer knows not to + // release it twice. + assert(exported.array.release == nullptr); + assert(exported.schema.release == nullptr); +} + +} // namespace + +int main() { + const std::vector pixels(4 * 4 * 3, 7); + const std::vector depth(4 * 4, 1500.0F); + const std::string state = "streaming"; + + for (int i = 0; i < 100; ++i) { + arrow_c::Exported image = + arrow_c::image(4, 4, "bgr8", pixels.data(), pixels.size()); + assert(image.array.length == 1); + assert(image.array.n_children == 4); + release(image); + + arrow_c::Exported depth_image = + arrow_c::image(4, 4, "32FC1", reinterpret_cast(depth.data()), + depth.size() * sizeof(float)); + release(depth_image); + + arrow_c::Exported values = arrow_c::float32(depth.data(), depth.size()); + assert(values.array.length == static_cast(depth.size())); + release(values); + + arrow_c::Exported text = arrow_c::string(state.data(), state.size()); + assert(text.array.length == 1); + release(text); + } + + return 0; +} diff --git a/tests/run.sh b/tests/run.sh new file mode 100755 index 0000000..aa46a79 --- /dev/null +++ b/tests/run.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Build and run the payload tests. +# +# They cover src/arrow only, which is the part of the node that has no ZED SDK, +# no CUDA and no dora dependency - so this runs anywhere, and is what CI runs. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(dirname "$SCRIPT_DIR")" +OUT_DIR="${REPO_DIR}/build/tests" + +mkdir -p "$OUT_DIR" + +# Sanitized: the release callbacks have to free the whole exported tree, and a +# leak there would be one per frame. +g++ -std=c++17 -Wall -Wextra -Werror -g -fsanitize=address,undefined \ + -I "${REPO_DIR}/src" \ + -o "${OUT_DIR}/arrow_test" \ + "${REPO_DIR}/src/arrow/array.cc" "${REPO_DIR}/tests/arrow_test.cc" +"${OUT_DIR}/arrow_test" +echo "ok arrow_test" + +# Unsanitized: pyarrow is the consumer standing in for dora, and loads the same +# code as a shared library. +g++ -std=c++17 -Wall -Wextra -Werror -O2 -shared -fPIC \ + -I "${REPO_DIR}/src" \ + -o "${OUT_DIR}/libzed_arrow.so" \ + "${REPO_DIR}/src/arrow/array.cc" "${REPO_DIR}/tests/arrow_export.cc" +uv run --no-project --with pyarrow python "${REPO_DIR}/tests/test_arrow.py" diff --git a/tests/test_arrow.py b/tests/test_arrow.py new file mode 100644 index 0000000..07dd786 --- /dev/null +++ b/tests/test_arrow.py @@ -0,0 +1,129 @@ +"""Round-trip every exported payload through a real Arrow consumer. + +The node builds its Arrow arrays by hand against the C data interface rather +than linking Arrow C++, so this is what shows that what dora imports on the +other side of `send_arrow_output` is what went in. It needs neither a camera nor +the ZED SDK. +""" + +import ctypes +import pathlib +import sys + +import pyarrow as pa + +LIBRARY = pathlib.Path(__file__).resolve().parent.parent / "build" / "tests" / "libzed_arrow.so" + +# ArrowArray is ten machine words and ArrowSchema nine, and both want the +# 8-byte alignment an array of uint64 gives. +WORDS = 16 +# Index of the `release` field in ArrowArray: a consumer that took ownership +# zeroes it, which is how this test sees the move happen. +RELEASE = 8 + +IMAGE_TYPE = pa.struct([ + ("width", pa.uint32()), + ("height", pa.uint32()), + ("encoding", pa.utf8()), + ("data", pa.large_binary()), +]) + + +def load(): + """Load the test library and declare the argument types of its exports.""" + library = ctypes.CDLL(str(LIBRARY)) + + library.zed_export_image.argtypes = [ + ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint32, ctypes.c_uint32, + ctypes.c_char_p, ctypes.c_char_p, ctypes.c_size_t, + ] + library.zed_export_float32.argtypes = [ + ctypes.c_void_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_float), ctypes.c_size_t, + ] + library.zed_export_string.argtypes = [ + ctypes.c_void_p, ctypes.c_void_p, ctypes.c_char_p, ctypes.c_size_t, + ] + return library + + +def structs(): + """Allocate the ArrowArray and ArrowSchema the producer fills in.""" + return (ctypes.c_uint64 * WORDS)(), (ctypes.c_uint64 * WORDS)() + + +def imported(array, schema): + """Import the pair as pyarrow does, taking ownership of both.""" + result = pa.Array._import_from_c(ctypes.addressof(array), ctypes.addressof(schema)) + assert array[RELEASE] == 0, "the consumer should have taken ownership of the array" + return result + + +def test_image(library): + """An image row carries its own dimensions, encoding and bytes.""" + array, schema = structs() + pixels = bytes(range(12)) + + library.zed_export_image( + ctypes.byref(array), ctypes.byref(schema), 2, 2, b"bgr8", pixels, len(pixels) + ) + result = imported(array, schema) + + assert result.type == IMAGE_TYPE, result.type + assert result.to_pylist() == [ + {"width": 2, "height": 2, "encoding": "bgr8", "data": pixels} + ] + + +def test_depth(library): + """A depth map is an image whose bytes are float32 millimetres.""" + array, schema = structs() + millimetres = [1500.0, 0.0, 250.5, 4000.0] + payload = pa.array(millimetres, type=pa.float32()).buffers()[1].to_pybytes() + + library.zed_export_image( + ctypes.byref(array), ctypes.byref(schema), 2, 2, b"32FC1", payload, len(payload) + ) + row = imported(array, schema).to_pylist()[0] + + assert row["encoding"] == "32FC1" + assert pa.Array.from_buffers( + pa.float32(), 4, [None, pa.py_buffer(row["data"])] + ).to_pylist() == millimetres + + +def test_float32(library): + """Intrinsics and the point cloud go out as plain float32 values.""" + array, schema = structs() + values = (ctypes.c_float * 4)(700.5, 700.5, 640.0, 360.0) + + library.zed_export_float32(ctypes.byref(array), ctypes.byref(schema), values, 4) + result = imported(array, schema) + + assert result.type == pa.float32() + assert result.to_pylist() == [700.5, 700.5, 640.0, 360.0] + + +def test_string(library): + """The camera state goes out as one UTF-8 string.""" + array, schema = structs() + + library.zed_export_string(ctypes.byref(array), ctypes.byref(schema), b"streaming", 9) + result = imported(array, schema) + + assert result.type == pa.utf8() + assert result.to_pylist() == ["streaming"] + + +def main(): + """Run every test, reporting the first failure.""" + library = load() + + for test in (test_image, test_depth, test_float32, test_string): + test(library) + print(f"ok {test.__name__}") + + return 0 + + +if __name__ == "__main__": + sys.exit(main())