From c1a7d70ba617ce1877ea0f0e79833bbc89674ea9 Mon Sep 17 00:00:00 2001 From: fondef Date: Tue, 15 Sep 2026 10:52:08 -0500 Subject: [PATCH 1/2] add typed Arrow payloads and a dora-image-view dataflow dora-image-view takes std/media/v1/Image, and dora's compatibility graph has no edge from std/core/v1/UInt8 to it, so wiring the two failed `dora validate` whatever the node actually sent. Its flat-buffer fallback would not have decoded these frames either: it reads width/height metadata, and this node only published shape. image and depth now go out as the Image struct, camera_info and point_cloud as Float32 arrays and camera_state as a string, all built against the Arrow C data interface in src/arrow so the node still links no Arrow C++. Shape, dtype and encoding metadata are dropped where the payload describes itself; intrinsics, frame_id and both timestamps are unchanged. tests/run.sh covers the packing without a camera or the SDK, round-tripping each array through pyarrow and rebuilding it under the sanitizers. Breaking: image_bgr is renamed to image and every payload changes shape. dataflow_plot.yml is replaced by dataflow_view.yml, which wires the viewer from a tag and repeats its contract in input_types so validate checks both sides of the wiring. --- .gitea/workflows/validate.yml | 24 ++++- CMakeLists.txt | 3 +- dataflow.yml | 4 +- dataflow_plot.yml | 27 ----- dataflow_view.yml | 46 +++++++++ dora-node.yml | 43 ++++++-- src/arrow/array.cc | 179 ++++++++++++++++++++++++++++++++++ src/arrow/array.h | 75 ++++++++++++++ src/main.cc | 85 ++++++++-------- tests/arrow_export.cc | 36 +++++++ tests/arrow_test.cc | 54 ++++++++++ tests/run.sh | 29 ++++++ tests/test_arrow.py | 129 ++++++++++++++++++++++++ 13 files changed, 648 insertions(+), 86 deletions(-) delete mode 100644 dataflow_plot.yml create mode 100644 dataflow_view.yml create mode 100644 src/arrow/array.cc create mode 100644 src/arrow/array.h create mode 100644 tests/arrow_export.cc create mode 100644 tests/arrow_test.cc create mode 100755 tests/run.sh create mode 100644 tests/test_arrow.py diff --git a/.gitea/workflows/validate.yml b/.gitea/workflows/validate.yml index e7e8878..24ed5ef 100644 --- a/.gitea/workflows/validate.yml +++ b/.gitea/workflows/validate.yml @@ -1,5 +1,6 @@ -# Checks both dataflows against the typed contract in dora-node.yml. This runs -# no node, so it needs neither a camera nor the ZED SDK. +# Two gates, neither of which needs a camera, the ZED SDK or CUDA: the dataflows +# are checked against the typed contract in dora-node.yml without running the +# node, and the payload packing is compiled and tested on its own. name: validate on: [push, pull_request] @@ -28,9 +29,22 @@ jobs: # strict_types: true in each dataflow turns a type warning into a # non-zero exit, so a mismatched annotation fails here rather than at - # runtime. The plot file is checked too: it wires the same outputs, and a - # rename that breaks only the viewer should still fail the build. + # runtime. The viewer file is checked too: it wires the same outputs, and + # a rename that breaks only the consumer should still fail the build. - name: Validate the dataflows run: | dora validate dataflow.yml - dora validate dataflow_plot.yml + dora validate dataflow_view.yml + + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install uv + run: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.local/bin" >> "$GITHUB_PATH" + + - name: Test the payload packing + run: tests/run.sh diff --git a/CMakeLists.txt b/CMakeLists.txt index 0fbd2c8..e2dc86f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,5 +1,5 @@ cmake_minimum_required(VERSION 3.21) -project(dora_zed VERSION 0.2.0 LANGUAGES C CXX) +project(dora_zed VERSION 0.3.0 LANGUAGES C CXX) set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_FLAGS "-fPIC") @@ -24,6 +24,7 @@ link_directories(${ZED_LIBRARY_DIR}) add_executable(dora_zed src/main.cc + src/arrow/array.cc src/zed/camera.cc src/zed/capture.cc ${node_bridge} diff --git a/dataflow.yml b/dataflow.yml index a1001e2..fd64efa 100644 --- a/dataflow.yml +++ b/dataflow.yml @@ -3,7 +3,7 @@ # A consumer wires the node from a tag instead — see README.md. # # The camera alone, with no viewer and no Python dependency. For a run that -# also displays the stream, use dataflow_plot.yml. +# also displays the stream, use dataflow_view.yml. strict_types: true nodes: @@ -13,7 +13,7 @@ nodes: inputs: tick: dora/timer/millis/100 outputs: - - image_bgr + - image - depth - point_cloud - camera_info diff --git a/dataflow_plot.yml b/dataflow_plot.yml deleted file mode 100644 index 570cada..0000000 --- a/dataflow_plot.yml +++ /dev/null @@ -1,27 +0,0 @@ -# The camera plus an opencv-plot viewer on image_bgr. -# -# Separate from dataflow.yml because the viewer pulls dora-hub and installs it -# with pip, and needs a display: the camera on its own should cost neither. -# dora has no cross-dataflow wiring, so the camera node is declared here too -# rather than referenced — run this file or dataflow.yml, not both. -strict_types: true - -nodes: - - id: zed-camera - path: bin/dora_zed - build: scripts/build.sh - inputs: - tick: dora/timer/millis/100 - outputs: - - image_bgr - - depth - - point_cloud - - camera_info - - camera_state - - - id: plot - git: https://github.com/dora-rs/dora-hub.git - build: pip install ./node-hub/opencv-plot - path: opencv-plot - inputs: - image: zed-camera/image_bgr diff --git a/dataflow_view.yml b/dataflow_view.yml new file mode 100644 index 0000000..11d44df --- /dev/null +++ b/dataflow_view.yml @@ -0,0 +1,46 @@ +# The camera into dora-image-view, and the worked example of wiring this node to +# a consumer. +# +# dora has no cross-dataflow wiring, so the camera node is declared here too +# rather than referenced — run this file or dataflow.yml, not both. +# +# Run it with `--uv`, which gives the viewer its own managed environment: +# +# dora run dataflow_view.yml --uv +# +# Fetching the viewer needs anonymous read access to its repository. dora-cli +# 1.0.1 fetches through libgit2 with no credential callback, so ssh:// cannot +# authenticate at all; on Gitea, marking the repository public is not enough +# either, since a repository inside a private organization answers an anonymous +# clone with 401. +strict_types: true + +nodes: + - id: zed-camera + path: bin/dora_zed + build: scripts/build.sh + inputs: + tick: dora/timer/millis/100 + outputs: + - image + - depth + - point_cloud + - camera_info + - camera_state + + - id: view + git: https://gitea.liria.cl/liria/dora-image-view.git + tag: v0.1.0 + build: uv pip install . + path: dora-image-view + inputs: + image: zed-camera/image + depth: zed-camera/depth + camera_info: zed-camera/camera_info + # The viewer's own manifest says this, but a `git:` node's manifest is not + # read at validate time — only at build. Repeating it here is what makes the + # two contracts checked against each other now rather than at runtime. + input_types: + image: std/media/v1/Image + depth: std/media/v1/Image + camera_info: std/core/v1/Float32 diff --git a/dora-node.yml b/dora-node.yml index 336e30a..ccf2916 100644 --- a/dora-node.yml +++ b/dora-node.yml @@ -7,33 +7,60 @@ apiVersion: 1 name: dora-zed namespace: liria -description: ZED stereo camera source publishing a BGR image, a depth map, intrinsics and an optional point cloud +description: ZED stereo camera source publishing an image, a depth map, intrinsics and an optional point cloud categories: [sensor] keywords: [zed, stereolabs, camera, depth, stereo, jetson] runtime: cpp entrypoint: bin/dora_zed build: scripts/build.sh +dora: ">=1.0" inputs: tick: type: std/core/v1/UInt64 required: true - description: Each tick triggers one camera grab. Connect to dora/timer/millis/N. + description: >- + Publishes the newest frame the capture thread has grabbed, or nothing if + none arrived since the last tick. It does not drive the camera: capture is + free-running at ZED_FPS, so this sets the publish rate and anything + captured between two ticks is dropped. Connect to dora/timer/millis/N. outputs: - image_bgr: - type: std/core/v1/UInt8 - description: BGR image bytes, [H, W, 3]. Shape and encoding are in the metadata. + image: + type: std/media/v1/Image + description: >- + Captured frame as an Arrow struct {width, height, encoding, data}. The + encoding is always bgr8, three bytes per pixel. depth: - type: std/core/v1/Float32 - description: Per-pixel depth in millimetres, [H, W]. Missing depth is 0, never inf or nan. + type: std/media/v1/Image + description: >- + Depth map as an Arrow struct {width, height, encoding, data}, one channel + per pixel -- dora has no depth type, and a depth map is an image whose + pixels are distances, which is how ROS spells it too. The encoding is + 32FC1 and the `units` metadata is mm. Missing depth is 0, never inf or + nan. camera_info: type: std/core/v1/Float32 description: Intrinsics [fx, fy, cx, cy], distortion in the metadata. Published once per camera session. point_cloud: type: std/core/v1/Float32 - description: XYZRGBA point cloud. Published only when ZED_POINT_CLOUD is on. + description: XYZRGBA point cloud, shape in the metadata. Published only when ZED_POINT_CLOUD is on. camera_state: type: std/core/v1/String description: One of opening, streaming, recovering. Published on every change. + +example: | + - id: zed-camera + git: https://gitea.liria.cl/liria/dora-zed.git + tag: v0.3.0 + build: scripts/build.sh + path: bin/dora_zed + inputs: + tick: dora/timer/millis/100 + outputs: + - image + - depth + - point_cloud + - camera_info + - camera_state diff --git a/src/arrow/array.cc b/src/arrow/array.cc new file mode 100644 index 0000000..3336f90 --- /dev/null +++ b/src/arrow/array.cc @@ -0,0 +1,179 @@ +#include "arrow/array.h" + +#include +#include +#include +#include + +namespace arrow_c { +namespace { + +// Producer-owned storage for one schema node. The struct's `format`, `name` and +// `children` pointers point in here, which is why it outlives the call that +// built it and is freed by the release callback rather than by the builder. +struct SchemaData { + std::string format; + std::string name; + std::vector children; +}; + +// Producer-owned storage for one array node. `owned` holds the bytes this +// node's own buffers point into - offset arrays, and scalars that have no +// storage elsewhere. The payload itself is borrowed from the caller. +struct ArrayData { + std::vector owned; + std::vector buffers; + std::vector children; +}; + +// A node's release callback releases its children before freeing itself, so a +// consumer that only releases the root - which is all the interface requires - +// still frees the whole tree. +void release_schema(ArrowSchema* schema) { + auto* data = static_cast(schema->private_data); + + for (ArrowSchema* child : data->children) { + if (child->release != nullptr) { + child->release(child); + } + delete child; + } + + delete data; + schema->release = nullptr; +} + +void release_array(ArrowArray* array) { + auto* data = static_cast(array->private_data); + + for (ArrowArray* child : data->children) { + if (child->release != nullptr) { + child->release(child); + } + delete child; + } + + delete data; + array->release = nullptr; +} + +void set_schema(ArrowSchema& schema, const char* format, const char* name, + std::vector children) { + auto* data = new SchemaData{format, name, std::move(children)}; + + schema.format = data->format.c_str(); + schema.name = data->name.c_str(); + schema.metadata = nullptr; + // Nullable everywhere, which is what pyarrow's defaults produce: the same + // struct built by a Python node has to compare equal to this one. + schema.flags = ARROW_FLAG_NULLABLE; + schema.n_children = static_cast(data->children.size()); + schema.children = data->children.empty() ? nullptr : data->children.data(); + schema.dictionary = nullptr; + schema.release = release_schema; + schema.private_data = data; +} + +// `buffers` may point into `owned`: moving a vector keeps its heap block, so +// pointers taken before the move stay valid after it. +void set_array(ArrowArray& array, int64_t length, std::vector owned, + std::vector buffers, std::vector children) { + auto* data = new ArrayData{std::move(owned), std::move(buffers), std::move(children)}; + + array.length = length; + // Nothing this node sends is null, so the validity bitmap - the first + // buffer of every node below - is left out entirely. + array.null_count = 0; + array.offset = 0; + array.n_buffers = static_cast(data->buffers.size()); + array.n_children = static_cast(data->children.size()); + array.buffers = data->buffers.data(); + array.children = data->children.empty() ? nullptr : data->children.data(); + array.dictionary = nullptr; + array.release = release_array; + array.private_data = data; +} + +// One field of a struct, owned by its parent. +struct Child { + ArrowSchema* schema; + ArrowArray* array; +}; + +Child new_child() { + return {new ArrowSchema{}, new ArrowArray{}}; +} + +void fill_uint32(ArrowSchema& schema, ArrowArray& array, const char* name, uint32_t value) { + std::vector owned(sizeof(value)); + std::memcpy(owned.data(), &value, sizeof(value)); + const void* values = owned.data(); + + set_schema(schema, "I", name, {}); + set_array(array, 1, std::move(owned), {nullptr, values}, {}); +} + +// One string, offsets and characters in a single allocation. +void fill_utf8(ArrowSchema& schema, ArrowArray& array, const char* name, const char* text, + size_t size) { + const int32_t offsets[2] = {0, static_cast(size)}; + + std::vector owned(sizeof(offsets) + size); + std::memcpy(owned.data(), offsets, sizeof(offsets)); + std::memcpy(owned.data() + sizeof(offsets), text, size); + const void* offsets_buffer = owned.data(); + const void* values_buffer = owned.data() + sizeof(offsets); + + set_schema(schema, "u", name, {}); + set_array(array, 1, std::move(owned), {nullptr, offsets_buffer, values_buffer}, {}); +} + +// One byte string. Only the offsets are owned; the bytes are the caller's. +void fill_large_binary(ArrowSchema& schema, ArrowArray& array, const char* name, + const uint8_t* data, size_t size) { + const int64_t offsets[2] = {0, static_cast(size)}; + + std::vector owned(sizeof(offsets)); + std::memcpy(owned.data(), offsets, sizeof(offsets)); + const void* offsets_buffer = owned.data(); + + set_schema(schema, "Z", name, {}); + set_array(array, 1, std::move(owned), {nullptr, offsets_buffer, data}, {}); +} + +} // namespace + +Exported image(uint32_t width, uint32_t height, const char* encoding, const uint8_t* data, + size_t size) { + Child w = new_child(); + fill_uint32(*w.schema, *w.array, "width", width); + + Child h = new_child(); + fill_uint32(*h.schema, *h.array, "height", height); + + Child e = new_child(); + fill_utf8(*e.schema, *e.array, "encoding", encoding, std::strlen(encoding)); + + Child d = new_child(); + fill_large_binary(*d.schema, *d.array, "data", data, size); + + Exported out; + set_schema(out.schema, "+s", "", {w.schema, h.schema, e.schema, d.schema}); + set_array(out.array, 1, {}, {nullptr}, {w.array, h.array, e.array, d.array}); + return out; +} + +Exported float32(const float* values, size_t count) { + Exported out; + set_schema(out.schema, "f", "", {}); + set_array(out.array, static_cast(count), {}, {nullptr, values}, {}); + return out; +} + +Exported string(const char* text, size_t size) { + Exported out; + fill_utf8(out.schema, out.array, "", text, size); + return out; +} + +} // namespace arrow_c diff --git a/src/arrow/array.h b/src/arrow/array.h new file mode 100644 index 0000000..e3502f0 --- /dev/null +++ b/src/arrow/array.h @@ -0,0 +1,75 @@ +#pragma once + +#include +#include + +// The Arrow C data interface, verbatim from the Arrow specification. +// +// These two structs are the whole ABI dora's send_arrow_output takes, so +// defining them here is what lets this node publish typed Arrow arrays without +// linking Arrow C++ - a heavy dependency for a producer that exports one array +// per output and never reads one. +#ifndef ARROW_C_DATA_INTERFACE +#define ARROW_C_DATA_INTERFACE + +#define ARROW_FLAG_DICTIONARY_ORDERED 1 +#define ARROW_FLAG_NULLABLE 2 +#define ARROW_FLAG_MAP_KEYS_SORTED 4 + +struct ArrowSchema { + const char* format; + const char* name; + const char* metadata; + int64_t flags; + int64_t n_children; + struct ArrowSchema** children; + struct ArrowSchema* dictionary; + + void (*release)(struct ArrowSchema*); + void* private_data; +}; + +struct ArrowArray { + int64_t length; + int64_t null_count; + int64_t offset; + int64_t n_buffers; + int64_t n_children; + const void** buffers; + struct ArrowArray** children; + struct ArrowArray* dictionary; + + void (*release)(struct ArrowArray*); + void* private_data; +}; + +#endif // ARROW_C_DATA_INTERFACE + +namespace arrow_c { + +// One array with the schema describing it. +// +// dora moves both structs out of here and calls their release callbacks once it +// has copied the payload into shared memory, so an Exported is built, sent +// once, and never touched again. +struct Exported { + ArrowArray array{}; + ArrowSchema schema{}; +}; + +// std/media/v1/Image: one row of {width, height, encoding, data}. +// +// The payload is borrowed rather than copied, so it has to stay valid until the +// send returns. A capture-thread frame buffer does: the event loop holds the +// frame across the call. +Exported image(uint32_t width, uint32_t height, const char* encoding, const uint8_t* data, + size_t size); + +// std/core/v1/Float32: `count` values, borrowed on the same terms as image(). +Exported float32(const float* values, size_t count); + +// std/core/v1/String: one UTF-8 string. Copied, since every string this node +// sends is a short state name. +Exported string(const char* text, size_t size); + +} // namespace arrow_c diff --git a/src/main.cc b/src/main.cc index e9cd256..1ae38b0 100644 --- a/src/main.cc +++ b/src/main.cc @@ -1,12 +1,15 @@ +#include "arrow/array.h" #include "zed/capture.h" #include #include +#include #include #include #include #include +#include #include namespace { @@ -47,9 +50,18 @@ void set_frame_stamps(Metadata& metadata, const zed::Frame& frame) { metadata.set_timestamp("host_timestamp_ns", frame.host_ns); } -void report(const char* output, const DoraResult& result) { +// Publish one Arrow array. dora moves both structs out and releases them once +// the payload is in shared memory, so the buffers they borrow only have to +// outlive this call - which the frame the caller is holding does. +void send(DoraNode& dora_node, const char* id, arrow_c::Exported payload, + rust::Box metadata) { + auto result = send_arrow_output(dora_node.send_output, id, + reinterpret_cast(&payload.array), + reinterpret_cast(&payload.schema), + std::move(metadata)); + if (!std::string(result.error).empty()) { - std::cerr << "Error sending " << output << ": " << std::string(result.error) << std::endl; + std::cerr << "Error sending " << id << ": " << std::string(result.error) << std::endl; } } @@ -90,17 +102,11 @@ int main() { std::string state; if (capture.take_state_change(state)) { - rust::Slice state_slice{ - reinterpret_cast(state.data()), state.size()}; - auto state_metadata = new_metadata(); - state_metadata->set_string("dtype", "string"); - state_metadata->set_string("encoding", "utf8"); state_metadata->set_timestamp("host_timestamp_ns", host_timestamp_ns()); - report("camera_state", - send_output_with_metadata(dora_node.send_output, "camera_state", state_slice, - std::move(state_metadata))); + send(dora_node, "camera_state", arrow_c::string(state.data(), state.size()), + std::move(state_metadata)); } // No new frame since the last tick: the camera is slower than the timer, @@ -109,44 +115,41 @@ int main() { continue; } - // --- image_bgr --- - rust::Slice image_slice{frame.image_bgr.data(), frame.image_bgr.size()}; + auto width = static_cast(frame.width); + auto height = static_cast(frame.height); + // --- image --- + // The payload carries its own width, height and encoding, so none of + // that travels as metadata: what is left is what describes the frame + // rather than the picture. auto image_metadata = new_metadata(); - image_metadata->set_string("dtype", "uint8"); - image_metadata->set_string("encoding", "bgr8"); - image_metadata->set_string("layout", "HWC"); - image_metadata->set_list_int("shape", make_vector_int({frame.height, frame.width, 3})); set_intrinsics(*image_metadata, frame.calib); set_frame_stamps(*image_metadata, frame); - report("image_bgr", send_output_with_metadata(dora_node.send_output, "image_bgr", - image_slice, std::move(image_metadata))); + send(dora_node, "image", + arrow_c::image(width, height, "bgr8", frame.image_bgr.data(), frame.image_bgr.size()), + std::move(image_metadata)); // --- depth --- - rust::Slice depth_slice{ - reinterpret_cast(frame.depth.data()), - frame.depth.size() * sizeof(float)}; - + // A depth map is an image whose pixels are distances, which is how ROS + // spells it too: 32FC1 is one float32 per pixel, and `units` says what + // those floats measure. auto depth_metadata = new_metadata(); - depth_metadata->set_string("dtype", "float32"); - depth_metadata->set_string("layout", "HW"); depth_metadata->set_string("units", "mm"); - depth_metadata->set_list_int("shape", make_vector_int({frame.height, frame.width})); set_intrinsics(*depth_metadata, frame.calib); set_frame_stamps(*depth_metadata, frame); - report("depth", send_output_with_metadata(dora_node.send_output, "depth", depth_slice, - std::move(depth_metadata))); + send(dora_node, "depth", + arrow_c::image(width, height, "32FC1", + reinterpret_cast(frame.depth.data()), + frame.depth.size() * sizeof(float)), + std::move(depth_metadata)); // --- point_cloud --- if (frame.has_point_cloud) { - rust::Slice pc_slice{ - reinterpret_cast(frame.point_cloud.data()), - frame.point_cloud.size() * sizeof(float)}; - + // Unlike the two images, a cloud is a flat array of values: its + // grid shape is only in the metadata. auto pc_metadata = new_metadata(); - pc_metadata->set_string("dtype", "float32"); pc_metadata->set_string("channels", "XYZRGBA"); pc_metadata->set_string("units", "mm"); pc_metadata->set_string("layout", "HWC"); @@ -155,26 +158,22 @@ int main() { set_intrinsics(*pc_metadata, frame.calib); set_frame_stamps(*pc_metadata, frame); - report("point_cloud", send_output_with_metadata(dora_node.send_output, "point_cloud", - pc_slice, std::move(pc_metadata))); + send(dora_node, "point_cloud", + arrow_c::float32(frame.point_cloud.data(), frame.point_cloud.size()), + std::move(pc_metadata)); } // --- camera_info --- if (frame.camera_info_first) { - float camera_data[4] = {frame.calib.fx, frame.calib.fy, frame.calib.cx, - frame.calib.cy}; - rust::Slice cam_slice{ - reinterpret_cast(camera_data), sizeof(camera_data)}; + const float camera_data[4] = {frame.calib.fx, frame.calib.fy, frame.calib.cx, + frame.calib.cy}; auto cam_metadata = new_metadata(); - cam_metadata->set_string("dtype", "float32"); - cam_metadata->set_string("layout", "C"); - cam_metadata->set_list_int("shape", make_vector_int({4})); cam_metadata->set_list_float("distortion", make_distortion(frame.calib.distortion)); set_frame_stamps(*cam_metadata, frame); - report("camera_info", send_output_with_metadata(dora_node.send_output, "camera_info", - cam_slice, std::move(cam_metadata))); + send(dora_node, "camera_info", arrow_c::float32(camera_data, 4), + std::move(cam_metadata)); } } diff --git a/tests/arrow_export.cc b/tests/arrow_export.cc new file mode 100644 index 0000000..9c594f8 --- /dev/null +++ b/tests/arrow_export.cc @@ -0,0 +1,36 @@ +// C entry points for tests/test_arrow.py. +// +// The Python side cannot call the C++ builders, so each one is wrapped here and +// the payload is passed in from the test rather than fixed in this file. + +#include "arrow/array.h" + +#include + +namespace { + +void hand_over(void* array, void* schema, const arrow_c::Exported& exported) { + // The caller owns the two structs, exactly as dora does: it reads them and + // is then responsible for releasing what they point at. + std::memcpy(array, &exported.array, sizeof(exported.array)); + std::memcpy(schema, &exported.schema, sizeof(exported.schema)); +} + +} // namespace + +extern "C" { + +void zed_export_image(void* array, void* schema, uint32_t width, uint32_t height, + const char* encoding, const uint8_t* data, size_t size) { + hand_over(array, schema, arrow_c::image(width, height, encoding, data, size)); +} + +void zed_export_float32(void* array, void* schema, const float* values, size_t count) { + hand_over(array, schema, arrow_c::float32(values, count)); +} + +void zed_export_string(void* array, void* schema, const char* text, size_t size) { + hand_over(array, schema, arrow_c::string(text, size)); +} + +} // extern "C" diff --git a/tests/arrow_test.cc b/tests/arrow_test.cc new file mode 100644 index 0000000..e5a2b92 --- /dev/null +++ b/tests/arrow_test.cc @@ -0,0 +1,54 @@ +// Release has to free the whole tree: the array nodes, the schema nodes, and +// the storage their buffers point into. Nothing here asserts that directly - +// the sanitizers tests/run.sh builds this with are the assertion, and a node +// that leaks per frame is a node that dies on a long run. + +#include "arrow/array.h" + +#include +#include +#include +#include + +namespace { + +void release(arrow_c::Exported& exported) { + exported.array.release(&exported.array); + exported.schema.release(&exported.schema); + + // A released struct marks itself, which is how a consumer knows not to + // release it twice. + assert(exported.array.release == nullptr); + assert(exported.schema.release == nullptr); +} + +} // namespace + +int main() { + const std::vector pixels(4 * 4 * 3, 7); + const std::vector depth(4 * 4, 1500.0F); + const std::string state = "streaming"; + + for (int i = 0; i < 100; ++i) { + arrow_c::Exported image = + arrow_c::image(4, 4, "bgr8", pixels.data(), pixels.size()); + assert(image.array.length == 1); + assert(image.array.n_children == 4); + release(image); + + arrow_c::Exported depth_image = + arrow_c::image(4, 4, "32FC1", reinterpret_cast(depth.data()), + depth.size() * sizeof(float)); + release(depth_image); + + arrow_c::Exported values = arrow_c::float32(depth.data(), depth.size()); + assert(values.array.length == static_cast(depth.size())); + release(values); + + arrow_c::Exported text = arrow_c::string(state.data(), state.size()); + assert(text.array.length == 1); + release(text); + } + + return 0; +} diff --git a/tests/run.sh b/tests/run.sh new file mode 100755 index 0000000..aa46a79 --- /dev/null +++ b/tests/run.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Build and run the payload tests. +# +# They cover src/arrow only, which is the part of the node that has no ZED SDK, +# no CUDA and no dora dependency - so this runs anywhere, and is what CI runs. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(dirname "$SCRIPT_DIR")" +OUT_DIR="${REPO_DIR}/build/tests" + +mkdir -p "$OUT_DIR" + +# Sanitized: the release callbacks have to free the whole exported tree, and a +# leak there would be one per frame. +g++ -std=c++17 -Wall -Wextra -Werror -g -fsanitize=address,undefined \ + -I "${REPO_DIR}/src" \ + -o "${OUT_DIR}/arrow_test" \ + "${REPO_DIR}/src/arrow/array.cc" "${REPO_DIR}/tests/arrow_test.cc" +"${OUT_DIR}/arrow_test" +echo "ok arrow_test" + +# Unsanitized: pyarrow is the consumer standing in for dora, and loads the same +# code as a shared library. +g++ -std=c++17 -Wall -Wextra -Werror -O2 -shared -fPIC \ + -I "${REPO_DIR}/src" \ + -o "${OUT_DIR}/libzed_arrow.so" \ + "${REPO_DIR}/src/arrow/array.cc" "${REPO_DIR}/tests/arrow_export.cc" +uv run --no-project --with pyarrow python "${REPO_DIR}/tests/test_arrow.py" diff --git a/tests/test_arrow.py b/tests/test_arrow.py new file mode 100644 index 0000000..07dd786 --- /dev/null +++ b/tests/test_arrow.py @@ -0,0 +1,129 @@ +"""Round-trip every exported payload through a real Arrow consumer. + +The node builds its Arrow arrays by hand against the C data interface rather +than linking Arrow C++, so this is what shows that what dora imports on the +other side of `send_arrow_output` is what went in. It needs neither a camera nor +the ZED SDK. +""" + +import ctypes +import pathlib +import sys + +import pyarrow as pa + +LIBRARY = pathlib.Path(__file__).resolve().parent.parent / "build" / "tests" / "libzed_arrow.so" + +# ArrowArray is ten machine words and ArrowSchema nine, and both want the +# 8-byte alignment an array of uint64 gives. +WORDS = 16 +# Index of the `release` field in ArrowArray: a consumer that took ownership +# zeroes it, which is how this test sees the move happen. +RELEASE = 8 + +IMAGE_TYPE = pa.struct([ + ("width", pa.uint32()), + ("height", pa.uint32()), + ("encoding", pa.utf8()), + ("data", pa.large_binary()), +]) + + +def load(): + """Load the test library and declare the argument types of its exports.""" + library = ctypes.CDLL(str(LIBRARY)) + + library.zed_export_image.argtypes = [ + ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint32, ctypes.c_uint32, + ctypes.c_char_p, ctypes.c_char_p, ctypes.c_size_t, + ] + library.zed_export_float32.argtypes = [ + ctypes.c_void_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_float), ctypes.c_size_t, + ] + library.zed_export_string.argtypes = [ + ctypes.c_void_p, ctypes.c_void_p, ctypes.c_char_p, ctypes.c_size_t, + ] + return library + + +def structs(): + """Allocate the ArrowArray and ArrowSchema the producer fills in.""" + return (ctypes.c_uint64 * WORDS)(), (ctypes.c_uint64 * WORDS)() + + +def imported(array, schema): + """Import the pair as pyarrow does, taking ownership of both.""" + result = pa.Array._import_from_c(ctypes.addressof(array), ctypes.addressof(schema)) + assert array[RELEASE] == 0, "the consumer should have taken ownership of the array" + return result + + +def test_image(library): + """An image row carries its own dimensions, encoding and bytes.""" + array, schema = structs() + pixels = bytes(range(12)) + + library.zed_export_image( + ctypes.byref(array), ctypes.byref(schema), 2, 2, b"bgr8", pixels, len(pixels) + ) + result = imported(array, schema) + + assert result.type == IMAGE_TYPE, result.type + assert result.to_pylist() == [ + {"width": 2, "height": 2, "encoding": "bgr8", "data": pixels} + ] + + +def test_depth(library): + """A depth map is an image whose bytes are float32 millimetres.""" + array, schema = structs() + millimetres = [1500.0, 0.0, 250.5, 4000.0] + payload = pa.array(millimetres, type=pa.float32()).buffers()[1].to_pybytes() + + library.zed_export_image( + ctypes.byref(array), ctypes.byref(schema), 2, 2, b"32FC1", payload, len(payload) + ) + row = imported(array, schema).to_pylist()[0] + + assert row["encoding"] == "32FC1" + assert pa.Array.from_buffers( + pa.float32(), 4, [None, pa.py_buffer(row["data"])] + ).to_pylist() == millimetres + + +def test_float32(library): + """Intrinsics and the point cloud go out as plain float32 values.""" + array, schema = structs() + values = (ctypes.c_float * 4)(700.5, 700.5, 640.0, 360.0) + + library.zed_export_float32(ctypes.byref(array), ctypes.byref(schema), values, 4) + result = imported(array, schema) + + assert result.type == pa.float32() + assert result.to_pylist() == [700.5, 700.5, 640.0, 360.0] + + +def test_string(library): + """The camera state goes out as one UTF-8 string.""" + array, schema = structs() + + library.zed_export_string(ctypes.byref(array), ctypes.byref(schema), b"streaming", 9) + result = imported(array, schema) + + assert result.type == pa.utf8() + assert result.to_pylist() == ["streaming"] + + +def main(): + """Run every test, reporting the first failure.""" + library = load() + + for test in (test_image, test_depth, test_float32, test_string): + test(library) + print(f"ok {test.__name__}") + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) -- 2.54.0 From a4cd493330f7174ea486cc70769e172657eb4031 Mon Sep 17 00:00:00 2001 From: fondef Date: Tue, 15 Sep 2026 10:52:12 -0500 Subject: [PATCH 2/2] docs: shorten the README and correct how the tick is described The README carried explanations that belong in AGENTS.md; it now follows the shape of the sibling node repositories and is a quarter shorter. The tick description was also wrong: it does not trigger a grab. Capture is free-running at ZED_FPS and a tick forwards the newest frame there is, so the tick sets the publish rate, and frames captured between two ticks are dropped. The usage snippet listed four of the five outputs, which read as an omission rather than the choice it is. --- AGENTS.md | 28 +++-- README.md | 339 +++++++++++++++++++++++------------------------------- 2 files changed, 166 insertions(+), 201 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8f7ca9f..dfc0fe8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,10 +4,18 @@ This file provides guidance to AI coding assistants when working with code in th ## What this is -A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame and publishes it over the dora dataflow: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Camera management lives in `src/zed/` and the dora event loop in `src/main.cc`. +A dora-rs C++ node that interfaces with ZED stereo cameras. Capture is free-running on its own thread at `ZED_FPS`; on each tick the event loop publishes the newest frame it has produced, or nothing if none arrived since the last tick. What goes out: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Every payload is a typed Arrow array — the two frames as `std/media/v1/Image`, the rest as `std/core/v1/Float32` and `.../String` — which is what lets the node feed [dora-image-view](https://gitea.liria.cl/liria/dora-image-view) and anything else built against those types. Camera management lives in `src/zed/`, payload packing in `src/arrow/`, and the dora event loop in `src/main.cc`. The node is a camera source only. It ran inference in-process through `v0.1.0`, under `src/engines/`; that was removed so the node has no inference dependency and builds on any platform the ZED SDK supports. The removed code remains reachable at tag `v0.1.0`. +## Test + +```bash +tests/run.sh +``` + +Needs `g++` and `uv`; no camera, no ZED SDK, no CUDA. + ## Build ```bash @@ -28,8 +36,8 @@ For a build without installing the SDK stack on the host, see the `Dockerfile` ( ```bash dora up -dora start dataflow.yml # camera alone -dora start dataflow_plot.yml # camera plus the opencv-plot viewer +dora start dataflow.yml # camera alone +dora start dataflow_view.yml --uv # camera plus dora-image-view ``` The two dataflows both declare the camera node, so run one at a time. @@ -42,20 +50,22 @@ Camera behavior is configured entirely through environment variables (see README - **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: `open()` (one attempt, returns success), warmup, grab (image + depth + point cloud), calibration, device timestamp, blank-frame check, `close()` and `reset()` (close plus `sl::Camera::reboot()` where the model allows it). Exposes `handle()` for engine access. - **`src/zed/capture.h/cc`** — `zed::Capture` and `zed::Frame`. Owns the camera on a dedicated thread: open, warmup, grab, blank/stall detection, backoff and escalation all live here, along with the packing of every payload into owned buffers. The newest frame is handed to the reader by swapping under a mutex (`take_frame`), which also recycles the previous frame's buffers so steady-state capture does not reallocate. **Nothing outside this file may call the ZED SDK** — it blocks unboundedly on a disconnected camera, which is what froze the node when the loop drove the camera directly. - **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace. `parse_depth_mode` runs in `Config::from_env()`, not in the constructor, so an unrecognised `ZED_DEPTH_MODE` exits before the node joins the dataflow rather than falling back to a default. -- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()`, then sends `image_bgr` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `pack_depth` copies the depth map into a tightly packed `[H, W]` float buffer (sl::Mat rows can be padded) and clears `inf`/`nan` to `0`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds). Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension. The loop never touches the SDK: it drains `take_state_change` and `take_frame` from the capture thread and publishes. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node. +- **`src/arrow/array.h/cc`** — Builds the Arrow arrays the node publishes, against the Arrow C data interface: the two ABI structs are defined in the header rather than pulled from Arrow C++, which would be a heavy dependency for a producer that exports one array per output and never reads one. `image()` packs an `std/media/v1/Image` row, `float32()` and `string()` the primitive payloads. The payload bytes are **borrowed**, not copied — they only have to outlive the send, since dora IPC-encodes the array into shared memory inside `send_arrow_output` and then calls the release callbacks. Each node's release frees its children before itself, so releasing the root frees the tree. +- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected) it drains `take_state_change` and `take_frame` from the capture thread and publishes `image` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `send()` hands one `arrow_c::Exported` to `send_arrow_output`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds); nothing about a frame's shape or encoding travels as metadata any more, since the Image struct carries it. The loop never touches the SDK. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node. - **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue. - **`scripts/build.sh`** — Configure/compile/install in one shell-free-callable step. Referenced by both dataflows, the `Dockerfile` and the README. -- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why both dataflows build from the checkout. Type URNs name the element type (`std/core/v1/UInt8`, `.../Float32`, `.../String`) because every payload is a raw buffer described by its metadata, not an Arrow struct. +- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why both dataflows build from the checkout. `image` and `depth` are `std/media/v1/Image`; dora's compatibility graph has no edge from `std/core/v1/UInt8` to it, so a consumer declaring the struct rejects a producer declaring the buffer whatever it actually sends — the manifest has to move with the payload. - **`dataflow.yml`** — Example dora dataflow: the camera alone, no viewer and no Python dependency. Built from this checkout with `path:` + `build:` and `strict_types: true`. A consumer wires the node from a tag instead (see README) — dora 1.0 pins a source with the mutually exclusive `tag:`, `rev:` or `branch:`. -- **`dataflow_plot.yml`** — The same camera node plus an `opencv-plot` viewer on `image_bgr`. Split out because the viewer pulls dora-hub over `pip` and needs a display. dora 1.0 has no cross-dataflow wiring (an input's `source` must resolve to a node in the same descriptor), so the camera node is duplicated here rather than referenced, and the two files are run one at a time. -- **`.gitea/workflows/validate.yml`** — CI gate running `dora validate` on both dataflows on push and pull request. Needs no camera and no ZED SDK. +- **`dataflow_view.yml`** — The same camera node plus [dora-image-view](https://gitea.liria.cl/liria/dora-image-view) on `image`, `depth` and `camera_info`, fetched from a tag. Run it with `--uv`, which gives the viewer its own managed environment. The viewer's manifest is not read at validate time (only a `path:` node's is), so the file repeats its contract in `input_types:` to get the two checked against each other. dora 1.0 has no cross-dataflow wiring (an input's `source` must resolve to a node in the same descriptor), so the camera node is duplicated here rather than referenced, and the two files are run one at a time. +- **`tests/run.sh`** — Builds and runs the payload tests: `tests/arrow_test.cc` under `-fsanitize=address,undefined` (the release callbacks have to free the whole tree), and `tests/test_arrow.py`, which imports the exported arrays through pyarrow's C data interface and checks the values. `tests/arrow_export.cc` is the C shim the Python side calls. Covers `src/arrow` only, so it needs no camera, no ZED SDK and no dora. +- **`.gitea/workflows/validate.yml`** — CI gates on push and pull request: `dora validate` on both dataflows, and `tests/run.sh`. Neither needs a camera, the ZED SDK or CUDA. - **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`. ## Key APIs -- **dora C++ API** (``): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`. +- **dora C++ API** (``): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_arrow_output()`, `new_metadata()`. - **ZED SDK** (``): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`. -- Output data is sent as raw byte slices (`rust::Slice`) with typed metadata (shape, dtype, encoding). +- Output data is sent as Arrow arrays: `send_arrow_output` takes an `ArrowArray`/`ArrowSchema` pair by pointer and moves both out, so a payload is built once with `src/arrow` and handed over. ## Tools diff --git a/README.md b/README.md index 4380ebe..b577a48 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # dora-zed -Dora node for ZED stereo cameras. Publishes a BGR image and a depth map on every frame, plus camera intrinsics, a camera state, and an optional point cloud. +Dora node for ZED stereo cameras. On every tick it publishes a colour image and a depth +map as `std/media/v1/Image`, plus camera intrinsics, a camera state, and an optional +point cloud. A source for a vision pipeline: it takes a tick and produces frames. ## Usage @@ -9,100 +11,169 @@ Add to your dataflow: ```yaml nodes: - id: zed-camera - git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git - tag: v0.2.0 + git: https://gitea.liria.cl/liria/dora-zed.git + tag: v0.3.0 build: scripts/build.sh path: bin/dora_zed inputs: tick: dora/timer/millis/100 outputs: - - image_bgr + - image - depth - point_cloud - camera_info - camera_state ``` -### Runnable examples +The node is C++: building it needs the ZED SDK and CUDA on the machine that runs the +dataflow — see [Build](#build). An output left out of `outputs:` is simply not published, +so a dataflow that never turns the point cloud on can drop that line. -Two dataflows live here, and they are run one at a time — dora has no cross-dataflow -wiring, so each declares the camera node itself: +### Runnable dataflows | File | Contents | Run | |---|---|---| -| [`dataflow.yml`](dataflow.yml) | The camera alone | `dora run dataflow.yml` | -| [`dataflow_plot.yml`](dataflow_plot.yml) | The camera plus an `opencv-plot` viewer on `image_bgr` | `dora run dataflow_plot.yml` | +| [`dataflow.yml`](dataflow.yml) | The camera alone, no viewer and no Python dependency | `dora run dataflow.yml` | +| [`dataflow_view.yml`](dataflow_view.yml) | The camera plus [dora-image-view][view] | `dora run dataflow_view.yml --uv` | -The viewer is a separate file because it pulls [dora-hub](https://github.com/dora-rs/dora-hub) -and installs it with `pip`, and it needs a display. Neither is a cost the camera should -carry on a headless robot, so `dataflow.yml` has no Python dependency at all. +[view]: https://gitea.liria.cl/liria/dora-image-view -Both build the node from the checkout (`path:` + `build:`) rather than from a tag, which -is what lets `dora validate` pick up the manifest described below. +`dataflow_view.yml` is the worked example of consuming this node: a window opens with +the colour image and the depth map side by side, the camera model captioned across them. +Press `q` to quit. It needs a display, and `--uv` so the viewer gets its own managed +environment. -### Node manifest +dora has no cross-dataflow wiring, so both files declare the camera node themselves: run +one at a time. Both build it from the checkout, which is what lets `dora validate` check +them against [`dora-node.yml`](dora-node.yml) — the typed contract a consumer wires +against, and what [CI](.gitea/workflows/validate.yml) validates on every push. -[`dora-node.yml`](dora-node.yml) declares the node's inputs and outputs with type URNs. -No source package is shared between node repositories, so it is the only contract a -consumer has, and it is what makes a wiring mistake fail at build time instead of at -runtime: +## Inputs -``` -$ dora validate dataflow.yml - node "zed-camera": injected 6 contract type(s) from .../dora-node.yml -All type annotations OK. -``` - -Types are `std/core/v1/UInt8` for `image_bgr`, `std/core/v1/Float32` for `depth`, -`camera_info` and `point_cloud`, and `std/core/v1/String` for `camera_state`; every -payload is a raw buffer whose shape and dtype are in the metadata, so the URN names the -element type rather than a struct. The `tick` input is `std/core/v1/UInt64`, matching how -dora types timer inputs. - -`strict_types: true` in both dataflows makes a type warning a non-zero exit. -[`.gitea/workflows/validate.yml`](.gitea/workflows/validate.yml) validates both on every -push and pull request; it needs no camera and no ZED SDK, since validation never runs the -node. - -## Dependencies - -| Dependency | Version | Notes | -|---|---|---| -| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time | -| [CMake](https://cmake.org/download/) | 3.21+ | | -| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` | -| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK | -| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings | - -### Installation scripts - -| Platform | Script | +| Input | Description | |---|---| -| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` | -| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` | -| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` | +| `tick` | Publishes the newest frame captured since the last tick, or nothing. Wire to `dora/timer/millis/N`. | -> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release). +The tick does not drive the camera. Capture is free-running on its own thread, paced by +the SDK to `ZED_FPS`, and each frame overwrites whatever the previous one left +uncollected; a tick forwards the newest there is, and publishes nothing when none arrived +since the last one. So the tick sets the publish rate, never the capture rate: below +`ZED_FPS` it drops frames — the default 100 ms tick against the default 15 fps keeps two +out of every three — and above it, the ticks that find nothing publish nothing. +`dora/timer/millis/66` at 15 fps forwards every frame, each at most one capture period +old and none of them twice. -## Local build +## Outputs + +| Output | Payload | Metadata | +|---|---|---| +| `image` | `std/media/v1/Image` struct, `bgr8` | intrinsics, `frame_id`, timestamps | +| `depth` | `std/media/v1/Image` struct, `32FC1` | `units=mm`, intrinsics, `frame_id`, timestamps | +| `camera_info` | `float32` `[fx, fy, cx, cy]` | `distortion`, `frame_id`, timestamps | +| `point_cloud` | `float32` XYZRGBA, off by default | `shape`, `channels`, `units=mm`, intrinsics, timestamps | +| `camera_state` | `string` | `host_timestamp_ns` | + +`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`, +`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself, +so nothing about its shape travels as message metadata. A depth map is an image whose +pixels are distances, as ROS also treats it: `32FC1` is one float32 per pixel, `units` +says what those floats measure, and `0` means missing depth — the payload contains no +`inf` and no `nan`. `depth` is published on every frame `image` is, at the same size and +`frame_id`. + +`camera_info` is published **once per camera session** — with the first frame after the +camera opens, and again after a reconnection — since the intrinsics are fixed for a given +camera and resolution. Every frame also carries `fx`, `fy`, `cx` and `cy` in its +metadata, so a consumer that started after the camera did still ends up with them. + +Every output of a frame carries the same `frame_id` and both clocks as dora `timestamp` +metadata: `device_timestamp_ns`, when the camera captured the frame, and +`host_timestamp_ns`, when this node published it — both Unix-epoch nanoseconds. Two nodes +reading the same frame agree on the device clock; the rest of the dataflow shares the +host one. + +### Camera state + +`camera_state` lets a consumer tell a camera that has not delivered its first frame yet +from one that has been lost: + +| State | Meaning | +|---|---| +| `opening` | The camera has not delivered a usable frame yet in this run | +| `streaming` | Frames are being published | +| `recovering` | The camera was streaming and has stopped; the node is bringing it back | + +The camera runs on its own thread, since the SDK can block indefinitely inside `grab()` +and `open()` on a disconnected camera and would otherwise freeze the event loop. A camera +that delivers no usable frame for 5 seconds is recovered, reopening on a 1, 2, 5 then 10 +second backoff and escalating to `sl::Camera::reboot()` past the first attempt; an +all-black frame counts as a failed grab, since a camera whose link has dropped can keep +reporting success while handing back a uniform one. Uncollected frames are dropped, so a +consumer always gets the newest one. + +## Configuration + +Settings are environment variables, which a dataflow can carry per node, keeping them +with the wiring rather than in whatever shell starts it: + +```yaml + env: + ZED_RESOLUTION: 720 + ZED_FPS: 30 +``` + +A node's `env:` wins over the environment the dataflow was started in; variables it does +not set still pass through. + +| Variable | Default | Meaning | +|---|---|---| +| `ZED_RESOLUTION` | `1080` | `720`, `1080` or `2000` | +| `ZED_FPS` | `15` | Frame rate | +| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY` or `NONE`. An unrecognised value is a startup error, not a fallback. | +| `ZED_DEPTH_MIN_MM` | `200` | Near end of the depth range, in millimetres | +| `ZED_DEPTH_MAX_MM` | `600` | Far end of the depth range, in millimetres | +| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) | +| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) | +| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. | +| `ZED_FLIP` | `ON` | `ON` or `OFF` | +| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup | +| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud | +| `ZED_PC_MODE` | `FULL` | `FULL` is a `[H, W, 4]` grid; `HALF` is a stride-2 downsample. Only applies with `ZED_POINT_CLOUD` on. | + +The defaults match the reference camera driver the perception stack was tuned against, and +the resolved depth settings are logged at startup. + +The point cloud is off because it is expensive: roughly 14.7 MB per frame at 1080p, paid +in retrieval and in a copy whether or not anything consumes it. Consumers that need 3D +back-project `depth` with the published intrinsics instead. + +## Build ```bash -git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git -cd dora-zed scripts/build.sh ``` -Binary installed to `bin/dora_zed`. +The binary is installed to `bin/dora_zed`. The script is also the dataflow's `build:` +command, since dora 1.0 runs build commands directly rather than through a shell — `cd` +and `&&` are not available there. The first build clones dora-rs and compiles its C++ +bindings with Cargo, which takes a while; later builds are incremental, and +`DORA_ROOT_DIR` builds against a local dora checkout instead. -`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build -commands directly rather than through a shell, so `cd`, `&&` and command substitution are -not available there; keeping the sequence in one script is what makes the dataflow build -work. It configures a `Release` build, which is also what links the optimized dora -bindings. +**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x at +`/usr/local/zed` (it bundles OpenCV), the CUDA Toolkit it needs, and a stable Rust +toolchain for the dora bindings. -### Containerized build +| Platform | Install the toolchain with | +|---|---| +| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` | +| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` | +| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1`, plus the [ZED SDK][sdk] by hand | -[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed. +[sdk]: https://www.stereolabs.com/developers/release + +[`Dockerfile`](Dockerfile) carries the same toolchain on Ubuntu 24.04 — the newest release +Stereolabs ships a ZED SDK for — so the host needs none of it, only an NVIDIA driver and +the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html): ```bash docker build -t dora-zed-build . @@ -110,134 +181,18 @@ docker run --rm --gpus all --user "$(id -u):$(id -g)" \ -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build ``` -The container runs the same `scripts/build.sh`. +It runs the same `scripts/build.sh`. The named volume keeps the Cargo cache across runs +and `--user` keeps `build/` and `bin/` owned by you. x86-64 only: on Jetson, run +[`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) and build natively. -The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs. -`--user` keeps `build/` and `bin/` owned by you rather than root. +## Test -x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the -device and build natively — there is no ARM64 image here. +```bash +tests/run.sh +``` -Needs an NVIDIA driver and the -[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) -on the host, as the ZED SDK does anyway. The image sets -`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC -(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host -only when `video` is in the capability list. - -## Configuration - -All behavior is controlled via environment variables. - -### Camera - -| Variable | Default | Description | -|---|---|---| -| `ZED_RESOLUTION` | `1080` | `720`, `1080`, or `2000` | -| `ZED_FPS` | `15` | Frame rate | -| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY`, or `NONE`. An unrecognised value is a startup error, not a fallback. | -| `ZED_DEPTH_MIN_MM` | `200` | Minimum depth (mm) | -| `ZED_DEPTH_MAX_MM` | `600` | Maximum depth (mm) | -| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) | -| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) | -| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. | -| `ZED_FLIP` | `ON` | `ON` or `OFF` | -| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup | - -The resolved depth mode, both confidence thresholds and the depth range are logged at -startup. The defaults match the reference camera driver the perception stack was tuned -against, so depth quality on the robot matches what it expects. - -### Point cloud - -Off by default. With `ZED_POINT_CLOUD` unset the XYZRGBA measure is not retrieved at all -and no `point_cloud` output is published — at 1080p a full cloud is roughly 14.7 MB per -frame, and retrieving it costs capture time whether or not anything consumes it. -Consumers that need 3D back-project `depth` with the published intrinsics instead. - -| Variable | Default | Description | -|---|---|---| -| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud | -| `ZED_PC_MODE` | `FULL` | Only applies when `ZED_POINT_CLOUD` is on | - -| `ZED_PC_MODE` | Shape | Description | -|---|---|---| -| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 1080p) | -| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) | - -The cloud is copied out of the SDK buffer, since the capture thread and the event loop are -different threads. That copy is the main reason to leave `ZED_POINT_CLOUD` off unless a -consumer needs it. - -## Inputs - -| Input | Description | -|---|---| -| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. | - -## Outputs - -| Output | Type | Description | -|---|---|---| -| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, timestamps. | -| `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. | -| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud, only when `ZED_POINT_CLOUD` is on. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. | -| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. Published **once per camera session** — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution. | -| `camera_state` | `string` | UTF-8 state name, published on each change. Metadata: `dtype=string`, `encoding=utf8`, `host_timestamp_ns`. | - -### Camera state and recovery - -`camera_state` lets a downstream node tell a camera that has not delivered its first -frame yet from one that has been lost: - -| State | Meaning | -|---|---| -| `opening` | The camera has not delivered a usable frame yet in this run | -| `streaming` | Frames are being published | -| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back | - -The camera runs on its own thread. The ZED SDK can block for an unbounded time inside -`grab()` and `open()` when the camera is disconnected — measured blocking indefinitely on -a ZED-M even with `async_grab_camera_recovery` set — so driving it from the dora event -loop would freeze the node: it would stop reading events, leaving `Stop` unanswered and -the recovery logic itself unreachable. The event loop only reads the newest frame the -capture thread has already produced, and therefore never calls into the SDK. - -Reopening is paced by a backoff of 1, 2, 5 then 10 seconds, the last repeated. Frames that -are not collected before the next one arrives are dropped: a consumer always gets the -newest frame rather than a queue of stale ones. - -The node treats an all-black frame as a failed grab: a camera whose link has dropped does -not necessarily fail its grabs, and can keep reporting success while handing back a -uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30 -seconds if the SDK reports it is reconnecting the link itself, since taking the handle -away mid-reconnect aborts a recovery that was already under way. - -The first recovery attempt closes the handle. Past that the device itself is wedged, and -the node escalates to `sl::Camera::reboot()`, which resets it over USB without unplugging -it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED -Mini the node closes the handle and lets the backoff give the endpoint time to clear. -Every attempt and every escalation is logged. - -### Timestamps - -Every frame output — `image_bgr`, `depth`, `point_cloud` and `camera_info` — carries both -clocks, as dora `timestamp` metadata parameters. `camera_state` carries -`host_timestamp_ns` only, since it is not tied to a frame. - -| Key | Epoch | Unit | Source | -|---|---|---|---| -| `device_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE)` — when the camera captured the frame | -| `host_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `std::chrono::system_clock` on the host — when this node published the frame | - -Both are needed. The device timestamp is the capture instant and is the value two -nodes reading the same frame agree on; the host timestamp is the one the rest of the -dataflow and the session recorder share. On a Jetson the wall clock can step when NTP -synchronises after boot, so the device timestamp is recorded alongside the host one -rather than in place of it. - -The node calls `sl::setTimestampClock(SYSTEM_CLOCK)` before opening the camera, which -is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock. - -Every output for a frame also carries the same `frame_id`, an integer counter starting -at 0. +Unit tests over `src/arrow`, which packs every payload this node sends: each exported +Arrow array is round-tripped through pyarrow to check what a consumer reads back, and +built again under the sanitizers to check that releasing one frees it. They need neither a +camera, the ZED SDK nor dora, which is what lets CI run them; everything past the packing +is covered by running a dataflow against a real camera. -- 2.54.0