From 8c3b3aac9288e1e9826e9984547b02821219681e Mon Sep 17 00:00:00 2001 From: Cristhian Date: Tue, 15 Sep 2026 06:56:49 -0300 Subject: [PATCH] rewrite the README It had drifted out of agreement with itself. The manifest section still said `image` is std/core/v1/UInt8 and depth and camera_info are std/core/v1/Float32, four lines above a table correctly saying both are std/media/v1/Image, and the worked example for a producer showed the flat-buffer form the node moved away from. Someone following it would have written a producer that fails validate. It was also close to twice the length of the one next door, mostly rationale: why a dataflow carries no strict_types, how a producer migrates, what a git: node needs and why. That belongs in the file it explains or in the commit that made the decision, not in front of someone trying to wire a node. 1007 words, from 1749. Every encoding, environment variable and CLI flag in it was checked against the code. --- README.md | 251 ++++++++++++++++++------------------------------------ 1 file changed, 81 insertions(+), 170 deletions(-) diff --git a/README.md b/README.md index 92a4d6d..cacc0fd 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,12 @@ # dora-image-view Dora node that displays a colour image and a depth map side by side in one OpenCV -window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a -vision pipeline: it produces no outputs. +window, with bounding box, text and camera intrinsics overlays. A display sink for +debugging a vision pipeline: it consumes frames and produces no outputs. Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot) -from dora-hub, brought up to the dora 1.0 node contract and packaged with `uv`. +from dora-hub, brought up to the dora 1.0 node contract, renamed, and changed to take +the typed Arrow image struct rather than a flat buffer plus metadata. ## Usage @@ -15,126 +16,44 @@ Add to your dataflow: nodes: - id: view git: https://gitea.liria.cl/liria/dora-image-view.git - tag: v0.1.0 + tag: v0.1.1 build: uv pip install . path: dora-image-view inputs: - image: camera/image_bgr + image: camera/image depth: camera/depth camera_info: camera/camera_info bbox: detector/bbox text: detector/text ``` -Run the dataflow with `--uv` (`dora run dataflow.yml --uv`) so the node gets its own -managed environment. +Run the dataflow with `--uv` (`dora run dataflow.yml --uv`): the build command is +`uv pip install .`, so the node gets its own managed environment and never installs into +an ambient one. -Every input is optional, and each of `image` and `depth` fills one panel of the window, -tiled left to right in that order. Wiring one of them is a plain viewer; -wiring several shows them side by side, scaled to a common height. `bbox`, `text` and `camera_info` -are overlays rather than panels — boxes are drawn on the `image` panel, in its own -pixels, and the caption on the composite. +Every input is optional. `image` and `depth` each fill a panel, tiled left to right and +scaled to a common height, so wiring one is a plain viewer and wiring both shows them +side by side. `bbox`, `text` and `camera_info` are overlays rather than panels: boxes on +the image panel in its own pixels, caption and camera model on the composite. -### End-to-end test +### Runnable dataflows -[`dataflow.yml`](dataflow.yml) is self-contained. [`tests/source.py`](tests/source.py) -is a synthetic source node that emits an animated frame, a depth map of the same scene, -the camera intrinsics, the box around the square moving in the frame, and a caption — -every input this node draws — so it needs no camera, no detector and no network. It is -a test fixture rather than a worked example; the realistic wiring is the block at the -top of this file. Run it to watch the node work: +Run either with `dora run --uv`: -```bash -dora run dataflow.yml --uv -``` +| File | Contents | +|---|---| +| [`dataflow.yml`](dataflow.yml) | A synthetic source driving every input. What CI runs. | +| [`dataflow_webcam.yml`](dataflow_webcam.yml) | A real camera, via [dora-webcam][webcam] | -A window opens with a square bouncing across a gradient, boxed and labelled. Press `q` -to quit. +[webcam]: https://gitea.liria.cl/liria/dora-webcam -It builds the viewer from this checkout (`path:` + `build:`) rather than from a tag, -which is what lets `dora validate` pick up the manifest described below. +`dataflow.yml` needs no camera, no detector and no network: a window opens with a square +bouncing across a gradient, boxed and labelled, beside its depth map. Press `q` to quit. +CI runs it headless on every push, which is what keeps it from rotting. -CI runs the same dataflow headless on every push, which is what makes it a gate rather -than a demo that rots. - -Set `SOURCE_FRAMES` to stop the source after a fixed number of frames instead of running -until you quit; that plus `CI=true` is how the same dataflow runs headless as a test: - -```bash -CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv -``` - -`SOURCE_WIDTH` and `SOURCE_HEIGHT` (default 640x480) set the frame size. - -### With a real camera - -[`dataflow_webcam.yml`](dataflow_webcam.yml) wires -[dora-webcam](https://gitea.liria.cl/liria/dora-webcam) into this node, and is the -worked example of consuming something that is not a test fixture: - -```bash -dora run dataflow_webcam.yml --uv -``` - -The camera publishes `std/media/v1/Image` and this node draws it. `depth`, -`camera_info`, `text` and `bbox` are simply unwired — every input is optional, so a -plain viewer is the result. - -It carries no `strict_types`, deliberately. A camera publishes an image and nothing -else, and `dora validate --strict` warns about every *typed* input this node declares -that a dataflow leaves unwired. The strictness belongs to `dataflow.yml`, which drives -all of them. - -Verified end to end: dora clones dora-webcam at the pinned tag, builds both nodes, and -runs them against a real camera to a clean exit. - -**A `git:` node needs anonymous read access.** dora-cli 1.0.1 fetches through libgit2 -with no credential callback, so `ssh://` cannot authenticate at all and anything asking -for credentials fails with `remote authentication required but no callback set`. On -Gitea, marking a repository public is not sufficient — a repository inside a private -organization is internal and answers an anonymous clone with `401`, so the organization -has to be public too. - -### Node manifest - -[`dora-node.yml`](dora-node.yml) declares the node's inputs with type URNs. No source -package is shared between node repositories, so it is the only contract a consumer has, -and it is what makes a wiring mistake fail at build time instead of at runtime. The -example source is typed inline in the dataflow with `output_types:`, so `dora validate` -checks the two against each other: - -```text -$ dora validate dataflow.yml - node "view": injected 4 contract type(s) from .../dora-node.yml -All type annotations OK. -``` - -There is one manifest because this repository ships one node. The example source is a -test fixture rather than something anyone installs, so it has none, and runs through -the interpreter to keep `dora validate` from matching the viewer's manifest against it. - -`image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and -`text` is `std/core/v1/String`; the payloads are raw buffers described by their -metadata, so the URN names the element type rather than a struct. `bbox` is deliberately -left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the -node reads either. - -## Behavior - -The node keeps a current frame, bbox set and text string. Each input updates its slice -of that state: - -- An **`image`** event decodes the frame and immediately redraws: it draws every stored - bbox (green rectangle plus `label, conf` text), draws the stored text at the top left, - resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled - `Dora Node: dora-image-view`. Press `q` in the window to quit. -- A **`depth`** event colour-maps the distances onto the depth panel and redraws. -- A **`camera_info`** event stores the pinhole model and draws it under the caption. -- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next - redraw. - -Setting `CI=true` suppresses the display: the node still decodes every input but opens -no window. That is what makes it safe to run in a pipeline test. +`dataflow_webcam.yml` is the worked example of consuming a real producer. A `git:` node +needs **anonymous** read access to fetch — on Gitea that means a public organization, not +just a public repository. ## Inputs @@ -142,59 +61,42 @@ no window. That is what makes it safe to run in a pipeline test. |---|---|---| | `image` | `std/media/v1/Image` struct | none — the struct describes itself | | `depth` | `std/media/v1/Image` struct, one channel | `units` (defaults to `mm`) | -| `camera_info` | `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` | -| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` | +| `camera_info` | `[fx, fy, cx, cy]` or a 9-element row-major `K` | `distortion` | +| `bbox` | Arrow struct `{bbox, conf, labels}` | `format`: `xyxy` or `xywh` | | `text` | Arrow array of size 1 | — | `image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`, -`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes -itself, so nothing about its shape has to travel as message metadata. +`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself, +so nothing about its shape travels as message metadata. -Depth is an image whose pixels are distances — dora's type registry has no depth type, -and this is how ROS spells it too. Its encodings are `32FC1`/`float32` and -`16UC1`/`mono16`/`uint16`, one channel per pixel, and `units` metadata still switches -millimetres to metres. +Image encodings: `bgr8`, `rgb8`, `yuv420`, `avif`, and `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/ +`png` as an encoded byte stream. Depth encodings: `32FC1`/`float32` and +`16UC1`/`mono16`/`uint16`, one channel per pixel — dora's registry has no depth type, and +a depth map is an image whose pixels are distances, as ROS also treats it. `0` and +non-finite values mean missing depth and draw black. + +`camera_info` is also read from the metadata of any frame carrying `fx`, `fy`, `cx` and +`cy`, so **wiring it is optional even for a camera that publishes it** — which covers a +camera that publishes it once per session and a viewer started afterwards. The node needs +none of it to draw; it is held for whatever turns depth into 3D. Older payloads are still decoded, so a producer can move at its own pace: a flat `UInt8` -buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32` -buffer with `shape`/`units` metadata for `depth`. +buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32` buffer +with `shape`/`units` for `depth`. Note that dora's type compatibility is directional, so +a producer still *declaring* `std/core/v1/UInt8` fails `dora validate` against an +`std/media/v1/Image` input whatever it sends — its manifest has to move too. -**Migrating a producer.** dora's type compatibility is directional — `Image` satisfies a -`Bytes` input, but a `std/core/v1/UInt8` output wired into an `std/media/v1/Image` input -is a `dora validate` error. A producer still declaring `std/core/v1/UInt8` or -`std/core/v1/Float32` therefore has to change its manifest as well as its payload. - -Supported `image` `encoding` values: `bgr8`, `rgb8`, `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/`png` -(encoded byte stream), `yuv420`, `avif`. `width` and `height` are only needed by the raw -encodings — the rest carry their own dimensions. An unsupported encoding raises. - -Supported `format` values: `xyxy` and `xywh` (centre-based, converted to `xyxy`). - -`depth` is in millimetres unless `units` is `m`. In a depth map, `0` -and any non-finite value mean missing depth and are drawn black. A payload sent as a -raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the -`Float32` contract, and is reinterpreted rather than converted. - -### Camera intrinsics - -`camera_info` carries the pinhole model the depth map is interpreted through, either as -the four values `[fx, fy, cx, cy]` or as a nine-element row-major `K`, which is how ROS -spells the same model in `sensor_msgs/CameraInfo`. Distortion coefficients are read from -the `distortion` metadata. - -The node displays the model and holds it. It needs none of it to draw a panel; it is -there for what reads it next — turning a depth map into a point cloud needs `fx`, `fy`, -`cx` and `cy` for every pixel, and nothing else in a dataflow carries them. - -Intrinsics are also picked up from the metadata of any `image` or `depth` frame that -carries `fx`, `fy`, `cx` and `cy`, so **wiring `camera_info` is optional even for a -camera that publishes it**. That matters because a camera typically publishes -`camera_info` once per session: a viewer started after the camera would otherwise never -see it. +Publishing to it: ```python -node.send_output("image", pa.array(frame.ravel()), - {"width": 640, "height": 480, "encoding": "bgr8"}) +image = pa.array( + [{"width": 640, "height": 480, "encoding": "bgr8", "data": frame.tobytes()}], + type=pa.struct([ + ("width", pa.uint32()), ("height", pa.uint32()), + ("encoding", pa.utf8()), ("data", pa.large_binary()), + ]), +) +node.send_output("image", image, {}) node.send_output("bbox", pa.array([{ "bbox": np.array([...]), # flattened, four values per box @@ -209,7 +111,19 @@ node.send_output("text", pa.array(["hello"])) None — it is a display sink. -## Environment variables +## Configuration + +Settings are environment variables, which a dataflow can carry per node, keeping them +with the wiring rather than in whatever shell starts it: + +```yaml + env: + VIEW_DEPTH_MIN_MM: 200 + VIEW_DEPTH_MAX_MM: 600 +``` + +A node's `env:` wins over the environment the dataflow was started in; variables it does +not set still pass through. | Variable | Default | Meaning | |---|---|---| @@ -218,18 +132,16 @@ None — it is a display sink. | `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. | | `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. | -Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set. Both are also -available as `--view-width` and `--view-height`; the environment variable wins when both -are given. +Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set; both are also +available as `--view-width` and `--view-height`, and the environment variable wins. -Each end of the depth range is auto-ranged over the valid pixels of the current frame -unless it is pinned, so pinning one end leaves the other automatic. Auto-ranging makes -an unconfigured sensor legible at the cost of colours that shift from frame to frame; -pin both ends to compare frames against each other. The range covers the full extent of -the valid pixels rather than a percentile of them, so a small near object against a -large wall stays visible. +Each end of the depth range is auto-ranged over the current frame's valid pixels unless +pinned, so pinning one end leaves the other automatic. Auto-ranging makes an +unconfigured sensor legible at the cost of colours that shift between frames; pin both +ends to compare frames against each other. -`CI=true` suppresses the display window; it is not a configuration knob. +`CI=true` suppresses the window while still decoding every input, which is what makes the +node safe inside a pipeline test. It is not a configuration knob. ## Build @@ -238,13 +150,12 @@ uv sync ``` Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64, -so the node installs with no system packages and no compiler on any of them — including -AVIF support, which comes from `pillow-avif-plugin` rather than OpenCV. +so the node installs with no system packages and no compiler — including AVIF support, +which comes from `pillow-avif-plugin` rather than OpenCV. **Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), Python 3.11+, `uv`, and a -display to show the window on (or `CI=true` to run without one). On a bare Linux image -`opencv-python` also needs the libGL shared libraries (`libgl1`, `libglib2.0-0`), which -it links at import time. +display (or `CI=true` to run without one). On a bare Linux image `opencv-python` needs +the libGL shared libraries (`libgl1`, `libglib2.0-0`), which it links at import time. ## Test @@ -252,6 +163,6 @@ it links at import time. uv run pytest ``` -Unit tests over decoding, the bbox format conversion and the headless draw path; they -need neither a camera, a dataflow nor a display. The headless dataflow run above covers -the rest — the manifest, the wiring and the event loop. +Unit tests over decoding, depth, intrinsics, the bbox conversion and the headless draw +path; they need neither a camera, a dataflow nor a display. The headless `dataflow.yml` +run covers the rest — the manifest, the wiring and the event loop.