liria/dora-image-view
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

rewrite the README
validate / validate (pull_request) Successful in 59m14s
Details

Browse Source 
It had drifted out of agreement with itself. The manifest section still said
`image` is std/core/v1/UInt8 and depth and camera_info are std/core/v1/Float32,
four lines above a table correctly saying both are std/media/v1/Image, and the
worked example for a producer showed the flat-buffer form the node moved away
from. Someone following it would have written a producer that fails validate.

It was also close to twice the length of the one next door, mostly rationale:
why a dataflow carries no strict_types, how a producer migrates, what a git:
node needs and why. That belongs in the file it explains or in the commit that
made the decision, not in front of someone trying to wire a node.

1007 words, from 1749. Every encoding, environment variable and CLI flag in it
was checked against the code.
This commit is contained in:
cristhian
2026-09-15 06:56:49 -03:00
parent a371bb07ba
commit 8c3b3aac92
1 changed files with 81 additions and 170 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+81 -170
View File
@@ -1,11 +1,12 @@
# dora-image-view # dora-image-view
Dora node that displays a colour image and a depth map side by side in one OpenCV Dora node that displays a colour image and a depth map side by side in one OpenCV
window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a window, with bounding box, text and camera intrinsics overlays. A display sink for
vision pipeline: it produces no outputs. debugging a vision pipeline: it consumes frames and produces no outputs.
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot) Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
from dora-hub, brought up to the dora 1.0 node contract and packaged with `uv`. from dora-hub, brought up to the dora 1.0 node contract, renamed, and changed to take
the typed Arrow image struct rather than a flat buffer plus metadata.
## Usage ## Usage
@@ -15,126 +16,44 @@ Add to your dataflow:
nodes: nodes:
- id: view - id: view
git: https://gitea.liria.cl/liria/dora-image-view.git git: https://gitea.liria.cl/liria/dora-image-view.git
tag: v0.1.0 tag: v0.1.1
build: uv pip install . build: uv pip install .
path: dora-image-view path: dora-image-view
inputs: inputs:
image: camera/image_bgr image: camera/image
depth: camera/depth depth: camera/depth
camera_info: camera/camera_info camera_info: camera/camera_info
bbox: detector/bbox bbox: detector/bbox
text: detector/text text: detector/text
``` ```
Run the dataflow with `--uv` (`dora run dataflow.yml --uv`) so the node gets its own Run the dataflow with `--uv` (`dora run dataflow.yml --uv`): the build command is
managed environment. `uv pip install .`, so the node gets its own managed environment and never installs into
an ambient one.
Every input is optional, and each of `image` and `depth` fills one panel of the window, Every input is optional. `image` and `depth` each fill a panel, tiled left to right and
tiled left to right in that order. Wiring one of them is a plain viewer; scaled to a common height, so wiring one is a plain viewer and wiring both shows them
wiring several shows them side by side, scaled to a common height. `bbox`, `text` and `camera_info` side by side. `bbox`, `text` and `camera_info` are overlays rather than panels: boxes on
are overlays rather than panels — boxes are drawn on the `image` panel, in its own the image panel in its own pixels, caption and camera model on the composite.
pixels, and the caption on the composite.
### End-to-end test ### Runnable dataflows
[`dataflow.yml`](dataflow.yml) is self-contained. [`tests/source.py`](tests/source.py) Run either with `dora run <file> --uv`:
is a synthetic source node that emits an animated frame, a depth map of the same scene,
the camera intrinsics, the box around the square moving in the frame, and a caption —
every input this node draws — so it needs no camera, no detector and no network. It is
a test fixture rather than a worked example; the realistic wiring is the block at the
top of this file. Run it to watch the node work:
```bash | File | Contents |
dora run dataflow.yml --uv |---|---|
``` | [`dataflow.yml`](dataflow.yml) | A synthetic source driving every input. What CI runs. |
| [`dataflow_webcam.yml`](dataflow_webcam.yml) | A real camera, via [dora-webcam][webcam] |
A window opens with a square bouncing across a gradient, boxed and labelled. Press `q` [webcam]: https://gitea.liria.cl/liria/dora-webcam
to quit.
It builds the viewer from this checkout (`path:` + `build:`) rather than from a tag, `dataflow.yml` needs no camera, no detector and no network: a window opens with a square
which is what lets `dora validate` pick up the manifest described below. bouncing across a gradient, boxed and labelled, beside its depth map. Press `q` to quit.
CI runs it headless on every push, which is what keeps it from rotting.
CI runs the same dataflow headless on every push, which is what makes it a gate rather `dataflow_webcam.yml` is the worked example of consuming a real producer. A `git:` node
than a demo that rots. needs **anonymous** read access to fetch — on Gitea that means a public organization, not
just a public repository.
Set `SOURCE_FRAMES` to stop the source after a fixed number of frames instead of running
until you quit; that plus `CI=true` is how the same dataflow runs headless as a test:
```bash
CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv
```
`SOURCE_WIDTH` and `SOURCE_HEIGHT` (default 640x480) set the frame size.
### With a real camera
[`dataflow_webcam.yml`](dataflow_webcam.yml) wires
[dora-webcam](https://gitea.liria.cl/liria/dora-webcam) into this node, and is the
worked example of consuming something that is not a test fixture:
```bash
dora run dataflow_webcam.yml --uv
```
The camera publishes `std/media/v1/Image` and this node draws it. `depth`,
`camera_info`, `text` and `bbox` are simply unwired — every input is optional, so a
plain viewer is the result.
It carries no `strict_types`, deliberately. A camera publishes an image and nothing
else, and `dora validate --strict` warns about every *typed* input this node declares
that a dataflow leaves unwired. The strictness belongs to `dataflow.yml`, which drives
all of them.
Verified end to end: dora clones dora-webcam at the pinned tag, builds both nodes, and
runs them against a real camera to a clean exit.
**A `git:` node needs anonymous read access.** dora-cli 1.0.1 fetches through libgit2
with no credential callback, so `ssh://` cannot authenticate at all and anything asking
for credentials fails with `remote authentication required but no callback set`. On
Gitea, marking a repository public is not sufficient — a repository inside a private
organization is internal and answers an anonymous clone with `401`, so the organization
has to be public too.
### Node manifest
[`dora-node.yml`](dora-node.yml) declares the node's inputs with type URNs. No source
package is shared between node repositories, so it is the only contract a consumer has,
and it is what makes a wiring mistake fail at build time instead of at runtime. The
example source is typed inline in the dataflow with `output_types:`, so `dora validate`
checks the two against each other:
```text
$ dora validate dataflow.yml
node "view": injected 4 contract type(s) from .../dora-node.yml
All type annotations OK.
```
There is one manifest because this repository ships one node. The example source is a
test fixture rather than something anyone installs, so it has none, and runs through
the interpreter to keep `dora validate` from matching the viewer's manifest against it.
`image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and
`text` is `std/core/v1/String`; the payloads are raw buffers described by their
metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
node reads either.
## Behavior
The node keeps a current frame, bbox set and text string. Each input updates its slice
of that state:
- An **`image`** event decodes the frame and immediately redraws: it draws every stored
bbox (green rectangle plus `label, conf` text), draws the stored text at the top left,
resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
`Dora Node: dora-image-view`. Press `q` in the window to quit.
- A **`depth`** event colour-maps the distances onto the depth panel and redraws.
- A **`camera_info`** event stores the pinhole model and draws it under the caption.
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
redraw.
Setting `CI=true` suppresses the display: the node still decodes every input but opens
no window. That is what makes it safe to run in a pipeline test.
## Inputs ## Inputs
@@ -142,59 +61,42 @@ no window. That is what makes it safe to run in a pipeline test.
|---|---|---| |---|---|---|
| `image` | `std/media/v1/Image` struct | none — the struct describes itself | | `image` | `std/media/v1/Image` struct | none — the struct describes itself |
| `depth` | `std/media/v1/Image` struct, one channel | `units` (defaults to `mm`) | | `depth` | `std/media/v1/Image` struct, one channel | `units` (defaults to `mm`) |
| `camera_info` | `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` | | `camera_info` | `[fx, fy, cx, cy]` or a 9-element row-major `K` | `distortion` |
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` | | `bbox` | Arrow struct `{bbox, conf, labels}` | `format`: `xyxy` or `xywh` |
| `text` | Arrow array of size 1 | — | | `text` | Arrow array of size 1 | — |
`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`, `image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`,
`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes `height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself,
itself, so nothing about its shape has to travel as message metadata. so nothing about its shape travels as message metadata.
Depth is an image whose pixels are distances — dora's type registry has no depth type, Image encodings: `bgr8`, `rgb8`, `yuv420`, `avif`, and `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/
and this is how ROS spells it too. Its encodings are `32FC1`/`float32` and `png` as an encoded byte stream. Depth encodings: `32FC1`/`float32` and
`16UC1`/`mono16`/`uint16`, one channel per pixel, and `units` metadata still switches `16UC1`/`mono16`/`uint16`, one channel per pixel — dora's registry has no depth type, and
millimetres to metres. a depth map is an image whose pixels are distances, as ROS also treats it. `0` and
non-finite values mean missing depth and draw black.
`camera_info` is also read from the metadata of any frame carrying `fx`, `fy`, `cx` and
`cy`, so **wiring it is optional even for a camera that publishes it** — which covers a
camera that publishes it once per session and a viewer started afterwards. The node needs
none of it to draw; it is held for whatever turns depth into 3D.
Older payloads are still decoded, so a producer can move at its own pace: a flat `UInt8` Older payloads are still decoded, so a producer can move at its own pace: a flat `UInt8`
buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32` buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32` buffer
buffer with `shape`/`units` metadata for `depth`. with `shape`/`units` for `depth`. Note that dora's type compatibility is directional, so
a producer still *declaring* `std/core/v1/UInt8` fails `dora validate` against an
`std/media/v1/Image` input whatever it sends — its manifest has to move too.
**Migrating a producer.** dora's type compatibility is directional — `Image` satisfies a Publishing to it:
`Bytes` input, but a `std/core/v1/UInt8` output wired into an `std/media/v1/Image` input
is a `dora validate` error. A producer still declaring `std/core/v1/UInt8` or
`std/core/v1/Float32` therefore has to change its manifest as well as its payload.
Supported `image` `encoding` values: `bgr8`, `rgb8`, `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/`png`
(encoded byte stream), `yuv420`, `avif`. `width` and `height` are only needed by the raw
encodings — the rest carry their own dimensions. An unsupported encoding raises.
Supported `format` values: `xyxy` and `xywh` (centre-based, converted to `xyxy`).
`depth` is in millimetres unless `units` is `m`. In a depth map, `0`
and any non-finite value mean missing depth and are drawn black. A payload sent as a
raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
`Float32` contract, and is reinterpreted rather than converted.
### Camera intrinsics
`camera_info` carries the pinhole model the depth map is interpreted through, either as
the four values `[fx, fy, cx, cy]` or as a nine-element row-major `K`, which is how ROS
spells the same model in `sensor_msgs/CameraInfo`. Distortion coefficients are read from
the `distortion` metadata.
The node displays the model and holds it. It needs none of it to draw a panel; it is
there for what reads it next — turning a depth map into a point cloud needs `fx`, `fy`,
`cx` and `cy` for every pixel, and nothing else in a dataflow carries them.
Intrinsics are also picked up from the metadata of any `image` or `depth` frame that
carries `fx`, `fy`, `cx` and `cy`, so **wiring `camera_info` is optional even for a
camera that publishes it**. That matters because a camera typically publishes
`camera_info` once per session: a viewer started after the camera would otherwise never
see it.
```python ```python
node.send_output("image", pa.array(frame.ravel()), image = pa.array(
{"width": 640, "height": 480, "encoding": "bgr8"}) [{"width": 640, "height": 480, "encoding": "bgr8", "data": frame.tobytes()}],
type=pa.struct([
("width", pa.uint32()), ("height", pa.uint32()),
("encoding", pa.utf8()), ("data", pa.large_binary()),
]),
)
node.send_output("image", image, {})
node.send_output("bbox", pa.array([{ node.send_output("bbox", pa.array([{
"bbox": np.array([...]), # flattened, four values per box "bbox": np.array([...]), # flattened, four values per box
@@ -209,7 +111,19 @@ node.send_output("text", pa.array(["hello"]))
None — it is a display sink. None — it is a display sink.
## Environment variables ## Configuration
Settings are environment variables, which a dataflow can carry per node, keeping them
with the wiring rather than in whatever shell starts it:
```yaml
env:
VIEW_DEPTH_MIN_MM: 200
VIEW_DEPTH_MAX_MM: 600
```
A node's `env:` wins over the environment the dataflow was started in; variables it does
not set still pass through.
| Variable | Default | Meaning | | Variable | Default | Meaning |
|---|---|---| |---|---|---|
@@ -218,18 +132,16 @@ None — it is a display sink.
| `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. | | `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. |
| `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. | | `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. |
Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set. Both are also Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set; both are also
available as `--view-width` and `--view-height`; the environment variable wins when both available as `--view-width` and `--view-height`, and the environment variable wins.
are given.
Each end of the depth range is auto-ranged over the valid pixels of the current frame Each end of the depth range is auto-ranged over the current frame's valid pixels unless
unless it is pinned, so pinning one end leaves the other automatic. Auto-ranging makes pinned, so pinning one end leaves the other automatic. Auto-ranging makes an
an unconfigured sensor legible at the cost of colours that shift from frame to frame; unconfigured sensor legible at the cost of colours that shift between frames; pin both
pin both ends to compare frames against each other. The range covers the full extent of ends to compare frames against each other.
the valid pixels rather than a percentile of them, so a small near object against a
large wall stays visible.
`CI=true` suppresses the display window; it is not a configuration knob. `CI=true` suppresses the window while still decoding every input, which is what makes the
node safe inside a pipeline test. It is not a configuration knob.
## Build ## Build
@@ -238,13 +150,12 @@ uv sync
``` ```
Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64, Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64,
so the node installs with no system packages and no compiler on any of them — including so the node installs with no system packages and no compiler — including AVIF support,
AVIF support, which comes from `pillow-avif-plugin` rather than OpenCV. which comes from `pillow-avif-plugin` rather than OpenCV.
**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), Python 3.11+, `uv`, and a **Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), Python 3.11+, `uv`, and a
display to show the window on (or `CI=true` to run without one). On a bare Linux image display (or `CI=true` to run without one). On a bare Linux image `opencv-python` needs
`opencv-python` also needs the libGL shared libraries (`libgl1`, `libglib2.0-0`), which the libGL shared libraries (`libgl1`, `libglib2.0-0`), which it links at import time.
it links at import time.
## Test ## Test
@@ -252,6 +163,6 @@ it links at import time.
uv run pytest uv run pytest
``` ```
Unit tests over decoding, the bbox format conversion and the headless draw path; they Unit tests over decoding, depth, intrinsics, the bbox conversion and the headless draw
need neither a camera, a dataflow nor a display. The headless dataflow run above covers path; they need neither a camera, a dataflow nor a display. The headless `dataflow.yml`
the rest — the manifest, the wiring and the event loop. run covers the rest — the manifest, the wiring and the event loop.