dora-image-view/README.md

# dora-image-view

Dora node that displays a colour image and a depth map side by side in one OpenCV
window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a
vision pipeline: it produces no outputs.

Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
from dora-hub, brought up to the dora 1.0 node contract and packaged with `uv`.

## Usage

Add to your dataflow:

```yaml
nodes:
  - id: view
    git: ssh://git@gitea.liria.cl:5001/liria/dora-image-view.git
    tag: v0.1.0
    build: pip install .
    path: dora-image-view
    inputs:
      image: camera/image_bgr
      depth: camera/depth
      camera_info: camera/camera_info
      bbox: detector/bbox
      text: detector/text
```

Run the dataflow with `--uv` (`dora run dataflow.yml --uv`) so the node gets its own
managed environment.

Every input is optional, and each of `image` and `depth` fills one panel of the window,
tiled left to right in that order. Wiring one of them is a plain viewer;
wiring several shows them side by side, scaled to a common height. `bbox`, `text` and `camera_info`
are overlays rather than panels — boxes are drawn on the `image` panel, in its own
pixels, and the caption on the composite.

### End-to-end test

[`dataflow.yml`](dataflow.yml) is self-contained. [`tests/source.py`](tests/source.py)
is a synthetic source node that emits an animated frame, a depth map of the same scene,
the camera intrinsics, the box around the square moving in the frame, and a caption —
every input this node draws — so it needs no camera, no detector and no network. It is
a test fixture rather than a worked example; the realistic wiring is the block at the
top of this file. Run it to watch the node work:

```bash
dora run dataflow.yml --uv
```

A window opens with a square bouncing across a gradient, boxed and labelled. Press `q`
to quit.

It builds the viewer from this checkout (`path:` + `build:`) rather than from a tag,
which is what lets `dora validate` pick up the manifest described below.

CI runs the same dataflow headless on every push, which is what makes it a gate rather
than a demo that rots.

Set `SOURCE_FRAMES` to stop the source after a fixed number of frames instead of running
until you quit; that plus `CI=true` is how the same dataflow runs headless as a test:

```bash
CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv
```

`SOURCE_WIDTH` and `SOURCE_HEIGHT` (default 640x480) set the frame size.

### Node manifest

[`dora-node.yml`](dora-node.yml) declares the node's inputs with type URNs. No source
package is shared between node repositories, so it is the only contract a consumer has,
and it is what makes a wiring mistake fail at build time instead of at runtime. The
example source is typed inline in the dataflow with `output_types:`, so `dora validate`
checks the two against each other:

```text
$ dora validate dataflow.yml
  node "view": injected 4 contract type(s) from .../dora-node.yml
All type annotations OK.
```

There is one manifest because this repository ships one node. The example source is a
test fixture rather than something anyone installs, so it has none, and runs through
the interpreter to keep `dora validate` from matching the viewer's manifest against it.

`image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and
`text` is `std/core/v1/String`; the payloads are raw buffers described by their
metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
node reads either.

## Behavior

The node keeps a current frame, bbox set and text string. Each input updates its slice
of that state:

- An **`image`** event decodes the frame and immediately redraws: it draws every stored
  bbox (green rectangle plus `label, conf` text), draws the stored text at the top left,
  resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
  `Dora Node: dora-image-view`. Press `q` in the window to quit.
- A **`depth`** event colour-maps the distances onto the depth panel and redraws.
- A **`camera_info`** event stores the pinhole model and draws it under the caption.
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
  redraw.

Setting `CI=true` suppresses the display: the node still decodes every input but opens
no window. That is what makes it safe to run in a pipeline test.

## Inputs

| Input | Payload | Metadata |
|---|---|---|
| `image` | `std/media/v1/Image` struct | none — the struct describes itself |
| `depth` | `std/media/v1/Image` struct, one channel | `units` (defaults to `mm`) |
| `camera_info` | `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` |
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
| `text` | Arrow array of size 1 | — |

`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`,
`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes
itself, so nothing about its shape has to travel as message metadata.

Depth is an image whose pixels are distances — dora's type registry has no depth type,
and this is how ROS spells it too. Its encodings are `32FC1`/`float32` and
`16UC1`/`mono16`/`uint16`, one channel per pixel, and `units` metadata still switches
millimetres to metres.

Older payloads are still decoded, so a producer can move at its own pace: a flat `UInt8`
buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32`
buffer with `shape`/`units` metadata for `depth`.

**Migrating a producer.** dora's type compatibility is directional — `Image` satisfies a
`Bytes` input, but a `std/core/v1/UInt8` output wired into an `std/media/v1/Image` input
is a `dora validate` error. A producer still declaring `std/core/v1/UInt8` or
`std/core/v1/Float32` therefore has to change its manifest as well as its payload.

Supported `image` `encoding` values: `bgr8`, `rgb8`, `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/`png`
(encoded byte stream), `yuv420`, `avif`. `width` and `height` are only needed by the raw
encodings — the rest carry their own dimensions. An unsupported encoding raises.

Supported `format` values: `xyxy` and `xywh` (centre-based, converted to `xyxy`).

`depth` is in millimetres unless `units` is `m`. In a depth map, `0`
and any non-finite value mean missing depth and are drawn black. A payload sent as a
raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
`Float32` contract, and is reinterpreted rather than converted.

### Camera intrinsics

`camera_info` carries the pinhole model the depth map is interpreted through, either as
the four values `[fx, fy, cx, cy]` or as a nine-element row-major `K`, which is how ROS
spells the same model in `sensor_msgs/CameraInfo`. Distortion coefficients are read from
the `distortion` metadata.

The node displays the model and holds it. It needs none of it to draw a panel; it is
there for what reads it next — turning a depth map into a point cloud needs `fx`, `fy`,
`cx` and `cy` for every pixel, and nothing else in a dataflow carries them.

Intrinsics are also picked up from the metadata of any `image` or `depth` frame that
carries `fx`, `fy`, `cx` and `cy`, so **wiring `camera_info` is optional even for a
camera that publishes it**. That matters because a camera typically publishes
`camera_info` once per session: a viewer started after the camera would otherwise never
see it.

```python
node.send_output("image", pa.array(frame.ravel()),
                 {"width": 640, "height": 480, "encoding": "bgr8"})

node.send_output("bbox", pa.array([{
    "bbox": np.array([...]),    # flattened, four values per box
    "conf": np.array([...]),
    "labels": np.array([...]),
}]), {"format": "xyxy"})

node.send_output("text", pa.array(["hello"]))
```

## Outputs

None — it is a display sink.

## Environment variables

| Variable | Default | Meaning |
|---|---|---|
| `VIEW_WIDTH` | composite width | Resize the displayed composite to this width, in pixels. |
| `VIEW_HEIGHT` | composite height | Resize the displayed composite to this height, in pixels. |
| `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. |
| `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. |

Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set. Both are also
available as `--view-width` and `--view-height`; the environment variable wins when both
are given.

Each end of the depth range is auto-ranged over the valid pixels of the current frame
unless it is pinned, so pinning one end leaves the other automatic. Auto-ranging makes
an unconfigured sensor legible at the cost of colours that shift from frame to frame;
pin both ends to compare frames against each other. The range covers the full extent of
the valid pixels rather than a percentile of them, so a small near object against a
large wall stays visible.

`CI=true` suppresses the display window; it is not a configuration knob.

## Build

```bash
uv sync
```

Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64,
so the node installs with no system packages and no compiler on any of them — including
AVIF support, which comes from `pillow-avif-plugin` rather than OpenCV.

**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), Python 3.11+, `uv`, and a
display to show the window on (or `CI=true` to run without one). On a bare Linux image
`opencv-python` also needs the libGL shared libraries (`libgl1`, `libglib2.0-0`), which
it links at import time.

## Test

```bash
uv run pytest
```

Unit tests over decoding, the bbox format conversion and the headless draw path; they
need neither a camera, a dataflow nor a display. The headless dataflow run above covers
the rest — the manifest, the wiring and the event loop.