# dora-image-view

Dora node that displays a colour image and a depth map side by side in one OpenCV
window, with bounding box, text and camera intrinsics overlays. A display sink for
debugging a vision pipeline: it consumes frames and produces no outputs.

Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
from dora-hub, brought up to the dora 1.0 node contract, renamed, and changed to take
the typed Arrow image struct rather than a flat buffer plus metadata.

## Usage

Add to your dataflow:

```yaml
nodes:
  - id: view
    git: https://gitea.liria.cl/liria/dora-image-view.git
    tag: v0.1.1
    build: uv pip install .
    path: dora-image-view
    inputs:
      image: camera/image
      depth: camera/depth
      camera_info: camera/camera_info
      bbox: detector/bbox
      text: detector/text
```

Run the dataflow with `--uv` (`dora run dataflow.yml --uv`): the build command is
`uv pip install .`, so the node gets its own managed environment and never installs into
an ambient one.

Every input is optional. `image` and `depth` each fill a panel, tiled left to right and
scaled to a common height, so wiring one is a plain viewer and wiring both shows them
side by side. `bbox`, `text` and `camera_info` are overlays rather than panels: boxes on
the image panel in its own pixels, caption and camera model on the composite.

### Runnable dataflows

Run either with `dora run <file> --uv`:

| File | Contents |
|---|---|
| [`dataflow.yml`](dataflow.yml) | A synthetic source driving every input. What CI runs. |
| [`dataflow_webcam.yml`](dataflow_webcam.yml) | A real camera, via [dora-webcam][webcam] |

[webcam]: https://gitea.liria.cl/liria/dora-webcam

`dataflow.yml` needs no camera, no detector and no network: a window opens with a square
bouncing across a gradient, boxed and labelled, beside its depth map. Press `q` to quit.
CI runs it headless on every push, which is what keeps it from rotting.

`dataflow_webcam.yml` is the worked example of consuming a real producer. A `git:` node
needs **anonymous** read access to fetch — on Gitea that means a public organization, not
just a public repository.

## Inputs

| Input | Payload | Metadata |
|---|---|---|
| `image` | `std/media/v1/Image` struct | none — the struct describes itself |
| `depth` | `std/media/v1/Image` struct, one channel | `units` (defaults to `mm`) |
| `camera_info` | `[fx, fy, cx, cy]` or a 9-element row-major `K` | `distortion` |
| `bbox` | Arrow struct `{bbox, conf, labels}` | `format`: `xyxy` or `xywh` |
| `text` | Arrow array of size 1 | — |

`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`,
`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself,
so nothing about its shape travels as message metadata.

Image encodings: `bgr8`, `rgb8`, `yuv420`, `avif`, and `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/
`png` as an encoded byte stream. Depth encodings: `32FC1`/`float32` and
`16UC1`/`mono16`/`uint16`, one channel per pixel — dora's registry has no depth type, and
a depth map is an image whose pixels are distances, as ROS also treats it. `0` and
non-finite values mean missing depth and draw black.

`camera_info` is also read from the metadata of any frame carrying `fx`, `fy`, `cx` and
`cy`, so **wiring it is optional even for a camera that publishes it** — which covers a
camera that publishes it once per session and a viewer started afterwards. The node needs
none of it to draw; it is held for whatever turns depth into 3D.

Older payloads are still decoded, so a producer can move at its own pace: a flat `UInt8`
buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32` buffer
with `shape`/`units` for `depth`. Note that dora's type compatibility is directional, so
a producer still *declaring* `std/core/v1/UInt8` fails `dora validate` against an
`std/media/v1/Image` input whatever it sends — its manifest has to move too.

Publishing to it:

```python
image = pa.array(
    [{"width": 640, "height": 480, "encoding": "bgr8", "data": frame.tobytes()}],
    type=pa.struct([
        ("width", pa.uint32()), ("height", pa.uint32()),
        ("encoding", pa.utf8()), ("data", pa.large_binary()),
    ]),
)
node.send_output("image", image, {})

node.send_output("bbox", pa.array([{
    "bbox": np.array([...]),    # flattened, four values per box
    "conf": np.array([...]),
    "labels": np.array([...]),
}]), {"format": "xyxy"})

node.send_output("text", pa.array(["hello"]))
```

## Outputs

None — it is a display sink.

## Configuration

Settings are environment variables, which a dataflow can carry per node, keeping them
with the wiring rather than in whatever shell starts it:

```yaml
    env:
      VIEW_DEPTH_MIN_MM: 200
      VIEW_DEPTH_MAX_MM: 600
```

A node's `env:` wins over the environment the dataflow was started in; variables it does
not set still pass through.

| Variable | Default | Meaning |
|---|---|---|
| `VIEW_WIDTH` | composite width | Resize the displayed composite to this width, in pixels. |
| `VIEW_HEIGHT` | composite height | Resize the displayed composite to this height, in pixels. |
| `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. |
| `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. |

Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set; both are also
available as `--view-width` and `--view-height`, and the environment variable wins.

Each end of the depth range is auto-ranged over the current frame's valid pixels unless
pinned, so pinning one end leaves the other automatic. Auto-ranging makes an
unconfigured sensor legible at the cost of colours that shift between frames; pin both
ends to compare frames against each other.

`CI=true` suppresses the window while still decoding every input, which is what makes the
node safe inside a pipeline test. It is not a configuration knob.

## Build

```bash
uv sync
```

Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64,
so the node installs with no system packages and no compiler — including AVIF support,
which comes from `pillow-avif-plugin` rather than OpenCV.

**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), Python 3.11+, `uv`, and a
display (or `CI=true` to run without one). On a bare Linux image `opencv-python` needs
the libGL shared libraries (`libgl1`, `libglib2.0-0`), which it links at import time.

## Test

```bash
uv run pytest
```

Unit tests over decoding, depth, intrinsics, the bbox conversion and the headless draw
path; they need neither a camera, a dataflow nor a display. The headless `dataflow.yml`
run covers the rest — the manifest, the wiring and the event loop.