dora-image-view/README.md

dora-image-view

Dora node that displays a colour image and a depth map side by side in one OpenCV window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a vision pipeline: it produces no outputs.

Based on opencv-plot from dora-hub, brought up to the dora 1.0 node contract and packaged with uv.

Usage

Add to your dataflow:

nodes:
  - id: view
    git: ssh://git@gitea.liria.cl:5001/liria/dora-image-view.git
    tag: v0.1.0
    build: pip install .
    path: dora-image-view
    inputs:
      image: camera/image_bgr
      depth: camera/depth
      camera_info: camera/camera_info
      bbox: detector/bbox
      text: detector/text

Run the dataflow with --uv (dora run dataflow.yml --uv) so the node gets its own managed environment.

Every input is optional, and each of image and depth fills one panel of the window, tiled left to right in that order. Wiring one of them is a plain viewer; wiring several shows them side by side, scaled to a common height. bbox, text and camera_info are overlays rather than panels — boxes are drawn on the image panel, in its own pixels, and the caption on the composite.

End-to-end test

dataflow.yml is self-contained. tests/source.py is a synthetic source node that emits an animated frame, a depth map of the same scene, the camera intrinsics, the box around the square moving in the frame, and a caption — every input this node draws — so it needs no camera, no detector and no network. It is a test fixture rather than a worked example; the realistic wiring is the block at the top of this file. Run it to watch the node work:

dora run dataflow.yml --uv

A window opens with a square bouncing across a gradient, boxed and labelled. Press q to quit.

It builds the viewer from this checkout (path: + build:) rather than from a tag, which is what lets dora validate pick up the manifest described below.

CI runs the same dataflow headless on every push, which is what makes it a gate rather than a demo that rots.

Set SOURCE_FRAMES to stop the source after a fixed number of frames instead of running until you quit; that plus CI=true is how the same dataflow runs headless as a test:

CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv

SOURCE_WIDTH and SOURCE_HEIGHT (default 640x480) set the frame size.

Node manifest

dora-node.yml declares the node's inputs with type URNs. No source package is shared between node repositories, so it is the only contract a consumer has, and it is what makes a wiring mistake fail at build time instead of at runtime. The example source is typed inline in the dataflow with output_types:, so dora validate checks the two against each other:

$ dora validate dataflow.yml
  node "view": injected 4 contract type(s) from .../dora-node.yml
All type annotations OK.

There is one manifest because this repository ships one node. The example source is a test fixture rather than something anyone installs, so it has none, and runs through the interpreter to keep dora validate from matching the viewer's manifest against it.

image is std/core/v1/UInt8, depth and camera_info are std/core/v1/Float32 and text is std/core/v1/String; the payloads are raw buffers described by their metadata, so the URN names the element type rather than a struct. bbox is deliberately left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either.

Behavior

The node keeps a current frame, bbox set and text string. Each input updates its slice of that state:

• An image event decodes the frame and immediately redraws: it draws every stored bbox (green rectangle plus label, conf text), draws the stored text at the top left, resizes to VIEW_WIDTHxVIEW_HEIGHT if both are set, and shows it in a window titled Dora Node: dora-image-view. Press q in the window to quit.
• A depth event colour-maps the distances onto the depth panel and redraws.
• A camera_info event stores the pinhole model and draws it under the caption.
• A bbox or text event only updates stored state; it is drawn on the next redraw.

Setting CI=true suppresses the display: the node still decodes every input but opens no window. That is what makes it safe to run in a pipeline test.

Inputs

Input	Payload	Metadata
image	std/media/v1/Image struct	none — the struct describes itself
depth	std/media/v1/Image struct, one channel	units (defaults to mm)
camera_info	Flat Float32 array, [fx, fy, cx, cy] or a 9-element K	distortion, distortion_model
bbox	Arrow struct array, element [0] with fields bbox, conf, labels	format
text	Arrow array of size 1	—

image and depth are both std/media/v1/Image: an Arrow struct of width: UInt32, height: UInt32, encoding: Utf8 and data: LargeBinary. The payload describes itself, so nothing about its shape has to travel as message metadata.

Depth is an image whose pixels are distances — dora's type registry has no depth type, and this is how ROS spells it too. Its encodings are 32FC1/float32 and 16UC1/mono16/uint16, one channel per pixel, and units metadata still switches millimetres to metres.

Older payloads are still decoded, so a producer can move at its own pace: a flat UInt8 buffer with width/height/encoding metadata for image, and a flat Float32 buffer with shape/units metadata for depth.

Migrating a producer. dora's type compatibility is directional — Image satisfies a Bytes input, but a std/core/v1/UInt8 output wired into an std/media/v1/Image input is a dora validate error. A producer still declaring std/core/v1/UInt8 or std/core/v1/Float32 therefore has to change its manifest as well as its payload.

Supported image encoding values: bgr8, rgb8, jpeg/jpg/jpe/bmp/webp/png (encoded byte stream), yuv420, avif. width and height are only needed by the raw encodings — the rest carry their own dimensions. An unsupported encoding raises.

Supported format values: xyxy and xywh (centre-based, converted to xyxy).

depth is in millimetres unless units is m. In a depth map, 0 and any non-finite value mean missing depth and are drawn black. A payload sent as a raw byte slice — which is what a C++ node does — arrives as UInt8 despite the Float32 contract, and is reinterpreted rather than converted.

Camera intrinsics

camera_info carries the pinhole model the depth map is interpreted through, either as the four values [fx, fy, cx, cy] or as a nine-element row-major K, which is how ROS spells the same model in sensor_msgs/CameraInfo. Distortion coefficients are read from the distortion metadata.

The node displays the model and holds it. It needs none of it to draw a panel; it is there for what reads it next — turning a depth map into a point cloud needs fx, fy, cx and cy for every pixel, and nothing else in a dataflow carries them.

Intrinsics are also picked up from the metadata of any image or depth frame that carries fx, fy, cx and cy, so wiring camera_info is optional even for a camera that publishes it. That matters because a camera typically publishes camera_info once per session: a viewer started after the camera would otherwise never see it.

node.send_output("image", pa.array(frame.ravel()),
                 {"width": 640, "height": 480, "encoding": "bgr8"})

node.send_output("bbox", pa.array([{
    "bbox": np.array([...]),    # flattened, four values per box
    "conf": np.array([...]),
    "labels": np.array([...]),
}]), {"format": "xyxy"})

node.send_output("text", pa.array(["hello"]))

Outputs

None — it is a display sink.

Environment variables

Variable	Default	Meaning
VIEW_WIDTH	composite width	Resize the displayed composite to this width, in pixels.
VIEW_HEIGHT	composite height	Resize the displayed composite to this height, in pixels.
VIEW_DEPTH_MIN_MM	auto	Pin the near end of the depth colour range, in millimetres.
VIEW_DEPTH_MAX_MM	auto	Pin the far end of the depth colour range, in millimetres.

Resizing only happens when both VIEW_WIDTH and VIEW_HEIGHT are set. Both are also available as --view-width and --view-height; the environment variable wins when both are given.

Each end of the depth range is auto-ranged over the valid pixels of the current frame unless it is pinned, so pinning one end leaves the other automatic. Auto-ranging makes an unconfigured sensor legible at the cost of colours that shift from frame to frame; pin both ends to compare frames against each other. The range covers the full extent of the valid pixels rather than a percentile of them, so a small near object against a large wall stays visible.

CI=true suppresses the display window; it is not a configuration knob.

Build

uv sync

Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64, so the node installs with no system packages and no compiler on any of them — including AVIF support, which comes from pillow-avif-plugin rather than OpenCV.

Requirements: dora 1.0+ (verified with dora-cli 1.0.1), Python 3.11+, uv, and a display to show the window on (or CI=true to run without one). On a bare Linux image opencv-python also needs the libGL shared libraries (libgl1, libglib2.0-0), which it links at import time.

Test

uv run pytest

Unit tests over decoding, the bbox format conversion and the headless draw path; they need neither a camera, a dataflow nor a display. The headless dataflow run above covers the rest — the manifest, the wiring and the event loop.