AGENTS.md

@@ -4,7 +4,7 @@ This file provides guidance to AI coding assistants when working with code in th
 
 ## What this is
 
-A dora-rs Python node that displays a video stream with overlays: it keeps a current frame, bbox set and text string, and every `image` event redraws all three into an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say.
+A dora-rs Python node that displays a video stream with overlays: it keeps one panel per frame stream (`image`, `depth`) plus a bbox set and text string, and every frame event redraws the lot as one side-by-side composite in an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say.
 
 Everything the example dataflow needs is in this checkout. Do not reintroduce a dependency on another repository — the example must keep running with no camera, no detector and no network.
 
@@ -25,9 +25,10 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv   # the same, headless
 
 - **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node.
 - **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
-- **`dora_image_view/view.py`** — `View` (the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded, which is what makes the node safe in a pipeline test.
+- **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one.
+- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. Two things are deliberate: `as_float32()` **reinterprets** a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`; and the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
 - **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is.
-- **`examples/source.py`** — The synthetic source node: an animated frame, the box around the square in it, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout.
+- **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout.
 - **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either.
 - **`examples/dora-node.yml`** — Manifest for the source node, and not optional. `dora validate` matches a manifest to a `path:` node by entrypoint; with no manifest beside `examples/source.py`, the root `dora-node.yml` is tried against it, does not match its entrypoint, and warns — which `strict_types: true` turns into a failure. Declaring the source's types here rather than as `output_types:` in `dataflow.yml` is also what makes them *checked*: an inline annotation shadows the manifest, so the manifest would be injected but never compared.
 - **`dataflow.yml`** — The example and the CI gate, with `strict_types: true`. Both nodes are `path:` nodes with a manifest, so both contracts are injected and compared against each other; `bbox` is annotated on neither side, matching the manifests.
@@ -41,7 +42,19 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv   # the same, headless
 
 ## A caveat on typed optional inputs
 
-`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped.
+`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map at all: `depth` is typed, so the example has to drive it.
+
+## Checking what it draws
+
+The gates prove the node does not crash, not that the picture is right — the colormap-ranging defect found while adding depth rendered as a plausible flat panel and passed every test. Compose a frame offline and look at it:
+
+```python
+view = View(headless=True)
+view.panels["depth"] = depth.colorize(depth.to_millimetres(storage, metadata))
+cv2.imwrite("/tmp/frame.png", view.compose())
+```
+
+`compose()` is public for exactly this: it returns the composite without needing a window.
 
 ## Rules
 

README.md

@@ -1,8 +1,8 @@
 # dora-image-view
 
-Dora node that overlays bounding boxes, labels and text on an image input and displays
-it in an OpenCV window. A display sink for debugging a vision pipeline: it produces no
-outputs.
+Dora node that displays a colour image and a depth map side by side in one OpenCV
+window, with bounding box and text overlays. A display sink for debugging a
+vision pipeline: it produces no outputs.
 
 Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
 from dora-hub, brought up to the dora 1.0 node contract and packaged with `uv`.
@@ -19,7 +19,8 @@ nodes:
     build: pip install .
     path: dora-image-view
     inputs:
-      image: camera/image
+      image: camera/image_bgr
+      depth: camera/depth
       bbox: detector/bbox
       text: detector/text
 ```
@@ -27,9 +28,11 @@ nodes:
 Run the dataflow with `--uv` (`dora run dataflow.yml --uv`) so the node gets its own
 managed environment.
 
-Wire `image` from any node publishing one of the encodings below, and `bbox` from any
-detector publishing the struct below. Both inputs are optional: an `image` alone is a
-plain viewer.
+Every input is optional, and each of `image` and `depth` fills one panel of the window,
+tiled left to right in that order. Wiring one of them is a plain viewer;
+wiring several shows them side by side, scaled to a common height. `bbox` and `text`
+are overlays rather than panels — boxes are drawn on the `image` panel, in its own
+pixels, and the caption on the composite.
 
 ### Runnable example
 
@@ -72,10 +75,11 @@ $ dora validate dataflow.yml
 All type annotations OK.
 ```
 
-`image` is `std/core/v1/UInt8` and `text` is `std/core/v1/String`; the payloads are raw
-buffers described by their metadata, so the URN names the element type rather than a
-struct. `bbox` is deliberately left untyped — senders disagree on whether its numeric
-fields are 32- or 64-bit, and the node reads either.
+`image` is `std/core/v1/UInt8`, `depth` is `std/core/v1/Float32` and `text` is
+`std/core/v1/String`; the payloads are raw buffers described by their
+metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
+left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
+node reads either.
 
 ## Behavior
 
@@ -86,8 +90,9 @@ of that state:
   bbox (green rectangle plus `label, conf` text), draws the stored text at the top left,
   resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
   `Dora Node: dora-image-view`. Press `q` in the window to quit.
+- A **`depth`** event colour-maps the distances onto the depth panel and redraws.
 - A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
-  `image` redraw.
+  redraw.
 
 Setting `CI=true` suppresses the display: the node still decodes every input but opens
 no window. That is what makes it safe to run in a pipeline test.
@@ -97,6 +102,7 @@ no window. That is what makes it safe to run in a pipeline test.
 | Input | Payload | Metadata |
 |---|---|---|
 | `image` | Flat `UInt8` array | `width`, `height`, `encoding` |
+| `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` |
 | `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
 | `text` | Arrow array of size 1 | — |
 
@@ -106,6 +112,11 @@ encodings — the rest carry their own dimensions. An unsupported encoding raise
 
 Supported `format` values: `xyxy` and `xywh` (centre-based, converted to `xyxy`).
 
+`depth` is in millimetres unless `units` is `m`. In a depth map, `0`
+and any non-finite value mean missing depth and are drawn black. A payload sent as a
+raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
+`Float32` contract, and is reinterpreted rather than converted.
+
 ```python
 node.send_output("image", pa.array(frame.ravel()),
                  {"width": 640, "height": 480, "encoding": "bgr8"})
@@ -127,11 +138,21 @@ None — it is a display sink.
 
 | Variable | Default | Meaning |
 |---|---|---|
-| `VIEW_WIDTH` | image width | Resize the displayed frame to this width, in pixels. |
-| `VIEW_HEIGHT` | image height | Resize the displayed frame to this height, in pixels. |
+| `VIEW_WIDTH` | composite width | Resize the displayed composite to this width, in pixels. |
+| `VIEW_HEIGHT` | composite height | Resize the displayed composite to this height, in pixels. |
+| `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. |
+| `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. |
 
-Resizing only happens when both are set. Both are also available as `--view-width` and
-`--view-height`; the environment variable wins when both are given.
+Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set. Both are also
+available as `--view-width` and `--view-height`; the environment variable wins when both
+are given.
+
+Each end of the depth range is auto-ranged over the valid pixels of the current frame
+unless it is pinned, so pinning one end leaves the other automatic. Auto-ranging makes
+an unconfigured sensor legible at the cost of colours that shift from frame to frame;
+pin both ends to compare frames against each other. The range covers the full extent of
+the valid pixels rather than a percentile of them, so a small near object against a
+large wall stays visible.
 
 `CI=true` suppresses the display window; it is not a configuration knob.
 

dataflow.yml

@@ -19,6 +19,7 @@ nodes:
       tick: dora/timer/millis/33
     outputs:
       - image
+      - depth
       - bbox
       - text
 
@@ -27,5 +28,6 @@ nodes:
     build: pip install .
     inputs:
       image: source/image
+      depth: source/depth
       bbox: source/bbox
       text: source/text

dora-node.yml

@@ -7,9 +7,9 @@
 apiVersion: 1
 name: dora-image-view
 namespace: liria
-description: Overlays bounding boxes, labels, and text on an image input and displays it in an OpenCV window
+description: Displays an image and a depth map side by side in an OpenCV window, with bounding box and text overlays
 categories: [visualization, debug]
-keywords: [opencv, view, bbox, bounding-box, image, visualization, object-detection]
+keywords: [opencv, view, bbox, bounding-box, image, depth, visualization, object-detection]
 
 runtime: python
 entrypoint: dora-image-view
@@ -41,10 +41,26 @@ inputs:
     description: >-
       Text string to overlay at the top-left corner. Stored and drawn on the
       next image redraw.
+  depth:
+    type: std/core/v1/Float32
+    required: false
+    description: >-
+      Depth map as a flat Float32 buffer, described by the metadata
+      {shape: [height, width], units}. Millimetres unless `units` says `m`;
+      0 and non-finite values are missing depth and draw black. Colour-mapped
+      into its own panel, which receiving this input redraws.
 
 outputs: {}
 
 env:
+  VIEW_DEPTH_MIN_MM:
+    type: float
+    description: >-
+      Pin the near end of the depth colour range, in millimetres. Each end is
+      auto-ranged over the valid pixels of the current frame when it is unset.
+  VIEW_DEPTH_MAX_MM:
+    type: float
+    description: Pin the far end of the depth colour range, in millimetres.
   VIEW_WIDTH:
     type: int
     description: Resize the displayed frame to this width (pixels). Defaults to the image input width.
@@ -59,6 +75,7 @@ example: |
     build: pip install .
     path: dora-image-view
     inputs:
-      image: camera/image
+      image: camera/image_bgr
+      depth: camera/depth
       bbox: detector/bbox
       text: detector/text

dora_image_view/depth.py

@@ -0,0 +1,126 @@
+"""Decoding and colouring of the `depth` input.
+
+A depth map is a single-channel distance image, not a picture: it is turned into
+something displayable by mapping a distance range onto a colormap. Decoding
+normalises to float32 millimetres with 0 for missing, so the colouring only ever
+sees one representation.
+"""
+
+import os
+
+import cv2
+import numpy as np
+
+# Perceptually uniform and readable in both directions, unlike JET.
+COLORMAP = cv2.COLORMAP_TURBO
+
+MISSING = 0.0
+
+
+def to_millimetres(storage, metadata: dict) -> np.ndarray:
+    """Decode one `depth` payload into a float32 array of millimetres.
+
+    Missing depth is 0. Non-finite values are normalised to 0 as well, so
+    everything downstream only has to test for it.
+    """
+    depth = as_float32(storage)
+    height, width = _hw(metadata, depth.size)
+
+    if depth.size < height * width:
+        raise RuntimeError(
+            f"depth payload holds {depth.size} values, "
+            f"expected at least {height * width} for the declared shape",
+        )
+
+    depth = depth[: height * width].reshape(height, width).copy()
+    depth[~np.isfinite(depth)] = MISSING
+
+    if units(metadata) == "m":
+        depth *= 1000.0
+
+    return depth
+
+
+def colorize(depth_mm: np.ndarray) -> np.ndarray:
+    """Map millimetres onto the colormap, leaving missing depth black."""
+    low, high = resolve_range(depth_mm)
+
+    normalised = np.clip((depth_mm - low) / (high - low), 0.0, 1.0)
+    frame = cv2.applyColorMap((normalised * 255).astype(np.uint8), COLORMAP)
+    frame[depth_mm <= MISSING] = 0
+    return frame
+
+
+def resolve_range(depth_mm: np.ndarray) -> tuple[float, float]:
+    """Pick the millimetre range to map onto the colormap.
+
+    Each end is taken from the environment if it is set there and auto-ranged
+    over the valid pixels otherwise, so pinning one end does not force the
+    other. Auto-ranging re-reads the frame every time, which is what makes an
+    unconfigured sensor legible and also what makes the colours shift between
+    frames.
+
+    The auto range is the full extent of the valid pixels, not a percentile of
+    them. Trimming the tails reads as the safer choice but breaks the scene
+    this viewer exists for: a small object in front of a large flat wall is a
+    few percent of the pixels, so both tails land on the wall, the range
+    collapses, and the object -- the thing being looked at -- clips to one
+    colour with everything else.
+    """
+    low = _env("VIEW_DEPTH_MIN_MM")
+    high = _env("VIEW_DEPTH_MAX_MM")
+
+    if low is None or high is None:
+        valid = depth_mm[depth_mm > MISSING]
+        auto_low, auto_high = (
+            (float(valid.min()), float(valid.max())) if valid.size else (0.0, 1.0)
+        )
+        low = auto_low if low is None else low
+        high = auto_high if high is None else high
+
+    # An empty or single-valued frame would otherwise divide by zero.
+    if high <= low:
+        high = low + 1.0
+
+    return float(low), float(high)
+
+
+def as_float32(storage) -> np.ndarray:
+    """Read the payload as float32, whether it arrived as floats or as bytes.
+
+    A node that sends the map as a raw byte slice -- dora-zed does -- produces a
+    UInt8 arrow array even though its manifest declares Float32, so the buffer
+    has to be reinterpreted rather than converted.
+    """
+    buffer = storage.to_numpy()
+
+    if buffer.dtype == np.uint8:
+        return buffer.view(np.float32)
+
+    return buffer.astype(np.float32, copy=False)
+
+
+def _hw(metadata: dict, size: int) -> tuple[int, int]:
+    shape = metadata.get("shape")
+
+    if shape is not None and len(shape) >= 2:
+        return int(shape[0]), int(shape[1])
+
+    height, width = metadata.get("height"), metadata.get("width")
+    if height and width:
+        return int(height), int(width)
+
+    raise RuntimeError(
+        "depth needs a `shape` [height, width] or `height`/`width` metadata; "
+        f"got a payload of {size} values and neither",
+    )
+
+
+def units(metadata: dict) -> str:
+    """Read the distance unit the sender declared, defaulting to millimetres."""
+    return str(metadata.get("units", "mm")).lower()
+
+
+def _env(name: str) -> float | None:
+    value = os.getenv(name)
+    return float(value) if value else None

dora_image_view/main.py

@@ -1,15 +1,16 @@
 """A dora display sink: overlays bounding boxes and text on an image input.
 
-An `image` event decodes the frame and immediately redraws. `bbox` and `text`
-events only update stored state, which the next `image` event draws.
+`image` and `depth` each fill one panel of the window and redraw. `bbox` and
+`text` events only update stored state, which the next redraw draws.
 """
 
 import argparse
 import os
 
+import numpy as np
 from dora import Node
 
-from dora_image_view import bbox, decode
+from dora_image_view import bbox, decode, depth
 from dora_image_view.view import View
 
 
@@ -36,14 +37,12 @@ def main() -> None:
         event_id = event["id"]
         metadata = event["metadata"]
 
-        if event_id == "image":
-            frame = decode.to_bgr(
-                event["value"],
-                metadata.get("encoding", "bgr8").lower(),
-                metadata.get("width"),
-                metadata.get("height"),
-            )
-            if view.show(frame):
+        panel = _decode_panel(event_id, event["value"], metadata)
+
+        if panel is not None:
+            name, frame = panel
+            view.panels[name] = frame
+            if view.show():
                 break
         elif event_id == "bbox":
             view.bboxes = bbox.parse(event["value"], metadata.get("format", "xyxy").lower())
@@ -53,9 +52,29 @@ def main() -> None:
             print(f"ignoring unexpected input `{event_id}`")
 
 
+def _decode_panel(event_id: str, value, metadata: dict) -> tuple[str, np.ndarray] | None:
+    """Decode a frame-bearing input into the panel name and frame it fills.
+
+    Returns None for an input that carries no frame, which is what tells the
+    event loop to treat it as overlay state instead.
+    """
+    if event_id == "image":
+        return "image", decode.to_bgr(
+            value,
+            metadata.get("encoding", "bgr8").lower(),
+            metadata.get("width"),
+            metadata.get("height"),
+        )
+
+    if event_id == "depth":
+        return "depth", depth.colorize(depth.to_millimetres(value, metadata))
+
+    return None
+
+
 def _parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(
-        description="OpenCV plotter: draws text and bounding boxes on an image.",
+        description="OpenCV viewer: image and depth, with bbox and text overlays.",
     )
     parser.add_argument(
         "--name",

dora_image_view/view.py

@@ -1,4 +1,4 @@
-"""The display state: the frame, the overlays drawn on it, and the window."""
+"""The display state: the panels, the overlays drawn on them, and the window."""
 
 from dataclasses import dataclass, field
 
@@ -11,6 +11,10 @@ WHITE = (255, 255, 255)
 FONT = cv2.FONT_HERSHEY_SIMPLEX
 FONT_SCALE = 0.5
 
+# Left to right in the composite. Fixed rather than insertion-ordered so a panel
+# does not move when a stream drops a frame.
+PANELS = ("image", "depth")
+
 
 @dataclass
 class Bboxes:
@@ -23,9 +27,11 @@ class Bboxes:
 
 @dataclass
 class View:
-    """Overlay state, plus the window it is displayed in.
+    """Panel state, plus the window the panels are displayed in.
 
-    `headless` suppresses the window; the node still decodes every input.
+    Each panel holds the most recent frame of one stream, as BGR. They are
+    tiled left to right, so wiring only `depth` shows only depth. `headless`
+    suppresses the window; the node still decodes every input.
     """
 
     width: int | None = None
@@ -33,13 +39,17 @@ class View:
     headless: bool = False
     bboxes: Bboxes = field(default_factory=Bboxes)
     text: str = ""
+    panels: dict[str, np.ndarray] = field(default_factory=dict)
 
-    def show(self, frame: np.ndarray) -> bool:
-        """Draw the overlays on `frame`, resize it, and display it.
+    def show(self) -> bool:
+        """Draw the overlays, tile the panels, and display them.
 
         Returns whether the viewer asked to quit by pressing `q`.
         """
-        self._draw(frame)
+        frame = self.compose()
+
+        if frame is None:
+            return False
 
         if self.width and self.height:
             frame = cv2.resize(frame, (self.width, self.height))
@@ -50,7 +60,42 @@ class View:
         cv2.imshow(WINDOW, frame)
         return cv2.waitKey(1) & 0xFF == ord("q")
 
-    def _draw(self, frame: np.ndarray) -> None:
+    def compose(self) -> np.ndarray | None:
+        """Tile the populated panels into one frame, tallest panel setting the height."""
+        panels = [(name, self.panels[name]) for name in PANELS if name in self.panels]
+
+        if not panels:
+            return None
+
+        # The boxes are in image pixels, so they are drawn before any scaling.
+        if "image" in self.panels:
+            self._draw_bboxes(self.panels["image"])
+
+        tallest = max(panel.shape[0] for _, panel in panels)
+        scaled = [self._label(_scale(panel, tallest), name) for name, panel in panels]
+
+        frame = scaled[0] if len(scaled) == 1 else cv2.hconcat(scaled)
+
+        # On the composite, not on a panel: the caption describes the frame as a
+        # whole, and a panel it was drawn on might not be the leftmost one.
+        cv2.putText(frame, self.text, (20, 20), FONT, FONT_SCALE, WHITE, 1)
+        return frame
+
+    def _label(self, panel: np.ndarray, name: str) -> np.ndarray:
+        """Name a panel, so two side by side are not read as one picture."""
+        if len(self.panels) > 1:
+            cv2.putText(
+                panel,
+                name,
+                (10, panel.shape[0] - 10),
+                FONT,
+                FONT_SCALE,
+                WHITE,
+                1,
+            )
+        return panel
+
+    def _draw_bboxes(self, frame: np.ndarray) -> None:
         for corners, conf, label in zip(
             self.bboxes.corners,
             self.bboxes.conf,
@@ -69,4 +114,11 @@ class View:
                 1,
             )
 
-        cv2.putText(frame, self.text, (20, 20), FONT, FONT_SCALE, WHITE, 1)
+
+def _scale(panel: np.ndarray, height: int) -> np.ndarray:
+    """Match a panel to the composite height, keeping its aspect ratio."""
+    if panel.shape[0] == height:
+        return panel
+
+    width = max(round(panel.shape[1] * height / panel.shape[0]), 1)
+    return cv2.resize(panel, (width, height))

examples/dora-node.yml

@@ -12,7 +12,7 @@
 apiVersion: 1
 name: source
 namespace: liria
-description: Synthetic image, bbox and text source for the dora-image-view example
+description: Synthetic image, depth, bbox and text source for the dora-image-view example
 categories: [debug]
 keywords: [example, test, synthetic]
 
@@ -39,6 +39,12 @@ outputs:
   text:
     type: std/core/v1/String
     description: A fixed caption.
+  depth:
+    type: std/core/v1/Float32
+    description: >-
+      Depth of the same scene in millimetres, [height, width]: a flat wall with
+      the square standing in front of it.
+
 
 env:
   SOURCE_WIDTH:

examples/source.py

@@ -1,8 +1,11 @@
 """Synthetic source node: this repository's stand-in for a camera and a detector.
 
-Emits an animated frame, the box around the square moving in it, and a caption --
-the three inputs dora-image-view draws -- so the example dataflow needs no camera, no
-detector and no network.
+Emits an animated frame, a depth map of the same scene, the box around the square
+moving in it, and a caption -- every input dora-image-view draws -- so the example
+dataflow needs no camera, no detector and no network.
+
+The depth map describes the same scene as the image: a flat back wall with the square
+standing in front of it.
 """
 
 import os
@@ -21,9 +24,12 @@ BOX = 96
 SPEED = 6
 CAPTION = "dora-image-view synthetic source"
 
+WALL_MM = 2000.0
+SQUARE_MM = 800.0
+
 
 def main() -> None:
-    """Send one image, bbox and text per tick."""
+    """Send one image, depth map, bbox and text per tick."""
     node = Node("source")
     sent = 0
 
@@ -46,6 +52,12 @@ def main() -> None:
             }]),
             {"format": "xyxy"},
         )
+        depth = render_depth(corners)
+        node.send_output(
+            "depth",
+            pa.array(depth.ravel()),
+            {"shape": [HEIGHT, WIDTH], "dtype": "float32", "layout": "HW", "units": "mm"},
+        )
         node.send_output("text", pa.array([CAPTION]))
 
         sent += 1
@@ -69,5 +81,13 @@ def render(index: int) -> tuple[np.ndarray, np.ndarray]:
     return frame, corners
 
 
+def render_depth(corners: np.ndarray) -> np.ndarray:
+    """Depth of the scene in `render`: a flat wall with the square in front of it."""
+    depth = np.full((HEIGHT, WIDTH), WALL_MM, dtype=np.float32)
+    left, top, right, bottom = (int(value) for value in corners)
+    depth[top:bottom, left:right] = SQUARE_MM
+    return depth
+
+
 if __name__ == "__main__":
     main()

tests/test_dora_image_view.py

@@ -7,7 +7,7 @@ import numpy as np
 import pytest
 from PIL import Image
 
-from dora_image_view import bbox, decode
+from dora_image_view import bbox, decode, depth
 from dora_image_view.view import Bboxes, View
 
 WIDTH, HEIGHT = 12, 8
@@ -130,6 +130,97 @@ def test_headless_show_draws_without_a_window():
         labels=np.array(["cat"]),
     )
     frame = np.zeros((32, 32, 3), dtype=np.uint8)
+    view.panels["image"] = frame
 
-    assert view.show(frame) is False
+    assert view.show() is False
     assert frame.any(), "the overlays should have marked the frame"
+
+
+def test_show_with_no_panels_is_a_no_op():
+    assert View(headless=True).show() is False
+
+
+def test_panels_are_tiled_left_to_right_at_a_common_height():
+    view = View(headless=True)
+    view.panels["image"] = np.zeros((20, 10, 3), dtype=np.uint8)
+    # Half the height and half the width: scaling to the common height should
+    # double both, so it contributes 20 columns, not 10.
+    view.panels["depth"] = np.zeros((10, 10, 3), dtype=np.uint8)
+
+    composite = view.compose()
+
+    assert composite.shape[0] == 20
+    assert composite.shape[1] == 10 + 20
+
+
+def test_depth_decodes_millimetres_and_reshapes():
+    values = np.array([100.0, 200.0, 300.0, 400.0], dtype=np.float32)
+
+    decoded = depth.to_millimetres(FakeStorage(values), {"shape": [2, 2]})
+
+    np.testing.assert_allclose(decoded, [[100.0, 200.0], [300.0, 400.0]])
+
+
+def test_depth_reinterprets_a_float32_payload_sent_as_bytes():
+    values = np.array([1500.0, 2500.0], dtype=np.float32)
+
+    decoded = depth.to_millimetres(
+        FakeStorage(values.view(np.uint8)),
+        {"shape": [1, 2], "dtype": "float32"},
+    )
+
+    np.testing.assert_allclose(decoded, [[1500.0, 2500.0]])
+
+
+def test_depth_converts_metres_to_millimetres():
+    decoded = depth.to_millimetres(
+        FakeStorage(np.array([1.5, 2.0], dtype=np.float32)),
+        {"shape": [1, 2], "units": "m"},
+    )
+
+    np.testing.assert_allclose(decoded, [[1500.0, 2000.0]])
+
+
+def test_depth_normalises_non_finite_values_to_missing():
+    values = np.array([np.inf, np.nan, 500.0, 0.0], dtype=np.float32)
+
+    decoded = depth.to_millimetres(FakeStorage(values), {"shape": [2, 2]})
+
+    np.testing.assert_allclose(decoded, [[0.0, 0.0], [500.0, 0.0]])
+
+
+def test_depth_without_a_shape_is_an_error():
+    with pytest.raises(RuntimeError, match="shape"):
+        depth.to_millimetres(FakeStorage(np.zeros(4, dtype=np.float32)), {})
+
+
+def test_colorize_leaves_missing_depth_black_and_colours_the_rest():
+    depth_mm = np.array([[0.0, 500.0], [1000.0, 1500.0]], dtype=np.float32)
+
+    frame = depth.colorize(depth_mm)
+
+    assert frame.shape == (2, 2, 3)
+    np.testing.assert_array_equal(frame[0, 0], [0, 0, 0])
+    assert frame[1, 1].any(), "valid depth should be coloured"
+
+
+def test_depth_range_is_pinned_by_the_environment(monkeypatch):
+    monkeypatch.setenv("VIEW_DEPTH_MIN_MM", "200")
+    monkeypatch.setenv("VIEW_DEPTH_MAX_MM", "600")
+
+    assert depth.resolve_range(np.array([[1000.0]], dtype=np.float32)) == (200.0, 600.0)
+
+
+def test_one_pinned_end_leaves_the_other_auto_ranged(monkeypatch):
+    monkeypatch.setenv("VIEW_DEPTH_MIN_MM", "0")
+
+    low, high = depth.resolve_range(np.array([[100.0, 900.0]], dtype=np.float32))
+
+    assert low == 0.0
+    assert high > 0.0
+
+
+def test_depth_range_of_an_empty_frame_does_not_divide_by_zero():
+    low, high = depth.resolve_range(np.zeros((2, 2), dtype=np.float32))
+
+    assert high > low