AGENTS.md

@@ -26,9 +26,11 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv   # the same, headless
 - **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node.
 - **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. `unpack()` accepts two payload shapes — the `std/media/v1/Image` struct (`width`, `height`, `encoding`, `data`), and the flat `UInt8` buffer described by message metadata that a `std/core/v1/UInt8` producer sends. A struct describes itself, so its metadata is ignored; everything after unpacking is common.
 
-  **The declared type is still `std/core/v1/UInt8` and the reason is interop.** Type compatibility is directional: `Image` satisfies a `Bytes` input, but a `std/core/v1/UInt8` output wired into an `std/media/v1/Image` input is a validate error. dora-zed declares `image_bgr` as `std/core/v1/UInt8`, so flipping this manifest to `Image` breaks the camera until dora-zed is migrated. Accepting both shapes first is what makes that migration orderable: producers move, then the contract. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
+  `IMAGE_TYPE` is the one definition of that struct; the example source and the tests build payloads from it rather than respelling the schema.
+
+  **Interop cost of the contract being `Image`.** Type compatibility is directional: `Image` satisfies a `Bytes` input, but a `std/core/v1/UInt8` output wired into an `std/media/v1/Image` input is a validate error. A producer still declaring `std/core/v1/UInt8` — dora-zed's `image_bgr` at time of writing — has to change its manifest, not just its payload, before it can feed this node. The decoder accepting the flat form does not rescue it: the failure is at validate, before anything runs. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
 - **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one.
-- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. One thing is deliberate: the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
+- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. Depth arrives as `std/media/v1/Image` with a single-channel encoding (`DEPTH_DTYPES`) because dora's registry has no depth type and a depth map *is* an image of distances — the same reading ROS takes. `units` stays in message metadata: the struct has no field for it. One thing is deliberate: the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
 - **`dora_image_view/payload.py`** — `as_float32()`: reinterprets a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`. Shared by `depth` and `intrinsics`, which both meet it.
 - **`dora_image_view/intrinsics.py`** — `Intrinsics`, `parse()` for the `camera_info` payload and `from_metadata()` for the copy senders attach to each frame. Both exist because dora-zed publishes the same numbers twice, and the per-frame copy is the one that survives a viewer starting after the camera did — `camera_info` goes out **once per session**. A nine-element row-major K is accepted alongside the compact four values, since that is how ROS spells the same model. The node needs none of this to draw; it is held for whatever turns depth into 3D, and drawn so a wrong camera model is visible rather than merely plausible.
 - **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is.

README.md

@@ -106,25 +106,31 @@ no window. That is what makes it safe to run in a pipeline test.
 
 | Input | Payload | Metadata |
 |---|---|---|
-| `image` | `std/media/v1/Image` struct, **or** a flat `UInt8` array | none for the struct; `width`, `height`, `encoding` for the flat form |
-| `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` |
+| `image` | `std/media/v1/Image` struct | none — the struct describes itself |
+| `depth` | `std/media/v1/Image` struct, one channel | `units` (defaults to `mm`) |
 | `camera_info` | Flat `Float32` array, `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` |
 | `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
 | `text` | Arrow array of size 1 | — |
 
-Two image payload shapes are accepted. The typed one is `std/media/v1/Image`, an Arrow
-struct of `width: UInt32`, `height: UInt32`, `encoding: Utf8` and `data: LargeBinary`,
-which carries inside the payload what the flat form carries beside it; when it is
-received the message metadata is not consulted at all. The flat form is a raw `UInt8`
-buffer described by `width`, `height` and `encoding` metadata, which is what a node
-declaring `std/core/v1/UInt8` sends.
+`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`,
+`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes
+itself, so nothing about its shape has to travel as message metadata.
 
-The declared contract is still `std/core/v1/UInt8`, so a producer sending the struct
-should keep declaring that until every producer in the dataflow has moved: dora's type
-compatibility is directional, and a `std/core/v1/UInt8` output wired into an
-`std/media/v1/Image` input fails `dora validate`.
+Depth is an image whose pixels are distances — dora's type registry has no depth type,
+and this is how ROS spells it too. Its encodings are `32FC1`/`float32` and
+`16UC1`/`mono16`/`uint16`, one channel per pixel, and `units` metadata still switches
+millimetres to metres.
 
-Supported `encoding` values: `bgr8`, `rgb8`, `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/`png`
+Older payloads are still decoded, so a producer can move at its own pace: a flat `UInt8`
+buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32`
+buffer with `shape`/`units` metadata for `depth`.
+
+**Migrating a producer.** dora's type compatibility is directional — `Image` satisfies a
+`Bytes` input, but a `std/core/v1/UInt8` output wired into an `std/media/v1/Image` input
+is a `dora validate` error. A producer still declaring `std/core/v1/UInt8` or
+`std/core/v1/Float32` therefore has to change its manifest as well as its payload.
+
+Supported `image` `encoding` values: `bgr8`, `rgb8`, `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/`png`
 (encoded byte stream), `yuv420`, `avif`. `width` and `height` are only needed by the raw
 encodings — the rest carry their own dimensions. An unsupported encoding raises.
 

dataflow.yml

@@ -40,9 +40,9 @@ nodes:
     # bbox is deliberately absent: it is untyped on the viewer's side too,
     # because senders disagree on 32- vs 64-bit numeric fields.
     output_types:
-      image: std/core/v1/UInt8
+      image: std/media/v1/Image
       text: std/core/v1/String
-      depth: std/core/v1/Float32
+      depth: std/media/v1/Image
       camera_info: std/core/v1/Float32
 
   - id: view

dora-node.yml

@@ -21,13 +21,14 @@ dora: ">=1.0"
 
 inputs:
   image:
-    type: std/core/v1/UInt8
+    type: std/media/v1/Image
     required: false
     description: >-
-      Base image to draw on: a flat UInt8 buffer described by the metadata
-      {width, height, encoding}. Supported encodings: bgr8, rgb8,
-      jpeg/jpg/jpe/bmp/webp/png (encoded bytes), yuv420, avif. Receiving this
-      input triggers a redraw and display.
+      Base image to draw on: an Arrow struct {width, height, encoding, data}.
+      Supported encodings: bgr8, rgb8, jpeg/jpg/jpe/bmp/webp/png (encoded
+      bytes), yuv420, avif. Receiving this input triggers a redraw and display.
+      A flat UInt8 buffer described by {width, height, encoding} metadata is
+      still decoded, for producers that have not moved to the struct.
   bbox:
     required: false
     description: >-
@@ -53,13 +54,17 @@ inputs:
       this input -- they are picked up from there, which is what covers a
       camera that published its intrinsics before this node started.
   depth:
-    type: std/core/v1/Float32
+    type: std/media/v1/Image
     required: false
     description: >-
-      Depth map as a flat Float32 buffer, described by the metadata
-      {shape: [height, width], units}. Millimetres unless `units` says `m`;
-      0 and non-finite values are missing depth and draw black. Colour-mapped
-      into its own panel, which receiving this input redraws.
+      Depth map as an Arrow struct {width, height, encoding, data}, one channel
+      per pixel. Supported encodings: 32FC1/float32 and 16UC1/mono16/uint16 --
+      dora has no depth type, and a depth map is an image whose pixels are
+      distances, which is how ROS spells it too. Millimetres unless the `units`
+      metadata says `m`; 0 and non-finite values are missing depth and draw
+      black. Colour-mapped into its own panel, which receiving this input
+      redraws. A flat Float32 buffer with {shape, units} metadata is still
+      decoded, for producers that have not moved to the struct.
 
 outputs: {}
 

dora_image_view/decode.py

@@ -13,6 +13,7 @@ import io
 import cv2
 import numpy as np
 import pillow_avif  # noqa: F401  # registers the AVIF plugin with Pillow
+import pyarrow as pa
 from PIL import Image
 
 # Encodings that carry their own dimensions, so `width`/`height` metadata is not
@@ -22,9 +23,16 @@ SELF_DESCRIBING = frozenset({"jpeg", "jpg", "jpe", "bmp", "webp", "png"})
 SUPPORTED = SELF_DESCRIBING | {"bgr8", "rgb8", "yuv420", "avif"}
 
 
-# The fields of std/media/v1/Image. A payload carrying them describes itself and
-# its metadata is not consulted.
-IMAGE_FIELDS = ("width", "height", "encoding", "data")
+# std/media/v1/Image. A payload carrying these fields describes itself, so its
+# message metadata is not consulted. Exported as an Arrow type so a producer --
+# the example source, the tests -- builds it from one definition.
+IMAGE_TYPE = pa.struct([
+    ("width", pa.uint32()),
+    ("height", pa.uint32()),
+    ("encoding", pa.utf8()),
+    ("data", pa.large_binary()),
+])
+IMAGE_FIELDS = tuple(field.name for field in IMAGE_TYPE)
 
 
 def to_bgr(storage, metadata: dict) -> np.ndarray:
@@ -68,7 +76,7 @@ def to_bgr(storage, metadata: dict) -> np.ndarray:
 
 def unpack(storage, metadata: dict) -> tuple[np.ndarray, str, int | None, int | None]:
     """Split a payload of either shape into buffer, encoding, width and height."""
-    if _is_image_struct(storage):
+    if is_image_struct(storage):
         # One image per message: the struct is a single-row array.
         row = storage[0].as_py()
         return (
@@ -86,7 +94,7 @@ def unpack(storage, metadata: dict) -> tuple[np.ndarray, str, int | None, int |
     )
 
 
-def _is_image_struct(storage) -> bool:
+def is_image_struct(storage) -> bool:
     """Whether the payload is an Arrow struct with the std/media/v1/Image fields."""
     fields = getattr(getattr(storage, "type", None), "fields", None)
 

dora_image_view/depth.py

@@ -11,6 +11,7 @@ import os
 import cv2
 import numpy as np
 
+from dora_image_view.decode import is_image_struct
 from dora_image_view.payload import as_float32
 
 # Perceptually uniform and readable in both directions, unlike JET.
@@ -18,6 +19,18 @@ COLORMAP = cv2.COLORMAP_TURBO
 
 MISSING = 0.0
 
+# Single-channel depth encodings and the pixel type each names. The ROS spellings
+# are the ones senders use; a depth map is an image whose pixels are distances,
+# which is why it arrives as std/media/v1/Image rather than a type of its own --
+# dora's registry has no depth type.
+DEPTH_DTYPES = {
+    "32fc1": np.float32,
+    "float32": np.float32,
+    "16uc1": np.uint16,
+    "mono16": np.uint16,
+    "uint16": np.uint16,
+}
+
 
 def to_millimetres(storage, metadata: dict) -> np.ndarray:
     """Decode one `depth` payload into a float32 array of millimetres.
@@ -25,8 +38,7 @@ def to_millimetres(storage, metadata: dict) -> np.ndarray:
     Missing depth is 0. Non-finite values are normalised to 0 as well, so
     everything downstream only has to test for it.
     """
-    depth = as_float32(storage)
-    height, width = _hw(metadata, depth.size)
+    depth, height, width = _values(storage, metadata)
 
     if depth.size < height * width:
         raise RuntimeError(
@@ -87,6 +99,27 @@ def resolve_range(depth_mm: np.ndarray) -> tuple[float, float]:
     return float(low), float(high)
 
 
+def _values(storage, metadata: dict) -> tuple[np.ndarray, int, int]:
+    """Read the payload of either shape into float32 values plus its shape."""
+    if is_image_struct(storage):
+        row = storage[0].as_py()
+        encoding = str(row["encoding"]).lower()
+        dtype = DEPTH_DTYPES.get(encoding)
+
+        if dtype is None:
+            raise RuntimeError(
+                f"unsupported depth encoding: {encoding}; "
+                f"expected one of {', '.join(sorted(DEPTH_DTYPES))}",
+            )
+
+        values = np.frombuffer(row["data"], dtype=dtype).astype(np.float32)
+        return values, int(row["height"]), int(row["width"])
+
+    values = as_float32(storage)
+    height, width = _hw(metadata, values.size)
+    return values, height, width
+
+
 def _hw(metadata: dict, size: int) -> tuple[int, int]:
     shape = metadata.get("shape")
 

examples/source.py

@@ -16,6 +16,8 @@ import numpy as np
 import pyarrow as pa
 from dora import Node
 
+from dora_image_view.decode import IMAGE_TYPE
+
 WIDTH = int(os.getenv("SOURCE_WIDTH", "640"))
 HEIGHT = int(os.getenv("SOURCE_HEIGHT", "480"))
 # 0 runs until the viewer quits; a positive count is what lets CI run the
@@ -55,11 +57,7 @@ def main() -> None:
             )
 
         frame, corners = render(sent)
-        node.send_output(
-            "image",
-            pa.array(frame.ravel()),
-            {"width": WIDTH, "height": HEIGHT, "encoding": "bgr8", **INTRINSICS_METADATA},
-        )
+        node.send_output("image", image_struct(frame, "bgr8"), INTRINSICS_METADATA)
         node.send_output(
             "bbox",
             pa.array([{
@@ -72,14 +70,8 @@ def main() -> None:
         depth = render_depth(corners)
         node.send_output(
             "depth",
-            pa.array(depth.ravel()),
-            {
-                "shape": [HEIGHT, WIDTH],
-                "dtype": "float32",
-                "layout": "HW",
-                "units": "mm",
-                **INTRINSICS_METADATA,
-            },
+            image_struct(depth, "32FC1"),
+            {"units": "mm", **INTRINSICS_METADATA},
         )
         node.send_output("text", pa.array([CAPTION]))
 
@@ -88,6 +80,24 @@ def main() -> None:
             break
 
 
+def image_struct(frame: np.ndarray, encoding: str) -> pa.Array:
+    """Pack a frame as std/media/v1/Image.
+
+    The struct carries the width, height and encoding a receiver needs, so none
+    of it has to travel as message metadata. `frame` may be any dtype: the
+    encoding names how to read the bytes back.
+    """
+    return pa.array(
+        [{
+            "width": frame.shape[1],
+            "height": frame.shape[0],
+            "encoding": encoding,
+            "data": frame.tobytes(),
+        }],
+        type=IMAGE_TYPE,
+    )
+
+
 def render(index: int) -> tuple[np.ndarray, np.ndarray]:
     """Draw frame `index` and return it with the xyxy corners of the square in it."""
     frame = np.zeros((HEIGHT, WIDTH, 3), dtype=np.uint8)

tests/test_dora_image_view.py

@@ -298,12 +298,7 @@ def _image_struct(frame: np.ndarray, encoding: str = "bgr8"):
             "encoding": encoding,
             "data": frame.tobytes(),
         }],
-        type=pa.struct([
-            ("width", pa.uint32()),
-            ("height", pa.uint32()),
-            ("encoding", pa.utf8()),
-            ("data", pa.large_binary()),
-        ]),
+        type=decode.IMAGE_TYPE,
     )
 
 
@@ -336,3 +331,44 @@ def test_a_decoded_image_struct_frame_is_writable(frame):
     decoded[0, 0] = (1, 2, 3)
 
     assert tuple(decoded[0, 0]) == (1, 2, 3)
+
+
+def test_depth_is_decoded_from_an_image_struct():
+    values = np.array([[100.0, 200.0], [300.0, 400.0]], dtype=np.float32)
+
+    # No shape metadata: the struct carries its own dimensions.
+    decoded = depth.to_millimetres(_image_struct(values, "32FC1"), {})
+
+    np.testing.assert_allclose(decoded, values)
+
+
+def test_depth_image_struct_reads_a_uint16_encoding():
+    values = np.array([[1500, 2500]], dtype=np.uint16)
+
+    decoded = depth.to_millimetres(_image_struct(values, "16UC1"), {})
+
+    np.testing.assert_allclose(decoded, [[1500.0, 2500.0]])
+
+
+def test_depth_image_struct_still_honours_the_units_metadata():
+    values = np.array([[1.5, 2.0]], dtype=np.float32)
+
+    decoded = depth.to_millimetres(_image_struct(values, "32FC1"), {"units": "m"})
+
+    np.testing.assert_allclose(decoded, [[1500.0, 2000.0]])
+
+
+def test_depth_image_struct_with_an_unsupported_encoding_is_rejected():
+    values = np.zeros((1, 2), dtype=np.float32)
+
+    with pytest.raises(RuntimeError, match="unsupported depth encoding"):
+        depth.to_millimetres(_image_struct(values, "rgb8"), {})
+
+
+def test_a_decoded_depth_struct_is_writable():
+    values = np.array([[100.0, 0.0]], dtype=np.float32)
+
+    decoded = depth.to_millimetres(_image_struct(values, "32FC1"), {})
+    decoded[0, 0] = 42.0
+
+    assert decoded[0, 0] == 42.0