diff --git a/AGENTS.md b/AGENTS.md index d2ff4b2..17ac4f7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ This file provides guidance to AI coding assistants when working with code in th ## What this is -A dora-rs Python node that displays a video stream with overlays: it keeps a current frame, bbox set and text string, and every `image` event redraws all three into an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say. +A dora-rs Python node that displays a video stream with overlays: it keeps one panel per frame stream (`image`, `depth`) plus a bbox set and text string, and every frame event redraws the lot as one side-by-side composite in an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say. Everything the example dataflow needs is in this checkout. Do not reintroduce a dependency on another repository — the example must keep running with no camera, no detector and no network. @@ -25,9 +25,10 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless - **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node. - **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see. -- **`dora_image_view/view.py`** — `View` (the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded, which is what makes the node safe in a pipeline test. +- **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one. +- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. Two things are deliberate: `as_float32()` **reinterprets** a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`; and the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging. - **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is. -- **`examples/source.py`** — The synthetic source node: an animated frame, the box around the square in it, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout. +- **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout. - **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either. - **`examples/dora-node.yml`** — Manifest for the source node, and not optional. `dora validate` matches a manifest to a `path:` node by entrypoint; with no manifest beside `examples/source.py`, the root `dora-node.yml` is tried against it, does not match its entrypoint, and warns — which `strict_types: true` turns into a failure. Declaring the source's types here rather than as `output_types:` in `dataflow.yml` is also what makes them *checked*: an inline annotation shadows the manifest, so the manifest would be injected but never compared. - **`dataflow.yml`** — The example and the CI gate, with `strict_types: true`. Both nodes are `path:` nodes with a manifest, so both contracts are injected and compared against each other; `bbox` is annotated on neither side, matching the manifests. @@ -41,7 +42,19 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless ## A caveat on typed optional inputs -`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. +`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map at all: `depth` is typed, so the example has to drive it. + +## Checking what it draws + +The gates prove the node does not crash, not that the picture is right — the colormap-ranging defect found while adding depth rendered as a plausible flat panel and passed every test. Compose a frame offline and look at it: + +```python +view = View(headless=True) +view.panels["depth"] = depth.colorize(depth.to_millimetres(storage, metadata)) +cv2.imwrite("/tmp/frame.png", view.compose()) +``` + +`compose()` is public for exactly this: it returns the composite without needing a window. ## Rules diff --git a/README.md b/README.md index f16d00e..34f9fee 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # dora-image-view -Dora node that overlays bounding boxes, labels and text on an image input and displays -it in an OpenCV window. A display sink for debugging a vision pipeline: it produces no -outputs. +Dora node that displays a colour image and a depth map side by side in one OpenCV +window, with bounding box and text overlays. A display sink for debugging a +vision pipeline: it produces no outputs. Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot) from dora-hub, brought up to the dora 1.0 node contract and packaged with `uv`. @@ -19,7 +19,8 @@ nodes: build: pip install . path: dora-image-view inputs: - image: camera/image + image: camera/image_bgr + depth: camera/depth bbox: detector/bbox text: detector/text ``` @@ -27,9 +28,11 @@ nodes: Run the dataflow with `--uv` (`dora run dataflow.yml --uv`) so the node gets its own managed environment. -Wire `image` from any node publishing one of the encodings below, and `bbox` from any -detector publishing the struct below. Both inputs are optional: an `image` alone is a -plain viewer. +Every input is optional, and each of `image` and `depth` fills one panel of the window, +tiled left to right in that order. Wiring one of them is a plain viewer; +wiring several shows them side by side, scaled to a common height. `bbox` and `text` +are overlays rather than panels — boxes are drawn on the `image` panel, in its own +pixels, and the caption on the composite. ### Runnable example @@ -72,10 +75,11 @@ $ dora validate dataflow.yml All type annotations OK. ``` -`image` is `std/core/v1/UInt8` and `text` is `std/core/v1/String`; the payloads are raw -buffers described by their metadata, so the URN names the element type rather than a -struct. `bbox` is deliberately left untyped — senders disagree on whether its numeric -fields are 32- or 64-bit, and the node reads either. +`image` is `std/core/v1/UInt8`, `depth` is `std/core/v1/Float32` and `text` is +`std/core/v1/String`; the payloads are raw buffers described by their +metadata, so the URN names the element type rather than a struct. `bbox` is deliberately +left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the +node reads either. ## Behavior @@ -86,8 +90,9 @@ of that state: bbox (green rectangle plus `label, conf` text), draws the stored text at the top left, resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled `Dora Node: dora-image-view`. Press `q` in the window to quit. +- A **`depth`** event colour-maps the distances onto the depth panel and redraws. - A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next - `image` redraw. + redraw. Setting `CI=true` suppresses the display: the node still decodes every input but opens no window. That is what makes it safe to run in a pipeline test. @@ -97,6 +102,7 @@ no window. That is what makes it safe to run in a pipeline test. | Input | Payload | Metadata | |---|---|---| | `image` | Flat `UInt8` array | `width`, `height`, `encoding` | +| `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` | | `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` | | `text` | Arrow array of size 1 | — | @@ -106,6 +112,11 @@ encodings — the rest carry their own dimensions. An unsupported encoding raise Supported `format` values: `xyxy` and `xywh` (centre-based, converted to `xyxy`). +`depth` is in millimetres unless `units` is `m`. In a depth map, `0` +and any non-finite value mean missing depth and are drawn black. A payload sent as a +raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the +`Float32` contract, and is reinterpreted rather than converted. + ```python node.send_output("image", pa.array(frame.ravel()), {"width": 640, "height": 480, "encoding": "bgr8"}) @@ -127,11 +138,21 @@ None — it is a display sink. | Variable | Default | Meaning | |---|---|---| -| `VIEW_WIDTH` | image width | Resize the displayed frame to this width, in pixels. | -| `VIEW_HEIGHT` | image height | Resize the displayed frame to this height, in pixels. | +| `VIEW_WIDTH` | composite width | Resize the displayed composite to this width, in pixels. | +| `VIEW_HEIGHT` | composite height | Resize the displayed composite to this height, in pixels. | +| `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. | +| `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. | -Resizing only happens when both are set. Both are also available as `--view-width` and -`--view-height`; the environment variable wins when both are given. +Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set. Both are also +available as `--view-width` and `--view-height`; the environment variable wins when both +are given. + +Each end of the depth range is auto-ranged over the valid pixels of the current frame +unless it is pinned, so pinning one end leaves the other automatic. Auto-ranging makes +an unconfigured sensor legible at the cost of colours that shift from frame to frame; +pin both ends to compare frames against each other. The range covers the full extent of +the valid pixels rather than a percentile of them, so a small near object against a +large wall stays visible. `CI=true` suppresses the display window; it is not a configuration knob. diff --git a/dataflow.yml b/dataflow.yml index cbded38..22731c8 100644 --- a/dataflow.yml +++ b/dataflow.yml @@ -19,6 +19,7 @@ nodes: tick: dora/timer/millis/33 outputs: - image + - depth - bbox - text @@ -27,5 +28,6 @@ nodes: build: pip install . inputs: image: source/image + depth: source/depth bbox: source/bbox text: source/text diff --git a/dora-node.yml b/dora-node.yml index 67ae067..f9dc7bf 100644 --- a/dora-node.yml +++ b/dora-node.yml @@ -7,9 +7,9 @@ apiVersion: 1 name: dora-image-view namespace: liria -description: Overlays bounding boxes, labels, and text on an image input and displays it in an OpenCV window +description: Displays an image and a depth map side by side in an OpenCV window, with bounding box and text overlays categories: [visualization, debug] -keywords: [opencv, view, bbox, bounding-box, image, visualization, object-detection] +keywords: [opencv, view, bbox, bounding-box, image, depth, visualization, object-detection] runtime: python entrypoint: dora-image-view @@ -41,10 +41,26 @@ inputs: description: >- Text string to overlay at the top-left corner. Stored and drawn on the next image redraw. + depth: + type: std/core/v1/Float32 + required: false + description: >- + Depth map as a flat Float32 buffer, described by the metadata + {shape: [height, width], units}. Millimetres unless `units` says `m`; + 0 and non-finite values are missing depth and draw black. Colour-mapped + into its own panel, which receiving this input redraws. outputs: {} env: + VIEW_DEPTH_MIN_MM: + type: float + description: >- + Pin the near end of the depth colour range, in millimetres. Each end is + auto-ranged over the valid pixels of the current frame when it is unset. + VIEW_DEPTH_MAX_MM: + type: float + description: Pin the far end of the depth colour range, in millimetres. VIEW_WIDTH: type: int description: Resize the displayed frame to this width (pixels). Defaults to the image input width. @@ -59,6 +75,7 @@ example: | build: pip install . path: dora-image-view inputs: - image: camera/image + image: camera/image_bgr + depth: camera/depth bbox: detector/bbox text: detector/text diff --git a/dora_image_view/depth.py b/dora_image_view/depth.py new file mode 100644 index 0000000..86733ce --- /dev/null +++ b/dora_image_view/depth.py @@ -0,0 +1,126 @@ +"""Decoding and colouring of the `depth` input. + +A depth map is a single-channel distance image, not a picture: it is turned into +something displayable by mapping a distance range onto a colormap. Decoding +normalises to float32 millimetres with 0 for missing, so the colouring only ever +sees one representation. +""" + +import os + +import cv2 +import numpy as np + +# Perceptually uniform and readable in both directions, unlike JET. +COLORMAP = cv2.COLORMAP_TURBO + +MISSING = 0.0 + + +def to_millimetres(storage, metadata: dict) -> np.ndarray: + """Decode one `depth` payload into a float32 array of millimetres. + + Missing depth is 0. Non-finite values are normalised to 0 as well, so + everything downstream only has to test for it. + """ + depth = as_float32(storage) + height, width = _hw(metadata, depth.size) + + if depth.size < height * width: + raise RuntimeError( + f"depth payload holds {depth.size} values, " + f"expected at least {height * width} for the declared shape", + ) + + depth = depth[: height * width].reshape(height, width).copy() + depth[~np.isfinite(depth)] = MISSING + + if units(metadata) == "m": + depth *= 1000.0 + + return depth + + +def colorize(depth_mm: np.ndarray) -> np.ndarray: + """Map millimetres onto the colormap, leaving missing depth black.""" + low, high = resolve_range(depth_mm) + + normalised = np.clip((depth_mm - low) / (high - low), 0.0, 1.0) + frame = cv2.applyColorMap((normalised * 255).astype(np.uint8), COLORMAP) + frame[depth_mm <= MISSING] = 0 + return frame + + +def resolve_range(depth_mm: np.ndarray) -> tuple[float, float]: + """Pick the millimetre range to map onto the colormap. + + Each end is taken from the environment if it is set there and auto-ranged + over the valid pixels otherwise, so pinning one end does not force the + other. Auto-ranging re-reads the frame every time, which is what makes an + unconfigured sensor legible and also what makes the colours shift between + frames. + + The auto range is the full extent of the valid pixels, not a percentile of + them. Trimming the tails reads as the safer choice but breaks the scene + this viewer exists for: a small object in front of a large flat wall is a + few percent of the pixels, so both tails land on the wall, the range + collapses, and the object -- the thing being looked at -- clips to one + colour with everything else. + """ + low = _env("VIEW_DEPTH_MIN_MM") + high = _env("VIEW_DEPTH_MAX_MM") + + if low is None or high is None: + valid = depth_mm[depth_mm > MISSING] + auto_low, auto_high = ( + (float(valid.min()), float(valid.max())) if valid.size else (0.0, 1.0) + ) + low = auto_low if low is None else low + high = auto_high if high is None else high + + # An empty or single-valued frame would otherwise divide by zero. + if high <= low: + high = low + 1.0 + + return float(low), float(high) + + +def as_float32(storage) -> np.ndarray: + """Read the payload as float32, whether it arrived as floats or as bytes. + + A node that sends the map as a raw byte slice -- dora-zed does -- produces a + UInt8 arrow array even though its manifest declares Float32, so the buffer + has to be reinterpreted rather than converted. + """ + buffer = storage.to_numpy() + + if buffer.dtype == np.uint8: + return buffer.view(np.float32) + + return buffer.astype(np.float32, copy=False) + + +def _hw(metadata: dict, size: int) -> tuple[int, int]: + shape = metadata.get("shape") + + if shape is not None and len(shape) >= 2: + return int(shape[0]), int(shape[1]) + + height, width = metadata.get("height"), metadata.get("width") + if height and width: + return int(height), int(width) + + raise RuntimeError( + "depth needs a `shape` [height, width] or `height`/`width` metadata; " + f"got a payload of {size} values and neither", + ) + + +def units(metadata: dict) -> str: + """Read the distance unit the sender declared, defaulting to millimetres.""" + return str(metadata.get("units", "mm")).lower() + + +def _env(name: str) -> float | None: + value = os.getenv(name) + return float(value) if value else None diff --git a/dora_image_view/main.py b/dora_image_view/main.py index dbc984f..fb5f0c9 100644 --- a/dora_image_view/main.py +++ b/dora_image_view/main.py @@ -1,15 +1,16 @@ """A dora display sink: overlays bounding boxes and text on an image input. -An `image` event decodes the frame and immediately redraws. `bbox` and `text` -events only update stored state, which the next `image` event draws. +`image` and `depth` each fill one panel of the window and redraw. `bbox` and +`text` events only update stored state, which the next redraw draws. """ import argparse import os +import numpy as np from dora import Node -from dora_image_view import bbox, decode +from dora_image_view import bbox, decode, depth from dora_image_view.view import View @@ -36,14 +37,12 @@ def main() -> None: event_id = event["id"] metadata = event["metadata"] - if event_id == "image": - frame = decode.to_bgr( - event["value"], - metadata.get("encoding", "bgr8").lower(), - metadata.get("width"), - metadata.get("height"), - ) - if view.show(frame): + panel = _decode_panel(event_id, event["value"], metadata) + + if panel is not None: + name, frame = panel + view.panels[name] = frame + if view.show(): break elif event_id == "bbox": view.bboxes = bbox.parse(event["value"], metadata.get("format", "xyxy").lower()) @@ -53,9 +52,29 @@ def main() -> None: print(f"ignoring unexpected input `{event_id}`") +def _decode_panel(event_id: str, value, metadata: dict) -> tuple[str, np.ndarray] | None: + """Decode a frame-bearing input into the panel name and frame it fills. + + Returns None for an input that carries no frame, which is what tells the + event loop to treat it as overlay state instead. + """ + if event_id == "image": + return "image", decode.to_bgr( + value, + metadata.get("encoding", "bgr8").lower(), + metadata.get("width"), + metadata.get("height"), + ) + + if event_id == "depth": + return "depth", depth.colorize(depth.to_millimetres(value, metadata)) + + return None + + def _parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( - description="OpenCV plotter: draws text and bounding boxes on an image.", + description="OpenCV viewer: image and depth, with bbox and text overlays.", ) parser.add_argument( "--name", diff --git a/dora_image_view/view.py b/dora_image_view/view.py index dd66d14..bfed187 100644 --- a/dora_image_view/view.py +++ b/dora_image_view/view.py @@ -1,4 +1,4 @@ -"""The display state: the frame, the overlays drawn on it, and the window.""" +"""The display state: the panels, the overlays drawn on them, and the window.""" from dataclasses import dataclass, field @@ -11,6 +11,10 @@ WHITE = (255, 255, 255) FONT = cv2.FONT_HERSHEY_SIMPLEX FONT_SCALE = 0.5 +# Left to right in the composite. Fixed rather than insertion-ordered so a panel +# does not move when a stream drops a frame. +PANELS = ("image", "depth") + @dataclass class Bboxes: @@ -23,9 +27,11 @@ class Bboxes: @dataclass class View: - """Overlay state, plus the window it is displayed in. + """Panel state, plus the window the panels are displayed in. - `headless` suppresses the window; the node still decodes every input. + Each panel holds the most recent frame of one stream, as BGR. They are + tiled left to right, so wiring only `depth` shows only depth. `headless` + suppresses the window; the node still decodes every input. """ width: int | None = None @@ -33,13 +39,17 @@ class View: headless: bool = False bboxes: Bboxes = field(default_factory=Bboxes) text: str = "" + panels: dict[str, np.ndarray] = field(default_factory=dict) - def show(self, frame: np.ndarray) -> bool: - """Draw the overlays on `frame`, resize it, and display it. + def show(self) -> bool: + """Draw the overlays, tile the panels, and display them. Returns whether the viewer asked to quit by pressing `q`. """ - self._draw(frame) + frame = self.compose() + + if frame is None: + return False if self.width and self.height: frame = cv2.resize(frame, (self.width, self.height)) @@ -50,7 +60,42 @@ class View: cv2.imshow(WINDOW, frame) return cv2.waitKey(1) & 0xFF == ord("q") - def _draw(self, frame: np.ndarray) -> None: + def compose(self) -> np.ndarray | None: + """Tile the populated panels into one frame, tallest panel setting the height.""" + panels = [(name, self.panels[name]) for name in PANELS if name in self.panels] + + if not panels: + return None + + # The boxes are in image pixels, so they are drawn before any scaling. + if "image" in self.panels: + self._draw_bboxes(self.panels["image"]) + + tallest = max(panel.shape[0] for _, panel in panels) + scaled = [self._label(_scale(panel, tallest), name) for name, panel in panels] + + frame = scaled[0] if len(scaled) == 1 else cv2.hconcat(scaled) + + # On the composite, not on a panel: the caption describes the frame as a + # whole, and a panel it was drawn on might not be the leftmost one. + cv2.putText(frame, self.text, (20, 20), FONT, FONT_SCALE, WHITE, 1) + return frame + + def _label(self, panel: np.ndarray, name: str) -> np.ndarray: + """Name a panel, so two side by side are not read as one picture.""" + if len(self.panels) > 1: + cv2.putText( + panel, + name, + (10, panel.shape[0] - 10), + FONT, + FONT_SCALE, + WHITE, + 1, + ) + return panel + + def _draw_bboxes(self, frame: np.ndarray) -> None: for corners, conf, label in zip( self.bboxes.corners, self.bboxes.conf, @@ -69,4 +114,11 @@ class View: 1, ) - cv2.putText(frame, self.text, (20, 20), FONT, FONT_SCALE, WHITE, 1) + +def _scale(panel: np.ndarray, height: int) -> np.ndarray: + """Match a panel to the composite height, keeping its aspect ratio.""" + if panel.shape[0] == height: + return panel + + width = max(round(panel.shape[1] * height / panel.shape[0]), 1) + return cv2.resize(panel, (width, height)) diff --git a/examples/dora-node.yml b/examples/dora-node.yml index 689e872..93439cf 100644 --- a/examples/dora-node.yml +++ b/examples/dora-node.yml @@ -12,7 +12,7 @@ apiVersion: 1 name: source namespace: liria -description: Synthetic image, bbox and text source for the dora-image-view example +description: Synthetic image, depth, bbox and text source for the dora-image-view example categories: [debug] keywords: [example, test, synthetic] @@ -39,6 +39,12 @@ outputs: text: type: std/core/v1/String description: A fixed caption. + depth: + type: std/core/v1/Float32 + description: >- + Depth of the same scene in millimetres, [height, width]: a flat wall with + the square standing in front of it. + env: SOURCE_WIDTH: diff --git a/examples/source.py b/examples/source.py index 6a8f799..704b741 100644 --- a/examples/source.py +++ b/examples/source.py @@ -1,8 +1,11 @@ """Synthetic source node: this repository's stand-in for a camera and a detector. -Emits an animated frame, the box around the square moving in it, and a caption -- -the three inputs dora-image-view draws -- so the example dataflow needs no camera, no -detector and no network. +Emits an animated frame, a depth map of the same scene, the box around the square +moving in it, and a caption -- every input dora-image-view draws -- so the example +dataflow needs no camera, no detector and no network. + +The depth map describes the same scene as the image: a flat back wall with the square +standing in front of it. """ import os @@ -21,9 +24,12 @@ BOX = 96 SPEED = 6 CAPTION = "dora-image-view synthetic source" +WALL_MM = 2000.0 +SQUARE_MM = 800.0 + def main() -> None: - """Send one image, bbox and text per tick.""" + """Send one image, depth map, bbox and text per tick.""" node = Node("source") sent = 0 @@ -46,6 +52,12 @@ def main() -> None: }]), {"format": "xyxy"}, ) + depth = render_depth(corners) + node.send_output( + "depth", + pa.array(depth.ravel()), + {"shape": [HEIGHT, WIDTH], "dtype": "float32", "layout": "HW", "units": "mm"}, + ) node.send_output("text", pa.array([CAPTION])) sent += 1 @@ -69,5 +81,13 @@ def render(index: int) -> tuple[np.ndarray, np.ndarray]: return frame, corners +def render_depth(corners: np.ndarray) -> np.ndarray: + """Depth of the scene in `render`: a flat wall with the square in front of it.""" + depth = np.full((HEIGHT, WIDTH), WALL_MM, dtype=np.float32) + left, top, right, bottom = (int(value) for value in corners) + depth[top:bottom, left:right] = SQUARE_MM + return depth + + if __name__ == "__main__": main() diff --git a/tests/test_dora_image_view.py b/tests/test_dora_image_view.py index 429db69..1fe81be 100644 --- a/tests/test_dora_image_view.py +++ b/tests/test_dora_image_view.py @@ -7,7 +7,7 @@ import numpy as np import pytest from PIL import Image -from dora_image_view import bbox, decode +from dora_image_view import bbox, decode, depth from dora_image_view.view import Bboxes, View WIDTH, HEIGHT = 12, 8 @@ -130,6 +130,97 @@ def test_headless_show_draws_without_a_window(): labels=np.array(["cat"]), ) frame = np.zeros((32, 32, 3), dtype=np.uint8) + view.panels["image"] = frame - assert view.show(frame) is False + assert view.show() is False assert frame.any(), "the overlays should have marked the frame" + + +def test_show_with_no_panels_is_a_no_op(): + assert View(headless=True).show() is False + + +def test_panels_are_tiled_left_to_right_at_a_common_height(): + view = View(headless=True) + view.panels["image"] = np.zeros((20, 10, 3), dtype=np.uint8) + # Half the height and half the width: scaling to the common height should + # double both, so it contributes 20 columns, not 10. + view.panels["depth"] = np.zeros((10, 10, 3), dtype=np.uint8) + + composite = view.compose() + + assert composite.shape[0] == 20 + assert composite.shape[1] == 10 + 20 + + +def test_depth_decodes_millimetres_and_reshapes(): + values = np.array([100.0, 200.0, 300.0, 400.0], dtype=np.float32) + + decoded = depth.to_millimetres(FakeStorage(values), {"shape": [2, 2]}) + + np.testing.assert_allclose(decoded, [[100.0, 200.0], [300.0, 400.0]]) + + +def test_depth_reinterprets_a_float32_payload_sent_as_bytes(): + values = np.array([1500.0, 2500.0], dtype=np.float32) + + decoded = depth.to_millimetres( + FakeStorage(values.view(np.uint8)), + {"shape": [1, 2], "dtype": "float32"}, + ) + + np.testing.assert_allclose(decoded, [[1500.0, 2500.0]]) + + +def test_depth_converts_metres_to_millimetres(): + decoded = depth.to_millimetres( + FakeStorage(np.array([1.5, 2.0], dtype=np.float32)), + {"shape": [1, 2], "units": "m"}, + ) + + np.testing.assert_allclose(decoded, [[1500.0, 2000.0]]) + + +def test_depth_normalises_non_finite_values_to_missing(): + values = np.array([np.inf, np.nan, 500.0, 0.0], dtype=np.float32) + + decoded = depth.to_millimetres(FakeStorage(values), {"shape": [2, 2]}) + + np.testing.assert_allclose(decoded, [[0.0, 0.0], [500.0, 0.0]]) + + +def test_depth_without_a_shape_is_an_error(): + with pytest.raises(RuntimeError, match="shape"): + depth.to_millimetres(FakeStorage(np.zeros(4, dtype=np.float32)), {}) + + +def test_colorize_leaves_missing_depth_black_and_colours_the_rest(): + depth_mm = np.array([[0.0, 500.0], [1000.0, 1500.0]], dtype=np.float32) + + frame = depth.colorize(depth_mm) + + assert frame.shape == (2, 2, 3) + np.testing.assert_array_equal(frame[0, 0], [0, 0, 0]) + assert frame[1, 1].any(), "valid depth should be coloured" + + +def test_depth_range_is_pinned_by_the_environment(monkeypatch): + monkeypatch.setenv("VIEW_DEPTH_MIN_MM", "200") + monkeypatch.setenv("VIEW_DEPTH_MAX_MM", "600") + + assert depth.resolve_range(np.array([[1000.0]], dtype=np.float32)) == (200.0, 600.0) + + +def test_one_pinned_end_leaves_the_other_auto_ranged(monkeypatch): + monkeypatch.setenv("VIEW_DEPTH_MIN_MM", "0") + + low, high = depth.resolve_range(np.array([[100.0, 900.0]], dtype=np.float32)) + + assert low == 0.0 + assert high > 0.0 + + +def test_depth_range_of_an_empty_frame_does_not_divide_by_zero(): + low, high = depth.resolve_range(np.zeros((2, 2), dtype=np.float32)) + + assert high > low