add depth and camera intrinsics, and adopt the typed Arrow image #4

Merged
cristhian merged 14 commits from depth-panel into main 2026-09-15 09:36:17 +00:00
12 changed files with 268 additions and 33 deletions
Showing only changes of commit 5a0f13d4db - Show all commits
+5 -3
View File
@@ -4,7 +4,7 @@ This file provides guidance to AI coding assistants when working with code in th
## What this is ## What this is
A dora-rs Python node that displays a video stream with overlays: it keeps one panel per frame stream (`image`, `depth`) plus a bbox set and text string, and every frame event redraws the lot as one side-by-side composite in an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say. A dora-rs Python node that displays a video stream with overlays: it keeps one panel per frame stream (`image`, `depth`) plus a bbox set, a text string and the camera model, and every frame event redraws the lot as one side-by-side composite in an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say.
Everything the example dataflow needs is in this checkout. Do not reintroduce a dependency on another repository — the example must keep running with no camera, no detector and no network. Everything the example dataflow needs is in this checkout. Do not reintroduce a dependency on another repository — the example must keep running with no camera, no detector and no network.
@@ -26,7 +26,9 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless
- **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node. - **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node.
- **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see. - **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
- **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one. - **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one.
- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. Two things are deliberate: `as_float32()` **reinterprets** a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`; and the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging. - **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. One thing is deliberate: the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
- **`dora_image_view/payload.py`** — `as_float32()`: reinterprets a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`. Shared by `depth` and `intrinsics`, which both meet it.
- **`dora_image_view/intrinsics.py`** — `Intrinsics`, `parse()` for the `camera_info` payload and `from_metadata()` for the copy senders attach to each frame. Both exist because dora-zed publishes the same numbers twice, and the per-frame copy is the one that survives a viewer starting after the camera did — `camera_info` goes out **once per session**. A nine-element row-major K is accepted alongside the compact four values, since that is how ROS spells the same model. The node needs none of this to draw; it is held for whatever turns depth into 3D, and drawn so a wrong camera model is visible rather than merely plausible.
- **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is. - **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is.
- **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout. - **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout.
- **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either. - **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either.
@@ -42,7 +44,7 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless
## A caveat on typed optional inputs ## A caveat on typed optional inputs
`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map at all: `depth` is typed, so the example has to drive it. `dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map and a camera_info at all: both are typed, so the example has to drive them.
## Checking what it draws ## Checking what it draws
+24 -4
View File
@@ -1,7 +1,7 @@
# dora-image-view # dora-image-view
Dora node that displays a colour image and a depth map side by side in one OpenCV Dora node that displays a colour image and a depth map side by side in one OpenCV
window, with bounding box and text overlays. A display sink for debugging a window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a
vision pipeline: it produces no outputs. vision pipeline: it produces no outputs.
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot) Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
@@ -21,6 +21,7 @@ nodes:
inputs: inputs:
image: camera/image_bgr image: camera/image_bgr
depth: camera/depth depth: camera/depth
camera_info: camera/camera_info
bbox: detector/bbox bbox: detector/bbox
text: detector/text text: detector/text
``` ```
@@ -30,7 +31,7 @@ managed environment.
Every input is optional, and each of `image` and `depth` fills one panel of the window, Every input is optional, and each of `image` and `depth` fills one panel of the window,
tiled left to right in that order. Wiring one of them is a plain viewer; tiled left to right in that order. Wiring one of them is a plain viewer;
wiring several shows them side by side, scaled to a common height. `bbox` and `text` wiring several shows them side by side, scaled to a common height. `bbox`, `text` and `camera_info`
are overlays rather than panels — boxes are drawn on the `image` panel, in its own are overlays rather than panels — boxes are drawn on the `image` panel, in its own
pixels, and the caption on the composite. pixels, and the caption on the composite.
@@ -75,8 +76,8 @@ $ dora validate dataflow.yml
All type annotations OK. All type annotations OK.
``` ```
`image` is `std/core/v1/UInt8`, `depth` is `std/core/v1/Float32` and `text` is `image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and
`std/core/v1/String`; the payloads are raw buffers described by their `text` is `std/core/v1/String`; the payloads are raw buffers described by their
metadata, so the URN names the element type rather than a struct. `bbox` is deliberately metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
node reads either. node reads either.
@@ -91,6 +92,7 @@ of that state:
resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
`Dora Node: dora-image-view`. Press `q` in the window to quit. `Dora Node: dora-image-view`. Press `q` in the window to quit.
- A **`depth`** event colour-maps the distances onto the depth panel and redraws. - A **`depth`** event colour-maps the distances onto the depth panel and redraws.
- A **`camera_info`** event stores the pinhole model and draws it under the caption.
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next - A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
redraw. redraw.
@@ -103,6 +105,7 @@ no window. That is what makes it safe to run in a pipeline test.
|---|---|---| |---|---|---|
| `image` | Flat `UInt8` array | `width`, `height`, `encoding` | | `image` | Flat `UInt8` array | `width`, `height`, `encoding` |
| `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` | | `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` |
| `camera_info` | Flat `Float32` array, `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` |
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` | | `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
| `text` | Arrow array of size 1 | — | | `text` | Arrow array of size 1 | — |
@@ -117,6 +120,23 @@ and any non-finite value mean missing depth and are drawn black. A payload sent
raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
`Float32` contract, and is reinterpreted rather than converted. `Float32` contract, and is reinterpreted rather than converted.
### Camera intrinsics
`camera_info` carries the pinhole model the depth map is interpreted through, either as
the four values `[fx, fy, cx, cy]` or as a nine-element row-major `K`, which is how ROS
spells the same model in `sensor_msgs/CameraInfo`. Distortion coefficients are read from
the `distortion` metadata.
The node displays the model and holds it. It needs none of it to draw a panel; it is
there for what reads it next — turning a depth map into a point cloud needs `fx`, `fy`,
`cx` and `cy` for every pixel, and nothing else in a dataflow carries them.
Intrinsics are also picked up from the metadata of any `image` or `depth` frame that
carries `fx`, `fy`, `cx` and `cy`, so **wiring `camera_info` is optional even for a
camera that publishes it**. That matters because a camera typically publishes
`camera_info` once per session: a viewer started after the camera would otherwise never
see it.
```python ```python
node.send_output("image", pa.array(frame.ravel()), node.send_output("image", pa.array(frame.ravel()),
{"width": 640, "height": 480, "encoding": "bgr8"}) {"width": 640, "height": 480, "encoding": "bgr8"})
+2
View File
@@ -20,6 +20,7 @@ nodes:
outputs: outputs:
- image - image
- depth - depth
- camera_info
- bbox - bbox
- text - text
@@ -29,5 +30,6 @@ nodes:
inputs: inputs:
image: source/image image: source/image
depth: source/depth depth: source/depth
camera_info: source/camera_info
bbox: source/bbox bbox: source/bbox
text: source/text text: source/text
+14 -2
View File
@@ -7,9 +7,9 @@
apiVersion: 1 apiVersion: 1
name: dora-image-view name: dora-image-view
namespace: liria namespace: liria
description: Displays an image and a depth map side by side in an OpenCV window, with bounding box and text overlays description: Displays an image and a depth map side by side in an OpenCV window, with bounding box, text and camera intrinsics overlays
categories: [visualization, debug] categories: [visualization, debug]
keywords: [opencv, view, bbox, bounding-box, image, depth, visualization, object-detection] keywords: [opencv, view, bbox, bounding-box, image, depth, intrinsics, camera-info, visualization, object-detection]
runtime: python runtime: python
entrypoint: dora-image-view entrypoint: dora-image-view
@@ -41,6 +41,17 @@ inputs:
description: >- description: >-
Text string to overlay at the top-left corner. Stored and drawn on the Text string to overlay at the top-left corner. Stored and drawn on the
next image redraw. next image redraw.
camera_info:
type: std/core/v1/Float32
required: false
description: >-
The pinhole model the depth map is interpreted through: [fx, fy, cx, cy],
or a nine-element row-major K as ROS spells it in sensor_msgs/CameraInfo.
Distortion coefficients are read from the `distortion` metadata. Stored
and displayed, and held for readers that turn depth into 3D. A sender
that also puts fx/fy/cx/cy in each frame's metadata does not need to wire
this input -- they are picked up from there, which is what covers a
camera that published its intrinsics before this node started.
depth: depth:
type: std/core/v1/Float32 type: std/core/v1/Float32
required: false required: false
@@ -77,5 +88,6 @@ example: |
inputs: inputs:
image: camera/image_bgr image: camera/image_bgr
depth: camera/depth depth: camera/depth
camera_info: camera/camera_info
bbox: detector/bbox bbox: detector/bbox
text: detector/text text: detector/text
+2 -15
View File
@@ -11,6 +11,8 @@ import os
import cv2 import cv2
import numpy as np import numpy as np
from dora_image_view.payload import as_float32
# Perceptually uniform and readable in both directions, unlike JET. # Perceptually uniform and readable in both directions, unlike JET.
COLORMAP = cv2.COLORMAP_TURBO COLORMAP = cv2.COLORMAP_TURBO
@@ -85,21 +87,6 @@ def resolve_range(depth_mm: np.ndarray) -> tuple[float, float]:
return float(low), float(high) return float(low), float(high)
def as_float32(storage) -> np.ndarray:
"""Read the payload as float32, whether it arrived as floats or as bytes.
A node that sends the map as a raw byte slice -- dora-zed does -- produces a
UInt8 arrow array even though its manifest declares Float32, so the buffer
has to be reinterpreted rather than converted.
"""
buffer = storage.to_numpy()
if buffer.dtype == np.uint8:
return buffer.view(np.float32)
return buffer.astype(np.float32, copy=False)
def _hw(metadata: dict, size: int) -> tuple[int, int]: def _hw(metadata: dict, size: int) -> tuple[int, int]:
shape = metadata.get("shape") shape = metadata.get("shape")
+89
View File
@@ -0,0 +1,89 @@
"""The pinhole camera model a depth map is interpreted through.
The node does not need intrinsics to draw anything -- it displays them, and
holds them for whatever reads them next. Back-projecting a depth map into a
point cloud is the obvious such reader: every pixel needs fx, fy, cx and cy to
become an XYZ point, and nothing else in a dataflow carries them.
Two senders are handled because dora-zed publishes the same numbers twice: once
per camera session on `camera_info`, and again in the metadata of every frame.
The per-frame copy is what makes a viewer that started after the camera did
still end up with intrinsics.
"""
from dataclasses import dataclass
from dora_image_view.payload import as_float32
# [fx, fy, cx, cy], the four numbers dora-zed publishes. Also accepted is a
# nine-element row-major K, which is how ROS spells the same model in
# sensor_msgs/CameraInfo.
COMPACT = 4
K_MATRIX = 9
# Indices of fx, fy, cx, cy in a row-major K:
# [fx, 0, cx,
# 0, fy, cy,
# 0, 0, 1]
K_INDICES = (0, 4, 2, 5)
@dataclass(frozen=True)
class Intrinsics:
"""A pinhole model: focal lengths and principal point, both in pixels."""
fx: float
fy: float
cx: float
cy: float
distortion: tuple[float, ...] = ()
model: str = ""
def __str__(self) -> str:
"""Render for the overlay, short enough to fit across a panel."""
return f"fx {self.fx:.1f} fy {self.fy:.1f} cx {self.cx:.1f} cy {self.cy:.1f}"
def parse(storage, metadata: dict) -> Intrinsics:
"""Read one `camera_info` payload.
Accepts the four-element [fx, fy, cx, cy] dora-zed sends and the
nine-element row-major K that ROS does.
"""
values = as_float32(storage)
if values.size >= K_MATRIX:
fx, fy, cx, cy = (float(values[index]) for index in K_INDICES)
elif values.size >= COMPACT:
fx, fy, cx, cy = (float(value) for value in values[:COMPACT])
else:
raise RuntimeError(
f"camera_info holds {values.size} values, "
f"expected {COMPACT} as [fx, fy, cx, cy] or {K_MATRIX} as K",
)
return Intrinsics(fx, fy, cx, cy, _distortion(metadata), _model(metadata))
def from_metadata(metadata: dict) -> Intrinsics | None:
"""Read intrinsics a sender attached to a frame, or None if it attached none."""
try:
fx, fy, cx, cy = (
float(metadata["fx"]),
float(metadata["fy"]),
float(metadata["cx"]),
float(metadata["cy"]),
)
except (KeyError, TypeError, ValueError):
return None
return Intrinsics(fx, fy, cx, cy, _distortion(metadata), _model(metadata))
def _distortion(metadata: dict) -> tuple[float, ...]:
values = metadata.get("distortion") or ()
return tuple(float(value) for value in values)
def _model(metadata: dict) -> str:
return str(metadata.get("distortion_model", ""))
+11 -3
View File
@@ -1,7 +1,8 @@
"""A dora display sink: overlays bounding boxes and text on an image input. """A dora display sink: overlays bounding boxes and text on an image input.
`image` and `depth` each fill one panel of the window and redraw. `bbox` and `image` and `depth` each fill one panel of the window and redraw. `bbox`,
`text` events only update stored state, which the next redraw draws. `text` and `camera_info` events only update stored state, which the next redraw
draws.
""" """
import argparse import argparse
@@ -10,7 +11,7 @@ import os
import numpy as np import numpy as np
from dora import Node from dora import Node
from dora_image_view import bbox, decode, depth from dora_image_view import bbox, decode, depth, intrinsics
from dora_image_view.view import View from dora_image_view.view import View
@@ -37,6 +38,11 @@ def main() -> None:
event_id = event["id"] event_id = event["id"]
metadata = event["metadata"] metadata = event["metadata"]
# A sender that puts intrinsics on every frame keeps them current even
# if the camera_info message itself was published before this node
# started, which is when dora-zed sends it.
view.intrinsics = intrinsics.from_metadata(metadata) or view.intrinsics
panel = _decode_panel(event_id, event["value"], metadata) panel = _decode_panel(event_id, event["value"], metadata)
if panel is not None: if panel is not None:
@@ -44,6 +50,8 @@ def main() -> None:
view.panels[name] = frame view.panels[name] = frame
if view.show(): if view.show():
break break
elif event_id == "camera_info":
view.intrinsics = intrinsics.parse(event["value"], metadata)
elif event_id == "bbox": elif event_id == "bbox":
view.bboxes = bbox.parse(event["value"], metadata.get("format", "xyxy").lower()) view.bboxes = bbox.parse(event["value"], metadata.get("format", "xyxy").lower())
elif event_id == "text": elif event_id == "text":
+18
View File
@@ -0,0 +1,18 @@
"""Reading raw dora payloads, independently of what they mean."""
import numpy as np
def as_float32(storage) -> np.ndarray:
"""Read a payload as float32, whether it arrived as floats or as bytes.
A node that sends its payload as a raw byte slice -- which is what a C++
node does -- produces a UInt8 arrow array even though its manifest declares
Float32, so the buffer has to be reinterpreted rather than converted.
"""
buffer = storage.to_numpy()
if buffer.dtype == np.uint8:
return buffer.view(np.float32)
return buffer.astype(np.float32, copy=False)
+10
View File
@@ -5,6 +5,8 @@ from dataclasses import dataclass, field
import cv2 import cv2
import numpy as np import numpy as np
from dora_image_view.intrinsics import Intrinsics
WINDOW = "Dora Node: dora-image-view" WINDOW = "Dora Node: dora-image-view"
GREEN = (0, 255, 0) GREEN = (0, 255, 0)
WHITE = (255, 255, 255) WHITE = (255, 255, 255)
@@ -40,6 +42,7 @@ class View:
bboxes: Bboxes = field(default_factory=Bboxes) bboxes: Bboxes = field(default_factory=Bboxes)
text: str = "" text: str = ""
panels: dict[str, np.ndarray] = field(default_factory=dict) panels: dict[str, np.ndarray] = field(default_factory=dict)
intrinsics: Intrinsics | None = None
def show(self) -> bool: def show(self) -> bool:
"""Draw the overlays, tile the panels, and display them. """Draw the overlays, tile the panels, and display them.
@@ -79,6 +82,13 @@ class View:
# On the composite, not on a panel: the caption describes the frame as a # On the composite, not on a panel: the caption describes the frame as a
# whole, and a panel it was drawn on might not be the leftmost one. # whole, and a panel it was drawn on might not be the leftmost one.
cv2.putText(frame, self.text, (20, 20), FONT, FONT_SCALE, WHITE, 1) cv2.putText(frame, self.text, (20, 20), FONT, FONT_SCALE, WHITE, 1)
# Showing the camera model is the point of receiving it here: a depth
# map read through the wrong intrinsics looks entirely plausible, and
# this is where that becomes visible.
if self.intrinsics is not None:
cv2.putText(frame, str(self.intrinsics), (20, 40), FONT, FONT_SCALE, WHITE, 1)
return frame return frame
def _label(self, panel: np.ndarray, name: str) -> np.ndarray: def _label(self, panel: np.ndarray, name: str) -> np.ndarray:
+6 -1
View File
@@ -12,7 +12,7 @@
apiVersion: 1 apiVersion: 1
name: source name: source
namespace: liria namespace: liria
description: Synthetic image, depth, bbox and text source for the dora-image-view example description: Synthetic image, depth, camera_info, bbox and text source for the dora-image-view example
categories: [debug] categories: [debug]
keywords: [example, test, synthetic] keywords: [example, test, synthetic]
@@ -39,6 +39,11 @@ outputs:
text: text:
type: std/core/v1/String type: std/core/v1/String
description: A fixed caption. description: A fixed caption.
camera_info:
type: std/core/v1/Float32
description: >-
[fx, fy, cx, cy] for the frames above. Published once, on the first tick,
the way a real camera publishes it once per session.
depth: depth:
type: std/core/v1/Float32 type: std/core/v1/Float32
description: >- description: >-
+27 -4
View File
@@ -5,7 +5,9 @@ moving in it, and a caption -- every input dora-image-view draws -- so the examp
dataflow needs no camera, no detector and no network. dataflow needs no camera, no detector and no network.
The depth map describes the same scene as the image: a flat back wall with the square The depth map describes the same scene as the image: a flat back wall with the square
standing in front of it. standing in front of it. `camera_info` is published once, on the first tick, the way a
real camera publishes it once per session -- which is exactly the case the viewer has
to cope with by reading intrinsics from frame metadata as well.
""" """
import os import os
@@ -27,9 +29,17 @@ CAPTION = "dora-image-view synthetic source"
WALL_MM = 2000.0 WALL_MM = 2000.0
SQUARE_MM = 800.0 SQUARE_MM = 800.0
# A plausible pinhole for a frame this size.
FX = FY = float(WIDTH)
CX, CY = WIDTH / 2, HEIGHT / 2
DISTORTION = [0.0, 0.0, 0.0, 0.0, 0.0]
# Attached to every frame as well as published once, which is what dora-zed does
# and what lets a viewer that started late still learn the camera model.
INTRINSICS_METADATA = {"fx": FX, "fy": FY, "cx": CX, "cy": CY, "distortion": DISTORTION}
def main() -> None: def main() -> None:
"""Send one image, depth map, bbox and text per tick.""" """Send one image, depth map, bbox and text per tick, and camera_info once."""
node = Node("source") node = Node("source")
sent = 0 sent = 0
@@ -37,11 +47,18 @@ def main() -> None:
if event["type"] != "INPUT": if event["type"] != "INPUT":
continue continue
if sent == 0:
node.send_output(
"camera_info",
pa.array(np.array([FX, FY, CX, CY], dtype=np.float32)),
{"shape": [4], "dtype": "float32", "layout": "C", "distortion": DISTORTION},
)
frame, corners = render(sent) frame, corners = render(sent)
node.send_output( node.send_output(
"image", "image",
pa.array(frame.ravel()), pa.array(frame.ravel()),
{"width": WIDTH, "height": HEIGHT, "encoding": "bgr8"}, {"width": WIDTH, "height": HEIGHT, "encoding": "bgr8", **INTRINSICS_METADATA},
) )
node.send_output( node.send_output(
"bbox", "bbox",
@@ -56,7 +73,13 @@ def main() -> None:
node.send_output( node.send_output(
"depth", "depth",
pa.array(depth.ravel()), pa.array(depth.ravel()),
{"shape": [HEIGHT, WIDTH], "dtype": "float32", "layout": "HW", "units": "mm"}, {
"shape": [HEIGHT, WIDTH],
"dtype": "float32",
"layout": "HW",
"units": "mm",
**INTRINSICS_METADATA,
},
) )
node.send_output("text", pa.array([CAPTION])) node.send_output("text", pa.array([CAPTION]))
+60 -1
View File
@@ -7,7 +7,7 @@ import numpy as np
import pytest import pytest
from PIL import Image from PIL import Image
from dora_image_view import bbox, decode, depth from dora_image_view import bbox, decode, depth, intrinsics
from dora_image_view.view import Bboxes, View from dora_image_view.view import Bboxes, View
WIDTH, HEIGHT = 12, 8 WIDTH, HEIGHT = 12, 8
@@ -224,3 +224,62 @@ def test_depth_range_of_an_empty_frame_does_not_divide_by_zero():
low, high = depth.resolve_range(np.zeros((2, 2), dtype=np.float32)) low, high = depth.resolve_range(np.zeros((2, 2), dtype=np.float32))
assert high > low assert high > low
def test_camera_info_reads_the_compact_four_value_form():
values = np.array([600.0, 601.0, 320.0, 240.0], dtype=np.float32)
model = intrinsics.parse(FakeStorage(values), {"distortion": [0.1, 0.2]})
assert (model.fx, model.fy, model.cx, model.cy) == (600.0, 601.0, 320.0, 240.0)
assert model.distortion == (0.1, 0.2)
def test_camera_info_reads_a_ros_k_matrix():
# Row-major K: [fx, 0, cx, 0, fy, cy, 0, 0, 1].
k = np.array(
[600.0, 0.0, 320.0, 0.0, 601.0, 240.0, 0.0, 0.0, 1.0],
dtype=np.float32,
)
model = intrinsics.parse(FakeStorage(k), {})
assert (model.fx, model.fy, model.cx, model.cy) == (600.0, 601.0, 320.0, 240.0)
def test_camera_info_reinterprets_a_payload_sent_as_bytes():
values = np.array([600.0, 601.0, 320.0, 240.0], dtype=np.float32)
model = intrinsics.parse(FakeStorage(values.view(np.uint8)), {})
assert model.fx == 600.0
assert model.cy == 240.0
def test_camera_info_that_is_too_short_is_an_error():
with pytest.raises(RuntimeError, match="camera_info"):
intrinsics.parse(FakeStorage(np.zeros(3, dtype=np.float32)), {})
def test_intrinsics_are_read_from_frame_metadata():
model = intrinsics.from_metadata(
{"fx": 600.0, "fy": 601.0, "cx": 320.0, "cy": 240.0},
)
assert model is not None
assert (model.fx, model.cy) == (600.0, 240.0)
def test_frame_metadata_without_intrinsics_yields_none():
assert intrinsics.from_metadata({"width": 640, "height": 480}) is None
def test_the_camera_model_is_drawn_on_the_composite():
view = View(headless=True)
view.panels["image"] = np.zeros((40, 80, 3), dtype=np.uint8)
without = view.compose().copy()
view.intrinsics = intrinsics.Intrinsics(600.0, 601.0, 320.0, 240.0)
with_model = view.compose()
assert not np.array_equal(without, with_model), "the model should be drawn"