add depth and camera intrinsics, and adopt the typed Arrow image #4
@@ -4,7 +4,7 @@ This file provides guidance to AI coding assistants when working with code in th
|
|||||||
|
|
||||||
## What this is
|
## What this is
|
||||||
|
|
||||||
A dora-rs Python node that displays a video stream with overlays: it keeps one panel per frame stream (`image`, `depth`) plus a bbox set and text string, and every frame event redraws the lot as one side-by-side composite in an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say.
|
A dora-rs Python node that displays a video stream with overlays: it keeps one panel per frame stream (`image`, `depth`) plus a bbox set, a text string and the camera model, and every frame event redraws the lot as one side-by-side composite in an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say.
|
||||||
|
|
||||||
Everything the example dataflow needs is in this checkout. Do not reintroduce a dependency on another repository — the example must keep running with no camera, no detector and no network.
|
Everything the example dataflow needs is in this checkout. Do not reintroduce a dependency on another repository — the example must keep running with no camera, no detector and no network.
|
||||||
|
|
||||||
@@ -26,7 +26,9 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless
|
|||||||
- **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node.
|
- **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node.
|
||||||
- **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
|
- **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
|
||||||
- **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one.
|
- **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one.
|
||||||
- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. Two things are deliberate: `as_float32()` **reinterprets** a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`; and the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
|
- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. One thing is deliberate: the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
|
||||||
|
- **`dora_image_view/payload.py`** — `as_float32()`: reinterprets a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`. Shared by `depth` and `intrinsics`, which both meet it.
|
||||||
|
- **`dora_image_view/intrinsics.py`** — `Intrinsics`, `parse()` for the `camera_info` payload and `from_metadata()` for the copy senders attach to each frame. Both exist because dora-zed publishes the same numbers twice, and the per-frame copy is the one that survives a viewer starting after the camera did — `camera_info` goes out **once per session**. A nine-element row-major K is accepted alongside the compact four values, since that is how ROS spells the same model. The node needs none of this to draw; it is held for whatever turns depth into 3D, and drawn so a wrong camera model is visible rather than merely plausible.
|
||||||
- **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is.
|
- **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is.
|
||||||
- **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout.
|
- **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout.
|
||||||
- **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either.
|
- **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either.
|
||||||
@@ -42,7 +44,7 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless
|
|||||||
|
|
||||||
## A caveat on typed optional inputs
|
## A caveat on typed optional inputs
|
||||||
|
|
||||||
`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map at all: `depth` is typed, so the example has to drive it.
|
`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map and a camera_info at all: both are typed, so the example has to drive them.
|
||||||
|
|
||||||
## Checking what it draws
|
## Checking what it draws
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
# dora-image-view
|
# dora-image-view
|
||||||
|
|
||||||
Dora node that displays a colour image and a depth map side by side in one OpenCV
|
Dora node that displays a colour image and a depth map side by side in one OpenCV
|
||||||
window, with bounding box and text overlays. A display sink for debugging a
|
window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a
|
||||||
vision pipeline: it produces no outputs.
|
vision pipeline: it produces no outputs.
|
||||||
|
|
||||||
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
|
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
|
||||||
@@ -21,6 +21,7 @@ nodes:
|
|||||||
inputs:
|
inputs:
|
||||||
image: camera/image_bgr
|
image: camera/image_bgr
|
||||||
depth: camera/depth
|
depth: camera/depth
|
||||||
|
camera_info: camera/camera_info
|
||||||
bbox: detector/bbox
|
bbox: detector/bbox
|
||||||
text: detector/text
|
text: detector/text
|
||||||
```
|
```
|
||||||
@@ -30,7 +31,7 @@ managed environment.
|
|||||||
|
|
||||||
Every input is optional, and each of `image` and `depth` fills one panel of the window,
|
Every input is optional, and each of `image` and `depth` fills one panel of the window,
|
||||||
tiled left to right in that order. Wiring one of them is a plain viewer;
|
tiled left to right in that order. Wiring one of them is a plain viewer;
|
||||||
wiring several shows them side by side, scaled to a common height. `bbox` and `text`
|
wiring several shows them side by side, scaled to a common height. `bbox`, `text` and `camera_info`
|
||||||
are overlays rather than panels — boxes are drawn on the `image` panel, in its own
|
are overlays rather than panels — boxes are drawn on the `image` panel, in its own
|
||||||
pixels, and the caption on the composite.
|
pixels, and the caption on the composite.
|
||||||
|
|
||||||
@@ -75,8 +76,8 @@ $ dora validate dataflow.yml
|
|||||||
All type annotations OK.
|
All type annotations OK.
|
||||||
```
|
```
|
||||||
|
|
||||||
`image` is `std/core/v1/UInt8`, `depth` is `std/core/v1/Float32` and `text` is
|
`image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and
|
||||||
`std/core/v1/String`; the payloads are raw buffers described by their
|
`text` is `std/core/v1/String`; the payloads are raw buffers described by their
|
||||||
metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
|
metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
|
||||||
left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
|
left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
|
||||||
node reads either.
|
node reads either.
|
||||||
@@ -91,6 +92,7 @@ of that state:
|
|||||||
resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
|
resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
|
||||||
`Dora Node: dora-image-view`. Press `q` in the window to quit.
|
`Dora Node: dora-image-view`. Press `q` in the window to quit.
|
||||||
- A **`depth`** event colour-maps the distances onto the depth panel and redraws.
|
- A **`depth`** event colour-maps the distances onto the depth panel and redraws.
|
||||||
|
- A **`camera_info`** event stores the pinhole model and draws it under the caption.
|
||||||
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
|
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
|
||||||
redraw.
|
redraw.
|
||||||
|
|
||||||
@@ -103,6 +105,7 @@ no window. That is what makes it safe to run in a pipeline test.
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `image` | Flat `UInt8` array | `width`, `height`, `encoding` |
|
| `image` | Flat `UInt8` array | `width`, `height`, `encoding` |
|
||||||
| `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` |
|
| `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` |
|
||||||
|
| `camera_info` | Flat `Float32` array, `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` |
|
||||||
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
|
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
|
||||||
| `text` | Arrow array of size 1 | — |
|
| `text` | Arrow array of size 1 | — |
|
||||||
|
|
||||||
@@ -117,6 +120,23 @@ and any non-finite value mean missing depth and are drawn black. A payload sent
|
|||||||
raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
|
raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
|
||||||
`Float32` contract, and is reinterpreted rather than converted.
|
`Float32` contract, and is reinterpreted rather than converted.
|
||||||
|
|
||||||
|
### Camera intrinsics
|
||||||
|
|
||||||
|
`camera_info` carries the pinhole model the depth map is interpreted through, either as
|
||||||
|
the four values `[fx, fy, cx, cy]` or as a nine-element row-major `K`, which is how ROS
|
||||||
|
spells the same model in `sensor_msgs/CameraInfo`. Distortion coefficients are read from
|
||||||
|
the `distortion` metadata.
|
||||||
|
|
||||||
|
The node displays the model and holds it. It needs none of it to draw a panel; it is
|
||||||
|
there for what reads it next — turning a depth map into a point cloud needs `fx`, `fy`,
|
||||||
|
`cx` and `cy` for every pixel, and nothing else in a dataflow carries them.
|
||||||
|
|
||||||
|
Intrinsics are also picked up from the metadata of any `image` or `depth` frame that
|
||||||
|
carries `fx`, `fy`, `cx` and `cy`, so **wiring `camera_info` is optional even for a
|
||||||
|
camera that publishes it**. That matters because a camera typically publishes
|
||||||
|
`camera_info` once per session: a viewer started after the camera would otherwise never
|
||||||
|
see it.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
node.send_output("image", pa.array(frame.ravel()),
|
node.send_output("image", pa.array(frame.ravel()),
|
||||||
{"width": 640, "height": 480, "encoding": "bgr8"})
|
{"width": 640, "height": 480, "encoding": "bgr8"})
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ nodes:
|
|||||||
outputs:
|
outputs:
|
||||||
- image
|
- image
|
||||||
- depth
|
- depth
|
||||||
|
- camera_info
|
||||||
- bbox
|
- bbox
|
||||||
- text
|
- text
|
||||||
|
|
||||||
@@ -29,5 +30,6 @@ nodes:
|
|||||||
inputs:
|
inputs:
|
||||||
image: source/image
|
image: source/image
|
||||||
depth: source/depth
|
depth: source/depth
|
||||||
|
camera_info: source/camera_info
|
||||||
bbox: source/bbox
|
bbox: source/bbox
|
||||||
text: source/text
|
text: source/text
|
||||||
|
|||||||
+14
-2
@@ -7,9 +7,9 @@
|
|||||||
apiVersion: 1
|
apiVersion: 1
|
||||||
name: dora-image-view
|
name: dora-image-view
|
||||||
namespace: liria
|
namespace: liria
|
||||||
description: Displays an image and a depth map side by side in an OpenCV window, with bounding box and text overlays
|
description: Displays an image and a depth map side by side in an OpenCV window, with bounding box, text and camera intrinsics overlays
|
||||||
categories: [visualization, debug]
|
categories: [visualization, debug]
|
||||||
keywords: [opencv, view, bbox, bounding-box, image, depth, visualization, object-detection]
|
keywords: [opencv, view, bbox, bounding-box, image, depth, intrinsics, camera-info, visualization, object-detection]
|
||||||
|
|
||||||
runtime: python
|
runtime: python
|
||||||
entrypoint: dora-image-view
|
entrypoint: dora-image-view
|
||||||
@@ -41,6 +41,17 @@ inputs:
|
|||||||
description: >-
|
description: >-
|
||||||
Text string to overlay at the top-left corner. Stored and drawn on the
|
Text string to overlay at the top-left corner. Stored and drawn on the
|
||||||
next image redraw.
|
next image redraw.
|
||||||
|
camera_info:
|
||||||
|
type: std/core/v1/Float32
|
||||||
|
required: false
|
||||||
|
description: >-
|
||||||
|
The pinhole model the depth map is interpreted through: [fx, fy, cx, cy],
|
||||||
|
or a nine-element row-major K as ROS spells it in sensor_msgs/CameraInfo.
|
||||||
|
Distortion coefficients are read from the `distortion` metadata. Stored
|
||||||
|
and displayed, and held for readers that turn depth into 3D. A sender
|
||||||
|
that also puts fx/fy/cx/cy in each frame's metadata does not need to wire
|
||||||
|
this input -- they are picked up from there, which is what covers a
|
||||||
|
camera that published its intrinsics before this node started.
|
||||||
depth:
|
depth:
|
||||||
type: std/core/v1/Float32
|
type: std/core/v1/Float32
|
||||||
required: false
|
required: false
|
||||||
@@ -77,5 +88,6 @@ example: |
|
|||||||
inputs:
|
inputs:
|
||||||
image: camera/image_bgr
|
image: camera/image_bgr
|
||||||
depth: camera/depth
|
depth: camera/depth
|
||||||
|
camera_info: camera/camera_info
|
||||||
bbox: detector/bbox
|
bbox: detector/bbox
|
||||||
text: detector/text
|
text: detector/text
|
||||||
|
|||||||
@@ -11,6 +11,8 @@ import os
|
|||||||
import cv2
|
import cv2
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
|
from dora_image_view.payload import as_float32
|
||||||
|
|
||||||
# Perceptually uniform and readable in both directions, unlike JET.
|
# Perceptually uniform and readable in both directions, unlike JET.
|
||||||
COLORMAP = cv2.COLORMAP_TURBO
|
COLORMAP = cv2.COLORMAP_TURBO
|
||||||
|
|
||||||
@@ -85,21 +87,6 @@ def resolve_range(depth_mm: np.ndarray) -> tuple[float, float]:
|
|||||||
return float(low), float(high)
|
return float(low), float(high)
|
||||||
|
|
||||||
|
|
||||||
def as_float32(storage) -> np.ndarray:
|
|
||||||
"""Read the payload as float32, whether it arrived as floats or as bytes.
|
|
||||||
|
|
||||||
A node that sends the map as a raw byte slice -- dora-zed does -- produces a
|
|
||||||
UInt8 arrow array even though its manifest declares Float32, so the buffer
|
|
||||||
has to be reinterpreted rather than converted.
|
|
||||||
"""
|
|
||||||
buffer = storage.to_numpy()
|
|
||||||
|
|
||||||
if buffer.dtype == np.uint8:
|
|
||||||
return buffer.view(np.float32)
|
|
||||||
|
|
||||||
return buffer.astype(np.float32, copy=False)
|
|
||||||
|
|
||||||
|
|
||||||
def _hw(metadata: dict, size: int) -> tuple[int, int]:
|
def _hw(metadata: dict, size: int) -> tuple[int, int]:
|
||||||
shape = metadata.get("shape")
|
shape = metadata.get("shape")
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
"""The pinhole camera model a depth map is interpreted through.
|
||||||
|
|
||||||
|
The node does not need intrinsics to draw anything -- it displays them, and
|
||||||
|
holds them for whatever reads them next. Back-projecting a depth map into a
|
||||||
|
point cloud is the obvious such reader: every pixel needs fx, fy, cx and cy to
|
||||||
|
become an XYZ point, and nothing else in a dataflow carries them.
|
||||||
|
|
||||||
|
Two senders are handled because dora-zed publishes the same numbers twice: once
|
||||||
|
per camera session on `camera_info`, and again in the metadata of every frame.
|
||||||
|
The per-frame copy is what makes a viewer that started after the camera did
|
||||||
|
still end up with intrinsics.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
from dora_image_view.payload import as_float32
|
||||||
|
|
||||||
|
# [fx, fy, cx, cy], the four numbers dora-zed publishes. Also accepted is a
|
||||||
|
# nine-element row-major K, which is how ROS spells the same model in
|
||||||
|
# sensor_msgs/CameraInfo.
|
||||||
|
COMPACT = 4
|
||||||
|
K_MATRIX = 9
|
||||||
|
|
||||||
|
# Indices of fx, fy, cx, cy in a row-major K:
|
||||||
|
# [fx, 0, cx,
|
||||||
|
# 0, fy, cy,
|
||||||
|
# 0, 0, 1]
|
||||||
|
K_INDICES = (0, 4, 2, 5)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Intrinsics:
|
||||||
|
"""A pinhole model: focal lengths and principal point, both in pixels."""
|
||||||
|
|
||||||
|
fx: float
|
||||||
|
fy: float
|
||||||
|
cx: float
|
||||||
|
cy: float
|
||||||
|
distortion: tuple[float, ...] = ()
|
||||||
|
model: str = ""
|
||||||
|
|
||||||
|
def __str__(self) -> str:
|
||||||
|
"""Render for the overlay, short enough to fit across a panel."""
|
||||||
|
return f"fx {self.fx:.1f} fy {self.fy:.1f} cx {self.cx:.1f} cy {self.cy:.1f}"
|
||||||
|
|
||||||
|
|
||||||
|
def parse(storage, metadata: dict) -> Intrinsics:
|
||||||
|
"""Read one `camera_info` payload.
|
||||||
|
|
||||||
|
Accepts the four-element [fx, fy, cx, cy] dora-zed sends and the
|
||||||
|
nine-element row-major K that ROS does.
|
||||||
|
"""
|
||||||
|
values = as_float32(storage)
|
||||||
|
|
||||||
|
if values.size >= K_MATRIX:
|
||||||
|
fx, fy, cx, cy = (float(values[index]) for index in K_INDICES)
|
||||||
|
elif values.size >= COMPACT:
|
||||||
|
fx, fy, cx, cy = (float(value) for value in values[:COMPACT])
|
||||||
|
else:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"camera_info holds {values.size} values, "
|
||||||
|
f"expected {COMPACT} as [fx, fy, cx, cy] or {K_MATRIX} as K",
|
||||||
|
)
|
||||||
|
|
||||||
|
return Intrinsics(fx, fy, cx, cy, _distortion(metadata), _model(metadata))
|
||||||
|
|
||||||
|
|
||||||
|
def from_metadata(metadata: dict) -> Intrinsics | None:
|
||||||
|
"""Read intrinsics a sender attached to a frame, or None if it attached none."""
|
||||||
|
try:
|
||||||
|
fx, fy, cx, cy = (
|
||||||
|
float(metadata["fx"]),
|
||||||
|
float(metadata["fy"]),
|
||||||
|
float(metadata["cx"]),
|
||||||
|
float(metadata["cy"]),
|
||||||
|
)
|
||||||
|
except (KeyError, TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
return Intrinsics(fx, fy, cx, cy, _distortion(metadata), _model(metadata))
|
||||||
|
|
||||||
|
|
||||||
|
def _distortion(metadata: dict) -> tuple[float, ...]:
|
||||||
|
values = metadata.get("distortion") or ()
|
||||||
|
return tuple(float(value) for value in values)
|
||||||
|
|
||||||
|
|
||||||
|
def _model(metadata: dict) -> str:
|
||||||
|
return str(metadata.get("distortion_model", ""))
|
||||||
+11
-3
@@ -1,7 +1,8 @@
|
|||||||
"""A dora display sink: overlays bounding boxes and text on an image input.
|
"""A dora display sink: overlays bounding boxes and text on an image input.
|
||||||
|
|
||||||
`image` and `depth` each fill one panel of the window and redraw. `bbox` and
|
`image` and `depth` each fill one panel of the window and redraw. `bbox`,
|
||||||
`text` events only update stored state, which the next redraw draws.
|
`text` and `camera_info` events only update stored state, which the next redraw
|
||||||
|
draws.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
@@ -10,7 +11,7 @@ import os
|
|||||||
import numpy as np
|
import numpy as np
|
||||||
from dora import Node
|
from dora import Node
|
||||||
|
|
||||||
from dora_image_view import bbox, decode, depth
|
from dora_image_view import bbox, decode, depth, intrinsics
|
||||||
from dora_image_view.view import View
|
from dora_image_view.view import View
|
||||||
|
|
||||||
|
|
||||||
@@ -37,6 +38,11 @@ def main() -> None:
|
|||||||
event_id = event["id"]
|
event_id = event["id"]
|
||||||
metadata = event["metadata"]
|
metadata = event["metadata"]
|
||||||
|
|
||||||
|
# A sender that puts intrinsics on every frame keeps them current even
|
||||||
|
# if the camera_info message itself was published before this node
|
||||||
|
# started, which is when dora-zed sends it.
|
||||||
|
view.intrinsics = intrinsics.from_metadata(metadata) or view.intrinsics
|
||||||
|
|
||||||
panel = _decode_panel(event_id, event["value"], metadata)
|
panel = _decode_panel(event_id, event["value"], metadata)
|
||||||
|
|
||||||
if panel is not None:
|
if panel is not None:
|
||||||
@@ -44,6 +50,8 @@ def main() -> None:
|
|||||||
view.panels[name] = frame
|
view.panels[name] = frame
|
||||||
if view.show():
|
if view.show():
|
||||||
break
|
break
|
||||||
|
elif event_id == "camera_info":
|
||||||
|
view.intrinsics = intrinsics.parse(event["value"], metadata)
|
||||||
elif event_id == "bbox":
|
elif event_id == "bbox":
|
||||||
view.bboxes = bbox.parse(event["value"], metadata.get("format", "xyxy").lower())
|
view.bboxes = bbox.parse(event["value"], metadata.get("format", "xyxy").lower())
|
||||||
elif event_id == "text":
|
elif event_id == "text":
|
||||||
|
|||||||
@@ -0,0 +1,18 @@
|
|||||||
|
"""Reading raw dora payloads, independently of what they mean."""
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
|
||||||
|
def as_float32(storage) -> np.ndarray:
|
||||||
|
"""Read a payload as float32, whether it arrived as floats or as bytes.
|
||||||
|
|
||||||
|
A node that sends its payload as a raw byte slice -- which is what a C++
|
||||||
|
node does -- produces a UInt8 arrow array even though its manifest declares
|
||||||
|
Float32, so the buffer has to be reinterpreted rather than converted.
|
||||||
|
"""
|
||||||
|
buffer = storage.to_numpy()
|
||||||
|
|
||||||
|
if buffer.dtype == np.uint8:
|
||||||
|
return buffer.view(np.float32)
|
||||||
|
|
||||||
|
return buffer.astype(np.float32, copy=False)
|
||||||
@@ -5,6 +5,8 @@ from dataclasses import dataclass, field
|
|||||||
import cv2
|
import cv2
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
|
from dora_image_view.intrinsics import Intrinsics
|
||||||
|
|
||||||
WINDOW = "Dora Node: dora-image-view"
|
WINDOW = "Dora Node: dora-image-view"
|
||||||
GREEN = (0, 255, 0)
|
GREEN = (0, 255, 0)
|
||||||
WHITE = (255, 255, 255)
|
WHITE = (255, 255, 255)
|
||||||
@@ -40,6 +42,7 @@ class View:
|
|||||||
bboxes: Bboxes = field(default_factory=Bboxes)
|
bboxes: Bboxes = field(default_factory=Bboxes)
|
||||||
text: str = ""
|
text: str = ""
|
||||||
panels: dict[str, np.ndarray] = field(default_factory=dict)
|
panels: dict[str, np.ndarray] = field(default_factory=dict)
|
||||||
|
intrinsics: Intrinsics | None = None
|
||||||
|
|
||||||
def show(self) -> bool:
|
def show(self) -> bool:
|
||||||
"""Draw the overlays, tile the panels, and display them.
|
"""Draw the overlays, tile the panels, and display them.
|
||||||
@@ -79,6 +82,13 @@ class View:
|
|||||||
# On the composite, not on a panel: the caption describes the frame as a
|
# On the composite, not on a panel: the caption describes the frame as a
|
||||||
# whole, and a panel it was drawn on might not be the leftmost one.
|
# whole, and a panel it was drawn on might not be the leftmost one.
|
||||||
cv2.putText(frame, self.text, (20, 20), FONT, FONT_SCALE, WHITE, 1)
|
cv2.putText(frame, self.text, (20, 20), FONT, FONT_SCALE, WHITE, 1)
|
||||||
|
|
||||||
|
# Showing the camera model is the point of receiving it here: a depth
|
||||||
|
# map read through the wrong intrinsics looks entirely plausible, and
|
||||||
|
# this is where that becomes visible.
|
||||||
|
if self.intrinsics is not None:
|
||||||
|
cv2.putText(frame, str(self.intrinsics), (20, 40), FONT, FONT_SCALE, WHITE, 1)
|
||||||
|
|
||||||
return frame
|
return frame
|
||||||
|
|
||||||
def _label(self, panel: np.ndarray, name: str) -> np.ndarray:
|
def _label(self, panel: np.ndarray, name: str) -> np.ndarray:
|
||||||
|
|||||||
@@ -12,7 +12,7 @@
|
|||||||
apiVersion: 1
|
apiVersion: 1
|
||||||
name: source
|
name: source
|
||||||
namespace: liria
|
namespace: liria
|
||||||
description: Synthetic image, depth, bbox and text source for the dora-image-view example
|
description: Synthetic image, depth, camera_info, bbox and text source for the dora-image-view example
|
||||||
categories: [debug]
|
categories: [debug]
|
||||||
keywords: [example, test, synthetic]
|
keywords: [example, test, synthetic]
|
||||||
|
|
||||||
@@ -39,6 +39,11 @@ outputs:
|
|||||||
text:
|
text:
|
||||||
type: std/core/v1/String
|
type: std/core/v1/String
|
||||||
description: A fixed caption.
|
description: A fixed caption.
|
||||||
|
camera_info:
|
||||||
|
type: std/core/v1/Float32
|
||||||
|
description: >-
|
||||||
|
[fx, fy, cx, cy] for the frames above. Published once, on the first tick,
|
||||||
|
the way a real camera publishes it once per session.
|
||||||
depth:
|
depth:
|
||||||
type: std/core/v1/Float32
|
type: std/core/v1/Float32
|
||||||
description: >-
|
description: >-
|
||||||
|
|||||||
+27
-4
@@ -5,7 +5,9 @@ moving in it, and a caption -- every input dora-image-view draws -- so the examp
|
|||||||
dataflow needs no camera, no detector and no network.
|
dataflow needs no camera, no detector and no network.
|
||||||
|
|
||||||
The depth map describes the same scene as the image: a flat back wall with the square
|
The depth map describes the same scene as the image: a flat back wall with the square
|
||||||
standing in front of it.
|
standing in front of it. `camera_info` is published once, on the first tick, the way a
|
||||||
|
real camera publishes it once per session -- which is exactly the case the viewer has
|
||||||
|
to cope with by reading intrinsics from frame metadata as well.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
@@ -27,9 +29,17 @@ CAPTION = "dora-image-view synthetic source"
|
|||||||
WALL_MM = 2000.0
|
WALL_MM = 2000.0
|
||||||
SQUARE_MM = 800.0
|
SQUARE_MM = 800.0
|
||||||
|
|
||||||
|
# A plausible pinhole for a frame this size.
|
||||||
|
FX = FY = float(WIDTH)
|
||||||
|
CX, CY = WIDTH / 2, HEIGHT / 2
|
||||||
|
DISTORTION = [0.0, 0.0, 0.0, 0.0, 0.0]
|
||||||
|
# Attached to every frame as well as published once, which is what dora-zed does
|
||||||
|
# and what lets a viewer that started late still learn the camera model.
|
||||||
|
INTRINSICS_METADATA = {"fx": FX, "fy": FY, "cx": CX, "cy": CY, "distortion": DISTORTION}
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
"""Send one image, depth map, bbox and text per tick."""
|
"""Send one image, depth map, bbox and text per tick, and camera_info once."""
|
||||||
node = Node("source")
|
node = Node("source")
|
||||||
sent = 0
|
sent = 0
|
||||||
|
|
||||||
@@ -37,11 +47,18 @@ def main() -> None:
|
|||||||
if event["type"] != "INPUT":
|
if event["type"] != "INPUT":
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
if sent == 0:
|
||||||
|
node.send_output(
|
||||||
|
"camera_info",
|
||||||
|
pa.array(np.array([FX, FY, CX, CY], dtype=np.float32)),
|
||||||
|
{"shape": [4], "dtype": "float32", "layout": "C", "distortion": DISTORTION},
|
||||||
|
)
|
||||||
|
|
||||||
frame, corners = render(sent)
|
frame, corners = render(sent)
|
||||||
node.send_output(
|
node.send_output(
|
||||||
"image",
|
"image",
|
||||||
pa.array(frame.ravel()),
|
pa.array(frame.ravel()),
|
||||||
{"width": WIDTH, "height": HEIGHT, "encoding": "bgr8"},
|
{"width": WIDTH, "height": HEIGHT, "encoding": "bgr8", **INTRINSICS_METADATA},
|
||||||
)
|
)
|
||||||
node.send_output(
|
node.send_output(
|
||||||
"bbox",
|
"bbox",
|
||||||
@@ -56,7 +73,13 @@ def main() -> None:
|
|||||||
node.send_output(
|
node.send_output(
|
||||||
"depth",
|
"depth",
|
||||||
pa.array(depth.ravel()),
|
pa.array(depth.ravel()),
|
||||||
{"shape": [HEIGHT, WIDTH], "dtype": "float32", "layout": "HW", "units": "mm"},
|
{
|
||||||
|
"shape": [HEIGHT, WIDTH],
|
||||||
|
"dtype": "float32",
|
||||||
|
"layout": "HW",
|
||||||
|
"units": "mm",
|
||||||
|
**INTRINSICS_METADATA,
|
||||||
|
},
|
||||||
)
|
)
|
||||||
node.send_output("text", pa.array([CAPTION]))
|
node.send_output("text", pa.array([CAPTION]))
|
||||||
|
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import numpy as np
|
|||||||
import pytest
|
import pytest
|
||||||
from PIL import Image
|
from PIL import Image
|
||||||
|
|
||||||
from dora_image_view import bbox, decode, depth
|
from dora_image_view import bbox, decode, depth, intrinsics
|
||||||
from dora_image_view.view import Bboxes, View
|
from dora_image_view.view import Bboxes, View
|
||||||
|
|
||||||
WIDTH, HEIGHT = 12, 8
|
WIDTH, HEIGHT = 12, 8
|
||||||
@@ -224,3 +224,62 @@ def test_depth_range_of_an_empty_frame_does_not_divide_by_zero():
|
|||||||
low, high = depth.resolve_range(np.zeros((2, 2), dtype=np.float32))
|
low, high = depth.resolve_range(np.zeros((2, 2), dtype=np.float32))
|
||||||
|
|
||||||
assert high > low
|
assert high > low
|
||||||
|
|
||||||
|
|
||||||
|
def test_camera_info_reads_the_compact_four_value_form():
|
||||||
|
values = np.array([600.0, 601.0, 320.0, 240.0], dtype=np.float32)
|
||||||
|
|
||||||
|
model = intrinsics.parse(FakeStorage(values), {"distortion": [0.1, 0.2]})
|
||||||
|
|
||||||
|
assert (model.fx, model.fy, model.cx, model.cy) == (600.0, 601.0, 320.0, 240.0)
|
||||||
|
assert model.distortion == (0.1, 0.2)
|
||||||
|
|
||||||
|
|
||||||
|
def test_camera_info_reads_a_ros_k_matrix():
|
||||||
|
# Row-major K: [fx, 0, cx, 0, fy, cy, 0, 0, 1].
|
||||||
|
k = np.array(
|
||||||
|
[600.0, 0.0, 320.0, 0.0, 601.0, 240.0, 0.0, 0.0, 1.0],
|
||||||
|
dtype=np.float32,
|
||||||
|
)
|
||||||
|
|
||||||
|
model = intrinsics.parse(FakeStorage(k), {})
|
||||||
|
|
||||||
|
assert (model.fx, model.fy, model.cx, model.cy) == (600.0, 601.0, 320.0, 240.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_camera_info_reinterprets_a_payload_sent_as_bytes():
|
||||||
|
values = np.array([600.0, 601.0, 320.0, 240.0], dtype=np.float32)
|
||||||
|
|
||||||
|
model = intrinsics.parse(FakeStorage(values.view(np.uint8)), {})
|
||||||
|
|
||||||
|
assert model.fx == 600.0
|
||||||
|
assert model.cy == 240.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_camera_info_that_is_too_short_is_an_error():
|
||||||
|
with pytest.raises(RuntimeError, match="camera_info"):
|
||||||
|
intrinsics.parse(FakeStorage(np.zeros(3, dtype=np.float32)), {})
|
||||||
|
|
||||||
|
|
||||||
|
def test_intrinsics_are_read_from_frame_metadata():
|
||||||
|
model = intrinsics.from_metadata(
|
||||||
|
{"fx": 600.0, "fy": 601.0, "cx": 320.0, "cy": 240.0},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert model is not None
|
||||||
|
assert (model.fx, model.cy) == (600.0, 240.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_frame_metadata_without_intrinsics_yields_none():
|
||||||
|
assert intrinsics.from_metadata({"width": 640, "height": 480}) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_camera_model_is_drawn_on_the_composite():
|
||||||
|
view = View(headless=True)
|
||||||
|
view.panels["image"] = np.zeros((40, 80, 3), dtype=np.uint8)
|
||||||
|
|
||||||
|
without = view.compose().copy()
|
||||||
|
view.intrinsics = intrinsics.Intrinsics(600.0, 601.0, 320.0, 240.0)
|
||||||
|
with_model = view.compose()
|
||||||
|
|
||||||
|
assert not np.array_equal(without, with_model), "the model should be drawn"
|
||||||
|
|||||||
Reference in New Issue
Block a user