add depth and camera intrinsics, and adopt the typed Arrow image #4
@@ -4,7 +4,7 @@ This file provides guidance to AI coding assistants when working with code in th
|
||||
|
||||
## What this is
|
||||
|
||||
A dora-rs Python node that displays a video stream with overlays: it keeps one panel per frame stream (`image`, `depth`) plus a bbox set and text string, and every frame event redraws the lot as one side-by-side composite in an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say.
|
||||
A dora-rs Python node that displays a video stream with overlays: it keeps one panel per frame stream (`image`, `depth`) plus a bbox set, a text string and the camera model, and every frame event redraws the lot as one side-by-side composite in an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say.
|
||||
|
||||
Everything the example dataflow needs is in this checkout. Do not reintroduce a dependency on another repository — the example must keep running with no camera, no detector and no network.
|
||||
|
||||
@@ -26,7 +26,9 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless
|
||||
- **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node.
|
||||
- **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
|
||||
- **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one.
|
||||
- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. Two things are deliberate: `as_float32()` **reinterprets** a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`; and the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
|
||||
- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. One thing is deliberate: the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
|
||||
- **`dora_image_view/payload.py`** — `as_float32()`: reinterprets a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`. Shared by `depth` and `intrinsics`, which both meet it.
|
||||
- **`dora_image_view/intrinsics.py`** — `Intrinsics`, `parse()` for the `camera_info` payload and `from_metadata()` for the copy senders attach to each frame. Both exist because dora-zed publishes the same numbers twice, and the per-frame copy is the one that survives a viewer starting after the camera did — `camera_info` goes out **once per session**. A nine-element row-major K is accepted alongside the compact four values, since that is how ROS spells the same model. The node needs none of this to draw; it is held for whatever turns depth into 3D, and drawn so a wrong camera model is visible rather than merely plausible.
|
||||
- **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is.
|
||||
- **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout.
|
||||
- **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either.
|
||||
@@ -42,7 +44,7 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless
|
||||
|
||||
## A caveat on typed optional inputs
|
||||
|
||||
`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map at all: `depth` is typed, so the example has to drive it.
|
||||
`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map and a camera_info at all: both are typed, so the example has to drive them.
|
||||
|
||||
## Checking what it draws
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# dora-image-view
|
||||
|
||||
Dora node that displays a colour image and a depth map side by side in one OpenCV
|
||||
window, with bounding box and text overlays. A display sink for debugging a
|
||||
window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a
|
||||
vision pipeline: it produces no outputs.
|
||||
|
||||
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
|
||||
@@ -21,6 +21,7 @@ nodes:
|
||||
inputs:
|
||||
image: camera/image_bgr
|
||||
depth: camera/depth
|
||||
camera_info: camera/camera_info
|
||||
bbox: detector/bbox
|
||||
text: detector/text
|
||||
```
|
||||
@@ -30,7 +31,7 @@ managed environment.
|
||||
|
||||
Every input is optional, and each of `image` and `depth` fills one panel of the window,
|
||||
tiled left to right in that order. Wiring one of them is a plain viewer;
|
||||
wiring several shows them side by side, scaled to a common height. `bbox` and `text`
|
||||
wiring several shows them side by side, scaled to a common height. `bbox`, `text` and `camera_info`
|
||||
are overlays rather than panels — boxes are drawn on the `image` panel, in its own
|
||||
pixels, and the caption on the composite.
|
||||
|
||||
@@ -75,8 +76,8 @@ $ dora validate dataflow.yml
|
||||
All type annotations OK.
|
||||
```
|
||||
|
||||
`image` is `std/core/v1/UInt8`, `depth` is `std/core/v1/Float32` and `text` is
|
||||
`std/core/v1/String`; the payloads are raw buffers described by their
|
||||
`image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and
|
||||
`text` is `std/core/v1/String`; the payloads are raw buffers described by their
|
||||
metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
|
||||
left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
|
||||
node reads either.
|
||||
@@ -91,6 +92,7 @@ of that state:
|
||||
resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
|
||||
`Dora Node: dora-image-view`. Press `q` in the window to quit.
|
||||
- A **`depth`** event colour-maps the distances onto the depth panel and redraws.
|
||||
- A **`camera_info`** event stores the pinhole model and draws it under the caption.
|
||||
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
|
||||
redraw.
|
||||
|
||||
@@ -103,6 +105,7 @@ no window. That is what makes it safe to run in a pipeline test.
|
||||
|---|---|---|
|
||||
| `image` | Flat `UInt8` array | `width`, `height`, `encoding` |
|
||||
| `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` |
|
||||
| `camera_info` | Flat `Float32` array, `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` |
|
||||
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
|
||||
| `text` | Arrow array of size 1 | — |
|
||||
|
||||
@@ -117,6 +120,23 @@ and any non-finite value mean missing depth and are drawn black. A payload sent
|
||||
raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
|
||||
`Float32` contract, and is reinterpreted rather than converted.
|
||||
|
||||
### Camera intrinsics
|
||||
|
||||
`camera_info` carries the pinhole model the depth map is interpreted through, either as
|
||||
the four values `[fx, fy, cx, cy]` or as a nine-element row-major `K`, which is how ROS
|
||||
spells the same model in `sensor_msgs/CameraInfo`. Distortion coefficients are read from
|
||||
the `distortion` metadata.
|
||||
|
||||
The node displays the model and holds it. It needs none of it to draw a panel; it is
|
||||
there for what reads it next — turning a depth map into a point cloud needs `fx`, `fy`,
|
||||
`cx` and `cy` for every pixel, and nothing else in a dataflow carries them.
|
||||
|
||||
Intrinsics are also picked up from the metadata of any `image` or `depth` frame that
|
||||
carries `fx`, `fy`, `cx` and `cy`, so **wiring `camera_info` is optional even for a
|
||||
camera that publishes it**. That matters because a camera typically publishes
|
||||
`camera_info` once per session: a viewer started after the camera would otherwise never
|
||||
see it.
|
||||
|
||||
```python
|
||||
node.send_output("image", pa.array(frame.ravel()),
|
||||
{"width": 640, "height": 480, "encoding": "bgr8"})
|
||||
|
||||
@@ -20,6 +20,7 @@ nodes:
|
||||
outputs:
|
||||
- image
|
||||
- depth
|
||||
- camera_info
|
||||
- bbox
|
||||
- text
|
||||
|
||||
@@ -29,5 +30,6 @@ nodes:
|
||||
inputs:
|
||||
image: source/image
|
||||
depth: source/depth
|
||||
camera_info: source/camera_info
|
||||
bbox: source/bbox
|
||||
text: source/text
|
||||
|
||||
+14
-2
@@ -7,9 +7,9 @@
|
||||
apiVersion: 1
|
||||
name: dora-image-view
|
||||
namespace: liria
|
||||
description: Displays an image and a depth map side by side in an OpenCV window, with bounding box and text overlays
|
||||
description: Displays an image and a depth map side by side in an OpenCV window, with bounding box, text and camera intrinsics overlays
|
||||
categories: [visualization, debug]
|
||||
keywords: [opencv, view, bbox, bounding-box, image, depth, visualization, object-detection]
|
||||
keywords: [opencv, view, bbox, bounding-box, image, depth, intrinsics, camera-info, visualization, object-detection]
|
||||
|
||||
runtime: python
|
||||
entrypoint: dora-image-view
|
||||
@@ -41,6 +41,17 @@ inputs:
|
||||
description: >-
|
||||
Text string to overlay at the top-left corner. Stored and drawn on the
|
||||
next image redraw.
|
||||
camera_info:
|
||||
type: std/core/v1/Float32
|
||||
required: false
|
||||
description: >-
|
||||
The pinhole model the depth map is interpreted through: [fx, fy, cx, cy],
|
||||
or a nine-element row-major K as ROS spells it in sensor_msgs/CameraInfo.
|
||||
Distortion coefficients are read from the `distortion` metadata. Stored
|
||||
and displayed, and held for readers that turn depth into 3D. A sender
|
||||
that also puts fx/fy/cx/cy in each frame's metadata does not need to wire
|
||||
this input -- they are picked up from there, which is what covers a
|
||||
camera that published its intrinsics before this node started.
|
||||
depth:
|
||||
type: std/core/v1/Float32
|
||||
required: false
|
||||
@@ -77,5 +88,6 @@ example: |
|
||||
inputs:
|
||||
image: camera/image_bgr
|
||||
depth: camera/depth
|
||||
camera_info: camera/camera_info
|
||||
bbox: detector/bbox
|
||||
text: detector/text
|
||||
|
||||
@@ -11,6 +11,8 @@ import os
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
from dora_image_view.payload import as_float32
|
||||
|
||||
# Perceptually uniform and readable in both directions, unlike JET.
|
||||
COLORMAP = cv2.COLORMAP_TURBO
|
||||
|
||||
@@ -85,21 +87,6 @@ def resolve_range(depth_mm: np.ndarray) -> tuple[float, float]:
|
||||
return float(low), float(high)
|
||||
|
||||
|
||||
def as_float32(storage) -> np.ndarray:
|
||||
"""Read the payload as float32, whether it arrived as floats or as bytes.
|
||||
|
||||
A node that sends the map as a raw byte slice -- dora-zed does -- produces a
|
||||
UInt8 arrow array even though its manifest declares Float32, so the buffer
|
||||
has to be reinterpreted rather than converted.
|
||||
"""
|
||||
buffer = storage.to_numpy()
|
||||
|
||||
if buffer.dtype == np.uint8:
|
||||
return buffer.view(np.float32)
|
||||
|
||||
return buffer.astype(np.float32, copy=False)
|
||||
|
||||
|
||||
def _hw(metadata: dict, size: int) -> tuple[int, int]:
|
||||
shape = metadata.get("shape")
|
||||
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
"""The pinhole camera model a depth map is interpreted through.
|
||||
|
||||
The node does not need intrinsics to draw anything -- it displays them, and
|
||||
holds them for whatever reads them next. Back-projecting a depth map into a
|
||||
point cloud is the obvious such reader: every pixel needs fx, fy, cx and cy to
|
||||
become an XYZ point, and nothing else in a dataflow carries them.
|
||||
|
||||
Two senders are handled because dora-zed publishes the same numbers twice: once
|
||||
per camera session on `camera_info`, and again in the metadata of every frame.
|
||||
The per-frame copy is what makes a viewer that started after the camera did
|
||||
still end up with intrinsics.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from dora_image_view.payload import as_float32
|
||||
|
||||
# [fx, fy, cx, cy], the four numbers dora-zed publishes. Also accepted is a
|
||||
# nine-element row-major K, which is how ROS spells the same model in
|
||||
# sensor_msgs/CameraInfo.
|
||||
COMPACT = 4
|
||||
K_MATRIX = 9
|
||||
|
||||
# Indices of fx, fy, cx, cy in a row-major K:
|
||||
# [fx, 0, cx,
|
||||
# 0, fy, cy,
|
||||
# 0, 0, 1]
|
||||
K_INDICES = (0, 4, 2, 5)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Intrinsics:
|
||||
"""A pinhole model: focal lengths and principal point, both in pixels."""
|
||||
|
||||
fx: float
|
||||
fy: float
|
||||
cx: float
|
||||
cy: float
|
||||
distortion: tuple[float, ...] = ()
|
||||
model: str = ""
|
||||
|
||||
def __str__(self) -> str:
|
||||
"""Render for the overlay, short enough to fit across a panel."""
|
||||
return f"fx {self.fx:.1f} fy {self.fy:.1f} cx {self.cx:.1f} cy {self.cy:.1f}"
|
||||
|
||||
|
||||
def parse(storage, metadata: dict) -> Intrinsics:
|
||||
"""Read one `camera_info` payload.
|
||||
|
||||
Accepts the four-element [fx, fy, cx, cy] dora-zed sends and the
|
||||
nine-element row-major K that ROS does.
|
||||
"""
|
||||
values = as_float32(storage)
|
||||
|
||||
if values.size >= K_MATRIX:
|
||||
fx, fy, cx, cy = (float(values[index]) for index in K_INDICES)
|
||||
elif values.size >= COMPACT:
|
||||
fx, fy, cx, cy = (float(value) for value in values[:COMPACT])
|
||||
else:
|
||||
raise RuntimeError(
|
||||
f"camera_info holds {values.size} values, "
|
||||
f"expected {COMPACT} as [fx, fy, cx, cy] or {K_MATRIX} as K",
|
||||
)
|
||||
|
||||
return Intrinsics(fx, fy, cx, cy, _distortion(metadata), _model(metadata))
|
||||
|
||||
|
||||
def from_metadata(metadata: dict) -> Intrinsics | None:
|
||||
"""Read intrinsics a sender attached to a frame, or None if it attached none."""
|
||||
try:
|
||||
fx, fy, cx, cy = (
|
||||
float(metadata["fx"]),
|
||||
float(metadata["fy"]),
|
||||
float(metadata["cx"]),
|
||||
float(metadata["cy"]),
|
||||
)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
return Intrinsics(fx, fy, cx, cy, _distortion(metadata), _model(metadata))
|
||||
|
||||
|
||||
def _distortion(metadata: dict) -> tuple[float, ...]:
|
||||
values = metadata.get("distortion") or ()
|
||||
return tuple(float(value) for value in values)
|
||||
|
||||
|
||||
def _model(metadata: dict) -> str:
|
||||
return str(metadata.get("distortion_model", ""))
|
||||
+11
-3
@@ -1,7 +1,8 @@
|
||||
"""A dora display sink: overlays bounding boxes and text on an image input.
|
||||
|
||||
`image` and `depth` each fill one panel of the window and redraw. `bbox` and
|
||||
`text` events only update stored state, which the next redraw draws.
|
||||
`image` and `depth` each fill one panel of the window and redraw. `bbox`,
|
||||
`text` and `camera_info` events only update stored state, which the next redraw
|
||||
draws.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -10,7 +11,7 @@ import os
|
||||
import numpy as np
|
||||
from dora import Node
|
||||
|
||||
from dora_image_view import bbox, decode, depth
|
||||
from dora_image_view import bbox, decode, depth, intrinsics
|
||||
from dora_image_view.view import View
|
||||
|
||||
|
||||
@@ -37,6 +38,11 @@ def main() -> None:
|
||||
event_id = event["id"]
|
||||
metadata = event["metadata"]
|
||||
|
||||
# A sender that puts intrinsics on every frame keeps them current even
|
||||
# if the camera_info message itself was published before this node
|
||||
# started, which is when dora-zed sends it.
|
||||
view.intrinsics = intrinsics.from_metadata(metadata) or view.intrinsics
|
||||
|
||||
panel = _decode_panel(event_id, event["value"], metadata)
|
||||
|
||||
if panel is not None:
|
||||
@@ -44,6 +50,8 @@ def main() -> None:
|
||||
view.panels[name] = frame
|
||||
if view.show():
|
||||
break
|
||||
elif event_id == "camera_info":
|
||||
view.intrinsics = intrinsics.parse(event["value"], metadata)
|
||||
elif event_id == "bbox":
|
||||
view.bboxes = bbox.parse(event["value"], metadata.get("format", "xyxy").lower())
|
||||
elif event_id == "text":
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
"""Reading raw dora payloads, independently of what they mean."""
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
def as_float32(storage) -> np.ndarray:
|
||||
"""Read a payload as float32, whether it arrived as floats or as bytes.
|
||||
|
||||
A node that sends its payload as a raw byte slice -- which is what a C++
|
||||
node does -- produces a UInt8 arrow array even though its manifest declares
|
||||
Float32, so the buffer has to be reinterpreted rather than converted.
|
||||
"""
|
||||
buffer = storage.to_numpy()
|
||||
|
||||
if buffer.dtype == np.uint8:
|
||||
return buffer.view(np.float32)
|
||||
|
||||
return buffer.astype(np.float32, copy=False)
|
||||
@@ -5,6 +5,8 @@ from dataclasses import dataclass, field
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
from dora_image_view.intrinsics import Intrinsics
|
||||
|
||||
WINDOW = "Dora Node: dora-image-view"
|
||||
GREEN = (0, 255, 0)
|
||||
WHITE = (255, 255, 255)
|
||||
@@ -40,6 +42,7 @@ class View:
|
||||
bboxes: Bboxes = field(default_factory=Bboxes)
|
||||
text: str = ""
|
||||
panels: dict[str, np.ndarray] = field(default_factory=dict)
|
||||
intrinsics: Intrinsics | None = None
|
||||
|
||||
def show(self) -> bool:
|
||||
"""Draw the overlays, tile the panels, and display them.
|
||||
@@ -79,6 +82,13 @@ class View:
|
||||
# On the composite, not on a panel: the caption describes the frame as a
|
||||
# whole, and a panel it was drawn on might not be the leftmost one.
|
||||
cv2.putText(frame, self.text, (20, 20), FONT, FONT_SCALE, WHITE, 1)
|
||||
|
||||
# Showing the camera model is the point of receiving it here: a depth
|
||||
# map read through the wrong intrinsics looks entirely plausible, and
|
||||
# this is where that becomes visible.
|
||||
if self.intrinsics is not None:
|
||||
cv2.putText(frame, str(self.intrinsics), (20, 40), FONT, FONT_SCALE, WHITE, 1)
|
||||
|
||||
return frame
|
||||
|
||||
def _label(self, panel: np.ndarray, name: str) -> np.ndarray:
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
apiVersion: 1
|
||||
name: source
|
||||
namespace: liria
|
||||
description: Synthetic image, depth, bbox and text source for the dora-image-view example
|
||||
description: Synthetic image, depth, camera_info, bbox and text source for the dora-image-view example
|
||||
categories: [debug]
|
||||
keywords: [example, test, synthetic]
|
||||
|
||||
@@ -39,6 +39,11 @@ outputs:
|
||||
text:
|
||||
type: std/core/v1/String
|
||||
description: A fixed caption.
|
||||
camera_info:
|
||||
type: std/core/v1/Float32
|
||||
description: >-
|
||||
[fx, fy, cx, cy] for the frames above. Published once, on the first tick,
|
||||
the way a real camera publishes it once per session.
|
||||
depth:
|
||||
type: std/core/v1/Float32
|
||||
description: >-
|
||||
|
||||
+27
-4
@@ -5,7 +5,9 @@ moving in it, and a caption -- every input dora-image-view draws -- so the examp
|
||||
dataflow needs no camera, no detector and no network.
|
||||
|
||||
The depth map describes the same scene as the image: a flat back wall with the square
|
||||
standing in front of it.
|
||||
standing in front of it. `camera_info` is published once, on the first tick, the way a
|
||||
real camera publishes it once per session -- which is exactly the case the viewer has
|
||||
to cope with by reading intrinsics from frame metadata as well.
|
||||
"""
|
||||
|
||||
import os
|
||||
@@ -27,9 +29,17 @@ CAPTION = "dora-image-view synthetic source"
|
||||
WALL_MM = 2000.0
|
||||
SQUARE_MM = 800.0
|
||||
|
||||
# A plausible pinhole for a frame this size.
|
||||
FX = FY = float(WIDTH)
|
||||
CX, CY = WIDTH / 2, HEIGHT / 2
|
||||
DISTORTION = [0.0, 0.0, 0.0, 0.0, 0.0]
|
||||
# Attached to every frame as well as published once, which is what dora-zed does
|
||||
# and what lets a viewer that started late still learn the camera model.
|
||||
INTRINSICS_METADATA = {"fx": FX, "fy": FY, "cx": CX, "cy": CY, "distortion": DISTORTION}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Send one image, depth map, bbox and text per tick."""
|
||||
"""Send one image, depth map, bbox and text per tick, and camera_info once."""
|
||||
node = Node("source")
|
||||
sent = 0
|
||||
|
||||
@@ -37,11 +47,18 @@ def main() -> None:
|
||||
if event["type"] != "INPUT":
|
||||
continue
|
||||
|
||||
if sent == 0:
|
||||
node.send_output(
|
||||
"camera_info",
|
||||
pa.array(np.array([FX, FY, CX, CY], dtype=np.float32)),
|
||||
{"shape": [4], "dtype": "float32", "layout": "C", "distortion": DISTORTION},
|
||||
)
|
||||
|
||||
frame, corners = render(sent)
|
||||
node.send_output(
|
||||
"image",
|
||||
pa.array(frame.ravel()),
|
||||
{"width": WIDTH, "height": HEIGHT, "encoding": "bgr8"},
|
||||
{"width": WIDTH, "height": HEIGHT, "encoding": "bgr8", **INTRINSICS_METADATA},
|
||||
)
|
||||
node.send_output(
|
||||
"bbox",
|
||||
@@ -56,7 +73,13 @@ def main() -> None:
|
||||
node.send_output(
|
||||
"depth",
|
||||
pa.array(depth.ravel()),
|
||||
{"shape": [HEIGHT, WIDTH], "dtype": "float32", "layout": "HW", "units": "mm"},
|
||||
{
|
||||
"shape": [HEIGHT, WIDTH],
|
||||
"dtype": "float32",
|
||||
"layout": "HW",
|
||||
"units": "mm",
|
||||
**INTRINSICS_METADATA,
|
||||
},
|
||||
)
|
||||
node.send_output("text", pa.array([CAPTION]))
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ import numpy as np
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from dora_image_view import bbox, decode, depth
|
||||
from dora_image_view import bbox, decode, depth, intrinsics
|
||||
from dora_image_view.view import Bboxes, View
|
||||
|
||||
WIDTH, HEIGHT = 12, 8
|
||||
@@ -224,3 +224,62 @@ def test_depth_range_of_an_empty_frame_does_not_divide_by_zero():
|
||||
low, high = depth.resolve_range(np.zeros((2, 2), dtype=np.float32))
|
||||
|
||||
assert high > low
|
||||
|
||||
|
||||
def test_camera_info_reads_the_compact_four_value_form():
|
||||
values = np.array([600.0, 601.0, 320.0, 240.0], dtype=np.float32)
|
||||
|
||||
model = intrinsics.parse(FakeStorage(values), {"distortion": [0.1, 0.2]})
|
||||
|
||||
assert (model.fx, model.fy, model.cx, model.cy) == (600.0, 601.0, 320.0, 240.0)
|
||||
assert model.distortion == (0.1, 0.2)
|
||||
|
||||
|
||||
def test_camera_info_reads_a_ros_k_matrix():
|
||||
# Row-major K: [fx, 0, cx, 0, fy, cy, 0, 0, 1].
|
||||
k = np.array(
|
||||
[600.0, 0.0, 320.0, 0.0, 601.0, 240.0, 0.0, 0.0, 1.0],
|
||||
dtype=np.float32,
|
||||
)
|
||||
|
||||
model = intrinsics.parse(FakeStorage(k), {})
|
||||
|
||||
assert (model.fx, model.fy, model.cx, model.cy) == (600.0, 601.0, 320.0, 240.0)
|
||||
|
||||
|
||||
def test_camera_info_reinterprets_a_payload_sent_as_bytes():
|
||||
values = np.array([600.0, 601.0, 320.0, 240.0], dtype=np.float32)
|
||||
|
||||
model = intrinsics.parse(FakeStorage(values.view(np.uint8)), {})
|
||||
|
||||
assert model.fx == 600.0
|
||||
assert model.cy == 240.0
|
||||
|
||||
|
||||
def test_camera_info_that_is_too_short_is_an_error():
|
||||
with pytest.raises(RuntimeError, match="camera_info"):
|
||||
intrinsics.parse(FakeStorage(np.zeros(3, dtype=np.float32)), {})
|
||||
|
||||
|
||||
def test_intrinsics_are_read_from_frame_metadata():
|
||||
model = intrinsics.from_metadata(
|
||||
{"fx": 600.0, "fy": 601.0, "cx": 320.0, "cy": 240.0},
|
||||
)
|
||||
|
||||
assert model is not None
|
||||
assert (model.fx, model.cy) == (600.0, 240.0)
|
||||
|
||||
|
||||
def test_frame_metadata_without_intrinsics_yields_none():
|
||||
assert intrinsics.from_metadata({"width": 640, "height": 480}) is None
|
||||
|
||||
|
||||
def test_the_camera_model_is_drawn_on_the_composite():
|
||||
view = View(headless=True)
|
||||
view.panels["image"] = np.zeros((40, 80, 3), dtype=np.uint8)
|
||||
|
||||
without = view.compose().copy()
|
||||
view.intrinsics = intrinsics.Intrinsics(600.0, 601.0, 320.0, 240.0)
|
||||
with_model = view.compose()
|
||||
|
||||
assert not np.array_equal(without, with_model), "the model should be drawn"
|
||||
|
||||
Reference in New Issue
Block a user