add camera_info input for the pinhole model
Nothing in a dataflow carries fx, fy, cx and cy except the camera, and every pixel of a depth map needs them to become a 3D point. The node now accepts them so whatever computes that next has them to hand. They are read from two places because dora-zed publishes them twice: the camera_info message, and the metadata of every frame. The per-frame copy is not redundant -- camera_info goes out once per camera session, so a viewer started after the camera would otherwise never see it, and reading frame metadata means wiring the input is optional even for a camera that publishes it. A nine-element row-major K is accepted alongside the compact [fx, fy, cx, cy], since that is how ROS spells the same model in sensor_msgs/CameraInfo. The model is drawn under the caption rather than only stored. A depth map read through the wrong intrinsics looks entirely plausible, and this is where that becomes visible. as_float32 moves to a payload module: camera_info arrives as a raw byte slice for the same reason the depth map does, and the reinterpretation is now shared rather than written twice. Verified with dora-cli 1.0.1: validate passes with four contracts injected on each side, the dataflow runs to a clean exit, and a composed frame was rendered and inspected with the model drawn on it.
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
# dora-image-view
|
||||
|
||||
Dora node that displays a colour image and a depth map side by side in one OpenCV
|
||||
window, with bounding box and text overlays. A display sink for debugging a
|
||||
window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a
|
||||
vision pipeline: it produces no outputs.
|
||||
|
||||
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
|
||||
@@ -21,6 +21,7 @@ nodes:
|
||||
inputs:
|
||||
image: camera/image_bgr
|
||||
depth: camera/depth
|
||||
camera_info: camera/camera_info
|
||||
bbox: detector/bbox
|
||||
text: detector/text
|
||||
```
|
||||
@@ -30,7 +31,7 @@ managed environment.
|
||||
|
||||
Every input is optional, and each of `image` and `depth` fills one panel of the window,
|
||||
tiled left to right in that order. Wiring one of them is a plain viewer;
|
||||
wiring several shows them side by side, scaled to a common height. `bbox` and `text`
|
||||
wiring several shows them side by side, scaled to a common height. `bbox`, `text` and `camera_info`
|
||||
are overlays rather than panels — boxes are drawn on the `image` panel, in its own
|
||||
pixels, and the caption on the composite.
|
||||
|
||||
@@ -75,8 +76,8 @@ $ dora validate dataflow.yml
|
||||
All type annotations OK.
|
||||
```
|
||||
|
||||
`image` is `std/core/v1/UInt8`, `depth` is `std/core/v1/Float32` and `text` is
|
||||
`std/core/v1/String`; the payloads are raw buffers described by their
|
||||
`image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and
|
||||
`text` is `std/core/v1/String`; the payloads are raw buffers described by their
|
||||
metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
|
||||
left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
|
||||
node reads either.
|
||||
@@ -91,6 +92,7 @@ of that state:
|
||||
resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
|
||||
`Dora Node: dora-image-view`. Press `q` in the window to quit.
|
||||
- A **`depth`** event colour-maps the distances onto the depth panel and redraws.
|
||||
- A **`camera_info`** event stores the pinhole model and draws it under the caption.
|
||||
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
|
||||
redraw.
|
||||
|
||||
@@ -103,6 +105,7 @@ no window. That is what makes it safe to run in a pipeline test.
|
||||
|---|---|---|
|
||||
| `image` | Flat `UInt8` array | `width`, `height`, `encoding` |
|
||||
| `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` |
|
||||
| `camera_info` | Flat `Float32` array, `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` |
|
||||
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
|
||||
| `text` | Arrow array of size 1 | — |
|
||||
|
||||
@@ -117,6 +120,23 @@ and any non-finite value mean missing depth and are drawn black. A payload sent
|
||||
raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
|
||||
`Float32` contract, and is reinterpreted rather than converted.
|
||||
|
||||
### Camera intrinsics
|
||||
|
||||
`camera_info` carries the pinhole model the depth map is interpreted through, either as
|
||||
the four values `[fx, fy, cx, cy]` or as a nine-element row-major `K`, which is how ROS
|
||||
spells the same model in `sensor_msgs/CameraInfo`. Distortion coefficients are read from
|
||||
the `distortion` metadata.
|
||||
|
||||
The node displays the model and holds it. It needs none of it to draw a panel; it is
|
||||
there for what reads it next — turning a depth map into a point cloud needs `fx`, `fy`,
|
||||
`cx` and `cy` for every pixel, and nothing else in a dataflow carries them.
|
||||
|
||||
Intrinsics are also picked up from the metadata of any `image` or `depth` frame that
|
||||
carries `fx`, `fy`, `cx` and `cy`, so **wiring `camera_info` is optional even for a
|
||||
camera that publishes it**. That matters because a camera typically publishes
|
||||
`camera_info` once per session: a viewer started after the camera would otherwise never
|
||||
see it.
|
||||
|
||||
```python
|
||||
node.send_output("image", pa.array(frame.ravel()),
|
||||
{"width": 640, "height": 480, "encoding": "bgr8"})
|
||||
|
||||
Reference in New Issue
Block a user