cristhian
8c3b3aac92
validate / validate (pull_request) Successful in 59m14s
It had drifted out of agreement with itself. The manifest section still said `image` is std/core/v1/UInt8 and depth and camera_info are std/core/v1/Float32, four lines above a table correctly saying both are std/media/v1/Image, and the worked example for a producer showed the flat-buffer form the node moved away from. Someone following it would have written a producer that fails validate. It was also close to twice the length of the one next door, mostly rationale: why a dataflow carries no strict_types, how a producer migrates, what a git: node needs and why. That belongs in the file it explains or in the commit that made the decision, not in front of someone trying to wire a node. 1007 words, from 1749. Every encoding, environment variable and CLI flag in it was checked against the code.
169 lines
6.6 KiB
Markdown
169 lines
6.6 KiB
Markdown
# dora-image-view
|
|
|
|
Dora node that displays a colour image and a depth map side by side in one OpenCV
|
|
window, with bounding box, text and camera intrinsics overlays. A display sink for
|
|
debugging a vision pipeline: it consumes frames and produces no outputs.
|
|
|
|
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
|
|
from dora-hub, brought up to the dora 1.0 node contract, renamed, and changed to take
|
|
the typed Arrow image struct rather than a flat buffer plus metadata.
|
|
|
|
## Usage
|
|
|
|
Add to your dataflow:
|
|
|
|
```yaml
|
|
nodes:
|
|
- id: view
|
|
git: https://gitea.liria.cl/liria/dora-image-view.git
|
|
tag: v0.1.1
|
|
build: uv pip install .
|
|
path: dora-image-view
|
|
inputs:
|
|
image: camera/image
|
|
depth: camera/depth
|
|
camera_info: camera/camera_info
|
|
bbox: detector/bbox
|
|
text: detector/text
|
|
```
|
|
|
|
Run the dataflow with `--uv` (`dora run dataflow.yml --uv`): the build command is
|
|
`uv pip install .`, so the node gets its own managed environment and never installs into
|
|
an ambient one.
|
|
|
|
Every input is optional. `image` and `depth` each fill a panel, tiled left to right and
|
|
scaled to a common height, so wiring one is a plain viewer and wiring both shows them
|
|
side by side. `bbox`, `text` and `camera_info` are overlays rather than panels: boxes on
|
|
the image panel in its own pixels, caption and camera model on the composite.
|
|
|
|
### Runnable dataflows
|
|
|
|
Run either with `dora run <file> --uv`:
|
|
|
|
| File | Contents |
|
|
|---|---|
|
|
| [`dataflow.yml`](dataflow.yml) | A synthetic source driving every input. What CI runs. |
|
|
| [`dataflow_webcam.yml`](dataflow_webcam.yml) | A real camera, via [dora-webcam][webcam] |
|
|
|
|
[webcam]: https://gitea.liria.cl/liria/dora-webcam
|
|
|
|
`dataflow.yml` needs no camera, no detector and no network: a window opens with a square
|
|
bouncing across a gradient, boxed and labelled, beside its depth map. Press `q` to quit.
|
|
CI runs it headless on every push, which is what keeps it from rotting.
|
|
|
|
`dataflow_webcam.yml` is the worked example of consuming a real producer. A `git:` node
|
|
needs **anonymous** read access to fetch — on Gitea that means a public organization, not
|
|
just a public repository.
|
|
|
|
## Inputs
|
|
|
|
| Input | Payload | Metadata |
|
|
|---|---|---|
|
|
| `image` | `std/media/v1/Image` struct | none — the struct describes itself |
|
|
| `depth` | `std/media/v1/Image` struct, one channel | `units` (defaults to `mm`) |
|
|
| `camera_info` | `[fx, fy, cx, cy]` or a 9-element row-major `K` | `distortion` |
|
|
| `bbox` | Arrow struct `{bbox, conf, labels}` | `format`: `xyxy` or `xywh` |
|
|
| `text` | Arrow array of size 1 | — |
|
|
|
|
`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`,
|
|
`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself,
|
|
so nothing about its shape travels as message metadata.
|
|
|
|
Image encodings: `bgr8`, `rgb8`, `yuv420`, `avif`, and `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/
|
|
`png` as an encoded byte stream. Depth encodings: `32FC1`/`float32` and
|
|
`16UC1`/`mono16`/`uint16`, one channel per pixel — dora's registry has no depth type, and
|
|
a depth map is an image whose pixels are distances, as ROS also treats it. `0` and
|
|
non-finite values mean missing depth and draw black.
|
|
|
|
`camera_info` is also read from the metadata of any frame carrying `fx`, `fy`, `cx` and
|
|
`cy`, so **wiring it is optional even for a camera that publishes it** — which covers a
|
|
camera that publishes it once per session and a viewer started afterwards. The node needs
|
|
none of it to draw; it is held for whatever turns depth into 3D.
|
|
|
|
Older payloads are still decoded, so a producer can move at its own pace: a flat `UInt8`
|
|
buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32` buffer
|
|
with `shape`/`units` for `depth`. Note that dora's type compatibility is directional, so
|
|
a producer still *declaring* `std/core/v1/UInt8` fails `dora validate` against an
|
|
`std/media/v1/Image` input whatever it sends — its manifest has to move too.
|
|
|
|
Publishing to it:
|
|
|
|
```python
|
|
image = pa.array(
|
|
[{"width": 640, "height": 480, "encoding": "bgr8", "data": frame.tobytes()}],
|
|
type=pa.struct([
|
|
("width", pa.uint32()), ("height", pa.uint32()),
|
|
("encoding", pa.utf8()), ("data", pa.large_binary()),
|
|
]),
|
|
)
|
|
node.send_output("image", image, {})
|
|
|
|
node.send_output("bbox", pa.array([{
|
|
"bbox": np.array([...]), # flattened, four values per box
|
|
"conf": np.array([...]),
|
|
"labels": np.array([...]),
|
|
}]), {"format": "xyxy"})
|
|
|
|
node.send_output("text", pa.array(["hello"]))
|
|
```
|
|
|
|
## Outputs
|
|
|
|
None — it is a display sink.
|
|
|
|
## Configuration
|
|
|
|
Settings are environment variables, which a dataflow can carry per node, keeping them
|
|
with the wiring rather than in whatever shell starts it:
|
|
|
|
```yaml
|
|
env:
|
|
VIEW_DEPTH_MIN_MM: 200
|
|
VIEW_DEPTH_MAX_MM: 600
|
|
```
|
|
|
|
A node's `env:` wins over the environment the dataflow was started in; variables it does
|
|
not set still pass through.
|
|
|
|
| Variable | Default | Meaning |
|
|
|---|---|---|
|
|
| `VIEW_WIDTH` | composite width | Resize the displayed composite to this width, in pixels. |
|
|
| `VIEW_HEIGHT` | composite height | Resize the displayed composite to this height, in pixels. |
|
|
| `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. |
|
|
| `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. |
|
|
|
|
Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set; both are also
|
|
available as `--view-width` and `--view-height`, and the environment variable wins.
|
|
|
|
Each end of the depth range is auto-ranged over the current frame's valid pixels unless
|
|
pinned, so pinning one end leaves the other automatic. Auto-ranging makes an
|
|
unconfigured sensor legible at the cost of colours that shift between frames; pin both
|
|
ends to compare frames against each other.
|
|
|
|
`CI=true` suppresses the window while still decoding every input, which is what makes the
|
|
node safe inside a pipeline test. It is not a configuration knob.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
uv sync
|
|
```
|
|
|
|
Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64,
|
|
so the node installs with no system packages and no compiler — including AVIF support,
|
|
which comes from `pillow-avif-plugin` rather than OpenCV.
|
|
|
|
**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), Python 3.11+, `uv`, and a
|
|
display (or `CI=true` to run without one). On a bare Linux image `opencv-python` needs
|
|
the libGL shared libraries (`libgl1`, `libglib2.0-0`), which it links at import time.
|
|
|
|
## Test
|
|
|
|
```bash
|
|
uv run pytest
|
|
```
|
|
|
|
Unit tests over decoding, depth, intrinsics, the bbox conversion and the headless draw
|
|
path; they need neither a camera, a dataflow nor a display. The headless `dataflow.yml`
|
|
run covers the rest — the manifest, the wiring and the event loop.
|