Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 cristhian
|
8c3b3aac92 |
@@ -1,11 +1,12 @@
|
||||
# dora-image-view
|
||||
|
||||
Dora node that displays a colour image and a depth map side by side in one OpenCV
|
||||
window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a
|
||||
vision pipeline: it produces no outputs.
|
||||
window, with bounding box, text and camera intrinsics overlays. A display sink for
|
||||
debugging a vision pipeline: it consumes frames and produces no outputs.
|
||||
|
||||
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
|
||||
from dora-hub, brought up to the dora 1.0 node contract and packaged with `uv`.
|
||||
from dora-hub, brought up to the dora 1.0 node contract, renamed, and changed to take
|
||||
the typed Arrow image struct rather than a flat buffer plus metadata.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -15,126 +16,44 @@ Add to your dataflow:
|
||||
nodes:
|
||||
- id: view
|
||||
git: https://gitea.liria.cl/liria/dora-image-view.git
|
||||
tag: v0.1.0
|
||||
tag: v0.1.1
|
||||
build: uv pip install .
|
||||
path: dora-image-view
|
||||
inputs:
|
||||
image: camera/image_bgr
|
||||
image: camera/image
|
||||
depth: camera/depth
|
||||
camera_info: camera/camera_info
|
||||
bbox: detector/bbox
|
||||
text: detector/text
|
||||
```
|
||||
|
||||
Run the dataflow with `--uv` (`dora run dataflow.yml --uv`) so the node gets its own
|
||||
managed environment.
|
||||
Run the dataflow with `--uv` (`dora run dataflow.yml --uv`): the build command is
|
||||
`uv pip install .`, so the node gets its own managed environment and never installs into
|
||||
an ambient one.
|
||||
|
||||
Every input is optional, and each of `image` and `depth` fills one panel of the window,
|
||||
tiled left to right in that order. Wiring one of them is a plain viewer;
|
||||
wiring several shows them side by side, scaled to a common height. `bbox`, `text` and `camera_info`
|
||||
are overlays rather than panels — boxes are drawn on the `image` panel, in its own
|
||||
pixels, and the caption on the composite.
|
||||
Every input is optional. `image` and `depth` each fill a panel, tiled left to right and
|
||||
scaled to a common height, so wiring one is a plain viewer and wiring both shows them
|
||||
side by side. `bbox`, `text` and `camera_info` are overlays rather than panels: boxes on
|
||||
the image panel in its own pixels, caption and camera model on the composite.
|
||||
|
||||
### End-to-end test
|
||||
### Runnable dataflows
|
||||
|
||||
[`dataflow.yml`](dataflow.yml) is self-contained. [`tests/source.py`](tests/source.py)
|
||||
is a synthetic source node that emits an animated frame, a depth map of the same scene,
|
||||
the camera intrinsics, the box around the square moving in the frame, and a caption —
|
||||
every input this node draws — so it needs no camera, no detector and no network. It is
|
||||
a test fixture rather than a worked example; the realistic wiring is the block at the
|
||||
top of this file. Run it to watch the node work:
|
||||
Run either with `dora run <file> --uv`:
|
||||
|
||||
```bash
|
||||
dora run dataflow.yml --uv
|
||||
```
|
||||
| File | Contents |
|
||||
|---|---|
|
||||
| [`dataflow.yml`](dataflow.yml) | A synthetic source driving every input. What CI runs. |
|
||||
| [`dataflow_webcam.yml`](dataflow_webcam.yml) | A real camera, via [dora-webcam][webcam] |
|
||||
|
||||
A window opens with a square bouncing across a gradient, boxed and labelled. Press `q`
|
||||
to quit.
|
||||
[webcam]: https://gitea.liria.cl/liria/dora-webcam
|
||||
|
||||
It builds the viewer from this checkout (`path:` + `build:`) rather than from a tag,
|
||||
which is what lets `dora validate` pick up the manifest described below.
|
||||
`dataflow.yml` needs no camera, no detector and no network: a window opens with a square
|
||||
bouncing across a gradient, boxed and labelled, beside its depth map. Press `q` to quit.
|
||||
CI runs it headless on every push, which is what keeps it from rotting.
|
||||
|
||||
CI runs the same dataflow headless on every push, which is what makes it a gate rather
|
||||
than a demo that rots.
|
||||
|
||||
Set `SOURCE_FRAMES` to stop the source after a fixed number of frames instead of running
|
||||
until you quit; that plus `CI=true` is how the same dataflow runs headless as a test:
|
||||
|
||||
```bash
|
||||
CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv
|
||||
```
|
||||
|
||||
`SOURCE_WIDTH` and `SOURCE_HEIGHT` (default 640x480) set the frame size.
|
||||
|
||||
### With a real camera
|
||||
|
||||
[`dataflow_webcam.yml`](dataflow_webcam.yml) wires
|
||||
[dora-webcam](https://gitea.liria.cl/liria/dora-webcam) into this node, and is the
|
||||
worked example of consuming something that is not a test fixture:
|
||||
|
||||
```bash
|
||||
dora run dataflow_webcam.yml --uv
|
||||
```
|
||||
|
||||
The camera publishes `std/media/v1/Image` and this node draws it. `depth`,
|
||||
`camera_info`, `text` and `bbox` are simply unwired — every input is optional, so a
|
||||
plain viewer is the result.
|
||||
|
||||
It carries no `strict_types`, deliberately. A camera publishes an image and nothing
|
||||
else, and `dora validate --strict` warns about every *typed* input this node declares
|
||||
that a dataflow leaves unwired. The strictness belongs to `dataflow.yml`, which drives
|
||||
all of them.
|
||||
|
||||
Verified end to end: dora clones dora-webcam at the pinned tag, builds both nodes, and
|
||||
runs them against a real camera to a clean exit.
|
||||
|
||||
**A `git:` node needs anonymous read access.** dora-cli 1.0.1 fetches through libgit2
|
||||
with no credential callback, so `ssh://` cannot authenticate at all and anything asking
|
||||
for credentials fails with `remote authentication required but no callback set`. On
|
||||
Gitea, marking a repository public is not sufficient — a repository inside a private
|
||||
organization is internal and answers an anonymous clone with `401`, so the organization
|
||||
has to be public too.
|
||||
|
||||
### Node manifest
|
||||
|
||||
[`dora-node.yml`](dora-node.yml) declares the node's inputs with type URNs. No source
|
||||
package is shared between node repositories, so it is the only contract a consumer has,
|
||||
and it is what makes a wiring mistake fail at build time instead of at runtime. The
|
||||
example source is typed inline in the dataflow with `output_types:`, so `dora validate`
|
||||
checks the two against each other:
|
||||
|
||||
```text
|
||||
$ dora validate dataflow.yml
|
||||
node "view": injected 4 contract type(s) from .../dora-node.yml
|
||||
All type annotations OK.
|
||||
```
|
||||
|
||||
There is one manifest because this repository ships one node. The example source is a
|
||||
test fixture rather than something anyone installs, so it has none, and runs through
|
||||
the interpreter to keep `dora validate` from matching the viewer's manifest against it.
|
||||
|
||||
`image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and
|
||||
`text` is `std/core/v1/String`; the payloads are raw buffers described by their
|
||||
metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
|
||||
left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
|
||||
node reads either.
|
||||
|
||||
## Behavior
|
||||
|
||||
The node keeps a current frame, bbox set and text string. Each input updates its slice
|
||||
of that state:
|
||||
|
||||
- An **`image`** event decodes the frame and immediately redraws: it draws every stored
|
||||
bbox (green rectangle plus `label, conf` text), draws the stored text at the top left,
|
||||
resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
|
||||
`Dora Node: dora-image-view`. Press `q` in the window to quit.
|
||||
- A **`depth`** event colour-maps the distances onto the depth panel and redraws.
|
||||
- A **`camera_info`** event stores the pinhole model and draws it under the caption.
|
||||
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
|
||||
redraw.
|
||||
|
||||
Setting `CI=true` suppresses the display: the node still decodes every input but opens
|
||||
no window. That is what makes it safe to run in a pipeline test.
|
||||
`dataflow_webcam.yml` is the worked example of consuming a real producer. A `git:` node
|
||||
needs **anonymous** read access to fetch — on Gitea that means a public organization, not
|
||||
just a public repository.
|
||||
|
||||
## Inputs
|
||||
|
||||
@@ -142,59 +61,42 @@ no window. That is what makes it safe to run in a pipeline test.
|
||||
|---|---|---|
|
||||
| `image` | `std/media/v1/Image` struct | none — the struct describes itself |
|
||||
| `depth` | `std/media/v1/Image` struct, one channel | `units` (defaults to `mm`) |
|
||||
| `camera_info` | `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` |
|
||||
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
|
||||
| `camera_info` | `[fx, fy, cx, cy]` or a 9-element row-major `K` | `distortion` |
|
||||
| `bbox` | Arrow struct `{bbox, conf, labels}` | `format`: `xyxy` or `xywh` |
|
||||
| `text` | Arrow array of size 1 | — |
|
||||
|
||||
`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`,
|
||||
`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes
|
||||
itself, so nothing about its shape has to travel as message metadata.
|
||||
`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself,
|
||||
so nothing about its shape travels as message metadata.
|
||||
|
||||
Depth is an image whose pixels are distances — dora's type registry has no depth type,
|
||||
and this is how ROS spells it too. Its encodings are `32FC1`/`float32` and
|
||||
`16UC1`/`mono16`/`uint16`, one channel per pixel, and `units` metadata still switches
|
||||
millimetres to metres.
|
||||
Image encodings: `bgr8`, `rgb8`, `yuv420`, `avif`, and `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/
|
||||
`png` as an encoded byte stream. Depth encodings: `32FC1`/`float32` and
|
||||
`16UC1`/`mono16`/`uint16`, one channel per pixel — dora's registry has no depth type, and
|
||||
a depth map is an image whose pixels are distances, as ROS also treats it. `0` and
|
||||
non-finite values mean missing depth and draw black.
|
||||
|
||||
`camera_info` is also read from the metadata of any frame carrying `fx`, `fy`, `cx` and
|
||||
`cy`, so **wiring it is optional even for a camera that publishes it** — which covers a
|
||||
camera that publishes it once per session and a viewer started afterwards. The node needs
|
||||
none of it to draw; it is held for whatever turns depth into 3D.
|
||||
|
||||
Older payloads are still decoded, so a producer can move at its own pace: a flat `UInt8`
|
||||
buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32`
|
||||
buffer with `shape`/`units` metadata for `depth`.
|
||||
buffer with `width`/`height`/`encoding` metadata for `image`, and a flat `Float32` buffer
|
||||
with `shape`/`units` for `depth`. Note that dora's type compatibility is directional, so
|
||||
a producer still *declaring* `std/core/v1/UInt8` fails `dora validate` against an
|
||||
`std/media/v1/Image` input whatever it sends — its manifest has to move too.
|
||||
|
||||
**Migrating a producer.** dora's type compatibility is directional — `Image` satisfies a
|
||||
`Bytes` input, but a `std/core/v1/UInt8` output wired into an `std/media/v1/Image` input
|
||||
is a `dora validate` error. A producer still declaring `std/core/v1/UInt8` or
|
||||
`std/core/v1/Float32` therefore has to change its manifest as well as its payload.
|
||||
|
||||
Supported `image` `encoding` values: `bgr8`, `rgb8`, `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/`png`
|
||||
(encoded byte stream), `yuv420`, `avif`. `width` and `height` are only needed by the raw
|
||||
encodings — the rest carry their own dimensions. An unsupported encoding raises.
|
||||
|
||||
Supported `format` values: `xyxy` and `xywh` (centre-based, converted to `xyxy`).
|
||||
|
||||
`depth` is in millimetres unless `units` is `m`. In a depth map, `0`
|
||||
and any non-finite value mean missing depth and are drawn black. A payload sent as a
|
||||
raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
|
||||
`Float32` contract, and is reinterpreted rather than converted.
|
||||
|
||||
### Camera intrinsics
|
||||
|
||||
`camera_info` carries the pinhole model the depth map is interpreted through, either as
|
||||
the four values `[fx, fy, cx, cy]` or as a nine-element row-major `K`, which is how ROS
|
||||
spells the same model in `sensor_msgs/CameraInfo`. Distortion coefficients are read from
|
||||
the `distortion` metadata.
|
||||
|
||||
The node displays the model and holds it. It needs none of it to draw a panel; it is
|
||||
there for what reads it next — turning a depth map into a point cloud needs `fx`, `fy`,
|
||||
`cx` and `cy` for every pixel, and nothing else in a dataflow carries them.
|
||||
|
||||
Intrinsics are also picked up from the metadata of any `image` or `depth` frame that
|
||||
carries `fx`, `fy`, `cx` and `cy`, so **wiring `camera_info` is optional even for a
|
||||
camera that publishes it**. That matters because a camera typically publishes
|
||||
`camera_info` once per session: a viewer started after the camera would otherwise never
|
||||
see it.
|
||||
Publishing to it:
|
||||
|
||||
```python
|
||||
node.send_output("image", pa.array(frame.ravel()),
|
||||
{"width": 640, "height": 480, "encoding": "bgr8"})
|
||||
image = pa.array(
|
||||
[{"width": 640, "height": 480, "encoding": "bgr8", "data": frame.tobytes()}],
|
||||
type=pa.struct([
|
||||
("width", pa.uint32()), ("height", pa.uint32()),
|
||||
("encoding", pa.utf8()), ("data", pa.large_binary()),
|
||||
]),
|
||||
)
|
||||
node.send_output("image", image, {})
|
||||
|
||||
node.send_output("bbox", pa.array([{
|
||||
"bbox": np.array([...]), # flattened, four values per box
|
||||
@@ -209,7 +111,19 @@ node.send_output("text", pa.array(["hello"]))
|
||||
|
||||
None — it is a display sink.
|
||||
|
||||
## Environment variables
|
||||
## Configuration
|
||||
|
||||
Settings are environment variables, which a dataflow can carry per node, keeping them
|
||||
with the wiring rather than in whatever shell starts it:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
VIEW_DEPTH_MIN_MM: 200
|
||||
VIEW_DEPTH_MAX_MM: 600
|
||||
```
|
||||
|
||||
A node's `env:` wins over the environment the dataflow was started in; variables it does
|
||||
not set still pass through.
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|---|---|---|
|
||||
@@ -218,18 +132,16 @@ None — it is a display sink.
|
||||
| `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. |
|
||||
| `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. |
|
||||
|
||||
Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set. Both are also
|
||||
available as `--view-width` and `--view-height`; the environment variable wins when both
|
||||
are given.
|
||||
Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set; both are also
|
||||
available as `--view-width` and `--view-height`, and the environment variable wins.
|
||||
|
||||
Each end of the depth range is auto-ranged over the valid pixels of the current frame
|
||||
unless it is pinned, so pinning one end leaves the other automatic. Auto-ranging makes
|
||||
an unconfigured sensor legible at the cost of colours that shift from frame to frame;
|
||||
pin both ends to compare frames against each other. The range covers the full extent of
|
||||
the valid pixels rather than a percentile of them, so a small near object against a
|
||||
large wall stays visible.
|
||||
Each end of the depth range is auto-ranged over the current frame's valid pixels unless
|
||||
pinned, so pinning one end leaves the other automatic. Auto-ranging makes an
|
||||
unconfigured sensor legible at the cost of colours that shift between frames; pin both
|
||||
ends to compare frames against each other.
|
||||
|
||||
`CI=true` suppresses the display window; it is not a configuration knob.
|
||||
`CI=true` suppresses the window while still decoding every input, which is what makes the
|
||||
node safe inside a pipeline test. It is not a configuration knob.
|
||||
|
||||
## Build
|
||||
|
||||
@@ -238,13 +150,12 @@ uv sync
|
||||
```
|
||||
|
||||
Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64,
|
||||
so the node installs with no system packages and no compiler on any of them — including
|
||||
AVIF support, which comes from `pillow-avif-plugin` rather than OpenCV.
|
||||
so the node installs with no system packages and no compiler — including AVIF support,
|
||||
which comes from `pillow-avif-plugin` rather than OpenCV.
|
||||
|
||||
**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), Python 3.11+, `uv`, and a
|
||||
display to show the window on (or `CI=true` to run without one). On a bare Linux image
|
||||
`opencv-python` also needs the libGL shared libraries (`libgl1`, `libglib2.0-0`), which
|
||||
it links at import time.
|
||||
display (or `CI=true` to run without one). On a bare Linux image `opencv-python` needs
|
||||
the libGL shared libraries (`libgl1`, `libglib2.0-0`), which it links at import time.
|
||||
|
||||
## Test
|
||||
|
||||
@@ -252,6 +163,6 @@ it links at import time.
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
Unit tests over decoding, the bbox format conversion and the headless draw path; they
|
||||
need neither a camera, a dataflow nor a display. The headless dataflow run above covers
|
||||
the rest — the manifest, the wiring and the event loop.
|
||||
Unit tests over decoding, depth, intrinsics, the bbox conversion and the headless draw
|
||||
path; they need neither a camera, a dataflow nor a display. The headless `dataflow.yml`
|
||||
run covers the rest — the manifest, the wiring and the event loop.
|
||||
|
||||
Reference in New Issue
Block a user