cristhian
92abe8a446
A dora-node.yml describes exactly one node and is the unit `dora hub publish` operates on, which expects it beside pyproject.toml. This repository ships one node, so it should carry one manifest. examples/dora-node.yml described a test fixture that nobody installs and that has no package of its own -- a manifest for something that is not a node. The source's outputs are now annotated with `output_types:` in the dataflow, which is the documented way to type an output without a manifest: "only annotated outputs are type-checked". It runs as `path: python` + `args:` because `dora validate` matches a manifest to a `path:` node by entrypoint, so a path node would be tried against the root manifest, fail to match `dora-image-view`, and warn -- fatal under strict_types. Through the interpreter there is nothing to match. The type gate is unchanged in strength: the viewer's contract is still injected and compared against the source's annotations, and a deliberate Float32 -> String swap still fails validate. The cost is that the fixture no longer gets a managed venv of its own and runs against the project environment. That applies to the fixture only; the shipped node still uses path: + build:, which is what a consumer exercises. This supersedes the approach in "fix strict-type validation by giving the example source a manifest" and the doc commit that cited the manifest-next-to-a-path-node rule as justification.
206 lines
8.4 KiB
Markdown
206 lines
8.4 KiB
Markdown
# dora-image-view
|
|
|
|
Dora node that displays a colour image and a depth map side by side in one OpenCV
|
|
window, with bounding box, text and camera intrinsics overlays. A display sink for debugging a
|
|
vision pipeline: it produces no outputs.
|
|
|
|
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
|
|
from dora-hub, brought up to the dora 1.0 node contract and packaged with `uv`.
|
|
|
|
## Usage
|
|
|
|
Add to your dataflow:
|
|
|
|
```yaml
|
|
nodes:
|
|
- id: view
|
|
git: ssh://git@gitea.liria.cl:5001/liria/dora-image-view.git
|
|
tag: v0.1.0
|
|
build: pip install .
|
|
path: dora-image-view
|
|
inputs:
|
|
image: camera/image_bgr
|
|
depth: camera/depth
|
|
camera_info: camera/camera_info
|
|
bbox: detector/bbox
|
|
text: detector/text
|
|
```
|
|
|
|
Run the dataflow with `--uv` (`dora run dataflow.yml --uv`) so the node gets its own
|
|
managed environment.
|
|
|
|
Every input is optional, and each of `image` and `depth` fills one panel of the window,
|
|
tiled left to right in that order. Wiring one of them is a plain viewer;
|
|
wiring several shows them side by side, scaled to a common height. `bbox`, `text` and `camera_info`
|
|
are overlays rather than panels — boxes are drawn on the `image` panel, in its own
|
|
pixels, and the caption on the composite.
|
|
|
|
### Runnable example
|
|
|
|
[`dataflow.yml`](dataflow.yml) is self-contained. [`examples/source.py`](examples/source.py)
|
|
is a synthetic source node that emits an animated frame, the box around the square
|
|
moving in it, and a caption — the three inputs this node draws — so the example needs
|
|
no camera, no detector and no network:
|
|
|
|
```bash
|
|
dora run dataflow.yml --uv
|
|
```
|
|
|
|
A window opens with a square bouncing across a gradient, boxed and labelled. Press `q`
|
|
to quit.
|
|
|
|
It builds the viewer from this checkout (`path:` + `build:`) rather than from a tag,
|
|
which is what lets `dora validate` pick up the manifest described below.
|
|
|
|
Set `SOURCE_FRAMES` to stop the source after a fixed number of frames instead of running
|
|
until you quit; that plus `CI=true` is how the same dataflow runs headless as a test:
|
|
|
|
```bash
|
|
CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv
|
|
```
|
|
|
|
`SOURCE_WIDTH` and `SOURCE_HEIGHT` (default 640x480) set the frame size.
|
|
|
|
### Node manifest
|
|
|
|
[`dora-node.yml`](dora-node.yml) declares the node's inputs with type URNs. No source
|
|
package is shared between node repositories, so it is the only contract a consumer has,
|
|
and it is what makes a wiring mistake fail at build time instead of at runtime. The
|
|
example source is typed inline in the dataflow with `output_types:`, so `dora validate`
|
|
checks the two against each other:
|
|
|
|
```
|
|
$ dora validate dataflow.yml
|
|
node "view": injected 4 contract type(s) from .../dora-node.yml
|
|
All type annotations OK.
|
|
```
|
|
|
|
There is one manifest because this repository ships one node. The example source is a
|
|
test fixture rather than something anyone installs, so it has none, and runs through
|
|
the interpreter to keep `dora validate` from matching the viewer's manifest against it.
|
|
|
|
`image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and
|
|
`text` is `std/core/v1/String`; the payloads are raw buffers described by their
|
|
metadata, so the URN names the element type rather than a struct. `bbox` is deliberately
|
|
left untyped — senders disagree on whether its numeric fields are 32- or 64-bit, and the
|
|
node reads either.
|
|
|
|
## Behavior
|
|
|
|
The node keeps a current frame, bbox set and text string. Each input updates its slice
|
|
of that state:
|
|
|
|
- An **`image`** event decodes the frame and immediately redraws: it draws every stored
|
|
bbox (green rectangle plus `label, conf` text), draws the stored text at the top left,
|
|
resizes to `VIEW_WIDTH`x`VIEW_HEIGHT` if both are set, and shows it in a window titled
|
|
`Dora Node: dora-image-view`. Press `q` in the window to quit.
|
|
- A **`depth`** event colour-maps the distances onto the depth panel and redraws.
|
|
- A **`camera_info`** event stores the pinhole model and draws it under the caption.
|
|
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
|
|
redraw.
|
|
|
|
Setting `CI=true` suppresses the display: the node still decodes every input but opens
|
|
no window. That is what makes it safe to run in a pipeline test.
|
|
|
|
## Inputs
|
|
|
|
| Input | Payload | Metadata |
|
|
|---|---|---|
|
|
| `image` | Flat `UInt8` array | `width`, `height`, `encoding` |
|
|
| `depth` | Flat `Float32` array, `[H, W]` | `shape`, `units` |
|
|
| `camera_info` | Flat `Float32` array, `[fx, fy, cx, cy]` or a 9-element K | `distortion`, `distortion_model` |
|
|
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
|
|
| `text` | Arrow array of size 1 | — |
|
|
|
|
Supported `encoding` values: `bgr8`, `rgb8`, `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/`png`
|
|
(encoded byte stream), `yuv420`, `avif`. `width` and `height` are only needed by the raw
|
|
encodings — the rest carry their own dimensions. An unsupported encoding raises.
|
|
|
|
Supported `format` values: `xyxy` and `xywh` (centre-based, converted to `xyxy`).
|
|
|
|
`depth` is in millimetres unless `units` is `m`. In a depth map, `0`
|
|
and any non-finite value mean missing depth and are drawn black. A payload sent as a
|
|
raw byte slice — which is what a C++ node does — arrives as `UInt8` despite the
|
|
`Float32` contract, and is reinterpreted rather than converted.
|
|
|
|
### Camera intrinsics
|
|
|
|
`camera_info` carries the pinhole model the depth map is interpreted through, either as
|
|
the four values `[fx, fy, cx, cy]` or as a nine-element row-major `K`, which is how ROS
|
|
spells the same model in `sensor_msgs/CameraInfo`. Distortion coefficients are read from
|
|
the `distortion` metadata.
|
|
|
|
The node displays the model and holds it. It needs none of it to draw a panel; it is
|
|
there for what reads it next — turning a depth map into a point cloud needs `fx`, `fy`,
|
|
`cx` and `cy` for every pixel, and nothing else in a dataflow carries them.
|
|
|
|
Intrinsics are also picked up from the metadata of any `image` or `depth` frame that
|
|
carries `fx`, `fy`, `cx` and `cy`, so **wiring `camera_info` is optional even for a
|
|
camera that publishes it**. That matters because a camera typically publishes
|
|
`camera_info` once per session: a viewer started after the camera would otherwise never
|
|
see it.
|
|
|
|
```python
|
|
node.send_output("image", pa.array(frame.ravel()),
|
|
{"width": 640, "height": 480, "encoding": "bgr8"})
|
|
|
|
node.send_output("bbox", pa.array([{
|
|
"bbox": np.array([...]), # flattened, four values per box
|
|
"conf": np.array([...]),
|
|
"labels": np.array([...]),
|
|
}]), {"format": "xyxy"})
|
|
|
|
node.send_output("text", pa.array(["hello"]))
|
|
```
|
|
|
|
## Outputs
|
|
|
|
None — it is a display sink.
|
|
|
|
## Environment variables
|
|
|
|
| Variable | Default | Meaning |
|
|
|---|---|---|
|
|
| `VIEW_WIDTH` | composite width | Resize the displayed composite to this width, in pixels. |
|
|
| `VIEW_HEIGHT` | composite height | Resize the displayed composite to this height, in pixels. |
|
|
| `VIEW_DEPTH_MIN_MM` | auto | Pin the near end of the depth colour range, in millimetres. |
|
|
| `VIEW_DEPTH_MAX_MM` | auto | Pin the far end of the depth colour range, in millimetres. |
|
|
|
|
Resizing only happens when both `VIEW_WIDTH` and `VIEW_HEIGHT` are set. Both are also
|
|
available as `--view-width` and `--view-height`; the environment variable wins when both
|
|
are given.
|
|
|
|
Each end of the depth range is auto-ranged over the valid pixels of the current frame
|
|
unless it is pinned, so pinning one end leaves the other automatic. Auto-ranging makes
|
|
an unconfigured sensor legible at the cost of colours that shift from frame to frame;
|
|
pin both ends to compare frames against each other. The range covers the full extent of
|
|
the valid pixels rather than a percentile of them, so a small near object against a
|
|
large wall stays visible.
|
|
|
|
`CI=true` suppresses the display window; it is not a configuration knob.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
uv sync
|
|
```
|
|
|
|
Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64,
|
|
so the node installs with no system packages and no compiler on any of them — including
|
|
AVIF support, which comes from `pillow-avif-plugin` rather than OpenCV.
|
|
|
|
**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), Python 3.11+, `uv`, and a
|
|
display to show the window on (or `CI=true` to run without one). On a bare Linux image
|
|
`opencv-python` also needs the libGL shared libraries (`libgl1`, `libglib2.0-0`), which
|
|
it links at import time.
|
|
|
|
## Test
|
|
|
|
```bash
|
|
uv run pytest
|
|
```
|
|
|
|
Unit tests over decoding, the bbox format conversion and the headless draw path; they
|
|
need neither a camera, a dataflow nor a display. The headless dataflow run above covers
|
|
the rest — the manifest, the wiring and the event loop.
|