add opencv-plot node for dora dataflows
A display sink that overlays bounding boxes and text on an image input and shows it in an OpenCV window. Based on opencv-plot from dora-hub, brought up to the dora 1.0 node contract and packaged with uv. Python rather than Rust: the opencv Rust crate generates its bindings from the OpenCV headers on the build machine, so it needs libopencv-dev and libclang-dev per machine and per architecture and does not cross-compile. Every dependency here ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64. Covers every encoding the upstream node did — bgr8, rgb8, jpeg/jpg/jpe/bmp/ webp/png, yuv420 and avif — plus both bbox formats, the text overlay, PLOT_WIDTH/PLOT_HEIGHT and headless operation under CI=true.
This commit is contained in:
cesar
@@ -0,0 +1,164 @@
|
||||
# dora-plot
|
||||
|
||||
Dora node that overlays bounding boxes, labels and text on an image input and displays
|
||||
it in an OpenCV window. A display sink for debugging a vision pipeline: it produces no
|
||||
outputs.
|
||||
|
||||
Based on [`opencv-plot`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/opencv-plot)
|
||||
from dora-hub, brought up to the dora 1.0 node contract and packaged with `uv`.
|
||||
|
||||
## Usage
|
||||
|
||||
Add to your dataflow:
|
||||
|
||||
```yaml
|
||||
nodes:
|
||||
- id: plot
|
||||
git: ssh://git@gitea.liria.cl:5001/liria/dora-plot.git
|
||||
tag: v0.1.0
|
||||
build: pip install .
|
||||
path: opencv-plot
|
||||
inputs:
|
||||
image: camera/image
|
||||
bbox: detector/bbox
|
||||
text: detector/text
|
||||
```
|
||||
|
||||
Run the dataflow with `--uv` (`dora run dataflow.yml --uv`) so the node gets its own
|
||||
managed environment.
|
||||
|
||||
### Runnable example
|
||||
|
||||
[`dataflow.yml`](dataflow.yml) wires a webcam and a fixed caption from dora-hub into
|
||||
the viewer:
|
||||
|
||||
```bash
|
||||
dora run dataflow.yml --uv
|
||||
```
|
||||
|
||||
It builds the node from this checkout (`path:` + `build:`) rather than from a tag,
|
||||
which is what lets `dora validate` pick up the manifest described below.
|
||||
|
||||
`bbox` is left unwired there, because every bbox source is a detector and pulling one
|
||||
in would put a model download in front of a webcam preview. To see boxes, add
|
||||
[`dora-yolo`](https://github.com/dora-rs/dora-hub/tree/main/node-hub/dora-yolo) between
|
||||
the camera and the viewer:
|
||||
|
||||
```yaml
|
||||
- id: detector
|
||||
git: https://github.com/dora-rs/dora-hub.git
|
||||
build: pip install ./node-hub/dora-yolo
|
||||
path: dora-yolo
|
||||
inputs:
|
||||
image: camera/image
|
||||
outputs:
|
||||
- bbox
|
||||
|
||||
- id: plot
|
||||
path: opencv-plot
|
||||
build: pip install .
|
||||
inputs:
|
||||
image: camera/image
|
||||
bbox: detector/bbox
|
||||
```
|
||||
|
||||
To display a [dora-zed](https://gitea.liria.cl/liria/dora-zed) camera instead, wire
|
||||
`image: zed-camera/image_bgr` — that node publishes `bgr8` with the `width`, `height`
|
||||
and `encoding` metadata this one expects.
|
||||
|
||||
### Node manifest
|
||||
|
||||
[`dora-node.yml`](dora-node.yml) declares the node's inputs with type URNs. No source
|
||||
package is shared between node repositories, so it is the only contract a consumer has,
|
||||
and it is what makes a wiring mistake fail at build time instead of at runtime:
|
||||
|
||||
```
|
||||
$ dora validate dataflow.yml
|
||||
node "plot": injected 2 contract type(s) from .../dora-node.yml
|
||||
All type annotations OK.
|
||||
```
|
||||
|
||||
`image` is `std/core/v1/UInt8` and `text` is `std/core/v1/String`; the payloads are raw
|
||||
buffers described by their metadata, so the URN names the element type rather than a
|
||||
struct. `bbox` is deliberately left untyped — senders disagree on whether its numeric
|
||||
fields are 32- or 64-bit, and the node reads either.
|
||||
|
||||
## Behavior
|
||||
|
||||
The node keeps a current frame, bbox set and text string. Each input updates its slice
|
||||
of that state:
|
||||
|
||||
- An **`image`** event decodes the frame and immediately redraws: it draws every stored
|
||||
bbox (green rectangle plus `label, conf` text), draws the stored text at the top left,
|
||||
resizes to `PLOT_WIDTH`x`PLOT_HEIGHT` if both are set, and shows it in a window titled
|
||||
`Dora Node: opencv-plot`. Press `q` in the window to quit.
|
||||
- A **`bbox`** or **`text`** event only updates stored state; it is drawn on the next
|
||||
`image` redraw.
|
||||
|
||||
Setting `CI=true` suppresses the display: the node still decodes every input but opens
|
||||
no window. That is what makes it safe to run in a pipeline test.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Payload | Metadata |
|
||||
|---|---|---|
|
||||
| `image` | Flat `UInt8` array | `width`, `height`, `encoding` |
|
||||
| `bbox` | Arrow struct array, element `[0]` with fields `bbox`, `conf`, `labels` | `format` |
|
||||
| `text` | Arrow array of size 1 | — |
|
||||
|
||||
Supported `encoding` values: `bgr8`, `rgb8`, `jpeg`/`jpg`/`jpe`/`bmp`/`webp`/`png`
|
||||
(encoded byte stream), `yuv420`, `avif`. `width` and `height` are only needed by the raw
|
||||
encodings — the rest carry their own dimensions. An unsupported encoding raises.
|
||||
|
||||
Supported `format` values: `xyxy` and `xywh` (centre-based, converted to `xyxy`).
|
||||
|
||||
```python
|
||||
node.send_output("image", pa.array(frame.ravel()),
|
||||
{"width": 640, "height": 480, "encoding": "bgr8"})
|
||||
|
||||
node.send_output("bbox", pa.array([{
|
||||
"bbox": np.array([...]), # flattened, four values per box
|
||||
"conf": np.array([...]),
|
||||
"labels": np.array([...]),
|
||||
}]), {"format": "xyxy"})
|
||||
|
||||
node.send_output("text", pa.array(["hello"]))
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
None — it is a display sink.
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `PLOT_WIDTH` | image width | Resize the displayed frame to this width, in pixels. |
|
||||
| `PLOT_HEIGHT` | image height | Resize the displayed frame to this height, in pixels. |
|
||||
|
||||
Resizing only happens when both are set. Both are also available as `--plot-width` and
|
||||
`--plot-height`; the environment variable wins when both are given.
|
||||
|
||||
`CI=true` suppresses the display window; it is not a configuration knob.
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
```
|
||||
|
||||
Every dependency ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64,
|
||||
so the node installs with no system packages and no compiler on any of them — including
|
||||
AVIF support, which comes from `pillow-avif-plugin` rather than OpenCV.
|
||||
|
||||
**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), Python 3.11+, `uv`, and a
|
||||
display to show the window on (or `CI=true` to run without one).
|
||||
|
||||
## Test
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
The tests cover decoding, the bbox format conversion and the headless draw path, so they
|
||||
need neither a camera, a dataflow nor a display.
|
||||
Reference in New Issue
Block a user