cristhian
8a7bda5867
Both inputs now declare dora's Arrow struct {width, height, encoding, data}
rather than a flat buffer plus loose metadata. The payload describes itself, so
nothing about its shape has to be agreed out of band, and the frame travels as
one Arrow value instead of a buffer whose meaning lives in a sidecar dict.
Depth uses the same type. dora's registry has no depth type, and a depth map is
an image whose pixels are distances -- the reading ROS takes as well -- so it
arrives with a single-channel encoding: 32FC1/float32 or 16UC1/mono16/uint16.
`units` stays in message metadata because the struct has no field for it.
decode.IMAGE_TYPE is the single definition of the struct; the example source
and the tests build payloads from it rather than respelling the schema.
The older shapes are still decoded -- a flat UInt8 buffer with
width/height/encoding for image, a flat Float32 buffer with shape/units for
depth -- so a producer's payload can move before its manifest does. That is
only half a migration path: type compatibility is directional, so a producer
still declaring std/core/v1/UInt8 fails validate against an
std/media/v1/Image input regardless of what it sends, and has to update its
manifest too.
Verified: 41 tests, validate clean with 4 contracts injected, the dataflow runs
to a clean exit, and a composite rendered through the struct path is identical
to the one rendered before the migration.
75 lines
11 KiB
Markdown
75 lines
11 KiB
Markdown
# AGENTS.md
|
|
|
|
This file provides guidance to AI coding assistants when working with code in this repository.
|
|
|
|
## What this is
|
|
|
|
A dora-rs Python node that displays a video stream with overlays: it keeps one panel per frame stream (`image`, `depth`) plus a bbox set, a text string and the camera model, and every frame event redraws the lot as one side-by-side composite in an OpenCV window. It is a display sink — it produces no outputs. [README.md](README.md) documents the inputs, the encodings and the environment variables; this file covers only what the code and the README do not say.
|
|
|
|
Everything the example dataflow needs is in this checkout. Do not reintroduce a dependency on another repository — the example must keep running with no camera, no detector and no network.
|
|
|
|
**It is Python on purpose.** The node was first written in Rust against the `opencv` crate; that crate generates its bindings from the OpenCV headers installed on the build machine, so it needs `libopencv-dev` and `libclang-dev` on every machine and every architecture and does not cross-compile. Every dependency here ships prebuilt wheels for linux-x86_64, linux-aarch64 and macos-arm64, so `uv sync` is the whole install. Do not reintroduce a dependency that has to be compiled from source.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
uv sync --group dev # deps plus pytest and ruff
|
|
uv run pytest && uv run ruff check . # no camera/dataflow/display needed
|
|
dora run dataflow.yml --uv # the example, in a window
|
|
CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless
|
|
```
|
|
|
|
`dora build --uv` / `dora run --uv` rewrites each manifest's `build: pip install .` to `uv pip install .` and gives the node its own managed venv under `.dora/python-envs/<node-id>/`, so what it imports at runtime is what was installed at build time.
|
|
|
|
## Architecture
|
|
|
|
- **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node.
|
|
- **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. `unpack()` accepts two payload shapes — the `std/media/v1/Image` struct (`width`, `height`, `encoding`, `data`), and the flat `UInt8` buffer described by message metadata that a `std/core/v1/UInt8` producer sends. A struct describes itself, so its metadata is ignored; everything after unpacking is common.
|
|
|
|
`IMAGE_TYPE` is the one definition of that struct; the example source and the tests build payloads from it rather than respelling the schema.
|
|
|
|
**Interop cost of the contract being `Image`.** Type compatibility is directional: `Image` satisfies a `Bytes` input, but a `std/core/v1/UInt8` output wired into an `std/media/v1/Image` input is a validate error. A producer still declaring `std/core/v1/UInt8` — dora-zed's `image_bgr` at time of writing — has to change its manifest, not just its payload, before it can feed this node. The decoder accepting the flat form does not rescue it: the failure is at validate, before anything runs. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
|
|
- **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one.
|
|
- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. Depth arrives as `std/media/v1/Image` with a single-channel encoding (`DEPTH_DTYPES`) because dora's registry has no depth type and a depth map *is* an image of distances — the same reading ROS takes. `units` stays in message metadata: the struct has no field for it. One thing is deliberate: the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
|
|
- **`dora_image_view/payload.py`** — `as_float32()`: reinterprets a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`. Shared by `depth` and `intrinsics`, which both meet it.
|
|
- **`dora_image_view/intrinsics.py`** — `Intrinsics`, `parse()` for the `camera_info` payload and `from_metadata()` for the copy senders attach to each frame. Both exist because dora-zed publishes the same numbers twice, and the per-frame copy is the one that survives a viewer starting after the camera did — `camera_info` goes out **once per session**. A nine-element row-major K is accepted alongside the compact four values, since that is how ROS spells the same model. The node needs none of this to draw; it is held for whatever turns depth into 3D, and drawn so a wrong camera model is visible rather than merely plausible.
|
|
- **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is.
|
|
- **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout.
|
|
- **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either.
|
|
- **`dataflow.yml`** — The example and the CI gate, with `strict_types: true`. The viewer is a `path:` node, so its manifest is injected; the source is typed inline with `output_types:` and the two are compared against each other. `bbox` is annotated on neither side, matching the manifest.
|
|
|
|
**One manifest, on purpose.** A `dora-node.yml` describes exactly one node (the schema is `NodeManifest`, singular) and is the unit `dora hub publish` operates on — it expects one beside `pyproject.toml`. This repository ships one node, so it has one manifest. The example source is a test fixture and gets none.
|
|
|
|
That is why the source runs as `path: python` + `args:` rather than `path: examples/source.py`. `dora validate` matches a manifest to a `path:` node **by entrypoint**, so a path node here would be tried against the root manifest, would fail to match its `dora-image-view` entrypoint, and would warn — and `strict_types: true` makes that warning fatal. Going through the interpreter leaves nothing to match. The cost is that the fixture gets no managed venv of its own (`.dora/python-envs/` stays empty for it) and runs against the project environment instead; the shipped node still uses `path:` + `build:`, which is the mechanism a consumer actually uses. Do not "fix" this back into a path node without re-reading this paragraph.
|
|
- **`.gitea/workflows/validate.yml`** — CI: lints, runs the unit tests, validates the dataflow and runs it headless. Needs no camera and no display. One job on purpose — the gates themselves take about a second, and are dwarfed by preparing the runner.
|
|
|
|
## Why CI installs libgl1
|
|
|
|
`opencv-python` links against libGL at import time, so `import cv2` fails on the bare runner image and CI installs the shared libraries first. At roughly sixteen seconds it is the most expensive step in the job, several times everything else combined.
|
|
|
|
`opencv-python-headless` needs none of them, but it also has no `imshow`, which is this node's whole purpose. Removing the step therefore means making the GUI wheel an optional extra — headless by default, `pip install .[gui]` for a real window — and that is a change to what a consumer installs, not just to CI. It has not been done.
|
|
|
|
## On `output_framing`
|
|
|
|
`output_framing: {image: arrow-ipc}` sets a per-output wire format — `raw` (the default, plain Arrow buffer layout) or `arrow-ipc` (self-describing: schema plus record batches). It is *not* required for images: a struct payload was verified to arrive identically under both framings between Python nodes, and `dora validate` does not ask for it when an output is annotated `std/media/v1/Image`. Reach for `arrow-ipc` when the wire has to be self-describing — crossing languages, or recording — not because a payload is an image.
|
|
|
|
## A caveat on typed optional inputs
|
|
|
|
`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map and a camera_info at all: both are typed, so the example has to drive them, and a new output it sends needs a matching entry in the dataflow's `output_types:`.
|
|
|
|
## Checking what it draws
|
|
|
|
The gates prove the node does not crash, not that the picture is right — the colormap-ranging defect found while adding depth rendered as a plausible flat panel and passed every test. Compose a frame offline and look at it:
|
|
|
|
```python
|
|
view = View(headless=True)
|
|
view.panels["depth"] = depth.colorize(depth.to_millimetres(storage, metadata))
|
|
cv2.imwrite("/tmp/frame.png", view.compose())
|
|
```
|
|
|
|
`compose()` is public for exactly this: it returns the composite without needing a window.
|
|
|
|
## Rules
|
|
|
|
- Do not mention AI tools in commit messages or documentation.
|