accept std/media/v1/Image payloads
dora's typed image is an Arrow struct of width, height, encoding and data -- it carries inside the payload what this node has been reading out of message metadata. The decoder now takes either shape: a struct describes itself and its metadata is ignored, a flat UInt8 buffer is described by width/height/ encoding beside it, and everything after unpacking is common. The declared contract stays std/core/v1/UInt8, deliberately. Type compatibility is directional -- Image satisfies a Bytes input, but a std/core/v1/UInt8 output wired into an std/media/v1/Image input is a validate error -- and dora-zed declares image_bgr as std/core/v1/UInt8. Flipping this manifest now would break the camera. Accepting both shapes first is what lets the producers move before the contract does. output_framing is not part of this. It selects the wire format, raw or arrow-ipc, and is not required for images: a struct payload was verified to arrive identically under both framings, and validate does not ask for arrow-ipc when an output is annotated std/media/v1/Image. It earns its keep when the wire must be self-describing, such as crossing languages or recording.
This commit is contained in:
@@ -24,7 +24,9 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless
|
||||
## Architecture
|
||||
|
||||
- **`dora_image_view/main.py`** — Argument parsing and the dora event loop, nothing else. The env var wins over the CLI flag for `VIEW_WIDTH`/`VIEW_HEIGHT`. The upstream node spells these `PLOT_WIDTH`/`PLOT_HEIGHT`; they were renamed with the node, so a dataflow copied from upstream needs its env keys changed. `Node(args.name)` rather than `Node()` so the node also runs standalone as a dynamic node.
|
||||
- **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
|
||||
- **`dora_image_view/decode.py`** — `to_bgr()`: every supported encoding to a writable BGR array. `unpack()` accepts two payload shapes — the `std/media/v1/Image` struct (`width`, `height`, `encoding`, `data`), and the flat `UInt8` buffer described by message metadata that a `std/core/v1/UInt8` producer sends. A struct describes itself, so its metadata is ignored; everything after unpacking is common.
|
||||
|
||||
**The declared type is still `std/core/v1/UInt8` and the reason is interop.** Type compatibility is directional: `Image` satisfies a `Bytes` input, but a `std/core/v1/UInt8` output wired into an `std/media/v1/Image` input is a validate error. dora-zed declares `image_bgr` as `std/core/v1/UInt8`, so flipping this manifest to `Image` breaks the camera until dora-zed is migrated. Accepting both shapes first is what makes that migration orderable: producers move, then the contract. Two behaviours are deliberate, not bugs: `bgr8` **copies** (the payload may be a read-only view on shared memory, and the frame is annotated in place), and `yuv420` converts with `COLOR_YUV420p2RGB` rather than `...2BGR` — the upstream node converts to RGB and then displays it as BGR, and that channel swap is part of the behaviour callers already see.
|
||||
- **`dora_image_view/view.py`** — `View` (the panels, the overlay state and the window) and `Bboxes`. `headless` (from `CI=true`) skips only the window: every input is still decoded and composed, which is what makes the node safe in a pipeline test. Panels are tiled in the fixed order in `PANELS`, not in arrival order, so a panel does not jump when a stream drops a frame. Boxes are drawn on the `image` panel before scaling, because they are in image pixels; the caption is drawn on the composite, because the panel it would otherwise land on may not be the leftmost one.
|
||||
- **`dora_image_view/depth.py`** — `to_millimetres()` and `colorize()`. Decoding normalises to float32 millimetres with 0 for missing, so the colouring only ever sees one representation. One thing is deliberate: the auto range is the valid pixels' full min/max, **not** a percentile. Trimming the tails looks safer and breaks the exact scene this node exists for — a small object in front of a large wall is a few percent of the pixels, so both tails land on the wall and the object clips away. A rendered check caught that; do not reintroduce percentile ranging.
|
||||
- **`dora_image_view/payload.py`** — `as_float32()`: reinterprets a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`. Shared by `depth` and `intrinsics`, which both meet it.
|
||||
@@ -45,6 +47,10 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless
|
||||
|
||||
`opencv-python-headless` needs none of them, but it also has no `imshow`, which is this node's whole purpose. Removing the step therefore means making the GUI wheel an optional extra — headless by default, `pip install .[gui]` for a real window — and that is a change to what a consumer installs, not just to CI. It has not been done.
|
||||
|
||||
## On `output_framing`
|
||||
|
||||
`output_framing: {image: arrow-ipc}` sets a per-output wire format — `raw` (the default, plain Arrow buffer layout) or `arrow-ipc` (self-describing: schema plus record batches). It is *not* required for images: a struct payload was verified to arrive identically under both framings between Python nodes, and `dora validate` does not ask for it when an output is annotated `std/media/v1/Image`. Reach for `arrow-ipc` when the wire has to be self-describing — crossing languages, or recording — not because a payload is an image.
|
||||
|
||||
## A caveat on typed optional inputs
|
||||
|
||||
`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map and a camera_info at all: both are typed, so the example has to drive them, and a new output it sends needs a matching entry in the dataflow's `output_types:`.
|
||||
|
||||
Reference in New Issue
Block a user