accept std/media/v1/Image payloads

dora's typed image is an Arrow struct of width, height, encoding and data --
it carries inside the payload what this node has been reading out of message
metadata. The decoder now takes either shape: a struct describes itself and
its metadata is ignored, a flat UInt8 buffer is described by width/height/
encoding beside it, and everything after unpacking is common.

The declared contract stays std/core/v1/UInt8, deliberately. Type
compatibility is directional -- Image satisfies a Bytes input, but a
std/core/v1/UInt8 output wired into an std/media/v1/Image input is a validate
error -- and dora-zed declares image_bgr as std/core/v1/UInt8. Flipping this
manifest now would break the camera. Accepting both shapes first is what lets
the producers move before the contract does.

output_framing is not part of this. It selects the wire format, raw or
arrow-ipc, and is not required for images: a struct payload was verified to
arrive identically under both framings, and validate does not ask for
arrow-ipc when an output is annotated std/media/v1/Image. It earns its keep
when the wire must be self-describing, such as crossing languages or
recording.
This commit is contained in:
2026-09-15 05:25:18 -03:00
parent 92abe8a446
commit a6fa3ce32b
5 changed files with 133 additions and 24 deletions
+48 -5
View File
@@ -1,4 +1,12 @@
"""Decoding of the `image` input into a BGR frame that can be drawn on."""
"""Decoding of the `image` input into a BGR frame that can be drawn on.
Two payload shapes are accepted, because dora has both in the field. The typed
one is `std/media/v1/Image`: an Arrow struct carrying width, height, encoding
and data together, so the frame is self-describing. The older one is a flat
UInt8 buffer whose dimensions and encoding travel beside it in the message
metadata, which is what a node declaring `std/core/v1/UInt8` sends -- dora-zed
among them. Everything after unpacking is identical.
"""
import io
@@ -14,13 +22,18 @@ SELF_DESCRIBING = frozenset({"jpeg", "jpg", "jpe", "bmp", "webp", "png"})
SUPPORTED = SELF_DESCRIBING | {"bgr8", "rgb8", "yuv420", "avif"}
def to_bgr(storage, encoding: str, width: int | None, height: int | None) -> np.ndarray:
# The fields of std/media/v1/Image. A payload carrying them describes itself and
# its metadata is not consulted.
IMAGE_FIELDS = ("width", "height", "encoding", "data")
def to_bgr(storage, metadata: dict) -> np.ndarray:
"""Decode one `image` payload into a writable BGR array.
`width` and `height` come from the message metadata and are only required by
the raw encodings, which carry no dimensions of their own.
Accepts either an `std/media/v1/Image` struct or a flat buffer described by
the message metadata.
"""
buffer = storage.to_numpy().astype(np.uint8)
buffer, encoding, width, height = unpack(storage, metadata)
if encoding == "bgr8":
# Copy so the frame can be annotated: to_numpy() may be a view on the
@@ -53,6 +66,36 @@ def to_bgr(storage, encoding: str, width: int | None, height: int | None) -> np.
raise RuntimeError(f"unsupported image encoding: {encoding}")
def unpack(storage, metadata: dict) -> tuple[np.ndarray, str, int | None, int | None]:
"""Split a payload of either shape into buffer, encoding, width and height."""
if _is_image_struct(storage):
# One image per message: the struct is a single-row array.
row = storage[0].as_py()
return (
np.frombuffer(row["data"], dtype=np.uint8),
str(row["encoding"]).lower(),
row["width"],
row["height"],
)
return (
storage.to_numpy().astype(np.uint8),
str(metadata.get("encoding", "bgr8")).lower(),
metadata.get("width"),
metadata.get("height"),
)
def _is_image_struct(storage) -> bool:
"""Whether the payload is an Arrow struct with the std/media/v1/Image fields."""
fields = getattr(getattr(storage, "type", None), "fields", None)
if fields is None:
return False
return all(name in {field.name for field in fields} for name in IMAGE_FIELDS)
def _reshape(buffer: np.ndarray, width: int | None, height: int | None, encoding: str):
width, height = _dimensions(width, height, encoding)
return _take(buffer, width * height * 3, encoding).reshape(height, width, 3)