From 92abe8a446b34b194bb5e7ab23c92456223fc669 Mon Sep 17 00:00:00 2001 From: Cristhian Date: Tue, 15 Sep 2026 05:21:36 -0300 Subject: [PATCH] refactor: type the example source inline instead of shipping a manifest A dora-node.yml describes exactly one node and is the unit `dora hub publish` operates on, which expects it beside pyproject.toml. This repository ships one node, so it should carry one manifest. examples/dora-node.yml described a test fixture that nobody installs and that has no package of its own -- a manifest for something that is not a node. The source's outputs are now annotated with `output_types:` in the dataflow, which is the documented way to type an output without a manifest: "only annotated outputs are type-checked". It runs as `path: python` + `args:` because `dora validate` matches a manifest to a `path:` node by entrypoint, so a path node would be tried against the root manifest, fail to match `dora-image-view`, and warn -- fatal under strict_types. Through the interpreter there is nothing to match. The type gate is unchanged in strength: the viewer's contract is still injected and compared against the source's annotations, and a deliberate Float32 -> String swap still fails validate. The cost is that the fixture no longer gets a managed venv of its own and runs against the project environment. That applies to the fixture only; the shipped node still uses path: + build:, which is what a consumer exercises. This supersedes the approach in "fix strict-type validation by giving the example source a manifest" and the doc commit that cited the manifest-next-to-a-path-node rule as justification. --- AGENTS.md | 9 ++++-- README.md | 13 +++++---- dataflow.yml | 23 ++++++++++++++- examples/dora-node.yml | 66 ------------------------------------------ 4 files changed, 36 insertions(+), 75 deletions(-) delete mode 100644 examples/dora-node.yml diff --git a/AGENTS.md b/AGENTS.md index cbe5614..c03eb2a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,8 +32,11 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless - **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is. - **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It exists so the example dataflow exercises all three inputs without a camera or a detector, and so CI can run the dataflow rather than only validate it. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout. - **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either. -- **`examples/dora-node.yml`** — Manifest for the source node, and not optional. A manifest describes exactly one node (the schema is `NodeManifest`, singular), so the second node in this repository gets a second manifest; a `dora-node.yml` next to a plain `path:` node is [the documented way](https://dora-rs.ai/dora/hub/node-manifest) to type a node without publishing it. `dora validate` matches a manifest to a `path:` node by entrypoint; with no manifest beside `examples/source.py`, the root `dora-node.yml` is tried against it, does not match its entrypoint, and warns — which `strict_types: true` turns into a failure. Declaring the source's types here rather than as `output_types:` in `dataflow.yml` is also what makes them *checked*: an inline annotation shadows the manifest, so the manifest would be injected but never compared. -- **`dataflow.yml`** — The example and the CI gate, with `strict_types: true`. Both nodes are `path:` nodes with a manifest, so both contracts are injected and compared against each other; `bbox` is annotated on neither side, matching the manifests. +- **`dataflow.yml`** — The example and the CI gate, with `strict_types: true`. The viewer is a `path:` node, so its manifest is injected; the source is typed inline with `output_types:` and the two are compared against each other. `bbox` is annotated on neither side, matching the manifest. + + **One manifest, on purpose.** A `dora-node.yml` describes exactly one node (the schema is `NodeManifest`, singular) and is the unit `dora hub publish` operates on — it expects one beside `pyproject.toml`. This repository ships one node, so it has one manifest. The example source is a test fixture and gets none. + + That is why the source runs as `path: python` + `args:` rather than `path: examples/source.py`. `dora validate` matches a manifest to a `path:` node **by entrypoint**, so a path node here would be tried against the root manifest, would fail to match its `dora-image-view` entrypoint, and would warn — and `strict_types: true` makes that warning fatal. Going through the interpreter leaves nothing to match. The cost is that the fixture gets no managed venv of its own (`.dora/python-envs/` stays empty for it) and runs against the project environment instead; the shipped node still uses `path:` + `build:`, which is the mechanism a consumer actually uses. Do not "fix" this back into a path node without re-reading this paragraph. - **`.gitea/workflows/validate.yml`** — CI: lints, runs the unit tests, validates the dataflow and runs it headless. Needs no camera and no display. One job on purpose — the gates themselves take about a second, and are dwarfed by preparing the runner. ## Why CI installs libgl1 @@ -44,7 +47,7 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless ## A caveat on typed optional inputs -`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map and a camera_info at all: both are typed, so the example has to drive them. +`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map and a camera_info at all: both are typed, so the example has to drive them, and a new output it sends needs a matching entry in the dataflow's `output_types:`. ## Checking what it draws diff --git a/README.md b/README.md index c54d76a..0ec34b0 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ dora run dataflow.yml --uv A window opens with a square bouncing across a gradient, boxed and labelled. Press `q` to quit. -It builds both nodes from this checkout (`path:` + `build:`) rather than from a tag, +It builds the viewer from this checkout (`path:` + `build:`) rather than from a tag, which is what lets `dora validate` pick up the manifest described below. Set `SOURCE_FRAMES` to stop the source after a fixed number of frames instead of running @@ -66,16 +66,19 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv [`dora-node.yml`](dora-node.yml) declares the node's inputs with type URNs. No source package is shared between node repositories, so it is the only contract a consumer has, and it is what makes a wiring mistake fail at build time instead of at runtime. The -example source has a manifest of its own, so `dora validate` checks the two against -each other: +example source is typed inline in the dataflow with `output_types:`, so `dora validate` +checks the two against each other: ``` $ dora validate dataflow.yml - node "source": injected 2 contract type(s) from .../examples/dora-node.yml - node "view": injected 2 contract type(s) from .../dora-node.yml + node "view": injected 4 contract type(s) from .../dora-node.yml All type annotations OK. ``` +There is one manifest because this repository ships one node. The example source is a +test fixture rather than something anyone installs, so it has none, and runs through +the interpreter to keep `dora validate` from matching the viewer's manifest against it. + `image` is `std/core/v1/UInt8`, `depth` and `camera_info` are `std/core/v1/Float32` and `text` is `std/core/v1/String`; the payloads are raw buffers described by their metadata, so the URN names the element type rather than a struct. `bbox` is deliberately diff --git a/dataflow.yml b/dataflow.yml index c94f3d4..b6331fd 100644 --- a/dataflow.yml +++ b/dataflow.yml @@ -11,7 +11,21 @@ strict_types: true nodes: - id: source - path: examples/source.py + # Run through the interpreter rather than as `path: examples/source.py`. + # + # A dora-node.yml describes exactly one node and is what gets published + # (`dora hub publish` expects it beside pyproject.toml), and this + # repository ships one node: dora-image-view. The example source is a test + # fixture, not a node anyone installs, so it gets no manifest. + # + # That has a mechanical consequence. `dora validate` matches a manifest to + # a `path:` node by entrypoint, so a path node here would be tried against + # the root manifest, would not match its `dora-image-view` entrypoint, and + # would warn -- which strict_types turns into a failure. Going through the + # interpreter leaves nothing for it to match, and `output_types` below is + # the documented way to type an output without one. + path: python + args: examples/source.py # The source needs dora-rs, numpy and pyarrow; installing the project gives # it exactly the versions pinned in uv.lock. build: pip install . @@ -23,6 +37,13 @@ nodes: - camera_info - bbox - text + # bbox is deliberately absent: it is untyped on the viewer's side too, + # because senders disagree on 32- vs 64-bit numeric fields. + output_types: + image: std/core/v1/UInt8 + text: std/core/v1/String + depth: std/core/v1/Float32 + camera_info: std/core/v1/Float32 - id: view path: dora-image-view diff --git a/examples/dora-node.yml b/examples/dora-node.yml deleted file mode 100644 index 354c903..0000000 --- a/examples/dora-node.yml +++ /dev/null @@ -1,66 +0,0 @@ -# Manifest for the example source node. -# -# A manifest describes exactly one node, so a second node in this repository -# gets a second manifest. This is the documented way to type a node you have -# not published: "a dora-node.yml sitting next to a plain `path:` node is -# picked up automatically" -- https://dora-rs.ai/dora/hub/node-manifest -# -# It is not optional here. `dora validate` matches a manifest to a `path:` node -# by entrypoint; with no manifest beside examples/source.py the repository-root -# one is tried against it, does not match its `dora-image-view` entrypoint, and -# warns -- which `strict_types: true` turns into a failure. -# -# Declaring the types here rather than as `output_types:` in dataflow.yml is -# what makes them checked: an inline annotation shadows the manifest, so the -# manifest would be injected but never compared against dora-image-view's. -apiVersion: 1 -name: source -namespace: liria -description: Synthetic image, depth, camera_info, bbox and text source for the dora-image-view example -categories: [debug] -keywords: [example, test, synthetic] - -runtime: python -entrypoint: source.py -build: pip install . -dora: ">=1.0" - -inputs: - tick: - required: true - description: Timer tick; one frame is emitted per tick. - -outputs: - image: - type: std/core/v1/UInt8 - description: >- - Animated bgr8 frame as a flat UInt8 buffer, with metadata - {width, height, encoding}. - bbox: - description: >- - The box around the moving square: an Arrow struct {bbox, conf, labels} - with metadata {format: xyxy}. Untyped to match the dora-image-view input. - text: - type: std/core/v1/String - description: A fixed caption. - camera_info: - type: std/core/v1/Float32 - description: >- - [fx, fy, cx, cy] for the frames above. Published once, on the first tick, - the way a real camera publishes it once per session. - depth: - type: std/core/v1/Float32 - description: >- - Depth of the same scene in millimetres, [height, width]: a flat wall with - the square standing in front of it. - -env: - SOURCE_WIDTH: - type: int - description: Frame width in pixels. Defaults to 640. - SOURCE_HEIGHT: - type: int - description: Frame height in pixels. Defaults to 480. - SOURCE_FRAMES: - type: int - description: Stop after this many frames. 0, the default, runs until the viewer quits.