diff --git a/AGENTS.md b/AGENTS.md index e2dc6af..1823adf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,13 +34,13 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless - **`dora_image_view/payload.py`** — `as_float32()`: reinterprets a `uint8` payload rather than converting it, because a C++ node sending a raw byte slice produces a `UInt8` arrow array despite declaring `Float32`. Shared by `depth` and `intrinsics`, which both meet it. - **`dora_image_view/intrinsics.py`** — `Intrinsics`, `parse()` for the `camera_info` payload and `from_metadata()` for the copy senders attach to each frame. Both exist because dora-zed publishes the same numbers twice, and the per-frame copy is the one that survives a viewer starting after the camera did — `camera_info` goes out **once per session**. A nine-element row-major K is accepted alongside the compact four values, since that is how ROS spells the same model. The node needs none of this to draw; it is held for whatever turns depth into 3D, and drawn so a wrong camera model is visible rather than merely plausible. - **`dora_image_view/bbox.py`** — Parses the `bbox` struct into xyxy corners. `xywh` is centre-based (x, y are the box centre). `_flat()` unwraps the single-element list that `pa.array([{...}])` produces around each field, and reads a field stored flat as-is. -- **`examples/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the box around the square, and a caption. It is both the demo and the CI gate, and the second role has been shaping it: it publishes depth and camera_info because those inputs are *typed*, and `dora validate --strict` fails on a typed input the dataflow does not wire. Before adding an output here, know which of the two reasons you are adding it for. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout. +- **`tests/source.py`** — The synthetic source node: an animated frame, a depth map of the same scene (a flat wall with the square in front of it), the intrinsics, the box around the square, and a caption. A **test fixture**, not an example — it lives in `tests/` because that is what it is, and it publishes depth and camera_info because those inputs are *typed* and `dora validate --strict` fails on a typed input the dataflow does not wire. It is a dora node rather than a pytest case because it exists to exercise the wiring and the event loop, which only a running dataflow covers; pytest does not collect it, since the filename does not start with `test_`. `SOURCE_FRAMES` bounds the run, which is what lets CI wait for a clean exit instead of killing it on a timeout. - **`dora-node.yml`** — Node manifest (`apiVersion: 1`). `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why the example builds from the checkout. `image` and `text` carry type URNs naming the element type because each payload is a raw buffer described by its metadata, not an Arrow struct. `bbox` is deliberately **untyped**: senders disagree on whether its numeric fields are 32- or 64-bit, and the node reads either. -- **`dataflow.yml`** — The example and the CI gate, with `strict_types: true`. The viewer is a `path:` node, so its manifest is injected; the source is typed inline with `output_types:` and the two are compared against each other. `bbox` is annotated on neither side, matching the manifest. +- **`dataflow.yml`** — The end-to-end test, with `strict_types: true`. It sits at the repository root, not beside the fixture, because a node's working directory is the dataflow file's directory and both nodes build with `pip install .`, which needs `pyproject.toml` there. The viewer is a `path:` node, so its manifest is injected; the source is typed inline with `output_types:` and the two are compared against each other. `bbox` is annotated on neither side, matching the manifest. **One manifest, on purpose.** A `dora-node.yml` describes exactly one node (the schema is `NodeManifest`, singular) and is the unit `dora hub publish` operates on — it expects one beside `pyproject.toml`. This repository ships one node, so it has one manifest. The example source is a test fixture and gets none. - That is why the source runs as `path: python` + `args:` rather than `path: examples/source.py`. `dora validate` matches a manifest to a `path:` node **by entrypoint**, so a path node here would be tried against the root manifest, would fail to match its `dora-image-view` entrypoint, and would warn — and `strict_types: true` makes that warning fatal. Going through the interpreter leaves nothing to match. The cost is that the fixture gets no managed venv of its own (`.dora/python-envs/` stays empty for it) and runs against the project environment instead; the shipped node still uses `path:` + `build:`, which is the mechanism a consumer actually uses. Do not "fix" this back into a path node without re-reading this paragraph. + That is why the source runs as `path: python` + `args:` rather than `path: tests/source.py`. `dora validate` matches a manifest to a `path:` node **by entrypoint**, so a path node here would be tried against the root manifest, would fail to match its `dora-image-view` entrypoint, and would warn — and `strict_types: true` makes that warning fatal. Going through the interpreter leaves nothing to match. The cost is that the fixture gets no managed venv of its own (`.dora/python-envs/` stays empty for it) and runs against the project environment instead; the shipped node still uses `path:` + `build:`, which is the mechanism a consumer actually uses. Do not "fix" this back into a path node without re-reading this paragraph. - **`.gitea/workflows/validate.yml`** — CI: lints, runs the unit tests, validates the dataflow and runs it headless. Needs no camera and no display. One job on purpose — the gates themselves take about a second, and are dwarfed by preparing the runner. ## Why CI installs libgl1 @@ -55,7 +55,7 @@ CI=true SOURCE_FRAMES=30 dora run dataflow.yml --uv # the same, headless ## A caveat on typed optional inputs -`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `examples/source.py` publishes a depth map and a camera_info at all: both are typed, so the example has to drive them, and a new output it sends needs a matching entry in the dataflow's `output_types:`. +`dora validate --strict` injects a contract type for *every* typed manifest input and then warns `input_types key "..." not found in inputs list` for any the dataflow does not wire. So a typed optional input forces the example dataflow to wire it. If a typed input is added to the manifest, either wire it in `dataflow.yml` or leave it untyped. This is why `tests/source.py` publishes a depth map and a camera_info at all: both are typed, so the fixture has to drive them, and a new output it sends needs a matching entry in the dataflow's `output_types:`. ## Checking what it draws diff --git a/README.md b/README.md index 40e034f..567fd1f 100644 --- a/README.md +++ b/README.md @@ -35,13 +35,14 @@ wiring several shows them side by side, scaled to a common height. `bbox`, `text are overlays rather than panels — boxes are drawn on the `image` panel, in its own pixels, and the caption on the composite. -### Runnable example +### End-to-end test -[`dataflow.yml`](dataflow.yml) is self-contained. [`examples/source.py`](examples/source.py) +[`dataflow.yml`](dataflow.yml) is self-contained. [`tests/source.py`](tests/source.py) is a synthetic source node that emits an animated frame, a depth map of the same scene, the camera intrinsics, the box around the square moving in the frame, and a caption — -every input this node draws — so the example needs no camera, no detector and no -network: +every input this node draws — so it needs no camera, no detector and no network. It is +a test fixture rather than a worked example; the realistic wiring is the block at the +top of this file. Run it to watch the node work: ```bash dora run dataflow.yml --uv @@ -53,11 +54,8 @@ to quit. It builds the viewer from this checkout (`path:` + `build:`) rather than from a tag, which is what lets `dora validate` pick up the manifest described below. -This example is also the end-to-end gate: CI runs the same dataflow headless on every -push, which is what keeps it from rotting. It earns its place twice over — it is the -only worked example of the *producing* side of these contracts, which is the fiddly -half of wiring this node. The realistic wiring, against a real camera, is the block at -the top of this file. +CI runs the same dataflow headless on every push, which is what makes it a gate rather +than a demo that rots. Set `SOURCE_FRAMES` to stop the source after a fixed number of frames instead of running until you quit; that plus `CI=true` is how the same dataflow runs headless as a test: diff --git a/dataflow.yml b/dataflow.yml index 28ade4b..4d09992 100644 --- a/dataflow.yml +++ b/dataflow.yml @@ -1,6 +1,11 @@ -# The repository's own example, and its end-to-end test: a synthetic source -# feeding the viewer. Everything it needs is in this checkout -- no camera, no -# detector, no network -- so it is also what CI runs. +# The end-to-end test: a synthetic source feeding the viewer. Everything it +# needs is in this checkout -- no camera, no detector, no network -- which is +# what lets CI run it rather than only validate it. +# +# It lives at the repository root rather than beside tests/source.py because a +# node's working directory is the directory of the dataflow file, and both +# nodes build with `pip install .`, which needs pyproject.toml in that +# directory. # # Both nodes are built from the checkout with `path:` + `build:`, which is what # lets `dora validate` pick up the typed contract in dora-node.yml. A consumer @@ -11,7 +16,7 @@ strict_types: true nodes: - id: source - # Run through the interpreter rather than as `path: examples/source.py`. + # Run through the interpreter rather than as `path: tests/source.py`. # # A dora-node.yml describes exactly one node and is what gets published # (`dora hub publish` expects it beside pyproject.toml), and this @@ -25,7 +30,7 @@ nodes: # interpreter leaves nothing for it to match, and `output_types` below is # the documented way to type an output without one. path: python - args: examples/source.py + args: tests/source.py # The source needs dora-rs, numpy and pyarrow; installing the project gives # it exactly the versions pinned in uv.lock. build: pip install . diff --git a/examples/source.py b/tests/source.py similarity index 86% rename from examples/source.py rename to tests/source.py index 7dde418..50db001 100644 --- a/examples/source.py +++ b/tests/source.py @@ -1,8 +1,13 @@ -"""Synthetic source node: this repository's stand-in for a camera and a detector. +"""Synthetic source node: the test's stand-in for a camera and a detector. -Emits an animated frame, a depth map of the same scene, the box around the square -moving in it, and a caption -- every input dora-image-view draws -- so the example -dataflow needs no camera, no detector and no network. +Emits an animated frame, a depth map of the same scene, the intrinsics, the box +around the square moving in it, and a caption -- every input dora-image-view draws -- +so dataflow.yml needs no camera, no detector and no network. + +This is a test fixture, not an example of how to wire the node; see the README for +that. It is a dora node rather than a pytest case because what it exists to exercise +is the wiring and the event loop, which only a running dataflow covers. pytest does +not collect it: the filename does not start with `test_`. The depth map describes the same scene as the image: a flat back wall with the square standing in front of it. `camera_info` is published once, on the first tick, the way a