cristhian
a4cd493330
The README carried explanations that belong in AGENTS.md; it now follows the shape of the sibling node repositories and is a quarter shorter. The tick description was also wrong: it does not trigger a grab. Capture is free-running at ZED_FPS and a tick forwards the newest frame there is, so the tick sets the publish rate, and frames captured between two ticks are dropped. The usage snippet listed four of the five outputs, which read as an omission rather than the choice it is.
199 lines
9.1 KiB
Markdown
199 lines
9.1 KiB
Markdown
# dora-zed
|
||
|
||
Dora node for ZED stereo cameras. On every tick it publishes a colour image and a depth
|
||
map as `std/media/v1/Image`, plus camera intrinsics, a camera state, and an optional
|
||
point cloud. A source for a vision pipeline: it takes a tick and produces frames.
|
||
|
||
## Usage
|
||
|
||
Add to your dataflow:
|
||
|
||
```yaml
|
||
nodes:
|
||
- id: zed-camera
|
||
git: https://gitea.liria.cl/liria/dora-zed.git
|
||
tag: v0.3.0
|
||
build: scripts/build.sh
|
||
path: bin/dora_zed
|
||
inputs:
|
||
tick: dora/timer/millis/100
|
||
outputs:
|
||
- image
|
||
- depth
|
||
- point_cloud
|
||
- camera_info
|
||
- camera_state
|
||
```
|
||
|
||
The node is C++: building it needs the ZED SDK and CUDA on the machine that runs the
|
||
dataflow — see [Build](#build). An output left out of `outputs:` is simply not published,
|
||
so a dataflow that never turns the point cloud on can drop that line.
|
||
|
||
### Runnable dataflows
|
||
|
||
| File | Contents | Run |
|
||
|---|---|---|
|
||
| [`dataflow.yml`](dataflow.yml) | The camera alone, no viewer and no Python dependency | `dora run dataflow.yml` |
|
||
| [`dataflow_view.yml`](dataflow_view.yml) | The camera plus [dora-image-view][view] | `dora run dataflow_view.yml --uv` |
|
||
|
||
[view]: https://gitea.liria.cl/liria/dora-image-view
|
||
|
||
`dataflow_view.yml` is the worked example of consuming this node: a window opens with
|
||
the colour image and the depth map side by side, the camera model captioned across them.
|
||
Press `q` to quit. It needs a display, and `--uv` so the viewer gets its own managed
|
||
environment.
|
||
|
||
dora has no cross-dataflow wiring, so both files declare the camera node themselves: run
|
||
one at a time. Both build it from the checkout, which is what lets `dora validate` check
|
||
them against [`dora-node.yml`](dora-node.yml) — the typed contract a consumer wires
|
||
against, and what [CI](.gitea/workflows/validate.yml) validates on every push.
|
||
|
||
## Inputs
|
||
|
||
| Input | Description |
|
||
|---|---|
|
||
| `tick` | Publishes the newest frame captured since the last tick, or nothing. Wire to `dora/timer/millis/N`. |
|
||
|
||
The tick does not drive the camera. Capture is free-running on its own thread, paced by
|
||
the SDK to `ZED_FPS`, and each frame overwrites whatever the previous one left
|
||
uncollected; a tick forwards the newest there is, and publishes nothing when none arrived
|
||
since the last one. So the tick sets the publish rate, never the capture rate: below
|
||
`ZED_FPS` it drops frames — the default 100 ms tick against the default 15 fps keeps two
|
||
out of every three — and above it, the ticks that find nothing publish nothing.
|
||
`dora/timer/millis/66` at 15 fps forwards every frame, each at most one capture period
|
||
old and none of them twice.
|
||
|
||
## Outputs
|
||
|
||
| Output | Payload | Metadata |
|
||
|---|---|---|
|
||
| `image` | `std/media/v1/Image` struct, `bgr8` | intrinsics, `frame_id`, timestamps |
|
||
| `depth` | `std/media/v1/Image` struct, `32FC1` | `units=mm`, intrinsics, `frame_id`, timestamps |
|
||
| `camera_info` | `float32` `[fx, fy, cx, cy]` | `distortion`, `frame_id`, timestamps |
|
||
| `point_cloud` | `float32` XYZRGBA, off by default | `shape`, `channels`, `units=mm`, intrinsics, timestamps |
|
||
| `camera_state` | `string` | `host_timestamp_ns` |
|
||
|
||
`image` and `depth` are both `std/media/v1/Image`: an Arrow struct of `width: UInt32`,
|
||
`height: UInt32`, `encoding: Utf8` and `data: LargeBinary`. The payload describes itself,
|
||
so nothing about its shape travels as message metadata. A depth map is an image whose
|
||
pixels are distances, as ROS also treats it: `32FC1` is one float32 per pixel, `units`
|
||
says what those floats measure, and `0` means missing depth — the payload contains no
|
||
`inf` and no `nan`. `depth` is published on every frame `image` is, at the same size and
|
||
`frame_id`.
|
||
|
||
`camera_info` is published **once per camera session** — with the first frame after the
|
||
camera opens, and again after a reconnection — since the intrinsics are fixed for a given
|
||
camera and resolution. Every frame also carries `fx`, `fy`, `cx` and `cy` in its
|
||
metadata, so a consumer that started after the camera did still ends up with them.
|
||
|
||
Every output of a frame carries the same `frame_id` and both clocks as dora `timestamp`
|
||
metadata: `device_timestamp_ns`, when the camera captured the frame, and
|
||
`host_timestamp_ns`, when this node published it — both Unix-epoch nanoseconds. Two nodes
|
||
reading the same frame agree on the device clock; the rest of the dataflow shares the
|
||
host one.
|
||
|
||
### Camera state
|
||
|
||
`camera_state` lets a consumer tell a camera that has not delivered its first frame yet
|
||
from one that has been lost:
|
||
|
||
| State | Meaning |
|
||
|---|---|
|
||
| `opening` | The camera has not delivered a usable frame yet in this run |
|
||
| `streaming` | Frames are being published |
|
||
| `recovering` | The camera was streaming and has stopped; the node is bringing it back |
|
||
|
||
The camera runs on its own thread, since the SDK can block indefinitely inside `grab()`
|
||
and `open()` on a disconnected camera and would otherwise freeze the event loop. A camera
|
||
that delivers no usable frame for 5 seconds is recovered, reopening on a 1, 2, 5 then 10
|
||
second backoff and escalating to `sl::Camera::reboot()` past the first attempt; an
|
||
all-black frame counts as a failed grab, since a camera whose link has dropped can keep
|
||
reporting success while handing back a uniform one. Uncollected frames are dropped, so a
|
||
consumer always gets the newest one.
|
||
|
||
## Configuration
|
||
|
||
Settings are environment variables, which a dataflow can carry per node, keeping them
|
||
with the wiring rather than in whatever shell starts it:
|
||
|
||
```yaml
|
||
env:
|
||
ZED_RESOLUTION: 720
|
||
ZED_FPS: 30
|
||
```
|
||
|
||
A node's `env:` wins over the environment the dataflow was started in; variables it does
|
||
not set still pass through.
|
||
|
||
| Variable | Default | Meaning |
|
||
|---|---|---|
|
||
| `ZED_RESOLUTION` | `1080` | `720`, `1080` or `2000` |
|
||
| `ZED_FPS` | `15` | Frame rate |
|
||
| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY` or `NONE`. An unrecognised value is a startup error, not a fallback. |
|
||
| `ZED_DEPTH_MIN_MM` | `200` | Near end of the depth range, in millimetres |
|
||
| `ZED_DEPTH_MAX_MM` | `600` | Far end of the depth range, in millimetres |
|
||
| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) |
|
||
| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) |
|
||
| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. |
|
||
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
|
||
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
|
||
| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud |
|
||
| `ZED_PC_MODE` | `FULL` | `FULL` is a `[H, W, 4]` grid; `HALF` is a stride-2 downsample. Only applies with `ZED_POINT_CLOUD` on. |
|
||
|
||
The defaults match the reference camera driver the perception stack was tuned against, and
|
||
the resolved depth settings are logged at startup.
|
||
|
||
The point cloud is off because it is expensive: roughly 14.7 MB per frame at 1080p, paid
|
||
in retrieval and in a copy whether or not anything consumes it. Consumers that need 3D
|
||
back-project `depth` with the published intrinsics instead.
|
||
|
||
## Build
|
||
|
||
```bash
|
||
scripts/build.sh
|
||
```
|
||
|
||
The binary is installed to `bin/dora_zed`. The script is also the dataflow's `build:`
|
||
command, since dora 1.0 runs build commands directly rather than through a shell — `cd`
|
||
and `&&` are not available there. The first build clones dora-rs and compiles its C++
|
||
bindings with Cargo, which takes a while; later builds are incremental, and
|
||
`DORA_ROOT_DIR` builds against a local dora checkout instead.
|
||
|
||
**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x at
|
||
`/usr/local/zed` (it bundles OpenCV), the CUDA Toolkit it needs, and a stable Rust
|
||
toolchain for the dora bindings.
|
||
|
||
| Platform | Install the toolchain with |
|
||
|---|---|
|
||
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
|
||
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
|
||
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1`, plus the [ZED SDK][sdk] by hand |
|
||
|
||
[sdk]: https://www.stereolabs.com/developers/release
|
||
|
||
[`Dockerfile`](Dockerfile) carries the same toolchain on Ubuntu 24.04 — the newest release
|
||
Stereolabs ships a ZED SDK for — so the host needs none of it, only an NVIDIA driver and
|
||
the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html):
|
||
|
||
```bash
|
||
docker build -t dora-zed-build .
|
||
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
|
||
-v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
|
||
```
|
||
|
||
It runs the same `scripts/build.sh`. The named volume keeps the Cargo cache across runs
|
||
and `--user` keeps `build/` and `bin/` owned by you. x86-64 only: on Jetson, run
|
||
[`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) and build natively.
|
||
|
||
## Test
|
||
|
||
```bash
|
||
tests/run.sh
|
||
```
|
||
|
||
Unit tests over `src/arrow`, which packs every payload this node sends: each exported
|
||
Arrow array is round-tripped through pyarrow to check what a consumer reads back, and
|
||
built again under the sanitizers to check that releasing one frees it. They need neither a
|
||
camera, the ZED SDK nor dora, which is what lets CI run them; everything past the packing
|
||
is covered by running a dataflow against a real camera.
|