dora-zed/README.md

# dora-zed

Dora node for ZED stereo cameras. Publishes a BGR image and a depth map on every frame, plus camera intrinsics, a camera state, and an optional point cloud.

## Usage

Add to your dataflow:

```yaml
nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    tag: v0.2.0
    build: scripts/build.sh
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
    outputs:
      - image_bgr
      - depth
      - point_cloud
      - camera_info
      - camera_state
```

### Runnable examples

Two dataflows live here, and they are run one at a time — dora has no cross-dataflow
wiring, so each declares the camera node itself:

| File | Contents | Run |
|---|---|---|
| [`dataflow.yml`](dataflow.yml) | The camera alone | `dora run dataflow.yml` |
| [`dataflow_plot.yml`](dataflow_plot.yml) | The camera plus an `opencv-plot` viewer on `image_bgr` | `dora run dataflow_plot.yml` |

The viewer is a separate file because it pulls [dora-hub](https://github.com/dora-rs/dora-hub)
and installs it with `pip`, and it needs a display. Neither is a cost the camera should
carry on a headless robot, so `dataflow.yml` has no Python dependency at all.

Both build the node from the checkout (`path:` + `build:`) rather than from a tag, which
is what lets `dora validate` pick up the manifest described below.

### Node manifest

[`dora-node.yml`](dora-node.yml) declares the node's inputs and outputs with type URNs.
No source package is shared between node repositories, so it is the only contract a
consumer has, and it is what makes a wiring mistake fail at build time instead of at
runtime:

```
$ dora validate dataflow.yml
  node "zed-camera": injected 6 contract type(s) from .../dora-node.yml
All type annotations OK.
```

Types are `std/core/v1/UInt8` for `image_bgr`, `std/core/v1/Float32` for `depth`,
`camera_info` and `point_cloud`, and `std/core/v1/String` for `camera_state`; every
payload is a raw buffer whose shape and dtype are in the metadata, so the URN names the
element type rather than a struct. The `tick` input is `std/core/v1/UInt64`, matching how
dora types timer inputs.

`strict_types: true` in both dataflows makes a type warning a non-zero exit.
[`.gitea/workflows/validate.yml`](.gitea/workflows/validate.yml) validates both on every
push and pull request; it needs no camera and no ZED SDK, since validation never runs the
node.

## Dependencies

| Dependency | Version | Notes |
|---|---|---|
| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time |
| [CMake](https://cmake.org/download/) | 3.21+ | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |

### Installation scripts

| Platform | Script |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |

> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release).

## Local build

```bash
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
scripts/build.sh
```

Binary installed to `bin/dora_zed`.

`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build
commands directly rather than through a shell, so `cd`, `&&` and command substitution are
not available there; keeping the sequence in one script is what makes the dataflow build
work. It configures a `Release` build, which is also what links the optimized dora
bindings.

### Containerized build

[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.

```bash
docker build -t dora-zed-build .
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
    -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
```

The container runs the same `scripts/build.sh`.

The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs.
`--user` keeps `build/` and `bin/` owned by you rather than root.

x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the
device and build natively — there is no ARM64 image here.

Needs an NVIDIA driver and the
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
on the host, as the ZED SDK does anyway. The image sets
`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC
(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host
only when `video` is in the capability list.

## Configuration

All behavior is controlled via environment variables.

### Camera

| Variable | Default | Description |
|---|---|---|
| `ZED_RESOLUTION` | `1080` | `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY`, or `NONE`. An unrecognised value is a startup error, not a fallback. |
| `ZED_DEPTH_MIN_MM` | `200` | Minimum depth (mm) |
| `ZED_DEPTH_MAX_MM` | `600` | Maximum depth (mm) |
| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) |
| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) |
| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |

The resolved depth mode, both confidence thresholds and the depth range are logged at
startup. The defaults match the reference camera driver the perception stack was tuned
against, so depth quality on the robot matches what it expects.

### Point cloud

Off by default. With `ZED_POINT_CLOUD` unset the XYZRGBA measure is not retrieved at all
and no `point_cloud` output is published — at 1080p a full cloud is roughly 14.7 MB per
frame, and retrieving it costs capture time whether or not anything consumes it.
Consumers that need 3D back-project `depth` with the published intrinsics instead.

| Variable | Default | Description |
|---|---|---|
| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud |
| `ZED_PC_MODE` | `FULL` | Only applies when `ZED_POINT_CLOUD` is on |

| `ZED_PC_MODE` | Shape | Description |
|---|---|---|
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 1080p) |
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |

The cloud is copied out of the SDK buffer, since the capture thread and the event loop are
different threads. That copy is the main reason to leave `ZED_POINT_CLOUD` off unless a
consumer needs it.

## Inputs

| Input | Description |
|---|---|
| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. |

## Outputs

| Output | Type | Description |
|---|---|---|
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, timestamps. |
| `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. |
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud, only when `ZED_POINT_CLOUD` is on. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. |
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. Published **once per camera session** — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution. |
| `camera_state` | `string` | UTF-8 state name, published on each change. Metadata: `dtype=string`, `encoding=utf8`, `host_timestamp_ns`. |

### Camera state and recovery

`camera_state` lets a downstream node tell a camera that has not delivered its first
frame yet from one that has been lost:

| State | Meaning |
|---|---|
| `opening` | The camera has not delivered a usable frame yet in this run |
| `streaming` | Frames are being published |
| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back |

The camera runs on its own thread. The ZED SDK can block for an unbounded time inside
`grab()` and `open()` when the camera is disconnected — measured blocking indefinitely on
a ZED-M even with `async_grab_camera_recovery` set — so driving it from the dora event
loop would freeze the node: it would stop reading events, leaving `Stop` unanswered and
the recovery logic itself unreachable. The event loop only reads the newest frame the
capture thread has already produced, and therefore never calls into the SDK.

Reopening is paced by a backoff of 1, 2, 5 then 10 seconds, the last repeated. Frames that
are not collected before the next one arrives are dropped: a consumer always gets the
newest frame rather than a queue of stale ones.

The node treats an all-black frame as a failed grab: a camera whose link has dropped does
not necessarily fail its grabs, and can keep reporting success while handing back a
uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30
seconds if the SDK reports it is reconnecting the link itself, since taking the handle
away mid-reconnect aborts a recovery that was already under way.

The first recovery attempt closes the handle. Past that the device itself is wedged, and
the node escalates to `sl::Camera::reboot()`, which resets it over USB without unplugging
it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED
Mini the node closes the handle and lets the backoff give the endpoint time to clear.
Every attempt and every escalation is logged.

### Timestamps

Every frame output — `image_bgr`, `depth`, `point_cloud` and `camera_info` — carries both
clocks, as dora `timestamp` metadata parameters. `camera_state` carries
`host_timestamp_ns` only, since it is not tied to a frame.

| Key | Epoch | Unit | Source |
|---|---|---|---|
| `device_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE)` — when the camera captured the frame |
| `host_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `std::chrono::system_clock` on the host — when this node published the frame |

Both are needed. The device timestamp is the capture instant and is the value two
nodes reading the same frame agree on; the host timestamp is the one the rest of the
dataflow and the session recorder share. On a Jetson the wall clock can step when NTP
synchronises after boot, so the device timestamp is recorded alongside the host one
rather than in place of it.

The node calls `sl::setTimestampClock(SYSTEM_CLOCK)` before opening the camera, which
is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock.

Every output for a frame also carries the same `frame_id`, an integer counter starting
at 0.