dora-zed/README.md

# dora-zed

Dora node for ZED stereo cameras. Publishes BGR images, point clouds and camera intrinsics.

## Usage

Add to your dataflow:

```yaml
nodes:
  - id: zed-camera
    git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
    tag: v0.2.0
    build: scripts/build.sh
    path: bin/dora_zed
    inputs:
      tick: dora/timer/millis/100
    outputs:
      - image_bgr
      - depth
      - point_cloud
      - camera_info
      - camera_state
```

See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.

## Dependencies

| Dependency | Version | Notes |
|---|---|---|
| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time |
| [CMake](https://cmake.org/download/) | 3.21+ | |
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |

### Installation scripts

| Platform | Script |
|---|---|
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |

> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release).

## Local build

```bash
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
cd dora-zed
scripts/build.sh
```

Binary installed to `bin/dora_zed`.

`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build
commands directly rather than through a shell, so `cd`, `&&` and command substitution are
not available there; keeping the sequence in one script is what makes the dataflow build
work. It configures a `Release` build, which is also what links the optimized dora
bindings.

### Containerized build

[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.

```bash
docker build -t dora-zed-build .
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
    -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
```

The container runs the same `scripts/build.sh`.

The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs.
`--user` keeps `build/` and `bin/` owned by you rather than root.

x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the
device and build natively — there is no ARM64 image here.

Needs an NVIDIA driver and the
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
on the host, as the ZED SDK does anyway. The image sets
`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC
(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host
only when `video` is in the capability list.

## Configuration

All behavior is controlled via environment variables.

### Camera

| Variable | Default | Description |
|---|---|---|
| `ZED_RESOLUTION` | `1080` | `720`, `1080`, or `2000` |
| `ZED_FPS` | `15` | Frame rate |
| `ZED_DEPTH_MODE` | `NEURAL` | `NEURAL`, `NEURAL_PLUS`, `NEURAL_LIGHT`, `PERFORMANCE`, `QUALITY`, or `NONE`. An unrecognised value is a startup error, not a fallback. |
| `ZED_DEPTH_MIN_MM` | `200` | Minimum depth (mm) |
| `ZED_DEPTH_MAX_MM` | `600` | Maximum depth (mm) |
| `ZED_CONFIDENCE_THRESHOLD` | `60` | Reject depth less certain than this (1–100, lower is stricter) |
| `ZED_TEXTURE_CONFIDENCE_THRESHOLD` | `100` | Reject depth on surfaces less textured than this (1–100) |
| `ZED_DEPTH_FILL` | `false` | Interpolate depth over holes. Overrides both confidence thresholds. |
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |

The resolved depth mode, both confidence thresholds and the depth range are logged at
startup. The defaults match the reference camera driver the perception stack was tuned
against, so depth quality on the robot matches what it expects.

### Point cloud

Off by default. With `ZED_POINT_CLOUD` unset the XYZRGBA measure is not retrieved at all
and no `point_cloud` output is published — at 1080p a full cloud is roughly 14.7 MB per
frame, and retrieving it costs capture time whether or not anything consumes it.
Consumers that need 3D back-project `depth` with the published intrinsics instead.

| Variable | Default | Description |
|---|---|---|
| `ZED_POINT_CLOUD` | `false` | Retrieve and publish the point cloud |
| `ZED_PC_MODE` | `FULL` | Only applies when `ZED_POINT_CLOUD` is on |

| `ZED_PC_MODE` | Shape | Description |
|---|---|---|
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 1080p) |
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |

## Inputs

| Input | Description |
|---|---|
| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. |

## Outputs

| Output | Type | Description |
|---|---|---|
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, timestamps. |
| `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. |
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud, only when `ZED_POINT_CLOUD` is on. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. |
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. Published **once per camera session** — with the first frame after the camera opens, and again after a reconnection — not on every frame, since the intrinsics are fixed for a given camera and resolution. |
| `camera_state` | `string` | UTF-8 state name, published on each change. Metadata: `dtype=string`, `encoding=utf8`, `host_timestamp_ns`. |

### Camera state and recovery

`camera_state` lets a downstream node tell a camera that has not delivered its first
frame yet from one that has been lost:

| State | Meaning |
|---|---|
| `opening` | The camera has not delivered a usable frame yet in this run |
| `streaming` | Frames are being published |
| `recovering` | The camera was streaming and has stopped; the node is trying to bring it back |

Recovery runs inside the dora event loop rather than in a retry loop of its own, so the
node keeps reading events — and honours `Stop` — while the camera is down. Reopening
never blocks the loop between attempts: the tick timer paces them, and a backoff deadline
of 1, 2, 5 then 10 seconds decides which ticks actually try.

The node treats an all-black frame as a failed grab: a camera whose link has dropped does
not necessarily fail its grabs, and can keep reporting success while handing back a
uniform frame. A camera that delivers no usable frame for 5 seconds is recovered — 30
seconds if the SDK reports it is reconnecting the link itself, since taking the handle
away mid-reconnect aborts a recovery that was already under way.

The first recovery attempt closes the handle. Past that the device itself is wedged, and
the node escalates to `sl::Camera::reboot()`, which resets it over USB without unplugging
it. That call reaches the ZED 2 and newer only; on the first-generation ZED and the ZED
Mini the node closes the handle and lets the backoff give the endpoint time to clear.
Every attempt and every escalation is logged.


Consumers that need 3D back-project `depth` with the published intrinsics rather than
consuming `point_cloud`; the point cloud is not a prerequisite for 3D.

### Timestamps

Every output carries both clocks, as dora `timestamp` metadata parameters:

| Key | Epoch | Unit | Source |
|---|---|---|---|
| `device_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE)` — when the camera captured the frame |
| `host_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `std::chrono::system_clock` on the host — when this node published the frame |

Both are needed. The device timestamp is the capture instant and is the value two
nodes reading the same frame agree on; the host timestamp is the one the rest of the
dataflow and the session recorder share. On a Jetson the wall clock can step when NTP
synchronises after boot, so the device timestamp is recorded alongside the host one
rather than in place of it.

The node calls `sl::setTimestampClock(SYSTEM_CLOCK)` before opening the camera, which
is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock.

Every output for a frame also carries the same `frame_id`, an integer counter starting
at 0.