cesarandClaude Opus 5
15134dc12e
Publish sl::MEASURE::DEPTH as float32 [H, W] millimetres on every frame that carries image_bgr, with the intrinsics, frame_id and timestamps the other outputs carry. The map is copied out rather than published in place: sl::Mat rows can be padded, so the buffer is not necessarily H * W floats. inf and nan are cleared to 0 in the same pass, matching the reference driver — a hole is meaningful and a consumer cannot back-project a non-finite value. Consumers that need 3D can now back-project depth with the published intrinsics instead of receiving a point cloud. Also drops the stale objects entry from the dataflow outputs, missed when detection was removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7g2cBJ5e3r15ZbJ9WZn8c
151 lines
6.0 KiB
Markdown
151 lines
6.0 KiB
Markdown
# dora-zed
|
||
|
||
Dora node for ZED stereo cameras. Publishes BGR images, point clouds and camera intrinsics.
|
||
|
||
## Usage
|
||
|
||
Add to your dataflow:
|
||
|
||
```yaml
|
||
nodes:
|
||
- id: zed-camera
|
||
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
|
||
tag: v0.2.0
|
||
build: scripts/build.sh
|
||
path: bin/dora_zed
|
||
inputs:
|
||
tick: dora/timer/millis/100
|
||
outputs:
|
||
- image_bgr
|
||
- depth
|
||
- point_cloud
|
||
- camera_info
|
||
```
|
||
|
||
See [`dataflow.yml`](dataflow.yml) for a complete runnable example with a camera display node.
|
||
|
||
## Dependencies
|
||
|
||
| Dependency | Version | Notes |
|
||
|---|---|---|
|
||
| [dora-rs](https://github.com/dora-rs/dora) | 1.0+ | Verified against `dora-cli 1.0.1`; the C++ bindings are built from source at configure time |
|
||
| [CMake](https://cmake.org/download/) | 3.21+ | |
|
||
| [ZED SDK](https://www.stereolabs.com/developers/release) | 5.x | Bundles OpenCV; default path `/usr/local/zed` |
|
||
| [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) | — | Required by ZED SDK |
|
||
| [Rust toolchain](https://rustup.rs) | stable | Builds dora-rs C++ bindings |
|
||
|
||
### Installation scripts
|
||
|
||
| Platform | Script |
|
||
|---|---|
|
||
| Ubuntu 24.04 x86_64 | `sudo scripts/install_linux_x64.sh` |
|
||
| NVIDIA Jetson (ARM64) | `sudo scripts/install_linux_arm.sh` |
|
||
| Windows x64 (PowerShell admin) | `scripts/install_windows.ps1` |
|
||
|
||
> **Windows:** Install the ZED SDK manually from [stereolabs.com](https://www.stereolabs.com/developers/release).
|
||
|
||
## Local build
|
||
|
||
```bash
|
||
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
|
||
cd dora-zed
|
||
scripts/build.sh
|
||
```
|
||
|
||
Binary installed to `bin/dora_zed`.
|
||
|
||
`scripts/build.sh` is also the `build:` command in the dataflow. dora 1.0 runs build
|
||
commands directly rather than through a shell, so `cd`, `&&` and command substitution are
|
||
not available there; keeping the sequence in one script is what makes the dataflow build
|
||
work. It configures a `Release` build, which is also what links the optimized dora
|
||
bindings.
|
||
|
||
### Containerized build
|
||
|
||
[`Dockerfile`](Dockerfile) provides the full toolchain — CUDA, ZED SDK and Rust — on Ubuntu 24.04, the newest release Stereolabs ships a ZED SDK for. No camera is needed.
|
||
|
||
```bash
|
||
docker build -t dora-zed-build .
|
||
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
|
||
-v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build
|
||
```
|
||
|
||
The container runs the same `scripts/build.sh`.
|
||
|
||
The first run clones dora-rs and compiles it with Cargo; the named volume keeps that cache across runs.
|
||
`--user` keeps `build/` and `bin/` owned by you rather than root.
|
||
|
||
x86-64 only. On Jetson, run [`scripts/install_linux_arm.sh`](scripts/install_linux_arm.sh) on the
|
||
device and build natively — there is no ARM64 image here.
|
||
|
||
Needs an NVIDIA driver and the
|
||
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
|
||
on the host, as the ZED SDK does anyway. The image sets
|
||
`NVIDIA_DRIVER_CAPABILITIES=compute,utility,video`: `libsl_zed.so` links against NVDEC
|
||
(`libnvcuvid`) and NVENC (`libnvidia-encode`), which the container runtime mounts from the host
|
||
only when `video` is in the capability list.
|
||
|
||
## Configuration
|
||
|
||
All behavior is controlled via environment variables.
|
||
|
||
### Camera
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `ZED_RESOLUTION` | `720` | `720`, `1080`, or `2000` |
|
||
| `ZED_FPS` | `15` | Frame rate |
|
||
| `ZED_DEPTH_MODE` | `NEURAL` | `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
|
||
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth (mm) |
|
||
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth (mm) |
|
||
| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
|
||
| `ZED_FLIP` | `ON` | `ON` or `OFF` |
|
||
| `ZED_WARMUP_FRAMES` | `30` | Frames to discard on startup |
|
||
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Reconnection interval (s) |
|
||
|
||
### Point cloud
|
||
|
||
| `ZED_PC_MODE` | Shape | Description |
|
||
|---|---|---|
|
||
| `FULL` (default) | `[H, W, 4]` | Full resolution grid (~14.7 MB/frame at 720p) |
|
||
| `HALF` | `[H/2, W/2, 4]` | Stride-2 downsample (~4× smaller) |
|
||
|
||
## Inputs
|
||
|
||
| Input | Description |
|
||
|---|---|
|
||
| `tick` | Triggers a camera grab. Connect to `dora/timer/millis/N`. |
|
||
|
||
## Outputs
|
||
|
||
| Output | Type | Description |
|
||
|---|---|---|
|
||
| `image_bgr` | `uint8 [H, W, 3]` | BGR image. Metadata: `shape`, `encoding=bgr8`, intrinsics, timestamps. |
|
||
| `depth` | `float32 [H, W]` | Per-pixel depth in millimetres, published on every frame `image_bgr` is. Metadata: `dtype`, `layout=HW`, `shape`, `units=mm`, intrinsics, timestamps. `H` and `W` match `image_bgr` for the same `frame_id`. Missing depth is `0` — the payload contains no `inf` and no `nan`. Holes are not interpolated unless `ZED_DEPTH_FILL` is set. |
|
||
| `point_cloud` | `float32 [*, 4]` | XYZRGBA point cloud. Shape depends on `ZED_PC_MODE`. Metadata: `units=mm`, intrinsics, timestamps. |
|
||
| `camera_info` | `float32 [4]` | `[fx, fy, cx, cy]`. Metadata: `distortion` coefficients, timestamps. |
|
||
|
||
Consumers that need 3D back-project `depth` with the published intrinsics rather than
|
||
consuming `point_cloud`; the point cloud is not a prerequisite for 3D.
|
||
|
||
### Timestamps
|
||
|
||
Every output carries both clocks, as dora `timestamp` metadata parameters:
|
||
|
||
| Key | Epoch | Unit | Source |
|
||
|---|---|---|---|
|
||
| `device_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `sl::Camera::getTimestamp(TIME_REFERENCE::IMAGE)` — when the camera captured the frame |
|
||
| `host_timestamp_ns` | Unix epoch (UTC) | nanoseconds | `std::chrono::system_clock` on the host — when this node published the frame |
|
||
|
||
Both are needed. The device timestamp is the capture instant and is the value two
|
||
nodes reading the same frame agree on; the host timestamp is the one the rest of the
|
||
dataflow and the session recorder share. On a Jetson the wall clock can step when NTP
|
||
synchronises after boot, so the device timestamp is recorded alongside the host one
|
||
rather than in place of it.
|
||
|
||
The node calls `sl::setTimestampClock(SYSTEM_CLOCK)` before opening the camera, which
|
||
is what makes the device timestamp a Unix-epoch value rather than an SDK-default clock.
|
||
|
||
Every output for a frame also carries the same `frame_id`, an integer counter starting
|
||
at 0.
|