cesarandClaude Opus 5
5822a870bb
Hardware testing showed the tick-driven recovery state machine did not work. async_grab_camera_recovery does not make grab() prompt on a disconnected ZED-M under SDK 5.3: grabs took ~3 s each while the link was down and then blocked indefinitely. The event loop stopped turning, so Stop went unanswered, the 30 s patience timer and the backoff were never reached, and replugging the camera did nothing. Measured: 122 s frozen, then dora stop needed 10 s and a forced kill. Move everything that can block onto a capture thread. zed::Capture owns the camera and runs open, warmup, grab, blank and stall detection, backoff and escalation; it packs each frame into owned buffers and hands the newest one to the reader by swapping under a mutex. The dora event loop now only drains that buffer and publishes, so it never calls into the SDK and stays responsive however long the SDK blocks. The newest frame wins and uncollected frames are dropped, which is what a consumer wants from a camera source. Swapping rather than assigning returns the previous frame's buffers to the capture thread, so steady state does not reallocate. The point cloud is now copied out of the SDK buffer rather than published in place, since the reader is another thread. On exit the loop waits 2 s for the thread and then _Exit()s if it is still inside the SDK, so Stop is always answered promptly. This also removes the blocking warmup from the event loop: it stays a blocking burst, which is harmless on the capture thread. Verified on a ZED-M: unplug leaves the node responsive and logs the backoff 1, 2, 5 then 10 s; replug gives "camera recovered after 13 attempt(s)" and publishing resumes without restarting the dataflow; dora stop with the camera unplugged returns in 0.70 s with the thread joining gracefully. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7g2cBJ5e3r15ZbJ9WZn8c
63 lines
6.8 KiB
Markdown
63 lines
6.8 KiB
Markdown
# AGENTS.md
|
|
|
|
This file provides guidance to AI coding assistants when working with code in this repository.
|
|
|
|
## What this is
|
|
|
|
A dora-rs C++ node that interfaces with ZED stereo cameras. On each tick it grabs a frame and publishes it over the dora dataflow: a BGR image and a depth map every frame, camera intrinsics once per camera session, a camera state on every transition, and an XYZRGBA point cloud when `ZED_POINT_CLOUD` is on. Camera management lives in `src/zed/` and the dora event loop in `src/main.cc`.
|
|
|
|
The node is a camera source only. It ran inference in-process through `v0.1.0`, under `src/engines/`; that was removed so the node has no inference dependency and builds on any platform the ZED SDK supports. The removed code remains reachable at tag `v0.1.0`.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
scripts/build.sh
|
|
```
|
|
|
|
The binary is installed to `bin/dora_zed` relative to the repo root. The script is also
|
|
the dataflow's `build:` command: dora 1.0 executes build commands directly rather than
|
|
through a shell, so `cd`, `&&` and command substitution do not work there.
|
|
|
|
**Requirements:** dora 1.0+ (verified with `dora-cli 1.0.1`), CMake 3.21+, ZED SDK 5.x (at `/usr/local/zed`), CUDA Toolkit, Rust stable toolchain (for dora-rs C++ bindings).
|
|
|
|
The first build clones dora-rs from GitHub and compiles `dora-node-api-cxx` via Cargo. This takes a while. Subsequent builds are incremental. If `DORA_ROOT_DIR` is set to a local dora checkout, it builds against that instead.
|
|
|
|
For a build without installing the SDK stack on the host, see the `Dockerfile` (Ubuntu 24.04 — the newest release with a ZED SDK; there is no Stereolabs build for 26.04). Run the compile with `--gpus all`.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
dora up
|
|
dora start dataflow.yml
|
|
```
|
|
|
|
Camera behavior is configured entirely through environment variables (see README.md for the full list: `ZED_RESOLUTION`, `ZED_FPS`, `ZED_DEPTH_MODE`, etc.).
|
|
|
|
## Architecture
|
|
|
|
- **`src/util/env.h`** — Shared inline env-variable parsing helpers (`get_env_int`, `get_env_str`, `get_env_bool`, `get_env_float`).
|
|
- **`src/zed/camera.h`** — `zed::Config`, `zed::Calibration`, and `zed::Camera` class. Camera wraps the ZED SDK lifecycle: `open()` (one attempt, returns success), warmup, grab (image + depth + point cloud), calibration, device timestamp, blank-frame check, `close()` and `reset()` (close plus `sl::Camera::reboot()` where the model allows it). Exposes `handle()` for engine access.
|
|
- **`src/zed/capture.h/cc`** — `zed::Capture` and `zed::Frame`. Owns the camera on a dedicated thread: open, warmup, grab, blank/stall detection, backoff and escalation all live here, along with the packing of every payload into owned buffers. The newest frame is handed to the reader by swapping under a mutex (`take_frame`), which also recycles the previous frame's buffers so steady-state capture does not reallocate. **Nothing outside this file may call the ZED SDK** — it blocks unboundedly on a disconnected camera, which is what froze the node when the loop drove the camera directly.
|
|
- **`src/zed/camera.cc`** — Implementation. SDK enum parsers live in an anonymous namespace. `parse_depth_mode` runs in `Config::from_env()`, not in the constructor, so an unrecognised `ZED_DEPTH_MODE` exits before the node joins the dataflow rather than falling back to a default.
|
|
- **`src/main.cc`** — Dora event loop and output publishing only. On any input event (intended to be driven by a tick timer; the id is not inspected): calls `camera.grab()`, then sends `image_bgr` and `depth`, plus `point_cloud` when `ZED_POINT_CLOUD` is on and `camera_info` on the first frame of a camera session (`camera_info_pending`, re-armed on reconnection). `pack_depth` copies the depth map into a tightly packed `[H, W]` float buffer (sl::Mat rows can be padded) and clears `inf`/`nan` to `0`. `set_intrinsics` and `set_frame_stamps` stamp every output of a frame with the same intrinsics, `frame_id` and the two timestamps (`device_timestamp_ns`, `host_timestamp_ns`; both Unix-epoch nanoseconds). Point cloud output varies by mode: `FULL` sends the raw `sl::Mat` pointer (no copy); `HALF` downsamples by 2 in each dimension. The loop never touches the SDK: it drains `take_state_change` and `take_frame` from the capture thread and publishes. On exit it calls `capture.stop()` and waits 2 s for the thread; if the thread is still inside a blocking SDK call it gives up and `_Exit(0)`s rather than let dora time out and kill the node.
|
|
- **`DoraTargets.cmake`** — Handles building or fetching dora-rs and generating the CXX bridge files. Two paths: local (`DORA_ROOT_DIR`) or git clone. Produces `dora_node_api_cxx` library and the `node_bridge.cc` glue.
|
|
- **`scripts/build.sh`** — Configure/compile/install in one shell-free-callable step. Referenced by `dataflow.yml`, the `Dockerfile` and the README.
|
|
- **`dora-node.yml`** — Node manifest (`apiVersion: 1`): typed input/output contract, entrypoint, build command. `dora validate` injects these types into any dataflow that reaches the node by `path:`; a `git:`-sourced node does **not** pick it up, which is why `dataflow.yml` builds from the checkout. Type URNs name the element type (`std/core/v1/UInt8`, `.../Float32`, `.../String`) because every payload is a raw buffer described by its metadata, not an Arrow struct.
|
|
- **`dataflow.yml`** — Example dora dataflow, built from this checkout with `path:` + `build:` and `strict_types: true`. A consumer wires the node from a tag instead (see README) — dora 1.0 pins a source with the mutually exclusive `tag:`, `rev:` or `branch:`.
|
|
- **`.gitea/workflows/validate.yml`** — CI gate running `dora validate dataflow.yml` on push and pull request. Needs no camera and no ZED SDK.
|
|
- **`Dockerfile`** — Ubuntu 24.04 x86-64 build environment (CUDA 12.8, ZED SDK 5.2, Rust). No ARM64/Jetson image; build natively there. No camera needed; run the compile with `--gpus all`. Sets `NVIDIA_DRIVER_CAPABILITIES=compute,utility,video` — the `video` capability is what makes the container runtime mount NVDEC/NVENC, which `libsl_zed.so` links against. Rust lives at `/usr/local/{rustup,cargo}` rather than `/root` so the build can run as the invoking user. `docker build -t dora-zed-build . && docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD:/src" -v dora-zed-cargo:/usr/local/cargo/registry dora-zed-build`.
|
|
|
|
## Key APIs
|
|
|
|
- **dora C++ API** (`<dora-node-api.h>`): `init_dora_node()`, `dora_node.events->next()`, `event_type()`, `send_output_with_metadata()`, `new_metadata()`.
|
|
- **ZED SDK** (`<sl/Camera.hpp>`): `sl::Camera`, `sl::InitParameters`, `sl::RuntimeParameters`, `sl::Mat`.
|
|
- Output data is sent as raw byte slices (`rust::Slice<const uint8_t>`) with typed metadata (shape, dtype, encoding).
|
|
|
|
## Tools
|
|
|
|
- **Issue tracker:** Gitea. Use the Gitea MCP tools (`mcp__gitea__*`) to read and manage issues, pull requests, and other repository operations.
|
|
|
|
## Rules
|
|
|
|
- Do not mention AI tools in commit messages or documentation.
|