cristhian aguilera
6e6dc8ef27
Add engines::ZedNative class wrapping ZED SDK's TensorRT-optimized object detection. On every frame grab, detected objects (label, confidence, 3D position, 2D/3D bounding boxes) are serialized as a flat float32 array and sent as a new "objects" dora output with frame_id correlation to image_bgr. Extract shared env-var helpers into src/util/env.h and expose sl::Camera& via Camera::handle() for engine access. Closes #3
92 lines
3.2 KiB
Markdown
92 lines
3.2 KiB
Markdown
# dora-zed
|
||
|
||
Dora C++ node for ZED stereo cameras. Captures frames and publishes BGR images, XYZRGBA point clouds, camera intrinsics, and 3D object detections.
|
||
|
||
## Usage via `git:` in a dataflow
|
||
|
||
```yaml
|
||
nodes:
|
||
- id: zed-camera
|
||
git: ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
|
||
build: mkdir -p build && cd build && cmake .. && make install
|
||
path: bin/dora_zed
|
||
inputs:
|
||
tick: dora/timer/millis/100
|
||
outputs:
|
||
- image_bgr
|
||
- point_cloud
|
||
- camera_info
|
||
- objects
|
||
```
|
||
|
||
Then run:
|
||
|
||
```bash
|
||
dora build dataflow.yml
|
||
dora up
|
||
dora start dataflow.yml
|
||
```
|
||
|
||
## Local build
|
||
|
||
```bash
|
||
git clone ssh://git@gitea.liria.cl:5001/liria/dora-zed.git
|
||
cd dora-zed
|
||
mkdir -p build && cd build
|
||
cmake ..
|
||
make install
|
||
```
|
||
|
||
The binary is installed to `bin/dora_zed` relative to the repository root.
|
||
|
||
## Dependencies
|
||
|
||
| Dependency | Minimum version | Notes |
|
||
|---|---|---|
|
||
| CMake | 3.21 | |
|
||
| ZED SDK | 4.x | Default path: `/usr/local/zed` |
|
||
| CUDA Toolkit | Required by ZED SDK | |
|
||
| OpenCV | 4.x | |
|
||
| Rust toolchain | stable | Used to build dora-rs C++ bindings |
|
||
|
||
## Environment variables
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `ZED_RESOLUTION` | `720` | Camera resolution: `720`, `1080`, or `2000` |
|
||
| `ZED_FPS` | `15` | Camera frame rate |
|
||
| `ZED_DEPTH_MODE` | `NEURAL` | Depth mode: `PERFORMANCE`, `QUALITY`, `NEURAL`, or `NONE` |
|
||
| `ZED_DEPTH_MIN_MM` | `10` | Minimum depth distance in millimeters |
|
||
| `ZED_DEPTH_MAX_MM` | `500` | Maximum depth distance in millimeters |
|
||
| `ZED_DEPTH_FILL` | `false` | Enable depth fill mode |
|
||
| `ZED_FLIP` | `ON` | Image flip: `ON` or `OFF` |
|
||
| `ZED_WARMUP_FRAMES` | `30` | Number of warmup frames to discard |
|
||
| `ZED_RETRY_INTERVAL_SECONDS` | `5` | Seconds between reconnection attempts |
|
||
| `ZED_DETECTION_MODEL` | `MULTI_CLASS_BOX_MEDIUM` | Detection model: `MULTI_CLASS_BOX_FAST`, `MULTI_CLASS_BOX_MEDIUM`, or `MULTI_CLASS_BOX_ACCURATE` |
|
||
| `ZED_DETECTION_CONFIDENCE` | `40` | Detection confidence threshold (0-100) |
|
||
| `ZED_DETECTION_TRACKING` | `false` | Enable object tracking across frames (enables positional tracking) |
|
||
|
||
## Outputs
|
||
|
||
### `image_bgr`
|
||
BGR image as raw bytes. Metadata includes `shape` (HxWx3), `encoding` (`bgr8`), camera intrinsics (`fx`, `fy`, `cx`, `cy`), `distortion`, `frame_id`, and `timestamp_ns`.
|
||
|
||
### `point_cloud`
|
||
XYZRGBA point cloud as float32 bytes. Metadata includes `shape` (HxWx4), `units` (`mm`), and camera intrinsics.
|
||
|
||
### `camera_info`
|
||
Four float32 values: `[fx, fy, cx, cy]`. Metadata includes `distortion` coefficients.
|
||
|
||
### `objects`
|
||
Detected objects as a flat float32 array with shape `[num_objects, 37]`. Each row contains 37 floats:
|
||
|
||
| Index | Field | Description |
|
||
|---|---|---|
|
||
| 0 | `label_id` | Object class (0=PERSON, 1=VEHICLE, 2=BAG, 3=ANIMAL, 4=ELECTRONICS, 5=FRUIT_VEGETABLE, 6=SPORT) |
|
||
| 1 | `confidence` | Detection confidence (0-100) |
|
||
| 2-4 | `x, y, z` | 3D center position in mm |
|
||
| 5-12 | `bb2d_*` | 2D bounding box: 4 corners × 2 pixel coords |
|
||
| 13-36 | `bb3d_*` | 3D bounding box: 8 corners × 3 world coords (mm) |
|
||
|
||
Metadata includes `frame_id` and `timestamp_ns` matching the corresponding `image_bgr`, plus `image_width` and `image_height` for interpreting 2D bounding box pixel coordinates. When no objects are detected, an empty array with shape `[0, 37]` is sent.
|