163 lines
4.4 KiB
Markdown
163 lines
4.4 KiB
Markdown
# Dora Voice Control Node
|
|
|
|
Dora node that processes Spanish voice commands and translates them into robot actions. Supports multiple robot types via robot subfolders.
|
|
|
|
## Features
|
|
|
|
- Spanish voice command parsing (rule-based or LLM)
|
|
- Robot adapter pattern for different gripper types
|
|
- Real-time web debug interface
|
|
- Command queue management
|
|
- Workspace bounds validation
|
|
|
|
## File Structure
|
|
|
|
```text
|
|
dora_voice_control/
|
|
├── main.py # Thin orchestrator
|
|
│
|
|
├── core/ # Shared logic
|
|
│ ├── behavior.py # RobotBehavior with actions
|
|
│ ├── config.py # Configuration classes
|
|
│ ├── node.py # Dora adapter + dispatcher + context
|
|
│ ├── robot.py # RobotAdapter base
|
|
│ ├── robot_io.py # Pose/status/image handlers + command queue
|
|
│ ├── scene.py # Scene state + notifier + objects handler
|
|
│ ├── state.py # Thread-safe shared state
|
|
│ └── voice.py # Voice input + parsing + intents
|
|
│
|
|
├── robots/ # Robot-specific implementations
|
|
│ └── littlehand/ # Vacuum gripper robot
|
|
│ ├── adapter.py # Vacuum adapter
|
|
│ ├── actions.py # Action vocabulary
|
|
│ └── behavior.py # Behavior binding
|
|
│
|
|
└── web/ # Web interface
|
|
├── api.py # FastAPI server
|
|
├── models.py # Pydantic models
|
|
└── templates.py # HTML template
|
|
```
|
|
|
|
## Robot Adapters
|
|
|
|
Set `ROBOT_TYPE` to select the robot package:
|
|
|
|
| Type | Grab Command | Release Command |
|
|
|------|--------------|-----------------|
|
|
| `littlehand` (alias: `vacuum`) | `vacuum_on` | `vacuum_off` |
|
|
|
|
To add a new robot, create a new subfolder under `robots/` with its adapter and behavior, then register it in `robots/__init__.py`.
|
|
|
|
## Inputs/Outputs
|
|
|
|
| Input | Type | Description |
|
|
|-------|------|-------------|
|
|
| `voice_in` | string | Voice command text |
|
|
| `tcp_pose` | array | Robot pose [x, y, z, roll, pitch, yaw] |
|
|
| `objects` | JSON | Detected objects |
|
|
| `status` | JSON | Command execution status |
|
|
| `image_annotated` | array | Camera image |
|
|
|
|
| Output | Type | Description |
|
|
|--------|------|-------------|
|
|
| `robot_cmd` | JSON | Robot command |
|
|
| `voice_out` | JSON | Response to user |
|
|
| `scene_update` | JSON | Scene state |
|
|
|
|
## Supported Commands (Spanish)
|
|
|
|
| Command | Action | Example |
|
|
|---------|--------|---------|
|
|
| `subir` | Move up | "sube" |
|
|
| `bajar` | Move down | "baja" |
|
|
| `tomar` | Grab object | "agarra el cubo rojo" |
|
|
| `soltar` | Release object | "suelta en la caja azul" |
|
|
| `ir` | Go to object | "ve al cilindro" |
|
|
| `reiniciar` | Reset | "reinicia" |
|
|
|
|
## Environment Variables
|
|
|
|
```bash
|
|
# Robot Configuration
|
|
ROBOT_TYPE=littlehand # "littlehand" (alias: "vacuum")
|
|
|
|
# Web API
|
|
API_ENABLED=true
|
|
API_PORT=9001
|
|
|
|
# TCP Parameters
|
|
TCP_OFFSET_MM=63.0
|
|
APPROACH_OFFSET_MM=50.0
|
|
STEP_MM=20.0
|
|
|
|
# LLM (optional)
|
|
LLM_PROVIDER=rules # "rules", "gemini", "ollama"
|
|
GOOGLE_API_KEY=your_key
|
|
|
|
# Initial Position
|
|
INIT_ON_START=true
|
|
INIT_X=300.0
|
|
INIT_Y=0.0
|
|
INIT_Z=350.0
|
|
|
|
# Safety
|
|
DRY_RUN=false
|
|
WORKSPACE_MIN_Z=0
|
|
WORKSPACE_MAX_Z=500
|
|
```
|
|
|
|
## Web Debug Interface
|
|
|
|
Access at `http://localhost:8080`:
|
|
|
|
- Camera view with detections
|
|
- Real-time status (pose, objects, queue)
|
|
- Send manual commands
|
|
- View parse results
|
|
|
|
## API Endpoints
|
|
|
|
```bash
|
|
# Status
|
|
curl http://localhost:8080/api/status
|
|
|
|
# Objects
|
|
curl http://localhost:8080/api/objects
|
|
|
|
# Send command
|
|
curl -X POST http://localhost:8080/api/command \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"text": "sube"}'
|
|
|
|
# Clear queue
|
|
curl -X POST http://localhost:8080/api/queue/clear
|
|
```
|
|
|
|
## Dataflow Example
|
|
|
|
```yaml
|
|
- id: voice
|
|
build: uv pip install -e dora_voice_control
|
|
path: dora_voice_control/dora_voice_control/main.py
|
|
env:
|
|
ROBOT_TYPE: "vacuum"
|
|
API_ENABLED: "true"
|
|
inputs:
|
|
voice_in: iobridge/text_out
|
|
tcp_pose: robot/tcp_pose
|
|
objects: detector/objects
|
|
status: robot/status
|
|
outputs:
|
|
- robot_cmd
|
|
- voice_out
|
|
- scene_update
|
|
```
|
|
|
|
## Adding a New Robot
|
|
|
|
1) Create `dora_voice_control/dora_voice_control/robots/<robot_name>/` with:
|
|
- `adapter.py` implementing a `RobotAdapter`
|
|
- `actions.py` defining action aliases (can reuse defaults)
|
|
- `behavior.py` binding the behavior class
|
|
2) Register it in `dora_voice_control/dora_voice_control/robots/__init__.py`
|