Adjust ROI usage and add voice control docs
This commit is contained in:
100
dora_voice_control/docs/add_robot.md
Normal file
100
dora_voice_control/docs/add_robot.md
Normal file
@@ -0,0 +1,100 @@
|
||||
# Add a New Robot
|
||||
|
||||
This project uses a simple adapter + behavior pattern.
|
||||
|
||||
## 1) Create a robot adapter
|
||||
|
||||
Implement the common command interface:
|
||||
|
||||
- File: `dora_voice_control/dora_voice_control/robots/<robot_name>/adapter.py`
|
||||
- Base class: `RobotAdapter` (`dora_voice_control/dora_voice_control/core/robot.py`)
|
||||
|
||||
Example (vacuum style):
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from ...core.robot import RobotAdapter
|
||||
from ...core.state import RobotStep
|
||||
|
||||
NAME = "my_robot"
|
||||
ALIASES = {"my_robot", "my_alias"}
|
||||
|
||||
class MyRobotAdapter(RobotAdapter):
|
||||
def grab(self) -> List[RobotStep]:
|
||||
return [RobotStep(action="vacuum_on", payload={})]
|
||||
|
||||
def release(self) -> List[RobotStep]:
|
||||
return [RobotStep(action="vacuum_off", payload={})]
|
||||
|
||||
def move(self, payload: Dict[str, Any]) -> List[RobotStep]:
|
||||
return [RobotStep(action="move_to_pose", payload=payload)]
|
||||
|
||||
def reset_tool(self) -> List[RobotStep]:
|
||||
return [RobotStep(action="vacuum_off", payload={})]
|
||||
```
|
||||
|
||||
The `action` strings must match what your robot node understands.
|
||||
|
||||
## 2) Create robot actions
|
||||
|
||||
- File: `dora_voice_control/dora_voice_control/robots/<robot_name>/actions.py`
|
||||
- Use `ActionInfo` to define names, aliases, and requirements.
|
||||
|
||||
```python
|
||||
from ...core.behavior import ActionInfo
|
||||
|
||||
MY_ACTIONS = {
|
||||
"tomar": ActionInfo(name="tomar", aliases=["toma"], requires_object=False),
|
||||
"soltar": ActionInfo(name="soltar", aliases=["suelta"], requires_object=False),
|
||||
}
|
||||
```
|
||||
|
||||
## 3) Create robot behavior
|
||||
|
||||
- File: `dora_voice_control/dora_voice_control/robots/<robot_name>/behavior.py`
|
||||
- Subclass `RobotBehavior` and implement `action_handlers()`.
|
||||
|
||||
```python
|
||||
from typing import Callable
|
||||
from ...core.behavior import ActionContext, RobotBehavior
|
||||
from .actions import MY_ACTIONS
|
||||
|
||||
class MyRobotBehavior(RobotBehavior):
|
||||
ACTIONS = MY_ACTIONS
|
||||
|
||||
def action_tomar(self, ctx: ActionContext) -> bool:
|
||||
self._queue_steps(ctx, self.robot_adapter.grab())
|
||||
return True
|
||||
|
||||
def action_soltar(self, ctx: ActionContext) -> bool:
|
||||
self._queue_steps(ctx, self.robot_adapter.release())
|
||||
return True
|
||||
|
||||
def action_handlers(self) -> dict[str, Callable[[ActionContext], bool]]:
|
||||
return {
|
||||
"tomar": self.action_tomar,
|
||||
"soltar": self.action_soltar,
|
||||
}
|
||||
```
|
||||
|
||||
## 4) Register the robot
|
||||
|
||||
- File: `dora_voice_control/dora_voice_control/robots/__init__.py`
|
||||
|
||||
Add a resolver entry that maps `ROBOT_TYPE` to your adapter/behavior.
|
||||
|
||||
## 5) Update dataflow
|
||||
|
||||
Set `ROBOT_TYPE` in your dataflow:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
ROBOT_TYPE: "my_robot"
|
||||
```
|
||||
|
||||
## Safety notes
|
||||
|
||||
- Keep bounds checks in behavior methods (`_queue_move` already checks workspace limits).
|
||||
- For real hardware, validate with a staged plan: simulation → dry-run → full run.
|
||||
44
dora_voice_control/docs/voice_control_node.md
Normal file
44
dora_voice_control/docs/voice_control_node.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Voice Control Node
|
||||
|
||||
This node turns voice intents into robot commands and publishes them to the Dora graph.
|
||||
|
||||
## Dataflow wiring
|
||||
|
||||
Typical wiring (from `dataflow_voice_control_ulite6_zed.yml`):
|
||||
|
||||
- Inputs
|
||||
- `voice_in` (speech/commands)
|
||||
- `tcp_pose` (current robot TCP pose)
|
||||
- `objects` (detected objects)
|
||||
- `status` (robot status)
|
||||
- `image_annotated` (debug image)
|
||||
- Outputs
|
||||
- `robot_cmd` (robot command queue)
|
||||
- `scene_update` (scene debug)
|
||||
- `voice_out` (debug/feedback)
|
||||
|
||||
## Runtime flow
|
||||
|
||||
1) Voice input is parsed into an intent (action + optional object/color/size).
|
||||
2) The scene is queried for a target object if the action requires it.
|
||||
3) `RobotBehavior.execute()` validates preconditions and dispatches to a handler.
|
||||
4) The handler queues `RobotStep` commands via the adapter.
|
||||
5) `CommandQueueService` sends commands on `robot_cmd`.
|
||||
|
||||
## Key files
|
||||
|
||||
- `dora_voice_control/dora_voice_control/main.py`
|
||||
- Node wiring, intent processing, and dispatch loop.
|
||||
- `dora_voice_control/dora_voice_control/core/behavior.py`
|
||||
- Base behavior, action validation, handler dispatch.
|
||||
- `dora_voice_control/dora_voice_control/robots/`
|
||||
- Robot-specific adapters and behaviors.
|
||||
|
||||
## Environment variables (common)
|
||||
|
||||
- `ROBOT_TYPE`: robot selector (e.g., `vacuum`, `littlehand`)
|
||||
- `COMMAND_OUTPUT`: output port name for robot commands (default `robot_cmd`)
|
||||
- `INIT_ON_START`: queue initial reset + move to home (default `true`)
|
||||
- `INIT_X/Y/Z`, `INIT_ROLL/PITCH/YAW`: home pose
|
||||
|
||||
See `dora_voice_control/README.md` for full configuration.
|
||||
Reference in New Issue
Block a user