_run_control never drained the job queue while the arm had not been reached, so
every request that arrived during an outage waited for a connection that might
never come. That breaks the one-response-per-request rule the node states as
absolute, and it was circular: requests are queued before the arm is reached
because recover is what a caller sends then, yet recover is served by the
control thread and so was the one service that could never run while down.
Serve the queue during the wait between connection attempts. Non-motion services
are tried on the arm, which is what lets recover and clear reach a controller
that answers while refusing to serve. A motion is refused rather than held: the
arm is somewhere unknown and nobody is watching, so a held motion would fire
unattended the moment the link came back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the command/status JSON pair with dora's own service pattern:
one request, one response, correlated by the request_id dora puts in the
message metadata. Motions take a wait flag, and their lifecycle is
published as the arm's state on motion_state rather than as a per-goal
stream — the arm is one resource, so a new motion preempts the one in
play and a state describes that exactly.
motion_state is keyed by request_id and publishes accepted before the
control thread reaches the motion. Both exist to close one race: a caller
polling a bare moving/idle flag right after its request would read "not
moving" and conclude the motion had already finished.
Three threads, because the SDK blocks. The control thread carries out one
job at a time and may block; the reporter thread only ever reads the
report cache, so telemetry keeps flowing through a ten-second move; the
stopper thread exists because a stop fits nowhere else — it must not
queue behind the motion it aborts, and emergency_stop re-commands the
controller for up to three seconds, which is far too long to stop routing
a dataflow.
The event loop never calls anything that can block, get_state included:
that one is allowed there only because it reads the cache. The node is
built before the arm threads start, so INIT_MODE cannot move a robot
before the process has joined the dataflow, and a restart leaves the arm
where it is rather than driving it home unasked. Shutdown halts the arm
first — a motion lives in the controller, and dropping the connection
does not abort it.
cristhianandClaude Opus 5