fix requests queued while the arm is down never being answered
_run_control never drained the job queue while the arm had not been reached, so every request that arrived during an outage waited for a connection that might never come. That breaks the one-response-per-request rule the node states as absolute, and it was circular: requests are queued before the arm is reached because recover is what a caller sends then, yet recover is served by the control thread and so was the one service that could never run while down. Serve the queue during the wait between connection attempts. Non-motion services are tried on the arm, which is what lets recover and clear reach a controller that answers while refusing to serve. A motion is refused rather than held: the arm is somewhere unknown and nobody is watching, so a held motion would fire unattended the moment the link came back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+39
-1
@@ -339,11 +339,49 @@ class Control:
|
||||
except RobotUnavailableError as error:
|
||||
log(f"{error}; retrying in {self.config.reconnect_interval_s:.0f}s")
|
||||
self._set_state(UNAVAILABLE)
|
||||
self._stopping.wait(self.config.reconnect_interval_s)
|
||||
self._serve_while_down(self.config.reconnect_interval_s)
|
||||
return
|
||||
self._started = True
|
||||
self._set_state(READY)
|
||||
|
||||
def _serve_while_down(self, seconds: float) -> None:
|
||||
"""Answer what can be answered without a serving arm, until it is time to retry.
|
||||
|
||||
This is the wait between connection attempts, and it serves the queue
|
||||
rather than sleeping through it. Sitting silent is the one thing a caller
|
||||
must never get: an arm that is down is an outcome to report, not a reason
|
||||
to stop answering, and a request left in the queue here would be answered
|
||||
only if the arm ever came back.
|
||||
"""
|
||||
deadline = time.monotonic() + seconds
|
||||
while not self._stopping.is_set():
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
return
|
||||
try:
|
||||
job = self._jobs.get(timeout=min(remaining, 0.1))
|
||||
except queue.Empty:
|
||||
continue
|
||||
if isinstance(job, ServiceCall):
|
||||
self._serve_without_arm(job)
|
||||
|
||||
def _serve_without_arm(self, call: ServiceCall) -> None:
|
||||
"""Answer one request that arrived before the arm was serving.
|
||||
|
||||
A motion is refused outright. Holding it would be worse than silence:
|
||||
the arm is somewhere unknown, nobody is watching, and the motion would
|
||||
fire unattended the moment the link came back.
|
||||
|
||||
Everything else is tried on the arm, because a controller that answers
|
||||
while refusing to serve is exactly what `recover` and `clear` are for,
|
||||
and refusing them here would leave the one state they exist to escape
|
||||
with no way out of it.
|
||||
"""
|
||||
if call.moves_the_arm:
|
||||
self._finish_motion(call, FAILED, False, "the arm is not serving yet")
|
||||
return
|
||||
self._respond(call, self._attempt(lambda: services.run(self.robot, call.row)))
|
||||
|
||||
def _start_position(self) -> None:
|
||||
"""Put the arm where INIT_MODE says it should start, if anywhere."""
|
||||
if self.restarted and self.config.init_mode != "none":
|
||||
|
||||
Reference in New Issue
Block a user