Troubleshooting

If the arms move in a way you do not expect, press the emergency stop first. See Teleoperation.

Start hereCopy Anchor Link

Two things answer most questions:

1. Doctor. Run it in the directory that holds robot.toml:

   Copy

   arc doctor robot.toml

   Each FAIL line says what is missing and, where there is one, the command that fixes it. See Doctor failures.

2. The session log. Every arc run writes one file to logs/ next to robot.toml. The last lines before the client stopped usually name the cause. Under systemd the same output is in journalctl -u arc.

EnrollmentCopy Anchor Link

A robot that is not enrolled stops like this, and the config commands print the same line:

[yam-relay] ERROR: ~/robot/robot.toml: this robot is not enrolled and [relay] cert is not set. Run: arc enroll

A wrong or rotated key:

[arc] ERROR: enrollment key not accepted: it is unknown or revoked; check the key in your account (server: enrollment key not accepted)

The robot shows RevokedCopy Anchor Link

A robot is revoked when someone selects Revoke in its row, or when the enrollment key is rotated. Rotating the key revokes every robot that enrolled with the old key.

On the robot, arc run keeps running but cannot check in. It prints this every 30 seconds:

[yam-relay] check-in refused (403): this robot's cert (sha256=1a613f90...) is not registered with the relay host, or [relay] robot_id does not match it - server says: client certificate is not registered to any robot; retrying in 30s

The config commands are refused the same way:

[arc] config sync: relay.example.com:443 answered 403 to POST /config/sync: {"error": "client certificate is not registered to any robot"}

What to do: enroll the robot again with the current key and the same name, then restart arc run. With the same name the robot keeps its id. Enroll has the command under "Enroll again".

The robot shows OfflineCopy Anchor Link

Online, but no session startsCopy Anchor Link

The robot shows Online in the console, but Northstar cannot start a session on it.

The likely cause is that outbound UDP is blocked. The robot checks in over TCP port 443, which is why it shows Online. It then has to reach its relay over UDP, on a port from 4433 to 4440. Many office firewalls allow web traffic only.

How to tell: in the session log, a checked in as line is followed by Connect failed (...); retrying in 2.0s, again and again, and no Connected to relay line ever appears.

What to do:

• Ask your network team to allow outbound UDP from the robot computer to ports 4433 to 4440. See How it works.
• Make sure no proxy on the path replaces TLS certificates.
• To confirm the cause, connect the robot computer through a network without the firewall, for example a phone hotspot, and start again.

If the log does show Connected to relay and Streaming., the robot side is working. Tell Northstar the time and the robot id.

Settings sync messagesCopy Anchor Link

arc run prints one config sync: line at each start, and arc config pull prints the same line. See how settings sync.

At a start, a line that reports a problem ends with ; starting from the local file. The robot still starts.

arc config status says state: unknown with the same reason when it cannot read the state:

[arc] config: ~/robot/robot.toml
[arc] local hash: 862bb792...
[arc] state: unknown, relay.example.com:443 answered 403 to GET /config: {"error": "client certificate is not registered to any robot"}

On the Robot settings page:

Doctor failuresCopy Anchor Link

A file with blanks:

[arc] FAIL  config: ~/robot/robot.toml: 5 blank value(s) to fill in before this file can be used: [arms].can_right, [arms].can_left, [[cameras]][0].serial, [[cameras]][1].index, [[cameras]][2].index. Fill them in with arc setup, or edit the file, or arc run will not start

Doctor exits with code 1 when any line is FAIL. A WARN does not fail it.

The arms do not moveCopy Anchor Link

First check that an operator has taken control. The arms do not follow anyone until then. See Teleoperation.

arc doctor --identify reads the arm positions with the motors off. Do not run it while arc run or anything else is driving the arms.

Camera problemsCopy Anchor Link

While a camera is down the rest keeps running. The arms hold until the operator resumes.

Ask Northstar for helpCopy Anchor Link

Send these with your message:

• The session log of the run that had the problem: the file from logs/ next to robot.toml.
• The output of arc doctor robot.toml.
• The version: the output of arc --version.
• The robot id, as shown on the Robots page.
• When it happened, with the time zone, and what you saw.
• For a settings problem, the output of arc config status.
• For a network problem on Wi-Fi, the output of journalctl -t arc-net-watchdog --since today.

Do not send your enrollment key or the file robot_key.pem. Northstar never needs them.