Ingotron 0.9.7 user guide

This guide takes an administrator from a newly installed Robot Vault appliance to a connected robot, a reviewed recording, and a signed dataset. Commands in this guide run on the server or host PC unless stated otherwise.

Know which machine you are using

One Linux server runs the full Ingotron appliance. It stores and processes recordings and hosts the admin web interface. An administrator uses a browser on the server or another trusted PC to open https://<server-hostname-or-IP>:8003.

Each robot has its own Linux computer. Install only the Ingotron robot agent on that computer. The robot agent watches for completed MCAP files and uploads them to the server.

All sudo ingotron ... commands in this guide run on the server. The robot enrollment command in Set up each robot computer runs on that robot. See the configuration reference for port settings.

Set up the server admin interface

First install the appliance on the server by following Set up the server or host PC. Do not run that installer on a robot.

The server installer currently fails closed on RHEL-family and Fedora hosts with SELinux Enforcing until Ingotron ships its packaged SELinux policy. Never disable SELinux as a workaround. Use another supported host or wait for the policy package.

At the end of installation, the terminal prints an HTTPS address, a SHA-256 certificate fingerprint, and a one-time claim code. Keep that terminal open.

1. Open the printed address, normally https://<appliance-address>:8003.
2. The browser initially shows a certificate warning because the appliance uses a private CA. Compare the certificate fingerprint with the installer output before continuing.
3. Enter the claim code in the setup page.
4. Choose the first administrator password, then sign in.

The code is secret, expires, and can be used once. If it expired or may have been exposed, replace it from the host:

sudo ingotron claim-token

To claim the appliance from the server terminal and verify the full certificate chain, run:

sudo ingotron claim

It prompts for the new administrator password. For automation, prefer a root-readable password file:

sudo ingotron claim --password-file /root/ingotron-admin-password

Delete that file securely after the claim succeeds. Do not put the password on the command line. If claim or sign-in recovery is needed later, see Administrator recovery.

Trust the appliance CA

A certificate warning before claim is expected, but do not continue until the fingerprint matches. To install the CA in a browser, OS, or managed trust store, export its public certificate:

sudo ingotron tls ca --out ingotron-ca.crt
sudo ingotron tls fingerprint --ca

Transfer only ingotron-ca.crt, verify the fingerprint out of band, and follow your platform's CA-import procedure. Never copy a key from /opt/ingotron/state/tls/private.

Learn the interface with the demo

Check the appliance, then load the sample data:

sudo ingotron status
sudo ingotron demo

The demo sends three simulated AMR episodes through the normal upload and processing path. It damages one episode so you can see how the quality checks report a problem. The CLI prints a link to the episode view. Some results may take a moment to appear.

Running the demo again is safe because Ingotron recognizes files it already has.

Navigate Robot Vault

The admin interface has these pages:

• Overview shows appliance health, recent activity, fleet state, and items that need attention.
• Intake shows each upload and its processing state. Filter by robot, status, or time, and open an upload for topic and capture details.
• Explore charts messages by robot, type, and time. It also provides a read-only SQL console for advanced exploration and CSV export.
• Review shows episodes, separate DEFECT and RISK scores, evidence, task labels, topic composition, media state, and inclusion decisions.
• Verification shows dataset cuts, export status, manifests, signatures, data cards, evaluations, and provenance.
• Robots creates and manages robot identities and enrollment.

Configure the appliance

Open Admin at https://<server-hostname-or-IP>:8003/ui/admin to view the appliance settings. 0.9.7 shows four values. Only Health-check timeout and Time-index cache size can be edited in the browser. Change either value, select Review changes, inspect the restart summary, and then select Apply reviewed plan. The page waits for the query service to return and confirms the saved values. If it reports that settings changed elsewhere, reload and create a fresh plan.

Allow raw media exports and Keep garbage collection in dry-run mode are read-only in this page. They are controlled by the signed release or the root operator because changing either policy can have irreversible consequences.

Review and Verification are part of the complete Robot Vault edition shipped with this appliance. Actions that change fleet or dataset state require the appropriate administrator or operator role.

An upload can be complete but intentionally absent from processing after a 0.9.0 adoption or upgrade if it predates source-digest enforcement. This is not a reason to force a processor retry. Ask the root operator to run sudo ingotron integrity list and choose the documented re-upload, quarantine, or owner-attestation workflow. Owner attestation is shown as a policy basis; it is not equivalent to a robot-supplied SHA-256.

Read the two quality scores correctly

Robot Vault keeps DEFECT and RISK separate:

• DEFECT asks whether the recording is structurally wrong, for example missing calibration, stream dropouts, timing disagreement, or motion that conflicts with odometry.
• RISK asks whether technically valid data may train poorly, for example blur, frozen images, idle density, or jerky trajectories.

A low score flags evidence for a human; it does not delete data. Exclusion from a dataset is an explicit, auditable decision.

Explore safely

The SQL console accepts read-only queries over the derived Parquet lake. A useful first query is:

SELECT topic, count(*) AS messages
FROM all_parquet
GROUP BY topic
ORDER BY messages DESC
LIMIT 20;

Write statements are rejected. Use narrow time and robot filters for large collections.

Set up each robot computer

Repeat this section for every robot. Never reuse an enrollment command for a second robot. The robot must resolve the server address, have a synchronized clock, and reach the server on TCP 8003 for enrollment and TCP 50051 for MCAP upload over gRPC and TLS.

On the server admin interface

Open the appliance URL printed by the installer and sign in. The direct Robots URL is normally https://<server-hostname-or-IP>:8003/ui/robots. Add robot is the normal setup path.

1. Open Robots and select Add robot.
2. Enter a stable robot identifier and display name.
3. Enter the recording directory on that robot. The default is /data/robot_data.
4. Generate an enrollment command. The one-use code has a 15-minute default lifetime.
5. Copy the complete command. It includes the recording directory and is a secret until it has been used or has expired.

On the robot computer

The robot bootstrap currently supports Linux x86-64 with systemd, OpenSSL 3, curl, tar, and the minimum glibc version listed in the signed agent manifest. Sign in to the intended robot with an account allowed to use sudo, then run the command copied from the server. It has this form:

curl -fsSL https://get.ingotron.com/robot | sudo sh -s -- \
  --vault https://<server-hostname-or-IP>:8003 --code 'ign1...' --root '/data/robot_data'

This command installs the robot agent and a systemd timer. The enrollment code includes the appliance CA pin. The robot fetches the public CA, verifies the pin, and only then exchanges the code for a credential scoped to that robot.

Treat an unexpired enrollment command as a secret. Do not paste it into a ticket or group chat. If the wrong host may have seen it, revoke or replace the code in the UI. A generic "invalid or expired" response is intentional and does not disclose whether an identity exists.

Confirm the connection and first upload

The agent watches the generated --root directory for completed .mcap files. It waits for a file to stop changing before uploading it and retries interrupted transfers. The server recognizes a file it already has, so retrying the same upload does not create another copy.

Return to Robots on the server and wait for the robot to show enrolled and online activity. Place a completed .mcap file in the watched directory on the robot, then open Intake on the server. The robot page and Intake view show whether the appliance has received data. Only recognized ROS 2 schemas produce queryable rows, but the original MCAP is still retained when ingest accepts it.

Re-enroll, disable, or decommission

Use Re-enroll after rebuilding or reimaging a robot, losing its credential, or changing the appliance CA. Confirm the warning, generate a new one-time command, and run it on that robot. You cannot reuse a code after enrollment.

Disable a robot to stop uploads without deleting its history. Before decommissioning a robot, check which audit and data records your retention policy requires. Never copy another robot's credential.

Review a recording

1. Open Intake and confirm the upload advances to completed processing. If an adopted legacy upload is complete but excluded, use the integrity review workflow above rather than repeatedly retrying it.
2. Open Review and filter by robot and capture time.
3. Inspect DEFECT and RISK separately, then open evidence for surprising checks.
4. Correct the task label if needed. Labels feed language-conditioned datasets, so a plausible but wrong label is not harmless metadata.
5. Exclude unsuitable episodes deliberately, with an operator note where your process requires one.

The appliance records media as raw/unscrubbed because this release does not include a GPU PII-scrubbing sidecar. Do not assume faces, licence plates, screens, voices, or location data were removed. See Security.

Create and collect a dataset

In Review, apply the robot, time, task, status, and quality filters you want, then select Create cut. A cut is an immutable selection whose identifier is derived from its canonical manifest. Robot Vault signs that manifest with the appliance's Ed25519 dataset-signing key.

In Verification:

1. Open the cut and review the resolved filter and episode count.
2. Watch each requested export reach a terminal state.
3. Read the data card, quality summary, PII state, and provenance.
4. Download the export, canonical manifest, signature, and public key together.
5. Verify the signature before moving the data into a training system.

LeRobot v3 is the normal structured export. Whole-source MCAP passthrough is intentionally strict: camera-bearing, attached, metadata-bearing, or otherwise unreviewed source is refused unless its exact digest was explicitly approved. A policy refusal can appear as a terminal failed/dead-letter export while a LeRobot export of the same cut succeeds. Read the recorded reason before retrying.

An export can also refuse raw media when the operator has closed the unscrubbed-media gate, or episodes below a configured quality floor. Those checks do not rewrite or silently omit source evidence.

The signature proves the manifest bytes and the object digests named by it. It does not prove that the underlying recording was truthful, that a human label is correct, or that the appliance signing key was never compromised. Preserve the downloaded public key and provenance with the dataset.

Everyday checks

sudo ingotron status
sudo ingotron logs --tail 100
sudo ingotron logs query --tail 100
sudo ingotron version

Use the operations guide for backups, upgrades, address or TLS changes, and removal. Do not operate an appliance with raw docker compose: the CLI owns the required overlay order, project name, state, and recovery journal.

Administrator recovery

If the appliance has not yet been claimed, this replaces the old claim code:

sudo ingotron recover-admin

After claim, the same command repairs or recreates the appliance administrator without reopening public setup. If the separate Keycloak master bootstrap operator is the problem, rotate it with:

sudo ingotron auth reset-bootstrap-credential

Run these recovery commands only as root on the server. Record their use according to your incident procedure.

Troubleshooting

The browser shows a certificate warning

Compare sudo ingotron tls fingerprint --query with the certificate shown by the browser. If they differ, stop: you may be using the wrong address or seeing an intercepting endpoint. If they match, import the CA as described above. An address change may require sudo ingotron set-address; do not hand-edit URLs.

The page loads but sign-in loops or returns to the wrong address

The browser URL, OIDC issuer, redirect URLs, and certificate SANs move as one unit. On the host run:

sudo ingotron status
sudo ingotron set-address https://new-name.example:8003

See Change the appliance address before using --allow-absent or --force.

A service is unhealthy

sudo ingotron status
sudo ingotron logs --tail 200
sudo ingotron logs query --tail 200

Look first at the earliest dependency that is unhealthy (PostgreSQL, MinIO, Keycloak, migrations), not only at a downstream symptom.

A robot cannot enroll

• Confirm TCP 8003 and 50051 reach the same appliance address.
• Confirm the robot clock and DNS are reasonable.
• Generate a new code; codes expire and can be used only once.
• Do not strip or edit the ign1... value.
• If the appliance CA changed, use Re-enroll for that robot.

A robot enrolled but data does not appear

Check the robot's systemd service and timer, then check Robots, Intake, and sudo ingotron logs ingest --tail 200. Confirm .mcap files are under the configured recording tree and are no longer being written. A disabled or decommissioned robot cannot upload.

Processing or exports appear stuck

Processing continues in the background. Check Intake first, then the processor and exporter logs:

sudo ingotron logs processor --tail 200
sudo ingotron logs exporter --tail 200

For a terminal export, read its recorded refusal or error rather than blindly retrying. Common expected refusals involve raw PII-bearing media, quality floors, or whole-source MCAP policy.

The appliance moved or its CA was rotated

Use sudo ingotron set-address for an address change. Routine leaf renewal does not change the CA pin. sudo ingotron tls rotate-ca --yes does: every robot must receive a fresh Re-enroll command afterward.

Next steps

• Operations
• Configuration reference
• Security and limitations
• Release notes

Interrupted uploads in 0.9.7

With both server and robot agent updated to 0.9.7, a retry keeps the chunks the server has confirmed, including after a robot or server restart. The agent sends only missing chunks. Keep the original recording unchanged until upload completion; the server checks the full file's SHA-256 before marking it done. An inactive partial upload expires after seven days and then starts again.

The robot Files and History views distinguish server-saved progress at the last check from bytes sent in the current attempt. A recording's receipt, integrity, and processing status are separate, so an uploaded recording may still be waiting for processing. Older servers use full-file retries.

Update both the appliance and robot agent through the signed release before using durable partial resume.