Ingotron 0.9.7 operations guide

This runbook is for the root-capable operator of a single-host Robot Vault appliance. It covers the installed 0.9.7 appliance, not the legacy checkout stack or the separate bring-your-own-OIDC production Compose file. The appliance runs on the server or host PC. Robot computers run only the robot agent. See the user guide for the deployment layout and initial setup.

Operating contract

Use /usr/local/bin/ingotron for appliance operations. Every command requires root because it controls Docker and root-owned state.

sudo ingotron status
sudo ingotron version
sudo ingotron <command> --help

Do not run the released Compose files individually. The CLI combines them in the required order with the base stack, bundled Keycloak setup, TLS settings, and, for adopted installs, the retained volume mapping. It also sets the project name to ingotron, selects the installed release and environment, and updates the recovery journal. Running docker compose up, down, or pull yourself can start the evaluation setup or create a new, empty set of volumes. For routine operator changes, use Admin → Configuration in the appliance UI. Root operators can use the matching narrow sudo ingotron config commands when the UI is unavailable.

The full appliance runs on one server only. Admin UI actions may come from any trusted browser, and each robot runs only the small agent. On RHEL-family and Fedora hosts, an Enforcing SELinux policy is currently a deliberate installer and upgrade refusal until the packaged Ingotron policy is available. Do not disable SELinux; use a supported host or wait for that policy.

Files and state

Mutable data also lives in seven Docker volumes recorded under volume_map in install.json: pg_data, minio_data, uploads, processor_work, exporter_work, query_work, and demo_data. Never infer their names on an adopted installation.

Health and logs

sudo ingotron status
sudo ingotron status --json
sudo ingotron logs --tail 200
sudo ingotron logs query --tail 200
sudo ingotron logs ingest --tail 200
sudo ingotron version --json

status checks more than container state: it reports the installed and running release, claim state, HTTPS/OIDC readiness, ingest TLS readiness, certificate SAN drift, legacy-token grace, and interrupted work. Treat a warning or non-zero result as an operational condition, not cosmetic output.

logs passes its arguments to the canonical docker compose logs invocation. Service names commonly useful for diagnosis are postgres, minio, keycloak, ingest, processor, exporter, query, and gc.

Managed configuration

Open https://<server-hostname-or-IP>:8003/ui/admin and use the Configuration panel. It shows four values:

Edit an allowed value, select Review changes, inspect the affected service, then select Apply reviewed plan. The manager validates the candidate, recreates the owning service when needed, waits for health, and restores the previous configuration if the check fails. A plan is tied to the current revision, release, and running service set. If any of those changed, reload the panel and review a fresh plan.

Applying the cache setting restarts the query service. A brief disconnect while it restarts is expected; the UI polls until it can confirm the saved value. If confirmation does not arrive, check status and query logs before retrying. Do not edit /opt/ingotron/state/env or use raw Compose to recover a failed change. The next configuration request performs any safe interrupted-transaction recovery; if the CLI reports that recovery needs attention, keep the stack stopped and follow that message before making another change.

Backup

Routine CLI backup

Run a backup before every upgrade or risky change, and on a schedule appropriate to your recovery-point objective:

sudo ingotron backup

The command writes a timestamped directory under /opt/ingotron/state/backups/. To write under another parent directory:

sudo ingotron backup --out /srv/offhost-staging/ingotron

Each backup contains:

• checksummed custom-format dumps of both robotvault and keycloak;
• minio-objects.tar, a complete object-API mirror of the configured product bucket;
• env, optional env.meta.json, install.json, and the adopted install's compose.override.yml when one exists;
• service secrets, including keycloak_db_password and the irreplaceable dataset-signing key;
• the appliance CA, CA private key, and listener material; and
• backup.json, whose exact artifact allowlist records every file's digest and size, the release version, schema version, and Keycloak image digest, plus backup.json.hmac, which authenticates the exact manifest bytes.

Publication fails if the backup contains an unlisted file or directory, misses a listed artifact, or contains a symbolic link or special file. Restore applies the same exact-inventory check before it trusts any payload.

The first backup creates /opt/ingotron/backup-auth.key as a root-owned mode-0600 file outside appliance state. The key is intentionally absent from every backup. Copy it to a separate protected location before relying on the archive for disaster recovery:

sudo install -m 0600 -o root -g root \
  /opt/ingotron/backup-auth.key /secure/off-host/ingotron-backup-auth.key

Do not put the key under /opt/ingotron/state or beside a backup. The CLI refuses a key under appliance state because the same archive must never contain both a manifest and the secret needed to forge its authentication tag.

The Keycloak bootstrap password is deliberately omitted because recovery rotates it. Protect the archive as highly sensitive anyway: it contains database credentials, the dataset-signing private key, and the appliance CA private key. Encrypt it, restrict access, copy it off the appliance, and test a restore.

A standalone backup has a short write outage. The CLI records the exact subset of query, ingest, processor, exporter, gc, robot, and keycloak that is running, durably journals that set, and stops only those services. It takes both database dumps and mirrors the MinIO product bucket while the writers are stopped, then restarts and health-checks exactly the recorded set. A service that was stopped beforehand stays stopped. If the command or host dies after quiescing, the next backup replays that exact restart recipe before it attempts a new recovery point.

By default, the CLI retains the latest three recovery points. Set the ring for the selected output root with --retain:

sudo ingotron backup --out /srv/offhost-staging/ingotron --retain 7

The output root is a trust boundary. It must resolve to a root-owned real directory that is not group- or world-writable. Retention never crosses that root. Your off-host policy must not depend on the local ring.

Use a root-owned executable post-hook to encrypt, copy, or upload a completed recovery point. The hook receives the recovery-point directory as its first argument and in INGOTRON_BACKUP_DIR. It must write a separate destination; the CLI validates the authenticated local source again after the hook returns:

sudo ingotron backup --post-hook /usr/local/sbin/ingotron-backup-offhost

For example, an operator-owned hook can stream a tar archive into age using an off-host recipient, then upload only the encrypted result. Protect the hook as root-owned and not group- or world-writable. Never place an encryption private key in the recovery point or the hook command line.

Scheduling is opt-in. The install command writes and enables the repository's systemd service and timer; status proves that all files exist and reports the enabled and active state:

sudo ingotron backup schedule install --calendar '*-*-* 02:15:00' \
  --randomized-delay 15m --out /srv/offhost-staging/ingotron \
  --retain 7 --post-hook /usr/local/sbin/ingotron-backup-offhost
sudo ingotron backup schedule status

Install creates or resolves the selected output root and refuses /home, /tmp, /var/tmp, protected operating-system trees, broad roots, symbolic links, or any group/world-writable ancestor. /tmp and /var/tmp are specifically unusable because the service has a private temporary-filesystem namespace. It proves the root is writable and establishes the separate backup-authentication key before enabling anything. That key must resolve outside the selected output root so backup media cannot forge authenticated manifests. The service makes the filesystem read-only by default, with explicit write access only to that resolved output root, appliance state, and authentication-key file. Scheduled hooks must place any local encrypted/copy artifact beneath the selected output root, or stream it directly off-host; other local write paths remain unavailable by design.

The timer is persistent, so a missed event runs after boot. Every attempt updates /opt/ingotron/state/backup-status.json. It also writes these textfile metrics to /opt/ingotron/state/metrics/ingotron-backup.prom:

• ingotron_backup_last_success_timestamp_seconds;
• ingotron_backup_last_failure_timestamp_seconds; and
• ingotron_backup_last_run_success.

backup schedule status reports the latest attempt and both last-success and last-failure epochs. It returns success only when the timer is enabled and active and the latest recorded attempt succeeded; an active timer before its first recovery point, or after a failed attempt, remains unhealthy.

The appliance does not currently scrape this textfile into its bundled Prometheus. Connect an operator-owned textfile collector before relying on an alert derived from it.

Set the schedule from the required recovery-point objective (RPO). The maximum routine RPO is roughly the calendar interval plus randomized delay and the time until the encrypted off-host copy succeeds. Set the recovery-time objective (RTO) only from a timed disposable restore with representative database and object volume; object transfer speed is usually the dominant variable.

Optional cold infrastructure archive

The CLI recovery point contains the databases, product bucket, appliance identity, and configuration needed for product recovery. A cold archive of every volume can additionally preserve caches, work directories, and the exact storage-engine state. Archive the volumes named in the authoritative volume_map only while all appliance containers are stopped. The exact snapshot mechanism depends on the Docker volume driver. Prefer a storage-native snapshot when available.

For Docker's local volume driver on Linux:

1. Run sudo ingotron backup first and copy that backup off-host.
2. Read /opt/ingotron/state/install.json and record all seven exact values in volume_map. On a fresh install they normally begin with ingotron_; on an adopted install they usually retain the legacy project prefix.
3. Create a root-only archive directory on a filesystem with enough free space.
4. Stop the containers, archive the volume mountpoints, and start exactly the containers that were running.

The following is a template, not a discovery script. Replace every value from install.json before running it:

sudo bash
set -euo pipefail

archive_dir=/srv/offhost-staging/ingotron-cold-20260819T120000Z
mkdir -m 0700 "$archive_dir"

volumes=(
  ingotron_pg_data
  ingotron_minio_data
  ingotron_uploads
  ingotron_processor_work
  ingotron_exporter_work
  ingotron_query_work
  ingotron_demo_data
)

mapfile -t running < <(
  docker ps -q --filter label=com.docker.compose.project=ingotron
)
((${#running[@]} > 0)) || { echo 'no running Ingotron containers' >&2; exit 1; }

restart_running() { docker start "${running[@]}" >/dev/null || true; }
trap restart_running EXIT
docker stop "${running[@]}" >/dev/null

for volume in "${volumes[@]}"; do
  driver=$(docker volume inspect --format '{{.Driver}}' "$volume")
  [ "$driver" = local ] || {
    echo "$volume uses $driver; use that driver's snapshot procedure" >&2
    exit 1
  }
  mountpoint=$(docker volume inspect --format '{{.Mountpoint}}' "$volume")
  tar --acls --xattrs --numeric-owner -C "$mountpoint" \
    -cpf "$archive_dir/$volume.tar" .
done

restart_running
trap - EXIT
exit
sudo ingotron status

Do not archive a live PostgreSQL or MinIO volume with a filesystem copy. Keep the matching CLI backup, volume archives, exact volume map, version, and checksums together. Practice this procedure and its inverse on a disposable host before relying on it; the 0.9.7 CLI does not automate cold-volume restore.

An S3-compatible, versioned off-host replication of the robotics-data bucket can replace the MinIO volume archive if it preserves every object and its integrity metadata. Validate restores, not just successful copy jobs.

Restore

List local backup IDs:

sudo ls -1 /opt/ingotron/state/backups

Restore a 0.9.7 CLI backup by ID:

sudo install -m 0600 -o root -g root \
  /secure/off-host/ingotron-backup-auth.key /opt/ingotron/backup-auth.key
sudo ingotron restore --backup 20260819T120000Z --yes
sudo ingotron status

restore first authenticates backup.json with the separate key and checks the exact inventory. It also validates the recorded release, install.json, required recovery secrets and TLS files, optional adopted-volume override, and the Compose model rendered from the backup's own configuration. These checks finish before the stack is stopped.

The command then copies env, metadata, install.json, the optional override, secrets, and TLS into private same-filesystem staging. After shutdown, it swaps that state into place as a durable transaction and re-renders keycloak.conf. If the state commit fails, the CLI rolls back the prior state; if that rollback cannot finish, it preserves the private transaction for repair. The PostgreSQL volume remains untouched in either case. Only a completed state transaction permits the database reset. Restore then recreates the recorded PostgreSQL volume, restores both databases, reapplies scoped roles and grants, and verifies isolation and usability. It starts MinIO alone, replaces the configured product bucket with the authenticated minio-objects.tar, repairs the Keycloak bootstrap credential, and starts the backup's matching release. A missing key, a wrong authentication tag, an invalid object archive, an unavailable release, or configuration that does not render is refused before shutdown.

If a backup was created with --out, copy its complete timestamped directory back under /opt/ingotron/state/backups/ with root-only ownership before using restore; the command accepts an ID, not an arbitrary path.

Understand the boundary:

• The database and product bucket are both replaced by the authenticated recovery point.
• Database rows and object writes created after that backup are removed.
• Dataset cuts created after the backup disappear from the restored registry and from MinIO.
• Pre-0.9.1 single-database backups are refused because they cannot restore a coherent claimed appliance.

Restoring a complete cold archive is a disaster-recovery procedure: restore the seven volumes under the exact names in volume_map, restore the matching /opt/ingotron/state (or CLI backup), reattach the release, then verify status, sign-in, an existing recording, dataset signature, and a test robot upload.

Disposable restore verification

Never test a restore against the production appliance. Provision a disposable host with separate DNS, networks, and fresh Docker volumes; install the exact release recorded by the backup; copy the full timestamped directory under /opt/ingotron/state/backups/; and restore the separately escrowed backup-auth.key. Keep robots and production clients unable to reach this host. Then time:

sudo ingotron restore --backup 20260819T120000Z --yes
sudo ingotron status

Verify sign-in, the expected database counts, one original object digest, one processed recording, one dataset signature, and a disposable robot upload. Record total restore time, backup age at the test start, database and object bytes, host profile, and the recovery-point ID. Those measurements establish the tested RTO and RPO. Destroy the disposable host after preserving redacted evidence. A successful backup job without this restore drill is copy evidence, not recovery evidence.

Upgrade

The normal command follows the signed latest channel:

sudo ingotron upgrade --show-config-changes

To request a specific signed version:

sudo ingotron upgrade --version 0.9.7 --show-config-changes

The CLI verifies the channel, manifest, bundle, and digests. It checks version and platform compatibility, disk space, configuration changes, and supported migrations. Before switching releases, it downloads the files and images, stops writes, and creates a backup. The release switch is atomic. It either completes or leaves the previous release active. The CLI then applies the product and Keycloak migrations and waits for both HTTPS/OIDC and ingest TLS health checks to pass.

Plan a maintenance window even though downloads happen first. Do not use --no-backup unless you accept that post-migration rollback and Keycloak recovery may be impossible. To roll back a Keycloak image change, you must restore a backup, even when the Robot Vault schema does not change.

If the command or host is interrupted, do not improvise with Compose:

sudo ingotron status
sudo ingotron upgrade --resume

The fsynced journal tells the CLI whether it is safe to continue, roll back the release pointer, or require a restore. Resume is phase-correct and never takes a second backup: MIGRATION_DONE continues at the Keycloak boundary; KEYCLOAK_MIGRATION_DONE, CUTOVER, and HEALTH continue at cutover; and COMMIT_STARTED completes only commit metadata. A ROLLBACK_STARTED record continues the exact recorded rollback. Operator rollback recovery uses sudo ingotron rollback --yes, not upgrade --resume.

The upgrade keeps an edge-bound snapshot of env metadata, secrets, TLS, and Keycloak configuration after a successful no-door upgrade. Plain rollback stops the fixed Compose project, restores that exact snapshot, activates the recorded previous release, and removes the snapshot only after a healthy, durably journaled ROLLED_BACK.

The upgrade health timeout defaults to 300 seconds. Override it for a slow host:

sudo ingotron upgrade --health-timeout 600

Allowed range is 60 to 3600 seconds. The installed environment's INGOTRON_HEALTH_TIMEOUT provides the persistent default.

Roll back

If the CLI reports that both the product schema and Keycloak image are reversible:

sudo ingotron rollback --yes

If either change is not reversible, the CLI refuses a plain rollback. Use the backup ID printed by the failed upgrade:

sudo ingotron rollback --restore-backup 20260819T120000Z --yes

This is intentionally destructive to database changes after that backup. It also replaces the product bucket with the same recovery point; review the restore boundary above.

Adopt a 0.9.0 checkout installation

Adoption is for the legacy 0.9.0 checkout-based evaluation stack on the same host. Back it up first and make sure it is in the detected, healthy shape. Run:

curl -fsSL https://get.ingotron.com/install | sudo INGOTRON_ADOPT=1 sh

If a 0.9.7 CLI is already staged, the explicit equivalent is:

sudo ingotron adopt --from /absolute/path/to/legacy/checkout

Adoption records the old checkout and keeps its existing volume names. It imports the environment and signing identity, stores the volume mapping, and moves appliance control to /opt/ingotron. It does not rename or copy the volumes. Keep the old checkout until you have checked the appliance, sign-in, recordings, dataset verification, and a robot upload.

Legacy 0.9.0 used one fleet-wide ingest token. Adoption provides a 30-day grace period so existing uploaders do not stop immediately. Add/re-enroll every robot through the UI to move it to a per-robot credential before the deadline. Do not extend the shared-token path as a permanent topology.

Resolve legacy upload integrity after upgrade or adoption

The upgrade and adoption preflights report completed, unprocessed uploads that lack source-side SHA-256 proof. They are not silently processed: migration 025 keeps every unresolved row excluded until an owner records one disposition. List the unresolved queue after the new release is healthy:

sudo ingotron integrity list
sudo ingotron integrity list --json

list shows only unresolved rows. Use --all to include recorded dispositions and their actor, policy, reason, replacement link, and timestamp. Choose exactly one path for each upload:

1. Re-upload and supersede (preferred). Re-upload the original recording through the current robot agent with its client-computed SHA-256. Wait for that new upload to complete, copy its upload UUID from Intake details, then link the excluded legacy row to it:

   sudo ingotron integrity supersede \
     --upload-id LEGACY_UUID \
     --superseding-upload-id VERIFIED_REUPLOAD_UUID \
     --actor 'operator@example.com' \
     --policy 'legacy-integrity/v1' \
     --reason 'robot re-uploaded the retained source with a client digest' \
     --yes

   The replacement must be a different completed upload for the same robot and its basis must be client_sha256. The original remains excluded.

2. Quarantine/exclude. Retain the source record without allowing it into processing:

   sudo ingotron integrity quarantine \
     --upload-id LEGACY_UUID \
     --actor 'operator@example.com' \
     --policy 'legacy-integrity/v1' \
     --reason 'source-side digest evidence is unavailable; retain for audit only' \
     --yes

3. Owner-approved attestation. Use this only when a documented owner policy explicitly accepts the legacy source despite the missing client digest:

   sudo ingotron integrity attest \
     --upload-id LEGACY_UUID \
     --actor 'data-owner@example.com' \
     --policy 'approved-legacy-import/2026-08' \
     --reason 'accepted under the cited controlled-import review' \
     --yes

   This changes processing eligibility with basis owner_attestation; it does not claim that the robot supplied a digest. Equality between expected_sha256 and sha256_hash is not evidence: migration 018 populated the former from the latter for legacy rows.

All three commands require an actor, a policy, and a reason. None can be empty. Each command appends one database record. Running the exact same command again is safe, but the CLI refuses a different decision for that upload. Do not put secrets or unnecessary personal data in the audit fields. They are stored in the database and backups. The CLI does not copy their plaintext into its operation log or nested Docker/psql command arguments.

Prometheus reports current state through:

• robot_vault_legacy_upload_integrity_backlog: unresolved completed, unprocessed, unverified rows only;
• robot_vault_legacy_upload_integrity_dispositions{disposition=...}: recorded superseded, quarantined, and owner_attested decisions; and
• robot_vault_legacy_upload_integrity_metrics_refresh_success: 1 only when the latest bounded Postgres refresh succeeded. A NaN backlog with refresh success 0 is unknown state, not an empty queue.

Release alert rules also cover query saturation, capacity rejections and timeouts, export dead letters, the unresolved integrity backlog, and a failed integrity-metric refresh. The repository's Alertmanager routing template splits critical and warning alerts but deliberately contains no destination. An operator must add and test a real delivery integration before treating an active Prometheus rule as a human notification. Validate every routing or rule edit with make test-observability-config; it runs pinned promtool and amtool checks plus the repository's shipped-metric contract.

There is no release-scraped metric yet for queue age, disk or object capacity, certificate expiry, or the backup textfile. Monitor those through the systems that own them. The release rules intentionally omit placeholder metrics that would create false coverage.

Reattach after a retained uninstall

A non-purge uninstall keeps state and all volumes. Reinstall the CLI and attach the signed release to that exact state with:

curl -fsSL https://get.ingotron.com/install | sudo INGOTRON_REATTACH=1 sh

Do not use the normal fresh-install command when retained state exists. After reattachment, check status, sign-in, TLS fingerprints, an existing dataset, and a robot upload.

Change the appliance address

set-address is the only supported way to change the public URL. It updates APPLIANCE_URL, every bundled-OIDC endpoint, listener certificates/SANs, and the running services as one recoverable operation:

sudo ingotron set-address https://vault.example.com:8003
sudo ingotron status

An IP literal or host:port is accepted and normalized to HTTPS. HTTP, URL paths, credentials, query strings, and fragments are refused.

Normally the CLI requires the new address to be present/reachable. For a planned DNS or interface move where it cannot exist yet:

sudo ingotron set-address --allow-absent https://vault-new.example.com:8003

Use --force only to recover an already unhealthy stack after collecting a backup and logs; it bypasses a safety precondition, not validation of the new URL. If interrupted, re-run the same command.

Changing the address reissues listener certificates under the same CA. Update DNS/firewall rules and robot configuration as planned. A stable CA does not require fleet-wide trust replacement, but robots must still reach the new name/address.

TLS operations

Inspect the CA, listener fingerprints, expiry, SANs, and drift:

sudo ingotron tls status
sudo ingotron tls fingerprint --ca
sudo ingotron tls fingerprint --query
sudo ingotron tls fingerprint --ingest
sudo ingotron tls ca --out ingotron-ca.crt

Routine maintenance keeps the appliance CA and reissues only missing, expiring, or drifted leaves:

sudo ingotron tls ensure
sudo ingotron tls regen --restart

Additional SANs can be requested during a deliberate leaf regeneration:

sudo ingotron tls regen --san vault.example.com --san 10.20.30.40 --restart

Changing the CA requires every robot and browser to trust a new certificate. Use it only for disaster recovery, not routine renewal:

sudo ingotron backup
sudo ingotron tls rotate-ca --yes

The old pin is no longer valid. Open Robots in the UI and create a new Re-enroll command for every robot. Redistribute the new public CA to browsers and managed clients after verifying its fingerprint out of band.

Administrator and identity recovery

Before claim, generate a replacement setup code. After claim, repair or recreate the appliance administrator without reopening setup:

sudo ingotron recover-admin

Rotate the separate Keycloak master bootstrap operator credential with:

sudo ingotron auth reset-bootstrap-credential

The rotation stops every Keycloak node and creates a temporary ingotron-recovery-<16 random hex characters> operator. Its throwaway password is generated independently from the new durable password and exists only in a root-owned transient file. After Keycloak starts, the temporary operator recreates or rotates ingotron-bootstrap; the CLI proves the durable login before deleting every ingotron-recovery-* account, including leftovers from an earlier failed attempt. A failure before that proof retains the current temporary account to avoid lockout; a successful retry removes it.

A direct reset owns its BOOTSTRAP_CREDENTIAL_RESET_STARTED and BOOTSTRAP_CREDENTIAL_RESET journal records. Restore runs the same rotation without those nested records, so RESTORE_STARTED remains the active recovery phase until the CLI writes RESTORED.

These commands are host-local and root-only. They are not a substitute for backing up both databases and the state directory.

Robot fleet operations

Follow Set up each robot computer for initial enrollment and the first upload.

• Use Robots → Add robot for every distinct identity.
• Use Re-enroll after reimage, credential loss, suspected compromise, or CA rotation. A fresh one-use code is required.
• Disable a robot to stop its credential while retaining history.
• Decommission according to your retention policy; it does not imply secure deletion of every historical object.
• Appliance ingest is TLS plus per-robot bearer authentication. It is not mTLS. Do not distribute the legacy fleet-wide INGEST_AUTH_TOKEN to new robots.

After address, firewall, or certificate work, test both surfaces: HTTPS on TCP 8003 and robot gRPC/TLS on TCP 50051.

Uninstall

Retain data and identity

sudo ingotron uninstall --yes

This stops/removes containers, release payloads, and the CLI, but keeps all named volumes and /opt/ingotron/state, including claim state, credentials, dataset-signing key, appliance CA, adopted-volume override, and backups. It also keeps /opt/ingotron/backup-auth.key, which is required to authenticate those backups. It can be reversed with the reattach command above.

Permanently purge

sudo ingotron uninstall --purge --yes

This irreversibly removes every volume and /opt/ingotron, including local backups. It destroys the dataset-signing identity and appliance CA. Previously distributed datasets remain verifiable only if recipients retained the matching public key; every robot must enroll again into any replacement appliance.

Add --images only if the pinned container images should also be removed:

sudo ingotron uninstall --purge --yes --images

Take and verify an off-host CLI recovery point or complete cold archive first. A local backup under /opt/ingotron/state/backups does not survive purge, and neither does /opt/ingotron/backup-auth.key. Confirm the separately protected copy of that key is readable before destroying the appliance.

Troubleshooting map

For unresolved failures, preserve sudo ingotron status --json, relevant CLI and service logs, installed version, timestamps, and a description of the last successful operation. Redact passwords, tokens, claim codes, private keys, and backup contents before sharing diagnostic material.

See also configuration ownership, security, and the user guide.

Partial upload recovery in 0.9.7

Version 0.9.7 requires migration 026 and the matching ingest image, robot agent, toolkit, and ingest object-store policy. Use the signed appliance upgrade so migrations and scoped permissions are applied with the new services. Keep source recordings until upload completion.

Partial uploads survive connection loss and process restarts. Sessions expire after seven days without activity. The server owns expired-session cleanup; generic object-store cleanup protects registered multipart sessions. Full-file SHA-256 verification still gates completion. The schema change requires backup restore for rollback to a release that predates migration 026.