Ingotron 0.9.7 security guide

Robot Vault 0.9.7 runs on one self-hosted server and is intended for trusted robot networks. This guide explains where its protections end, which controls operators must provide, and what this release does not support. The appliance does not make an untrusted network safe.

Security model at a glance

The appliance exposes two host ports:

PostgreSQL, MinIO, Redis, internal Keycloak listeners, metrics, and maintenance endpoints are not published by the appliance overlay. Service-to-service traffic inside the private Docker network is generally plaintext and relies on host/network isolation plus scoped database and object-store credentials.

Protect the Docker host as the primary trust boundary. Root, Docker-daemon access, or read access to /opt/ingotron/state is effectively appliance administrator access and can expose customer data, credentials, signing keys, and the private CA.

Configuration manager trust boundary

The Admin UI exposes four settings, but only the health-check timeout and time-index cache size are reversible operator changes. The raw-media export policy and garbage-collection dry-run policy are displayed for awareness and remain controlled by the signed release or the root operator. Every UI change is validated, reviewed, tied to a current revision, and applied as a bounded transaction with health-gated rollback.

The query service reaches the root-owned configuration manager through a private Unix socket mounted at /run/ingotron. The query container receives neither the Docker socket nor writable appliance state. The manager therefore has a narrow command surface, but the query service is still part of the appliance trusted computing base: code execution in query can ask the manager to apply the two allowed settings. The host manager does not independently re-check the browser user's OIDC role. Protect the signed release, the query service, TCP 8003, and the Docker host accordingly; the UI's operator role is an application control, not a separate host-level authorization boundary.

The appliance reserves numeric GID 10001 for its manager socket and mounted service files. Installation creates the system group ingotron-secrets at that GID when it is absent. Installation and manager activation refuse a conflicting group name or GID, explicit group members, or host accounts whose primary GID is 10001. Existing accounts and groups are not reassigned. Resolve the reported collision before retrying; do not add human or unrelated service accounts to this group, because membership grants access to the manager's host-control API. The numeric container identity remains 10001:10001.

If an apply request disconnects while query restarts, use status and the query logs, then reload the Admin panel. Do not infer success from a dropped HTTP connection or immediately submit the same plan again.

Install and update trust

The one-line bootstrap is small enough to fetch the signed release metadata but does not trust transport alone. It verifies the signed update channel, release manifest, bundle hash/size, and artifact digests against an embedded Ed25519 release public key. The installed CLI repeats that process on upgrade.

Use only:

curl -fsSL https://get.ingotron.com/install | sudo sh

Inspect the fetched bootstrap first if your policy forbids piping network content to a privileged shell. Do not bypass signature/digest refusals, edit an installed release, substitute unpinned images, or use REGISTRY and ROBOT_VAULT_VERSION overrides. A verified release is immutable under /opt/ingotron/releases; mutable state is separate.

Version 0.9.7 currently supports Linux x86-64 only. Keep the host kernel, Docker Engine, and base OS patched under your own maintenance policy.

RHEL-family and Fedora hosts with SELinux Enforcing are refused by the installer and upgrade path until Ingotron ships a packaged SELinux policy for the host-managed configuration socket. This is fail-closed by design. Never disable SELinux to make an install proceed.

Appliance claim and administrators

Installation creates a cryptographically random, one-time claim code before setup becomes available. The installer prints it to the root terminal and stores it only in protected state. The code expires. The appliance accepts it once and invalidates it in the same operation that creates the first administrator. It is not a URL parameter. Do not log it or send it in a ticket.

Prefer claiming from the server terminal when possible:

sudo ingotron claim --password-file /root/ingotron-admin-password

Delete the password file after success. If a code may have leaked, replace it with sudo ingotron claim-token. sudo ingotron recover-admin is a root-only recovery path; after claim it does not reopen public setup. The separate Keycloak master bootstrap credential can be rotated with sudo ingotron auth reset-bootstrap-credential.

Use individual administrator/operator identities where the UI permits it. Grant the least role required, remove departed users promptly, and protect access to server recovery commands with the same controls as root.

Internal admin portal (separate service)

The Cloudflare admin-portal/ Worker is separate from the appliance UI and is dark by default. If it is given a route, keep Cloudflare Access in front of it and retain the Worker's own gate as a second layer. Its login limiter allows eight attempts per location per minute and its API limiter allows 240 requests per minute; a missing or failing limiter fails closed. These are abuse brakes, not a replacement for Access, a firewall, or a strong secret.

Set ADMIN_PASSWORD only as a Worker secret. It must be 32–256 visible ASCII characters; an unset or shorter value leaves the portal unsatisfiable rather than weakening the gate. Rotate it during onboarding, staff changes, or any suspected disclosure by generating a new random value and running npx wrangler secret put ADMIN_PASSWORD from admin-portal/. Changing the secret invalidates existing HMAC-signed sessions, so verify a fresh login and remove the old copy from local files and operator transcripts.

TLS and browser trust

Each appliance creates an ECDSA P-256 private CA and separate query and ingest leaf certificates. The CA is valid for ten years; leaf certificates are issued for 397 days and the CLI can reissue them before expiry or when SANs drift. Private keys are root-owned and mounted read-only into the services that need them.

Because the CA is private, a browser does not trust a new appliance by default. Compare the certificate fingerprint printed by the installer with the browser before accepting a warning, or import the public CA through a managed trust store:

sudo ingotron tls fingerprint --ca
sudo ingotron tls fingerprint --query
sudo ingotron tls ca --out ingotron-ca.crt

Move the public address only with sudo ingotron set-address; it keeps the canonical URL, bundled OIDC endpoints, redirect behavior, and certificate SANs together. Never copy files from the private TLS directory to clients.

HSTS defaults off because pinning a browser to HTTPS before its operator has established trust in a private CA can make recovery worse. Enforce HSTS at a managed upstream boundary only when every client trust store and recovery path is controlled.

Routine leaf regeneration keeps the CA unchanged. tls rotate-ca --yes replaces the CA trusted by the fleet. Every browser and client needs the new public CA, and every robot must be re-enrolled. Take a backup and treat the change as a security incident or disaster-recovery event.

Robot identity and enrollment

Robots connect with server-authenticated TLS, and each robot gets its own bearer credential. The appliance does not use mutual TLS. This differs from the separate bring-your-own-OIDC/mTLS production setup in older documentation.

An operator first creates the robot in the UI, then generates a one-time ign1... enrollment code. It expires after 15 minutes by default. The code links the intended robot to the appliance CA fingerprint. The robot installer:

1. downloads the public CA certificate, subject to a response-size limit;
2. checks the CA fingerprint from the code before sending the code;
3. exchanges the one-use code over the pinned TLS channel; and
4. stores the returned per-robot credential in a root-protected location.

Keep enrollment commands out of shell transcripts, tickets, chat, and shared automation logs. If an unexpired code leaks, revoke/replace it. If a robot token or host is compromised, disable the robot and issue a Re-enroll command only after the host is trusted again. A token from one robot must never be copied to another.

Fresh 0.9.7 appliances deny the old fleet-wide shared ingest token. Adoption of 0.9.0 provides a recorded 30-day grace period solely to migrate existing robots. Complete per-robot enrollment before it expires.

Secrets and key material

Critical state includes:

• database and object-store credentials;
• robot provisioning/internal credentials;
• Keycloak database and bootstrap credentials;
• the dataset-signing private key;
• the appliance CA private key and TLS leaves; and
• claim, install, and configuration ownership state.

It lives under /opt/ingotron/state, normally root-only, with narrowly group-readable mounted files for non-root containers. Keycloak passwords use mounted secret files rather than container environment values. Do not weaken permissions, put secrets in command-line arguments, commit state to version control, or expose Docker's API remotely.

The dataset-signing private key establishes the provenance identity of every cut. Losing it prevents future signatures under that identity; stealing it permits forged manifests that appear to come from the appliance. The CA private key establishes browser and robot server trust. Back up both encrypted and off-host, with access logging and separation from ordinary application users.

Data protection and privacy

Robot recordings can contain faces, licence plates, voices, screens, location, facility layout, proprietary behavior, and other sensitive information. Version 0.9.7 does not include the GPU PII-scrubbing sidecar. The appliance marks derived media as raw because it has not been scrubbed.

The 0.9.7 baseline blocks raw, unscrubbed exports with ALLOW_UNSCRUBBED=false. If a root-controlled policy enables raw exports, the export records which raw media it contains. This setting can expose sensitive data. Deployments that must block raw media should set it to false and test that the export is refused. Changing it does not remove data already ingested. Whole-source MCAP passthrough has additional exact file-digest and content checks that this setting cannot bypass.

The appliance does not provide application-level encryption at rest. Database, MinIO, Docker volume, cold archive, and backup confidentiality depend on the host/storage layer. Use encrypted disks and encrypted off-host backups, protect swap and crash dumps, restrict physical access, and define retention/deletion procedures appropriate to the data.

Garbage collection starts in dry-run mode. Review it against a tested backup before enabling deletion. Decommissioning a robot or excluding an episode does not by itself prove secure erasure from every database row, object, backup, or downstream dataset.

Backups and recovery

sudo ingotron backup contains both database dumps, the complete configured MinIO product bucket, credentials, the dataset signing key, and the CA private key. It is therefore as sensitive as the live appliance. Keep its separate manifest-authentication key outside the backup media, and encrypt every off-host copy as described in Operations.

Test restores on an isolated host. Preserve the matching release/version, configuration, volume map, public signing key, and checksums. Do not test using production DNS or a network where restored robots could connect to two copies of the same identity.

Legacy upload-integrity boundary

Uploads accepted before source-digest enforcement can be completed while integrity_verified=false. They remain in object storage but are excluded from processing. Migration 018 copied some server-recorded final hashes into expected_sha256; equality between those two server-held values is not proof that the robot declared the digest and 0.9.7 never auto-promotes it.

Upgrade/adoption preflight and the robot_vault_legacy_upload_integrity_backlog runtime gauge expose unresolved rows. The root operator must use sudo ingotron integrity to record one append-only disposition: a separate client-SHA-verified re-upload, quarantine, or an owner-approved attestation. Attestation satisfies the processing gate under owner_attestation; it is a policy decision, not cryptographic source verification. Define who may approve it and what evidence/reason your policy requires before using that option.

The database records actor, policy, reason, decision, replacement (when any), and timestamp. Repeating the exact command is safe. The appliance refuses a conflicting decision for that upload. Database backups include these audit fields. Do not enter passwords, tokens, health information, or unnecessary personal data. The CLI does not write their plaintext to ingotron.log or nested Docker/psql argv. See Operations for the reviewed commands.

Network and host hardening checklist

• Place TCP 8003 behind a firewall/VPN or trusted management network; expose it only to intended users.
• Allow TCP 50051 only from robot networks that need ingest. Enrollment also uses the HTTPS appliance surface. INGEST_BIND_HOST may narrow the listener to a particular host interface, but it is not an access-control boundary; enforce the robot CIDR allow-list in the host firewall as well.
• Do not publish internal Compose ports or Docker's socket/API.
• Use stable DNS and time synchronization on the host and robots.
• Restrict SSH/root and Docker-group membership; audit privileged sessions.
• Enable encrypted storage and protected, tested off-host backups.
• Import and verify the appliance CA through managed channels instead of teaching users to ignore certificate warnings.
• Review sudo ingotron status, service logs, administrator changes, robot enrollment/disable events, export decisions, and upgrade history.
• Clear every legacy upload-integrity backlog item by recording a disposition. Alert on a non-zero backlog or a failed metric refresh.
• Keep GC_DRY_RUN=true until deletion evidence is reviewed.
• Finish legacy shared-token migration before the adoption grace expires.

Current limitations

Treat these as design constraints for 0.9.7:

• single Linux x86-64 host; no high-availability control plane;
• private appliance CA; no bundled public-CA/ACME automation;
• HSTS disabled by default;
• trusted private Docker network, with plaintext service-to-service traffic on that network;
• TLS plus per-robot bearer authentication, not appliance mTLS;
• no application-level encryption at rest or built-in key-management service;
• no bundled PII-scrubbing backend;
• routine CLI backup includes the configured product bucket, but backup media still requires encryption and off-host protection;
• no arbitrary appliance configuration or restart command; only the narrow, health-gated settings boundary described above; and
• no claim that Internet exposure without an external firewall/VPN/WAF is a supported security posture.

Reporting a vulnerability

Report suspected vulnerabilities privately to hello@ingotron.com with the affected version, impact, and reproduction steps. Do not include customer data, passwords, tokens, claim/enrollment codes, private keys, or backup archives. Avoid public disclosure until a safe remediation path has been coordinated.

For operational incidents, first preserve timestamps, ingotron status --json, the installed version, and relevant redacted logs. See the operations troubleshooting map.