Deployment Runbook
Daily health checks, restart procedures, maintenance schedule, troubleshooting quick reference, and escalation contacts for IT administrators.
This runbook is for IT administrators and shift supervisors responsible for the Capped AI appliance day-to-day. Keep it accessible on-site. For operator-facing guidance, see the Operator Training Guide.
Daily health check
Run these checks at the start of each day shift before operators begin handovers.
| Check | How to verify | Expected result |
|---|---|---|
| Appliance reachable | Ping appliance IP from plant network | Response <1ms |
| Capped AI web UI loads | Open https://<appliance-ip> in browser | Login screen renders |
| OPC-UA historian feed | Admin dashboard → Integrations → Status | Green — last data <60s ago |
| GPU inference service | Admin dashboard → System → Inference | vLLM running, GPU utilization shown |
| Disk space | Admin dashboard → System → Storage | >20% free on data volume |
| WORM log integrity | Admin dashboard → Audit → Log integrity | Last check: Passed |
Starting a handover session (operator-initiated)
This is handled by operators on their tablets. See the Operator Training Guide. IT staff do not need to intervene unless the system is unavailable.
Appliance restart procedure
Only restart the appliance if directed by Capped AI support or during a scheduled maintenance window. In-progress handovers will be lost if not signed before restart.
Scheduled maintenance windows
| Frequency | Task | Duration | Responsible |
|---|---|---|---|
| Weekly (Sat 02:00–04:00) | Software update check (auto-applied if approved) | Up to 30 min | Capped AI Ops Cloud (auto) |
| Monthly | Disk usage audit — archive old handovers if >80% full | 1 hour | IT admin |
| Monthly | WORM log export to encrypted USB backup | 30 min | IT admin |
| Quarterly | Hardware visual inspection (cables, LEDs, fan noise) | 30 min | IT admin |
| Annually | Full appliance health report sent to Capped AI support | 1 hour | IT admin + Capped AI |
Software update procedure
Capped AI Ops Cloud notifies your IT admin contact via email when an update is available. Updates include release notes and a SHA-256 hash for verification.
- Log in to Admin dashboard → System → Updates
- Review the release notes and confirm the update window with the shift supervisor
- Click Approve and schedule — the appliance will apply the update during the next configured maintenance window (default: 02:00 local time)
- After the update, verify all services are green and run the daily health check
- To roll back: Admin dashboard → System → Updates → Rollback (available for 48 hours after each update)
Troubleshooting quick reference
Handover generation fails or times out
- Check GPU inference: Admin dashboard → System → Inference. If vLLM is stopped, click Restart Inference Service.
- Check OPC-UA historian feed. If disconnected, see OPC-UA reconnect procedure below.
- If GPU temperature exceeds 85°C, check physical ventilation around the 1U appliance. Ensure 1U of clearance above and below in the rack.
- If neither resolves the issue, contact Capped AI support with the request ID shown in the error message.
OPC-UA historian feed disconnected
- Admin dashboard → Integrations → Historian → Reconnect. This retries the OPC-UA session without a full restart.
- If reconnect fails, verify firewall rule: TCP 4840 from Capped AI appliance IP to DCS historian IP must be open.
- Verify DCS historian service is running (contact your DCS engineer).
- If historian credentials have changed (certificate rotation), update them in Admin dashboard → Integrations → Edit credentials.
Operator cannot log in to Capped AI
- Verify the operator tablet can reach the appliance IP (ping test).
- Check the HTTPS certificate: browsers will show a certificate warning if the self-signed cert has expired. Renew via Admin dashboard → System → TLS Certificate → Renew.
- Reset the operator PIN via Admin dashboard → Users → select operator → Reset PIN.
Disk space below 20%
- Admin dashboard → System → Storage → Archive handovers. Select handovers older than 90 days to export to encrypted USB and remove from primary storage.
- The WORM log is append-only and cannot be trimmed. Archiving moves handovers to a signed export file; the WORM log entry remains immutable.
- Contact Capped AI support if you need a storage expansion kit (additional NVMe drive).
Appliance does not respond (full outage)
- Check physical power LEDs on the front panel. Both PSUs should show green.
- Access the IPMI management interface from the IT management network to view hardware status without logging in to the OS.
- If hardware fault is confirmed (GPU, disk, or PSU failure), call Capped AI hardware support immediately. Do not attempt to swap components without Capped AI guidance.
- During appliance downtime, operators revert to manual paper-based handovers. Handover data is safe on the RAID-1 mirror.
Escalation contacts
| Severity | Criteria | Contact |
|---|---|---|
| P1 — Production down | Appliance fully unavailable during active shift | Capped AI on-call: +1 (305) 000-0000 (24/7) |
| P2 — Degraded | Handovers generating but slow (>60s), or OPC-UA data stale | hello@capped.ai — response <2h |
| P3 — Minor | UI glitch, single operator login issue, non-urgent questions | hello@capped.ai — response next business day |
| Hardware fault | GPU/disk/PSU failure indicated by IPMI or front-panel LED | Capped AI hardware support: hardware@capped.ai |
Backup and recovery
The appliance maintains a RAID-1 mirror of the data volume. In the event of a single disk failure, the system continues operating on the surviving disk. Capped AI ships a replacement NVMe drive within 5 business days (Standard) or 48 hours (Enterprise).
For off-site backups, export the encrypted handover archive monthly via Admin dashboard → System → Export. The export is AES-256 encrypted and signed with the plant manager's key. Store on an encrypted USB in your plant safe.