# AI Biz Manual Provisioning and Recovery Runbook **Service:** Own the Stack **Domain:** `ownthestack.io` **Validated pilot client:** Dr. Rasheeda Hall **Pilot slug:** `drhall` **Pilot workspace:** `https://drhall.ownthestack.io` **Environment:** Production **Runbook status:** Pilot-validated manual provisioning and optimized recovery process, with post-pilot security, monitoring, backup-freshness, and notification verification completed through September 30, 2026 --- ## 1. Purpose and Scope This runbook documents the **manual AI Biz customer provisioning, backup, key-escrow, and recovery process** validated during the first client implementation for Dr. Rasheeda Hall. The goal is to provide a repeatable operational procedure that can be followed by an administrator today. It is also the manual reference process that future provisioning and recovery automation should reproduce. This runbook covers: - Client Docker Compose environment template - DigitalOcean client droplet creation - Client firewall and tags - Coolify server registration and deployment - Porkbun DNS and HTTPS - n8n owner account and Community Edition activation - Uptime Kuma monitoring - Nightly PostgreSQL and n8n data backups to Backblaze B2 - Backup retention and repository pruning - Restic recovery-credential escrow - `N8N_ENCRYPTION_KEY` escrow using SOPS and age - Fresh-server restore drill - Secure SSH-tunnel access to a restored workspace - Restore validation and cleanup - Post-pilot client security, monitoring, notification, backup-freshness, and OS security-update verification ### 1.1 Out of Scope This document does **not** define the automated provisioning implementation, control-plane state machine, GoHighLevel webhook handling, portal development, or API orchestration. Those items belong in a separate automation implementation document. ### 1.2 Source of Truth The manual procedure in this runbook is aligned with the current AI Biz Build Specification, while preserving the steps that were actually implemented and tested during the Dr. Hall pilot. Where the pilot sequence differed from the target standard, this runbook uses the **standard process for future manual deployments** and records the pilot difference separately in the validation record. --- ## 2. Standard Conventions ### 2.1 Client Variables Use the following placeholders throughout this runbook. | Variable | Meaning | Dr. Hall Example | |---|---|---| | `` | Customer display name | Dr. Rasheeda Hall | | `` | Lowercase client identifier | `drhall` | | `` | Client droplet public IPv4 | Record privately | | `` | Control droplet public IPv4 | Record privately | | `` | Service domain | `ownthestack.io` | | `` | Fleet-pinned n8n version | `2.38.5` during pilot | | `` | Temporary restore droplet IP | Temporary only | > **Placeholder rule:** Values shown inside angle brackets such as `` are placeholders. Replace them with the real value before running any command. Do not copy commands containing unresolved angle-bracket placeholders directly into a shell. ### 2.2 Permanent Shared Components The following are shared AI Biz infrastructure and should be **reused**, not recreated for every client: - DigitalOcean project: `AI Biz` - Coolify control server and project - Coolify provisioning SSH keypair - Standard client firewall: `ai-biz-client-firewall` - Backblaze backup bucket: `kwagency-automation-backups` - Backblaze escrow bucket: `kwagency-ai-biz-key-escrow` - Agency age identity on the Control server - Uptime Kuma instance on the Control server ### 2.3 Security Rules Never place real secrets in: - Git repositories - Runbooks - Screenshots or screen recordings - Shared chat messages - `.env.example` This includes: - `PG_PASSWORD` - `N8N_ENCRYPTION_KEY` - `N8N_LICENSE_ACTIVATION_KEY` - Backblaze key IDs and application keys - Restic repository passwords - age private identities - Private SSH keys - Authentication tokens Use NordPass or the approved secret store for human-managed credentials. --- # Part I — Manual Client Provisioning ## 3. Build the Reusable Client Environment Template ### 3.1 Purpose Use one Docker Compose template for all AI Biz clients. Client-specific values are supplied through environment variables rather than hard-coded into the template. Required variables: ```text SLUG DOMAIN N8N_VERSION PG_PASSWORD N8N_ENCRYPTION_KEY N8N_LICENSE_ACTIVATION_KEY ``` ### 3.2 Docker Compose Template Create `docker-compose.yml`: ```yaml services: n8n: image: docker.n8n.io/n8nio/n8n:${N8N_VERSION} restart: unless-stopped environment: - N8N_HOST=${SLUG}.${DOMAIN} - N8N_PROTOCOL=https - WEBHOOK_URL=https://${SLUG}.${DOMAIN}/ - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY} - N8N_LICENSE_ACTIVATION_KEY=${N8N_LICENSE_ACTIVATION_KEY} - DB_TYPE=postgresdb - DB_POSTGRESDB_HOST=postgres - DB_POSTGRESDB_DATABASE=n8n - DB_POSTGRESDB_USER=n8n - DB_POSTGRESDB_PASSWORD=${PG_PASSWORD} - EXECUTIONS_DATA_PRUNE=true - EXECUTIONS_DATA_MAX_AGE=336 - N8N_DIAGNOSTICS_ENABLED=false - N8N_RUNNERS_ENABLED=true - GENERIC_TIMEZONE=America/New_York volumes: - n8n_data:/home/node/.n8n depends_on: - postgres deploy: resources: limits: memory: 1200M postgres: image: postgres:16-alpine restart: unless-stopped environment: - POSTGRES_DB=n8n - POSTGRES_USER=n8n - POSTGRES_PASSWORD=${PG_PASSWORD} volumes: - pg_data:/var/lib/postgresql/data deploy: resources: limits: memory: 512M volumes: n8n_data: pg_data: ``` > **Important:** Always pin `N8N_VERSION`. Never use `latest` for production client deployments. > **Important:** PostgreSQL is internal only. Do not publish port `5432`. n8n is exposed externally only through Coolify's HTTPS proxy, not by publishing port `5678` on the DigitalOcean firewall. ### 3.3 Environment-Variable Reference Create `.env.example` only as a reference: ```dotenv SLUG=client-slug DOMAIN=ownthestack.io N8N_VERSION=x.x.x PG_PASSWORD=CHANGE_ME N8N_ENCRYPTION_KEY=CHANGE_ME N8N_LICENSE_ACTIVATION_KEY=CHANGE_ME ``` Do not commit real values. ### 3.4 Template Validation Confirm before deployment: - [ ] n8n version is variable-based and pinned - [ ] PostgreSQL uses `postgres:16-alpine` - [ ] n8n uses PostgreSQL rather than SQLite - [ ] PostgreSQL has no public port - [ ] `n8n_data` persistent volume exists - [ ] `pg_data` persistent volume exists - [ ] PostgreSQL password is not hard-coded - [ ] n8n encryption key is not hard-coded - [ ] n8n license key is not hard-coded - [ ] Execution pruning is enabled - [ ] n8n memory limit is configured - [ ] No production credentials are stored in the template --- ## 4. Create the Client Droplet in DigitalOcean ### 4.1 Create the Droplet In DigitalOcean, create the client droplet under the existing **AI Biz** project. Standard configuration: | Setting | Value | |---|---| | Droplet name | `` | | Project | `AI Biz` | | Region | New York, unless operational needs require otherwise | | OS | Ubuntu 24.04 LTS x64 | | Memory | 2 GB | | CPU | 1 vCPU | | Environment | Production | The standard client floor is **2 GB / 1 vCPU**. Larger droplets are an operational exception based on actual client load, not a separate customer-facing plan. ### 4.2 Assign the Coolify Provisioning SSH Key During droplet creation, select the registered Coolify provisioning public key. Pilot key label: ```text ai-biz-coolify-provisioning ``` The matching private key remains stored in Coolify/control infrastructure. Never copy or expose the private key. ### 4.3 Apply Required Tags Add: ```text client: managed:true ``` For Dr. Hall: ```text client:drhall managed:true ``` ### 4.4 Record Client Infrastructure Details Record privately: ```text Client name Client slug Droplet name DigitalOcean project Region OS Size SSH provisioning key label Tags Public IPv4 Droplet ID ``` ### 4.5 Validation - [ ] Droplet is active - [ ] Droplet is under `AI Biz` - [ ] Ubuntu 24.04 LTS is installed - [ ] Size is at least 2 GB / 1 vCPU - [ ] Coolify provisioning SSH key is authorized - [ ] `client:` tag is present - [ ] `managed:true` tag is present --- ## 5. Apply the DigitalOcean Client Firewall ### 5.1 Standard Firewall Reuse the existing firewall: ```text ai-biz-client-firewall ``` Do not create a new firewall for every client unless the customer has a special security requirement. ### 5.2 Required Inbound Rules | Service | Protocol | Port | Source | |---|---|---:|---| | SSH | TCP | 22 | Control server public IP only (`/32`) | | HTTP | TCP | 80 | All IPv4 / IPv6 | | HTTPS | TCP | 443 | All IPv4 / IPv6 | Do **not** publicly allow: ```text 5678 n8n 5432 PostgreSQL ``` Keep the standard outbound rules enabled unless agency security policy explicitly changes them. Client droplets require outbound access for package/container downloads, workflow APIs, DNS, HTTPS services, monitoring, and backup operations. ### 5.3 Attach the Firewall In DigitalOcean: ```text Networking → Firewalls → ai-biz-client-firewall → Add Droplet ``` Attach ``. ### 5.4 Validation - [ ] Firewall attached to client droplet - [ ] TCP 22 restricted to Control server - [ ] TCP 80 publicly reachable - [ ] TCP 443 publicly reachable - [ ] TCP 5678 not publicly exposed - [ ] TCP 5432 not publicly exposed --- ## 6. Register and Deploy the Client in Coolify ### 6.1 Register the Server Open: ```text https://coolify.ownthestack.io ``` Use: ```text Project: Automation Service Environment: production ``` Go to: ```text Servers → Add Server ``` Configure: ```text Name: Description: AI Biz client server IP Address: Private Key: existing Coolify provisioning key SSH User: root SSH Port: 22 Dedicated Build Server: No ``` Run **Validate Connection**. Expected: ```text Coolify SSH validation: SUCCESS ``` ### 6.2 Create the Docker Compose Resource Go to: ```text Projects → Automation Service → production → Add Resource → Docker Compose ``` Select the client server and paste the template from Section 3. Recommended resource name: ```text -n8n ``` ### 6.3 Configure Environment Variables In Coolify, set: ```text SLUG= DOMAIN=ownthestack.io N8N_VERSION= PG_PASSWORD= N8N_ENCRYPTION_KEY= N8N_LICENSE_ACTIVATION_KEY= ``` Generate a unique `PG_PASSWORD` and `N8N_ENCRYPTION_KEY` for every client. The Community Edition key is agency-managed. Use an agency-controlled key with remaining activation capacity. If registration cannot be completed immediately, do not hold up workspace availability; record the issue and complete registration afterward. ### 6.4 Deploy Click **Deploy** and monitor the startup logs. Expected images: ```text docker.n8n.io/n8nio/n8n: postgres:16-alpine ``` Expected resources: ```text n8n postgres n8n_data pg_data ``` ### 6.5 Validate PostgreSQL In the PostgreSQL runtime logs, confirm: ```text database system is ready to accept connections ``` Verify there are no repeating authentication failures, initialization errors, or restart loops. ### 6.6 Validate n8n In n8n runtime logs, confirm the exact pinned version and successful startup. For the Dr. Hall pilot: ```text Version: 2.38.5 ``` ### 6.7 Deployment Validation In **General → Compose Resources**, verify both resources remain running. During the Dr. Hall pilot, Coolify could show `Running (no healthcheck)` or `Running (unknown)` before explicit application health checks were added; this is acceptable only when the containers remain running and runtime logs show successful startup. For the pilot, the n8n startup output also referenced: ```text https://drhall.ownthestack.io ``` Public access is validated only after DNS and proxy configuration. - [ ] Client server registered in Coolify - [ ] SSH validation successful - [ ] Compose resource created - [ ] n8n running - [ ] PostgreSQL running - [ ] Persistent volumes created - [ ] Exact n8n version confirmed - [ ] PostgreSQL accepts connections - [ ] No critical startup errors --- ## 7. Configure Porkbun DNS and HTTPS ### 7.1 Create the Client A Record DNS for `ownthestack.io` is hosted at Porkbun. Open: ```text https://porkbun.com ``` Log in using the credentials stored in NordPass and open the DNS records for: ```text ownthestack.io ``` Create: ```text Type: A Host: Answer / Value: TTL: Default ``` For Dr. Hall: ```text drhall.ownthestack.io → Dr. Hall client droplet ``` Do not point the record to the Control server. ### 7.2 Verify DNS Resolution From a workstation: ```powershell Resolve-DnsName .ownthestack.io ``` or: ```powershell nslookup .ownthestack.io ``` Confirm the resolved IPv4 matches ``. ### 7.3 Assign the Domain in Coolify Open the client's n8n resource and add: ```text https://.ownthestack.io ``` Expected routing: ```text Internet :443 ↓ .ownthestack.io ↓ Coolify Proxy ↓ n8n:5678 ``` Confirm: ```text Internal port: 5678 HTTP → HTTPS redirect: Enabled ``` If Coolify reports pending changes, apply them with a normal **Restart**. Do not use a **Restart (pull latest)** action during a normal configuration change because the production n8n version is pinned. If the hostname initially returns: ```text no available server ``` that means traffic is reaching the Coolify proxy but the hostname does not yet have a valid routing rule. Re-check the domain assignment, internal port `5678`, save the configuration, and perform the normal restart before testing again. ### 7.4 Verify TLS and Application Access Open: ```text https://.ownthestack.io ``` Verify: - [ ] HTTPS loads - [ ] Valid certificate is presented - [ ] No certificate mismatch - [ ] No “Not secure” warning - [ ] HTTP redirects to HTTPS - [ ] n8n interface loads --- ## 8. Configure the n8n Workspace and Community Edition ### 8.1 Create the Owner Account Open the client workspace and create the owner using the client's confirmed email address. ```text Email: Password: ``` Store the password in NordPass or the approved password manager. Verify: - [ ] Owner account created - [ ] Logout/login works - [ ] Workspace loads normally ### 8.2 Community Edition License Standard License registration is handled by the agency, not by the client. Standard process: ```text Agency-controlled email ↓ Request/receive Community Edition key ↓ Store key securely ↓ Select key with remaining capacity ↓ Set N8N_LICENSE_ACTIVATION_KEY in Coolify ↓ Deploy/restart n8n ↓ Verify registration ``` The current agency process tracks license keys and available activations. Do not put actual license values into this runbook, Git, screenshots, or recordings. ### 8.3 Verify Registration In n8n: ```text Settings → Usage and plan ``` Confirm Community Edition registration is active. Expected: ```text Community Edition: REGISTERED Activation method: ENVIRONMENT VARIABLE Client action required: NO Workspace access: WORKING ``` > **Pilot note:** Dr. Hall was initially deployed before the agency-managed key was added. The key was then added to Coolify and n8n was restarted. Future manual deployments should set the agency key before the first deployment whenever it is available. Community Edition registration is not required for the workspace to function. If registration fails during provisioning: ```text License registration failed ↓ Record/log the issue ↓ Continue client provisioning ↓ Workspace goes live ↓ Retry registration later ``` License registration must not delay client workspace availability. --- ## 9. Add the Client to Uptime Kuma ### 9.1 Create the Monitor Use the existing Uptime Kuma instance on Control. Create: ```text Monitor Type: HTTP(s) Friendly Name: - n8n URL: https://.ownthestack.io Heartbeat Interval: 60 seconds Request Method: GET ``` Do not enable **Ignore TLS/SSL errors**. Enable certificate-expiry monitoring if available. ### 9.2 Validation Confirm: - [ ] Monitor URL is correct - [ ] Interval is 60 seconds - [ ] Status remains `UP` - [ ] HTTPS certificate is valid - [ ] No timeout errors - [ ] No SSL/TLS errors ### 9.3 Agency Notification Status The Dr. Hall HTTPS monitor is active and agency email delivery has now been validated. On **September 30, 2026**, the agency Resend account was used to create a dedicated Uptime Kuma sending key. The verified agency sender domain is `kwagency.team`. Validated SMTP settings: ```text Provider: Resend SMTP Host: smtp.resend.com Port: 2465 Security: TLS Username: resend Sender: alerts@kwagency.team Recipient: approved agency notification inbox ``` Operational note: - Port `465` timed out from the DigitalOcean-hosted Uptime Kuma instance. - Resend port `2465` was then tested and succeeded. - The Uptime Kuma test email was received by the agency inbox. - The saved notifier is named `AI Biz - Agency Email Alerts` and is assigned to the Dr. Hall monitor. - Certificate Expiry Notification is enabled for the Dr. Hall HTTPS monitor. - Certificate-expiry reminder intervals are configured for **7, 14, and 21 days** under Uptime Kuma notification settings. - The Resend API key itself is secret and must remain in the approved secret store. Never place it in this runbook. Slack notification delivery was **not** configured or tested during this verification. Do not represent Slack delivery as validated unless it is separately implemented and tested. --- # Part II — Backup and Recovery Preparation ## 10. Configure Nightly Backups to Backblaze B2 ### 10.1 Backup Scope Each client backup contains: 1. PostgreSQL logical dump using `pg_dump` 2. `n8n_data` volume archive 3. Client-side encryption through Restic 4. Storage in Backblaze B2 under `backups/` Retention: ```text Nightly backups: 30 daily restore points Monthly backups: 12 monthly restore points ``` ### 10.2 Confirm the Client and Install Restic Access the client server through the Coolify terminal and verify the hostname first: ```bash hostname ``` For the Dr. Hall pilot: ```text drhall ``` Then install Restic: ```bash apt update apt install -y restic restic version ``` The Dr. Hall pilot used Restic `0.16.4`. Record the installed version during future deployments rather than assuming it is identical. ### 10.3 Create Backup Directories For Dr. Hall: ```bash mkdir -p /etc/backup mkdir -p /var/backups/drhall chmod 700 /etc/backup chmod 700 /var/backups/drhall ``` For another client, replace `drhall` with the client slug. ### 10.4 Generate a Dedicated Restic Repository Password ```bash openssl rand -base64 48 > /etc/backup/restic-password chmod 600 /etc/backup/restic-password ``` Verify permissions only: ```bash ls -l /etc/backup/restic-password ``` Never print the password. ### 10.5 Create a Restricted Backblaze Backup Key In Backblaze B2, create a dedicated application key. Naming pattern: ```text ai-biz--backup ``` Configuration: ```text Bucket: kwagency-automation-backups Access: Read and Write File prefix: backups// Expiration: None, unless agency policy changes ``` Store the generated credentials in NordPass. ### 10.6 Configure the Restic Environment Create: ```bash install -m 600 /dev/null /etc/backup/b2.env nano /etc/backup/b2.env ``` Add: ```bash B2_ACCOUNT_ID='' B2_ACCOUNT_KEY='' RESTIC_REPOSITORY='b2:kwagency-automation-backups:backups/' RESTIC_PASSWORD_FILE='/etc/backup/restic-password' ``` Load: ```bash set -a source /etc/backup/b2.env set +a ``` Verify only non-secret values: ```bash echo "$RESTIC_REPOSITORY" echo "$RESTIC_PASSWORD_FILE" ``` ### 10.7 Initialize and Verify the Repository ```bash restic init restic snapshots restic check ``` Expected repository check: ```text no errors were found ``` ### 10.8 Identify the PostgreSQL Container and n8n Volume Use: ```bash docker ps --format 'table {{.Names}}\t{{.Image}}' docker volume ls ``` Record the actual Coolify-generated PostgreSQL container name and n8n volume name. For the pilot, these names were generated by Coolify and therefore should **not** be treated as reusable constants for future clients. The PostgreSQL Docker volume is not copied directly. The database is backed up with `pg_dump`. ### 10.9 Perform the First Manual Backup Set the client-specific values: ```bash TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S") BACKUP_DIR="/var/backups//$TIMESTAMP" POSTGRES_CONTAINER='' N8N_VOLUME='' mkdir -p "$BACKUP_DIR" ``` Create the PostgreSQL dump: ```bash docker exec "$POSTGRES_CONTAINER" \ pg_dump -U n8n -d n8n -Fc \ > "$BACKUP_DIR/n8n-postgres.dump" ``` Archive the n8n data volume: ```bash docker run --rm \ -v "$N8N_VOLUME:/source:ro" \ -v "$BACKUP_DIR:/backup" \ alpine:3.20 \ sh -c 'tar -czf /backup/n8n-data.tar.gz -C /source .' ``` Verify: ```bash ls -lh "$BACKUP_DIR" tar -tzf "$BACKUP_DIR/n8n-data.tar.gz" ``` Both files must exist and have non-zero size: ```text n8n-postgres.dump n8n-data.tar.gz ``` ### 10.10 Upload the Manual Backup ```bash restic backup "$BACKUP_DIR" \ --tag client: \ --tag nightly \ --tag manual-test ``` Verify: ```bash restic snapshots restic ls restic check ``` Confirm both backup artifacts are present in the snapshot. --- ## 11. Create the Permanent Nightly Backup Job ### 11.1 Backup Script Create a per-client script, for example: ```text /usr/local/sbin/backup-drhall ``` Use the following standard structure and replace the client/container/volume values: ```bash #!/usr/bin/env bash set -Eeuo pipefail umask 077 CLIENT="drhall" ENV_FILE="/etc/backup/b2.env" BACKUP_ROOT="/var/backups/${CLIENT}" LOG_DIR="/var/log/ai-biz-backup" LOG_FILE="${LOG_DIR}/${CLIENT}-backup.log" POSTGRES_CONTAINER="" N8N_VOLUME="" TIMESTAMP="$(date -u '+%Y-%m-%d_%H-%M-%S')" BACKUP_DIR="${BACKUP_ROOT}/${TIMESTAMP}" mkdir -p "$LOG_DIR" chmod 700 "$LOG_DIR" exec > >(tee -a "$LOG_FILE") 2>&1 log() { echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] $*" } cleanup() { local rc=$? if [ "$rc" -eq 0 ]; then if [ -d "${BACKUP_DIR:-}" ]; then rm -rf "$BACKUP_DIR" fi else log "ERROR: Backup failed with exit code ${rc}." log "Temporary files kept for troubleshooting: ${BACKUP_DIR:-unknown}" fi } trap cleanup EXIT if [ ! -f "$ENV_FILE" ]; then log "ERROR: Backup environment file not found: $ENV_FILE" exit 1 fi set -a source "$ENV_FILE" set +a : "${B2_ACCOUNT_ID:?B2_ACCOUNT_ID is not configured}" : "${B2_ACCOUNT_KEY:?B2_ACCOUNT_KEY is not configured}" : "${RESTIC_REPOSITORY:?RESTIC_REPOSITORY is not configured}" : "${RESTIC_PASSWORD_FILE:?RESTIC_PASSWORD_FILE is not configured}" if ! docker inspect "$POSTGRES_CONTAINER" >/dev/null 2>&1; then log "ERROR: PostgreSQL container not found: $POSTGRES_CONTAINER" exit 1 fi if [ "$(docker inspect -f '{{.State.Running}}' "$POSTGRES_CONTAINER")" != "true" ]; then log "ERROR: PostgreSQL container is not running." exit 1 fi if ! docker volume inspect "$N8N_VOLUME" >/dev/null 2>&1; then log "ERROR: n8n Docker volume not found: $N8N_VOLUME" exit 1 fi mkdir -p "$BACKUP_DIR" chmod 700 "$BACKUP_DIR" log "Starting backup for ${CLIENT}." log "Creating PostgreSQL dump." docker exec "$POSTGRES_CONTAINER" \ pg_dump -U n8n -d n8n -Fc \ > "${BACKUP_DIR}/n8n-postgres.dump" if [ ! -s "${BACKUP_DIR}/n8n-postgres.dump" ]; then log "ERROR: PostgreSQL dump is empty." exit 1 fi log "PostgreSQL dump completed." log "Archiving n8n data volume." docker run --rm \ -v "${N8N_VOLUME}:/source:ro" \ -v "${BACKUP_DIR}:/backup" \ alpine:3.20 \ sh -c 'tar -czf /backup/n8n-data.tar.gz -C /source .' if [ ! -s "${BACKUP_DIR}/n8n-data.tar.gz" ]; then log "ERROR: n8n data archive is empty." exit 1 fi log "n8n data archive completed." log "Uploading encrypted backup to Backblaze B2." RESTIC_TAGS=(--tag "client:${CLIENT}") if [ "$(date -u '+%d')" = "01" ]; then RESTIC_TAGS+=(--tag "monthly") else RESTIC_TAGS+=(--tag "nightly") fi restic backup "$BACKUP_DIR" "${RESTIC_TAGS[@]}" log "Backup uploaded successfully." log "Applying backup retention policy." restic forget \ --tag "client:${CLIENT},nightly" \ --group-by host \ --keep-daily 30 restic forget \ --tag "client:${CLIENT},monthly" \ --group-by host \ --keep-last 12 log "Backup retention completed." log "Backup for ${CLIENT} completed successfully." ``` Make executable and verify syntax: ```bash chmod 700 /usr/local/sbin/backup-drhall bash -n /usr/local/sbin/backup-drhall ``` No output from `bash -n` means the syntax is valid. ### 11.2 Test the Script ```bash /usr/local/sbin/backup-drhall restic snapshots restic check tail -50 /var/log/ai-biz-backup/drhall-backup.log ``` Expected: ```text Backup uploaded successfully. Backup retention completed. Backup for drhall completed successfully. ``` ### 11.3 Validate Retention Before Automation Dry-run nightly retention: ```bash restic forget \ --tag "client:drhall,nightly" \ --group-by host \ --keep-daily 30 \ --dry-run ``` Dry-run monthly retention: ```bash restic forget \ --tag "client:drhall,monthly" \ --group-by host \ --keep-last 12 \ --dry-run ``` ### 11.4 Nightly systemd Service Create `/etc/systemd/system/drhall-backup.service`: ```ini [Unit] Description=Dr. Hall n8n Nightly Backup Requires=docker.service After=docker.service network-online.target Wants=network-online.target [Service] Type=oneshot User=root ExecStart=/usr/local/sbin/backup-drhall ``` ### 11.5 Nightly systemd Timer Create `/etc/systemd/system/drhall-backup.timer`: ```ini [Unit] Description=Run Dr. Hall n8n Backup Nightly [Timer] OnCalendar=*-*-* 02:15:00 America/New_York Persistent=true RandomizedDelaySec=10m [Install] WantedBy=timers.target ``` Enable: ```bash systemctl daemon-reload systemctl enable --now drhall-backup.timer systemctl status drhall-backup.timer --no-pager systemctl list-timers drhall-backup.timer ``` Expected timer state: ```text active (waiting) ``` ### 11.6 Test Through systemd ```bash systemctl start drhall-backup.service systemctl status drhall-backup.service --no-pager ``` Expected: ```text status=0/SUCCESS ``` `inactive (dead)` afterward is normal for a `Type=oneshot` service. ### 11.7 Weekly Restic Prune Create `/usr/local/sbin/prune-drhall-backup`: ```bash #!/usr/bin/env bash set -Eeuo pipefail umask 077 set -a source /etc/backup/b2.env set +a echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] Starting Dr. Hall repository prune." restic prune echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] Dr. Hall repository prune completed." ``` ```bash chmod 700 /usr/local/sbin/prune-drhall-backup ``` Create `/etc/systemd/system/drhall-prune.service`: ```ini [Unit] Description=Dr. Hall Restic Repository Prune After=network-online.target Wants=network-online.target [Service] Type=oneshot User=root ExecStart=/usr/local/sbin/prune-drhall-backup ``` Create `/etc/systemd/system/drhall-prune.timer`: ```ini [Unit] Description=Run Dr. Hall Restic Prune Weekly [Timer] OnCalendar=Sun *-*-* 03:30:00 America/New_York Persistent=true RandomizedDelaySec=15m [Install] WantedBy=timers.target ``` Enable and test: ```bash systemctl daemon-reload systemctl enable --now drhall-prune.timer systemctl start drhall-prune.service systemctl status drhall-prune.service --no-pager systemctl list-timers drhall-prune.timer ``` ### 11.8 Final Backup Verification ```bash set -a source /etc/backup/b2.env set +a restic snapshots restic check systemctl list-timers drhall-backup.timer drhall-prune.timer ``` Required: ```text Backup snapshots: PRESENT Repository integrity: PASS Nightly timer: ACTIVE Weekly prune timer: ACTIVE ``` --- ## 12. Configure n8n Encryption-Key Escrow `N8N_ENCRYPTION_KEY` is required to decrypt credentials stored by n8n. It must be recoverable independently of the client droplet. ### 12.1 Escrow Design ```text Client n8n server │ │ N8N_ENCRYPTION_KEY ▼ SOPS + age public recipient │ ▼ Encrypted escrow file │ ▼ Separate Backblaze B2 escrow bucket Control server └── Holds the private age identity for recovery ``` The age private key must never be copied to the client server. ### 12.2 Shared Backblaze Escrow Bucket Reuse: ```text kwagency-ai-biz-key-escrow ``` Configuration: ```text Private bucket Default encryption enabled Object Lock disabled unless policy changes ``` ### 12.3 Create a Restricted Client Escrow Application Key Naming pattern: ```text ai-biz--key-escrow ``` Restrict it to: ```text Bucket: kwagency-ai-biz-key-escrow Access: Read and Write Allow List All Bucket Names: Enabled during the validated Dr. Hall pilot File prefix: / Expiration: None, unless agency policy changes ``` Store the generated credentials in NordPass. ### 12.4 One-Time Agency age Identity on Control If the agency age identity does not already exist, install age and SOPS on Control and generate it once. Install age and curl: ```bash apt update apt install -y age curl age --version age-keygen --version ``` Install the approved SOPS version. The pilot used `3.13.3` and verified the downloaded binary against the published checksum before installation: ```bash cd /tmp SOPS_VERSION="3.13.3" curl -fLO "https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.amd64" curl -fLO "https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.checksums.txt" grep "sops-v${SOPS_VERSION}.linux.amd64$" \ "sops-v${SOPS_VERSION}.checksums.txt" | sha256sum -c - install -m 0755 \ "sops-v${SOPS_VERSION}.linux.amd64" \ /usr/local/bin/sops rm -f "/tmp/sops-v${SOPS_VERSION}.linux.amd64" rm -f "/tmp/sops-v${SOPS_VERSION}.checksums.txt" sops --version ``` Generate the agency age identity: ```bash install -d -m 700 /etc/ai-biz/age age-keygen -o /etc/ai-biz/age/keys.txt chmod 600 /etc/ai-biz/age/keys.txt age-keygen -y /etc/ai-biz/age/keys.txt \ > /etc/ai-biz/age/recipient.txt chmod 644 /etc/ai-biz/age/recipient.txt ``` Files: ```text keys.txt PRIVATE — Control only recipient.txt PUBLIC — may be copied to client servers ``` Store a secure disaster-recovery copy of the private identity in the approved agency secret store. ### 12.5 Install age and SOPS on the Client On the client server: ```bash apt update apt install -y age curl cd /tmp SOPS_VERSION="3.13.3" curl -fLO "https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.amd64" curl -fLO "https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.checksums.txt" grep "sops-v${SOPS_VERSION}.linux.amd64$" "sops-v${SOPS_VERSION}.checksums.txt" | sha256sum -c - install -m 0755 "sops-v${SOPS_VERSION}.linux.amd64" /usr/local/bin/sops rm -f "/tmp/sops-v${SOPS_VERSION}.linux.amd64" rm -f "/tmp/sops-v${SOPS_VERSION}.checksums.txt" age --version sops --version ``` The Dr. Hall pilot used SOPS `3.13.3` and verified the downloaded binary before installation. ### 12.6 Install the Public Recipient on the Client Create: ```bash install -d -m 755 /etc/ai-biz/age nano /etc/ai-biz/age/recipient.txt chmod 644 /etc/ai-biz/age/recipient.txt ``` Paste only the public `age1...` recipient from Control. Never copy `/etc/ai-biz/age/keys.txt` to the client. ### 12.7 Confirm the Runtime n8n Encryption Key Exists Identify the n8n container: ```bash N8N_CONTAINER=$(docker ps --format '{{.Names}} {{.Image}}' \ | awk '$2 ~ /n8nio\/n8n:/ {print $1; exit}') ``` Check without displaying the secret: ```bash docker inspect "$N8N_CONTAINER" \ --format '{{range .Config.Env}}{{println .}}{{end}}' \ | grep -q '^N8N_ENCRYPTION_KEY=' \ && echo "N8N_ENCRYPTION_KEY: PRESENT" \ || echo "N8N_ENCRYPTION_KEY: MISSING" ``` ### 12.8 Create the Encrypted Escrow File ```bash install -d -m 700 /etc/ai-biz/escrow AGE_RECIPIENT="$(cat /etc/ai-biz/age/recipient.txt)" N8N_KEY="$( docker inspect "$N8N_CONTAINER" \ --format '{{range .Config.Env}}{{println .}}{{end}}' \ | sed -n 's/^N8N_ENCRYPTION_KEY=//p' \ | head -n1 )" export N8N_KEY ``` Encrypt directly without creating a plaintext JSON file: ```bash python3 - <<'PY' | \ sops encrypt \ --age "$AGE_RECIPIENT" \ --input-type json \ --output-type json \ /dev/stdin \ > /etc/ai-biz/escrow/-n8n-encryption-key.enc.json import json import os print(json.dumps({"N8N_ENCRYPTION_KEY": os.environ["N8N_KEY"]})) PY ``` Immediately clear temporary variables: ```bash unset N8N_KEY unset AGE_RECIPIENT chmod 600 /etc/ai-biz/escrow/-n8n-encryption-key.enc.json ``` ### 12.9 Verify the Encrypted File ```bash grep -q '"sops"' /etc/ai-biz/escrow/-n8n-encryption-key.enc.json \ && echo "SOPS metadata: PRESENT" grep -q 'ENC\[' /etc/ai-biz/escrow/-n8n-encryption-key.enc.json \ && echo "Encrypted payload: PRESENT" ``` Do not decrypt the escrow file on the client server. ### 12.10 Configure Restricted B2 Escrow Credentials Create: ```bash install -m 600 /dev/null /etc/ai-biz/escrow/b2-escrow.env ``` Contents: ```bash B2_ESCROW_KEY_ID='' B2_ESCROW_APPLICATION_KEY='' B2_ESCROW_BUCKET='kwagency-ai-biz-key-escrow' B2_ESCROW_PREFIX='' ``` ### 12.11 Install the B2 CLI For the pilot, the B2 CLI was installed into an isolated Python virtual environment: ```bash apt update apt install -y python3-venv python3 -m venv /opt/b2-cli /opt/b2-cli/bin/pip install --upgrade pip /opt/b2-cli/bin/pip install b2 ln -sf /opt/b2-cli/bin/b2 /usr/local/bin/b2 which b2 b2 version ``` Expected command path: ```text /usr/local/bin/b2 ``` ### 12.12 Load Escrow Credentials Safely ```bash set -a source /etc/ai-biz/escrow/b2-escrow.env set +a export B2_APPLICATION_KEY_ID="$B2_ESCROW_KEY_ID" export B2_APPLICATION_KEY="$B2_ESCROW_APPLICATION_KEY" ``` Verify without displaying values: ```bash [ -n "$B2_APPLICATION_KEY_ID" ] && echo "B2 Key ID: LOADED" [ -n "$B2_APPLICATION_KEY" ] && echo "B2 Application Key: LOADED" ``` Verify access to the client escrow prefix before uploading: ```bash b2 ls "b2://kwagency-ai-biz-key-escrow//" ``` An empty result is acceptable before the first upload as long as there is no authorization error. Do not run B2 CLI commands that print authorization tokens/application keys into shared logs or documentation. In particular, the validated procedure loads credentials through the protected environment file rather than using an interactive authorization command that could echo sensitive material. ### 12.13 Upload the Encrypted Escrow File Record the encrypted-file checksum: ```bash sha256sum /etc/ai-biz/escrow/-n8n-encryption-key.enc.json ``` Upload: ```bash b2 file upload \ kwagency-ai-biz-key-escrow \ /etc/ai-biz/escrow/-n8n-encryption-key.enc.json \ /-n8n-encryption-key.enc.json ``` Verify: ```bash b2 ls "b2://kwagency-ai-biz-key-escrow//" ``` Clear shell variables afterward: ```bash unset B2_APPLICATION_KEY_ID unset B2_APPLICATION_KEY unset B2_ESCROW_KEY_ID unset B2_ESCROW_APPLICATION_KEY ``` ### 12.14 Verify Recovery from Control Configure the restricted escrow credentials on Control in a root-only file: ```bash install -d -m 700 /etc/ai-biz/escrow install -m 600 /dev/null /etc/ai-biz/escrow/b2-escrow.env ``` Store only the approved client escrow credentials in that file, then load them: ```bash set -a source /etc/ai-biz/escrow/b2-escrow.env set +a export B2_APPLICATION_KEY_ID="$B2_ESCROW_KEY_ID" export B2_APPLICATION_KEY="$B2_ESCROW_APPLICATION_KEY" ``` Confirm the encrypted object is visible, then download it: ```bash b2 ls "b2://kwagency-ai-biz-key-escrow//" b2 file download \ "b2://kwagency-ai-biz-key-escrow//-n8n-encryption-key.enc.json" \ "/etc/ai-biz/escrow/-n8n-encryption-key.downloaded.enc.json" chmod 600 /etc/ai-biz/escrow/-n8n-encryption-key.downloaded.enc.json ``` Compare SHA-256 against the value recorded before upload. Then verify decryption without printing the actual key: ```bash export SOPS_AGE_KEY_FILE="/etc/ai-biz/age/keys.txt" sops decrypt \ /etc/ai-biz/escrow/-n8n-encryption-key.downloaded.enc.json \ | python3 -c ' import json, sys data = json.load(sys.stdin) key = data.get("N8N_ENCRYPTION_KEY", "") print("N8N encryption key decryption: PASS" if key else "N8N encryption key decryption: FAIL") ' ``` Required: ```text Checksum verification: PASS N8N encryption key decryption: PASS ``` --- ## 13. Escrow the Restic Repository Password for Recovery A backup repository is not independently recoverable if the Restic password exists only on the client droplet. The pilot therefore escrowed the Restic password using the agency age public key and the separate B2 escrow bucket. ### 13.1 Prerequisite The client server must already have the agency public age recipient at: ```text /etc/ai-biz/age/recipient.txt ``` The private age identity remains on Control only. ### 13.2 Encrypt the Existing Restic Password On the client server: ```bash age \ -r "$(cat /etc/ai-biz/age/recipient.txt)" \ -o /etc/ai-biz/escrow/-restic-password.age \ /etc/backup/restic-password ``` Protect: ```bash chmod 600 /etc/ai-biz/escrow/-restic-password.age ``` Do not display the original password. ### 13.3 Upload to the Escrow Bucket Load the restricted escrow credentials configured in Section 12 and upload: ```bash b2 file upload \ kwagency-ai-biz-key-escrow \ /etc/ai-biz/escrow/-restic-password.age \ /-restic-password.age ``` Verify: ```bash b2 ls "b2://kwagency-ai-biz-key-escrow//" ``` ### 13.4 Recover the Restic Password on Control On Control, load the client escrow B2 credentials from the root-only escrow environment configured in Section 12, then download the encrypted Restic-password object: ```bash set -a source /etc/ai-biz/escrow/b2-escrow.env set +a export B2_APPLICATION_KEY_ID="$B2_ESCROW_KEY_ID" export B2_APPLICATION_KEY="$B2_ESCROW_APPLICATION_KEY" install -d -m 700 /etc/ai-biz/escrow b2 file download \ "b2://kwagency-ai-biz-key-escrow//-restic-password.age" \ "/etc/ai-biz/escrow/-restic-password.downloaded.age" chmod 600 /etc/ai-biz/escrow/-restic-password.downloaded.age ``` Decrypt it using the Control server's private age identity: ```bash install -d -m 700 /etc/ai-biz/restic age \ --decrypt \ -i /etc/ai-biz/age/keys.txt \ -o /etc/ai-biz/restic/-restic-password \ /etc/ai-biz/escrow/-restic-password.downloaded.age chmod 600 /etc/ai-biz/restic/-restic-password ``` Validate without printing the password: ```bash [ -s /etc/ai-biz/restic/-restic-password ] \ && echo "Restic password recovery: PASS" \ || echo "Restic password recovery: FAIL" ``` Create the Control-side client backup recovery environment using the dedicated backup application-key credentials stored in NordPass: ```bash install -m 600 /dev/null /etc/ai-biz/restic/-b2.env nano /etc/ai-biz/restic/-b2.env ``` Contents: ```bash B2_ACCOUNT_ID='' B2_ACCOUNT_KEY='' RESTIC_REPOSITORY='b2:kwagency-automation-backups:backups/' RESTIC_PASSWORD_FILE='/etc/ai-biz/restic/-restic-password' ``` Load it and verify independent repository access from Control: ```bash set -a source /etc/ai-biz/restic/-b2.env set +a restic snapshots restic check ``` Required: ```text Restic password recovery: PASS Repository opens successfully restic check: no errors were found ``` --- # Part III — Restore and Disaster Recovery Drill ## 14. Restore Scope and Safety This section documents the **validated scratch restore drill** used to prove that a client backup can be recovered to a fresh droplet. The restore drill must not modify the production client server or production DNS. > **Important:** A real production disaster cutover would require additional promotion/cutover steps. The pilot validated recovery to an isolated scratch environment; production DNS cutover was intentionally not performed. Target restore time: ```text ≤ 60 minutes ``` ### 14.1 Optimized Recovery Procedure — Validated in Drill #2 The second Dr. Hall restore drill established the **preferred operational recovery path**. The full manual recovery procedure in Sections 15–24 is intentionally retained as the baseline/fallback procedure and as the source process that future automated provisioning and recovery should map and automate. For a timed restore drill: ```text START = immediately before creating the fresh DigitalOcean restore droplet STOP = only after all three are true: 1. restored n8n /healthz = PASS 2. existing owner login = PASS 3. RESTORE TEST - SAFE workflow = PASS ``` Cleanup is performed after the timer stops. Before starting the timer, verify on Control: ```bash for f in \ /etc/ai-biz/age/keys.txt \ /etc/ai-biz/age/recipient.txt \ /etc/ai-biz/restic/-restic-password \ /etc/ai-biz/restic/-b2.env do if [ -s "$f" ]; then echo "PASS: $f" else echo "MISSING: $f" fi done set -a source /etc/ai-biz/restic/-b2.env set +a restic snapshots restic check ``` Verify the selected snapshot contains both restore artifacts: ```bash restic ls \ | grep -E 'n8n-postgres\.dump|n8n-data\.tar\.gz' ``` Verify that the encrypted n8n key escrow is recoverable without printing the key: ```bash SOPS_AGE_KEY_FILE=/etc/ai-biz/age/keys.txt \ sops decrypt \ /etc/ai-biz/escrow/-n8n-encryption-key.downloaded.enc.json \ | python3 -c ' import json, sys data = json.load(sys.stdin) print( "PASS: N8N encryption key recoverable" if data.get("N8N_ENCRYPTION_KEY") else "FAIL: N8N encryption key unavailable" ) ' ``` For the Dr. Hall pilot, validate the permanent recovery helpers before a timed drill: ```bash bash -n /opt/ai-biz-recovery/restore-drhall-remote.sh bash -n /usr/local/sbin/ai-biz-restore-drhall ls -l \ /opt/ai-biz-recovery/restore-drhall-remote.sh \ /usr/local/sbin/ai-biz-restore-drhall ``` Expected permissions: ```text -rwx------ ``` The optimized restore is started from Control with: ```bash umask 077 LOG="/root/-restore-test--$(date -u '+%Y%m%dT%H%M%SZ').log" set -o pipefail /usr/local/sbin/ai-biz-restore-drhall \ \ latest \ 2>&1 | tee "$LOG" RC=${PIPESTATUS[0]} echo echo "Restore helper exit code: $RC" echo "Restore log: $LOG" ``` Required successful helper result: ```text RESTORE AUTOMATION: PASS Restore helper exit code: 0 n8n health: PASS Active workflows: 0 ``` The helper automates the repetitive server-side recovery work, including Restic restore, PostgreSQL restore, workflow deactivation, n8n data restore, encryption-key validation, exact n8n version startup, and `/healthz` checking. It does **not** modify production DNS, activate restored workflows, destroy droplets, or delete backups/escrow. The timer continues after the helper finishes until owner login and the safe workflow test are both complete. --- ### 14.2 Procedure A — Manual Baseline from Restore Drill #1 Sections **15–24** document the full step-by-step manual recovery path validated during **Restore Drill #1**. Restore Drill #1 was functionally successful but took: ```text 1 hour 28 minutes 40 seconds ``` and therefore did not meet the `≤ 60 minutes` recovery SLO. This manual procedure must remain in the runbook even after optimization because it documents the atomic recovery actions that the future automated provisioning/recovery system will need to reproduce, including: ```text Fresh scratch infrastructure Temporary SSH trust/access Backup repository access Snapshot restore PostgreSQL restore Workflow deactivation n8n_data restore Encryption-key recovery and comparison Exact n8n version startup Local-only health validation Two-stage workstation access Owner-login verification Safe workflow verification Cleanup ``` The optimized helper used in Drill #2 automates many of these same actions, but the manual baseline remains the authoritative fallback and implementation reference. --- ## 15. Prepare Control for Manual Restore Drill #1 The following sections preserve the **full manual baseline used for Restore Drill #1**. This path is retained both as the operational fallback and as the atomic source process for future recovery automation. ### 15.1 Recovery Components Recovery depends on three independent components: ```text 1. Backblaze B2 backup repository ├── PostgreSQL database dump └── n8n_data archive 2. Restic recovery credentials ├── Backblaze backup application-key credentials └── Restic repository password 3. n8n encryption-key escrow └── Original N8N_ENCRYPTION_KEY ``` Target: ```text Full client recovery to fresh droplet: ≤ 60 minutes ``` The restore environment must remain isolated from production throughout the drill. ### 15.2 Pre-Restore Requirements Before beginning, confirm Control has access to: ```text Restic repository Recovered Restic repository password Backblaze backup credentials Private age identity Backblaze key-escrow access Coolify DigitalOcean ``` Permanent Dr. Hall recovery files include: ```text /etc/ai-biz/age/keys.txt /etc/ai-biz/age/recipient.txt /etc/ai-biz/restic/drhall-restic-password /etc/ai-biz/restic/drhall-b2.env /etc/ai-biz/escrow/ ``` Secret files should remain mode `600` and secret directories mode `700`. Never print their contents. ### 15.3 Verify the Backup Repository from Control On Control: ```bash set -a source /etc/ai-biz/restic/drhall-b2.env set +a restic snapshots restic check ``` Select the newest successful nightly snapshot. Required: ```text no errors were found ``` Do not proceed if repository integrity fails. ### 15.4 Record the Restore Start Time Immediately before creating the restore environment: ```bash date -u ``` Record: ```text Restore Test Started: ``` For Drill #1 the recorded start was: ```text Thu Sep 17 15:19:11 UTC 2026 ``` --- ## 16. Create and Isolate the Fresh Restore Droplet ### 16.1 Create the Scratch Droplet Create a temporary droplet in the existing DigitalOcean `AI Biz` project. For the validated Dr. Hall manual drill: ```text Hostname: drhall-restore-test Operating System: Ubuntu 24.04 LTS Size: 2 GB RAM / 1 vCPU Region: Same region as production when practical Authentication: Existing Coolify provisioning SSH public key Firewall: ai-biz-client-firewall ``` Allowed inbound access: ```text 22 SSH from authorized management source 80 HTTP 443 HTTPS ``` Do not publicly expose: ```text 5678 n8n 5432 PostgreSQL ``` ### 16.2 Keep the Restore Environment Isolated Do not change production DNS. Production remains: ```text drhall.ownthestack.io ↓ Production Dr. Hall server ``` Scratch restore environment: ```text drhall-restore-test └── No production DNS ``` Do not: - modify production DNS; - route production traffic to the restore server; - add the scratch server to the production Uptime Kuma monitor; - expose PostgreSQL publicly; - expose n8n port `5678` publicly. ### 16.3 Register the Restore Server in Coolify In Coolify: ```text Servers → Add Server ``` Use: ```text Name: drhall-restore-test SSH key: existing Coolify provisioning key ``` Validate the server. Required: ```text Server validation: SUCCESS Docker connectivity: SUCCESS ``` Using the Coolify terminal on the scratch server: ```bash hostname docker ps df -h / ``` Expected hostname: ```text drhall-restore-test ``` At this point only normal Coolify infrastructure should normally be present; there should be no restored n8n/PostgreSQL stack yet. --- ## 17. Create Temporary Control-to-Restore SSH Access — Drill #1 Method This temporary key is used to copy recovery files and create the Control → restore tunnel. On Control: ```bash ssh-keygen \ -t ed25519 \ -f /root/.ssh/drhall-restore-test \ -N "" \ -C "temporary-drhall-restore-test" ``` This creates: ```text /root/.ssh/drhall-restore-test Private key /root/.ssh/drhall-restore-test.pub Public key ``` Never display or copy the private key. Display only the public key: ```bash cat /root/.ssh/drhall-restore-test.pub ``` Using the Coolify terminal on `drhall-restore-test`, append that public key to: ```text /root/.ssh/authorized_keys ``` Then: ```bash chmod 700 /root/.ssh chmod 600 /root/.ssh/authorized_keys ``` Test from Control: ```bash ssh \ -i /root/.ssh/drhall-restore-test \ root@ \ hostname ``` Expected: ```text drhall-restore-test ``` > **Drill-history note:** Drill #1 used the direct SSH command above. `StrictHostKeyChecking=accept-new` and explicit stale-`known_hosts` handling were introduced and validated during Drill #2; those steps are documented in Procedure B rather than retroactively changing the recorded Drill #1 procedure. --- ## 18. Restore the Backup Artifacts Manually ### 18.1 Install Restic and Create Protected Directories On `drhall-restore-test`: ```bash apt update apt install -y restic restic version install -d -m 700 /etc/restore install -d -m 700 /var/restore/drhall ``` ### 18.2 Transfer Restic Recovery Files from Control From Control: ```bash scp \ -i /root/.ssh/drhall-restore-test \ /etc/ai-biz/restic/drhall-restic-password \ root@:/etc/restore/restic-password ``` Then: ```bash scp \ -i /root/.ssh/drhall-restore-test \ /etc/ai-biz/restic/drhall-b2.env \ root@:/etc/restore/drhall-b2.env ``` On the restore server: ```bash chmod 600 /etc/restore/restic-password chmod 600 /etc/restore/drhall-b2.env ``` Update the copied Restic password-file path: ```bash sed -i \ 's#^RESTIC_PASSWORD_FILE=.*#RESTIC_PASSWORD_FILE=/etc/restore/restic-password#' \ /etc/restore/drhall-b2.env ``` Verify only the non-secret values: ```bash grep '^RESTIC_REPOSITORY=' /etc/restore/drhall-b2.env grep '^RESTIC_PASSWORD_FILE=' /etc/restore/drhall-b2.env ``` Do not display the B2 credentials or Restic password. ### 18.3 Verify Repository Access from the Restore Server ```bash set -a source /etc/restore/drhall-b2.env set +a restic snapshots restic check ``` The same snapshots visible from Control should appear. Required: ```text no errors were found ``` ### 18.4 Restore the Selected Snapshot ```bash restic restore \ --target /var/restore/drhall ``` Locate the restored files: ```bash find /var/restore/drhall -type f -ls ``` Expected: ```text n8n-postgres.dump n8n-data.tar.gz ``` Set the actual restored backup directory: ```bash RESTORED_BACKUP_DIR="$( find /var/restore/drhall/var/backups/drhall \ -type f -name 'n8n-postgres.dump' \ -printf '%h\n' | head -n1 )" ``` Verify: ```bash ls -lh "$RESTORED_BACKUP_DIR" ``` Both restore artifacts must have non-zero size. ### 18.5 Validate the n8n Data Archive ```bash tar -tzf "$RESTORED_BACKUP_DIR/n8n-data.tar.gz" ``` Typical contents include: ```text ./ ./config ./storage/ ./nodes/ ./n8nEventLog.log ``` Do not start n8n yet. ### 18.6 Validate the PostgreSQL Dump Use PostgreSQL 16 tooling: ```bash docker run --rm \ -v "$RESTORED_BACKUP_DIR:/backup:ro" \ postgres:16-alpine \ pg_restore -l /backup/n8n-postgres.dump | head -30 ``` A valid custom-format PostgreSQL archive should display its table of contents. Do not continue if it reports: ```text input file does not appear to be a valid archive ``` --- ## 19. Restore PostgreSQL and n8n Data Manually ### 19.1 Create Isolated Docker Resources ```bash docker network create drhall_restore_net docker volume create drhall_restore_pg_data docker volume create drhall_restore_n8n_data ``` Verify: ```bash docker volume ls | grep drhall_restore ``` Expected: ```text drhall_restore_n8n_data drhall_restore_pg_data ``` ### 19.2 Generate a Temporary PostgreSQL Password The original production PostgreSQL password is not required for a logical `pg_dump` restore. ```bash umask 077 PG_PASSWORD="$(openssl rand -hex 32)" cat > /etc/restore/postgres.env < \ 'install -d -m 700 /etc/restore && umask 077 && cat > /etc/restore/n8n-encryption-key && chmod 600 /etc/restore/n8n-encryption-key' ``` Verify on the restore server without printing the key: ```bash [ -s /etc/restore/n8n-encryption-key ] \ && echo "Recovered N8N encryption key: PRESENT" \ || echo "Recovered N8N encryption key: MISSING" ``` Expected: ```text Recovered N8N encryption key: PRESENT ``` ### 20.2 Verify the Escrow Key Against the Restored n8n Configuration ```bash docker run --rm \ -v drhall_restore_n8n_data:/data:ro \ -v /etc/restore/n8n-encryption-key:/run/escrow-key:ro \ python:3.12-alpine \ python3 -c ' import json with open("/data/config") as f: config = json.load(f) with open("/run/escrow-key") as f: escrow_key = f.read().strip() config_key = config.get("encryptionKey") if not config_key: print("Restored config encryption key: NOT PRESENT") elif config_key == escrow_key: print("Escrow key matches restored n8n config: PASS") else: print("Escrow key matches restored n8n config: FAIL") ' ``` Required: ```text Escrow key matches restored n8n config: PASS ``` If the result is `FAIL`, do not start n8n. --- ## 21. Start the Restored n8n Instance in Local-Only Mode — Manual Drill ### 21.1 Create the Complete Restore Environment File Load the temporary PostgreSQL environment: ```bash set -a source /etc/restore/postgres.env set +a ``` Load the recovered n8n key: ```bash N8N_KEY="$(cat /etc/restore/n8n-encryption-key)" ``` Create: ```bash umask 077 cat > /etc/restore/n8n.env < ``` If an existing authorized key works, reuse it and skip temporary workstation-key creation. If authentication fails, continue below. ### 22.2 Create a Temporary Windows-to-Control Key On Windows PowerShell: ```powershell ssh-keygen -t ed25519 ` -f "$env:USERPROFILE\.ssh\ai-biz-control" ` -C "temporary-ai-biz-control-access" ``` This creates: ```text ai-biz-control Private key ai-biz-control.pub Public key ``` Never display or share the private key. Display only the public key: ```powershell Get-Content "$env:USERPROFILE\.ssh\ai-biz-control.pub" ``` Append the public key to Control: ```text /root/.ssh/authorized_keys ``` Then on Control: ```bash chmod 600 /root/.ssh/authorized_keys ``` Test from Windows: ```powershell ssh ` -i "$env:USERPROFILE\.ssh\ai-biz-control" ` root@ ` hostname ``` Expected: ```text control ``` ### 22.3 Create the Control-to-Restore n8n Tunnel On Control: ```bash ssh -f -N -T \ -o ExitOnForwardFailure=yes \ -i /root/.ssh/drhall-restore-test \ -L 127.0.0.1:15679:127.0.0.1:5678 \ root@ ``` Verify: ```bash curl -fsS http://127.0.0.1:15679/healthz && echo ``` Expected: ```json {"status":"ok"} ``` ### 22.4 Create the Windows-to-Control Tunnel From Windows PowerShell: ```powershell ssh -N -T ` -o ExitOnForwardFailure=yes ` -i "$env:USERPROFILE\.ssh\ai-biz-control" ` -L 5679:127.0.0.1:15679 ` root@ ``` Keep this PowerShell window open. Open: ```text http://localhost:5679 ``` Connection path: ```text Browser http://localhost:5679 ↓ Windows SSH tunnel ↓ Control:127.0.0.1:15679 ↓ Restore server:127.0.0.1:5678 ↓ Restored n8n ``` --- ## 23. Validate and Complete Manual Restore Drill #1 ### 23.1 Verify the Existing Owner Account Open: ```text http://localhost:5679 ``` Use the existing client n8n owner credentials. Verify: ```text Restore login page PASS Existing owner login PASS n8n dashboard PASS ``` Do not create another owner account unless specifically required. ### 23.2 Perform the Safe Workflow Execution Test Create: ```text RESTORE TEST - SAFE ``` Use: ```text Manual Trigger ↓ Edit Fields ``` Set: ```text restore_test = success ``` Do not use webhooks, schedules, Gmail, HTTP Request, external APIs, production databases, or any real client integration. Execute manually. Expected output: ```json { "restore_test": "success" } ``` Record: ```text Workflow execution test: PASS ``` Do not activate the workflow. ### 23.3 Confirm the Test Workflow Remains Inactive On the restore server: ```bash docker exec drhall-restore-postgres \ psql -U n8n -d n8n -c \ 'SELECT name, active FROM workflow_entity;' ``` Expected: ```text RESTORE TEST - SAFE | f ``` `f` confirms the workflow remains inactive. ### 23.4 Record Completion Time ```bash date -u ``` Record: ```text Restore Test Completed: ``` For Drill #1: ```text Thu Sep 17 16:47:51 UTC 2026 ``` Calculate: ```text Restore Duration = Restore Completed - Restore Started ``` Compare against: ```text Restore SLO: ≤ 60 minutes ``` Functional recovery and restore timing are recorded separately. ### 23.5 Manual Restore Success Criteria A successful manual restore confirms: ```text Fresh restore server created PASS Coolify validation PASS Backup repository accessible PASS Restic repository password recovered PASS Repository integrity PASS Latest backup restored PASS PostgreSQL restored PASS n8n_data restored PASS N8N_ENCRYPTION_KEY recovered PASS Encryption-key verification PASS Exact production n8n version starts PASS Health endpoint PASS Existing owner account accessible PASS Stored credentials decrypt PASS / N/A Safe workflow executes PASS Test workflow remains inactive PASS Production environment remains untouched PASS ``` The restore-duration SLO is evaluated separately. --- # Part IV — Restore Drill #1 Cleanup ## 24. Cleanup After the Manual Baseline Restore This cleanup procedure belongs specifically to **Restore Drill #1**, the fully manual recovery baseline. Drill #1 did **not** use the optimized recovery helper scripts, the managed SSH control socket, or the Drill #2 scratch copy of `/opt/ai-biz-recovery`. Keep this cleanup separate from the optimized Drill #2 cleanup in Section 27. Cleanup begins only after the Drill #1 completion time has been recorded. ### 24.1 Stop the Restore Containers On the scratch restore server: ```bash docker stop drhall-restore-n8n docker stop drhall-restore-postgres ``` Verify: ```bash docker ps ``` Only the normal Coolify infrastructure should remain, for example: ```text coolify-proxy coolify-sentinel ``` If the scratch droplet is going to be destroyed immediately, the Docker volumes do not need to be manually deleted during Drill #1 cleanup. ### 24.2 Remove Temporary Restore Secrets and Working Files On the scratch restore server: ```bash rm -rf /etc/restore rm -rf /var/restore/drhall ``` Verify: ```bash test ! -e /etc/restore \ && echo "Restore secrets: REMOVED" test ! -e /var/restore/drhall \ && echo "Restore working data: REMOVED" ``` Do not remove anything from the production `drhall` server. ### 24.3 Close the Windows Workstation-to-Control Tunnel In the Windows PowerShell window running the tunnel: ```text Ctrl+C ``` Verify that: ```text http://localhost:5679 ``` no longer loads the restored n8n instance. ### 24.4 Close the Control-to-Restore Tunnel Drill #1 used a normal background SSH tunnel rather than the managed control-socket method introduced in Drill #2. On Control: ```bash ss -ltnp | grep 15679 ``` Identify the SSH process listening on: ```text 127.0.0.1:15679 ``` Terminate **only that restore-tunnel SSH process**: ```bash kill ``` Verify: ```bash ss -ltnp | grep 15679 \ || echo "Restore tunnel: CLOSED" ``` Expected: ```text Restore tunnel: CLOSED ``` ### 24.5 Remove Temporary Windows Workstation Access from Control If Drill #1 created the temporary Windows → Control key with the comment: ```text temporary-ai-biz-control-access ``` remove only that entry from Control: ```bash sed -i \ '/temporary-ai-biz-control-access$/d' \ /root/.ssh/authorized_keys ``` Ensure permissions remain correct: ```bash chmod 600 /root/.ssh/authorized_keys ``` Verify: ```bash grep 'temporary-ai-biz-control-access' \ /root/.ssh/authorized_keys \ || echo "Temporary workstation access: REMOVED" ``` Expected: ```text Temporary workstation access: REMOVED ``` Do not remove unrelated permanent SSH keys. ### 24.6 Delete the Temporary Windows Key Pair On Windows PowerShell: ```powershell Remove-Item "$env:USERPROFILE\.ssh\ai-biz-control" Remove-Item "$env:USERPROFILE\.ssh\ai-biz-control.pub" ``` Verify: ```powershell Get-ChildItem "$env:USERPROFILE\.ssh" ``` The temporary Drill #1 key pair should no longer exist. ### 24.7 Remove the Temporary Control-to-Restore SSH Key On Control: ```bash rm -f /root/.ssh/drhall-restore-test rm -f /root/.ssh/drhall-restore-test.pub ``` Verify: ```bash test ! -e /root/.ssh/drhall-restore-test \ && echo "Temporary restore private key: REMOVED" test ! -e /root/.ssh/drhall-restore-test.pub \ && echo "Temporary restore public key: REMOVED" ``` ### 24.8 Remove the Scratch Server from Coolify In Coolify: ```text Servers → drhall-restore-test → Danger → Delete Server ``` Delete only: ```text drhall-restore-test ``` Do not select the production `drhall` server. For the manual drill, remove the Coolify server entry before destroying the DigitalOcean droplet. ### 24.9 Destroy the DigitalOcean Scratch Droplet In DigitalOcean: ```text AI Biz → drhall-restore-test → Destroy ``` Verify both the hostname and IP before confirming destruction. Do not destroy the production `drhall` droplet. ### 24.10 Verify Production Is Still Healthy After cleanup: - open the normal production Dr. Hall URL; - confirm n8n loads normally; - confirm Uptime Kuma reports the production workspace as healthy; - confirm production DNS was not changed by the drill. ### 24.11 Drill #1 Cleanup Completion Checklist Record: ```text Restore Drill #1 cleanup: Restore containers STOPPED Restore secrets/files REMOVED Windows → Control tunnel CLOSED Control → restore tunnel CLOSED Temporary Windows authorization REMOVED Temporary Windows key pair REMOVED Temporary Control → restore key pair REMOVED Coolify scratch server REMOVED DigitalOcean scratch droplet DELETED Production client instance UNTOUCHED Production DNS UNCHANGED Nightly backups RETAINED Encryption-key escrow RETAINED Restic recovery escrow RETAINED ``` > **Drill #1 note:** no optimized helper files existed on the scratch server, and the Drill #2 managed tunnel socket / per-drill `known_hosts` cleanup was not part of the original validated Drill #1 cleanup. --- # Part V — Optimized Restore Procedure (Restore Drill #2) ## 25. Install the Validated Optimized Recovery Helper Scripts Restore Drill #2 used two permanent Control-side scripts: ```text /opt/ai-biz-recovery/restore-drhall-remote.sh /usr/local/sbin/ai-biz-restore-drhall ``` The two script files above are the validated Restore Drill #2 helpers and must remain on Control unless they are deliberately superseded by a tested newer version. These scripts are **Dr. Hall pilot implementations**. They intentionally contain `drhall`-specific names because this is the exact pilot flow that was validated. When future automated provisioning/recovery is implemented, parameterize the client slug, image version, paths, ports, and recovery metadata rather than duplicating these hard-coded values across customers. The scripts do not replace the manual baseline in Sections 15–24. They automate repeatable parts of that baseline. ### 25.1 Remote Restore Worker Create on Control: ```bash install -d -m 700 /opt/ai-biz-recovery nano /opt/ai-biz-recovery/restore-drhall-remote.sh ``` Validated Drill #2 worker: ```bash #!/usr/bin/env bash set -Eeuo pipefail umask 077 SNAPSHOT_ID="${1:-latest}" CLIENT="drhall" N8N_VERSION="2.38.5" RESTORE_ETC="/etc/restore" RESTORE_ROOT="/var/restore/${CLIENT}" NETWORK="drhall_restore_net" PG_VOLUME="drhall_restore_pg_data" N8N_VOLUME="drhall_restore_n8n_data" PG_CONTAINER="drhall-restore-postgres" N8N_CONTAINER="drhall-restore-n8n" START_EPOCH="$(date +%s)" log() { printf '[%s] %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" } elapsed() { local now now="$(date +%s)" echo "$((now - START_EPOCH))" } fail() { log "ERROR: $*" exit 1 } HOSTNAME_SHORT="$(hostname -s)" # Safety protection: never allow the restore worker to run # against the production Dr. Hall server or Control. if [[ "$HOSTNAME_SHORT" == "drhall" || "$HOSTNAME_SHORT" == "control" ]]; then fail "Refusing to run on protected host: ${HOSTNAME_SHORT}" fi # Require the scratch-server marker created by Control. if [[ ! -f "${RESTORE_ETC}/.ai-biz-scratch-target" ]] || \ [[ "$(cat "${RESTORE_ETC}/.ai-biz-scratch-target")" != "$CLIENT" ]]; then fail "Scratch-target marker is missing or invalid." fi # Required recovery material. for f in \ "${RESTORE_ETC}/drhall-b2.env" \ "${RESTORE_ETC}/restic-password" \ "${RESTORE_ETC}/n8n-encryption-key" do [[ -s "$f" ]] || fail "Required recovery file missing: $f" done # Coolify validation should already have installed Docker. command -v docker >/dev/null 2>&1 \ || fail "Docker is not installed. Validate/register the server in Coolify first." docker info >/dev/null 2>&1 \ || fail "Docker is not available." # Install only missing recovery utilities. MISSING_PKGS=() command -v restic >/dev/null 2>&1 || MISSING_PKGS+=(restic) command -v curl >/dev/null 2>&1 || MISSING_PKGS+=(curl) command -v python3 >/dev/null 2>&1 || MISSING_PKGS+=(python3) command -v openssl >/dev/null 2>&1 || MISSING_PKGS+=(openssl) if (( ${#MISSING_PKGS[@]} > 0 )); then log "Installing required packages: ${MISSING_PKGS[*]}" apt-get update -qq DEBIAN_FRONTEND=noninteractive \ apt-get install -y -qq "${MISSING_PKGS[@]}" fi chmod 600 \ "${RESTORE_ETC}/drhall-b2.env" \ "${RESTORE_ETC}/restic-password" \ "${RESTORE_ETC}/n8n-encryption-key" # The Control copy points to Control's password path. # Change it to the temporary restore-server path. sed -i \ 's#^RESTIC_PASSWORD_FILE=.*#RESTIC_PASSWORD_FILE=/etc/restore/restic-password#' \ "${RESTORE_ETC}/drhall-b2.env" set -a source "${RESTORE_ETC}/drhall-b2.env" set +a : "${RESTIC_REPOSITORY:?RESTIC_REPOSITORY is not configured}" : "${RESTIC_PASSWORD_FILE:?RESTIC_PASSWORD_FILE is not configured}" log "Checking backup repository access." restic snapshots >/dev/null # Start clean if the helper is re-run after a failed attempt. log "Resetting scratch restore resources." docker rm -f \ "$N8N_CONTAINER" \ "$PG_CONTAINER" \ >/dev/null 2>&1 || true docker volume rm \ "$N8N_VOLUME" \ "$PG_VOLUME" \ >/dev/null 2>&1 || true docker network rm \ "$NETWORK" \ >/dev/null 2>&1 || true rm -rf "$RESTORE_ROOT" install -d -m 700 "$RESTORE_ROOT" # --------------------------------------------------------- # Restore latest/selected Restic snapshot # --------------------------------------------------------- RESTORE_START="$(date +%s)" log "Restoring snapshot: ${SNAPSHOT_ID}" restic restore \ "$SNAPSHOT_ID" \ --target "$RESTORE_ROOT" RESTORE_END="$(date +%s)" DUMP_FILE="$( find "$RESTORE_ROOT" \ -type f \ -name 'n8n-postgres.dump' \ -print -quit )" [[ -n "$DUMP_FILE" && -s "$DUMP_FILE" ]] \ || fail "PostgreSQL dump was not restored." BACKUP_DIR="$(dirname "$DUMP_FILE")" N8N_ARCHIVE="${BACKUP_DIR}/n8n-data.tar.gz" [[ -s "$N8N_ARCHIVE" ]] \ || fail "n8n data archive was not restored." tar -tzf "$N8N_ARCHIVE" >/dev/null \ || fail "n8n data archive validation failed." # --------------------------------------------------------- # Create isolated Docker environment # --------------------------------------------------------- log "Creating isolated Docker resources." docker network create "$NETWORK" >/dev/null docker volume create "$PG_VOLUME" >/dev/null docker volume create "$N8N_VOLUME" >/dev/null # --------------------------------------------------------- # PostgreSQL # --------------------------------------------------------- PG_PASSWORD="$(openssl rand -hex 32)" cat > "${RESTORE_ETC}/postgres.env" </dev/null docker run -d \ --name "$PG_CONTAINER" \ --restart unless-stopped \ --network "$NETWORK" \ --env-file "${RESTORE_ETC}/postgres.env" \ -v "${PG_VOLUME}:/var/lib/postgresql/data" \ postgres:16-alpine \ >/dev/null for _ in $(seq 1 60); do if docker exec "$PG_CONTAINER" \ pg_isready -U n8n -d n8n \ >/dev/null 2>&1 then break fi sleep 1 done docker exec "$PG_CONTAINER" \ pg_isready -U n8n -d n8n \ >/dev/null 2>&1 \ || fail "PostgreSQL did not become ready." DB_START="$(date +%s)" log "Restoring PostgreSQL database." docker cp \ "$DUMP_FILE" \ "$PG_CONTAINER:/tmp/n8n-postgres.dump" docker exec "$PG_CONTAINER" \ pg_restore \ --exit-on-error \ --clean \ --if-exists \ --no-owner \ --no-privileges \ -U n8n \ -d n8n \ /tmp/n8n-postgres.dump # Safety requirement: no restored workflow may automatically run. log "Disabling all restored workflows before n8n startup." docker exec "$PG_CONTAINER" \ psql \ -v ON_ERROR_STOP=1 \ -U n8n \ -d n8n \ -c \ 'UPDATE workflow_entity SET active = false WHERE active = true;' \ >/dev/null ACTIVE_COUNT="$( docker exec "$PG_CONTAINER" \ psql \ -At \ -U n8n \ -d n8n \ -c \ 'SELECT COUNT(*) FROM workflow_entity WHERE active = true;' )" [[ "$ACTIVE_COUNT" == "0" ]] \ || fail "One or more restored workflows remain active." DB_END="$(date +%s)" # --------------------------------------------------------- # n8n persistent data # --------------------------------------------------------- DATA_START="$(date +%s)" log "Restoring n8n data volume." docker run --rm \ -v "${N8N_VOLUME}:/target" \ -v "${BACKUP_DIR}:/backup:ro" \ alpine:3.20 \ sh -c \ 'cd /target && tar -xzf /backup/n8n-data.tar.gz' N8N_MOUNTPOINT="$( docker volume inspect \ -f '{{.Mountpoint}}' \ "$N8N_VOLUME" )" [[ -f "${N8N_MOUNTPOINT}/config" ]] \ || fail "Restored n8n config file is missing." # --------------------------------------------------------- # Validate recovered encryption key # --------------------------------------------------------- log "Verifying recovered n8n encryption key." python3 \ - \ "${N8N_MOUNTPOINT}/config" \ "${RESTORE_ETC}/n8n-encryption-key" \ <<'PY' import json import sys config_path, key_path = sys.argv[1], sys.argv[2] with open(config_path, encoding="utf-8") as f: config = json.load(f) with open(key_path, encoding="utf-8") as f: escrow_key = f.read().strip() if not escrow_key: raise SystemExit("Recovered encryption key is empty") if config.get("encryptionKey") != escrow_key: raise SystemExit( "Escrow key does not match restored n8n config" ) print("Encryption-key verification: PASS") PY DATA_END="$(date +%s)" # --------------------------------------------------------- # Build temporary restore n8n environment # --------------------------------------------------------- set -a source "${RESTORE_ETC}/postgres.env" set +a N8N_KEY="$(cat "${RESTORE_ETC}/n8n-encryption-key")" cat > "${RESTORE_ETC}/n8n.env" </dev/null docker run -d \ --name "$N8N_CONTAINER" \ --restart no \ --memory 1200m \ --network "$NETWORK" \ --env-file "${RESTORE_ETC}/n8n.env" \ -v "${N8N_VOLUME}:/home/node/.n8n" \ -p 127.0.0.1:5678:5678 \ "docker.n8n.io/n8nio/n8n:${N8N_VERSION}" \ >/dev/null HEALTH_OK=0 for _ in $(seq 1 90); do if curl -fsS \ http://127.0.0.1:5678/healthz \ >/dev/null 2>&1 then HEALTH_OK=1 break fi sleep 1 done if [[ "$HEALTH_OK" != "1" ]]; then docker logs \ --tail 100 \ "$N8N_CONTAINER" \ >&2 || true fail "n8n health check did not pass." fi N8N_END="$(date +%s)" # --------------------------------------------------------- # Recovery summary # --------------------------------------------------------- USER_COUNT="$( docker exec "$PG_CONTAINER" \ psql \ -At \ -U n8n \ -d n8n \ -c \ 'SELECT COUNT(*) FROM "user";' )" CREDENTIAL_COUNT="$( docker exec "$PG_CONTAINER" \ psql \ -At \ -U n8n \ -d n8n \ -c \ 'SELECT COUNT(*) FROM credentials_entity;' )" WORKFLOW_COUNT="$( docker exec "$PG_CONTAINER" \ psql \ -At \ -U n8n \ -d n8n \ -c \ 'SELECT COUNT(*) FROM workflow_entity;' )" log "RESTORE AUTOMATION: PASS" printf '\nRecovery summary\n' printf ' Snapshot restore: %ss\n' \ "$((RESTORE_END - RESTORE_START))" printf ' PostgreSQL restore: %ss\n' \ "$((DB_END - DB_START))" printf ' n8n data + key verify: %ss\n' \ "$((DATA_END - DATA_START))" printf ' n8n startup + health: %ss\n' \ "$((N8N_END - N8N_START))" printf ' Remote helper total: %ss\n' \ "$(elapsed)" printf ' Users restored: %s\n' \ "$USER_COUNT" printf ' Credentials restored: %s\n' \ "$CREDENTIAL_COUNT" printf ' Workflows restored: %s\n' \ "$WORKFLOW_COUNT" printf ' Active workflows: %s\n' \ "$ACTIVE_COUNT" printf ' n8n health: PASS\n' printf ' n8n local endpoint: http://127.0.0.1:5678\n' ``` Protect and syntax-check: ```bash chmod 700 /opt/ai-biz-recovery/restore-drhall-remote.sh bash -n /opt/ai-biz-recovery/restore-drhall-remote.sh ``` Expected from `bash -n`: ```text No output ``` ### 25.2 Control-Side Restore Orchestrator Create: ```bash nano /usr/local/sbin/ai-biz-restore-drhall ``` Validated Drill #2 orchestrator: ```bash #!/usr/bin/env bash set -Eeuo pipefail umask 077 RESTORE_IP="${1:-}" SNAPSHOT_ID="${2:-latest}" SSH_KEY="${SSH_KEY:-/root/.ssh/drhall-restore-test-2}" SSH_USER="root" WORKER="/opt/ai-biz-recovery/restore-drhall-remote.sh" RESTIC_ENV="/etc/ai-biz/restic/drhall-b2.env" RESTIC_PASSWORD="/etc/ai-biz/restic/drhall-restic-password" ESCROW_FILE="/etc/ai-biz/escrow/drhall-n8n-encryption-key.downloaded.enc.json" AGE_KEY="/etc/ai-biz/age/keys.txt" usage() { echo \ "Usage: ai-biz-restore-drhall [SNAPSHOT_ID|latest]" exit 2 } [[ -n "$RESTORE_IP" ]] || usage [[ -s "$SSH_KEY" ]] || { echo "ERROR: SSH key not found: $SSH_KEY" >&2 exit 1 } for f in \ "$WORKER" \ "$RESTIC_ENV" \ "$RESTIC_PASSWORD" \ "$ESCROW_FILE" \ "$AGE_KEY" do [[ -s "$f" ]] || { echo "ERROR: Required file missing: $f" >&2 exit 1 } done SSH_OPTS=( -i "$SSH_KEY" -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new ) SCP_OPTS=( -i "$SSH_KEY" -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new ) REMOTE="${SSH_USER}@${RESTORE_IP}" printf 'Checking restore target...\n' REMOTE_HOST="$( ssh \ "${SSH_OPTS[@]}" \ "$REMOTE" \ 'hostname -s' )" printf 'Restore host: %s\n' "$REMOTE_HOST" # Never allow the orchestrator to target production or Control. if [[ "$REMOTE_HOST" == "drhall" || \ "$REMOTE_HOST" == "control" ]] then echo \ "ERROR: Refusing to run against protected host: $REMOTE_HOST" \ >&2 exit 1 fi # Restore-test hosts must follow this naming standard. if [[ ! "$REMOTE_HOST" =~ ^drhall-restore ]]; then echo \ "ERROR: Restore host must begin with drhall-restore. Found: $REMOTE_HOST" \ >&2 exit 1 fi # Create protected restore directories and safety marker. ssh \ "${SSH_OPTS[@]}" \ "$REMOTE" \ 'install -d -m 700 /etc/restore /opt/ai-biz-recovery && printf "%s\n" drhall > /etc/restore/.ai-biz-scratch-target && chmod 600 /etc/restore/.ai-biz-scratch-target' printf 'Transferring recovery files...\n' scp \ "${SCP_OPTS[@]}" \ "$RESTIC_PASSWORD" \ "$REMOTE:/etc/restore/restic-password" scp \ "${SCP_OPTS[@]}" \ "$RESTIC_ENV" \ "$REMOTE:/etc/restore/drhall-b2.env" scp \ "${SCP_OPTS[@]}" \ "$WORKER" \ "$REMOTE:/opt/ai-biz-recovery/restore-drhall-remote.sh" ssh \ "${SSH_OPTS[@]}" \ "$REMOTE" \ 'chmod 600 /etc/restore/restic-password /etc/restore/drhall-b2.env && chmod 700 /opt/ai-biz-recovery/restore-drhall-remote.sh' printf 'Recovering n8n encryption key from escrow...\n' SOPS_AGE_KEY_FILE="$AGE_KEY" \ sops decrypt "$ESCROW_FILE" \ | python3 -c ' import json import sys data = json.load(sys.stdin) key = data.get("N8N_ENCRYPTION_KEY", "") if not key: raise SystemExit( "N8N_ENCRYPTION_KEY missing from escrow" ) sys.stdout.write(key) ' \ | ssh \ "${SSH_OPTS[@]}" \ "$REMOTE" \ 'umask 077 cat > /etc/restore/n8n-encryption-key chmod 600 /etc/restore/n8n-encryption-key' printf 'Starting optimized restore...\n' ssh \ -t \ "${SSH_OPTS[@]}" \ "$REMOTE" \ "/opt/ai-biz-recovery/restore-drhall-remote.sh '$SNAPSHOT_ID'" ``` Protect and syntax-check: ```bash chmod 700 /usr/local/sbin/ai-biz-restore-drhall bash -n /usr/local/sbin/ai-biz-restore-drhall ``` Expected: ```text No output ``` Safe usage test without a target IP: ```bash /usr/local/sbin/ai-biz-restore-drhall ``` Expected: ```text Usage: ai-biz-restore-drhall [SNAPSHOT_ID|latest] ``` ### 25.3 Script Safety Checks Before relying on the helper, verify these protections are still present: ```bash grep -nE 'protected host|scratch-target|restic restore|pg_restore|active = false|Encryption-key verification|127\.0\.0\.1:5678|RESTORE AUTOMATION' /opt/ai-biz-recovery/restore-drhall-remote.sh grep -nE 'protected host|drhall-restore|scratch-target|Transferring recovery files|Recovering n8n encryption key|Starting optimized restore' /usr/local/sbin/ai-biz-restore-drhall ``` The orchestrator intentionally includes: ```text StrictHostKeyChecking=accept-new ``` inside both its SSH and SCP option arrays. This was validated during Drill #2 and avoids failing on the first connection to a newly created scratch host while still refusing a changed host key for an already-known host. --- ## 26. Procedure B — Optimized Recovery Used in Restore Drill #2 This is the step-by-step procedure that produced the validated **36 minute 31 second** end-to-end result. ### 26.1 Preflight on Control Verify the permanent recovery files: ```bash for f in /etc/ai-biz/age/keys.txt /etc/ai-biz/age/recipient.txt /etc/ai-biz/restic/drhall-restic-password /etc/ai-biz/restic/drhall-b2.env do if [ -s "$f" ]; then echo "PASS: $f" else echo "MISSING: $f" fi done test -d /etc/ai-biz/escrow && echo "PASS: escrow directory present" || echo "MISSING: escrow directory" ``` Load the repository environment: ```bash set -a source /etc/ai-biz/restic/drhall-b2.env set +a restic snapshots restic check ``` Verify the intended snapshot contains the two required backup artifacts: ```bash restic ls | grep -E 'n8n-postgres\.dump|n8n-data\.tar\.gz' ``` Verify permissions: ```bash stat -c '%a %n' /etc/ai-biz/escrow/drhall-n8n-encryption-key.downloaded.enc.json /etc/ai-biz/restic/drhall-restic-password /etc/ai-biz/restic/drhall-b2.env /etc/ai-biz/age/keys.txt ``` Required: ```text 600 ``` Verify escrow decryption without printing the secret: ```bash SOPS_AGE_KEY_FILE=/etc/ai-biz/age/keys.txt sops decrypt /etc/ai-biz/escrow/drhall-n8n-encryption-key.downloaded.enc.json | python3 -c ' import json, sys data = json.load(sys.stdin) print( "PASS: N8N encryption key recoverable" if data.get("N8N_ENCRYPTION_KEY") else "FAIL: N8N encryption key unavailable" ) ' ``` Verify tools: ```bash for cmd in ssh scp restic sops python3 openssl; do command -v "$cmd" >/dev/null && echo "PASS: $cmd" || echo "MISSING: $cmd" done ``` ### 26.2 Start the End-to-End Restore Timer Immediately before creating the fresh DigitalOcean droplet: ```bash date -u '+RESTORE DRILL START: %Y-%m-%dT%H:%M:%SZ' ``` The validated measurement definition is: ```text START = immediately before fresh restore droplet creation STOP = n8n /healthz PASS + existing owner login PASS + safe workflow execution PASS ``` Cleanup is measured separately. ### 26.3 Create and Validate the Scratch Droplet Create: ```text Name: drhall-restore-test- OS: Ubuntu 24.04 LTS Size: 2 GB / 1 vCPU Project: AI Biz SSH key: Coolify provisioning key Firewall: ai-biz-client-firewall ``` Do not create a production DNS record or public `5678`/`5432` rule. Register the scratch server in Coolify and validate it so Docker is available. ### 26.4 Create Temporary Control-to-Restore SSH Access On Control: ```bash ssh-keygen -t ed25519 -f /root/.ssh/drhall-restore-test- -N "" -C "temporary-drhall-restore-test-" ``` Add only the public key to the scratch server. On the first connection: ```bash ssh-keygen -F ``` If a stale entry exists and the IP has been verified as the new scratch droplet: ```bash ssh-keygen -R ``` Then: ```bash ssh -i /root/.ssh/drhall-restore-test- -o BatchMode=yes -o StrictHostKeyChecking=accept-new root@ hostname ``` Verify Docker: ```bash ssh -i /root/.ssh/drhall-restore-test- -o BatchMode=yes root@ 'hostname && docker --version' ``` ### 26.5 Run the Optimized Restore Helper The orchestrator defaults to the Drill #2 key name. For another drill number, either create the expected key name or override `SSH_KEY` explicitly. Example with an explicit key: ```bash umask 077 LOG="/root/drhall-restore-test--$(date -u '+%Y%m%dT%H%M%SZ').log" set -o pipefail SSH_KEY="/root/.ssh/drhall-restore-test-" /usr/local/sbin/ai-biz-restore-drhall latest 2>&1 | tee "$LOG" RC=${PIPESTATUS[0]} echo echo "Restore helper exit code: $RC" echo "Restore log: $LOG" ``` Required result: ```text RESTORE AUTOMATION: PASS Restore helper exit code: 0 Active workflows: 0 n8n health: PASS ``` Do not stop the end-to-end timer yet. ### 26.6 Create the Control-to-Restore Tunnel On Control: ```bash ssh -M -S /run/drhall-restore-test--tunnel.sock -fN -i /root/.ssh/drhall-restore-test- -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -L 127.0.0.1:15679:127.0.0.1:5678 root@ ``` Verify: ```bash curl -fsS http://127.0.0.1:15679/healthz ``` Expected: ```json {"status":"ok"} ``` ### 26.7 Create Temporary Windows-to-Control Access if Needed If the previous drill's temporary Windows key was removed during cleanup, create a new one. On Windows PowerShell: ```powershell ssh-keygen -t ed25519 ` -f "$env:USERPROFILE\.ssh\drhall-restore-test--control" ` -N '""' ` -C "temporary-drhall-restore-test--control" ``` Display only the `.pub` file and append that public key to Control's `/root/.ssh/authorized_keys`. Test from Windows to the **Control public IP**: ```powershell ssh ` -i "$env:USERPROFILE\.ssh\drhall-restore-test--control" ` -o BatchMode=yes ` -o StrictHostKeyChecking=accept-new ` root@ ` hostname ``` Expected: ```text control ``` A direct Windows → restore-server SSH test may time out by design because SSH to the scratch server is restricted to Control. ### 26.8 Create the Windows-to-Control Tunnel From Windows PowerShell: ```powershell ssh -N -T ` -i "$env:USERPROFILE\.ssh\drhall-restore-test--control" ` -o ExitOnForwardFailure=yes ` -o ServerAliveInterval=30 ` -L 5679:127.0.0.1:15679 ` root@ ``` Leave the PowerShell session open. Browse to: ```text http://localhost:5679 ``` ### 26.9 Validate Owner Login and Safe Workflow Log in with the existing Dr. Hall owner credentials. Create: ```text RESTORE TEST - SAFE ``` with: ```text Manual Trigger ↓ Edit Fields ↓ restore_test = success ``` Execute manually. Required: ```text Existing owner login: PASS restore_test = success Workflow active: false ``` ### 26.10 Stop and Calculate the Timer Immediately after the safe workflow passes: ```bash END="$(date -u '+%Y-%m-%dT%H:%M:%SZ')" echo "RESTORE DRILL COMPLETE: $END" ``` Calculate from the recorded start time and compare against: ```text ≤ 60 minutes ``` Drill #2 result: ```text Start: 2026-09-22T15:36:14Z Complete: 2026-09-22T16:12:45Z Duration: 36 minutes 31 seconds SLO: PASS Margin: 23 minutes 29 seconds under target ``` The helper itself completed in: ```text 2 minutes 21 seconds ``` The remaining end-to-end time included scratch infrastructure creation, Coolify validation, SSH access preparation, workstation access, owner login, and safe workflow validation. --- ## 27. Cleanup After the Optimized Restore Drill #2 This cleanup procedure belongs specifically to **Restore Drill #2** and should be followed after the optimized restore timer has stopped. Unlike Drill #1, Drill #2 used: - the permanent Control-side optimized recovery helpers; - a temporary copy of the remote worker on the scratch server; - a per-drill Control → restore SSH key; - `StrictHostKeyChecking=accept-new`; - a managed SSH control socket for the Control → restore tunnel; - a fresh temporary Windows → Control SSH key because the Drill #1 workstation key had already been removed; - a restore execution log on Control. The Drill #2 cleanup therefore has additional steps. Do not replace it with the Drill #1 cleanup. ### 27.1 Preserve the Drill #2 Restore Log on Control Retain the optimized restore log long enough to document the drill result and troubleshoot any later questions: ```text /root/drhall-restore-test--.log ``` For the validated Drill #2 execution, the log recorded the helper stages and the `RESTORE AUTOMATION: PASS` result. This file is on **Control**, not on the scratch server. ### 27.2 Remove the Scratch Restore Containers, Volumes, Network, Secrets, and Worker Copy From Control, connect to the scratch server using the per-drill restore key: ```bash ssh \ -i /root/.ssh/drhall-restore-test- \ root@ \ ' docker rm -f drhall-restore-n8n drhall-restore-postgres \ >/dev/null 2>&1 || true docker volume rm \ drhall_restore_n8n_data \ drhall_restore_pg_data \ >/dev/null 2>&1 || true docker network rm drhall_restore_net \ >/dev/null 2>&1 || true rm -rf /etc/restore rm -rf /var/restore/drhall rm -rf /opt/ai-biz-recovery echo "Scratch restore data cleanup: PASS" ' ``` The command above removes only the **scratch-server copy** of: ```text /opt/ai-biz-recovery ``` Do **not** remove the permanent Control-side helpers: ```text /opt/ai-biz-recovery/restore-drhall-remote.sh /usr/local/sbin/ai-biz-restore-drhall ``` Those are validated recovery assets and must remain on Control. ### 27.3 Close the Managed Control-to-Restore Tunnel Drill #2 created the Control → restore tunnel with an SSH control socket. Close it from Control: ```bash ssh \ -S /run/drhall-restore-test--tunnel.sock \ -O exit \ root@ ``` Verify: ```bash ss -lntp | grep 15679 \ || echo "Control restore tunnel closed: PASS" ``` Expected: ```text Control restore tunnel closed: PASS ``` If the socket file remains after the SSH master connection exits, confirm no tunnel process is using it before removing the stale socket file. ### 27.4 Close the Windows Workstation-to-Control Tunnel In the Windows PowerShell session running: ```powershell ssh -N -T ... ``` press: ```text Ctrl+C ``` Verify: ```text http://localhost:5679 ``` no longer loads the restored workspace. ### 27.5 Remove the Temporary Windows Authorization from Control The Drill #2 Windows key comment follows: ```text temporary-drhall-restore-test--control ``` On Control, remove only that per-drill authorization: ```bash tmp="$(mktemp)" grep -v \ 'temporary-drhall-restore-test--control' \ /root/.ssh/authorized_keys \ > "$tmp" install -m 600 "$tmp" /root/.ssh/authorized_keys rm -f "$tmp" ``` Verify: ```bash grep \ 'temporary-drhall-restore-test--control' \ /root/.ssh/authorized_keys \ || echo "Temporary Windows authorization removed: PASS" ``` Expected: ```text Temporary Windows authorization removed: PASS ``` Do not remove unrelated permanent SSH keys. If a temporary backup such as: ```text /root/.ssh/authorized_keys.before-restore-drill ``` was created, verify the live `authorized_keys` file is correct and remove the safety copy later when it is no longer operationally needed. ### 27.6 Delete the Temporary Windows Key Pair On Windows PowerShell: ```powershell Remove-Item ` "$env:USERPROFILE\.ssh\drhall-restore-test--control" Remove-Item ` "$env:USERPROFILE\.ssh\drhall-restore-test--control.pub" Write-Host "Temporary Windows restore keys removed: PASS" ``` Verify they no longer exist: ```powershell Get-ChildItem ` "$env:USERPROFILE\.ssh\drhall-restore-test--control*" ``` The command should return no matching per-drill files. ### 27.7 Remove the Temporary Control-to-Restore Key Pair On Control: ```bash rm -f \ /root/.ssh/drhall-restore-test- \ /root/.ssh/drhall-restore-test-.pub echo "Temporary Control restore keys removed: PASS" ``` Do not remove the permanent Coolify provisioning key or unrelated agency SSH keys. ### 27.8 Remove the Scratch Server from Control `known_hosts` Because Drill #2 used: ```text StrictHostKeyChecking=accept-new ``` the scratch server's host key may now be stored in Control's `known_hosts`. After confirming the scratch target is the server being removed: ```bash ssh-keygen -R ``` This cleanup is specific to the disposable scratch host and prevents stale host-key entries from affecting a later restore drill if the IP is reused. Do not globally disable SSH host-key checking. ### 27.9 Remove the Scratch Server from Coolify In Coolify: ```text Servers → drhall-restore-test- → Danger → Delete Server ``` Delete only the scratch restore server. Confirm the production `drhall` server remains registered. ### 27.10 Destroy the DigitalOcean Scratch Droplet In DigitalOcean: ```text AI Biz → drhall-restore-test- → Destroy ``` Verify the scratch hostname and IP before confirming. Do not destroy the production `drhall` droplet. ### 27.11 Verify Production and Permanent Recovery Assets After destroying the scratch server: - verify the normal Dr. Hall production URL; - verify Uptime Kuma reports production healthy; - confirm production DNS remains unchanged; - confirm the nightly backup repository remains accessible; - confirm escrow data remains intact; - confirm the permanent optimized helpers remain on Control. Verify the helpers: ```bash test -x /opt/ai-biz-recovery/restore-drhall-remote.sh \ && echo "Remote restore worker retained: PASS" test -x /usr/local/sbin/ai-biz-restore-drhall \ && echo "Control restore orchestrator retained: PASS" ``` ### 27.12 Drill #2 Cleanup Completion Checklist Record: ```text Restore Drill #2 cleanup: Restore execution log on Control RETAINED Scratch restore containers REMOVED Scratch restore Docker volumes REMOVED Scratch restore Docker network REMOVED Scratch restore secrets/files REMOVED Scratch remote-worker copy REMOVED Control → restore tunnel CLOSED Windows → Control tunnel CLOSED Temporary Windows authorization REMOVED Temporary Windows key pair REMOVED Temporary Control → restore key pair REMOVED Scratch known_hosts entry REMOVED Coolify scratch server REMOVED DigitalOcean scratch droplet DELETED Production client instance UNTOUCHED Production DNS UNCHANGED Nightly backups RETAINED Encryption-key escrow RETAINED Restic recovery escrow RETAINED Permanent remote restore worker RETAINED Permanent Control restore orchestrator RETAINED ``` The Drill #2 cleanup is complete only when the temporary infrastructure and access paths are gone while the production environment, backup/escrow assets, restore log, and permanent Control-side recovery helpers remain intact. --- ## 28. Permanent Recovery Assets — Do Not Delete The following are permanent recovery assets on Control and/or Backblaze: ```text /etc/ai-biz/age/keys.txt /etc/ai-biz/age/recipient.txt /etc/ai-biz/restic/-restic-password /etc/ai-biz/restic/-b2.env /etc/ai-biz/escrow/ /opt/ai-biz-recovery/restore-drhall-remote.sh /usr/local/sbin/ai-biz-restore-drhall ``` Backblaze: ```text Backup repository: backups// N8N encryption-key escrow: /-n8n-encryption-key.enc.json Restic password escrow: /-restic-password.age ``` These are disaster-recovery assets and must not be removed during routine restore-drill cleanup. --- # Part VI — Operational Checklists and Pilot Validation ## 29. New Client Manual Provisioning Checklist ### Infrastructure - [ ] Client droplet created in `AI Biz` - [ ] 2 GB / 1 vCPU minimum - [ ] Ubuntu 24.04 LTS - [ ] Coolify provisioning SSH key assigned - [ ] `client:` tag applied - [ ] `managed:true` tag applied - [ ] `ai-biz-client-firewall` attached - [ ] SSH restricted to Control - [ ] Ports 5678 and 5432 not publicly exposed ### Coolify / n8n - [ ] Client server registered and validated - [ ] Reusable Compose template deployed - [ ] `N8N_VERSION` pinned - [ ] Unique `PG_PASSWORD` configured - [ ] Unique `N8N_ENCRYPTION_KEY` configured - [ ] Agency `N8N_LICENSE_ACTIVATION_KEY` configured when available - [ ] PostgreSQL healthy - [ ] n8n healthy ### DNS / TLS - [ ] Porkbun A record created - [ ] DNS resolves to client droplet - [ ] Domain assigned to n8n in Coolify - [ ] HTTP redirects to HTTPS - [ ] TLS certificate valid ### Client Workspace - [ ] Owner account created with client email - [ ] Login verified - [ ] Community Edition registration verified ### Security / OS Maintenance - [ ] DigitalOcean monitoring agent enabled and active - [ ] CPU alert configured at 80% threshold - [ ] RAM alert configured at 80% threshold - [ ] Disk alert configured at 80% threshold - [ ] `fail2ban` installed, enabled, and active - [ ] `sshd` jail active in `fail2ban` - [ ] Effective SSH configuration shows `PasswordAuthentication no` - [ ] Effective SSH configuration shows `KbdInteractiveAuthentication no` - [ ] Public-key SSH authentication verified - [ ] Controlled password-only SSH attempt fails - [ ] `unattended-upgrades` installed - [ ] `apt-daily.timer` and `apt-daily-upgrade.timer` enabled - [ ] Recent unattended-upgrade execution confirmed in logs ### Monitoring - [ ] Uptime Kuma HTTPS monitor added - [ ] 60-second interval - [ ] HTTPS valid - [ ] Monitor stable `UP` - [ ] Agency email notifier assigned - [ ] Test notification delivered to approved agency inbox - [ ] Certificate Expiry Notification enabled - [ ] Certificate-expiry reminders configured for 7, 14, and 21 days - [ ] Backup Freshness Push monitor created - [ ] Backup freshness threshold set to 26 hours - [ ] Backup checker timer enabled and active - [ ] Healthy backup state tested (`UP`) - [ ] Controlled stale-backup state tested (`DOWN`) - [ ] DOWN email notification delivered - [ ] Recovery `UP` notification delivered ### Backup / Recovery - [ ] Dedicated Restic password generated - [ ] Restricted Backblaze backup key created - [ ] Restic repository initialized - [ ] Manual backup verified - [ ] Nightly backup timer active - [ ] Retention verified - [ ] Weekly prune timer active - [ ] Restic password escrowed - [ ] n8n encryption key escrowed - [ ] Control can recover both recovery secrets - [ ] `restic check` passes - [ ] Recovery helper scripts pass `bash -n` - [ ] First-connection host-key handling validated with `StrictHostKeyChecking=accept-new` - [ ] Temporary Windows → Control key procedure validated when no workstation key exists - [ ] Two-stage SSH tunnel validated - [ ] Existing owner login validated on restored workspace - [ ] Safe workflow execution validated - [ ] Restore duration recorded against the ≤60-minute SLO - [ ] Quarterly restore drill tracked as a recurring ClickUp task - [ ] Quarterly drill result logged and scratch environment cleaned up --- ## 30. Dr. Hall Pilot Validation Record ### 30.1 Provisioning Validated during the first AI Biz manual implementation: ```text Dedicated client droplet PASS Required tags PASS DigitalOcean firewall PASS Coolify registration PASS n8n + PostgreSQL deployment PASS Pinned n8n version PASS Porkbun DNS PASS HTTPS / Coolify proxy PASS Owner account PASS Community Edition registration PASS Uptime Kuma HTTPS monitoring PASS ``` ### 30.2 Backup and Escrow ```text PostgreSQL pg_dump PASS n8n_data archive PASS Encrypted Restic/B2 backup PASS 30-day daily retention CONFIGURED 12-month monthly retention CONFIGURED Nightly systemd backup PASS Weekly Restic prune PASS Repository integrity PASS Restic password recovery PASS N8N encryption-key escrow PASS Independent Control decryption PASS ``` ### 30.3 Initial Restore Drill The first full Dr. Hall restore drill was performed on **September 17, 2026 UTC**. ```text Restore Test Started: Thu Sep 17 15:19:11 UTC 2026 Restore Test Completed: Thu Sep 17 16:47:51 UTC 2026 Total Duration: 1 hour 28 minutes 40 seconds ``` Results: ```text Fresh DigitalOcean droplet PASS Coolify scratch-server registration PASS Backblaze repository access PASS Restic password recovery PASS Repository integrity PASS Latest nightly snapshot restore PASS PostgreSQL 16 restore PASS n8n_data restore PASS N8N_ENCRYPTION_KEY recovery PASS Escrow/config key verification PASS n8n 2.38.5 startup PASS Local health check PASS Owner workspace access PASS Stored credential decryption N/A — no stored credentials Safe workflow execution PASS RESTORE TEST - SAFE remained inactive PASS Public restore exposure NONE Production Dr. Hall environment UNTOUCHED Functional Restore Result PASS Restore SLO ≤ 60 minutes Restore SLO Result NOT MET ``` The first manual restore exceeded the target by **28 minutes 40 seconds** because the procedure was being set up, tested, verified, troubleshot, and documented step by step during the initial validation. ### 30.4 Optimized Restore Drill #2 The second full Dr. Hall restore drill was performed on **September 22, 2026 UTC** using the optimized recovery helper and the same functional completion criteria as Drill #1. ```text Restore Drill #2 Started: 2026-09-22T15:36:14Z Restore Drill #2 Completed: 2026-09-22T16:12:45Z Total Duration: 36 minutes 31 seconds Restore SLO: ≤ 60 minutes Restore SLO Result: PASS Margin Under SLO: 23 minutes 29 seconds ``` Validated results: ```text Fresh DigitalOcean droplet PASS Coolify scratch-server validation PASS Temporary Control → restore SSH PASS StrictHostKeyChecking=accept-new VALIDATED Latest nightly snapshot restore PASS PostgreSQL 16 restore PASS n8n_data restore PASS N8N_ENCRYPTION_KEY recovery PASS Escrow/config key verification PASS n8n 2.38.5 startup PASS Local /healthz PASS Two-stage SSH tunnel PASS Temporary Windows → Control key PASS Existing owner login PASS Stored credential decryption N/A — no stored credentials Safe workflow execution PASS RESTORE TEST - SAFE remained inactive PASS Public restore exposure NONE Production Dr. Hall environment UNTOUCHED Functional Restore Result PASS Restore SLO Result PASS ``` Optimized helper timing: ```text Snapshot restore: 3 seconds PostgreSQL restore: 3 seconds n8n data + key verification: 1 second n8n startup + health: 97 seconds Remote helper total: 141 seconds (2 minutes 21 seconds) ``` Comparison: ```text Drill #1: 1 hour 28 minutes 40 seconds Drill #2: 36 minutes 31 seconds Improvement: 52 minutes 09 seconds Reduction: approximately 59% ``` Drill #2 also captured two operational details that must remain in the recovery procedure: 1. The first Control → scratch SSH connection may require `StrictHostKeyChecking=accept-new`. 2. If the previous temporary workstation key was removed during cleanup, create a fresh temporary Windows → Control key for the new drill. The Windows workstation connects to **Control**, not directly to the scratch restore server. The **bring-the-restore-below-60-minutes** task is therefore validated as complete for the Dr. Hall pilot recovery process. --- ## 31. Post-Pilot Operational Verification — Dr. Hall This section records the step-by-step operational verification performed after the original Dr. Hall manual pilot. The checks were completed across **September 29–30, 2026**. Server timestamps shown in command output are UTC unless otherwise noted. The purpose of this verification phase was to close the client-server security, monitoring, maintenance, notification, and backup-monitoring items before beginning provisioning automation or any new access-service work. ### 31.1 Verification Summary The following original post-pilot items were verified: ```text DigitalOcean monitoring agent PASS DigitalOcean CPU 80% alert policy CONFIGURED DigitalOcean RAM 80% alert policy CONFIGURED DigitalOcean Disk 80% alert policy CONFIGURED fail2ban PASS fail2ban sshd jail PASS Password-based SSH authentication DISABLED / PASS Public-key SSH authentication ENABLED / PASS Unattended Ubuntu security upgrades PASS Uptime Kuma agency email delivery PASS TLS certificate-expiry notification PASS Nightly Restic backup execution PASS Weekly Restic prune PASS Control → B2 repository access PASS 26-hour backup-freshness monitoring PASS Backup stale DOWN notification PASS Backup recovery UP notification PASS Backup freshness systemd scheduling PASS Quarterly restore-drill tracking method CLICKUP / APPROVED ``` Two limitations must remain explicit: 1. The DigitalOcean CPU/RAM/disk **policies were verified as configured**, but production was not deliberately stressed to cross the 80% thresholds solely to prove delivery of a DigitalOcean resource-alert email. 2. Uptime Kuma **email** notification delivery was tested successfully. Slack delivery was not configured or tested during this verification. ### 31.2 DigitalOcean Monitoring Agent and Resource Alerts #### Step 1 — Confirm the DigitalOcean metrics agent on the client droplet On the Dr. Hall production droplet: ```bash hostname systemctl is-enabled do-agent systemctl is-active do-agent systemctl status do-agent --no-pager ``` Validated result: ```text Hostname: drhall do-agent enabled: enabled do-agent active: active Result: PASS ``` The DigitalOcean Insights page also showed current CPU, load, memory, disk, and bandwidth metrics for the `drhall` droplet. #### Step 2 — Confirm DigitalOcean alert policies In DigitalOcean, verify the Resource Alerts assigned to `drhall`: ```text AIBiz – Dr Hall – CPU 80% CPU Utilization Percent is above 80% for 5 min AIBiz – Dr Hall – RAM 80% Memory Utilization Percent is above 80% for 5 min AIBiz – Dr Hall – Disk 80% Disk Utilization Percent is above 80% for 5 min ``` Validated result: ```text CPU policy present and assigned to drhall PASS RAM policy present and assigned to drhall PASS Disk policy present and assigned to drhall PASS Agency email recipient configured PASS ``` Do not intentionally load-test or fill the production droplet merely to trigger these policies. A real delivery test can be recorded later if a safe non-production method is approved. ### 31.3 Verify `fail2ban` `fail2ban` was initially absent on the production droplet. Install it only if the package/service is not already present: ```bash apt update apt install -y fail2ban python3-systemd ``` Verify: ```bash fail2ban-client --version systemctl is-enabled fail2ban systemctl is-active fail2ban systemctl status fail2ban --no-pager fail2ban-client status fail2ban-client status sshd ``` Validated result: ```text Fail2Ban version: 1.0.2 Service enabled: yes Service active: yes Configured jails: 1 Active jail: sshd Result: PASS ``` At verification time there were no currently banned addresses. That is not an error; the important result is that the service and `sshd` jail are active and monitoring SSH authentication failures. ### 31.4 Verify Key-Only SSH Authentication #### Step 1 — Inspect the effective SSH configuration On the client droplet: ```bash sshd -T | grep -Ei \ '^(passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication|permitrootlogin) ' ``` Validated effective values: ```text permitrootlogin yes pubkeyauthentication yes passwordauthentication no kbdinteractiveauthentication no ``` #### Step 2 — Validate SSH configuration syntax ```bash sshd -t && echo "SSH configuration syntax: PASS" ``` Expected/validated result: ```text SSH configuration syntax: PASS ``` #### Step 3 — Locate the active configuration directives ```bash grep -RniE \ '^(PasswordAuthentication|KbdInteractiveAuthentication|PubkeyAuthentication|PermitRootLogin)' \ /etc/ssh/sshd_config /etc/ssh/sshd_config.d/ 2>/dev/null ``` The production configuration included `PasswordAuthentication no` in the cloud-init/Ubuntu SSH include files. #### Step 4 — Perform a controlled password-only negative test From Control, explicitly disable public-key authentication for the test connection: ```bash ssh \ -o BatchMode=yes \ -o PubkeyAuthentication=no \ -o PreferredAuthentications=password \ -o KbdInteractiveAuthentication=no \ -o ConnectTimeout=10 \ root@ true ``` Validated result: ```text Permission denied (publickey). ``` Result: **PASS** — password and keyboard-interactive authentication are disabled, while authorized public-key SSH remains available. `PermitRootLogin yes` was not changed during this verification because the validated management path still uses root with an authorized key. If this is later hardened to `prohibit-password`, validate Coolify and recovery access before applying the change. ### 31.5 Verify Unattended Security Updates On the client droplet: ```bash dpkg -l unattended-upgrades cat /etc/apt/apt.conf.d/20auto-upgrades systemctl is-enabled apt-daily.timer systemctl is-enabled apt-daily-upgrade.timer systemctl list-timers --all apt-daily.timer apt-daily-upgrade.timer tail -30 /var/log/unattended-upgrades/unattended-upgrades.log ``` Validated configuration: ```text APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Unattended-Upgrade "1"; ``` Validated result: ```text unattended-upgrades package installed PASS apt-daily.timer enabled PASS apt-daily-upgrade.timer enabled PASS Recent unattended upgrades in log PASS ``` Recent logs showed successful package upgrades on September 28 and September 29, 2026. ### 31.6 Configure and Verify Uptime Kuma Agency Email Delivery #### Step 1 — Use the agency Resend account The agency Resend account contains the verified domain: ```text kwagency.team ``` Create a dedicated sending key for Uptime Kuma. Store it only in the approved secret store. Do not place the real key in this runbook, screenshots, recordings, or shared chat. #### Step 2 — Configure the Uptime Kuma SMTP notifier Validated notifier configuration: ```text Friendly Name: AI Biz - Agency Email Alerts SMTP Host: smtp.resend.com Port: 2465 Security: TLS Username: resend Password: From: alerts@kwagency.team To: ``` The first test using port `465` returned a connection timeout from the DigitalOcean-hosted environment. The port was changed to Resend SMTP port `2465`, after which the Uptime Kuma test succeeded. #### Step 3 — Verify actual email delivery Click **Test** in Uptime Kuma and confirm receipt in the approved agency inbox. Validated result: ```text SMTP connection on 2465 PASS Uptime Kuma test action PASS Agency test email received PASS ``` Save the notifier and assign `AI Biz - Agency Email Alerts` to the `Dr. Hall - n8n` monitor. ### 31.7 Verify TLS Certificate-Expiry Notifications Open the `Dr. Hall - n8n` monitor and confirm: ```text Monitor Type: HTTP(s) URL: https://drhall.ownthestack.io Notification: AI Biz - Agency Email Alerts Certificate Expiry Notification: enabled ``` In **Settings → Notifications**, verify the TLS certificate-expiry reminder intervals: ```text 7 days 14 days 21 days ``` Validated result: ```text Agency notifier assigned PASS Certificate Expiry Notification enabled PASS 7/14/21-day reminder intervals PASS ``` ### 31.8 Re-Verify Nightly Backup and Weekly Prune #### Step 1 — Verify the systemd backup and prune timers On the Dr. Hall droplet: ```bash systemctl list-timers --all \ drhall-backup.timer \ drhall-prune.timer systemctl status \ drhall-backup.timer \ drhall-prune.timer \ --no-pager ``` Validated result: ```text drhall-backup.timer enabled / active (waiting) drhall-prune.timer enabled / active (waiting) ``` #### Step 2 — Verify a recent backup execution ```bash journalctl -u drhall-backup.service \ --since "3 days ago" --no-pager | tail -50 ``` The September 29 run showed: ```text PostgreSQL dump completed PASS n8n data archive completed PASS Encrypted upload to Backblaze B2 PASS Restic snapshot saved PASS Retention policy completed PASS Backup service completed successfully PASS ``` The September 29 snapshot was `16c76e6f`. By the later backup-freshness test on September 30, a newer snapshot `74e45991` was present. #### Step 3 — Verify the weekly prune ```bash journalctl -u drhall-prune.service \ --since "2026-09-27" --no-pager | tail -40 ``` Validated September 27 result: ```text Repository indexes loaded PASS Snapshots evaluated PASS Packs processed/repacked PASS Index rebuilt PASS Repository prune completed PASS ``` ### 31.9 Verify Control Can Independently Read the B2 Repository On Control, first verify that the recovery files exist without printing their contents: ```bash test -s /etc/ai-biz/restic/drhall-b2.env \ && echo "Backup environment: PRESENT" \ || echo "Backup environment: MISSING" test -s /etc/ai-biz/restic/drhall-restic-password \ && echo "Restic password: PRESENT" \ || echo "Restic password: MISSING" ``` Load the repository environment and verify snapshot access: ```bash set -a source /etc/ai-biz/restic/drhall-b2.env set +a restic snapshots --json --tag client:drhall \ | jq -r 'sort_by(.time) | last |\n "Snapshot: \(.id[0:8])\\nTime: \(.time)\\nHost: \(.hostname)"' ``` Validated result during initial verification: ```text Snapshot: 16c76e6f Time: 2026-09-29T06:17:35.629554858Z Host: drhall ``` Result: **PASS** — Control can independently read the Dr. Hall Restic repository in Backblaze B2. ### 31.10 Implement and Verify the 26-Hour Backup-Freshness Alert The Build Specification requires an alert if the last successful backup becomes more than 26 hours old. No existing backup-freshness systemd timer, cron task, or Uptime Kuma backup monitor was found on Control, so a dedicated monitor and checker were added. #### Step 1 — Confirm no existing checker conflicts On Control: ```bash systemctl list-timers --all \ | grep -Ei 'backup|restic|watch|monitor|alert' systemctl list-unit-files \ | grep -Ei 'backup|restic|watch|monitor|alert' sudo crontab -l sudo grep -RIlEi \ 'backup.stale|backup.age|restic snapshots|26.hour|26h' \ /etc/cron.d \ /etc/cron.daily \ /usr/local/sbin \ /opt/ai-biz-recovery \ 2>/dev/null ``` The only relevant match was the existing recovery helper. No dedicated backup-freshness monitor was present. #### Step 2 — Create the Uptime Kuma Push monitor Create: ```text Monitor Type: Push Friendly Name: Dr. Hall - Backup Freshness (26h) Heartbeat Interval: 3600 seconds Notification: AI Biz - Agency Email Alerts ``` Keep the Push URL/token private. If a Push URL or token is ever exposed in a screenshot, chat, ticket, or recording, rotate it before production use. If the installed Uptime Kuma version does not provide a token-reset option, delete and recreate only the affected Push monitor, then update the protected Control-side environment file with the new token. #### Step 3 — Store the Push endpoint securely on Control ```bash install -d -m 700 /etc/ai-biz/monitoring nano /etc/ai-biz/monitoring/drhall-backup-monitor.env ``` Store only the private local Push endpoint with the actual token: ```bash PUSH_URL='http://127.0.0.1:3001/api/push/' ``` Protect the file: ```bash chmod 600 /etc/ai-biz/monitoring/drhall-backup-monitor.env chown root:root /etc/ai-biz/monitoring/drhall-backup-monitor.env ``` Verify without exposing the token: ```bash grep -q '^PUSH_URL=' /etc/ai-biz/monitoring/drhall-backup-monitor.env \ && echo "Push URL: PRESENT" \ || echo "Push URL: MISSING" stat -c '%U:%G %a %n' \ /etc/ai-biz/monitoring/drhall-backup-monitor.env ``` Validated result: ```text Push URL: PRESENT root:root 600 /etc/ai-biz/monitoring/drhall-backup-monitor.env ``` #### Step 4 — Verify local Control → Uptime Kuma connectivity ```bash curl -sS -o /dev/null \ -w 'Local Kuma HTTP status: %{http_code}\\n' \ --connect-timeout 5 \ http://127.0.0.1:3001/ ``` Validated result: ```text Local Kuma HTTP status: 302 ``` The `302` response confirms that the local Uptime Kuma web service is reachable; the root path redirects normally. #### Step 5 — Create the backup-freshness checker Create: ```bash nano /usr/local/sbin/check-drhall-backup-freshness.sh ``` Validated script: ```bash #!/usr/bin/env bash set -uo pipefail PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CLIENT="drhall" THRESHOLD_HOURS="${THRESHOLD_HOURS:-26}" RESTIC_ENV="/etc/ai-biz/restic/drhall-b2.env" RESTIC_PASSWORD_FILE="/etc/ai-biz/restic/drhall-restic-password" MONITOR_ENV="/etc/ai-biz/monitoring/drhall-backup-monitor.env" log() { echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] $*" } push_status() { local status="$1" local message="$2" curl -fsS \ --max-time 15 \ --get "$PUSH_URL" \ --data-urlencode "status=${status}" \ --data-urlencode "msg=${message}" \ >/dev/null } for file in "$RESTIC_ENV" "$RESTIC_PASSWORD_FILE" "$MONITOR_ENV"; do if [[ ! -s "$file" ]]; then log "ERROR: Required file missing or empty: $file" exit 1 fi done set -a source "$RESTIC_ENV" source "$MONITOR_ENV" set +a export RESTIC_PASSWORD_FILE if [[ -z "${PUSH_URL:-}" ]]; then log "ERROR: PUSH_URL is not configured." exit 1 fi if ! SNAPSHOT_JSON="$( restic snapshots \ --json \ --tag "client:${CLIENT}" )"; then MESSAGE="Backup check failed: unable to read Dr. Hall Restic repository" log "DOWN: $MESSAGE" if ! push_status "down" "$MESSAGE"; then log "ERROR: Unable to send DOWN status to Uptime Kuma." fi exit 1 fi LATEST_TIME="$( printf '%s' "$SNAPSHOT_JSON" | jq -r 'sort_by(.time) | last | .time // empty' )" LATEST_ID="$( printf '%s' "$SNAPSHOT_JSON" | jq -r 'sort_by(.time) | last | .id[0:8] // empty' )" if [[ -z "$LATEST_TIME" || -z "$LATEST_ID" ]]; then MESSAGE="Backup check failed: no Dr. Hall snapshot found" log "DOWN: $MESSAGE" if ! push_status "down" "$MESSAGE"; then log "ERROR: Unable to send DOWN status to Uptime Kuma." fi exit 1 fi LATEST_EPOCH="$(date -d "$LATEST_TIME" +%s 2>/dev/null || true)" NOW_EPOCH="$(date -u +%s)" if [[ -z "$LATEST_EPOCH" ]]; then MESSAGE="Backup check failed: invalid snapshot timestamp" log "DOWN: $MESSAGE" if ! push_status "down" "$MESSAGE"; then log "ERROR: Unable to send DOWN status to Uptime Kuma." fi exit 1 fi AGE_SECONDS=$(( NOW_EPOCH - LATEST_EPOCH )) AGE_HOURS=$(( AGE_SECONDS / 3600 )) if (( AGE_SECONDS < 0 )); then MESSAGE="Backup check failed: latest snapshot timestamp is in the future" log "DOWN: $MESSAGE" if ! push_status "down" "$MESSAGE"; then log "ERROR: Unable to send DOWN status to Uptime Kuma." fi exit 1 fi if (( AGE_SECONDS > THRESHOLD_HOURS * 3600 )); then MESSAGE="Backup stale: latest snapshot ${LATEST_ID} is ${AGE_HOURS}h old (threshold ${THRESHOLD_HOURS}h)" log "DOWN: $MESSAGE" if ! push_status "down" "$MESSAGE"; then log "ERROR: Unable to send DOWN status to Uptime Kuma." fi exit 1 fi MESSAGE="Backup healthy: latest snapshot ${LATEST_ID} is ${AGE_HOURS}h old" log "UP: $MESSAGE" if ! push_status "up" "$MESSAGE"; then log "ERROR: Unable to send UP status to Uptime Kuma." exit 1 fi exit 0 ``` Set permissions and validate syntax: ```bash chown root:root /usr/local/sbin/check-drhall-backup-freshness.sh chmod 750 /usr/local/sbin/check-drhall-backup-freshness.sh bash -n /usr/local/sbin/check-drhall-backup-freshness.sh \ && echo "Backup checker syntax: PASS" \ || echo "Backup checker syntax: FAIL" stat -c '%U:%G %a %n' \ /usr/local/sbin/check-drhall-backup-freshness.sh ``` Validated result: ```text Backup checker syntax: PASS root:root 750 /usr/local/sbin/check-drhall-backup-freshness.sh ``` #### Step 6 — Perform the first real healthy check ```bash /usr/local/sbin/check-drhall-backup-freshness.sh echo "Exit code: $?" ``` Validated September 30 result: ```text UP: Backup healthy: latest snapshot 74e45991 is 7h old Exit code: 0 ``` Uptime Kuma changed from `Unknown` to `Up`. #### Step 7 — Create the systemd service and timer Create `/etc/systemd/system/drhall-backup-freshness.service`: ```ini [Unit] Description=Check Dr. Hall Backup Freshness After=network-online.target Wants=network-online.target [Service] Type=oneshot ExecStart=/usr/local/sbin/check-drhall-backup-freshness.sh User=root ``` Create `/etc/systemd/system/drhall-backup-freshness.timer`: ```ini [Unit] Description=Run Dr. Hall Backup Freshness Check Every 30 Minutes [Timer] OnBootSec=5min OnUnitActiveSec=30min Persistent=true Unit=drhall-backup-freshness.service [Install] WantedBy=timers.target ``` Validate and enable: ```bash systemctl daemon-reload systemd-analyze verify \ /etc/systemd/system/drhall-backup-freshness.service \ /etc/systemd/system/drhall-backup-freshness.timer systemctl enable --now drhall-backup-freshness.timer ``` Verify: ```bash systemctl is-enabled drhall-backup-freshness.timer systemctl is-active drhall-backup-freshness.timer systemctl list-timers drhall-backup-freshness.timer --all ``` Validated state: ```text enabled active ``` The first automatic service execution also completed successfully: ```text UP: Backup healthy: latest snapshot 74e45991 is 7h old ``` #### Step 8 — Safely test the stale/DOWN condition Do **not** stop, corrupt, or delay the real backup. The checker supports a one-command threshold override solely for controlled testing. Temporarily stop only the freshness timer so it cannot immediately overwrite the test state: ```bash systemctl stop drhall-backup-freshness.timer ``` Run a single simulated stale check with a `0`-hour threshold: ```bash THRESHOLD_HOURS=0 /usr/local/sbin/check-drhall-backup-freshness.sh echo "Exit code: $?" ``` Validated result: ```text DOWN: Backup stale: latest snapshot 74e45991 is 7h old (threshold 0h) Exit code: 1 ``` The following were then verified: ```text Uptime Kuma monitor changed to DOWN PASS DOWN message included snapshot/age PASS Agency DOWN email received PASS ``` #### Step 9 — Restore the production 26-hour state and verify recovery Run the checker normally, with no override: ```bash /usr/local/sbin/check-drhall-backup-freshness.sh echo "Exit code: $?" ``` Validated recovery result: ```text UP: Backup healthy: latest snapshot 74e45991 is 8h old Exit code: 0 ``` The Uptime Kuma monitor returned to `Up`, and the agency recovery notification was received. Restart/verify the timer: ```bash systemctl start drhall-backup-freshness.timer systemctl is-enabled drhall-backup-freshness.timer systemctl is-active drhall-backup-freshness.timer systemctl list-timers drhall-backup-freshness.timer --all ``` Final validated state: ```text Timer enabled: PASS Timer active: PASS 30-minute schedule: PASS Production threshold: 26 hours Healthy state: PASS Controlled stale state: PASS DOWN email delivery: PASS Recovery UP state: PASS Recovery email delivery: PASS ``` ### 31.11 Quarterly Restore Drill Tracking The Build Specification requires a quarterly restore drill that restores a client backup to a scratch droplet, verifies login and a safe workflow execution, records the result, and cleans up the scratch environment. The Dr. Hall recovery procedure itself has already been validated through Restore Drill #1 and Restore Drill #2. On September 30, 2026, Kimi confirmed that the recurring quarterly drill can be tracked in **ClickUp**. Standard tracking method: ```text Tracking system: ClickUp Task type: Recurring operational task Frequency: Quarterly Execution: Manual restore drill using the validated recovery procedure ``` Each quarterly task should require evidence of: - scratch droplet creation; - latest backup restore; - owner/login verification; - safe workflow execution; - total recovery duration; - result against the current restore SLO; - cleanup of temporary infrastructure and access; - confirmation that production remained untouched. At the time of this runbook revision, the **tracking method is approved**. If the recurring ClickUp task has not yet been created, create it as an administrative follow-up rather than representing it as already scheduled. ### 31.12 Final Operational Status and Remaining Administrative Follow-Up The original client-server verification items for Dr. Hall have now been completed and documented. The backup-freshness alert and certificate-expiry notification path have also been validated. Remaining Build-Spec / administrative follow-up: ```text Create recurring quarterly ClickUp restore-drill task TO DO if not yet created Slack notification delivery NOT CONFIGURED / NOT TESTED Monthly reboot maintenance window NOT YET DEFINED / VERIFIED DigitalOcean real threshold email delivery NOT FORCED ON PRODUCTION ``` The absence of a forced DigitalOcean threshold test does not invalidate the confirmed policy configuration. Do not deliberately stress production solely to generate a resource alert. The Build Specification also calls for a **monthly maintenance window for operating-system reboots**. Unattended security patching is verified, but a recurring reboot window and its communication process were not defined or tested during this verification. Confirm the operating schedule/owner before representing that requirement as complete. No payment automation or new access services were started during this verification phase. Future provisioning/recovery automation remains a separate implementation phase governed by Section 32. ### 31.13 Build-Spec Items Explicitly Deferred to the Automation / Control-Plane Phase The Dr. Hall manual runbook validates the client provisioning, security, monitoring, backup, escrow, and recovery behaviors that exist today. The following Build Specification items are **future-state control-plane or fleet-operations requirements** and were not implemented or validated by this manual verification: ```text Kuma status sync → instances.health / portal health DEFERRED TO CONTROL-PLANE PHASE Backup success/failure → backups table / admin portal DEFERRED TO CONTROL-PLANE PHASE Fleet n8n version state in control plane DEFERRED TO AUTOMATION PHASE Staging → canary → fleet ring rollout DEFERRED TO AUTOMATION PHASE Pre-update per-client pg_dump orchestration DEFERRED TO AUTOMATION PHASE Portal announcement workflow for maintenance/reboots DEFERRED TO PORTAL/OPS PHASE ``` The manual 26-hour Uptime Kuma freshness monitor provides an operational stale-backup alert for the current pilot, but it does **not** replace the Build Specification's future `backups` table / admin-portal reporting requirement. Likewise, the working HTTPS monitor does not replace the future `instances.health` synchronization requirement. These distinctions should remain explicit so future automation does not treat the manual checks as implementation of the control-plane features. --- ## 32. Future Automation Handoff This runbook is the validated **manual provisioning and recovery reference** for AI Biz. Future automated provisioning and recovery should be designed from **both validated recovery paths** in this runbook: - **Procedure A / Restore Drill #1:** the complete manual baseline in Sections 15–23, followed by its own validated cleanup in Section 24. This is the clearest source-of-truth for the individual recovery actions and manual fallback behavior. - **Procedure B / Restore Drill #2:** the optimized helper scripts and execution flow in Sections 25–26, followed by its own validated cleanup in Section 27. This shows which repetitive actions were safely automated and how the end-to-end SLO was reduced below 60 minutes. - **Cleanup:** Drill #1 and Drill #2 intentionally have separate cleanup procedures. Future automation must model the correct cleanup for the recovery path being executed instead of treating cleanup as one generic sequence. Future automation should reproduce the validated host-key handling, temporary SSH access controls, two-stage tunnel, backup/escrow recovery, workflow deactivation, exact-version startup, health validation, owner verification, safe workflow verification, cleanup, and measured completion criteria. Automation architecture, APIs, queues, control-plane state, billing-trigger implementation, and generalized multi-client parameterization should be documented separately.