English {#english}

PREGO v4.0 Operations Runbook Summary

For full details see §14 Runbook in the implementation plan. Below is a quick reference.

Frequently used workflows

Situation	Action
Pulumi Up (per region)	Actions → Pulumi Up → Run workflow, set `region`: sg / us / eu. Or push infra changes to main.
Pulumi Destroy	Actions → Pulumi Destroy → enter `tenant_id`, `region`, `confirm`: DESTROY. Production destroy requires approval.
Pulumi Preview (PR)	Runs automatically on PRs that change `prego-pulumi/**`. Matrix: sg, us, eu.

When something fails

Failure	Where to check
Pulumi Up failed	Actions logs, Pulumi Cloud lock/state. If needed, run `pulumi refresh` / `pulumi up` manually (same stack).
Provision failed	Control Plane → provision_jobs, trace_id → GET /trace/:trace_id for LogPath. Check Slack alerts.
Stripe Webhook duplicate	Ignored idempotently via provider_events. Subscription stays Active during 7-day grace.
Manual Hard Purge	Not recommended for tenants < 30 days. If required: workers/purge-job `POST /purge` or wait for Cron. Order: §5.8 (Hetzner → R2 → D1, keep audit_logs).

Secrets and environment

List and rotation: GitHub secrets. Recommended: HCLOUD_* 90d, PULUMI 180d, CLOUDFLARE 90d.
After rotating: Run once manually any workflow that uses that secret to verify.

Manual worker triggers

Worker	Manual run
Usage Aggregator	`POST /aggregate`
Cycle Close	`POST /cycle-close`
Purge Job	`POST /purge`
Autoscaler	`POST /run`

(Worker URLs: Wrangler or Cloudflare dashboard.)

Reference

Control Plane Worker (prego-control-plane): Separate repo. POST /v1/tenants (202+job_id), Stripe webhook, POST /internal/provision-complete callback and KV updates. On workflow completion call provision-complete with that Worker URL + Bearer INTERNAL_API_KEY. See Implementation index prego-control-plane. Deploy and dashboard: Prego Control Plane dashboard.
Data plane resource tuning: Resource optimization safe adoption, Quick reference.
Hetzner server (step 1): Pulumi Step 1 Hetzner.
prego-docker image verify: Prego Docker image verify.
MariaDB 10.6→10.11: MariaDB migration.
Frappe v16 rollback: Frappe v16 rollback.
API Control Plane runbook: Provisioning failure, DDoS / traffic spike. Index: API Control Plane implementation plan §19.
client-web deploy (OTP): Client web deploy OTP verify.
Passcode reset flow: Passcode reset flow verification.
OTP and email: OTP and email consolidated.
Monitoring (Cloudflare): Cloudflare-based monitoring. Dashboard: GET /internal/dashboard, GET /internal/metrics/summary (Bearer). Email flow: GET /internal/email-flow-dashboard — see Prego Control Plane dashboard. Hybrid: Server metrics collection.
Implementation plan §14, §14.1; Deployment checklist; Implementation index.

flowchart LR
  subgraph Triggers
    A[Pulumi Up]
    B[Pulumi Destroy]
    C[Provision]
    D[Workers]
  end
  subgraph On failure
    E[Actions / Pulumi Cloud]
    F[Control Plane / trace_id]
    G[Slack]
  end
  A --> E
  B --> E
  C --> F
  C --> G
  D --> D

한국어 {#korean}

PREGO v4.0 운영 Runbook 요약

상세는 구현 기획서 §14 Runbook 참조. 아래는 빠른 참조.

자주 쓰는 워크플로

상황	조치
Pulumi Up (리전별)	Actions → Pulumi Up → Run workflow, `region`: sg / us / eu. 또는 main에 infra 변경 push.
Pulumi Destroy	Actions → Pulumi Destroy → `tenant_id`, `region`, `confirm`: DESTROY 입력. production-destroy 승인 필요.
Pulumi Preview (PR)	`prego-pulumi/**` 변경 PR 시 자동. matrix: sg, us, eu.

실패 시 확인

실패	확인 위치
Pulumi Up 실패	Actions 로그, Pulumi Cloud lock/state. 필요 시 수동 `pulumi refresh` / `pulumi up` (동일 스택).
Provision 실패	Control Plane → provision_jobs, trace_id → GET /trace/:trace_id 로 LogPath. Slack 알림 확인.
Stripe Webhook 중복	provider_events 멱등으로 무시됨. 7일 Grace 동안 Active 유지.
Hard Purge 수동	30일 미만은 비권장. 예외 시 workers/purge-job `POST /purge` 또는 Cron 대기. 순서: §5.8 (Hetzner → R2 → D1, audit_logs 유지).

Secrets·환경

목록·로테이션: github-secrets. 권장: HCLOUD_* 90일, PULUMI 180일, CLOUDFLARE 90일.
로테이션 후: 해당 Secret을 쓰는 워크플로 1회 수동 실행으로 검증.

Worker 수동 트리거

Worker	수동 실행
Usage Aggregator	`POST /aggregate`
Cycle Close	`POST /cycle-close`
Purge Job	`POST /purge`
Autoscaler	`POST /run`

(각 Worker 배포 URL은 Wrangler/Cloudflare 대시보드에서 확인.)

참조

Control Plane Worker (prego-control-plane): 별도 레포. POST /v1/tenants(202+job_id), Stripe webhook, POST /internal/provision-complete 콜백·KV 갱신. 워크플로 완료 시 해당 Worker URL + Bearer INTERNAL_API_KEY로 provision-complete 호출. IMPLEMENTATION_INDEX prego-control-plane 항목. 배포·대시보드 접속: prego-control-plane-dashboard — URL, 최초 배포(D1/KV), INTERNAL_API_KEY 설정.
리소스 절감 (Data Plane): resource-optimization-safe-adoption-plan — Redis·Gunicorn·CDN·Docker 제한 적용 순서. 요약: resource-optimization-quick-reference.
1단계 Hetzner 서버 생성 (로컬): pulumi-step1-hetzner-server
prego-docker 이미지 검증·취약점: prego-docker-image-verify — 앱 포함 검증, Trivy 스캔·정책.
MariaDB 10.6→10.11 마이그레이션: mariadb-106-to-1011-migration — v16 전제 DB 업그레이드.
Frappe v16 롤백: frappe-v16-rollback — v16→v15 복귀 절차.
API Control Plane Runbook: provisioning-failure (테넌트 생성 실패), ddos-traffic-spike (DDoS·트래픽 급증). 인덱스·상세: api-control-plane-implementation-plan §19.
client-web 배포 (x.pregoi.com OTP 반영): client-web-deploy-otp-verify-email — OTP 6칸 숫자 노출·에러 시 초기화·붙여넣기 반영 빌드 배포 및 검증.
Passcode 재설정 플로우 검증: passcode-reset-flow-verification — request-reset → 이메일 링크 → reset-passcode → 로그인 완료 검증, CLIENT_WEB_BASE_URL·ZUPLO_EMAIL_* 확인.
OTP·이메일 (플로우·템플릿·Zuplo): otp-and-email-consolidated — 발신 경로(Auth Worker / Frappe), 템플릿 위치·가독성, Zuplo 호출. OTP 메일은 Auth Worker만 발송.
모니터링(Cloudflare 기반): cloudflare-based-monitoring-plan — Analytics Engine·D1·Workers 대시보드·Logpush/R2. 대시보드: GET /internal/dashboard, GET /internal/metrics/summary (Bearer). 이메일 플로우 검증: GET /internal/email-flow-dashboard — prego-control-plane-dashboard §이메일 플로우 검증 대시보드. E2E(발송+OTP 검증 자동)는 Auth Worker OTP_TEST_EMAILS 설정 후 플로우 B 탭에서 사용. Edge(Phase 2): Secrets CF_ANALYTICS_READ_TOKEN·CF_ACCOUNT_ID 설정 시 Row 3 메트릭 포함. Hybrid: POST /internal/server-metrics — server-metrics-collection. D1 migration 0007.
구현 기획서 §14, §14.1
DEPLOYMENT_CHECKLIST
IMPLEMENTATION_INDEX