Lesson 35 — Monitoring, health checks and backups: A responsive application does not prove that workers process jobs, scheduled tasks finish or backups can be restored. Observe each promise separately and assign ownership for responding to failures.
1. Liveness, readiness and business completion
The default /up route checks application boot, not every dependency. The internal taskflow:health-check command adds SELECT1, cache read/write and private-storage read/write probes. A living worker may consume the wrong queue; an active timer may miss intended work. Observe completion rather than just process existence.
| Signal | Question | Response |
|---|---|---|
| HTTP errors/p95 | Are user requests completing correctly and promptly? | Correlate release, sanitized logs and database |
| Oldest pending job/failures | Is queue work progressing? | Inspect consumers/dependencies before retries |
| Completion heartbeat | Did expected scheduled work finish? | Inspect timer, locks, timezone and exit status |
| Backup/restore age | Is recovery data recent and rehearsed? | Investigate backup pipeline and restore drill |
Choose thresholds and windows from system SLOs, not arbitrary copied values. Each alert needs an owner, priority, contact path and runbook. No external monitoring provider or real alert routing is configured by this lesson.
2. Probes without leaking secrets
php artisan taskflow:health-check
php artisan taskflow:security-check
The first checks dependencies; the second checks configuration. HealthCheck prints component PASS/FAIL labels and exits nonzero on failure, omitting exception strings that may disclose credentials or paths.
// Excerpt from HealthCheck; each closure is caught independently by the command.
'database' => fn () => DB::select('select 1') !== [],
'cache' => function (): bool {
$key = 'taskflow:health:'.Str::uuid();
try {
return Cache::put($key, 'probe', 30) && Cache::get($key) === 'probe';
} finally {
Cache::forget($key);
}
},HealthCheck imports DB/Cache/Storage facades and Str. Storage probes use UUID files under .health on the private local disk, read back the content and remove their own file in finally. Cache keys have a30-second TTL; the command never flushes the store. These are small write probes, not entirely read-only diagnostics. A killed process can leave a file behind; verify age and provenance before cleanup.
Use the application's identity and environment. Passing array cache does not prove a shared store works, and deploy-user permissions do not prove FPM permissions. Configure dependency-client timeouts and an external execution deadline; the command imposes no overall timeout. Do not expose a public debug endpoint or create a fleet-wide restart loop from transient dependency failures.
3. Start backup planning from recovery requirements
Agree acceptable lost writes (RPO) and recovery duration (RTO) before selecting frequency, retention and destinations. TaskFlow needs its database, private PDFs, securely recoverable APP_KEY/other necessary secrets and matching code/lockfiles. An intact database may still be unusable if encryption keys are lost.
Database metadata and files are not one transaction. Choose controlled write quiescence or a consistent snapshot strategy, then reconcile references. Encrypt backups, restrict access, keep copies outside the host and address deletion resistance as required. Never place backups under public or print secrets in logs.
4. An isolated SQLite restore drill
BackupRestoreDrillTest migrates in-memory SQLite and creates a task/attachment fixture. VACUUM INTO produces a snapshot; a separate restored database and file tree are checked for integrity, foreign keys, task title, attachment references/size and SHA256. All paths belong to the test fake disk; no development or production database is read.
// Inside BackupRestoreDrillTest, after migrating isolated in-memory SQLite:
$disk = Storage::fake('restore-drill');
$snapshot = $disk->path('snapshot.sqlite');
DB::statement('VACUUM INTO '.DB::getPdo()->quote($snapshot));
$disk->copy('snapshot.sqlite', 'restored.sqlite');
$restored = new PDO('sqlite:'.$disk->path('restored.sqlite'));
$this->assertSame('ok', $restored->query('PRAGMA integrity_check')->fetchColumn());
$this->assertSame([], $restored->query('PRAGMA foreign_key_check')->fetchAll());This excerpt is not a production backup script. VACUUM INTO writes a separate snapshot and leaves the source unchanged; its destination must be new or empty, and the connection must not have an open transaction. Database snapshots do not include files, so the test restores and hashes them separately. Do not blindly copy a running SQLite file while ignoring WAL. Other engines require their own backup/recovery tools.
5. Restore acceptance before reconnecting services
- Select a backup and matching code, with new isolated database/storage targets.
- Block outgoing mail/webhooks and leave workers, scheduler and Reverb disabled until reviewed.
- Restore data and secrets securely; verify integrity, counts and sampled attachments.
- Boot the application and test login, policies, task reads, authorized downloads and decryption where relevant.
- Decide which queued work may replay to avoid unintended duplicate side effects.
- Handle restored tokens/sessions: snapshots may resurrect credentials revoked after the backup.
- Record elapsed recovery time, checkpoint, estimated lost writes and acceptance owner.
Never overwrite production merely to rehearse. Backup exit0 and matching checksums do not prove application recovery. Remove only identified drill data afterward, not broad paths.
6. Current evidence and limits
php artisan test --filter=HealthCheckTest
php artisan test --filter=BackupRestoreDrillTest
php artisan test
vendor/bin/pint --test
The suite passes102 tests/465 assertions and Pint. Health tests cover success, redacted database errors and cache mismatch; the restore test checks fixture data/files. Concurrent WAL traffic, encrypted offsite storage, secret recovery, restored HTTP application boot, production volume, RPO/RTO and unattended backup scheduling remain untested. The operator checklist is taskflow/deploy/monitoring-backup.md.
Exercise: assign SLOs and owners to each signal, simulate dependency failure on staging and produce a restore report including unmet criteria. Next comes controlled dependency management and framework upgrades.
References: SQLite VACUUM INTO, Laravel deployment and health route. Navigation: Lesson 34 · Roadmap.




No comments yet. Be the first to share your thoughts.