Under the Hood: How the Flag Caretaker was built

Earlier this week, the Flag Caretaker was announced, a new automated flag-hygiene system designed to help Teams and Ultra subscribers clean up stale and dormant feature flags.

Building a system that analyzes millions of evaluations, determines code safety, and generates contextual prompts, all without codebase access, was a fascinating engineering challenge.

Here is a look under the hood at how the Flag Caretaker was designed and built.

The Design Philosophy: Pure domain, zero I/O

When designing the status classifier, the goal was to ensure the classification logic was completely testable and predictable. The decision engine was separated from the database and APIs by implementing it as a pure function in Go:

func EvaluateStatus(flag *Flag, auditEvents []*AuditEvent, totalHits int64, now time.Time) (newStatus string, changed bool)

By passing all dependencies (the flag data, a list of audit events for the last 30 days, the 30-day evaluation traffic count from BigQuery, and the current time), the function has no side effects. It’s incredibly fast, deterministic, and can be unit tested across dozens of permutations without mocking a database.

The Classification State Machine

The Caretaker classifies flags into one of four states: healthy, stale, dormant, or ignored (snoozed). To minimize false positives, the classifier evaluates flags using a strict set of rules.

                detector (daily 2AM)                user action
                ─────────────────────               ───────────
created ──► healthy ──► [stale | dormant] ──► (badge + tab appear)
                            │                          │
                            │  snooze 30/60/90d        ▼
                            └──────────────► ignored ──┘
                                              │ (snooze expires
                                              │  or user unsnoozes)
                                              ▼
                                    re-evaluated → stale/dormant/healthy

1. Snooze Check (ignored)

If a user has snoozed the flag in the UI (CaretakerIgnoredUntil > now), the evaluation immediately short-circuits. The flag remains in the ignored state until the timer expires.

2. Dormancy Detection (dormant)

A flag is classified as dormant when it has effectively served no traffic:

• Age Guard: The flag must be at least 60 days old. Newly created flags are skipped.
• Traffic Check: The last evaluation hit is either 0 (never hit) or older than 60 days.

3. Staleness Detection (stale)

Determining if a flag is stale (and safe to delete) is much more complex. The Caretaker requires five distinct conditions to hold true:

1. Age Guard: The flag must be at least 30 days old.
2. Traffic Pinned: The traffic percentage must be locked at exactly 0 or exactly 100. If it’s serving a percentage rollout (e.g., 50%), it’s still active and therefore healthy.
3. High Traffic: The flag must have served at least 1,000 evaluations over the last 30 days. This prevents low-traffic or staging-only flags from being falsely flagged.
4. Recently Evaluated: The flag must have been evaluated at least once in the last 7 days.
5. Untouched: No admin has toggled the flag or changed its traffic configuration in the last 30 days. The flag’s audit log is scanned to verify this.

If all five conditions are met, the flag is marked as stale. If none of the conditions for dormant or stale match, the flag is classified as healthy.

A Gotcha: Numeric Normalisation

Firestore decodes all numbers as int64, but in-memory audit logs and SDKs can use varying types (like int8 or int). In early staging runs, comparing a Firestore audit event int64(100) to an in-memory event int(100) returned a false mismatch, incorrectly indicating a flag configuration change.

This was solved by writing a normalization helper toFloat to standardize numeric types before comparison, falling back to plain equality for strings and booleans.

Backend Architecture

The Caretaker system is split into three layers:

1. Platform Layer (platform/caretaker/): Houses the pure domain logic (EvaluateStatus), prompt templates, and testing fixtures.
2. Controller Layer (controllers/caretaker/): Handles business logic, verifies subscriber entitlements, and coordinates snooze/unsnooze operations.
3. Persistence Layer (PatchCaretakerStatus): Writes status changes back to Firestore, invalidates the cache, and logs audit events.

┌─────────────────────────────────────────────────────────┐
│              Daily Cron Job (Cloud Run)                 │
└────────────────────────────┬────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────┐
│               BigQuery Telemetry Batch                  │
└────────────────────────────┬────────────────────────────┘
                             │ (Fetch 30-day hit counts)
                             ▼
┌─────────────────────────────────────────────────────────┐
│                 Platform Layer (Pure)                   │
│   - EvaluateStatus(...)                                 │
│   - BuildPrompt(...)                                    │
└────────────────────────────┬────────────────────────────┘
                             │ (Classify stale/dormant)
                             ▼
┌─────────────────────────────────────────────────────────┐
│              Persistence Layer (Firestore)              │
│   - PatchCaretakerStatus                                │
│   - Cache Invalidation & Audits                         │
└─────────────────────────────────────────────────────────┘

Cache Invalidation & Snooze Bugs

During development, two nasty state bugs were uncovered:

• The Cache Race: Originally, the project-wide flag list was cached asynchronously in a goroutine (go r.Cache.Set(...)). However, this raced with Cache.Delete(...) during updates, occasionally overwriting the cache with outdated flag states. This call was made synchronous to guarantee consistency.
• The Unsnooze Trap: When users clicked “Unsnooze”, the snooze timer was cleared, but the flag would temporarily revert to a generic status. This was resolved by introducing a CaretakerPreSnoozeStatus field. Now, when a flag is snoozed, the active status is captured and restored atomically upon unsnoozing.

High-Performance Detector Job

Rather than checking flags one by one (which would saturate Firestore and API calls), a high-performance, containerized Go binary (caretaker-detector) was built to run daily at 2:00 AM.

1. Batched BigQuery queries: The job starts by running a single, optimized query to fetch the 30-day evaluation counts for all flags in one round trip.
2. Streaming Firestore reads: Projects are streamed one at a time using StreamAllAdmin to keep memory usage bounded, preventing the container from scaling out of control.
3. Group Flag Proxies: Group flags contain multiple environments (e.g., staging and production) in a single configuration. The detector evaluates these using a lightweight proxy adaptor that maps the maximum traffic and last-hit timestamps across all environments, allowing the same pure EvaluateStatus logic to be reused.

Constructing the Prompt Templates

To generate the AI prompts, Go’s built-in text/template engine is used. Depending on whether a flag is stale or dormant, the Caretaker renders a template with details about the flag: who last changed it, how many hits it received, and its traffic rules.

For stale flags, the instructions branch based on the traffic percentage:

• If the flag is locked at 100% traffic, the prompt tells the AI agent that the true code block is permanent and instructions it to delete the conditional wrapper and the false path.
• If locked at 0% traffic, it does the opposite.

For dormant flags, since no traffic is active, the prompt instructs the AI to carefully evaluate both branches and pick the one matching current business logic, and when in doubt, retain the safer defensive path and leave a standard developer TODO for review.

Daily Digests via Loops

Every run, the detector aggregates newly stale or dormant flags. Instead of spamming users, it groups these flags by organization.

If an organization is on a Teams plan or above, the system gathers all active administrators and sends each of them a single digest email via the Loops transactional API, summarizing the newly affected flags. In DRY_RUN mode no digests are sent at all, and in non-production environments the email payload is logged rather than sent to the API.

Looking Forward

By combining a pure domain engine, batched telemetry, and a safe, prompt-based cleanup design, the Flag Caretaker helps you eliminate codebase rot without introducing security or performance issues.

Give the Caretaker a try, and stay tuned as more intelligence tools are built to streamline release pipelines.

— JK @ RocketFlag