There's a specific kind of fear that hits you when you're about to deploy code that controls how real money moves. Not "oops, the page crashed" money. Hundreds of crores. Thousands of accounts. Options trades where a partial execution doesn't just lose money — it leaves someone with an unhedged position and theoretically unlimited risk.

I built the portfolio allocation engine at a Y Combinator-backed fintech that automated options trading. This is the story of how I shipped it without blowing up anyone's account — and the estimation disaster that almost derailed the whole thing.

The Problem: One Portfolio Wasn't Enough

Users on our platform could only invest in a single trading strategy. If that strategy had a bad week, their entire capital took the hit. No diversification. Churn was climbing.

The product ask was straightforward: let users invest across multiple model portfolios simultaneously. Spread risk. Reduce churn.

The engineering reality was anything but straightforward.

The Constraint That Changed Everything

Options trading has a property that makes it fundamentally different from, say, allocating money across mutual funds. Most options strategies involve hedged positions — a spread, for example, where you buy one contract and sell another. The two legs protect each other.

If you only execute one leg, the user is left holding a naked position. Depending on the market, that exposure can be unlimited.

This meant the allocation algorithm had a hard constraint: either a user trades ALL their assigned strategies for the week, or they trade NONE. There's no "let's do 3 out of 5 and skip the rest." Partial allocation is financially dangerous.

I remember sitting with this constraint and thinking — this isn't a software problem anymore. This is a "someone's life savings" problem.

The Two-Pass Algorithm

If you squint at this problem from an algorithms perspective, it's a variant of the 0/1 Knapsack problem with an all-or-nothing constraint. Each user has a "knapsack" (their available capital) and a set of "items" (strategies, each with a capital cost). The twist: unlike classic knapsack where you maximize value by selecting a subset, here the constraint is you must fit ALL items or take NONE. There's no partial selection. This simplifies the optimization (no need to find the optimal subset) but makes the feasibility check critical — get it wrong, and someone has an unhedged position.

More formally, this is a constraint satisfaction problem (CSP): given a user's capital state and a set of strategy requirements, determine if a valid assignment exists where all strategies are funded simultaneously. Pass 1 is the feasibility check. Pass 2 is the assignment.

The allocation engine ran as a Cloud Scheduler-triggered daily job. Every morning before market open, it processed all 2,500+ accounts and decided: who trades today, and how much capital goes where.

Pass 1 — Eligibility (Constraint Satisfaction)

For each user, compute projected available capital:

projected_available = current_balance
                    - capital_locked_in_open_positions
                    - margin_reserved_for_pending_orders
                    + capital_releasing_from_today's_expiries

Then sum up the capital required across ALL assigned strategies for the week. If projected_available < total_required_across_all_strategies, the user is marked ineligible. They sit out the entire week. No partial bets.

This is the CSP feasibility check — can all constraints (strategy capital requirements) be satisfied simultaneously given the available resources (user capital)? If not, the entire assignment is rejected. No greedy "fit as many as possible."

The tricky term is capital_releasing_from_today's_expiries. A position expiring today could settle in profit (releasing more capital than expected) or in loss (consuming more). At the time the algorithm runs — early morning, before market open — we don't know the settlement outcome yet. I initially used last closing price to estimate this. Shadow mode later revealed this was insufficient (more on that below).

Pass 2 — Capital Distribution (Round-Robin Scheduling)

For eligible users, capital is distributed across their assigned strategies using a pattern borrowed from round-robin scheduling — the same algorithm OS schedulers use to allocate CPU time across processes, adapted here for capital allocation across strategies with temporal constraints:

for each eligible user:
    strategies = user.assigned_strategies  # sorted by execution_window
    remaining_capital = user.projected_available

    for each strategy in strategies:
        required = strategy.compute_capital_requirement(user)
        if remaining_capital >= required:
            allocate(user, strategy, required)
            remaining_capital -= required
        else:
            # This shouldn't happen — Pass 1 already verified total affordability
            # If it does, it's a bug. Log, alert, skip user entirely.
            rollback_all_allocations(user)
            break

The round-robin ordering matters because strategies have different execution windows. A strategy that trades at 9:30 AM needs capital allocated before one that trades at 2:00 PM. The sort ensures capital is committed in execution order — same principle as priority scheduling with deadlines, where time-constrained jobs get resources first.

The entire job — Pass 1 + Pass 2 for 2,500 users — completes in under a minute. It's CPU-light; the work is mostly database reads (balances, positions, strategy configs) and in-memory computation. Writes are the allocation decisions themselves, persisted to the database and picked up by the execution pipeline when trading windows open.

The algorithm itself wasn't the hard part. The hard part was trusting it.

I Didn't Trust My Own Code

We were managing ₹300Cr+ in real transaction volume across 2,500+ accounts. I'd written the algorithm, tested it, reviewed it. But I couldn't bring myself to flip the switch and let it control real trades on day one.

So I built shadow mode — also known as dark launching or shadow traffic testing in the broader industry.

If you've worked with deployment strategies, you've probably seen blue-green deployments (two identical environments, traffic switches atomically), canary releases (route a small percentage of traffic to the new version), and feature flags (toggle behavior at runtime). Shadow mode sits in a different spot on this spectrum. Unlike canary, where the new code serves real traffic for a subset of users, shadow mode has zero production impact. The new algorithm processes real inputs but its outputs go nowhere — they're logged, not acted upon. The old algorithm remains fully in control.

The reason I chose shadow mode over a canary approach: this isn't a system where "5% of users get the new version" is safe. In a web app, if the canary serves a bad page to 5% of users, you roll back and those users refresh. Here, if the new allocation is wrong for even one user, they could end up with an unhedged options position. The cost of a single bad decision isn't a degraded experience — it's real financial loss. Shadow mode was the only deployment pattern where I could validate against real production data with zero risk.

The idea is simple: run the old algorithm and the new algorithm simultaneously. The old one controls real trades. The new one runs silently, logs every decision it would have made, and writes the output to a separate store. Every morning, I'd compare the two outputs manually.

We ran shadow mode for 2–3 weeks. And it caught things I wouldn't have found in testing.

The timing edge case: A user's balance was technically sufficient at the time the algorithm ran. But they had positions expiring later that same day. Depending on whether those positions settled in profit or loss, the balance could swing either way. The new algorithm was saying "eligible" based on a snapshot, but by the time trades actually executed, the numbers had shifted. Shadow mode surfaced this because I could compare "what the algorithm decided at 8 AM" against "what actually happened by 3:30 PM."

The rounding problem: Small-balance users near the eligibility threshold were flickering between eligible and ineligible across runs due to floating-point rounding in the capital calculation. Not a bug you'd catch in a unit test with clean numbers.

Both of these would have caused real problems — either users trading when they shouldn't, or users being incorrectly blocked. Shadow mode was the safety net that let me find them without anyone's money on the line.

After 2–3 weeks, I gradually migrated users — effectively moving from shadow mode to a staged rollout, shifting user cohorts from the old algorithm to the new one. This is the standard canary progression, but only after shadow mode had already validated correctness.

The Estimation Disaster

Here's the part I'm less proud of.

This project was part of a larger rewrite — let's call it the Bundle Service v3. Original estimate: 1 month. One week for the tech spec, one week for development, one week for QA, one week for release.

It took 2.5 months.

Three things I didn't anticipate:

1. Cross-team blast radius

The allocation engine didn't exist in isolation. It touched the onboarding service, the PnL generation pipeline, the analytics system — roughly 5 dependent services owned by different people. I estimated as if I was only changing my service. I wasn't.

2. Legacy code tax

The existing system had deprecated data models, dead code paths, and flows that weren't idempotent. This was the real blocker for shadow mode. Idempotency — the property that running an operation multiple times produces the same result as running it once — is a fundamental requirement in distributed systems (you'll find it in any discussion of exactly-once semantics, Kafka message processing, or payment systems). In our case, the old allocation flow had side effects: it mutated shared state as it ran, so running it twice could produce different results or double-count allocations. You can't safely run old and new algorithms in parallel if the old one has side effects you don't fully understand. I had to make the existing flows idempotent first — adding idempotency keys, making operations safely re-runnable, ensuring that re-processing a user didn't create duplicate allocations — before I could even begin the shadow mode comparison. Cleaning this up took weeks I hadn't budgeted for.

3. Shadow mode wasn't in the original plan

When I sat down and really thought about deploying to ₹300Cr+ in live money, I realized I couldn't skip the parallel validation phase. It was the right call — but it added significant time, and I hadn't scoped it upfront because I hadn't fully internalized the risk when I gave the initial estimate.

The 2.5x overrun was my estimation failure. I own that.

What I Changed After This

I stopped estimating before I understood the full blast radius. For any cross-service change now, the first step is mapping every impacted service and talking to its owner before committing to a timeline. This is where the tech spec practice came from — I started writing detailed design documents for every feature, getting alignment in review before writing a line of code.

I added a "legacy tax" to every estimate. In a fast-moving startup, there's always more legacy than you think. If a feature touches old code, I add buffer. Always.

I plan the rollout before the code. Shadow mode should have been in the original estimate. For anything touching money or critical paths, the deployment strategy is now part of the spec, estimated alongside development — not added as an afterthought.

I escalate early with data. When I realized at the one-week mark that the scope was bigger than expected, I went to my manager with specifics: here are all the services impacted, here are the touch points, here's why shadow mode is non-negotiable for a system handling this much money, and here's the revised timeline. Because I showed up with data instead of vague "it's taking longer," the extended timeline was understood — not a surprise.

The Patterns Behind the Design

Looking back, this project was a crash course in applying well-known engineering patterns to a domain where the cost of getting them wrong is measured in money, not error rates.

Constraint Satisfaction / 0-1 Knapsack — Pass 1 eligibility check. Framing the problem correctly — this is feasibility checking, not optimization. All-or-nothing means you reject early instead of trying to find the "best partial fit."

Round-Robin Scheduling — Pass 2 capital distribution. Borrowed from OS process scheduling. Temporal ordering (execution windows) maps directly to deadline-based priority scheduling.

Shadow / Dark Launch — Pre-production validation. Zero-risk validation against real data. Chosen over canary because even one bad allocation has unbounded financial cost.

Staged Rollout / Canary — Post-shadow migration. After shadow mode validated correctness, users were migrated in cohorts — the standard canary progression.

Idempotency — Legacy code cleanup. Required before shadow mode could work. Operations needed to be safely re-runnable — same principle behind idempotency keys in payment APIs.

None of these patterns are novel. They're textbook. But knowing when to apply which pattern, and understanding why shadow mode was necessary here but a canary might suffice elsewhere — that's the engineering judgment part that doesn't show up in a design patterns book.

The Outcome

The allocation engine shipped. Multi-portfolio investing went live across 2,500+ accounts. No user money was impacted during the rollout. Shadow mode caught real edge cases that would have caused problems. Churn reduced.

And I learned that the scariest part of shipping critical software isn't the algorithm. It's the moment you realize your estimate was wrong, your scope is blowing up, and people's money is on the other side of your deploy button.

The algorithm was the easy part. Earning the right to trust it — that was the real engineering.

I'm a Senior Backend Engineer with 6+ years of experience building distributed systems in Go and Python. I write about the messy reality of backend engineering at mrinal.dev.

I'm deep in interview prep right now.

DSA, system design, machine coding — the whole sacred ritual of proving I can reverse a linked list under pressure. I've been doing this for weeks. Waking up early, solving graph problems, designing parking lots on whiteboards in my head, rehearsing how I'd explain the tradeoffs between eventual consistency and strong consistency to a stranger who holds my career in their hands for 45 minutes.

Meanwhile, the tool I'm chatting with could probably clear the round I'm prepping for.

I don't say that to be dramatic. I say it because I tested it. I gave Claude a machine coding problem — the kind where you're supposed to design a system from scratch in Go under time pressure — and it produced something clean, extensible, and honestly better-structured than what I'd have written under stress. In less time than it takes me to set up the folder structure.

There's a quiet existential comedy playing out in every engineer's head right now. If you're in tech, you've felt it. That low hum of cognitive dissonance between "I need to grind harder to get hired" and "the thing I'm grinding to prove I can do is being automated while I grind."

I've been sitting with this discomfort for a while now. And the analogy that keeps coming back to me — the one that actually helps me think clearly about what's happening — isn't from tech at all. It's from the 15th century.

The Scribes and the Press

When Gutenberg introduced the printing press around 1440, it didn't make scribes disappear overnight. For a while, both coexisted. Scribes kept copying manuscripts. The press kept printing books. Some scribes probably scoffed at the quality of early printed text — rough, inconsistent, lacking the artistry of hand-lettered pages.

But the thing is, the scribe's core skill — "I can copy text beautifully" — went from being the entire job to being worth exactly nothing. Not gradually. Not over centuries. Within a generation.

What survived? The people who understood the content. Editorial judgment. Knowing what's worth saying. The ability to curate, verify, and make decisions about what gets published and why. The scholars. The thinkers. The people whose value was never in the act of writing itself but in the thinking behind it.

The scribes? They're now a Wikipedia footnote.

Software Engineering's Gutenberg Moment

I believe software engineering is entering its printing press moment.

The "I can build it" era is winding down. AI is getting terrifyingly good at translating specs into working code. It doesn't get tired. It doesn't argue about tabs vs spaces. It doesn't need a standup. It doesn't have a bad day where it writes sloppy code because it had a fight with its roommate or slept poorly.

And it's getting better fast. Not linearly. The kind of fast where the gap between "impressive parlor trick" and "genuinely useful production tool" collapsed in about 18 months.

But here's where the analogy gets interesting — because it's not all doom. It's actually clarifying.

What the Press Can't Do

I spent the last year and a half as a Senior Backend Engineer at a fintech startup. We built trade automation — real money, real markets, real consequences. And I can tell you with certainty what AI cannot do, because I've lived in the gap between "technically works" and "trusted in production."

It can't know what you don't know yet.

We had a project — a service migration that was scoped at one month. It turned into two and a half. Not because the code was hard. Because the cross-service dependencies were invisible until we were knee-deep in them. Five services. Legacy fields that nobody documented. Allocation logic that had edge cases baked in by engineers who left two years ago. No AI could have predicted that from the spec. You had to live in the system to feel where it would break.

It can't wake up at 3 AM with a gut feeling.

We had a token refresh system that took 20 minutes every morning for 3,000 users. On paper, it worked. In practice, it was a ticking time bomb — one slow morning away from users missing market open. The decision to re-architect it — to move to Cloud Tasks with Redis caching and parallel generation — didn't come from a prompt. It came from the accumulated anxiety of watching those morning logs and thinking, "this won't survive the next 2x."

It can't navigate organizational chaos.

Software doesn't exist in a vacuum. It exists inside companies with politics, competing priorities, limited budgets, and humans who disagree. Getting a system from "I built it" to "it's deployed, monitored, and someone will wake up if it breaks" requires navigating a landscape that is fundamentally social, not technical.

It can't own the consequences.

When real money moves through a buggy allocation algorithm, someone has to answer for it. Not "explain what went wrong in a post-mortem" — actually own it. Feel the weight of it. Carry the scar tissue into the next design decision. That accountability is non-transferable.

The Shift

The shift isn't from "engineer" to "no engineer." It's from "I write the code" to "I know what's right, I can verify the machine got it right, and my name is on it when it breaks."

This is a profound change in what it means to be good at this job. The skills that got you hired in 2020 — raw coding speed, framework knowledge, the ability to implement a B-tree from scratch — are becoming table stakes at best and irrelevant at worst.

The skills that will matter are harder to test in interviews and harder to acquire quickly:

Systems thinking. The ability to see a problem in its full context — business constraints, technical tradeoffs, organizational dynamics, failure modes — and frame it correctly before anyone writes a line of code.

Domain expertise. Understanding what "correct" means in your specific domain. In fintech, that's knowing what SEBI compliance means for your order execution pipeline. In healthcare, it's understanding why a 99.9% accuracy rate might not be good enough. In logistics, it's knowing that your routing algorithm needs to account for the fact that drivers take lunch breaks.

Verification and judgment. The more powerful AI gets at generating solutions, the more critical it becomes to have humans who can evaluate those solutions. This is almost paradoxical — AI doesn't eliminate the need for expertise, it raises the stakes of it.

The Asymmetry

Here's the thought I keep coming back to.

The asymmetry isn't capability — it's intent.

AI can do anything I ask, but it has no reason to ask anything of itself. And even when it learns to self-correct — to catch its own bugs, to iterate without prompting — someone still has to decide: is this worth building at all?

That question doesn't come from intelligence. It comes from living in the world and giving a damn.

It comes from watching your users struggle with a clunky interface and feeling personally offended. From noticing that your morning token refresh is getting slower and deciding, on your own, without anyone asking, that you're going to fix it. From reading a regulatory announcement and immediately thinking about what it means for your architecture.

That's not a skill. That's not a capability. It's something closer to caring — and it's the one thing that can't be automated, no matter how good the models get.

So What Do I Do About It?

I'm still solving graph problems. The game today demands it. The interview system hasn't caught up to the reality shift, and I need a job, so I play by the rules as they exist.

But I'm keeping one eye on which of my skills are "scribe" skills and which are "scholar" skills.

The scribe skills: memorizing API signatures, implementing standard patterns from scratch, writing boilerplate, configuring infrastructure by hand. These have a shelf life. Not zero — not yet — but the expiry date is visible.

The scholar skills: knowing what to build, spotting what's wrong with someone else's solution (or an AI's solution), navigating the messy human process of shipping software inside an actual organization, carrying the weight of production systems that move real money. These don't expire. They compound.

The scholars did just fine after Gutenberg.

The scribes are now a Wikipedia footnote.

Choose wisely.

Or, probably, maybe I'm wrong. It all goes to shit from here.

If you're a software engineer wrestling with the same questions, I'd love to hear how you're thinking about it. What skills are you investing in? What are you letting go of? Drop a comment or reply to this email.

And if you're hiring senior backend engineers who think about systems this way — well, my DMs are open.

Table of Contents

1. Why I Wanted My Own AI Agent

2. Setting Up Open Claw on Digital Ocean

3. The N8N Layer — Why Open Claw Never Touches My Keys

4. Automation 1: Daily Bookmark Cleanup (Raindrop)

5. Automation 2: Daily Planner + Calendar Nudges

6. Automation 3: On-Call Monitoring That Actually Helps (PoC)

7. The Honest Downsides

8. Key Takeaways

Why I Wanted My Own AI Agent

I was doing the same things every single day. Manually.

Every morning: open Raindrop, scroll through yesterday's saved bookmarks, delete the dead ones, file the rest into collections. Every morning: open Google Calendar, mentally map out my day, set reminders for the important meetings. Every on-call shift: get an alert at 2 AM, SSH into a box, grep through logs, figure out which service is screaming, message the team on Discord with what I found.

None of this required creativity. None of it required judgment that couldn't be codified. It was pure process — the kind of work that makes you feel busy without being productive.

I was already using Claude Code for my development cycles — it's become core to how I build software. But since I'm on the Claude Max subscription, I can't use Claude APIs directly. So I loaded $20 onto my OpenRouter account and pointed Open Claw at a cheap model for these automations. The tasks don't need frontier intelligence — bookmark categorization and log parsing work fine with smaller models. What I wanted was a push-based agent that runs on my infrastructure, on my schedule, doing things I'd otherwise have to remember to do.

Why Self-Hosted

Three reasons:

1. Privacy. My monitoring logs contain service names, error traces, and infrastructure details. I'm not piping that through a third-party hosted AI without controlling the environment.

2. Always-on. I don't want to open a browser tab and paste things in. I want the agent running 24/7, triggered by crons and alerts, doing its job whether I'm awake or not.

3. Cost. A $4/month Digital Ocean droplet running Open Claw costs less than a single month of most AI SaaS tools. I bring my own API keys.

Why Open Claw

Open Claw checked the boxes: open source, self-hostable via Docker, supports multiple LLM backends, has a clean API for programmatic access, and plays well with external tools. It's not the only option — there's LibreChat, LobeChat, and a dozen others — but Open Claw's function calling and tool-use support made it the best fit for automation workflows.

Setting Up Open Claw on Digital Ocean

The setup is surprisingly straightforward. Here's what I'm running:

Droplet Specs

• Size: $4/month (1 vCPU, 512MB RAM, 10GB SSD)

• Region: Bangalore (BLR1) — closest to me

• OS: Ubuntu 22.04 LTS

• Stack: Docker Compose (Open Claw + N8N) · Tailscale for access

Installation

I went with a basic Ubuntu flavor — no fancy pre-built images, no marketplace one-click installs. Just a clean Linux box and a terminal.

I kept a running log of every command I ran during setup — every apt install, every config file edit, every Docker command. Once everything was working, I asked Claude to compile all those commands and changes into a clean, reproducible setup report. That report lives in a GitHub repo now.

Why this matters: if this droplet dies tomorrow, or I want to move to a different cloud provider, I don't have to remember what I did three months ago. I just follow the report. Fifteen minutes and I'm back up. It's infrastructure-as-documentation — not as elegant as Terraform, but a lot more practical for a single $4 VPS.

# SSH into the droplet
ssh root@your-droplet-ip

# Install Docker
curl -fsSL https://get.docker.com | sh

# Clone and start Open Claw
git clone https://github.com/open-claw/open-claw.git
cd open-claw
cp .env.example .env
# Edit .env with your LLM API keys

docker compose up -d

What's Running

The docker-compose.yml runs two containers: Open Claw (:8080) and N8N (:5678), both bound to the Tailscale interface only. No Nginx needed — Tailscale handles encryption at the network layer. Watchtower auto-updates the containers. Total setup time: about 30 minutes.

Cost Breakdown

| Item                                 | Monthly Cost     |
|--------------------------------------|------------------|
| Digital Ocean Droplet (512MB)        | $4.00            |
| Domain (amortized)                   | ~$1.00           |
| LLM API via OpenRouter (cheap model) | ~$3-5            |
| **Total**                            | **~$8-10/month** |

Less than a Netflix subscription. For an AI agent that runs 24/7.

Locking It Down — Tailscale for Zero-Trust Access

Once everything was running, I did something that made the whole setup significantly more secure: I turned off all external connections to the droplet.

No public ports. No exposed Nginx. No open firewalls. The only way to reach my Open Claw instance is through Tailscale — a mesh VPN that creates a private network between my devices.

Here's what I did:

1. Installed Tailscale on both my MacBook and the Digital Ocean droplet.

2. Dropped all inbound firewall rules — the droplet no longer listens on any public IP.

3. Bound Open Claw and N8N to the Tailscale interface only — they're accessible via Tailscale IPs (e.g., 100.x.y.z:8080), not the public internet.

The result: my droplet is invisible to the internet. Port scans return nothing. There's no attack surface. The only device that can talk to it is my laptop, authenticated through Tailscale's WireGuard-based identity system.

This also means I don't need Nginx for TLS anymore — Tailscale handles encryption at the network layer. One less container, one less thing to maintain, one less cert to renew.

For a $4 VPS running an AI agent with access to my bookmarks, calendar, and monitoring logs — even through N8N's read-only layer — this felt like the right level of paranoia.

The N8N Layer — Why Open Claw Never Touches My Keys

This is the part I want to be very clear about, because it's the most important architectural decision I made.

Open Claw does not have any API keys, credentials, or tokens. Not for Raindrop. Not for Google Calendar. Not for my monitoring stack. Not for Discord. Nothing.

All external API calls go through N8N workflows. N8N is the only component that stores credentials, and it runs in its own Docker container with its own encrypted credential store.

How It Works

1. Open Claw decides it needs to do something (e.g., "fetch today's calendar events").

2. Open Claw calls an N8N webhook — a simple HTTP POST with a JSON payload describing the intent.

3. N8N receives the webhook, authenticates against the relevant API using its stored credentials, executes the request, and returns the result.

4. Open Claw receives processed data. Never credentials.

Why This Matters

• Open Claw gets compromised? Attacker gets zero credentials. N8N is a separate container, separate network, separate credential store.

• All access is read-only. The API tokens stored in N8N have the minimum permissions needed — read-only for Raindrop, read-only for calendar, read-only for logs. No write, no delete, no mutate.

• Every call is auditable. N8N logs every workflow execution with timestamps, payloads, and responses. If something looks wrong, I can trace exactly what happened.

Think of it as the principle of least privilege applied to AI agents. Open Claw is powerful, but it's powerful within a sandbox. It can orchestrate, but it can't authenticate.

Automation 1: Daily Bookmark Cleanup (Raindrop)

I save 10-20 bookmarks a day. Articles, GitHub repos, documentation pages, random things I find interesting. Within a week, my Raindrop "Unsorted" collection has 100+ items, half of which I've already read or are dead links.

Before Open Claw, I'd spend 15-20 minutes every morning manually cleaning this up. Now I don't.

What Happens Every Morning at 6 AM

A cron triggers an N8N workflow that kicks off the following chain:

1. Fetch all unsorted bookmarks — N8N calls Raindrop's API, pulls everything in the "Unsorted" collection.

2. Check for dead links — N8N hits each URL with a HEAD request. Anything that returns a 404, 410, or times out after 5 seconds gets flagged.

3. Detect duplicates — Open Claw normalizes URLs (strips tracking params, www prefixes, trailing slashes) and identifies duplicates across all collections.

4. Auto-categorize — Open Claw reads the page title and URL pattern and assigns each bookmark to the right collection: "Engineering," "Design," "Tools," "Reading List," etc. This is where the LLM shines — it understands that a GitHub repo about "distributed tracing" belongs in "Engineering," not "Tools."

5. Execute and report — N8N moves bookmarks to their collections, removes dead links and duplicates, and Open Claw posts a summary to Discord.

What the Discord Summary Looks Like

📋 Raindrop Daily Cleanup — Mar 12, 2026

Processed: 18 bookmarks
├── Categorized: 12 (5 Engineering, 3 Tools, 2 Reading, 2 Design)
├── Dead links removed: 3
├── Duplicates removed: 2
└── Skipped (manual review): 1

Needs your attention:
• "Building a Real-Time..." — couldn't determine category (ambiguous)

Time taken: 14 seconds

Fifteen minutes of daily busywork, gone. Every morning I wake up to a clean bookmark library and a one-paragraph summary of what changed.

Automation 2: Daily Planner + Calendar Nudges

This one sounds simple — "pull calendar events and make a plan" — but it's become the automation I rely on most.

Morning Plan (7 AM)

Every morning, Open Claw pulls my Google Calendar events via N8N and generates a structured daily plan. Not just a list of meetings — an actual plan:

🗓 Your Day — Mar 12, 2026 (Thursday)

08:00 - 09:30  Deep work block (no meetings)
               → Suggested: Continue the API migration (you left off at the auth middleware)

09:30 - 10:00  Standup (Engineering)
               → 3 attendees · Google Meet link attached

10:00 - 12:00  Deep work block
               → Suggested: Review open PRs (4 pending, oldest is 3 days)

12:00 - 13:00  Lunch

13:00 - 13:30  1:1 with [manager name]
               → Last week you discussed: Q2 roadmap priorities

14:00 - 16:00  Deep work block
               → Suggested: Write the tech spec for the notification service refactor

16:00 - 16:30  Team sync (Backend)
               → Agenda shared in #backend-team

17:00          End of scheduled day

The deep work suggestions are based on context from previous days — Open Claw remembers what I was working on and picks up where I left off.

Calendar Nudges (Throughout the Day)

This is the part that makes it feel like having a personal assistant. Fifteen minutes before every calendar event, Open Claw pings me on Discord:

⏰ In 15 minutes: 1:1 with [name]
📍 Google Meet: [link]
📝 Last time you discussed: Q2 roadmap priorities, hiring timeline
💡 You had a follow-up action: Share the capacity planning doc

It's not just "you have a meeting" — it's "here's the context you need to walk in prepared." The follow-up action reminder alone has saved me from showing up to 1:1s having forgotten what I promised to do.

End of Day (9 PM)

A quick summary of what was planned vs. what happened:

📊 Day Summary — Mar 12, 2026

Meetings attended: 3/3
Deep work blocks used: 2/3 (missed the 14:00-16:00 block)
Unplanned: 1 ad-hoc call (30 min)

Tomorrow's first event: Standup at 09:30

Automation 3: On-Call Monitoring That Actually Helps (PoC)

This is the big one. And I want to be upfront: this is currently in PoC mode. It works. My team and I use it. But it's not production-hardened, and I'm still iterating on the reliability.

The Problem

On-call is exhausting not because of the alerts themselves, but because of the triage ritual that follows every alert:

1. Get paged.

2. SSH into the relevant box or open the monitoring dashboard.

3. Grep through logs for the error signature.

4. Figure out which service is actually broken vs. which one is just complaining because an upstream dependency is broken.

5. Formulate a hypothesis.

6. Message the team on Discord with what you found and what you think should happen next.

Steps 2-5 are mechanical. They follow a pattern. The experienced on-call engineer is faster at them not because they're smarter, but because they've seen the patterns before and know where to look.

That's exactly what Open Claw is good at.

The Architecture

Alert (Zenduty)
    │
    ▼
Open Claw (receives alert webhook)
    │
    │ "I need logs for service X, last 30 minutes"
    ▼
N8N Workflow
    │
    │ (uses read-only credentials stored in N8N)
    ▼
Log Store (Loki/ELK/CloudWatch)
    │
    │ returns filtered logs
    ▼
N8N → Open Claw
    │
    │ analyzes logs, identifies patterns
    ▼
Discord Message
    │
    "Here's what I found, here's what I think is wrong,
     here are the suggested next steps"

The critical detail: Open Claw never authenticates against the log store directly. It sends an intent ("get me logs for service X") to an N8N workflow. N8N has the read-only credentials. N8N makes the API call. N8N returns the data. Open Claw only ever sees log text — never tokens, never connection strings, never credentials.

What Happens When an Alert Fires

1. Alert arrives — Zenduty sends a webhook to Open Claw with the alert metadata: service name, severity, error message, timestamp.

2. Open Claw triggers log retrieval — It sends a request to N8N: "Get me the last 30 minutes of logs for payment-service, filtered for ERROR and WARN levels."

3. N8N fetches logs — Using its stored read-only credentials, N8N queries the log store, filters the results, and returns them to Open Claw.

4. Open Claw analyzes — This is where the LLM does its thing. It reads the logs and:

• Identifies the error pattern (e.g., "connection refused to inventory-service on port 8443")

• Maps the dependency chain (payment-service depends on inventory-service)

• Checks if the upstream service is also alerting (often the real problem is two levels up)

• Formulates a hypothesis ("inventory-service is down, causing payment-service to fail on inventory checks")

5. Posts to Discord — A structured triage summary lands in the team's incident channel:

🚨 Alert Triage — payment-service (P2)

Triggered: 2026-03-12 02:14 AM IST
Duration: Alert active for 4 minutes

Root Cause (probable):
  inventory-service is returning connection refused on port 8443.
  payment-service depends on inventory-service for stock validation
  during checkout. All checkout attempts are failing.

Impacted Services:
  • payment-service (direct) — checkout flow broken
  • order-service (indirect) — new orders can't be created
  • notification-service (unaffected) — no dependency on inventory

Evidence:
  • 47 "connection refused" errors in payment-service logs (last 15 min)
  • inventory-service last healthy log entry: 02:10 AM
  • No deployment detected in last 2 hours

Suggested Next Steps:
  1. Check inventory-service pod status (likely crashed or OOMKilled)
  2. If OOMKilled, restart with increased memory limit
  3. If networking issue, check service mesh / DNS resolution
  4. Once inventory-service recovers, payment-service should auto-heal
     (circuit breaker will close after 3 successful health checks)

⚠️ This is an AI-generated triage. Verify before acting.

How the Team Uses It

Right now, it's me and two other engineers on the team using this. We don't blindly follow the suggestions — we treat Open Claw's triage like a junior engineer's analysis: usually directionally correct, occasionally wrong on specifics, always worth reading before diving in.

The time savings are real. Before Open Claw, the triage step alone took 10-20 minutes per alert. Now we get a structured analysis within 60 seconds of the alert firing. We skip straight to verification and action.

What's Still PoC About It

• Hallucination risk. Open Claw sometimes identifies the wrong root cause, especially for failure modes it hasn't seen before. The "verify before acting" disclaimer isn't optional.

• Log volume limits. For services that produce massive log volumes, the 30-minute window can exceed what the LLM can process. I'm working on smarter filtering.

• Single-cluster only. It currently works for one Kubernetes cluster. Multi-cluster support needs more N8N workflow plumbing.

The Honest Downsides

1. The $4 Droplet Has Limits

512MB of RAM is tight. Open Claw and N8N together use most of it at idle. During a monitoring triage (when logs are being processed), it can spike and hit swap. I'm considering upgrading to the $6 droplet (1GB RAM) but haven't needed to yet.

2. Hallucinations Are Real

Open Claw is an LLM wrapper at its core. It can — and does — get things wrong. The bookmark categorization is wrong maybe 5% of the time (fixable with a quick manual correction). The monitoring triage is wrong maybe 15-20% of the time (which is why it's still a PoC with a human-in-the-loop).

3. Self-Hosting Means Self-Maintaining

Docker containers crash. N8N workflows break when APIs change their response format. I spend maybe 30 minutes a month on maintenance — not bad, but not zero.

4. The Calendar Planner Doesn't Know What I Actually Did

It can tell me what was scheduled, but it can't tell me if I actually did the deep work I planned. It's a planner, not a tracker. I still rely on my own discipline for execution.

5. N8N Adds Complexity

An extra container, extra configuration, extra failure surface. But the security isolation is worth it. I'd rather debug an N8N workflow than worry about my monitoring credentials sitting inside an LLM's context window.

Key Takeaways

1. Self-hosting an AI agent is surprisingly accessible. A $4 droplet, Docker Compose, and 30 minutes of setup. You don't need a GPU. You don't need Kubernetes. You need a VPS and docker compose up.

2. Never give your AI agent direct access to credentials. Use N8N (or any workflow tool) as a middleware layer. Open Claw sends intents, N8N executes authenticated calls. Credentials never leave N8N's vault.

3. Start with low-stakes automations. Bookmark cleanup was my first automation because the worst case was a miscategorized link — not a production incident. Build trust in the system before giving it anything consequential.

4. Read-only credentials are your friend. Every API token stored in N8N has the minimum permissions needed. Even if everything goes wrong, the blast radius is "it read some data it shouldn't have," not "it deleted my production database."

5. The monitoring PoC saves real time. 10-20 minutes of manual triage per alert, reduced to 60 seconds of reading an AI-generated summary. Multiply that by 3-4 alerts per on-call shift, and the math works out fast.

6. Human-in-the-loop is not optional for high-stakes automations. Bookmark cleanup can be fully autonomous. On-call triage cannot. Open Claw suggests, humans decide. That boundary is load-bearing.

7. The $10/month total cost pays for itself in the first week. Between bookmark cleanup (15 min/day), daily planning (10 min/day), and on-call triage (10-20 min/alert), the time savings are measured in hours per week.

Running on a Digital Ocean droplet that costs less than a coffee. If you want to try something similar, start with the bookmark automation — it's the easiest win and the lowest risk.

Connect with me at [mrinal.dev](https://mrinal.dev). I'm always happy to talk about self-hosted AI, automation workflows, and engineering productivity.

The Problem That Started It All

Here's a cycle every competitive programmer knows too well:

1. Spend 45 minutes solving a dynamic programming problem on LeetCode.

2. Feel great about it. Move on.

3. See the same pattern three weeks later in a contest. Draw a complete blank.

4. Re-solve it from scratch. Again.

I was stuck in this loop for months. I'd solve 200+ problems but could barely recall the approaches for half of them. The issue wasn't that I wasn't practicing — it was that I wasn't revisiting at the right time.

I tried maintaining spreadsheets. I tried setting calendar reminders. I even tried sticky notes on my monitor. Nothing stuck — ironic, given that the whole goal was to make things stick.

---

What Inspired Me to Build Revise

Three things converged:

1. The Forgetting Curve Hit Me Hard

In 1885, Hermann Ebbinghaus demonstrated that we forget roughly 70% of newly learned information within 24 hours if we don't review it. But — and this is the key insight — each review at the right moment makes the memory decay slower. The optimal review schedule isn't "every day" or "every week." It's a curve that stretches out exponentially: 1 day, then 6 days, then 15 days, then 35 days, and so on.

I already knew about this from using Anki for medical terminology during college. But I never connected it to coding until I realized the pattern: solving problems and forgetting them was literally the Ebbinghaus curve playing out in real-time.

2. Anki Was Too Much Friction

Anki is incredible for flashcards. But for coding problems, it's clunky. I didn't want to create a flashcard for every LeetCode problem I solved. I didn't want to write the "front" and "back" of a card. I just wanted to:

Solve a problem

Click a button

Have the system figure out when I should revisit it

That's it. No card creation. No manual data entry. Just track and schedule.

3. I Wanted It Where I Already Was

I didn't want to switch to a separate app every time I finished a problem. I wanted the tracking to happen right there — on LeetCode, on Codeforces, on HackerRank. A browser extension that auto-detects what I'm working on felt like the only workflow that wouldn't get abandoned after a week.

So I built Revise.

---

What Revise Actually Does

Revise is two things:

1. A browser extension (Chrome and Safari) that sits in your toolbar and lets you track anything you study with one click.

2. A web dashboard that shows you what's due for revision, your stats, and your full learning history.

The extension auto-detects 10+ learning platforms — LeetCode, Codeforces, HackerRank, CodeChef, GeeksForGeeks, InterviewBit, AtCoder, NeetCode, AlgoMonster, and DesignGurus. Navigate to any problem on these sites, and the extension picks up the URL and title automatically.

But here's the thing — it's not just for coding. Add any custom platform from the dashboard settings, and you can track Khan Academy lessons, design tutorials, language exercises, whatever you're learning. The SM-2 algorithm doesn't care what the subject is. It only cares about how well you remembered it.

---

The Science Behind It — SM-2 Algorithm

Revise uses the SM-2 algorithm — the same algorithm that powers Anki and SuperMemo, two of the most successful spaced repetition tools ever built.

Here's how it works in plain English:

1. You rate your recall from 1 to 5 stars after revisiting something:

⭐ (1) — Complete blackout. Couldn't recall anything.

⭐⭐ (2) — Struggled heavily. Got it wrong.

⭐⭐⭐ (3) — Got it right, but with significant effort.

⭐⭐⭐⭐ (4) — Got it right with some hesitation.

⭐⭐⭐⭐⭐ (5) — Perfect recall. Effortless.

2. The algorithm adjusts two things:

Interval — how many days until your next review. Starts at 1 day, then 6 days, then grows exponentially based on your ratings.

Easiness Factor (EF) — a per-item difficulty multiplier. Items you consistently rate low get reviewed more frequently. Items you find easy gradually space out to weeks, then months.

3. If you rate 1 or 2 stars (quality < 3 in SM-2 terms), the algorithm resets the interval to 1 day. You start over. This is harsh but effective — it ensures you don't advance material you haven't actually internalized.

The math behind the easiness factor update:

EF' = EF + (0.1 - (5 - q) * (0.08 + (5 - q) * 0.02))

Where `q` is the quality rating (0-5) and `EF` never drops below 1.3. This formula ensures that the EF converges towards a stable value that reflects the true difficulty of that specific item for you.

The result: things you find easy come back in 30, 60, 90 days. Things you struggle with come back tomorrow. Over time, your entire library of learned material is maintained with minimal daily effort.

---

How to Get Started (Under 60 Seconds)

Step 1: Sign Up

Go to revise.mrinal.dev and click Get Started Free.

Enter your email. No password needed — Revise uses magic link authentication. You'll get an email with a link. Click it. You're in.

No credit card. No trial period. No catch. It's free and open source.

Step 2: Install the Browser Extension

Chrome:

1. Download `extension.zip` from GitHub

2. Unzip it

3. Go to `chrome://extensions`

4. Enable Developer mode (top right)

5. Click Load unpacked and select the unzipped folder

6. Pin the extension from the puzzle icon in your toolbar

Safari: Available on request — reach out at dmrinal626@gmail.com for the build.

Step 3: Start Tracking

That's it. Navigate to any supported platform, click the extension, and you're tracking.

---

Using the Browser Extension

The extension is designed to get out of your way. Here's the workflow:

1. Navigate to a Problem

Go to any problem on a supported platform. The extension auto-detects the platform, URL, and page title.

2. Start the Timer

Click Start Timer when you begin working on the problem. The timer runs in the background — you can switch tabs, take breaks, whatever. Pause and resume as needed.

3. Rate and Save

When you're done, stop the timer. The extension asks you to:

Set the difficulty (Easy / Medium / Hard)

Rate your recall (1-5 stars) — this feeds directly into the SM-2 algorithm

Add notes (optional) — jot down the key insight, the pattern used, or a reminder for future you

Hit Save. The SM-2 algorithm immediately calculates when you should revisit this problem and schedules it.

4. Check What's Due

The extension badge shows how many items are due for revision today. Click it to see the list. When you revisit a problem, rate your recall again — the algorithm adjusts and reschedules.

---

The Dashboard — Your Learning Command Center

The dashboard at revise.mrinal.dev/dashboard gives you the full picture:

What You'll See:

Items Tracked — total number of problems/topics you're tracking

Difficulty Breakdown — pie chart of Easy / Medium / Hard distribution

Platform Distribution — how many items from each platform

Revision Schedule — upcoming reviews charted over time

Daily Activity — your study streak and daily patterns

Full History Table — filterable, sortable table of everything you've ever tracked, with next review dates, ratings, time spent, and notes

Revising from the Dashboard

The dashboard highlights items that are due today. Click on any item to revisit it — the original URL opens in a new tab. After you've reviewed it, update your recall rating directly from the dashboard. The SM-2 algorithm recalculates and updates the schedule.

CSV Export

Need your data elsewhere? Export your entire history as a CSV with one click.

---

Adding Custom Platforms

Revise auto-detects 10+ platforms out of the box, but you're not limited to those. From the dashboard settings, you can add any website:

1. Go to Settings in the dashboard

2. Click Add Custom Platform

3. Enter a name (e.g., "Khan Academy") and a URL pattern (e.g., `khanacademy.org`)

4. Save

Now when you visit Khan Academy and click the extension, it auto-detects the platform name, just like the built-in ones.

This makes Revise useful beyond coding. Track:

Math exercises on Khan Academy

Design tutorials on Figma Learn

Language lessons on Duolingo or Pimsleur

System design articles on your favorite blog

Literally anything with a URL

---

The Tech Stack

For the curious, here's what powers Revise:

Browser Extension (Chrome / Safari)
        |
        |  REST API
        v
   FastAPI Server  →  Supabase (Postgres + Auth)
        |
        v
   Web Dashboard (Jinja2 templates)

Backend: Python + FastAPI — lightweight, async, and fast

Database: Supabase (managed PostgreSQL) with Row Level Security — each user's data is completely isolated

Auth: Supabase Magic Links — no passwords to store, no password resets to build

Frontend: Server-rendered Jinja2 templates — no SPA framework, no build step, just HTML/CSS/JS

Extension: Vanilla JavaScript — no React, no Webpack, just a popup that does its job

Hosting: The dashboard runs on Vercel/Docker, the API on any container platform

I deliberately kept the stack simple. No Next.js. No Redux. No GraphQL. The entire server is a single `main.py` file with a few helper modules. The extension is under 500 lines of JavaScript. Simple tools, used well.

---

Self-Hosting for Developers

Revise is fully open source under the MIT license. If you want to self-host:

1. Clone the repo:

git clone https://github.com/the-mrinal/code-revision-tracker.git

2. Set up Supabase: Create a free project at supabase.com and run the SQL schema from the README.

3. Configure environment variables:

SUPABASE_URL=your-project-url
   SUPABASE_KEY=your-anon-key
   SUPABASE_SERVICE_KEY=your-service-key

4. Run with Docker:

docker-compose up

5. Update the extension to point to your server URL.

Full setup instructions are in the GitHub README.

---

What's Next

Revise is already useful — I use it daily and it's changed how I retain information from coding practice. But there's more I want to build:

Mobile app — check what's due and do quick reviews on the go

Collaborative revision — share revision lists with study groups

AI-powered hints — when you revisit a problem, get a nudge about the approach without seeing the full solution

More analytics — retention rate over time, weakest topics, optimal study times

Firefox extension — because not everyone uses Chrome

---

Try It

If you're tired of forgetting what you've learned — whether it's coding problems, math concepts, or anything else — give Revise a shot.

[revise.mrinal.dev](https://revise.mrinal.dev) — sign up in 10 seconds, no password, no payment.

[GitHub](https://github.com/the-mrinal/code-revision-tracker) — star the repo, open an issue, or contribute.

The best time to start retaining what you learn was yesterday. The second best time is today.

---

Built by Mrinal. If you found this useful, share it with someone who's stuck in the solve-forget-repeat loop.

Table of Contents

1. The Origin Story

2. The Three Properties — Precisely Defined

3. Why "Pick 2 of 3" Is a Dangerous Oversimplification

4. Deep Dive: Google Cloud Firestore (CP)

5. Deep Dive: Apache Cassandra (AP with Tunable Consistency)

6. Deep Dive: Redis Cluster (CP-Biased, but Leaky)

7. Side-by-Side Comparison

8. The PACELC Extension — What CAP Doesn't Tell You

9. Choosing the Right System for Your Use Case

10. Key Takeaways

11. References

The Origin Story

In autumn 1998, Eric Brewer — then a professor at UC Berkeley — introduced what he called the CAP principle. He formally presented it as a conjecture at the ACM Symposium on Principles of Distributed Computing (PODC) in 2000. Two years later, in 2002, Seth Gilbert and Nancy Lynch at MIT published a formal mathematical proof, transforming Brewer's conjecture into a full-blown theorem.

The theorem made a deceptively simple claim: a distributed data store cannot simultaneously provide more than two out of three guarantees — Consistency, Availability, and Partition Tolerance.

This one sentence reshaped how an entire generation of engineers thought about distributed systems. It also led to one of the most widespread misunderstandings in computer science.

The Three Properties — Precisely Defined

Before we can understand the trade-offs, we need to be precise about what each property actually means. The colloquial definitions that float around blog posts and interview prep sites are often subtly wrong, and those subtle errors lead to real architectural mistakes.

Consistency (C) — Linearizability

Consistency in CAP is not the same "C" as in ACID. ACID consistency means a transaction moves the database from one valid state to another. CAP consistency is far more specific.

CAP Consistency means linearizability: all operations appear to execute instantaneously at some point between their invocation and their response, and that order is consistent across all nodes. In practical terms:

• If Client A writes balance = 500 and gets an acknowledgment, then Client B reading balance from any node must see 500 (or a later value).

• There is a single, globally agreed-upon order of operations.

• There are no "stale reads." Ever.

This is an extremely strong guarantee. Most systems that claim to be "consistent" are actually providing weaker models like sequential consistency, causal consistency, or read-your-writes consistency — none of which satisfy CAP's definition.

Availability (A) — Every Non-Failing Node Responds

Availability in CAP means every request received by a non-failing node must result in a response. Not an error. Not a timeout. A response with data.

Crucially, the response doesn't have to contain the latest data. An available system can return stale data. It just can't return nothing or an error.

This definition is also stronger than it sounds. It means:

• You cannot redirect the client to another node (that would be a form of unavailability for the original node).

• You cannot return an error code saying "try again later."

• You cannot hang indefinitely waiting for the partition to heal.

Partition Tolerance (P) — Surviving Network Splits

A network partition occurs when communication between two groups of nodes is completely lost. Messages sent between the groups are either dropped or delayed indefinitely.

Partition Tolerance means the system continues to function (in whatever degraded mode it chooses) when such a partition occurs. It doesn't crash. It doesn't corrupt data. It makes a deliberate choice about how to handle the situation.

Here's the part most people miss: partition tolerance is not optional. In any system deployed across multiple machines, network partitions will happen. Switches fail. Cables get cut. Cloud availability zones lose connectivity. The question is never "should we tolerate partitions?" — it's "what do we sacrifice when a partition occurs?"

This is why the real CAP trade-off is: when a partition happens, do you sacrifice Consistency or Availability?

Why "Pick 2 of 3" Is a Dangerous Oversimplification

You've probably seen the CAP Venn diagram — three overlapping circles labeled C, A, and P, with systems placed in the overlapping regions: CP, AP, or CA. This model is not just simplistic — it's actively misleading. Here's why:

1. The Trade-off Only Applies During Partitions

During normal operation (no partitions), a well-designed system can provide all three properties simultaneously. Firestore gives you strong consistency and high availability and partition tolerance — right up until a partition actually occurs.

The CAP theorem only constrains behavior during a partition event. Since partitions are (thankfully) rare in well-maintained infrastructure, a system spends the vast majority of its life in a state where CAP doesn't force any trade-off at all.

2. The Choice Is Granular, Not Global

A single system doesn't have to make one permanent choice. Different subsystems, different data types, even different queries can make different trade-offs:

• Your user authentication might be strongly consistent (CP behavior) because showing the wrong session state is dangerous.

• Your activity feed might be eventually consistent (AP behavior) because a 2-second delay in seeing a new post is harmless.

• Your shopping cart might use session-level consistency — consistent within one user's session, eventually consistent across the system.

Cassandra embodies this perfectly with per-query consistency levels.

3. Brewer Himself Corrected the Record

In 2012, twelve years after his original conjecture, Brewer published a retrospective article in IEEE Computer titled "CAP Twelve Years Later: How the 'Rules' Have Changed." In it, he explicitly stated:

• The "2 of 3" framing is misleading because it oversimplifies the tension among the three properties.

• Partitions are rare, so there is little reason to forfeit C or A when the system is not partitioned.

• The choice between C and A can occur many times within the same system at very fine granularity.

• System designers should focus on partition management and recovery techniques rather than accepting permanent trade-offs.

4. CA Systems Don't Exist in Distributed Contexts

The "CA" region of the Venn diagram (consistent + available, not partition tolerant) is essentially a single-node system or a system that gives up when a partition occurs. A traditional single-server PostgreSQL instance is "CA" — but the moment you distribute it, you must handle partitions, pushing you into CP or AP territory.

Deep Dive: Google Cloud Firestore (CP)

What Firestore Is

Firestore is Google's fully managed, serverless NoSQL document database. It evolved from the Google Cloud Datastore and is now the primary database for Firebase. It stores data as documents organized into collections, with support for subcollections, rich querying, and real-time listeners.

Architecture

Single-Region Deployment

In single-region mode, Firestore distributes your data across multiple zones within a single Google Cloud region. Each data "split" (a partition of your dataset) is replicated across zones.

The replication is managed by the Paxos consensus algorithm — the same algorithm that underpins many of Google's internal systems (Chubby, Spanner, Megastore). Paxos ensures that a majority of replicas must agree before a write is committed.

Each split has its own Paxos leader, and leaders can be in different zones. This means there's no single point of failure at the zone level.

SLA: 99.99% availability.

Multi-Region Deployment

Multi-region Firestore distributes data across three locations:

1. Region A — Full read-write replica of the entire database.

2. Region B — Full read-write replica of the entire database.

3. Region C — A witness replica that participates in Paxos voting but doesn't store a full copy of the data.

The witness replica is what makes this clever. It breaks ties in Paxos voting without the cost of maintaining a third full replica. The system can survive the complete loss of any single region.

SLA: 99.999% availability (five nines — roughly 5 minutes of downtime per year).

Trade-off: Multi-region writes are significantly slower because the write must be committed across geographically distant regions before acknowledgment.

Consistency Model: Strong (Serializable)

Firestore provides serializable isolation — the strongest isolation level. This means:

• All transactions appear to execute in some serial order.

• Each transaction sees a consistent snapshot of the database.

• Reads always return the most recently committed value.

This is achieved through Paxos-based synchronous replication. A write is not acknowledged to the client until a majority of replicas have durably committed it. This means:

• No stale reads.

• No dirty reads.

• No phantom reads.

• No lost updates.

Every transaction gets a commit timestamp, and Firestore guarantees that operations are serialized in commit-timestamp order.

What Happens During a Partition

When a network partition separates Firestore replicas, the system's CP nature becomes visible:

1. Writes may become unavailable: If the Paxos leader for a data split can't reach a majority of replicas (because they're on the other side of the partition), it cannot commit writes. The write request will timeout or return an error.

2. Reads from the majority partition succeed: If you're on the side of the partition that has the Paxos majority, reads and writes continue normally.

3. The minority side becomes unavailable: Nodes on the minority side of the partition cannot form a Paxos majority. They will refuse to serve reads (because they can't guarantee the data is current) and refuse to accept writes (because they can't commit them).

4. No data divergence: Because the minority side refuses to operate, there's never a situation where both sides of the partition accept conflicting writes. When the partition heals, there's nothing to reconcile.

This is the essence of a CP system: correctness over responsiveness. Firestore would rather give you an error than give you wrong data.

Multi-Region Conflict Handling

In multi-region deployments, if the same document is edited concurrently in different regions (a rare edge case that can occur briefly), Firestore uses last-write-wins (LWW) resolution based on commit timestamps. However, this is minimized by the Paxos protocol — true concurrent writes to the same document are serialized before they can conflict.

Real-World Implications

When to choose Firestore's approach:

• Financial applications where showing a wrong balance is worse than showing an error.

• Inventory systems where overselling (due to stale reads) has real cost.

• Authentication/authorization where stale permission data is a security risk.

• Any system where "temporarily unavailable" is better than "silently wrong."

What you sacrifice:

• Write latency in multi-region setups (cross-region Paxos rounds).

• Availability during partitions (affected clients get errors).

• Throughput on heavily-contested documents (serialization bottleneck).

Deep Dive: Apache Cassandra (AP with Tunable Consistency)

What Cassandra Is

Apache Cassandra is a distributed wide-column store originally developed at Facebook for inbox search, later open-sourced in 2008. It's designed for high write throughput, horizontal scalability, and continuous availability across multiple data centers. It has no single point of failure by design.

Architecture

Fully Peer-to-Peer

Unlike Firestore (which has Paxos leaders) or Redis Cluster (which has master nodes), Cassandra is fully decentralized. Every node is identical in function. There is no master, no leader, no coordinator that the system depends on. Any node can accept any request for any piece of data.

When a client sends a request, the node that receives it becomes the coordinator for that request. The coordinator determines which nodes hold replicas of the requested data (using consistent hashing) and forwards the request to them.

The Gossip Protocol

Nodes discover and monitor each other through an epidemic gossip protocol:

• Every 1 second, each node initiates a gossip round.

• It selects up to 3 random nodes to exchange state information with.

• The exchanged information includes: cluster membership, node liveness, load information, and schema versions.

• Information propagates exponentially — in a 100-node cluster, all nodes converge on new information within seconds.

This peer-to-peer gossip is how Cassandra avoids needing a centralized membership service. It's also how nodes detect failures: if a node hasn't been heard from in a configurable period, gossip marks it as "down."

Data Distribution: Consistent Hashing

Cassandra uses consistent hashing with virtual nodes (vnodes) to distribute data:

1. Each piece of data has a partition key.

2. The partition key is hashed to a position on a "ring" (the hash space 0 to 2^63 -1).

3. Each node owns a range of the ring.

4. The data is stored on the node that owns the range containing the hash value.

5. Replicas are placed on subsequent nodes in the ring, governed by the replication factor (RF).

With RF=3, each piece of data exists on three different nodes. Which three depends on the replication strategy — SimpleStrategy for single-datacenter, NetworkTopologyStrategy for multi-datacenter (where you specify RF per datacenter).

Consistency Model: Tunable Consistency

This is Cassandra's most powerful — and most dangerous — feature. You choose the consistency level per operation.

Consistency Levels Explained

For writes:

| Level          | Nodes That Must Acknowledge        | Behavior                                                                                                |
|----------------|------------------------------------|---------------------------------------------------------------------------------------------------------|
| `ANY`          | 1 (including hinted handoff)       | Fastest. Write succeeds even if all replicas are down (hint stored on coordinator). Weakest durability. |
| `ONE`          | 1 replica                          | Fast. Minimal durability guarantee.                                                                     |
| `TWO`          | 2 replicas                         | Moderate.                                                                                               |
| `THREE`        | 3 replicas                         | Slower but more durable.                                                                                |
| `QUORUM`       | ⌊RF/2⌋ + 1 replicas                | The sweet spot for most use cases.                                                                      |
| `LOCAL_QUORUM` | Quorum within the local datacenter | Avoids cross-DC latency.                                                                                |
| `EACH_QUORUM`  | Quorum in each datacenter          | Strong cross-DC guarantee.                                                                              |
| `ALL`          | Every replica                      | Slowest. Any single replica failure means the write fails.                                              |

For reads:

The same levels apply. ONE means only one replica needs to respond. QUORUM means a majority must respond and the most recent value (by timestamp) is returned.

The Strong Consistency Formula

Cassandra can provide strong consistency (equivalent to linearizability for single-key operations) if:

R + W > RF

Where:

• R = read consistency level (number of nodes that must respond to a read)

• W = write consistency level (number of nodes that must acknowledge a write)

• RF = replication factor (total number of replicas)

Example with RF=3:

• QUORUM reads (R=2) + QUORUM writes (W=2): 2 + 2 = 4 > 3. Strong consistency.

• ONE reads (R=1) + ALL writes (W=3): 1 + 3 = 4 > 3. Strong consistency (fast reads, slow writes).

• ALL reads (R=3) + ONE writes (W=1): 3 + 1 = 4 > 3. Strong consistency (slow reads, fast writes).

• ONE reads (R=1) + ONE writes (W=1): 1 + 1 = 2 < 3. Eventually consistent. You might read stale data.

Why this formula works: If W nodes acknowledged a write and R nodes are queried for a read, and R+W > RF, then at least one node must be in both the write set and the read set. That node has the latest value. Cassandra picks the most recent value (by timestamp) from the responding nodes, so the latest write is always returned.

The Per-Query Trade-off in Action

This is where Cassandra's design shines for real applications. Consider an e-commerce platform:

// Writing a new order — use QUORUM for durability
INSERT INTO orders (...) VALUES (...) USING CONSISTENCY QUORUM;

// Reading order status for the customer who just placed it — use QUORUM for freshness
SELECT * FROM orders WHERE order_id = ? USING CONSISTENCY QUORUM;

// Reading product recommendations — use ONE for speed (stale data is fine)
SELECT * FROM recommendations WHERE user_id = ? USING CONSISTENCY ONE;

// Writing analytics events — use ANY for maximum throughput
INSERT INTO analytics (...) VALUES (...) USING CONSISTENCY ANY;

Same cluster, same data model, four different consistency trade-offs based on business requirements.

The Three Repair Mechanisms

Because Cassandra allows writes to proceed without all replicas acknowledging them, replicas can (and do) drift out of sync. Three mechanisms bring them back:

1. Hinted Handoff (Write-Path Repair)

When a write is destined for a replica that's currently unreachable:

1. The coordinator stores the write as a "hint" in its own local storage.

2. The hint contains the full mutation data and the target node's identity.

3. When the coordinator detects (via gossip) that the target node is back, it replays the hint.

Limitations:

• Hints are stored for a maximum of 3 hours by default (max_hint_window_in_ms).

• If the target node is down for longer than that, hints are discarded and the data gap remains.

• Hints consume disk space on the coordinator node.

• Hinted handoff is "best effort" — it doesn't guarantee repair.

2. Read Repair (Read-Path Repair)

When a read query contacts multiple replicas and they return different versions of the data:

1. Cassandra compares timestamps across the responding replicas.

2. It returns the newest version to the client.

3. In the background, it sends the newest version to any replicas that had stale data.

This means every read is also a potential repair operation. However, it only repairs data that's actually being read — cold data that nobody queries stays inconsistent.

3. Anti-Entropy Repair (Background Full Repair)

This is the heavyweight mechanism. Run manually via nodetool repair:

1. Nodes build Merkle trees — hash trees where each leaf represents a range of data.

2. Nodes exchange and compare Merkle trees.

3. Differences are identified and the most recent version is propagated.

This is the only mechanism that guarantees all data across all replicas eventually converges. It's computationally expensive and should be run periodically (Cassandra documentation recommends at least once within the gc_grace_seconds window — default 10 days — to prevent zombie data resurrection from deleted records).

What Happens During a Partition

This is where Cassandra's AP nature is visible:

1. Both sides of the partition continue operating. Nodes accept reads and writes regardless of whether they can communicate with nodes on the other side.

2. Data diverges. If the same row is written on both sides of the partition, both writes succeed (no coordination is possible). This creates conflicting versions.

3. Conflict resolution: Last Write Wins (LWW). When the partition heals and replicas sync up, conflicts are resolved by timestamp. The write with the latest timestamp wins. The "losing" write is silently discarded.

4. Clock synchronization matters. Because LWW depends on timestamps, Cassandra assumes NTP (Network Time Protocol) is running on all nodes. Clock skew can cause the "wrong" write to win. In practice, this means running NTP is not optional — it's a critical infrastructure requirement.

5. Lightweight Transactions (LWT) for CP behavior. For operations that absolutely require linearizability (like a unique username check), Cassandra offers LWT using the Paxos protocol. But LWT is 4x slower than regular writes due to the Paxos round trips and should be used sparingly.

Real-World Implications

When to choose Cassandra's approach:

• High write throughput requirements (IoT sensor data, event logging, time-series data).

• Multi-datacenter or multi-region deployments where you need writes to succeed everywhere.

• Use cases where "always writable" is more important than "always correct."

• Systems with clear consistency requirements that vary by operation.

What you sacrifice:

• No strong consistency by default (you must configure it deliberately).

• LWW conflict resolution can silently lose data during partitions.

• Operational complexity — you must run anti-entropy repairs, manage tombstones, tune consistency levels.

• Lightweight Transactions are expensive and limited.

Deep Dive: Redis Cluster (CP-Biased, but Leaky)

What Redis Cluster Is

Redis Cluster is Redis's built-in horizontal scaling solution, available since Redis 3.0 (2015). It automatically shards data across multiple Redis master nodes, with each master having one or more replica nodes for failover.

Unlike Firestore (managed service) or Cassandra (peer-to-peer), Redis Cluster follows a master-replica architecture with gossip-based failure detection.

Architecture

Sharding with Hash Slots

Redis Cluster divides the keyspace into 16,384 hash slots. Each master node owns a subset of these slots.

When a client issues a command:

1. The key is hashed using CRC16.

2. The hash is taken modulo 16,384 to determine the slot.

3. The command is routed to the master that owns that slot.

4. If the client connects to the wrong node, it receives a MOVED redirect to the correct one.

Replication Model: Asynchronous

This is the single most important architectural detail of Redis Cluster, and it's the source of both its speed and its consistency weakness.

When a write arrives at a master:

1. Master executes the write in memory.

2. Master sends acknowledgment to the client.

3. Master asynchronously replicates the write to its replicas.

Step 2 happens before step 3. The client is told "write succeeded" before any replica has seen the data. This makes writes extremely fast but creates a window of vulnerability.

Gossip Protocol

Similar to Cassandra, Redis Cluster nodes communicate via gossip:

• Nodes periodically ping random other nodes.

• They exchange cluster state: which nodes are alive, which slots are assigned where, which nodes are failing.

• The gossip port is the data port + 10000 (e.g., if data port is 6379, gossip port is 16379).

What Happens During a Partition

Phase 1: Detection

When a master node can't reach another node, it waits for cluster-node-timeout (commonly configured at 15,000ms / 15 seconds) before marking it as potentially failed (PFAIL).

When enough nodes agree that a node is unreachable (via gossip), it's marked as FAIL.

Phase 2: Majority vs. Minority

The cluster determines which side of the partition has the majority of master nodes:

• Majority partition: Continues operating. Clients connected to this side experience no disruption (for slots owned by masters in this partition).

• Minority partition: Stops accepting writes. Masters on this side detect that they can't reach a majority and begin rejecting write operations after cluster-node-timeout.

This is the CP behavior: the minority side sacrifices availability to prevent split-brain.

Phase 3: Failover (If a Master Is Lost)

If a master is on the minority side (or completely down), the majority side initiates failover:

1. Replicas of the lost master detect the failure.

2. A replica election begins. Replicas request votes from remaining masters.

3. A replica needs votes from a majority of masters to win the election.

4. The winning replica promotes itself to master and takes ownership of the lost master's hash slots.

5. The cluster announces the new topology via gossip.

Timing: Failover typically completes within cluster-node-timeout + 1-2 seconds for the election.

The Data Loss Problem

Because of asynchronous replication, Redis Cluster has a well-documented data loss window. Here's the exact scenario:

Time T0: Client sends WRITE to Master A
Time T1: Master A executes WRITE, stores in memory
Time T2: Master A sends OK to Client  ← Client thinks write is safe
Time T3: Master A begins replicating to Replica A1
Time T4: Master A crashes before replication completes  ← Write is lost
Time T5: Replica A1 is promoted to master (WITHOUT the write from T1)

The write acknowledged at T2 is permanently lost. The client received confirmation, but the data never made it to any surviving node.

How Large Is This Window?

The window is typically tiny — milliseconds between T2 and T3. But under heavy load or network congestion, it can grow. And when it bites, it bites hard: the client has no way to know the write was lost.

Mitigation Strategies

1. The WAIT Command

SET user:123:balance 500
WAIT 1 5000

WAIT numreplicas timeout blocks the client until the specified number of replicas have acknowledged the write, or the timeout (in milliseconds) expires.

Caveat: WAIT reduces the data loss window but doesn't eliminate it entirely. Even if WAIT succeeds, the replica could crash before the next persistence cycle. WAIT also doesn't make the write transactional — it's an acknowledgment that the replica received the data, not that it's durably persisted.

2. min-replicas-to-write Configuration

min-replicas-to-write 1
min-replicas-max-lag 10

This tells the master to refuse writes if fewer than min-replicas-to-write replicas are connected with a replication lag of at most min-replicas-max-lag seconds.

This prevents the master from accepting writes when it's isolated (no replicas reachable), but it's a blunt instrument — it makes the master unavailable when replicas are unreachable, shifting the system further toward CP behavior.

3. CLUSTER FAILOVER (Manual Failover)

For planned maintenance, the CLUSTER FAILOVER command performs a safe failover:

1. The replica tells the old master to stop accepting writes.

2. The replica waits until it has processed all data from the master's replication stream.

3. Only then does the replica promote itself.

This guarantees zero data loss but only works for planned operations, not crashes.

The Split-Brain Scenario

Despite Redis Cluster's CP bias, a split-brain can occur in edge cases:

1. Network partitions a cluster into two halves of equal size (even number of masters).

2. Neither side has a clear majority.

3. If cluster-require-full-coverage is set to no, both sides might continue serving reads for their local slots.

Mitigation: Always maintain an odd number of master nodes. This ensures one side always has a clear majority.

Real-World Implications

When to choose Redis Cluster's approach:

• Caching layers where speed is paramount and occasional data loss is tolerable.

• Session stores where sessions can be regenerated.

• Rate limiters and counters where approximate counts are acceptable.

• Real-time leaderboards where losing a few score updates during a failure is acceptable.

What you sacrifice:

• Guaranteed durability for every write (async replication gap).

• Perfect consistency during failover windows.

• Simplicity — managing a Redis Cluster requires understanding hash slots, failover mechanics, and split-brain prevention.

What you should NOT use Redis Cluster for:

• Sole source of truth for financial transactions.

• Primary database for data that can't be regenerated.

• Systems where every single write must survive any failure.

Side-by-Side Comparison

| Dimension                      | Firestore                                | Cassandra                                 | Redis Cluster                                 |
|--------------------------------|------------------------------------------|-------------------------------------------|-----------------------------------------------|
| **CAP Bias**                   | CP                                       | AP (tunable toward CP)                    | CP (with caveats)                             |
| **Consistency Model**          | Strong (Serializable)                    | Eventual (Tunable per-query)              | Strong for single-key (async replication gap) |
| **Replication Protocol**       | Paxos (synchronous)                      | Gossip + async replication                | Gossip + async replication                    |
| **Replication Type**           | Synchronous                              | Asynchronous                              | Asynchronous                                  |
| **During Partition**           | Minority side becomes unavailable        | Both sides keep serving                   | Minority side stops accepting writes          |
| **Data Loss During Partition** | None                                     | Possible (LWW conflict resolution)        | Possible (unacknowledged writes)              |
| **Conflict Resolution**        | No conflicts (Paxos prevents them)       | Last-Write-Wins (LWW) by timestamp        | Last replica promoted wins                    |
| **Topology**                   | Managed (Paxos leaders per split)        | Peer-to-peer (no masters)                 | Master-replica per shard                      |
| **Write Latency (Normal)**     | Low (single-region), High (multi-region) | Very low at ONE, Moderate at QUORUM       | Very low                                      |
| **Consistency Control**        | Fixed (always strong)                    | Per-query (ONE to ALL)                    | Fixed (async + WAIT for tuning)               |
| **Failure Detection**          | Managed by Google                        | Gossip protocol (1s intervals)            | Gossip + cluster-node-timeout                 |
| **Repair Mechanism**           | Paxos consensus (automatic)              | Hinted handoff, read repair, anti-entropy | Replica promotion + full resync               |
| **Operational Overhead**       | None (fully managed)                     | High (repairs, tuning, tombstones)        | Medium (slot management, failover)            |

The PACELC Extension — What CAP Doesn't Tell You

CAP tells you what happens during partitions. It says nothing about normal operation. But systems make trade-offs all the time, not just during failures.

In 2010, Daniel Abadi introduced the PACELC theorem in a blog post (later formally published in a 2012 paper) to fill this gap:

If there is a Partition (P), how does the system trade off between Availability and Consistency (A and C)? Else (E), in normal operation, how does the system trade off between Latency and Consistency (L and C)?

This gives us a much richer classification:

| System                 | If Partition (PAC)                                  | Else (ELC)                                                               | Full Classification            |
|------------------------|-----------------------------------------------------|--------------------------------------------------------------------------|--------------------------------|
| **Firestore**          | PC (chooses Consistency)                            | EC (chooses Consistency, accepts higher latency for multi-region writes) | PC/EC                          |
| **Cassandra (QUORUM)** | PA (chooses Availability at ONE, PC-like at QUORUM) | EL (chooses low Latency at ONE) or EC (chooses Consistency at QUORUM)    | PA/EL or PC/EC depending on CL |
| **Redis Cluster**      | PC (minority unavailable)                           | EL (chooses low Latency via async replication)                           | PC/EL                          |

Why PACELC Matters

Redis Cluster and Firestore are both "CP" under CAP. But they behave very differently in normal operation:

• Firestore (PC/EC): Even when everything is fine, it pays a latency cost for consistency (multi-region Paxos rounds).

• Redis Cluster (PC/EL): When everything is fine, it chooses speed (async replication), accepting a small durability risk.

CAP can't distinguish these two very different design philosophies. PACELC can.

Choosing the Right System for Your Use Case

Decision Framework

Choose a CP system (like Firestore) when:

• Incorrect data is worse than unavailable data.

• You're dealing with financial transactions, inventory counts, or access control.

• Regulatory requirements mandate strong consistency.

• Your users would rather see an error page than wrong information.

Choose an AP system (like Cassandra) when:

• Unavailability is worse than stale data.

• You need writes to succeed across multiple regions without cross-region latency.

• Your data has natural conflict resolution (timestamps, counters, CRDTs).

• Your system can tolerate and recover from temporary inconsistency.

Choose a CP-biased-but-fast system (like Redis Cluster) when:

• You need sub-millisecond latency.

• The data is either regenerable or not the primary source of truth.

• You can tolerate rare data loss during failures.

• Your use case is caching, sessions, rate limiting, or real-time counters.

Common Architectural Pattern: Use Multiple Systems

Most production systems don't rely on a single database. A common pattern:

┌─────────────────────────────────────────────────┐
│                  Application Layer               │
├──────────┬──────────────────┬───────────────────┤
│  Firestore                  │    Cassandra       │    Redis Cluster    │
│  (Source of Truth)   │    (Analytics/Logs) │    (Cache/Sessions) │
│  - User profiles     │    - Event streams  │    - API responses   │
│  - Orders            │    - Time-series    │    - Rate limits     │
│  - Account balances  │    - IoT sensor data│    - Leaderboards    │
│  Strong Consistency  │    High Write Volume│    Sub-ms Latency    │
└──────────────────────┴──────────────────────┴───────────────────────┘

Each database handles the data it's best suited for. The CAP trade-off isn't a constraint — it's a design tool.

Key Takeaways

1. The CAP theorem isn't "pick 2 of 3." It's "when a partition happens (and it will), do you sacrifice consistency or availability?" During normal operation, you can have all three.

2. Partition tolerance is mandatory. In any distributed system, network partitions are inevitable. The real engineering question is how you handle them.

3. Firestore chooses consistency at all costs. It uses Paxos for synchronous replication, guaranteeing that every read returns the latest write. During partitions, it sacrifices availability — the minority side goes offline rather than serving stale data.

4. Cassandra chooses availability by default, but lets you tune it. Its per-query consistency levels let you balance speed vs. correctness for each operation. The formula R + W > RF gives you strong consistency when you need it, and maximum throughput when you don't.

5. Redis Cluster is CP in theory but has a consistency gap in practice. Asynchronous replication creates a window where acknowledged writes can be lost. It's blazing fast but not suitable as a sole source of truth for critical data.

6. PACELC is the better framework. It captures what systems do during normal operation (latency vs. consistency trade-off), not just during partitions.

7. Real architectures use multiple systems. Match each data type to the database whose trade-offs align with that data's requirements.

References

• Brewer, Eric. "CAP Twelve Years Later: How the 'Rules' Have Changed." IEEE Computer, 2012.

• Gilbert, Seth & Lynch, Nancy. "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services." ACM SIGACT News, 2002.

• Abadi, Daniel. "Consistency Tradeoffs in Modern Distributed Database System Design." IEEE Computer, 2012.

• Google Cloud. "Understand Reads and Writes at Scale — Firestore." Firebase Documentation.

• Apache Software Foundation. "Apache Cassandra Documentation — Configuring Data Consistency."

• Redis Ltd. "Redis Cluster Specification." Redis Documentation.

• Kleppmann, Martin. Designing Data-Intensive Applications. O'Reilly Media, 2017.

If you found this useful, connect with me — I'm always happy to talk distributed systems.

Table of Contents

1. The Problem with "Vibe Coding"

2. The Workflow — End to End

3. Phase 1: Research — Understanding Before Building

4. Phase 2: Tech Spec — Claude as a Thinking Partner

5. Phase 3: Planning — Test-Driven Development with Intent

6. Phase 4: Implementation — Tests First, Code Second

7. The CLAUDE.md File — Your Project's Long-Term Memory

8. What Changed in How I Think About Development

9. The Honest Downsides

10. Key Takeaways

---

The Problem with "Vibe Coding"

There's a term floating around the AI-assisted development world — "vibe coding." Open a terminal, start prompting, let the AI write code, accept what looks right, iterate until it works. Ship it.

I tried this. For about two weeks. Here's what happened:

1. Claude would generate plausible-looking code that compiled and passed a few basic cases.

2. I'd ship it. Tests were sparse because I hadn't thought deeply about edge cases — I was "vibing."

3. Two days later, a bug would surface in production. The kind of bug that only shows up when a service handles a race condition between two concurrent gRPC calls.

4. I'd go back and realize the AI-generated code had made an assumption about the codebase that wasn't true. It didn't know that our order service processes refunds asynchronously, or that our inventory service has eventual consistency guarantees.

The problem wasn't Claude. The problem was me. I was using a powerful tool without a process.

Vibe coding is to AI-assisted development what copy-pasting from Stack Overflow was to traditional development. It works for toy problems. It falls apart the moment you're dealing with a production codebase with 40+ microservices, shared libraries, and a deployment pipeline that doesn't forgive half-baked changes.

So I built a process. And it fundamentally changed how I think about writing software.

---

The Workflow — End to End

Here's the full pipeline, from the moment a product requirement lands on my desk to the moment I open a pull request:

Each phase has a clear input, a clear output, and a clear handoff. Let me walk through each one.

---

Phase 1: Research — Understanding Before Building

A PRD lands from the product team. Let's say the requirement is: "Allow merchants to set custom refund policies per product category."

Before I write a single line of code — before I even think about a solution — I need to understand what exists. In a codebase with dozens of Go and Python microservices, this isn't trivial. The refund logic might touch the order service, the payment gateway adapter, the notification service, and a shared protobuf definition that six other teams consume.

This is where `hf-research` comes in.

What hf-research Does

`hf-research` is a custom slash command built on top of Claude Code (using the HumanLayer framework). When I invoke it, it spawns parallel sub-agents that fan out across the codebase — searching for relevant files, tracing function calls, reading configuration, and mapping dependencies.

I'll run it like this:

/hf-research How does the current refund flow work end-to-end?
Which services are involved? Where are refund policies defined?

Within minutes, I get back a structured research document that tells me:

Which files handle refund logic — with exact file paths and line numbers.

Which services communicate during a refund — the gRPC call chain, the event bus messages, the database writes.

Where refund policies are currently hardcoded or configured — maybe it's a constant in the order service, maybe it's a database table, maybe it's a feature flag.

Historical context — if there are design documents, past research notes, or architectural decisions recorded in the codebase.

Why This Matters

Without this step, I'd spend 2-3 hours manually grep-ing through the codebase, opening files, tracing imports, reading old PRs. With `hf-research`, that becomes 5-10 minutes of automated exploration followed by 15 minutes of me reading and internalizing the results.

But the real value isn't speed — it's completeness. When I manually explore a codebase, I follow my intuition. I check the places I think are relevant. I miss the edge case where the notification service also has refund-related logic because it sends different emails based on refund type. The sub-agents don't have this bias. They search everything.

Brainstorming with Claude

Once the research document is ready, I don't just read it and move on. I use it as the foundation for a conversation with Claude:

Based on this research, here's what I'm thinking:
- The current refund flow is tightly coupled to the order service
- Refund policies are hardcoded as constants
- We need to make them configurable per product category

What are the different ways we could architect this?
What are the trade-offs?

Claude has now read the entire relevant codebase. It's not hallucinating about a generic refund system — it's reasoning about our refund system, with our service boundaries, our data models, our deployment constraints. The brainstorming is grounded.

This conversation typically produces 2-3 viable approaches. I evaluate them, push back, ask follow-up questions. By the end, I have a clear mental model of the solution space.

---

Phase 2: Tech Spec — Claude as a Thinking Partner

With the research done and approaches brainstormed, I write the tech spec. This isn't optional. This isn't a formality. This is the most important document in the entire process.

What the Tech Spec Contains

Every tech spec I write follows the same structure:

1. Current Problem

What's broken, limited, or missing? Why can't the current system handle the new requirement? This section forces me to articulate the gap clearly.

2. What We're Solving

Scope the solution. What's in scope? What's explicitly not in scope? This prevents scope creep during implementation.

3. Why the Current Implementation Won't Scale

This is where the research pays off. Because I've already mapped the current system, I can explain precisely why hardcoded refund policies won't work — maybe the product catalog is growing from 12 categories to 200, or maybe a new regulatory requirement demands per-category refund windows.

4. Proposed Solution

The recommended approach, explained in detail. Data model changes, API contracts, service interactions, migration strategy.

5. Alternative Approaches

At least two other approaches I considered, with honest pros and cons for each. This shows the reviewer that I didn't just pick the first idea — I evaluated the solution space.

6. C3/C4 Diagrams

Architecture diagrams at the appropriate level. C3 (component) diagrams for how services interact. C4 (code) diagrams for complex internal flows.

How Claude Helps Write the Tech Spec

Claude doesn't write the tech spec for me. It writes it with me. The distinction matters.

I'll draft the "Current Problem" section, then ask Claude to review it: "Does this accurately describe the current refund flow based on what we found in the research?" Claude catches things I miss — maybe I forgot that the payment gateway has a 30-day refund window constraint, or that the event schema includes fields we'd need to update.

For the "Alternative Approaches" section, Claude is especially valuable. It generates approaches I wouldn't have considered — maybe a policy engine pattern, or an event-sourced approach, or leveraging an existing configuration service. I evaluate each one against our constraints (team size, timeline, operational complexity) and make the call.

For diagrams, I describe what I want and Claude generates the Mermaid or PlantUML markup. I render it, review it, iterate.

Leadership Review

The tech spec goes through a review process. Senior engineers and engineering leadership read it, ask questions, push back on architectural decisions, and either approve or request changes.

This is a human-only step. Claude doesn't attend the review meeting. But because Claude helped me think through the problem comprehensively, I walk into the review with answers to questions I might not have anticipated otherwise.

Once approved, we move to planning.

---

Phase 3: Planning — Test-Driven Development with Intent

This is where the process diverges from how most people use AI coding tools.

Most developers using AI assistants go straight from "I understand the problem" to "generate the code." I don't. I plan the test-driven development process first.

What hf-plan Does

`hf-plan` takes the approved tech spec and produces a detailed implementation plan. But it's not just a task list. It's a TDD-oriented plan — it tells you what tests to write and in what order, before describing any implementation code.

The output looks something like this:

## Implementation Plan

### Step 1: Define test cases for RefundPolicyService
- Test: CreatePolicy with valid category returns policy ID
- Test: CreatePolicy with duplicate category returns conflict error
- Test: GetPolicy for existing category returns correct policy
- Test: GetPolicy for nonexistent category returns default policy
- Test: UpdatePolicy validates refund window bounds (1-90 days)
- Test: DeletePolicy with active orders returns rejection

### Step 2: Implement RefundPolicyService to pass all tests

### Step 3: Define test cases for order service refund flow changes
- Test: ProcessRefund uses category-specific policy when available
- Test: ProcessRefund falls back to default when no category policy
- Test: ProcessRefund rejects if outside category refund window
...

Why Tests First in the Plan

The plan doesn't say "implement RefundPolicyService" and then "add tests." It says "define test cases for RefundPolicyService" and then "implement RefundPolicyService to pass all tests."

This ordering is deliberate. When you think about tests first, you're thinking about behavior. What should this service do? What are the edge cases? What should it reject? You're defining the contract before you write the implementation.

When you think about implementation first, you're thinking about structure. How should I organize this code? What patterns should I use? These are important questions, but they should come after you know what the code needs to do.

---

Phase 4: Implementation — Tests First, Code Second

With the plan approved, `hf-implement` takes over. And this is where TDD actually happens.

How hf-implement Works

`hf-implement` reads the plan and executes it step by step. For each step:

1. Write the tests first. Claude generates test files based on the test cases defined in the plan. These tests compile but fail — the implementation doesn't exist yet.

2. Write the implementation. Claude writes the production code to make the tests pass. It runs the tests after each significant change to verify progress.

3. Iterate until green. If a test fails, Claude reads the error, adjusts the implementation, and re-runs. This cycle continues until all tests pass.

Why This Works Better Than "Generate Everything"

When Claude writes tests and implementation simultaneously (the "vibe coding" approach), the tests tend to be shaped around the implementation. They test what the code does, not what it should do. You get 100% pass rate and zero confidence.

When tests are written first from the plan, they're shaped around the requirements. The implementation has to conform to the tests, not the other way around. If Claude's implementation has a bug, the pre-written tests catch it. If Claude's implementation makes an incorrect assumption about the codebase, the tests (which were designed with the research and tech spec in mind) catch it.

I've seen this prevent real bugs. Claude once generated an implementation that used an in-memory cache for refund policy lookups — clean, fast, elegant code. The pre-written tests included a case for "policy updated in database, next lookup returns updated policy." The cached implementation failed that test. Without TDD, that cache staleness bug would've made it to production.

---

The CLAUDE.md File — Your Project's Long-Term Memory

Every project I work on has a `CLAUDE.md` file at the root. This is Claude Code's project-level memory — instructions, conventions, and context that persist across sessions.

What Goes in CLAUDE.md

# Project: Order Service

## Architecture
- Go 1.22, gRPC, PostgreSQL
- Part of the commerce platform (12 services)
- Communicates with: payment-service, inventory-service, notification-service

## Conventions
- All new endpoints need integration tests in /tests/integration/
- Use structured logging (slog) — no fmt.Println
- Error types defined in /pkg/errors/ — don't create new ones without discussion
- Database migrations in /migrations/ — always reversible

## Known Gotchas
- The RefundProcessor runs async — don't assume refund completion is synchronous
- payment-service has a 30-second timeout on refund API calls
- inventory-service uses eventual consistency — reads might be stale by up to 2 seconds

## Testing
- Unit tests: `go test ./...`
- Integration tests: `make test-integration` (requires local Docker)
- E2E tests: `make test-e2e` (requires staging environment)

Why It Matters

Without `CLAUDE.md`, every new Claude Code session starts from zero. Claude has to re-discover that your project uses structured logging, or that the payment service has a 30-second timeout, or that migrations must be reversible. With `CLAUDE.md`, Claude starts every session with institutional knowledge.

Think of it as onboarding documentation — but for your AI pair programmer. The same way you'd tell a new team member "hey, don't use fmt.Println, we use slog," you tell Claude once in `CLAUDE.md` and it remembers forever.

I update `CLAUDE.md` whenever I discover a new gotcha or establish a new convention. It's a living document. Over months, it accumulates the kind of tacit knowledge that usually only lives in senior engineers' heads.

---

What Changed in How I Think About Development

Using Claude Code with this process hasn't just made me faster. It's changed how I approach problems.

I Write More Tech Specs, Not Fewer

You'd think that having an AI write code would reduce the need for planning. The opposite happened. Because Claude can implement so fast, the bottleneck shifted from "writing code" to "knowing what code to write." The tech spec became the highest-leverage artifact in the entire process.

A well-written tech spec means Claude produces the right code on the first try. A vague tech spec means Claude produces plausible-looking wrong code that takes longer to debug than writing it from scratch.

I Think in Tests Before I Think in Code

TDD existed long before AI. But I'll be honest — I was a "write code first, add tests later" developer for most of my career. The friction of writing tests was high enough that I'd skip them when I was in a flow state.

With `hf-implement` handling the typing, the friction disappeared. Writing tests first costs me almost nothing in terms of time. And the quality improvement is obvious. I'll never go back.

I Treat the Codebase as Readable, Not Just Writable

Before Claude Code, my relationship with unfamiliar parts of the codebase was: avoid them. If a feature required touching a service I'd never worked on, I'd groan and spend half a day getting oriented.

Now, `hf-research` makes any part of the codebase approachable in minutes. I'm no longer afraid of unfamiliar code. I actively seek out context from services I don't own, because I know Claude can map the dependencies and surface the relevant details faster than I can.

This has made me a better systems thinker. I understand how our services interact at a level I never did before, because the cost of understanding dropped to near zero.

---

The Honest Downsides

This wouldn't be a useful blog post if I pretended everything was perfect.

1. Context Window Limits Are Real

Large codebases don't fit in a context window. `hf-research` mitigates this by using sub-agents that each focus on a subset of the codebase, but there are still moments where Claude loses track of a detail from earlier in the conversation. The fix is good `CLAUDE.md` files and breaking work into smaller, focused sessions.

2. You Still Need to Read the Code

Claude generates code. You must review it. Every line. Not "glance at it and approve" — actually read it, understand it, and verify it does what you intended. The tests help (they catch functional bugs), but they don't catch architectural mistakes, security issues, or performance problems. That's still your job.

3. The Research Isn't Always Complete

`hf-research` searches aggressively, but it can miss things. Undocumented side effects, implicit dependencies, tribal knowledge that isn't in the codebase. When the research seems too clean, I get suspicious and manually verify the critical paths.

4. Not Every Task Needs This Process

Quick bug fixes, one-line config changes, README updates — these don't need a four-phase workflow. I use the full process for features that touch multiple services or involve architectural decisions. For everything else, I just use Claude Code directly without the ceremony.

---

Key Takeaways

1. "Vibe coding" doesn't scale. It works for prototypes and personal projects. For production codebases with multiple services and teams, you need a process.

2. Research before you build. Use `hf-research` (or any systematic approach) to understand the current codebase before proposing changes. The 15 minutes you spend on research saves hours of debugging wrong assumptions.

3. Write the tech spec. Claude is an excellent thinking partner for tech specs — brainstorming approaches, evaluating trade-offs, generating diagrams. But the decisions are yours. The tech spec is the document where you commit to an approach.

4. Plan your TDD before implementing. Don't let the AI decide what to test. Define your test cases in the plan, then let the AI write them. Tests should be shaped by requirements, not by implementation.

5. Tests first, code second. `hf-implement` writes tests before writing production code. This catches bugs that would otherwise sail through — including bugs in AI-generated code.

6. CLAUDE.md is your project's institutional memory. Invest in it. Update it. It compounds over time and makes every Claude Code session more effective.

7. The human is still the architect. Claude Code is the most powerful implementation tool I've ever used. But it doesn't make architectural decisions. It doesn't evaluate business trade-offs. It doesn't know which approach your team can maintain. That's your job — and it always will be.

---

If you want to try this workflow, start with a `CLAUDE.md` file in your project. That single step will improve every interaction you have with Claude Code. Then gradually add structure — research, tech specs, TDD planning — as the complexity of your work demands it.

Connect with me at [mrinal.dev](https://mrinal.dev). I'm always happy to talk about engineering workflows.mrinal.dev. I'm always happy to talk about engineering workflows.*