CASE STUDY
ROU_WITCH
A production-grade, multi-database High Availability daemon.
PROJECT BRIEF
Witch is a single binary that manages database clusters handling leader election, automatic failover, and replication monitoring. It embeds a complete Raft consensus algorithm, replacing the need for an external distributed coordination service.
TIMEFRAME
2025 - Present
ROLE
Systems architecture, implementation, release engineering
STACK
PROBLEM
Highly available database clusters often require external coordination systems like etcd, Consul, or ZooKeeper. That extra layer increases operational weight, widens the failure surface, and makes split-brain prevention dependent on infrastructure that operators now have to secure, observe, back up, upgrade, and recover.
The daemon had to remain database-agnostic without reducing failover guarantees to the lowest common denominator.
Leader promotion could not depend only on process liveness; replication state and fencing confidence had to participate.
Operators needed a single binary deployment path with predictable configuration and limited runtime dependencies.
The design had to tolerate network partitions without allowing two writable primaries to exist at the same time.
APPROACH
Witch embeds consensus and cluster supervision directly into a small, auditable daemon. It keeps database topology, leader election, promotion gates, replication health, and fencing behavior inside one operational unit while remaining database-adapter driven.
Discovery
The first pass separated database availability problems from coordination-service problems. Many HA setups fail because the coordination layer is treated as invisible until it needs the same operational maturity as the database itself.
We documented the normal operator workflow: bootstrap, node join, planned switchover, unplanned failover, network isolation, replica lag, stale primary fencing, and manual recovery after a disputed election.
Modeling
A node is not promotable simply because it is alive. Promotion requires consensus state, freshness checks, role confidence, and fencing posture to agree.
That model turned failover from a best-effort script into a constrained state machine that can be inspected and tested.
Implementation
Witch keeps core consensus generic while pushing database-specific operations into adapters. The daemon can ask whether a replica is healthy, promote a candidate, demote a stale primary, or report replication lag without hard-coding one database protocol.
This made PostgreSQL, MySQL, and future engines possible without rewriting the election logic.
EVIDENCE
Dependencies Removed
1 coordination tier
The design removes the need to operate an additional consensus service beside the database cluster.
Primary Failure Mode
Split-brain prevention
Promotion requires quorum, replication checks, and fencing posture rather than simple health polling.
Deployment Unit
Single binary
Cluster supervision, election, and adapters ship as one operational artifact.
ARCHITECTURE
The embedded Raft layer is responsible for the cluster agreement surface: terms, votes, leadership, and committed topology changes.
Database state is never allowed to bypass consensus when a role transition can affect write safety.
Adapters expose capabilities such as health checks, replication lag inspection, promotion, demotion, and read-only enforcement.
The core daemon makes decisions from normalized signals while each adapter handles the database-specific command surface.
The system treats unknown primary state as dangerous. Fencing and failsafe modes are designed to avoid optimistic promotion when the old leader cannot be confidently neutralized.
This is the difference between availability as uptime theater and availability as data integrity under stress.
FIELD MATERIAL
SEQUENCE
01
Cataloged split-brain, replica lag, stale primary, quorum loss, and operator recovery paths.
02
Implemented leader election and membership behavior with simulated database health states.
03
Separated database-specific commands from cluster decision logic.
04
Added fencing posture, failsafe modes, logging, and deterministic recovery paths.
OUTCOME
The architecture removes a full coordination dependency from small and mid-sized database clusters while preserving deterministic leader elections, safer failover, and explicit recovery modes.
Simpler topology
Operators can run a database HA cluster without adding a separate distributed coordination service.
Sharper failure behavior
Promotion is controlled by explicit gates rather than optimistic liveness checks.
Portable model
The adapter layer makes the architecture reusable across database engines.
LESSONS
01
High availability is mostly a discipline of refusing ambiguous state.
02
A smaller deployment topology is only better when it preserves the safety guarantees operators actually need.
03
Database-agnostic design works only when the core model is strict and adapter contracts are honest.