The contract said $12M over two years. What it actually said, once you read between the lines of the RFP, was: the thing we’re replacing cannot go down during an election, and the thing that replaces it cannot go down during an election, and between the old thing and the new thing there has to be a seamless handoff that nobody notices. New Jersey holds elections constantly — presidential, gubernatorial, congressional, county, municipal, school board, special. At the time the contract started, the next general election was about eighteen months out, and there were roughly forty smaller elections scheduled between then and now. None of them could be disrupted.

The old system was twenty-four Oracle instances — a mix of 9, 10, and 11 — running on Windows servers scattered across the state’s twenty-one counties plus a handful of state-level aggregation nodes. Licensing costs were eating the budget. The vendor relationship had soured. The DBAs running each county’s instance had different levels of skill and different opinions about how to run things. The replication between counties was, charitably, a topic of active debate among the people responsible for it. The state wanted off.

I was brought in to design and execute the migration onto MariaDB 10.1, running on Linux, with a topology that would let any county keep working during a network partition and resynchronize cleanly when connectivity came back. We delivered three months early. Twelve years later the platform is still running, having maintained 100% uptime through every local and state election cycle since launch.

This is a retrospective on the technical decisions that made that possible, with the usual caveats that hindsight makes everything look cleaner than it was.

The design constraint that drove everything

Most database architecture work starts with a question like “how much throughput do we need” or “what’s the read/write ratio.” Election systems don’t work that way. The throughput is trivial — a county registering voters handles at most a few hundred transactions an hour, and even on election day the reads per second are well within what a single modest server could serve. The constraint that mattered was: what happens when the network goes down?

New Jersey’s network infrastructure, like most state infrastructure, was not built with the assumption that it would always be available. County offices connected back to state systems over links of varying quality. Fiber cuts happened. Routers failed. Weather took things out. The old Oracle-based replication assumed a working link most of the time, and when the link went down, the affected instance would pause replication and queue up changes locally. When the link came back, the changes would replay. This worked until it didn’t — conflicts could accumulate, replay could fail, and a county could spend hours or days offline while the DBAs sorted it out.

For an election system, this is unacceptable in a specific way: a voter who shows up on election day and cannot be found in the rolls has no remedy later. The damage is done at the moment of the outage. Any replacement system had to let the county keep accepting writes during a partition, because the alternative was disenfranchising voters.

That decision — that every site must accept writes during partition — cascaded through every other architectural choice we made. It ruled out primary-replica with failover, because failover during a partition still leaves the isolated site read-only. It ruled out consensus-based replication (Galera was available but we evaluated and rejected it), because Galera requires a quorum to accept writes, which by definition the isolated site won’t have. What it left was multi-master asynchronous replication with conflict detection and semi-automated reconciliation. This is the unfashionable option in database architecture, and it’s unfashionable for good reasons — but for this specific problem shape, it was the right choice.

The topology we landed on

Thirteen sites, arranged as a partial mesh. Not all twenty-one counties had their own site — several smaller counties shared a regional site to keep operational overhead manageable. Each site ran a MariaDB 10.1 instance configured as a master, with replication streams flowing to every other site. Writes could happen at any site. Reads were served locally.

The replication topology was not a simple ring or star. A ring is fragile — one broken link and half the cluster is cut off from the other half. A star puts too much load on the hub and makes the hub a single point of failure. What we ended up with was closer to a small-world network: each site replicated to three or four peers, with redundant paths so that any single link or site failure still left every surviving site reachable through at least two independent routes. When we simulated link failures in staging, the cluster reconvergence time after a restored link was typically under ninety seconds.

The auto-increment offsets were configured per-site to ensure no two sites could ever generate the same primary key. This is the standard multi-master trick — site 1 uses offsets of 1, 14, 27, ..., site 2 uses 2, 15, 28, ..., and so on — but with thirteen sites and room for future growth we used a wider stride than usual. This produced sparse key spaces, which occasionally surprised developers reading raw data but never caused operational problems.

Conflict detection was implemented at the application layer rather than the database layer. The voter registration workflow is constrained enough that most conflicts were detectable by the business rules themselves — you cannot register the same person in two counties simultaneously, and the rules for moving a voter between counties are procedural. The few classes of conflict that could slip through the application rules were caught by a reconciliation process that ran on each site and surfaced anomalies for human review. The number of genuine conflicts over a year of operation was in the low tens, all resolvable by a clerk. This is the payoff for narrow-domain applications — you can let the database replication be less clever because the application is disciplined.

The Oracle-to-MariaDB migration itself

The data migration was the part everyone worried about and which turned out to be the second-hardest problem, well behind the topology design.

Oracle 9 through 11 has data-type behaviors that MariaDB does not replicate exactly. NUMBER with arbitrary precision does not map cleanly to any MySQL-family integer or decimal type. Oracle’s handling of empty strings as NULL is idiosyncratic. The date and timestamp types have different precision and timezone semantics. Anyone who has done this migration knows the shape of the problems; it’s the sheer volume of edge cases that makes it tedious.

I wrote the ETL in Perl. Not because Perl was my first choice in 2014, but because the team had Perl expertise, the existing Oracle-side tooling was already Perl, and the transformation logic was straightforward text-munging that Perl handles well. Python would have worked. So would Java. The choice of language mattered less than the discipline of the pipeline: every table got a schema-mapping file that documented the source column, the destination column, the transformation rule, and the edge-case handling. Every row that failed transformation got written to a quarantine table with enough context to diagnose it. Every batch was reversible.

The migration ran in three passes. First, a full historical load from Oracle into staging MariaDB instances, executed during low-usage windows over several weeks. Second, a daily delta sync that kept the staging instances current while we tested. Third, the cutover itself, which was a coordinated few-hour window where the old Oracle instances were set to read-only, a final delta was applied, and the counties switched to querying the new MariaDB instances.

The cutover happened on a Saturday morning. It went cleanly. I was in the operations center with coffee and a printout of the runbook, watching the connection counts drain from the old system and rise on the new one.

The Windows-to-Linux conversion

This was a larger political problem than technical. County IT staff were, on average, comfortable with Windows and not comfortable with Linux. The migration wasn’t just moving data; it was asking people to learn a new operating system. Some embraced this, some resisted it, most tolerated it.

The technical work was standard: Ubuntu LTS, hardened to the state’s security baseline, MariaDB installed from official repositories, configured via Puppet modules that we wrote specifically for the election system. The less-standard work was the training program. I wrote and delivered training materials for county-level SysAdmins who would be running these boxes. The materials assumed no prior Linux experience. They covered everything from “here’s how you SSH in” through “here’s how you read a MariaDB slow query log and figure out what’s slow.” I still have one of the training binders in a box somewhere.

The training was the part of the project I was least prepared for and ended up finding most rewarding. The county DBAs were not database specialists; they were government IT staff with a lot of other responsibilities. Teaching them enough to run their instance competently was a different skill from designing the instance in the first place. I learned a lot about how to write operational documentation for people who will be reading it at 2 AM on election night.

What election day actually looks like

The first major test was the 2015 general election. By that point we’d been live for six months and had survived a dozen smaller elections. The general was the one where the system’s behavior would matter most.

The day ran without incident. Peak query load was roughly 4x the normal daily average — high but well within what we’d sized for. Two counties had brief network issues during the day; both kept operating locally and resynced when the links came back. Nobody noticed the system was doing anything special, which was the point.

The thing I remember most clearly is that I wasn’t needed. I was on standby. I spent most of the day reading a book. By the time the polls closed, the system had processed more transactions than it usually saw in a week, and nothing had broken. The county DBAs I’d trained were the ones running it.

This is the outcome you want from good infrastructure work: the specialist who built it isn’t needed on the day it matters, because the system runs itself. Twelve years later, the same is true.

What I’d do differently

A few things, with the caveat that 2014 was a different time.

I’d use a younger MariaDB. We chose 10.1 because it was the most recent stable release with features we needed; 10.2 would have given us better JSON handling and window functions that we ended up wanting later. Picking a slightly newer release would have traded some initial stability risk for longer useful life.

I’d separate the operational database from the reporting database earlier. The voter registration system grew reporting demands over time that started to contend with OLTP traffic, and retrofitting a read-only replica topology a year after launch was more work than designing it in from the start.

I’d invest more in automated cutover testing. The cutover itself went fine, but we tested it manually in staging and I was never as confident as I wanted to be that the production cutover would match. Modern infrastructure would let me run dozens of full-scale cutover rehearsals; we ran three.

The fundamental architectural decisions — multi-master, asynchronous, application-layer conflict detection, mesh topology — I’d make the same way again. Twelve years of uninterrupted production operation is as close to proof as you get in this business.

← All case studies  ·  Engage me on similar work