How a Load Test Saved a Florida County Election Call Center in Four Lines of Code

How a Load Test Saved a Florida County Election Call Center in Four Lines of Code

The thorn in this county’s democratic process was their phone system. Every election season, poor communication crippled the county’s ability to register voters, confirm voters’ IDs and report back polling station statistics. During peak voting hours, citizens dialing in to get information about where they could vote wouldn’t be able to reach an operator, or would be told the number was disconnected.

To make the voting process smoother for Florida officials and residents, Acuity Technologies stepped in to clean up the telecom-mess using Twilio. Josh Anderson (pictured right), CEO and founder of Acuity Technologies, is a Florida native, but more importantly, he’s a problem solver.

Josh was challenged with uncovering the failure points across a call center reaching 350 voting stations, piping out thousands of calls to one telecom provider and one carrier. With only four lines of Twilio code, he was able to solve the county’s telecom woes, save them thousands of dollars, save voters time, and bridge the communication gap between carrier and telecom provider.

The Problem at Hand

The first step in fixing Florida’s communication problem was to identify the key parts in play, and the role they served:

• • 350 Polling Stations – The county has a network of polling stations, all of which have the ability to dial into a call center. There, agents record key metrics on election day, verify votes, and authenticate voter IDs.
  • The telecom provider – the telecom provider (a third party vendor who will also remain anonymous) was responsible to provide the SIP trunks that would take the calls, and route them to the carrier so they could be initiated.
  • The carrier– The carrier takes the telecom provider’s volume of SIP trunks, and SIP requests, and makes sure each call is successful.

The second step in solving the problem was dropping the hammer. Josh asked the county’s telecom provider to run a hammer test, but was unceremoniously denied. Instead, he wrote four short lines of Twilio code, and had a hammer test ready to run in minutes.

Dropping The Hammer: The Load Test Of Truth

The goal of the “hammer test” was to load the entire call center with its maximum volume of calls for 14 hours (the average high traffic duration during election season), identify the points of failure, and fix them. It was a game-day drill for Josh and the county.

Josh’s Twilio script called the 180 SIP trunk lines in succession, placing them in a Twilio Queue. Once all 180 lines were connected, Josh hung a few up and called them right back. The call flow was designed to mimic the waves of calls on election day.

The total cost of the test: less than $20.

The Red Flag, and Miscommunication

Minutes into the hammer test, a red flag popped up, and then another, and another. The telecom provider was dropping calls and returning an incorrect 404 error, essentially saying that telecom provider didn’t recognize its endpoint (The Carrier) and could not initiate the call. But, the telecom provider did know it’s endpoints – it had already called them during the test.

Josh looked at his Twilio account portal and confirmed he had the right endpoints. The next step was figuring out the real problem, mistakenly labeled as a 404. At this point, the Twilio hammer test began to serve the role of detective as well as load tester.

Reaching the Fire Alarm: How to Trigger a Fail Safe

It turned out the real problem was a discrepancy between the telecom’s promised number of SIP trunks, and their number of provided SIP trunks. The telecom provider had 120 SIP trunks, the carrier had 125. With this set up, the Telecom Provider was incapable of hitting the call threshold to trigger the carrier’s failsafe switch. When it was overloaded, it spat out 404s instead of hitting the fail-safe.

Josh could load all of the telecom providers SIP trunks, but that still wasn’t enough to flip the carrier’s failsafe. Imagine a child who is too short to pull the fire alarm — this was the equivalent in telecom terms.

Josh worked out the capacity issues with the telecom provider to make they could successfully work with the carrier. Using Twilio Queue, he re-ran the hammer test a second time and kept a packed Queue going for 30 minutes with no errors.

Playing telecom detective is no easy task. But using Twilio, Josh’s task was much more manageable. With the new voting communication system in place, Josh hopes citizens can vote with confidence, and get their election day questions answered easily.