The Computer Voting Revolution Is Already Buggy, Hackable and Obsolete

I voted! I think...

Technology is not always the answer. Especially when their is a lack of human interaction and oversight. JL

Michael Riley and colleagues report in Bloomberg:

The real threat is a violation of trust: that Election Day mishaps borne of outdated, poorly engineered technology will confirm and amplify the fear pervading this campaign. Multiple lawsuits over the past 10 years have alleged that voting machines and computerized tabulators have been used to steal or suppress votes—deepening the distrust of a system some see as stacked against them.

Six days after Memphis voters went to the polls last October to elect a mayor and other city officials, a local computer programmer named Bennie Smith sat on his couch after work to catch up on e-mail.

The vote had gone off about as well as elections usually do in Memphis, which means not well at all. The proceedings were full of the technical mishaps that have plagued Shelby County, where Memphis is the seat, since officials switched to electronic voting machines in 2006. Servers froze, and the results were hours late. But experts at the county election commission assured both candidates and voters that the problems were minor and the final tabulation wasn’t affected.
That story might have held up if Smith, a financial software developer and church organist, hadn’t been conducting an election night experiment. In his free time, Smith crunches voter-turnout data with programs he’s written to help local politicians target their direct-mail campaigns. Like Smith, most of his clients are black, and he had bet a friend 10 candy bars that the polling place at Unity Christian Church, a black congregation a mile from Graceland, would have a big turnout. The precinct, No. 77-01, is a Democratic stronghold and has one of the largest concentrations of African American voters in a city known for racially fractured politics. Smith’s guess: 600 votes. When the polls closed at 7 p.m., he was at Unity Christian and snapped some photos with his BlackBerry of the precinct’s poll tape—literally a tally of the votes printed on white paper tape and posted on a church window. Since the printouts come directly from the voting machines at each location, election officials consider it the most trustworthy count. According to the tape, Smith’s guess was close: 546 people had cast ballots.When he got an e-mail a week later with Shelby County’s first breakdown of each precinct’s voting, he ran down the list to the one precinct where he knew the tally for sure. The count for Unity Christian showed only 330 votes. Forty percent of the votes had disappeared.
If you’re an election official, losing votes is a very big deal, but it presents a special problem in Tennessee. Most counties in the state don’t keep paper records of ballots, so there are no physical votes locked in a room somewhere, ready to be recounted.
When underperforming voting equipment in Florida nearly created a constitutional crisis in the 2000 presidential race, officials at least knew what went wrong. The aging Votomatic machines were supposed to be cleaned regularly, which election officials in several counties failed to do. So when voters choosing between Al Gore and George W. Bush inserted their readable punch-card ballots into the devices, they often created a half-punched piece of chaff rather than a clean cut, and entered the term “hanging chad” into the American political lexicon.

Shelby County uses a GEMS tabulator—for Global Election Management System—which is a personal computer installed with Diebold software that sits in a windowless room in the county’s election headquarters. The tabulator is the brains of the system. It monitors the voting machines, sorts out which machines have delivered data and which haven’t, and tallies the results. As voting machines check in and their votes are included in the official count, each machine’s status turns green on the GEMS master panel. A red light means the upload has failed.
At the end of Memphis’s election night in October 2015, there was no indication from the technician running Shelby County’s GEMS tabulator that any voting machine hadn’t checked in or that any votes had gone missing, according to election commission e-mails obtained by Bloomberg Businessweek. Yet as county technicians followed up on the evidence from Smith’s poll-tape photo, they discovered more votes that never made it into the election night count, all from precincts with large concentrations of black voters.
For the members of Congress, who in 2002 provided almost $4 billion to modernize voting technology through the Help America Vote Act, or HAVA—Congress’s response to Bush v. Gore—this probably wasn’t the result they had in mind. But voting by computer has been a technological answer in search of a problem. Those World War II-era pull-lever voting machines may not have been the most elegant of contraptions, but they were easy to use and didn’t crash. Georgia, which in 2002 set out to be an early national model for the transition to computerized voting, shows the unintended consequences. It spent $54 million in HAVA funding to buy 20,000 touchscreen voting machines from Diebold, standardizing its technology across the state. Today, the machines are past their expected life span of 10 years. (With no federal funding in sight, Georgia doesn’t expect to be able to replace those machines until 2020.) The vote tabulators are certified to run only on Windows 2000, which Microsoft stopped supporting six years ago. To support the older operating system, the state had to hire a contractor to custom-build 100 servers—which, of course, are more vulnerable to hacking because they can no longer get current security updates.

Smith

Photographer: Julian Berman for Bloomberg Businessweek

After California declared almost all of the electronic voting machines in the state unfit for use in 2007 for failing basic security tests, San Diego County put its decertified machines in storage. It has been paying the bill to warehouse them ever since: No one wants to buy them, and county rules prohibit throwing millions of dollars’ worth of machines in the trash bin.
This muddle is about to collide head-on with one of the most incendiary presidential campaigns in modern U.S. history, one in which the candidates have already questioned whether votes will be counted properly. Donald Trump warned supporters in Columbus, Ohio, on Aug. 1 that “we’d better be careful, because that election is going to be rigged.” Hillary Clinton and top Democrats have accused Russia of trying to manipulate the election by hacking. FBI Director James Comey, testifying before Congress on Sept. 28, said that states should be vigilant against online intrusions “because there’s no doubt that some bad actors have been poking around.”
The real threat isn’t a thrown election. Nationwide electoral fraud would be extremely difficult to pull off, mostly because votes in the U.S. are tallied by more than 7,000 counties and townships. Hacking enough of them to tip the balance would be a monumental undertaking—and one certain to be detected. (Tabulators are designed not to be connected to the internet at all.) Rather, the risk is a violation of trust: that Election Day mishaps borne of outdated, poorly engineered technology will confirm and amplify the fear pervading this campaign. In Shelby County, multiple lawsuits over the past 10 years have alleged that voting machines and computerized tabulators have been used to steal or suppress votes—deepening the distrust of a system some locals see as stacked against them.
Smith was never one of those. His energies went into building data analytics that candidates would need if they wanted to stop complaining and get elected. But after the votes disappeared from Unity Christian last October, something changed. “I kind of knew this would be a place where this could happen, but this morbid feeling came over me—are you serious?” he remembers thinking. “Is this how politics is supposed to work? Is anybody’s vote safe?”

The voting technology business, after a frenetic decade of mergers, acquisitions, and renamings, is dominated by just a few companies: Election Systems & Software, or ES&S, and Dominion Voting Systems are the largest. Neither has much in common with the giants of computing. Apple, Dell, IBM, and HP have all steered clear of the sector, which generates, according to an analysis by Harvard professor Stephen Ansolabehere, about $300 million in annual revenue. For context, Apple generates about $300 million in revenue every 12 hours.

During the dramatic Florida recount, Mark Earley was an election official in Leon County, which is mostly made up of the city of Tallahassee, the state capital. That put Earley at the center of a global spectacle, in charge of counting the 103,000 votes in the county, which were cast on optically scanned ballots, a novelty then. State troopers armed with machine guns stood outside the courthouse, protecting the proceedings from crowds of screaming protesters and international TV crews.

Earley

Photographer: Julian Berman for Bloomberg Businessweek

Earley knew the controversy would create a big opportunity for voting tech companies, and they began hiring local officials like him. Sandra Mortham, a former Florida secretary of state, was hired by ES&S, based in Omaha. (Mortham also represented the Florida Association of Counties, and before long ES&S was the only voting system endorsed by the association.) Earley took a job at Global Election Systems, a smaller Canadian company that he thought had better products.
Global Election Systems, with U.S. offices in McKinney, Texas, was sold to Diebold in 2002, as companies merged to chase the HAVA billions. Earley went to Diebold as well, where he liked the travel and the chance to share what he’d learned with officials in other states. But this was no Silicon Valley, with its stock options and office juice bars. Managers at Diebold’s election division in McKinney went to CiCis Pizza for the all-you-can-eat buffet. “You had to try and go when it was busy—that way they had to keep replacing the food,” Earley says. “Otherwise it got cold and stiff.”
By 2006 every state but New York had dumped their pull-lever and punch-card machines in favor of computerized voting. The voting tech vendors rushed systems to market, often without adequate testing, to meet procurement deadlines set by hundreds of counties and states. According to Earley, the systems often had software flaws or too little memory, problems the company’s executives figured could be fixed later.
Before Global was sold, Earley says, its executives were frantically trying to solve the problem of recurring revenue. Consumers were willing to replace mobile phones or computers every two or three years to get the latest features, creating big profits and fast innovation cycles. County buyers wanted electronic voting machines to last a decade or more. Earley believes Global ran into trouble because its products were too reliable, so there were too few returning customers.
He says the competition solved the revenue problem by focusing less on making equipment and more on long-term contracts. It was an enhancement of the old razors-and-blades strategy: Sell the razors cheap and make money on the blades, and make even more money by making the razors so hard to use that customers pay you to give them a shave. When Allen County, Ohio, replaced its old voting machines in 2005 with equipment from ES&S, officials didn’t realize they’d also be stuck with a service fee of $40,000 per year to help run an election system that handled about 70,000 votes. “When we found out the cost, our jaws just about hit the floor,” says Ken Terry, who was election director there until this year.

Advertisement

To top it off, Terry discovered that the county was paying top dollar for antiquated technology. It wasn’t until the machines were purchased, and in place, that county officials realized their new system ran on software written in 1996. After counting paper ballots with an optical scanner, the data had to be transferred to a server using Zip drives—a storage format developed when pagers and AOL dial-up were still in vogue. When Allen County tried to replace the disks in 2012, they were so hard to find that officials had to ask ES&S for a set. “They were in this shrink-wrapped package,” Terry says, “and when we opened it, there was a coupon that expired in 1999.”
ES&S declined to be interviewed for this story but provided written answers to questions. Kathy Rogers, senior vice president for government relations, wrote that many local election officials are satisfied customers who see digital voting as a big improvement over the old-fashioned kind. The company stands by the performance record of its equipment and services. “At ES&S we place as much emphasis on sustainability of currently fielded systems as we do on development and research of new systems,” she wrote.
Election officials now have more ways than ever to screw up a vote. South Carolina elections are run on ES&S machines that use cartridges—like the ones for old Nintendo game consoles—to transfer votes to a tabulator. Poll workers put the cartridge in a slot in the machine at the start of voting; after polls close, all cartridges must be delivered to the tabulator room, where they are plugged in and their data downloaded. In 2010 workers at two precincts in the state capital of Columbia mixed up cartridges and lost 1,127 votes, or almost two-thirds of the precincts’ total.
Ohio Secretary of State Jennifer Brunner sued Diebold following the 2008 primaries after 11 counties using the company’s AccuVote-TSX voting machines and GEMS tabulator dropped votes. The company claimed the problem was the result of the antivirus program the counties were using. After a 10-month fight, Diebold conceded the lost votes were the result of a software bug. The bug was fixed in later versions, and more than half of Ohio counties received free or discounted voting machines and software as part of the settlement.
“This is the strangest niche of IT that I’ve ever come across,” says Merle King, executive director for the Center for Election Systems at Georgia’s Kennesaw State University, which runs the state’s vote machine testing program. “Whatever you think you know about IT, you have to check it at the door. It’s legacy stuff, but it’s legacy in weird ways. This is legacy stuff that as you start to tease it apart goes back decades.”
With little chance of another infusion of federal funds from a constipated Congress, the industry consolidated some more. Diebold rebranded its elections division as Premier Elections Solutions in 2007, then sold the business two years later to ES&S. Antitrust action by the U.S. Department of Justice forced ES&S to sell some of those assets to Toronto-based Dominion.
What the industry hasn’t done is invest much in updating the hardware on which millions of Americans will cast their votes in November. In conversations with officials at statehouses and county offices, the device makers often point fingers at the same federal law that greatly expanded the market for digital voting in the first place. HAVA created the U.S. Election Assistance Commission, which sets federal standards for computerized voting equipment that most states use as a benchmark. Those standards haven’t changed much since 2005.
Whoever is to blame, the result is that many of the old systems were simply repackaged as they passed from one company to the next. Even a cosmetic upgrade was too much in some cases. On its website, ES&S displays a picture of its AccuVote-TSX touchscreen voting machine. Emblazoned on the front of the machine is the name Diebold, a company that stopped making voting equipment seven years ago. (ES&S’s Rogers says its agreement with the Justice Department allows the company to sell AccuVote only to customers who already use them.)
In 2014 a presidential commission that assessed the state of voting technology, interviewing hundreds of local election officials over two years, issued a devastating judgment in its final report: “Jurisdictions do not have the money to purchase new machines, and legal and market constraints prevent the development of machines they would want even if they had funds.”

Advertisement

The AccuVote-TSX touchscreen voting machines used in Shelby County are essentially folding easels with a computer screen. They stand about 5 feet tall, weigh about as much as a hibachi grill, and record votes on a removable data card about the size of a credit card. When the polls close, the votes from the five or six machines at a typical polling place are totaled and printed on a tally tape, which is posted somewhere visible. The cards are then sealed in a clear security bag and driven to one of six upload centers, where they are individually inserted into a transmitter and the data sent to GEMS over a closed network.

At 7 p.m. on Oct. 8, 2015, the polls for Memphis’s municipal elections closed, and tallies began rolling in to Shelby County’s election operations center, a wide building on the eastern outskirts of the city, across from a county penal farm ringed with concertina wire. The problems began almost immediately.
About 15 minutes into counting, votes stopped coming into the GEMS tabulator from the precincts. Shelby County was using two GEMS databases that night instead of one, a troublesome configuration because each memory card can upload only to the database it was programmed on for security reasons. Problems merging the two sets of data created at least a two-hour reporting delay that night, according to a postelection analysis by ES&S. But by 8:45 p.m. the system was up and running again.
The county also happened to be using new software to post results to the web, which is a different program than the one that calculates the official tally. A worker copying the wrong file in the race for city clerk of court published the wrong vote count, which stood uncorrected until around 9 p.m.
Watching from Fitzgerald’s Cigar Lounge, Wanda Halbert, a two-term city councilwoman, was munching on a deviled egg when officials corrected the numbers. Halbert had left her City Council seat to run for clerk, a job that oversees the collection of some $5 million in traffic fines handed out annually—real power in Memphis. She and her supporters were stunned as the top three candidates’ vote totals suddenly changed, dropping her into second place.

Halbert

Photographer: Julian Berman for Bloomberg Businessweek

Climbing into a friend’s car, Halbert sped to the county election commission’s offices, where she cursed out officials and demanded answers. The profane exchange was caught by a local TV news crew before Halbert calmed down. She stayed until the final results were tallied around 2 a.m. Halbert, who is black, lost to the top vote-getter, who is white, by a slim margin, 1,367 votes.
Herman Morris, then the Memphis city attorney, sent out an e-mail the following day congratulating election officials on quickly resolving the night’s various crises and noting that, “notwithstanding the intense media scrutiny, harassment, bias and spin, you all performed magnificently.”
That was before Smith’s smartphone image of the Unity Christian vote count surfaced. Smith grew up poor in the city’s College Park neighborhood, a high-crime stretch of single-family homes in north Memphis. As a kid, he managed to stay out of the violence that engulfed his family and friends, reading Malcolm X and tinkering with toy trains and other electronics. Smith eventually moved into a gated community, raising three girls, but he never lost the habit of questioning authority and “this knack for the rub—figuring out why things didn’t line up.”
He decided to show the poll tape to Norma Lester, a friend on the election commission. The photo, which Smith gave Lester on Oct. 17, the week before commissioners were scheduled to certify the election, was smoking-gun proof that the votes had disappeared somewhere between Unity Christian’s voting machines and the GEMS tabulator that spat out the official election night tally.
What the county’s election administrators did with that information isn’t entirely clear, mostly because the county has decided not to clarify it. Shortly after the election results were certified, Halbert filed a lawsuit over irregularities in the tally, and an attorney representing the election commission cited the legal action in explaining that officials wouldn’t comment about anything that happened that night.
Internal e-mails and other documents related to the Oct. 8 election were given to Bloomberg Businessweek by Carol Chumney, a local attorney, who got them through a series of open records requests. They show that Joe Young, Shelby County’s deputy administrator of elections, went hunting for answers shortly after Lester got Smith’s poll-tape photo. He looked at server logs and other data that gave a picture of how GEMS operated on election night, and he found the problem was much worse. At first it looked like votes were missing from not just one precinct but 20. After more investigation, he appeared to narrow that number to four. Not all of the precincts are named in the e-mail, but a master record for the voting machines shows missing uploads at four polling places on election night, all in areas with large concentrations of black voters. Three are located at black churches: Mt. Zion Missionary Baptist, Mississippi Boulevard Christian, and Unity Christian; the fourth is at Gaston Park, a community center in a mostly black neighborhood south of downtown. The weird thing is, the GEMS system recognized at least some of the missing votes—stored on the memory cards of seven voting machines—as already counted when officials tried to reload them on Oct. 19, according to an e-mail exchange between Young and operations manager Darral Brown. But it was clear from Smith’s poll tape and other data dug up by Young that they hadn’t been. In all, 1,001 votes had been dropped from the election night count, according to the master record, including almost 400 from an early voting center at Mt. Zion, the most from any single polling place.
Young wasn’t rewarded for his effort. In fact, on Oct. 19 he got a sharp rebuke from his boss, Shelby County Administrator of Elections Richard Holden, who had asked Young to investigate a separate issue. Holden refused repeated interview requests by phone, e-mail, and in person.

Photographer: Julian Berman for Bloomberg Businessweek

Discussion of the missing votes stops in the e-mails after that. There’s no indication the county looked further into how the votes got lost or why the GEMS system failed. One other thing Holden and Young didn’t do: They didn’t explain how widespread the problem was to the five county election commissioners, according to Commissioner Anthony Tate, one of two Democrats on the panel. As far as the five commissioners knew, the only precinct with missing votes was Unity Christian; not three other precincts, too. If they had been informed of the full scale of the fiasco, Tate says, he would not have voted to certify the election along with the other commissioners on Oct. 23.
Given that this is Memphis, where the political history is fraught with a legacy of election tampering and cronyism that favored wealthy, white elites, it would be easy to conclude that something nefarious was going on. That’s what Halbert’s lawsuit is seeking to prove. King, the Kennesaw professor and an expert in GEMS software, says the answer is probably much simpler: It’s what happens when hard-to-use technology is deployed sloppily.
Shelby election officials added the votes after the fact, using the data on the memory cards located the day of Young’s search. Around 300 votes were added on Oct. 19 and an additional 700 or so over the next two weeks, according to a master tabulation report which was also obtained by Chumney. In the court clerk’s race, the missing votes were divided among four African American candidates, including Halbert, who gained 225 votes on the winner, not enough to change the result.
It may never be possible to say exactly what went wrong on election night last October, since officials in Shelby County didn’t conduct a public investigation, but one possibility stands out. Among the documents released to Chumney is a user’s manual for the county’s version of GEMS. It shows they’re using a version of the software that contains the bug known to drop votes, the subject of that 10-month investigation in Ohio in 2008. The software flaw creates exactly the situation described in the e-mails by Young and other officials, one that has been well-known for eight years. Diebold didn’t replace the flawed versions outside of Ohio, and for counties to do so on their own was expensive. Some counties in Virginia and Georgia still use the problem software, as well. But they employ special protocols to make sure that votes aren’t dropped, officials in both states say. In Georgia, that includes comparing tabulated precinct results with each physical poll tape—essentially replicating Smith’s experiment, but for every precinct in every county.
Tennessee law requires counties there to do it as well, but Shelby County stopped the practice several years ago to save money, according to a deposition in an earlier election lawsuit. The process was replaced with an audit of 10 percent of polling places, which failed to catch any problem with the October vote. The county’s election commissioners were also not told that Shelby uses a version of GEMS known to lose votes, Tate says. “I’m shocked and baffled as to why that information was not disclosed to us,” he says.
For Smith, who prides himself on being sober and analytical, it’s tough to know what to think. It’s certainly odd that the missing votes were all in areas with high concentrations of black voters. What is clear, he says, is that local officials didn’t try to get to the bottom of the problem, or figure out whether it’s also occurred in past elections, or determine what elections over the past several years might also be compromised. If these problems were affecting white votes, Smith says, “there’d be some smoke in the city.”

Following every major election since introducing computerized voting 10 years ago, Shelby County has been sued, often with the defendants alleging some version of election fraud through hacking or data manipulation. So far, none of the defendants has been successful.

Before running for city clerk, Halbert spent 25 years as an administrative assistant at FedEx and eight years on the City Council. She is now cleaning houses, trying to earn enough on top of her savings to fund her lawsuit, which alleges that problems with Shelby County’s election systems “date back at least a decade and are pervasive, severe, chronic, and persistent.” The only attorney she could afford is a personal injury lawyer; Shelby County is represented by John Ryder, a local attorney who’s also general counsel for the Republican National Committee and one of the most high-powered election lawyers in the country.
“We just have to stop this happening in Shelby County,” says Halbert. “If I did lose, I want it to be fair and square. We really don’t know who the hell won” the election.
The case was going Halbert’s way, at first. In early filings, the county conceded some of the software it uses is so old or obscure it doesn’t even know who makes it. The suit stalled in May, when District Court Judge Jim Kyle abruptly recused himself, citing that his wife is running for office in November. A new judge was appointed in July, but Halbert says she’s running low on money.
Shelby County officials say they’re exploring whether to buy new voting machines, but replacing all of them would cost about $20 million. Linda Phillips, the county’s new administrator of elections (Holden, who oversaw the October election, retired at the end of last year), said in a local TV interview that the current system needs only small changes, mostly voter education about confusing ballots. “The basics are there,” Phillips said.
Reverend Eric Lowell Winston, pastor at Mt. Zion, the polling place where the most votes were lost in October, says election officials have already lost the trust of many black voters in Memphis. Not having a clear answer for what happened, or pretending the problem doesn’t exist, only feeds suspicion. “I think it insults the intelligence of our community,” he says. “As if we don’t see or understand. But we do understand, we understand perfectly well what’s going on.”