Mailchimp Sends a Billion Emails A Day - And That's the Easy Part

Yes, you have a reputation score, even for senders of email spam. JL

Klint Finley reports in Wired:

The real problem for Mailchimp is making sure that those emails end up in people’s inboxes, not their spam folders.
Mailchimp sends more than a billion emails a day. That’s its business. The company operates a service that lets anyone send out messages en masse to their customers or followers. And by the end of the year, Mailchimp executives expects its servers will be handling way more. But sending that amount of email isn’t the problem that keeps the company’s engineers busy all day.
Sending email is a fairly well understood engineering problem. Need to send more? Just add more servers. The real problem for Mailchimp is making sure that those emails end up in people’s inboxes, not their spam folders. “There’s a big difference between sending email and delivering email,” says vice president of operations Joe Uhl.What many people don’t realize is that today’s spam filters don’t just scan an email for questionable keywords, like references to pharmaceutical products or porn. Nor do they look merely at the email address of the sender. Crucially, they also look at the servers sending the email. Most of today’s biggest email services, such as Gmail, Yahoo Mail and Outlook.com, use reputation scoring to rank the likely spamminess of a server that’s sending an email. Think of it as a sort of credit rating for email senders.
Reputation scoring is a big part of how email providers keep your inbox from completely overflowing with spam. It also helps them protect you from phishing scams and malware. But these little-known barriers also make it much harder for just anyone to send group emails, even in small amounts. That’s where companies like Mailchimp and rivals such as ConstantContact and AWeber come in, and how they try to distinguish themselves. They don’t just send email—they try to deliver it.

Batch Processing

The thing is, at a time when email overload threatens to render inboxes useless, there are some mass emails people still want to get. For example, Jerry Seinfeld’s show Comedians in Cars Getting Coffee uses Mailchimp to give subscribers an advance look at new episodes. But Mailchimp knows that if too many of its customers use its service to send spam, the major email providers will blacklist the company’s IP addresses and domain names. All those Comedians in Cars emails would end up in spam folders. Mailchimp would be useless.
To fight that possibility, Mailchimp will break up a mass emailing into batches, Uhl says. That enables the company to get some feedback from the first batch before making a full commitment. “Internally we call it taste-testing,” he says. “We send increasingly large batches and see how they perform.”

‘There’s a big difference between sending email and delivering email.’

From those initial batches, Mailchimp can see how many recipients open the email and click on what’s inside; how many unsubscribe or complain about abuse. If most people open the emails right away, that’s a signal to Mailchimp that the user isn’t spamming their list. But if lots of recipients unsubscribe or complain, the company knows that something is wrong, and it can prevent the next batch from going out. “If it’s performing poorly, we’ll kill it before it’s done,” Uhl says. “If we’re doubtful, we’ll send it out slowly.”
This approach has its own challenges, Uhl says. A billion emails per day generates millions of clicks, all of which Mailchimp needs to be processed nearly instantly. Yet the scale at which Mailchimp operates actually provides it with a few advantages when it comes to spotting spam. When a user imports a new list of email addresses, Mailchimp can compare that list to lists used by people caught using the company’s service for spamming. If the lists are very similar, that’s a good sign that the new user is a spammer too.

Too Small to Succeed

Less spam sounds great. But reputation-scoring of email servers establishes a problematic hierarchy where established companies enjoy a dominant advantage. In theory, you could bypass the Gmails and Mailchimps of the world by setting up your own email server on any old laptop. You could use it to send out an email newsletter or even create your own web-based email service for your friends and family. But under the current reputation-scoring system, most of the email you send from that server will wind up in people’s spam folders, even if you’re meticulous about ensuring spam never goes out from your server.
These days, new email servers don’t start with a reputation score of zero, says email marketing and deliverability consultant Laura Atkins. They start at less than zero. That’s because all too often, new email servers aren’t set up intentionally by the owners of the server but by spammers using malware to install email software on unsuspecting users’ computers. For email providers, protecting their users from spam, phishing schemes, and malware is a top priority, so they treat all new servers with suspicion.

Email providers treat all new servers with suspicion.

Atkins says truly well-intentioned email senders can find ways around the deliverability problem. “You start small, sending a small amount, maybe a few hundred,” she says. Most of that will end up spam folders, but if you’re sending email that people are actually expecting, they will move it out of their spam folders and into their inboxes. “That’s a huge message to the email provider that they miscategorized something,” she says. From there, you steadily increase the amount of email you send each week, making sure you only send email that people actually want.
But for most people and businesses, that’s more work than they want to manage. So they flock to Microsoft, Gmail, and Mailchimp send their email. Sometimes going big is easier than staying small.