What We Found Scanning Millions of Email Systems

As a company focused on email technologies (check us out here and here), we strive to really understand how email is being used. And so we built something new: a system to scan and categorize millions of email systems. The result? Surprising insights regarding email usage, security and performance. In this post, we announce some key findings. In subsequent posts, we will discuss other topics such as who is winning the Google vs. Microsoft battle for email, who the top hosted service providers are, cool pie charts and more. Did I mention we’re hiring?

Scanning Millions of Email Systems

In order to scan millions of email systems, you first need to find millions of domains. And to find domains, you can’t just query DNS and get a complete list of domains: you need to crawl the internet. That is exactly what we did using a combination of cloud computing, NoSQL database technology, and PageRank-like algorithms. We classified each domain into a specific category based on content (ex: health care vs. personal blog). We queried MX records in DNS to obtain the IP address of associated SMTP servers. Another classification algorithm takes care of analyzing SMTP banners, as well as other protocol responses to determine the type of email system used by each domain. The result: a scalable SMTP classification engine with a 85%+ success rate for millions of domain.

Many Email Systems Lack Protection

We found that 0.4% of email systems had an open relay. Multiply this figure by millions of domains and you end up with thousands of systems available to botnets and spammers. However, we believe that some of those SMTP servers were false positives. Some servers may accept relay commands, but in fact ignore submitted emails (we can’t confirm either way because, being good citizens, our probing logic doesn’t actually submit emails). Others SMTP servers may also be honey pots used to identify attack vectors. Another key finding: only 78% of email systems supported TLS authentication, leaving 22% of SMTP servers with only clear text authentication available.

The World’s Leading Email Systems

Overall, Exim (20%), PostFix (16%), Google Apps (9%) and GoDaddy (4%) lead the market. However, many large hosted service providers (i.e. hosting hundreds of thousands of domains) include a free or low-cost email package. Focus on companies with 200+ employees and you end up with a very different picture. Enterprise solutions such as Microsoft Exchange (14.7%), Google Postini (11%) or Symantec Cloud (8%) lead the charge. Postfix remains a serious contender with 7% of the enterprise market. Venerable SendMail only represents 5% of the enterprise market and 3% overall. As for the overall leader, Exim, it drops from 20% overall, to 2% of companies with 200+ employees.

A Unique Situation: Universities

Microsoft’s Live@edu program offers free, comprehensive access to its cloud Exchange 2010 platform to educational institutions. Google has a similar program providing free access to its Google Apps email platform. So it’s not surprising to see Google Apps (12%) and Microsoft Exchange (12%) emerge as neck-and-neck contenders for the education sector. Interestingly, many universities rely on PostFix (11%) or even enterprise solutions such as Barracuda (9%) or Postini (4%). Also, there was a much larger variety of email platforms being deployed at universities, including custom ones. This is not surprising because several known email platforms were initially developed by the Computer Science departments of leading universities.

Email Delivery Speed: Not Sub-Second

What about performance? How much time does it take to deliver an email from point A to point B? To deliver an email, you need to go through different steps including looking up MX records in DNS, establishing a connection and then performing an SMTP transaction. Of course you also need to send the data, but let’s ignore that part. We found that, on average, it took 0.3 second to establish a connection, and 1.4 seconds to complete an SMTP transaction. The good news is that 34% of domains could complete an SMTP transaction in under 1 second. The bad news is that 42% of domains took more than 3 seconds. So if you’re sending an email to a friend, don’t expect sub-second delivery (if you do, give SMTP Logic a try).

Email Reliability: Lots of Redundancy But Poor Maintenance

Email delivery was designed to be reliable. SMTP servers will retry delivery several times before giving up (typically after a few days). Domain administrators can specify more than 1 SMTP server in DNS so as to provide redundancy. As expected, for companies with more than 200 employees, the average number of MX records is 2.4, so enterprise do realize the benefits of more than one MX record. However DNS maintenance is poor with 5% of SMTP servers not responding. These are probably “forgotten” MX entries left behind due to book keeping issues.

Posted in Email, SMTP Logic
3 comments on “What We Found Scanning Millions of Email Systems
  1. Daniel says:

    Very interesting and good info! Thank you for summing this up! best regards

  2. […] Reliability of Email Systems SMTP Logic scanned millions of email systems and found most to be lacking protection, maintenance, and speed. Share this: This entry was posted in IT News Roundup. Bookmark the permalink. ← IT Operations News Roundup — Sept 24 to 30th Cancel Reply […]

  3. […] What We Found Scanning Millions of Email Systems: A run-down of all the information that we collected after we scanned and categorized millions of […]

Comments are closed.

Stay Tuned

Get every new post delivered to your Inbox.

%d bloggers like this: