Amazon SES Deliverability Test Results

YAHOO_window_on

EmailExpert has been testing the new service from Amazon, the Amazon Simple Email Service. A cloud based barebones ESP similar to a service like smtp.com or SendGrid but far cheaper.

We will be publishing more findings over the course of this week. We wondered about deliverability from the cloud, and reading statements like this:

“Amazon SES has been invaluable to our business by letting us no longer worry about email deliverability ” said Patrick Lightbody, Director of Product Management at Neustar, Inc. “For our self-service SaaS business, the email signup process must be flawless. Any delay in delivery can have significant impact on our signup rates, which require email verification, and Amazon SES has already improved those signup rates by about 10%.” Meant we were keen to test the service for ourselves, which is what we did.

One of the first things we did was to subject Amazon to the spam analysis tests offered by Litmus App

These tests check to see inbox placement across a wide variety of ISP’s and a number of spam filters. The tests are fairly exhaustive and it would appear that Amazon passes the ‘deliverability’ test with flying colours, based on these initial tests. Our test messages sent via Amazon SES consistently achieved inbox placement. We will continue testing at larger volumes and provide further details and analysis shortly.

We have already conducted our own testing of inbox placement to yahoo.co.uk and a number of other services with the same results repeated, consistent inbox placement.  With Facebook email arrives in “other” as is expected until added to the inbox rules. It should be noted email was sent using the command line interface from our server and we did not utilise the API which would be required to sign messages with DKIM.

Spam analysis & inbox placement

MessageLabs Passed

MessageLabs is a  corporate server-side spam filter. It is in use by many large organisations .

Postini Passed with a score of 20.69868

Postini is another server-side spam filter, owned by Google.

Barracuda Passed with a score of 0.6

Barracuda Spam Firewall is an expensive hardware spam filter that is installed by large organisations within their own datacentres.

Gmail Passed

The filter offered as part of Google’s Gmail service. Recipients can aid delivery by adding the sender’s email address to their address book. That’s not been done here, still the mail was inboxed.

Yahoo! Mail Passed

The filter offered as part of Yahoo’s mail service. Recipients can aid delivery by adding the sender’s email address to their address book. That’s not been done here, still the mail was inboxed.

SpamAssassin Passed with a score of 2.2

A very popular open source spam filter. Here we are testing against version 3.2, using its default settings.

Mobile Me Passed

The filter offered as part of Mobile Me service. Recipients can aid delivery by adding the sender’s email address to their address book. That’s not been done here, still the mail was inboxed.

Sender Policy Framework Passed

Sender Policy Framework (SPF) is an e-mail sending validation system designed to prevent e-mail spam by confirming the sender is permitted to send via their host. SPF allows administrators to specify which hosts are allowed to send e-mail for a domain.

DKIM Failed

No surprise as EmailExpert made no attempt to sign with DKIM

DomainKeys Failed

No surprise as EmailExpert made no attempt to sign with Domain

Sender ID Passed

SenderID is very similar to Sender Policy Framework (SPF) with a few additions.

GMX Passed

The filter offered as part of GMX service. Recipients can aid delivery by adding the sender’s email address to their address book. That’s not been done here, still the mail was inboxed.

Hushmail Passed

The filter offered as part of Hushmail service. Recipients can aid delivery by adding the sender’s email address to their address book. That’s not been done here, still the mail was inboxed.

FastMail Passed

The filter offered as part of Fastmail service. Recipients can aid delivery by adding the sender’s email address to their address book. That’s not been done here, so this represents a more pessimistic spam scoring..

Lycos Passed

The filter offered as part of Lycos service. Recipients can aid delivery by adding the sender’s email address to their address book. That’s not been done here, still the mail was inboxed.

Mail.com Passed

The filter offered as part of Mail.com service. Recipients can aid delivery by adding the sender’s email address to their address book. That’s not been done here, still the mail was inboxed.

Further Results

You can view the results of the tests for yourself here: http://emailexpert.litmus.com/pub/6d05dbb

Last updated by at .

Andrew Bonar

The founder of emailexpert.org, Andrew Bonar currently resides not far from Sydney in Australia where he performs his primary role as Postmaster for self-service ESP Campaign Monitor

In the past two years alone Andrew has been responsible for the delivery of in excess of 120 Billion messages. With more than 15 years of industry experience, Andrew is widely recognised as a leader in the field of message sending, deliverability and compliance.

In 1996, he co-founded the UK’s oldest privately held ISP, Cheapnet Ltd. In 1998, launched the UK’s first privately held eCommerce payment systems: eBanx Ltd, and in 2003 he launched two of the very first ESP’s in Europe: MailPhoenix and eMailGenie.

From 2006 Andrew served as an independent consultant at organisations throughout Europe, the Middle East, Asia Pacific and the US. More recently serving as Worldwide Director of Deliverability at Emailvision, managing deliverability operations in 22 Countries. Andrew continues developing and evangelising best practices in permission-based marketing with clients and industry associations and travels extensively in Asia, Europe and North America to fulfil these obligations.

9 Comments

  1. Martin said:

    Hi, I use EZS3.com to make it easy to use AWS for videos. We send a lot of mail, is there a service that would make the integration with Amazon easier, perhaps something open source?

    • Andrew Bonar said:

      Hi Phil

      Sorry for the confusion…
      They do support DKIM. There are a couple of methods you can use to communicate with AWS SES one of which is the API.

      Only the API supports DKIM signing. What I said was “we did not utilise the API which would be required to sign messages with DKIM.”

      Therefore as we did not sign the emails we sent, the DKIM checks failed.

      Hope that clarifies.

  2. Pingback: Emailblog.eu » Amazon launches email solution Simple Email Service

  3. Jordie van Rijn said:

    Thats great, so the test is more than just testing the e-mail. As litmus also has the capability to send a test via them (and not directly to the ISPs) it was a quick mistake to make on my part. thanks for clearing that up.

  4. Jordie van Rijn said:

    It seems that the spam test by Litmus is a nice way to test deliverability (problems). I guess the filter for outgoing email that Amazon SES uses itsself works in a similar way for a part.

    But the Litmus test checks the contents of the mail, setup of the domain validation, etc. If i am not mistaken its not so much the reputation score, because otherwise you couldn’t test it via Litmus but had to send it yourself. Again a good pre-flight check, but the reputation check has to follow.

    • Andrew Bonar said:

      Hi Jordie
      You are mistaken though, we did not test “via litmus”, we sent the tests direct from SES to the ISP’s, Litmus then checks for inbox placement and shows a screengrab for proof (as is indicated in the image for this post).

      It is very much the reputation score, the email is tested against every major spam filter and for inbox placement at almost every major ISP – that is my yardstick for deliverability.

      The message was sent from the SES cloud to GMX, Yahoo, Hotmail etc. It was also tested against MessageLabs, Postini, Barracuda and SpamAssasin. The tests were extensive, and repeated.

Comments are closed.

Top