Test Email sending with cypress

See also Wesley Aptekar-Cassels' post End-to-end testing emails for an alternative approach that uses playwright and a local SMTP server.

The company I work for uses its own little email newsletter software.

I wanted to write an automated test to make sure the newsletter works correctly. In the spirit of full e2e testing, that meant verifying that newsletters are actually sent out, and that the unsubscribe link at the bottom of every email works correctly.

To that end, I used the imap npm package. If a package has 100,000+ downloads a week and the last commit is 5 years ago, that's a good sign that it's either feature-complete or abandoned. And, well, imap isn't abandonded seeing as they're still answering issues (shoutout to mscdex).

Problem #1: figuring out IMAP

When you subscribe to our newsletter, you immediately get an email asking you to confirm your subscription. This is standard practice to prevent sending newsletters to people who don't really want them. When you click the confirm link, you get another email congratulating you on your choice. That second email contains an unsubscribe link.

The e2e test consisted of the following steps:

To keep the test efficient, I wanted a notification as soon as an email lands in the test inbox. imap (the library) has an overwhelming amount of methods and options, which are very well documented but not really understandable if you're someone who really has no clue how IMAP (the protocol) works. Long story short, you need to:

Here's the mostly complete code for your reading displeasure:

const Imap = require('imap');
const imap = new Imap({
  // you can probably get these values from your email provider
  user, password, host, port, tls,
});
/**
* connects only if we aren't already connected
* @returns {Promise}
*/
function connectIMAP() {
  if (imap.state !== 'authenticated') {
    return new Promise((resolve, reject) => {
      imap.once('ready', () => resolve());
      imap.once('error', err => reject(err));
      imap.connect();
    });
  }
  return Promise.resolve();
}
/**
* waits for a new email and returns its entire content as a string
* @returns {Promise}
*/
function getEmail() {
  return new Promise((resolve, reject) => {
    imap.openBox('INBOX', true, (err, box) => {
      if (err) {
        reject(err);
      }
      imap.once('mail', () => {
        const fetch = imap.seq.fetch(box.messages.total + ':*', { bodies: '' });
        fetch.on('message', (msg, seqno) => {
          msg.on('body', (stream, info) => {
            let content = '';
            stream.on('data', chunk => {
              content += chunk.toString('utf-8');
            });
            stream.once('end', () => {
              resolve(content);
            });
          });
        });
      });
      imap.subscribeBox('INBOX', err => {
        if (err) {
          reject(err);
        }
      });
    });
  });
}
// here's your email:
connectIMAP().then(() => getEmail())

Problem #2: Invalid Version

This is what I got running that first test:

The following error originated from your test code, not from Cypress.
  > Invalid Version:
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.

We all love a helpful error message.

After spending way too much time researching, I found out that this issue means "you're trying to run node.js code in the browser, dummy.". Which is because cypress runs its tests entirely in the browser, unlike traditional selenium-based test frameworks.

To use node.js apis in cypress, you need to create a plugin in the aptly named plugins.js file, and call it from your test code with cy.task() . Here's what that plugin looks like in its simplest form:

/**
* @type {Cypress.PluginConfig}
*/
module.exports = (on, config) => {
  on('task', {
    async "mail:receive"() {
      await connectIMAP();
      return await getLatestEmail();
    },
  });
  return config;
};

It doesn't get better in the actual test code:

it('can subscribe to the newsletter', () => {
  cy.visit('http://localhost:54321/newsletter');
  // enter fancy test email and click 'subscribe' here 
  // you probably need a higher timeout than the default 60,000 ms
  cy.task('mail:receive', null, {timeout: 240_000})
    .then(email => {
      // check that email is ok and click confirm link
      // wait for second email
      return cy.task('mail:receive', null, {timeout: 240_000});
    })
    .then(secondEmail => {
      // check that second email is ok and click unsubscribe link
    });
});

You can't use await in the test code because, well, you can't have everything in life.

Problem #3: Quoted-printable

At this point, it was like 1 am and I was really counting on that test working. And it did! If by "work", you mean, "produced a new fun error". While I was able to received emails, the parsing still failed, because the email content looked like this:

// excerpt:
Viele Gr=C3=BC=C3=9Fe,

This is supposed to say "Viele Grüße" which is German for "Best Regards", but all the Umlauts were messed up. At first sight, this seems like an encoding issue, but I double-checked and the Email was sent as UTF-8 and I read it as UTF-8.

Well, turns out there are encodings on top of encodings. We live in a wonderful world.

This email used quoted-printable encoding, which is a way to convert utf-8 text into something that can be transmitted over a system that isn't 8-bit clean. Tl;dr, if you send a byte with the highest bit set, and it comes out the other end with the highest bit unset, then that system isn't 8-bit-clean. The reason such systems exist is because that's not an issue if you only use ASCII characters, and historically a lot of apps only used ASCII characters.

That aside, I now needed a way to decode that quoted-printable into actual utf-8 text. Fortunately, there's a library for that. Another one where the last commit was five years ago!

That library returns a byte stream, which needs to be utf-8 decoded again:

const quotedPrintable = require('quoted-printable');
const utf8 = require('utf8');
// ...
utf8.decode(quotedPrintable.decode(content));

With that, the test worked!

Final thoughts

In my setup, it takes on average ~100 seconds for the email to arrive. I'd imagine that different email providers could be faster (or slower) than that. Either way, testing mail will take more time than your usual "click around on a website" e2e tests. If you're in an environment with very frequent code changes, you might want to limit when you run those specific tests.

Hope this post helped you, or at least was interesting. I'm going to bed now. love you all. gn