gmail.js: Using api.get.email_source() fails in MS Edge

@josteink I think we’re on the same page here - I meant we should retire the sync version in favor of async as I assume using xhr route will return the email data asynchronously 😃

WRT compatibility, if we can make that function return the data synchronously (without having the library deal with/incorporating all the callback structure/overhead), that’d be a win, otherwise we should retire the function.

That seems a bit rash IMO. Allow me argue why.

Overview

Oh noes. I’ve added headers. It seems this will be a long message 😃

When we want to download the raw MIME-message, the URL we generate (and which we are allowed to make, as a script running on https://mail.google.com/, is to another resource on the same domain.

As @KartikTalwar points out, that’s not where the data is actually stored, so we are redirected by a HTTP 302 to the actual domain hosting the data: mail-attachment.googleusercontent.com.

This means we are crossing a domain-boundary, but since the request originated on mail.google.com, that’s OK, as long as the receiving host (mail-attachment.googleusercontent.com) is also willing to accept our request.

This is what the preflight check is doing. It’s checking for CORS headers, which will list mail.google.com as a valid source for requests. If this header is present in the response, the browser will proceed with the actual, real request.

So for all good, all standard. All browsers are (or should be) doing this. It’s pretty much the spec for how things should work. (Or at least so I’ve been told).

Issues

Now this is where things get tricky, because we have 3 different issues, 2 of them which can easily get conflated, and those are both related to caching:

• 405 response from mail-attachment.googleusercontent.com because of bad request.
• 401 response from mail-attachment.googleusercontent.com because of invalid authentication cookies.
• Character encoding issues due to downloading as text.

The third issue is not really directly related to the others, but if we’re making any changes or API-changes in response to the first ones, it should be considered too, so I’m including it anyway 😃

Issue 1: Bad request

Let’s tackle the bad request first:

The workaround we have for this right now, is sending {cache: false} to jQuery.ajax(), but that unfortunately only seems to work in Chrome right now.

I only know that it works and was baffled by it… so please enlighten me 😃

From what I can tell, jQuery.ajax() is attempting to speed things up by exploiting cached content. Towards this goal, it might add additional HTTP-headers:

https://github.com/jquery/jquery/blob/1.10.2/jquery.js#L8048-L8080

My guess is that mail-attachment.googleusercontent.com isn’t expecting these headers and thus responds with “405 Bad Request”.

By manually implementing a XHR downloader which does none of that fancy stuff jQuery does, I’ve been able to completely bypass this problem. As far as I can tell, it only exists because of jQuery. So let’s not use jQuery 😃

Issue 2: Failing “unauthenticated” requests.

And when we solve that, it eventually leads us to the 401 unathenticated response…

So before diving into the specifics, just to get it over with, let’s remind ourselves of some basics: Just like I can’t take (or guess!) a URL from your Gmail, run it in my browser and grab your email, Google does the same for it’s raw MIME-messages and email-attachments.

To get any of your personal data, I need to be authenticated to Gmail as you. On the web, this is done through the use of cookies.

Yes I know you knew that. But it’s easy to forget. And now we all have it in mind, and will give it proper attention. So let’s proceed with the actual issue.

When trying to (manually via XHR) download the email_source, the initial response will succeed for a single given emailID. And it will succeed for other emailIDs too.

But if you try to request the email_source for the same emailID more than once, things will start failing. With a 401.

To find the reason for this, we need to inspect the actual browser requests and responses:

The request get’s redirected, and as we can see at this point, we have lots of cookies!

One of those cookies (or several, in combination) represents our authenticated login to Gmail.

We follow the redirect to mail-attachment.googleusercontent.com, and get a new response. This response however tells us to go obsolete some of our cookies. Immediately!

NOTE: The following request was issued at 7:53:03 GMT, despite the misleading Date: and Expired: headers:

That means that if we repeat the original request (unmodified) to mail.google.com, the browser will most likely remember the 302, and go directly to mail-attachment.googleusercontent.com.

But if we re-request the same resource on mail-attachment.googleusercontent.com, we will now have a different set of cookies, since we obsoleted some on our last request!

And so, it seems we’re suddenly unauthenticated!

A few observations:

I’ve observed the following:

• Using manual XHR over jQuery.ajax() does less “sophisticated” tricks, and seems to work consistently better.
• Our cookies getting obsoleted immediately may partially explain why we have observed different behaviour on sync-vs-async requests?
• Applying a cache-busting query-parameter seems to somehow bypass the issue of getting unauthenticated requests once we actually manage to cross the CORS barrier, although I haven’t been able to determine why. That is: I don’t see where we get our cookies re-set to a valid value.
• Also: Downloading MIME-messages as string/text-data may introduce text-encoding issues when passed along to a real MIME-parser, as nested messages may have had their original encoding overridden (For instance: MIME message with message originaly in Windows-1252, downloaded, decoded, passed to a string in unicode form. Passed along as a base-64 encoded UTF-8 stream to a real MIME-engine… Which then tries to decode non-latin letters in a Windows 1252 section, but which are now UTF-8. Ooops)…

In conclusion:

• Let’s remove jQuery in favour of manual XHR downloads.
• We need to implement manual cache-busting to allow repeatable downloads.
• To avoid character-encoding issues with email-source, we should have an API/option to permit binary downloads (Uint8Array).

All in agreement say “yes!”, and I’ll do my best to make it so 😃