rsbadger

Unfortunately not - the links were indexed and shown in Bing search results

I do have a robots.txt to block this directory. But Bing only listens to that for what to crawl, not what to index.

I’m fairly certain they are. My links ended up indexed in Bing search results. The only place they were ever rendered was in private emails to users. Bing should not be indexing that.

Exactly that.

This is very useful information. You’d really hope that private emails would be excluded by default…

The only purpose of this link was to verify that the email address is valid. Once it’s verified, you can login.

Many people (most?) prefer to signup to services by email address. To do so, those email addresses must be verified. How would you verify it without sending them an email link?

You’re right that it was a bit of an oversight on my behalf, as the links were only generated after a verified human user action (signup) I had assumed the 1 time links to their email would be safe. But regardless of the link action, it shouldn’t be passing that data to Bingbot to crawl and (possibly) index in search engine results. Private email data should not be shared with search engine crawlers IMO.

I would be pretty mortified if browsers were using user browser sessions to scan content and pass it to bingbot…? What about if you’re browsing something local? Or your bank account?

Almost all of them. A good example is Dropbox link you send to someone. I could generate this link to a private file in my Dropbox, email it you, and Bing (may) index it.

https://www.dropbox.com/s/vucien2ns8jktga/denim%20bodywarmer...

I doubt many people realise this when they email "private" links...

Exactly - seems the safest way is to explicitly block known bots by user agent from even reaching pages you don't want indexed.

You may be seeing a stale version, try this: https://shoprocket.io/robots.txt?bypass=1

(I made a lot of changes today when testing all, including "visit as Bingbot" from their webmaster tools with and without the URL blocked by robots.txt)

Quite possibly…

They were indexed in Bing results, I’ve shared the URL of that in this thread

I don't think that's very surprising for most people, the real takeaway is that not only will Bing read your emails, but they may also index any links you send and serve them in search results.

Exactly. I also noticed Bing had accessed some non microsoft tokens too. Even gmail accounts were affected. I assume some people have connected their gmail account to the outlook client?

Exactly. From what I've heard today it sounds like most apps have an extra step between the email link and the login, usually a JS step, to check for bots.

Not adding that was my downfall I think.

AFAIK, most of those "private" links are actually just "unlisted" but they're still public. I'm sure Bing is indexing those too...

Absolutely

Yeh I was a bit reluctant to post that as it doesn't look great for my app! But here's the results: https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2F...

Yeh I think you're right, an extra JS step on the page that the email link leads to would help a lot.

They are now - I didn't think I had to as all the pages are naturally behind a login, it never crossed my mind that Bing would follow email links, let alone index them in search results.

yeh...

https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2F...

Scanning something for malware and publishing it in search results seem like 2 completely different things to me...?

It was blocked by robots.txt but Bing chose to ignore it. I even tried "blocking" the URLs in Bing webmaster tools today and this was the response:

"Block request denied We found that the URL submitted for block is important for Bing users and hence cannot be blocked through Bing Webmaster Tools.

We recommend that the best way to block URLs in this scenario is to add NOINDEX meta-tag to the HTML header of the page."

Essentially yes - but "previewing" link and sending that link to Bingbot to crawl and index is another matter.

Imagine you send someone a "private" link to a file...Bing sees that and indexes it for the world to see. Not cool.

Exactly what I was thinking - what really worries me is it also seems to have happened to a lot of @gmail users too. I still can't figure out how Bing managed to find email tokens sent to gmail. Maybe users who connected their gmail account to Outlook...?

Yup this is true - I was just being lazy. But what surprised me was that Bing actually indexed them. (even though my robots.txt said not to)

I found them in Bing results

That’s actually a very good point, when I made the static image our rating was 5 stars, as the only negative review had been removed (because the user was complaining about not getting emails from us, but they had just given us the wrong address so easily fixed) we had that review flagged, but it was eventually added again by trustpilot so the score was reduced a bit. I’m still trying to get that user to update their rating as they’ve been using shoprocket happily since they entered the right email address!