Outlook 2016’s New POP3 Bug Deletes Your Emails
wp.josh.com
wp.josh.com
The main problem with POP is that unless you do something clever, determining the changes in the mailbox from time t0 to time t1 is both conceptually difficult and computationally expensive. This is because generic POP has no concept of a message UID, so there's no principled way to diff between states. In a very real sense, this means that POP is somewhat broken for the most important use case for the client: syncing to server changes.
The UIDL extension adds a "UID" but it's just an MD5 hash of the contents -- meaning that multiple copies of a message appear to be the same message. And you can ask for just the headers -- which means you can get the message-id header -- but this is still very expensive to do repeatedly (say, every 5 minutes) on a 100,000 message POP store. And you can't ask for just the Message-Id header, which would fix the problem.
Even if you have valid UIDs -- which you won't -- you still have to run a diff algorithm. Typical dynamic programming algorithms are O(N^2), which obviously sucks big time for a 100,000 message POP store.
For Inky (http://inky.com) we use a clever linear-time diff algorithm based in part on [Meyers 86]: An O(ND) Difference Algorithm and Its Variations. [Burns & Long 97] A Linear Time, Constant Space Differencing Algorithm is also a good treatment. But I know Outlook and Thunderbird both use non-linear-time algorithms to diff, so "leave messages on server" gets increasingly (non-linearly) expensive as the mailbox size grows on the server.
A few other points on comments made in this thread:
- POP is still widely used. In the US, for example, Comcast has finally migrated to IMAP, but Verizon is still POP only.
- POP is, from the server standpoint, a very simple protocol, and it is highly amenable to automated testing, as others here have pointed out. For our own testing we generate both patterned and random mailbox modification sequences, then have the test client cooperate with the test server to ensure that the client has (independently) correctly determined what's happened to the (test) mailbox. This is a perfect example of a situation where investing significant effort into automated testing pays off -- and where a TDD approach to development would also work well.
That sounds like a server problem; to quote the RFC,
The server should never reuse an unique-id in a given maildrop, for as long as the entity using the unique-id exists.
Why would you hash the contents? The arrival time should be unique, assuming no two messages could arrive at exactly the same time. That doesn't require any hashing.
I think POP could've been far better designed, without growing into the complexity of IMAP, with just a few little changes like this.
That's why we've had IMAP for a couple of decades now (and available anywhere for over a decade). POP simply wasn't designed for this use case.
That's just good advice for dealing with modern software, for which fixes, breakage, and feature churn are all mixed in a single awful stream. "Newer" does not mean "better."
Life with up to date tools can be real good. So unless I'm simply the only one using good tools, updates can be good.
My primary OS is also Arch Linux, and while it's certainly stable, it's not without its warts. Failure to update regularly on a rolling release distro can have absolutely disastrous consequences (though you only have yourself to blame), and a healthy dose of caution is strongly recommended whenever a major update to important software is in the pipeline (think KDE4 to KDE5 transition). I think this sort of bug (Outlook) illustrates the importance of having an abundance of caution with new software where the failure modes may not be well understood by merit of its relative youth. But with rapid releases, I think the problem is a bit more focused on the end user: Someone who is unable or unwilling to take the risk of updates causing material harm to their workflow or consuming time they can't afford in order to fix potential problems should look for more conservative release cycles. I don't think it's really a matter of "good" versus "bad" tools; that may be part of it, but I can't help myself from thinking it's a matter of misplaced expectations.
That is, it's easy to fall into the mindset of erroneously believing that faster, more rapid updates is always better without fully appreciating their impact. (I've done this more times than I'm willing to admit.)
I do think, and maybe I'm wrong (which I usually am), that those of us who tend toward using rolling release distros have a bit of a bias and a rose tint to our glasses. We almost innately know what the risks are, and I think we take that for granted by assuming most others will freely accept such risks and appreciate the occupational hazards that go hand in hand with change. Not everyone has the same degree of patience, nor the same goals or motives. I think our optimism for and evangelizing of software that pushes rapid releases (like Arch, as an example) can help create an aura that lulls those we influence into expectations that don't mesh well with their use case, their personality, or their constraints. Is that a bad thing? I don't think so (evangelizing is important), because I do agree with you: Updates can be good. I just think we're all too happy to espouse advantages while sometimes glossing over potential drawbacks (guilty again as charged!). :)
Anyway, I should apologize: I didn't mean to wax philosophical. It's late in my timezone, and I saw another Arch user who provoked me into a short essay. I agree with you and username223, but I don't have any real answer. I do think that sometimes we ought to be more cautious with our advice and perhaps weigh context more heavily than our excitement allows. (I made the mistake once of suggesting Arch to someone who really ought to use something with sturdier training wheels. My only saving grace is that he never got around to installing it.)
But with stagnating platforms, like Windows where tipping point was Windows XP (or Windows 7), it goes downhill with every update since then, the bubble bursted, now the user is the product, and your files get screened and searched by the platform owners. You can run software from 1985 (Win 1.0) on Windows 7 (32bit, but it's just an arbitrary limitation to prevent 16bit applications from running on 64bit OS, the competition can do that see Wine and ReactOS). Almost every developer already moved on to the web or emerging new platforms like the market share dominating Android or the second most popular one, iOS/OSX. End users are smart, they don't buy into old antique burned platforms of yesteryear anymore. They application landscape is changing as well. And for everyone who is still happy with their old software, there is little reason to update, and it won't get better on sinking platforms.
Another thing I'd point out - Microsoft almost backed themselves into a corner worshiping at the shrine of backwards compatibility - to the point it was difficult to move their platform or their ecosystems software forward to use more modern, more secure and more reliable methods - so unless you've been very forward looking from the start (see IBM System Z) there is a real, fundamental and painful engineering cost to maintain a line to yesterday without great sacrifices to tomorrow.
I'd argue that the PC (be it Windows, OSX or Linux) is here to stay for the foreseeable future - it may not be the platform for everyone - but for many workloads and applications the web, or mobile simply will not do.
Can you elaborate? System Z always looks curious, but I don't think that many people who aren't involved with mainframes professionally had a chance to even look at it
It must be nice in your filter bubble.
Some genius at Google decided to make the phone vibrate and beep every time there is an open WiFi spot, or you need to sign up to a known one
Really
This kind of crap (not the only one) almost justifies the extra price for iOS
- When I get called, half of the time I don't see the name of the person when they are in my address book.
- Music controls (which were already very basic) usually stopped working a certain amount of time after the last restart.
- Music randomly pauses when browsing the web at the same time (as in, I have to open up the music notification card and press 'Play' again).
- E-Mail notifications have been flaky for some reason.
- Active display music controls are worthless again. Google decided to add a 'favorite' button. Motorola just picks the first three buttons from the notification card. Now you can't go to the next track anymore.
Security updates were also still at the 1 November 2015 level. After being stuck for half a year on a buggy 5.0 release before getting 5.1, I know this is going to take months to fix, if ever (Lenovo probably doesn't care about the 2014 anymore).
I am now back to an iPhone after a Nexus 4, Moto X 2013, and Moto X 2014.
15 years ago when people had problems with their graphics cards, the standard "fix" was to update the driver. Now we're still updating drivers. Weren't the problems supposed to have been fixed many years ago? It seems they introduce as many new bugs as they fix, making the net effect of updates useless as far as bugs go.
For security related bugs, stop using C++ for internet facing software.
Your statement makes sense if you believe that graphics card manufacturers were still working on drivers for 15 year old cards and 15 year old APIs.
Manufacturers are always putting out new cards, and there are new APIs to support. Writing drives for dozens of chips spanning multiple generations of graphics architectures to work on multiple OSs and supporting multiple APIs is incredibly challenging.
Even if there was just one chip and one OS and one API to support, getting maximum performance across multiple applications is non-trivial. Drivers have a lot of heuristics to try to maximize performance, and manufacturers invest heavily in tuning those heuristics even to specific games to get the best results.
If you aren't a gamer, then, yeah, you can run three year old drivers and it won't affect the speed of scrolling the text in your browsers or whatever.
I still use Visual Source Safe for my personal projects. It's never failed me in 20 years. Right click, check out, right click checkin. Perfect every time.
1st reaction: Horrified that you are still using VSS.
2nd reaction: Was that if it does what you need it to, well, then, who am I to judge? It doesn't matter whether the tool was created yesterday or 20 years ago.
3rd reaction: Does it really do what you need it to do? I would guess that maybe svn/git has some simple featured that would significantly improve your life.
4th reaction: Wasn't VSS known as being super buggy? Is it just luck that you've escaped failure in 20 years?
And yes, VSS is very buggy, although the bugs might not show up for a single-user environment always following the happy path.
I think your reaction is more horrifying than me not having a compelling reason to change the way I've been doing things without flaw for decades.
I do however get the objection to newer is better, I keep getting besieged to move my projects from svn to git - to which I usually respond "Why, tell me what feature we need in git, that svn doesn't do?" I've yet to get an answer.
- Familiarity. Everyone uses git nowadays; like it or not, svn projects are the odd ones out.
- Fast, offline querying of the project's history. With git, they have a full copy of the project's history in their local computer, while with svn, any query has to go to the server. This helps a lot when chasing regressions, or just when browsing the changes between one release and the other.
- Easy branching. Branches in git are more lightweight than branches in svn, and git's merge functionality is quite good. When they want to propose some change to your code, they can just create a branch in their local copy, make the changes they want, publish the branch somewhere, and ask you to merge it; this is made even easier by sites like github. With svn, unless they have an account in your svn server, they have to do it the old-fashioned way.
As well as the oft-unjustly-maligned check in/check out model, two other things it has going for it are that administration is pretty easy and it has both a command line client and a GUI client. (Which might sound like a ridiculous thing to say, but I have used some systems that have one but not the other - hopefully very rare these days though.)
This isn't really a recommendation for it, though, unless they've put a huge amount of work into it over the past fifteen-odd years (which somehow I doubt).
To get the same check in/check out workflow, you could try Perforce (though I heard from somebody that did it on AWS that the initial setup can be a bit fiddly) or SVN (though check in/check out evidently isn't quite how it's designed to be used, because (a) this is not the default, and (b) the performance can be pretty crappy).
I may have to borrow that line at my day job sometimes.
A friend and I, him no longer on Facebook, me without it at this point, both have stickies on our cameras all day that we peel off to talk on Facetime. And as we were chatting, he mentioned someone who was a power user but didn't know much. He was downright promiscuous with the software he used, just, terrible. He never even thought of reading a EULA, had every app that fit on his phone (while my friend and I want nothing to do with smartphones), knew how to use all the features...Perhaps we're slightly paranoid users, but it's these tiny religions that keep us from spilling our guts to the whole planet. And it happens, and there's no undo. I know what computers can do, and it's not fun to think about what happens when yours is commandeered. Especially if you look at the situation antagonistically and as a programmer.
I think of old applications, and old computers, the way I think of generic pills: you know what you're getting, the luster is gone from the competitor that once made it look like shit based on vaporware promises, you can trust it no problem, and it's super cheap. And you're used to it, you know the rules, you know how it works. New hardware is expensive, but new software is worse: you pay either in loss of data, highly-targeted advertising, price discrimination when you finally buy something, irresponsibility with data, worrying about getting hacked, getting hacked, countermeasures, getting spammed, or in a mundane but expensive way as with the $5 a month that Danish guy said every app should charge. But most apps just can't man up and charge you a fee, or tell you to go screw, instead, they play it sleazy, they want to traffic a tiny part of you.
Right click, branch, right click, merge, right click, tire fire, right click, throw computer out window.
Wait! What?! Make sure you do not run out of disk space or do any of the other things that instantly corrupt your VSS repo. Hopefully you were being sarcastic.
Well, to each their own I suppose
Software developers know this, which is why they maximize the scariness of all security fixes, and minimize your ability to separate security fixes from all other software changes. How much churn did you have to swallow to fix "algorithmic complexity attacks" you never saw, which would at worst result in a DoS? Did anyone actually attack your hash tables?
My experience was that most clients sucked.
For example, one (Eudora?) would move IMAP messages by copy and delete (which may have been idiomatic, as I say it was a long time ago). But it wouldn't check the success of the copy operation, so a failure to copy would result in a delete, not a move.
Seems they still suck in 2016, for what seem like trivial reasons.
I mean, surely this sort of protocol interaction is very, very amenable to automated testing of some sort. We had a bunch of automated regression tests for our mail servers, written in C, back in the late nineties.
I'm genuinely uncertain whether to blame incompetence or another attempt at the strategies spelled out in the Halloween Documents: http://www.catb.org/esr/halloween/faq.html
For a program like this, there ought to be a Sacred Core that Does Very Little and has implementations of key protocols that can’t be touched without the blessing of about 5 senior engineers and the personal seal of the CEO or some such.
In other words, it should be unbelievably hard to screw with parts of the program that are crucial, while making changes that shouldn’t have anything to do with it. (Heck, for all we know, they were adding Windows 10 Tiles™ when this screw-up occurred.)
Look, I know that mistakes can be made. But in this case, I just can't think of a single excuse that would be satisfactory.
Look what they do with IMAP support in Outlook 2016. Pre-loading just the "Subject" (a feature supported since 1990s) got dropped, instead of the whole email incl attachments is downloaded.
And now we learn POP3 got crippled too. The common Microsoft tactics to lock-in end-user and enterprise customers to proprietary ever changing protocols only fully supported by their most recent server software - their Exchange and Outlook.com/Office365 cloud eco-system.
Microsoft wants you to upgrade to Windows 10 on PC, adopt it on smartphones (even it has just 1.1% market share), on servers, SQL-Server 2016, Exchange 2016, Office365 (subscription based Office 2016 client, cloud based SharePoint 2016 = OneDrive for Business on Azure), Skype for Business (=rebranded Lync), etc. Oh, and they ask you to install their telemetry services for Office on your clients, to get a "full picture". The telemetry and other phone home stuff cannot be turned off, except in the expensive LTSB license version. And several IPs and URLs are white lists in the kernel mode network layer, in all versions.
It's up to you, to help Microsoft to create another monopoly. Or you out smart them and say no.
To be fair, oOo and LO always warn you when you want to save a file in MS Office format. Mind you, you can turn the notification off.
Food for thought: one of my employers now deploys OpenOffice as well as MS Office after years of resisting that because of fears of user confusion. Guess what? No confusion apparent.
Oh, you use Autodesk products, you'll need Windows for that.
"We created a shared Word doc / PowerPoint in OneDrive for the report" - only editable with MS Office.
My choice is work or no work.
It's so sad, because LTSB is quite a good system. My home system was eligible for Windows 10 upgrade, which I did - then loaded LTSB as soon as I found out about it and replaced it, so my conscience is somehow clear about running it.
It reminds me of good old Windows 7 days and updates aren't breaking the system weekly.
You still have to disable various telemetry crap though.
That said, yeah, this is something they definitely should be testing. The fact that the article updated yesterday with a new knowledge base article and the status of the problem says they are looking into it, means they've been notified but not yet fixed it. If they haven't stopped that update from rolling out, someone at MS may come back to very unhappy managers on Monday, and have to explain why they thought leaving a data destroying bug live over the weekend was the right call...
I was trying to fix email for a friend recently. Her UK ISP's email service is a rebranded Yahoo Mail account.
It works fine on her iPad, but LiveMail in Win7 (she doesn't want Win10) regularly chokes on attachments.
Turns out LiveMail can't cope if there's spam in a Yahoo trash account. And this bug has been around for years now.
Not sure if incompetence, corporate dementia, or some nefarious plot to get users to switch to <whatever Microsoft's mail service is called this year.>
Either way, I suggested she switch to Gmail.
Empirically, it seems to be less of this problem occurring on the connection to the Exchange server, back when I was forced to use before. (Though at that time didn't have had so much of the connection, but had plenty of issues some involving broken database and such, I wouldn't touch with a ten foot pole myself...)
I've found eM Client to be a good alternative to Outlook and WLM.
UPD: I was just about to persuade client to move to Outlook 2016 from 2011. Glad I did not.