Hacker News Reader

Having followed the history of Docker since its LXC roots and it decided to write its own LibContainer library the only thing I want to point out is:

> It's not correct to say Docker was designed "not" to do persistent data.

Actually...looking at a lot of things "missing from" or "difficult in" Docker can more or less be traced back to Docker's overriding philosophy of container runtimes. Things like not having an init process and optimizing user experience around single "Run" commands.

Having relatively recently done a container retrospective, following the path of:

OpenVZ -> LXC -> Docker -> now (CoreOS, Rancher, LXD, Mesos, etc.)

I can't help but feel as if some of the "limitations" of Docker were more or less designed.

I'm not saying it's a bad thing, being a Python enthusiast I see value in simplicity/having only one way of doing things...but I just wanted to point out that where you draw the line for what is considered "designed" starts to get blurry in areas of Docker's chosen implementation.

Don't forget that DotCloud/Docker was originally basically a clone of Heroku, where "12 factor" dictated that all persistent data was stored in (non-containerized) Postgres.

That's incorrect. Dotcloud ran both stateless and stateful services on the same container fabric, unlike Heroku and every other PaaS. Specifically you could run a managed containerized postgres, mysql, mongodb, redis, rabbitmq, and custom stateful tcp services. Docker was a result of that design.

Docker was specifically designed to NOT impose a model of ephemeral containers. For example, by default 'docker run' preserves all data.

We use https://github.com/l3nz/whaleware for persistent, database-oriented workload with persistent data. We used to keep persistent data on a data volume, but now mount a native directory instead (2x faster).

Might not be the True And Approved Docker Way, but who cares? it works well, and has been in production for a number of years now. No data loss ever.

> note there is also devicemapper

You know that trick the post mentions about how you have to keep removing images to stop your machines running out of disk space? Yeah, removing images doesn't actually reclaim disk space on devicemapper (!!):

https://github.com/docker/docker/issues/3182

Oh, and devicemapper is the only natively supported storage backend on the AWS official AMI.

You have to use devicemapper in direct-lvm mode in order to get free space back. I agree it's not the best out-of-box experience, but so far we haven't had any problems running ECS on CentOS instances since switching to direct-lvm.

Yes, although loopback mode for devicemapper is the out of the box configuration for Docker even though Docker themselves doesn't recommend it for production. I had tons of issues w/devicemapper on CentOS 7 (kernel panics causing AWS to mark the host as unhealthy and kill it off) that I was finally only able to resolve by switching to AUFS under Ubuntu 16.04, which has worked pretty well for me. Knock on wood, AUFS has been pretty solid so far.

except when it comes to a many-level-image, AUFS performance degrades dramatically

Yup, this is how I wound up with a 45 GB image on my development machine after futzing around with Docker for two hours. Does the same with OS X.

> It's not correct to say Docker was designed "not" to do persistent data.

Any monkey can run a bunch of stateless systems. It's the stateful ones that have the worthwhile problems.

> If you hit this many problems with any given tech, I would suggest you should be looking for outside help from someone that has experience in the area.

I suggest using something that works for you and your business instead. If the only thing that works is $SOLUTION, but you have these difficulties, then yes by any means look for help.

Edit: Ok, I feel compelled to extend my reply as I've been downvoted without comment, which is terrible.

Here's the thing: You're not forced to use Docker just because it's the next hyped thing. It is great and yes, I use it. However there is more than way of doing things and the only reason to choose one technology over another is if it's better for the business.

Anything else is just ignoring reality.

I did not downvote you but it sounds like the author works at one of those horrible places where developers get to do things that make no sense in production without undergoing an operational review. He probably cannot change that (change would have to come from devs in a company like this), so your advice boils down to finding a new job.

More specifically, it sounds like it's one of those places where developers got to do things in production that didn't make sense.

Anonymous downvoting is pretty popular around here. I agree it's a very rude thing to do. Also, just an FYI, I was scolded the other day for acknowledging the downvoting in an edit.

Edit: case and point

Please don't be surprised by this behavior. It's in the guidelines to do this and is what most of us would prefer.

> Please resist commenting about being downvoted. It never does any good, and it makes boring reading.

> Please don't bait other users by inviting them to downvote you or announce that you expect to get downvoted.

By "most of us" who exactly are you referring to? I wasn't aware there was some kind of user survey etc that was used to determine that stuff. I thought one person just added that to the bottom of the comments guidelines. Was there a vote or something? I'm very curious how you know that is what "most of us would prefer".

Can you point to any evidence that people enjoy talking about downvoting? More people are allowed to upvote than are allowed to downvote so if your post is negative, you're clearly doing something the majority of people reading it disagree with.

Can you point to any evidence that people enjoy being down voted without an explanation? Clearly the fact that there has to be two entries in a fairly short comment guidelines points to the reality that many people have expressed that they do not like being down voted without any explanation.

And since it is such a frequent occurrence, would that not point to the fact that people do enjoy talking about being down voted?

well the article had some issue's especially around coreos, but it's good that somebody points it out.

Actually CoreOS produced pretty stable stuff: etcd, fleet. Actually with fleet/etcd combined with some cgroups i see barely a reason to use docker, I mean on Cloud's its actually another useless layer, at least for me, maybe not in Google scale.

container engines provide apis that orchestration softwares use to manage containers in all kinds of ways. In small setups with not much logistic going on i guess you can manage everything with fleet and some systemd units

Minor: Ubuntu 14.04 can run an LTS 4.4 kernel via the linux-generic-lts-xenial package.

> - Docker do not own CoreOS and did not create the distro

Not just that, CoreOS is basically made to push systemd-nspawn as a container framework as hard as possible.

rkt which uses systemd-nspawn was started after docker shifted focus from just the containers to the whole environment

Is that a problem?

I would like to add another factual error to the list - you can get linux 4.4 on debian stable with the backports instead of switching to an unstable release, and the current ubuntu LTS (16.04) runs 4.4 as well.

It seems that they started to use docker while it was changing a lot, I didn't have any issue in the past few months with docker, but I don't use it in production.

1) You've been running Docker for a few months. It could still be changing a lot, a few months is no proof one way or the other.

2) You are not running it in production. Poster is. All sorts of issues you don't care about on your dev box matter a lot in production.

IOW, your experience can best be summarized by "just trying out Docker in a totally different way and it works great!" Irrelevant to this post.

"Issues with storage" is an understatement. Linux has major issues with storage. I see no way forward for Linux in that area unless they adopt ZFS.

How many filesystems has the Linux world created in the last 20 years? Of those how many are rotting piles of tire fire?

Filesystems are hard to get right.

ZFS is the way forward. For cross compatability, for reliability, for stability, for lots of use cases.

Btrfs is another tire to throw on the burning pile of other filesystems that won't work out in Linux.

Let the down votes begin.

But Linux has some major architectural issues that eventually the Linux faithful will have to admit to.

Containers.

Filesystems.

Observability. Though bless him Brendan Gregg is trying his hardest to help here.

Portability.

And this is what is so painful for the rest of us to watch or deal with. This inability for the Linux folks to admit these glaring architectural problems. Problems they refuse to look outside their bubble to see how others solved these problems and adopt solutions that have already solved these problems. They just want to continue to bury their heads in NIH soil. And double down on trying to solve these issues, poorly, on their own without any awareness of how others have solved these issues. People outside Linux land just might have the right ideas on how to solve these problems!

But I expect no one in that camp to acknowledge this and down vote away.

Sigh.

I can see a lot of inflammatory content here - vague accusations about "tire fire" filesystems and how Linux users are "faithful" zealots, blind to the truth, that will have to "admit" their folly and implicitly humble themselves before "the rest of us", whoever they are. And baiting a bad response, as you ask people to "downvote away" as if you want to inspire anger. All of this is needlessly confrontational. You don't need to insult your opposition to make your point.

You may view them as inflammatory. But it's coming from a POV of pain having to deal with broken architecture, piss poor portability and general pain of hacking around poorly engineered Linux architecture.

Frustration, pain are more appropriate terms from my pov than inflammatory. But ymmv.

It's like "Come on folks! You can do better than this! Please! Just stop! Think! Please. I'm begging you."

Regardless of whether your point is true or not (I agree that the state of Linux is suboptimal), this kind of attitude in which you spend more on venting your emotions than on actually describing issues, will not help you get your point across, and will not help making the world better. In the worst case you may even give BSD people a bad name by making the public perceive them as a bunch of emotional haters.

> ZFS is the way forward. For cross compatability, for reliability, for stability, for lots of use cases.

I'm not entirely convinced we should settle on ZFS just yet. It's fantastic and quite possibly the best option right now but it has a few limitations:

- The Linux implentation seems to have issues with releasing memory from the ARC back to the system

- The only way to expand a zpool is to add a new vdev. Pools are essentially a RAID0 of vdevs so if a single vdev fails, your entire server fails. You can mirror or RAID within a vdev but this means that the reliability of a vdev is the reliability of your entire pool. The problem here is that you can't just add 1 or 2 new disks since adding a 1 or 2 disk vdev would be data-suicide. For smaller servers, this is silly.

BTRFS looked like it was getting there but it proved buggy and unreliable. Personally, I'm waiting for bcachefs: https://www.patreon.com/bcachefs

There's nothing wrong with a 2-disk vdev. Mirrored instead of RAIDZ is probably the best choice for "smaller servers."

I strongly disagree. If I have an 8 disk server at home with mirroring, the pool could only take a single disk failure. The usable space would be 50% of the total disk. Further, I'd have to buy any extra disks in pairs.

If instead I ran RAID6, I'd have 80% of the disk available and I could add disks in single disk increments.

I think ZFS makes great sense for businesses that can throw money at disks but for smaller businesses or home servers it's kinda bad.

I think it's not a very good idea to design storage solutions around how inexpensive it is to add capacity to them. Your 9+-drive RAID6 is going to take forever to rebuild; 4+ mirrored vdevs (or mirrored RAID of course) will not be a problem at all.

I run a FreeNAS device at home (ZFS underneath).

It was kind of a pain to configure (albeit quite flexible) but it's been pretty nice overall, already survived 1 disk failure and a capacity upgrade (during which I had to resilver after every individual disk upgrade, which was time-consuming, but after the last disk got upgraded, the extra space finally showed up)

You also wouldn't get the same I/O performance a stripped mirror will give you. You can get extra resiliency using a three disk mirror.

Disk is cheap. There's no reason to design like it's not.

Disk is at least ~$0.023/GB. Whether it's "cheap" or not depends on how much of it you need.

Plus, disk may be cheap but servers to house it are not (the kind of servers you'd run in your house, I know you can get cheap SC846 off eBay).

Servers can be inexpensive. It really depends on your use case. You don't have to drop 50K on a box to make a killer ZFS storage array.

My scale starts at 40PB with a "P". Disks aren't cheap.

I use stripped mirrors for most of my ZFS deployments. Disk is cheap. If you want more resilience do a three way mirror.

I'd extend your point on zpool expansion to ZFS's general inflexibility when it comes to reshaping pools or rebalancing ZRAIDs.

I'm sure it's not an issue in enterprise environments, but I've personally been bit by ZFS's inability to shrink or rebalance pools (even offline) several times now through personal use.

Those sort of use cases need to be handled if ZFS is to become a more general use filesystem.

> Personally, I'm waiting for bcachefs

tires++

;-)

And this is what is so painful for the rest of us to watch or deal with. This inability for the Linux folks to admit these glaring architectural problems.

Alright, the suspense is killing me: what OS is the one that gets it right? I ask this to "the rest of us" as you put it.

SmartOS, FreeBSD, Tredly, etc

Tredly is "just" a FreeBSD distribution. And Linux can't have ZFS in-tree because CDDL.

It is not just a version of FreeBSD. The base is FreeBSD but the tooling for containers is not stock. It's value added bits that are shaping up to be very nice for containers on a more stable, secure, and storage friendly stack.

Yes, but the storage parts which the discussions are about are from FreeBSD

Ah, you must be from the alternate universe where most of us have heard of more than one of those!

What you've heard of, and what's workable and useful, do not necessarily intersect.

From the username, x86 BSD perhaps?

NixOS intrigues me as an attempt to create a completely deterministic build.

Many Linux users in particular are pretty well aware of the shortcomings of Linux's filesystem scenario, as well as the fact we never had features like DTrace. (I still think the over-proliferation of Linux tracing tools that fit 10,000 different case is a bit ridiculous.) As someone who worked on a distro, I've been acutely aware of many of these things, for example, as have our users (because they inevitably come to us for advice, suggestions, or bug reports).

I'm not sure what reality you live in where Linux users like, actively deny things like lacking an appropriate COW filesystem with low latency, or things like the fact it took us a decade to get only close to DTrace. Your own mind? 6 month Ubuntu users? Internet forums where you talk to other BSD users and nobody else?

> Let the down votes begin.

Don't worry, I'm sure people will oblige.

I like ZFS,

However XFS is perfectly good for storage. I've used it on 15pbs worth of fileservers with no particular nonsense.

I wouldn't touch BTRFS with a barge pole, not because of any inherent stability issues (although that is a big factor) its the utter nastyness of the tooling.

ZFS is a joy to setup, and its simple, logical and the man pages tell you useful things.

BTRFS, not so much

My favorite XFS workflow:

1. filesystem is corrupted, once again

2. try to repair it

3. oh right, the repair tool can not replay the journal

4. try to mount it

5. admit after 3 hours of nothing that the journal-replay code triggered on mount actually can really not deal with corruption

6. reboot the server to get the filesystem unstuck

7. rerun repair, this time throwing away the journal

8. look at the empty filesystem with everything in lost+found

9. restore from backup

The team I'm in only runs 1'400 servers, yet this happens regularly.

point taken. I have had xfs_check run out of memory because $reasons. This has reminded me of that. But then we also had lustre, so you know, by comparison XFS is a paragon of stability (We also had clustered XFS....)

But, mostly it happens because the fileservers dont have UPS's (The pipeline tools are almost exclusively COW, and backups are tested many times a week.)

I think we paid for support on XFS from either redhat or SGI, but I can't remember, I left that place a year or so ago.

I've never had the balls to run ZFS on large(100tb+) arrays. Last time I tried the way slab handling was translated from solaris cause many problems (but that was more than 4 years ago. ) Plus the support is a bit odd, you either go with oracle(fuck that), or one of the openZFS lot.

To get the best performance/stability you ideally need to let ZFS do everything, instead of letting the enclosure do the raid 6 (4 * 14 disk raid 6 with 4 spares). This of course is a break from the norm and to be treated with utmost suspicion.

Have you seen the GPFS raid replacement? thats quite sexy.

I run ZFS at home, because it works, and has bitrot checking.

Ah yes, ZFS the saviour.

Just as long as it does't get anywhere near capacity.

I assume you are referring to the same limitations almost every FS on earth to date suffers from when you consume about 90% capacity?

What a useless comment. If you are running out of capacity adding disks to a zpool is incredibly mind numbingly easy.

"zfs add poolname mirror disk1 disk2"

Every filesystem has performance hit the floor when you consume all available capacity. This is not unique to ZFS but to well, reality.

No I'm talking 100% capacity, shit happens and I do t want to fight the fs when it does.

100% is a lot better than ~80% (looking at you btrfs)

Sir, you have spoken well. What OS/distro should I be looking at?

Enterprise: Solaris and use zones. Internal/Personal/POC: FreeBSD 11 and use jails.

Both get you ZFS and something lighter-weight than VMs. Need Linux? Bite the bullet and use VMs.

I agree with this. SmartOS would be my choice over Solaris. I think FreeBSD is getting there. With things like Tredly. The building blocks are solid and available they just lack some polish.

So if you have the skill FreeBSD. Otherwise SmartOS makes a killer setup for containers unmatched by anyone else.

A warning, though: don't try to use FreeBSD with SCSI tape drives, in my truncated experience because of what I found experience a few months ago, the driver is capable of writing without error data it then can't read back (but can be read back by Linux).

Maybe not so important in the container context, but sooner or later, somewhere you need persistence, and tape offers certain persistence features you pretty much can't get elsewhere, especially at its media price points.

(Plus, if they got this wrong, when it's really not that hard to get right (I've done SCSI at this level before), I wonder what else they have.)

well, checkout hypercontainer (https://hypercontainer.io/) that runs on Linux, uses VMs and performs like containers.

No killing yourself necessary, really ;)

Solaris is discontinued but Illumos and SmartOS have taken up the torch.

That would come as a surprise. Solaris is alive and well thank god https://www.oracle.com/solaris/solaris11/index.html

See? Solaris is dead

CoreOS, as stated by the roadmap of the article. /s

Init systems.

I think Docker can somewhat be seen as a generalized term like Linux. Linux doesn't write EXT4 but it is often referred to as "Linux filesystem".

Docker has been actively trying to become to household brand name for containers so it's difficult for me to give them a pass with your line of reasoning.

> Re Persistent Data: You are probably going to have a tough time if you try to do persistent data. Containers really excel at stateless tasks(or temporary state like caching). Certainly it's doable but for someone early in their adoption the proper toe in the water is probably the stateless business logic not the state layer.

That is Dockers problem, they even want to be on Windows, I mean come on. But thats docker the company for you.

Can you blame them for wanting to break more into the Microsoft ecosystem? Besides, Docker's company mentality aligns pretty closely with a company like Microsoft, there's probably a lot for them to learn. A huge plus from a business perspective, but it's easy to see why it wouldn't sit well with open source advocates.

Yup I blame them for that. (I is freedom software advocate)

Not a company of good taste in my book.

I thought the whole Windows thing was totally stupid (I is also freedom software advocate). I've been a big fan of Docker for about three years, put loads of stuff into production at a handful of different companies, most recently using native Swarm features that came with 1.12. Personally, I thought the Windows thing was a gimmick. But what I see (as someone that has access to the Docker community through open source) is this enormous group of people who are chained to Windows technologies. And for years and years they've watched all the cool kids play with the cool toys, from the sideline. And what Docker-for-Windows has done -- with help from a new kind of Microsoft -- is expose all of these very eager people to containerization and microservices and open source and free software more generally. Having been exposed to loads of these people (and there are an enormous number of them), I think it means a lot of important things about bringing open source and free software into the fold for these people. Docker-for-Windows opens the door for a yuge number of developers to appease their enterprise leadership by sticking with the Microsoft brand, while also having the whole world of free software at their fingertips. I can see this world on the horizon in which open source and free software concepts have broken into the Microsoft ecosystem, and I believe it comes waving the Docker flag. I am still extremely suspicious of Microsoft as a company, and wouldn't pay for Windows if my life depended on it, but you can't say they aren't doing some interesting things over there at Microsoft lately.

There is time and resources spent on improving free and open source software, and then there is time and resources spent improving proprietary software for another business.

Pick one.

Now, that's not the reality even slightly.

Yeah, I flagged the article for being grossly inaccurate. I've had a lot of frustrations with Docker myself, and will happily switch to Rkt after it matures a bit more. Docker surely deserves a lot of criticism, but it should at least be valid and well-informed.