GlusterFS 3.5 released
gluster.org
gluster.org
If you feel like you're a good C programmer have a look at the codebase and prepare to have your mind expanded.
It's a very nice piece of software and a very strong proof that not all successful development is done in SV (Almost all of GlusterFS before the buy-out was written in India, not sure what the current situation is).
The main failure I was getting involved VMs. I know I could do something with openstack, but this is mainly a platform to experiment, so I wanted to be able to support any kind of VM.
Having said that, starting with v3.4.0 gfapi can be used if your Qemu supports it (it's the case with CentOS 6.4+ or Ubuntu 14.04), therefore bypassing the FUSE layer. This should give you an improved performance.
The iXSystems guys are doing really interesting stuff, too.
Our primary use case for Gluster is in serving persistent filesystems for Drupal. Our customers store potentially millions of files on their GlusterFS clusters.
We've built a number of tools/processes to help protect Gluster against failures in EC2 (for instance fencing network traffic at the iptables layer to help protect GlusterFS clients from hanging talking to down nodes), as well as to help our team perform common tasks (resizing clusters, moving customers from cluster to cluster, etc.). We haven't necessarily hit blocker issues recovering from underlying hardware fails, but our team is definitely very experienced with many possible failure modes.
Overall GlusterFS has been very reliable over the years and our research has shown it is the best option out there for when our customers can't use something such as S3 directly.
If you want more details or would love to hack on a 8000+ node EC2 cluster running things such as GlusterFS feel free to ping me.
Disclaimer, I've been at companies with thousands of servers for my past 3 jobs. I've never once used any "cloud" service other than Linode for my personal VPS and ganeti for the Oregon open source lab VM donated to the gnome foundation (I'm a sysadmin alum for gnome.org)
All of our enterprise customers receive a highly available setup running on multiple nodes--thus, we have the need for a persistent filesystem attached to multiple EC2 instances. We utilize GlusterFS to ensure all of our clients have the filesystem capabilities their apps may need.
Performance gets better with every release, but the speed of `ls` (Gluster uses `stat()` to trigger replication/checking) is still woeful at times. 3.5 will hopefully make that a lot better. In general, Gluster is not the right answer if you have lots of small files.
Having the 'bricks' use the regular filesystems was a very smart decision, it means you don't end up in binary hell when things go south.
Split brain was one of the weaker points back in the days, I am not current with the software so no idea if and how much this has improved, someone with more recent experience would be a better guide there.
(Disclaimer: I'm a GlusterFS developer)
For some reasons it usually failed on high load, resulting in bad split-brains, extremely slow IO and loss of data. Very bad. We contacted Red Hat regarding that and got comments that gluster was not VM ready yet. We ditched gluster and went for plane proven old iSCSI.
http://thr3ads.net/gluster-users/2011/06/480298-Enomaly-user...
This bug (and others) would happen when moving files and/or folders over top of existing ones, ie:
mkdir -p foo/bar/baz && \
mkdir -p foo/bar/tmp && \
touch foo/bar/baz/file foo/bar/tmp/file && \
mv foo/bar/tmp/file foo/bar/baz/ && \
rm -rf foo/bar/tmp
Also, filesystems are hard, I get it, so no hard feelings :)
Gluster is nice, but you have to accept the fact that it is slow, and upgrades are painful. (We had issues where clients and servers had to be updated at the same time, to avoid interoperability issues.)
The fact that I keep my "bricks" on top of RAID(6) also helped with that.
(1) How do you secure GlusterFS traffic? Does GlusterFS use some kind of encryption on the wire or do you have to manually setup a VPN?
(2) How well does GlusterFS work if you host bricks on web servers? If data are sharded per user, can you make a web server always have fast access to that user data (can you guarantee a copy of the data is always hosted locally on that web server?) -- or is this something that only happens indirectly through the OS cache?
(3) Do you need something more from the underlying filesystem to guarantee data integrity, like running GlusterFS on ZFS? Or is GlusterFS enough?
Although I have not used the feature, there is support for SSL, look it up. I have also heard about people trying it over OpenVPN though not sure how successful they were.
> (2) How well does GlusterFS work if you host bricks on web servers? If data are sharded per user, can you make a web server always have fast access to that user data (can you guarantee a copy of the data is always hosted locally on that web server?) -- or is this something that only happens indirectly through the OS cache?
Don't have first hand experience with this, but I can recommend you Joe Julian's blog, i.e. http://joejulian.name/blog/optimizing-web-performance-with-g...
> (3) Do you need something more from the underlying filesystem to guarantee data integrity, like running GlusterFS on ZFS? Or is GlusterFS enough?
GlusterFS relies on the underlying filesystem to do its job, the recommended one is XFS with 512 bytes inode size.
The native client still uses it though and this is unlikely to change too soon, afaik.
The performance hit is disputable, there aren't really better alternatives. Most distributed filesystems out there are using FUSE too (XtreemFS, MooseFS) and I am not touching Lustre.
There are issues with everything, your dismissing it like that is not really fair or constructive.
It should be possible to avoid the userspace to kernelspace transition in the client by using LD_PRELOAD with a shim library as well.
The performance hit is disputable, there aren't really better alternatives. Most distributed filesystems out there are using FUSE too (XtreemFS, MooseFS) and I am not touching Lustre.
The Ceph distributed filesystem has an in-kernel client which is upstream. XtreemFS and MooseFS are designed for a different use-case (WAN filesystem) and aren't really directly comparable.
Lustre runs the majority of the world's supercomputers. It seems a little unfair to dismiss them without at least giving a reason why. They have made some progress towards getting their kernel client upstream recently.
I was a Ceph developer for a while. I read about Red Hat's recent acquisition of Inktank with interest. (Inktank was founded by Sage Weil, the creator of Ceph, to commercialize the filesystem.) Since Red Hat previously acquired the main company behind Gluster, this makes things a little-- how shall we say?-- interesting. It's unclear to me whether Red Hat will want to support two distributed filesystems going forward, or whether they will try to streamline things.
> Lustre runs the majority of the world's supercomputers. It seems a
> little unfair to dismiss them without at least giving a reason
> why. They have made some progress towards getting their kernel
> client upstream recently.
Also, there is no clear upgrade path between different versions: the recommended procedure is to dump everything to another filesystem and then reload after the upgrade. There is no replication handled at the FS level. When running Lustre on top of ext4, you have to use the Lustre's own versions of the ext4 utilities, which replace the ext4 utilities in the system...
In short: Lustre sacrified a lot of ops-friendliness for the sake of high-performance. This is quite the opposite choice that GlusterFS has made.
(Edit: formatting)
We (I'm a GlusterFS developer) actually did this for a while once, and people have recently started talking about doing it again. With GFAPI it shouldn't even be that complicated, but there are still problems e.g. around fork(2).
"The Ceph distributed filesystem has an in-kernel client which is upstream."
...and it's probably not a coincidence that the file system component is the one piece of Ceph that still hasn't reached production readiness. Development velocity counts. The fact that it uses FUSE has rarely been the cause of performance problems in GlusterFS. More often than not, the real culprit is (relative) lack of caching, which is fixable in user space.
"XtreemFS and MooseFS are designed for a different use-case (WAN filesystem)"
Pretty true for XtreemFS - for which I have the utmost respect and which I promote at every opportunity - but MooseFS targets exactly the same use case as GlusterFS. OK, a subset, because they don't have all the features and integrations we do. ;) I'd also add PVFS/OrangeFS, which is contemporaneous with Lustre. It doesn't use FUSE, but has its own user-space "interceptor" which is equivalent.
"Lustre runs the majority of the world's supercomputers."
It runs a lot of the world's biggest supercomputers, because those people can afford to keep full-time Lustre developers on staff to baby-sit it. Not ops staff - developers to apply the latest patches, add their own, etc. I spent over two years at SiCortex trying to make Lustre usable for our customers. At that time, I believe LLNL had four Lustre developes. ORNL had slightly less. Cray, DDN, etc. each had their own as well. When it works, Lustre can be great. On the other hand, few users can afford to devote that level of staff to running a distributed file system. Those that can't will find themselves in the weeds with MDS meltdowns and "@@@ processing error" messages all the time.
Because of this, I'd say it's Lustre that's not really targeting the same market or use case as GlusterFS. We never encounter them head to head, in either the corporate or community context. The "performance at any cost" market is a hard place to make a living, and it barely overlaps at all with the "performance plus usability" market.
"It's unclear to me whether Red Hat will want to support two distributed filesystems going forward"
Why not? They've supported multiple local file systems for a long time, and there's an even bigger overlap there. When you look at both Ceph and GlusterFS in terms of distributed data systems rather than just file systems specifically, maybe things will look a bit different. Now we're talking about block and object as well as files. Maybe we're talking about integrating distributed storage with distributed computation in ways not covered by any of those interfaces. We're certainly talking about users having their own preferences in each of these areas. If there are enough APIs, and enough users who prefer one over the other for a certain kind of data or vertical market, then it makes quite a bit of sense to continue maintaining two separate systems. On the other hand, of course we want to reduce the number of components we'll have to maintain, and there will be plenty of technology sharing back and forth. Only time will tell which way the balance shifts.
MooseFS targets exactly the same use case as GlusterFS.
Yeah, actually you are quite right. I was getting MooseFS confused with a different filesystem. MooseFS looks like it has a GFS heritage. Kind of like what I am working on now... HDFS.
...and it's probably not a coincidence that the file system component is the one piece of Ceph that still hasn't reached production readiness.
Ceph's filesystem component is reasonably stable. It's the multi-MDS use case which still had problems (at least a few years ago when I was working on the project.) The challenge was coordinating multiple metadata servers to do dynamic subtree partitioning and other distributed algorithms. Ceph has a FUSE client which you can use if you don't want to install kernel modules, and a library API.
It seems that Lustre has an in-kernel server. This might have led to some of the maintenance difficulties people seem to have with it. (I never worked on Lustre myself.) I don't think this discredits the idea of in-kernel clients, which are different beasts entirely. Especially when the client is in the mainline kernel, rather than an out-of-tree patch like with Lustre in the past.
It's a tough market out there for clustered filesystems. HPC is shrinking due to government sequesters and budget freezes. The growth area seems to be Hadoop, but most users prefer to just run HDFS, since it has a proven track record and is integrated well into the rest of the project. Moving into the object store world is one option, but that is also a very crowded space. It will be interesting to see how things unfold.
> By "problems with fork," you mean the possibility of
> forking without that environment variable set via exevpe
> or similar?
With regard to in-kernel clients, I'm not trying to discredit them, but they're not the only viable alternative. User-space clients have their place too, as I'm sure you know if you work on HDFS. Every criticism of FUSE clients goes double for JVM clients, but both can actually work and perform well. It seemed like some people were dissing GlusterFS because of something they'd heard from someone else, who'd heard something from someone else, based on versions of FUSE before 2.8 (when reverse invalidations and a lot of performance improvements went in). This being HN, it seemed like they were just repeating FUD uncritically, so I sought to correct it. The fact that GlusterFS uses FUSE is simply not a big issue affecting its performance.