Hacker News Reader

>If everything is more or less stock Linux, what value does Cumulus actually add for it to be worth the licensing costs?

Cumulus's value is that it's the translation between Linux kernel bridging/routing, and the hardware accelerated routing performed by the Broadcom Trident family. Somebody has to sign Broadcom's agreements and program for it, and Cumulus is doing that cheaper than anybody else as far as I can tell.

Another value is that you can now choose between all sorts of switch vendors, and not be as susceptible to lock-in from that angle.

Retail pricing on a white box 48x 10gB + 4x 40Gb switch is less than $6k, and an annual Cumulus license is $1k. If Juniper can match or beat that, I would love to see it, but I doubt I ever will, because you have to spend days cultivating a relationship with Juniper to even start getting decent pricing. And really, JunOS, despite being nice, puts Juniper at a disadvantage over Linux as the net OS. I'd even pay a small premium to use Linux as it is so much more programmable and makes it so much easier to get new racks up and going.

You can buy an EX4550 new in the $6K range, albeit just with 32x 10Gb ports; you'd need to buy the modules for 40Gb or additional 10Gb ports. This is just on the grey market (check eBay), no negotiation or relationship required. If you need support, J-Care service is $525/yr MSRP for the EX4550, with a reinstatement fee cost for the same for products out of warranty. You'd be able to pay off the modules with the difference in licensing/support costs over time.

Maybe I'm just bad at negotiating, but I haven't been able to get vendors to quote me less than double the cost of the Cumulus hardware quotes I've seen (primarily Quanta). Also, looking at support costs, the quotes I have for Cumulus are about 20% cheaper than the cheapest quotes I've seen from Juniper/Dell, just for software support.

That said, NBD or sooner hardware replacement plans are a bit weird with Cumulus. I think they expect you to keep onsite spares, which still ends up being cheaper.

I think if Cumulus becomes more popular, we might see some community code that implements validation and confirmed commits and more features. If we decide to go with Cumulus, and we are thinking about it, we might write something like that.

In theory, the value of Cumulus is (1) support and (2) their proprietary switchd. (Edit: I also second the other comments that Cumulus licensing appears to be really cheap. But if you're looking for "CentSwitch" you're out of luck.)

Also, how exactly is Cumulus separating the control plane and data plane?

They're not. Cumulus is kind of anti-SDN.

"Proprietary" means "if you ever have issues with that that our support won't be able to solve for you in meaningful time span, you're basically out of luck".

We're running a gateway router on Vyatta/VyOS and with every single hardware and software change we had some issues. Most were relatively minor (like default settings being not appropriate for our workload), already documented/solved by someone on the 'net. Some had required reading source code and debugging/profiling to see what goes on. Must also admit, some were beyond my understanding and were mysteriously solved with some shamanism and voodoo magic (like randomly tinkering with firmware and driver version combos).

I wouldn't characterize us as "anti-SDN", instead I would say that Cumulus Linux is not an SDN solution by itself.

We work closely with all the major SDN vendors for use-cases that are virtualization-centric or cloud-centric. That said, there are use-cases like Hadoop clusters that have very little need for an SDN layer, just very fast L3 IP connectivity.

It is stock Linux (Debian based), but behind the scenes, we're programming a hardware forwarding ASIC to do the actual packet forwarding. This is why we run on white-box switches and not servers, see our HCL for details: http://cumulusnetworks.com/support/linux-hardware-compatibil...

This is how a 1U switch drawing less than 200W can forward 2+ terabits/sec. A server full of 10gig/40gig NICs would be 50x slower and draw way more power.

I prefer JunOS to IOS/NX-OS as well, but both of their CLIs are optimized for managing switches with hand-written (or PERL script written) config files. Most Cumulus customers use automation tools. We've had folks use Chef, Puppet, CFEngine, Ansible, Saltstack, as well as a few mega-scale customers who have home-grown automation tools. The idea is to use whatever you're already using to automate servers to automate the switches; often there is substantial sharing between the server automation scripts and the network automation.

The volume is quite high already, Cumulus Linux alone is managing well over 1 million 10G ports today, and we're not the entire white box OS option. Our software pricing is available on our website: http://cumulusnetworks.com/product/pricing/ and some of our hardware partners publish price (keep in mind, this is web orders of quantity 1): http://whiteboxswitch.com/collections/all-switches

But the biggest cost savings come from automating the management, and not having to buy vendor-locked optics. Compare the prices here (again, quantity 1!): $30 for a SFP+ 1M DAC cable: http://www.fiberyes.com/sfp-cable-cab-10gsfp-p1m-30 $59 for a SFP+ SR optic module: http://www.fiberyes.com/10gsfp-transceiver-axs85-192-m3 to what you're you're paying cisco or Juniper.

We don't really separate the control and data planes. We don't really consider ourselves an SDN company, we enable many approaches to SDN by building very high performance fabrics that you can run your SDN layer on top of. We work closely with VMware NSX (formerly Nicira), Midokura, Nuage, and PLUMgrid. Some of our higher scale customers have their own SDN solution.

<Disclosure: I am co-founder and CTO of Cumulus>

Hi Nolan, thank you for responding to my question. In my naivete I presumed support for the ASICs would end up in the Linux kernel or at least made available due to GPL requirements, but I now see another post mentioned you have a proprietary switching daemon, so I guess it extends a lot further than just driver support.

Thanks for the link; I've seen similar pricing on the Quantas before, but only from obscure online retailers, who are often well below prices from more trustworthy sources. Would you recommend whiteboxswitch.com as a reliable source?

I don't think the optics is a fair selling point however, as you can easily purchase 3rd party optics programmed to be Cisco, Juniper, etc. compatible. We currently buy Juniper coded SFP+ SR optics for <$25 from a distributor in China.

We've been working with the kernel folks on trying to get support for forwarding ASICs into the kernel, but it is probably going to take a while. For now, ASIC programming is done from userspace by a daemon we call "switchd", which observes changes in the kernel's state, and programs the hardware accordingly.

whiteboxswitch.com is a fine option, though bm-switch.com seems to have lower web pricing at the moment.

Most of our customers are concerned about support, and thus aren't interested in unsupported optics. I guess as long as Juniper doesn't find out... =)