bbrazil

I'm talking about internal libraries of a company, where there is no proven library with a stable interface as it's continuously evolving - similar to a young library in the OSS world that has no reasonable substitute. It might be something like a new parameter to a function, or an additional method call you need to make in a given version.

It's very common to have breaking changes that you need to carefully manage in a company's internal code, and where managing that as a full-on deprecation isn't worth the effort. Moving faster is often the better choice compared to never refactoring and improving the codebase.

Debs and Rpms are fine for configuring the machine, as soon as you have it where machines and application services aren't a 1:1 relationship that will lead you in the direction of dependency hell. As each application will want slightly different versions you'd have to be very careful which version you installed.

You can cheat a little. For example I am using some system debs for python packages, but if there were more than a handful of python utility packages that can get unmanageable fast. That leads naturally to the fat jar/virtualenv/static binary approach with manually managed dependencies which gives you isolation between applications, and all the associated costs.

http://deb.robustperception.io/ is my nightly build. It's one of the few things I want to run on the very bleeding edge of.

A bit over a year ago I researched what "Devops" means, and the answer seemed to be Developers being able to push code to production without having to involve Operations. This sounds like a good goal to have (presuming you've got good unittests etc.) as it removes an unnecessary friction for developers.

What it doesn't include is all the higher-value work the people on the Operational side tend to provide, like thinking about rollbacks, machine failure, network failure, provisioning, capacity planning, change/configuration management, security, monitoring etc - which is not to say that all developers don't think about these things, their focus tends to be on developing product rather than these non-functional requirements.

I'm presently in a world of Ubuntu LTS+a few manual backports for similar reasons as the authors. My home systems are ansible managed, giving me things like wireshark installed everywhere if I need it and new machines automatically hooked into Prometheus monitoring (which is much easier with my debs). I've seen what happens if you try to manually manage machines, and know that a small bit of upfront work will bring dividends later.

Writing the core code is just one critical step in running software long term, let's not forget the rest of the critical steps to keep it running in a sane way in the future.

Rented gear in someone else's racks, and it's only that cheap due to Supermicro. I did calculate what the providers premium was, and it didn't seem too unreasonable given the services such as remote hands and taking care of replacements that were being provided.

> it's not a panacea because it tends to get very expensive, very quickly.

I've investigated colocs near some AWS locations and found AWS to be around 20% cheaper than the cheapest viable option. With AWS's occasional price drops on top of that and general flexibility it's not a difficult choice.

One thing I found was that most providers were trying to sell vertical reliability (e.g. hardware RAID1), with a matching price. Our redundancy is horizontal at the application level (e.g. Zookeeper, Cassandra) so this has no value for us.

We'd like it to run on Windows, however none of the core developers use Windows.

https://github.com/prometheus/prometheus/issues/505

We haven't head back recently, so it's possible it's all working now.

My conjecture on Nearline is that it's unused disk space on disks that are mostly I/O bound.

How many servers and metrics are you expecting?

I know for prometheus.io we can handle at least 1M metrics per server.

This presentation by the author explains the thinking behind it: https://speakerdeck.com/kennethreitz/python-for-humans

That can mean that the server hasn't got data it needs loaded and is returning empty requests, so it's good to spot those.

I find that https://xkcd.com/radiation/ always helps put things in perspective.

> There is a strong argument to be made that libraries should not vendor their own dependencies, but as the author of several library / command combo packages that rely extensively on 3rd party libraries I’m not convinced of that blanket assertion.

We ran into this with prometheus.io and godep. Vendoring in our libraries meant that types didn't match between the vendored version that the main server had and the vendored version coming form the library.

If there's a way to solve this, that'd be great but it means you're back to choosing which version is correct.

Pricing: https://aws.amazon.com/codecommit/pricing/

$1 per user/month, with plenty of free storage (10GB) each. 2k git requests/month sounds a bit low though if you're doing anything automated. There is a good free tier.

Previously: https://news.ycombinator.com/item?id=478632

Shaft is a well known anime studio, famous for several series including https://en.wikipedia.org/wiki/Puella_Magi_Madoka_Magica and https://en.wikipedia.org/wiki/Monogatari_(series) shown here.

We've been doing this in our makefiles and exposing it to the user https://github.com/prometheus/prometheus/blob/0e383b0b9f4ff0...

Cloudflare have been putting out quite a few high-quality posts recently.

I did my final year project of my degree on B-Trees (Cache Conscious Optimisation of STL Data Structures).

The short version is that you want to tune to page size, inserts/reads tend to be faster than RB-Trees but deletes are slower. This was true across a variety of chips and cpu architectures.

One way to think of it is why would you use Cassandra when you already have a filesystem?

Those databases are quite raw, to be useful for a use case like monitoring you need to build on top of them to allow for ability to get data from agents, do some processing and enable efficient querying of the data.

Prometheus uses a file per timeseries, with two levels of delta encoding to keep the data small. This is our second major storage iteration, and seems to be doing pretty well with a single server able to handle over 2M timeseries.

See http://prometheus.io/docs/introduction/faq/#why-does-prometh...

This has been in the works for years, so telecoms have had plenty of time to get ready. They've also been offering much better roaming rates the past few years within Europe.

I'm looking forward to InfluxDB cluster so we can use it as long term storage for Prometheus.io. I'd like to focus on our strengths at high-throughput low latency metric processing, rather than having to solve the very hard problem of creating and maintaining a distributed storage system.

Generic SNMP support for Prometheus is ongoing, see https://github.com/brian-brazil/snmp_exporter

If anyone knows of a MIB-parsing library I can use in Python or Go I'd appreciate a pointer (Mibble is GPLv2, which isn't compatible with the Apache 2.0 license Prometheus uses).

https://lock.cmpxchg8b.com/sophail.pdf and https://lock.cmpxchg8b.com/sophailv2.pdf

Context managers are quite powerful, you can use them for more than resource closing. They're handy to make instrumentation easier too, for example by allowing you put the code you want to count exceptions in in a with block as shown in http://www.slideshare.net/brianbrazil/python-ireland-monitor...

This is even more relevant today now that we have "VDUs" everywhere.

That'd still have you going through the kernel on every event to handle the UDP packet, keeping in userspace is more efficient (on the order of 10ns of CPU).

I hadn't realised the PHP link, things make more sense now.

You're getting into IPC then, which is a fun topic alright e.g. https://github.com/prometheus/client_ruby/issues/9 and https://github.com/prometheus/client_python/issues/30

Have you considered pushing the aggregation into the applications, rather than doing it across the network?

Having whatever library is sending data to statsd, instead keep counters/gauges in memory and then expose that on a regular basis would greatly reduce the data volumes involved as it's O(timeseries*frequency) rather than O(events).

This is the approach we take with Prometheus, and based on the statsd setups of some people who've come talking to us there's scope for a reduction in network load of at least an order of magnitude without having to do any downsampling.