bbrazil

> Another tricky problem is if people use 8.8.8.8 – a CDN provider such as Akamai will see your request as coming from Europe, and will serve the request out of a European datacenter as opposed to the node in Nairobi.

https://tools.ietf.org/html/draft-vandergaast-edns-client-ip... was designed to handle this exact problem.

Obfuscation is still useful.

For example if a sysadmin is investigating a problem they're less likely to accidentally see a user's data in human-readable form, it also provides a level of defence against unsophisticated attackers.

People cannot access some internet sites, depending on who they have internet connectivity with.

https://blog.cloudflare.com/route-leak-incident-on-october-2... has more information from a previous time this happened. A key idea is that internet routing depends a lot on trust, and it's possible for a single misconfigured site to cause serious issues across the internet.

Nanog discussion: http://mailman.nanog.org/pipermail/nanog/2015-June/076187.ht...

The 3-times rule is still the roughly case in Ireland when you're offered something, and what the "ah go on go on go on" in the TV Show Father Ted was satirising.

What's interesting is when customs collide. For example my German relatives wonder why we keep asking them if they want something when they've already said no, whereas I'm waiting for them to offer another two times before I can say yes.

https://github.com/prometheus/prometheus/tree/master/console... has all the public examples for console templates.

> - Assuming metrics are stored in memory, they'll be lost if the app or server recycles.

Similarly if the server dies before sending the metrics, so that's a draw.

> The flip side is no automatic discovery and dynamic scaling where a server going down might not necessarily be a bad thing.

That's more about top-down vs. bottom-up target discovery than the scraping itself, see http://www.boxever.com/push-vs-pull-for-monitoring

As of Prometheus 0.14.0 which was released earlier in the week, Prometheus supports target discovery. http://prometheus.io/blog/2015/06/01/advanced-service-discov...

> Security. Not super thrilled with exposing something that says "Hey come get my data".

That's a fair concern, though I'd expect your metric endpoint not to be internet accessible (and some of the push options will have a HTTP endpoint for debugging).

This has been done for some time already via DKIM and DMARC, which anyone can configure.

https://support.google.com/a/answer/174124 https://support.google.com/a/answer/2466580

For Prometheus's console templates (basically offering a html templating engine to end users to create monitoring consoles with), I used Go's own templating language as nothing else seemed mature at the time.

Go is strongly typed, and so is it's templating language. This means that you need to take a much more rigorous approach to using them, as a single unhandled edge case value can break the entire page. Contrast this with something like Jinja2 where if you mess up it'll only break a small bit of the page.

I managed to produce a usable template system, but it's still a tad more verbose than I'd like. It'd have been nice if a more scripting-language style templating language had been an option.

That's the approach we take, documented at http://www.boxever.com/gracefully-shutting-down-haproxy-with...

The challenge is that whether HAProxy's own support for graceful restarts works depends on which OS and kernel version you're running and what network socket features are supported, so ensuring that's all kosher is a bit of a challenge. I took the approach that it's simpler to do something that takes a bit longer without having to worry about all that.

There's a few small things you can do over the unix socket, but in general you need to restart haproxy to change it's config.

While the phone doesn't meet it's stated spec, it's still pretty amazing that tech that'd have been top of the line a few years back now costs less than a night out.

One potential issue is that you don't know you'll get the same HTTP response next time.

> The irony is that Docker is based on software that underpins Google’s vast online empire.

Doesn't Rocket also use cgroups?

I'd recommend My Philosophy on Alerting by Rob Ewaschuk - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... as a good approach to take.

We've fleshed this out a bit more in the Prometheus best practices at http://prometheus.io/docs/practices/alerting/

Taking this approach at our company greatly reduced the alert count and improved responsiveness with no degradation in service.

"in anger" means used in a real-life situation, not just playing around.

> Monitoring first: we have a strong desire to increase this but we're a bit at a loss as to what a good light-weight solution is here

Prometheus is designed with this in mind. Instrumentation is easy to add and takes care of things like concurrency and state maintenance for you, so you can sprinkle it anywhere safely. http://www.slideshare.net/brianbrazil/python-ireland-monitor... goes into this a bit more.

What's base 32 used for?

HAProxy is fundamentally a L3/L4 proxy, with some specific HTTP features (all the nice throttling stuff can use only L3/L4 signals for example).

Looks like Prometheus support is in the works too: https://github.com/grafana/grafana-plugins/pull/13

That repo also has Atlas, Zabbix and KairosDB plugins.

If that was the craziest thing I was doing, I'd be doing well :)

> cc would be reachable via 50.jfoo.ubar.cc.borg.google.com.

I've implemented similar at my current job, as that sort of naming is very convenient. http://www.boxever.com/using-google-apps-openid-connect-with... has a sketch of how to do this with Apache as a reverse proxy with Google Auth, though we're using a PAC file now going to a HTTPS forward proxy to avoid limitations of SSL wildcard certs.

One of the changes hinted at here is that you can now specify that a task will always run using the 'always' tag - handy for things like ec2_facts which was a long requested feature (https://github.com/ansible/ansible/issues/3157)

That's a security point, not a reliability point. They will be more reliable, as you don't need to talk to e.g. MySQL that might be down or slow. The disadvantage is that they're difficult to revoke.

Cryptography is used to verify them.

We've recently been looking at adding JWT, and one of the things I looked out for was that we were protected against downgrade attacks. This was done both as a check of the current library code, and an explicit check on our side that the alg is what we expect to protect against future library changes.

Such a token doesn't require you to have a database to lookup and maintain the user session/permissions in, which can lead to more reliable and lower latency designs.

Be wary if you do anything with signed data before you verify the signature.

http://www.thoughtcrime.org/blog/the-cryptographic-doom-prin...

This could be handy for package distribution, is there mention of the propagation delay?

With a 3s average response time and presumably replication across multiple disks, you've plenty of scope to schedule requests in a way that they won't affect performance.

In 3s on one disk you can do roughly 5k seeks and read up to 300MB. They only need to do 1 seek and read 4MB.

My guess would be disk drives that are otherwise used for I/O intensive purposes, and where there's tons of unused/wasted space.

4MB/s is only around 4% of the bandwidth of a modern 1TB hard disk.