roskilli

I was at a wedding when the question was posted, hence taking forever to reply (apologies!).

I am of course biased, I want to say that up front. For Uber we wouldn’t really want to use Thanos for a few reasons but one primary one is that Thanos is upfront that they don’t really optimize for latency with metrics that don’t currently reside on disk, which can take significantly more time coming from S3 rather being mmapd on a local disk. We have historical metrics needed for anomaly detection (5 weeks of data) queried very frequently and the model of downloading the S3 data for each request or caching it locally (which would fill up the disk of query nodes quite quickly, since we have petabytes of metrics data) doesn’t really scale for us. Also we’re conscious of having to pay all the AWS bandwidth costs considering the dataset is in the petabytes and we run things both on premise and cloud.

Anyway I could definitely talk at length about this, perhaps we should write up something and put it on our wiki.

Yeah, we apply the roll up rules at ingestion time (the coordinator has an embedded m3aggregator in process).

Theo came and have a talk at our office when he was in NY one time, it was really great to dig into IRONdb and hear all about it.

Many similarities for sure, unfortunately we are somewhat strict on the OS we run (so it can be standardized across the fleet) and thus we can’t really run ZFS so we have to checksum all our data ourselves, etc.

Wow you have some epic karma AA - haha, cheers!

Yup indeed, we have to make this more friendly because you have to curl an HTTP/JSON endpoint right now to set the configuration (we’ll release an embedded UI for policy rules editing soon, called m3ctl).

Basically though when you set retention mapping rules, metrics are downsampled to whatever resolution you choose (10s, 1min, 10min, etc) and then stored for whatever time the retention the namespace you have setup for that policy.

And with respect to my comment about disk space being the bottleneck - unless you have a huge dataset you actually might not be bottlenecked by disk, it’s just we’ve encountered that at times with our really high cardinality data sets.

So if you look at a lot of LSM databases they basically optimize for write throughput by design but also ultimately spend a lot of time on compaction of sstable like file volumes. So we avoid a lot of compaction of volumes together by making volumes time window based rather than other limits like heap size, etc. This means that your resources ultimately do less work and should cost less. The flip side is that you do need to make sure you have enough heap for your mutable data (which is fine in the steady write case, but for back fills means you should be more careful and use libraries to coordinate the backfills)

Hey, it is yes - by default all metrics are unaggregated, it’s only until you set aggregation and mapping rules that downsampling occurs.

It’s mainly just disk space that is the primary bottleneck.

M3QL still is the primary query language internally, however the query service is still being rebuilt in open source M3 - so it’s not available just yet.

As to why we’ve kept it, there’s been such a large amount of functions we’ve added over time that don’t really have an alternative in say PromQL, etc - so we aim to offer both PromQL and M3QL in open source land and letting end users use either one. I don’t think either are better than the other one, they’re just different flavors - function expansion vs pipe based, etc.

Here’s the list of functions, you can kind of get a sense of what I’m talking about just by looking at the list: http://m3db.github.io/m3/query_engine/architecture/functions...

Author here and a few other M3ers around too, please do ask any questions. Ultimately we'd just like to make the project useful to others, so any and all feedback is appreciated, thank you!

Can't take credit for that one, this one was a 99designs competition: https://99designs.com.au/logo-design/contests/establish-bran...

I really liked the rabbit, entry #12 TBH.

It’s definitely interesting to see the different systems being built for monitoring across the different tech co’s.

M3 aggregator, Uber’s metrics aggregation tier is similar, except it has inbuilt replication and leader election on top of etcd to avoid any SPOF during deployments, failed instances, etc. Also it uses Cormode-Muthukrishnan for estimating percentiles by default, it has support for T-Digest too. Although these days submitting histogram bucket aggregates all the way from the client to aggregator then to storage is more popular as you can estimate percentiles across more dimensions and time windows at query time quite cheaply. You need to choose your buckets carefully though.

It too is open source, but needs some help to make it plug into other stacks more easily: https://github.com/m3db/m3aggregator

Also the other "fun" approach:

  for _, elem := range elems { elem := elem; go func() { /* use elem */ }() }

  (&article).legs = 99

This would not work actually because it would only be modifying the copy that exists in the inner loop of the for loop. You'd be taking a pointer to the for loop stack variable and modifying it.

If they were already references as you first mention that way simply article.legs = 99 would work yes.

If you look at the numbers I have to say that's a little bit silly to say. They have $6.6 bill cash on hand, they are burning ~$2.5 bill per year, although quarter to quarter net loss fell 9% last quarter. They very likely have 2 years, likely more, to start making profits each quarter.

If they really need to jettison more expensive parts of the business they can sell off their autonomous unit and their VTOL investments, along with a ton of other fat they can trim.

Your viewpoint does not seem connected with reality IMHO.

What I wouldn't give to have this in Go. Literally have been playing with Mmap system calls recreating something similar with mixed results...

It's here: https://pdfs.semanticscholar.org/3a6b/0506b8b922d7e6cc933a39...

MIPS is a great RISC instruction set to learn and has a ton of good emulators out there too, highly recommended.

Likely the strong generics while still being terse and inbuilt functional methods on collection types. Also being able to add extensions to any class is super helpful.

You could write one pretty simply on top of something like Ringpop:

https://github.com/uber/ringpop

There's also a Go implementation of Ringpop coming right around the corner too.

> Multiple versions of the same library will cause problems ... This is a limitation of ... but probably also a sane one

> Another issue (hat tip to onetruekarl) comes up when 2 vendored dependencies also vendor their own copies of another depencency with the same name ...

So I still don't see why people are content with having a vendoring solution that doesn't support multiple versions of the same package and strict ASCII name conflicts. I understand this is an experiment but it begs the question why it doesn't want to at least eventually bite off doing powerful things (it seems to handwave the hard problems with a future spec that kind might solve it - seemingly something probably orthogonal to this experiment).

I have really enjoyed what NPM did with a simple package.json file and enabling recursive dependencies. It seems like if go were to add a simple file that maps package name to package URL with a decent support for different URL protocols you could build the same recursive dependency graph and depend on multiple versions of the same package and also different packages with the same name.

I'm not usually a guy that asks for a code review to be done directly by pinging the reviewers, but I've found on certain projects there are colleagues who will unfortunately _never_ review any diffs I have with them listed even as the sole reviewers until I ask them.

This is unfortunate but means that I fall behind by however long I don't ask them to review my code for :( Others are great and I don't have to ping them directly and they can tell how urgent a review is or isn't usually.

Not commenting specifically on any situations discussed here but just realizing I do sit on the other side of the fence from time to time.

If you only care about iOS and a layer on the backend that can sit between your actual app server and the client you could consider Jetstream:

https://github.com/uber/jetstream

https://github.com/uber/jetstream-ios

It's a considerably different model however. More about realtime updates.

Surely the riders install base is quite valuable if both were to have as good tech and self driving cars

Coffee:

[45,69,82,82,89,0,35,72,82,73,83,84,77,65,83,1].map((x) -> String.fromCharCode x+32).join('')

MSR per-chance?

I love it "Available for iPhone. More platforms coming."

I really think he hit the nail on the head with his memo when leaving MSFT...

If you are on call/scheduled for the deployment you may be required from time to time to come online and deploy in off-peak times. This type of maintenance should be extremely rare - in that it requires making your service entirely unavailable, most deployments should be able to be done in a rolling manner and not affect ongoing service availability.

Also usually you are on call every so-often so if your team/company happens to do this often it's not always you performing the deployment.

In regards to "being in the office": most people will do this from home after sleeping most of the night and waking up to an alarm and then coming in a little later the next day. There are a few hardcore ones out there that prefer to pull an all-nighter and do so from the office, although my mileage has been those are quite few and far between.

Listening to a Podcast about procrastination instead of coding on one of the rare days I get to the office before 8am... interesting suggestion you make mon frere

Swift should theoretically deliver faster apps with less message passing overall and other big improvements.

At the end of the day the latest phones need to keep up with competitors and added features and resimplified experiences that are more costly to implement cannot be avoided. The only way to really avoid it is disallowing older phones to upgrade - which is far less desirable IMO.

Pretty sure Snow Leopard was a release aimed at doing just that, rewriting a lot of the core system libraries for x86_64 and also clearing house on a lot of old cruft. No new real features were delivered with the release.

I would also add that callback signatures should always follow:

  function (err, result) {
    //...
  }