oavdeev

I have once built an interactive calculator[1] for this exact problem, maybe someone else will find it useful too

[1] https://observablehq.com/@oavdeev/parallel-task-latency-calc...

Yeah QuickSight is their shot at that market. But (anecdotally) I don't think it getting much traction. The fact that they didn't have any flashy announcements about it at re:Invent this year somewhat corroborates that. There is something in Amazon DNA that prevents them from building great UX-centric products.

Besides, without something like Google Sheets or Excel it is not that useful, inevitably you have a ton of small data across the org that is stored in spreadsheets.

One obvious play for Amazon is to just pay a ton of cash and acquire something like Airtable, but they don't seem to be for sale.

That set of people is not that small; it is basically what people in business analyst roles do all day. They know Excel very well and maybe even a bit of python/js, have a good understanding of the business problem at hand, but not technical enough to do full on custom reports from scratch.

Data Studio fits squarely in this market of tools for them, and that space is growing and pretty crowded. There a multiple big businesses built on producing tools just like that, for example Tableau (public company valued at ~10B as of today).

Granted, Data Studio is more basic but it is free and nicely integrates with the rest of Google data product suite (Analytics, Sheets, BigQuery). I don't think it will go away, and I think it is actually the strongest part of google cloud offering (and the biggest gap in AWS cloud product lineup).

It does; unfortunately they don't have a nice way to just disallow overwrites (that's why I was curious how you implemented that)

How did you do "no overwrite" bit? By enabling versioning?

I suspect it works well for business travellers though; they are not that price sensitive (as long as the fare is within the company policy).

Here's lstopo output for the largest one, for good measure: https://instaguide.io/info.html?type=a1.4xlarge#tab=lstopo though there's nothing terribly surprising.

Here's lstopo output on a 4-core AWS instance to illustrate your point: https://instaguide.io/info.html?type=c5.xlarge#tab=lstopo

If you're using AWS anyway, I think using their managed Postgres / RDS could make this a lot easier operationally. No need for custom syncing/backup scripts, easier to scale the storage up.

To the larger point, while I totally agree with the premise -- most apps will never need to scale beyond one large instance, I'm not exactly sure what's the actual tradeoff is. If you're writing a simple CRUD app, it is not really controversial that it shouldn't need any complex distributed stuff anyway, it is just a simple python/Go app and a data store.

Most "fancy" things outside that single process paradigm, such as avoiding using local disk and using S3/RDS/document stores, using containers for deployment, etc. usually have more to do with making the app operationally simpler and easy to recover from instance failures than scaling per se.

To be fair, AWS gives you bigger discounts for RI than GCP's sustained use. That makes sense; someone gotta pay for the risk.

This said, GCPs committed use discounts still seem to be a better deal than convertible RIs.

Looking at the run-of-the-mill 64GB RAM standard instance:

  AWS 1yr all upfront:            $0.458
  GCP commited use:               $0.47878
  AWS 1yr no upfront:             $0.491
  GCP sustained use:              $0.532
  AWS 1yr no upfront convertible: $0.564

edit: confused commited vs sustained use for a second there

I mean.. I get it, there isn't a lot to work with, it is a thing that _stores files_ after all, and it most closely resembles a normal filesystem.

But it does make GCP's storage product naming even more confusing overall (after "Cloud Storage" vs "Cloud Datastore" mentioned above).

So now they have two cloud storage products, called Cloud Firestore and Cloud Filestore? Their branding team is on top of their game, as always.

You're talking about a hypothetical fund with 20/20 hindsight.

Alpine.ai | ML engineer, Full Stack engineer | San Francisco | ONSITE

We're well-funded, early stage startup in SF. You have a chance to join our team as one of the founding engineers.

Did you know that Amazon Echo is the best selling product on all of Amazon this holiday season? We're building a platform for conversational product search, to make these 450M voice-enabled devices home for brands and big e-commerce players.

If you're a senior machine learning engineer (ideally, with experience or interest in NLP), or a full stack engineer who wants to work on our ML platform -- we'd love to talk to you.

Check out open positions at our website https://alpine.ai/careers/. Happy to tell you more by email oleg@alpine.ai or over a coffee.

Alpine.ai | San Francisco | ML and full stack engineers | ONSITE

Did you know that Echo is the best selling product on all of Amazon this holiday season? We're building a platform for conversational product search, to make these 450M voice-enabled devices home for brands and big e-commerce players.

We're well-funded, early stage startup in SF. You have a chance to join our team as one of the founding engineers.

If you're a full stack engineer who wants to work on our ML platform, or a senior machine learning engineer -- ideally, with experience or interest in NLP -- we'd love to talk to you.

Check out open positions at our website https://alpine.ai/careers/. Happy to tell you more by email oleg@alpine.ai or over a coffee.

We've looked at Airflow. Two main things why we went with Luigi are

* time-based and dependency-based scheduling do not mix well in our experience, if you try to do both at the same time you end up with a very complicated execution model. Luigi is much simpler, it is basically a makefile. To handle time one can just add dummy dependencies where "exists()" function returns true if now()>X, and/or just run the whole thing by cron.

* We heavily use dynamic dependencies in Luigi, where one job decides how many childs to spawn and what parameters to use at run time. Makes it easy to do e.g. mapreduce style jobs purely in Luigi.

Yes, schema wise that's pretty close.

However, last time I looked that didn't seem like a use case InfluxDB invested a ton of effort optimizing for, especially if you tried to store a lot of separate "event histories" either as separate measurements or by using tags. By "a lot" I mean 100K-100M histories. It may have improved since.

Besides, there is somewhat fundamental tradeoff between allowing efficient realtime granular writes, which I believe a priority for InfluxDB, and building efficient indexed store for millions of histories/event trails, which is more of a TrailDB use case. Kind of OLAP vs OLTP.

Exact same photo was making rounds in Russian Facebook few weeks ago, judging by the door and the handle I bet it is indeed Moscow, not London.

As in this case, no one saw the guy actually skimming cards. Also if he was doing really that, why not hide POS in a bag or his pocket?

This problem definitely existed with official 14.04 (HVM) AMI, though I haven't re-tested this recently, they may have fixed it. It did have some kind of SR-IOV driver but it was too old.

Stock Ubuntu needs SR-IOV driver to get to the actual bandwidth limit on ec2, it makes a lot of difference. We routinely get to ~2 Gbps down from S3 with that setup (using largest instance types).

edit: Gbps not GBps

We wrote our own scheduler, called Quentin, which we may opensource. Essentially it is a simple queue that takes care of running containers and managing pools of instances that containers run on by integrating with AWS autoscaling.

It is a very straightforward service, for example it doesn't do retries (done by Luigi), it is not highly available and not distributed. That makes things much easier to maintain, and we don't need HA since jobs inputs/outputs are in S3 anyway. Luigi makes it easy to restart parts of the pipeline if scheduler ever goes down.

Yeah, that's my point: if you're happy with this all-or-nothing logic, you can just use dummy target files.

Allowing steps to partially succeed seems like adding quite a lot of complexity, and I'm not sure where it would be beneficial.

Multiple outputs are tricky, what you going to do if you run a step and it somehow produces 2 out of 3 expected outputs? I mean, you need a way to decide whether whole thing failed or not, easiest way is indeed to have dummy targets.

Anyway, take a look at https://github.com/spotify/luigi, it is basically make-like tool geared toward data pipelines. We are experimenting with it for our data pipeline (which is quite sizeable, ~30TB/day gzipped, thousands of files, a bunch of processing steps) and while it lacks some things, especially in the UI, this approach seems to work quite well.

> There's this new idea where you can have an operating system, run the command 'apt-get install foobar', and BAM all of a sudden foobar is installed.

Nah dude, that's never gonna fly. See, as mentioned in the top comment, no one uses and knows the OS they are developing for nowadays (though only windows users are looked down upon for some strange reason).

It is always more fun to not spend precious minutes reading about how to create an apt package, but reinvent the wheel for the hundredth time and implement another under-featured and buggy package manager in our current favorite language, and then add workaround after workaround to actually make it somewhat close feature wise to apt or rpm (i.e. usable).