nathanwallace

A completely fair point ... and one we debated at length.

Ultimately we decided that the convenience for users and documentation of a single location outweighed the benefits of following the basedir spec, particularly when deployed across operating systems. We looked at other tools, including terraform [1], in reaching a final decision. FWIW, we do let you customize the location of the "install dir" [2].

Hopefully the benefits of Steampipe outweigh the "peeve" factor in this case <grin>

1 - https://www.terraform.io/docs/cli/config/config-file.html 2 - https://steampipe.io/docs/reference/env-vars#steampipe_insta...

Steampipe is a new project we launched in Jan 2021. Turbot is leading the build, and has been working with cloud APIs since 2014. We're bootstrapped, and have a lot less than 100 people, but are hiring (engineering, marketing, tech writing) if anyone would like to get involved :-)

We currently have 37 open source plugins [1] like AWS, GCP, GitHub, Kubernetes, Hacker News, etc. More are in development by Turbot and community members [2]. Each plugin has many tables, e.g. 223 for AWS [3], 23 for GitHub [4].

The 200+ on the home page is referring to tables. Most tables have a single API source behind them, but others like aws_s3_bucket [5] have ~10 API calls for each row to collect related data like tags, versioning, etc.

(We're excited about the rapid growth of our plugins / tables, but can see that if you read it as plugin == data source then the 200+ would be wildly impressive.)

1 - https://hub.steampipe.io/plugins 2 - https://github.com/topics/steampipe 3 - https://hub.steampipe.io/plugins/turbot/github/aws 4 - https://hub.steampipe.io/plugins/turbot/github/tables 5 - https://hub.steampipe.io/plugins/turbot/aws/tables/aws_s3_bu...

YOW! Data 2021 (https://yowconference.com/data/) was an online conference held in May. Talks are all now available on YouTube (https://www.youtube.com/playlist?list=PLIpl4GKFQR6e2iVY2DZbp...).

If you are interested in Postgres FDWs, please check out Steampipe (https://steampipe.io). It's an open source project with a Go-based plugin model (similar to Terraform) to instantly query cloud services (AWS, GCP, GitHub, Slack, etc) using SQL. TBH, we've been completely shocked how far we can push this model querying real-time against APIs.

We're bullish on the ELT use case and think of virtual tables as "Data Rainbows" - structured, ephemeral access to cloud data. I spoke about this concept for YOW Data (https://youtu.be/2BNzIU5SFaw?t=183). Still structuring our thoughts here, so feedback and ideas would be greatly appreciated.

Disclaimers: Steampipe is open source. I'm a lead on the project. I can't stand listening to my own talk.

Agree that adding NIST controls would be great - so far we've focused on CIS, PCI and HIPAA.

That's exactly one of the use cases we are targeting. Beyond the standard benchmarks, Steampipe has a simple HCL + SQL language to write your own controls [1]. It feels like writing Terraform, but is for operational and security controls.

Hope you can give it a try, we'd love your feedback and suggestions!

1 - https://steampipe.io/docs/using-steampipe/writing-controls

Steampipe has open source cost and compliance mods for AWS, Azure, GCP, GitHub and Zoom. Full docs are available at https://hub.steampipe.io/mods

Disclaimer - I'm a lead on the team. We'd love your feedback and are excited to see what other custom controls people build!

Steampipe automatically installs and manages an embedded Postgres instance for you. It's shocking how lightweight that is - another way in which Postgres is awesome.

Running "steampipe query" will start and run embedded Postgres for the duration of the session. Using "steampipe service start" will run it in the background for access from your favorite 3rd party SQL tool.

As a child in Australia we had Danish backpackers come to stay at our place. I recall taking one set of visitors to the zoo, and specifically to see the Platypus since such an animal seemed too strange to be real. Thing is, the enclosure was closed for maintenance that day ... reinforcing their disbelief :-)

If you like SQL interfaces then please also checkout Steampipe (https://steampipe.io).

Use SQL to instantly query cloud resources like AWS, GitHub, Yahoo Finance, etc (no CSV yet). It's written in Go and uses Postgres Foreign Data Wrappers (similar to SQLite virtual tables).

Disclaimer: It's open source. I'm a lead on the project.

I wrote a Postgres FDW in Go to query Twitter using SQL - https://hub.steampipe.io/plugins/turbot/twitter

(Steampipe is an open source CLI to query cloud APIs using SQL. No DB required!)

Use SQL to query Hacker News with Steampipe - https://hub.steampipe.io/plugins/turbot/hackernews

It's an open source Postgres FDW using Go - https://github.com/turbot/steampipe-plugin-hackernews

Cross-region replication of data is always significantly harder, definitely consider if cross-AZ would be sufficient for your case (which it often is). Postgres can definitely be configured for a cross-region scenario, but the approach varies a lot depending on your recovery needs.

Postgres is incredibly flexible and scales beyond what most projects will ever need. We use it both as a very large scale CMDB store for https://turbot.com and as an small embedded DB for https://steampipe.io.

Use cases where we hit road bumps earlier were:

1. High churn data. Postgres copies data for each rite update and then vacuums to clean the old versions. These use cases dominated our DB load and moving them out helped a lot.

2. Large JSON blobs. Postgres automatically compresses and will pull large (~>8kB) columns into a separate “toast” table. This makes sense but also means they are often a real bottleneck if you expect to work with larger JSON blobs as you core model. pg is great for JSON, but if you are using it at scale do pre-work and testing of this factor and how it plays with your model.

3. Of course, long term archive of data objects is much more cost efficient and scalable in an object store like S3.

Bet on Postgres. Whatever you use case it will work great until you are large enough to really understand your data and optimize for your business.

Joins are supported, both between the cloud tables and with local data. Local data can be created in the public schema, which is available for writing & table creation. Writing & updates to cloud tables are not supported. It works using an embedded Postgres instance with foreign data wrappers implemented by Steampipe ... you could try adding SQLite to it as a Postgres FDW. Hope that answers your questions and helps!

Agree! We've just been doing a simple Spotify playlist before all hands, shared over Zoom. Works great with everyone just quietly hanging out together for a while and sets the mood for the meeting. Most recent was a 1992 theme for our internal launch celebration of https://steampipe.io (a tool to query cloud resources using SQL).

True. It was an incredibly powerful model, but developers like files they can see.

I mentioned in my earlier comment a "hand-drawn diagram of the architecture" - it was in fact a picture of the exact layout of this database including all the pointer blocks etc between the classes, methods, versions, etc.

One project I did was to convert that file format & pointer model to support larger database sizes of 16GB (IIRC). Fun exploring and updating that code. For testing I built a massive fake DB with random classes, methods, etc. The crazy thing was when that fake repo zipped down to almost nothing because it had so much repetition in the names!

Fun fact ... "VisualAge for Java was implemented in Smalltalk and the Team facility used ENVY/Developer under the hood" [1]. Of course eventually that product and team morphed into Eclipse.

I was lucky enough to play as small part as a college-age intern working at an Object Technology International [2] Lab in Sydney.

I soaked in the pair programming, watching the screen flash by. I read every book they gave me (e.g. The Toyota Way, this was before XP / Agile books existed). I was amazed they actually paid me to work on these problems. I was so proud to make contributions and still remember fondly the hand-drawn diagram of the architecture passed down like a relic through the team. I am still inspired by their passion and commitment to delivery.

I definitely would not be the person I am today without the career defining experience working for Adrian Cho [3] and Dave Thomas [4] during that time. Periods like that are rare in a career - grab them if you can!

1 - https://paulhammant.com/2017/09/01/smalltalk-envy/ 2 - https://en.wikipedia.org/wiki/Object_Technology_Internationa... 3 - https://www.adriancho.com/ 4 - https://www.davethomas.net/

Career change is often best one variable at a time. For example, change company size but stay in the same industry. Double jumps are generally harder.

Perhaps, focus on startups that reduce the number of change variables. For example, startups focused on satisfying the needs of enterprise IT customers should value your inside knowledge of the pain points.

Keep looking - the right thing will be out there. (For example, my company, Turbot, is actively seeking remote employees with enterprise experience and a pent up hunger to get stuff done.)

The Google Calculator returns incorrect results for the range:

  16384.69 - 16384.58 (0.10999999999)
  16385.69 - 16385.58 (0.10999999999)
  ...
  32766.69 - 32766.58 (0.10999999999)
  32767.69 - 32767.58 (0.10999999999)

Results are correct on either side of this range:

  16383.69 - 16383.58 (0.11)
  32768.69 - 32768.58 (0.11)