huy

OP here. Sharing a bit of history on DBML.

Originated from dbdiagram.io, a simple tool that we built to help you draw ER diagram by using writing code [1]. We shared that project on HN about a year back. Since then it has been growing organically and reach over 80k users.

When we launched, what our users like most about that tool is the ability to write simple DSL language and get the chart visualized. Then over time, we started to see our users using that DSL language to document their internal database structure (without the necessary need to visualize it). They were using it as a database documentation/definition language.

Realized this is something we can add more value to the community, DBML was then born as a simple language to help you design, document and communicate your database structure with other developers. DBML is also open-sourced and would welcome contributions [2].

Basically with DBML you can: - Have a simple db.dbml file to document your database schemas, with proper context and descriptions - Your dbml file becomes the central source of truth of your database definition - Using simple CLI command to generate SQL creation file from DBML

[1] https://news.ycombinator.com/item?id=18066881

[2] http://github.com/holistics/dbml

Thanks for sharing. I understand the definition, but I'd say that by documenting these market interactions down, we're giving our colleagues a better intuition of the market/industry.

Granted it will be some further steps before they actually acquire these tacit knowledge, but it goes a long way into showing them what's it like.

Thanks! We actually built this in-house using D3js. Maybe we can look into open sourcing the rendering component.

Thanks! Can you work around by changing that to a number and try again?

Thanks for this! Looks like we haven't handled all the custom syntax cases properly. Will sort it out shortly!

For now, a workaround is to remove those lambda and it should work!

It's string-reading, we use PegJS for that :) https://pegjs.org/

Thanks! This diagram generated by rails-erd is indeed very nice. Something we should definitely look into to improve our visualization.

I think both tools serve slightly different purpose. How we're different is that we started by being a tool to help you draw databae diagrams by typing code. And adding Rails support was added rather quickly recently.

Thanks for the love! Yes since rails convention is clear-cut and standardized, we can read those implicit conversions in the backend and convert to necessary relationships.

Thanks! It seems like we haven't catered for more complex table syntax (when you added extra stuff like `force: delele` et al). We'll look into this and fix them.

Really appreciate your feedback here!

Thanks! We will support DDL of individual SQL databases (postgres, mysql, etc). I.e. you can copy your CREATE TABLE statements here and we will translate them.

Thanks for the feedback on the showing of relations by default!

OP here, let me share a bit on the backstory of this.

1 odd month ago, our interns were starting to build this little project, our goal is to integrate into the current product (a cloud-based BI), so our users could easily understand their database structure to analyze them.

After a few weeks, we realized that it could potentially become a standalone application to help data analysts/developers design or share their own database diagrams, so we decided to bring it out as a free tool.

The feedback so far from our early users has been really great, so we decided to launch here on HN. Please give it a try and share with us your feedback!

I work for a BI startup (competitor to Domo), so as part of my work, we did our own analysis of Domo product, and also chanced across prospects/customers that are current/ex-Domo users.

Domo is very "business users friendly", they do this by:

- Very pretty (and drag-and-drop capable) visualization. Not as compared to Tableau though, but still.

- Native mobile apps

- Their Domo cache + datasets concept, that makes it easy to load simple data into Domo and visualize

- Their commenting system?

I suspect that it's these things + all the marketing that made them sell well in certain contexts.

They have the dataset concept that ties to their cache, which makes it easy for business users to upload a file and "just explore", but then joining these datasets together is not quite straightforward. And you have to load each dataset one by one. So it's good for very basic use cases, but for advanced stuff that requires proper data preparation, it starts to fall behind.

Also soon Domo customers would run into performance problem, since Domo hosts the data for the customers.

Besides that, I feel like the entire product is a patch work, without proper thorough design/thinking. Feels like they hire a lot of different product managers, each responsible for one feature, but the features they don't fit together very well.. Some I even find redundant (seriously what's the 3D visualization of datasets/data sources in the product homepage for?).

It's been very interesting for me to observe the space. Our startup, we took the directly opposite approach to Domo in building our business: totally bootstraped with no funding. We've been profitable since year 1, working with customers from unicorn startups, public companies to fast-growing startups. We're very conscious/aware about the space to shape our product strategy.

I think ultimately the team and the product are the most important in the long-term success. I can say that among the solutions, we're most impressed by Tableau, PowerBI and Looker.

In case you've read this far, do check us out: https://www.holistics.io

If you want something that's affordable, yet can serve the full need of what open-source tools can't, do check us out - https://www.holistics.io

We've worked with both small, medium to big tech unicorns. I built the product during my role as data engineer at previous tech startup's employer (we got acquired), so we understood well what are the features desired by SME/startup.

As I read through your list, I feel like our product (https://holistics.io) can serve you well on these fronts

- We are both ETL + BI, we help you get data into your DW easily (CSV Upload, DB Sync, etc)

- Support for dashboard-wide filters, model joins, pivots

For complex time comparison (last week, QTD, YTD) we have metrics spreadsheets: https://docs.holistics.io/metrics/

It really depends on your budget, data volume and nature. So each person can only offer advice.

I did this a while back at my previous employer/startup (it got acquired), we used AWS, PostgreSQL, Hadoop/Hive, using SQL + custom Ruby script for ETL + processing. You can read some of that writing here https://engineering.viki.com/blog/2014/data-warehouse-and-an...

If I can distill them into bullet points, there probably are:

- SQL is great, stick with it.

- We started with PostgreSQL and scales a long way with it (using techniques like table partitioning, unlogged table, etc), then slowly split into Redshift, and later Hadoop/Hive.

- Handling events/behavioural data (JSON/semi-structure at large volume) are very different from handling transactional data (structured, lower volume).

- Take a more lean/incremental approach to it: get a basic DW setup, load data that you can immediately act on, act on it (build reports, run analysis, show to management), then repeat.

You can also check out my startup: https://www.holistics.io , where I turned those experiences above into a data platform that automating DW + BI.

We allow you to do "lean ETL" on top of customers' DW infrastructure (be it Redshift, PostgreSQL, BigQuery or others). We work with small startup to unicorn tech companies.

To add on to your list:

https://www.holistics.io (I work here).

You can try us out.

https://www.holistics.io

Some benefits:

- Designed with the analysts who's comfortable with SQL in mind, thus extremely flexible.

- Extra visualizations catered for product/marketing analytics: Conversion Funnel, Cohort Retention

- We have in-built ETL that helps analysts load data from Excel, CSV, Google Sheets to your reporting DB without bugging engineers

Wanted to share about my BI startup here for those who're interested too: holistics.io - we're a small, self-funded BI startup in Asia (Singapore). Started 2 years ago and have customers around the globe, from small startups to big unicorns.

https://www.holistics.io

We're not free or open-source, but a lot more affordable than other competitors. Our pricing starts with $50/mo for 5 users. We have a few customers who moved to us from open-source alternatives or other commercial competitors.

We support PostgreSQL, MySQL, SQL Server, Redshift, Greenplum, BigQuery, Presto, etc.

If you try it out let us know you're from HN and your first 2 months are on us.

I really like Redash, it’s one of the early tools that introduce this concept of turn SQL into chart to developers, and also teach developers to learn and write better SQL, altogether without any cost. I evaluated Redash during my past company back in 2013 (we were also using Tableau), but due to some Redash’s lack of features (no support for filters, lack of permission control, sporadic performance), we went and build something inhouse with similar approach (turn SQL into charts).

And inspired by the same path Redash founder took, that internal project turned into a startup by itself.

We’re relatively new but getting good momentum. Some of our customers went with us after evaluating both. While we don’t have a self-hosted open-source version, our pricing only starts at 49$/mo for up to 5 users (pretty affordable for startups IMHO).

You can check it out here: https://www.holistics.io

While Quicksight is game-changing in their pricing and trying to disrupt the usually-overpriced BI market (the same thing it did very successfully with Amazon Redshift and data-warehouse market), as the other comment pointed out their product is still not up-to-par with existing ones.

If you're looking for affordable BI platform for startup, check out Holistics [0] (note: I'm cofounder), we are very tech startup-friendly, and our pricing starts at $49/mo for up to 5 users. Our customers range from 3-person shop to 600+ person unicorn startup.

[0] https://www.holistics.io

Check out Holistics.io (disclaimer: I'm a cofounder). While I can't say we can solve all your listed problems above, we provide enough tooling on top of your DW to help you pre-process (clean, aggregate) them.

https://www.holistics.io/data-preparation/

And on the part about firehose data, you might already know this, but Kafka and their line of work should be aligned with what you're after.

I think this won't work because Redis' sorted set only allow members with different values. In this case the value mutations could be the same.

OP here, thanks a lot for the overwhelming suggestions, I'm trying to slowly digest all of them.

To give a little bit more context about the problem. This is related to our data analytics infrastructure that we're building (we did a technical writeup of it here - http://engineering.viki.com/blog/2014/data-warehouse-and-ana... )

If you look at our data infrastructure diagram (in the posted link), the thing we're trying to improve is the hydration system (where it takes in a record in real-time and try to inject more time-sensitive information into it).

E.g. When a user watches a video (thus a video_play event sent), we want to know if it's a free user or a paid user. Since the user could be a free user today and upgrade to paid tomorrow, the only way to correctly attribute the play event to free/paid bucket is to inject that status right right into the message when it's received.

Building the system this way (using the hydration service) makes our service very prone to error and indeterministic (since you only have a short window to hydrate the message, and you can't replay a hydration).

That's why we're looking at building a historical lookup service that remembers all the different changes of a data object over time, so that replaying a hydration becomes deterministic.

At the moment we're processing around 100M records a day and growing. That translates to about 100M read requests. The key size should be around 1M. Not a lot but still at some scale that puts us in the position to think about scalability and performance.

Before asking this question, we did look around and also looked into how we'd build it using existing database technology. We thought of 3 different ways using relational DB (PostgreSQL), simple K-V store (Redis/Riak), and Cassandra (with column family). Yet we want to get more opinions of you guys :)

Thanks for the pointer. I'm reading more about Secondary Index and Range Key in DynamoDB and it seems to fit well to what we're looking for. The only slight concern at the moment is it's AWS-hosted service so we have to be reliant on AWS (and its cost structure), so we might have to do some cost analysis there. But other than that it seems to fit the use case!

Thanks. We did study this article a while back while building our data infrastructure. We wrote about our infrastructure here: engineering.viki.com/blog/2014/data-warehouse-and-analytics-infrastructure-at-viki/

In fact, to give you a little bit more background on the problem, if you look at our data infrastructure diagram (in the posted link), the thing we're trying to improve is the hydration system (where it takes in a record in real-time and try to inject more time-sensitive information into it).

E.g. When a user watches a video (thus a video_play event sent), we want to know if it's a free user or a paid user. Since the user could be a free user today and upgrade to paid tomorrow, the only way to correctly attribute the play event to free/paid bucket is to inject that status right right into the message when it's received.

Building the system this way (using the hydration service) makes our service very prone to error and indeterministic (since you only have a short window to hydrate the message, and you can't replay a hydration).

That's why we're looking at building a historical lookup service that remembers all the different changes of a data object over time, so that replaying a hydration becomes deterministic.

At the moment we're processing around 100M records a day and growing. Not a lot but still at some scale that puts us in the position to think about scalability and performance.

I do think s/he has a point. Some people would want to login and manage their upload files.

Though you can pretty much upload files without having to login whatsoever. Pretty much similar to Github Gist, or Imgur.

This is very cool. Thank you!

Thanks a lot! I think that is a "killer feature" too. Will look into implementing that.

Thanks! That would be something we consider adding next :-)