Thanks for sharing about Seektable. Neat that you’ve already built out a search function, and even aside from that it looks like a really interesting tool.
1,361 karma · joined June 21, 2014
https://delphihq.com
Thanks for sharing about Seektable. Neat that you’ve already built out a search function, and even aside from that it looks like a really interesting tool.
I've spent the past 4 years working as a data analyst and analytics engineer. One of the biggest pain points for analysts is that, even with tools like Looker and Tableau, they still spend a ton of time answering fairly simple questions from less-technical users instead of doing high-value analytics.
Often as a data analyst I've felt like a computer translating requests from English into SQL. That got me wondering - could an actual computer do it instead?
For a while, I thought the answer was "no" (or at least not in the near future). But then OpenAI's Codex[0] was released. I started using GitHub Copilot[1] every day while writing SQL in Visual Studio, and it felt like magic. The autocomplete was getting my code right a lot of the time, even for complex SQL queries.
So I decided I'd give this a go. I've built a demo using the Codex API and dbt Labs's "Jaffle Shop" example database, which will be familiar to many in the Analytics Engineering community. It's a simple database, but the results are promising. The demo app gets a lot of queries right, and it does so without any database-specific logic.
The only input needed is a schema.yml file describing the tables and columns (which any dbt project would have) and 2 or 3 example queries, so I am hoping to open up testing to more data warehouses and users in the near future.
I hope you enjoy using the demo. If you have any feedback, ideas, or would like to test it out on your own data warehouse, my email is in my bio!
[0] https://openai.com/blog/openai-codex/ [1] https://copilot.github.com/
One thing I’m not sure about though: can you use the metrics outside of the native tool, and if so how?
That is, I see Looker as a BI tool, not a metrics layer, since you mainly use the metrics you define inside Looker, not in other tools. On the other hand, something like MetriQL[0] is a pure metrics layer that can supposedly be used anywhere.
Is this both? If so, some better documentation around how to use the metrics layer would be helpful (or maybe I just didn’t look in the right place).
But I'm not sure that has the policy implications they think it does. No one in the US (or outside of Australia/NZ/parts of Asia) have really been _trying_ to eradicate Covid.
But at some point, we will have to figure out what the goal is of our Covid policies.
At first (Spring 2020) the goal was to "flatten the curve" i.e. prevent hospitals from being overwhelmed. We largely did that.
Next, the goal was to postpone as many infections as possible until vaccines were widely available. They are.
So the question is: what comes next? Perpetual restrictions and lockdowns every time there is a flare-up to try and prevent as many (mostly unvaccinated) deaths and hospitalizations as possible? Or do we just accept that everyone is going to get it eventually and we've done all we can to minimize the risk (to people who choose vaccination)?
The title is strongly associated with the dbt community, so it could imply you’re using dbt for your data modeling (not necessarily a bad thing, as it sounds like it would be a good tool for your use case).
I am interested to see how long it’ll take someone like Fivetran to buy one of these new “reverse ETL” companies so they can be a full end-to-end solution.
I used to work in AdTech and keep an eye on the industry. Most of the responses to those developments have been around consent management and first-party authentication, which are disappointing both from a privacy lens and as an advertiser (they often rely on signed-in users which are usually going to be a small % of traffic).
To me, this shift towards privacy feels like an opportunity to reinvent online advertising.
My core question is: what if contextual advertising (based on the content of the site/page you’re on) could be as good or better than all the creepy tracking that happens now? I think the answer is that with recent developments in natural language processing (NLP), it can be.
Thanks for reading, and I’m happy to answer any questions!
That's not to say these laws are bad, just that the giant additional privacy risk they pose is kind of funny to think about.
The fact that researchers in Wuhan were studying (some of?) these features is stronger evidence when combined with the location. However, it also seems reasonable that features of viruses researchers study because of their likelihood to increase human transmission would show up in a naturally occurring virus with a lot of human transmission.
But conditional on being in a pandemic the chance that the virus that arises to cause it is like SARS-CoV-2 is super high!!
There’s no conspiracy, just statistics”[0]
That said, the fact that the outbreak started so close to the Wuhan Institute of Virology is certainly suggestive that it escaped from the lab, though that doesn’t mean it was necessarily “man made.”
[0]https://twitter.com/epiellie/status/1281960135796101120?s=21
It’ll probably be the next big “feature” I work on.
UPDATE: There is a back button now.
This was supposed to be Day 2, but ended up taking me about 3-4 days to get to the current state.
The problem I wanted to solve is that basically, I am addicted to Twitter. I get my news from Twitter, so I'm constantly pulling it up and scrolling through it throughout the day. But there's a lot of noise on Twitter--a ton of low quality content, mean-spirited comments, and so on. I wanted a way to stay on top of the news that met the following criteria:
1. One place to see new articles/posts from a bunch of different sources. I didn't want to have to pull up each news site individually.
2. No comments or noise--just the latest headlines.
3. Customizable - I can add new sources I come across, group them in different ways by subject.
4. Public - If someone else added some interesting news sites, it would be cool to see that and find something new to follow myself.
5. No authentication - Handling passwords and user state is a pain. Just let everyone see everything!
So I made "PaperPup" (the name came from the idea of your dog fetching you the morning paper). While there are other RSS aggregators out there, a lot of them feel clunky to me and don't necessarily have the public aspect I wanted. In technical terms, it's a Rails API backend with a "kind-of-vanilla" HTML/CSS/JQuery single-page front-end. As I mentioned I'm not really a software engineer, so this is nothing fancy; I wanted to make sure my skills on some of the basics were still up-to-date rather than experimenting with whatever the hot new languages/frameworks are these days.
Anyway, I hope you all like it! If you have any suggestions or questions (technical, feature-wise, or other apps to use), I'd love to hear them.
2. Bubble - no-code!
3. Stripe - already big but has the potential to be the next Google/FB/MSFT etc
Economists also have a big causality literature which might be less accessible for HN folks but I think is still interesting and important. For a good intro to all that, I suggest Scott Cunningham’s “Causal Inference: The Mixtape.”[2]
I also love how everything is a table. Looker does this, and it's great (though Chartio's implementation looks better). Tableau does not, which makes certain things a headache.
I do wonder how well this will scale to large datasets and columnar databases, though. Showing a preview of the tables is really cool and useful, but it is not what data warehouses like Redshift are designed for. The preview probably wouldn't load in time for my users to see it unless they just sit there waiting for it.
I was lucky that my first job was as a support engineer at a data-centric tech company, which is where I learned these. I've often thought about how to teach them to data analysts coming from a non-engineering background. This is comprehensive but clear and would be a perfect resource for training someone like that. Thank you!
For a smaller company, it makes a lot of sense for us to use off-the-shelf tools like these rather than rolling our own pipelines. Every once in a while we run into issues where Stitch isn't quite as flexible as we'd like (can't set dist/sortkeys etc), but that's the tradeoff you make for its simplicity.
DBT is amazing and I can't recommend it highly enough!
Looker works for analytics, but we're starting to do more data-sciency work, and it doesn't have built-in Jupyter notebooks or anything like that. Does anyone have a solution they use and like for hosting and sharing notebooks alongside a stack like this?
I could actually see this being pretty useful as an ETL layer to go along with Metabase if someone were trying to build a free/open source BI stack.