totalhack

From our good friend GPT-4 who can summarize it better than I can...

A semantic data layer is an abstraction layer that sits between the raw data stored in databases and the applications or analytics tools that use this data. The purpose of this layer is to provide a consistent, business-friendly interface to data that might be stored in a variety of formats, tables, or systems. By doing so, the semantic data layer helps in simplifying complex data structures into more meaningful, understandable models. Key Features:

    Unified View: It provides a unified view of data from multiple sources, making it easier for users to access and understand the data without having to know the underlying structure or complexity.

    Data Governance: It can enforce business rules, security policies, and data quality measures, ensuring that data is consistent, compliant, and accessible only by authorized users.

    Flexibility: A semantic layer is often designed to be flexible, allowing business users to adapt to changes in the data model without requiring changes to the applications themselves.

    Query Simplification: The layer simplifies the process of querying data by providing a more user-friendly way of accessing and manipulating data, often through a drag-and-drop interface or other graphical tools.

    Decoupling: It decouples application development from data source changes. When underlying data changes, you don't necessarily have to update all the applications that use it; you might only need to update the semantic layer.

    Data Integration: It can integrate data from multiple sources, providing a single "source of truth" for business users and applications.

Purpose:

    Simplification: Make data more accessible and easier to understand for non-technical users.

    Consistency: Ensure that everyone is working from the same definitions and business rules.

    Efficiency: Reduce the time and complexity involved in generating reports, analytics, and other data-driven functions.

    Data Quality: Help to enforce data quality and governance policies.

    Security: Provide a mechanism for enforcing security rules on who can see or modify data.

    Adaptability: Enable quicker adaptation to changes in business needs or data structures.

By providing a semantic data layer, organizations can ensure that their data is not only high-quality and secure but also that it can be easily used for making informed decisions. This is particularly valuable in today's data-driven business environment.

I think natural language to SQL is a worthwhile goal and will have its use cases, but perhaps fine-tuning a model on a semantic layer and having it talk to a semantic data modeling API will yield a more useful/practical result. You may even find the semantic layer makes a natural language approach moot in many cases.

In Zillion* I added an experimental feature that uses OpenAI to form a report API call from natural language. Fine tuning (vs prompt tuning) this on the semantic model with gpt-3.5 would probably yield some notable improvement in abilities, as I view it as more of a toy feature at the moment.

* https://github.com/totalhack/zillion

Just to throw it out there, an alternative approach is to have the AI talk to a semantic layer. There are pros and cons to this and in practice you'd probably want an agent that can intelligently decide whether to call your semantic layer or go through some other process that may involve multiple queries, function calls, etc.

IMO a semantic layer makes it so easy to query that natural language querying is often not beneficial, even for business users. I take a hybrid approach of the semantic layer solving >95% of use cases and custom SQL as needed in backend processing.

https://github.com/totalhack/zillion

Aren't they comparing to an old version of Nuxt too? From what I've seen on Twitter Nuxt 3 should be pretty close, no firsthand experience though.

You can incorrectly join on any columns you want. That's not really the fault of the column types IMO, that's a problem in the layers above.

As someone who wants stuff like this to succeed, I must say you lost me at DDR4 and ~1" thick.

In some cases, you can preempt criticism by being critical of your own work first in a public manner, such as maybe calling out something that you anticipate another dev might not like and explaining why you did it that way. That said, you may just run into people who like to complain and nitpick, dealing with them is more about being mentally and emotionally prepared to stay calm and grounded and try to steer conversations to a productive place.

Haven't had trouble into the low hundreds of millions on a single RDS server with 16GB of ram. YMMV.

This is a huge upgrade...but one major annoyance with their new interface is some keyboard shortcuts conflict with browser shortcuts. I'm looking at my repo and I ctrl+f to search the current file and start typing my search query assuming it's working...but it's actually focused on the file browser and started trying to edit the file instead?!?

Interested in learning more about the codesandbox workflow. We use Webflow for the flexibility and abilities it gives a small team with some non-engineers, but I want to move off of it if I can find a solution that maintains some of the benefits while addressing some of the drawbacks. Is there any content you can point me to that inspired this?

I didn't realize GitHub would struggle with this. I'll consider cleaning that up, thanks!

Cool, do you have an example showing a config/setup that can do dynamic joins? All the examples I see have joins explicitly defined at the cube level in the yaml config.

Zillion uses networkx to create a graph of tables and relationships so it sounds like it's doing something similar.

Views are a nice guardrail but can also get in the way! I've experienced this frustration first hand (and seen it with my business users) when using tools that focus on such a premise. Guess it depends how complex your data model is and how quickly your business is evolving / adding to that model. In the move-fast-break-stuff phase of a company they can get annoying at least.

I can't vouch for cube itself as I haven't used it but can confidently say such tools are highly valuable. I built one for use in my own business and have operated other businesses on similar tools.

It brings all data together, provides a consistent interface, and is way faster than writing SQL (though there will still be use cases for that). There is some up front cost to getting configured but it pays off in my case at least.

https://github.com/totalhack/zillion

Does cube support dynamically figuring out joins or is each cube a hard coded set of joins?

My similar, much less polished project doesn't require you to specify joins ahead of time outside of optionally defining a tree-like table lineage: https://github.com/totalhack/zillion

Built it for me, but available to all -- Zillion: a python data modeling and analytics library.

https://github.com/totalhack/zillion

Excellent. If I could recommend some inspiration, NVD3 (no longer maintained) had the exact set of features I found useful, as does billboard.js (https://naver.github.io/billboard.js/demo/#Chart.AreaChart). Namely the ability to hide/focus/show certain series from the legend, showing all series in the interactive tooltip, and (NVD3) highlighting in the tooltip which series you are directly hovered over. These all become helpful when you start charting lots of stuff at once. Thanks for being the bedrock of all of this stuff!

I believe their moat is that they have early access to OpenAIs newer models as they come out so they are always a bit ahead. Combined with some big partnerships they have already landed perhaps, which could yield highly valuable proprietary datasets to train v2 of whatever they are doing.

Apologies if I missed examples, how about interactive tooltip and interactive legend support? I'm sure it's doable since so many libs based on d3 do it.

I've always wondered why solutions like modular, "tiny", or even 3d printed homes haven't taken off more when there is such a lack of affordable housing. I know some of it is stigma, some lack of land or cost of land near more densely populated areas...but it's definitely not a quality issue (excluding 3d print, that's too new).

It would be a way better use of money than renting for many people. But how do we get around the "trailer park" stigma? Maybe that's a pipe dream.

I don't necessarily do the "broken" part but always leave a note or marker for myself about where I left off (with special syntax highlighting). I write out thoughts and options as I go in my org file notes so it's useful for rehydrating when I come back.

Tangential trick: walk away more often, especially when in the more intense problem solving or architectural phases.

I got a Thinkbook 13s that is basically the same specs as this but was half the price and not razor thin. With the Intel 12th gen chips that's arguably a good thing though if you want to be able to let them fly a bit before thermal throttling.

Ubuntu has been great on it aside from some occasional heat issues while sleeping. Running 6.x kernel now and just hoping that disappears with one of these updates but it happens rarely anyway. Overall very happy with it. Keyboard is way better than my last MacBook pro too.

Was gonna flag this same thing. Maybe their hoping for help from the open source community?

Tried Milvus first and never really got it off the ground. Ended up with Qdrant for its simplicity.

Almost none of the complaints are true of Webflow, FWIW. It's by no means perfect, but there are many use cases where it is a much smarter decision than contracting or hiring a site engineer.

Sprinkle in some custom widgets as needed and you can get pretty far (albeit not with optimal page speed performance).

My hope is open research and open source collaboration will continue to lead to breakthroughs, most importantly lowering the barrier for entry to training such capable models.

It's still relatively early days for this technology; if model research and processing power developments find an order of magnitude or two efficiency gain over the next decade maybe OpenAI's closed approach will no longer matter. Maybe that's wishful thinking though.

Translation: we can't figure out a good way to do this. Let's get community feedback/help.

Got a Thinkbook 13s Gen 4 running Ubuntu and it's been solid. Excellent specs for the price, battery life is the only thing that's mediocre but I'm plugged in most of the time. Oh and it tries to turn itself into a nuclear reactor occasionally while sleeping.

If I think an open source library question might have value to someone else I'll always open an issue on GitHub, even if they try to push to discord.

Maybe if discord communities added AI chat agents trained on their docs and discord discussions you'd have less of an information black hole.

Wish there was a better org mode plugin in vs code.

I've had malort! It's definitely worse!