Seems like it's doing better than GPT-4o in most benchmarks though I'd like to see if its speed is comparable or not. Also, eagerly awaiting the LMSYS blind comparison results!
631 karma · joined April 7, 2023
prasoon [at] getlumen [dot] dev
Seems like it's doing better than GPT-4o in most benchmarks though I'd like to see if its speed is comparable or not. Also, eagerly awaiting the LMSYS blind comparison results!
The other way is to get an API key from OpenAI or Anthropic and then, you can pay per token which is incredibly cheap - load it up with $5 and it can easily last you months if you're not using it much.
As a dev, I cannot recommend these better models enough. The ROI on my $10-$20 that I spend every month is easily in hundreds to thousands of dollars.
Prompting does help though - if you ask the model to check it's own answer, it can often catch errors it made and improve. But the interesting part is precisely how does the model succeed at all at any spatial reasoning task? Surely it hasn't seen all spatial reasoning tasks in its training set. So, what's the other hypothesis? As you say, people suggest that there could be a world model that is constructed internally by the LLM.
However, as you can see right here in this thread that people disagree that you need a world model for solving such tasks.
Others have argued against the "inner world model" theory and suggested that solving reasoning tasks is merely an extension of the "stochastic parrot" scenario - i.e., they claim that no such world model exists and that the model has rote memorized these reasoning scenarios.
Another interesting bit of experiment people did when the GPT-4 class models launched were to test out spatial awareness. For eg, you could describe with words a construction made of blocks, spheres and so on and then ask questions about the stability of the structure. The newer models more often than not got it right.
Of course, that hasn't stopped people from making tired old claims of "stochastic parrot" and so forth but I think if your model can solve such reasoning problem (spatial or otherwise) then there's something really interesting to discover. I'm glad that folks at Anthropic are really trying to create a better understanding of these models (https://www.anthropic.com/research/mapping-mind-language-mod...)
Think about it - if you're a criminal, and you know about chat control, why would you risk your chats being leaked at all? Why wouldn't you use a different app that you know to be more secure (this already happens for any serious crime already btw)
It's precisely the law-abiding people whose privacy will be invaded for no-gain
See my comment here: https://news.ycombinator.com/item?id=40560873
Once accepted, it's almost impossible to repeal the law by the EP because they will need the approval of the EC and Council of the EU, both of which have members that are not directly elected by the people. EC and Council of the EU is also that body which so far has proven to be far more authoritarian in it's approach.
So, the parent is largely correct.
You can come down to SF and take a waymo today. It works great.
We're different in a couple of ways:
- You don't need SQL to transform and manipulate data in Pretzel - we have visual blocks where you can construct a transform chain (though we do support manual SQL blocks now!)
- We have better CSV parsing (I spent a fair bit of time on this!). See one of our test case CSVs here that breaks (https://imgur.com/a/O8XMfET)
- We allow plotting!
- We support PRQL - which IMO is a better way to build data pipelines (but we also support SQL)
- We have AI Blocks! You can use your API keys (or our server) to get SQL from text which you can edit
- We're currently adding support for Python via Pyodide (pyodide.org) so you can do complex data transforms, even train simple ML models right inside the browser!
- Lastly, we'll be adding a ton more features to Pretzel over the coming months (many of them have been asked for in this thread actually!)
Support for multiple files is crucial - we'll be adding that soon. Right now, we're rethinking our whole task execution architecture so any major features will have to wait until then.
It's almost a meta thing but I think dev-tools have been getting better and better over the last 10-20 years while data tools seem to be stuck in the past. Devs, when then see friction, go build something to solve it while this tendency is much less common in data scientists, for eg.
https://news.ycombinator.com/item?id=39732135
Didn't know about OpenRefine - but one major difference is Pretzel is web-based meaning no installation necessary (important for work laptops, for eg). We're also going to be adding a ton of features to this - think Jupyter notebooks crossed with Metabase crossed with Datasette/Pretzel.
However, there's a couple of major differences:
- Datasette - to me - is a tool for devs - it takes a fair bit of tech know-how to understand how to load new data in datasette, install plugins and deploy an instance. I wanted Pretzel to have a "low-ceiling, high-floor" - meaning the tool should work well for non-technical folks but also shouldn't hinder technical folks and power users
- Datasette has a unique way to explore data based on "facets" where you can dig through data. It's a powerful way to explore but I personally don't think Datasette is the best tool for complex data transforms - its strengths lie in exploring datasets instead.
- For accessibility and speed, we wanted something browser local and fast - hence the focus on DuckDB and canvas rendered tables. Datasette requires setting up an instance and that's additional friction for non-tech folks and isn't particularly designed for large datasets (is my impression - I might be wrong)
- Lastly, we have ton of future plans - including in-browser python support via WASM (Pyodide), local LLM support, ability to connect databases etc.
I think there's space for both tools!
Tad viewer looks perfect for embedding in Pretzel! Is it easy to embed it in web apps? I’m on mobile right now and did a quick search and didn’t find anything. I’ll definitely be in touch!
So, it should be pretty straightforward to let folks drop a SQLite file instead of a CSV file! I haven't been able to get extensions to work on WASM so far though but I will definitely take a look to see if we can make this work!
This is quite interesting - we've not explored really large files so far and being honest, we haven't thought that far either. Didn't know about CBOR! I will have to look deeper into how this can save on memory. I was wondering though, since WASM memory is limited to 4GB, if I have sufficiently large memory on my compute device, at least one tab should be able to handle 1gb+ files too, correct?
Alright, did a bit of digging and the flamegraph points to a problem with the Table component. I'd hoped that BlueprintJS tables would be performant enough for our usecase but apparently not!
We'll try to move to canvas based rendering ASAP - that should fix any lags in the filter UI
I really liked that interface and haven't really found anything near as useable yet. Perspective JS (a free library by JP Morgan, you can try it here: https://perspective.finos.org/block/) has some really cool functionality in this direction but it has its one data processing engine, query language, rendering engine etc so we couldn't have used that for this project.
But, this is an interesting thought - I definitely would want to see this functionality in Pretzel - the only question is how to prioritize this. If you know of any performant table libraries that support collapsible pivots out-of-the-box, I'd love to integrate that. Alternatively, we'll write our own!
- I don't have excel on a Mac so I need to use Google Sheets. Google Sheets crashes on any file over 30-40 MB (at least on my macbook air)
- I think this type of chained data transform is a much better (more code-like?) way of doing transforms. You can see exactly how you got to your end-result. Vs, for example, in Google Sheets, you're modifying data in-place and your transform history is only accessible via Undo and Redo. This is far more reproducible short of using a python script.
(I've worked as a data scientist for a while and I suppose that influences how I think about chained data transforms vs the in-place Excel way too!)
But this is just for now - the long term vision is: Being able to easily switch between visual no-code blocks, SQL and Python in the same browser-local "notebook"/"workbook" that's easily shareable. I think it doesn't take much convincing that such a tool would be more powerful than Excel for certain workflows!
That makes sense. I ran into some old issues about Pivot but it seems it's not been implemented yet. And yes, s-strings wouldn't work - the DuckDB pivot statement looks like this:
PIVOT ⟨dataset⟩
ON ⟨columns⟩
USING ⟨values⟩
GROUP BY ⟨rows⟩
So, we'd have to pass the `dataset` so far into the pivot statement. This is where CTEs come in handy but happy to hear if there's a better solution :)