3,444 karma · joined July 30, 2013
There isn’t really a much more productive web dev setup than Rails + your favorite LLM tool. Will take time to earn Gen Z back to Rails though and away from Python/TS or Go/Rust.
I gave up on using GCP even though the products like BigQuery are way better just because I got burned too many times like with the Google Domains -> Squarespace transition.
I’m thinking of switching back to a bare metal provider now like Vultr or DO (would love to know what people are using these days I haven’t used bare metal providers since ~2012).
Also, completely unrelated does anyone know what the best scraping proxy is these days for side projects (data journalism, archiving etc.)?
Like all the people surprised by Deepseek when it has been clear for the last 2 years there is no moat in foundation models and all the value is in 1) high quality data that becomes more valuable as the internet fills with AI junk 2) building the UX on top that will make specific tasks faster.
LLMs do not think, they do not perform logic they are approximating thought. The reason why CoT works is because of the main feature of LLMs, they are extremely good at picking reasonable next tokens based on the context.
LLM are good and always have been good at three types of tasks:
- Closed form problems where the answer is in the prompt (CoT, Prompt Engineering, RAG)
- Recall from the training set as the Parameter space increases (15B -> 70B -> almost 1T now)
- Generalization and Zero shot tasks as a result of the first two (this is also what causes hallucinations which is a feature not a bug, we want the LLM to imitate thought not be a Q&A expert system from 1990)
If you keep being fooled by LLM thinking they are AGI after every impressive benchmark and everyone keeps telling you that in practice LLM are not good at tasks that are poorly defined, require niche knowledge, or require a special mental model that is on you.
I use LLM every day I speed up many tasks that would take 5-15 mins down to 10-120 seconds (worst case for re-prompts). Many times my tasks take longer than if I had done it myself because it’s not my work im just copying it. But overall I am more productive because of LLM.
Does LLM speeding up your work mean that LLM can replace Humans?
Personally I still don’t think LLM can replace Humans at the same level of quality because they are imitating thought not actually thinking. Now the question among the corporate overlords is will you reduce operating costs by XX% per year (wages) but reducing the quality of service for customers. The last 50 years have shown us the answer…
And I don’t even know what to say about the president and First Lady releasing official meme coins…
I wish there was a way to either shebang something like this or build a wheel that has the full venv inside.
My usual process is Dev tools -> Copy as CURL -> delete unnecessary headers -> translates to requests in python (these days I just use ChatGPT) -> wrap in python sdk for managing auth etc.
The OP’s correlation features are really nice though.
I really like the typescript graphic walker: https://github.com/Kanaries/graphic-walker
So I guess that’s who it’s for.
I’ve only spent an hour with it though obviously.
I would highly recommend parents consider teaching their kids using processing (p5.js), it’s super visual but still “real” code so you still build that muscle memory of thinking in loops and typing out real code: https://p5js.org/tutorials/
There’s lots of art and games to be inspired by: https://openprocessing.org/browse?time=anytime&type=tags&q=g...
And there’s some great books from dan shiffman on it that are super visual but still teach programming concepts: http://learningprocessing.com/ https://natureofcode.com/
When you start out programming that young it is hard to go from idea to thing on the screen doing what you want. So the advantage of using processing is it keeps kids engaged and removes the frustration of not making progress since everything is visual you’re always moving around stuff on the screen every frame so it’s quick and easy feedback.
I’ll take a try this weekend as well.
> In qSQL this is: aj[`sym`time; t; q], which means perform an asof-join on t, looking up the nearest match from table q based on the sym and time column.
> In standard SQL, again you’ll have difficulty: sql nearest date, sql closest date even just the closest lesser date isn’t elegant. One solution would be:
> WITH cte AS (SELECT t.sym, t.time, q.bid, ROW_NUMBER() OVER (PARTITION BY t.ID, t.time ORDER BY ABS(DATEDIFF(dd, t.time, p.time))) AS rowNum FROM t LEFT JOIN q ON t.sym = q.sym) SELECT sym,time,bid FROM cte WHERE rowNum = 1
> It’s worth pointing out this is one of the queries that is typically extremely slow (minutes) on row-oriented databases compared to column-oriented databases (at most a few seconds).
This is a really nice example but I think it’s more about this as of join being a really useful operation, it appears both pandas https://pandas.pydata.org/docs/reference/api/pandas.merge_as... And duckdb https://duckdb.org/docs/guides/sql_features/asof_join.html
Pandas:
> pd.merge_asof(t, q, on='time', by='sym', direction='backward')
Duckdb:
> SELECT t.sym, t.time, q.value FROM t ASOF JOIN q ON t.sym = q.sym AND t.time >= q.time;
So it seems more like this is benefitting from a useful feature of time series databases rather than the features of an APL-family language.
Personally I find the pandas syntax to be the most straightforward here.
I also think there is some merit to “high syntactical density” clearly if you can see the entire code in one place instead of having to navigate through many files or sections that’s beneficial. (Heavily discussed in the last big HN thread: https://news.ycombinator.com/item?id=38981639)
I also think JQ has proven the merit of tacit functional languages in that you can concisely write arbitrary transforms/queries on json that can be more expressive than SQL (many SQL engines have added JSONPath anyway). And I also think postfix is great for processing pipelines.
But I am not totally convinced in the approach of APL/J/Q/KDB for the combination of terse style + prefix + tacit because it makes the code so difficult to read. I think if you took an approach similar to JQ where instead of relying on symbols operators were just human readable words it would be easier to get us half way there to trying out the verb, adverbs etc. approach of the APL family. The problem with making it human readable text is that you lose the conciseness which is part of the draw of the APL family as they want to have a high syntax density and analogous code to mathematical expressions.
The syntax has the same problem as perl in that you have to learn too many symbols that are hard to look up. And this combined with the tacit style makes it difficult to parse what the code is doing.
I think ruby went in the radically opposite direction in that it provides a similar feature set to perl with similar functional features but everything is human readable text. I feel like there’s room for a human readable version of J that relies less on syntactical sugar.
I do think the direction Klong has gone with the syntax is an improvement: https://t3x.org/klong/ambiguity.html
I’m curious if anyone has a go-to article/example of why an array language would be a good choice for a particular problem over other languages?
I know the second chapter of J for C programmers is kinda like that but I didn’t really find it convincing: https://www.jsoftware.com/help/jforc/culture_shock.htm#_Toc1...
> count::{#x}
These two expressions are not valid in R or Python. They are implicitly declaring a function without declaring the arguments, it is implicitly iterating over the array and returning the result (all features of array languages to make it more concise).
Python’s + and / are a function of two objects (dyadic) and are syntactic sugar (they can’t be treated as function values, thats what the operator module is for): https://docs.python.org/3/reference/datamodel.html#object.__...
The array languages optimize for concise code so a lot of things are done implicitly that are done explicitly in Python.
The python equivalent:
> _sum = lambda x : functools.reduce(operator.add, x)
> _count = lambda x: functools.reduce(lambda acc, _: acc + 1, x, 0)
Of course in python and R there are built ins for these already (sum(x) and len(x)) but this is just to show what array languages are doing for you.
It’s basically a standardized way to wrap you Openapi client with a standard tool format then plug it in to your locally running AI tool of choice.
The real difference in approaches is between short lived environments that you redeploy from scratch all the time and long lived environments we nurse back to health with runbooks.
You can use lambda, kube, etc. or chef, puppet etc. but you end up at this same crossroad.
Just starting a process and keeping it alive for a long time is easy to get started with but eventually you have to pay the runbook tax. Instead you could pay the kubernetes tax or the nomad tax at the start instead of the 12am ansible tax later.
A common thing to hear in frontend these days is “you’re just rebuilding X” “you mean you want to rebuild X” where X is some trivial flavor of the month library.
All of the famous tools, databases and frameworks you see today was built by someone who said they can do it better and then they built a community around it.
They are showing a 20-30% improvement over Polars, Clickhouse and Duckdb. But those 3 tools are SOTA in this area and generally rank near eachother in every benchmark.
So 20-30% improvement over that cluster makes me interested to know what techniques they are using to achieve that over their peers.
I agree that pandas does not have the best designed api in comparison to say dplyr but it also has a lot of functionality like pivot, melt, unstack that are often not implemented by other libraries. It’s also existed for more than a decade at this point so there’s a plethora of resources and stackoverflow questions.
On top of that, these days I just use ChatGPT to generate some of my pandas tasks. ChatGPT and other coding assistants know pandas really well so it’s super easy.
But I think if you get to know Pandas after a while you just learn all the weird quirks but gain huge benefits from all the things it can do and all the other libraries you can use with it.
Like if you wanted to implement a search engine or recommendation system it’s not clear to me how you retrieve a training set.
I’m still new to the erlang/elixir community and I haven’t run it in prod yet but this is my experience coming from Java, Node, and Python.
It is just a way to wrap your python code into a self bootstrapping executable: https://ofek.dev/pyapp/latest/runtime/
You can get it setup pretty easily with hatch: https://hatch.pypa.io/1.9/plugins/builder/app/
But yea I agree the problem with python is there isn’t a tool like deno that just does everything that has to do with packaging, locking, venvs, and building a cross platform binary.
Big companies are like government bureaucracies now.
Its especially annoying since many use binary message formats and there isnt a great way to document an arbitrary binary message protocol.
A couple techniques im trying out:
- websocat and wsrepl for reverse engineering interactively: https://github.com/doyensec/wsrepl
- kaitai struct for documenting the binary message formats: https://kaitai.io/