476 karma · joined September 16, 2024
kian at sturdystatistics.com
https://github.com/ghodouss https://sturdystatistics.com/
I run a small company and needed to budget solid amount of chunk of time for next year to dig into improving this component of our system. I respect your perspective around holding high standards, but I do think it's worth getting excited about and celebrating reliable performant software that demonstrates consistent competence.
There was interesting stuff here: the human readable format, auto publication. But the tone and framing bashing a reliable open source project really turned me off.
1. To what does 71% accuracy refers? I assume picking which team will win in any given matchup?
2. How does 70% compare to say, picking the team with the better record to win, or some simple time weighted rolling average of SRS record (metric that factors in opponent difficulty)? Because a solid percentage of games are between teams of extremely different talent levels, especially as tanking becomes more prominent later in the season.
We do offer an api layer (the website is a light layer above this) over the low level statistics code focused on making it super easy to apply to language data if you are interested in playing around with it: https://docs.sturdystatistics.com
Above C we built a python wrapper to help construct arbitrary Dirichlet and Pitman-Yor Processes graphs.
From there we have some python wrappers and store it all in a hierarchical DuckDB schema for fast query access.
The site itself is actually just a light wrapper around our API that simplifies this process.
We have some more technical write-ups on the internals of the model that are not hosted publicly (we have some on-going publication efforts applying those model to scRNA sequencing). But feel free to shoot me an email (in my profile) and I'd be happy to send over some of our more technical documents.
Our tool works a little different than LLM style tools. We are doing a bulk search — for academic search, ~1000 papers — and then training a hierarchical Bayesian model to organize the results. Once the model trains, it provides a visual representation of the high level themes that you can then use to explore the results.
The trade off is we are willing to lower the relevance filter to enable a broad set of exploration.
In the technical notes I sort of laid out our model graph on the document branch. We also have a topic branch that is also structured hierarchically: Uniform Prior → High Level Topic Word → Granular Topics → Document Lever Variation in Topics. We just directly visualize that hierarchical representation in the sunburst.
The low level model graph is all written in C and exports granular annotations of the model graph. We use the model output to annotate the original text data. We do some work to store these hierarchical results in a SQL queryable format in DuckDB.
What's cool about this process is it's all annotation based. You can query data at the topic level, analyze topics and sql, and at any point pull up the exact excerpts to which the high level data refers.
Curious what you've been using it to search for?
Under the hood, this model resembles LDA, but replaces its Dirichlet priors with Pitman–Yor Processes (PYPs), which better capture the power-law behavior of word distributions. It also supports arbitrary hierarchical priors, allowing metadata-aware modeling.
For example, in an earnings-transcript corpus, a typical LDA might have a flat structure: Prior → Document
Our model instead uses a hierarchical graph: Uniform Prior → Global Topics → Ticker → Quarter → Paragraph
This hierarchical structure, combined with the PYP statistics, consistently yields more coherent and fine-grained topic structures than standard LDA does. There’s also a “fast mode” that collapses some hierarchy levels for quicker runs; it’s a handy option if you’re curious to see the impact hierarchy has on the model results (or in a rush).
Edit: Got home and checked the error logs. There was a very long search query with no results. Bug on my end to not return an error in that case.
If you were hoping to use the citation network, it needs the url as input rather than the title.
Example: https://platform.sturdystatistics.com/deepdive?search_type=e...
On a random day, a light went off in my head that hierarchy perfectly addresses the stratification vs aggregation problems that arise in bandits. Unfortunately I’ve never had a chance to apply this (and thus see the issues) in a relevant setting since.
For those not wanting to read the whole story, GitHub repo with basic benchmarks and utilities is here: https://github.com/Sturdy-Statistics/duckdb-sparse-array-lis...
One example is rare class prediction on long form text data eg phone calls, podcasts, transcripts. Other networks including neural networks and LLMs are either not flexible enough or require far too much data to achieve the necessary performance. Structured hierarchical modeling is the balance between those two extremes.
Another example is in genomic analysis. Similarly high dimensional, noisy, low data. Additionally, you don’t actually care about the predictions, you want to understand what genes or sets of genes are driving phenotypic behaviors.
I’d be happy to go into more depth via email or chat if this is something you are interested in (on my profile).
Some useful reads
[1] https://sturdystatistics.com/articles/text-classification
I would say his work with Stan and his writings, along with theorists like Radford Neal, really opened the door to a computational approach to hierarchical modeling. And I think this is a meaningfully different field.
Edit: Haha I just found the textbook and I’m remembering now that I actually worked through sections of it back when I was working through BDA several years back.
It’s just a relatively dense book. There’s some other really good suggestions in this thread, most of which I’ve heard good things about. If you have a background in programming, I’d suggest Bayesian Methods for Hackers as a really good starting point. But you can also definitely tackle this book head on, and it will be very rewarding.