HNHacker News
TopNewBestAskShowJobs

kianN

476 karma · joined September 16, 2024

I like building hierarchical Bayesian models in C.

kian at sturdystatistics.com

https://github.com/ghodouss https://sturdystatistics.com/

submissionscomments
kianN··on Data-at-Rest Encryption in DuckDB
We are in the separate ACL/encryption key bucket. We provide a Bayesian data analytics platform/api for other companies. Each company can have hundreds to thousands of datasets ("indices") each of which has a separate encryption key, and those keys are also stored encrypted with an organizational level key that is rotated daily.
kianN··on Data-at-Rest Encryption in DuckDB
I'm more amazed that they released this as part of their open-source offering (not clear from my above comment). Encryption is a standard lever for open-source projects to monetize.

I run a small company and needed to budget solid amount of chunk of time for next year to dig into improving this component of our system. I respect your perspective around holding high standards, but I do think it's worth getting excited about and celebrating reliable performant software that demonstrates consistent competence.

kianN··on Data-at-Rest Encryption in DuckDB
I’m just continually amazed by the DuckDB team. We had built out a naive solution with OpenSSL to encrypt duckdb files, but that lead to a 2x runtime cost for first time queries and used up a lot of ram because we were encrypting/decrypting the entire file all at once. It seems like because DuckDB is encrypting at the page level and leveraging modern processors native AES operations, they are able to perform read/writes at practically no cost.
kianN··on Zohran Mamdani Just Inherited the NYPD Surveillance State
https://archive.ph/7Y5wV
kianN··on Deepnote, a Jupyter alternative, is going open source
I agree with this. I have no problem with a technical discussion comparing the features and discussing Jupyter’s shortcomings. But the Jupyter job postings and contribution graph screenshots felt both ill-spirited and not particularly relevant.

There was interesting stuff here: the human readable format, auto publication. But the tone and framing bashing a reliable open source project really turned me off.

kianN··on Show HN: DeepShot – NBA game predictor with 70% accuracy using ML and stats
Awesome thank you for the info!
kianN··on Show HN: DeepShot – NBA game predictor with 70% accuracy using ML and stats
Haven’t dug the repo too much yet but looks like a super cool project. I’ have a few questions:

1. To what does 71% accuracy refers? I assume picking which team will win in any given matchup?

2. How does 70% compare to say, picking the team with the better record to win, or some simple time weighted rolling average of SRS record (metric that factors in opponent difficulty)? Because a solid percentage of games are between teams of extremely different talent levels, especially as tanking becomes more prominent later in the season.

kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
I think there is so much value and room to grow by leveraging a statistical foundation. We’re still iterating really quickly on the low level C code on a variety of applications (pharma, scRNA, text) so it might be a while before we release it standalone.

We do offer an api layer (the website is a light layer above this) over the low level statistics code focused on making it super easy to apply to language data if you are interested in playing around with it: https://docs.sturdystatistics.com

kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
Would love to chat. My email is in my profile if you want to drop me a line.
kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
We built our own collapsed Gibbs sampler in C: PyMC/Stan are use HMC which scales only to a few hundred parameters and we are modeling millions.

Above C we built a python wrapper to help construct arbitrary Dirichlet and Pitman-Yor Processes graphs.

From there we have some python wrappers and store it all in a hierarchical DuckDB schema for fast query access.

The site itself is actually just a light wrapper around our API that simplifies this process.

kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
We do! We have a (very) high level overview focused on applying this model to language on our blog: https://blog.sturdystatistics.com/posts/technology/.

We have some more technical write-ups on the internals of the model that are not hosted publicly (we have some on-going publication efforts applying those model to scRNA sequencing). But feel free to shoot me an email (in my profile) and I'd be happy to send over some of our more technical documents.

kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
That's actually an application we've had a lot of success in. This framework allows you to really easily traverse the graph at a thematic level (with sql filtering if needed), then for any high level theme, you can pull up granular excerpts. This site itself is actually just a thin wrapper over our API (https://docs.sturdystatistics.com/).
kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
So this may have been something worth mentioning above, but the hacker news search is exact match.
kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
Quick update: I ran into a rate limit issue for one of my data sources. Apologies to anyone who has hit errors in the past 15 minutes. I think the issue should be resolved.
kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
I see that the model has not yet finished training: I think you are referring to the "Raw Search Results Section".

Our tool works a little different than LLM style tools. We are doing a bulk search — for academic search, ~1000 papers — and then training a hierarchical Bayesian model to organize the results. Once the model trains, it provides a visual representation of the high level themes that you can then use to explore the results.

The trade off is we are willing to lower the relevance filter to enable a broad set of exploration.

kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
Awesome so glad the result were helpful! What's cool is because it's built on hierarchical Bayesian sampling, it is extremely robust to any input — it just kinda works.
kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
The URL is unique to your search and saves it's state!

In the technical notes I sort of laid out our model graph on the document branch. We also have a topic branch that is also structured hierarchically: Uniform Prior → High Level Topic Word → Granular Topics → Document Lever Variation in Topics. We just directly visualize that hierarchical representation in the sunburst.

The low level model graph is all written in C and exports granular annotations of the model graph. We use the model output to annotate the original text data. We do some work to store these hierarchical results in a SQL queryable format in DuckDB.

What's cool about this process is it's all annotation based. You can query data at the topic level, analyze topics and sql, and at any point pull up the exact excerpts to which the high level data refers.

Curious what you've been using it to search for?

kianN··on Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
Some statistical notes for those interested:

Under the hood, this model resembles LDA, but replaces its Dirichlet priors with Pitman–Yor Processes (PYPs), which better capture the power-law behavior of word distributions. It also supports arbitrary hierarchical priors, allowing metadata-aware modeling.

For example, in an earnings-transcript corpus, a typical LDA might have a flat structure: Prior → Document

Our model instead uses a hierarchical graph: Uniform Prior → Global Topics → Ticker → Quarter → Paragraph

This hierarchical structure, combined with the PYP statistics, consistently yields more coherent and fine-grained topic structures than standard LDA does. There’s also a “fast mode” that collapses some hierarchy levels for quicker runs; it’s a handy option if you’re curious to see the impact hierarchy has on the model results (or in a rush).

kianN··on Microsoft to Get 27% of OpenAI, Access to AI Models Until 2032
Additional link from Reuters: https://www.reuters.com/business/microsoft-openai-reach-new-...
kianN··on Spotting AI Articles
It’s a pretty complex topic with a lot of variance. I think an accumulation of detailed examples is the first step towards empirical data.
kianN··on OpenAI researcher announced GPT-5 math breakthrough that never happened
Haha that’s embarrassing! The progress bars are an estimate. If a paper has a lot of citations, it may take a bit longer than the duration of the bars but it will hopefully finish relatively soon!

Edit: Got home and checked the error logs. There was a very long search query with no results. Bug on my end to not return an error in that case.

If you were hoping to use the citation network, it needs the url as input rather than the title.

kianN··on OpenAI researcher announced GPT-5 math breakthrough that never happened
If you’re interested in a literature review tool, I built a public one for some friends in grad school that uses hierarchical mixture models to organize bulk searches and citation networks.

Example: https://platform.sturdystatistics.com/deepdive?search_type=e...

kianN··on AI Keeps Shifting Right: Coping with the Limitations of Large Language Models
Hi author here. The goal of this post was to do a data driven analysis on the investment trends of companies that are leveraging LLMs. I used YC company descriptions to get a granular sense of the use cases in which LLMs are gaining and losing momentum. I used YC as a proxy for the VC industry. I do question how representative it is of the VC industry as a whole but my impression is it's a good leading indicator.
kianN··on Introduction to Multi-Armed Bandits (2019)
Haha I’ve actually never heard of that field. My work was focused on applying Chinese restaurant process models to text analysis. But very curious what you were working on?
kianN··on Introduction to Multi-Armed Bandits (2019)
I’ve actually run into the exact same issue. At the time we similarly had to scrap bandits. Since then I’ve had the opportunity to do a fair amount of research into hierarchical dirichelete processes in an unrelated field.

On a random day, a light went off in my head that hierarchy perfectly addresses the stratification vs aggregation problems that arise in bandits. Unfortunately I’ve never had a chance to apply this (and thus see the issues) in a relevant setting since.

kianN··on Reducing Data Lake Storage by 52% with SciPy-Style Sparse Arrays in DuckDB
Hi all, author here. I originally was just planning to put together a few helper functions to support sparse arrays in DuckDB, but there ended up being a lot of interesting lessons in the process of benchmarking storage and optimizing queries.

For those not wanting to read the whole story, GitHub repo with basic benchmarks and utilities is here: https://github.com/Sturdy-Statistics/duckdb-sparse-array-lis...

kianN··on Bayesian Data Analysis, Third edition (2013) [pdf]
My applications have focused on noisy, high dimensional small datasets in which it is either very expensive or impossible to get more data.

One example is rare class prediction on long form text data eg phone calls, podcasts, transcripts. Other networks including neural networks and LLMs are either not flexible enough or require far too much data to achieve the necessary performance. Structured hierarchical modeling is the balance between those two extremes.

Another example is in genomic analysis. Similarly high dimensional, noisy, low data. Additionally, you don’t actually care about the predictions, you want to understand what genes or sets of genes are driving phenotypic behaviors.

I’d be happy to go into more depth via email or chat if this is something you are interested in (on my profile).

Some useful reads

[1] https://sturdystatistics.com/articles/text-classification

[2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5028368/

kianN··on Bayesian Data Analysis, Third edition (2013) [pdf]
Totally agree and great point that hierarchical models have been around for a long time; however, these were primarily analytical, leveraging conjugate priors or requiring pretty extensive integration.

I would say his work with Stan and his writings, along with theorists like Radford Neal, really opened the door to a computational approach to hierarchical modeling. And I think this is a meaningfully different field.

kianN··on Bayesian Data Analysis, Third edition (2013) [pdf]
I might checkout Statistical Rethinking given how frequently it is being recommended!

Edit: Haha I just found the textbook and I’m remembering now that I actually worked through sections of it back when I was working through BDA several years back.

kianN··on Bayesian Data Analysis, Third edition (2013) [pdf]
I don’t mean for the bar to sound too high. I think working through khan academy’s full probability, calculus and linear algebra courses would give you a strong foundation. I worked through this book having just completed the equivalent courses in college.

It’s just a relatively dense book. There’s some other really good suggestions in this thread, most of which I’ve heard good things about. If you have a background in programming, I’d suggest Bayesian Methods for Hackers as a really good starting point. But you can also definitely tackle this book head on, and it will be very rewarding.

← PreviousPage 2 of 4Next →