72 karma · joined January 20, 2012
Currently working on making the US government work better and document digitization.
You can email me at christianmoscardi@gmail.com - always happy to talk!
"[per Treil, LADW is for] a student who, while not yet very familiar with abstract reasoning, is willing to study more rigorous mathematics than what is presented in a “cookbook style” calculus type course."
But yeah it's really attempting to introduce you to higher mathematics rather than get you comfortable doing linear algebra per se.
Like I said, I'm essentially continually prompting to refine the material. LLMs certainly continue to append, and never cut back. It just keeps spitting out additional content at me. So that's a bit annoying too. But I can basically get figure out what's going on with a few extra promps.
If youre curious what i've got so far... just be warned it is quite literally AI slop plus me continually prompting for clarification/cleanup etc. : https://github.com/cmoscardi/ai-for-ai
I pretty regularly work with social science researchers who have a need for something like this... will keep it in mind. For a bit we thought of setting something like this up within the Census Bureau, in fact. I have some stories about routing engines from my time there...
People from the federal civic tech nexus started them up over the past decade as they termed out of 18F, USDS, and PIF
The approach I share here (fine-tuning recent deep learning models) is the first one that's gotten me anything resembling high-quality OCR on these particular noisy historical documents. OCRing these has been something of a white whale for me for several years (except, a white whale that I have spent comparatively little time on).
At this point I think I am reasonably competent in OCR, but no expert... Curious for your thoughts.
1. I believe the IR jargon for getting a JSON of this form is Key Information Extraction (KIE). MS has an out-of-the-box model for this. I just tried the screenshot and it did a pretty good (but not perfect) job. It didn't get every form field, but most. MS sort-of has a flow for fine-tuning, but it really leaves a lot to be desired IMO. Curious if this would be "good enough" to satisfy the use case.
2. In terms of just OCR (i.e. getting the text/numeric strings correct), MS is known to be the best on typed text at the moment [1]. Handwriting is a different beast... but it looks like MS is doing a very good job there (and SOTA on handwriting is very good). In particular, it got all the numbers in that screenshot correct.
If you want to see the results from MS on the screenshot in this blog post, here's the entire JSON blob. A bit of a behemoth but the key/value stuff is in there: https://gist.github.com/cmoscardi/8c376094181451a49f0c62406e...
[1] https://mindee.github.io/doctr/latest/using_doctr/using_mode...
Using the models underlying a library like this, there's maybe room for fine-tuning as well if you have a set of documents with specific semantic boundaries that current approaches don't capture. (And you spend an hour drawing bounding boxes to make that happen).
5 minute deck: https://github.com/codingitforward/cdfdemoday2018/blob/maste...
Feel free to shoot me a message.
This confused me as well, since I've always used the normal central-limit-based confidence interval for binary outcomes.
[1] https://en.wikipedia.org/wiki/Binomial_proportion_confidence...
(I am no expert in the analytic underpinnings of the beta distribution or precisely how it is the conjugate prior to the binomial -- or, rigorously speaking, what conjugate prior means -- but the formula here lines up with his formula :P )
I'd love for it to be made obsolete though :)
This site gives me an interface to say, "oh yeah, I /do/ need VLC on this laptop, come to think of it", rather than having to lazy-evaluate that when I have a video downloaded and realize I don't currently have a means of playing it.
I spend a fair bit of time doing DevOps work, and this isn't meant to be some sort of configuration management or provisioning tool. This is for desktop / dev users, which is, I think, quite a different use case (but an increasingly relevant one). I have a dotfiles repo, which is to say I'm familiar with the "document things in scripts and repos" game - but it's not clear to me that something like a script-as-documentation is even a useful process to go through when any given computer I set up is going to be used differently. That laptop playing VLC is very different than the web dev box I spin up to mess around with a new project.
The motivation was basically a version of ninite[1] for Linux. So very desktop-focused. I actually got in touch with the ninite folks, who used to run a Linux installer [2]. They said there wasn't enough interest and package managers are "good enough." I respectfully disagree on that point - I think that having the ability to point-and-click for the core, most important packages you want to install is really useful when you're spinning up a new desktop from scratch.
I was running into this problem after basically doing ad-hoc setup for every new machine I turned on. Sure - I could write a shell script / list of packages and save it to my dotfiles repo, but where would the fun be in that? :)
(Plus, package lists on every machine are different)
In particular, the Muen Kernel report itself [1] explains:
By implementing the kernel in SPARK and proving the absence of runtime errors, we have shown that the kernel is free from exceptions. While these proofs provide some evidence to the correctness claim of the implementation, the application of these particular formal methods do not provide any assurances beyond the error free execution of the kernel. Proving functional properties such as the correspondence of the scheduler to a given formal specification is necessary to further raise the confidence in systems based on the Muen kernel.
In other-words, we don't yet have formal confirmation that this thing actually does what we might expect it to - just that its execution is bug-free.
I think in this instance, ethics and economic incentives align - your expertise is part of your value-prop to clients, and having a good grasp of your limits makes you substantially more valuable. You get things done on-time, with clear expectations set, and do a prompt and thorough job providing what you're being paid for.
Personally, as a natural optimist, one thing I need to be especially careful about is not saying "yeah that's doable!" and setting expectations too high - it can potentially lead to a messy situation. That said, one of my favorite parts of freelancing is that I get confronted with new, unique challenges that I've never dealt with before - there's a balance to be struck here.
Yeah, most of our curriculum uses regular old matplotlib. We only use this technique specifically in our D3 curriculum. In general I can't think of many use cases where this is actually better than just having a couple of files, aside from instructional ones.
Then, there are IPython Notebooks, which give you the HTML, cell-based frontend for executing code.
Jupyter is the v2/generalization for both of these things - there is the Jupyter shell, which is a plugin-friendly REPL, and Jupyter notebooks, which allow you to run code across a variety of langauges. AFAIK so far, you can only run one language per Jupyter notebook (though having dug around the source code, the possibility for multiple languages on a per-cell basis is very much there).
I still find myself using IPython and Jupyter interchangably, which probably doesn't help the confusion.
In terms of who uses this stuff: IPython notebooks are very popular across academia and for lectures/talks. Check out all the cool learning material here: https://github.com/ipython/ipython/wiki/A-gallery-of-interes...
Literally their first example is embedding D3...
http://sharing.beakernotebook.com/gist/anonymous/e21582541d7...
then again, i'm just trying to become an all-around better fullstack developer for my current employment situation - so i'm mostly using (and pull-requesting) frameworks rather than writing my own
given this perspective, i'm much more interested in the entire [and rapidly expanding] toolchain for modern js development - and learning the language's esoterica [and i am the sort of person who takes pleasure that stuff] as i become an all-around better js dev
That said, if you're in the market and inexperienced [or maybe even experienced], take a prospective car to a professional mechanic. Don't buy a junk car and expect to be able to fix everything yourself, unless you want a fun side project which will require a ton of man-hours [and more money than you'd expect] to complete. But on the overall, doing your own maintenance and fixing smaller problems you encounter [timing belt issues? no biggie!] is very very enjoyable.