HNHacker News
TopNewBestAskShowJobs

kncvetko

4 karma · joined April 11, 2020

submissionscomments
kncvetko··on Is Jev the general-purpose classifier we've been waiting for?
Is Jev the general-purpose classifier we’ve been waiting for?

That would be the thing that could create a real dent in the universe: a classifier that can classify without first collecting a massive statistical dataset and training it with whatever algorithm fits best; be it BERT or good old logistic regression.

Long story short: no.

The more nuanced answer is: it depends.

Over the years, we’ve built classification systems, expert systems, and scoring models for credit scoring, financial services, and tax tech under German regulation. So we decided to run a smoke test with Jev on what looks, at first glance, like a fairly simple problem: bank transaction classification.

This sits at the foundation of accounting and almost every downstream financial workflow.

At first, it looks trivial. A transaction comes in, you look at the counterparty, purpose, amount, maybe a few metadata fields, and assign a category.

In practice, it gets difficult very quickly.

The same merchant can belong to different categories depending on context. A transfer can be revenue, a reimbursement, an intercompany transfer, tax, payroll, or simply a reconciliation entry. The correct classification can depend on the company, its chart of accounts, historical behavior, VAT treatment, or what happened before and after the transaction. Buying milk means something very different for a restaurant than it does for a physician.

So while bank transaction classification looks like a simple benchmark, it is actually a pretty good test of whether a general-purpose classifier can deal with ambiguity, context, and domain-specific constraints.

For our smoke test, we tried three things.

First, we used google/gemma-2-2b-it as a baseline, running on a consumer Apple laptop with MLX as the platform against ~1,000 real-world transactions.

The result was pretty much what we expected: fast, cheap, but with 17.5% accuracy a terrible quality.

Then we adapted the same google/gemma-2-2b-it model using ~4,000 real-world transactions as the training set and the ~1,000 transactions as the validation set.

The results were as expected: slow, cheap, but at 85.3% accuracy, still not accurate enough for this use case.

Then we tried Jev.

The results were sobering: fast, cheap, but with 40.7% accuracy it's basically useless for this use case.

And that is the problem for us.

The main value proposition we see in Jev is classification without a human in the loop and without having to collect data and fine-tune a model first. In our business context, our smoke test didn't get us there.

None of the approaches is good enough to solve a hard problem in a real-world business without a human in the loop.

Of course, we have very specific use cases. There are probably plenty of classification problems where world knowledge is enough and Jev works well. Just not for the kind of problems we have to solve. For those, the boring old lesson still seems to apply: you need data, you need context, and the algorithm is probably the least of your problems.

We’re looking forward to seeing where Jev goes from here, especially around adaptation and fine-tuning. There is definitely a need for a better developer experience and for classifiers that are faster and cheaper than misusing a general-purpose LLM for a classification problem.

kncvetko··on Apple Google Contact Tracing API – need for interoperability and standardisation
Author here:

Uploading diagnosis keys to health authorities backends of each realm you‘ve visited is unfortunately not a trivial requirement. To prevent fraud and pranksters health authorities would need define a trusted set of apps, which requires a complicated process or architecture. But a trusted set of apps in itself won’t prevent fraud. The critical transaction is the push of keys after a positive diagnosis which needs to be secured extra and the trust propagated to other health authorities. Our approach reduces the complexity by focusing on the securing the push by the proposed signed one time token.

The proposed solution tries to be compatible with the Contact tracing API. In terms of GDPR the sensitive data set could be the diagnosis keys and to some extend the geohashes. However both data sets are aggregated and anonymized so that a contact service backend provider can‘t match it with a specific user. Hence we don‘t see significant limitations for the proposed solution by GDPR in the EU.

The proposed solutions assumes that there won’t global standardized apps. Even the EU struggles to standardize it’s efforts and a frictioned set of tracking apps might be reality. We think that this will limit the effectiveness of tracking as soon as travel restrictions are reduced and shall be addressed in an early stage of development.

We use the SDL capabilities of graphQL to define the required mutations, queries and types as technical but platform agnostic documentation of the proposed interfaces.

kncvetko··on Apple and Google partner on Covid-19 contact tracing technology
As far as I understand the actual risk assessment is done by the Contact Tracing Framework.

I wonder why this design decision was made. The risk assessment will change and the apps done by health organization have the expertise on that subject and shall do the assessment and not the CT framework. It will require an update of the OS to get latest findings published.

Privacy and other technical decisions are sound and legit.

Has somebody some background information on the reasoning driving that design decision?