Nvidia – open-source auto-ontology
github.com
github.com
"Auto Ontology" is sort of a poor name.
Here's the problem:
Your company has data scattered across multiple types of databases. People who make decisions want to ask questions about that data.
The old way is they would go to a Data Analyst, and say, "Get me a report on so and so", and it was the Analyst's job to iron out the right series of queries and one-off scripts to generate the needed info.
With a system like this, the idea is that all your database models have been introspected and stored in a central metadata catalogue.
When a question is asked like "What were the top selling items in the month of May?", the metadata catalogue tells you "Products are stored in FOO_DB" and "Product_Sales are stored in BAR DB" and so you go and fetch the right data from the right place and do whatever in-memory calculations you need to do.
What the system provides is the implementation code for:
- Introspecting each of your DB's
- Aggregating model + metadata information in a central, virtual DB (VDB)
- Generating dialect-specific SQL from natural language queries and routing them to the right DB for the underlying table
- Combining results in-memory when a natural language query requires multiple datasources
- Handling proper permissions, etc
A GOOD system for doing this also integrates with your external API sources (OpenAPI, GraphQL, gRPC, whatever) and also documentation sources (cough, I work on this, cough).
I tried to make it through the README and it was just unmotivated slop. Perhaps it reads better to agents. It is like the Van Halen rider story - if the authors didn't care to write a good README, what are the odds that they paid attention to the rest of the large codebase?
https://github.com/NVIDIA/auto-ontology/blob/main/auto_ontol...
> The problem it’s trying to solve is that generating valid SQL doesn’t necessarily produce the right answer. An AI might find a column called revenue, but still choose the wrong dataset or misunderstand whether that figure includes things like refunds. Auto Ontology adds information about what the data means to help it make those decisions.
and then the one paragraph summary at the end:
> My reading: it’s a database chat application built around a reusable catalogue of business meaning. The question worth testing is whether that catalogue makes answers measurably more correct—and how much human work it takes to get the definitions right.
If they can't do that, it's a signal they didn't care enough about the details to rely on this project, and there is more cool software than ever I can spend my finite time on.
That may not be particular fair, and it may be a great project, but the bar is just much higher now.
The fact that I had to throw an LLM at it to cut down the slop and turn it into something more succinct and comprehensible is nuts. For myself, I'd be embarrassed if I released something with that kind of README on it.
These are optional but it allows manual override when schema design is poor
Example is a transaction application with a backing database: The users conduct information interactions with the application. The application stores relatively sparse data into the transactional database. The database has just enough metadata to route data from transactions but doesn't have the full meaning.
Example database column "updated_date" with type timestamp: Without looking at the data, how do you know if this column is populated when the application first stores data in a row? How do you know the timezone of the timestamp as a user would see it?
Real world example: Patient height in app and api is SI and both note the units. Patient height in the data warehouse is US Customary with no unit notation. The app and api make elaborate conversions for the underlying transaction database's US customary storage.
I think the promise of an LLM ontology is to read application code to understand the context for any particular database column. The linked software is not that.
Something like autoontology would help with human enforced governance for "blessed" data sets. You still need the operational rigor and data maturity to build said data sets and drive the transformation to adopt them.
This is effectively a tooling and labeling layer. But it will not magically solve the messy people issues that have always been at the root of this.
I guess it all comes back to the fact that they don't have a clear division between data they have learned about the world, instructions and data under analysis.
You can ask them only to consider the 5 facts in front of them, but they won't do that, and cannot do that.
Depending on how you combine imperfect systems, you can lower or increase error rates.
> data they have learned about the world, instructions and data under analysis.
I don't think there really is a clear distinction between these things though.
And I don't mean this in a gotcha way - these things apply to people too. Not that that means they're like people, but we have successfully built enormous systems out of fallible people, and other imperfect things.
The goal is rarely "eliminate all errors" but "have a thing that's useful, given a possibility of things going wrong".
Well, in principle yes one system can compensate for another, in practice, errors tend to compound and combine in interesting ways, not go away, particularly in systems like this which are unpredictable and generate slightly different responses each time without human intervention. They are not reliable in the way we're used to considering and don't work well without supervision from careful humans at present.
I don't think there really is a clear distinction between these things though.
Yes your second point is an interesting one - the divisions between these things are arbitrarily imposed on the world, but we do have these distinctions for a reason, even if they are not completely objective.
I do think this is a significant problem specific to LLMs though (that I at least see when using them) - we don't have a way to constrain them in what knowledge they consider. This can be a bonus (they are good at connecting faraway bits of maths in interesting ways), but it can be a significant problem when asking for a summary of a data - confabulations and data from outside the context easily leak into the output of the LLM. That is undesirable behaviour and we have no reliable way to constrain it.
I'm not sure that this particular effort is going to help because it doesn't change or constrain the behaviour of the actual machines and the behaviour is the problem, not the data they are fed.
Ah but the point here IIUC is moving from "let an LLM figure out the right way of querying the data and what you mean with your terms from a wide span of different documents and sources, which it has to perform multiple searches on" to "let an LLM figure out the right way of querying your data with a coherent and machine readable ontology".
While the conversion from "load of data" to "defined structure" is error prone, you now have the second step operating on the same converted thing, and you have ways of fixing that over time (the corrections).
By splitting the task into two distinct phases, you can both interpret and fix the middle step, and remove a large class of mistakes and/or differences caused by different interpretations of the same documents.
There are really two answers to it IMO that this boils down to:
1. This process may be less error prone for that first step than the LLM you're using. That could reduce errors. 2. This process means the first step is run once rather than every time which means you cut out one area where there can be differences each time.
> Yes your second point is an interesting one - the divisions between these things are arbitrarily imposed on the world, but we do have these distinctions for a reason, even if they are not completely objective.
You're right, I think there are useful groupings, but I think any effort to fully isolate them is doomed to fail because they inherently blend. Your instructions contain things that can only be understood in context of world knowledge and the data. The data only makes sense with certain bits of world knowledge.
We also know that these things happen with humans (again just to pre-empt how others may read this and not saying you are accusing me of this, I'm not gotcha-ing here) and so we often have to be careful about how we ask questions, how different cultural backgrounds impact understanding and so forth. We have to take care about how we structure things.
> I'm not sure that this particular effort is going to help because it doesn't change or constrain the behaviour of the actual machines and the behaviour is the problem, not the data they are fed.
I think this effort does address that. A really key thing is that this is about turning human readable questions into SQL that then only query the actual data and you see the result of that query - it allows you to constrain things more (at both points).
Computer systems, though, can have that distinction. If you read data supplied by the user, and then process it via behavior in code, and process that result against administratively-set data in the database, the separation between user-supplied data, administratively-configured data, and instructions (code) is extremely clear. Even if you are doing something that could be nondeterministic, e.g. incorporating randomness or doing sentiment analysis or whatnot.
I don't think the oft-repeated "AI is fallible just like humans are" applies here. Computer systems are an area where that separation can be achieved. We know this because we've achieved it a lot. LLMs, powerful as they are, are a massive step back in that regard. Their benefits might outweigh that compromise, but it is, as they say, "worth naming" anyway.
No it is not.
They are three places you can put things, amongst an infinite other number of places you can, but they are not some inherent core distinctions.
Is a javascript file code or data? If I read it to measure file size and don't execute it, is it ever code? If I read in a file and do different things in the code based on it, and never alter that file, how is that any different from having if (x) baked into the code? Why is it user supplied data if we read in a file and do things based on the contents yet code when I read in js and execute stuff in a VM?
They are just different places with different access controls that alter the outcome. There's no magical distinction between code, data and admin data or anything else other than really who gets to change what. And then there's obviously not just three named buckets of "stuff", it's as many as you want.
> I don't think the oft-repeated "AI is fallible just like humans are" applies here.
You're missing the point here, the point is we have built big things with fallible components. Therefore the idea that a component is fallible does not just from that fact mean we can't do useful things with it. And it's a very relevant point if we're looking at "where can we use this in place of humans" - if we had a human there before we were not starting from a perfectly reliable point.
Has training them to do it been tried and proven to fail?
I saw this same mentality from a lot of people every year for the last 4 years. All the limitations they clung to keep falling. We're not limited to LLMs either, and LLMs themselves aren't as limited as people have been imagining.
I think 2027 is the last year where "but AI can't do X!" is a viable mentality. Like, yes, as a casual user there are limitations that affect us in day to day use and the surface level concerns are what impact people most today. But at some point there must be a threshold at which even the casual users start to get a little curious what's likely to be possible in the limit.
Happy to admit relative ignorance of the internals and hear a more informed opinion, but it is not convincing to hear that they'll inevitably solve all the problems and we'll live in the best of all possible worlds next year, without hearing a proposal as to how they'd do that.
These machines are trained to be convincing and addictive, very similar to our social media platforms now, that leads to a lot of over-optimism about what they might be able to do one day, with the next frontier model.
I think the 'reliability add' is from how it's presented back to the user. If you can trust the agent to at least say what tool it called, then you can mark a chart as being the canonical output of "get_projected_profit" tool vs "get_realised_profit" tool and everyone knows it hasn't come down to the current dice roll of randomness what the agent itself did this time.
You can give it instructions and 5k tokens later it will do exact opposite you told it to.
i want to use these semantic / KG tech but i find them so confusing!
it's local first, agent-friendly tooling; you have single cli to access it (mcp is there as well if that's your thing)
everything is persisted in simple local text files that you can actually read (or sqlite database if you don't care / have more data)
ie there is no integration, k8s-style nonsense etc, you can just use it on project or on your system user level and know exactly what's there, you have control over it, you can delete it, move it, extract parts you find useful and move on etc
because it has first class support for uncertainty
evidence is first class concern
supports evolution of belief (statements can be revised, retracted etc. without destroying history)
knowledge is executable (rules, actions, automations – all operate on the same claim model)
it works well for knowledge that keeps changing (including support for querying knowledge as of some past time)
it's great for llm sessions, ie. simple things like comment on a claim being part of a claim itself, which means it appears when working with knowledge, which allows to support weird/human/natural-language aspects that are not directly part of the model
...
rdf/owl/sparql is great for large, public, structured datasets (ie. medical, scientific or some regulatory, standardized ontologies)
cave is great for evolving knowledge, ie. in most cases i use it by just dropping it in and forgetting about it – i don't care how/what llm will model, i just care it'll solve some problems faster when aiding itself with it as a tool; if i want some structured dataset out of it i may occasionally ask it to model it around homogeneous schema i can extract, but in most cases i really don't care
But you are putting broad but is more opinionated design structure around it. I will have a look thanks.
I intended to share Wikidata as an example of an ontology that can be reviewed and maintained, rather than an example of an LLM maintained ontology.
I did not do a good job of communicating this, so appreciate the clarification!
reminds me of http://www.thai-language.com/ (it's safe). it was around long before Google Translate got good enough. i don't know the authors/maintainers, but afaik, everything is entered manually. this is the "analytical" approach to language. most translations before Google Translate went full ML used an analytical approach.
anyway, long story short, turns out "analyzing" a language to "understand" it GREATLY underperforms probabilistic/statistical methods like ML.
also reminds me of trying to label parts of speech in Vietnamese. there's a school of thought that finds it useful...then there's another school of thought that plainly points out words that can be verbs, nouns, adjectives all in different contexts.
- you need some engine to support your owl/rdf/sparql, and all available are lower quality/perf/features than pg
- its easier for LLMs to talk standard SQL compared so some version of sparql
I’ve used AWS Neptune as a pretty reasonable managed sparql query engine but for most my needs actually just local jsonld mastered in git is plenty for almost APO knowledge graphs I have.
I’ve found LLMs are exceptionally proficient at authoring sparql, and quite clever. This isn’t surprising given there’s plenty of sparql in the corpus, and it’s semantics aren’t HUGELY different than SQL. Likewise they do fine writing datalog etc.
it is obviously not OSS like postgres.
There’s also a lot of books on how to develop ontologies as well as online resources. They are all in the corpus and tasking a frontier pro level model with building a process and mental model document with references inline for grounding works really well as well for priming.
SQL > Cypher > SPARQL
You can do RDF/SHACL/OWL with LPG/Cypher based databases.
Ontolite (sqlite) and ontobug (ladybug) are examples.
How about we start by open-sourcing the NVIDIA user-space libraries: libcuda, libcudart, libOpenCL, libnvtx, as well as nvidia-smi and the CUDA and OpenCL profilers.
In software, an ontology is just a machine-readable definition of the entities in a system - their names, properties, relationships etc. This tool analyzes a system and produces such a definition for agents to use.
Imagine if you will, 10 groups inside a broader org. Each of those groups have teams that work on different stuff, but at the org level and cross team level managers and leadership still need to make sense of it all and if you ever want to connect the dots across work you need a common language.
Now multiply that problem across a large enterprise if you want a company brain or any of the context to be trustworthy for usage at scale or for high accuracy and trust work.
This is a a tooling solution for the schema and government layer that enables that so you can have an empowered human say "what team X calls Foo, and what team Y call Bar should map to our approved definition for Baz and here is how we validate their data against that shared classification."
An ontology in this kind of context means a more precise machine readable spec of this and how it all relates to each other. You can totally build one by hand. The idea here is I think that it tries to figure it out itself and can take corrections over time, so you can ask natural language questions and get more reliable results rather than hoping an llm works out what "customers this month" means the same way each time over a wide span of data rather than having a much clearer API to work against.
2. Related to 1., your codebase or source material may hold different interpretations of the same definition (what "revenue" means can differ between Sales and Finance). The ontology forces you to choose one, or to state when to use each. Without that, the LLM may pick one a bit randomly.
3. It's a governance tool for humans. It gives the company a human-readable, "deterministic" source of truth to agree on what means what, then version it and evolve it.
Before: domain expert wants to ask a question about the data, but which of these hundred tables actually has the right “revenue per User” column? Ask a DE to help slice the data and bless the queries.
After: ask your agent. The agent looks at the ontology and can usually find the best blessed data. To enable this you need to write down all the tribal knowledge that lived in the DE/DS head; agents can take the first pass at proposing and updating metadata.
Under the covers the DE and DS roles switch to reviewing the underlying ontology, instead of working directly with business stakeholders on one-off queries.
It looks a lot like English, but it doesn't have the same rhythm human English has. It's as if a human tried and failed to be concise.
Seriously though, it'd be nice if we started using a more shared and less vendor-specific jargon to refer to what these latest features are. We're getting into JVM vs. .NET Execution Engine distinctions and they just muddy the water on what goes on in the market.
Now there are times when I let the model keep its own docs in its own language, for its own future reference; but, it is not intended for human distribution due to its odd selection of words and phrases (aka slop).
Or: Agentic analytics is gated on sufficient reliabilty.
Or reallllly short and hard to understand: Reliability gates agentic analytics.
And nvidia probably doesn’t want to say “probabilistic responses including hallucinations preclude reliable ai analytics”
As it is the sentence communicates very little. Things are unreliable for a lot of different reasons. Hardware? Software? Users? Intentional and required use of RNGs? Claude presumes the reader has all the same weights activated.
People are really seeing 'AI' in everything now. You can't tell.
Humans are now hallucinating what is, or is not, AI.
People are "gatekeeping" how we use the word "gate".
I am convinced this is a result of RLVR and synthetic data. LLMs optimize not for human readers, but for passing the LLM evaluators in their training runs, and end up developing sophistry.
The game I was using for this uses a standard memo pads mechanic to fill in the story and many were barely stumbling through it in their own native language.
So it's my impression that this is not entirely Claude's fault.
Conservatively, I'll blame it on "dyslexia".
> If a tool is worth building and sharing, surely it’s also worth writing a brief summary of what it’s supposed to achieve in your own words?
Next the complaint will be any such description looks like AI slop.
It's quite common for LLMs to mix up values between columns in a table. It's often referred to as a structural hallucination, since it can happen with all kinds of relationships that form structures, not just columns in a table.
> Agents can retrieve documents and query databases, but reliability gates agentic analytics: a revenue metric answered faster is worth nothing if the number is wrong.
Absolutely no human would ever write this kind of slop. I have no idea what this even means. Is this really the first thing you want users to read?
I wish people would take a fucking minute to at least figure out what their main selling point is and just write it, you know, like we used to.
Just adding a few words takes it so far...
All the four nodes on the right side have either only incoming or only outgoing arrows.
What? now I can't draw boxes and arrows without it being slop?
Maybe typical marketing power points are worthless. But I called it slop because it didn't help me at all understanding what they are doing.
this reads like an attempt at poetry
Nvidia, I get you're all-in on AI, but please at least find a human to write some copy. It's both overly verbose and entirely lacking of important info at the same time.