Hacker News Reader

Comment by simonw | Hacker News Reader

I've been playing with some ideas for creating a SQLite database of classes, functions and suchlike found in Python code, so I can analyze my codebases with SQL queries.

I've had some good initial results with https://github.com/davidhalter/jedi - which is the Python introspection library that powers various editor autocomplete implementations. I have a prototype which uses that to create a SQL database of functions, classes and places that they are used.

I've also been playing with https://github.com/github/semantic - it can parse Python, JavaScript and other languages and offers a --json-symbols option which dumps out a JSON object showing the symbols (functions, variables etc) found in the code.

View on HN

Oh this sounds cool! When I was writing this post I was thinking about ASTs and transforming them into domain-specific semantic graphs.

E.g. that `run_config` example would be a generic "method call" node in an AST, but a domain-specific AST crawler could recognize that a "method call node whose name is `run_config`" should be replaced with a semantically-meaningful `run_config` node.

And maybe that would be an interesting way to build up a conceptual graph of a codebase?

I'm with you on this one. I'd actually suggest https://github.com/CoatiSoftware/Sourcetrail could be extended to do this, though I haven't found the time yet for my own codebases. For example https://github.com/CoatiSoftware/SourcetrailPythonIndexer and under the hood the file format is SQLite: https://github.com/CoatiSoftware/SourcetrailDB

Although I understand what we're trying to achieve here is easier code navigation and manipulation on a much broader scale with preferably a single tool, I couldn't shake the feeling that some of the examples you gave have existing solutions.

The `run_config` one stood out the most. Isn't it perfectly feasible to have a `jobs` and a related `run_configs` table, instead of them only being objects? You could even put them in a separate database, much like one does with the auditing tables in a Rails application, or with reporting tables that need to be accessible by the likes of PowerBI. That separate database could have it's own light-weight GUI, something that just plugs your data structure into the interface, like Django's admin back-end.

Sure, but then you'd have the "config" for a job separate from its implementation. If I'm looking at the code for a job, how do I easily find when it's scheduled to run? If I'm looking at the config database, how do I easily get to the implementation of the job?

Putting aside the feasibility of migrating a large codebase that exists one way or the other, my point here is that these shouldn't be separate tools.

do you have a repo? this sounds cool

Reply on news.ycombinator.com