30 karma · joined May 27, 2016
(I put in a timeline (histogram) of photos, and keyword search, to serve these kinds of needs.)
The major pain point is LSP integration, which has to be closely tied to the build system, since it's only by building that the LSP server can know a file's dependencies. Everything is all neatly available with dune. We've cobbled something together a bit with buck2 but it's not as nice.
I kind of don't want to use DOMParser because it's browser-only... my web-scrapers have to evolve every few years as the underlying web pages change, so I really want CI tests, so it's easiest to have something that works in node.
I figure that a combination of wetware and software is the current sweet spot. My brain usually has enough associations and context to turn every photo search into a time or place filter - "I think it was downtown last year" or "some time in summer at home" or "it had my wife in it". The photo storage system need only provide search/filter on date and place to narrow it down to a few hundred thumbnails, plus machine-learning to tag people. Which is basically what iOS provides, no more, no less.
Any other up-front categorization or tagging is basically wasted effort.
For me, I wrote a bulk tool that renames my photo file names by reverse geocoding the GPS information via Open Street Maps. That way I can do text search for place, as well as 2d map search. It's at https://unto.me
How it started -- I was swimming along the shore of the lake, figured that if I saved nothing for the journey back then the worst that could happen is I run out of energy and have to haul myself into someone's back yard and ask them for help. That wouldn't be too bad, so I might as well. It's amazing how much I can trick my brain+body into going further than it otherwise would.
The MIT license is a license about copyrighted software, allowing people to use/modify/publish that software. But a file format isn't a piece of software.
Are you open-sourcing the specification document for the file format? (people are still free to write software that reads+writes the file format even if the specification document isn't open-sourced).
Are you open-sourcing your particular library for reading the file format? (I'm confused here, because you stressed that the file format was so simple, so I'd have expected it easy and maybe even desirable for many people to come up with libraries for reading+writing the file foramt?)
(1) I wanted something where colleagues could easily share links in workplace chat with each other, so we could cooperatively investigate bugs.
(2) For LSP we're often concerned with responsiveness, and I thought the best way to indicate times when viewing a log is with whitespace gaps between log messages in proportion to their time gap.
(3) For LSP we have lots of interleaved activity going on, and I wanted to have visual "threads" connecting related logs.
(4) As the post and lnav say, interactivity is everything. I tried to take it a step further with (1) javascript, (2) playground-style updates as you type, (3) autocomplete which "learns" what fields are available from structured logs.
My tool runs all in the browser. (I spent effort figuring out how people can distribute it safely and use it for their own confidential logs too). It's fast enough up to about 10k lines of logs.
If you have a tech savvy audience they can also view your thing in an iframe with only sandbox="allow-scripts" to prove that it's not making network requests.
I wrote an html/js log viewer with those security models https://GitHub.com/ljw1004/seaoflogs - it handles up to 10kline log files decently, all locally.
This seems like a solution where rich people get the good things and poor people don't. I wouldn't call that fair.
In a democracy, everyone has an equal weight, an equal say, in the governance of the country. We often call that fair, or sometimes just.
In capitalism, every dollar has an equal weight, an equal say in what gets produced or built or done.
Autocomplete isn't really helped by the AST. Here instead is how it works:
(1) Query the backend "at the specified line+col in the file, what context are we in?"
(2) The context might be a qualified identifier, e.g. "foo.|" or "foo.ba|" or "MyFoo::|", in which case the context will have to tell you the type of the thing to the left of the qualifier. You then make a second request to the backend to ask for a list of members of that type. (This requires you have some common language with the backend for referring to types).
(3) Or, the context might be an unqualified identifier, e.g. "|" or "fo|". In this case you have to assemble a list of all possible identifiers in this position. (You might also want to assemble keywords here, but it's hardly necessary). This list will include all local variables; for this, you can use the AST so long as the semantics of your language aren't too tricky about what variables are in scope, or you might query the backend for them. The list will also include every single top-level symbol in your entire project. (Even in languages which require import or #include statements, you still want to list top-level identifiers that haven't yet been imported, because that's what a nice IDE experience is -- you help the user type this and then help them auto-insert the import).
This fact means that autocomplete has to be powered by the build system, has to know about all files in the project. Also, for large projects, there might be countless top-level identifiers across the project, maybe in the tens of thousands, so autocomplete needs some serious pre-compute assistance.
So how would one achieve any part of the above with an AST? The AST solely helps with part of the first step, allowing you to identify which AST node you're in. Then you still have to query the backend using the exact same API as above to learn the type of the qualifier to the left of the current AST node. Then you still have to query the backend to find all members of that type. And you still have to query the backend for its build-system knowledge of every type in the file.
In other words, the AST has barely helped you at all in your quest to implement autocomplete.