Hacker News Reader

The problems I've seen with tools that try to load everything once and monitor for incremental changes -- is it's hard to get RIGHT. These tools tend to be buggy, or worse tell you it's not them that's buggy, it's you that's "doing it wrong" somehow, trying to work the way you want.

The beauty of simple tools is it's easier to make simple tools that work really really well. From loading everything at once, to considering text editors vs "IDE", which is really part of what you're talking about too. When we have new languages and platforms all the time these days... it's important to be able to get high-quality tools that don't take forever to develop.

Sure, simple tools are great when simple is what you need. But here they took a simple approach to a problem that's essentially complex, which means they lost a user.

This is an important point for tools, like type checkers, that are supposed to help deal with complexity. Type checking doesn't really matter in a small code base. The bigger you get, the more it pays off. But their type checking tool gets worse with the code base size. This is self-limiting in a way that can be fatal. If your tool doesn't matter for small code bases and it can't be used on large ones, that's a problem. It's very hard to know in advance that one's project will grow to a medium size and then stick there.

The problem is that our editors are at the core still "text" editors. That forces the tools (IDEs) to continuously synchronize the text with the parsed AST.

It's trivial to get an editor right if it edits the AST directly. You skip the whole tokenization/parsing step which is where most complexity is. However, such an editor will need specific plugins for each language, and these plugins will be easier to write for languages which have a standardized AST (Haskell comes to mind). Also, users would have to give up part of the formatting (whitespace etc), and some people are very religious about that :(

The counterpoint to this is that if you're not using text editors then you lose the universal property that most source code is plain text which can be edited with your preferred text editor. Plain text seems to be the most tool-agnostic representation, which for many people is a big plus. Notice that in this day and age there are still people complaining about "bloated" IDEs...

I think there's a useful distinction between serialization format and working format (that is, what it looks like in RAM). For a plain text editor, they're the same. But that's pretty unusual.

There's reason we couldn't have an editor that works with the AST while saving everything out in plain text for interchange purposes.

True. But the property you mention -- that for plain text editors the serialization and working formats are the same -- is actually a pretty useful and convenient property! I think this is why it's extremely unlikely there will ever be widespread adoption of a different format for source code. Niche tools will remain niche forever, in my opinion.

I think we wouldn't even have to give up whitespace, comments, and other things that get left out of ASTs. I think the problem there is that ASTs are normally thought of a one-way, compiler-friendly representation. But if we start thinking of them as a round-trip representation that is also friendly to programmers, it doesn't seem impossible to include comments and formatting as hints in the AST that compilers and interpreters ignore. Sort of the same way that many HTML and XML parsers keep track of comments and whitespace even though that generally gets ignored by code that uses the parse trees.

It's "trivial" once you've done a bunch of non-trivial work to make it trivial. :)

Facebook's Flow and Hack typecheckers both do this.

If anybody has used these for serious coding, I'd love to hear more about the experience.

We've been using Flow at Streak. Compared to my last experience with TypeScript, it's been really great for how easy it is to add into an existing codebase. Nothing about our build process had to change because we were already using Babel (which by default strips out Flow types and supports JSX) with Browserify. Flow understands node-isms and follows how commonjs and es6 modules include each other, and only enforces its type checking on files that have opted in with the magic /* @flow */ comment, so we didn't have to stop the world and make everything play nice with it in one go. We're gradually opting in more of our codebase over time. Its type system maps to javascript really well, supporting things like optional types, disjoint unions, intersection types, and polymorphic functions and classes.

The annoying things with Flow: No Windows support yet which is the one thing holding me back from transitioning some personal projects over, but it sounds like progress is coming there[1]. There doesn't yet seem to be any real plans for letting 3rd party NPM modules bundle their own type definitions and have them be automatically used[2]. (Flow doesn't have issues with untyped imports at least. They're just treated as the any type.) Editor integration is pretty bad still. There's an Atom plugin, but it's buggy. This doesn't affect me too much as the "flow" command for type checking is quick to use (on the first run, it starts a long-running daemon in the background which watches your whole project for changes and keeps their type information cached) in the terminal I already always have open for interacting with git.

[1]: https://github.com/facebook/flow/issues/6 [2]: https://github.com/facebook/flow/issues/593

Yeah, although I don't know how parallel it is because OCaml. I think they might have gotten a multiprocess architecture working?

"Please god somebody build tools that take advantage of that."

Like, an IDE?

An IDE's a good start. But IDEs tend towards monolithic, which can mean relatively low adoption. I'd much rather see independent tools that can be plugged into IDEs. E.g., in this case, I'd rather that the type checker ran as a daemon. In its base mode, it would just produce text output. But it would also have some IDE-friendly protocol so that it could be easily bundled.

Yeah some tools are kind of like what you describe... for example, FindBugs is a great lint-type tool for Java, that you can run standalone from command line (ie as a part of a build, or CI), but also has plugins for all major Java IDEs. I have it set to run every time I save a file (on that file only) and here is a kick, it does not run the command-line version, it just runs within the IDE so it is fast enough to be imperceptible.

Do any IDEs support large (million+ LOC) code bases?

I guess they do? Eclipse itself is many millions of lines of code, and far as I can tell, it seems that the prescribed way to develop for it, is to use Eclipse itself.

I mean, if you search for a string, Eclipse comes back quick because it indexed all your files. Run a grep? It comes back 6X slower (on my adhoc test)

"iteration speed being critical ... But it's something I rarely see featured in tool design."

Go? They've always had this as a top level goal.

I'm not very familiar with Go, but from what I understand Go is the antithesis of this feature in many ways. AFAIK the tools are all stand-alone command line applications which you have to run every time you want to check whether you're on the right track. Go tools which generate code are an even worse example. If your language good abstraction capabilities your development environment could understand your intentions much better than with generated code.

Go does not take the suggested approach, but certainly seems to prioritize iteration speed by making each step of the staged approach fast.

> AFAIK the tools are all stand-alone command line applications which you have to run every time you want to check whether you're on the right track.

I really don't understand this distinction. My tools run those commands for me. I type in my editor, and it tells me if the compilation fails. My editor is also far more responsive and uses far less ram than the typical java IDE, so it feels faster to me, not slower.