Comment by still_grokking | Hacker News Reader

Comment by still_grokking | Hacker News Reader

Parent Full thread

This is imho a pure tooling problem.

We still use a loosely representation (text) for a semantically very rich data format (code). When you look at this objectively that's madness.

We should finally use proper editors for our semantically rich data format instead of "dumb" text editors. (Which are by now actually not so dumb, and which do again completely crazy things to re-extract the semantic information from a representation that inherently erased this information while we're used a loosely text representation).

We should imho eventually switch to "binary source code" and start using proper tools to edit this data.

What we do currently is like editing Excel sheets in a hex editor…

Binary source code (or at least something in the ballpark of XML tech) would also solve a lot of other problems: For example the rendering of the data on screen would be independent form the actual data on disc. This means bitching around "code style" and even partly syntax could finally end. The stuff on screen could be rendered with the help of some "style sheets". Everybody would get the preferred look & feel for their code. (We do this already partly with syntax highlighting; we "just" need to extend this concept). Also things like version control, dependencies, and such, could be build in. Proper indices for searching (and refactoring code) could be also added (things that our editors today compute anyway, but in some ad hoc and very language specific way). Compilation of such code would be faster as all the parsing wouldn't be needed (as it would happen already in the editor; actually, also what is already done today, only that we than again throw away this information for no reason, so the compiler "may" regenerate it over and over on every run, instead that this stuff ends up in a kind of database which could be even part of the "binary source code" format).

I guess some innovative editor features nobody even considered until now would be also enable this way.

It's time for progress!

(Most likely, as almost always, someone will come up with some references to very old systems that did already this because they were ahead of their time; let's see what new stuff I'm going to learn this time :-D).

Having done a fair bit of programming outside of text, text is not "loose" or less "rich" than the programs it is used to describe. It's actually far more explicit and easier to grok than alternative representations of programs. Problems like code style have been solved by linters and auto-formatters.

A lot of the ideas you're pitching have been pitched before and died for the same reason: parsing text is easy for humans and computers. Although I'd quibble with some minor points, like "compilation of such code would be faster". The major bottlenecks in a compiler are codegen, optimization, and linking compilation units. Even bad parsers are extremely fast.

The stuff on screen would be still (rich) text, of course. Only that it would be a rendering of the actually underlying data structure.

Linters and auto-formatters didn't solve anything. They are just a constant PITA, and a source of arguments.

And to quibble a little bit: I said "faster". I did not say how much… Just parsing is indeed very cheep by now.

But a major performance issue was unmentioned: Type checking. In languages with advanced type systems that's the most costly part. Putting all that information into a database and carry it along "the code" would make things significantly faster. It would be like there would be only incremental compilation of the few line you're just editing.

I've actually worked on this problem quite a bit and taken this approach, but it's not as obvious as you're making it out to be - I'm going to talk in generalizations about optimization, which is obviously bad so take this at face value.

Type checking is usually dwarfed by macro expansion, import resolution, optimization, and code generation. There's very little I/O, not that much work to do, and it is pretty straightforward to cache (*). There's a trap to thinking that complex syntax and type systems are potential bottlenecks - it's true the compiler has to do more work, but it's still not that much compared to all the other time it has to waste doing mundane things.

An example is import resolution. In order to type check a compilation unit you need to type check its imports. In order to do that, the compilation unit and its dependencies needs to be macro expanded. If the language supports some kind of scripted macro-programming then you need to evaluate the macros and possibly cannot cache the results (performance pitfall #1). If the language does not support explicit imports, then the entire dependency graph may need be scanned and checked to resolve the imports (performance pitfall #2).

This is a case where the simplest and most expressive language designs can have the absolute worst performance regardless of the compiler's design - the language definition forces the compiler to do a ton of mundane work, in particular I/O and cache invalidation. A central in-memory database holding a binary representation of the AST does not solve this problem and is not necessarily faster than redoing all of the work up front.

In my experience, something as mundane as "import x from y" has a much larger impact over the performance of a compiler than any type checker or parser, and incrementalism strategies are tightly coupled to the design of the language being compiled and not the storage layer. It has to be approached holistically for the language being compiled.

Features like "style sheets" for code are just linter and auto formatter config files. It may be useful to reuse the compiler internals for this, but in practice every tooling author eventually learns not to do that. You will need to modify the IR generated by the compiler, and it probably doesn't have an API that's anything close to what you need for that problem

(*) Of course I'm glossing over type systems that allow higher-kinded dependent types, but those are going to suck to check the same that macro heavy code sucks and currently isn't that widely used.

You're talking about C++ issues here. Those are very specific to C++.

No other languages besides C/C++ have such a broken import and dependency mechanism. No other language than C++ has such broken "macros" like the template mechanism.

Also I was explicitly talking about advanced type systems. C/C++ have a very shallow and simple type system.

But OTOH C/C++ have the most advanced optimizers.

So the whole picture gets quite bended when looking at it form that perspective.

Scala for example has an IR called TASTy which can be stored along the compile artifacts. It's the AST after type checking (Typed Abstract Syntax Tree). Recompiling form that is very fast as it's (almost) only the relatively cheep code gen step.

Also incremental compilation works differently for a lot of languages because those languages have small and independent compilation units. The situation in C/C++ is again a very specific mess as there are really no modules (until now). Usually you don't need to potentially recompile the world on every code change.

> Features like "style sheets" for code are just linter and auto formatter config files.

No, they aren't.

If I "change the style of my code" the actual code gets changed!

In case I would change the style sheet used for rendering only a local config value would get switched. The "binary code" on disc would stay exactly the same. (Still I could than have "Python with braces" after the switch if this is what I prefer to look at).

> It may be useful to reuse the compiler internals for this, but in practice every tooling author eventually learns not to do that.

Everybody involved in the "LSP revolution" would disagree, I think.

The whole point of LSPs is to reuse the already existing tools (usually the compiler) built by the language devs.

I'm not talking about C++ at all!

My entire point is that caching is hard and tightly coupled to the design of the language, and things like typechecking are not really that slow compared to the other work a compiler has to do. Choosing an in memory representation is not really a secret recipe for performance or utility, designing the way the cache gets updated and what it stores is.

I think the idea of a core language with multiple syntaxes is interesting, but I'd contest that the system to implement it is any faster.

> Everybody involved in the "LSP revolution" would disagree, I think.

I reached this conclusion after writing a language server and researching other language servers. The tide is changing to where the primary compilers for languages are query-oriented compilers like Roslyn that allow for this kind of thing, but they aren't nearly as common as batch compilers so language server authors find themselves writing everything from scratch. Interestingly though, big portions like type checkers can be reused if they have stable input/output IRs because they're not slow (however, compiler authors are loathe to expose those details or make stability guarantees there).

> references to very old systems that did already this

Here's one: Alice Pascal

> In a syntax directed editor, you edit a program, not a piece of text. The editor works directly on the program as a tree -- matching the syntax trees by which the language is structured. The units you work with are not lines and characters but terms, expressions, statements and blocks.

https://www.templetons.com/brad/alice.html

I know what source code is and I know what binary code is. But I’ve never heard of binary source code, and my Google search doesn’t turn up anything either.

Can you elucidate?

Most code editors deal with abstract syntax trees for linting, etc. Think HTML DOM but to define the syntax relations of the code itself.

I'm assuming OP talks about saving something like that format directly, instead of text. Then information like 'this method is being called with these two parameters' isn't mixed with presentational info like whether you separated the parameters with newlines, or whether you use tabs or spaces. The downside being that it would not necessarily be immediately readable, the text editors would have to parse it and present it in a readable format, kinda like what happens now with, say, excel sheets.

Even currently code isn't "immediately readable". It shows up, but until the IDE ingested the project it does not provide any features besides basic string editing, which is mostly useless.

So no new issue here.

But reading in a project would be much much faster if the IDE would just need to read a rich binary format!

> Even currently code isn't "immediately readable"

I read code in text editors (not full IDEs) all the time...

> it does not provide any features besides basic string editing, which is mostly useless.

I also do write code in text editors all the time. Not always - I do use IDEs sometimes. But at least half the time, I would say.

I use vim a lot, and all I want there is syntax highlighting. I would hate it if I was forced to use a heavy IDE everytime I want to read/edit some source code. Also I like grepping into sources without having to run an IDE that can search through binary code.

Just a proper binary format for code. (Needs to be invented, of course ;-)).

Using Unicode for that is like pocking around in hex when compared to what we do actual with code in an IDE.

Edit: I just see the sibling has a even better explanation.

Reply on news.ycombinator.com