Let’s build a browser engine (2014) | Hacker News Reader

Let’s build a browser engine (2014)

limpet.net

188 pointschiefofgxbxl47 comments

An other very lightweight weight engine in c is dillo. Unfortunately now its development has stopped.

There's also NetSurf.

As much as saying this is probably going to get me a lot of hate from web developers, the world needs more browser engines. Simpler ones, maybe HTML+CSS only with no scripting. The idea of the Web as a flexible hyperlinked document system and not an application platform needs to gain more support. IMHO if your site is information-centric, and it's not readable in these "document-only" browsers, you're doing it wrong.

I don't understand this hatred of Javascript. The only websites I've felt were actually bloated are news sites with a lot of ads, but that's not a problem with Javascript as much as it is a problem with excessive ads.

What counts as information-centric? A lot of basic things (commenting, searching, liking a post) require Javascript. If you want to use pretty animations, there's a high probability you need Javascript.

Making information-centric sites only use HTML/CSS would significantly decrease the capabsilities and attractiveness of the sites.

> The only websites I've felt were actually bloated are news sites with a lot of ads, but that's not a problem with Javascript as much as it is a problem with excessive ads.

The problem is JS has too much power in the browser, and too little consideration for security. It can effectively take control away from the user, there's virtually no way to know what it's doing without source code audits, which are prohibitive, and the security vulnerabilities are legion.

> What counts as information-centric? A lot of basic things (commenting, searching, liking a post) require Javascript.

None of these actually require JS.

> > What counts as information-centric? A lot of basic things (commenting, searching, liking a post) require Javascript.

> None of these actually require JS.

Can you imagine a facebook doing a page reload/refresh every time you click to like a post? Or without loading more content on demand every time you scroll the page?

> Can you imagine a facebook doing a page reload/refresh every time you click to like a post?

You're stuck thinking about Facebook as if it still had long lists of posts with infinite scroll. The UX would be completely different when the design constraints are different.

For instance, instead of infinite scrolling, you might show one post at a time with clickable previews of the last and next posts. A like doing a full postback isn't a big deal with this approach, particularly with judicious use of anchors. Certainly not as slick, but perfectly usable.

Put me in the group of users who despise infinite scrolling --- I would much prefer a paged interface (like the way it was before IS became popular) because it gives you a sense of where you are, and more importantly, an O(1) way to resume where you left off.

(I suppose the companies like IS because it has an addictive property, but I suspect me and others who see through that don't like it at all. Relatedly, the other popular concept of a "feed" also conjures up images of farm animals munching away at a trough; perhaps that is the real intent...)

Usable, with a worse user experience, to what end? It would be nice to have more non-js built-in browser behavior, though.

A slightly worse user experience on Facebook, for a significantly better and more secure user experience on the web overall. I'm not sure that's such a terrible tradeoff.

Agreed on built-in browser behaviour though. Chrome pushing more input types a few years ago was a great thing.

Yes. Forums worked this way for years. It was fine.

Making information-centric sites only use HTML/CSS would significantly decrease the capabsilities and attractiveness of the sites.

For me, and probably many others, the "attractiveness" of sites that don't use JS is far higher. Searching the Web for information with JS enabled is like visiting a library full of books that will randomly turn their pages, jump around, and scream at you like those in the Harry Potter world.

Intrusive ads are a problem, but not the only problem.

My basic issue with JavaScript is when it's used to move stuff around after the page is done loading, or when it significantly delays when loading finishes.

Voting on posts, and well done search autocomplete can be nicely done with JavaScript, commenting itself works fine without (I don't think there's anything JavaScript where I'm writing this?)

Somedays it feels like I spend as much time waiting for pages to load in 2018 as I did in 2000, and pages certainly look prettier, but don't impart any more information.

I've come across just straightforward documents that require JavaScript to display.

I started a weblog a couple of days ago, and the joy of publishing generated markdown documents on a fileserver is as big as any of my JavaScript ventures.

Interesting. I've used Dillo before on my N900, but I didn't know it had its own engine; I always assumed it was Yet Another Webkit.

Heh no. Its origins go way back before webkit started: https://www.levien.com/gzilla/

Probably very little of that original code in the current version though.

I loved this walk through back when it came out and even made a ham handed attempt to write the example code in Go https://github.com/radiofreejohn/gobinson

It is a bummer it ended with a to be continued that never happened, I remember checking up on it from time to time.

Honestly I feel like he covered the hardest and weirdest things (parsing html and CSS is just weird but not hard. Layout and painting is (IMO) pretty hard and there are a lot of places to get lost.)

Scripting is mostly about hooking up a VM to the other stuff and there are a thousand and one articles on writing VMs.

For networking you should just use libcurl. Now that the URL parser is part of the public API you’re better off with it anyway. Plus the networking should be done outside of the engine anyway.

No doubt. The author owes me nothing, it is a great series. I just felt like I was left hanging a bit by the conclusion. :)

I feel the same way. I took a stab at reimplementing it in C https://github.com/wernsey/robinson-c

My Readme.md contains a lot of TODOs, which I thought I'd be able to tackle on my own, but sadly never got around to.

So the blog post ends with a promise of sections about:

* inline layout

* text rendering.

* networking.

* scripting.

LAYOUT

I think here could be useful to take a look at something like Facebook's Yoga [1] although maybe not great to start. There's this little layout demo that could be interesting to investigate [2].

TEXT

Regarding text probably the keyword to learn about is "rasterization". I found a few tutorials back in the day [3], [4]. In general rendering text is a PITA and you don't want to do it no matter what :-) The only lightweight solution I found was stb's truetype [3].

So if you want to render text yourself you'll need:

- a parser for a font format format (e.g. TTF, PITA) to have a way to get the curve definitions for each font.

- a rasterizer (could be a fun project for getting something reasonable and probably slow performant)

- a curve filling algorithm and plugging it to your rasterizer, which will probably be able to do humble things like drawing filled triangles and other simple shapes (PITA! :-p)

- a word and paragraph layout engine (PITA again :-)

Overall I'm happy half understanding the steps involved and not having to deal with all the messy steps in the middle.

NETWORKING

I'm super lazy in this area, and I would rather learn something like ZeroMQ to posix sockets... back in the day I followed Beej's tutorial which was nice. [6] Then you can connect your sockets to some pre-existing HTTP parser which are a dime a dozen these days [7] and you are all set! :-p

SCRIPTING

Not much to say here... hundreds of languages available. Probably Lua is a nice alternative to JavaScript, but there are also plenty of lightweight JavaScript implementations these days (I think there is a nice little one called V7). If you want to learn how to actually implement an interpreter there are plenty of resources too. If I went this route I would start by implementing some version of lisp or scheme for simplicity.

1: https://yogalayout.com/

2: https://github.com/randrew/layout

3: https://github.com/NotCamelCase/RasterizationInOneWeekend

4: http://www.scratchapixel.com/index.php

5: https://github.com/nothings/stb/blob/master/stb_truetype.h

6: https://beej.us/guide/bgnet/

7: https://github.com/search?q=http+parser&ref=opensearch

>So if you want to render text yourself you'll need (...)

Or possibly use SDL or a similar graphics library meant for game development.

You would still probably have to write a lot of code but you would get a rasterizer, font loading, simple primitives and image rendering in a cross-platform compatible API.

If you’re going to use a library just use straight freetype (what SDL wraps.) It’s very easy to use.

Sure, but SDL gives you more on top of just font handling.

I'm just saying, it's not necessary to reinvent all of the wheels.

Amazing follow up. Thanks so much.

I bet it's possible to achieve layout completeness (in the same vein as turing completeness) with a much simpler markup and styling language.

I'm actually more interested in implementing a toy OS or a toy compiler for a toy language because the browser seems to be a more of very specific complex mess of mistakes and legacy choices rather then a serious theoretical concept. Although I'm sure you'll still learn a lot if you implement one.

You might be interested in STEPS Toward the Reinvention of Programming [1].

Rather than "layout complete" I would be interested in a select set of well thought approaches to layout. Something like flexbox+grid might be the best. Flexbox could be used for simple layouts that flow in one direction while grid could be used for master layouts and complex widgets.

1: http://www.vpri.org/pdf/tr2012001_steps.pdf

Nice, I read that report some time ago and since I've been kind of obsessed with the idea of model and language driven development. It really seems like a much better way of writing software.

I don't know about the "completeness" part but the idea of making a brower-like rendering system without the burden of having to support old standards and data formats _is_ appealing. I think this is what projects like QML [1], JavaFX [2], Avalonia [3] and more recently Flutter [4] have done (also... Flash! does anybody remember it anymore? One minute of silence please...).

It is too bad none of these projects decouple well enough the layout and rendering part, they are really very coupled to their respective environments and all of them are super idiosyncratic and make cross-language development almost impossible or unpractical (except maybe Qt but it is not quite the same model as QML), as opposed to the openness of the web platform. I think maybe all of them use Skia on the back though?

Some of these projects are similar to the browser in that they allow controlling the content using an scripting language (JavaScript for QML). What do you get when you mix a rasterizer plus a scripting system or VM plus a layout system plus some IO primitives? Macromedia Flash! Just kidding, you get a browser. Or either one :-).

1: http://doc.qt.io/qt-5/qmlapplications.html

2: https://openjfx.io/

3: http://avaloniaui.net/

4: https://flutter.io/docs/resources/technical-overview

With regard to browser specific problems, I came across these articles from this "Meyerovich" guy talking about some opportunities to optimize and parallelize the layout process. Honestly a lot of this stuff flies over my head (formulating the layout problem with attribute grammars? I don't even know how to use attribute grammars for the "normal" things... :-p). I have the idea of maybe coming back to these papers some day with more time and patience and go over the references, etc:

https://lmeyerov.github.io/projects/pbrowser/pubfiles/paper....

https://lmeyerov.github.io/projects/pbrowser/pubfiles/login....

https://lmeyerov.github.io/projects/pbrowser/hotpar09/paper....

There is also https://www.netsurf-browser.org/

Small as a mouse, fast as a cheetah and available for free. NetSurf is a multi-platform web browser for RISC OS, UNIX-like platforms (including Linux), Mac OS X, and more.

Have you ever tried building it? It’s small compared to Firefox but not “small as a mouse.”

Lynx or (maybe) linemode might qualify for that.

It takes a while, but it's not too bad. Even my old Atom netbook managed the build in, if memory serves, under an hour - launched the build before lunch, was done afterwards - more than what I expected for sure, but even elinks takes its sweet time.

And I believe that they might just be referring to NetSurf's memory footprint instead than its code size. It's much heavier than Dillo, but it's currently using only 118 MB with 10 open tabs.

Another problem would be the delivery. Yes, writing something that behaves like a browser would be nice, but it took 10-15 years for browsers to be present on enough computers to make a clear difference, and there is currently no reason to assume that a new technology that is the-same-but-simpler would have nearly as much success.

Of course, the solution would probably be to compile that code to something that works in actual browsers, which would be somewhat ironic :)

I would say, if you managed to cross compile your new browser into a statically linked binary executable weighting a few hundred kilobytes, it would become trivial to distribute and a very attractive “platform”.

You could probably easily render it in a regular browser using JS as well. Instant universal deployment.

They can bake in dual rendering solutions into the browser and wait for the better solution to takeover. Very similar to what is happening with JS and web assembly.

This was a very useful guide for me a couple years ago. I had to print html documents (thankfully, just a reasonably-well-defined subset) from a .NET Winforms app, but driving a browser to do it was too slow and flaky. So I used the OP as the starting point for a C# layout engine.

Is it open source? Thinking about the logic for auto boxing text nodes adjacent to elements...

Can you elaborate on "auto boxing text nodes" ? Not sure what you mean. Are you trying to shape paragraphs of text? (like [1]).

By the way, anything having to do with text is more complicated to handle that one would imagine [2].

1: https://github.com/bramstein/typeset#variable-line-width

2: https://xxyxyz.org/line-breaking/

This is what I'm referring to:

https://www.google.com/search?q=css+anonymous+block+box

Ie. when having text adjacent to an element (which is a block element), that text is automatically inserted into a box which can't be styled or referred to, but yet help to layout the document.

    <div>
      I'm in a box
      <div></div>
      Me too
    </div>

wkhtmltopdf could have been a shout

Just implementing the <video> tag is a career.

What happened to Matt Brubeck?

He still works on browsers, but blogs less. You can see his contributions on GitHub.

Should probably say [2014]

Yes.

Reply on news.ycombinator.com