I Built My Own Shitty Static Site Generator
erikwinter.nl
erikwinter.nl
I just built a small web application framework that I can build some basic personal apps on top of. Just like the author's, it's "shitty". It lacks features, doesn't handle edge cases, isn't particularly secure or performant.
None of that matters. It's locked up behind HTTP authentication and it's for my use only. It has exactly the features I desire, and none that I don't. I understand it top to bottom. It was fun to write and extend. It serves my needs perfectly. I didn't have to fight with, or conform to someone else's notion of how software should be built.
You go into any experienced woodworker’s shop, you can divide it into maybe 20% of the tools that see the vast majority of the action—table saws, band saws, milling machines, proper hammers. But then there is a halo of 60% or more of the things there are little helpful things—doodads, mallets, jigs, often even shelving and work surfaces—such that she uses a bunch of these on any build she undertakes, and she made them herself. “Why would I stumble around with someone else's mallet when it’s about the same cost and about half a day’s work to machine my own mallet down in a router that has exactly the weight and shape that I like it to have? Pays for itself.”
We are so far from this ideal. All of our software breaks because we imported left-pad and somebody unpublished left-pad. Wasn't even a world changing perfectly V8-on-x64-tuned performant version of left-pad, but we gotta “not reinvent the wheel.”
I don't think we do it for no reason, there is something about software itself right now that is amateurish. We’re still writing subroutines working on data structures; our brains are FORTRAN even if we call it Java, and whatever that is makes it really hard to decouple the tools that we are using to build software from the final outcome that has been built. The woodworker has a sense of the “shape” of the artifact, we routinely build the equivalent of gorgeous wooden tables that start spewing garbage onto their shiny surfaces after three weeks unattended, “oh, I didn’t realize that the built-in trash chutes could get clogged.” Wait—what? “Yeah, we allocate more wood at the bottom of the table to make up for the wood that we remove at the top of the table every night so that the surface is always clean.” Your user doesn't clean the table? “Hahaha you’re silly, users could not possibly have the technical knowledge to keep anything clean, they are not trained in sponges, and I especially could not trust the with something as beautiful and delicate as this table.” The problem has a beautiful sort of fractal quality to it, every zoom-in is more WTF.
You can reinvent the wheel as many times you want. If you however do that at work, your colleagues and your replacement after you left the job likely won't think too kindly of you.
> The IKEA effect is a cognitive bias in which consumers place a disproportionately high value on products they partially created. The name refers to Swedish manufacturer and furniture retailer IKEA, which sells many items of furniture that require assembly. A 2011 study found that subjects were willing to pay 63% more for furniture they had assembled themselves, than for equivalent pre-assembled items.
https://en.wikipedia.org/wiki/IKEA_effect
I think this effect also applies to developers. They tend to value self-written applications subconsciously higher than "pre-assembled" applications.
I don't think that this is the same about Ikea, though. If I bought Ikea table, I can't really adapt it, unless I'm advanced woodsman with proper tools. If I'm just ordinary guy with screwdriver, I can just follow the script to build that table.
I think that Ikea analogue is someone building his vim environment from other's plugins following guides. He does not know C, so he can't hack vim, he does not know vim configuration language, so he can't hack plugins.
Therefore, I think the IKEA effect holds true in software development, because you rely on "pre-assembled" items, but I agree that you have much larger degree or freedom on how you build your application in e.g. C compared to a IKEA build. Maybe, this higher degree of freedom even strengthens the effect.
- Frameworks and tools are almost always ill-fitting. You have many features you don't need, you lack many features you need. You have dependencies you don't want, code that you don't know, models that might not match your own (and might not be explained well enough, or might not even be consistent with themselves anyway).
- Frameworks and tools are often leaky. They don't solve problems perfectly anyway. They substitute the problem of having to develop a custom solution to the problem with having to figure out how to use the tool effectively, plus any problems derived of the shortcomings of the tool itself, which are often harder to solve on a tool you didn't develop yourself. The better you want to do something, the more disappointed you become with leaky solutions. And other people leaks are far more uncomfortable than your own leaks.
- Some people just want to have control and know what they are doing, even if they need to put much more work on it. Not productive, but more consistent.
- Writing your own tools deepens your understanding of the landscape. This is particularly significant in a world where we have hundreds of frameworks and tools, but very few coherent explanations of the context they work on and the problems they really try to solve. We have reached the point where if someone wants to make a non-trivial website, we tell them to learn React instead of sending them to learn how the internet and websites work so they can choose a tool for their needs afterwards. And this piles up and giving coherent explanations of any landscape becomes harder because no landscape is coherent anymore.
- Related to the previous, there's a certain trend that emerges where the more dependent we become on frameworks and tools, the more the problems in a given scenario are ignored or solved by adding more abstration layers and fancier features on tools, instead of working on the root underlying issues, and the problems become fuzzier. Working on the actual problems by yourself helps you see how much cruft and dirt there is behind even well established technologies... and some of us are masochists and want to know about all that sh*t, because if it remains hidden there's little chance to fix it.
How about compilers?
That's not what I'm trying to do (to compare them), but people who like to write things from scratch often blame frameworks e.g. they think they are leaky or don't solve problems perfectly anyway.
My question is why they think they can rely on interpreters or compilers, because they also have limitations, bugs, CVEs, leaky abstractions and other known problems. They also form an unknown variable, because most haven't read the source of the compiler or interpreter they use.
I'd much rather inherit a project that contains 'sprintf("%010d", i)', or equivalent, than one with yet another dependency. Especially when repeated a thousand times over the course on an entire project.
Every dependency is one that I must understand all the possible return types, failure modes, and upgrade paths. Every function is one that I must read. The needle must be quite clearly on the benefits side in order to be worth it.
You shouldn't reinvent Django if other members of your team already know Django, as you're imposing a learning cost in doing so.
You rather obviously shouldn't reinvent the Linux kernel, as doing so is an enormous undertaking. It's probably a huge waste of resources, your solution will probably be far inferior, and you're likely to fail anyway.
You shouldn't reimplement TLS, as you will almost certainly do so incorrectly, and it would take a great deal of effort. It's a waste of resources and will worsen your cybersecurity risk profile.
Why should you not? You only learn by developing. Precautions follow with such, as likes not running in production. But how else do you learn if you do not?
If your interested in learning about secure sockets and the rest, you should reimplement TLS. Because that will teach you the inner workings of TLS.
The vibe "you shouldn't" crushes the development side of things.
> "Shouldn't" is the wrong word to use. Why should you not?
I'll assume that here you're not referring to reimplementing purely as a learning exercise.
I already said why: you will almost certainly do so incorrectly, and this will worsen your cybersecurity risk profile. For serious work, you should use a mature battle-tested implementation like everyone else. The slightest error in a cryptography codebase can lead to severe security vulnerabilities. This is well understood and much has been written on the topic of don't roll your own crypto, e.g. https://security.stackexchange.com/a/18198/
> The vibe "you shouldn't" crushes the development side of things.
No, it's solid cybersecurity advice. It's irresponsible to make use of either an amateur cryptographic scheme, or an amateur implementation of a cryptographic scheme, in a serious codebase. Reimplementing TLS purely as a learning exercise is of course still to be encouraged.
More generally, there is no culture of you shouldn't posing a problem in the software development world. Nuclear engineers and bridge engineers are taught you shouldn't... but software developers are taught just go for it.
If I create a mature, well thought piece of encryption. There is no irresponsibly in wanting to use it in production. Yes, I should have it validated and if that validation comes back as something with no thought, buggy and disastrous then that would be irresponsible of me. And one could argue it's irresponsible to run it without validation too. To which I would agree upon.
I still disbelieve that those who create something being told: "you should not be allowed in production ever" and only run "battle-proven" is the wrong approach. Create something, validate it and then run it.
Besides battle-proven systems had to be ran in production in the first place and that those battle-tested systems still have vulnerabilities themselves, ie: HeartBleed. Again which was immune with Mbed TLS/PolarSSL that someone had reimplemented TLS. Validation is the key that's required.
"You should create your own implementation of TLS but you should should not run it in production as that would be irresponsible until you have validation". Not just to blanket "you should t" end of. Above is what should be passed to developers. If you feel your work is good enough, then pay the price for validation.
Besides what makes an expert if you don't make it for yourself?
> What makes an expert if you don't make it for yourself?
You study the field, like any other field. This doesn't collide with what I've said.
In this context there needs to be a fairly bright line between learning, and producing real-world cryptographic systems. It might be instructive to have engineering students build an airbag system, but you don't then put it in your car.
> If I create a mature, well thought piece of encryption
Unless you're a professional cryptographer, someone like Bruce Schneier, Tanja Lange, or Filippo Valsorda, it's best to assume that you haven't created a well thought out cryptographic solution. See Schneier's Law. [0] If you have a PhD related to cryptography, and/or a history of employment as a cryptography specialist at a major technology company, then you may have a solid enough grasp of the field to be taken seriously, but short of that, you should leave cryptography to the experts.
It's really hard to get the theory just right, and it's also really hard to get the implementation just right. Fortunately there are existing out-of-the-box solutions that do all the things we want: secure channels, secure file encryption, authentication, etc.
> Yes, I should have it validated
We have a validation process: standards bodies. For instance, in the TLS 1.3 standard, they introduced the requirement for supporting the x25519 algorithm. That algorithm was developed by a team of professional cryptographers, not by a well-meaning dabbler, and it has been subject to careful scrutiny by the cryptographic community.
After standardisation, we see the algorithm implemented in the few trustworthy TLS libraries (e.g. OpenSSL and Google's Tink), which we then adopt for use in the real world.
Serious organisations do not play around with this stuff, they only use trusted standard algorithms. Microsoft/Apple/Amazon/Google have crypto teams who are qualified to write their own implementations of the standard algorithms. The rest of us then use those implementations. Microsoft's Active Directory is backed by the standard Kerberos crypto protocol, for instance.
> I still disbelieve that those who create something being told: "you should not be allowed in production ever" and only run "battle-proven" is the wrong approach. Create something, validate it and then run it.
We agree in a sense, it's just that the bar for considering it battle proven is set very, very high, and for good reason. Developing crypto isn't like styling a webpage with CSS. It's technically challenging to do correctly, it's difficult to know if you've done it correctly, and the consequences of getting it wrong are severe.
> Besides battle-proven systems had to be ran in production in the first place and that those battle-tested systems still have vulnerabilities themselves, ie: HeartBleed. Again which was immune with Mbed TLS/PolarSSL that someone had reimplemented TLS. Validation is the key that's required.
I don't know what 'validation' is meant to mean here. If we had a way to easily detect such issues, we would use it. Again, it's extremely difficult to get this stuff just right. The smallest defect can have terrible consequences.
This applies even when we're doing everything correctly. As you say, we see issues even in major implementations. That doesn't mean that using amateur crypto code is a good idea. It isn't. Every cryptographer agrees that it's a terrible idea to do that. Sometimes aeronautical engineers build dangerous aircraft, but that doesn't mean we let amateurs have a go.
> If you feel your work is good enough, then pay the price for validation.
That isn't how it works. Cryptography is an academic discipline, it makes advances through slow-moving academic publishing and standards bodies, not by paying for a code-review. If you really want to make a contribution to the field, you'll need to make a career of it.
Again though, in a sense there's little need. Much of the best crypto software in the world is Free and Open Source.
If you want to implement TLS as an exercise, or make a neat cryptographic 'toy' program of some sort, then great, but don't gamble anything of value on it (user data, say).
[0] https://www.schneier.com/blog/archives/2011/04/schneiers_law...
My mind is primarily focused on the practical viewpoint of a experienced Systems Operator/Admin rather then a Cryptologist.
Lets build our own, make it work and push it live and see what crumbles. I amware that TLS and the likes are all very specialist and to the level as you wouldn't give a Ferrari owner keys to a Jet Fighter. What did spark the flame for me is that be "Free and Opensource" until it comes to crypto which then becomes a touch area of "leave it for the experts" but with full reason makes sense in a way. Maybe because I know little on the matter, that is what makes a large problem.
Learn something new everyday. Thank you for your time on the matter.
A good attitude for a weekend project, but not a good attitude for cybersecurity.
> "Free and Opensource" until it comes to crypto which then becomes a touch area of "leave it for the experts" but with full reason makes sense in a way.
It's still very beneficial for it to be Free and Open Source. It can be studied by hobbyists and students, it can be audited and analyzed by anyone at all, or even by automated systems.
Glad it's been helpful.
I've encountered ~100% custom codebases that are unmaintainable, and I've encountered ~100% custom codebases that are extremely maintainable, even for a new person joining a team. And we're not talking about leftPad-esque libs; entire frameworks made in-house.
It's rare, and there are reasons to avoid it (locking yourself into your own tech, not benefiting from community innovation, not being "on trend" with common dev conventions). But ease of maintenance is not one of those reasons. If it's hard to maintain, that's on you.
I do agree fully that programmers should strive to understand every layer, and building your own tools is a great way to get experience that helps with that. Yes, left-pad is unnecessarily granular abstraction. However, in each domain, we inevitably depend on other software, and that breadth of that graph is part of the reason for why you can accomplish so much with so few people writing software.
The problem is that many things a developer considers tools (as in: "the best tool for the job") are actually components.
The equivalent of a woodworker's tool for a software developer would be an editor. Only you or your team might care about which particular brand of sawing machine or editor you use. And they are not an integral part of the end result.
A component for a woodworker would be a hinge or drawer rails. You would probably not appreciate it if your woodworker uses hinges that they made themselves. They are not worth the time to build. They're probably not very reliable. And if they fail they might not be easily replaced or repaired except by the person that made them. So you would only use custom components if the part you need is otherwise unavailable or does not meet specific needs.
Are you a woodworker and making a cabinet for yourself? Go ahead and make your own hinges, but for paid projects you probably should rely on standard components as much as possible; only doing custom woodwork or software engineering where it adds value.
Hinges - like everything else on CarpentryHub - are either in a state of permanent development or abandoned. So you design your project for Hinge 13.2, but then you come back a month later - just before you open your Etsy store - and discover that Hinge x.n has now become HingeOS, which is breaking change.
It also has its own HingeAPI, which is poorly documented, somewhere.
You were used to Hinge's ScrewHole 5.3. But that's now buried inside HingeAPI, which calculates your project budget and lifetime carbon footprint for you, but no longer accepts external screwhole positions, because it works its own values from the complex project specification you have to supply - defined in DirkLang, which is a hot new carpentry specification language only ten people know, and replaces HammerLang, which was fifty five years old and very popular but had a lot of frankly questionable design choices.
And the values HingeAPI returns aren't mutable, because mutability is bad practice.
I too look forward to the 2021 YC batch
To absolutely torture the analogy: say you have a general hinge that you can attach to any door. And that's fantastic, but it's a lot more complex than it needs to be (it being generalised necessitates this) and it isn't actually completely optimal in most cases (it just works ok, ish). You can use Super Hinge on all your paid projects, and it will work, it's just that it's often better to make a hinge that exactly fits the required specifications.
There are for sure economic factors behind choices in how software is built, for sure.
Back to the analogy and actual tools -- what employer is going to pay you to write your own editor to use to write the software the employer wants? It's not just that they are mean and choose not to pay you for this, it is not a cost they could sustain (for all but the biggest employers I guess. Google, sure, can have an editor team. Still can't let every individual programmer build their own editor)
To go back to the static website, I really like the Wiki format (Current favorite is http://pmwiki.org ). I picked it for three reasons 1) It's a very powerful wiki 2) I can convert the site into static pages. This lets me build in the wiki format, then publish out. 3) There is a cookbook (a tool another user wrote) to make PDF. You can select the pages in page order and produce a PDF. To that base I've added my custom items.
Pmwiki in it's own right is simple (a router), but allows for cookbooks (custom extensions) that others can add. One of the interesting things is the current lead developer doesn't really add features to the core, they also create cookbooks to add on. And the core is pretty soft, if you don't like the markup, you can change it.
A long way to go to say that I can see writing software like the OP article. But if you should look at a core tool and see if you can build around that. @Molf uses hinges as an example. You can do a lot around a stock hinge (paint, bevel the edges, add "hand crafted tool marks", a process that is easier to do than starting from ground zero.
You can also think of lower-level components, like nails and glue, which even more general and standardized than something like a hinge, maybe these are like standard-library or builtins for a language?
I would just use "personal".
err...
Because, let's face it, most software out there (especially most software built by individual hobbyists) is ... let's say not great.
I realize that this is a cultural thing. In the US for example they tend to favor soft language more. They would call it "beta", "early access" or "personal project".
As an European I tend to see it more as... well, "shitty".
And that is fine. It is in fact refreshing not to have to filter out the marketing buzzwords like "innovative" or "minimalistic". The title immediately put me in a good disposition towards the author. It feels honest.
When I peek into your average open source project (made, mostly, by hobbyists) I see good to great quality. Whenever any private code gets leaked, or on your average company, or just inspecting websites, you see the horrors. Just my experience here :)
[1] Spaniard, I do believe our websites might be on the bottom of quality of Europe though, so my experience might be different from other "European"s :)
In general I agree with you, open source tends to have better quality than proprietary software.
I will point out however that Open Source sees a lot of contribution from non-hobbyists too. That's my case, in fact: my salary for the last 10 years came from working on open source projects.
http://scripting.com/davenet/1995/09/03/wemakeshittysoftware...
Ha Ha!
But, it works, it has rare features I need, features that AFAIK most of the famous ones do not and could not be easily added (well, at least when I checked)
That said, while it started 100% hand written I did add stuff over the years. For example I had my own templating system but switched to 'handlebars' at some point. I don't actually remember if there was a good reason or not. A long time ago I wanted to add RSS/Atom support. Sure that should be easy to do on my own (i've done it in the past) but whatever library I looked into was easy without too many dependencies. The latest, I added jsdom to process a few more automated transformations.
The word "shitty" really does describe the code and I'm embarrassed to show it really but it works and I have better things to do than refactor to make it clean.
Sometimes an existing wheel design is perfect for our use case, or at least good enough that creating a new one isn't worth the effort. Sometimes a tweaked wheel is what we need. Sometimes a radical departure from established wheel design is required to make a new machine possible.
Build vs buy is an eternal question, and it's reasonable to examine and reexamine it for any project. But "reinvent the wheel" is just hyperbole.
You greatly overestimate people's knowledge of wheels.
Your complaint about the phrase seems to be that it doesn't well describe the process of adjusting another design to work better for a specific use-case. But that's not what the phrase is trying to describe, it's literally describing the problem where people reinvent that which absolutely does not need to be reinvented.
In the case of rewriting an application feature for feature in another language, that may or may not be a good idea. Maybe we're switching from an interpreted language to a compiled language because we need better performance. Maybe we're replacing a huge pile of impenetrable bash with a well organized Python app. Maybe we're taking advantage of a library that's only available in another language. Maybe there's a good reason for the rewrite but it's so much work that it's still not worth it. Maybe it's just a engineers wanting to use the latest toys because it'll look good on their resumes.
It's totally legit to question the need for a given project. But the phrase "reinvent the wheel" is just scornful insult. It adds nothing to the discussion, and in fact makes it harder to discuss trade-offs.
I think that over the time all the software starts creeping with bunch of features that most people dont want or need and strange decisions from the past hunts it.
By way of example, my hand-rolled blog has a README file which lists these things in it. Whenever you pick up a new SSG or create your own you should make sure that you have a good plan for at least these things:
1. How to start writing a new post. For me, that's the script to fill out the template, plus the instructions to run the dev server.
2. Frontmatter fields supported. If there's too many to list all of them, at least list every one you've used before, since you'll probably use those again.
3. Tools for writing. For me, that's a few scripts to do things like properly import media.
4. Publishing. This is a full checklist including "verify mobile", "verify excerpt on posts page", "run publish/deploy", "submit to these sites".
One of the reasons your personal SSG feels better than anything is because you know every nook and cranny in it. That feeling will go away after about 6 months, and you'll thank yourself for having written good documentation.
e.g. Ruby: https://rubygems.org/gems/front-matter
Python: https://pypi.org/project/python-frontmatter/
Javascript: https://www.npmjs.com/package/front-matter
PHP: https://packagist.org/packages/webuni/front-matter
Go: https://github.com/gernest/front
Elixir: https://github.com/sebastiandedeyne/yaml_front_matter
Clojure: https://github.com/liquidz/frontmatter
And many of those languages have more than one library.
Like it or not front matter in Markdown files is a bit of a standard these days.
Personally, I find it a bit silly to optimize for outputting things in many different formats when you'll have to deal with tweaking the output for each format to get it right. And it's not a one and done deal. You'll still find things going wrong on occasion. :)
Personally, I've used a number of static site generators and I ended up in the opposite direction. I needed a website where I could write posts and not have to worry about more than just writing and posting, no deployment, no plain text intermediate languages between me and the end result.
I ended up using Wordpress. I was already familiar with it, knew how to self-host it, and was comfortable modifying a theme to suit my purposes.
Writing and posting new articles is so much easier using rich text. I generally type up my post in Apple Notes first. Copying and posting to Wordpress saves most of the formatting. Inserting images is drag and drop.
Large numbers of good plugins meant it was easy to find something to implement features like having a click-through warning on 18+ posts. Install a plugin, configure it, and I'm done.
Wordpress works well for me, but a lot of programmers are rightfully horrified. Wordpress is popular and complex, which results in a long history of security issues requiring mitigation. It's inefficient and needs caching. It needs to be backed up because everything is in a database, not plain text. As a result, it requires more systems knowledge than a static website.
Static site generators and applications like Wordpress are tools.
Everyone ultimately looks for a tool that works for them. My goal is to write posts, not fiddle endlessly. Other people love the fiddling.
So, if you’re looking to build and maintain a website, think about what’s important to you. What’s your skill set, interests, preferred workflow, and goals?
I know that there are some websites that take care of this… for money. Unfortunately $20/month is not in the budget for one-off sites.
But you can do exactly that right now. For example:
- use wordpress as your data backend in next.js
- not exactly a static site generator, but you can use w3tc plugin to perform full page caching with very long expiry time, which essentially turns your wordpress site into a static site. You can also do this using cloudflare "cache everything" page rule, but it must be paired with another page rule to skip caching if wp cookie found. Similarly, you can add web app firewall rules on cloudflare to disallow login access except for you.
Also for whatever reason even with the right headers I could never force CloudFlare to actually cache everything. I think it will just keep a copy of a visited page in some servers, but not in all, and definitely it won’t cache the whole site at once.
There is also an option in cloudflare to pre-cache everything on their edge servers so visitors will never hit the origin server, but it's not available in free plan (I think it's enterprise only plan).
I forget the exact set of wget options but you can find blog posts about how to do this.
I'm actually working on basically exactly what you described (using WP2Static), and we're planning for a competitive $10-15/mo price, but we're not quite ready for launch yet.
Hand off WordPress / Django CMS to untechy people (either hosted or installed on their pc locally), setup a daily scraping job that updates a static website. By using free static hosts (eg netlify) and free scheduled processes somewhere (eg heroku) you can do it for free.
With a SSG I just have gitlab host it for me which is everything sorted and git does the backups.
The trick with Wordpress is to accept that somethings wont be as slick as if you spent $1-10k of your own time building perfect static site generator and customising it beautifully. You have to do what most non-IT people do and make do. But I feel like I rarely have to make do with Wordpress - I can get speed, I can customise a bit with PHP sometimes (although I avoid that) and there are a tonne of plugins.
You can build many "no-code" level solutions in Wordpress for free.
But the biggest reason I use Wordpress is I can't handle the friction of treating my website like code. My author hat hates programming and needs a faster feedback loop than flipping back and forth between different windows to see what's happening.
Being able to click Edit, fix a typo, and hit Publish, without any other steps, is what I need.
That's before I get into all of the features Wordpress supports. I installed the JetPack plugin and now I have mailing lists, analytics, monitoring, and a bunch of other things with No Extra Work.
For any website, the complexity has to go somewhere. If you're a programmer with simple needs, an SSG is easier than wrangling an angry kraken like Wordpress.
There are many horror stories of people using git as their backup strategy and finding out a corrupted object or a goofed up rebase not noticed right away meant a mass panic looking for an old clone.
If you want to backup your site, do it in addition to git.
Wordpress is inherently unstable on the long term because plugins must be upgraded (anyone who’s had a vulnerability issue knows it). But there is no other decently simple technology.
Writing posts is slightly more of a pain than with wordpress but its nothing compared to the pain of either updating wordpress or having it hacked.
I don't have recent experience here but, last time I used WP (10+ years ago), they had caching plugins. Isn't that kind of the same? If not, couldn't you just yank out the editor of WP if you think it's more pleasant and put it into anything else you wanted?
My knowledge is about 5 years old at this point, but the idea still scares me silly.
I could slap something together with a few lines of Make or Python that does exactly what I need, in an hour or two. I could even hook it up to AWS and have it update automatically.
Doing all of that is less time consuming and requires less effort than reading through the grav or hugo documentation, installing and setting everything up, and troubleshooting the inevitable problems.
The only issue I have is that doing front-end web development/design is not something I enjoy, so grav and hugo's existing library of themes is very convenient. Convenient enough to put up with the other annoyances.
Here is a static page generator implemented in a 200 LOC Makefile: https://github.com/patrickbr/hagel
Disclosure: My wife developed this project.
@cat $(dir $@).$(notdir $@).$(notdir $<).newlesc.tmp | tr -d '\n' | sed -e 's|\([$(ALLWD_VAR_CHRS)]*\)={{|\n\1={{|g' -e 's|\\\([^n]\)|\\\\\1|g' -e 's|/|\\/|g' -e 's|\&|\\&|g' | sed 's|\([$(ALLWD_VAR_CHRS)]*\)={{\(.*\)}}| s/$$$(strip $(subst .,,$(suffix $(patsubst %.info,%.category,$<)))){\1}/\2/|' > $@.var.sub.tmp
Edit: Now I see in the readme:
> Error messages are for the most part cryptic errors from sed. Most of them are entirely useless to you.
A quite honest readme & project :- )
They tend to oversimplify complex problems. They fail to recognize that perhaps they haven't thought of every single twist and turn in the path. These comments believe building something is a straight line, and that because they can think it up and do some napkin math, everyone else's work is invalid.
It's easy to say "I can build x in x time" until the hard work starts and you realize reality isn't that simple.
they're saying "does exactly what I need". this poster probably doesn't need all the twists and turns that these tools address. many of us could build our own "tool for X" that does just enough of what we need, but it's almost never remotely suitable for use by anyone else, let alone a growing community.
Omakase is sometimes desired when you don’t have the will to put up with another new ecosystem to learn.
Amen - Hugo is a total pain to work with in my experience. Many hours pulling hair out trying to do very simple things (like a page that appears if the user just enters a directory (e.g. "mysite.com/stuff/" and not "mysite.com/stuff/page"))
I found Jekyll to be much, much, much, much easier to work with than Hugo - I ported a site from dokuwiki and with Jekyll I was done within a day (complete with writing a custom theme), but with Hugo I spent days and days just wrestling trying to get basic things to work before I had even started theming before giving up and just using Jekyll.
I am sure there are people who need compilation in 20ms vs 1000ms, but I can wait a couple of seconds if it means I don't have to spend days messing with Hugo just to get it to do basic things.
But if you don't do everything the specific Hugo way - boom everything renders blank, and there are no warnings or errors.
There are some novel things though, like, I use markdown files, but separate sections with ``` so I can use different inputs like ```MD for markdown, or ```JSON for pure JSON data, or ```PHP for PHP code. The VS Code highlights the portion of the code correctly, so I can work on many formats/languages on the same page. The .md file looks like this:
```INI META
includes = docs/index.html.md menu menuTop
title = Theme
base = docs/docs.html
slug = theme
```
```JSON menu
[
{ "label": "Demo", "url": "/demo/features" },
{ "label": "Try it!", "url": "/try.html?d=/data/try.json" },
{ "label": "Docs", "url": "/docs" }
]
```
```MD BODY
# Customize the story
Below are the parameters to customize the style of the story.
```
---
includes: docs/index.html.md menu menuTop
title: Theme
base: docs/docs.html
slug: theme
menu:
- label: Demo
url: /demo/features
- label: Try it!
url: /try.html?d=/data/try.json
- label: Docs
url: /doc
---
# Customize the story
Below are the parameters to customize the style of the story.
For example, for documentation, I use markdown, but for the website, that I want to control the layout better, I use JSON only, and when I need to create complex operation (as a consult to an external database), I use PHP. The markdown is very limiting, this was the motivation to create my own.
While this is true for nearly all software, I missed being in control of when to upgrade, and I also missed a few little features here and there. The result? I shopped around for it a bit and then decided to roll out my own: HastySite (https://hastysite.h3rald.com) which now powers a bunch of sites of mine like https://h3rald.com and https://min-lang.org
Actually I ended up writing my own markdown processor and my own programming language before doing that (which also needed a Readline/line noise replacement, and wrappers for a regexp and compression library) but... hey, that's part of the fun isn't it? That's the very reason why I love being a programmer: it's not about the final result, it's about how you get there and what you learn along the way.
So kudos to you sir, I wish you all the best and that you build more things yourself.
I used it to replace a code section containing dot code with a png.
or using comments like <!--more--> to generate site Teasers.
https://github.com/enter-haken/book/blob/master/lib/generato...
You can use pandoc to convert markdown in almost every format you like.
I use it to write letters. https://hake.one/blog/letter.html
If you have access to the AST, you can integrate as much metadata as you like.
The complexity in most SSGs comes from them being generic solutions.
If you have a fairly simple site and know what you need, you probably get it build with a few lines of JS and a GH action.
Using a generic SSG is like learning CMake.
Althought most of the times using a generic SSG feels more like learning the autotools than CMake.
Inspired by https://github.com/giuseppeg/xm I ended up writing a absolutely minimal no-frills PHP snippet to turn the PHP file itself in to an html page.
https://github.com/gnyman/piss / https://blog.nyman.re/2020/10/11/introducing-piss-a.html
Doing it yourself is of course not an option for most people. But as the author and many other commenters here, programming is a hobby so the journey is as important as the destination.
My issue was mostly long term backup. When you think about keeping your writing around decades--ie, longer than you might keep a site--there's no solution I loved.
I also love Scrivener for actually writing. So I built a SSG that takes exports from Scrivener and turns that into webpages.
I get to use my writing tool of choice + I have full backups and the ability to recreate the site on demand.
Also, Scrivener has metadata as part of the UI, so no awkward front matter. Just some custom fields (as the software already expects).
Definitely a fun exercise. I bet I still have the code around somewhere.
(The QB version was succeeded by a series of shitty SSGs: one written in Visual Basic, one written in Java, and one written in C#, before the current iteration which is backed by a dynamic website.)
I don't have the same need as the author though. If I want to post something that is not a blog post (e.g. a page that showcase a particular personal project), I'll just post it as a standalone page and add whatever custom stuff the standalone page needs in the theme.
In my case, I stuck with markdown and still used frontmatter to add metadata to posts. Since I'm using asp.net core, I can use the markdown library Markdig which supports frontmatter, but I can also just fallback to the name of the markdown file (regex out the date and title), since I don't need much information from metadata. Adding middleware to asp.net core is fairly easy as well, so I can simply create some simple middleware to create static files as requests come through.
The only annoying case I had was how to manage syncing between git or gitlab (I created post retrieval clients for both) and already generated static files. I have a fairly forced method at the moment, where almost every generated file is wiped when a new git commit shows up, but I get to find those solutions myself since it's mine (!).
Definitely agree with the sentiments here. Although, I feel that the fact there's already working products out there makes it less incentivized to do projects like this. The new programmers I know seem to be more interested in doing something new or creating software you can't just get from some SaaS. I guess when I created my first website I didn't have that worry, there was no squarespace or whatever else, so there was the double benefit of creating something both simple and new.
When I initially started googling around for custom static site generators, I could not find much content so I just wrote a quick post on it: https://smalldata.tech/blog/2018/08/16/building-a-simple-sta...
Only like, 3 seem to have any significant number of themes and half of them don't seem to work with the latest version of the generator.
The first is fugitive [1]. It came to life because I just learned about git hooks and thought of using them for generating a blog with all of its meta data in Git itself. I wanted to have only Git as a dependency so I wrote the whole thing in pure sh (not even using some convenient Bash features). It was for fun and for just myself initially but it had maybe like two dozen users in the end, some of them even contributed to the project or maintained their own forks.
The second one is not released. It's what I use for my own web page [2]. It's a make file + a few Bash scripts that make use of xml2 parse an HTML template and content files, then uses the GNU coreutils (mostly sed) to insert the content in the template and adjust a few things, and finally use 2xml to build the final HTML document. It also has two additional scripts: one that can take an XML file and build either an RSS feed or an HTML output to be inserted into a page, and one that can build an HTML representation of my list of publications.
The third and most recent one is called cigala [3]. It was written for people around me with no web development experience but who needed a simple web page (typically people in academia outside CS). It's a single PHP file that uses SQLite to store the CMS content and is able to build static HTML files whe you want to publish your website. It was originally run from my own server where I created accounts manually each time someone asked me to. But it now is available for larger use and is easy to install as well.
I wanted to extend markdown with latex and graphviz, and wrote an extensible markdown that grabs output from code execution (https://github.com/smasher164/mexdown).
Doesn’t mean Hugo, 11eventy, Jekyll etc aren’t good it’s just an area that could still have mass improvement I think
The first one was some kind of macro processor to insert text in files. I do not remember the syntax, but in each file it was like: %headermacro% ...the content... %footer%. Then the macros were replaced by layout data, turning each file into a generated static page.
Then I realized that is the wrong way around. All the content files had the same macros. It is very redundant. So I wrote a new one, that did it the other way around. Now there was a layout file with placeholders {{$WRITEX content/data}} and it would load the data from an XML content file and insert it in the layout file to generate the output file. And it had loops and conditions. Afair I could even write my "Makefile" in that language.
Then I wrote a webscraper and implemented XPath 2, because nothing else supported that. Then XPath 2 is very close to XQuery, so I implemented that as well. Now XQuery is very similar to the second generator, but with a w3 standarized syntax. The above insert expression simply becomes {content/data}. So now I use the webscraper as site generator.
So I currently have my own generators written in .Net Core, Go, Node, Python, Ruby, and PHP.
Of them all, Go is by far the fastest, Node the simplest, .Net Core the cleanest, PHP the hackiest, Python the nicest, but Ruby the most balanced in all those areas.
The Ruby one uses Markdown with YAML frontmatter, all in one script file that can be dropped directly into any folder and committed as part of my sites.
If you're curious (and remember the Ruby one is not the best) you can see it in github at https://github.com/kcartlidge/ruthless and running my site at https://kcartlidge.com (using Netlify).
Eventhough markdown is popular, its not a standard speicification. If author wanted to work with something which works for generating output for a website, pdf file or an ebook, he should have looked into a proper standard like RestructuredText (RST).
I think rst is much cleaner and better specification than markdown. Given its a specification there are multiple implementations available. So may be community should adopt it with more open arms. Still wonder like xsd vs relaxng in this case also markdown being inferior implementation without specification is winning over rst (restructuredText) format and specification.
I still use it for my module documentation, but I still use Jekyll for my personal sites instead of something like Sphinx (which is another element of hell for me - the errors when your RST is wrong are worse than useless;
> "You have an error somewhere between _here_ and _here_"
"You just pointed to everything"
> "Yes I did"
It's quite niche, though.
And if you don't want the Ruby dependency, AsciiDoctorJ is very usable.
Not mine, but a (nice) demo: https://www.mentalnodes.com/ and more advanced guide: https://nesslabs.com/tiddlywiki-static-website-generator
Basically it was a PHP website where there would be snippets of PHP at the top of the page followed by html with the occasional `<?php echo $var; ?>`.
Then I wrote a simple bash script that curled the website from localhost and saved everything to a local folder (kind of like how Jekyll has _site). Then the script used the neocities gem [1] to automatically upload.
I now use Jekyll, but I've thought about making my own because only I will use it and I'll understand it from top to bottom. With Jekyll whenever I need to make a change I have to re-read the docs to figure out what I need to do.
Same. I think lots of people would appreciate this. Writing is naturally private (in draft) before it may become public (published).
I am tempted to follow the author's path. But I don't think I need to go the whole way. I just need a preprocessor of sorts. I have a folder of markdown files I edit with Zettlr. I could flip a `published: true` flag in the frontmatter. Then iterate over each file and conditionally copy them somewhere to be dealt with by an off-the-shelf SSG (11ty is the current favourite).
We had two main issues... one is that links to dynamic pages would need to either be rewritten to static ones, or the dev ops would have to our 301 redirects for the list of dynamic URLs which we rendered, to the corresponding static html pages.
The other issue is that various aspects of the page would be related to authenticated sessions, and should NOT appear in the static site generated resources.
Any ideas about these issues?
Another alternative is to have your generator create an nginx config with the redirects. (W3 Total Cache for Wordpress does something like this)
As for hiding certain element, maybe appending some simple css to the generated pages to hide them is enough? something like `.login-form {display: none !important;}`
* Extract meta data like categories
* TOC generator
* Building a search index for Algolia
I used 11ty for a project recently and liked it a lot. It's very simple and the nice thing is that it understands several templating languages out of the box. You can basically mix them as well if you want or simply pick your favorite. Would use again: https://www.11ty.dev/
The trade offs between any SSG and platforms like WordPress or Twitter are clearer. They sit on a spectrum of predefined needs, and figuring out what matches your needs is the key (and also the cause for analysis paralysis).
The author is pedantic and seems to just not want to use other tools. That’s fine. But this is a dumb road for others to follow.
Remember the entire thing started because they don’t like front matter. They could literally have written a script to parse front matter, which are literally a few plain text lines.
Then they wanted asciidoc but without the dependency. Except their own code will likely have an asciidoc dependency unless they wrote their own entire asciidoc parser. In which case they could also have done this and plugged it into the same script.
“You don’t want to get stuck into the thinking of others,” they scream into the wires, followed by hobbling themselves into the most minimal viable product straight jacket imaginable.
Fool programmers and their time are soon parted.
I started down the Go template route and quickly realized that pandoc + some other standard tools got me everything _I_ needed. It was a great learning experience for sure!
I had no idea what I was doing. Didn't look at how other SSG's work... I just made some template HTML files, stuffed some content in YAML/Markdown frankenstein files, and then used JavaScript string.split().join() calls to stitch together the output files
It worked! It is very satisfying to have one build command that generates the entire site and pushes it to gh-pages for hosting.
Some years ago everybody was using Apache, a terrible implementation
Without backing that statement up and explaining why. Your post came across as simple flame-bait. Nevertheless, please keep posting. Contrary opinions count. Without them, the world becomes the ultimate echo chamber.
It is famously mod_php that isn't good in this regard and will naively start one interpreter per request. Since PHP 5.4 (released in 2011) everyone should just use PHP in FPM mode instead. The fact that mod_php only supported preforking caused a lot of spite and angst against PHP the language, which was understandable but perhaps a bit misguided.
None of this has anything to do with Apache, of course. No web server should have any trouble serving static pages at any speed and you could do much worse than Apache.