Why are video games graphics challenging? Productionizing rendering algorithms
bartwronski.com
bartwronski.com
What’s worse is that all of these techniques across all of the mentioned areas are evolving and changing as hardware changes. It’s impossible for one person to keep up anymore.
The art pipeline is pretty much cookie cutter from films with an added step of simplification of the mesh and baking normals. The rest is up to the engine and graphics programmers to get working as intended, the downstream of this is the design team.
It’s gotten easier in the animation department. Easier in the texturing department. Easier due to painting and PBR, but it’s gotten exponentially more difficult in the core engineering, engine level department. It’s a murder pit of the mind.
I’ve written a few game engines. I don’t write game engines.
How do you measure "complex", anyway?
So the hard part is to make aproximations that run in REAL TIME (unlike scientific simulations, that have some time to run).
And how do you measure complex? Well, how about the number of changing variables involved?
Simulating and solving chess is very easy in comparison. You have only very limited and fixed variables. Simulating a physic sandbox game with advanced chemistry and even biology that is supposed to run in real time - seems impossible at this moment on the other hand. Too many connected details.
I'm trying to think of an example, but I believe I remember reading about a game that purposefully used a slightly lighter gravity for some models to help the player feel more powerful (?) than other characters. I wish I could remember, but the idea is the same. Physics that are dynamic could be a fun/useful gameplay mechanic and I hope we don't see those types of games go away because everyone decides it's too complicated/not worth it to use a different physics algorithm than the one shipped in some console SDK.
Depends what you want from them. I've written a small quest and dialog system for my indie game (never finished, as most of my side projects :( ), and it was quite involved.
I used a graph programming library ( https://github.com/ajuc/pefjs ) to create graphs of nodes where each edge is a blocking condition (is the $MONSTER alive? is player near $NPC?) and each node is next step in a quest (with some actions if needed).
Then there was a in-memory database allowing to record arbitrary metadata that quests might require. Stuff like "have the player killed that monster", "does the player have the quest item equipped", "have the player talked with one of these 4 people about X", "has any NPC seen the player near the $PLACE during $EVENT".
I briefly considered adding gossips - if $NPC1 seen player commit murder and $NPC2 meets $NPC1 - then $NPC2 remembers that too, and remember who told them. But then I calculated how much memory that would need and it's staggering. And players won't notice anyway most likely. But it would be so cool.
Once you start recording data about data (does $NPC2 know about the fact that $NPC1 seen player commit murder?) it's hard to know where to stop.
Anyway, even without these - there's lots of that stuff and it needs indexing and filtering. I considered using some in-memory SQL database or datalog, but eventually just rolled my own with basic hashmap indexing.
Then there was dialog system and I wanted more variety than the same responses every time, so I created sets of possible greetings and goodbyes, confirmations, negations, reaction to repeatedly asking the same question, etc. Then the system chooses randomly from the list (each "kind" of NPC has separate list - one for working class characters, one for strangers, one for scholarly types, and special lists for important NPCs).
You can also parametrize dialogs with data from quest database - so you can for example ask the player about his new $COLOR $BRAND vehicle when he comes to see the NPC the first time after the purchase.
There's lots of interesting stuff to do there, even if it's not visual.
I did 3d graphics too even some shaders for commercial projects (not games - enhanced vision devices for partially blind people). I did some basic (archais by now) 3d rendering techniques (BSP trees, portals, quadtrees) in my hobby games.
I don't think one is inherently harder or more cool than the other.
You can look at Dwarf Fortress for gameplay/quest/dialogs and Cyberpunk 2077 for graphic, and it would be as cool and hard as each other IMHO.
Then when I check if NPC x knows about EVENT y I check who he met between time of the event and now and look at importance (witnessed murder would be high) and if there's a path for the information to pass - then he knows.
But it's too much work to do in every "tick" of the quest process. It should only be checked when something changed (someone who knows meets someone else).
It became too complicated system for just a few usages and some background gossip. But I think for a big RPG it could be great. Would allow more reactive world and more interesting quest design.
I think the main reason AAA games don't do this is voice overs. If we had good emotional speech synthesis a lot of new possibilities would open. So indies have an advantage here because text-only is still accepted.
I think where such a system would shine is in background banter. The player has to observe the NPCs telling each other these facts so he can appreciate it. That also opens the door for the player stepping in to prevent the spread of information. With the right setting that could be a major game mechanic
I think we are getting here with natural language understanding.
Rather having to write statements and handle the logic on a per-agent basis. We could keep a database of events with a brief description of what occurred, and use some NLU engine to use that description to form statements and opinions on the subject at hand.
It could even incorporate sentiment by having favorably weights between groups. So let's say, elves hate dwarves, and an event occurred where "Belegost (dwarven city) was razed (terrible action) by armies of Gondolin(elven city." The event name would run though a NLP system to which would make elven characters see the event more favorable than dwarven ones.
You could even model the spread of information.
At the rate the industry is going, I give you a whopping 1 in 1000 chances of matching your current wage making games.
The only people making real money are: a) corporate execs, b) people who get lucky with a big indie hit on Steam/etc, c) people lucky enough to work at a company with profit sharing (very rare)
The comment your replied to make it clear they find game programming interesting and they have a passion for it.
What on earth does making the same wage have to do with it?
If we can make a liveable wage doing what we enjoy, there are scores of people that couldn't care less about a massive wage.
The fact that, on Earth, passion only gets you so far. And it can easily wane off, given bad circumstances.
And of course, on Earth at least,
(a) "passion to create AAA games"
is not the same as
(b) "passion to be an employee on a AAA game company, with impossible schedules, idiotic management, unpaid overtime, and lower salary that I could get elsewhere".
In the end, whatever their passion, the AAA game developer is just a cog in a team (the head / driver of the team would be 1 in 20 or 1 in 50 devs).
I guess “we” pay for the fun with lower salaries and/or poorer work life balance. At least sometimes, eh hem.
The only gotcha is you can't really transfer those skills into other jobs, and the market for computational physics is rather sparse.
Of course, there are also super interesting and technically challenging jobs in ML out there.
Game programming maybe.
For game playing, I'd prefer games focused on interesting gameplay and world-building, rather than the 1000th derivative game in a 3D-graphics race...
Just like having a database holding DNA data that a robot in a chemical lab needs to get hold of in less than X ms for the planning software not to go astray when locating where to get the plate from.
It is a matter to look for challenges.
Thanks to demoscene and early industry contacts I have seen the other side of the garden, and decided it wasn't that green after all.
The amount of complexity that was slowly introduced for the sake of better graphics makes entering the field now a daunting task.
I have about a dozen books, each thousands of pages long. People who have been writing games since the 90s added this knowledge incrementally. Learning it now is such a mess.
I am partially disappointed with myself, partially with the state at which the field has arrived.
Don't forget technical artists, who implement and use tools to make art content run well on game engines. This could mean optimizing shaders, tweaking content streaming, tweaking occluders, etc.
Two examples.
https://store.steampowered.com/app/417290/Ghost_of_a_Tale/
Made almost entirely by one person
https://store.steampowered.com/app/242760/The_Forest/
Small team
There are tons of others. Sure, someone had to solve those issues but they mostly packaged the solutions into Unreal, Unity, etc.
It's like concentrating on making cameras instead of making movies. Sure there are new techniques that might give your movie some special edge (like when they invented bullet time for The Matrix) but a good story (or a well designed game) doesn't really need the technical tricks. In fact some of the most famous games are designed around the limits of the tech (either the limits of the hardware or the limits of their particular engine).
Note I've written several AAA game engines and have friends that work on AAA engines for others. They are all admitting / lamenting now that engines are basically a commodity and their skills are not needed. They're turning into scripters for the artist and designers and management is pushing to just license the engines.
1) Using an existing product like Unity or Unreal Engine or GameMaker. 2) Using their own engine but with very basic graphics.
"Ghost of a Tale" uses Unity. "The Forest" uses Unity.
So yeah, it's not "that hard" if you use a product which has already had millions of developer hours put into it like Unity or UnrealEngine.
Take away any prepackaged game engine and see if either of those two developer teams would have been able to make those games. :)
There are engines that will allow you to do a 3d FPS and make it look very good without ever having to worry about any low level barycentric coordinate issues.
Isn't that a good thing?
"When you're an adult you won't always have a calculator with you !"
Do you make the same comment about someone using standard libraries or frameworks? How productive would you be without system libraries or even an OS?
I was more making the point that starting by building an engine is a huge commitment and you need the skills and times (and the reason) to do that.
> So yeah, it's not "that hard" if you use a product which has already had millions of developer hours put into it like Unity or UnrealEngine.
What point are you making? The GGP correctly claimed it's "not hard" for small teams to make games. The games they referenced were made by small teams. The fact those games used an off the shelf game engine is immaterial. There's no "point" to make about it.
Your "point" comes off as a qualitative judgement about those developers using an off the shelf engine. If you've got no problem with someone using Unity what point are you really making?
Using someone else's hard work doesn't mean that its easy, just that you don't have to do that work yourself because others have done it for you.
> Your "point" comes off as a qualitative judgement about those developers using an off the shelf engine.
I didn't read it like that at all. In most cases, for most teams (certainly small ones), using an off the shelf engine is absolutely the right call, exactly because it means you don't have to solve the hard graphics problems yourself. But using an off the shelf engine does not mean that those hard problems don't exist or aren't hard, just that you can outsource them to the engine vendor.
From the point of view of a game developer, having good graphics is "easy" because Unity and Unreal exist. The fact Unreal and Unity exist enables thousands of developers to make games that couldn't otherwise. Making a "point" that a small game developer didn't write their entire stack is just shitty gatekeeping.
But this is not what the article, and this comment section, is about.
Unity is popular with the indies, but Unreal and Frostbite have powered games with collective sales of several hundreds of billions of dollars.
For a truly small games developer leveraging modern technology, look at Wube, the studio behind Factorio. Or Intoversion, the creators of Prison Architect. I think they each still have less than 20 people. Those games are not graphics beasts but they do rely heavily on modern CPU/memory speeds.
More telling, could they have sold better with a massive marketing campaign?
I don't think a lot of the complexity that goes into graphics is essential, though.
I do think top end will remain top end. However, I also think many games can be done without putting graphics rendering at the top of the budget. Axiom Verge is a recent find of mine that I feel hits this well. Thimbleweed Park is another good one here.
Which is not to say that either of those were easy. I doubt they were.
But I disagree with the higher level idea - that this means that things become simple. It's the other way around - they became so complex that a few person studio cannot write a new, general engine (see how many people programmed Doom or Quake which were state of the art back then!). At Unity there are a few thousands of just programmers working on the engine (yes!). This is not FAANG size, but I don't know if there is a team at FAANG that has a few thousands of engineers working on a single product. You need a team of this size to make a multi-platform, universal engine that is still not good enough to make AAA or open-world games.
Also my personal experience at my last games job (I left games) was the complete opposite of what you suggest as "engines being commodities" (at least this is at odds with you writing that they are working on AAA engines- sounds like they might be "using" engines and not writing in-house ones?). At Sony Santa Monica we had a team of ~30 programmers, out of which ~8 graphics people (just for a single platform and a single game) and I was constantly frustrated with how impossible it was just to catch up on all necessary state of the art techniques with a team of that size.
There are people much smarter than me who spend for example a half of a year on something as obscure as "multiscattering rough diffuse BRDF" - and a larger engine has hundreds of "features" like this.
My teenager doesn't need to know about cylinders and pistons and fuel injection and air-fuel ratios and timing and radio reception and electronic motors and A/C compressors and alternators and power brakes and traction control and ...
Just needs to know the interfaces to control and maintain the car.
We've made driving so simple that even a 16-year-old can do it.
https://web.archive.org/web/20201228071137/https://bartwrons...
I'd like to add to this that another reason game graphics are challenging is schools don't teach it. If you get a compsci degree and focus on video game graphics, the most you'll likely do in the courses is make an incredibly unoptimized software raytracer render some shiny spheres, a task that could not be farther from actual game engine development.
With general programming GPU, like CUDA, we can have the flexibility of software and the performance of traditional hardware pipelines.
Hopefully we will soon be able to add some neural networks in the rendering pipeline. Instead of computing expensive light effects, we could be faking them. For example the foliage could probably be handled better by just rendering a semantic mask indicating where the grass is, and let the neural network do some in-painting.
Neural rendering seems like a great way to get some speed-ups. On a powerful machine you render with High quality (eventually cinematographic quality), and you render the same scene with very low quality, and you train a neural network to convert the low quality into the high quality. At game time, you render low-quality and predict with the neural network the high quality image. It's a trade-off of sacrificing accuracy to gain speed but because humans are bad at evaluating accuracy, it's poised to be worth it.
The additional advantage of using neural rendering is that you don't need to spend as much creating the assets. You can get rid of one of the biggest mistake in 3D history, the invention of the triangle mesh. Representing objects as a list of textured triangle is just a bad representation. The main reason they are bad is that they are 2D surfaces in a 3D world which introduce some geometrical constraints that need to be solved explicitly when you deform the object, or you will get some artifacts
Alternative representations like dense or sparse point-clouds either in feature-space or color-space, or implicit fields representation don't suffer from these. Their main issue was the need to be converted back to triangles for rendering. These representations are continuously deform-able, which means you can use machine learning to infer them.
Lighting in games is also benefiting from sparse voxel based global illumination.
GPUs are very efficient at pushing triangles and the tooling and skill to build meshes is well established. Games like Teardown are showing alternative ways but when development time and budgets are so tight most games will use what’s there rather than risk new technologies.
This is also how Trespasser dealt with far geometry in 1998 https://en.wikipedia.org/wiki/Trespasser_(video_game)#Develo...
Started top be used again with introduction of directx 9.0 under a name Impostors http://www.tommironty.fi/screenshots/Imposters/dynamic_2d_im... https://software.intel.com/content/www/us/en/develop/article... https://developer.nvidia.com/gpugems/gpugems3/part-iv-image-...
here at work in COD warzone: https://www.youtube.com/watch?t=691&v=mvdTtl27TpM
Temporal AA is the most general and mature solution, but at the same time, many gamers hate it due to introducing some blurriness through imperfect resampling (and makes some very small population motion sick) - just web-search for "temporal antialiasing reddit" and see the sentiment of many users.
Maybe it technically could push more polygons, but it looked like crap.
But yeah, I agree that their tech seems to be a clever way to index and access large amounts of point cloud data, allowing them to stream from disk just what is needed for the current view -- a clever database more or less.
But for all the claims they made about how it will revolutionise everything, their demo's were pretty damn bad.
My "way in" was to just accept a generic offer and then later switch team to work inside Research (when you don't need to pass N layers of "recruiter abstraction" and can talk with colleagues and present your past work freely it becomes much easier). But still, among my collaborators maybe 5% don't have a PhD?
All of this was a surprise to me and made me question my life path and life choices. I'm originally from Poland in Eastern Europe and the only reason to consider a CS PhD would be if you wanted to teach. And I even considered it, but never occurred to me to do it for any kind of R&D work - mostly because of quality of Polish academic "research" being very low, much worse than the industry. And I am still in lots of self-doubt and questioning if I belong in this environment. Questions like "who was your advisor / what was your grad group?" when introducing yourself to new people become almost micro-aggressions. Also my focus on R&D ("solving unsolved problems") is very different than academic incentives ("use PhD interns to do projects, then write papers and get citations").
Brilliant creators of Dreams didn't care if software like Maya or 3ds Max are not compatible with their approach - as it wasn't the target audience. They didn't care for scalability of the tools, pipelines, and the renderer in terms of business practices, parallelism, finishing on time, reusing existing technology, packing with certain features. If your core feature is something unique like this, many other constraints can just go away. Is it the future of rendering? I don't think so. Btw. Media Molecule's Alex Evans (amazing person, he used to create beautiful art for demoscene) has long history of innovative rendering tech - like LBP used lots of voxel and volumetric representations (look for "Voxels in LittleBigPlanet 2" Siggraph talk) on PS3 and it also didn't catch up. Different use-cases = different constraint.
See also another example I mentioned in the post - Claybook, it uses SDFs for rendering representation (and physics!), but is also a very unique and special case.
And btw. artists absolutely hate modeling with nurbs, splines and anything similar. :) They love sculpting in ZBrush, or editing polys directly (for man-made and city objects). SDFs are +/- compatible with it (but come at huge memory and disk space costs - think of it this way, with meshes you encode "surface", so O(N^2) complexity, with SDFs volume, O(N^3), implicit representations not really...