Pixar's Render Farm
twitter.com
twitter.com
Cores and memory were indeed what mattered, and as long as frames could be fitted into memory then the machine was left to render the frame and then save it back out to shared disk. There was no virtualization, just raw compute hardware running at full capacity 24/7. The scene was already split into different layers and rendered as sequences of frames that were then composited into final products.
First job, first mini-datacenter in a new building, and there I am asking contractors to knock a hole in the building to fit air handlers due to the heat we were expecting to generate. Was great fun.
It taught me a lot about automated systems management.
Whats the hardware? How much electric energy goes into rendering a frame or a whole movie? How do they provision it (as they keep #cores fixed)? They only talk about cores, do they even use GPUs? What's running on the machines? What did they optimize lately?
So many questions! Maybe someone from Pixar's systems department is reading this :)?
Usually in crunch time if there's not enough nodes in the render farm they might rent nodes connecting them to their network for a period of time, or they might use the cloud, or they might get budget to increase their render farms.
From what I've seen the Cloud is extremely expensive for beefy machines with GPUs, but, you can see that some companies use it if you google [0] [1].
GPUs can be used for some workflows in modern studios but I would bet the majority of it is CPUs, those machines are usually running a Linux distro and the render processes (like vray / prman , etc.). Everything runs from a big NFS cluster.
[0] https://deadline.com/2020/09/weta-digital-pacts-with-amazon-...
[1] https://www.itnews.com.au/news/dreamworks-animation-steps-to...
Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) performance. "Industrial" grade hardware (Xeon/Epyc + Quadro) it's under 12 months. We chose consumer grade bare metal.
On thing that was half surprising, half calculated in our decision was despite the operational overhead how much less stressful running your own hardware is. When we ran experimentally on the cloud, a misrender could cost us 900 euro, and sometimes we'd have to render 3 times or more for a single client. Bringing us from healthily profitable to losing money. The stress of having to get it right the first time sucked.
So no fancy answer, our 3D experts got TeamViewer access to the nodes running Windows Pro. Sometimes our renders fail on patch Tuesday because I forgot to reapply the no-reboot hack.
We're professionalizing now at 12 nodes, we got to the point where the 3D experts don't need to TeamViewer in, so we're swapping them to headless Linux. No idea on the update management yet, but they're clean nodes running Ubuntu server.
If you have a steady load cloud makes little sense. It only makes sense if you have a tight deadline (as is not that uncommon with video and VFX) and can't fit it within your deployed capacity.
They’ve been working on a GPU version of RenderMan for a couple of years.
https://renderman.pixar.com/news/renderman-xpu-development-u...
> Whats the hardware?
It varies. They have several generations of equipment, but I can say it was all Intel based, and high core count. I don't know how different the render infra was to the workstation infra. I think the total core count (aggregate of render, workstation, and leased) was ~60K cores. And they effectively need to double that over the coming years (trying to remember one of the last meetings I was in) for the productions they have planned.
> How much electric energy goes into rendering a frame or a whole movie?
A lot. The render farm is pretty consistently running at high loads as they produce multiple shows (movies, shorts, episodics) simultaneously so that there really isn't idle times. I don't have numbers, though.
> How do they provision it
Not really sure how to answer this question. But in terms of rendering, to my knowledge shots are profiled by the TDs and optimized for their core counts. So different sequences will have different rendering requirements (memory, cores, hyperthreading etc). This is all handled by the render farm scheduler.
> What's running on the machines?
RHEL. And a lot of Pixar proprietary code (along with the commercial applications).
> They only talk about cores, do they even use GPUs?
For rendering, not particularly. The RenderMan denoiser is capable of being used on GPUs, but I can't remember if the render specific nodes have any in them. The workstation systems (which are also used for rendering) are all on-prem VDI.
Though with RenderMan 24 due out in Q1 2021 will include RenderMan XPU, which is a GPU (CUDA) based engine. Initially it'll be more of a workstation facing product to allow artists to iterate more quickly (it'll also replace their internal CUDA engine used in their propriety look-dev tool Flow, which was XPU's predecessor), but it will eventually be ready for final-frame rendering. There is still some catchup that needs to happen in the hardware space, though NVLink'ed RTX8000's does a reasonable job.
A small quote on the hardware/engine:
>> In Pixar’s demo scenes, XPU renders were up to 10x faster than RIS on one of the studio’s standard artist workstations with a 24-core Intel Xeon Platinum 8268 CPU and Nvidia Quadro RTX 6000 GPU.
If I remember correctly that was the latest generation (codenamed Pegasus) initially given to the FX department. Hyperthreading is usually disabled and the workstation itself would be 23-cores as they reserve one for the hypervisor. Each workstation server is actually two+1, one workstation per CPU socket (with NUMA configs and GPU passthrough) plus a background render vm that takes over at night. The next-gen workstations they were negotiating with OEMs for before COVID happened put my jaw on the floor.
ex VFX sysadmin here. I'm not sure if they use their own scheduler or not. IF they do, they use tractor(might be tractor 2 now), which looks after putting the processes in the right places. Think K8s, but actually easy to use, well documented and reliable. (just not distributed, but then it scales way higher and is nowhere near as chatty)
They would have a whole bunch of machines, some old some new, some with extra memory, for particle sims, some with extra cores for just plain rendering. Each machine will be separated into slots, which are made up of a fixed number of cores. Normally memory is guarded but CPU is not (ie, you only get 8 gigs of ram, but as much CPU as you can consume. Context switching the CPU is fast, memory not so much.) I'm not sure on how pixar does it, but at a large facility like ilm/framestore/dneg the farm will be split into shows, with guaranteed minimum allocation of cores. this is controlled by the scheduler. crucially it'll be over subscribed, so jobs are ordered by priority.
As for actual hardware provisioning, thats quite cool. In my experience there will be a bringup script that talks to the iLo/iDrac/other management system. When a machine is plugged in, it'll be seen by the bringup script, download the xml/config/other goop that tells the bios how to configure and boot from the network, connect to the imaging system and install whatever version of linux they have.
As for power per frame, each frame will be made up of different plates, so if you have a water sim, that'll be rendered separately, along with other assets. These can then be combined afterwards in nuke to tweak and make pretty without having to render everything again.
That being said, a crowd shot with lots of characters with hair, or a water/smoke/ice effect can take 25+ hours per frame to render. So think a 100core/thread machine redlining for 25 hours, plus a few hundreds TB of spinny disk. (then it'll be tweak 20ish times)
optimisations wise, I suspect it's mostly getting software to play nice on the same machine, or beating TDs to make better use of assets, or adjusting the storage to make sure its not being pillaged too much.
Cheers
I moved to a large profitable financial news paper, which had cute scaling issues (ie they were all solved, so engineers tried to find new and interesting ways to unsolved them )
I then moved to a startup that made self building machine readable maps, which allowed me to play with scale again, but on AWS (alas no real hardware). We were then bought out by a FAAMG company, so now I'm getting bored but being paid loads to do so.
Once the golden handcuffs have been broken, I'd like to go back, but only if I can go home at 5 every day...
I find VFX really fun as far as job! Sometimes I do think about leaving mostly for Pay reasons but the pay has been decent enough recently (basically FAANG base pay without RSU/Bonus...).
It is interesting how we have a lot of big scale problems that goes unrecognized, I find the problems really challenging. Compared to when I worked in the software industry we had a team 10x as big for a problem 100x simpler.
Outside of some big tech companies, biology, oil industry and finance, I cannot imagine many companies having such a big scale on number of cores/memory/disk.
Working in Pipeline I haven't found crazy hours yet, has been mostly a 8h/day job that I can disconnect after I am done. Also with Covid some people even switched to 4 days weeks which is quite interesting.
Anyho, thanks for sharing your perspective!
It works, and roughly does what it says on the tin, but most of the bigger studios (other than MPC and DNeg who do use it) have better custom solutions.
From my experience (animated movies) GPU is still very experimental because of how limited it can scale, and definitely not use in render farm.
And I'm not even talking about the cost.
GPU rendering demos focus on speed, but of the biggest problem with full feature is flexibility. The more complex your image is, the more problems/artifacts you will "create" on it. Your render time can be x100 faster, if you need to spend two days to fix a problem for each shot, the quality VS speed ratio completely fall over.
Everything get easier, slowly, so maybe one day we will have a 100% GPU farm on big budget projects, but for now, CPU is the most predictable way to manage large scale rendering for both sides (sysadmin/artists).
https://mobile.twitter.com/benedictevans/status/766822192197...
> FreeBSD Used to Generate Spectacular Special Effects
For the then upcoming movie The Matrix
> Manex Visual Effects used 32 Dell Precision 410 Dual P-II/450 Processor systems running FreeBSD as the core CG Render Farm.
[0]: https://www.freebsd.org/news/press-rel-1.html
edit: added that it was for The Matrix.
https://www.slashfilm.com/cool-stuff-a-look-at-pixar-and-luc...
Basically though, Pixar doesn't have the scale to make custom chips (the entire Pixar and even "Disney all up" scale is pretty small compared to say a single Google or Amazon cluster).
Until recently GPUs also didn't have enough memory to handle production film rendering, particularly the amount of textures used per frame (which even on CPUs are handled out-of-core with a texture cache, rather than "read it all in up front somehow"). I think the recent HBM-based GPUs will make this a more likely scenario, especially when/if OptiX/RTX gains a serious texture cache for this kind of usage. Even still, however, those GPUs are extremely expensive. For folks that can squeeze into the 16 GiB per card of the NVIDIA T4, it's just about right.
tl;dr: The economics don't work out. You'll probably start seeing more and more studios using GPUs (particularly with RTX) for shot work, especially in VFX or shorts or simpler films, but until the memory per card (here now!) and $/GPU (nope) is competitive it'll be a tradeoff.
High end VFX/CG usually tessellates geometry down to micropolygon, so you roughly have 1 quad (or two triangles) per pixel in terms of geometry density, so you can often have > 150,000,000 polys in a scene, along with per vertex primvars to control shading, and many textures (which can be paged fairly well with shade on hit).
Using ray tracing pretty much means having all that in memory at once (paging sucks in general of geo and accel structures, it's been tried in the past) so that intersection / traversal is fast.
Doing lookdev on individual assets (i.e. turntables) is one place where GPU rendering can be used as the memory requirements are much smaller, but only if the look you get is identical to the one you get using CPU rendering, which isn't always the case (some of the algorithms are hard to get working correctly on GPUs, i.e. volumetrics).
Renderman (the renderer Pixar use, and create in Seattle) isn't really GPU ready yet (they're attempting to release XPU this year I think).
Can you speak to any competitive advantages a vfx-centric gpu cloud provider may have over commodity AWS? Even the RenderMan XPU looks to be OSL / Intel AVX-512 SIMD based. Thanks!
Supercharging Pixar's RenderMan XPU™ with Intel® AVX-512
And GPU rendering has been gaining momentum over the past few years, but the biggest bottleneck until recently was availabe VRAM. Big budget VFX scenes can often take 40-120 GB of memory to keep everything accessible during the raytrace process, and unless a renderer supports out-of-core memory access, then the speed up you may have gained from the GPU gets thrown out the window from swapping data
Uncompressed, it’s 93Gb of render data, plus 130Gb of animation data if you want to render the entire shot instead of a single frame.
From what I’ve seen elsewhere, that’s not unusual at all for a modern high end animated scene.
Didn't you guys get the memo from B. Gates that no one will ever need more than 640k?
I disagree with this takeaway. But full disclosure I’m biased: I work on OptiX. There is a reason Pixar and Arnold and Vray and most other major industry renderers are moving to the GPU, because the trends are clear and because it has recently become ‘worth it’. Many renderers are reporting factors of 2-10 for production scale scene rendering. (Here’s a good example: https://www.youtube.com/watch?v=ZlmRuR5MKmU) There definitely are tradeoffs, and you’ve accurately pointed out several of them - memory constraints, paging, micropolygons, etc. Yes, it does take a lot of engineering to make the best use of the GPU, but the scale of scenes in production with GPUs today is already firmly well past being limited to turntables, and the writing is on the wall - the trend is clearly moving toward GPU farms.
So I'm well aware of the trade offs. As I mentioned, for lookdev and small scenes, GPUs do make sense currently (if you're willing the pay the penalty of getting code to work on both CPU and GPU, and GPU dev is not exactly trivial in terms of debugging / building compared to CPU dev).
But until GPUs exist with > 64 GB RAM, for rendering large scale scenes, it's just not worth it given the extra burdens (increased development costs, heterogeneous sets of machines in the farm, extra debugging, support), so for high-end scale, we're likely 3/4 years away yet.
It's not just about them existing, they need to be cost effective.
They are really expensive though. But chassis and rackspace also isn't free. If one beefy node with a couple GPUs can replace have a rack of CPU only Nodes the GPUs are totally worth it.
I'm not too familiar with 3D rendering but in other workloads the GPU speedup is so huge that if its possible to offload to the GPU it made sense to do it from a economical perspective.
If 8k gaming becomes a real thing you can expect work to be done towards a solution, but until then not so much.
Edit: added [desktop] preceding PCIE
The trend is pretty clear, though. The size of scenes than can be done on the GPU today is large and growing fast, both because of improving engineering and because of increasing GPU memory speed & size. It’s just a fact that a lot of commercial work is already done on the GPU, and that most serious commercial renderers already support GPU rendering.
It’s fair to point out that the largest production scenes are still difficult and will remain so for a while. There are decent examples out there of what’s being done in production with GPUs already:
https://www.chaosgroup.com/vray-gpu#showcase
Are GPUs starting to be used at earlier points in the pipeline? Yes, absolutely, but they always were to a degree in previs and modelling (via rasterisation). They are gradually becoming more useable at more steps in pipelines, but they're not there yet for high-end studios.
In some cases, if a studio's happy using an off-the-shelf renderer with the stock shaders (so no custom shaders at all - at least until OSL is doing batching and GPU stuff, or until MDL actually supports production renderer stuff) studios can use GPUs further down the pipeline, and currently that's smaller scale stuff from what I gather talking to friends who are using Arnold GPU. Certainly the hero-level stuff at Weta / ILM / Framestore isn't being done with with GPUs, as they require custom shaders, and they aren't going to be happy with just using the stock shaders (which are much better than stock shaders from 6/7 years ago, but still far from bleeding edge in terms of BSDFs and patterns).
Even from what I hear at Pixar with their lookdev Flow renderer things aren't completely rosy on the GPU front, although it is at least getting some use, and the expectation is XPU will take over there, but I don't think it's quite ready yet.
Until a studio feels GPU rendering can be used for a significant amount of the renders (that they do, for smaller studios, the fidelity will be less, so the threshold will be lower for them), I think it's going to be a chicken-and-egg problem of not wanting to invest in GPUs on the farms (or even local workstations).
The folks at Framestore and many other shops already don’t do more than XX GiB per frame for their rendering. So for me, this comes down to “can we finally implement a good enough texture cache in optix/the community” which I understand Mark Leone is working on :).
The shader thing seems easy enough. I’m not worried about an OSL compiled output running worse than the C-side. Divergence is a real issue, but so many studios are now using just a handful of BSDFs with lots of textures to drive, that as long as you don’t force the shading to be “per object group” but instead “per shader, varying inputs is fine”, you’ll still get high utilization.
The 80 GiB parts will make it so that some shops could go fully in-core. I expect we’ll see that sooner than you’d think, just because people will start doing interactive work, never want to give it up, and then say “make that but better” for the finals.
(I'm aware of the OSL batching/GPU work that's taking place, but it remains to be seen how well that's going to work).
From what I've heard from friends in the industry (at other companies) who are using GPU versions of Arnold, the numbers are no-where near as good as the upper numbers you're claiming when rendering at final fidelity (i.e. with AOVs and Deep output), so again, the use-cases - at least for high-end VFX with GPU - are still mostly for lookdev and lighting blocking iterative workflow from what I understand. Which is still an advantage and provides clear benefits in terms of iteration time over CPU renderers, but it's not a complete win, and so far, only the smaller studios have started dipping their toes in the water.
Also, the advent of AMD Epyc has finally thrown some competitiveness back to CPU rendering, so it's now possible to get a machine with x2 as many cores for close to half the price, which has given CPU rendering a further shot in the arm.
Isn’t there a better Vray or Arnold comparison somewhere?
As in my summary comment, an A100 can now run real scenes, but will cost you ~$10k per card. For $10k, you get a lot more threads from AMD.
An AMD Rome/Milan part will give you 256 decent threads on a 2S box with a ton of RAM for say $20-25k at list price (e.g., a Dell power edge without any of their premium support or lots of flash). By comparison, the list price of just an A100 is $15k (and you still need a server to drive the thing).
So for shops shoving these into a data center they still need to do a cost/benefit tradeoff of “how much faster is this for our shows, can anyone else make use of it, how much power do these draw...”. If anything, the note about more and more software using CUDA is probably as important as “ray tracing is now sufficiently faster” since the lack of reuse has held them back (similar things for video encoding historically: if you’ve got a lot of cpus around, it was historically hard to beat for $/transcode).
The reason is that those softwares need to be sold to many, and a big part of studios are doing advertise and series. GPU rendering is perfect for them as they don't need/can't afford large scale render farms.
About your example, that not honest. It's full of instances and perfect use case for a "Wow" effect but it's not a production shot. Doing a production shot required complexity management on the long run, even for CPU rendering. On this side, GPU is more "constrained" than CPU, management is even more complex.
Not only that, they are massive, kickout a whole bunch of heat in new and interesting ways. worse still they depreciate like a mofo.
the tip top renderbox of today is next years comp box. a two generation old GPU is a pointless toaster.
My understanding is that for neural networks, the speedup is much more than 4x. Does anyone know why there's such a difference?
OTOH, Vfx rendering involves a varying scene with moving light sources, cameras, objects, textures, and physics. Much more dynamic interactions. This is a gross simplification but I hope it helps.
This is no longer true, and hasn’t been for around a decade. This is a left-over memory of when GPUs weren’t using IEEE 754 compatible floating point. That changed a long time ago, and today all GPUs are absolutely up to par with the IEEE standards. GPUs even took the lead for a while with the FMA instruction that was more accurate than what CPUs had, and Intel and other have since added FMA instructions to their CPUs.
There's a huge I/O bottleneck as well as you're reading huge textures (I've seen textures as big as 1 TB) and writing constantly to disk the result of the renderer.
Other than that, most of the tooling that modern studios use is off the shelf, for example, Autodesk Maya for Modelling or Sidefx Houdini for Simulations. If you had a custom architecture then you would have to ensure that every piece of software you use is optimized / works with that.
There are studios using GPUs for some workflows but most of it is CPUs.
Note though these are not single 1TB textures, they’re multiple sets of textures, plus all of the layers that constitute them. Some large robots In particular had 65k 4K textures if you count the layers.
I know it doesn't makes sense to tell your clients that what they are doing is nonsense, but if I saw something like that going on, the first thing I would do is chase down why it happened. Massive waste like that is extremely problematic while needing to make a sharper texture for some tiny piece that gets close to the camera is not a big deal.
Over texturing like this can be a good decision depending on the production. Asset creation often starts a long time before shots or cameras are locked down.
If you don’t know how an asset is to be used it makes sense to texture all of it upfront as if it will be full screen, 4K.
Taking an asset off final to ‘Upres’ it for a can be a pain in the ass and more costly than just detailing it up in the first place.
In isolation it’s a insane amount of detail and given perfect production planning it is normally not needed, but until directors lock down the scripts and shots it can be the simplest option.
This was easier to rely on in the days before ray tracing, when texture filtering was consistent because everything was from the camera. Ray differentials from incoherent rays aren't quite as forgiving.
> If you don’t know how an asset is to be used it makes sense to texture all of it upfront as if it will be full screen, 4K.
4k textures for large parts of the asset in the UV layout can be an acceptable amount of overkill. That's not the same as putting 65,000 4k textures on something because each little part is given its own 4k texture. I know that you know this, but I'm not sure why you would conflate those two things.
> Taking an asset off final to ‘Upres’ it for a can be a pain in the ass and more costly than just detailing it up in the first place
It is very rare that specific textures need to be redone like that and it is not a big deal.
650GB of textures for one asset drags everything from iterations to final renders to disk usage to disk activity to network usage down for every shot in a completely unnecessary way. There isn't a fine line between these things, there is a giant gap between that much excessive texture resolution and needing to upres some piece because it gets close to the camera.
> Asset creation often starts a long time before shots or cameras are locked down.
This is actually fairly rare.
> In isolation it’s a insane amount of detail and given perfect production planning it is normally not needed, but until directors lock down the scripts and shots it can be the simplest option.
That's rarely how the time line fits together. It's irrelevant though, because there is no world where 65,000 4k textures on a single asset makes sense. It's multiple orders of magnitude out of bounds of reality.
I am glad that you have that insane amount of scalability as a focus since you are making tools that people rely on heavily, and I wish way more people on the tools end thought like this. Still, it is about 1000x what would set off red flags in my mind.
I apologize on behalf of whoever told you that was necessary, because they need to learn how to work within reasonable resources (which is not difficult given modern computers), no matter what project or organization they are attached to.
Take for example, a large hero asset like King Kong.
Kong look development started many months before a script was locked down. Kong is 60ft tall, our leading lady is 5’2”.
We think we need shots where she’ll be standing in Kong’s hands, feet, be lifted up to his face, nose etc.
So we need fingers prints that will stand up at 4K renders, tear ducts, pores on the inside on the nose, etc etc but we don’t know. All of which will have to match shot plates in detail.
We could address each of these as the shots turn up and tell the director (who owns the company) he needs to wait a few days for his new shot, or you can break Kong into 500 patches and create a texture for each of the diffuse, 3 spec, 3 subsurface, 4 bump, dirt, blood, dust, scratch, fur, flow etc etc inputs to our shaders.
Let’s says we have 500 UDIM patches for Kong so we can sit our leading lady on the finger tips, and 20 channels to drive our shaders and effects systems.
When working the artist uses 6 paint layers for each channel ( 6 is a massive underestimate for most interesting texture work).
So we have 500 patches * 20 channels * 6 layers which gives us 60k images. Not all of these will need be at 4K however.
For Kong replace any hero asset where shots will be more placed “in and on” the asset rather than “at”. Heli carriers, oil rigs, elven great halls, space ships, giant robots.... The line between asset and environment is blurred at that point and maybe think “set” rather than “asset”
We both know that stuff isn't showing up on film and that the excess becomes a CYA case of the emperor's new clothes where no one wants to be the one to say it's ridiculous.
> When working the artist uses 6 paint layers for each channel ( 6 is a massive underestimate for most interesting texture work).
This is intermediary and not what is being talked about.
I would say it's the opposite. There is nothing necessary about 10,000 4k maps and definitely nothing typical. Workflows trade a certain amount of optimization for consistency, but not like this.
> patronizing who actually works in the industry.
I don't think I was patronizing. This person is valuable in that they are trying to make completely excessive situations work. Telling people (or demonstrating to them) they are being ridiculous is not his responsibility and is a tight rope to walk in his position.
> Also speaking as an authority that knows best
If I said that 2 + 2 = 4 would you ask about a math degree? This is an exercise in appeal to authority. This person and myself aren't even contradicting each other very much.
He is saying the extremes that he has seen, I'm saying that 10,000 pixels of texture data for each pixel in a frame is enormous excess.
The only contradiction is that he seems to think that because someone did it, it must be a neccesity.
Instead of confronting what I'm actually saying, you are trying to rationalize why you don't need to.
Usually the way VFX works is that technology (R&D) is very moved away from production. The artist job is getting the shot done regardless of technology and they have very short deadlines. They usually push the limits.
Digital artists are not very tech savvy in a lot of disciplines, it is not feasible to have a TD in the delivery deadlines of the shots for a show.
The person at Weta also told you how Weta actually worked in Kong which is very typical. You don't know upfront what you need. And you dismissed it as something unnecessary, still, is how every big VFX studio works. Do you feel that you know better and/or everyone is doing something wrong and hasn't really thought about it? If that is the case you might have a business opportunity for a more efficient VFX studio!
> how Weta actually worked in Kong which is very typical
It is not typical to have 10,000 4k maps on a creature. What has been typical when rendering at 2k is a set of 2k maps for face, torso, arms and legs. Maybe a single arm and leg can be painted and the UVs can be mirrored, though mostly texture painters will layout the UVs separately and duplicate the texture themselves to leave room for variations.
> it is not feasible to have a TD in the delivery deadlines of the shots for a show.
Actually most of the people working on shots are considered TDs. Specific asset work for some sequence with a hero asset is actually very common, which makes sense if you think about it from a story point of view of needing a visual change to communicate a change of circumstances.
4k rendering (was the 2017 king kong rendered in 4k?) and all the closeups of king kong mean that higher resolution maps and more granular sections are understandable, but it doesn't add up to going from 16 2k maps to 10,000 4k maps. Maps like diffuse, specular and subsurface albedo are also just multiplicative, so there is no reason to have multiple maps unless they need to be rebalanced against each other per shot (such as for variations).
You still never actually explained a problem or inconsistency with anything I've said.
> Actually most of the people working on shots are considered TDs.
That's not true in the studios I've been. TDs is usually reserved to more close to pipeline folks that aren't doing shot work (as in, delivering shots). They're supporting folks doing so.
For the record, I haven't downvoted you at all.
This was created with the requirement that the director be able to use it at will. Closeups. Set replacements, destruction, the works.
You don’t have a shot breakdown or camera list.
You’ve got 6 months of pre production to support 1000 shots. Once in production you will be the only texture artist supporting 30 TDS.
How do you spend your 6 months to make sure production runs smoothly?
I’m kinda interested in your experience of this stuff as the numbers you’re quoting for 2k work are, in my experience, waaaaaay off and are closer to how a high end games asset would currently be textured.
I don’t disagree with you that the numbers involved are crazy when taken in isolation but it is (or at least was 5 years ago) a very common workflow at ILM, Weta, Dneg, DD, R&H Framestore etc etc. The quoted high numbers are the very upper end but many thousands of assets on hundreds of productions have been textured at what I believe you would consider “insane” detail levels.
You clearly understand many of the issues involved but downplay the complexity in running high end assets in less than perfect production.
Unless the industry has changed dramatically in 5 years shot changes, per shot fixes, variants (clean, dirty, destroyed), shader tweaks, happen on every single show I’ve ever been part of.
Render time and storage is one factor as is individual artist iteration, but the real productivity killer is an inter discipline iteration.
Going from a “blurry texture” note in comp to a TD fix to a texture “upres” is potentially a 5 person, 4 day turn around. I would trade a whole bunch of cpu and storage to avoid that.
Computers are cheap, people are expensive, people coordinating even more so.
So that can easily end up being several hundred gigabytes of source image data. At rendertime, only the textures that are needed to render what's visible in the camera are loaded into memory and utilized, which typically ends up being a fraction of the source data.
Large scale terrains and environments typically make more use of procedural textures, and they may be cached temporarily in memory while the rendering process happens to speed up calculations
An entire movie at 2k, uncompressed floating point rgb would be about 4 terabytes.
This is probably true today, but leaves the wrong impression IMHO. The clear trend is moving toward GPUs, and surprisingly quickly. Maya & Houdini have release GPU simulators and renderers. RenderMan is releasing a GPU renderer this year. Most other third party renderers have already gone or are moving to the GPU for path tracing - Arnold, Vray, Redshift, Clarisse, etc., etc.
I'd imagine another reason is that Pixar uses off-the-shelf Digital Content Creation apps (DCCs) like Houdini and Maya in addition to their proprietary software, so while they could optimize some portions of their pipeline, it's probably better to develop for more general computing tasks. They also mention the ability to "ramp up" and "ramp down" as compute use changes over the course of a show
[1] https://devmesh.intel.com/projects/supercharging-pixar-s-ren...
[2] https://nvidianews.nvidia.com/news/pixar-animation-studios-l...
https://graphics.pixar.com/library/SoulRasterizingVolumes/pa...
On a personal note, I had a pretty visceral "anti-" reaction to the movie Soul. I just felt it too trite in its handling of themes that humankind has wrestled with since the dawn of time. And jazz is probably the most cinematic of musical tastes. Think of the intros to Woody Allen's Manhattan or Midnight in Paris. But it felt generic here.
That said the physically based rendering is state of the art! If you've ever taken the LIE toward the Queensborough Bridge as the sun sets across the skyscraper canyons of the city you know it is one of the most surreal tableaus in modern life. It's just incredible to see a pixel perfect globally illuminated rendering of it in an animated film, if only for the briefest of seconds ;)
A GPU enabled version of RenderMan is just coming out now. I imagine their farm usage after this could change.
https://gfxspeak.com/2020/09/11/animation-studios-renderman/
I’m purely speculating, but I think the main reason they haven’t been using GPUs until now is that RenderMan is very full featured, extremely scalable on CPUs, has a lot of legacy features, and it takes a metric ton of engineering to port and re-architect well established CPU based software over to the GPU.
From a different perspective, think about supercomputers - many supercomputers do indeed do relatively specific things (and I would assume some do run custom hardware), but the magic is in the interconnects - getting the data around effectively is where the black magic is.
Also, if you aren't particularly time bound, why bother? FPGAs require completely different types of engineers, and are generally a bit of pain to program for even ignoring how horrific some vendor tools are - your GPU code won't fail timing for example.
But, very soon after I left Lucas, ILM started pushing ray tracing a lot harder. Getting good quality results per ray is very difficult. Much easier to throw hardware at the problem and just cast a whole lot more rays. So, they moved over to being heavily GPU-based around that time. I do not know the specifics.
Is any of that true?
I can appreciate that working well when people are in the office, but I'm amazed that worked out for them when people moved to work from home. I have trouble getting some of my engineers to have a stable connection stable enough for VS Code's remote mode. I can't imagine trying to use a modern GUI over these connections.
Seems very strange to me.
They're pretty great overall and the bandwidth requirements aren't crazy high but it does max out your data usage if you're capped pretty quickly. The faster you can be, the better the experience.
Some studios like Imageworks don't even have the backend data center in the same location. So the thin clients connect to a center in Washington state when the studios are in LA and Vancouver.
I was an intern in Pixar systems for 2019 (at Blue Sky now), and we're also using a mix of PCoIP and NoMachine for home users. We finally figured out a quirk with our VPN terminal we sent home with people that was throttling connections, but the experience post-that fix is actually really good. There are a few things that can cause lag (such as moving apps like Chrome/Firefox), but for the most part unless your ISP is introducing problems it's pretty stable. And everyone with a terminal setup has two monitors, either 2*1920x1200 or 1920x1200+2560x1440.
I have a 300Mbps/35Mbps plan (turns into a ~250/35 on VPN) and it's great. We see bandwidth usage ranging from 1Mbps to ~80 on average. The vast majority being sub-20. There are some outliers that end up in mid-100s, but we still need to investigate those.
We did some cross country tests with our sister studio ILM over the summer and was hitting ~70-90ms latency which although not fantastic, was still plenty workable.
Interesting contrast with other companies that switched to VDI where it made very little sense. VMware + server racks + zero clients compared to desktops never made economic sense, at the time. But oftent here is some other factor that tips things in VDI's favour.
The Teradici stuff was great, and from an admin perspective having everything located in the DC made maintenance SO much better. Switching over to VDI is a long term goal for us at Blue Sky as well, but it'll take a lot more time and planning.
Are you still using a similar sort of setup to orchestrate the backend of this, and if so have you published anything about it? I've had a few people ask me about this sort of problem lately and there aren't too many great resources out there I can point people new to this sort of tech towards.
Here at Blue Sky we are in our infancy for thin client based work. Remote terminals aren't too new as they were used for contract workers and artists who needed to WFH on the prior show, but we don't have VDI as we still use deskside workstations. For COVID, the workstations have been retrofitted with Teradici remote workstation hostcards and we send the artists home with a VPN client and zero client, utilizing direct connect. It was enough to get us going, but we have a long road ahead in optimizing this stack and eventually (if our datacenters can handle it) switching over to VDI.
My team at WDAS mirrored pretty closely what Pixar did with VDI although we didn't fully switch to it for different reasons (power and heat constraints in the datacenter and price). IIRC the VDI hosts had static VMs and the teradici connection manager did all the smarts of routing user requests to a VM. There was no dynamic orchestration for us because we only had 60ish users using full VDI VMs, but even our plans of hundreds of users was still to use teradici and standard VMs on each host.
We rendered different than Pixar which also made our system a bit more static. We didn't have a separate render VM and instead rendered directly on the workstation VM when users were idle or disconnected.
At my workplace we have a smallish HPC center and ended up moving off of NFS at about 2PB of storage since we were starting to hit the limits of NFS (think 1TB of RAM and 88 cores on a single NFS server).
At rhythm and Hues, you could request footage all the way back from the founding of the studio for example.
CG work is fairly IO intensive for tasks like rendering where you're reading hundreds or even thousands of geometry caches per frame. But for other things, your IO isn't as frequent since it's not about constant r/w as there are long computational or artist time between saves and reads.
Also none of the high end commercial filers are single node - they're all clusters of varying sizes.
We even have a reference platform spec for some kind of industry wide baseline: https://vfxplatform.com/
The studio internally uses PCOIP boxes, which I don't like due the added tiny delay (I'm a bit like those developers who complain about miliseconds of latency on their text editors...). Anyways, for the work for home setup, we are using NoMachine, which doesn't feel any different from the PCoIP boxes - unless if using the MacOS client, which is much laggier than the Windows or Linux versions.
Actually, I went ahead and tried installing Nomachine on Google Cloud and Amazon AWS CPU only instances, and got the same responsiveness of my studio setup. No fancy setups or gpu encoding/decoding.
So if you have a Nuke license, you can do some pretty heavy 2D vfx for about 1USD/hour on a 96vcpu machine (performance similar to an AMD 32core) and 196GB of RAM, even without any GPU acceleration.
NoMachine looks like a good answer for people like me. Thanks for the pointer, I'll check it out.
I am not surprised that they “make the film fit the box”, because managing compute expenditures is such a big deal!
(Edit: When I say "simple wall clock", I'm talking about the elapsed time from start to finish for rendering one frame, disregarding how many other frames might be rendering at the same time. Throughput != 1/latency, and all that.)
Of course I'm sure things like I/O and art-team-workflow hugely complicate the story at this scale, but I still doubt there's a meaningful concept of "wall-clock time for one frame" that doesn't change with the number of available cores.
At the standard 24fps it takes you 24 days per film second which works out to 473 years for the average 2 hour long film which can't be right.
That also doesn't include rendering multiple shots at once. It's all about parallelism.
Finally, those frame counts for a film only assume final render. There's a whole slew of work in progress renders too, so a given shot may be rendered 10-20 times. Often they'll render every other frame to spot check and render at lower resolutions to get it back quick.
My experience running massive pipelines is that there's a limited amount of parallelization you can do. It's not like you can just slice the frame into rectangles and farm them out.
Funny thing, you sure can! Distributed rendering of single frames been a thing for a long time already.
Obviously doesn't work for every frame/scene/project, only if the main time is spent on actual rendering with CPU/GPU. Most of the times when doing distributed rendering, CPU isn't actually the bottleneck, but rather transferring the necessary stuff for the rendering (project/scene data structures that each worker needs).
36 is at the high end of things, and the histogram is more skewed towards the lower end than > 30 hours, but it's relatively common.
Frames can be parallelised, so multiple frames in a shot/sequence are rendered at once, on different machines.
Each and every asset, animation, lighting, texturing sim and final comp will go through a number of revisions before being accepted.
So in all actuality that final frame could have been rendered 20+ times.
VFX farms are huge, In 2014 I worked on one that was 36K cpus and about 15pb of storage. Its probably now in the 200k cpu mark.
Video games, desktop apps, web apps, etc. And now confirmed that it happens to movies at Pixar.
On the one hand, it’s wise to only expend effort making something faster up to a point. At some point, unless a human has to wait for the result, there is no reason to make something faster [1].
On the other hand, once something takes more than a minute or two, and the person who started it goes and does something else, it doesn’t matter how long it takes, as long as it’s done before you get back. Film shots usually render overnight, so as long as they’re done in the morning and as long as they don’t prevent something else from being rendered by the morning, it doesn’t necessarily need to go faster. Somewhere out there is a blog post I remember about writing renderers and how artists behave; it posits perhaps there’s a couple of thresholds. If something takes longer than ten seconds to render, they’re going to leave to get coffee. If something takes longer than ten minutes to render, they’re going to start it at night and check on it in the morning.
[1] I always like the way Michael Abrash framed it:
“Understanding High Performance: Before we can create high-performance code, we must understand what high performance is. The objective (not always attained) in creating high- performance software is to make the software able to carry out its appointed tasks so rapidly that it responds instantaneously, as far as the user is concerned. In other words, high-performance code should ideally run so fast that any further improvement in the code would be pointless.
“Notice that the above definition most emphatically does not say anything about making the software as fast as possible. It also does not say anything about using assembly language, or an optimizing compiler, or, for that matter, a compiler at all. It also doesn’t say anything about how the code was designed and written. What it does say is that high-performance code shouldn’t get in the user’s way—and that’s all.” (From the “Graphics Programming Black Book”)
- Feedback loops
- Cumulative end-to-end latency
The second is especially challenging as only a handful of coders saying “that’s good enough” can add up to perceptibly massive latency for the end user.
Recalling a bit more now, the actual point of the blog post I was thinking of, and not summarizing super accurately, was to try to make things faster to prevent the artists from getting out of their seat, precisely because the tool it was referring to was primarily a feedback loop interaction. The tool in question was the PDI lighting tool “Light”, which received an Academy award a few years back. https://www.oscars.org/sci-tech/ceremonies/2013
24 hours of wall clock time is excessive and the reality is that anything over 2 hours starts to get painful. If you can't render reliably over night, your iterations slow down to molasses and the more iterations you can do the better something will look. These times are usually inflated in articles. I would never accept 24 hours to turn around a typical frame as being necessary. If I saw people working with that, my top priority would be to figure out what is going on, because with zero doubt there would be a huge amount of nonsense under the hood.
When I was doing CG production shot work ~15 years ago, there were occasionally shots that ran 24 hours, but the average was more like 3 or 4 hours. The shots that took 24 hours or more usually caused people to investigate whether something was wrong.
I worked on one such shot that was taking more than 24 hours. A scene in the film Madagascar where the boat blows a horn and all the trees on the island blow over. The trees and plants were modeled for close-ups, including flowers with stamens and pistils, but the shot was a view of the whole island. One of my co-workers wrote a pre-render filter with only a few lines of code, to check if pieces of the geometry were smaller than a pixel, and if so just discard them. IIRC, render times immediately dropped from 24 hours to 8 hours.
I wrote about my experiences in more detail a couple of months ago:
https://blog.markjgsmith.com/2020/11/24/what-its-like-workin...
There’s also some render farm setup stuff in my portfolio on the blog if you’re interested.
The hardware side of things was incredible. So much expensive kit. There’s a lot of great hardware related comments in the thread, so I won’t go over that.
The thing that I found really cool was how all the software systems were setup for so many artists to collaborate. Though we didn’t talk about agile methodologies (it was back in 2003/2004), the ways we worked had a lot of similarity to a classic agile / scrum setup, with daily standups, though they were called ‘dailies’, and there was no notion of sprints (we were always sprinting!) or sprint planning, but similar planning was done by producers and department heads, instead of user stories / features, people worked on ‘shots’. At the digital intermediate place the unit of work tended to be reels since we were scanning and printing entire reels, combining all the shots that the vfx houses completed.
Everyone used version control, though it was subversion, I don’t think git was that popular back then. Artists worked on their shots, checking in their project files rather than rendered files, we could always re-render a shot if necessary. There were also cli tools to submit finished rendered files, which were automatically organised into a standard folder structure on shared storage. The artists Linux shells would load environment variables from a dB, and their applications (Shake/Maya/Houdini etc) would load these transparently, so they never had to worry about where things were stored. That was all automatic as long as they knew which shot they were working on.
It was a great place to learn about technology and collaborating on digital production at scale.
I’ve always thought there were a lot of setups and tools that could be applied to software development at scale. I’d love to work on that sort of project. Hint: I am available for hire :)
If there's a generally static scene with just characters walking through it, does the render take advantage of rendering the static parts for the whole scene once, and then overlay and recompute the small differences caused by the moving things in each individual sub frame?
Or, alternatively what "class" of optimizations does something like that fall into?
Is rendering of video games more similar to rendering for movies, or for VFX?
What are some of physics "cheats" that look good enough but massively reduce compute intensity?
What are some interesting scaling laws about compute intensity / time versus parameters that the film director may have to choose between? "Director X, you can have <x> but that means to fit in the budget, we can't do <y>"
Can anyone point to a nice introduction to some of the basic compute-relevant techniques that rendering uses? Thanks!
As for your questions:
> Is rendering of video games more similar to rendering for movies, or VFX?
This question is possibly based on an incorrect assumption that feature (animated) films are rendered differently than VFX. They're identical in terms of most tech stacks including rendering and the process is largely similar overall.
Games aren't really similar to either since they're raster based rather than pathtraced. The new RTX setups are bringing those worlds closer. However older rendering setups like REYES that Pixar used up until Finding Dory, are more similar to games raster pipelines. though that's trivializing the differences.
A good intro to rendering is reading Raytracing in a Weekend (https://raytracing.github.io/books/RayTracingInOneWeekend.ht...), and Matt Pharr's PBRT book (http://www.pbr-book.org/)
(I was also reading the OP which says "...Our world works quite a bit differently than VFX in two ways..." hence my curiosity)
VFX schedules are usually significantly more compressed, typically 6-12 months, so often times it is cheaper and faster to throw more compute power at a problem rather than paying a group of highly knowledgeable rendering engineers and technical artists to optimize it (although, VFX houses will still employ rendering engineers and technical artists that know about optimization). Pixar has a dedicated group of people called Lightspeed technical artists whose sole job is to optimize scenes so that they can be rendered and re-rendered faster.
Historically, Pixar is also notorious for not doing a lot of "post-work" to their rendered images (although they are slowly starting to embrace it on their most recent films). In other words, what you see on film is very close to what was produced by the renderer. In VFX, to save time, you often render different layers of the image separately and then composite them later in a software package like Nuke. Doing compositing later allows you to fix mistakes, or make adjustments in a faster way than completely re-rendering the entire frame.
A lot of VFX studios use off the shelf farm management solutions that package up a job as a whole to a node.
I don't believe core stealing like they describe is unique to Pixar, but is also not common outside Pixar either, which is what they allude to afaik. It's less an animation vs VFX comparison, as just studio vs studio infrastructure comparison.
Showcased by the yearly highlights reel that the Renderman team puts out.
I'm pretty sure one of those shots was from Alien:Covenant (2017)
From the detail of rendering they do, I'd say there's no such thing.
As in: characters walking will have radiosity and shadows and reflections so there's no such thing as "the background is the same, only the characters are moving" because it isn't.
Typically, most pathtracers use a technique called Monte Carlo Estimation, which means that they continuously loop over every pixel in an image, and average together the incoming light from randomly traced light paths. To calculate motion blur, they typically sample the scene at least twice (once at camera shutter open, and again at shutter close). Adaptive sampling rendering techniques will typically converge faster when there is less motion blur.
One of the biggest time-saving techniques lately, is machine learning powered image denoising [1]. This allows the renderer to compute significantly fewer samples, but then have a 2D post-process run over the image and guess what the image might look like if it had been rendered with higher samples.
Animated movies and VFX render each frame in terms of minutes and hours, while games need to render in milliseconds. Many of the techniques used in game rendering are approximations of physically based light transport, that look "good enough". But modern animated films and VFX are much closer to simulating reality with true bounced lighting and reflections.
Not any more. It used to be that frames were rendered in bits then composited to make the final image. However, you then need lots of tricks to reflect what would have happened to the background as a result of the foreground ... shadows, for instance. So now the entire scene is given to the renderer and the renderer is told to get on with it.
Regarding physics cheats, it depends on the renderer but basically none. AI despeckling is making a huge difference to render times, however.
Directors don't get involved in scaling laws and stuff like that. Basically a studio has a "look" that they'll quote around.
Compute relevant techniques? A renderer basically solves the rendering equation. https://en.wikipedia.org/wiki/Rendering_equation
And have a look at Mitsuba! http://rgl.epfl.ch/publications/NimierDavidVicini2019Mitsuba...
I think a key is keeping the amount of time to render a constant.
I ray trace quake demos for fun at a much much lower scale[0], and have professionally organized much bigger installs (I feel confident in saying even though I don't know Pixar's exact scale).
But I don't know state of the art rendering. I'm sure Pixar knows their workload much better than I do. I would be interested in hearing why, though.
[0] Youtube butchers the quality in compression, but https://youtu.be/0xR1ZoGhfhc . Live system at https://qpov.retrofitta.se/, code at https://github.com/ThomasHabets/qpov.
Edit: I see people are following the links. What a day to overflow Go's 64bit counter for time durations on the stats page. https://qpov.retrofitta.se/stats
I'll fix it later.
Ie there's some headroom left for rush jobs and a safety net, because full occupancy isn't great either.
But at that point it may get into RAM constraint, or some as yet unmentioned inter-frame dependency/caching.
Maybe they're not as parallelizable as I'd expect. E.g. if there's serial work to be done by reusing scene layout algorithms between frames.
each "polygon" could have a 16k texture on it. You're pulling TBs of textures and other assets in each frame.
Naively I would expect that (as is the case for my MUCH smaller scale system) that I can compensate for network/disk-bound and non-multithreaded stages by merely running two concurrent frames.
On a larger scale I would expect to be able to estimate RAM-cheap frames, and always have one of them running per machine, but at SCHED_IDLE priority, so that they only get CPU when the "main" frame is blocked on disk or network, or a non-parallelizable stage. By starving one frame of CPU, it's much more likely that it'll need CPU the short intervals when it's allowed to get it.
How the hell can we have so much tech and leave so many common use cases so fucking broken for so long
It’s a perfect example of how you don’t really decide how others use your products; people do.
Most 4k films till very recently actually have the digital elements at 2k DCI-ish resolutions and are upscaled. I can't speak to wether soul is rendered at 2k or 4k, but it wouldn't be surprising if it was 2k upscaled.
If a shot doesn't look good enough when upscaled, they re-render in 4k.
Actually, they used to upscale most of the movies using Nuke, but recently, started using deep learning for that. At Siggraph 2020, they gave a talk on this topic: https://s2020.siggraph.org/presentation/?id=gensub_443&sess=...
The color space will be acesCG most likely these days.
The master will have been rendered into a DCP.
What gets delivered to the streaming service is another thing.
what then ends up at your screen is even more of a guess.
A 4K dcp is a JPEG2000 stream[1], with 16bit colour. Something like 60-120 megabytes a second. Obviously this isn't practical to stream to consumers. 4K over a stream is always a balancing act, more often than not it drops to 1080.
[1] well normally it is
We're currently building an API for rendering with the major render engines like V-Ray, Corona etc. Therefore we automatically scale jobs and deliver the results to our customers favorite storage like a NAS / s3 bucket or a google drive.
Another factor is, that we can reduce the costs for the cloud rendering factor 2-3x compared to the public clouds by tapping into spare capacity of secure data centers around the world (connected to our platform).
The reason the rule of thumb is to throw more hardware at the problem is that most (good) engineers bias towards wanting to make things performant, in my experience often beyond the point where it’s ROI positive. But of course you should not take that rule of thumb as a universal law, rather it’s another reminder of a cognitive bias to keep an eye on.
This may not apply on smaller side projects or places where technology is secondary.
Every second spent rendering or processing, is time an artist is not working on a shot. Any savings in optimizations add up to incredible cost savings.
I reviewed this book (2014): https://www.fevrierdorian.com/blog/post/2014/08/24/Multithre...
Hardware can be overloaded quickly if you don't care about it. You still need some engineering to keep everything under control.
I suspect this assumption come from the fact CGI have a lot of different things to render so you try to get the hardware that "support" by and large the problems you will have: You can't focus optimization on every single problem, but you can optimize the 90% use case, so the 10% use case can represent 90% of artist time and keep deadline safe.
much more simple, scales way higher, has much better documentation.
However its not designed to do the same thing.
tractor is about cramming as much stuff onto a machine as possible as fast as possible. It also is designed to handle processes that fail.
it doesn't force you to use stupid network patterns, and has very rich dependency management. But its not decentralised. However its pretty stable.
Just ... Scroll ... Down ... Click where it says "read more" or "show more replies"
You're human; THE most adaptable creature known. Adapt!
I'm not saying that twitter UX is perfect, or even good. I AM saying that it is usable.
Once you scroll up, it sort of makes sense -- each tweet with a line connecting user icons but then suddenly the expanded tweet thread has the main tweet in a larger font, then the "retweet/like" controls below it, THEN another line of smaller-font tweets that comprise the thread. Then you get some limited number and have to click "more" for more.
The monochrome of it all reminds me of when GMail got rid of the very helpful colors-for-threads and went to grey on grey on grey.
It's not visually apparent at all.
I'm saying it's usable.
I'm saying that complaining about it makes people look like they think they're royalty who need everything just so or their whole day is ruined... And now they can't tell the butler to take Poopsie for a walk because they're so shaken by the experience.
The amount of stuff your brain has to filter in the form of user name, user tweet handle, additional tagged handlers, UI menus, UI buttons for replying, retweeting, liking, etc on every single code snippet makes your brain work way more than it should to read a page of text.
Just imagine if I had written this exact text in 3 separate HackerNews comments, and prepended each with a 1/ 2/ 3/ text, in addition to all the message UI, it would have been more difficult to read than a simple piece of text.
Because that's what I'm getting from you.
If I'm going to use something, it should be intuitive and usable. It should be fit for its purpose, especially with a myriad of single and multi purpose tools available to all. This doesn't feel like something I should justify too hard :-)
Twitter is not a necessity of life. I don't have to use it. They want me to use it and if so they can/should make it usable.
Its paradigm and user interface don't work for me personally (and particularly when people try to fit an article into something explicitly designed for a single sentence - it feels like a misuse of a tool,like hammering a screw) so I don't use it. And that's ok!
I don't feel they are morally obligated to make it usable by me. It's a private platform and they can do as they please.
But my wife is a store manager and taught me that "feedback is a gift" - if a customer will leave the store and never come back,she'd rather know why, rather then remain ignorant.
She may or may not choose to address it but being aware and informed is better than being ignorant of the reasons.
So at the end of it, rather than down vote, let me ask what is the actual crux of your argument? People shouldn't be discriminate? They should use optional things they dislike? They shouldn't share their preferences and feedback? Twitter is a great tool for long format essays? Or something we all may be missing?
Yes, their UI sucks. No, they haven't fixed it. I would not be surprised at all if that were due to necessity rather than incompetence. Even tiny changes at that scale make large differences.
And, honestly I would not be surprised if it were incompetence, either. The skill of the silicon valley developer (as demonstrated by the commenters here) does not impress me even a little.
I think it's a UI that's been designed for one very clear, specific use-case; that's been stretched beyond belief by people cramming it to fit radically different use-cases. Developers I'm sure are simply stuck; do they optimize for original use-case and its legions of users; or other use-cases and those legions of users; I genuinely believe they cannot make everybody happy.
But as a user:
1. As an author, you have a choice not to try to cram a square kitchen through a round sink (long-form articles onto twitter)
2. As a reader, you have a choice not to consume content from a platform not designed or well-supporting of that content.
There's a million lifetimes worth of fascinating, useful, interesting, readable content out there.
Puzzling through a maze that is Twitter long-form article threads... is not how I choose to spend my time :).
As I asked in another post, I'm not sure what your underlying point is - that the UI is good and fit for purpose? Or that we should put up with it? Or that we should be more understanding of it, and that implies not criticizing it?
This website is full of employees (and ex employees) of Google, Facebook, Amazon, Netflix, Twitter, and any other company that has done stuff at global scale.
Scaling provides constraints, we all understand that. The interface being completely miserable to use is not because of that constraint.
I understand fetching all the replies all the time is next to impossible, that's not what I was criticizing, I get the impression that Twitter is doing a good job performance wise anyway.
> And, honestly I would not be surprised if it were incompetence, either. The skill of the silicon valley developer (as demonstrated by the commenters here) does not impress me even a little.
I think incompetence is part of it, but not the only reason. Twitter probably wants its mobile users to interact through the native iOS/Android apps. It's not as obvious as with what Reddit is doing but making the website act flaky and a bit unpleasant to use is certainly a way to get more people to use the app.
I imagine the app looks and feels better. I never used it and I don't intend to, for unrelated reasons. But then I guess I don't really have a right to complain.
Plenty of much better, more readable content on the internet without submitting myself to that low quality shit show with a poor ui.
Sure, you /can/ cut your steak with a spoon and a fork but... it is just "painful" because those tools were not made for that. Would we think you are a delicate flower for asking for a knife? (or to make the analogy better, let's say you are at your friend's BBQ and he is giving you the steak for free).
I like twitter, and I use it of certain specific things for which a 200 character text is quite good at.
> Twitter was designed to have 280 characters max per message.
Twitter was designed to use 140 characters, plus room for 20 for a username to fix in the 160 char budget of an SMS message.
Their upping it to 280 later was capitulating to the fact that nobody actually wants to send SMS sized messages over the internet.
That doesn't work. Neither "read more" nor "show more replies" appears on the page [1]. Nor does "show replies", which turns out to be what you actually need to click once you go to a place it appears. In fact, there's no indication that there even are more replies, or that "replies" are actually the main content.
To see the content, it turns out you need to click on the original "1/" tweet, which takes you to what looks like a new page (but doesn't change the URL, so you can't link to it).
It is not usable in any real sense. I only spent the time to solve the puzzle as I was trying to make sense of your comment.
[1] https://imgur.com/a/BL9M74m (the centre column is shown in full, and there's nothing down further)
People were supposed to shoot short, simple, single messages and other people maybe react to this with their own short, single messages.
It’s time for Twitter to evolve in so many ways.
The character limit is the defining feature of Twitter.
But honestly I'd prefer if they spent some fucking time upstaffing their user service and abuse teams. And if they could finally ban Trump and his conspiracy nutcase followers/network.
I certainly don't think that HN should ban you because of it, though.
Except for the information that doesn't show up on other platforms, which is why it is a frequently linked site here?
"It's not a bug, it's a feature" I've heard since the late 80's and still works magic if you ask me.
As an authoring medium though, the character constraints force you to write succinct points that keep the reader engaged. You can focus your writing just on one point at a time, committing to them when you tweet, and you can stop anytime. If you're struggling with writing longer form pieces a tweet thread is a great on-ramp to get the outline together, which you can later expand into a post.
As a conversation medium, it's also nice to be able to focus conversation specifically on a particular point, rather than get jumbled together with a bunch of unrelated comments in the comments section at the end of a post.
I'm pretty sure the context isn't there in many cases, although I haven't figured out the rules exactly.
Twitter have tried to have their cake and eat it too by allowing a reddit or hn style tree (which is good for discussions) while having their ui present it as a linear feed (which is good for engagement). This has lead to their current solution where the branches are presented in a choose your own adventure way. Now because the linked tweet has three replies, it has to show all three of them and leave it to you to select which branch you want to go down by clicking on it or the show replies prompt (though it does give primacy to pixprin's replies to self).
So much for Pixar's render farm.
That just about sums up the insanity of the "modern web".
I shouldn't really say more.
Firstly, K8s has no concept of licenses, it also is exceptionally weak on dependencies. A job graph for a VFX job can be well over 100k nodes, something that would crash k8s.
Secondly, tractor (https://rmanwiki.pixar.com/display/TRA/Tractor+2) is exceptionally fast at dispatching jobs to machines. I suspect its in the order of 50k a second, if not more.
Thirdly, getting k8s to talk to 25k machines without saturating the network is almost impossible.
fourthly, it doesn't do to well on "normal" network, try getting decent network throughput on one of K8s batshit networking schemes(each server on a farm will have at a minimum 2 10 gig links, more likley 2 40gig)