How much different/better/worse is this compared to Kubernetes?
As a scheduler it's much more powerful in capabilities, scalability and performance. It was designed to react to different workload demands in huge clusters very quickly. Imagine a data center goes offline and you need to spin up the search engine while killing a map reduce with 20k workers. A borg cluster is war zone, priorities and quotas win.
CSIs exist because there are a billion vendors of "cloud storage" solutions, and apps need a way to decouple themselves from the details. If you just need a block device, your Kubernetes app can run against anything that implements that API; your app you developed on DigitalOcean will Just Work on GKE. That's the idea behind those.
For workload management, Borg feels very close to StatefulSets; your replica count could increase and decrease as you demanded it, but each task was individually named (0.your-job.cell). I ran all my jobs with semantics similar to Deployments (tasks did not care which id they had; they all showed up as load balancer targets) and I think Borg had different abstractions for things in Kubernetes like ReplicaSets (i.e., a particular [config,code] tuple that is running in production that logically makes up a deployment).
You didn't ask so I won't go into the details as it gets very complicated, but networking is vastly different. (Kubernetes tries to provide IPAM and connection-balancing that is transparent to workloads; Google preferred client-side "smart" load-balancing libraries. That meant that load-balancing complexity was outside the scope of Borg, but for Kubernetes, it's very much in scope. Often with confusing results, but I digress.)
My experience having used both extensively is that there are about the same to the end user. Before I worked at Google, I always let "someone else" handle deployment and maintenance of production. At Google, I felt like I could write and release a piece of software to Borg in the same day (and did once!) After Google, I struggled with a bunch of orchestrators until I landed on Kubernetes, which gives me similar confidence and ease of use. Kubernetes has more API surface, probably because it has more users. At Google you could declare by fiat how jobs were to be run. In the real world, people aren't going to use your thing if you don't support their favorite feature. I have tried the opinionated container orchestrators in the real world (Heroku, Convox, ECS) and they didn't make me happy, while Kubernetes did.
Early on (changing somewhat now), large scale and throughput were not big goals for K8S. Given most uses are with VMs, and pretty much all uses are tiny compared to Borg, this was a totally reasonable prioritization.
K8S fixes many many serious issues in Borg that are mostly unfixable at this point (I worked on Borg for many years), it's better in many ways.
For scale specifically, Borg has a big advantage in that the state database in internal (the master maintains it directly on a local disk). This is naturally much faster then using etcd (at least etcd today).
But there are big architectural differences too. Borg can split its master across multiple machines for example. The scheduler is a very tightly optimised piece of code. Borg clusters are managed by a central, dedicated specially trained team so there's basically no limit to the operational complexity that a Borg cluster can take on as long as they're all consistent - which they are.
To that extent, there's a hole left dealing with things that do need to consider state versus run networked API services. Kubernetes seems to have no good story here, whether it's the evolutionary progress happening around StatefulSets/PVC/PV, or a per appliance operator for you, and you, and you, which punch a hole as big as you like in the scheduling abstraction. Streaming for example is a notable pain, but pretty much every OSS project created in this century that has state needs a compensating tool, typically an operator, to function on Kubernetes. I'm not even sure at this point whether state is a design consideration that can be retrofitted—that's not a criticism of Kubernetes, but it is a complication and investment factor for stateful workloads. So what may happen should Kubernetes be one of those things that does in fact end up being a long term technology, is the entire software industry offloads state management to vendors (ie to a handful of cloud services), or something in open source reacts and is created to fill the infrastructure gap for state management.
In my personal opinion it is more valuable to adopt the Google model where no local file is considered critical, than it is to try to cram statefulness into an otherwise cloud-native stack. I feel that if you still care about specific files on specific disks then you really haven't fully adopted the meaning of cloud-native.
Jobs can write to local disk. Where do you think their executables came from? They can also use local disk as a scratch space for various things. The main use was writing out logs. Logs get written then picked up by a co-located job and saved out to disk clusters like Colossus, where they are then in turn picked up for processing.
However it's true that local disk was considered to be used only for transient data, most of the time. You could configure Borg to not work like that, so you could certainly run MySQL clusters on it and things, but that was a very unusual setup and not at all recommended. After all, then you'd have to manage backups and machine failure yourself.
This practice worked amazingly well when writing software entirely in house. So it was great for things like web search where there was no open source stack waiting to be adopted. It was disastrous for backwards compatibility with pre-written software, and I think this sort of thing contributed to Google's notoriously strong NIH syndrome. Using open source software at Google is hard; there are processes around it that must be satisfied, but more importantly, that software will expect POSIX file APIs to actually work and on Borg they don't. They appear to work, right up until your job gets evicted and then the data goes poof. To solve this you need to work with files using Google's own proprietary file VFS APIs that are backed by RPC clustered storage. GFS then Colossus presented a FS API that was only vaguely related to POSIX, so you couldn't even just hack together a quick bridge.
Net/net it was worth importing small in-memory libraries from the open source world, very rarely, but anything more substantial like a server - forget it. This could lead to absurd outcomes. I worked on a project there many years ago that would have really benefited from using Apache to do file serving, but Apache didn't integrate with Borg so I ended up using "static GWS" as it was called at the time. But static GWS had just enough features to serve websites written exactly how Googlers wrote it and nothing more, so it was a total fail at serving a third party developed website we'd agreed to host. Much pain.
One of the problems with Docker and container architectures in my mind is that they ultimately evolved out of Borg. The cgroups work in the kernel and other kernel features were developed by Paul Menage and others on the Borglet team for Google's internal use. Then the Docker guys picked them up and created this notion of containers that contain a whole OS - not at all how Google does it - but they kept the notion of transient local storage. Well that's a disaster, because normal software expects local disk to stick around. In my post-Google career I have witnessed more than one disaster caused by Docker of the form "whoops, we misconfigured our Dockerfiles and just erased our private key". There's no good justification for that. For most people systemd with some simple use of static linking would work just as well. That's what Borglets used to do - set up cgroups, have a simple base OS and then run statically linked binaries.
Just FYI, a lot changes in 5 years.