Exploring the .NET Core Runtime
mattwarren.org
mattwarren.org
I also love that people are doing write ups about the source of .NET Core and it's internals.
Me too!
BTW, if you want more on the same topic check out http://mattwarren.org/2018/07/12/Presentations-and-Talks-cov... and http://mattwarren.org/2018/01/22/Resources-for-Learning-abou...
Also Matt Warren's blog is super cool.
To understand why, we need to talk about 2 separate things:
---
1. Specialization is important for performance and when speaking of .NET, specialization for value types is subsumed in type erasure, but that's not necessarily the case. You can have compiler-driven or even runtime-driven specialization without reification.
In Scala for example we've had the "@specialize" annotation for a long time, with the compiler being able to specialize generic code just fine. It's not perfect, a better implementation eventually happened in Miniboxing [1] but it withered away due to lack of interest and the ongoing work happening in Dotty / Scala 3 and its newer TASTY distribution format, which should make specialization easier to accomplish.
Also there is on-going work to bring value types to the JVM and it's happening: http://mail.openjdk.java.net/pipermail/valhalla-spec-experts...
That said having specialization is pretty cool for fine control of the memory layout and Java developers have to resort to a lot of unsafe hacks for achieving the same thing. But if that cost was paid such that languages like Scala or Haskell could happen on top of the JVM, until they figure out how to do it such that everybody benefits, I think it was a cost worth paying.
Also consider that the lack of specialization forced the JVM 's engineers to get creative in other areas. For example the JVM has always been great at inlining code at runtime, even for megamorphic call sites. And the new GraalVM has super impressive abilities to eliminate boxing at runtime, which works for dynamic languages too: https://www.graalvm.org/
---
2. Reification is in fact about adding info about type parameters at runtime. This aids in using reflection to make a difference between List<int> and List<string>, but people miss the forest from the trees.
As a matter of fact such reflection is only needed because languages like Java or C# have very weak static type systems, compared with other languages in the ML family. With an expressive type system, you never need reflection capabilities.
In Haskell for example the question of whether something is a List<string> or List<int> never, ever happens. In Scala you sometimes need it, but much rarely and you can get it via a compiler-generated `ClassTag`, which is actually a much better approach, because it makes it clear in the signature, this being compile-time reflection. People also like being able to do "new T", however that need completely goes away via proper support for type classes, which both Haskell and Scala have, this being another special purpose band-aid.
And reification is actually a bad feature to have in the runtime, because it makes it hard for languages to support higher-kinded types, or to build dynamic languages.
F# does not do higher-kinded types and is less expressive for that reason than OCaml, Scala or Haskell and the primary reason for why it doesn't do higher-kinded types is because it would have to do type erasure by itself, thus forgoing the performance benefits and the interoperability it has with C#.
Ironically it is support for higher-kinded types in a language that increases its expressive capabilities to the point that you no longer need runtime reflection. In other words ... you're blaming Java for not having a band-aid that happened in C# due to their static type system being basically unsound and thus needing runtime guards and reflection. You quickly get over this once you'll start using a more expressive language ;-)
Scala, Kotlin, Clojure, Eta (Haskell), JRuby are the ones I care about and they've been flourishing.
I'm a pragmatic at heart btw, but pragmatism sometimes leads to ignorance. It's important to see what's available on the other side of the fence without bringing the preconceptions bag with you. Because you can then actually use the metaphorical "best tool for the job".
It's what I'm doing myself, or at least trying and .NET is without a doubt evolving in a direction I like. And I can also write entire paragraphs of what's better on .NET, I just don't find the generics reification to be one of those things.
The implication here is that you never need runtime type checking, which clearly isn't the case in ML family languages. Sum types for example have an abstract representation that must be resolved at runtime (for pattern matching).
C# also doesn't need reflection to resolve types (other than through the virtual route - i.e. inherited method resolution), this is a core feature of the CLR though (via `callvirt` [1] if I remember right) and not a weakness of C#'s type-system. C# bakes the concrete generic types into the generated IL.
[1] https://docs.microsoft.com/en-us/dotnet/api/system.reflectio...
Indeed, but that's plain tagging, which is basically an int and not the same thing.
"callvirt" is basically doing a virtual method lookup in a vtable. It's still at runtime, but we are talking about OOP polymorphism now and how late binding works. Languages with type classes don't need that either, but that's a separate discussion and note that I still like OOP.
> C# also doesn't need reflection to resolve types
No, the developers do, because the language is not expressive enough to express generic pieces of code without losing type info, so you need reflection for constraints (e.g. "new T" or "instanceOf" checks) and runtime guards for downcasting.
Classic example: express a "sum" function that works on any Array<A>.
And note that what F# is doing to solve this particular problem is a hack to workaround both the lack of higher kinded types and of type classes and that does not interoperate with C#, since it requires "inline functions" which are doing type erasure (and it's a big and ugly hack, speaking of ML, because inline functions are no longer values ;-)).
> C# bakes the concrete generic types into the generated IL.
Of course, but that too is part of the problem.
The type system is totally capable of it. The language doesn't make it easy, but the code below is essentially type-classes and class-instances if you squint. It's something the C# team are looking into now anyway, but the main point is the type system doesn't preclude it.
public interface Newable<A>
{
T New();
}
public struct NewableThing : Newable<Thing>
{
Thing New() => new Thing();
}
public class Foo
{
public A Bar<NewableA, A>() where NewableA : struct, Newable<A> =>
default(NewableA).New();
}
public interface Num<A>
{
A Zero();
A Add(A x, A y);
}
public struct NumInt : Num<int>
{
int Zero() => 0;
int Add(int x, int y) => x + y;
}
public struct NumFloat : Num<float>
{
float Zero() => 0;
float Add(float x, float y) => x + y;
}
public static A Sum<NumA, A>(A[] xs) where NumA : struct, Num<A> =>
xs.Fold(default(NumA).Zero(), (s, x) => default(NumA).Add(s, x));
int sumInt = Sum<NumInt, int>(new [] {1,2,3,4});
int sumFlt = Sum<NumFloat, float>(new [] {1.0,2.0,3.0,4.0});
That's a very hand wavey statement. It's the goal of all staticly typed compilers to resolve to concrete types at compile time. How is part of any problem? The CLR has the capability to generate new concrete types from generic types at runtime if required, but obviously tries to do as much of that work up front at compile time.
Next, you can't say that C# has a "weak" type system. It is mainly a strongly and statically typed by design. There are elements of weak typing in C# such as the dynamic keyword, and all C# type safety features can be bypassed if desired. It seems you prefer "weak" to mean "poor" but that is your definition, not the generally accepted one.
Further, it is completely false to indicate something to the effect that C#/CLR can't let other languages flourish on the platform. The CLR stands for "Common Language Runtime" and can run any language you can imagine, including weakly typed languages. It is a plain old finite stack machine at heart. Unlike JVM, the CLR is an ECMA standard and anyone can implement it and .NET sources are not GPL licensed.
Scala used to be available on CLR, and the reason it is not supported is more political than technical. You are again trying to make a technical justification for a political decision. Besides, Scala still only runs on JDK 8 which is funny because Oracle JDK will need a commercial license from 2019 to get patches so you're forced to switch to OpenJDK or pay for a Scala runtime.
The Java community has talked about value types forever, we know that. Good luck unwinding that decision, it is not that easy.
The cost for unsafe hacks to let Scala and Haskell work had more to do about the fact that .NET was not cross platform at that time. That was not the ecosystem impetus for for having Scala run on JVM. It was more because Java as a language is aging and lacks innovation which is well recognized. The cross platform argument against the CLR is gone now, so there is little reason why we shouldn't be able to run Scala and Haskell on CLR instead if you are after those languages.
Also, the point about "forcing JVM engineers to get creative" is not really a solid argument at all. They just have to work around the mistakes they made in the past (no value types + runtime type erasure). We also know for a fact that escape analysis in JVM doesn't really cover that many cases anyway in practice. You need to look at the generated code to tell if it works or not. Besides, it is not something that the CLR cannot implement. On the contrary, there is active effort in CLR towards Object Stack Allocation (you can Google for it).
The general point that reflection is "only" needed because of a "weak static type system" is also not true. You seem to confuse the shape of the API for getting runtime type information with the need for getting such information in the first place. You can try to build a polymorphic serializer in C++ which also has run time type erasure, it is not as easy as in C#. Eventually we need to carry over type information anyway, no matter what language.
Reflection is not about "adding info about type parameters at runtime". It is much more than that. First, we don't "add information" at runtime for existing types, we can inquire about it. This is super useful for many scenarios. Second, we can generate code at runtime as well with reflection, that is, MSIL op codes. This is also super useful.
Your use case of "express sum function over Array<A>" is not the direction C# is heading. We have LINQ for the expressiveness part which Java doesn't have apart from some lookalike hacks. Also, for performance, your sum function is going to run at glacier speed compared with .NET Core 3.0 where we will have SIMD instructions and we can execute dedicated AVX code paths depending on memory alignment, in managed code. This stuff used to be C++ only until now. I don't think your Scala and Haskell implementations stand a chance to perform close to that. And reusing a sum function over a read only span is a matter of a one line call. I like the pureness argument but in practice we need to consider performance as well.
The majority of the issues and gotchas listed in the generics FAQ for Java do not happen in C# at all. It is liberating.
Reflection is an advanced use case that some people use, but it is rare.
The lack of flourish in. NET overtime more to what happens when a company takes the reins of a stack and later stops working on it.
Microsoft for a while invested and designed JavaScript, Python and Ruby versions for .NET. But when conditions changed, there was no interest to keep the projects going. These are all observations from an outsider at the time the projects were defunded.
Because of the siloed approach to development at the time, and the lack of an external county around those efforts, bootstrapping a community to drive those on their own proved to be very hard. Ruby mostly died, Python is barely surviving.
The mood in the ecosystem went from "we can build these and speed them up" to "this is an ongoing cost, let us rather interop with the real implementations rather than find constant catch up".
Meanwhile, languages that Microsoft did not build did flourish, like PHP
This corporate phenomenon deserves a blog post on its own
But couldn't .NET not being a good target for dynamic languages be one of the reasons to stop working on it?
I keep hearing that .NET is a good platform for other languages, but there doesn't seem to be much empirical evidence for it (at least in dynamic languages).
Thanks! I'm glad you like it
We’ve been a C# house for several years, decades really, and I’ve always preferred it to JAVA so I’m actually excited for Core.
But we rarely use it. Not because it’s not great, rather because we’re more productive with flask or Django. For Core to really make sense for us, it’d would have to stop being so damn low level, but I guess that maybe it can’t without sacrificing too much efficiency. More importantly it needs better libraries for things that aren’t “built-in”.
I can certainly see why .NET developers welcome it, because they finally have good cross platform ability. At least until they need to do authentication on a non-standard SAML token, that though easily supported by ADFS but is a bitch in any .NET setup.
I know we aren’t most use cases, being the public sector and running a gazillion different tech stacks at once, but .NET has never played well once you stepped outside it’s comfortzone and it’s always been so low level that writing library extensions were a bitch. And that may have worked out, so far, but I just don’t see why people stick with it when there are more productive alternatives.
I say productive, because I don’t think .NET core is lacking technically, but delivering solutions on time and with minimum maintenance requirements afterward is just easier in python or JAVA and I’d imagine others as well.
But maybe I’m missing something?
I find I'm most productive in C#, but probably mostly because it's what I use every day and I like the tooling.
By the time we have an app running in Django, we’re not even finished with Entity modelling in a Core web-api.
I do think stuff like Blazor.Net is promising, but we’re not a technology company, we support thousands of employees who only care about how digitisation can make their lives easier as fast and as stable as possible.
.NET isn’t the best at that, at least not for us.
Dont get me wrong, I don’t dislike .NET Core, it think it’s great, I just don’t see how it benefits me.
>.NET isn’t the best at that, at least not for us.
You're comparing apples and oranges, djanjo is a web framework, for creating websites, .Net core is a cross platform compiler and the standard library that goes with the C# programming language, which includes some stuff for creating websites (along with desktop, CLI, services etc). Django might work best for you but I want to create a bunch of micro services I'm not going to use it, am I? Not to mention Django works well as a general purpose solution for general purpose websites, If you're building anysort of heavyweight, enterprises level web architecture you're going to want a hell of a lot more control that what django provides for you.
Obviously we don’t use Django for everything. But like with web-applications, a python script or a flask application is always more productive for us than .NET.
>Do people really use .NET core, and if so, why?
Could you please expand on this one?
> but delivering solutions on time and with minimum maintenance requirements afterward is just easier in python or JAVA
I would agree if you would replace Python/Java with Node, but with these languages I don't see it.
> on time and with minimum maintenance requirements afterward
This is my Dockerfile, it is serving me without changes (technically replacing 2.1 with 2.2 is a change) for more than 6 months.
# Build assets
FROM node:8-alpine AS assets
WORKDIR /app
COPY src/Website/Package.json ./Website/package.json
COPY src/Website/Gulpfile.js ./Website/
COPY src/Website/Static/. ./Website/Static
WORKDIR /app/Website
RUN npm install
RUN npm run build
# Build project
FROM microsoft/dotnet:2.2-sdk-alpine AS build
WORKDIR /app
COPY src/Website/*.csproj ./Website/
WORKDIR /app/Website
RUN dotnet restore
WORKDIR /app
COPY src/Website/. ./Website/
WORKDIR /app/Website
RUN dotnet publish -c Release -o dist
# Run project
FROM microsoft/dotnet:2.2-aspnetcore-runtime-alpine AS runtime
WORKDIR /app
COPY --from=build /app/Website/dist ./
COPY --from=assets /app/Website/Static/dist ./Static/dist
RUN rm /app/Data/Log/.gitkeep
ENTRYPOINT ["dotnet", "Website.dll"]
I think you might be.
Plus you get a decent static type system!
.NET is an ecosystem of languages, runtimes, and tooling with Visual Studio being regarded by many as the best IDE. Add in the vast MS suite of Windows, Office, Azure, SQL Server, and the rest and you have one of the most productive frameworks to build anything for desktop, mobile, games, server backends, and even databases.
We moved over to .NET Core since v1.0 to consolidate on Linux/K8S and it's been a great experience. ASP.NET Core is also a very good framework for building webapps, SPAs and APIs that has more functionality in the box than pretty much anything else we've seen.
Nuget is turning into a solid package management system and there are great 3rd-party projects like IdentityServer for authentication, have you looked at that?
I have been developing in C# for years and I still prefer Python's simplicity over C# and .NET. C# and .NET just seems to have too many concepts, indirection, and 'enterpriseyness' for my liking. This is alleviated by things like LINQ, but still an issue for me.
I can however offer a simple possible explanation for your experience: you are more proficient in other languages and frameworks because you know them better.
Yet another case of “Do people still watch TV? I haven’t owned a TV in 10 years”, style myopia.
I once read CLR via C# and it was difficult but amazing. Is all of the knowledge in that book now worthless? What about the IL, is the IL still the same?
I guess it makes sense. Before .NET Core, Silverlight was the most x-plat version of the Microsoft .NET Runtime(s) (excluding Xamarin/Mono), so it would've been a good starting point.
.NET Core has self contained deployments so you can bundle it with your app in a .zip.
For development or continuous integration SQL Server comes in a container now.
They first bundled (1.0) with xp, and every OS since. I don't know why they bothered considering practically every app has had to bundle newer versions with the installer. They could have at least pushed newer versions with windows update.
Don't get me wrong though, I worked to integrate it into Yocto because I love .NET/C#, but the build system needs a lot of love.
See https://github.com/dotnet/source-build#net-core-build-script... and https://github.com/dotnet/arcade#arcade