Five years of Scala and counting: debunking some myths
manuel.bernhardt.io
manuel.bernhardt.io
Scala compilation time is probably the worst problem I have with scala, and it still is frankly. I have seen no evidence that this has improved from personal experience and after talking with devs I know working in large scala code bases (Foursquare, Twitter).
Even if there are speed improvements in the scala compiler in future versions, the author doesn't address binary incompatibility between major scala versions either which, along with slow compilation times, makes it a serious PITA to upgrade to a scala version that might actually compiler faster.
[edit] hit submit too early last time, re-edited after regathering my thoughts.
Complexity or learning the language is still the biggest issue I see. That has not changed in 5 years.
That's not true. Most things are them same or easier than on Java8.
If you don't lear Lambda's on Java8 you are missing everything. And most things in Scala's are like Lambda map/foreach all over the place. and generators making things easier to read and easier to understand. Java8 doesn't have this future you end up writing flatmap(data -> return flatmap .. return flatmap return flatmpap) that's awkward.
On scala if you don't use the hard things the language is way easier than the most languages outside of the jvm ecosystem.
Sure, it'd be great to have binary compatibility, but I think the ability to make changes to the language unhindered by supporting byte code from previous languages is more valuable.
For example with the 2.11 release, I had three different non-trivial sized projects and for each I was able to simply change the scala version in my sbt file and everything worked just as it had before with no library or compile issues.
I will agree that it was a huge issue in the past with 2.7 - 2.9, but in my world it hasn't bitten me in a long time. Definitely not trying to say it isn't an issue or something that wouldn't be great to be fixed, but at this point I have a tough time seeing it being considered a barrier to entry in any way.
Also, I just throw in that for me compile time hasn't been an issue for me lately either. I work on reasonably sized projects, but definitely nothing on the twitter / foursquare scale.
For example I'm going to make the claim that SBT indeed isn't just a bunch of symbols, but rather one of the best build management tools in existence. For anybody going to challenge that I'm going to start mentioning the complete clusterfuck of other platforms and languages, like that of C/C++, or that of Python, or that of Javascript, or that of C#/.NET, or that of Haskell. SBT not only does demonstrably a better job than most, but works well out of the box and is relatively pain free. It has its own set of problems of course, but what tool doesn't.
Compatibility issues are real. But you know, the nice thing about those compatibility issues is that it forced the community to come up with solutions. At least Scala and SBT are addressing compatibility issues, whereas the transition from Java 6 to later versions is proving to be next to impossible and I don't think I have to mention Python 3 or ECMAScript 6.
Compilation times haven't been that much of an issue for us because our codebase is split into services and libraries that have a reasonable size. And on the switch from Scala 2.10 to 2.11 I definitely saw an improvement, with the claims of 30% being about right.
I do agree that they can do better, taking a look at similar languages with a slightly better story, like Haskell. However what people are usually missing is that this is usually a trade-off with compile-time safety.
The more static a language is, the harder it is to compile it. The compilation cost for a dynamic language like Javascript is zero, the cost being completely amortized with everything happening directly at runtime. When going from Javascript to Java or C#, for people that prefer Java or C#, people go from zero to a very noticeable and resource intensive AOT compilation step and many times this tradeoff is worth it (depending on whom you ask) for safety, performance or tooling reasons. Well, Scala is a more static language than Java, it infers a lot of things, it has better generics, it does implicits which can then model type-classes and even the infamous CanBuildFrom pattern and so on, giving you a nice balance between productivity and safety in a static language. And that doesn't come for free.
ECMAScript 6 is finalized and many people are running it today in production environments either using Node on the backend or on the front-end using a build process that integrates with something like Babel.
The following quote implies that you believe JavaScript is runtime evaluated, when in fact the standards body does not specify runtime evaluation or compilation. That is a decision left up to the implementation. All major browsers today and Node.js use just in time compilation for JavaScript rather than runtime evaluation.
"The compilation cost for a dynamic language like Javascript is zero, the cost being completely amortized with everything happening directly at runtime."
[EDIT]
I have added a link to the JavaScript entry on Wikipedia where you can verify that modern browsers compile rather than interprest JavaScript.
The tooling hasn't been the problem. There are compatibility libraries (ie: six), source transformation tools (ie: 2to3), and plenty of books, blog posts, and talks on how to manage supporting both version of Python or doing a full port.
If you want to port there's a well-tested and clear path. There are enough big libraries out there that have managed it. There is no real barrier as the OP suggested in their article.
Also, porting is made more difficult by the fact that Python is a dynamic language, so a lot of errors that could be caught at compile time will only be catchable through a thorough testing suite. I have seen little evidence to suggest that porting from Python 2 to 3 is actually straightfoward in the sense of not requiring significant effort proportional to the size of the code base.
The piece I quoted indicates to me that the parent possibly does not understand this about JavaScript since they stated the compilation cost of JavaScript "is zero" that implies that JavaScript is not compiled. If there is a compilation step then some cost is incurred no matter how low.
It was further supported that the parent may not understand JavaScript is typically compiled when they followed on with "cost being completely amortized with everything happening directly at runtime". That would indicate that all of the cost is incurred/paid at runtime rather than in the compile step which is simply false.
Also, that the parent bothered specifying "AOT" in the first place is proof enough the entire basis of your belief in their understanding of things is utterly groundless. This would be clearer to you if you understood that JIT compilation cost is amortized, as the parent alludes, and that it doesn't happen in "a" compilation step.
I've never heard of anyone having issues going from Java 6 to later versions, at the language level or JVM level.
Without community hacks.
Before I get the usual Sun CEO approved it reply:
http://nighthacks.com/roller/jag/entry/my_attitude_on_oracle...
Second, at least on lambdas they could generate inner classes for older devices and update DEX format now that Dex is compiled AOT to native code. So only the compiler dex2aot would be required to be updated.
Currently there is zero public feedback on the new compiler chain (Jack & Jill), but from the sources it appears it is Java 6.5 all the way.
They have some unit tests that appear to be related to Java 8, but no one knows what is their idea.
In a few years (by Java 10 timeframe), according to the actual roadmap which might still change, Java will have modules, a JNI replacement, reifeid generics, value types, new array type, GPGPU support, AOT compiler....
If nothing changes, assuming Android 4.4 devices will be gone, the only change will be from Java 6.5 to Java 7, if Android's team attitude is to stay like it currently is.
But at least they spend time doing those stupid devbytes videos, doing continuous buggy releases of the support library and Google services.
Android Java is definitely the second coming of J++/J#.
Compiler to native code? aren't the big performance gains done in the JVM?
The majority of the commercial third party JVM vendors do offer AOT as part of their toolchain.
Similarly .NET always had a JIT/AOT model. The novelty with .NET Native is static compilation and integration of the Visual C++ backend.
I don't if this is true, but I read somewhere that AOT was tabu at Sun.
Oracle is now improving Java to cover the use cases where it still fails short vs other alternatives, so having an AOT compiler in the reference JDK instead of forcing devs to get it from a third party is part of their roadmap.
How it will look like in the end, it is not 100% clear.
Well, we are running and further developing code written for Java 1.4 on Java 8 now. Zero problems.
Android is a special case, though. That's just Google being incapable of accepting a ruling and making a deal with Oracle. Instead they are pouting like little princesses and secretly developing their Swift copycat language.
sbt's biggest problems is dependency resolution times. and thas mostly caused by the fact of ivy.
compilation times aren't an issue. mostly its faster than most things.
This is one of the big points I always try to make to people. JS compilation is cheap and maybe it's faster to spit out some functional code. But all you've really done is pushed the rest of your work to tomorrow. Your type verification happens at runtime, and now instead of your bugs being compiler errors, your bugs are production errors that you've exposed your customers to.
With Scala I rarely need to use the debugger. With JS and PHP I have to constantly use the debugger to track down "what type does this variable actually have in this random edge case at runtime?"
If I spot my error during compile-time in scala and only during run-time in javascript that is still a win for javascript if I found the error 1 second after writing it in javascript because that's how long the page takes to load as opposed to 5 seconds in scala, because that's how long sbt takes to compile.
Absolute time to discover the failure from writing it is important, not the phases that the error is discovered in, and despite this article, I still find that on complex project structures sbt/scala is horribly slow.
You won't. Your users will. In production.
Statically typed languages make sure you don't ship with that kind of buggy code.
I'm now firmly of the opinion that static vs dynamic typed is a sideshow, the real value is shifting as many errors as possible to as early as possible (in terms of seconds from typing the error to finding it) by whatever means you can, and a powerful type system is one tool for doing this, but if it compromises the iteration cycle and actually shifts the discovery of errors later in time then it's undermined one of the major reasons for using it.
1) sbt incremental compilation is basically immediate. This facilitates tight feedback loops for running tests and using the repl
2) Scala's type system does so much more than fast-compiling languages (Go, Java, etc) that it's definitely worth it. What you lose in compile-time, you gain in speed to a correct and sound solution.
Also, the article left out probably my two favourite parts about Scala - awesome language level support for concurrency, and pattern matching. Whenever I go to languages without these features, I really miss them.
Above all else, what always gets to me with Scala is its powerful generics. It truly is first-class. I'm excited for macros as well.
If you literally just mean generics at runtime, they are no different than Java's, outside of support for specialization for primitive types.
> Perhaps the OP was referring to the type system instead?
Generics and the type system are pretty much the same thing? They only matter and have true power at compile time regardless of your language. So the fact that Scala's generics have absurd power at compile-time (way more than anything in C#) is 100% relevant.
I searched around a bit about HKT's and although I can't say I understand them, I can see how the ability to do something like provide a constructor over a type could be useful. It feels like it has a lot of use in writing very general-purpose libraries, or perhaps a DSL that can map to a wide variety of different types. Interesting!
The second thing that comes to mind is the ability to support variance. To quote Twitter Scala School: "A central question that comes up when mixing OO with polymorphism is: if T’ is a subclass of T, is Container[T’] considered a subclass of Container[T]?" Scala allows us to express this information using +/- as in `Container[+D]` allows for subclasses of `D`.
Finally, though not so much related, an interesting development in the type space is type macros (http://docs.scala-lang.org/overviews/macros/typemacros.html). This will give us a stronger "generic" approach to programming that allows us to write code to construct type hierarchies. In general, Scala is moving fast and always evolving.
You'd have to be more specific as to what your problems are concerning boxing & jvm. Lists are properly specialized such that a = List(1) == List<Int> and a(0) == Int. Of course, if you use a diverse set of types in the list, you'll have a `List<Any>` which then you would use `match` like so... match item { case i:Int => println("int " + i) case s:String => println("string " + s) }
Scala (my current language), world's apart! I was truly blind to a much higher, cleaner level of thinking/programming until I learned Scala.
Ten years ago I was using Perl to write web applications with HTML::Mason, in a consulting firm. I can't say I miss that :-).
I think operator overloading and implicit conversions are the biggest problem.
You look at Scala code and first don't get anything because it's full of library specific symbols.
Then these implicit conversions mangle your objects behind your back and you don't even know what's happening at run time.
I like the ideas behind these features, but as a beginner, they always scared me away after looking into "production code" :\
I had the feeling that "Programming in Scala" prized them as the killer feature, back in 2009 :)
I've seen code like this pop up way more than I'm comfortable admitting.
I think there are two better ways of evaluating a language in terms of good code/bad code:
1. Does the language (and its idioms) encourage good or bad code?
2. Does the language support strong enough abstraction facilities to allow good code to be written?
Now, of course, the problem is that people will disagree about what counts as good code, but these at least make you think about what's important in a language.
Edit: Formatting
But that's just about the definition, not what it's doing which is no good. Anyone can abuse a lot of these features, which depending upon who you'd ask, is better flexibility than not having them at all.
scala> def f(x: Float) = x.round
f: (x: Float)Int
scala> f(123456789)
res0: Int = 123456792
scala> f(Long.MaxValue)
res1: Int = 2147483647
Edit: replaced "you" with "one" to clarify
I looked into stuff like Lift and Play
get {
path("foo" / IntNumber) { itemId =>
{
complete("You just looked for foo #" + itemId)
}
}
path("foo") {
parameter('id.as[Int] ?) { (maybeId: Option[Int]) =>
{
...
}
}
}
I've been using Scala for years and I've had minimal contact with libraries using symbols in any significant way (most libraries I commonly depend on use no operator overloading at all). The only ones which come to mind are some of the linked list operations and Map(foo -> bar, abc -> def) stuff.
EDIT: added infix
This is why I ended up writing a series of articles on practical implicates. Implicits are pretty much unavoidable, but the scale's website documentation is appalling. But if they are explained through examples, and shows how, and when, it makes sense to use them, we can deal with them reasonably.
Not everytime. Sometimes it could be great. Just look at some BigDecimal Code of Java and other languages.
in Scala (and Python) it's:
BigDecimal(1.00) + BigDecimal(2.00)
new BigDecimal(1.00).add(new BigDecima(2.00) and so onon the surface it looks quite Ruby-esque, but a bunch of surprises lurking in wait because it's Java underneath
I use Scala where I can at work (mostly smaller projects where I am the only developer and can thus get away with it), but every place I've worked over the last few years has been staffed with developers who are actively hostile to the thought of using any language other than Java. And right now, Java is king in the Defense industry. It's the new Ada, except I'd rather be coding in Ada than Java.
On my own side projects, I don't use Scala, because I would never choose to target the JVM for greenfield development. But that's just me.
If I'm doing some front-end stuff, I'll use Javascript. I most certainly will not use something like GWT because I want my entire codebase to be Java-only.
I've used Ruby, Python, and others. I'd like to get back into Haskell at some point when I find the time. My current little side project is written using (mostly) Go and Javascript (I'll have to post a "Show HN" sometime, maybe).
Right now I'm really digging Go for network services :)
https://www.typesafe.com/resources/e-book/scala-for-the-impa...
Works for the author, I suppose. Me, I'd prefer to have a solid grasp of the abstractions I'm using -- and an equally solid understanding of how the programming constructs I'm using correspond to those abstractions -- before using them to crank out production code.
Rust doesn't have inheritance, let alone multiple inheritance. We do have traits, which I assume work similarly, it looks like.
On to the ones that are listed under "myths":
IDE support is coming along: https://www.rust-lang.org/ides.html
I rarely hear people say that Rust has too many features, or if they do, they're comparing it to a language with a GC.
Rust's compiler can be faster, and it's getting better all the time. https://raw.githubusercontent.com/sanxiyn/sandbox/master/rus...
Rust doesn't really have traditional OOP, so yeah, a lot of it does go out the window.
Rust doesn't have some Scala features like Higher Kinded Types that lead to this kind of perception, so it's not quite the same. Our type system can be hard for totally different reasons ;). The borrow checker stuff.
Rust traits are more like typeclasses. Scala traits more more like Java interfaces w/default definitions and some other things.
Scala traits can change behavior dependent on the order you specify them. Because they can specify fields and state.
Rust traits are as far as I know not like that.
(My real fear is that Scala is just a gateway drug and if I don't take it easy I'll be main lining Haskell eventually. )
Checked exceptions force you to think about error cases upfront, that's their value. More pain for developers, better software for users.
If you want your caller to handle an error, then that should be part of the return type (e.g. sum types).
Exceptions fix that by not bothering any of these users because the return type of the methods in-between is not affected.
I think Odersky had mentioned something of this kind. Anybody else remembers something of this sort?
Under the hood it's a complete overhaul, based on the DOT calculus[2][3]. Already compile times are faster than current Scala (without any optimizations applied), and tooling will improve as well (Dotty fuses current Scala's type parameter and type member implementation into one, thus less work for IDE presentation compilers to do, for example).
Implicits aren't going anywhere, thankfully, how else could Spark et al offer such wonderful DSLs ;-)
[1] https://github.com/lampepfl/dotty [2] http://www.cs.uwm.edu/%7Eboyland/fool2012/papers/fool2012_su... [3] http://arxiv.org/pdf/1510.05216v1
What does he mean by "architecting the application correctly"? I have a project in production which uses Play framework in Java and I'd be interesting in transitioning this to Scala? What should I look out for?
This is the only discussion I could find for such a migration: http://grokbase.com/t/gg/play-framework/151wb9eebq/migrate-s...
I am wondering what Scala experts would say (I don't know Scala sufficiently to trust my judgment)?
https://github.com/lampepfl/dotty/issues/818
I think the talk was pretty influential, but atleast the community is responding. I don't think it spells doom for Scala.
It looks to me like you can't trust Scala because the people behind it are ok with accepting clearly broken or poor quality implementations.
Whatever you decide, I'd recommend rounding out your view of Scala by consulting more sources than Paul's rants. There's a lot of hyperbole there. This of course doesn't mean he's wrong, and we've certainly listened to him and fixed some of the issues he complained about (I was in the front row of that talk).
Since it's often the case big and complex languages succeed, for some definition of success, I'm sure it will be as successful as Java.
For me, the biggest problem is how fractured the community is. Scala is not a very opinionated language at all, so work from different subcultures barely looks like the same language. Working with scalaz? You won't touch the standard library at all. Finagle? Twitter came up with a Futures library before it came standard with Scala, and there seems to be no interest in migrating out of that, so you have multiple libraries for basic concurrency (along with the scalaz version of course). Those groups, that use the language so differently, do not share many interests when it comes to the evolution of the language, what parts are good and bad. This leads to much infighting, and an environment where instead of learning Scala, what people learn is that the community is divided, and often very rude. It makes learning a challenge. Each community also has a different idea of what it is considered acceptable behavior to other human beings. The scala mailing lists has postings that make your average Linus Tovalds post seem sensible and polite.
Another part that makes learning a challenge is how different application level Scala and library level Scala are. In a language like Java, it's easy to go into the standard library, or into whatever popular library you want to use, and look under the covers, even and a newbie, and learn how to write idiomatic Java (whether that's good or not). In Scala, the standard library, and pretty much every major library out there, use language features that are not needed at all for regular use, but are very valuable for building libraries, and yet make the code very hard to understand for someone new. For the worst case of all, look at the source code of Shapeless: Even very seasoned Scala programmers find it difficult to understand, because it's attempting to do things that the designers never even expected people to do with the language.
Another problem, related to the previous one, is that since it's so easy to build DSLs in Scala, pretty much every major library does it. Internal DSLs, implemented around a bunch of implicit conversions that you need to understand pretty well to solve syntax errors. For instance, say you want to use json in your spray web service. There are two ways to do it: Follow a tutorial without understanding it and hope you do not have errors, or spend the time to understand the type class pattern.
And that leads to the last, problem: There are basic intro articles out there. There's information for experts. But the route from basic knowledge to mastery is a big cliff with very little information, and it's easy for people to give up. As it is now, Scala is a language better learned tribally: You have a few gurus, and when complicated questions happen, go ask one of the gurus to see if he can teach you about whatever problem you are having, or about how a feature really applies to your problem.
I love Scala, I really do, but if the community problems are not resolved, the rest of the problems will not get resolved. It will remain a hard language to learn, and other languages that right now do not offer all the rich features of Scala will just steal what they can and replace it instead, just because they have better, less fractured communities.
We absolutely want to resolve the issues you're referring to, and we welcome everyone's help and input to speed up this process. We're working on refining our process and our plans regarding library evolution, especially in light of the recent controversy around adding a JSON module to our platform.
The first (baby) step I've taken towards bringing the communities closer was embarking on modularizing the standard library in 2.11, which will continue in 2.13. The goal was to create room for other modules to compete with the ones that were originally in scala-library and to allow community maintainers to step forward and own a module (every module now has a community maintainer, who is in charge of everything, short of promoting staged artifacts to maven). I'm very pleased with how this turned out, and we'll continue this effort in 2.13, where we'll focus on modularizing the collections library. (We alternate compiler and library-focussed releases for smoother upgrades, 2.12 is focussed on making the most of Java 8's new VM features.)
Two more examples, in which I was not involved: the design of the 2.10 Future API, which was done in collaboration with Twitter; I'm also very excited about the prospect of including Reactive Streams in JDK 9!
Personally, I prefer to write a bit of boilerplate over relying too much on extreme implicit magic to eradicate every last line of unneeded code, but I think it's cool that you can do that it you like to. (Disclaimer: I did a lot of work on enabling this implicit magic, between implementing support for higher-kinded types and improving the interaction of implicit search, type inference and dependent method types.)
The whole Scala Team is very concerned with the experience of our community in our forums, and we do our best to step in when things get out of hand, including banning repeat offenders (which we don't do lightly). Because Scala fuses so many different PL concepts, it's not possible to make everybody happy all the time.
I really like Martin's quote on using abstraction responsibly:
> [...] trading abstraction capabilities for basic features > is exactly the opposite of my design philosophy. > My design philosophy is more like free speech. > We protect it even if it is abused. In fact free speech would be a > fiction if there were no abuses. In the same way, abstraction > capabilities would be a fiction if there was no over-abstraction. > I completely agree this approach comes with challenges. It is our > responsibility as good engineers to realize that abstractions have > benefits as well as costs, and to collectively develop designs where > the benefits exceed the costs.
Regarding learning the language, I believe we have a lot of excellent post-tutorial material out there, including many books and the Coursera classes. There are many stages of Scala mastery, but you don't have to get to the boss level in order to be productive. Part of the joy of Scala for me is that you can keep learning new features and improve your understanding of how it all fits together (quite nicely, if you ask me).
Each Scala run inside SBT-enabled project takes >10 secs minimum to run. Every single run. SBT make is invoking the full SBT startup sequence and is unable to reuse the session. This comes from the context of using Play Framework where you can't substitute SBT with something else.
I do hope that SBT server becomes a reality at some point and IntelliJ Scala plugin is able to fully use it.
The issue is with IntelliJ -- when I want to run Play app from IntelliJ, not from command prompt, to get easier debug support when running the app or junit tests. Running tests from IntelliJ is more convenient than using activator shell.
Yes, I could provide -jvm-debug flag to the activator command line but this works only when fork:=false, which can't be always used.
I always just keep a console open and compile from the command line, because IntelliJ compilation is huge waste of time. I rarely debug because that means I'd have to wait 2 minutes for IntelliJ to do it's thing before it launches.
Oh, and the test output is inconsistent. It often reports that all tests pass even when they are failing.
Other than that - and it is a huge inconvenience - IntelliJ is still awesome for Scala development. Everything else the editor is capable of easily justifies it's place in Scala development.
For me it's a strong competitor to Scala, if you want to use something better than Java.
A bunch of my code is interspersed with these function vals. It's useful when within a function when you repeatedly apply the same transformation in multiple places. It kinda makes the code easier to read once you stop making that distinction.
re:short functions - I am not as much in favor of them as I used to be before. IMO, splitting functions into smaller ones are fine as long as there is some "meaning" to splitting them ( mostly related to your domain logic ). But I have stopped splitting functions just to make it smaller. When your code is referentially transparent, I think the size of your function really matters less.
Scala initially is a product of a single remarkably well-trained mind, instead of committee of mediocrity.
Clojure, BTW, was a solo remake of Scala's approach (Java interloop, emphasis on collections, immutability, etc) while starting from Common Lisp (strong, dynamic typing) instead of Standard ML (strong, static) as its basis.
Actually, the Scala course on Coursera is very good start.
Hindley-Milner type inference?
I really miss having proper type inference in Scala. I'll admit, the "we will infer the types as whatever javac would have said they should be from left to right" is a lot more effective than one would think, but still feels rather primitive.
I try to be 90% scala, but I hope I get other stuff a fair shake...