Scaling Scala | Hacker News Reader

Scaling Scala

oreilly.com

93 pointsm124548 comments

What strikes me about the Scala community is how amazingly anti-fragile it is.

Odersky often uses criticism to refine future versions of the language. He is currently my favorite benevolent dictator.

With handling criticism it all comes down to the mental attitude:

Is your project just a tool to get a job done or do you draw some sense of self-worth and life purpose from it?

Former is a healthy attitude, latter would make your life miserable.

Before you start trying to scale with Scala/Spark, consider if my preferred progression makes more sense:

* < 50 million rows: Can your data fit into a standard RDBMS? Writing all your business logic in SQL might make more sense.

* < 1 billion rows: Can your data fit into a few flat files on disk, with some Python/Perl scripts to manipulate them?

* < 1 trillion rows: Can you use an off-the-shelf solution like Google BigQuery or Amazon Redshift? You can also use this rather than flat files depending on the skillset of your team.

* < 1 trillion rows: Can you engineer smart data structures & algorithms, write some custom C++, tune the compiler flags, inspect the resulting assembly, cache intermediate results on disk, etc. for a small number of core queries?

For anything larger, good luck. You seem to genuinely need a dedicated data engineering team, and similar tools may very well help you scale up. Even so, try to keep things simple. A single binary with well-defined inputs & outputs(even poorly written in C++) is intrinsically simpler than a distributed network(even if the Scala code itself is simpler and well-written), just due to deployment considerations.

If you need things to be accessible in real-time from a website, that may also require a sophisticated solution. Consider if you can queue the query and use one of the above approaches, however. It's even harder to deploy a real-time distributed query system with worst-case performance guaranteed to be queryable from the front-end of your website.

I'm averse to writing SQL simply because it's a bit harder to test business logic with and things get hard to read very quickly. Also consider the example where my logs are in json for whatever reason, now using SQL just makes things harder especially when I need to run some sort of transformation.

Also SQL is in clear text, which invites small manual changes outside of version control.

I find Scala/Spark easier to write and work with than any of the options you list (yes, SQL included). It gives you a real first-class programming language and lets you reuse the same code in your analysis/aggregation as you're using for your main product.

Two points: article is 5 months old, and it's fantastic: informed journalism rather than "one coder's opinion."

One of the main problems with Scala is that it has a clusterfuck of niche use features, which even experienced developers have problem using.

Scala core library is littered with :: +: and other nonsensical operators. Things like implicit may have few good use cases, but are mostly abused and make code hard to read. Blindly following immutability in even local function variables and stack makes code slow and unreadable.

Eventually you get to understand these operators (and, after all they essentially mean append/concatenate/join in one sense or another... so it's basically add stuff to other stuff), and as for implicits, they make code much easier to follow if they are used sparingly and for good reasons. What makes Scala code (with implicits) hard to follow can be type classes, and understanding how all the pieces of the language fit together in these cases.

As for immutability making things slow... Well, personally I start writing code for humans (and immutability makes the code much easier to read for humans). _If_ speed is an issue then I consider dropping immutability. So far have had no issues with Scala. Also depending on how much speed I need I wouldn't even think about using Scala.

That's what they say about perl. Keep trying and itll make sense!

Granted, scala can be written to read much easier than perl, but that argument in particular don't carry much weight imo.

The thing is, in Scala you can do really crazy and complicated stuff (see Shapeless, Cats, etc) that in other languages (with exceptions, of course) would be extremely hard or downright impossible.

In Scala, it is just hard, but doable, but implies code being hard to read because what you are doing _is_ hard. This does not detract from doing as much as you can to make it readable, but sometimes it is impossible, or at least impossible to someone not used to it.

I'm not good enough with Scala to understand some of it (yet, I hope) but this is something that is bugging me lately. Some things (in general, I'm not talking just about programming here) are hard. No matter how much sugar-coating, some stuff is hard to understand, complex and not everyone can get to the bottom of it. It just happens in some programming languages, happens in most of mathematics, physics and computer science.

> Scala core library is littered with :: +: and other nonsensical operators.

They are very useful. With the two you mentioned (:: and +:) they can be used for pattern matching in addition to concatenation (of a list / sequence), for example:

    scala> val list = 1 :: 2 :: Nil
    list: List[Int] = List(1, 2)


    scala> val x = list match {
         |   case head :: _ => head
         |   case _         => 0
         | }
    x: Int = 1

    scala> val seq = 1 +: Seq(2, 3) :+ 4
    seq: Seq[Int] = List(1, 2, 3, 4)

    scala> val y = seq match {
         |   case a +: _ :+ b => a + b
         |   case _           => 0
         | }
    y: Int = 5

It can be a bit confusing at first, especially for mutable vs immutable collection operators. But you end up remembering some, if not most of it, after using them a number of times.

and that's what I meant, when I said niche use cases. You can replace all operators with names and still have no difference. At some places code will be verbose, but on whole code will be more readable. Another example if you want is http://www.scala-lang.org/api/2.9.2/scala/sys/process/Proces.... All operators can be replaced by completely readable names like run, build, read, write.

Scala on whole is optimized to have lesser no. of characters in code, in contrast to verbosity of Java. Case classes and lambdas are what good came out of it, I understood the concept and now I can use them. Operator overloading and Implicit are what move pendulum to other side, they are not some novel concept and make very readable code, unreadable.

Why do we use operators at all? We can just as easily write out plus, minus, multiply, and divide.

Concatenating to a list and pattern matching are extremely frequent operations in Scala. Some of the operators they use have been in use for decades in the languages that influenced Scala.

Like many features across many languages it can be taken too far at times and people that enjoy Scala talk about them all the time. That said, the similarities of your example to operators I use at the command line every day makes it nice to read for me.

IMO the ProcessBuilder class is not a very good example, as the operators are designed to match the existing shell / bash ones (e.g. `#>>` is similar in behaviour to bash's `>>`).

The main long-term of Scala is nobody is making money on it. Lightbend tried (formerly called Typesafe), but haven't been that succesful and pivoted to other tools (Akka).

Java also stagnated, because money obssesed Oracle is not earning much money on it.

Good point. Seems like a language either needs to be non profit and open sourced, or profitable and closed source.

Non profit and closed source seems like a formula for doom and destruction.

> Non profit and closed source seems like a formula for doom and destruction.

Which is not the situation here.

It seems very hard to make money off of a language unless you can get support contracts from corporations. Which in the language rich world is very hard.

No company is trying to sell Scala on its own. Several startups emerged around major open source Scala projects and are doing fine. Lightbend employs core contributors to the compiler and stdlib. There are also very successful consulting shops around Scala technologies that are contributing where they can. The language and its ecosystem haven't seen anything but growth, I'm not worried.

Is Java really stagnated? I was under impression Java 8/9/10 have stable schedules and are delivering some long sought features like lambda, value types and REPL. It is much better than when Sun had lost all money and future was bleak.

Given that a fair few people at EPFL seem to work on it and there seems to be a tendency for thesis projects to be written on Scala or to be updates/improvements to Scala, that alone seems like a decent source of updates (if not for stability, though). Perhaps Scala can run on academic grants (into programming languages research, or just a small portion of researchers' other work)?

Also the work on Dotty seems to be progressing apace, folding the community in quite a bit (https://contributors.scala-lang.org and https://github.com/lampepfl/dotty/issues)

> (Spark, Mesos, Akka, Cassandra, Kafka), most of which has been written in Scala.

Regarding Kafka, I noticed a large amount of Scala API is deprecated and replaced with Java API in the latest version. There is about twice the Java code in Kafka as compared to Scala.

Mesos is C++.

Cassandra is Java.

So it seems to me Spark and Akka is mainly Scala. And in this field I am expecting Rust to make big inroads with high performance, no JVM like overhead for memory safety, and lately emphasis on async IO with tokio.

I wouldn't expect rust to take much scala market share. A big part of he scala advantage is being able to take advantage of the massive number of jvm libraries available.

The kafka client api moved to java because of fewer dependencies to distribute. The brokers are still a scala project.

I'm not quite sure why the Scalaz IRC behavior issue got such a focus here, but here's the chat logs of one of the incidents https://gist.github.com/betehess/77de5b4b35ae11801936

Idk, tony is my programming role model. Fantastic guy, extremely knowledgeable, explains things in very approachable way. Love him to death.

He's helped me fix MANY stupid type puzzles, and scalaz scumbags ran him out of the entire language.

I mainly do f# and erlang now :(

Scala has plenty of problems aside from the users. I can't begin to explain how terrible sbt is to people who haven't suffered its grasp. Sbt imo is the single most critical obstacle preventing widespread adoption. It's awful.

There's too much politics in the scalaverse. The people are too whiny, the tooling whines even more than the people, and its a damn shame because it's really great code at the end of the day. Its the most fascinating objfunctional I know of, though F# isn't bad at all. Great tooling too, but poor native support.

What do you find problematic with SBT ? I'm from the C++ camp and I just love SBT, simple and powerful, so I never really understood what the issue was with SBT (i've seen other people complain about it too)

Too many weird macros, too many non-english operators, too much weird bullshit with Ivy, that if you install scala from source sbt will fuck it up if you try and do coverage testing on different versions, build times are ridiculous (this has improved quite a bit since I first used sbt, but it's still embarrassingly slow compared to other build systems (FAKE comes to mind immediately, or rebar3 is real nice too, hell I still use rake for when I have an obnoxious code structure with ten million subdirs, and it's great if you use jruby)

What's good about it? What about sbt makes it better than any other build tool OTHER than the fact that it lets you use scala?

I don't mind that SBT uses a bunch of advanced concepts and type safety, that's fucking AWESOME, but it's so difficult to wrangle it to just build something correctly, it seems like there's such a huge time investment to learning it to a threshold where its features are useful enough to make that investment seem worth it.

Several years later, I still have not reached that point. It seemed like such a pain at first to learn other languages (Erlang syntax still confuses me sometimes) but then I started completing components in these languages and there were moments when I started working on prod builds, and using rake, FAKE, rebar3, etc, and it really put into perspective how much time I had been spending internally screaming at sbt to just DO IT already.

There will always be a special place in my heart for Scala as a language, and I think it's one of those languages we'll look back on in 3017 and think "Damn, we learned a lot then"

but I'm getting over it in my day-to-day work. I'm also being slowly enslaved by .NET corp overlords now, but I've been complaining about SBT for a LONG time before this, and a lot of other people have too, sometimes for the same reasons that I hate it, sometimes for their own special reasons.

It causes way too much friction for Lightbend to continue to look the other way. It's cool when you read the list of features, but write a complicated build and look me in the eyes and say you enjoyed it, you just can't.

why not just use Maven?

I don't want to trade a shit for a turd.

And I also don't want to learn another jvm build system. I've long awaited a day where scala native can start replacing things, but they've already put their sbt in their coffin.

https://github.com/cvogt/cbt [0] is under very active, heavy development and seems like a potential beacon of hope to SBT-haters (including myself).

The CBT author and community seem determined to genuinely make it a completely viable SBT replacement (it already works pretty well).

Given that during the earlier part of my career starting a new SBT project would involve 1 - 2 days of wrangling every time (there's always something simple that needs changing, that's inevitably harder than it need be in SBT), I can't wait to see CBT mature.

[0]: In their own words: 'fun, fast, intuitive, compositional, statically checked builds written in Scala'

It'd be the first build system for any JVM language that would be nice. I don't know what it is, but they all seem to come with various amounts of suckage (started with Java when GNU Make was the most advanced option, via ant, maven, gradle, buildr, sbt, ...). It must be the JVM. I worked with Scala for close to a decade now, and my conclusion is that its advantages (type stuff) are mostly theoretical, and its drawbacks (sbt, type stuff, the JVM, slow compiler, the mess of features) are mostly practical. I'm happy to have mostly moved on.

What do you use now? Have you tried F#?

While not on the same level as Cargo, I'd say l Leiningen is a very robust and nice build system for the JVM. Miles ahead of SBT.

Yeah but who would want to use clojure of they had a choice to use scala?

Lein is what I always wished sbcl had.

I've known about cbt but it's a hard sell at a corp, esp when lightbend doesn't seem to want anything to do with it. They'd rather pretend sbt works just fine and that we're all idiots rather than laying sbt to a graceful rest and using cbt.

The Scalaz vs Cats issue (and seeing what happens to them over time as a 'community judgement' on whether the CoC decision was correct, as in the article) is muddied further by the fact that Cats is different on a technical level too (it's not just a fork of Scalaz).

Cats is more approachable than Scalaz. Whilst you could ostensibly see it as an extension of their CoC principles, I think Cats mostly just benefits from being much, much newer and so has more freedom and power of hindsight.

The community both shrinks and grows. An example of the former:

https://www.reddit.com/r/programming/comments/6bh8xv/leaving...

Aren't alternative JVM languages subject to the same potential legal issues that sparked the Oracle lawsuit? Don't languages like Scala, Groovy, and Kotlin still make use of many Java APIs?

kotlin does not reimplement java API. Instead it have it's own "wrappers" for collections and primitive types. This wrappers are just platform specific types that is replaced directly to native ArrayList and others by compiler. This is more like just using Java API instead of reimplementing it. If Oracle will sue someone for using their API to build Java apps - it will be the end for everything.

True, and Scala is exactly the same. It uses Java APIs.

Apache Groovy is also the same.

Why is that in every discussion about Scala some Groovy fanboy pops out? sometimes with a ridiculous claim like "groovy is more popular", or "groovy has better java interoperability"

This account's comments have already been violating the HN guidelines. We ban accounts that do that. Please read the following and post civilly and substantively, or not at all.

https://news.ycombinator.com/newsguidelines.html

https://news.ycombinator.com/newswelcome.html

Probably not, as long as they distribute correctly under the terms of the license. Android didn't follow the license correctly (although they've since fixed that), which is why they ran into trouble.

Oracles lawsuit is centered around Dalvik which is a JVM implementation, I have a hard time understanding on how generating code and using an API is the same thing as that ? But I may miss some nuances here.

They are using GPL'ed version of those APIs, so no.

Google / Apache re-implemented the APIs in a way that conflicted with the GPL.

Reply on news.ycombinator.com