Project for bringing macros to Scala | Hacker News Reader

Project for bringing macros to Scala

scalamacros.org

39 pointsmike_esspe26 comments

I'm not that familiar with Scala, but the examples aren't very convincing. Why must `printf` be defined as a macro? Why not a plain old function? What are the benefits to the second example?

The other thing that bothers me is the empty 'Use cases' page. Scala macros might be a solution to a problem nobody has. A problem that people _do_ have is the breadth of Scala language, and macros will add to the conceptual dog pile.

You can make printf type safe with macro, so it will complain at compile time if you are passing incorrect type.

This lets you embed programs in other languages (DSLs) within your program that are parsed and compiled at compile time instead of run-time. This has the following benefits over run-time DSLs / languages embedded inside of strings:

1. The embedded language will be type checked at compile time. For example a regular expression macro could enforce that your regular expression has balanced parentheses in capture groups. This potentially lets you shift a class of errors from run-time to compile time.

2. The language will be parsed at compile time saving run-time cycles.

3. The host language can be extended and experimented with without adding new syntax. Features like this actually limit the growth of the "dog pile", since many language experiments features can be tested as macros first.

Some common DSL examples are regular expressions, parser generators, SQL / ORM generators and LINQ

An impressive example of what you could do with macros is LINQ - a static and type-safe DSL for querying collections of any kind. People did impressive things with it, like offloading the work to the GPU.

Another example would be about adding your own improvements to the time-system, improvements that are actually going to help you keep the code safe. Like the units of measure from F#.

However, adding powerful macros to a language that was not designed for macros and that is NOT simple ... I do think it is a bad idea. I'd be happy to see in Scala something like the code quotations from F#. Anything less is useless, anything more will make Scala way too complicated.

> Scala macros might be a solution to a problem nobody has

Pls read slide 10 of this document - http://scalamacros.org/talks/2011-10-18-WhatsUp.pdf

--Proposed Code snippet with macro ---

class Queryable[T, Repr](query: Query) { macro def filter(p: T => Boolean): Repr = <[ val b = $this.newBuilder b.query = Filter($this.query, $reify(p)) b.result ]> } val products = db.products products.filter(p => p.startsWith("foo")).toList --

My takeaways : 1. adding LINQ support to Scala via macros 2. macros virtualize XXX, where XXX = vals, vars, defs, types, classes, traits. So macro XXX can be used in place of XXX.

The author also says "We can tailor macros to fit your particular use case". He mentions OR-mapping, and a Scala-to-javascript compiler that might end up using these macro invocations, among other things.

Thanks for pointing out the weak spots of the content!

We've posted several compelling use cases enabled by macros (http://scalamacros.org/usecases/index.html) and added some more information to the main page that uncovers the inner beauty of the examples.

Btw, we realize the responsibility of adding a competely new feature to Scala, so every new macro-related idea is being weighed against the complexity it brings to the table. So far we've done away with several really interesting prospects in order to keep the overall proposal simple.

This is a significant breakthrough. Scala is once again showing the way forward. Instead of having ad hoc macros (which everyone knows are bad), Scala macros don't change the syntax of the language. Thus they are no more difficult to come to terms with than an API for a well-written library. In this sense ordinary API's are going to start looking like crippled Scala.

The other nice thing about these is that you don't need to lose type safety to use macros. That's really where the innovation is here.

I don't yet fully understand how this all works at the type-theoretic level. I'm imagining that expressions (as opposed to the values they represent) are thought of as belonging to higher kinded types. The macros then reject expression which aren't of the right kind.

I have to say, I am absolutely flawed by the comments so far about this. I think this is going to have an absolutely pivotal effect on future programming languages. This is how macros should always have been. I don't know of any other language which does this. It's a totally new area in language theory to explore.

This is how macros should always have been. I don't know of any other language which does this

They look very similar to Perl6 macros. See recent discussion here: http://news.ycombinator.com/item?id=3124920

Macros are bad, but in order to understand why they are bad it helps if you are not a language nut. Most language nuts have one shortcoming: they do not understand their consumer.

If you want proof you only need to look at the error messages generated by most compilers. They probably make perfect sense to someone who is trying to debug the compiler, but they usually leave the consumer, ie the programmer using the compiler, somewhat confused.

Macros are bad because they redefine the language. The point of having a language in the first place is to make it easy to communicate. This is only possible if the semantics are understood. If you do good language design the semantics are clear. If you do sloppy language design people have to look things up all the time.

Now, if everyone gets to extend the language in arbitrary ways, you have suddenly unloaded a massive burden onto your programmers. It is no longer enough to say that you know language X. In order to understand the software you have to understand the specific dialects (plural) of language X -- because for a non-trivial project with lots of people extending the language in various ways, you will end up with multiple dialects.

When you can no longer bring up a piece of code in your editor and see what it does without having to reference dozens of other files that may modify the semantics of the language (sometimes in non-obvious ways), you are going to be a lot less productive.

I think Joshua Bloch was absolutely right when he said that one of the really good decisions of Java was to NOT have macros. He made some of the same points I made above.

Now I understand that people think macros can lead to elegant code, but that code only has value if other people can understand it. If you need to re-learn the language, in big or small ways, to understand it, its value is very limited.

And again: what excites language nuts and what has real value is not always the same thing.

You make some valid points, but they would be better made if you toned down your rhetoric about people who work on programming languages.

I'm an engineer at heart. One of the hardest things that I have to do on a daily basis is to ask myself where what I think is cool conflicts with what the user needs. Or indeed if my cool idea will harm the user. As engineers we tend to focus on the cool bit. I too have fallen into that trap and I expect to do it many times in the future. But I try to be aware that this can and will happen.

I think asking language nuts (which isn't limited to those who write compilers, but also extends to practitioners with deep knowlede) to be aware of this is particularly important in that people look to them for enlightenment and advice -- and when they do harmful things it has a huge multiplier effect.

This is a very valid concern, and we keep it in mind. Several really fancy ideas were rejected simply because that added too much complexity or could get out of hand quickly. That's why, for one, the main page of the project says that "Scala macros ... cannot change syntax of Scala".

Some people from our team do have a painful experience with Lisp-style macros, so we try especially hard to find a minimally useful yet conceptually clean approach to compile-time metaprogramming (we are sure, though, that this sweet spot exists).

And that's why we stress the importance of feedback and appreciate it. Thanks for sharing your thoughts!

Macros have their place, but not as the primary basis of a language (sorry, Lisp), where most things unnecessarily become macros to avoid explicit function literals or quoting. Then to do anything reasonable you're using several language "extensions" at any one given time, which can only compose in ways the macro writers forsaw. Most of what is done with Lisp macros can be done with first-class function literals, in a way that makes it pretty clear that non-linear control flow is going on.

Macros in Scala? I don't know, the language already seems quite complex without introducing custom syntax. But if they're more of an occasionally-used compiler extension for specific internal DSLs rather than a by-example use-them-everywhere, I don't see them hurting.

That being said, I'm quite enjoying taking a break from JVM hell and doing some exploratory programming in Racket. If special forms are well documented, widely used, and syntax checked, the extra mental overhead isn't burdensome.

Macros are good, but in order to understand why there are good you have to lift your head out of the common mindset of most developers that spend half their days configuring their application in some poorly designed XML-based DSL that's only necessary because doing the same thing in their programming language is too awkward. They coincidentally spend the other half of their days writing code in the same basic procedural style they learned as a kid, copying and pasting bits of code everywhere and making a general highly-coupled mess because they don't how to do it any better.

It has been said that at any given point, every project implements their own programming language, usually as some kind of DSL, and they do it really really badly. I've seen this borne out in practice in EVERY commercial project I've worked on. I've seen it, I've lived, and I continue to live it every day. Macros are one of the ways out of this mess that we've created for ourselves. While a lot of macro research is still cutting edge, slowly different techniques are becoming more proven and starting to disseminate more and more into mainstream languages and runtimes. The problems you mention with macros have honestly already been dealt with in the past decade or two.

I've dealt with people like you at work before. Your kind are those that stare at any abstraction they don't understand with immediate suspicion and reject it without any real consideration. They refuse to learn new things or improve their programming abilities or toolset. While this in of itself isn't a crime, they hold the rest of the industry back by rejecting superior technologies as they come along because they refuse to take the time understand them.

Your kind and mine, sir, are in direct opposition.

Well said. I'm reminded of the pg's Blub paradox, and Rich Hickey's recent quote, "I can't read German; does that make it unreadable?" both at the same time.

I think a better analogy is struggling to understand odd (spoken) english dialects: yes, they are speaking english, but not the variant(s) you know.

I don't understand what you are trying to say.

On one hand you are saying that most projects you have worked on end up creating their own language, usually as some kind of DSL. Then you argue that the way to counter this is to ... give people more tools to craft DSLs.

So what are you trying to say?

PS: you seem to make a lot of assumptions about me based on ... what exactly? I don't believe we've met and whatever you are "in direct opposition" to is simply a straw man you have imagined.

I am an average programmer, I know C#, Java and dart. Can anyone explain macro's? Wikipedia/internet is not very helpfull because of ambiguity. Thanks!

We're all average programmers here that know C#, Java, and/or Dart, but here goes!

A macro, in its simplest form, is a function of type syntax -> syntax, or, a function that takes syntax as input and produces syntax as output. These functions are invoked at "compile time" as the compiler is processing your source code, and the resulting code is then compiled or interpreted along with the rest of the code. The way you write these functions, what these functions are capable of doing, and even the precise data type of syntax varies between the languages that implement macros.

The most powerful macro systems allow you to "extend" your language with new syntactic constructs and features. In this sense you can think of macros as a way for you to team up with the compiler to improve the language. Macros also allow you to generate repetitive code that would be tedious or error-prone to hand-write and maintain on your own.

The easiest way to understand good macro systems would be to learn about them from a Lisp-variant like Clojure, Scheme, or Common Lisp, as macros have typically been the domain of these languages. It is still an active area of language design research to try to effectively integrate a good macro system into non-Lisp languages. Scala's macro proposal is supposedly heavily inspired by a language called Nemerle, which runs on the .NET platform. I don't know much about Nemerle, but it appears to be innovating very well in this particular space.

The Dylan language (witch had Lisp syntax at first)probebly had one of the early macro-systems in a language with syntax. Its ok to work with but much harder then a normal lisp.

One of the designers of Dylan tried to push this in his new language called PLOT (Programming Language for Old Timers). It has pythonic syntax and provids a full powerd macro-system. Unfortently there is no implmentation of PLOT just a discription: http://users.rcn.com/david-moon/PLOT/

if you know C# and .net and want to learn macros then check out Nemerle. it will be a familiar and excellent setting to learn functional programming and metaprogramming.

The best way to learn macros is to learn a Lisp. Macros are easy to grok in Lisps. Regarding languages, I'd pick Clojure since you already know Java. In Lisp, you write ASTs directly. Syntax is very lightweight. For example, 3 + 4 * 5 is written as (+ 3 (* 4 5))-- the form compilers in other languages translate it into. The prefix notation and parentheses make parsing unambiguous; you don't have to worry about left vs. right associativity or order of operations.

The evaluation strategy for Lisp functions is to evaluate each of the inputs before making the function call. In the example above, (* 4 5) is evaluated first before the call to the function + even begins; once you have the intermediate form (+ 3 20) where all the parameters are fully-evaluated, the function call happens.

Why are macros important? Let's say that you want to write a short-circuit AND. You can't do that as a function. For example, (and false (/ 1 0)) will return false (never evaluating the right-hand expression) if you have short-circuit AND but it will raise an error (division by zero) if you don't. If that were a function, that call would (undesirably) fail. So what this means is that you can't write the short-circuiting AND as a function. You need something else that operates on expression forms, not values. In practice, it's a macro whose behavior is approximately like so:

(and <X> <Y>) => (if <X> <Y> false)

This is a transformation of code that happens during compilation, not a run-time function call.

It's worth noting that IF is not a function or a macro. It's a "special form" provided by the language. The evaluation of if-forms is not like the evaluation of function forms; some branches are never evaluated. It requires special-case handling.

That's one use for macros: writing control-flow primitives. There's a lot more to them as well: code generation to eliminate boiler-plate, DSL creation. Common Lisp, for example, has an object system built on macros, and a (controversial) LOOP macro that is its own DSL for expressing iterations, allowing idioms like (loop for i from 4 downto 1 collect (* i i)), which would return (16 9 4 1).

Macros can be very useful. They're pretty much essential because of how Lisp works, so every Lisper ends up writing a few, often to substantial benefit. That said, macros can also be abused to make write-only code. You have to be careful with them, and the maxim, "Never use a macro when a function will do", is certainly worth heeding.

Thanks. The first time an esoteric language actually seems interesting to learn ;)

A macro is a kind of function which is executed sometime before runtime, like at compile-time.

Observe (aaa bbb ccc). If you just look at it as a datastructure, it's a list of 3 elements. If you look at it as executable code, then aaa is the operator, which operates on bbb and ccc.

If aaa is that special kind of operator which is a macro, it can rearrange (aaa bbb ccc) to be something different, which is then executed. Anything. It replaces itself with something else. If you programmed in a language like that, you'd never have to wait for some language designer to create a new for() loop. Because you could easily write a loop which is then transformed into the usual for(i=0;...) loop.

Macros are for hiding common patterns in your code. Instead of typing out a given block of code mutatatis mutandis every time you need to use a slight variation of it, you code up a macro for it and just call the macro each time.

They work by not evaluating expressions but treating them as blocks of unevaluated code. The blocks of code can then be manipulated, transformed, split apart and reassembled all using the full power of the language. You aren't manipulating the values of those expressions, but the code itself.

The simplest example of a macro is a function which does not evaluate its arguments until you specifically tell it to do so. But they can be arbitrarily more complex than that.

Lisp is very flexible because it has no syntax to speak of and code is data. So manipulating the code is the same as manipulating data. With languages like Scala that have syntax, macros have to work with either strings representing the language elements or objects which represent them (expression trees as they are called in the post). In other words, they work with some abstract version of the syntax of the language.

You might be interested in reading my first talk about macros: http://scalamacros.org/talks/2011-09-27-ProjectKepler.pdf and taking a look at the newly posted use cases: http://scalamacros.org/usecases/index.html. If you get stuck, comment here or mail me at eugene.burmako@epfl.ch - I will explain the unclear stuff.

Reply on news.ycombinator.com