Hacker News Reader

This is exactly the field that I'm working on, so I'm quite interrested by this.

In my opinion, the problem is more of a tools problem than a language problem : One language that has one of the most interresting support for this kind of things is C, with coccinelle [1] [2].

Google has been working on absolutely amazing tools for C++ that applies programmatic refactorings in a distributed manner on absolutely enormous codebases [3].

This IMHO shows that this is a tools issue rather than a language issue. Coccinelle had to develop a swat of parsers for its project, and the Google project is using Clang as a basis for structural and semantic capabilities. C and C++, with their fragile type systems, preprocessor, horribly hard to parse syntax, might be the worse languages amongst typed languages to develop such projects on. And in spite of that, they are the ones for which such project exist, because they are the one with enough need for such tools.

Of course some languages are more amenable to such tools than others. Very strong static typing and a solid package system helps a lot. No metaprogramming helps too (because you don't have to handle the transformation).

I'm working on such tools for Ada, that is pretty much the perfect language for this as far as imperative languages go. The essential need for such tools is to have a compiler that exposes some services as an API, most notably the ability to explore the AST and query cross references for language entities. This was the fantastic insight of Clang/LLVM in my opinion.

[1] Coccinelle semantic patch language : http://lwn.net/Articles/315686/

[2] Presentation on refactoring with coccinelle http://video.rmll.info/videos/coccinelle-automated-refactori...

[3] Clang MapReduce -- Automatic C++ Refactoring at Google Scale http://www.youtube.com/watch?v=mVbDzTM21BQ

Interesting! I knew about clang and what Google is doing there. It is definitely related to the point I'm trying to convey.

I'll be taking a close look at Coccinelle. Thanks for the link.

And as you say, C and C++ "might be the worse languages amongst typed languages to develop such projects on," due to things like the preprocessor, hard to parse syntax, etc. So I'm saying, perhaps it is time to invent the programming language that is most amenable to this.

It's amazing what we can do if we put our best engineers onto the hard problem of achieving this with C / C++ code. Imagine what might be possible if we designed a language from scratch with the primary goal of making this easy. As far as I know, no programming language has ever been designed with this as its primary goal (I'd love to be corrected if I'm wrong).

I think it was either Grace Hopper or Barbara Liskov, that once made a statement that C entry into the mainstream computing has set the compiler development back to stone age.

> A programming language that could be PROGRAMMATICALLY REFACTORED would be a HUGE home run.

The go people are poking at this. They ship a tool, 'go fix', that can safely refactor your code to deal with backwards-incompatible API changes. Go also has a tool, 'go fmt', that most people use to enforce tabs vs spaces, etc., but has a 'rewrite' flag that you can use to do simple, arbitrary transformations.

They also ship some AST tools for go, though I haven't seen much use made of them.

One recent use of the Go parsing and printing packages is goimports, a tool that automatically adds and removes import statements based on the code you've written.

Try loading this snippet, checking the "Import" box, and clicking "Format": http://play.golang.org/p/jS4s_Xz26v

To use it locally:

    $ go get code.google.com/p/go.tools/cmd/goimports

I was on play.golang.org last night and didn't see that button.

When did that get added? That's brilliant!

We added it this week. :-)

Clojure has Slamhound. Yeah, I know you didn't want to hear about LISP...

ennef is not charlieflowers (in fact he is Andrew Gerrand, one of the official Go devs).

Plus, I love Lisp! It even has a lot of the characteristics that would allow what I'm talking about. I was just trying to articulate that what I'm saying requires more than just SEXP's and Lisp macros (although that does go a long way).

And Slamhound is definitely barking up the same tree. Very interesting!

Yeah, SEXPs and LISP macros to a full refactor aware environment is like getting to Riemann surfaces from set theory.*

*The small problem with this example is that in fact the latter has been achieved.

I suspect that this is a very clever joke. And possibly that I might be the butt of the joke. But I suspect that even so, the joke is so good that I'd still laugh heartily at it.

Unfortunately, I freely admit that the joke goes right over my head. I used to know something about Riemann surfaces, so I could invest some time and eventually get the joke perhaps, but I just don't have time now. :)

Sorry, the chain is just

Set Theory Natural Numbers Integers Rationals Real Numbers Complex Numbers Calculus Riemann Surfaces

(as the Apple ads say, some steps skipped)

The only point I was making was that the level you were thinking at was much higher than the level SEXPs are at. Trying to envisage how we get there remains hard. TL;DR I was agreeing with you. :)

I haven't delved into Go, but have occasionally "glanced" at it with curiosity. One of the most powerful principles I got a whiff of was that it was designed up front with the goal of providing a powerful tool chain -- an open parsing library and the like.

That provides some incredible power and may be the thing about Go that impresses me the most.

What if we also made programming language decisions about syntax, modularity constructs, etc., all with that goal in mind? (And perhaps Go has to a large degree).

Can this become part of Gofmt triggered by a flag? I love it, and feel it should really be part of an installation.

Goimports is a fork of gofmt, so you can use it as a drop-in replacement. We're considering whether to merge the functionality back into gofmt. Goimports is still pretty experimental, so this might not happen until Go 1.4, if it does at all.

Heck I'd be thrilled to have a smart identifier/type renamer. Maybe it's just me but I like things to be named properly but I tend to be very bad at it until its 3rd or 4th use. I also like short(ish) names, so many variables tend to have the similar names across scopes which makes it impossible to do a quick replace-all.

You should be able to specify a type name, identifier, or module and its' replacement (optionally file & line to be specific) via CLI. It should do the replacement automatically, and fail if it at all changes the semantics (e.g. clash or variable shadowing) and give an overridable warning on a non-semantics-changing possible problem (like an unused shadowed variable).

Sure, this is a normal feature for an IDE, but I feel like it should be simpler (and I don't like IDE bloat anyways). Maybe the tools you mentioned can already do this.

http://golang.org/pkg/go/ast/

this is a pretty powerful package.

This is the main thing functional programming theorists get excited about when going off the deep end about sufficiently smart compilers. (That and "proving a program is correct" which is a related problem). Basically all FP optimisations would instead work by automatically transforming a program.

The idea behind all these type systems is really to allow automated transforms which preserve the integrity of the program.

It's really too bad that Haskell, with it's referential transparency, doesn't have a way to rewrite things programmatically (well, none that I know of). Rewriting terms would be great in a language like that. It seems that when a typeclass has some laws, and you want to show that an instance of it obeys the laws, you have to bust out the pen and paper. Nothing wrong with that, but it seems like a language like that could support computer-aided rewriting.

Then there are languages whose evaluation is based on term rewriting, like the language Pure.

EDIT: You do have HLint, and the bot that rewrites expressions to point-free style.

I agree that functionally pure code could allow for amazing programmatic rewriting. Imagine coupling that with homoiconic syntax. Not necessarily to have Lisp-style macros, but rather so you could offer repl-based programmatic rewriting/refactoring tools, where the tools know that your code is pure and thus can be "reasoned about mathematically."

It's close: http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/rewri...

I acknowledge that is not precisely what you were referring to, but it is close.

Alright, looks like I may be downvoted to oblivion. I still believe I'm on to something.

Would any downvoters care to take a minute to share what you think I'm missing? Seems like the polite thing to do. I know you're busy, so make it short and sweet if you have to, but if I'm wrong/naive/ignorant, please, help me out.

There is a reason there are so many languages extant. They each serve a purpose.

What exactly do you think you're on to? Insight has built all of the things that you're hoping to replace. So what are you bringing to the table?

> Insight has built all of the things that you're hoping to replace.

Who/what is "insight"? I didn't follow this sentence.

> So what are you bringing to the table?

If this has been done or thought out before, possibly nothing. But I'd be OK with that, because I'd be excited to see the results. If you have a link or a search term, please pass it on.

Here's my question. What does "ground up" buy you? It is ideologically good, for sure. But it requires a viewpoint about future programming that is similar to existing programming: we'll write systems that are hard to maintain and are burdened with a lot of code doing relatively low-level data manipulations whose implementation can be automated.

But if our future programming environments are "smaller and sleeker" in all respects(as is anticipated by, for example, the VPRI work on STEPS) this would be the wrong optimization to make. The cost of maintenance will go down across the board because the new languages let us express the change with less effort, and then complicated refactoring becomes less necessary again.

(Counterpoint: We just build even more complex systems and then need better refactoring tools. An endless cycle...)

I love a lot of the principles behind VPRI and STEPS. But it strikes me as a very futuristic ideal, not something we're close to being able to use concretely and productively today.

And I get the irony! The exact same thing is true of my post. But I think I'm not trying to shoot as high as you are.

If we can find a way to make the VPRI/STEPS vision a reality, I'm all for it. I think what I'm proposing is a less ambitious interim step.

I still believe I'm on to something.

I didn't vote at all, but you state:

In other words, the original language has to capture more of the programmers intent (probably by inferring most of it).

But I think that your comment fails to provide (1) an argument why homoiconic languages such as Lisp and Prolog do not fit the bill, since they are easy to rewrite (code is data); and (2) how this hypothetical language differs over languages that are easy to parse and have simple semantics, such as Java and C#, if most of the intent needs to be inferred.

I'd rather like to see better libraries for e.g. Java (Groovy provides the REPL :)) to do this, than yet another language.

(1) Homoiconic languages are actually a smaller scale instance of exactly what I'm talking about. They were designed from the ground up to allow for compile-time code rewriting, and because that was the primary driving goal from the start, they have amazing code rewriting capabilities that other languages cannot begin to dream of.

But the distinction I was trying to draw is that I'm talking about going further with that idea. Homoiconic syntax only allows for local code re-writing, but there are many other kinds of things we'd like to programmatically transform on a large codebase.

Half-baked examples: We'd like to replace all use of new/delete with smart pointers, but only in source files that are in one particular section of the codebase. Or we'd like to move all our unit tests from being in a separate package/dll, to being included in the same dll but surrounded by compiler pragmas.

(2) How the language would differ -- I don't yet know. But just as homoiconicity says "let's allow our end goal of automatic code rewriting to dictate our syntax", what I'm trying to say is, "let's allow our end goal of automatic refactorability / transformability to dictate all the programming language design decisions we make."

I think probably many of the things Raphael_Amiard mentions in this thread would be a part of it: strong inferred typing, perhaps, and a strong package system. Perhaps the ability to write some kind that was intentionally agnostic about how it would be packaged, and write some code elsewhere that says "take that code over there, and package it thusly." It would need to know if certain snippets were functionally pure (sort of how D does with "pure nothrow"), because then it would know it have more freedom with how it could transform that code.

Hopefully that clarifies the idea. It's hand wavy and high level, but it's a new (to me) idea that's only at the hand wavy stage so far.

If you are serious and passionate about this idea, I recommend digging in deeper and thinking of more specific examples of how you can improve on the status quo. You say that you want to design from the ground up to be more transformable than anything else, but without specifics this is just a goal, and it's not clear that a new language is necessary or would buy you anything.

For example, take your idea about wanting to be able to replace all instantiations with a function call instead. There's no need to invent a new language for this; the AST of a program in most any language would have enough information to write this refactoring capability into a tool. The hard part would be expressing the refactoring itself; how do you specify exactly the pattern you are trying to match, and exactly what you want to rewrite it to, in a way that is easy to use?

Maybe some other ideas you have would require a new language, but it's hard to say unless you are more specific about what you want. You might find that a lot of your ideas are implementable on top of existing systems. Designing "from the ground up" has a certain appeal to it, but to justify the extreme amount of effort a new language would take, I think you'd want to be more specific about what the clean break would buy you.

Thanks for the comment.

I agree that the AST of a program provides the raw material that could empower these kinds of transformations. But certain language features make it hard (like the C preprocessor, to cherry pick an example).

And I guess I'm exploring the question: What would a language look like whose goal was to make it easy, and to increase the set of possible automatic refactorings?

Maybe it's just a thought experiment that won't lead anywhere concrete. But to me, it's a very interesting thought experiment that I've never heard posed before.

You're going on and on about how huge and important and valuable this insight is, like a kid saying "look how smart I am"; I suspect people are downvoting because of the tone rather than the content. Hell, the GRATUITOUS CAPITAL LETTERS in your post justify a downvote by themselves.

I don't think it's a great idea. It's been tried in various ways (you assert that C# wasn't designed from the ground up to be automatically refactored, but actually easy manipulation of the code by IDEs was a key design goal - not the only goal to be sure, but a major one), and it doesn't seem to have worked. But don't let me stop you; try and build something useful out of this. I suspect you'll quickly see the problems with the idea, but if it actually works then we'll all be very grateful.

Ouch. Sorry you read it that way. I don't mean it as "look how smart I am." I mean it as "wow, this idea excites me."

I think C# has more refactorability than most languages I've been exposed to. But I'm intrigued by the question of how much further could we go if we made that a primary goal.

Granted, it's just the merest kernel of an idea, and it's an extremely ambitious one at that. Not something I'm likely to tackle on my own -- more like a thought experiment at this point. Even if it failed, the attempt would be illuminating perhaps.

Anyway, thanks for leaving a comment.

I didn't read it that way. Not sure why someone showing enthusiasm would get downvoted here, but HN has a strange crowd.

What OP talks about is only syntactic transformation which leaves semantics unchanged and probably "know" nothing about semantics of the code being translated, or might be being aware of a few standard ideoms, the same way macro transforms in Lisps do.

Transforming from one language to another is also old idea, some Scheme systems transforms (compiles) into C first.

In both cases the transformations themselves must be defined precisely by a programmer in advance. The idea that a program could transform semantics of another program is still a fantasy.) Even for Haskell.

I know, and I agree that having one program transform the semantics of another gets into the realm of fantasy.

I don't want to eliminate the role of human developers from the process. Instead, I want to provide tools that the human developer who understands these semantics can use to express the desired transformations, and tools that will help carry out those transformations across a huge codebase. Tools that "magnify" the efforts of the human who has the deep understanding of the before and after semantics.

That's why Steve Yegge said some article that compilers writers should have a configuration in their compiler to export the annotated AST. Then the tools would get better quickly.

Many languages can already be programmatically refactored. Pretty much any .NET language, for example, and Java, of course.

Yes, I agree. I've spent many hours working in both C# and Java and using JetBrains products to help me shape the code.

But C# and Java weren't designed from the ground up around the primary goal of being amenable to such programmatic refactoring. So there are limits on what you can programmatically refactor.

But our industry has reached a point where "rearranging" large bases of working code might be a bigger, harder, and more important job than originally developing that code. So perhaps the most important characteristic of a programming language is its "refactorability."

(Update - I think this sums it up: Would you consider working in a clunkier programming language if you knew that it offered 10X more "refactorability" than the other languages you were considering?)

Anyway, thanks for the comment.

> Would you consider working in a clunkier programming language if you knew that it offered 10X more "refactorability" than the other languages you were considering?

I'd have a hard time believing that a more clunky language would be a more refactorable language.

Ha ha, good point. Probably not. But take Lisp's homoiconicity as an example. Some people regard it as less desirable syntax, and yet even a subset of people who feel that way still find that the macro system makes it worth it.

So, would you accept certain trade-offs in your programming language if it greatly magnified the programmatic refactorability of your code?

Well, in my case my projects are never that large - I'm just a designer who does small high-fidelity prototypes and lets people who are supposed to be more capable do the large scale implementations. So "being easy to make quick-and-dirty hack from scratch with" is more important for me.

Also, have you tried Julia? It's homoiconic but looks more like Python. Haven't seen any refactoring tools for it though.

Calm down.

>(I am surely not the first person to have this insight, but it's WAY ahead of being mainstream).

Great. Maybe you can actually bring it into the mainstream. With code.

Imagination is fun, but you're not doing more than imagining.

Oh, the downvotes make me feel good. HN hates its cynic culture, and at the same time, this guy gets a stupid immediate benefit of the doubt for his feelings.

He asked for real criticism, guys. His post is really high on imagination and low on delivery, and if you don't see that you should probably think about how many people have created our programming languages and contemplate doing better.

This person isn't having conversations with them.

Am I rude for requiring him to step up to his talk?

Edit:

>>In other words, I think this might be the killer feature that allows one programming language to outcompete most others.

This sentence alone is flagrant ignorance.

You're right, it is high on imagination and low on delivery. It's a looooong way from being concrete and actionable.

I can't step it up just yet -- all I have right now is the question, which I pose as a thought experiment.

This conversation is more akin to some physicists speculating on the limits of efficiency of solar cells, than some engineers talking about a concrete plan of action.

>>In other words, I think this might be the killer feature that allows one programming language to outcompete most others.

> This sentence alone is flagrant ignorance.

Not really ... it's just awkward wording on my part. I don't really believe this will lead to the One True Language to Rule Them All. I was just trying to say that it might be well worth sacrificing on certain language features in order to maximize refactorability.

So ... I don't agree with you, but I do appreciate you leaving a comment.

Cheers.