trotterdylan

Hey HN, I’m one of the creators. We’re always dealing with each WYSIWYG’s quirks, strengths, and licensing. It’s tedious. Being able to preview and compare them in one place really speeds up picking the right editor for each project. Hope it helps. Let us know if we’re missing something!

We hope to implement optimizations like you're describing eventually. But the core functionality needs a lot of work before we start down that path.

Correct, Grumpy cannot yet run Grumpy :)

Ugh, sorry about that. There's a couple issues here:

1. Lambda tuple args are not yet supported -- I actually didn't know that was a thing :\ -- https://github.com/google/grumpy/issues/17

2. Even if that worked properly, sorted() is not yet implemented: https://github.com/google/grumpy/issues/16

It's all good. Thanks for filing the issue. I'll get that fixed.

Whoops, just noticed you (or somebody) had already filed it: https://github.com/google/grumpy/issues/12

Unfortunately the native interface is still pretty immature so I can't guarantee things will work as they should. In this case, I think the problem is that you have a *Response object which is not itself a struct. I've rewritten this part of the code a couple times and I think this functionality was lost. I've filed: https://github.com/google/grumpy/issues/13

Yes, leveraging type hints for optimization purposes is a long term goal. Thanks for pointing me to that issue, I'll keep an eye on it.

One of the goals of open sourcing was to get feedback and work with outside folks so I'm definitely open to collaboration!

The idea is to continue to write code in Python. The transpiled code is not suitable for working with directly. That said, there is the possibility of rewriting bits and pieces in Go (e.g. performance critical stuff) and then call into it from Python. Sort of a hybrid approach.

> I would love to know how big the codebase is.

Sorry, can't be very specific, but rewriting all the frontend code would take a lot more effort than writing a new Python runtime :)

> It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces.

We do iteratively rewrite components as well. We are pursuing multiple strategies.

> Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)?

Most async operations are performed out-of-process by other servers.

> Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)?

I'd love to support async and await at some point.

> Of course being biased towards the JVM (since I know it so well) they could get really fast concurrency if they want with Jython today. Most of the Python tools already work with Jython (assuming 2.7).

We did also do an evaluation of Jython but there were a number of technical issues that made it unsuitable for our codebase and workload. One such example is this longstanding issue: http://bugs.jython.org/issue527524. I just noticed the very recent update on that thread that implemented the workaround outlined in 2010 by Jim Baker. We tried that workaround and found we got a huge performance hit on affected code. There were a few other general performance problems as well but I can't recall all the details.

Please note I'm not at all bashing Jython, I think it's a great project with a sound design, it just wasn't right for us.

> With Jython you could always drop down into Java (or any other JVM lang) if you need more speed as well C for cpython (or even C from Java). It is unclear what you do with Grumpy with performance critical code. Can you interface with Go code or is the plan C?

You can interface with Go code directly, e.g. from the blog post:

  from __go__.net.http import ListenAndServe, RedirectHandler
  handler = RedirectHandler('http://github.com/google/grumpy', 303)
  ListenAndServe('127.0.0.1:8080', handler)

Heh, I came across the namedtuple exec thing the other day when I was trying to get the collections module working :\

namedtuple will have to be implemented differently. I think it can be accomplished by defining the class with type()? Maybe with a metaclass...

I'd like to support 3.x at some point. See https://github.com/google/grumpy/issues/1

Reflection in Go is used minimally in Grumpy because it is slow. Currently importing Go packages into Python code is accomplished via the reflect module, but I think this will have to change to make such integration useful.

As has been said by others, Grumpy is not used in production at Google currently. There's still a lot of work to do -- especially on the standard library -- to support large real world codebases.

I don't know. I hope so. I think this is an area where Go can succeed and integration with existing Python libraries could be useful.

Although Grumpy is compiled, it is just as dynamic as Python, in that method dispatch involves dictionary lookups, etc.

The main reason why Grumpy's slower for most single threaded benchmarks is that most Python workloads involve creating and freeing a bunch of small Python objects. In Go, these objects are garbage that need to be GC'd in a very general way. In CPython, there are free lists, arenas and other optimizations for allocating small (especially immutable) objects. And cleaning up garbage in CPython involves pushing unreferenced objects back onto the free lists for later reuse.

exec/eval do not work, but all of the other dispatching and magic methods are supposed to work exactly like CPython (and if they don't, it's a bug/not yet implemented).

> > Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable. > What about stuff like literal_eval? Or even just monkeypatching with name.__dict__[param] = value ?

literal_eval could in principle be supported I think. name.__dict__[param] = value works as you'd expect:

  $ make run
  class A(object):
    pass
  a = A()
  a.__dict__['foo'] = 'bar'
  print a.foo
  bar

EDIT: fixed formatting

YouTube does not run on Grumpy. There is a lot of work left to do before Grumpy can run a large existing codebase.

It's been said a couple times in this thread already but basically, the YouTube Python codebase is very large and the cost of a rewrite is prohibitive.

Thanks for trying it out!

Yeah, Grumpy does not currently support old-style classes. Since all of our code internally requires new-style classes, this was not a high priority feature. It is something that we'll get to.

To be clear, Grumpy cannot yet run YouTube's Python codebase. There's still a lot of work to do on the standard library.

There are a handful of C extensions for JSON, protobufs, etc that YouTube uses, but mostly they're small utility functions written by us to optimize particularly hot code paths.

Interestingly, achieving performance parity with CPython is one of the biggest challenges of this project. There are certain things CPython does very fast like allocating and freeing many small objects.

I think the CPython AST module is written as a C extension module so currently it's a no-go. I don't think there's a fundamental reason Grumpy couldn't run a pure Python AST module, though.

Thanks for trying it out! And sorry about the lacking documentation. I'll be fleshing it out over the next little while.

Your assessment is right: the grumpc compiler takes a single Python file and spits out a Go package. Incidentally, this means you can import a Python module into Go code pretty easily.

I don't have a ready solution for building a large existing project but I'll write up a quick doc to outline the process. The trickiest bit is that the Python statement "import foo.bar" translates to a Go import: import "prefix/foo/bar". Currently prefix always points at the grumpc/lib directory so that's one way to integrate your code, but I need to make it more configurable.

It does not, yet. The standard library is very incomplete at this point.

> Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out.

Basically, we needed to support a large existing Python 2.7 codebase. See discussion here: https://github.com/google/grumpy/issues/1

> It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it.

There are restrictions. I'll update the README to make note of them. Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable.

> If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what?

It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated.

I'm not philosophically opposed to supporting C extensions. The additional complexity just was not deemed to be warranted since the YouTube frontend doesn't use a lot of C extensions.

In principle it's possible to implement something like JyNI (http://jyni.org/) or CPyExt (https://morepypy.blogspot.de/2010/04/using-cpython-extension...) to bridge the CPython and Grumpy APIs. In practice, marshalling data across the interface can be very expensive.