Hacker News Reader

Comment by toyg | Hacker News Reader

Is there anything about doing it the other way around, i.e. embed python code in Go...? I wish we could get the easy-deployment story of Go around the easy-development story of Python.

View on HN

This probably isn't what you're after, but interestingly there is a Google-developed Python dialect that can be embedded in a Go program, to facilitate runtime scriptability and programmable config files:

https://github.com/google/starlark-go

https://github.com/starlight-go/starlight

It seems to be used by the Delve debugger for example:

https://github.com/go-delve/delve/blob/master/Documentation/...

https://github.com/bazelbuild/starlark/blob/master/users.md

You might be interested in this: https://github.com/google/grumpy

(although it looks like it's not close to supporting the needed functionality to import a typical python library)

Grumpy was abruptly abandoned by Google, 3 years ago.

There is an active fork here: https://github.com/grumpyhome/grumpy

Python imports modules dynamically so even if we embedded python in Go, all the deployment woes of Python would still be around. Its kind of a fundamental problem with dynamically linked libraries and python being an interpreted language.

There isn't any fundamental problem, it's just about tooling. Interpreted languages typically do this by packing all the necessary modules into an executable binary together with an interpreter and preloading or faking import function to load them from memory. Dynamically linked libraries usually can be recompiled and relinked statically, sometimes you may need to fake dlopen() and add a namespace to functions as some people try to prevent static linking, and if recompiling is not a possibility, approaches like elf statifier [1] can be taken. It's too bad shipping a single self contained binary doesn't have a first class support in most languages as if deployment is somebody else's problem. But nothing actually prevents it.

[1] http://statifier.sourceforge.net/statifier/background.html

I'm also curious to know how people who use Python for data science and combine it with some other platform for web facing endpoints architect their apps in the cloud.

Python web server triggering lambda functions to process jobs

Where do you put the web server? What's the purpose of it?

Or you mean to use Python as the client facing API server? I meant how to architect if you use some other platform for the publicly exposed API. Would you build an internal Python API at the lambdas that run the actual data science computation, and invoke that from the client facing API/app?

I haven't done this specifically but it seems like a perfect use case for a smaller separate service. You'd probably submit jobs through an internal API (REST, gRPC, pick your poison) or through a job queue like Celery, if they're slower operations.

Yes that's what I thought too. Building a bunch of data analytics microservices that host a Python API and serve the results internally to the actual outwards facing API server.

The Go development story is already pretty easy. Easier than Python if you need concurrency or performance.

Reply on news.ycombinator.com