Show HN: FunctorFlow – An attempt to re-imagine Python libraries | Hacker News Reader

Show HN: FunctorFlow – An attempt to re-imagine Python libraries

functorflow.org

40 pointssamravshanov48 comments

The source for the primary package is not published, afaik. The primary entrypoint function is implemented in cython, resisting casual analysis. The stated purpose is to pull (unversioned!) code from the internet and run it, but no mechanism to trust or verify that code is even mentioned.

The advisability of the idea aside, the presentation here feels exceedingly and unnecessarily risky

Friend, (1) there is a versioning mechanism already in place (2) Cython is used to optimize the speed (3) FunctorFlow only works on Repl.it. We are in alpha. It does not work on your machine. Just go and install it. See if it works. It doesn't!

Lemme repeat once again:

FunctorFlow only works on Repl.it, not on your machine!

Currently making tutorials and assignments for Repl.it community

Ah - you might want to clarify that point then, the marketing copy on your site implies that is (will be) usable in any python environment.

Yes friend, it will be. Now it is not (while we are in alpha stage).

Hard no. My visceral reactions to this:

- Strings are not names. Names help tooling, strings make tooling harder.

- A function which takes a string to switch on where behavior radically differs between cases is an anti-pattern.

- Imports should be explicit and well organized, not magic and scattered throughout the code.

I’m all for improving Python, but as feedback on the minimal website, the impression this gives is you’re working on the npmification of Python libraries. (I don’t mean that as a good thing.)

> from functorflow import f, ff

Please don't give these functions simple names like "f". I often use these names as generic function names locally. I know I can use "import ... as" but doing so shouldn't be necessary and will result in everybody using different names, which will hurt consistency, readability and searchability of code.

I was hoping this would combine the best features of existing package managers, but instead this seems to have been made in a vacuum.

While making a good package manager/community around it is hard, there's plenty of hard earned knowledge that's been completely ignored here.

This seems to make dependency management harder (imports spilled all over the code base, versioning seems non-existent, documentation for this/plugins? Twitter is not documentation).

How does plugin authoring work? How does distribution and trust work?

What problem is this supposed to solve that isn't addressed by other package managers (or what does this do better than said package managers)

Hi friend, This is a humble beginning. Now docs are in YAML-like format and soon they will be published in a readable and interactive format on our new website. For now, I just wanted to share it earlier and to involve other developers in the process.

Thanks for understanding. ;)

Where are the docs? The “docs” link just forwards to a Twitter account.

How would one use this with existing Python packages? Or is this a parallel universe to PIP where libraries need to be written explicitly with FF?

The website doesn’t mention how FF packages are versioned or how version restrictions between dependencies are resolved.

I see this as quickly becoming full of “left-pad” sized packages, can you analyze a full project to fetch everything ahead of time?

Hi friend, This is a humble beginning. Now docs are in YAML-like format and soon they will be published in readable/interactive format on our new website. For now, I just wanted to share it earlier and to involve other developers in the process.

Thanks for understanding. ;)

This is just an awful idea all the way through. I'm baffled by how much effort had been put into this. It's almost as if the author went off grid and made this without ever asking anyone if it was a good idea without a second thought for whether it was needed at all. It's also obvious to anyone that you cannot just magically make version numbers disappear and expect everything to go well.

Friend, there is a versioning mechanism. I'll share details in our Discord group. Please, subscribe to our email list to participate in our alpha.

What's the target audience?

Looks convenient for notebook/repl style use (assuming the the editing environment still supports autocomplete/correct and other niceties). I might just be stodgy and change averse, but I'd be a bit less willing to try things like this in production.

Landing page or FAQ could possibly benefit from an explanation along these lines.

Hi friend,

Yes, do not try this in production yet.

Currently, this library works only on Repl.it for the exact reason.

This is why you see the headline with "an attempt". :)

I get that. I mean even the idea. Say in 5 years this is far more mature and stable, what use cases do you see it filling? Would they include production applications?

Yeah, this is pretty awful. I don't see myself ever using this.

1. What's the difference between this and just packaging a function as a pip package? The only one I see is just shaving off a few seconds and one less command.

2. Where is the code running from for each plugin? Eval'ing random code just seems like a bad idea. You can argue that it's the same as blindly installing a pip package, but this apparently happens each time on runtime and I see no option to check the code beforehand.

3. f and ff are pretty bad function names. I see what you're doing, but why?

4. Dep management issues aren't as common as you think they are.

5. Lots of these plugins are already in the standard library. XML to JSON is already possible quite trivially with a pip package. Not only that, but this seems like a less flexible version of that - and more obscure. (how to map xml to json?)

I'm giving this a hard pass for me, but looking forward to seeing my doubts cleared. Cheers.

I find how offended some of the people commenting are really compelling. What is the rationale behind this? Like why do it, then why again?

I'm still open to the possibility this could be brilliant.

Indeed. Thank you, friend!

I'll keep you updated.

How about in the thread? It would be really helpful.

Yes sure, in the thread!

I find the naming of this library confusing and almost clickbait. I don't see any kind of Functor (object?) in the API, furthermore what the heck is "flowing"? Seems like a copycat to "TensorFlow".

A better suited name which encodes the core feature, "installationless helper routines", could really help this library to become actually recognized for what it can provide.

Here's another comment about the API: I would prefer a syntax like

   f.name_of_function(args)

instead of

   f("name-of-function", args)

The syntactic sugar variant is easy to obtain with Python and has the advantage of much better readability.

For those developers who wish to earn an extra income in their free time while helping MOOC students complete their MOOC courses consider registering as an expert tutor here: https://moocteas-stage-web.azurewebsites.net/

Helping others learn while rewarded for doing so.

Lack of versioning support makes this absolutely unusable for anything but sandboxing. And since the imports are scattered all over the code, it's hard for security scanning tools to check for vulnerabilities and to update libraries regularly. Pip isn't perfect but at least piprot helps me keep libs up to date.

This tool was clearly designed in a vacuum.

Friend, there is a versioning mechanism. I'll share details in our Discord group. Please, subscribe to our email list to participate in our alpha.

And how do you expect people to learn that when your website doesn't mention it and your "docs" link is a Twitter account with XKCD comics?

This tool is clearly not ready for any use outside of a sandbox or toy.

Hi HN! Meet FunctorFlow ^_^

Think of it like lego blocks for your next tech-toys and hopefully (in the future) for your next big-thing ^_^.

If you love tinkering with new toys then give it a try with Repl.it: https://dev.to/t/functorflow.

If you want to meet cool kids like you then join our club at Discord by subscribing to our email-list.

...and follow updates at https://twitter.com/functorflow

Together, we will re-imagine and re-invent how Python apps should be developed.

Lemme know what you think!

Have fun!

- Sam

"Lego blocks" "tech toys" "tinkering" "cool kids" "next big-thing" -- perhaps they are your hopes for this project, but these are all vague, filler words that don't tell me what this thing actually does. If I take a look at it what do I see?

It's not until I get to the twitter account that I see that it looks like you're trying to create not just a different python package manager, but a new ecosystem that's not backwards compatible. I don't think this is a good idea. The Python package ecosystem is gigantic and one of the largest, most mature language ecosystems out there. If you're trying to make the way Python apps are developed better, try taking a look at alternate Python package managers like `pipenv`, `poetry` and `anaconda` and figure out what you like and dislike about them. Try making changes to them before you reinvent the wheel.

As it stands right now, this isn't even a wheel. It's an uninspired marketing page on twitter that is not doing a great job of convincing anyone it's better than the alternatives. In fact, it's not even clear that you're even aware of the alternatives.

https://twitter.com/functorflow/status/1157396077043167232

That diagrams is 90% about different versions of PYTHON and not python modules. So I have no idea what it has to do with the discussion at hand.

Even if it was about modules, if anything FunctorFlow makes it worse because you now have X+1 things creating modules (FunctorFlow has to live somewhere after all so it doesn't remove the previous system).

https://nixos.org/nixpkgs/manual/#python

Not an issue I've dealt with during the last 10 years in production. On OSX, homebrew python + standard virtual environments + pip mitigates that completely. On the Ubuntu side, system python + standard virtual environments + pip work fine as well. Anything else?

Well, this has all the signs of well-meaning but inexperienced developers seeing a market opportunity without appreciating the many difficult problems that will have to be solved to produce something that is secure, reliable and interoperable with current development ecosystems.

Rather than abuse them, we might perhaps use them as an example of the Dunning-Kruger effect.

So, aside from other comments, doesn't this break the already mediocre tab completion side of python? Scrolling through API docs to figure out what exactly that method or argument was called gets really tedious after the 5000th time.

Why?

Absolutely terrible. I can see why data science types who use / produce a lot of (typically throwaway) Python code might be enamored with this, but as far as the Python language goes, functorflow will be another nail in its coffin if it ever gains mass-acceptance. It will turn the already messy and inconsistent ecosystem into shambles that no sane person will want to build anything robust on top of.

With the mass migration of Python developers into Go, this is the last thing that Python needs.

Brother, those "data science types" are also people. Be respectful of other people.

Their comment seemed respectful to me. It was pointing out that some users of the platform have priorities other than maintainability and reproducibility.

I mean, data science is really the thing that people should be using Python for.

What? Python is arguably the go-to language in data (and other) science. With a massive ecosystem of mature, powerful libraries like numpy and scipy, you immediately and freely have access to a range of features, managed almost trivially with pip/virtualenv/conda. I don't think any other language has a larger number of performant ML libraries like tensorflow and pytorch, and more importantly, if you want source from the latest ML papers on arxiv, it will almost always be implemented in python.

Then the language itself offers trivial iteration, possibly in real time with tools like jupyter, and requires substantially less bootstrapping knowledge than something like C/C++, which creates a low barrier of entry for non-programmers, who are less likely to shoot themselves in their feet with memory management and such. And if you're concerned about performance, most of the standard data science packages are just wrapped C/C++ anyway.

Sure, it's not perfect, but it's practically the lingua franca of data science right now, and it fits the role quite well.

"I mean, data science is really the thing that people should be using Python for."

Did you read my comment?

I think your comment is coming over sarcastic.

> I can see why data science types who use / produce a lot of (typically throwaway) Python code might be enamored with this

I think that's honestly its killer app. I don't do data science, but I often use jupyter to write short prototypes, and make plots and such.

I've been looking for something that lets me:

1. Set up a jupyter notebook.

2. Run some tests or experiments, pulling in libraries as needed.

3. Ignore it for a few years.

4. Come back and run the same notebook without everything breaking.

If this could patch into Python's native import mechanism and let me specify the repo version as todays date, it'd be great.

    import functorflow
    functorflow.repo("2019-08-04")

    from ff.package_name.whatever import foo

If that gets you a replicable set of versions, it'd be pretty great.

Hmmm, that feels more like a jupyter extension than a standalone module. The extension could intercept imports automatically without the need for a prefix. Might require a custom kernel that does some magical per notebook lib mapping. Maybe it installs libs in version specific central folders and then does some symlinks to build a notebook specific lib folder based on the date specified. Feels more seamless but also more complicated.

I'd actually implement it as a transparent venv and use pinned versions in a requirements.txt – with hashes, of course.

That'd make sense. You'd need to pin all transitive dependencies as well and cache the venv. Not sure if you'd need to find older versions of new direct dependencies to avoid conflicts. For example, you run the sheet in 2015 and then again in 2019 but with a new import. That new import's latest version has a transitive dependency that you already pinned but has different version requirements than what you pinned. I wonder if you can embed the requirements.txt inside the sheet itself.

Only issue is that it'd be really space inefficient, my DS venvs clock in at 400+mb each so having one per sheet will probably quickly become unusable. Which is why I thought of some sort of smart system wide caching akin to maven/ivy. But I'd forgotten how complicated python dependencies (binaries, c code, etc.) were and how little api support pip had.

> You'd need to pin all transitive dependencies as well and cache the venv.

I think to do this stuff, you need hooks in Jupyter to setup and teardown the venv before it runs the kernel. (And generally Jupyter would want to clean up unused venvs to mitigate the teardown hook not firing.)

> Only issue is that it'd be really space inefficient

A venv is overkill. You can just run the kernel in a regular directory, and `pip install --target kernel_dir foo-bar-pkg` to put packages directly in it. As long as the linker sees it, third-party libraries will work. This technique is used in the serverless project[1] to bundle dependencies for use on AWS Lambda.

> Not sure if you'd need to find older versions of new direct dependencies to avoid conflicts.

Curation is a solution to this. Stackage[1] is popular in the Haskell community; they build a consistent version set of everything every night and curate stable releases periodically.

With curation, a date and a set of top-level packages is enough to pin your dependencies.

[1]: https://github.com/commercialhaskell/stackage#frequently-ask... [2]: https://github.com/UnitedIncome/serverless-python-requiremen...

Reply on news.ycombinator.com