Hacker News Reader

Comment by maxeonyx | Hacker News Reader

This would be solved if python used an (OS-specific) cache directory for its .pyc files. I have always disliked .pyc files... here's a concrete reason!

Question: what does python do if it doesn't have write permission in the current working directory? Not write the cache?

View on HN

It already (since 3.2, as the article points out) uses a cache directory. Why would an OS-specific one help?

You can also set PYTHONPYCACHEPREFIX if you want to use 'a mirror directory tree at this path, instead of in __pycache__ directories within the source tree' - https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPY...

The PEP mentions your case at https://www.python.org/dev/peps/pep-3147/#case-5-read-only-f... . But honestly, I don't really follow the answer. I think it means "ignore creation and write failures."

> Why would an OS-specific one help?

E.g., if you want to back up your home dir, and omit caches, since they can be regenerated. It's a lot easier if programs write their cache data to ~/.cache / $XDG_CACHE_HOME than if they intermix it / scatter it about.

But how does it help to have one directory for all Linux-based OSes (or one directory for RHEL, one for Ubuntu, one for Debian, etc.), and one for FreeBSD and one for OpenIndiana?

At least, that's what I interpret "OS-specific" to mean.

More along the line of the original comment you're replying to, if the cache was in, say, ~/.cache, then it won't get swept up in the repository's commits, since the cache data is no longer inside the repository's working directory. Then, it never gets uploaded to GitHub, and this security issue never happens.

I have seen a surprising number of people — some who are engineers by profession too, and ought to know better — just git add everything, and then commit it all without looking. One should review the diff one has staged to see if it is correct, but alas…

That's possible with 3.8's PYTHONPYCACHEPREFIX, yes?

Perhaps it's worthwhile for someone to blog about this more/promote this as a best practice? Though what's missing is the hook to connected it as appropriate for the given platform.

I see now that "OS-specific" was meant to be interpreted as "the OS-defined mechanism to find a cache directory", not "a cache directory which differs for each operating system".

I would not have been confused by the term "platform dependent", which is what Python's tmpdir documentation uses, as in: "The default directory is chosen from a platform-dependent list" at https://docs.python.org/3/library/tempfile.html?highlight=tm... .

Windows exists, too. Probably would be in %APPDATA% or %TMP%

I suspect his thought was to use something like "%UserProfile%/pycache/" on windows and "$HOME/.pycache/" on Linux.

That env var is new in 3.8. I've been looking forward to it to stop docker writing root owned cache files back to a bind mounted file system. I might set it to $HOME/.cache/python globally as well.

> what does python do if it doesn't have write permission in the current working directory? Not write the cache?

In the case where the interpreter can't find any place it has write access to to write the cache, it will not write it, yes. That means it will have to re-parse the source file into bytecode (and fail to write the bytecode to cache) every time it is loaded.

It makes more sense to use a second stream or named stream.

That sounds like a recipe for even more people accidentally storing and distributing that bytecode without wanting to do so, because it won't be immediately visible that it's there, many tools don't show or highlight the existence of those streams.

I'd argue that it's more appropriate to follow the 'explicit is better than implicit' from the 'guidelines' of https://www.python.org/dev/peps/pep-0020/ .

Reply on news.ycombinator.com