donaldstufft

There is a way to make a virtualenv relocatable but it has some issues where it doesn't always work very well. I'm not sure offhand what those issues are though.

You're also forgetting that Python itself is a core part of most modern Linux distributions. This means that even for a minimal install of an OS there are often times packages already inside your site-packages that your operating system depends on for a minimal set of functionality. Docker won't isolate you from these. It's a special kind of pain when you require one version of something, and your OS itself requires another version.

The recommended way to install pip has not been via easy_install for awhile. Generally we recommend using the ``get-pip.py`` script. Which with recent versions will also install setuptools for you.

<- PyPI Administrator.

I have plans for either turning PyPI into a build farm, or making a secondary service that acts as a build farm for PyPI. We've just been more focused on cleaning up other issues.

I have actually now asked Ernest, and that was basically the reason. Familiarity. He did most of the work and was more familiar so it enabled him to complete the work quicker. We still do use Chef on the python.org infra, just not for PyPI itself.

Importantly, every layer in the stack is responsible for it's own security. A consumer of this API should be making sure that it's optimally configured and configure it differently where it's not. Any project that isn't doing that should have security reports sent to it to tell it to do that and if they refuse they are guilty of the same sort of negligence as ruby core.

However the fact that other people should also be claiming responsibility for their own security does not absolve ruby of it's own responsibilities.

Realistically there is a value to porting to Python 3, in a year and some months Python2 will no longer be receiving security updates from Python Core. This will get taken care of by third parties for awhile but I fully suspect this support to be incomplete and eventually relegated only to RHEL.

I'm not sure I'd called M2Crypto a much better API. It more or less exposes the OpenSSL API which is also notoriously hard to get right. Further more it hasn't seen an update since 2011, the SSL certificate for the bug tracker expired in 2011, and it does not compile without patches against any modern OpenSSL (1.0+).

The state of crypto in Python is horrendous, hence the new library.

That's our goal at least. Personally I feel if you use our higher level API and get something wrong, then that's a bug in our code that allowed you to do that.

The lower level APIs we are exposing will hopefully be more easy to use as well, but they will not offer the same protections against mistakes the same as the higher level API would.

1) Not currently (and it's unlikely to be able to do so) there's an addon to add it which has been talked about getting it into core. I do believe it's documented to use HSTS and TLS if you want your site secured and recommends the ddon.

2) The above addon does have it. I have no idea what bitbucket does.

3) Not sure I understand what this means, you mean if there is a valid XSS on a subdomain?

Decoupling CSRF and Sessions has been a requirement for us for awhile now. We've spoken a little bit about optionally coupling them where if you have the session framework enabled it will couple them but if you don't it falls back on the current method.

I'm not sure I fully understand the proposed fix here, how does it differ from the application simply including random chunks of data inside the response?

This area of things isn't my strong suite, but assuming that this is analogous to just adding random data to the response, I believe that simply adding random data to the response can be worked around by doing more requests as using statistics to factor out the noise introduced.

If my understanding is wrong then excuse me :)

A browser will accept cookies from a HTTP response on a site that has HSTS set?

A CSRF request from a plaintext subdomain would not include a header and would fail Django's CSRF for lacking a referrer header (Strict referrer checking only available under HTTPS). Further more even a subdomain under TLS would fail to have the same origin.

A subdomain can set cookies this is true. This requires a XSS on the subdomain or allowing plaintext responses on subdomains. If you do not ensure both of these then besides being able to set (or in the case of XSS, read) the CSRF token you can also fixate the session, steal the session, preform a DoS using the size of the cookie, etc.

The solution is forced TLS with HSTS and includeSubdomains.

@homakov, so you can get a browser to trust a cookie from a HTTP response under a domain protected by HSTS?

I've just woken up and I haven't tested it, but off the cuff I believe HSTS will prevent the browser from trusting a plaintext HTTP response at all. So you cannot force a cookie if I understand the blog post correctly. You'd have to create a cookie inside of a verifiable HTTPS connection, which if you can do you've already executed a much worse attack.

An unrelated HTTP session cannot set a cookie for another domain (unless it's a subdomain in which you have the more serious issue of session theft or session fixation). The solution to both of these problems is HSTS with the includeSubDomains option.

Django's CSRF protection is perfectly fine other than issues with BREACH.

In any case that you can edit the CSRF token you already can execute a much stronger attack (MITM, XSS, etc). If you have a way to set your own arbitrary cookie that doesn't require a much work attack that already includes the ability to do arbitrary requests without them needing to be cross origin then I heartily suggest you report it.

The recently released pip 1.4 allows you to install from Wheels which does not use setup.py during install and executes no code from the package during install.

There's a difference between signing a tarball in it's entirety (in which case you'd get only one checksum) and checksumming every file inside the tarball and then signing that file.

> That's why you have to trust the author and NOT the source. The author is the first creator of the code and thus the person with the most ability to sign the code.

Source of Trust, not Source of The Thing You Downloaded. The author would still sign the package, it's just how do you get from where you're at to trusting that person. The way that browsers, most (All?) Linux distributions, Microsoft etc work is by hard baking a list of trust roots. This has the effect that we have in the modern CA system where because there's a hard baked system and the trust relationship is between the "author" and the source of trust that you can't reasonably not trust say Verisign or a significant portion of the internet breaks. It's about Trust Agility not about trusting the place you downloaded the package from. It's an idea similar to http://convergence.io/.

Interestingly enough I actually I have a rough sketch of an idea which will probably get laid out in a future post. I started to add it to this one but it felt disjointed and crammed to add it so I figured I'd wait till later :)

But you're right something somewhere has to give because nothing is 100%. I hoped to show people that things aren't 100% and just slapping some crypto in there isn't going to give the results they were hoping for.

There actually is a good bit of problem with this method even in the browser space.

For one, which key do you bundle? If it's a central root key see the part about Linux as that's essentially what they do. Also the part about not having the work force that the for profit CA vendors have is relevant here too.

1. Yes absolutely. I tried to not talk about specific signing implementations because this was mostly about that simply signing isn't enough, you need a trust model.

2. I'm actually a PyPI admin, a pip core developer, and one of the folks working in the python packaging space. This sort of heuristics is something I absolutely want to add. Right now I'm mostly focused on closing huge gaps in security wise (prior to 1.3.x pip downloaded everything over HTTP, PyPI did not have a trusted SSL cert, etc). But Yes! Making it easier for eyeballs is a good thing.

Hmm, maybe I wasn't clear. I actually meant It's unmanageable for the administrators/owners of PyPI to verify each author in the way that the owners of Debian/Red Hat verify their package maintainers. Basically that the repository can't act as a CA/Root of Trust because we simply don't have the man power.

You should be using setuptools now.

There's distutils which was the original one and is part of the standard library, setuptools which extends distutils and adds some very useful things like dependencies, then distribute split off from setuptools because of a number of reasons. Just recently distribute merged back into mainline setuptools and the distribute project is now done and it's maintainers have taken over setuptools.

In the middle there was also distutils2/packaging which were attempts at replacing the entire thing but have since been abandoned for a number of reasons such as trying to replace the entire toolchain at once.

Because it makes you depend on urls instead of abstract name/version requirements. easy_install does provide some tooling to get around it but in general it shouldn't be used.

I call them abstract for lack of a better word. Referenced by name?

It's not about providing versions of code that does or doesn't pass tests. It's about where that code is fetched from. If you reference the dependency by name then you can easily switch it out for something that looks like the code the author originally developed against, but actually has some changes (patches, bugfix, whatever).

Where to retrieve packages from is both a build and a deployment instruction. After all you need to fetch the built packages from soemwhere yes? In general "reference by name and version/abstract" allows you to easily swap in a built package for a source package as well. Otherwise you'd be stuck with whatever formats the author published because the dependencies would be pointing to a specific url (likely outside your control).

The very fact you can build a binary package and host it yourself works because of what I call abstract dependencies (for lack of a better name). If they wern't abstract then you'd have to fork the dependent package in order to point it at your built package/repository instead.

Sure that'd be good too :) This wasn't really aiming to be a comprehensive best practices guide to distributing things. Mostly trying to explain the differences between what a dependency in setup.py means vs a requirements.txt like file.

I have a hard time coming up with wording because sometimes applications are distributed as well and the "deployment" is just a config file + installing the dependencies. I hate using the term application because of that confusion but I haven't thought of a better way to word it yet :(

``setup.py#install_requires`` would be the gemspec file.

Ideally We'd have requirements.txt and requirements.txt.lock which would correspond to Gemfile and Gemfile.lock. However right now that doesn't really happen. So you're kind of stuck having requirements.txt play a double duty where for things where you need exact exact dependencies you'd check in a very exact requirements.txt and things where you have a supported set of deps (maybe more restricted ranges) and you expect people to Deploy by cloning/unzipping a tarball etc you'd use a less restrictive requirements.txt.

I would very much love to get a distinction between the two uses of requirements.txt so that we have a better correlation to Gemfile and Gemfile.lock

Actually you've misread it or I didn't explain it well enough :)

A private library/app can have abstract dependencies. As I said in the post it deals with the "deploy specific" side of a Python application. So each specific deployment would get it's own requirements.txt.

And yes requirements.txt is a pip only thing but the concept applies to the similar file in zc.buildout whose name escapes me at the moment and any other similar file.

Not yet anyways ;)