Skip to content
This repository was archived by the owner on Jul 16, 2020. It is now read-only.

pip over TUF

Jump to bottom
Trishank Karthik Kuppusamy edited this page Sep 21, 2013 · 49 revisions

NOTE: We are running a public TUF-secured PyPI mirror with our automation for creating, updating and destroying a TUF-secured PyPI mirror. If the demonstration does not work as expected, please wait until the mirror updates itself and try again later. Also, the PyPI mirror is not being kept continuously up-to-date (it is a static snapshot of PyPI on September 10 2013), but this will change soon. Thanks for your patience!

Set up virtualenv

GNU/Linux or Apple OS X

$ curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.10.1.tar.gz
$ tar xvfz virtualenv-1.10.1.tar.gz
$ python virtualenv-1.10.1/virtualenv.py --no-site-packages pipovertuf
$ source pipovertuf/bin/activate

Microsoft Windows

:: Download virtualenv-1.10.1.tar.gz from https://pypi.python.org/pypi/virtualenv
:: Extract virtualenv-1.10.1.tar.gz with http://www.haskell.org/haskellwiki/How_to_unpack_a_tar_file_in_windows
> python virtualenv-1.10.1\virtualenv.py --no-site-packages pipovertuf
> pipovertuf\Scripts\activate
:: On PowerShell, you should read: http://www.virtualenv.org/en/latest/#activate-script
:: On any shell, you should read: http://www.virtualenv.org/en/latest/#windows-notes

Install TUF and pip+TUF

$ sudo apt-get install python-dev libgmp-dev
$ pip install --upgrade https://github.com/theupdateframework/tuf/archive/v0.7.5.zip
$ pip install --upgrade https://github.com/theupdateframework/pip/archive/tuf-master.zip

(Note that here we replace the virtualenv copy of pip with our modified version of pip.)

Test the TUF-enabled pip client

$  pip install Django
[2013-09-21 19:37:48,707 UTC] [tuf.interposition] [INFO][info:[email protected]] Updater added for Configuration(netloc=pypi.python.org:80).
Downloading/unpacking Django
[2013-09-21 19:37:48,709 UTC] [tuf.interposition] [INFO][info:[email protected]] Found updater for hostname=pypi.python.org
[2013-09-21 19:37:48,709 UTC] [tuf.interposition] [INFO][info:[email protected]] Interposing for https://pypi.python.org/simple/Django/
[2013-09-21 19:37:48,711 UTC] [tuf.download] [INFO][_download_file:[email protected]] Downloading: http://mirror1.poly.edu/test-pypi/metadata/timestamp.txt
[2013-09-21 19:37:48,757 UTC] [tuf.download] [WARNING][_check_content_length:[email protected]] reported_length (1357) < required_length (2048)
[2013-09-21 19:37:48,758 UTC] [tuf.download] [WARNING][_check_downloaded_length:[email protected]] Downloaded 1357 bytes, but expected 2048 bytes. There is a difference of 691 bytes!
[2013-09-21 19:37:48,768 UTC] [tuf.download] [INFO][_download_file:[email protected]] Downloading: http://mirror1.poly.edu/test-pypi/metadata/release.txt.gz
[2013-09-21 19:37:48,857 UTC] [tuf.client.updater] [INFO][__check_hashes:[email protected]] The file's sha256 hash is correct: 8f930ddf9ce178f80bb89d8efed47345a767f48d0703600556b9a1d7b2f2fa19
[2013-09-21 19:37:48,933 UTC] [tuf.download] [INFO][_download_file:[email protected]] Downloading: http://mirror1.poly.edu/test-pypi/metadata/targets.txt
[2013-09-21 19:37:48,987 UTC] [tuf.client.updater] [INFO][__check_hashes:[email protected]] The file's sha256 hash is correct: 7794388bcdf7da7154a11cb0121f59b76a19ca33cb5d552a34460cdf4e380482
[2013-09-21 19:37:49,002 UTC] [tuf.download] [INFO][_download_file:[email protected]] Downloading: http://mirror1.poly.edu/test-pypi/metadata/targets/unclaimed.txt.gz
[2013-09-21 19:37:49,070 UTC] [tuf.client.updater] [INFO][__check_hashes:[email protected]] The file's sha256 hash is correct: db4338e8ae7f72d4d14f336ec08f103b4b442f2834cc6266c9347f71f081049f
[2013-09-21 19:37:49,333 UTC] [tuf.download] [INFO][_download_file:[email protected]] Downloading: http://mirror1.poly.edu/test-pypi/metadata/targets/unclaimed/bb0.txt
[2013-09-21 19:37:49,416 UTC] [tuf.client.updater] [INFO][__check_hashes:[email protected]] The file's sha256 hash is correct: 30fd99f71f57a80aa0a13e06e787612ea76075af7768371f6cff68f85f55d87f
[2013-09-21 19:37:49,455 UTC] [tuf.download] [INFO][_download_file:[email protected]] Downloading: http://mirror1.poly.edu/test-pypi/targets/simple/Django/index.html
[2013-09-21 19:37:49,501 UTC] [tuf.client.updater] [INFO][__check_hashes:[email protected]] The file's sha256 hash is correct: 5f3e004863d73c7375c7b8600dd705ca995b01bfc0c73668d3044bf56f005add
[2013-09-21 19:37:49,539 UTC] [tuf.interposition] [INFO][info:[email protected]] Found updater for hostname=pypi.python.org
[2013-09-21 19:37:49,539 UTC] [tuf.interposition] [INFO][info:[email protected]] Interposing for https://pypi.python.org/packages/source/D/Django/Django-1.5.2.tar.gz
[2013-09-21 19:37:49,539 UTC] [tuf.download] [INFO][_download_file:[email protected]] Downloading: http://mirror1.poly.edu/test-pypi/metadata/timestamp.txt
[2013-09-21 19:37:49,581 UTC] [tuf.download] [WARNING][_check_content_length:[email protected]] reported_length (1357) < required_length (2048)
[2013-09-21 19:37:49,582 UTC] [tuf.download] [WARNING][_check_downloaded_length:[email protected]] Downloaded 1357 bytes, but expected 2048 bytes. There is a difference of 691 bytes!
[2013-09-21 19:37:49,766 UTC] [tuf.download] [INFO][_download_file:[email protected]] Downloading: http://mirror1.poly.edu/test-pypi/metadata/targets/unclaimed/53c.txt
[2013-09-21 19:37:49,842 UTC] [tuf.client.updater] [INFO][__check_hashes:[email protected]] The file's sha256 hash is correct: e9982eff321ed9494a38a96b831009f317941239189ebe78d335ad4e81bee2da
[2013-09-21 19:37:49,880 UTC] [tuf.download] [INFO][_download_file:[email protected]] Downloading: http://mirror1.poly.edu/test-pypi/targets/packages/source/D/Django/Django-1.5.2.tar.gz
[2013-09-21 19:37:53,209 UTC] [tuf.client.updater] [INFO][__check_hashes:[email protected]] The file's sha256 hash is correct: 9a4b19adaaa096843425d426ffbeb928e85d861ff9c106527cb747dc67b434da
  Downloading Django-1.5.2.tar.gz (unknown size): 8.0MB downloaded
  Running setup.py egg_info for package Django
    
    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Installing collected packages: Django
  Running setup.py install for Django
    changing mode of build/scripts-2.7/django-admin.py from 644 to 755
    
    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    changing mode of /tmp/pipovertuf/bin/django-admin.py to 755
Successfully installed Django
[2013-09-21 19:37:59,578 UTC] [tuf.interposition] [INFO][info:[email protected]] Updater removed for Configuration(netloc=pypi.python.org:80).
Cleaning up...

TUF logging (which silently writes to a file by default) has been copied to the console so that you may see that TUF is actively working to protect your use of pip.

Stress testing pip without and with TUF

A simple Bash script to find out whether pip with TUF fails where pip without TUF does not.

How efficient is it?

Presently, downloading any package for the very first time will incur an overhead of about 111KB. This is computed as: 1.3KB (timestamp.txt) + 48.5KB bytes (release.txt.gz) + 3.5KB bytes (targets.txt) + 16KB bytes (targets/unclaimed.txt.gz) + 41.7KB (targets/unclaimed/BIN.txt) = 111KB.

Assuming no changes to target delegations, downloading any package from a new release will incur a smaller overhead of about 91KB. Computed as: 1.3KB (timestamp.txt) + 48.5KB bytes (release.txt.gz) + 41.7KB (targets/unclaimed/BIN.txt) = 91.5KB.

Assuming no changes to target delegations, downloading any package from an old release will incur a smaller overhead of about 43KB with 1.3KB (timestamp.txt) + 41.7KB (targets/unclaimed/BIN.txt) = 43KB. If the bin has already been downloaded, then the overhead will only be about 1.3KB (timestamp.txt).

The total amount of TUF metadata is 45MB (0.08%) for 54GB of PyPI data, and as you see from the calculations above, we only ever download a fraction of the TUF metadata to download any PyPI package.

How stable is it?

In a stress test of 1,972 (5.87%) randomly chosen packages from a total of 33,623 packages, we found that pip with TUF failed to install only 20 (1.01%) of those packages. We know the reasons for the failures which are documented in the next section. If you see a failure that is not explained by these known issues, please report it on our bug tracker.

Your stress tests results may vary because every test randomly shuffles the order of packages. Furthermore, we have not yet publicly shared our temporary workaround for an issue which affects the stress test results.

How does it work?

We refer to you our design document on securing PyPI and pip with TUF for more details.

Known issues

Clone this wiki locally