Fetchez le Python

UPDATED see the end.

The Python standard library has a module called getpass you can use to get a password from the prompt:

>>> import getpass
>>> password = getpass.getpass()
Password:          <-- non-echoed typing here
>>> print password
worked

That’s nice, and Distutils uses it to ask for your password when you register or upload a release at PyPI, if it’s not found in your pypirc file. But this is annoying to type and type again your password, so you end up saving it in clear text in pypirc. Thats sucks. And the getpass module gets pretty useless if you want to store and retrieve passwords from other places than the user brain.

But wait… we have the Keyring project now.. what about making getpass use Keyring so you can safely read a password from your favorite keyring (Keychain, KWallet, etc..) ?

I’ve started to write a new getpass module that could do this. But instead of adding a keyring dependency in it and struggling for months (years) to get the addition of Keyring into the stdlib, I have made getpass pluggable.

In my improved version, you can define in a small configuration file (getpass.cfg) an arbritrary function that will be used by getpass for the getpass.getpass API. Here’s such a file:

  [getpass]
  getpass-backend = keyring:get_pass_get_password

Here I am configuring get pass to use the get_pass_get_password function from the keyring package. That’s a function that gets installed in your Python once Keyring is installed.

This function has the same interface than the default getpass.getpass API and calls keyring.

The modified getpass module is here: http://bitbucket.org/tarek/getpass/

And works against the current trunk of Keyring.

What I would like to do now is to propose the small changes I’ve made in Python’s getpass for inclusion in the stdlib. They are backward compatible changes and offers a simple, yet powerfull way to extend getpass without adding any other module in the stdlib. And maybe adding a setpass in there too would make sense.

Update from python-ideas

So I brought up the idea in the mailing lists and it turns out (thanks to the folks at Python-ideas) that the way I want to introduce this feature is not good for these reasons :

1. getpass is just a function that is used to get a password from the prompt. you can consider it as a potential, dummy backend for Keyring for example. Trying to make it extendable just denaturates its original purpose.
2. the only use case right now in the stdlib is for Distutils, so it doesn’t really make sense to have a keyring in there. People can just use the Keyring project directly.
3. Now if other parts of the stdlib have the same need, it will be time to think about how it could be included in the stdlib level rather than in Distutils.

So, I’ll work for its inclusion at Distutils level rather thah on getpass level.

Read more | Comment

I am pretty happy with the Keyring project. I use it now everywhere in my Mercurial-based projects, thanks to mercurial_keyring.

There’s one other place I’ve started to use it: I needed a simple command-line based tool to save passwords and read them. The tool I’ve used so far was KeePass, but I need to run it then click on its UI. This is time consuming when I simply want to push a password in the clipboard to use it to unlock something.

So I’ve wrote these two very simple scripts that use Keyring to store and retrieve passwords

$ clvault-set blog
Set your password:
Password set.

$ clvault-get blog
The password has been copied in your clipboard

The code that copy the passwords in the clipboard was tested under Mac OS with its Keychain, but should work under Windows and Linux as well.

I think these scripts can be useful for people like me who spend most of their time in a bash prompt when they are not in Vim or Emacs. So I created a project called CLVault.. You can grab it at the PyPI: CLVault PyPI page

or install it like this with Pip:

$ pip install clvault

Let me know if it’s useful to you !

Read more | Comment

Pycon is coming soon. Here’s a list of possible topics I would like to work on during the sprint:

1. adding the features in Distutils I’ve mentioned in my earlier post.
2. work on the standalone release of Distutils, and make sure it works with 2.4, 2.5, etc so it can be distributed at PyPI. There are already installable nightly builds by the way.
3. Finish the buildbot work so Distutils is tested with more projects from PyPI
4. Continue the work on Distribute, and specifically the work on 0.7:
   1. finish the develop command that would work with non-eggs formats
   2. finish the configure/build/install command where all options are computed and saved by the configure command
5. Fix plenty of issues in the tracker
6. work on a geolocalisation feature for Pip, so the nearest mirror can be picked
7. …

Anyone interested in packaging sprinting at Pycon ? Let me know !

Read more | Comment

Edit: the discussion is still going on, so I’ve probably blogged that a little bit early (I was excited about it ). Stay tuned for the final output.

Brad Allen launched a thread in Distutils-SIG about packaging terminology confusion. In particular the usage of the word “package” in our community. Part of the confusion is because of the meaning of this word in Python (that is, a directory containing one of several Python modules, with a special one named __init__.py) and in some systems like Debian (there, a package is a distribution file for a library or an application).

This confusion was present in PEP 345 (which was started years ago, so that explains it) – and is present in Distutils documentation and also in PyPI (That is: Python Package Index).

I really like Tres Seaver’s definitions, because they match prefectly the reality:

• package means a Python package, (directory intended to be on sys.path, with an __init__.py.  We *never* mean a distributable or installable archive, except when “impedance matching” with folks who think in terms of operating system distributions.
• distribution is such a distributable / installable archive: either in source form (an ’sdist’), or one of the binary forms (egg., etc.). Any distribution may contain multiple packages (or even no packages, in the case of standalone scripts).
• project is the process / community which produces releases of a given set of software, identified by a name unique within PyPI’s namespace.  PyPI manages metadata about projects (names, owners) and their releases.  Every real project has at least one release.
• release is a set of one or more distributions of a project, each sharing the same version.  Some PyPI metadata is specific to a release, rather than a project.  Every release has at least one distribution.

And I really like Martin’s proposal in the thread (in Catalog-SIG since it was cross-posted): “PyPI would then be the Python Project Index.”

I’ll fix Distutils documentation on my side accordingly, as well as the guide we are building. Let’s promote these definitions

Read more | Comment

While PEP 345 and PEP 386 are waiting for the final approval, I am back at work on Distutils code work, PEP 376, Distribute, and the HitchHicker guide to Packaging. The latter is growing faster than I have expected, thanks to the contributions of John Gabriele. It has quite some content already. I think the guide is an important task, and I’ll try to focus on it in this first trimester.

Distutils 2.7 new features

Python 2.7 first beta version is around the corner, and once it’s reached we can’t add new features. So, besides the code that will be changed if the PEPs we worked on at Distutils-SIG are accepted, here’s a list of small features I’d like to introduce in Distutils:

• a test command, that just uses the new unittest discovery script to run unittest-compatible tests.
• a new option for sdist called ‘extra_files’, that will allow to list extra files to be included in the distribution. These files will not be installed by ‘install’, just be part of the distribution. This will allow including files like CHANGELOG, etc.. without having to use a MANIFEST template.
• a very basic pre/post commit hook for the install command. These hooks will be deactivated when any bdist_* command runs install to create the binary tree. Now for bdist_rpm own hooks, I guess the best way would be to make install consumes the same two options than bdist_rpm (pre-install, post-install) so a project will be able to define a hook that is used by RPM and/or python setup.py install

If you think about something that should be added in 2.7, speak up !

Read more | Comment

Here’s a short, 5 questions, 2009 Python meme. Copy-paste the questions, and blog your answers !

1. What’s the coolest Python application, framework or library you have discovered in 2009 ?

Twisted. I have used Twisted in small occasions in the past, but since I have changed my job, a fair amount of my daily work is around the Twisted framework. I also program in plain Python for some needs, and I use Pylons to build our website, but Twisted is the biggest part of our current application.

2. What new programming technique did you learn in 2009 ?

Network programming. I was a web programmer before, and now I am learning how to write programs that need to scale, to work using several computers. It’s pretty exciting, and sometimes pretty exhaustive because when some bugs occur, they are often leading to snowball effects. The code needs to be more rigourous, and I need to know exactly what is going on when interacting with memory or I/O.

3. What’s the name of the open source project you contributed the most in 2009 ? What did you do ?

Python. I am a commiter since December 24th 2008 (my birthday ). And I have been working almost exclusively on Distutils. I did around 200 commits in the trunk (not counting the backports and forwardports on the branches), and I have been mainly correcting bugs and improving Distutils code coverage, which I raised from 18% to up to 75%.

I have also added some minor new features. The most important ones are or will be added in the What’s new file hopefully. I also did a lot of work in the PEPs to prepare the future of packaging in Python, and wait for that work to be finished, to start to change Distutils more. While I did more commits in Distribute, Distutils was the most time-consuming project.

4. What was the Python blog or website you read the most in 2009 ?

Python Reddit. And I think I am not alone in that case. 90% of my blog hits come from Reddit

5. What are the three top things you want to learn in 2010 ?

I’d like to learn how to program the C part of the CPython core, and eventually to try to contribute some patches for CPython. I would also like to write my own toy programming language, just for fun. Last, I would like to learn Japanese. And all this comes after Distutils, Distribute, etc, so I am pretty sure I won’t find the time…
Read other people meme now (If you did one, let me know, I’d like to link it here):

• Daniel Greenfeld
• Lennart Regebro
• Martijn Faassen
• Jorgen Modin
• Chris McDonough
• Arc Riley
• Brian Jones
• Corey Goldberg
• Alex Clark
• Thibaud “Teebes” Morel l’Horset
• Matthew Wilkes
• Rok Garbas
• Michael Foord
• …

Read more | Comment

DON’T PANIC !

Following-up various discussions we had about the need to provide a guide for people that want to understand our rather complex packaging eco-system, and how they can use it,

I’ve decided to start The Hitchhiker’s Guide to Packaging. The goals are:

• consolidate in one single Sphinx-based powered website, tutorials and documentation on how to use the various packaging tools and how to package, release and distribute your Python application.
• provide guidelines to the community that are up-to-date with the latest work done in packaging
• see if it can be added in the Python documentation itself at some point when its mature, as Georg Brandl suggested.

If  you want to join this documentation effort, by adding your existing documentation to this guide, by writing new one, or by helping on copy editing or reviewing, let me know (#distutils on Freenode, or by mail)

The project repository is hosted on bitbucket: http://bitbucket.org/tarek/hitchhiker-guide-packaging/

I’ve also created a google group: http://groups.google.com/group/packaging-guide

Read more | Comment

Someone told me on IRC that it’s currently hard to follow what’s going on in the packaging front. The truth is that it’s almost impossible if you don’t read all mails posted in Distutils-SIG.

So here’s a quick wrap-up that can save you some time if you are not reading Distutils-SIG.

PEP 345 – Metadata 1.2

We are almost done with the update of PEP 345. This PEP is describing the Metadata fields for a distribution, that get added in the file named PKG-INFO. This file is inserted in all your distribution and also published at PyPI. It’s the ID card of your project.

We are adding these fields in the metadata:

• Maintainer : the maintainer’s name
• Maintainer-email : the maintainer’s email
• Requires-Python : What Python versions are compatible with this distribution
• Requires-External : A list of external dependencies, like “libpng”, “libxslt”
• Requires-Dist : A list of Python dependencies, from the names founded at PyPI. like “zope.interface”
• Provides-Dist : A list of additional distribution names this distribution provides (as a complement to the one provided in “Name”)
• Obsoletes-Dist : A list of Python dependencies that are incompatible with the current distribution

Another important change is environment markers. An environment marker is a marker that can be added at the end of a field after a semi-colon (‘;’), to add a condition about the execution environment.

Examples:

Requires-Dist: pywin32, bar > 1.0; sys.platform == 'win32'
Obsoletes-Dist: pywin31; sys.platform == 'win32'
Requires-Dist: foo; os.machine == 'i386'
Requires-Dist: bar; python_version == '2.4' or python_version == '2.5'
Requires-External: libxslt; 'linux' in sys.platform

This will allow developers to define different conditions depending on the target platform. Moreover, this will allow tools like Pip to get a list of all dependencies for a given project and a given platform just by querying PyPI, and with no downloads or build required !

Last, for all the fields that manipulates versions, PEP 345 will use the version scheme described in PEP 386.

PEP 386 – Version scheme

We’ve designed in PEP 386 a version scheme that works with most Python software we know about. This version scheme comes with a new version comparison algorithm that will be provided by Distutils.

The scheme is in pseudo-regexpr (read the PEP for more details):

N.N[.N]+[abc]N[.N]+[.postN+][.devN+]

Don’t be afraid ! It looks complex but it’s not. The apparent complexity is due to the fact that we need to be able to work with development versions and post-release versions.

There are good chances that your project already works with this version scheme. If you want to give it a shot, there’s a prototype you can play with in an external repo here: http://bitbucket.org/tarek/distutilsversion/

PEP 376 – Installation standard

PEP 376 is quite completed now. We have our “standard” for site-packages, we know how to query installed projects, and how to remove them.

The discussions are now focusing on the “data” problem. Which is : how to describe in Distutils, in a more elegant way, the data files you are using, such as images, man files etc.

This is required to provide to developers more control on how their data files are installed on the target system, and to the packagers more tools to re-package a Python distribution.

Wolodja Wentland has been doing a lot of work in this area and leads this “data” effort. You can follow the discussion on this work in the Python wiki, starting at: http://wiki.python.org/moin/Distutils/DiscussionOverview.

PEP 382 -Namespaces packages

Distribute comes with a namespace package system, that allows you to have packages under the same namespace, spread into several distributions.

That’s what Plone and Zope use to be able to release all those plone.* and zope.* distributions.

Martin von Loewis proposed to implement it in Python, and this is described in PEP 382.

We are now waiting for Martin to implement it, and are ready to drop in Distribute 0.7.x the namespace feature in favor of supporting the PEP 382 one.

Distutils redesign discussions

One thing that makes Distutils a bit hard to work with, is how commands are designed. David Cournapeau (from the Numpy project) gave us an example of a use case that makes it hard. He basically needs to run the “build” command knowing the finalized options from the “install” command.

In other words, when you call something like :

$ python setup.py install --prefix=/some/place

The install command will use the prefix option to cook some other options. The build command that needs all the options needs in that case to look over the install command to get the values.

This is not optimal because it means that a build command depends on an install command to run. It also makes options redundants from one command to the other.

The solution we are going to try is to create a new command, called configure, that will be in charge of building a file with all the options that are required by the build command and the install command.

This is not new. It has been implemented years ago in 4suite, and it’s the philosophy behind tools like scons, etc: a configure/make/make install principle applied to a Python project.

This redesign is going to occur in Distribute 0.7. Once it’s ready, if the community has tried it and gave us positive feedback, I’ll push it in Distutils.

It might happen before Python 2.7 is out, it might not.

Other topics

There are many other topics, like PyPI mirroring (PEP 381) etc. I’ll write a blog entry later for these.

Read more | Comment

We are still actively working in fixing all the remaining bugs in Distribute (our Setuptools fork).

But we have reached an important milestone this week: both virtualenv and zc.buildout now comes with an option to switch to Distribute.

In virtualenv:

$ virtualenv --distribute ENV

In zc.buildout, using its bootstrap.py file:

$ python bootstrap.py --distribute

Enjoy !

For those who may wonder why they should switch to Distribute over Setuptools, it’s quite simple:

• Distribute 0.6.x is a drop-in replacement for Setuptools
• Distribute is actively maintained, and has over 10 commiters
• Distribute 0.6.x offers Python 3 support !

And if you still struggle with packaging issues, the place to hang around to get some help is the #distutils IRC channel on Freenode.

Read more | Comment

While we are working at fixing bugs in 0.6.x, we are organizing a first online coding sprint for the 0.7.x series, and for various community tasks.

What are we going to do ?

• We need to finalize the 0.7.x renaming and splitting work
• We also need to define a more detailed roadmap for each splitted package (new features, deprecations, etc.)
• We need to build a better test environment, and see if we can set up a buildbot for our work
• We need to write a tutorial explaining how Distribute can be used in a project. (Python 3 support, Moving from Setuptools to Distribute, etc)

Who can join ?

There are no particular level required to participate, but we won’t be able in this session to teach to participants how to work with Mercurial, Python, etc. But if you want to help in the Documentation part or testing, it’s as important and useful as the code and we are welcoming you.

Last, if you are just interested in evaluating Distribute for your project, you are welcome to join, we will help you.

Where ?

Online, on IRC – Freenode – #distutils channel

When, how long and how ?

Possible times (all CEST +0200 UTC) :

• Sunday October 18th – 6 pm
• Tuesday October 20th – 7 pm
• Wenesday October 21th – 7 pm
• Friday October 23th – 7 pm

The sprint will be held for a minimum of 3 hours, and people will be free to leave when they want.

The most important thing is to be present on time when the sprint starts, so we can make groups and kick it off. If you want to join, add your name here: http://doodle.com/bfgv3yi3pi48buuv

Read more | Comment