Specifying Your Project's Version

Setuptools can work well with most versioning schemes; there are, however, a few special things to watch out for, in order to ensure that setuptools and EasyInstall can always tell what version of your package is newer than another version. Knowing these things will also help you correctly specify what versions of other projects your project depends on.

A version consists of an alternating series of release numbers and pre-release or post-release tags. A release number is a series of digits punctuated by dots, such as 2.4 or 0.5. Each series of digits is treated numerically, so releases 2.1 and 2.1.0 are different ways to spell the same release number, denoting the first subrelease of release 2. But 2.10 is the tenth subrelease of release 2, and so is a different and newer release from 2.1 or 2.1.0. Leading zeros within a series of digits are also ignored, so 2.01 is the same as 2.1, and different from 2.0.1.

Following a release number, you can have either a pre-release or post-release tag. Pre-release tags make a version be considered older than the version they are appended to. So, revision 2.4 is newer than revision 2.4c1, which in turn is newer than 2.4b1 or 2.4a1. Postrelease tags make a version be considered newer than the version they are appended to. So, revisions like 2.4-1 and 2.4pl3 are newer than 2.4, but are older than 2.4.1 (which has a higher release number).

A pre-release tag is a series of letters that are alphabetically before "final". Some examples of prerelease tags would include alpha, beta, a, c, dev, and so on. You do not have to place a dot before the prerelease tag if it's immediately after a number, but it's okay to do so if you prefer. Thus, 2.4c1 and 2.4.c1 both represent release candidate 1 of version 2.4, and are treated as identical by setuptools.

In addition, there are three special prerelease tags that are treated as if they were the letter c: pre, preview, and rc. So, version 2.4rc1, 2.4pre1 and 2.4preview1 are all the exact same version as 2.4c1, and are treated as identical by setuptools.

A post-release tag is either a series of letters that are alphabetically greater than or equal to "final", or a dash (-). Post-release tags are generally used to separate patch numbers, port numbers, build numbers, revision numbers, or date stamps from the release number. For example, the version 2.4-r1263 might denote Subversion revision 1263 of a post-release patch of version 2.4. Or you might use 2.4-20051127 to denote a date-stamped post-release.

Notice that after each pre or post-release tag, you are free to place another release number, followed again by more pre- or post-release tags. For example, 0.6a9.dev-r41475 could denote Subversion revision 41475 of the in- development version of the ninth alpha of release 0.6. Notice that dev is a pre-release tag, so this version is a lower version number than 0.6a9, which would be the actual ninth alpha of release 0.6. But the -r41475 is a post-release tag, so this version is newer than 0.6a9.dev.

For the most part, setuptools' interpretation of version numbers is intuitive, but here are a few tips that will keep you out of trouble in the corner cases:

• Don't use - or any other character than . as a separator, unless you really want a post-release. Remember that 2.1-rc2 means you've already released 2.1, whereas 2.1rc2 and 2.1.c2 are candidates you're putting out before 2.1. If you accidentally distribute copies of a post-release that you meant to be a pre-release, the only safe fix is to bump your main release number (e.g. to 2.1.1) and re-release the project.

• Don't stick adjoining pre-release tags together without a dot or number between them. Version 1.9adev is the adev prerelease of 1.9, not a development pre-release of 1.9a. Use .dev instead, as in 1.9a.dev, or separate the prerelease tags with a number, as in 1.9a0dev. 1.9a.dev, 1.9a0dev, and even 1.9.a.dev are identical versions from setuptools' point of view, so you can use whatever scheme you prefer.

• If you want to be certain that your chosen numbering scheme works the way you think it will, you can use the pkg_resources.parse_version() function to compare different version numbers:

  >>> from pkg_resources import parse_version
  >>> parse_version('1.9.a.dev') == parse_version('1.9a0dev')
  True
  >>> parse_version('2.1-rc2') < parse_version('2.1')
  False
  >>> parse_version('0.6a9dev-r41475') < parse_version('0.6a9')
  True
  

Once you've decided on a version numbering scheme for your project, you can have setuptools automatically tag your in-development releases with various pre- or post-release tags. See the following sections for more details:

• Tagging and "Daily Build" or "Snapshot" Releases
• Managing "Continuous Releases" Using Subversion
• The egg_info command

Using find_packages()

For simple projects, it's usually easy enough to manually add packages to the packages argument of setup(). However, for very large projects (Twisted, PEAK, Zope, Chandler, etc.), it can be a big burden to keep the package list updated. That's what setuptools.find_packages() is for.

find_packages() takes a source directory, and a list of package names or patterns to exclude. If omitted, the source directory defaults to the same directory as the setup script. Some projects use a src or lib directory as the root of their source tree, and those projects would of course use "src" or "lib" as the first argument to find_packages(). (And such projects also need something like package_dir = {'':'src'} in their setup() arguments, but that's just a normal distutils thing.)

Anyway, find_packages() walks the target directory, and finds Python packages by looking for __init__.py files. It then filters the list of packages using the exclusion patterns.

Exclusion patterns are package names, optionally including wildcards. For example, find_packages(exclude=["*.tests"]) will exclude all packages whose last name part is tests. Or, find_packages(exclude=["*.tests", "*.tests.*"]) will also exclude any subpackages of packages named tests, but it still won't exclude a top-level tests package or the children thereof. In fact, if you really want no tests packages at all, you'll need something like this:

find_packages(exclude=["*.tests", "*.tests.*", "tests.*", "tests"])

in order to cover all the bases. Really, the exclusion patterns are intended to cover simpler use cases than this, like excluding a single, specified package and its subpackages.

Regardless of the target directory or exclusions, the find_packages() function returns a list of package names suitable for use as the packages argument to setup(), and so is usually the easiest way to set that argument in your setup script. Especially since it frees you from having to remember to modify your setup script whenever your project grows additional top-level packages or subpackages.

"Eggsecutable" Scripts

Occasionally, there are situations where it's desirable to make an .egg file directly executable. You can do this by including an entry point such as the following:

setup(
    # other arguments here...
    entry_points = {
        'setuptools.installation': [
            'eggsecutable = my_package.some_module:main_func',
        ]
    }
)

Any eggs built from the above setup script will include a short excecutable prelude that imports and calls main_func() from my_package.some_module. The prelude can be run on Unix-like platforms (including Mac and Linux) by invoking the egg with /bin/sh, or by enabling execute permissions on the .egg file. For the executable prelude to run, the appropriate version of Python must be available via the PATH environment variable, under its "long" name. That is, if the egg is built for Python 2.3, there must be a python2.3 executable present in a directory on PATH.

This feature is primarily intended to support bootstrapping the installation of setuptools itself on non-Windows platforms, but may also be useful for other projects as well.

IMPORTANT NOTE: Eggs with an "eggsecutable" header cannot be renamed, or invoked via symlinks. They must be invoked using their original filename, in order to ensure that, once running, pkg_resources will know what project and version is in use. The header script will check this and exit with an error if the .egg file has been renamed or is invoked via a symlink that changes its base name.

Dependencies that aren't in PyPI

If your project depends on packages that aren't registered in PyPI, you may still be able to depend on them, as long as they are available for download as an egg, in the standard distutils sdist format, or as a single .py file. You just need to add some URLs to the dependency_links argument to setup().

The URLs must be either:

1. direct download URLs, or
2. the URLs of web pages that contain direct download links

In general, it's better to link to web pages, because it is usually less complex to update a web page than to release a new version of your project. You can also use a SourceForge showfiles.php link in the case where a package you depend on is distributed via SourceForge.

If you depend on a package that's distributed as a single .py file, you must include an "#egg=project-version" suffix to the URL, to give a project name and version number. (Be sure to escape any dashes in the name or version by replacing them with underscores.) EasyInstall will recognize this suffix and automatically create a trivial setup.py to wrap the single .py file as an egg.

The dependency_links option takes the form of a list of URL strings. For example, the below will cause EasyInstall to search the specified page for eggs or source distributions, if the package's dependencies aren't already installed:

setup(
    ...
    dependency_links = [
        "http://peak.telecommunity.com/snapshots/"
    ],
)

Declaring "Extras" (optional features with their own dependencies)

Sometimes a project has "recommended" dependencies, that are not required for all uses of the project. For example, a project might offer optional PDF output if ReportLab is installed, and reStructuredText support if docutils is installed. These optional features are called "extras", and setuptools allows you to define their requirements as well. In this way, other projects that require these optional features can force the additional requirements to be installed, by naming the desired extras in their install_requires.

For example, let's say that Project A offers optional PDF and reST support:

setup(
    name="Project-A",
    ...
    extras_require = {
        'PDF':  ["ReportLab>=1.2", "RXP"],
        'reST': ["docutils>=0.3"],
    }
)

As you can see, the extras_require argument takes a dictionary mapping names of "extra" features, to strings or lists of strings describing those features' requirements. These requirements will not be automatically installed unless another package depends on them (directly or indirectly) by including the desired "extras" in square brackets after the associated project name. (Or if the extras were listed in a requirement spec on the EasyInstall command line.)

Extras can be used by a project's entry points to specify dynamic dependencies. For example, if Project A includes a "rst2pdf" script, it might declare it like this, so that the "PDF" requirements are only resolved if the "rst2pdf" script is run:

setup(
    name="Project-A",
    ...
    entry_points = {
        'console_scripts':
            ['rst2pdf = project_a.tools.pdfgen [PDF]'],
            ['rst2html = project_a.tools.htmlgen'],
            # more script entry points ...
    }
)

Projects can also use another project's extras when specifying dependencies. For example, if project B needs "project A" with PDF support installed, it might declare the dependency like this:

setup(
    name="Project-B",
    install_requires = ["Project-A[PDF]"],
    ...
)

This will cause ReportLab to be installed along with project A, if project B is installed -- even if project A was already installed. In this way, a project can encapsulate groups of optional "downstream dependencies" under a feature name, so that packages that depend on it don't have to know what the downstream dependencies are. If a later version of Project A builds in PDF support and no longer needs ReportLab, or if it ends up needing other dependencies besides ReportLab in order to provide PDF support, Project B's setup information does not need to change, but the right packages will still be installed if needed.

Note, by the way, that if a project ends up not needing any other packages to support a feature, it should keep an empty requirements list for that feature in its extras_require argument, so that packages depending on that feature don't break (due to an invalid feature name). For example, if Project A above builds in PDF support and no longer needs ReportLab, it could change its setup to this:

setup(
    name="Project-A",
    ...
    extras_require = {
        'PDF':  [],
        'reST': ["docutils>=0.3"],
    }
)

so that Package B doesn't have to remove the [PDF] from its requirement specifier.

Accessing Data Files at Runtime

Typically, existing programs manipulate a package's __file__ attribute in order to find the location of data files. However, this manipulation isn't compatible with PEP 302-based import hooks, including importing from zip files and Python Eggs. It is strongly recommended that, if you are using data files, you should use the Resource Management API of pkg_resources to access them. The pkg_resources module is distributed as part of setuptools, so if you're using setuptools to distribute your package, there is no reason not to use its resource management API. See also Accessing Package Resources for a quick example of converting code that uses __file__ to use pkg_resources instead.

Non-Package Data Files

The distutils normally install general "data files" to a platform-specific location (e.g. /usr/share). This feature intended to be used for things like documentation, example configuration files, and the like. setuptools does not install these data files in a separate location, however. They are bundled inside the egg file or directory, alongside the Python modules and packages. The data files can also be accessed using the Resource Management API, by specifying a Requirement instead of a package name:

from pkg_resources import Requirement, resource_filename
filename = resource_filename(Requirement.parse("MyProject"),"sample.conf")

The above code will obtain the filename of the "sample.conf" file in the data root of the "MyProject" distribution.

Note, by the way, that this encapsulation of data files means that you can't actually install data files to some arbitrary location on a user's machine; this is a feature, not a bug. You can always include a script in your distribution that extracts and copies your the documentation or data files to a user-specified location, at their discretion. If you put related data files in a single directory, you can use resource_filename() with the directory name to get a filesystem directory that then can be copied with the shutil module. (Even if your package is installed as a zipfile, calling resource_filename() on a directory will return an actual filesystem directory, whose contents will be that entire subtree of your distribution.)

(Of course, if you're writing a new package, you can just as easily place your data files or directories inside one of your packages, rather than using the distutils' approach. However, if you're updating an existing application, it may be simpler not to change the way it currently specifies these data files.)

Automatic Resource Extraction

If you are using tools that expect your resources to be "real" files, or your project includes non-extension native libraries or other files that your C extensions expect to be able to access, you may need to list those files in the eager_resources argument to setup(), so that the files will be extracted together, whenever a C extension in the project is imported.

This is especially important if your project includes shared libraries other than distutils-built C extensions, and those shared libraries use file extensions other than .dll, .so, or .dylib, which are the extensions that setuptools 0.6a8 and higher automatically detects as shared libraries and adds to the native_libs.txt file for you. Any shared libraries whose names do not end with one of those extensions should be listed as eager_resources, because they need to be present in the filesystem when he C extensions that link to them are used.

The pkg_resources runtime for compressed packages will automatically extract all C extensions and eager_resources at the same time, whenever any C extension or eager resource is requested via the resource_filename() API. (C extensions are imported using resource_filename() internally.) This ensures that C extensions will see all of the "real" files that they expect to see.

Note also that you can list directory resource names in eager_resources as well, in which case the directory's contents (including subdirectories) will be extracted whenever any C extension or eager resource is requested.

Please note that if you're not sure whether you need to use this argument, you don't! It's really intended to support projects with lots of non-Python dependencies and as a last resort for crufty projects that can't otherwise handle being compressed. If your package is pure Python, Python plus data files, or Python plus C, you really don't need this. You've got to be using either C or an external program that needs "real" files in your project before there's any possibility of eager_resources being relevant to your project.

Dynamic Discovery of Services and Plugins

setuptools supports creating libraries that "plug in" to extensible applications and frameworks, by letting you register "entry points" in your project that can be imported by the application or framework.

For example, suppose that a blogging tool wants to support plugins that provide translation for various file types to the blog's output format. The framework might define an "entry point group" called blogtool.parsers, and then allow plugins to register entry points for the file extensions they support.

This would allow people to create distributions that contain one or more parsers for different file types, and then the blogging tool would be able to find the parsers at runtime by looking up an entry point for the file extension (or mime type, or however it wants to).

Note that if the blogging tool includes parsers for certain file formats, it can register these as entry points in its own setup script, which means it doesn't have to special-case its built-in formats. They can just be treated the same as any other plugin's entry points would be.

If you're creating a project that plugs in to an existing application or framework, you'll need to know what entry points or entry point groups are defined by that application or framework. Then, you can register entry points in your setup script. Here are a few examples of ways you might register an .rst file parser entry point in the blogtool.parsers entry point group, for our hypothetical blogging tool:

setup(
    # ...
    entry_points = {'blogtool.parsers': '.rst = some_module:SomeClass'}
)

setup(
    # ...
    entry_points = {'blogtool.parsers': ['.rst = some_module:a_func']}
)

setup(
    # ...
    entry_points = """
        [blogtool.parsers]
        .rst = some.nested.module:SomeClass.some_classmethod [reST]
    """,
    extras_require = dict(reST = "Docutils>=0.3.5")
)

The entry_points argument to setup() accepts either a string with .ini-style sections, or a dictionary mapping entry point group names to either strings or lists of strings containing entry point specifiers. An entry point specifier consists of a name and value, separated by an = sign. The value consists of a dotted module name, optionally followed by a : and a dotted identifier naming an object within the module. It can also include a bracketed list of "extras" that are required for the entry point to be used. When the invoking application or framework requests loading of an entry point, any requirements implied by the associated extras will be passed to pkg_resources.require(), so that an appropriate error message can be displayed if the needed package(s) are missing. (Of course, the invoking app or framework can ignore such errors if it wants to make an entry point optional if a requirement isn't installed.)

Defining Additional Metadata

Some extensible applications and frameworks may need to define their own kinds of metadata to include in eggs, which they can then access using the pkg_resources metadata APIs. Ordinarily, this is done by having plugin developers include additional files in their ProjectName.egg-info directory. However, since it can be tedious to create such files by hand, you may want to create a distutils extension that will create the necessary files from arguments to setup(), in much the same way that setuptools does for many of the setup() arguments it adds. See the section below on Creating distutils Extensions for more details, especially the subsection on Adding new EGG-INFO Files.

Using setuptools... Without bundling it!

Your users might not have setuptools installed on their machines, or even if they do, it might not be the right version. Fixing this is easy; just download ez_setup.py, and put it in the same directory as your setup.py script. (Be sure to add it to your revision control system, too.) Then add these two lines to the very top of your setup script, before the script imports anything from setuptools:

import ez_setup
ez_setup.use_setuptools()

That's it. The ez_setup module will automatically download a matching version of setuptools from PyPI, if it isn't present on the target system. Whenever you install an updated version of setuptools, you should also update your projects' ez_setup.py files, so that a matching version gets installed on the target machine(s).

By the way, setuptools supports the new PyPI "upload" command, so you can use setup.py sdist upload or setup.py bdist_egg upload to upload your source or egg distributions respectively. Your project's current version must be registered with PyPI first, of course; you can use setup.py register to do that. Or you can do it all in one step, e.g. setup.py register sdist bdist_egg upload will register the package, build source and egg distributions, and then upload them both to PyPI, where they'll be easily found by other projects that depend on them.

(By the way, if you need to distribute a specific version of setuptools, you can specify the exact version and base download URL as parameters to the use_setuptools() function. See the function's docstring for details.)

What Your Users Should Know

In general, a setuptools-based project looks just like any distutils-based project -- as long as your users have an internet connection and are installing to site-packages, that is. But for some users, these conditions don't apply, and they may become frustrated if this is their first encounter with a setuptools-based project. To keep these users happy, you should review the following topics in your project's installation instructions, if they are relevant to your project and your target audience isn't already familiar with setuptools and easy_install.

Network Access

If your project is using ez_setup, you should inform users of the need to either have network access, or to preinstall the correct version of setuptools using the EasyInstall installation instructions. Those instructions also have tips for dealing with firewalls as well as how to manually download and install setuptools.

Custom Installation Locations

You should inform your users that if they are installing your project to somewhere other than the main site-packages directory, they should first install setuptools using the instructions for Custom Installation Locations, before installing your project.

Your Project's Dependencies

If your project depends on other projects that may need to be downloaded from PyPI or elsewhere, you should list them in your installation instructions, or tell users how to find out what they are. While most users will not need this information, any users who don't have unrestricted internet access may have to find, download, and install the other projects manually. (Note, however, that they must still install those projects using easy_install, or your project will not know they are installed, and your setup script will try to download them again.)

If you want to be especially friendly to users with limited network access, you may wish to build eggs for your project and its dependencies, making them all available for download from your site, or at least create a page with links to all of the needed eggs. In this way, users with limited network access can manually download all the eggs to a single directory, then use the -f option of easy_install to specify the directory to find eggs in. Users who have full network access can just use -f with the URL of your download page, and easy_install will find all the needed eggs using your links directly. This is also useful when your target audience isn't able to compile packages (e.g. most Windows users) and your package or some of its dependencies include C code.

Subversion or CVS Users and Co-Developers

Users and co-developers who are tracking your in-development code using CVS, Subversion, or some other revision control system should probably read this manual's sections regarding such development. Alternately, you may wish to create a quick-reference guide containing the tips from this manual that apply to your particular situation. For example, if you recommend that people use setup.py develop when tracking your in-development code, you should let them know that this needs to be run after every update or commit.

Similarly, if you remove modules or data files from your project, you should remind them to run setup.py clean --all and delete any obsolete .pyc or .pyo. (This tip applies to the distutils in general, not just setuptools, but not everybody knows about them; be kind to your users by spelling out your project's best practices rather than leaving them guessing.)

Creating System Packages

Some users want to manage all Python packages using a single package manager, and sometimes that package manager isn't easy_install! Setuptools currently supports bdist_rpm, bdist_wininst, and bdist_dumb formats for system packaging. If a user has a locally- installed "bdist" packaging tool that internally uses the distutils install command, it should be able to work with setuptools. Some examples of "bdist" formats that this should work with include the bdist_nsi and bdist_msi formats for Windows.

However, packaging tools that build binary distributions by running setup.py install on the command line or as a subprocess will require modification to work with setuptools. They should use the --single-version-externally-managed option to the install command, combined with the standard --root or --record options. See the install command documentation below for more details. The bdist_deb command is an example of a command that currently requires this kind of patching to work with setuptools.

If you or your users have a problem building a usable system package for your project, please report the problem via the `mailing list`_ so that either the "bdist" tool in question or setuptools can be modified to resolve the issue.

Managing Multiple Projects

If you're managing several projects that need to use ez_setup, and you are using Subversion as your revision control system, you can use the "svn:externals" property to share a single copy of ez_setup between projects, so that it will always be up-to-date whenever you check out or update an individual project, without having to manually update each project to use a new version.

However, because Subversion only supports using directories as externals, you have to turn ez_setup.py into ez_setup/__init__.py in order to do this, then create "externals" definitions that map the ez_setup directory into each project. Also, if any of your projects use find_packages() on their setup directory, you will need to exclude the resulting ez_setup package, to keep it from being included in your distributions, e.g.:

setup(
    ...
    packages = find_packages(exclude=['ez_setup']),
)

Of course, the ez_setup package will still be included in your packages' source distributions, as it needs to be.

For your convenience, you may use the following external definition, which will track the latest version of setuptools:

ez_setup svn://svn.eby-sarna.com/svnroot/ez_setup

You can set this by executing this command in your project directory:

svn propedit svn:externals .

And then adding the line shown above to the file that comes up for editing.

Setting the zip_safe flag

For maximum performance, Python packages are best installed as zip files. Not all packages, however, are capable of running in compressed form, because they may expect to be able to access either source code or data files as normal operating system files. So, setuptools can install your project as a zipfile or a directory, and its default choice is determined by the project's zip_safe flag.

You can pass a True or False value for the zip_safe argument to the setup() function, or you can omit it. If you omit it, the bdist_egg command will analyze your project's contents to see if it can detect any conditions that would prevent it from working in a zipfile. It will output notices to the console about any such conditions that it finds.

Currently, this analysis is extremely conservative: it will consider the project unsafe if it contains any C extensions or datafiles whatsoever. This does not mean that the project can't or won't work as a zipfile! It just means that the bdist_egg authors aren't yet comfortable asserting that the project will work. If the project contains no C or data files, and does no __file__ or __path__ introspection or source code manipulation, then there is an extremely solid chance the project will work when installed as a zipfile. (And if the project uses pkg_resources for all its data file access, then C extensions and other data files shouldn't be a problem at all. See the Accessing Data Files at Runtime section above for more information.)

However, if bdist_egg can't be sure that your package will work, but you've checked over all the warnings it issued, and you are either satisfied it will work (or if you want to try it for yourself), then you should set zip_safe to True in your setup() call. If it turns out that it doesn't work, you can always change it to False, which will force setuptools to install your project as a directory rather than as a zipfile.

Of course, the end-user can still override either decision, if they are using EasyInstall to install your package. And, if you want to override for testing purposes, you can just run setup.py easy_install --zip-ok . or setup.py easy_install --always-unzip . in your project directory. to install the package as a zipfile or directory, respectively.

In the future, as we gain more experience with different packages and become more satisfied with the robustness of the pkg_resources runtime, the "zip safety" analysis may become less conservative. However, we strongly recommend that you determine for yourself whether your project functions correctly when installed as a zipfile, correct any problems if you can, and then make an explicit declaration of True or False for the zip_safe flag, so that it will not be necessary for bdist_egg or EasyInstall to try to guess whether your project can work as a zipfile.

Namespace Packages

Sometimes, a large package is more useful if distributed as a collection of smaller eggs. However, Python does not normally allow the contents of a package to be retrieved from more than one location. "Namespace packages" are a solution for this problem. When you declare a package to be a namespace package, it means that the package has no meaningful contents in its __init__.py, and that it is merely a container for modules and subpackages.

The pkg_resources runtime will then automatically ensure that the contents of namespace packages that are spread over multiple eggs or directories are combined into a single "virtual" package.

The namespace_packages argument to setup() lets you declare your project's namespace packages, so that they will be included in your project's metadata. The argument should list the namespace packages that the egg participates in. For example, the ZopeInterface project might do this:

setup(
    # ...
    namespace_packages = ['zope']
)

because it contains a zope.interface package that lives in the zope namespace package. Similarly, a project for a standalone zope.publisher would also declare the zope namespace package. When these projects are installed and used, Python will see them both as part of a "virtual" zope package, even though they will be installed in different locations.

Namespace packages don't have to be top-level packages. For example, Zope 3's zope.app package is a namespace package, and in the future PEAK's peak.util package will be too.

Note, by the way, that your project's source tree must include the namespace packages' __init__.py files (and the __init__.py of any parent packages), in a normal Python package layout. These __init__.py files must contain the line:

__import__('pkg_resources').declare_namespace(__name__)

This code ensures that the namespace package machinery is operating and that the current package is registered as a namespace package.

You must NOT include any other code and data in a namespace package's __init__.py. Even though it may appear to work during development, or when projects are installed as .egg files, it will not work when the projects are installed using "system" packaging tools -- in such cases the __init__.py files will not be installed, let alone executed.

You must include the declare_namespace() line in the __init__.py of every project that has contents for the namespace package in question, in order to ensure that the namespace will be declared regardless of which project's copy of __init__.py is loaded first. If the first loaded __init__.py doesn't declare it, it will never be declared, because no other copies will ever be loaded!)

Tagging and "Daily Build" or "Snapshot" Releases

When a set of related projects are under development, it may be important to track finer-grained version increments than you would normally use for e.g. "stable" releases. While stable releases might be measured in dotted numbers with alpha/beta/etc. status codes, development versions of a project often need to be tracked by revision or build number or even build date. This is especially true when projects in development need to refer to one another, and therefore may literally need an up-to-the-minute version of something!

To support these scenarios, setuptools allows you to "tag" your source and egg distributions by adding one or more of the following to the project's "official" version identifier:

• A manually-specified pre-release tag, such as "build" or "dev", or a manually-specified post-release tag, such as a build or revision number (--tag-build=STRING, -bSTRING)
• A "last-modified revision number" string generated automatically from Subversion's metadata (assuming your project is being built from a Subversion "working copy") (--tag-svn-revision, -r)
• An 8-character representation of the build date (--tag-date, -d), as a postrelease tag

You can add these tags by adding egg_info and the desired options to the command line ahead of the sdist or bdist commands that you want to generate a daily build or snapshot for. See the section below on the egg_info command for more details.

(Also, before you release your project, be sure to see the section above on Specifying Your Project's Version for more information about how pre- and post-release tags affect how setuptools and EasyInstall interpret version numbers. This is important in order to make sure that dependency processing tools will know which versions of your project are newer than others.)

Finally, if you are creating builds frequently, and either building them in a downloadable location or are copying them to a distribution server, you should probably also check out the rotate command, which lets you automatically delete all but the N most-recently-modified distributions matching a glob pattern. So, you can use a command line like:

setup.py egg_info -rbDEV bdist_egg rotate -m.egg -k3

to build an egg whose version info includes 'DEV-rNNNN' (where NNNN is the most recent Subversion revision that affected the source tree), and then delete any egg files from the distribution directory except for the three that were built most recently.

If you have to manage automated builds for multiple packages, each with different tagging and rotation policies, you may also want to check out the alias command, which would let each package define an alias like daily that would perform the necessary tag, build, and rotate commands. Then, a simpler script or cron job could just run setup.py daily in each project directory. (And, you could also define sitewide or per-user default versions of the daily alias, so that projects that didn't define their own would use the appropriate defaults.)

Generating Source Distributions

setuptools enhances the distutils' default algorithm for source file selection, so that all files managed by CVS or Subversion in your project tree are included in any source distribution you build. This is a big improvement over having to manually write a MANIFEST.in file and try to keep it in sync with your project. So, if you are using CVS or Subversion, and your source distributions only need to include files that you're tracking in revision control, don't create a a MANIFEST.in file for your project. (And, if you already have one, you might consider deleting it the next time you would otherwise have to change it.)

(NOTE: other revision control systems besides CVS and Subversion can be supported using plugins; see the section below on Adding Support for Other Revision Control Systems for information on how to write such plugins.)

If you need to include automatically generated files, or files that are kept in an unsupported revision control system, you'll need to create a MANIFEST.in file to specify any files that the default file location algorithm doesn't catch. See the distutils documentation for more information on the format of the MANIFEST.in file.

But, be sure to ignore any part of the distutils documentation that deals with MANIFEST or how it's generated from MANIFEST.in; setuptools shields you from these issues and doesn't work the same way in any case. Unlike the distutils, setuptools regenerates the source distribution manifest file every time you build a source distribution, and it builds it inside the project's .egg-info directory, out of the way of your main project directory. You therefore need not worry about whether it is up-to-date or not.

Indeed, because setuptools' approach to determining the contents of a source distribution is so much simpler, its sdist command omits nearly all of the options that the distutils' more complex sdist process requires. For all practical purposes, you'll probably use only the --formats option, if you use any option at all.

(By the way, if you're using some other revision control system, you might consider creating and publishing a revision control plugin for setuptools.)

Making your package available for EasyInstall

If you use the register command (setup.py register) to register your package with PyPI, that's most of the battle right there. (See the docs for the register command for more details.)

If you also use the upload command to upload actual distributions of your package, that's even better, because EasyInstall will be able to find and download them directly from your project's PyPI page.

However, there may be reasons why you don't want to upload distributions to PyPI, and just want your existing distributions (or perhaps a Subversion checkout) to be used instead.

So here's what you need to do before running the register command. There are three setup() arguments that affect EasyInstall:

url and download_url

These become links on your project's PyPI page. EasyInstall will examine them to see if they link to a package ("primary links"), or whether they are HTML pages. If they're HTML pages, EasyInstall scans all HREF's on the page for primary links

long_description

EasyInstall will check any URLs contained in this argument to see if they are primary links.

A URL is considered a "primary link" if it is a link to a .tar.gz, .tgz, .zip, .egg, .egg.zip, .tar.bz2, or .exe file, or if it has an #egg=project or #egg=project-version fragment identifier attached to it. EasyInstall attempts to determine a project name and optional version number from the text of a primary link without downloading it. When it has found all the primary links, EasyInstall will select the best match based on requested version, platform compatibility, and other criteria.

So, if your url or download_url point either directly to a downloadable source distribution, or to HTML page(s) that have direct links to such, then EasyInstall will be able to locate downloads automatically. If you want to make Subversion checkouts available, then you should create links with either #egg=project or #egg=project-version added to the URL. You should replace project and version with the values they would have in an egg filename. (Be sure to actually generate an egg and then use the initial part of the filename, rather than trying to guess what the escaped form of the project name and version number will be.)

Note that Subversion checkout links are of lower precedence than other kinds of distributions, so EasyInstall will not select a Subversion checkout for downloading unless it has a version included in the #egg= suffix, and it's a higher version than EasyInstall has seen in any other links for your project.

As a result, it's a common practice to use mark checkout URLs with a version of "dev" (i.e., #egg=projectname-dev), so that users can do something like this:

easy_install --editable projectname==dev

in order to check out the in-development version of projectname.

Managing "Continuous Releases" Using Subversion

If you expect your users to track in-development versions of your project via Subversion, there are a few additional steps you should take to ensure that things work smoothly with EasyInstall. First, you should add the following to your project's setup.cfg file:

[egg_info]
tag_build = .dev
tag_svn_revision = 1

This will tell setuptools to generate package version numbers like 1.0a1.dev-r1263, which will be considered to be an older release than 1.0a1. Thus, when you actually release 1.0a1, the entire egg infrastructure (including setuptools, pkg_resources and EasyInstall) will know that 1.0a1 supersedes any interim snapshots from Subversion, and handle upgrades accordingly.

(Note: the project version number you specify in setup.py should always be the next version of your software, not the last released version. Alternately, you can leave out the tag_build=.dev, and always use the last release as a version number, so that your post-1.0 builds are labelled 1.0-r1263, indicating a post-1.0 patchlevel. Most projects so far, however, seem to prefer to think of their project as being a future version still under development, rather than a past version being patched. It is of course possible for a single project to have both situations, using post-release numbering on release branches, and pre-release numbering on the trunk. But you don't have to make things this complex if you don't want to.)

Commonly, projects releasing code from Subversion will include a PyPI link to their checkout URL (as described in the previous section) with an #egg=projectname-dev suffix. This allows users to request EasyInstall to download projectname==dev in order to get the latest in-development code. Note that if your project depends on such in-progress code, you may wish to specify your install_requires (or other requirements) to include ==dev, e.g.:

install_requires = ["OtherProject>=0.2a1.dev-r143,==dev"]

The above example says, "I really want at least this particular development revision number, but feel free to follow and use an #egg=OtherProject-dev link if you find one". This avoids the need to have actual source or binary distribution snapshots of in-development code available, just to be able to depend on the latest and greatest a project has to offer.

A final note for Subversion development: if you are using SVN revision tags as described in this section, it's a good idea to run setup.py develop after each Subversion checkin or update, because your project's version number will be changing, and your script wrappers need to be updated accordingly.

Also, if the project's requirements have changed, the develop command will take care of fetching the updated dependencies, building changed extensions, etc. Be sure to also remind any of your users who check out your project from Subversion that they need to run setup.py develop after every update in order to keep their checkout completely in sync.

python setup.py egg_info -RDb "" sdist bdist_egg register upload

python setup.py alias -u release egg_info -RDb ""

python setup.py release sdist bdist_egg register upload

Distributing Extensions compiled with Pyrex

setuptools includes transparent support for building Pyrex extensions, as long as you define your extensions using setuptools.Extension, not distutils.Extension. You must also not import anything from Pyrex in your setup script.

If you follow these rules, you can safely list .pyx files as the source of your Extension objects in the setup script. setuptools will detect at build time whether Pyrex is installed or not. If it is, then setuptools will use it. If not, then setuptools will silently change the Extension objects to refer to the .c counterparts of the .pyx files, so that the normal distutils C compilation process will occur.

Of course, for this to work, your source distributions must include the C code generated by Pyrex, as well as your original .pyx files. This means that you will probably want to include current .c files in your revision control system, rebuilding them whenever you check changes in for the .pyx source files. This will ensure that people tracking your project in CVS or Subversion will be able to build it even if they don't have Pyrex installed, and that your source releases will be similarly usable with or without Pyrex.

Release Tagging Options

The following options can be used to modify the project's version string for all remaining commands on the setup command line. The options are processed in the order shown, so if you use more than one, the requested tags will be added in the following order:

--tag-build=NAME, -b NAME

Append NAME to the project's version string. Due to the way setuptools processes "pre-release" version suffixes beginning with the letters "a" through "e" (like "alpha", "beta", and "candidate"), you will usually want to use a tag like ".build" or ".dev", as this will cause the version number to be considered lower than the project's default version. (If you want to make the version number higher than the default version, you can always leave off --tag-build and then use one or both of the following options.)

If you have a default build tag set in your setup.cfg, you can suppress it on the command line using -b "" or --tag-build="" as an argument to the egg_info command.

--tag-svn-revision, -r

If the current directory is a Subversion checkout (i.e. has a .svn subdirectory, this appends a string of the form "-rNNNN" to the project's version string, where NNNN is the revision number of the most recent modification to the current directory, as obtained from the svn info command.

If the current directory is not a Subversion checkout, the command will look for a PKG-INFO file instead, and try to find the revision number from that, by looking for a "-rNNNN" string at the end of the version number. (This is so that building a package from a source distribution of a Subversion snapshot will produce a binary with the correct version number.)

If there is no PKG-INFO file, or the version number contained therein does not end with -r and a number, then -r0 is used.

--no-svn-revision, -R

Don't include the Subversion revision in the version number. This option is included so you can override a default setting put in setup.cfg.

--tag-date, -d

Add a date stamp of the form "-YYYYMMDD" (e.g. "-20050528") to the project's version number.

--no-date, -D

Don't include a date stamp in the version number. This option is included so you can override a default setting in setup.cfg.

(Note: Because these options modify the version number used for source and binary distributions of your project, you should first make sure that you know how the resulting version numbers will be interpreted by automated tools like EasyInstall. See the section above on Specifying Your Project's Version for an explanation of pre- and post-release tags, as well as tips on how to choose and verify a versioning scheme for your your project.)

For advanced uses, there is one other option that can be set, to change the location of the project's .egg-info directory. Commands that need to find the project's source directory or metadata should get it from this setting:

Other egg_info Options

--egg-base=SOURCEDIR, -e SOURCEDIR

Specify the directory that should contain the .egg-info directory. This should normally be the root of your project's source tree (which is not necessarily the same as your project directory; some projects use a src or lib subdirectory as the source root). You should not normally need to specify this directory, as it is normally determined from the package_dir argument to the setup() function, if any. If there is no package_dir set, this option defaults to the current directory.

egg_info Examples

Creating a dated "nightly build" snapshot egg:

python setup.py egg_info --tag-date --tag-build=DEV bdist_egg

Creating and uploading a release with no version tags, even if some default tags are specified in setup.cfg:

python setup.py egg_info -RDb "" sdist bdist_egg register upload

(Notice that egg_info must always appear on the command line before any commands that you want the version changes to apply to.)

Configuration File Options

Normally, settings such as options and aliases are saved to the project's local setup.cfg file. But you can override this and save them to the global or per-user configuration files, or to a manually-specified filename.

--global-config, -g

Save settings to the global distutils.cfg file inside the distutils package directory. You must have write access to that directory to use this option. You also can't combine this option with -u or -f.

--user-config, -u

Save settings to the current user's ~/.pydistutils.cfg (POSIX) or $HOME/pydistutils.cfg (Windows) file. You can't combine this option with -g or -f.

--filename=FILENAME, -f FILENAME

Save settings to the specified configuration file to use. You can't combine this option with -g or -u. Note that if you specify a non-standard filename, the distutils and setuptools will not use the file's contents. This option is mainly included for use in testing.

These options are used by other setuptools commands that modify configuration files, such as the alias and setopt commands.

Adding Commands

You can add new setup commands by defining entry points in the distutils.commands group. For example, if you wanted to add a foo command, you might add something like this to your distutils extension project's setup script:

setup(
    # ...
    entry_points = {
        "distutils.commands": [
            "foo = mypackage.some_module:foo",
        ],
    },
)

(Assuming, of course, that the foo class in mypackage.some_module is a setuptools.Command subclass.)

Once a project containing such entry points has been activated on sys.path, (e.g. by running "install" or "develop" with a site-packages installation directory) the command(s) will be available to any setuptools-based setup scripts. It is not necessary to use the --command-packages option or to monkeypatch the distutils.command package to install your commands; setuptools automatically adds a wrapper to the distutils to search for entry points in the active distributions on sys.path. In fact, this is how setuptools' own commands are installed: the setuptools project's setup script defines entry points for them!

Adding setup() Arguments

Sometimes, your commands may need additional arguments to the setup() call. You can enable this by defining entry points in the distutils.setup_keywords group. For example, if you wanted a setup() argument called bar_baz, you might add something like this to your distutils extension project's setup script:

setup(
    # ...
    entry_points = {
        "distutils.commands": [
            "foo = mypackage.some_module:foo",
        ],
        "distutils.setup_keywords": [
            "bar_baz = mypackage.some_module:validate_bar_baz",
        ],
    },
)

The idea here is that the entry point defines a function that will be called to validate the setup() argument, if it's supplied. The Distribution object will have the initial value of the attribute set to None, and the validation function will only be called if the setup() call sets it to a non-None value. Here's an example validation function:

def assert_bool(dist, attr, value):
    """Verify that value is True, False, 0, or 1"""
    if bool(value) != value:
        raise DistutilsSetupError(
            "%r must be a boolean value (got %r)" % (attr,value)
        )

Your function should accept three arguments: the Distribution object, the attribute name, and the attribute value. It should raise a DistutilsSetupError (from the distutils.error module) if the argument is invalid. Remember, your function will only be called with non-None values, and the default value of arguments defined this way is always None. So, your commands should always be prepared for the possibility that the attribute will be None when they access it later.

If more than one active distribution defines an entry point for the same setup() argument, all of them will be called. This allows multiple distutils extensions to define a common argument, as long as they agree on what values of that argument are valid.

Also note that as with commands, it is not necessary to subclass or monkeypatch the distutils Distribution class in order to add your arguments; it is sufficient to define the entry points in your extension, as long as any setup script using your extension lists your project in its setup_requires argument.

Adding new EGG-INFO Files

Some extensible applications or frameworks may want to allow third parties to develop plugins with application or framework-specific metadata included in the plugins' EGG-INFO directory, for easy access via the pkg_resources metadata API. The easiest way to allow this is to create a distutils extension to be used from the plugin projects' setup scripts (via setup_requires) that defines a new setup keyword, and then uses that data to write an EGG-INFO file when the egg_info command is run.

The egg_info command looks for extension points in an egg_info.writers group, and calls them to write the files. Here's a simple example of a distutils extension defining a setup argument foo_bar, which is a list of lines that will be written to foo_bar.txt in the EGG-INFO directory of any project that uses the argument:

setup(
    # ...
    entry_points = {
        "distutils.setup_keywords": [
            "foo_bar = setuptools.dist:assert_string_list",
        ],
        "egg_info.writers": [
            "foo_bar.txt = setuptools.command.egg_info:write_arg",
        ],
    },
)

This simple example makes use of two utility functions defined by setuptools for its own use: a routine to validate that a setup keyword is a sequence of strings, and another one that looks up a setup argument and writes it to a file. Here's what the writer utility looks like:

def write_arg(cmd, basename, filename):
    argname = os.path.splitext(basename)[0]
    value = getattr(cmd.distribution, argname, None)
    if value is not None:
        value = '\n'.join(value)+'\n'
    cmd.write_or_delete_file(argname, filename, value)

As you can see, egg_info.writers entry points must be a function taking three arguments: a egg_info command instance, the basename of the file to write (e.g. foo_bar.txt), and the actual full filename that should be written to.

In general, writer functions should honor the command object's dry_run setting when writing files, and use the distutils.log object to do any console output. The easiest way to conform to this requirement is to use the cmd object's write_file(), delete_file(), and write_or_delete_file() methods exclusively for your file operations. See those methods' docstrings for more details.

Adding Support for Other Revision Control Systems

If you would like to create a plugin for setuptools to find files in other source control systems besides CVS and Subversion, you can do so by adding an entry point to the setuptools.file_finders group. The entry point should be a function accepting a single directory name, and should yield all the filenames within that directory (and any subdirectories thereof) that are under revision control.

For example, if you were going to create a plugin for a revision control system called "foobar", you would write a function something like this:

def find_files_for_foobar(dirname):
    # loop to yield paths that start with `dirname`

And you would register it in a setup script using something like this:

entry_points = {
    "setuptools.file_finders": [
        "foobar = my_foobar_module:find_files_for_foobar"
    ]
}

Then, anyone who wants to use your plugin can simply install it, and their local setuptools installation will be able to find the necessary files.

It is not necessary to distribute source control plugins with projects that simply use the other source control system, or to specify the plugins in setup_requires. When you create a source distribution with the sdist command, setuptools automatically records what files were found in the SOURCES.txt file. That way, recipients of source distributions don't need to have revision control at all. However, if someone is working on a package by checking out with that system, they will need the same plugin(s) that the original author is using.

A few important points for writing revision control file finders:

• Your finder function MUST return relative paths, created by appending to the passed-in directory name. Absolute paths are NOT allowed, nor are relative paths that reference a parent directory of the passed-in directory.
• Your finder function MUST accept an empty string as the directory name, meaning the current directory. You MUST NOT convert this to a dot; just yield relative paths. So, yielding a subdirectory named some/dir under the current directory should NOT be rendered as ./some/dir or /somewhere/some/dir, but always as simply some/dir
• Your finder function SHOULD NOT raise any errors, and SHOULD deal gracefully with the absence of needed programs (i.e., ones belonging to the revision control system itself. It may, however, use distutils.log.warn() to inform the user of the missing program(s).

A Note Regarding Dependencies

If the project containing your distutils/setuptools extension(s) depends on any projects other than setuptools, you must also declare those dependencies as part of your project's setup_requires keyword, so that they will already be built (and at least temprorarily installed) before your extension project is built.

So, if for example you create a project Foo that includes a new file finder plugin, and Foo depends on Bar, then you must list Bar in both the install_requires and setup_requires arguments to setup().

If you don't do this, then in certain edge cases you may cause setuptools to try to go into infinite recursion, trying to build your dependencies to resolve your dependencies, while still building your dependencies. (It probably won't happen on your development machine, but it will happen in a full build pulling everything from revision control on a clean machine, and then you or your users will be scratching their heads trying to figure it out!)

Subclassing Command

Sorry, this section isn't written yet, and neither is a lot of what's below this point, except for the change log. You might want to subscribe to changes in this page to see when new documentation is added or updated.

XXX

ez_setup

XXX

setuptools.archive_util

XXX

setuptools.sandbox

XXX

setuptools.package_index

XXX