Documentation for the pip accelerator API¶
On this page you can find the complete API documentation of pip-accel 0.43.
A note about backwards compatibility¶
Please note that pip-accel has not yet reached a 1.0 version and until that time arbitrary changes to the API can be made. To clarify that statement:
- On the one hand I value API stability and I’ve built a dozen tools on top of pip-accel myself so I don’t think too lightly about breaking backwards compatibility :-)
- On the other hand if I see opportunities to simplify the code base or make things more robust I will go ahead and do it. Furthermore the implementation of pip-accel is dictated (to a certain extent) by pip and this certainly influences the API. For example API changes may be necessary to facilitate the upgrade to pip 1.5.x (the current version of pip-accel is based on pip 1.4.x).
In pip-accel 0.16 a completely new API was introduced and support for the old “API” was dropped. The goal of the new API is to last for quite a while but of course only time will tell if that plan is going to work out :-)
The Python API of pip-accel¶
Here are the relevant Python modules that make up pip-accel:
pip_accel
¶
Top level functionality of pip-accel.
The Python module pip_accel
defines the classes that implement the
top level functionality of the pip accelerator. Instead of using the
pip-accel
command you can also use the pip accelerator as a Python module,
in this case you’ll probably want to start by taking a look at
the PipAccelerator
class.
Wheel support¶
During the upgrade to pip 6 support for installation of wheels was added to
pip-accel. The pip-accel
command line program now downloads and installs
wheels when available for a given requirement, but part of pip-accel’s Python
API defaults to the more conservative choice of allowing callers to opt-in to
wheel support.
This is because previous versions of pip-accel would only download source distributions and pip-accel provides the functionality to convert those source distributions to “dumb binary distributions”. This functionality is exposed to callers who may depend on this mode of operation. So for now users of the Python API get to decide whether they’re interested in wheels or not.
Setuptools upgrade¶
If the requirement set includes wheels and setuptools >= 0.8
is not yet
installed, it will be added to the requirement set and installed together with
the other requirement(s) in order to enable the usage of distributions
installed from wheels (their metadata is different).
-
class
pip_accel.
PipAccelerator
(config, validate=True)¶ Accelerator for pip, the Python package manager.
The
PipAccelerator
class brings together the top level logic of pip-accel. This top level logic was previously just a collection of functions but that became more unwieldy as the amount of internal state increased. ThePipAccelerator
class is intended to make it (relatively) easy to build something on top of pip and pip-accel.-
__init__
(config, validate=True)¶ Initialize the pip accelerator.
Parameters: - config – The pip-accel configuration (a
Config
object). - validate –
True
to runvalidate_environment()
,False
otherwise.
- config – The pip-accel configuration (a
-
validate_environment
()¶ Make sure
sys.prefix
matches$VIRTUAL_ENV
(if defined).This may seem like a strange requirement to dictate but it avoids hairy issues like documented here.
The most sneaky thing is that
pip
doesn’t have this problem (de-facto) becausevirtualenv
copiespip
wherever it goes... (pip-accel
on the other hand has to be installed by the user).
-
initialize_directories
()¶ Automatically create local directories required by pip-accel.
-
clean_source_index
()¶ Cleanup broken symbolic links in the local source distribution index.
The purpose of this method requires some context to understand. Let me preface this by stating that I realize I’m probably overcomplicating things, but I like to preserve forward / backward compatibility when possible and I don’t feel like dropping everyone’s locally cached source distribution archives without a good reason to do so. With that out of the way:
- Versions of pip-accel based on pip 1.4.x maintained a local source
distribution index based on a directory containing symbolic links
pointing directly into pip’s download cache. When files were removed
from pip’s download cache, broken symbolic links remained in
pip-accel’s local source distribution index directory. This resulted
in very confusing error messages. To avoid this
clean_source_index()
cleaned up broken symbolic links whenever pip-accel was about to invoke pip. - More recent versions of pip (6.x) no longer support the same style of download cache that contains source distribution archives that can be re-used directly by pip-accel. To cope with the changes in pip 6.x new versions of pip-accel tell pip to download source distribution archives directly into the local source distribution index directory maintained by pip-accel.
- It is very reasonable for users of pip-accel to have multiple versions of pip-accel installed on their system (imagine a dozen Python virtual environments that won’t all be updated at the same time; this is the situation I always find myself in :-). These versions of pip-accel will be sharing the same local source distribution index directory.
- All of this leads up to the local source distribution index directory containing a mixture of symbolic links and regular files with no obvious way to atomically and gracefully upgrade the local source distribution index directory while avoiding fights between old and new versions of pip-accel :-).
- I could of course switch to storing the new local source distribution index in a differently named directory (avoiding potential conflicts between multiple versions of pip-accel) but then I would have to introduce a new configuration option, otherwise everyone who has configured pip-accel to store its source index in a non-default location could still be bitten by compatibility issues.
For now I’ve decided to keep using the same directory for the local source distribution index and to keep cleaning up broken symbolic links. This enables cooperating between old and new versions of pip-accel and avoids trashing user’s local source distribution indexes. The main disadvantage is that pip-accel is still required to clean up broken symbolic links...
- Versions of pip-accel based on pip 1.4.x maintained a local source
distribution index based on a directory containing symbolic links
pointing directly into pip’s download cache. When files were removed
from pip’s download cache, broken symbolic links remained in
pip-accel’s local source distribution index directory. This resulted
in very confusing error messages. To avoid this
-
install_from_arguments
(arguments, **kw)¶ Download, unpack, build and install the specified requirements.
This function is a simple wrapper for
get_requirements()
,install_requirements()
andcleanup_temporary_directories()
that implements the default behavior of the pip accelerator. If you’re extending or embedding pip-accel you may want to call the underlying methods instead.If the requirement set includes wheels and
setuptools >= 0.8
is not yet installed, it will be added to the requirement set and installed together with the other requirement(s) in order to enable the usage of distributions installed from wheels (their metadata is different).Parameters: - arguments – The command line arguments to
pip install ..
(a list of strings). - kw – Any keyword arguments are passed on to
install_requirements()
.
Returns: The result of
install_requirements()
.- arguments – The command line arguments to
-
setuptools_supports_wheels
()¶ Check whether setuptools should be upgraded to
>= 0.8
for wheel support.Returns: True
when setuptools 0.8 or higher is already installed,False
otherwise (it needs to be upgraded).
-
get_requirements
(arguments, max_retries=None, use_wheels=False)¶ Use pip to download and unpack the requested source distribution archives.
Parameters: - arguments – The command line arguments to
pip install ...
(a list of strings). - max_retries – The maximum number of times that pip will be asked
to download distribution archives (this helps to
deal with intermittent failures). If this is
None
thenmax_retries
is used. - use_wheels – Whether pip and pip-accel are allowed to use wheels
(
False
by default for backwards compatibility with callers that use pip-accel as a Python API).
Warning
Requirements which are already installed are not included in the result. If this breaks your use case consider using pip’s
--ignore-installed
option.- arguments – The command line arguments to
-
decorate_arguments
(arguments)¶ Change pathnames of local files into
file://
URLs with#md5=...
fragments.Parameters: arguments – The command line arguments to pip install ...
(a list of strings).Returns: A copy of the command line arguments with pathnames of local files rewritten to file://
URLs.When pip-accel calls pip to download missing distribution archives and the user specified the pathname of a local distribution archive on the command line, pip will (by default) not copy the archive into the download directory if an archive for the same package name and version is already present.
This can lead to the confusing situation where the user specifies a local distribution archive to install, a different (older) archive for the same package and version is present in the download directory and pip-accel installs the older archive instead of the newer archive.
To avoid this confusing behavior, the
decorate_arguments()
method rewrites the command line arguments given topip install
so that pathnames of local archives are changed intofile://
URLs that include a fragment with the hash of the file’s contents. Here’s an example:- Local pathname:
/tmp/pep8-1.6.3a0.tar.gz
- File URL:
file:///tmp/pep8-1.6.3a0.tar.gz#md5=19cbf0b633498ead63fb3c66e5f1caf6
When pip fills the download directory and encounters a previously cached distribution archive it will check the hash, realize the contents have changed and replace the archive in the download directory.
- Local pathname:
-
unpack_source_dists
(arguments, use_wheels=False)¶ Find and unpack local source distributions and discover their metadata.
Parameters: Returns: A list of
pip_accel.req.Requirement
objects.Raises: Any exceptions raised by pip, for example
pip.exceptions.DistributionNotFound
when not all requirements can be satisfied.This function checks whether there are local source distributions available for all requirements, unpacks the source distribution archives and finds the names and versions of the requirements. By using the
pip install --download
command we avoid reimplementing the following pip features:- Parsing of
requirements.txt
(including recursive parsing). - Resolution of possibly conflicting pinned requirements.
- Unpacking source distributions in multiple formats.
- Finding the name & version of a given source distribution.
- Parsing of
-
download_source_dists
(arguments, use_wheels=False)¶ Download missing source distributions.
Parameters: Raises: Any exceptions raised by pip.
-
get_pip_requirement_set
(arguments, use_remote_index, use_wheels=False)¶ Get the unpacked requirement(s) specified by the caller by running pip.
Parameters: - arguments – The command line arguments to
pip install ...
(a list of strings). - use_remote_index – A boolean indicating whether pip is allowed to connect to the main package index (http://pypi.python.org by default).
- use_wheels – Whether pip and pip-accel are allowed to use wheels
(
False
by default for backwards compatibility with callers that use pip-accel as a Python API).
Returns: A
pip.req.RequirementSet
object created by pip.Raises: Any exceptions raised by pip.
- arguments – The command line arguments to
-
transform_pip_requirement_set
(requirement_set)¶ Transform pip’s requirement set into one that pip-accel can work with.
Parameters: requirement_set – The pip.req.RequirementSet
object reported by pip.Returns: A list of pip_accel.req.Requirement
objects.This function converts the
pip.req.RequirementSet
object reported by pip into a list ofpip_accel.req.Requirement
objects.
-
install_requirements
(requirements, **kw)¶ Manually install a requirement set from binary and/or wheel distributions.
Parameters: - requirements – A list of
pip_accel.req.Requirement
objects. - kw – Any keyword arguments are passed on to
install_binary_dist()
.
Returns: The number of packages that were just installed (an integer).
- requirements – A list of
-
arguments_allow_wheels
(arguments)¶ Check whether the given command line arguments allow the use of wheels.
Parameters: arguments – A list of strings with command line arguments. Returns: True
if the arguments allow wheels,False
if they disallow wheels.Contrary to what the name of this method implies its implementation actually checks if the user hasn’t disallowed the use of wheels using the
--no-use-wheel
option (deprecated in pip 7.x) or the--no-binary=:all:
option (introduced in pip 7.x). This is because wheels are “opt out” in recent versions of pip. I just didn’t like the method namearguments_dont_disallow_wheels
;-).
-
create_build_directory
()¶ Create a new build directory for pip to unpack its archives.
-
clear_build_directory
()¶ Clear the build directory where pip unpacks the source distribution archives.
-
cleanup_temporary_directories
()¶ Delete the build directories and any temporary directories created by pip.
-
build_directory
¶ Get the pathname of the current build directory (a string).
-
-
class
pip_accel.
DownloadLogFilter
(name='')¶ Rewrite log messages emitted by pip’s
pip.download
module.When pip encounters hash mismatches it logs a message with the severity
CRITICAL
, however because of the interaction between pip-accel and pip hash mismatches are to be expected and handled gracefully (refer todecorate_arguments()
for details). TheDownloadLogFilter
context manager changes the severity of these log messages toDEBUG
in order to avoid confusing users of pip-accel.-
__enter__
()¶ Enable the download log filter.
-
__exit__
(exc_type=None, exc_value=None, traceback=None)¶ Disable the download log filter.
-
filter
(record)¶ Change the severity of selected log records.
-
-
class
pip_accel.
SetupRequiresPatch
(config, created_links=None)¶ Monkey patch to enable caching of setup requirements.
This context manager monkey patches
InstallRequirement.run_egg_info()
to enable caching of setup requirements. It works by creating a symbolic link called.eggs
in the source directory of unpacked Python source distributions which points to a shared directory inside the pip-accel data directory. This can only work on platforms that supportos.symlink()`()
but should fail gracefully elsewhere.The
SetupRequiresPatch
context manager doesn’t clean up the symbolic links because doing so would remove the link when it is still being used. Instead the context manager builds up a list of created links so that pip-accel can clean these up when it is known that the symbolic links are no longer needed.For more information about this hack please refer to issue 49.
-
__init__
(config, created_links=None)¶ Initialize a
SetupRequiresPatch
object.Parameters: - config – A
Config
object. - created_links – A list where newly created symbolic links are added to (so they can be cleaned up later).
- config – A
-
__enter__
()¶ Enable caching of setup requirements (by patching the
run_egg_info()
method).
-
__exit__
(exc_type=None, exc_value=None, traceback=None)¶ Undo the changes that enable caching of setup requirements.
-
-
class
pip_accel.
CustomPackageFinder
(find_links, index_urls, allow_all_prereleases=False, trusted_hosts=None, process_dependency_links=False, session=None, format_control=None, platform=None, versions=None, abi=None, implementation=None)¶ Custom
pip.index.PackageFinder
to keep pip off the internet.This class customizes
pip.index.PackageFinder
to enforce what the--no-index
option does for the default package index but doesn’t do for package indexes registered with the--index=
option in requirements files. Judging by pip’s documentation the fact that this has to be monkey patched seems like a bug / oversight in pip (IMHO).-
index_urls
¶ Dummy list of index URLs that is always empty.
-
dependency_links
¶ Dummy list of dependency links that is always empty.
-
-
class
pip_accel.
PatchedAttribute
(object, attribute, value, enabled=True)¶ Context manager to temporarily patch an object attribute.
This context manager changes the value of an object attribute when the context is entered and restores the original value when the context is exited.
-
__init__
(object, attribute, value, enabled=True)¶ Initialize a
PatchedAttribute
object.Parameters: - object – The object whose attribute should be patched.
- attribute – The name of the attribute to be patched (a string).
- value – The temporary value for the attribute.
- enabled –
True
to patch the attribute,False
to do nothing instead. This enables conditional attribute patching while unconditionally using thewith
statement.
-
__enter__
()¶ Change the object attribute when entering the context.
-
__exit__
(exc_type=None, exc_value=None, traceback=None)¶ Restore the object attribute when leaving the context.
-
-
class
pip_accel.
AttributeOverrides
(opts, **overrides)¶ AttributeOverrides
enables overriding of object attributes.During the pip 6.x upgrade pip-accel switched to using
pip install --download
which unintentionally broke backwards compatibility with previous versions of pip-accel as documented in issue 52.The reason for this is that when pip is given the
--download
option it internally enables--ignore-installed
(which can be problematic for certain use cases as described in issue 52). There is no documented way to avoid this behavior, so instead pip-accel resorts to monkey patching to restore backwards compatibility.AttributeOverrides
is used to replace pip’s parsed command line options object with an object that defers all attribute access (gets and sets) to the original options object but always reportsignore_installed
asFalse
, even after it was set toTrue
by pip (as described above).-
__init__
(opts, **overrides)¶ Construct an
AttributeOverrides
instance.Parameters: - opts – The object to which attribute access is deferred.
- overrides – The attributes whose value should be overridden.
-
__getattr__
(name)¶ Get an attribute’s value from overrides or by deferring attribute access.
Parameters: name – The name of the attribute (a string). Returns: The attribute’s value.
-
__setattr__
(name, value)¶ Set an attribute’s value (unless it has an override).
Parameters: - name – The name of the attribute (a string).
- value – The new value for the attribute.
-
pip_accel.config
¶
Configuration handling for pip-accel.
This module defines the Config
class which is used throughout the
pip accelerator. At runtime an instance of Config
is created and
passed down like this:
The PipAccelerator
class receives its configuration object from
its caller. Usually this will be main()
but when pip-accel is used
as a Python API the person embedding or extending pip-accel is responsible for
providing the configuration object. This is intended as a form of dependency
injection that enables non-default configurations to be injected into
pip-accel.
Support for runtime configuration¶
The properties of the Config
class can be set at runtime using
regular attribute assignment syntax. This overrides the default values of the
properties (whether based on environment variables, configuration files or hard
coded defaults).
Support for configuration files¶
You can use a configuration file to permanently configure certain options of
pip-accel. If /etc/pip-accel.conf
and/or ~/.pip-accel/pip-accel.conf
exist they are automatically loaded. You can also set the environment variable
$PIP_ACCEL_CONFIG
to load a configuration file in a non-default location.
If all three files exist the system wide file is loaded first, then the user
specific file is loaded and then the file set by the environment variable is
loaded (duplicate settings are overridden by the configuration file that’s
loaded last).
Here is an example of the available options:
[pip-accel] auto-install = yes max-retries = 3 data-directory = ~/.pip-accel s3-bucket = my-shared-pip-accel-binary-cache s3-prefix = ubuntu-trusty-amd64 s3-readonly = yes
Note that the configuration options shown above are just examples, they are not meant to represent the configuration defaults.
-
class
pip_accel.config.
Config
(load_configuration_files=True, load_environment_variables=True)¶ Configuration of the pip accelerator.
-
__init__
(load_configuration_files=True, load_environment_variables=True)¶ Initialize the configuration of the pip accelerator.
Parameters:
-
available_configuration_files
¶ A list of strings with the absolute pathnames of the available configuration files.
-
load_configuration_file
(configuration_file)¶ Load configuration defaults from a configuration file.
Parameters: configuration_file – The pathname of a configuration file (a string). Raises: Exception
when the configuration file cannot be loaded.
-
__setattr__
(name, value)¶ Override the value of a property at runtime.
Parameters: - name – The name of the property to override (a string).
- value – The overridden value of the property.
-
get
(property_name=None, environment_variable=None, configuration_option=None, default=None)¶ Internal shortcut to get a configuration option’s value.
Parameters: - property_name – The name of the property that users can set on
the
Config
class (a string). - environment_variable – The name of the environment variable (a string).
- configuration_option – The name of the option in the configuration file (a string).
- default – The default value.
Returns: The value of the environment variable or configuration file option or the default value.
- property_name – The name of the property that users can set on
the
-
cache_format_revision
¶ The revision of the binary distribution cache format in use (an integer).
This number is encoded in the directory name of the binary cache so that multiple revisions can peacefully coexist. When pip-accel breaks backwards compatibility this number is bumped so that pip-accel starts using a new directory.
-
source_index
¶ The absolute pathname of pip-accel’s source index directory (a string).
This is the
sources
subdirectory ofdata_directory
.
-
binary_cache
¶ The absolute pathname of pip-accel’s binary cache directory (a string).
This is the
binaries
subdirectory ofdata_directory
.
-
eggs_cache
¶ The absolute pathname of pip-accel’s eggs cache directory (a string).
This is the
eggs
subdirectory ofdata_directory
. It is used to cache setup requirements which avoids continuous rebuilding of setup requirements.
-
data_directory
¶ The absolute pathname of the directory where pip-accel’s data files are stored (a string).
- Environment variable:
$PIP_ACCEL_CACHE
- Configuration option:
data-directory
- Default:
/var/cache/pip-accel
if running asroot
,~/.pip-accel
otherwise
- Environment variable:
-
install_prefix
¶ The absolute pathname of the installation prefix to use (a string).
This property is based on
sys.prefix
except that whensys.prefix
is/usr
and we’re running on a Debian derived system/usr/local
is used instead.The reason for this is that on Debian derived systems only apt (dpkg) should be allowed to touch files in
/usr/lib/pythonX.Y/dist-packages
andpython setup.py install
knows this (see theposix_local
installation scheme in/usr/lib/pythonX.Y/sysconfig.py
on Debian derived systems). Because pip-accel replacespython setup.py install
it has to replicate this logic. Inferring all of this from thesysconfig
module would be nice but that module wasn’t available in Python 2.6.
-
python_executable
¶ The absolute pathname of the Python executable (a string).
-
auto_install
¶ Whether automatic installation of missing system packages is enabled.
True
if automatic installation of missing system packages is enabled,False
if it is disabled,None
otherwise (in this case the user will be prompted at the appropriate time).- Environment variable:
$PIP_ACCEL_AUTO_INSTALL
(refer tocoerce_boolean()
for details on how the value of the environment variable is interpreted) - Configuration option:
auto-install
(also parsed usingcoerce_boolean()
) - Default:
None
- Environment variable:
-
log_format
¶ The format of log messages written to the terminal.
- Environment variable:
$PIP_ACCEL_LOG_FORMAT
- Configuration option:
log-format
- Default:
DEFAULT_LOG_FORMAT
- Environment variable:
-
log_verbosity
¶ The verbosity of log messages written to the terminal.
- Environment variable:
$PIP_ACCEL_LOG_VERBOSITY
- Configuration option:
log-verbosity
- Default: ‘INFO’ (a string).
- Environment variable:
-
max_retries
¶ The number of times to retry
pip install --download
if it fails.- Environment variable:
$PIP_ACCEL_MAX_RETRIES
- Configuration option:
max-retries
- Default:
3
- Environment variable:
-
trust_mod_times
¶ Whether to trust file modification times for cache invalidation.
-
s3_cache_url
¶ The URL of the Amazon S3 API endpoint to use.
By default this points to the official Amazon S3 API endpoint. You can change this option if you’re running a local Amazon S3 compatible storage service that you want pip-accel to use.
- Environment variable:
$PIP_ACCEL_S3_URL
- Configuration option:
s3-url
- Default:
https://s3.amazonaws.com
For details please refer to the
pip_accel.caches.s3
module.- Environment variable:
-
s3_cache_bucket
¶ Name of Amazon S3 bucket where binary distributions are cached (a string or
None
).- Environment variable:
$PIP_ACCEL_S3_BUCKET
- Configuration option:
s3-bucket
- Default:
None
For details please refer to the
pip_accel.caches.s3
module.- Environment variable:
-
s3_cache_create_bucket
¶ Whether to automatically create the Amazon S3 bucket when it’s missing.
- Environment variable:
$PIP_ACCEL_S3_CREATE_BUCKET
- Configuration option:
s3-create-bucket
- Default:
False
For details please refer to the
pip_accel.caches.s3
module.- Environment variable:
-
s3_cache_prefix
¶ Cache prefix for binary distribution archives in Amazon S3 bucket (a string or
None
).- Environment variable:
$PIP_ACCEL_S3_PREFIX
- Configuration option:
s3-prefix
- Default:
None
For details please refer to the
pip_accel.caches.s3
module.- Environment variable:
-
s3_cache_readonly
¶ Whether the Amazon S3 bucket is considered read only.
If this is
True
then the Amazon S3 bucket will only be used forget()
operations (allput()
operations will be disabled).- Environment variable:
$PIP_ACCEL_S3_READONLY
(refer tocoerce_boolean()
for details on how the value of the environment variable is interpreted) - Configuration option:
s3-readonly
(also parsed usingcoerce_boolean()
) - Default:
False
For details please refer to the
pip_accel.caches.s3
module.- Environment variable:
-
s3_cache_timeout
¶ The socket timeout in seconds for connections to Amazon S3 (an integer).
This value is injected into Boto’s configuration to override the default socket timeout used for connections to Amazon S3.
- Environment variable:
$PIP_ACCEL_S3_TIMEOUT
- Configuration option:
s3-timeout
- Default:
60
(Boto’s default)
- Environment variable:
-
s3_cache_retries
¶ The number of times to retry failed requests to Amazon S3 (an integer).
This value is injected into Boto’s configuration to override the default number of times to retry failed requests to Amazon S3.
- Environment variable:
$PIP_ACCEL_S3_RETRIES
- Configuration option:
s3-retries
- Default:
5
(Boto’s default)
- Environment variable:
-
pip_accel.req
¶
Simple wrapper for pip and pkg_resources Requirement objects.
After downloading the specified requirement(s) pip reports a “requirement set” to pip-accel. In the past pip-accel would summarize this requirement set into a list of tuples, where each tuple would contain a requirement’s project name, version and source directory (basically only the information required by pip-accel remained).
Recently I’ve started using pip-accel as a library in another project I’m working on (not yet public) and in that project I am very interested in whether a given requirement is a direct or transitive requirement. Unfortunately pip-accel did not preserve this information.
That’s when I decided that next to pip’s pip.req.InstallRequirement
and setuptools’ pkg_resources.Requirement
I would introduce yet
another type of requirement object... It’s basically just a summary of the
other two types of requirement objects and it also provides access to the
original requirement objects (for those who are interested; the interfaces are
basically undocumented AFAIK).
-
class
pip_accel.req.
Requirement
(config, requirement)¶ Simple wrapper for the requirement objects defined by pip and setuptools.
-
__init__
(config, requirement)¶ Initialize a requirement object.
Parameters: - config – A
Config
object. - requirement – A
pip.req.InstallRequirement
object.
- config – A
-
__repr__
()¶ Generate a human friendly representation of a requirement object.
-
name
¶ The name of the Python package (a string).
This is the name used to register a package on PyPI and the name reported by commands like
pip freeze
. Based onpkg_resources.Requirement.project_name
.
-
version
¶ The version of the package that
pip
wants to install (a string).
The pathnames of the source distribution(s) for this requirement (a list of strings).
Note
This property is very new in pip-accel and its logic may need some time to mature. For now any misbehavior by this property shouldn’t be too much of a problem because the pathnames reported by this property are only used for cache invalidation (see the
last_modified
andchecksum
properties).
-
last_modified
¶ The last modified time of the requirement’s source distribution archive(s) (a number).
The value of this property is based on the
related_archives
property. If no related archives are found the current time is reported. In the balance between not invalidating cached binary distributions enough and invalidating them too frequently, this property causes the latter to happen.
-
checksum
¶ The SHA1 checksum of the requirement’s source distribution archive(s) (a string).
The value of this property is based on the
related_archives
property. If no related archives are found the SHA1 digest of the empty string is reported.
-
source_directory
¶ The pathname of the directory containing the unpacked source distribution (a string).
This is the directory that contains a
setup.py
script. Based onpip.req.InstallRequirement.source_dir
.
-
is_wheel
¶ True
when the requirement is a wheel,False
otherwise.Note
To my surprise it seems to be non-trivial to determine whether a given
pip.req.InstallRequirement
object produced by pip’s internal Python API concerns a source distribution or a wheel distribution.There’s a
pip.req.InstallRequirement.is_wheel
property but I’m currently looking at a wheel distribution whoseis_wheel
property returnsNone
, apparently because the requirement’surl
property is alsoNone
.Whether this is an obscure implementation detail of pip or caused by the way pip-accel invokes pip, I really can’t tell (yet).
-
is_transitive
¶ Whether the dependency is transitive (indirect).
True
when the requirement is a transitive dependency (a dependency of a dependency) orFalse
when the requirement is a direct dependency (specified on pip’s command line or in arequirements.txt
file). Based onpip.req.InstallRequirement.comes_from
.
-
is_direct
¶ The opposite of
Requirement.is_transitive
.
-
is_editable
¶ Whether the requirement should be installed in editable mode.
True
when the requirement is to be installed in editable mode (i.e. setuptools “develop mode”). Based onpip.req.InstallRequirement.editable
.
-
sdist_metadata
¶ Get the distribution metadata of an unpacked source distribution.
-
wheel_metadata
¶ Get the distribution metadata of an unpacked wheel distribution.
-
__str__
()¶ Render a human friendly string describing the requirement.
-
-
class
pip_accel.req.
TransactionalUpdate
(requirement)¶ Context manager that enables transactional package upgrades.
-
__init__
(requirement)¶ Initialize a
TransactionalUpdate
object.Parameters: requirement – A Requirement
object.
-
__enter__
()¶ Prepare package upgrades by removing conflicting installations.
-
__exit__
(exc_type=None, exc_value=None, traceback=None)¶ Finalize or rollback a package upgrade.
-
-
pip_accel.req.
escape_name
(requirement_name)¶ Escape a requirement’s name for use in a regular expression.
This backslash-escapes all non-alphanumeric characters and replaces dashes and underscores with a character class that matches a dash or underscore (effectively treating dashes and underscores equivalently).
Parameters: requirement_name – The name of the requirement (a string). Returns: The requirement’s name as a regular expression (a string).
-
pip_accel.req.
escape_name_callback
(match)¶ Used by
escape_name()
to treat dashes and underscores as equivalent.Parameters: match – A regular expression match object that captured a single character. Returns: A regular expression string that matches the captured character.
pip_accel.bdist
¶
Functions to manipulate Python binary distribution archives.
The functions in this module are used to create, transform and install from binary distribution archives (which are not supported by tools like easy_install and pip).
-
class
pip_accel.bdist.
BinaryDistributionManager
(config)¶ Generates and transforms Python binary distributions.
-
__init__
(config)¶ Initialize the binary distribution manager.
Parameters: config – The pip-accel configuration (a Config
object).
-
get_binary_dist
(requirement)¶ Get or create a cached binary distribution archive.
Parameters: requirement – A Requirement
object.Returns: An iterable of tuples with two values each: A tarfile.TarInfo
object and a file-like object.Gets the cached binary distribution that was previously built for the given requirement. If no binary distribution has been cached yet, a new binary distribution is built and added to the cache.
Uses
build_binary_dist()
to build binary distribution archives. If this fails with a build errorget_binary_dist()
will useSystemPackageManager
to check for and install missing system packages and retry the build when missing system packages were installed.
-
needs_invalidation
(requirement, cache_file)¶ Check whether a cached binary distribution needs to be invalidated.
Parameters: - requirement – A
Requirement
object. - cache_file – The pathname of a cached binary distribution (a string).
Returns: True
if the cached binary distribution needs to be invalidated,False
otherwise.- requirement – A
-
recall_checksum
(cache_file)¶ Get the checksum of the input used to generate a binary distribution archive.
Parameters: cache_file – The pathname of the binary distribution archive (a string). Returns: The checksum (a string) or None
(when no checksum is available).
-
persist_checksum
(requirement, cache_file)¶ Persist the checksum of the input used to generate a binary distribution.
Parameters: - requirement – A
Requirement
object. - cache_file – The pathname of a cached binary distribution (a string).
Note
The checksum is only calculated and persisted when
trust_mod_times
isFalse
.- requirement – A
-
build_binary_dist
(requirement)¶ Build a binary distribution archive from an unpacked source distribution.
Parameters: requirement – A Requirement
object.Returns: The pathname of a binary distribution archive (a string). Raises: BinaryDistributionError
when the original command and the fall back both fail to produce a binary distribution archive.This method uses the following command to build binary distributions:
$ python setup.py bdist_dumb --format=tar
This command can fail for two main reasons:
- The package is missing binary dependencies.
- The
setup.py
script doesn’t (properly) implementbdist_dumb
binary distribution format support.
The first case is dealt with in
get_binary_dist()
. To deal with the second case this method falls back to the following command:$ python setup.py bdist
This fall back is almost never needed, but there are Python packages out there which require this fall back (this method was added because the installation of
Paver==1.2.3
failed, see issue 37 for details about that).
-
build_binary_dist_helper
(requirement, setup_command)¶ Convert an unpacked source distribution to a binary distribution.
Parameters: - requirement – A
Requirement
object. - setup_command – A list of strings with the arguments to
setup.py
.
Returns: The pathname of the resulting binary distribution (a string).
Raises: BuildFailed
when the build reports an error (e.g. because of missing binary dependencies like system libraries).Raises: NoBuildOutput
when the build does not produce the expected binary distribution archive.- requirement – A
-
transform_binary_dist
(archive_path)¶ Transform binary distributions into a form that can be cached for future use.
Parameters: archive_path – The pathname of the original binary distribution archive. Returns: An iterable of tuples with two values each: - A
tarfile.TarInfo
object. - A file-like object.
This method transforms a binary distribution archive created by
build_binary_dist()
into a form that can be cached for future use. This comes down to making the pathnames inside the archive relative to the prefix that the binary distribution was built for.- A
-
install_binary_dist
(members, virtualenv_compatible=True, prefix=None, python=None, track_installed_files=False)¶ Install a binary distribution into the given prefix.
Parameters: - members –
An iterable of tuples with two values each:
- A
tarfile.TarInfo
object. - A file-like object.
- A
- prefix – The “prefix” under which the requirements should be
installed. This will be a pathname like
/usr
,/usr/local
or the pathname of a virtual environment. Defaults toConfig.install_prefix
. - python – The pathname of the Python executable to use in the shebang
line of all executable Python scripts inside the binary
distribution. Defaults to
Config.python_executable
. - virtualenv_compatible – Whether to enable workarounds to make the
resulting filenames compatible with
virtual environments (defaults to
True
). - track_installed_files – If this is
True
(not the default for this method because of backwards compatibility) pip-accel will createinstalled-files.txt
as required by pip to properly uninstall packages.
This method installs a binary distribution created by
build_binary_dist()
into the given prefix (a directory like/usr
,/usr/local
or a virtual environment).- members –
-
fix_hashbang
(contents, python)¶ Rewrite hashbangs to use the correct Python executable.
Parameters: - contents – The contents of the script whose hashbang should be fixed (a string).
- python – The absolute pathname of the Python executable (a string).
Returns: The modified contents of the script (a string).
-
update_installed_files
(installed_files)¶ Track the files installed by a package so pip knows how to remove the package.
This method is used by
install_binary_dist()
(which collects the list of installed files forupdate_installed_files()
).Parameters: installed_files – A list of absolute pathnames (strings) with the files that were just installed.
-
pip_accel.caches
¶
Support for multiple cache backends.
This module defines an abstract base class (AbstractCacheBackend
)
to be inherited by custom cache backends in order to easily integrate them in
pip-accel. The cache backends included in pip-accel are built on top of the
same mechanism.
Additionally this module defines CacheManager
which makes it
possible to merge the available cache backends into a single logical cache
which automatically disables backends that report errors.
-
class
pip_accel.caches.
CacheBackendMeta
(name, bases, dict)¶ Metaclass to intercept cache backend definitions.
-
__init__
(name, bases, dict)¶ Intercept cache backend definitions.
-
-
class
pip_accel.caches.
AbstractCacheBackend
(config)¶ Abstract base class for implementations of pip-accel cache backends.
Subclasses of this class are used by pip-accel to store Python distribution archives in order to accelerate performance and gain independence of external systems like PyPI and distribution sites.
Note
This base class automatically registers subclasses at definition time, providing a simple and elegant registration mechanism for custom backends. This technique uses metaclasses and was originally based on the article Using Metaclasses to Create Self-Registering Plugins.
I’ve since had to introduce some additional magic to make this mechanism compatible with both Python 2.x and Python 3.x because the syntax for metaclasses is very much incompatible and I refuse to write separate implementations for both :-).
-
__init__
(config)¶ Initialize a cache backend.
Parameters: config – The pip-accel configuration (a Config
object).
-
get
(filename)¶ Get a previously cached distribution archive from the cache.
Parameters: filename – The expected filename of the distribution archive (a string). Returns: The absolute pathname of a local file or None
when the distribution archive hasn’t been cached.This method is called by pip-accel before fetching or building a distribution archive, in order to check whether a previously cached distribution archive is available for re-use.
-
put
(filename, handle)¶ Store a newly built distribution archive in the cache.
Parameters: - filename – The filename of the distribution archive (a string).
- handle – A file-like object that provides access to the distribution archive.
This method is called by pip-accel after fetching or building a distribution archive, in order to cache the distribution archive.
-
__repr__
()¶ Generate a textual representation of the cache backend.
-
-
class
pip_accel.caches.
CacheManager
(config)¶ Interface to treat multiple cache backends as a single one.
The cache manager automatically disables cache backends that raise exceptions on
get()
andput()
operations.-
__init__
(config)¶ Initialize a cache manager.
Automatically initializes instances of all registered cache backends based on setuptools’ support for entry points which makes it possible for external Python packages to register additional cache backends without any modifications to pip-accel.
Parameters: config – The pip-accel configuration (a Config
object).
-
get
(requirement)¶ Get a distribution archive from any of the available caches.
Parameters: requirement – A Requirement
object.Returns: The absolute pathname of a local file or None
when the distribution archive is missing from all available caches.
-
put
(requirement, handle)¶ Store a distribution archive in all of the available caches.
Parameters: - requirement – A
Requirement
object. - handle – A file-like object that provides access to the distribution archive.
- requirement – A
-
generate_filename
(requirement)¶ Generate a distribution archive filename for a package.
Parameters: requirement – A Requirement
object.Returns: The filename of the distribution archive (a string) including a single leading directory component to indicate the cache format revision.
-
pip_accel.caches.local
¶
Local file system cache backend.
This module implements the local cache backend which stores distribution
archives on the local file system. This is a very simple cache backend, all it
does is create directories and write local files. The only trick here is that
new binary distribution archives are written to temporary files which are
then moved into place atomically using os.rename()
to avoid partial
reads caused by running multiple invocations of pip-accel at the same time
(which happened in issue 25).
-
class
pip_accel.caches.local.
LocalCacheBackend
(config)¶ The local cache backend stores Python distribution archives on the local file system.
-
get
(filename)¶ Check if a distribution archive exists in the local cache.
Parameters: filename – The filename of the distribution archive (a string). Returns: The pathname of a distribution archive on the local file system or None
.
-
put
(filename, handle)¶ Store a distribution archive in the local cache.
Parameters: - filename – The filename of the distribution archive (a string).
- handle – A file-like object that provides access to the distribution archive.
-
pip_accel.caches.s3
¶
Amazon S3 cache backend.
This module implements a cache backend that stores distribution archives in a
user defined Amazon S3 bucket. To enable this
backend you need to define the configuration option
s3_cache_bucket
and configure your Amazon S3 API
credentials (see the readme for details).
Using S3 compatible storage services¶
The Amazon S3 API has been implemented in several open source projects and
dozens of online services. To use pip-accel with an S3 compatible storage
service you can override the s3_cache_url
option. The
pip-accel test suite actually uses this option to test the S3 cache backend by
running FakeS3 in the background and pointing pip-accel at the FakeS3 server.
Below are some usage notes that may be relevant for people evaluating this
option.
- Secure connections
- Boto has to be told whether to make a “secure” connection to the S3 API and
pip-accel assumes the
https://
URL scheme implies a secure connection while thehttp://
URL scheme implies a non-secure connection. - Calling formats
- Boto has the concept of “calling formats” for the S3 API and to connect to the official Amazon S3 API pip-accel needs to specify the “sub-domain calling format” or the API calls will fail. When you specify a nonstandard S3 API URL pip-accel tells Boto to use the “ordinary calling format” instead. This differentiation will undoubtedly not be correct in all cases. If this is bothering you then feel free to open an issue on GitHub to make pip-accel more flexible in this regard.
- Credentials
- If you don’t specify S3 API credentials and the connection attempt to S3 fails with “NoAuthHandlerFound: No handler was ready to authenticate” pip-accel will fall back to an anonymous connection attempt. If that fails as well the S3 cache backend is disabled. It may be useful to note here that the pip-accel test suite uses FakeS3 and the anonymous connection fall back works fine.
A note about robustness¶
The Amazon S3 cache backend implemented in pip_accel.caches.s3
is
specifically written to gracefully disable itself when it encounters known
errors such as:
- The configuration option
s3_cache_bucket
is not set (i.e. the user hasn’t configured the backend yet). - The
boto
package is not installed (i.e. the user ranpip install pip-accel
instead ofpip install 'pip-accel[s3]'
). - The connection to the S3 API can’t be established (e.g. because API credentials haven’t been correctly configured).
- The connection to the configured S3 bucket can’t be established (e.g. because the bucket doesn’t exist or the configured credentials don’t provide access to the bucket).
Additionally CacheManager
automatically disables
cache backends that raise exceptions on
get()
and
put()
operations. The end
result is that when the S3 backend fails you will just revert to using the
cache on the local file system.
Optionally if you are using read only credentials you can disable
put()
operations by setting the configuration
option s3_cache_readonly
.
-
class
pip_accel.caches.s3.
S3CacheBackend
(config)¶ The S3 cache backend stores distribution archives in a user defined Amazon S3 bucket.
-
get
(filename)¶ Download a distribution archive from the configured Amazon S3 bucket.
Parameters: filename – The filename of the distribution archive (a string). Returns: The pathname of a distribution archive on the local file system or None
.Raises: CacheBackendError
when any underlying method fails.
-
put
(filename, handle)¶ Upload a distribution archive to the configured Amazon S3 bucket.
If the
s3_cache_readonly
configuration option is enabled this method does nothing.Parameters: - filename – The filename of the distribution archive (a string).
- handle – A file-like object that provides access to the distribution archive.
Raises: CacheBackendError
when any underlying method fails.
-
s3_bucket
¶ Connect to the user defined Amazon S3 bucket.
Called on demand by
get()
andput()
. Caches its return value so that only a single connection is created.Returns: A boto.s3.bucket.Bucket
object.Raises: CacheBackendDisabledError
when the user hasn’t definedConfig.s3_cache_bucket
.Raises: CacheBackendError
when the connection to the Amazon S3 bucket fails.
-
s3_connection
¶ Connect to the Amazon S3 API.
If the connection attempt fails because Boto can’t find credentials the attempt is retried once with an anonymous connection.
Called on demand by
s3_bucket
.Returns: A boto.s3.connection.S3Connection
object.Raises: CacheBackendError
when the connection to the Amazon S3 API fails.
-
get_cache_key
(filename)¶ Compose an S3 cache key based on
Config.s3_cache_prefix
and the given filename.Parameters: filename – The filename of the distribution archive (a string). Returns: The cache key for the given filename (a string).
-
check_prerequisites
()¶ Validate the prerequisites required to use the Amazon S3 cache backend.
Makes sure the Amazon S3 cache backend is configured (
Config.s3_cache_bucket
is defined by the user) andboto
is available for use.Raises: CacheBackendDisabledError
when a prerequisite fails.
-
-
class
pip_accel.caches.s3.
PatchedBotoConfig
¶ Monkey patch for Boto’s configuration handling.
Boto’s configuration handling is kind of broken on Python 3 as documented here. The
PatchedBotoConfig
class implements a context manager that temporarily patches Boto to work around the bug.Without this monkey patch it is impossible to configure the number of retries on Python 3 which makes the pip-accel test suite horribly slow.
-
__init__
()¶ Initialize a
PatchedBotoConfig
object.
-
get
(section, name, default=None, **kw)¶ Replacement for
boto.pyami.config.Config.get()
.
-
pip_accel.deps
¶
System package dependency handling.
The pip_accel.deps
module is an extension of pip-accel that deals
with dependencies on system packages. Currently only Debian Linux and
derivative Linux distributions are supported by this extension but it should be
fairly easy to add support for other platforms.
The interface between pip-accel and SystemPackageManager
focuses on
install_dependencies()
(the other methods are
used internally).
-
class
pip_accel.deps.
SystemPackageManager
(config)¶ Interface to the system’s package manager.
-
__init__
(config)¶ Initialize the system package dependency manager.
Parameters: config – The pip-accel configuration (a Config
object).
-
install_dependencies
(requirement)¶ Install missing dependencies for the given requirement.
Parameters: requirement – A Requirement
object.Returns: True
when missing system packages were installed,False
otherwise.Raises: DependencyInstallationRefused
when automatic installation is disabled or refused by the operator.Raises: DependencyInstallationFailed
when the installation of missing system packages fails.If pip-accel fails to build a binary distribution, it will call this method as a last chance to install missing dependencies. If this function does not raise an exception, pip-accel will retry the build once.
-
find_missing_dependencies
(requirement)¶ Find missing dependencies of a Python package.
Parameters: requirement – A Requirement
object.Returns: A list of strings with system package names.
-
find_known_dependencies
(requirement)¶ Find the known dependencies of a Python package.
Parameters: requirement – A Requirement
object.Returns: A list of strings with system package names.
-
find_installed_packages
()¶ Find the installed system packages.
Returns: A list of strings with system package names. Raises: SystemDependencyError
when the command to list the installed system packages fails.
-
installation_refused
(requirement, missing_dependencies, reason)¶ Raise
DependencyInstallationRefused
with a user friendly message.Parameters: - requirement – A
Requirement
object. - missing_dependencies – A list of strings with missing dependencies.
- reason – The reason why installation was refused (a string).
- requirement – A
-
confirm_installation
(requirement, missing_dependencies, install_command)¶ Ask the operator’s permission to install missing system packages.
Parameters: - requirement – A
Requirement
object. - missing_dependencies – A list of strings with missing dependencies.
- install_command – A list of strings with the command line needed to install the missing dependencies.
Raises: DependencyInstallationRefused
when the operator refuses.- requirement – A
-
pip_accel.utils
¶
Utility functions for the pip accelerator.
The pip_accel.utils
module defines several miscellaneous/utility
functions that are used throughout pip_accel
but don’t really belong
with any single module.
-
pip_accel.utils.
compact
(text, **kw)¶ Compact whitespace in a string and format any keyword arguments into the string.
Parameters: - text – The text to compact (a string).
- kw – Any keyword arguments to apply using
str.format()
.
Returns: The compacted, formatted string.
The whitespace compaction preserves paragraphs.
-
pip_accel.utils.
expand_path
(pathname)¶ Expand the home directory in a pathname based on the effective user id.
Parameters: pathname – A pathname that may start with ~/
, indicating the path should be interpreted as being relative to the home directory of the current (effective) user.Returns: The (modified) pathname. This function is a variant of
os.path.expanduser()
that doesn’t use$HOME
but instead uses the home directory of the effective user id. This is basically a workaround forsudo -s
not resetting$HOME
.
-
pip_accel.utils.
create_file_url
(pathname)¶ Create a
file:...
URL from a local pathname.Parameters: pathname – The pathname of a local file or directory (a string). Returns: A URL that refers to the local file or directory (a string).
-
pip_accel.utils.
find_home_directory
()¶ Look up the home directory of the effective user id.
Returns: The pathname of the home directory (a string). Note
On Windows this uses the
%APPDATA%
environment variable (if available) and otherwise falls back to~/Application Data
.
-
pip_accel.utils.
is_root
()¶ Detect whether we’re running with super user privileges.
-
pip_accel.utils.
get_python_version
()¶ Get a string identifying the currently running Python version.
This function generates a string that uniquely identifies the currently running Python implementation and version. The Python implementation is discovered using
platform.python_implementation()
and the major and minor version numbers are extracted fromsys.version_info
.Returns: A string containing the name of the Python implementation and the major and minor version numbers. Example:
>>> from pip_accel.utils import get_python_version >>> get_python_version() 'CPython-2.7'
-
pip_accel.utils.
makedirs
(path, mode=511)¶ Create a directory if it doesn’t already exist (keeping concurrency in mind).
Parameters: - path – The pathname of the directory to create (a string).
- mode – The mode to apply to newly created directories (an integer,
defaults to the octal number
0777
).
Returns: True
when the directory was created,False
if it already existed.Raises: Any exceptions raised by
os.makedirs()
except forerrno.EEXIST
(this error is swallowed andFalse
is returned instead).
-
pip_accel.utils.
same_directories
(path1, path2)¶ Check if two pathnames refer to the same directory.
Parameters: - path1 – The first pathname (a string).
- path2 – The second pathname (a string).
Returns: True
if both pathnames refer to the same directory,False
otherwise.
-
pip_accel.utils.
hash_files
(method, *files)¶ Calculate the hexadecimal digest of one or more local files.
Parameters: - method – The hash method (a string, given to
hashlib.new()
). - files – The pathname(s) of file(s) to hash (zero or more strings).
Returns: The calculated hex digest (a string).
- method – The hash method (a string, given to
-
pip_accel.utils.
replace_file
(src, dst)¶ Overwrite a file (in an atomic fashion when possible).
Parameters: - src – The pathname of the source file (a string).
- dst – The pathname of the destination file (a string).
-
class
pip_accel.utils.
AtomicReplace
(filename)¶ Context manager to atomically replace a file’s contents.
-
__init__
(filename)¶ Initialize a
AtomicReplace
object.Parameters: filename – The pathname of the file to replace (a string).
-
__enter__
()¶ Prepare to replace the file’s contents.
Returns: The pathname of a temporary file in the same directory as the file to replace (a string). Using this temporary file ensures that replace_file()
doesn’t fail due to a cross-device rename operation.
-
__exit__
(exc_type=None, exc_value=None, traceback=None)¶ Replace the file’s contents (if no exception occurred) using
replace_file()
.
-
-
pip_accel.utils.
requirement_is_installed
(expr)¶ Check whether a requirement is installed.
Parameters: expr – A requirement specification similar to those used in pip requirement files (a string). Returns: True
if the requirement is available (installed),False
otherwise.
-
pip_accel.utils.
is_installed
(package_name)¶ Check whether a package is installed in the current environment.
Parameters: package_name – The name of the package (a string). Returns: True
if the package is installed,False
otherwise.
-
pip_accel.utils.
uninstall
(*package_names)¶ Uninstall one or more packages using the Python equivalent of
pip uninstall --yes
.The package(s) to uninstall must be installed, otherwise pip will raise an
UninstallationError
. You can check for installed packages usingis_installed()
.Parameters: package_names – The names of one or more Python packages (strings).
-
pip_accel.utils.
match_option
(argument, short_option, long_option)¶ Match a command line argument against a short and long option.
Parameters: - argument – The command line argument (a string).
- short_option – The short option (a string).
- long_option – The long option (a string).
Returns:
-
pip_accel.utils.
is_short_option
(argument)¶ Check if a command line argument is a short option.
Parameters: argument – The command line argument (a string). Returns: True
if the argument is a short option,False
otherwise.
-
pip_accel.utils.
match_option_with_value
(arguments, option, value)¶ Check if a list of command line options contains an option with a value.
Parameters: - arguments – The command line arguments (a list of strings).
- option – The long option (a string).
- value – The expected value (a string).
Returns: True
if the command line contains the option/value pair,False
otherwise.
-
pip_accel.utils.
contains_sublist
(lst, sublst)¶ Check if one list contains the items from another list (in the same order).
Parameters: - lst – The main list.
- sublist – The sublist to check for.
Returns: True
if the main list contains the items from the sublist in the same order,False
otherwise.Based on this StackOverflow answer.
pip_accel.exceptions
¶
Exceptions for structured error handling.
This module defines named exceptions raised by pip-accel when it encounters error conditions that:
- Already require structured handling inside pip-accel
- May require structured handling by callers of pip-accel
Yes, I know, I just made your lovely and elegant Python look a whole lot like Java! I guess the message to take away here is that (in my opinion) structured error handling helps to build robust software that acknowledges failures exist and tries to deal with them (even if only by clearly recognizing a problem and giving up when there’s nothing useful to do!).
Hierarchy of exceptions¶
If you’re interested in implementing structured handling of exceptions reported by pip-accel the following diagram may help by visualizing the hierarchy:
-
exception
pip_accel.exceptions.
PipAcceleratorError
(text, **kw)¶ Base exception for all exception types explicitly raised by
pip_accel
.-
__init__
(text, **kw)¶ Initialize a
PipAcceleratorError
object.Accepts the same arguments as
compact()
.
-
-
exception
pip_accel.exceptions.
NothingToDoError
(text, **kw)¶ Custom exception raised on empty requirement sets.
Raised by
get_pip_requirement_set()
when pip doesn’t report an error but also doesn’t generate a requirement set (this happens when the user specifies an empty requirements file).
-
exception
pip_accel.exceptions.
EnvironmentMismatchError
(text, **kw)¶ Custom exception raised when a cross-environment action is attempted.
Raised by
validate_environment()
when it detects a mismatch betweensys.prefix
and$VIRTUAL_ENV
.
-
exception
pip_accel.exceptions.
UnknownDistributionFormat
(text, **kw)¶ Custom exception raised on unrecognized distribution archives.
Raised by
is_wheel
when it cannot discern whether a given unpacked distribution is a source distribution or a wheel distribution.
-
exception
pip_accel.exceptions.
BinaryDistributionError
(text, **kw)¶ Base class for exceptions related to the generation of binary distributions.
-
exception
pip_accel.exceptions.
InvalidSourceDistribution
(text, **kw)¶ Custom exception raised when a source distribution’s setup script is missing.
Raised by
build_binary_dist()
when the given directory doesn’t contain a Python source distribution.
-
exception
pip_accel.exceptions.
BuildFailed
(text, **kw)¶ Custom exception raised when a binary distribution build fails.
Raised by
build_binary_dist()
when a binary distribution build fails.
-
exception
pip_accel.exceptions.
NoBuildOutput
(text, **kw)¶ Custom exception raised when binary distribution builds don’t generate an archive.
Raised by
build_binary_dist()
when a binary distribution build fails to produce the expected binary distribution archive.
-
exception
pip_accel.exceptions.
CacheBackendError
(text, **kw)¶ Custom exception raised by cache backends when they fail in a controlled manner.
-
exception
pip_accel.exceptions.
CacheBackendDisabledError
(text, **kw)¶ Custom exception raised by cache backends when they require configuration.
-
exception
pip_accel.exceptions.
SystemDependencyError
(text, **kw)¶ Base class for exceptions related to missing system packages.
-
exception
pip_accel.exceptions.
DependencyInstallationRefused
(text, **kw)¶ Custom exception raised when installation of dependencies is refused.
Raised by
SystemPackageManager
when one or more known to be required system packages are missing and automatic installation of missing dependencies is disabled by the operator.
-
exception
pip_accel.exceptions.
DependencyInstallationFailed
(text, **kw)¶ Custom exception raised when installation of dependencies fails.
Raised by
SystemPackageManager
when the installation of missing system packages fails.
pip_accel.tests
¶
Test suite for the pip accelerator.
I’ve decided to include the test suite in the online documentation of the pip accelerator and I realize this may be somewhat unconventional... My reason for this is to enforce the same level of code quality (which obviously includes documentation) for the test suite that I require from myself and contributors for the other parts of the pip-accel project (and my other open source projects).
A second and more subtle reason is because of a tendency I’ve noticed in a lot of my projects: Useful “miscellaneous” functionality is born in test suites and eventually makes its way to the public API of the project in question. By writing documentation up front I’m saving my future self time. That may sound silly, but consider that writing documentation is a lot easier when you don’t have to do so retroactively.
-
pip_accel.tests.
setUpModule
()¶ Initialize verbose logging to the terminal.
-
pip_accel.tests.
tearDownModule
()¶ Cleanup any temporary directories created by
create_temporary_directory()
.
-
pip_accel.tests.
delete_read_only
(action, pathname, exc_info)¶ Force removal of read only files on Windows.
Based on http://stackoverflow.com/a/21263493/788200. Needed because of https://ci.appveyor.com/project/xolox/pip-accel/build/1.0.24.
-
pip_accel.tests.
create_temporary_directory
(**kw)¶ Create a temporary directory that will be cleaned up when the test suite ends.
Parameters: kw – Any keyword arguments are passed on to tempfile.mkdtemp()
.Returns: The pathname of a directory created using tempfile.mkdtemp()
(a string).
-
class
pip_accel.tests.
PipAccelTestCase
(methodName='runTest')¶ Container for the tests in the pip-accel test suite.
-
setUp
()¶ Reset logging verbosity before each test.
-
skipTest
(text, *args, **kw)¶ Enable backwards compatible “marking of tests to skip”.
By calling this method from a return statement in the test to be skipped the test can be marked as skipped when possible, without breaking the test suite when unittest.TestCase.skipTest() isn’t available.
-
initialize_pip_accel
(load_environment_variables=False, **overrides)¶ Construct an isolated pip accelerator instance.
The pip-accel instance will not load configuration files but it may load environment variables because that’s how FakeS3 is enabled on Travis CI (and in my local tests).
Parameters:
Test filename translation logic used by
pip_accel.req.Requirement.related_archives
.The
pip_accel.req.escape_name()
function generates regular expression patterns that match the given requirement name literally while treating dashes and underscores as equivalent. This test ensures that the generated regular expression patterns work as expected.
-
test_environment_validation
()¶ Test the validation of
sys.prefix
versus$VIRTUAL_ENV
.This tests the
validate_environment()
method.
-
test_config_object_handling
()¶ Test that configuration options can be overridden in the Python API.
-
test_config_file_handling
()¶ Test error handling during loading of configuration files.
This tests the
load_configuration_file()
method.
-
test_cleanup_of_broken_links
()¶ Verify that broken symbolic links in the source index are cleaned up.
This tests the
clean_source_index()
method.
-
test_empty_download_cache
()¶ Verify pip-accel’s “keeping pip off the internet” logic using an empty cache.
This test downloads, builds and installs pep8 1.6.2 to verify that pip-accel keeps pip off the internet when intended.
-
test_package_upgrade
()¶ Test installation of newer versions over older versions.
-
test_package_downgrade
()¶ Test installation of older versions over newer version (package downgrades).
-
test_s3_backend
()¶ Verify the successful usage of the S3 cache backend.
This test downloads, builds and installs pep8 1.6.2 several times to verify that the S3 cache backend works. It depends on FakeS3.
This test uses a temporary binary index which it wipes after a successful installation and then it installs the exact same package again to test the code path that gets a cached binary distribution archive from the S3 cache backend.
Warning
This test abuses FakeS3 in several ways to simulate the handling of error conditions (it’s not pretty but it is effective because it significantly increases the coverage of the S3 cache backend):
- First the FakeS3 root directory is made read only to force an error when uploading to S3. This is to test the automatic fall back to a read only S3 bucket.
- Then FakeS3 is terminated to force a failure in the S3 cache backend. This verifies that pip-accel handles the failure of an “optional” cache backend gracefully.
-
test_wheel_install
()¶ Test the installation of a package from a wheel distribution.
This test installs Paver 1.2.4 (a random package without dependencies that I noticed is available as a Python 2.x and Python 3.x compatible wheel archive on PyPI).
-
test_bdist_fallback
()¶ Verify that fall back from
bdist_dumb
tobdist
action works.This test verifies that pip-accel properly handles
setup.py
scripts that breakpython setup.py bdist_dumb
but supportpython setup.py bdist
as a fall back. This issue was originally reported based onPaver==1.2.3
in issue 37, so that’s the package used for this test.
-
test_installed_files_tracking
()¶ Verify that tracking of installed files works correctly.
This tests the
update_installed_files()
method.When pip installs a Python package it also creates a file called
installed-files.txt
that contains the pathnames of the files that were installed. This file enables pip to uninstall Python packages later on. Because pip-accel implements its own package installation it also creates theinstalled-files.txt
file, in order to enable the user to uninstall a package with pip even if the package was installed using pip-accel.
-
test_setuptools_injection
()¶ Test that
setup.py
scripts are always evaluated using setuptools.This test installs
docutils==0.12
as a sample package whosesetup.py
script uses distutils instead of setuptools. Because pip and pip-accel unconditionally evaluatesetup.py
scripts using setuptools instead of distutils the resulting installation should have an*.egg-info
metadata directory instead of a file (which is what this test verifies).
-
test_requirement_objects
()¶ Test the public properties of
pip_accel.req.Requirement
objects.This test confirms (amongst other things) that the logic which distinguishes transitive requirements from non-transitive (direct) requirements works correctly (and keeps working as expected :-).
-
test_editable_install
()¶ Test the installation of editable packages using
pip install --editable
.This test clones the git repository of the Python package pep8 and installs the package as an editable package.
We want to import the pep8 module to confirm that it was properly installed but we can’t do that in the process that’s running the test suite because it will fail with an import error. Python subprocesses however will import the pep8 module just fine.
This happens because
easy-install.pth
(used for editable packages) is loaded once during startup of the Python interpreter and never refreshed. There’s no public, documented way that I know of to refreshsys.path
(see issue 402 in the Gunicorn issue tracker for a related discussion).
-
test_setup_requires_caching
()¶ Test that
pip_accel.SetupRequiresPatch
works as expected.This test is a bit convoluted because I haven’t been able to find a simpler way to ensure that setup requirements can be re-used from the
.eggs
directory managed by pip-accel. A side effect inside the setup script seems to be required, but the setuptools sandbox forbids writing to files outside the build directory so an external command needs to be used ...
-
generate_package
(name, version, source_index, tracker_script, find_links=None, setup_requires=[])¶ Helper for
test_setup_requires_caching()
to generate temporary Python packages.
-
test_time_based_cache_invalidation
()¶ Test default cache invalidation logic (based on modification times).
When a source distribution archive is changed the cached binary distribution archive is invalidated and rebuilt. This test ensures that the default cache invalidation logic (based on modification times of files) works as expected.
-
test_checksum_based_cache_invalidation
()¶ Test alternate cache invalidation logic (based on checksums).
When a source distribution archive is changed the cached binary distribution archive is invalidated and rebuilt. This test ensures that the alternate cache invalidation logic (based on SHA1 checksums of files) works as expected.
-
check_cache_invalidation
(**overrides)¶ Test cache invalidation with the given option(s).
-
test_cli_install
()¶ Test the pip-accel command line interface by installing a trivial package.
This test provides some test coverage for the pip-accel command line interface, to make sure the command line interface works on all supported versions of Python.
-
test_cli_usage_message
()¶ Test the pip-accel command line usage message.
-
test_cli_as_module
()¶ Make sure
python -m pip_accel ...
works.
-
test_constraint_file_support
()¶ Test support for constraint files.
With the pip 7.x upgrade support for constraint files was added to pip. Due to the way this was implemented in pip the use of constraint files would break pip-accel as reported in issue 63. The issue was since fixed and this test makes sure constraint files remain supported.
-
test_empty_requirements_file
()¶ Test handling of empty requirements files.
Old versions of pip-accel would raise an internal exception when an empty requirements file was given. This was reported in issue 47 and it was pointed out that pip reports a warning but exits with return code zero. This test makes sure pip-accel now handles empty requirements files the same way pip does.
-
test_system_package_dependency_installation
()¶ Test the (automatic) installation of required system packages.
This test installs cffi 0.8.6 to confirm that the system packages required by cffi are automatically installed by pip-accel to make the build of cffi succeed.
Warning
This test forces the removal of the system package
libffi-dev
before it tries to install cffi, because without this nasty hack the test would only install required system packages on the first run, because on later runs the required system packages would already be installed. Because of this very non conventional behavior the test is skipped unless the environment variablePIP_ACCEL_TEST_AUTO_INSTALL=yes
is set (opt-in).
-
test_system_package_dependency_failures
()¶ Test that unsupported platforms are handled gracefully in system package dependency management.
-
-
pip_accel.tests.
wipe_directory
(pathname)¶ Delete and recreate a directory.
Parameters: pathname – The directory’s pathname (a string).
-
pip_accel.tests.
create_source_dist
(sources)¶ Create a source distribution archive from a Python package.
Parameters: sources – A dictionary containing a setup.py
script (a string).Returns: The pathname of the generated archive (a string).
-
pip_accel.tests.
uninstall_through_subprocess
(package_name)¶ Remove an installed Python package by running
pip
as a subprocess.Parameters: package_name – The name of the package (a string). This function is specifically for use in the pip-accel test suite to reliably uninstall a Python package installed in the current environment while avoiding issues caused by stale data in pip and the packages it uses internally. Doesn’t complain if the package isn’t installed to begin with.
-
pip_accel.tests.
find_installed_version
(package_name, encoding='UTF-8')¶ Find the version of an installed package (in a subprocess).
Parameters: package_name – The name of the package (a string). Returns: The package’s version (a string) or None
if the package can’t be found.This function is specifically for use in the pip-accel test suite to reliably determine the installed version of a Python package in the current environment while avoiding issues caused by stale data in pip and the packages it uses internally.
-
pip_accel.tests.
find_one_file
(directory, pattern)¶ Use
find_files()
to find a file and make sure a single file is matched.Parameters: - directory – The pathname of the directory to be searched (a string).
- pattern – The filename pattern to match (a string).
Returns: The matched pathname (a string).
Raises: AssertionError
when no file or more than one file is matched.
-
pip_accel.tests.
find_files
(directory, pattern)¶ Find files whose pathname contains the given substring.
Parameters: - directory – The pathname of the directory to be searched (a string).
- pattern – The filename pattern to match (a string).
Returns: A generator of pathnames (strings).
-
pip_accel.tests.
try_program
(program_name)¶ Test that a Python program (installed in the current environment) runs successfully.
This assumes that the program supports the
--help
option, because the program is executed with the--help
argument to verify that the program runs (--help
was chose because it implies a lack of side effects).Parameters: program_name – The base name of the program to test (a string). The absolute pathname will be calculated by combining sys.prefix
and this argument.Raises: AssertionError
when a test fails.
-
pip_accel.tests.
find_python_program
(program_name)¶ Get the absolute pathname of a Python program installed in the current environment.
Parameters: name – The base name of the program (a string). Returns: The absolute pathname of the program (a string).
-
pip_accel.tests.
generate_nonexisting_pathname
()¶ Generate a pathname that is expected not to exist.
Returns: A pathname (string) that doesn’t refer to an existing directory or file on the file system (assuming random.random()
does what it’s documented to do :-).
-
pip_accel.tests.
test_cli
(*arguments)¶ Test the pip-accel command line interface.
Runs pip-accel’s command line interface inside the current Python process by temporarily changing
sys.argv
, invoking thepip_accel.cli.main()
function and catchingSystemExit
.Parameters: arguments – The value that sys.argv
should be set to (a list of strings).Returns: The exit code of pip-accel
.
-
class
pip_accel.tests.
CaptureOutput
¶ Context manager that captures what’s written to
sys.stdout
.-
__init__
()¶ Initialize a string IO object to be used as
sys.stdout
.
-
__enter__
()¶ Start capturing what’s written to
sys.stdout
.
-
__exit__
(exc_type=None, exc_value=None, traceback=None)¶ Stop capturing what’s written to
sys.stdout
.
-
__str__
()¶ Get the text written to
sys.stdout
.
-
-
class
pip_accel.tests.
AptLock
¶ Cross-process locking for critical sections to enable parallel execution of the test suite.
-
class
pip_accel.tests.
FakeS3Server
(**options)¶ Subclass of
ExternalCommand
that manages a temporary FakeS3 server.-
__init__
(**options)¶ Initialize a
FakeS3Server
object.
-
root
= None¶ The pathname of the temporary directory used to store the files required to run the FakeS3 server (a string).
-