Building conda packages from scratch¶
Overview¶
In this tutorial we will build a conda package for Pyinstrument, but write the required files in the conda build recipe from scratch. We also explore the elements of a conda package in more detail.
Who is this for?¶
This tutorial is for Windows, Mac, and Linux users who wish to generate a conda package from scratch. Prior knowledge of conda-build and conda recipes is helpful.
You should have already completed Building conda packages with conda skeleton.
Building conda packages summary¶
This tutorial includes the following:
- Before you start
- What is a conda package?
- About conda build recipe
- Manually edit the file meta.yaml
- Write the build script files build.sh and bld.bat
- Build and install
- Convert package for use on all platforms
- Optional: How to use PyPI as the source instead of GitHub
- Optional: Upload new packages to Anaconda.org
- More resources
Before you start¶
You should already have installed Miniconda or Anaconda.
Install conda-build:
conda install conda-build
It is recommended that you use the latest versions of conda and conda-build. To upgrade both packages run:
conda upgrade conda
conda upgrade conda-build
Now you are ready to start building your own conda packages.
What is a conda package?¶
A conda package is a package that can be installed using the conda install [packagename]
command.
It includes a link to a tarball or bzipped tar archive (.tar.bz2) which contains metadata under the
info/
directory, and a collection of files which are installed directly into an install
prefix.
The format is identical across platforms and operating systems. During the install process,
files are extracted into the install prefix, except for files in the info/
directory. Installing the files of a conda package into an environment can be thought of as changing the
directory to an environment, then downloading and extracting the zip file and its dependencies – all with
the single conda install [packagename]
command.
About conda build recipe¶
Building a package requires a recipe. A conda build recipe is a flat directory which contains the following files:
meta.yaml
contains all the metadata in the recipe. Only package/name and package/version are required fields.build.sh
The Unix script that installs the files for the package on OS X and Linux, and is executed using the bash command.bld.bat
The build script that installs the files for the package on Windows, and is executed using cmd.run_test.py
Optional Python test file, a test script that will run automatically if it is part of the recipe.- Optional: patches that will be applied to the source.
- Other resources, which are not included in the source and cannot be generated by the build scripts.
NOTE: When you use the conda skeleton command, the first three files (meta.yaml
, build.sh
and bld.bat
)
are automatically generated for you. See the previous tutorial Building conda packages with conda skeleton for
more information.
Manually edit the file meta.yaml¶
Let’s start by making a new directory for this tutorial named pyinstrument
, and change to the new directory.
All users:
mkdir pyinstrument
cd pyinstrument
To create a new meta.yaml file, open your favorite editor. Create a new text file and insert the following information. A blank sample meta.yaml follows to make it easier to match up the information.
name | pyinstrument |
version | “0.13.1” (or latest from https://github.com/joerick/pyinstrument/releases) |
git_rev | v0.13.1 (or latest from https://github.com/joerick/pyinstrument/releases) |
git_url | https://github.com/joerick/pyinstrument.git |
imports | pyinstrument |
home | https://github.com/joerick/pyinstrument |
license | BSD |
license_file | LICENSE |
package:
name:
version:
source:
git_rev:
git_url:
requirements:
build:
- python
- setuptools
run:
- python
test:
imports:
-
about:
home:
license:
license_file:
When you are finished, save the file in the same pyinstrument directory as meta.yaml
. It should
match this meta.yaml file
.
NOTE: The version number must be specified as a string to allow correct sorting and comparison.
Write the build script files build.sh and bld.bat¶
The other two files you need for a build are
- build.sh shell script for Linux and OS X, and
- bld.bat batch file for Windows.
These two build files contain all the variables such as for 32-bit or 64-bit architecture (the ARCH
variable) and the build environment prefix (PREFIX). The two files build.sh
and bld.bat
files must be
in the same directory as your meta.yaml
file. In this tutorial you’re going to make both build.sh
and
bld.bat
so that other users can build the appropriate package for their architecture.
First we’ll write the build file for Windows and then the build file for Linux and OS X.
All users, in your favorite text editor, create a new file named bld.bat
and enter the text exactly as shown:
"%PYTHON%" setup.py install
if errorlevel 1 exit 1
NOTE: In bld.bat
, the best practice is to to add the if errorlevel 1 exit 1
after every command
so if the command fails, the build fails.
Save this new file bld.bat
to the same directory where you put your meta.yaml
.
All users, in your favorite text editor, create a new file named build.sh
and enter the text exactly as shown:
$PYTHON setup.py install # Python command to install the script.
Save your new build.sh file to the same directory where you put the meta.yaml file.
build.sh
is run with bash -x -e
(the -x
makes it echo each command that is run, and the -e
makes it exit whenever a command in the script returns nonzero exit status). You can revert this in the
script if you need to by using the set
command in build.sh.
Build and install¶
Now that you have your three new build files ready, you are ready to create your new package with
conda build
, and install the package on your local computer.
Run conda build:
conda build pyinstrument
When conda-build is finished, it displays the filename and the file’s location at the end of the build. In our case the file was saved to:
~/anaconda/conda-bld/linux-64/pyinstrument-0.13.1-py27_0.tar.bz2
NOTE: Save this path and file information for the next step. The exact path and filename will vary depending on your operating system, and whether you are using Anaconda or Miniconda. Conda-build tells you the exact location and filename.
Now install your newly built program on your local computer by using the use-local flag:
conda install --use-local pyinstrument
We know that Pyinstrument installed successfully if there are no error messages.
Convert package for use on all platforms¶
Now that you have built a package for your current platform with conda-build, you can convert it for use on
other platforms. This is why you made the two build files, build.sh
and bld.bat
.
Use the conda convert command with a platform specifier from the list {osx-64,linux-32,linux-64,win-32,win-64,all}. We will use the platform specifier all, as shown:
conda convert --platform all ~/anaconda/conda-bld/linux-64/pyinstrument-0.13.1-py27_0.tar.bz2 -o outputdir/
NOTE: change your path and filename to the exact path and filename you saved in Step 6.
Optional: How to use PyPI as the source instead of GitHub¶
What if you wanted to use PyPI or another repository instead of GitHub? There is little difference to conda-build
between building from Git versus building from a tarball on a repository like PyPI. Because the same source
is hosted on PyPI and GitHub, you can easily find a script on PyPI instead of GitHub. Simply replace this
source
section:
git_rev: v0.13.1
git_url: https://github.com/joerick/pyinstrument.git
With the following:
fn: pyinstrument-0.13.1.tar.gz
md5: e347036acc50720c0903dc2221b2605d
url: https://pypi.python.org/packages/source/p/pyinstrument/pyinstrument-0.13.1.tar.gz
NOTE: The md5 is found on the PyPI Pyinstrument page.
Optional: Upload new packages to Anaconda.org¶
After converting your files for use on other platforms, you may choose to upload your files to Anaconda.org, formerly known as binstar.org. It only takes a minute to do if you have a free Anaconda.org account.
If you haven’t already, open a free Anaconda.org account and record your new username and password.
Next, in your terminal window, run conda install anaconda-client
and enter your new Anaconda.org username and password.
Again in your terminal window, log into your Anaconda.org account with the command:
anaconda login
And upload your package to Anaconda.org:
anaconda upload ~/miniconda/conda-bld/linux-64/pyinstrument-0.12-py27_0.tar.bz
NOTE: Change your path and filename to the exact path and filename you saved in Step 6.
TIP: To save time, you can set conda to always automatically upload a successful build to Anaconda.org
with the command: conda config --set anaconda_upload yes
More resources¶
See more information about all the possible values that can go in the meta.yaml file on the Conda build recipe reference page.