======================================
Mirroring a PyPI repository (AER 2.31)
======================================

.. raw:: html

    <div class="section" id="before-you-begin">
    <h2>Before you begin<a class="headerlink" href="#before-you-begin" title="Permalink to this headline">¶</a></h2>
    <p>You need to have already completed <a class="reference external" href="install.html">installing and configuring your Anaconda Repository instance</a>.  Due to the size of Anaconda Repository, it is important that you have configured a file storage location with sufficient disk space.  If necessary please see the <a class="reference external" href="install.html#configure-the-server">instructions for setting the file storage location</a>.</p>
    <p>The full PyPI mirror requires approximately 120GB.</p>
    </div>
    <div class="section" id="the-mirror-command">
    <h2>The mirror command<a class="headerlink" href="#the-mirror-command" title="Permalink to this headline">¶</a></h2>
    <p>To create a PyPI mirror run:</p>
    <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">anaconda</span><span class="o">-</span><span class="n">server</span><span class="o">-</span><span class="n">sync</span><span class="o">-</span><span class="n">pypi</span>
    </pre></div>
    </div>
    <p>This will load all of the packages on pypi.python.org into the <cite>~pypi</cite> binstar user account.</p>
    <p>Verify that this is working by opening your browser to <code class="docutils literal"><span class="pre">http://your.anaconda.repository/pypi/~pypi</span></code></p>
    <p>NOTE: Replace your.anaconda.repository with the actual URL to your Anaconda Repository.</p>
    </div>
    <div class="section" id="customizing-the-mirror">
    <h2>Customizing the mirror<a class="headerlink" href="#customizing-the-mirror" title="Permalink to this headline">¶</a></h2>
    <p>It is possible to customize the mirror behavior by creating a configuration file such as <code class="docutils literal"><span class="pre">$PREFIX/etc/anaconda-server/mirror/pypi.yaml</span></code> and using the <code class="docutils literal"><span class="pre">--mirror-config</span></code> option:</p>
    <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">anaconda</span><span class="o">-</span><span class="n">server</span><span class="o">-</span><span class="n">sync</span><span class="o">-</span><span class="n">pypi</span> <span class="o">--</span><span class="n">mirror</span><span class="o">-</span><span class="n">config</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">binstar</span><span class="o">/</span><span class="n">mirrors</span><span class="o">/</span><span class="n">pypi</span><span class="o">.</span><span class="n">yaml</span>
    </pre></div>
    </div>
    <p>The following configuration options are available:</p>
    <table border="1" class="docutils">
    <colgroup>
    <col width="29%" />
    <col width="71%" />
    </colgroup>
    <thead valign="bottom">
    <tr class="row-odd"><th class="head">Name</th>
    <th class="head">Description</th>
    </tr>
    </thead>
    <tbody valign="top">
    <tr class="row-even"><td><code class="docutils literal"><span class="pre">user</span></code></td>
    <td>The local user under which the PyPI packages are
    imported. <strong>Default:</strong> <code class="docutils literal"><span class="pre">pypi</span></code></td>
    </tr>
    <tr class="row-odd"><td><code class="docutils literal"><span class="pre">whitelist</span></code></td>
    <td>A list of packages to mirror. Only packages listed are
    mirrored. If the list is empty, all packages are
    checked. <strong>Default:</strong> <code class="docutils literal"><span class="pre">[]</span></code></td>
    </tr>
    <tr class="row-even"><td><code class="docutils literal"><span class="pre">blacklist</span></code></td>
    <td>A list of packages to skip. The packages listed are
    ignored. <strong>Default:</strong> <code class="docutils literal"><span class="pre">[]</span></code></td>
    </tr>
    <tr class="row-odd"><td><code class="docutils literal"><span class="pre">latest_only</span></code></td>
    <td>Only download the latest versions of the packages.
    <strong>Default:</strong> <code class="docutils literal"><span class="pre">false</span></code>.</td>
    </tr>
    <tr class="row-even"><td><code class="docutils literal"><span class="pre">remote_url</span></code></td>
    <td>The URL of the PyPI mirror. <code class="docutils literal"><span class="pre">/pypi</span></code> is appended to
    build the XML RPC API URL, <code class="docutils literal"><span class="pre">/simple</span></code> for the simple
    index, and <code class="docutils literal"><span class="pre">/pypi/{package}/{version}/json</span></code> for the
    JSON API. <strong>Default:</strong> <code class="docutils literal"><span class="pre">https://pypi.python.org/</span></code></td>
    </tr>
    <tr class="row-odd"><td><code class="docutils literal"><span class="pre">xml_rpc_api_url</span></code></td>
    <td>A custom value for XML RPC URL. If this value is
    present, it takes precedence over the URL built using
    <code class="docutils literal"><span class="pre">remote_url</span></code>. <strong>Default:</strong> <code class="docutils literal"><span class="pre">null</span></code></td>
    </tr>
    <tr class="row-even"><td><code class="docutils literal"><span class="pre">simple_index_url</span></code></td>
    <td>A custom value for the simple index URL. If this value
    is present, it takes precedence over the URL built
    using <code class="docutils literal"><span class="pre">remote_url</span></code>. <strong>Default:</strong> <code class="docutils literal"><span class="pre">null</span></code></td>
    </tr>
    <tr class="row-odd"><td><code class="docutils literal"><span class="pre">use_xml_rpc</span></code></td>
    <td>Whether to use the XML RPC API as specified by
    <a class="reference external" href="https://www.python.org/dev/peps/pep-0381/">PEP381</a>.
    If this is set to <code class="docutils literal"><span class="pre">true</span></code>, the simple index is used
    to determine which packages to check. The simple index
    is also used if the XML RPC API fails for any reason.
    <strong>Default:</strong> <code class="docutils literal"><span class="pre">true</span></code></td>
    </tr>
    <tr class="row-even"><td><code class="docutils literal"><span class="pre">use_serial</span></code></td>
    <td>Whether to use the serial number provided by the XML
    RPC API. Only packages updated since the last serial
    saved will be checked. If this is set to false <strong>all
    PyPI packages are going to be checked for updates.</strong>
    <strong>Default:</strong> <code class="docutils literal"><span class="pre">true</span></code></td>
    </tr>
    <tr class="row-odd"><td><code class="docutils literal"><span class="pre">create_org</span></code></td>
    <td>Create the mirror user as an Organization instead of a
    regular user account. All superusers are added to the
    &#8220;Owners&#8221; group of the organization.
    <strong>Default:</strong> <code class="docutils literal"><span class="pre">false</span></code></td>
    </tr>
    </tbody>
    </table>
    <div class="section" id="example">
    <h3>Example:<a class="headerlink" href="#example" title="Permalink to this headline">¶</a></h3>
    <div class="highlight-yaml"><div class="highlight"><pre><span></span><span class="l l-Scalar l-Scalar-Plain">whitelist</span><span class="p p-Indicator">:</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">requests</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">six</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">numpy</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">simplejson</span>
    <span class="l l-Scalar l-Scalar-Plain">latest_only</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">true</span>
    <span class="l l-Scalar l-Scalar-Plain">remote_url</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">http://pypimirror.local/</span>
    <span class="l l-Scalar l-Scalar-Plain">use_xml_rpc</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">false</span>
    </pre></div>
    </div>
    </div>
    </div>
    <div class="section" id="configure-pip">
    <h2>Configure pip<a class="headerlink" href="#configure-pip" title="Permalink to this headline">¶</a></h2>
    <p>To configure pip to use this new mirror you must edit your <cite>~/.pip/pip.conf</cite> file:</p>
    <div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">[</span><span class="k">global</span><span class="p">]</span>
    <span class="n">index</span><span class="o">-</span><span class="n">url</span> <span class="o">=</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">your</span><span class="o">.</span><span class="n">anaconda</span><span class="o">.</span><span class="n">repo</span><span class="p">:</span><span class="o">&lt;</span><span class="n">port</span><span class="o">&gt;/</span><span class="n">pypi</span><span class="o">/~</span><span class="n">pypi</span><span class="o">/</span><span class="n">simple</span>
    </pre></div>
    </div>
    <p>NOTE: Replace <code class="docutils literal"><span class="pre">your.anaconda.repo</span></code> with the actual URL to your Anaconda Repository.</p>
    </div>
