The Python Language: Packaging

The official documentation has pages about building python packages1. Torborg also compiled a series of pages and examples about building a package2. In this note, I only provide some tips.

Private Python Packages

We can easily setup a private pypi service (e.g., pypicloud).

Install Packages from Private Pypi

To install packages inline, use

pip install -i https://$PYPI_USER:[email protected]/simple/ durst==0.0.5

To install packages from requirements.txt use,

pip install -r requirements.txt --trusted-host https://$PYPI_USER:[email protected]/simple --extra-index-url https://$PYPI_USER:[email protected]/simple

Publishing Packages Using GitHub Actions

We can easily publish python packages using GitHub Actions. Here is an example.

name: Publish Package to Private Pypi

on: [push]

jobs:
  build-n-publish:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.7
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install -r requirements-dev.txt          
      - name: Generate dist
        run: python setup.py sdist
      - name: Publish package
        if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
        uses: pypa/gh-action-pypi-publish@master
        with:
          user: ${{ secrets.private_pypi_username }}
          password: ${{ secrets.private_pypi_password }}
          repository_url: ${{ secrets.private_pypi_url }}

Test Python Packages Using GitHub Actions

Continuous testing packages before publishing is crucial for the stability of the packages.

name: Test Code with Pip

on: [push]

jobs:
  build:
    strategy:
      matrix:
        python-version: [3.6, 3.7, 3.8]
        os: [ubuntu-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install -r requirements-dev.txt
          # install black if available (Python 3.6 and above), and autopep8 for testing the pipe mode
          pip install black || true
          pip install autopep8 || true          
      - name: Lint with flake8
        run: |
          # stop the build if there are Python syntax errors or undefined names
          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
          # all Python files should follow PEP8
          # echo "flake8 all python"
          # flake8 durst tests
          # exit-zero treats all errors as warnings.  The GitHub editor is 127 chars wide
          echo "flake8 BLK"
          flake8 durst --count --exit-zero --max-complexity=18 --statistics --select BLK          
      - name: Install from source (required for the pre-commit tests)
        run: pip install .
      - name: Test with pytest
        run: pytest --cov=durst -W ignore::DeprecationWarning

Build Documentation Using GitHub Actions

The following configuration builds the documentation using sphinx and deploys the content to S3.

name: Test Build Docs

on:
  push:
    branches:
      - "**"
      - "!gh-pages"

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          submodules: true  # Fetch Hugo themes
          fetch-depth: 0    # Fetch all history for .GitInfo and .Lastmod

      - uses: ammaraskar/sphinx-action@master
        with:
          docs-folder: "docs/"

      - uses: actions/upload-artifact@v1
        with:
          name: DocumentationHTML
          path: docs/build/html/

      - name: Deploy to S3
        uses: jakejarvis/s3-sync-action@master
        with:
          args: --delete
        env:
          AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REGION: ${{ secrets.AWS_REGION }}
          SOURCE_DIR: 'public'

Building docs using mkdocs is similar. Here is a minimal example.

name: Publish Docs
on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: 3.x
      - run: pip install mkdocs
      - run: pip install mkdocs-material
      - run: pip install mkdocs-material-extensions
      - run: pip install mkdocs-autorefs
      - run: pip install mkdocstrings
      - run: pip install -e ".[all]"
      - run: mkdocs gh-deploy --force

Planted: by ;

wiki/programming-languages/python/packaging Links to:

L Ma (2021). 'The Python Language: Packaging', Datumorphism, 08 April. Available at: https://datumorphism.leima.is/wiki/programming-languages/python/packaging/.