Sign in

Python

This chapter serves as a guideline for writing and reviewing python code.

Setting up your environment

You are recommended to use pyenv with the pyenv-virtualenv plugin.

TLDR to get started

To install pyenv and pyenv-virtualenv on macOS, run:

brew install pyenv pyenv-virtualenv

To activate pyenv add this to your shell profile (~./bash_profile or similar):

eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

To install a python version and create a virtualenv for the repository, run:

pyenv install 3.6.5 # Use the correct version for the repository
pyenv virtualenv 3.6.5 my-repository-3.6.5

To automatically activate the correct virtualenv in the repository, run:

echo "my-repository-3.6.5" > ./my-respository/.python-version

Tools

These are our common tools to help us develop python applications.

Style Guide

PEP 20 -- The Zen of Python

Long time Pythoneer Tim Peters succinctly channels the BDFL's guiding principles for Python's design into 20 aphorisms, only 19 of which have been written down.

Source: PEP 20

PEP 20 -- The Zen of Python serves as a starting point for writing readable and maintainable pythonic code.

PEP 8 -- Style Guide for Python Code

PEP 8 -- Style Guide for Python Code is our style guide that dictates the details for how our python code should be written. To help govern these rules in our codebase we use a tool to lint our code. The default tool is flake8, but other tools are available.

Where PEP 8 allows for choice on how to write the code, our decisions will be documented here.

String Quotes

In Python, single-quoted strings and double-quoted strings are the same. This PEP does not make a recommendation for this. Pick a rule and stick to it. When a string contains single or double quote characters, however, use the other one to avoid backslashes in the string. It improves readability.

For triple-quoted strings, always use double quote characters to be consistent with the docstring convention in PEP 257.

Source: PEP 8

Our team prefers single-quoted strings.

Import formatting

Imports must be on separate lines.

Example
import os
import sys

Import Order

Imports are always put at the top of the file, just after any module comments and doc strings and before module globals and constants. Imports should be grouped with the order from most generic to least generic:

  • Standard library imports
  • Third-party imports
  • Application-specific imports

Each group is sorted using these rules:

  • import foo comes before from foo import bar
  • Imports are sorted lexicographically, ignoring case, according to each module's full package path.
Example
import logging
import sys
from collections import OrderedDict

import third_party
from third_party import bar
from third_party.bar import baz
from third_party.bar import Quux

import application
from application import bar
from application.bar import baz
from application.bar import Quux

Test Coverage

All Python code used in other modules must have full test coverage. Ideally all of the code should have full test coverage and all modules, classes, methods and functions should have their own unit tests.

Code intentionally left untested must be excluded from the tests using the test frameworks annotations.

Artifacts

Most of our Python applications generate 2 artifacts during build, a Python Wheel that install a command and a Docker image. The python wheel is installed in the Docker image and the command installed by the python wheel is set as the ENTRYPOINT of the Docker image.

Codebase setup

Here are the standard files needed to start a new python codebase.

.dockerignore

.git
build
dist

.editorconfig

This is our default EditorConfig setup.

root = true

[*]
charset = utf-8
end_of_line = lf
indent_size = 4
indent_style = space
insert_final_newline = false
max_line_length = 79
trim_trailing_whitespace = true

# YAML
[*.{yml,yaml}]
indent_size = 2

# JSON
[*.json]
indent_size = 2

[Makefile]
indent_style = tab

.gitignore

A good starting point for a .gitignore file can be generated by running:

curl -L -s https://www.gitignore.io/api/vim,linux,emacs,macos,python,windows,intellij+all > .gitignore

Then add this entry to the bottom of the .gitignore file.

# Pytest
.pytest_cache

# Pyenv
.python-version

README.md

# Application name

Application name generates new features in DDI.  Nullam fermentum massa
libero, ac venenatis velit egestas vitae. Vestibulum quis hendrerit urna,
nec maximus massa. Integer eleifend finibus sapien posuere faucibus. Proin
posuere, nisi vitae bibendum sollicitudin, ipsum dolor accumsan sem, id
eleifend mauris lectus vehicula leo. Cras id cursus enim. Nullam ut leo
consectetur, tempor metus ac, auctor tortor. Vivamus lobortis vulputate
risus vitae faucibus. Suspendisse feugiat libero sit amet tortor
condimentum auctor. Pellentesque erat lorem, elementum dignissim est at,
hendrerit gravida velit. Sed tristique id libero id imperdiet. Vestibulum
lacus ligula, faucibus sed scelerisque a, consequat ac urna. Curabitur nibh
tellus, faucibus quis est ut, suscipit auctor metus. Fusce volutpat
bibendum purus, nec mattis nibh ullamcorper sit amet. Quisque cursus
tincidunt lacus, ut ullamcorper arcu malesuada et. Morbi feugiat
sollicitudin mauris vel maximus. Ut rutrum, quam eget tempor interdum,
sapien diam dignissim nibh, ac semper tellus mauris ac mi.

## Prerequisite

* Configure your editor to use [EditorConfig][editorconfig]
* Python 3.6.x
* Virtual environment configured for this project

### Tips

Easy way on mac os to get a python virtual environment.

Install [pyenv][pyenv] and [pyenv-virtualenv][pyenv-virtualenv]

## Develop

Configure a python 3.6.x virtual envrironment and activate it then run:

```sh
make configure
```

Start the service with:

```sh
application-name -h
```

### Lint

```sh
make lint
```

### Test

Run all tests

```sh
make test
```

Run a single test suite

```sh
make test TESTS=tests/test_suite.py
```

### Build

#### Python Wheel

```sh
make build
```

#### Docker image

```sh
make docker
```

## References

* AWS SDK: [Boto 3][boto3]
* CLI library: [docopt][docopt]
* Test tool: [pytest][pytest], [pytest-cov][pytest-cov], [pytest-flask][pytest-flask]
* Code style guide tool: [flake8][flake8]

[boto3]: https://boto3.readthedocs.io
[docopt]: http://docopt.org
[editorconfig]: http://editorconfig.org
[flake8]: http://flake8.pycqa.org/
[pyenv-virtualenv]: https://github.com/pyenv/pyenv-virtualenv
[pyenv]: https://github.com/pyenv/pyenv
[pytest-cov]: https://pytest-cov.readthedocs.io
[pytest]: https://docs.pytest.org`

Dockerfile

This is an example Dockerfile that builds the Python application wheel and installs the application into a runtime image. The command installed by the Python wheel is set as the ENTRYPOINT. For more information see the Docker chapter.

FROM python:3.6-alpine as builder
ARG APP_VERSION

RUN apk add --update \
    make

# Add the code

WORKDIR /app
# Add files needed to install depencies
ADD Makefile README.md setup.py ./
# Install dependencies
RUN make configure APP_VERSION=${APP_VERSION}
# Add the code
ADD . .
# Build the codebase, will run tests etc
RUN make build APP_VERSION=${APP_VERSION}

FROM python:3.6-alpine
ARG APP_VERSION

# Copy the runtime artifact from the builder image
COPY --from=builder /app/dist /dist

# Install the runtime artifact
RUN pip install dist/application_name-${APP_VERSION}-py3-none-any.whl
RUN rm -rf dist

ENTRYPOINT [ "application_name" ]

Makefile

We drive our development workflow using a Makefile. Here is an example of the default targets we use for python code. This allows use to choose the tools used for build, linting, testing etc without changing how humans and CI interact with the development workflow. For more information see the Make chapter.

APP_VERSION = 0.0.1
COV_MIN = 100
IMAGE = docker-image-name
TAG = latest
TESTS = tests/
INTEGRATION_TESTS = integration_tests

build: clean configure lint test
	APP_VERSION=$(APP_VERSION) python3 setup.py bdist_wheel

clean:
	find . -name '__pycache__' -exec rm -rf {} +
	find . -name '*.pyc' -exec rm -f {} +
	find . -name '*.pyo' -exec rm -f {} +
	find . -name '*~' -exec rm -f {} +
	rm -rf build/
	rm -rf dist/
	rm -rf *.egg-info

configure:
	APP_VERSION=$(APP_VERSION) pip install -e ".[dev]"

lint:
	flake8

test:
	pytest \
		--cov=application_name \
		--cov-report html \
		--cov-report term-missing \
		--cov-fail-under $(COV_MIN) \
		$(TESTS)

integration-test:
	pytest $(INTEGRATION_TESTS)

docker:
	docker build \
		-t $(IMAGE):$(TAG) \
		--build-arg APP_VERSION=$(APP_VERSION) \
		-f Dockerfile .
	echo "$(IMAGE):$(TAG)"

.PHONY: build clean configure lint test integration-test docker

setup.cfg

[flake8]
exclude = .git,build,dist
max-line-length = 79
import-order-style = smarkets
application-import-names = application_name

setup.py

Setuptools is a collection of enhancements to the Python distutils that allow developers to more easily build and distribute Python packages, especially ones that have dependencies on other packages.

Source: Setuptools

This is an example of a baseline setup.py used by our projects.

from codecs import open
from os import getenv
from os import path

from setuptools import find_packages
from setuptools import setup

version = getenv('APP_VERSION')
here = path.abspath(path.dirname(__file__))

with open(path.join(here, 'README.md'), encoding='utf-8') as f:
    long_description = f.read()

setup(
    name='application_name',

    version=version,

    description='Short description',
    long_description=long_description,

    url='https://github.com/TeliaSoneraNorge/...',

    author='Telia Division X',

    license='CLOSED',

    classifiers=[
        'Development Status :: 3 - Alpha',
        'License :: OSI Approved :: Closed License',
        'Programming Language :: Python :: 3.6',
    ],

    keywords='',

    packages=find_packages(exclude=[
        'contrib',
        'docs',
        '*.tests',
        '*.tests.*',
        'tests.*',
        'tests',
        'integration_tests',
        'integration_tests.*'
        ]),

    # List of dependencies, with exact versions
    install_requires=[],

    # List additional groups of dependencies here (e.g. development
    # dependencies). You can install these using the following syntax,
    # for example:
    # $ pip install -e .[dev,test]
    extras_require={
        'dev': [
            'flake8',
            'flake8-import-order',
            'pytest',
            'pytest-cov',
            'wheel'
        ]
    },

    include_package_data=True,
    package_data={},

    entry_points={
        'console_scripts': [
            'application-name=application_name.__main__:main',
        ],
    },
)