{ "cells": [ { "cell_type": "markdown", "metadata": { "toc": true }, "source": [ "

Table of Contents

\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Automated Testing & Continuous Integration" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "
\n", " \n", "

Yesterday:\n", "

\n", "\n", "

In this lecture:\n", "

\n", " \n", "

Next Week:\n", "

\n", "

\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "

Contact details:

\n", "\n", "\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### By the end of this lecture you should:\n", " - Understand the basics of code testing and ways to automate it.\n", " - Understand the different sorts of test.\n", " - Have experience of Continuous Integration via [Travis-CI.com]()\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Automatic documentation\n", "\n", "Documentation is written for three broad groups of people:\n", "\n", "1. Users (\"I don't care how it does it, I care what it does.\")\n", "1. User-Developers (\"I build what I need.\")\n", "1. Developers (\"I make it, I don't use it\")\n", "\n", "These individuals have different needs from the documentation you provide to them. Users want to know what software can do, and how to run it for their own problems. Developers need to understand how code works, but not necessarily what it's intended for in a wider sense. User-developers need to understand both sides. Commercial software is generally written by pure developers, but a lot of scientific software (and many other small projects) is written by user-developers.\n", "\n", "For all groups, the most important piece of documentation in Python is the docstring, since it remains with the code it applies to, and connects to the Python online help. For developers & user-developers, additional comments in the body of the text may also be useful, however users may never look there.\n", "\n", "Although it is now generally acknowledged that the best place for living documentation is in the source code, near where it applies, since this gives the best probability of developers updating it in line with changes, it is still useful to maintain a full (electronic) manual, both for ease of reference, and to give a general project overview.\n", "\n", "Several tools have been developed to close this gap by automatically collating comments and calling signatures from the source code and converting it into a human readable \"manual\". Perhaps the most famous opensource cross-language tool is [Doxygen](). However we'll look further at Sphinx, which originated in the Python community and is the tool of choice on several large Python projects including SciPy, Django and [Python itself] (https://docs.python.org/3/about.html) \n", "\n", "### Sphinx\n", "\n", "You can install the core tool with the command\n", "\n", "```bash\n", "pip install sphinx\n", "```\n", "\n", "Sphinx works by converting [reStructured text](http://docutils.sourceforge.net/docs/user/rst/quickstart.html) into html pages or pdf files.\n", "\n", "The simplest use pattern is to use the fully automatic scanning tools to collect together all the docstrings into an index of APIs.\n", "\n", "This requires creating two files, the first containing the configuration options for the sphinx tool, and the second containing a skeleton reStructured text file into which to slot it:\n", "\n", "_`docs/conf.py`_:\n", "\n", "```python\n", "import sys\n", "import os\n", "\n", "sys.path.insert(0, os.path.abspath(os.sep.join('.','/mycoolproject')))\n", "\n", "project = 'MyCoolProject'\n", "extensions = ['sphinx.ext.autodoc']\n", "source_suffix = '.rst'\n", "master_doc = 'index'\n", "exclude_patterns = ['_build']\n", "autoclass_content = \"both\"\n", "```\n", "\n", "_`docs/index.rst`_:\n", "\n", "```rst\n", "mycoolproject\n", "=============\n", "\n", "This is just example text.\n", "\n", ".. automodule:: mycoolproject\n", " :members:\n", "```\n", "\n", "\n", "With this setup we can build a `html` version of the documentation with\n", "\n", "```bash\n", "sphinx-build docs docs/html\n", "```\n", "\n", "
\n", " \n", "

Warning

\n", "\n", "By Microsoft Windows uses `\\` as the separator symbol between levels in the directory tree. Meanwhile Linux and Mac OSX use `/`. This makes it almost impossible for notes to give paths that work on both sets of computers. In this section I use the \\*nix standard of `/` to match the sphinx documentation. Please remember to convert in your head on a Windows computer.\n", "\n", "
\n", "\n", "If this is successful, you should be able to open `./docs/html/index.html` to see documentation automatically generated from the docstrings in your project. Sphinx also supports other output formats (for example LaTeX) with the `-b` flag. A recipe to generate a `pdf` manual on a suitably configured Linux system is\n", "\n", "```bash\n", "sphinx-build -b latex docs .\n", "pdflatex MyCoolProject.tex\n", "pdflatex MyCoolProject.tex\n", "```\n", "\n", "which will generate `MyCoolProject.pdf`. We run LaTeX twice to ensure that references and citations (including the index) are set correctly. \n", "\n", "\n", "
\n", "\n", "

Exercise: Autodocumenting your module

\n", "\n", "\n", " \n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sandboxing & Virtual environments\n", "\n", "There are several situations in which it can be useful to \"[sandbox](https://docs.python.org/3.6/library/venv.html)\" code into its own space so that other package installations cannot interfere with it, and so that it cannot interfere with them.\n", "\n", " - Sometimes packages have mutually incompatible version dependencies, e.g. package A only works with version 1.1 of package B, while package C needs version 1.2\n", " - Sometimes packages have the same name, but do different things, creating a namespace clash.\n", " - Sometimes you need a clean environment to test your package dependencies to write your `requirements.txt` file.\n", " \n", "Python 3 comes with an [inbuilt library](https://docs.python.org/3.6/library/venv.html) called `venv` which can be used to create so-called \"virtual environments\". Inside a virtual environment, only the packages and tools you choose are available, at the version numbers you request.\n", "\n", "To create a new `venv` environment you can run a command like\n", "```bash\n", "python -m venv foo\n", "```\n", "or, on systems with both Python 2 and Python available,\n", "```bash\n", "python3 -m venv foo\n", "```\n", "\n", "This will create a new directory `./foo/` containing the files relevant to that virtual environment. To start the environment on Windows run\n", "```bash\n", "foo\\Scripts\\activate.bat\n", "```\n", "or on unix shell like systems\n", "```bash\n", "source foo/bin/activate\n", "```\n", "\n", "To turn it off again, run\n", "```bash\n", ".\\foo\\Scripts\\deactivate.bat\n", "```\n", "on Windows or just type\n", "```bash\n", "deactivate\n", "```\n", "on most other shells." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "

Exercise

\n", "

Create your own venv environment, giving it a name of your choice. Activate it. Note the difference it makes to your command prompt.

\n", "\n", "

Double check the linstalled package list using pip list. Install a package into the virtual environment (such as matplotlib) using pip. Check that the list of installed packages inside the environment changes.

\n", "\n", "

Install the package you wrote into your virtual environment.

\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Code Testing\n", "\n", "When writing software, especially scientific software, a key question is whether the code is correct, and provably correct. How do we check something for correctness? We test.\n", "\n", "### What is a test?\n", "\n", "The simplest kind of test is the ad hoc kind you run when hacking about with code. If I have created a new function `add(x, y)` which adds together two numbers and returns the result, then it might be a good idea to try something like\n", "```python\n", "print(add(1, 0) == 1)\n", "print(add(1, 1) == 2)\n", "```\n", "This can't catch every possible mode of failure. For example, an `add` defined function like\n", "```python\n", "def add(x, y):\n", " return x**2+y**2\n", "```\n", "would pass both the tests given above, despite not being returning `x+y`.\n", "\n", "The examples above are both examples of \"testing to pass\", in that we have written a statement we expect to return true, and will be happy if it does so. One important thing to check is that operations you expect to raise an exception error actually do so (i.e. testing to fail). There are usually ways of writing a test to fail as a test to pass if needed. E.g. for a `divide` operator:\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "def divide(x, y):\n", " \"\"\"return x divided by y.\"\"\"\n", " return x/y\n", "\n", "\n", "## Check we get an error when dividing by zero.\n", "try:\n", " divide(1, 0)\n", " print(False)\n", "except ZeroDivisionError:\n", " print(True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By writing our tests like this we can use an operator like `assert()` to turn the test from raising an exception if when it does what we want into raising an exception if it doesn't." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Test Driven Development\n", "\n", "The most extreme version of this philosopy is strict Test Driven Development (TDD), sometimes also called \"Red-Green\" testing. Here the idea is that when implementing new code *first* you write a test, ensuring that it fails (red), then *second* you write just enough code to pass the test you've writen (green). Finally you [refactor]() (i.e. rewrite) any code you believe can be improved (simplified or speeded up), _while ensuring that your existing tests pass or are fixed_. \n", "\n", "This is intended to be an iterative process, so once you are done with one test, you move on to the next. Let's have an example, pushing things as far as they can go:\n", "\n", "__Problem__: Write a function that returns the repeated elements in a list.\n", "\n", "So, first, we write a test.\n", "\n", "```python\n", "assert(f([0, 1, 1]) == 1)\n", "```\n", "\n", "Does it fail on a do nothing implementation?" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "False\n" ] } ], "source": [ "def f(x):\n", " \"\"\"Return the repeated elements in a list.\"\"\"\n", " pass\n", "\n", "\n", "print(f([0, 1, 1]) == 1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok, our test is failing, let's actually write some code. A quick way to catch repeated elements is to use a set." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "def f(x):\n", " \"\"\"Return the repeated elements in a list.\"\"\"\n", " vals = set()\n", " for _ in x:\n", " if _ in vals:\n", " return _\n", " else:\n", " vals.add(_)\n", "\n", "\n", "print(f([0, 1, 1]) == 1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our test is now passing. There's no old code to improve, so we either write another test, or move on. Let's try having a couple of repeated elements\n", "```python\n", "f([0,1,1,0]) == [0,1]\n", "```" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n", "False\n" ] } ], "source": [ "\n", "print(f([0, 1, 1]) == 1)\n", "print(f([0,1,1,0]) == [0,1])" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "False\n", "True\n" ] } ], "source": [ "def f(x):\n", " \"\"\"Return the repeated elements in a list.\"\"\"\n", " vals = set()\n", " out = []\n", " for _ in x:\n", " if _ in vals:\n", " out.append(_)\n", " else:\n", " vals.add(_)\n", " return sorted(out)\n", "\n", "\n", "print(f([0, 1, 1]) == 1)\n", "print(f([0, 1, 1, 0]) == [0, 1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We've broken the first test. But it's actually the test which is bad, so we'll fix it." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n", "True\n" ] } ], "source": [ "print(f([0, 1, 1]) == [1])\n", "print(f([0, 1, 1, 0]) == [0,1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok. Another iteration. What about if an element turns up 3 times?" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "False\n", "[0, 0, 1]\n" ] } ], "source": [ "print(f([0, 1, 1, 0, 0]) == [0, 1])\n", "print(f([0, 1, 1, 0, 0]))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n", "True\n", "True\n" ] } ], "source": [ "def f(x):\n", " \"\"\"Return the set of repeated elements in a list.\"\"\"\n", " vals = set()\n", " out = set()\n", " for _ in x:\n", " if _ in vals:\n", " out.add(_)\n", " else:\n", " vals.add(_)\n", " return out\n", "\n", "\n", "print(f([0, 1, 1]) == set([1]))\n", "print(f([0, 1, 1, 0]) == set([0, 1]))\n", "print(f([0, 1, 1, 0, 0]) == set([0, 1]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And so on. At some point you run out of tests to set and the code is finished." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "

Exercise

\n", "\n", "

Try and write your own attempt at TTD for the following problems:\n", "\n", "

    \n", "

  1. Write an implementation of a [fizz buzz](https://en.wikipedia.org/wiki/Fizz_buzz) function. This is a children's game, where players count around a circle in order. \n", "

      \n", "
    • However, any number divisible by 3 is replaced by the word \"fizz\".
      • \n", "
    • Any number divisible by 5 is replaced by the word \"buzz\".
      • \n", "
    • Numbers divisible by both become \"fizz buzz\".
      • \n", "

    \n", "\n", "

    Your function should accept an integer, x, and return either x, 'fizz', 'buzz' or 'fizz buzz' according to the rules above.

    2. \n", " \n", "
  2. Write a program to put the elements of an $n$ dimensional numpy array, $X$, into order of size.\n", "
      \n", "
    • The code should have output $Y$ with Y[i,...]<[j,...] for all i<j.
      • \n", "
    • The code should have output $Y$ with Y[k,i..]<[k,j,..] for all i<j and fixed k
      • \n", "
    • And so on.
      • \n", "
    • Note that this means when written in the form\n", "

      [[..[x_1, .., x_n], [x_n+1, .., x_2n], .., x_N]]

      \n", "we have x_n< x_n+i for all i>0.
      • \n", "
    3. \n", "\n", "

  3. Write a function to accept or reject a string as a candidate for a password based on the following criteria:\n", "

      \n", "
    • The string is between 8 and 1024 characters long.
      • \n", "
    • The string contains a number
      • \n", "
    • The string contains a capital letter
      • \n", "
    • The string contains none of the following characters: @<>!
      • \n", "
    \n", "
\n", "\n", "

Remember, you should come up with a test first, then write only enough new code to satisfy it, and fix your previous tests, before moving on to another test. Once the test passes, remember to have a look at your code to see if anything needs to be refactored.

\n", "\n", "

The goal here is to concentrate on the TDD process, but for completeness, [model answers](https://msc-acse.github/ACSE-1/lectures/lecture10-solutions.html) are available.

\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At this point you can hopefully see that \n", "1. TDD is a fine ideal for correct software engineering\n", "2. No sane person would write all their tests an that level of detail.\n", "\n", "However, there are some important ideas to pull out of the paradigm:\n", "\n", "1. Make sure your tests can fail.\n", "2. Consider both the usual and the corner cases." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ways to implement tests" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lets have a slighly more concrete example. We'll look for solutions to the quadratic equation\n", "$$ ax^2 + bx + c =0,$$\n", "where $x$ is a real number. The formula for solutions is\n", "$$ x=\\frac{-b \\pm \\sqrt{b^2-4ac}}{2a}.$$\n", "Now to write some code in a module file" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "import numpy as np # for the sqrt function\n", "\n", "def solve_quadratic(a,b,c):\n", " \"\"\"Solve a quadratic equation ax^2+bx+c=0 in the reals\"\"\"\n", " if 0>> mean([1, 5, 9)\n", " 5\n", " \n", " \"\"\"\n", " return sum(x)/len(x)\n", "\n", "if __name__ == \"__main__\":\n", " import doctest\n", " doctest.testmod()\n", "\n", "```\n", "\n", "Run the test by calling the module as a script:\n", "```bash\n", "python3 -m docstring_test\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the test suceeds it silently returns a successful exit code. If the test fails (e.g. we replace the out put of 5.0 in the example) then an error message is printed, looking like the following:\n", "\n", "```bash\n", "**********************************************************************\n", "File \"docstring_test.py\", line 6, in __main__.mean\n", "Failed example:\n", " mean([1, 5, 9])\n", "Expected:\n", " 3.0\n", "Got:\n", " 5.0\n", "**********************************************************************\n", "1 items had failures:\n", " 1 of 1 in __main__.mean\n", "***Test Failed*** 1 failures.\n", "```\n", "\n", "Doctests can also be run from plain text files as\n", "\n", "```bash\n", "import doctest\n", "doctest.testfile(\"example.txt\")\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "

Exercise:

\n", "\n", "

Write some tests using the doctest module inside your own module or script.

\n", "\n", "
    \n", "
  1. First write some tests which pass.
    2. \n", "
  2. Next write some tests which do not pass.
    3. \n", "
  3. Swap your work with another student (why not use github?).
    4. \n", "
  4. Fix their failing tests, either by editing the code or changing the tests.
    5. \n", "
\n", "\n", "

You can use some of the modules you wrote earlier in the week, or else write some new code.

\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### The `unittest` module\n", "\n", "It's still useful to automate things a bit more, so that we can . Python provides an inbuilt `unittest` module, which (with some work) can be used to build a test framework. It introduces the basic concept of the three stages of a test:\n", "1. Set up. We create anything which must already exist for a test to make sense.\n", "2. Running. The test itself operates\n", "3. Tear down. We clean up anything which won't get dealt with automatically" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "The file `unittest_example.py` contains the following code\n", "\n", "```python\n", "import unittest\n", "\n", "import numpy as np # for the sqrt function\n", "\n", "def solve_quadratic(a,b,c):\n", " \"\"\"Solve a quadratic equation ax^2+bx+c=0 in the reals\"\"\"\n", " if 0\n", "\n", "

Exercise:

\n", "\n", "Try this yourself by writing a unittest using the `unittest` module. Try breaking and fixing the test.\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Pytest\n", "\n", "The external `pytest` functions simplifies the actions of writing tests even further, as well as providing a more informative interface. Pytest can be installed with\n", "```bash\n", "pip install pytest\n", "```\n", "which adds a tool of the same name. This tool can be used to run both `doctest`s, (add the `--doctest-modules`) and unit tests based on the unit test module (just leave out the `unittest.main()`), as well as tests in its own format.\n", "\n", "We have created a [GitHub repository](https://github.com/jrper/CI) in which the file `pytest_example.py` contains the following code\n", "\n", "```python\n", "\n", "def test_solve_quadratic():\n", " \n", " assert(foo.solve_quadratic(1,0,-1)==(-1.0,1.0))\n", " assert(foo.solve_quadratic(1,0,0)==0.0)\n", " assert(foo.solve_quadratic(1,0,1) is None)\n", " \n", "```\n", "\n", "while the `foo.py` module contains our `solve_quadratic` function. We can run the pytest tests as well as the others using the following syntax:\n", "\n", "```bash\n", "py.test --doctest-modules pytest_example.py docstring_test.py unittest_example.py\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "

Exercise:

\n", "\n", "
    \n", "
  • Clone the repository using git, install pytest and run the tests.
    • \n", "
  • Try breaking some of the tests by editting the .py files.
    • \n", "
  • Write your own pytest test for some of your code.
    • \n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The many classes of test\n", "\n", "1. Unit tests: Small tests which confirm the behaviour of a single module or function. Inputs are \"mocked\" up to generate known outputs. \n", "1. Integration tests: Tests which combine multiple functions together.\n", "1. Feature/Functionality tests: Tests which confirm an entire feature is working successfully. For numerical codes, this often involves analytic solutions.\n", "1. Regression tests: Tests which check that fixed bugs stay fixed." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Continuous integration\n", "\n", "Once you have tests, you want to make sure that any public changes to the code don't break them. For code under version control, this means that you want to rerun your test suite on all new public merges, to ensure that the code installs and runs successfully on a fresh system. Continous integration (CI) frameworks tie together version control repositories and test suites so that this happens automatically whenever a commit to production code (i.e. master) is proposed, or even whenever any change is commited.\n", "\n", "The goal is to ensure that the production branch always works, so that pure users can always securely use it. Meanwhile, developers can feel secure in repeatedly merging from the master version and knowing whether their code works or not. At its furthest, under the doctrine of Extreme programming, work can merged multiple times a day among moderate ly sized teams." ] }, { "attachments": { "Unknown-2.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAB3CAMAAAA3iQuPAAAA/FBMVEX////08K/NMkrGs5rmzK1lbGf387EtMTbQMkru3J769rPJtpzx7a39+bX//7sAHCqBgGWHhGvWMUsoLTOIMkCEO0OzNkgAAAAzLDShnn7Wx5AAACE/PDpdYVEgKzLV0psjJzHw8PBeZGGpmofGM0o3O0Dk4KUYIC2fnnjXwaRSVVnZ2dqNjG9dWFN6eWgAAA+Xinuvo3keMTVhY2a4p5EAEyl6fH5ER0ttb3IAMDO1trcAABZSVEq1soZtZl8YHiVtbVrKy8wAEB1bMTqcM0SNj5GjpKVtNT+IfXBKMzpQTErDwJF6cmlXOD0NIidANTkAACozOEYWIDbHtIbiLhGiAAAV2ElEQVR4nO2dC3vaOLD3l0AtwiUNiNCSgC0CsrmHEMCEAAZzNySctt//u5wZ2dzS5tJu9/Tt++i/z2YTY/wsP41Go9FI/PPPrylbHnWnnbuSUHvaHZWzv/gkqZ9Stjy749bNjW0ZFopZ9s0nm5SmI9kA/62yoyn/dGNRStK961oMVRs4DqHUsj/RTlfy/6+ULU+5/WBwZ1jpa75oNOQqGo1q/VStl6a2TdvS/v8LZUdt8sB4b9jXIwBe8R0IGsCnrSo9h97QjsT/u5UddewH2qv1I2dR5Qj8rgFC0Qjw5/anO4n/twodDslUtEj0h+A9KSFFKwz5gyWdz+9Ttlt6oJmCFgq9Qt7DH9ULQ3LDZ+U//T/9/4lGbct2KpoC6BU98rLpK7oOLypRPdWzDel7fofA7G1SW0WE1Uf7mvKiQtoqIry/T6vwB3Mm6f9bZafU7hV0N7pRIjVdibwkXyS1cvtFKNIfGLQtXc+/U7ltG8PV1tdE+zHf2flLWkVThYjngcD4vxqd0Z/+3/+rNbqzWEXfxpaKb3irnF21Pv5QD4Wzfk1XdgNAIW2UJP1fV/nOaKUi2hZoSCOacsboPPG9Lsatwpk26G9H34im90271P3TH+Gv1ejO+NJXtP42wjxLfdF9ZwZNBE++U/6iVYieDiu7aEjr+1Y9m0v6v6byndVa+UIVfed1MsNThD+JB+NC0Ajub8HgSQLgR86d3c16X4sCfVN6nl9RuWO0Vkqon4ruvI6Z8uCfJIWvyQeD4pdJMigsP5L6uvNRyqoQCq3Stiljnl9Q2/gCkWN0Z8u+0IrfevDz8wYhdJMMxi8IKhHPC8v/RlKR3YibWoUUrWc7Mt7/ac1Iq4DAa1vD9/li9NsePqVqWsCnKgX4wu2cXvZqO6cf6ldgutvnD50//VH+OnX5VQw4RgfaPqEz5JcC/nySeDRSWuYzwlfTSkYdTxJjhH/aq+q7bINWgfdG+/Rm+qc/zF+mcukBbTikfT3bp256PRd+4/5eNSo7+LrD8MoVwh8ae6cfqUCcpCgx9kkOuj+jbPvBQZOPpoY7+NFVOnN5GjkzGOoQflVcsgvK6Wnta3+XeQO/A+OFEuoZ/E9/nr9KXcYFxDO29zrRPhlenkKg72kP37sQjZyeVq4O4K/OsRtENX7T/tMf6C9SuWSI2ZKik0P4TMDfBZ47+Fvcp6enKaOyg6/otRW+O1poUel43qvs7GYooEf71W3kiAyNGOB9Hf4tGezeoPjOC/iSctazS3/6M/01GnHiphSiqfNdoAnjZyv1DviOb69KTLwW0qg1+9Mf6i9Rtm172UllWDmAH3sXfH4Avz90Y59o5aEkJ7rv0shI9721E17Yw9drVwg/8jp8/mXPHmZo7pCh6KYhg/33KNuxzyMefGO1HW+VqAdffxF+RMAnSmib/w9pQ29pS+lfyRzPezSynVXUM/YHz/2ElIi2GrbeAz+taXrErXOAW7ylLSXiMOn136E2Gr4CvMFtPERgihqK6qtCZZjptSqnO6evRHfwt8U8An6a13rnhZWORW0CviJ6gpK6upOm/6bKJL06C+naahWJ9h+UM13rV4YmY5RREWqeuqR1fdVz4XPNqydB+ClOGPtipIeVgqZHdYAfWa00HdrPIXJd5U1N7Yy2SsVSfc13Vrha3dYypEUyt7c9So3h6XbEjRQyGS5SypRkMsJNRXR4MUYZZ6nbYdpgvVoh1Sv4FG3Vr8HTUq22zC2/JaLyXgb8RiSkr4YGN1pm7Pby8vRbrxfj7NzzO4oeu1LvGwj/vsGMfnRr+GmjUjW+nV5efqtkjK+EDfowIuhaYejAREtOc19XuX3Dhn0NnX6kkmbmMHUJ5EHfesNv35zW7RZ+5eNjYuKuZG3ULfy+Q1KnA4CPury8HfYYqYln6asat2UV22vqdr60aivdp4i8TobfCvC6LxKJAPxLzNy4po/w53lv6dyFLwyfVS5P0fKxhkrw/+ZwkSRSFH1VYawjy9heULdEH2oa5mTcOLOX2efRFC3Tuzy9ped7+Bd5UcQQP4BfMSAYBfju+BvBWdeQeQl++KnFSEtWEf5I5Q69cla+XfG9ovXOd7kEoD0E+JfpjPAoEfD5YPmeEL6C7E9jBrglh13uF78iNaO/f4hPqxmykup7dc3W/6QOS5CjfRLTD+Cfp2Ec7VExAAjOxBMEl7futcthGl7m6dP9YyKxr3v4ooiTPFgy03CsbMeuakc7HxB+5AB+jADi2NW3Lfybq708+H0THNUpc6IHtFNGwXeoqF6jtvQ8RyrftVbHWx+iBaNyAD9SQPu+dUN9N5jZy7t0axTwx+A1+L7oitvS8Rypy8nqGaTUIXxfaJWOAeS0cUD+G2qP/3L4BfpFpZWKHj3l9vi5iu7cyLnukWbM0Y4hgc2mvCgRIxe97zi3qfM0TTvpNFcpiBlC8JvK0iCT8VrlW+0KJmg+8T5cwIXoM3L84LPejcyxHWkGLv94w4+SEuu4EV3X+6nzocOZcXXVYoCcMfq9GFyn7i2GOTyv9DVdh2ATnNftd/Af5Hr6kWYPvWfwT2/5ub7qp5yvV1dfDcrTPVRmCHK3nj9TDV/JZDK9nsOJAa3w1YkVVnoFok8J/3XNHo4sX1FOLyvEHPRMJzOsVFKpfn+lo17cE+RK3LPq91OFSuV82HN6Q+dr7FI/2jIt4T/X7ME5hI955PPzSmqFKyNnZ9Eopuf38nk74FDeFd/Bxji8/Syq6Jq2gscMK7eaHpHwX9bM5gfRDhrvSvd5bH3e8Kl4Jy1AQ+ysfNsZoq72bSHeE8IVLfRd/YPpmq9nS/hHmlkM4QMuBBfR8HiFHcBQyEUNlgytAgKnUqnEaq7w94K4vMJ1E9EePnEqhtcGoWhEEzt0xeKiojkS/rFmtLVScOFD8wx9a6gIfNUvpCqx8/NML83ZVxiAr1pfWy3DUwuFlxjnTq92fh6DtsB22Dsbty/o2krHtcn0gww1j9TlV32xoWGAw6sntOlhptozIbBXnz5+NDC8dxM6DVeEbMheVNz1ZKiEp51eNVOLVfZPq1RqMdwkjRX7Ev6RRqUWluOH9H7FMbmLkhkfWx+fGPu8WS6Xj+P5+HG89NBv5hOhxFzA37bJZj4fj8ePj3D/58/MnYQxr2XMWkETk7YCt+QM90jZjj1An3B2pqeGnCJOtTG+EBuuJpNkMplPJuaPS4G6QeaTfDweT04uxsvPCN40S6VrkzQuMMOcFIJ3jUWrCBHKh6mV7wxipkiNWrKO4UjZNuMarjbVhj3Ts+4JojzBDYfB5GQ+dskj+mQeriTmyw34GY7cr68/fLjmjXEyuFfyYgPIS6WSyTkH/jBJG8bA+PWMYf7pT/v/mmbE6CuRNGYPqIf4JCi2egbzwBmcPHDmhDaWgD4/gSuqimyRO+qaq8vJbn9u/CTxSKBH4IvQNNfQBAQTEGSor0wZ7DzXyLRrkShnpHRd4mD2CbFIGETOYPHI+RodC3iWePJiub3yYatrU20kduiDyTHcYO5fdpvAJDCwpIjcJfRc2Y71VY/WDXJdArsHsxfLs3mwcArBi/ArwH6Tz08eP7vkPxyihWFiyz6eTy4/4x0fnuuaG319aJhyLeW5ZuQqdbZqETB7gmvjwXh8gvarepyB7/1FPvHYwMjFJb//adLG3Osq8ckjaVBy3DgefNbT+ml7KuE/V7nEjOEKHHVjk4wHwd8kNujnuQexxOnm4mLTEKu2JnmeURbsg/GT5AW4JkD/PXl8BuutOf0kY53vNbVwRCSNZX6Lngijx/GSY+i/2bigVcuy3Mkt7kqEKRVofhJH8kskj82ThjDnue1fm5Qy9UZWDf5I3RJFd3+C8c3mnqKFY5TIcPHEm3ZRjCyr6/W6LpTDH2tosbkYmO/vVWg/CwJQhv/C3Zyb+1EZHBe1qKzb+bFmAPgC0E8E+u38CNrAEX+Sai5cLBbDx0L2y3n6/r6hEmedCzeb9cViPShhXOoucTGYhIkRmxB54O9LmjGVTMCEHxuqW5CTdqqDRb3ZzFVVGAucuj/sP1bYn6taBCfDFG5e55r7NmnC2+rrQdVJc9GO0AdMS3qclzRT1c0kmJzfg68n3AHPknM5+gfgd5yFPxw+Bu/359Zg1qKR1nUB3u9vNrevCxX92AbcSwDJhNoLmhGcvEKwgsOsyesuS1TOYOl1M7znKgRQFybOhksA3u/eHPYvHHOQO2wk0QJVilM3KgsFX1DXZMvJ5PGecH6HgWV9Ry/HabUODmbtVBc7/M21A0MxFXEndJLt5QVllsFzz9xT0SQYaFJZIv5jdU1rmZg3nsz2HRFj42LLvlklYPbhumlAgLne8lx/5JjKxLyZCa1QD7vtlObtNlG/PBscigYREzFmylN+f6ByxyKPj0+kM52ZtIRz0S3m8KIKAy00AePtqUldow43HX4QxZfowKW9tqbZrEmNxbPRocUxyuemZcmDTr9TuW2BA3kqtacd2yImkKIDz8OE0ezBnzDrrvxP2xp4wwAzD2dPJA1DAnYSW8CnzpHph3M2PpJXc2tiyzD/mbIziCRVqzOaEpsMiEuqGd4ZLmCFgOeujEl/t00WDLsHuBxwPehRCPqdcM5h5rSNkc3xmLuw8G4yKBbXDx0J/1hdTollzrIjYq1zfo5GbTqH/MI5kxJw5yZVc2K4reLAUCJmFYZd9DtkLe5yYAqLfYjWj+CvBXy6CATqttyNe6xyyaLWXfeff6aMFANFkyP89DH8NAQ2lkWJIS7nMFd/baKBD1yrxo6C8L2V9PUR/IGBTUXrEv53yrYpU6fIZGqZgQAE5YDK5MfwuYfVhb8wPD8SCDSpuW2rQ/jFQ/pVJvoJwpdu51hdyzLdlIuAHxhQF9XRkGkewRftA6YM8AOOG8MLWx94pQp0UG/u6TdFXyqlc4HA4kEm8w+VLd1sZ54zSoHm2rXTxQH7HXx3wM0RLjpHM4C3C7/DBjAwh+vcMAxMN3NzsB+wmwQjTe6EA8WBXEk5Uvtm54a79Apo1oVP4et9uAixjicuopq64QZEfoSf+yj+cAfo3GK9XqzrzeZBKiKcowifVosBf1WeOHWo0c3dzhbL5AqM2X/lBfo7+PV1zpN7TTima2OBXicQENPXEj+YWIX3eSHsNQ7bjhBNR9ZLHShLDtK8WdOGQTHcMgWr5uGIewSWCR9OcoJ9wFTFALDeJvr9hwr7F1zcDl0JegknMre21/SohuaDDYSKqimSAfVnHAV48CfrKjvwOkDUnb9C0D9YQN9ohov7NsC0BBULum6wY8mj1vYaHe9HnlocAn0xyypRMYYee5L62iFYfYleB0zdhe91BMoYLu1S7vQG64W7rlKsc8tNApUYdBSY4MrllL2e2WHXYk0I9AWva97axvpo8fVFlbZsi5pe2UhJ9bxOINCz3NoSN9/AYTJm2fZVC+e/hmv219zCQLPp2HK8fVFZy6gHigvLLfMzqYpZg2YuN3A4s3DxfLcafm2mm1v4C/tZpQiWB3KOa7iu2V9TFeLMQLGucunyX5bJuJjjuuVmJW5UFwNHNUTVwkFt4PW1ybZeB+5v8evr72ukoCe4F2F2zOoB6D1rmVx4TTOb+WEutOZ0Z/wMuN8dgi9hyRRxdl4nEKiKohKxuoIlmcftcE1oldIFLqg7cor1msoWhXg9HF4Qr8y1VKoeeROc6eJ6eS5c3MP31weDquOk0dVgbTIucHmNcF2ifF2HpnGqVYdSGeW/ppLl4GQKTP+43BINnmOhQrUK8aT/gLygD5EMrqjXF+tB1asWwdVds8SpUw+vVcrE96XLdaxX1b1hHNhZDMIa5hr/tfA0jCL3eq7pL4IC4Vx4hz5cNxeBItD3F0V4j43gtoLJOczVwo7VaXdAUzncvqpsm2ON390ddxYDghUfYLxo8KJ6KowVa4Eikq5W6038y9+sD5iZyy0A9rqJBW3+fdUUNAPclKNWOVsul7PS4b+hbBc0Ko86Rr25SDPKuDNYo8GHXfKBYm6BqhrMHICBVzn2Esd4enpSjSqMwtgx9vMyaIrAwLr705/q71J2ag2a/txgUK/nvMqnonD0zTVXUZi1N5i3iUhtNDbj+WNDdUSKXzSA1wK4NLY9X6csXf77NDJ5rhhuutWXHnhM5DhGY4NbPTeENkSxPpAny3likswn50SFYGkX/hfdt9XJDfqb7Gjalj7/fcq27QWa7xahx57T+8eJ2OuZSMyXjftG4355kUjmT3ADYj5BGnTgP46EAlWxAa7bLllyI9x71eW4UnUcU+ZM2rjIi52eeJoptMAYTV5sW8Q9QfnJRgXXc/weQsrZWcdULVmq+W6V7+z1s3C+6bCGd5QposYmyOeDwf0XtAZPkssG4wv/Pg7F4XbW4RalVJ7o+KZ25jml7NiDFKvsfs9+2wLHX4sbxG9NJA7MZwdrMSuoc8q52lAJk6n8t9XeDotg+oMj9mvaeMz/4HuIn9G/aNxDNCS2tPC0OMehQeaPKpHZ5LdVptzDPyOseQC/zlWefIs9ajIZP5J7ITyZ5P5+nEwSVX5nynvU/URZSRTy3Flkm0HIDTil95P3sPdcUT45SSQuLi4Sk5OT+LghPf67lAXm1MKvjh9xeyEClgFnjDbu5286neM2AMXhn5N4skGlx3+fRir4a/aJz2DMNeo507YZTGEb88lPsT9Q/LEht2O9VzNLzFxt0ubEMlQYMJcXyaD4vvlfUTB5T+UpL+9VtmOR+fwzUS0K4MkSD9j5VfIIf96QhWrv14iryySeVkQ240ky/gr5+GR+8kazBJOqKg8aeb+yM6rO8WCdST7/usnH5/cXzydaz9gH56o0/J9Rtm01Evm33Xwwv3wjAA2eTIglPf5PaVRSN5PXwQvDTxBCGy/TDwbzE0JvZEbt5zTjjeWb89lg/lElFE8N+CH4IMy0LoiqyvnVTyo7JY03J1XxCVGrCwb049+Rx4zzZL5pqEQW6/y0yh2r8TyH+Z3hj6maKzoqdpLgnnswjqn+i/GGPj1Zd/KUl18QuH3yKv3gSWJD6WBtwgR4ORGJfdfVTAD8kjSeLFpqyw3/v6YuVxsXL0bxeOrjRqVVLCoZGOrn+SSZDyL4+XgD4J8s3pl25UkXvyygT+Y/CPRFwgzP9VKNhR+r18ILajU2j/P5+HFD1Cf7RoAvS/L/Rmj74wngDx5QD+bRs1zMlzCWetuyAsVclViqOFfcInfT0UgWSP17jYitbsYX4txqEAYwgH2M5yqDgTu5/SKvP7eoEkKsT6WuBP+blG3blko2y/EYT2dfLjebz+L0fFXl1S16Udjjx+UWyypJL/8blR21LdsWi7Kq+vRk2xYewUCcRdOPtVTFMO5/yy0GTpraMqz8/cp2pyXcaGWRUmc263bb1ieL4eZDPF1zMcCTR6j16VNpKkfY/0ZZoe0fo3aJWzefbm5sy76B/zKzM5Pk/++ULY+6s2m7PZ1hTCnB/7z+F6jC2gKPfriSAAAAAElFTkSuQmCC" } }, "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![Unknown-2.png](attachment:Unknown-2.png)\n", "\n", "As with test frameworks, a large number of CI solutions exist. We will introduce a web-based framework called [Travis-CI](travis-ci.com), which interacts nicely with GitHub.\n", "\n", "Travis is configured from a file called `.travis.yml` in the root directory of the Github repository. This is another example of a YAML file, similar to the `environment.yml` file used for `conda` dependencies. [In this case](https://github.com/jrper/CI), the file looks like\n", "\n", "```yml\n", "language: python\n", "python:\n", " - \"3.6\"\n", "# command to install dependencies\n", "install:\n", " - pip install -r requirements.txt\n", " - pip install .\n", "# command to run tests\n", "script: pytest pytest_example.py\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Here the first line specifies that we're using python (and thus that travis needs to create a virtual machine running python. The next lines say we want to test python 2.7 as well as python3.6. The `install:` command tells Travis to install extra python packages our code requires using the `pip` package manager. For this repository, our `requirements.txt` file contains only one line:\n", "\n", "```txt\n", "numpy>=1.13.0\n", "```\n", "\n", "Remember this says that we need a version of `numpy` at 1.13.0 or more recent.\n", "\n", "When the Travis tests run they contact GitHub and add a green tick (success) or red cross(failure) beside the commit when it appears. Optionally, by modifying settings one can make a passing Travis result a condition to merge a pull request into the master branch. " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "\n", "

Exercise:

\n", "\n", "Try this yourself:\n", "\n", "
    \n", "
  1. Fork the repository on GitHub.com
    2. \n", "
  2. Log into Travis] (which links with your GitHub account).
    3. \n", "
  3. Add the Travis GitHub app to your new repository.
    4. \n", "
  4. Trigger a build, (or wait for one to start automatically).
    5. \n", "
  5. Add some commits to your fork to break and fix the tests.
    6. \n", "
  6. Add the docstring and unittest tests to the .travis.yml file.
    7. \n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Extending CI tests\n", "\n", "There are several things that can be added into the automated CI testing by editting the `.travis.yml` file:\n", "- Check coding Standards :- Test for `pep8`/`pylint` errors by running the tools on the code. Both of them return a failing error code by default if they detect any issues with the code base, which means they can easily be hooked into the tests. \n", "- Documentation Standards :- If any pregenerated manuals/reference pdfs are created, make sure that they are up to date with the code. We can do this by rerunning the documentation tool and make sure it gives the same answer.\n", "- Check examples :- Any examples provided should be included in tests, especially for production code. This is easy for docstrings (use doctests), but for longer tests, you might have to build simplified feature testing.\n", "- Add tools to check [code coverage](https://en.wikipedia.org/wiki/Code_coverage)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "

In this lecture we learned:

\n", "\n", "
    \n", "
  • Writing a formal test
    • \n", "
  • The basics of Test Driven Development
    • \n", "
  • Automated testing using testing frameworks and py.test
    • \n", "
  • Continuous Integration using Travis-ci.com
    • \n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Further Reading\n", "\n", "- The python documentation page on [venv](https://docs.python.org/3.6/library/venv.html)\n", "- A fuller [sphinx tutorial](https://pythonhosted.org/an_example_pypi_project/sphinx.html)\n", "- Python documentaion on [unittest](https://docs.python.org/3/library/unittest.html) and [doctest](https://docs.python.org/3/library/doctest.html).\n", "- Pytest [documentation](https://pytest.readthedocs.io/en/latest/).\n", "- A software carpentry [lecture](https://intermediate-and-advanced-software-carpentry.readthedocs.io/en/latest/testing-python.html) on testing.\n", "- [Travis-ci.com tutorials](https://docs.travis-ci.com)." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n" ], "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# This cell sets the css styles for the rest of the notebook.\n", "# Unless you are particlarly interested in that kind of thing, you can safely ignore it\n", "\n", "from IPython.core.display import HTML\n", "\n", "def css_styling():\n", " styles = \"\"\"\n", "\"\"\"\n", " return HTML(styles)\n", "css_styling()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": true, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": true, "toc_position": {}, "toc_section_display": true, "toc_window_display": true } }, "nbformat": 4, "nbformat_minor": 2 }