Adding `skip()` and `only()` features to unittest

If you are like me, a python newbie just moved from web front-end world, you must have used Mocha before, the most commonly used test framework made by the genius TJ.

skip() and only() are the two most useful features for me among all the tools that Mocha provides. So when I started to write test cases in Python using Unittest, I felt so frustrated, because there’s nothing that can do the same as the skip() and only() did for me.

So I am going to teach you how to add these two features to unittest-testing in this article, but before we get there, I want to talk more about why these two features are important to your development.

Faster Feedback and Iteration

only

Imageing you are working in a project for your company, and your task today is adding a new feature that only needs you to write another simple utility function. It took you half an hour to finish, and now you need to test it!

I believe few people can write flawless code all at once and most people make minor or big mistakes here and there. Even one simple function needs to go through several debuggings, tests and reviews. What makes it even harder is that writing flawless test cases is not an easy job neither.

You will probably write your test cases from scratch and then run them. Things will not go well at begging, your code will throw mysterious exceptions and you will correct the mistakes and run the test again, back and forth. You will make all the test cases pass at the end of the day.

Just to make the situation even worse, there’s already hundreds of test cases in the project. To run your test cases, the framework has to go through all these hundreds of test cases together. Your test cases only take around 1s to finish, while testing the whole project will take minutes and not to mention the countless lines of irrelevant output logs that the serveral lines of yours will be buried in.

So it’s the perfect user case for the only() feature. It saves your time and lets you focus on the test cases that you care about.

The result of your test cases are the feedbacks of your code, the faster the feedback can be given to the change you make, the sooner your code can reach to the ready level.

The test-and-change is just like a small interation in our development process, the quicker it can go, the faster our development can move forward.

skip

There could be couple reasons for why you want to skip your test cases:

• the test cases depend on specific system environment, you need to skip them when certain requirements are not met
• the test cases depend on functions that are still under development or fail in some way. It is quite common in a TDD style development that test cases will be written before the development starts.

By skipping these unnecessary or unready test cases, you can be more concentrated, which leads to the same goal that the only() wants to achieve: making your development faster.

Examples in mocha

Before we dive into the Python world, I want to show you some examples about the way of using only() and skip() in Mocha and we are going to achieve something similar in Python.

In Mocha, you can make it only run specific test cases by appending a .only() to those test cases, like below:

describe('My Component', () => {
  it('should be very strong', () => { ... });
  it('should be very flexible', () => { ... });
  it.only('should be very agile', () => {
    /* I am still working on it! */
  })
});

Instead of just focusing on one test, you can also test the suite only:

describe.only('My Component', () => {
  it('should be very strong', () => { ... });
  it('should be very flexible', () => { ... });
  it('should be very agile', () => { ... })
});

.skip() follows the same way.

Unittest usage in Python

Being different from Mocha, for identify test case & suite, unittest requires that you wrap your test cases in a class drives from unittest.TestCase:

import unittest

class BaseTestCase(unittest.TestCase):
  def test_user_login(self):
    ...

So the way that unittest detects test cases follows two steps:
– find out all the classes that derive from unittest.TestCase
– get all the methods whoes name starts with test, and put them into the test case list.

Make our rules

It will be hard (at least for me) to change the whole syntax of unittest and make it like Mocha, but we can use a little trick to bypass the unittest’s own detection process and only test what we want. So before we make that change, let’s look at the rules I made:

• If the name of a test case class ends with __only, all the test methods of it will be put into the exclusive list
• If the name of a test method ends with __only, it will be put into the exclusive list
• If the name of a test case class ends with __skip, all the test methods of it will be ignored
• If the name of a test method ends with __skip, it will be ignored

The rules above will allow us to write test case like below:

import unittest

class BaseTestCase__only(unittest.TestCase):
  def test_user_login(self):
    ...

Or you can skip test cases with __skip:

import unittest

class BaseTestCase(unittest.TestCase):
  def test_user_this_is_not_finished__skip(self):
    ...
  def test_user_login(self):
    ...

Though it’s not as elegant as Mocha, it is quite straight forward and easy to use.

Core Concerpts in Unittest

There are several concerpts in Unittest that you need to know about:

• Test case: A test case is the smallest unit of testing. It checks for a specific response to a particular set of inputs.
• Test Suite: A test suite is a collection of test cases, test suites, or both. It is used to aggregate tests that should be executed together.
• test runner: A test runner is a component which orchestrates the execution of tests and provides the outcome to the user. The runner may use a graphical interface, a textual interface, or return a special value to indicate the results of executing the tests.

A test process can be simplified like below:

test_case_or_suite = /* you somehow managed to get them */
test_runner.run(test_case_or_suite)

A commonly used way of dicroverying test cases is using unittest’s built-in TestLoader:

import unittest

tests = unittest.TestLoader().discover('tests', pattern='test_*.py')
result = unittest.TextTestRunner(verbosity=2).run(tests)

if not result.wasSuccessful():
    sys.exit(1)

You can check out the signature for the discover, basically it goes through the ./tests to collect all the test cases and test suites.

Collecting test cases

According to the last section, we can see that the core steps of testing with unittest are:

• get target test cases or suites
• run the test cases or suites

Runing collected test cases are simple:

result = unittest.TextTestRunner(verbosity=2).run(testCases)

What we need to work out is how to get all the target test cases follow our rules. Now that we already have all the test cases or suites:

tests = unittest.TestLoader().discover('tests', pattern='test_*.py')

We just need to filter them with matching __only and __skip in their names!

the result tests is a list of test suites, so let’s create a function to iterate a test suites first:

def iterate_test_cases(test_suite_or_case):
    """Iterate through all of the test cases in 'test_suite_or_case'."""
    try:
        suite = iter(test_suite_or_case)
    except TypeError:
        yield test_suite_or_case
    else:
        for t in suite:
            for subtest in iterate_test_cases(t):
                yield subtest

Once you can separate a single test case from a test suite, you are able to get its name and the class name:

test_cases = unittest.TestLoader().discover('tests', pattern='test_team*.py')
for ts in iterate_test_cases(test_cases):
    case_name = ts.id()
    case_class_name = ts.__class__.__name__

Now, if we think about the relation between __only and __skip more carefully, we will find there are rules we missed:

• __skip and __only can not be applied to the same class or method at the same time
• if a class is marked as skipped, all the methods inside the class will be skipped regardless of whether they are marked with __only or not
• if a class is marked with __only, methods marked with __skip inside the class can still be skipped

So the code will be:

not_skipped_cases = []
only_cases = []
for ts in iterate_test_cases(test_cases):
    case_name = ts.id()
    case_class_name = ts.__class__.__name__
    if not (case_class_name.endswith('__skip') or case_name.endswith('__skip')):
        if case_class_name.endswith('__only') or case_name.endswith('__only'):
            only_cases.append(ts)
        else:
            not_skipped_cases.append(ts)

Now that we have filtered the list with __only and __skip, our final test cases wil be:

final_test_cases = []
if len(only_cases) > 0:
    final_test_cases = only_cases
else:
    final_test_cases = not_skipped_cases

Almost there! The last thing we need to do is wrapping all the test cases into a test suite:

test_suite = unittest.TestSuite()
test_suite.addTests(final_test_cases)

and run:

result = unittest.TextTestRunner(verbosity=2).run(test_suite)

You can check out the complete code here