Amount of information given by tests

The “amount of the information” given by failing tests about the broken functionality may vary because sometimes many tests may fail as a consequence of one single functionality broken. In the following example the amount of information tends to zero as the complexity of the tested object (a state machine) increases (in term of number of states), unless specific  programming practices and constructs are used (that in this case will be injecting the initial state by constructor ).

I have a state machine, that has an initial state A, that switch the state from A to B on input ‘ab’; and from B to C on input is ‘bc’

If there is no way to set initial state to B independently from the A->B transition then the tests become somehow dependents each other: the second tests need to do a setup that works only if the ab functionality, given by the first test, works.

So the amount of information given by a test failure respect to a broken functionality as the complexity of the state machine increases, tends to zero because every test need to start from the initial state.

In unit testing the problem is solved injecting the initial state by constructor, but in other type of tests the solution looks less obvious, with the cost of having tests with low information, and so low value.

BDD like specification of the behavior of the machine are:

Test 1 - “go to B, from A”

“given the state is A // (i.e. the initial state)

when the input is “ab”

then the final state should be “B”

Test 2 - “go to C, from B”’

given the state is B

when the input is “bc”

then the final state should be “C”

One way to prepare for the test 2 (the “given” part), that requires starting from B, is starting from A and going to B so “given the machine is in state B” were to become “given the machine is in state A, and the machine is given the input ‘ab’:

given the machine is in initial state A

and the machine is given the input ‘ab’ // going to state B

when the input is “bc”

then the final state should be C

In case of failure of the tests the amount of information provided by the failure is not always the same:

If the broken functionality is the  B->C transition, then only the second test is broken

If the broken functionality is the A->B transition, then both the tests are broken.

So we lack of one-to-one correspondence between failed test and broken functionality.

There is a fragility in tests: single broken functionality affects many tests as side effects.

It can be fixed if we are working at an object level, modifying the code in order to set the initial state by construction:

public class StateMachine {

public StateMachine() {

this(state.A)

}

public StateMachine(State state) {

this.state = state;

}

....

}

Now one broken state transition code affects only one state-transition test, and one broken test may means that only the “then” part doesn’t work (or that the constructor doesn’t, which will affect all the tests in that case)  so the “fragility”, (or “entropy”, or “unpredictability”, or “noise”....) is reduced.

A little bit more complicated is finding a “workaround” if we talk about tests from the user prospect, like in tests about the web navigation, where the “machine state” is basically a navigation path from the home page to the news, and from the news to the tech-news:

Test 1 - “see the news”“given connect to the home page

when I click on ‘news’

then the header of the page should be “news”

Test 2 - “see the tech news, subpage of news”

given I am on homepage

when I click on news

and I click on tech

then the header of the page should be “tech news”

If the “news” button does not work, both the tests are broken, as in the above example.

Here there is no a “straightforward” way to “set the state” relate to the “given” part in a way independent from the other functionalities, and so there is still the problem we talked about.

So for tests closer to domain it is more likely that the problem of “noisy” tests is reduced,

Keywords: “information theory” (Shannon), “Bayes Theorem”, “value of Information” (Book: “how to measure anything”).