In my last post I suggested some elements related to a generic "production system".
I want to suggest some example of how I would apply such hypothesis of definition of those elements like learning, experimentation, hazard, and value of experiment
It is still learning even if it has no value, i.e. the measures show us that there will be no improvement if we adopt this actions: we know the price of some machines but we can't afford them, or our production volume does not justify the action of buying such machines. So basically: learning is getting new information without regard to their value.
We make an hypothesis about the impact of some changes (in term of new actions), and adopt them in a safe to fail (i.e. controlled risk) way, in order to generate learning (particularly about the validation of the hypothesis). Example: we make assumptions about a direction of a code refactoring (introducing some pattern, establishing where is more strategical to apply the Open Closed principle), and we start in that direction with virtually zero risk ensured by unit tests, automated refactoring, and code versioning. The impact of those changes has measures (for example in term of facilitating, i.e. reducing the cost, of implementing some possible future features, or changing existing features). The extreme opposite situation is when we have few refactoring skill, or unit test support and so on, and in that case is more likely that such changes become a Hazard.
We have to make some changes directly in a production, affecting all the users, because a serious bug need to be fixed urgently, and not fixing it may seriously compromise the term of some contracts. We don't have a reasonable measure of the risk involved in the other side effects, because we don't have full control of the production environment.
A variation of the case is when we have A/B testing, which means that basically this same situation can be treated as an experimentation rather than an hazard.
Value of an experiment: the expected value of the information that the experiment will produce, minus the cost of the experiment.
Being the "expected value" basically a weighted average among possible events, any of them has a single value that starts from the “valueless” learning to the discovery of some important customer need.
This weighted average among the probability of all those events (including the cost of the execution of the action, or the kata for the needed improvements, or other "implementation costs") will give this value of the experiment. I don't mean necessarily that event, costs, and probabilities need to be strictly related to the mathematical probability theory framework. I'll consider also brainstorming sessions, deliberate discovery of known and unkowns using "gamestorming", pre-mortem, determination of confidence intervals given by calibrated estimators (ref: "how to measure anything") and so on...