Laboratory conditions don’t exist
Even when we don’t rely on our own prejudices,
belief systems and confidence levels and instead do
some research into what may be the best course of
action, we can still come undone by the environment
we choose to test in.
Entire industries exist to help mitigate the mistakes
we may make in our endeavours. Researchers,
social scientists and strategists of all sorts test
hypotheses, conduct double blind experiments and
enlist carefully selected polling of ‘typical’ subjects,
producing reams of data … even big data (the
corporate world’s new security blanket).
And yet, failure is everywhere.
We have often been wary of the true intentions of
much research and testing, suggesting (perhaps
unfairly) that this work is largely used as ‘screw-up
insurance’ — in other words, research conducted not
to inform, but as a defence should things go horribly
awry. An employee or consultant can hardly be held
responsible for failure if the research suggested
success was a more likely outcome.
However, even when the aspirations and the
participants involved in the research are noble and
rigorous, errors still persist. Part of the reason for this
is the choice of environment in which research
is conducted and the margins for error agreed
upon. So much of the research people do isn’t
conducted in the real world and the artificiality of
the environments we create can’t help but skew the
results. For example, if you ask someone about their
political ideals in a polling questionnaire, they are
likely to want to appear more caring, more intelligent
and more interested than they may a ctually be. As
a result, a lot of research suffers from much of the
same over-confidence in its results as our own best
estimates.
To be fair, big data has started to go a long way to
improving this process, given its real-world sourcing,
although, like all data, big data is only as powerful as
its interpretation and application.
Another possible solution lies in a more scientific
rather than corporate view of research; that is,
research that’s designed to generate information, not
conclusions. In other words, rather than looking only
to prove a hypothesis, we should also use research
as a way of identifying the threats to our hypotheses
and the conditions under which this proof may come
undone.
So instead of focusing on an outcome, we should be
focused on generating outcomes.