On Honesty

I am no zealot when it comes to research methods. I firmly believe that different approaches are suited to different situations. Experiment and case study, quantitative and qualitative methods, each have their place. There are some research methods where I must confess scepticism, so called 'grounded' approaches, because I cannot believe that it is possible to approach problems without some analytic framework, and questionnaires (excluding survey approaches) because they are so difficult to do well - question design, sample size and so on. Overall however, I regard myself as flexible and open to any strategy that will yield a practical insight into the engineering problems that concern me.

It is with regret therefore that, given the many choices available to researchers, I review so many papers (in computer science, my area of expertise) that are deeply flawed. The flaw lies with what I might describe as their essential 'honesty'. It seems to me that the core of any science, construed broadly, is a genuine scepticism and a personal commitment to get at the truth. Whatever the approach that is taken there must be a clear intent to try to falsify the hypothesis, subvert the framework or place limits on the applicability of a technique. I recognise that such an outcome may present difficulties for the individual but it is how the collective endeavour advances, and how we get nearer that fragile thing 'the actuality'. Too many papers seem to me to evade, whether deliberately or unconsciously, a process by which they actually confront their work with tests that might fail. In general, the longer the 'methodology' section of the paper, and the denser the statistical analysis, the more likely it is for this to be the case.

I have had students who genuinely struggle with this. Typical of this is a student who spends excessive time searching out a case study or an experimental situation that is 'just right' and then conditioning and cleaning data of all the awkward 'anomalous' real-world detail that would make their tool or technique difficult to apply. This is short of fraud, not least because they report doing this, but at the limit it may be difficult to distinguish fraud from the struggle with self-criticism. Engineering research demands, in particular, a clear and open engagement with the world 'as it is', any departure from this, does not simply place the integrity of the research at risk but renders the outcomes worthless.

I believe we are too tolerant of research that steers away from risking attempts at falsification. Indeed, there is too much research that while borrowing the 'form' of established research methods is organised so as to avoid a negative outcome. While this research might be engaging and technically interesting there is at its core a void.