The invisible deficit

I read with interest a tweet from Kent Beck the other day as it seemed to ring a bell: "the complexity created by a programmer is in inverse proportion to their ability to handle complexity". He followed up the tweet with a note on his Facebook page explaining that he'd been doing a code review of a developer's code and noticed that the guy didn't recognise that the solution he'd adopted was needlessly complex compared to the problem, but the developer simply couldn't see it. Kent finished with "The programmer least likely to be able to handle the extra complexity is exactly the one most likely to create it. Seems a little unfair. I'm interested in how to break this cycle, and whether it is even possible to break this cycle." (emphasis mine).

A link in the comments led me to the Dunning-Kruger effect, and a light bulb came on as I recognised a friend's comment from last week that none of the rest of the developers on his team were able to code for toffee, indeed this sprint all they had been tasked to do was to review his code, and by the time he'd left the office on Friday evening they'd failed to do even that! Over a beer he opined that he'd be better off bringing in a machete to work on Monday and just hacking them all to death and then he'd be able to work faster! Well, he is Italian, so I can forgive him the exaggeration. Having spent much of my time over the last three months interviewing Java developers for one of our clients I can empathise with my friend's feelings: there are a lot of mediocre developers out there, particularly in the Java world (I refer to Java as "the new COBOL").

So what is the Dunning-Kruger effect? Dr. David Dunning, a Cornell professor of social psychology, in 1996 read an article describing the arrest of a Pittsburgh bank robber called McArthur Wheeler; undisguised and in broad daylight the guy had attempted to hold up a bank, his photos were later released to the press and he was arrested and charged. He had been under the sadly mistaken belief that if he smeared his face in lemon juice it would make him invisible to video cameras. Yes, really. Not only was he too stupid to be a bank robber, he was too stupid to know that he was too stupid to be a bank robber!

Dunning wondered whether it was possible to measure a self-assessed level of competence against actual competence. Soon he and his graduate student, Justin Kruger, had organised a program of research and three years later their paper, “Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments,” was published.

Surprise surprise! Kruger and Dunning found that incompetent people will:

• Tend to overestimate their own level of skill;
• Fail to recognise genuine skill in others;
• Fail to recognise the extremity of their inadequacy.

In one fell swoop they have explained why every taxi driver you've ever met is both an expert in politics and international diplomacy, and why a psychology study showed 93% of drivers rate their driving ability as 'above average" [O. Svenson, Acta Psychologica, 1981]. The thing is, this knowledge isn't exactly new; empirically, wise men have known this for a long long time, for example:

• "One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision" - Bertrand Russell
• "Ignorance more frequently begets confidence than does knowledge" - Charles Darwin
• “The only true wisdom is to know that you know nothing.” - Socrates

What Dunning has identified is a form of cognitive bias (something I talk about a lot, in relation to software development) based on the metacognitive inability of the lowly-skilled to recognise their shortcomings. This correlates strongly with Noel Burch's "Four stages of competence":

1. Unconscious incompetence
2. Conscious incompetence
3. Conscious competence
4. Unconscious competence

What we are seeing is the first stage, and as Burch correctly identified, moving from stage 1 to stage 2 is complicated by the fact that being in stage one we are not even aware of the fact that there is something more to learn, like the blind spot on the retina our mind rationalises reality around our lack of awareness to protect our self-esteem. Indeed, as Dunning says: "You can call it self-deception, but it also goes by the names rationalisation, wishful thinking, defensive processing, self-delusion, and motivated reasoning. There is a robust catalogue of strategies people follow to believe what they want to, and we research psychologists are hardly done describing the shape or the size of that catalogue!"

Faced with such a situation, how are we to move from Burch's stage 1 to stage 2 and become aware of our incompetence? To get out of the trap of cluelessness, the not knowing what we do not know, requires two things:

1. Exposure to new information;  
2. The ability to recognise that new information as useful and pertinent.  

The former happens all the time, it is the latter that presents the problem. As Kent asks, is it even possible to break the cycle? It occurred to me that just as it is almost impossible to spot your own mistakes when writing, Agile has built up a body of practices that rely on having another pair of eyes with another set of experiences and world-views to see what you are doing and to help you become more aware of your own limitations - these include (but are not limited to):

• Peer review (not Agile per se. but worth mentioning)
• Pair-programming
• Retrospectives
• Three amigos
• Iteration showcase
• Cross-functional teams
• Face-to-face interaction
• Daily stand-ups
• Test-Driven Development (the computer is the observer)
• Continuous Integration (the build machine is the observer)

All of these things allow us opportunities for feedback, to see beyond our own limitations, and give us opportunities to grow and learn; I believe this focus on collaboration is one of the core strengths of Agile and why it is more than just a 'bandwagon' or 'silver bullet' and truly represents a change for the better in Software Development.