In 1951 Theodore Sturgeon was giving a talk about science fiction when someone in the audience noted that “90% of science fiction writing was crap”. Sturgeon shot back that “90% of everything is crap”.

This observation came to be known as Sturgeon’s Law.

Using the same standards that categorize 90% of science fiction as trash, crud, or crap, it can be argued that 90% of film, literature, consumer goods, etc. are crap. In other words, the claim (or fact) that 90% of science fiction is crap is ultimately uninformative, because science fiction conforms to the same trends of quality as all other artforms.

— Theodore Sturgeon

Sturgeon’s law is just as valuable when you are thinking about professionals as the old joke about doctors illustrates.

Q: What do you call a doctor who graduates at the bottom of his class?

A: Doctor

After years of pondering, I have come to believe that Sturgeon’s Law is more about the 10% that isn’t crap than the 90% that is. Forgetting the exact ratios for a moment, in every artform and every sporting contest, in every profession and every human endeavour, there is a distribution of quality in which a minority stands out as much better than the rest. We all know teachers who work that little bit harder to make their lessons interesting or the pharmacist who goes the extra mile to make sure that you understand your prescription. In my own profession—software engineering—only about 10% of engineers ever read a book about their craft once they leave college. This leads me to what I think is a more profound corollary to Sturgeon’s Law.

If you are a software engineer at the top of your game, you probably read books, attend conferences, study relentlessly to enhance your skills and engage in endless discussion on how to improve the state of the art of software craftsmanship.  Average software engineers do not do this but you are probably surrounded by the rare few who seek to be the best they can be. This is where the less obvious aspect of Sturgeon’s Law casts its insidious spell. When you meet people outside of your elite circle, they are more likely to be average than elite.

Most engineering managers have a similar disinclination to better themselves. The best of them are very good but most of them are not the best. Too many are merely average. When the very best of the software engineering profession looks at the competence of software engineering management, they sigh a little because most managers are not very good.

Oddly enough, the very best of software engineering managers have a symmetrical view of the engineers they manage. Most engineers are not very good either. This isn’t just true of software engineers and their managers; it’s true of QA engineers, product managers, and designers too. Most QA engineers think that most programmers suck at testing. It’s true. Most of them do. But most QA engineers suck at it too. To the best QA engineers, the average programmers seem like uncaring barbarians. And vice versa.

I should emphasize at this point that I am not suggesting that some people are better than others at everything. Even very great software engineers might suck at gardening or astronomy.

This corollary to Sturgeon’s Law pertains across so many domains where the best of folks surround themselves with other folks who are motivated to excel. The average person outside their circle contrasts poorly as a result. Consider a Kalenjin long distance runner from the Rift Valley in Kenya. He probably encounters other great runners almost every day of his life. Most of the people he runs with, eats with and loves with are also great runners. If our Kalenjin got on a plane to Helsinki, he’d probably be disappointed to find that most Finns are not great long-distance runners. Most of them are just average. Truth be told, most of the Kalenjin are probably average too but our guy doesn’t encounter them very often. He hangs with the elite when he is at home. Away from home he has less opportunity to be so picky. It was only when he went to Finland that he encountered so many people who were not elite runners. Finland has elite runners too of course (I fondly recall cheering for Lasse Virén at the Montreal Olympics) but you are not going to bump into them at the airport as you might when you are at the Kenyan School for Elite Runners.

It’s helpful at this point to remember that individuals are not statistics (The median is not the message in the memorable words of Stephen Gould). If you wanted to recruit a team of long-distance runners it wouldn’t be a great idea to fly off to the Rift Valley and round up the first 5 runners you came across. You’d probably have a very average running team. You’d be much better off choosing great runners wherever they hail from. Furthermore, if you had to choose between a Finnish runner and a Kenyan runner, there is no need to check their birth certificate or the colour of their skin when you decide which one should join your elite running team. You can actually race them and choose the one that runs the fastest. When you are dealing with individuals, you should concern yourself with their individual qualities, not with some arbitrary statistical correlation however accurate that may be. It might be true that the average Kenyan runs faster than the average Finn but—so what? You will never be in a situation where you have to choose between average individuals without some other evidence to inform your choice.

The insidious nature of statistical racism is magnified by confirmation bias. Once you have decided that Kenyans run faster than Finns, it’s all too easy to reinforce your prejudice by noticing all the data points that confirm your bias—Hey! There goes another fat, slow and lazy Finn!—and to overlook the occasions that contradict your instincts. There is a long, unfortunate history of people doing exactly this.

There’s a whole garden of isms that wither under Sturgeon’s steely gaze. My dentist (Hi, Dr Bobba!) has a lovely cartoon on the wall with a caption that says something like “Women will never make good dentists. Their wrists are too weak!”.

Put yourself in the perspective of a Victorian gentleman who just happens to be a very good dentist. All your friends are very good dentists. They all have a certain background in dentistry and very strong wrists. You probably have a quite distinct image of what a proper dentist looks like. The average woman of your acquaintance probably seems very un-dentist-like. She is probably very uninterested in dentistry and has very dainty wrists. If you had to choose a dentist based only on wrist strength, you’d be marginally better off choosing the dentist with the stronger wrists—in 1875. But in 2014, you can skip right past concerns about wrist strength and whether your dentist has the appropriate genitalia and just hire the one who is the best at dentistry. And Doctor Bobba *is* very good. Trust me on that.

More casually—in our everyday lives—we are surrounded at work by people who share a certain intellectual outlook on life. Maybe your colleagues are more interested in politics than the average citizen. Maybe your friends at the sports bar care an awful lot about the intricacies of the infield fly rule or exactly how many defenders need to be behind the ball before offside is called. They know more than the average Joe about sports and certainly more than the average wife. Does that mean that women don’t understand politics or sports? No, of course not. The median is not the message, remember? It means that the average wife—in fact, the average anyone—knows less about sports than the fanatics you hang out with at the sports bar.

Let’s try some more examples.

The average tourist who visits Paris from their friendly little town in Georgia will find most Parisians quite distant, abrupt and possibly rude. The literary Parisians that he encounters will surely conclude that tourists from Georgia know very little about French art and are quite uncultured.  If you repeat the experiment in the opposite direction and send a farmer to Atlanta from a little village in Provence I’ll wager the outcome will be identical. Repeat as necessary with Beijing, Nairobi, Melbourne and Rio.

The average kid who spent every evening of the 1970s browsing record stores for rare blues recordings is likely to be disappointed with the crap his kids listen to on The YouTube. And vice versa.

If you are really interested in US history, I bet you are disappointed with how little the kids of today know about your favourite topic. Guess what! They are disappointed in you too!

The average software developer who does not have a degree in computer science probably doesn’t know much about data structures and algorithms. Neither does the average CS graduate. Most of them slept through that class or forgot most of it the next day. More surprisingly, the average PhD is not very good at software engineering either unless they are working in their very narrow field of expertise. Of the best engineers I have ever worked with, only a few had a PhD or a Master’s degree in CS. Some had degrees in English or music and a good number had no degree at all. In fact, it’s quite amazing that many of the most famous people in software dropped out of college—or maybe that’s just my own confirmation bias playing tricks on me.

I expect the world would be a much happier place if people listened to their Uncle Sturgeon and relied on statistics and biases only when they prove useful. A statistical overview of a population can be helpful when you are deciding how to profitably market your new product or where to spend your campaign dollars or which college recruiting fair to attend. But if you are choosing an umpire for your baseball league or an anchor for your running team or a new hire for your software startup you’d do better to ignore the statistics and hire the individuals with the right skills for the job. To do otherwise is prejudice.