Yes Sherlock Holmes is a genius!

There is a simple answer to our question from yesterday. The question was, does one have to be a genius to reconstruct a hidden structure, such as a crime? The question arises because it seems one does not. Detecting clues and finding out what they mean depend on knowledge we already have. If one has all the knowledge required to read the clues one should be able to recreate the structure from which the clues flow. From this point of view, following clues is nothing more than applying knowledge we already have to the solution of a practical problem. No genius is required in this kind of activity.

What is our answer to this question?

The answer is the following.

Let's start with ciphers first. It is clear that we do not have to be a genius to crack a cipher if we know already what clues to watch out for. But what about a new type of cipher? When we meet with a new type of cipher, will we know what clues to watch out for? Polyalphabetic substitution ciphers were invented in the 15th Century. It had a reputation of being unbreakable for some 400 years until a general solution was discovered by Kasiski in the 19th. Why did it take so long? Why is a new type of cipher sometimes so hard to break? Why do we still remember Kasiski?

As everyone knows, a cipher hides the characteristics of a language as it is ordinarily used. Sometimes a new cipher hides these characteristics so well that we do not know where the clues are, if any. Clues are the characteristics of structures, disguised. A disguise can be so heavy that, unless we have special knowledge, we will never know what is behind it. And this is where the work of a genius comes in. A cryptanalyst (codebreaker) who wants to break a new type of cipher will have to study the cipher to see where its weak points are. A weak point is where the disguise the cipher puts up is thin enough to be penetrated. Armed with this knowledge a cryptanalyst can then crack any cipher of the same or a similar type.

Cracking a new type of cipher sometimes requires knowledge we do not yet have. It may take a genius to provide us with this missing knowledge.

Now to crimes. Crimes of a known type do not need a Sherlock Holmes. Crimes of a new type can defeat a Sherlock Holmes ... unless he can figure out where its weak points are. But here Sherlock Holmes is at a disadvantage when compared to a cryptanalyst. Take Kasiski as example. Kasiski knows how polyalphabetic substitution works. He knows how to construct a cipher of this type and use it to encrypt messages. What he wants to find out is, when given a message encrypted by a polyalphabetic substitution cipher, how he could work out the cleartext on his own without having been given that particular cipher. But in the case of a novel kind of crime we don't know how the crime 'works': we have never committed it ourselves nor do we know of anyone who has. If we have committed it ourselves or know of someone who has, we can study the crime from the inside and find out where the weak points are. Lacking the knowledge that such a study will provide, how can we solve the crime? Here we need Sherlock Holmes. He is a consulting detective. He specializes in crimes that other people cannot solve.

How does Sherlock Holmes do it? How can we solve a new type of crime?

The answer is part of popular culture: Set a thief to catch a thief! Put yourself in the place of the criminal and see how you would carry out the same crime. It takes a clever criminal to catch another clever criminal: Sherlock Holmes and Professor Moriarty--they are of the same mind, just working on different sides of the law.

I have always had an idea that I would have made a highly efficient criminal.

--Sherlock Holmes in Charles Augustus Milverton

Einstein said, ‘God does not play dice.’ Bohr is reputed to have replied, ‘Don’t tell God what to do!’ Was Einstein in the habit of talking to God?