Yes Sherlock Holmes is a genius!

There is a simple answer to our question from yesterday. The question was, does one have to be a genius to reconstruct a hidden structure, such as a crime? The question arises because it seems one does not. Detecting clues and finding out what they mean depend on knowledge we already have. If one has all the knowledge required to read the clues one should be able to recreate the structure from which the clues flow. From this point of view, following clues is nothing more than applying knowledge we already have to the solution of a practical problem. No genius is required in this kind of activity.

What is our answer to this question?

The answer is the following.

Let's start with ciphers first. It is clear that we do not have to be a genius to crack a cipher if we know already what clues to watch out for. But what about a new type of cipher? When we meet with a new type of cipher, will we know what clues to watch out for? Polyalphabetic substitution ciphers were invented in the 15th Century. It had a reputation of being unbreakable for some 400 years until a general solution was discovered by Kasiski in the 19th. Why did it take so long? Why is a new type of cipher sometimes so hard to break? Why do we still remember Kasiski?

As everyone knows, a cipher hides the characteristics of a language as it is ordinarily used. Sometimes a new cipher hides these characteristics so well that we do not know where the clues are, if any. Clues are the characteristics of structures, disguised. A disguise can be so heavy that, unless we have special knowledge, we will never know what is behind it. And this is where the work of a genius comes in. A cryptanalyst (codebreaker) who wants to break a new type of cipher will have to study the cipher to see where its weak points are. A weak point is where the disguise the cipher puts up is thin enough to be penetrated. Armed with this knowledge a cryptanalyst can then crack any cipher of the same or a similar type.

Cracking a new type of cipher sometimes requires knowledge we do not yet have. It may take a genius to provide us with this missing knowledge.

Now to crimes. Crimes of a known type do not need a Sherlock Holmes. Crimes of a new type can defeat a Sherlock Holmes ... unless he can figure out where its weak points are. But here Sherlock Holmes is at a disadvantage when compared to a cryptanalyst. Take Kasiski as example. Kasiski knows how polyalphabetic substitution works. He knows how to construct a cipher of this type and use it to encrypt messages. What he wants to find out is, when given a message encrypted by a polyalphabetic substitution cipher, how he could work out the cleartext on his own without having been given that particular cipher. But in the case of a novel kind of crime we don't know how the crime 'works': we have never committed it ourselves nor do we know of anyone who has. If we have committed it ourselves or know of someone who has, we can study the crime from the inside and find out where the weak points are. Lacking the knowledge that such a study will provide, how can we solve the crime? Here we need Sherlock Holmes. He is a consulting detective. He specializes in crimes that other people cannot solve.

How does Sherlock Holmes do it? How can we solve a new type of crime?

The answer is part of popular culture: Set a thief to catch a thief! Put yourself in the place of the criminal and see how you would carry out the same crime. It takes a clever criminal to catch another clever criminal: Sherlock Holmes and Professor Moriarty--they are of the same mind, just working on different sides of the law.

I have always had an idea that I would have made a highly efficient criminal.

--Sherlock Holmes in Charles Augustus Milverton

Einstein said, ‘God does not play dice.’ Bohr is reputed to have replied, ‘Don’t tell God what to do!’ Was Einstein in the habit of talking to God?

Is Sherlock Holmes a genius?

To decipher the messages from the Dancing Men Cipher Sherlock Holmes looked first for the most frequently occurring letter in these messages. He took this step because he knew that E is the most often used letter in the English alphabet.

When examining a crime scene one of the first things Sherlock Holmes does is to see if he could find any footprints. Footprints, he knows, can tell us a lot, including a person's height.

Detecting clues and finding out what they mean depends on knowledge already in our possession. If we did not know what footprints can tell us we would not look for footprints. If we did not know that E is the most frequently occurring letter we would not look for the most frequent Dancing Man. Now if we find enough clues we can reconstruct that structure which leave behind these clues, be that structure a crime, a secret message, or whatever. But if this is the case, Sherlock Holmes does not have to be a genius; anybody can do what he does; all we need do is make sure we know the same things as Sherlock Holmes. Take the Dancing Men Cipher again as example. Sherlock Holmes has shown us what we need to know to solve this cipher. Since he has already done this, next time we come across a similar cipher we should be able to solve it on our own. Is this not so?

All I want to know is who done it!

'Mr. Sherlock Holmes, your time is precious and my funds are limited. Instead of finding out what the butler was doing, and the maid, and the footman, and all the other people, why don't you simply concentrate on the murderer and find out who he is. That's all I want to know. For that I am willing to pay but not for anything having to do with the other people.'

Will Sherlock Holmes take on a commission like this?

Yet people often say, why don't scientists concentrate on discoveries that are practically useful and forget all the rest?

In solving the Dancing Men Cipher Sherlock Holmes was furnished with five messages from that cipher. Can we ask him, 'Don't bother with the other four; just tell me what the last one says.'?

(This is a follow-up on something I said in the last post.)

Why are some clues more specific than others?

In crime investigations fingerprints and DNA samples can provide important clues. Since no two people have the same set of fingerprints nor exactly the same DNA they identify individuals unambiguously. On the other hand, if a witness of a crime says the person running away has two arms and two legs, no one will marvel except perhaps at the naivete of this particular witness. Most people have two arms and two legs. Knowing that the suspect has two arms and two legs will not help us locate this person.

Why are some clues more specific than others? DNA is a specific clue; having two arms and two legs is not. (Indeed, having two arms and two legs is so useless as a clue that we might not want to call it a clue at all!) Now clues are the characteristics of structures, so one way to answer our question is to say, a clue is more specific if it is a characteristic found in a smaller number of structures. A particular DNA is the characteristic of just one person. Having two arms and two legs is a characteristic common to most human beings. DNA is a more specific clue than having two arms and two legs.

But this does not seem to be totally satisfactory. Consider the following. A murder investigation is nearing an end. Only two suspects are left: all the others have been ruled out. Of the two left only one is responsible for the crime; the other is innocent. So, who is the murderer? Which of the two?

Naturally, the thing to do at this point is look for more clues. Unfortunately in this case, none can be found. At the moment, everyone is still scratching their heads .... Then someone remembers: the person running away from the scene of the crime has two arms and two legs!

This wraps up the case. Why? Because of the two remaining suspects, one has lost a leg in an accident not so long ago.

Ordinarily, having two arms and two legs is not a specific clue but situations can arise in which it can become highly specific. In our example it is so specific that it points out who the guilty party is.

We have to re-think our question why some clues are more specific than others. We cannot say any more a clue is more specific if it is a characteristic shared by fewer structures. Most people have two arms and two legs; yet in our example having two arms and two legs became a significant clue!

It seems how specific a clue is depends on what stage of an investigation we have arrived at. In our example, at the beginning of the investigation having two arms and two legs is far from specific. Close to the end, it becomes highly specific. Why? What is going on?

I think what goes on is the following. We follow clues to reconstruct structures. Clues are the characteristics of structures but we do not gather all the characteristics first and then rebuild the structure. We do not ask, what are the characteristics of this crime? Then, after finding all of them, start to reconstruct the crime. Following clues is a narrowing-down process; we look for clues as we go along, to help us narrow down. In a murder investigation we do not start by suspecting everybody in the universe; we suspect only a tiny portion. As the investigation continues, as more clues are found, the number of suspects is whittled down until at the end the culprit is caught. But this is not yet the whole story. In order to pin down the murderer; in order to reconstruct the crime; we have to pin down the other players as well. Who are they? Where can we find them? What were they doing? What can they tell us? Are they telling the truth? In an investigation there are many things to pin down. A crime has many details (characteristics). The more of these details we can nail down, the clearer we will be as to who is responsible. Clues may not help us nail down all these details right away but they should at least help us converge on them.

What do we mean by 'converge on them'?

We mean having fewer interpretations ... We are converging on the things we want to know if our clues have fewer and fewer interpretations. If a clue tells us that the butler has to be in the room when the murder happened, we say this clue has nailed down the location of the butler: the clue has at this point only one interpretation. But sometimes a clue is not this specific. For example, it may only tell us that the butler is either in the room or the one next door. Clues are open to interpretations. The fewer interpretations a clue is open to, the closer it brings us to the truth. The more specific clues therefore are those that do not allow for much room in our interpretation. We have to remember, a clue is a characteristic of a structure, disguised. When the disguise is light it is easy to tell what it means. But when the disguise is heavy more than one interpretation of the clue will be possible. A characteristic under heavy disguise is a vague clue; a characteristic thinly disguised is more specific. But disguise is gradually removed as the investigation advances (things become clearer and clearer). At the beginning of an investigation knowing that the murderer has two arms and two legs would not help us but by the time the number of suspects is narrowed down to two this originally vague clue becomes highly specific.

Some clues are easy to notice ...

In A Study in Scarlet a man was killed in an unoccupied house. In the room in which the body was found there were five letters written in blood on the wall:

Rache

In this case we have a clue that is hard for anyone to miss. Of course, what it means is quite a different matter.

But not all clues jump out at us in this fashion. In cracking the Dancing Men Cipher, to find out which Dancing Man stands for the letter E Sherlock Holmes had to do a little statistical analysis.

The elliptical orbits of the planets around the sun was an important clue for Newton. It took astronomers centuries to extract this particular clue from the large body of observational data that had been collected. Part of the reason for this, as we all know, was that for the longest time they thought the data supported the view that the planets moved in circles around the earth.

Sometimes a clue is in open view and yet not noticed. Sherlock Holmes's professional colleagues from Scotland Yard often miss clues that Sherlock Holmes detects. The same clues are there for everybody to see but because they don't have the same background knowledge as Sherlock Holmes, they don't notice them. Sherlock Holmes can estimate the height of a person from the footprints that person leaves behind. Other detectives, not knowing there is a relation between a person's stride and that person's height, pay no attention to footprints.

Clues--what are they?

By following clues Sherlock Holmes can reconstruct a crime, a secret message, or even--if he works at it and is lucky enough--some law governing the workings of nature. What are they, these things we call clues? Why are they there? Why is it that they can lead us to things which initially we did not know?

Can we call just anything a clue? Criminals know that if they leave clues behind they run the risk of being caught. Can they, before they leave the crime scene, decree all clues to disappear?

Anyone who tries to play Sherlock Holmes will recognize the importance of clues. The word 'clue'--or some equivalent--they know is not a label we can apply to anything we please. Whether something is a clue in a particular case is a question with an objective answer. Indeed detectives distinguish between genuine clues and false clues (the practice at Sherlock Holmes's time was to call a false clue a blind). It is not a pleasant experience mistaking a false clue for genuine.

But what are they? What are clues? Why do they have this ability of leading us on, towards the truth?

Let's try to answer this question by going backwards; let's reason in the following way. Clues help us to reconstruct. To make things easier let us suppose the thing we want to reconstruct is not a crime but an ancient sailing ship. Some authors have mentioned this kind of sailing ship here and there but none has given a detailed account. In fact, so far as we know no one has come across even a single crude drawing. What should we do?

Naturally, the first thing we do is look for more information. We will scour books and museums and even archaeological sites to see what clues we can find, clues that will help us build this sailing ship.

What do we expect these clues to tell us if we should find any?
If we are to succeed in reconstructing this sailing ship, the clues we hope we will find will, eventually, have to tell us as many of the characteristics of this ancient sailing ship as we require. How big are these ships? How much can they hold? Can they go on the open sea or are they confined to shallow waters? We will have many questions like these and enough of them will have to be answered before we can go ahead and start building. This is of course a tall order, the reason why the chance of success in this kind of projects is seldom very high.

To build a sailing ship we need to know enough of its characteristics. A sailing ship is not a pile of timber with a piece of fabric thrown in. A sailing ship has a structure. Each kind of sailing ship has its own kind of structure with its own set characteristics. If we know enough of the characteristics of a sailing ship we can rebuild that kind of sailing ship. In the case of our sailing ship, there is no hope of our ever finding a blueprint. Because it is ancient we can find out its characteristics only through clues. Now the clues will not tell us directly what these characteristics are (because if they will, we will not call them clues) but eventually they will if they are to be useful at all.

So what are clues?

From our example we can see they are simply the characteristics of the structures we want to reconstruct, hidden behind some sort of disguise. We are using the word 'disguise' metaphorically to signify that clues do not tell us automatically what is behind them; we have to figure it out; we have to remove the disguise.

Which of course is usually the difficult part! But difficult or not, this is what clues are. They the characteristics of structures, disguised. This is the reason why by following them we can decipher secret messages. For those who send messages have to use a language. Now each language has its own structure with its own set of characteristics. For example, one of the characteristics of the English language is that of the twenty-six letters in the English alphabet E is the most frequently used. As we all know, this characteristic is often depended upon when deciphering English messages. In cracking the Dancing Men Cipher the first thing Sherlock Holmes did was to see which of the Dancing Men appeared most frequently in the intercepted messages. That Dancing Man is a clue. Behind it is the letter E.

Now just as a language has a structure, so does a crime. When a person is shot she falls. From the way she falls we can estimate the direction from which the shot comes. Thus it is that when Sherlock Holmes investigates a crime he will examine the crime scene carefully to see what clues he can find and basing on his knowledge of the characteristics of the various sorts of things he knows, figure out the meanings of these clues.

Scientists follow clues too. Why do the planets move around the sun in ellipses? Newton was able to work out mathematically that this is a characteristic of a universe governed by the theory of universal gravitation as he has formulated it, together with his three laws of motion.

In playing Sherlock Holmes we are trying to reconstruct structures we cannot see. To this end we have to follow clues because the clues are the characteristics of these structures, even though disguised.

No falsehoods in proofs!

In a mathematical proof we do not touch falsehoods at any point (see Note below for a minor exception). Any time a falsehood enters we are off the straight-and-narrow. In following clues, even when we are successful, the final product may not correspond exactly to the original.

---
Note: In a reductio ad absurdum proof we assume the opposite of the proposition to be proved. By showing that this assumption leads to a contradiction and therefore false, we conclude that the proposition to be proved must be true. In the case of the reductio then, we do touch a falsehood but only as an assumption. Moreover, the point of the reductio is to show how dangerous it is if we have any truckings with falsehoods: they lead to contradictions!

Truth and Impatience

People who think of themselves as serious about the truth are often impatient: they want to get at it right away because what is not true is false and no one in their right mind (they say) would want to touch falsehoods. But following clues is a tedious and even frustrating process: it requires a lot of patience. Even then the final product may not correspond exactly to the original.

Is sherlock Holmes not serious about the truth? Are we not serious about the truth when playing Sherlock Holmes?

Truth vs Partial Correspondence

A proposition is either true or false. A reconstruction obtained by following clues may not correspond exactly to the original, yet could be regarded as satisfactory or even valuable.

Proofs vs Reconstructions

In a mathematical proof we go from truth to truth to truth until we reach the conclusion. In reconstructing a crime, depending on how clues develop, we could go through many versions which differ in important ways from each other. In an earlier version the butler is the main suspect. In a later version he is totally innocent.

What does Sherlock Holmes aim at?

Sherlock Holmes has a method. What is he aiming at when he is exercising this method?

We all play detective sometimes. What should we be aiming at when we are playing detective?

I am asking this question because a method is employed to achieve some aim. If we know the aim we may be able to work out the method ourselves. So the question is, what are we aiming at when we are playing Sherlock Holmes?

Let's try out a few answers to see if any of them fits:

Solving a crime?
This can't be the right answer because, as we have pointed out in the last post, when we play detective we are not always trying to solve a crime: we could be trying to crack a cipher, for example.

Providing a solution to a mystery?
When we are playing Sherlock Holmes don't we always have some mystery that we want to solve? A mystery can be about anything, not necessarily a crime. A string of symbols that apparently doesn't make sense could be intriguing. Perhaps there is message behind it. If there is, can we find out? So we play Sherlock Holmes. Natural phenomena can be puzzling, so we engage in scientific research to find out what is really going on. Is this the right answer to our question then? When we play Sherlock Holmes are we aiming at solving some mystery?

I think the answer has to be no. Take the case of Watson's trip to the post-office: there is no mystery to it. If Sherlock Holmes wants to know where Watson went that morning all he has to do is ask.

The truth?
In solving crimes, in cracking ciphers, in scientific research--in all these endeavors we are aiming at the truth; are we not? In solving crimes we don't want to accuse the wrong person. In cracking ciphers we don't want to replace the true message with something we ourselves make up. In science we want to know what the world is really like; we don't want to deceive ourselves by adhering to superstitions.

Is this the right answer then? In playing Sherlock Holmes our aim is to get at the truth; is this not so?

Strange as it may sound, I don't think it is. Certainly Sherlock Holmes is interested in the truth but then most people are. Journalists get themselves into a habit of double checking their sources; why? To make sure they report the truth. So, yes Sherlock Holmes is interested in the truth but the kind of truth he aims at is more difficult to obtain than the kind that journalists report after double checking sources. Indeed, the kind of truth that Sherlock Holmes aims at is so difficult to obtain that usually no one would blame him if it were missing in some details. As Sherlock Holmes himself says, when he finds out what happens in a crime, usually he only knows in 'essentials', not in every single detail.

Truth of things hidden?
Why is it hard to obtain the kind of truth we seek when playing detective? Watson marvels at what Sherlock Holmes is able do; why? If Sherlock Holmes had accompanied Watson to the post-office, would Watson still marvel?

Clearly, the reason why the kind of truth detectives are interested in is difficult to obtain is that it is about things hidden, things that the detectives themselves cannot see. Now since this is the case, should we not say that our aim when playing Sherlock Holmes should be the truth of those things that we want to know but which are hidden?

This I think will not be an acceptable answer either. For when we play Sherlock Holmes we do not look for just anything that is hidden and try to see what we can find out about them; we look only for things for which we have clues. Not everything that is hidden can be known, only those that leave behind clues.

Perhaps then this is the answer to our question. Our question is, what should Sherlock Holmes aim at? It seems the right answer is, he should aim at truths of things hidden, which we want to know and for which there are clues.

Should we stop here? Should we take what has just been said as the answer?

We could but I suggest we do not. Detectives are practical people. The answer just given is not likely to mean much to them. It is very well to say we should get at the truth but how in practice do we do it when all that we have are clues? So instead, I suggest we adopt the following answer, which is slightly different but has the merit of being easier to understand. In playing Sherlock Holmes we should aim at the reconstruction of those things hidden, which we want to know, basing our reconstruction on clues. Put this way we can see that our answer conforms to actual practice. In practice, when we are trying to solve a crime we are in fact trying to recreate that crime (not by committing it again but in words or through play-acting). When we are cracking a cipher we are trying to recreate that cipher. And in the case of science, we are trying to recreate in symbolic form structures present in the universe. In all these instances, it is clear that we have to rely on clues. The more clues we have, and the more significant these clues, the more accurate and more complete the reconstruction. Once a reconstruction is in place we can tell what is true based on this reconstruction. Is it true that the butler fired the fatal shot? Our reconstruction tells us that it is. Did the butler intend to shoot the victim? We do not know because our reconstruction is incomplete (due to the lack of sufficient clues).

What Sherlock Holmes is able to do often surprises people. Sherlock Holmes wants to know about things hidden. But how can things hidden be known?! To say that they can seems to be a contradiction in terms. However, if we look at what Sherlock Holmes is aiming at slightly differently all the mystery disappears. The reconstruction of things whose originals no longer exist is done all time. It is just a matter of whether we have enough clues. The more clues we have the closer to success we are. Do we need complete success all the time? No, not when it is the truth we are looking for. Partial knowledge is better than complete ignorance.

Does Sherlock Holmes only solve crimes?

We all know that he does not. Using the same method that he applies in crime detection he does other things too. For example, there is this famous instance that Watson recounts about his visit to the post-office, something that he did on the spur of the moment and about which he had told no one. Exercising his Method Sherlock Holmes knew not only that Watson went to the post-office, but which post-office and what he was doing there.

"... you have been to the Wigmore Street Post-Office this morning, ... when there you dispatched a telegram."

"Right!" said I. "Right on both points! But I confess that I don't see how you arrived at it. It was a sudden impulse upon my part, and I have mentioned it to no one."

"It is simplicity itself," he remarked, chuckling at my surprise--"so absurdly simple that an explanation is superfluous; and yet it may serve to define the limits of observation and of deduction. Observation tells me that you have a little reddish mould adhering to your instep. Just opposite the Wigmore Street Office they have taken up the pavement and thrown up some earth, which lies in such a way that it is difficult to avoid treading in it in entering. The earth is of this peculiar reddish tint which is found, as far as I know, nowhere else in the neighbourhood. So much is observation. The rest is deduction." "How, then, did you deduce the telegram?"

"Why, of course I knew that you had not written a letter, since I sat opposite to you all morning. I see also in your open desk there that you have a sheet of stamps and a thick bundle of postcards. What could you go into the post-office for, then, but to send a wire? Eliminate all other factors, and the one which remains must be the truth."

--The Sign of Four

It is well known also that Sherlock Holmes solves ciphers. In The Dancing Men he was able to decipher a number of secret messages. How did he do it? He did it by using the same method that he used in solving crimes.

But Sherlock Holmes not only solves crimes and deciphers secret messages, he also has a strong interest in some of the sciences (including chemistry but not astronomy). What was he doing in his retirement besides tending bees? 'Of late I have been tempted to look into the problems furnished by nature rather than those more superficial ones for which our artificial state of society is responsible,' he says (The Final Problem). To Sherlock Holmes solving problems presented by nature is no different from solving problems presented by society: we use the same method.

From the above we see that solving crimes, tracking the whereabouts of Dr. Watson, cracking ciphers, engaging in scientific research--all these fall within the province of the detective inasmuch as the same method can be employed in all of them. When we are playing Sherlock Holmes it is not necessary that we be only interested in solving crimes. Indeed, whatever we may be interested in, so long as our interest requires that we employ the same method that Sherlock Holmes employs, we are playing Sherlock Holmes.

What this blog is about

Of all detectives Sherlock Holmes is the most famous. When we act like a detective we are often said to be playing Sherlock Holmes. What should we do if we want to play Sherlock Holmes? What should we do if we want to be a good detective? This is what this blog is about.