The Small Steps Principle

Clues allow us to reconstruct hidden structures. When a structure is complex we need many clues. For clues are the characteristics of structures. When a structure is complex, to mark it off from others like it--to mark it off, that is to say, from neighbouring structures--we will need as many of these characteristics / clues as we can. This is to say, in a complex investigation each clue can only give us a small piece of information about the structure which we want to reconstruct--small, that is to say, in relation to the whole. From footprints Sherlock Holmes can estimate a person's height. Now this is useful information but just one small item. He will need many more items, many more clues, if he is to identify the person.

In investigations our steps forward are guided by clues. When the structure we want to reconstruct is complex, to reconstruct this structure we will need many steps, each small. When playing Sherlock Holmes we therefore have to observe a Small Steps Principle: we take only small steps, never huge leaps. Failure to observe this principle will mean failure in the reconstruction.

The name the Small Steps Principle is my creation but its content is recognized in practice. In practice people know there is a limit to how much one can read into a clue. When an investigator exceeds the limit, we say he / she is taking too large a leap. In our cryptanalytic example used in earlier posts (reproduced below), to say SB might mean SH or TH or AR is permissible. But to say that SB alone tells us that the first two words are ARE AREAR is too large a leap.

SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI

The Small Steps Principle applies only to complex investigations, investigations in which there is a lot to be found out. But in the statement of the Principle given above, I said nothing about complex investigations. I will explain why later on.

The Small Steps Principle is important because it is easy to breach--not only by novices but even experienced investigators. Success can go to a person's head. An accomplished investigator, having met with success after success and perhaps too impatient for the next one, might let his / her guard down and, basing themselves on just one or two clues, start pronouncing how the investigation will turn out. This happens to Sherlock Holmes in The Yellow Face.

The events Sherlock Holmes were asked to investigate in The Yellow Face happened in a village called Norbury. Before he left London Sherlock Holmes already had a 'theory' as to what happened, a theory based on the interview he had with the client, a Mr. Munro. The surprising thing about this theory is that it is full of details but no explanation as to where these details come from.

“The facts, as I read them, are something like this [Sherlock Holmes says to Watson]: This woman was married in America. Her [first] husband developed some hateful qualities, or shall we say he contracted some loathsome disease and became a leper or an imbecile? She flies from him at last, returns to England, changes her name, and starts her life, as she thinks, afresh. She has been married three years [to Mr. Munro, her second husband] and believes that her position is quite secure, having shown her husband the death certificate of some man whose name she has assumed, when suddenly her whereabouts is discovered by her first husband, or, we may suppose, by some unscrupulous woman who has attached herself to the invalid. They write to the wife and threaten to come and expose her. She asks for a hundred pounds and endeavours to buy them off. They come in spite of it, and when the husband mentions casually to the wife that there are newcomers in the cottage, she knows in some way that they are her pursuers. She waits until her husband is asleep, and then she rushes down to endeavour to persuade them to leave her in peace. Having no success, she goes again next morning, and her husband meets her, as he has told us, as she comes out. She promises him then not to go there again, but two days afterwards the hope of getting rid of those dreadful neighbours was too strong for her, and she made another attempt, taking down with her the photograph which had probably been demanded from her. In the midst of this interview the maid rushed in to say that the master had come home, on which the wife, knowing that he would come straight down to the cottage, hurried the inmates out at the back door, into the grove of fir-trees, probably, which was mentioned as standing near. In this way he found the place deserted. I shall be very much surprised, however, if it is still so when he reconnoitres it this evening. What do you think of my theory?”

--The Yellow Face

Watson was surprised that his friend could have inferred so much from the interview and moreover be so certain about it. To the question, 'What do you think of my theory?' Watson had to answer the only way he could: “It is all surmise.”

Sherlock Holmes arrived at his theory right after the interview with his client. He had not yet set foot at Norbury; he had not yet examined 'the scene of the crime'. What did Norbury tell him when he got there? How did events turn out? Was blackmail involved?

At Norbury Sherlock Holmes discovered one of his biggest mistakes ever. In propounding his theory he breached the Small Steps Principle. Instead of small steps he was taking huge leaps.

Not another word did he say of the case until late that night, when he was turning away, with his lighted candle, for his bedroom.
“Watson,” said he, “if it should ever strike you that I am getting a little over-confident in my powers, or giving less pains to a case than it deserves, kindly whisper ‘Norbury’ in my ear, and I shall be infinitely obliged to you.”

--The Yellow Face

Let me now turn back to a point I raised earlier. The Small Steps Principle applies only to complex investigations but in my statement of the Small Steps Principle earlier, I said nothing about complex investigations. I think it is unnecessary because it can be taken as understood. For in simple investigations the only steps we can take are necessarily and in an absolute sense small. We need only two points to reconstruct a straight line. In a ten-letter word in which nine are known it takes one step to fill in the missing letter. In a simple investigation it is not likely that anyone who has any idea about clues would start on a long story after detecting one or two of them.

Even with complex investigations there is an exception to the Small Steps Principle. However, this exception is also well known and therefore can be taken as understood as well. The exception occurs in the case of a clue which represents a unique characteristic. Now the structure may be complex but once we recognize the clue to be the unique characteristic of this structure, we should be able to rattle off the rest of the characteristics / details of this structure if we know of them already. Alfred Hitchcock has a trademark silhouette. Give this to a cinema buff together with the question, who is this person? and the cinema buff will be able to tell you a long, long story.

Blinded-by-Progress Fallacy

In following clues we have to be careful with our claims. Because of the excitement of the chase, because of the surprising results clues sometimes afford, even experienced investigators will sometimes overstate. Now in an investigation when new clues appear it is an occasion for joy. But we should not say for this reason that ALL that we have done in the investigation up to that point must be correct. Such a claim is too broad: it overlooks the possibility of hidden mistakes, the kind we explained in the last post. Some old clues could have been incorrectly interpreted even though new clues appear. When new clues appear we can only say, what we have done so far in the investigation is largely correct.

I want to coin a new term here. If when new clues appear, we say ALL that we have done up to that point is correct, I shall say we have committed the blinded-by-progress fallacy. We are making progress when we manage to develop new clues from old but this progress does not entitle us to conclude that ALL that we have done in the investigation up to that point is correct.

Why largely?

In an investigation when new clues appear, we know we are moving in the right direction; that is, that what we have done so far in the investigation is largely correct. This is what the Right Direction Catechism tells us.

Some will ask, why the 'largely'? Are we hinting there could be mistakes in some of the early steps even though new clues appear?

This is indeed the case. Even though new clues appear, our interpretation of some of the old clues could be wrong. The point to notice here is, the formation of new clues does not always depend on all the old clues. If enough of the old clues have been correctly interpreted new clues could appear; it does not have to be the case that all the old clues have to be correctly interpreted. In the English language the letter X is not used all that often. Suppose in cryptanalyzing a message we make a mistake in deciphering X. Suppose X occurs only once in this message. Now when this is the case, the mistake could easily pass unnoticed. The one time that X occurs we make a mistake in interpreting it. But X does not occur again; it therefore cannot contribute to the formation of new clues. If the investigation nevertheless advances, it will be due to other clues. Even if X had been correctly interpreted, it still would not contribute to the formation of new clues since it occurs only once and does not occur a second time.

Some Comparisons

In making up a story, so long as our imagination does not run out, we could go on and on. In reasoning in mathematics, from a set of premises we could keep drawing conclusion one after another and never stop. But when following clues, if we make serious mistakes we hit a break wall. We cannot go through a brick wall.

The Brick Wall Catechism

The Brick Wall Catechism is a corollary of the Right Direction Catechism. The Right Direction Catechism tells us that when new clues appear, we must be moving in the right direction. Now ask the following: What happens if no new clues appear even though there is plenty of evidence? Answer: we must have made serious mistakes, mistakes serious enough to prevent us from interpreting the evidence properly, the reason why no new clues appear. New clues mean we have done things right. No new clues means we have made major mistakes.

In an investigation, when there are no new clues further gains in the investigation is impossible. When this happens investigators often say they have reached a dead end or hit a brick wall.

The Brick Wall Catechism:

What happens if we hit a Brick Wall?

We must have made serious mistakes.

In simple investigations Brick Walls are relatively easy to deal with: we simply turn back and correct the mistakes that cause them. In complex investigations a Brick Wall could be the occasion for a major crisis. In a complex investigation things can sometimes get murky. Now if we have plenty of evidence and we can make sense of it, new clues should appear. But none has! Could this be our fault? Maybe the new clues are right in front of our eyes except that for some reason we don't see them. True, it is possible that we might have made serious mistakes in the earlier stages but ... where are they?! Mistakes that escaped our attention at the time we made them could escape our attention again when we turn back and look for them.

In a complex investigation a Break Wall could stay with us for a long time, making life miserable. In The Man with the Twisted Lip things were so bad at one point for Sherlock Holmes that he had to ask Watson for help:

“You have a grand gift of silence, Watson .... It makes you quite invaluable as a companion. ‘Pon my word, it is a great thing for me to have someone to talk to, for my own thoughts are not over-pleasant.... It seems absurdly simple, and yet, somehow, I can get nothing to go upon. There’s plenty of thread, no doubt, but I can’t get the end of it into my hand. Now, I’ll state the case clearly and concisely to you, Watson, and maybe you can see a spark where all is dark to me.”

The Right Direction Catechism

In the last post I presented readers with the following cryptogram:

SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI

This cryptogram contains many clues, for example: SB in the first two words, GG in the fourth, THE in the second last (by 'clues' here I mean both genuine and false). Now these are clues we can see right from the beginning. If they are genuine they help us solve the cipher; if they are false they waste our time.

Suppose we follow up on some of these clues and have succeeded in fathoming their meaning (through trial and error). As a result, we are able to advance--that is, are able to decipher parts of the message. Now when this happens sometimes we see things which we did not see at the beginning of the investigation. We see, for example, a three-letter word the first two letters of which are TW, or a five-letter word beginning with WEE.

What are these?

They are NEW CLUES! clues that were not there at the beginning.

What do people say when they see new clues appearing this way?

They frequently say: we must be moving in the right direction!

What do they mean ... 'the right direction'?

They mean, what they have done so far, including their readings of the old clues must be largely right.

Why largely right?

This is easy to answer. If the old clues had been wrongly interpreted, the wrong interpretations could not come together to produce new clues! Notice that the new clues result from applying the interpretations of old clues to the cryptogram. How do we get to have TW at the beginning of a three-letter word? How else if it is not due to some old clues telling us where the Ts and Ws are. Clues are the characteristics of the hidden structure. A clue wrongly interpreted ascribes to the hidden structure a characteristic it does not possess. It is impossible for wrong characteristics to lead to right ones: a crocodile-head cannot suggest a horse-leg.

When we play Sherlock Holmes we are overjoyed, we are delighted, we are jubilant, when old clues lead to new ones. The appearance of new clues in this way tells us we are moving in the right direction.

The Right Direction Catechism:
How do you know when you are moving in the right direction?

When new clues appear.

Two ways to narrow down

SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI
What I have above is a message in cipher which even beginners can solve because there are so many clues. One they are likely to notice right away is furnished by the first two letters: SB. It should be easy to see there are a number of ways of interpreting these two letters. For example, we could think of SB as standing for SH, or TH, or AR. However, once we start making a few trials in deciphering the rest of this message we will soon discover that only one of these possibilities will work. Interestingly enough, this will happen long before we finish deciphering the whole message! This is to say, long before the end the possible interpretations of this particular clue will have been narrowed down to one.

Now this is a common phenomenon, as readers can attest from their own experience. In an investigation we do not have to wait till the end to have everything pegged down. As the investigation proceeds, it will become clearer and clearer what the earlier clues should mean. There is a simple reason for this. Our interpretation of any clue has to jibe with our interpretation of the other clues. If we start off the investigation with a wrong interpretation, after a few steps things will start looking awry because the wrong interpretation will not agree with the other clues, at which point we will have to backtrack and try a different interpretation.

If the structure we are trying to reconstruct is a horse we cannot start by giving it a crocodile's head.

In our example above SB at the beginning has many possible interpretations. However, if we make the effort to try to decipher this message, we will find that after a few steps the number of possible interpretations will be narrowed down to one. This is done through trial and error, as we have said. So here we have an example of the narrowing-down process which is common in investigations. In this case the narrowing down is achieved through trial and error.

But this is not the only way in which we narrow down. In an earlier post we mentioned Watson's trip to the post-office. Without Watson telling him Sherlock Holmes was able to determine that Watson went there to send a telegram. Understandably, Watson was surprised.

“How, then, did you deduce the telegram?”

“Why, of course I knew that you had not written a letter, since I sat opposite to you all morning. I see also in your open desk there that you have a sheet of stamps and a thick bundle of postcards. What could you go into the post-office for, then, but to send a wire? Eliminate all other factors, and the one which remains must be the truth.”

--The Sign of Four

In popular parlance what Sherlock Holmes is doing here is engage in a process of elimination. There are only three things Watson could do at the post-office: buy stamps, buy postcards, or send telegram. Sherlock Holmes narrows these three things down to one: Watson went to the post-office to send a telegram.

The process of elimination is a well-known technique among all those who follow clues but it is different from the SB example we explained earlier. In the SB example we land on the truth through trial and error: the true interpretation will fit in with the other clues and by doing so enable us to advance the investigation. In a process of elimination we have no direct support for the true answer; the only thing that tells us it is true is that all the other answers are false.

Two remarks I want to enter before we close, one for each kind of narrowing down.

When we plan on using trial and error to narrow down, sometimes we can be lucky enough to have picked up the right candidate at the very first trial so that there is no need to test the others. In this case there is only one trial and no error. Inasmuch as we would congratulate ourselves when this happens, I think I can assume it does not happen all that often.

Now the other remark, this one in connection with the process of elimination .... In employing this process we first list all the possibilities, then eliminate them one by one in the hope that only one will remain. When this goes as expected we have no complaints. But sometimes there are surprises. Sometimes we manage to eliminate all the possibilities!

We shall have more to say about both ways of narrowing down by and by.

Do criminals have to confess?

Some people think, unless criminals confess, we can have no idea what happened during a crime; we can have all sorts of theories but they will remain just theories; we can argue about these theories till the cows come home and still we would not know if any of them is true. The only way to know, according to these people, is if the criminals confess.

But people have been known to confess willingly to crimes they have not committed (as in the case of John Mark Karr confessing to the murder of Jonbenet Ramsey). When confessions are extracted through torture we have reason to doubt their veracity. But when they are offered voluntarily, are we to accept them at face value?

Crimes are sometimes complicated. More than one person could be involved. At the time the crime is occurring the participants could be at different places, unobservable by each other. In a case like this, can any of the criminals have a full picture of what happened? True, they will have a plan but will everyone be adhering to it? Criminals have a habit of doublecrossing each other. In the presence of such a tendency can we trust the confessions they provide?

How do we know whether a confession is true? How do we know what happened in a complicated case?

We can only find out by carrying out investigations--that is, by playing Sherlock Holmes.

Sherlock Holmes does not wait for criminals to confess; rather he sometimes uses what he has found out to induce criminals to confess:

“What do you wish me to do?”

“To give me a true account of all that happened at the Abbey Grange last night–-a true account, mind you, with nothing added and nothing taken off. I know so much already that if you go one inch off the straight, I’ll blow this police whistle from my window and the affair goes out of my hands forever.”

--The Abbey Grange

If we want to follow Sherlock Holmes in offering this kind of inducement we had better make sure what we have found out is largely true. Criminals are not all stupid; bluffing will work sometimes but not always.

A question arises. If Sherlock Holmes knows so much already, why does he ask for confessions? Just to tie up the loose ends? What if, in tidying up these loose ends, the criminal starts lying?

Good detectives welcome confessions and Sherlock Holmes is a good detective. Confessions voluntarily given are valuable not because they are invariably true but because sometimes they lead us to clues we have missed, thereby improving our reconstruction of what happened.

Take what criminals say always with a grain of salt, including confessions. They do not run our investigation; we do.

Correction

In an earlier post I said the following:

Clues are the characteristics of structures but we do not gather all the characteristics first and then rebuild the structure. We do not ask, what are the characteristics of this crime? Then, after finding all of them, start to reconstruct the crime. Following clues is a narrowing-down process; we look for clues as we go along, to help us narrow down.

What I have written above is not quite correct. Following clues is not always a narrowing-down process. Whether it is or not depends on the structures we are trying to reconstruct and the number and kinds of clues available. For example, suppose all parts of a straight line have faded away through the ravages of time except for two faint points. Do we need to go through a narrowing-down process to reconstruct the original line? No; all we need do is draw a line through those two points: the line drawn will correspond to the original line.

In crime detection, situations detectives face can sometimes come close to our example: all the clues needed are there and they are so obvious that it is easy to reconstruct what happened--no need to go through any narrowing-down process. They even have a term for it; they call a crime like this an open-and-shut case. (We have to be careful, however; sometimes an open-and-shut case could turn out to be more complicated than originally thought.)

In playing Sherlock Holmes sometimes we have to go through narrowing-down processes, sometimes we do not. Some investigations are complex; some, simple or simpler.

From our line example above we see also that sometimes all the clues we need can be collected first, before we carry out the reconstruction.

Unbreakable Ciphers

If we use ciphers and these ciphers leave behind clues they will be broken. Do we have ciphers that do not leave behind clues? Do we, in other words, have unbreakable ciphers?

Cryptologists have understood for a long time where clues come from, so yes there are unbreakable ciphers. These are ciphers that are not only practically unbreakable but theoretically unbreakable. When cryptologists talk about unbreakable ciphers without qualification they usually mean theoretically unbreakable.

A quick way to understand how we can construct an unbreakable cipher is the following. Encrypt a random string of alphabets using a simple substitution cipher (that is, anything resembling the Dancing Men Cipher). Examine the ciphertext thus obtained for clues. You will find that however hard you look, you will not discover any. Why? Because the ciphertext is just another random string. Clues are the characteristics of structures. A random string has no structure. A string without structure to begin with cannot acquire the structure of English or any other language after encipherment.

But we do not send each other random strings to communicate. When we send messages the cleartext is not a random string. However, if we can do something to the cipher so that it will always generate a random string as the ciphertext whether the cleartext is random or not, this cipher will be unbreakable--because there will be no clues.

How can we construct a cipher that will always generate random strings?

Simple! Just do the following. In encrypting your message change your cipher after every letter in a random way. If you proceed in this fashion your opponents, even if they know how you encrypt the first letter in your message, will have no way of knowing how you encrypt the second, and the third, and the fourth and so on. Needless to say, a cipher like this is very clumsy. For one thing, the intended receiver of your messages will have to know in advance the actual sequence of ciphers you use (which cipher for the first letter, which for the second, which for the third, and so on). Because they are clumsy unbreakable ciphers were not used all that often in the old days. But nowadays we have computers. What is clumsy for human beings is not clumsy for computers. Nowadays, theoretically unbreakable ciphers are more common than before.

Not so elementary, my dear Watson!

Sherlock Holmes sometimes speaks as though no genius is required in solving crimes, that it is just a matter of applying knowledge already in one's possession:

... I have a trade of my own. I suppose I am the only one in the world. I’m a consulting detective, if you can understand what that is. Here in London we have lots of government detectives and lots of private ones. When these fellows are at fault, they come to me, and I manage to put them on the right scent. They lay all the evidence before me, and I am generally able, by the help of my knowledge of the history of crime, to set them straight. There is a strong family resemblance about misdeeds, and if you have all the details of a thousand at your finger ends, it is odd if you can’t unravel the thousand and first ...

--A Study in Scarlet

The point Sherlock Holmes seems to be making here is that solving crimes is merely the application of knowledge already in our possession. He seems to have overlooked the fact that sometimes the knowledge required to locate clues and to make sense of them has yet to be discovered. Kasiski discovered the one weakness that all polyalphabetic substitution ciphers share. Before his discovery polyalphabetic substitution ciphers were thought to be unbreakable.

In Black Peter, to determine who the murderer was, Sherlock Holmes first had to make good a piece of knowledge he found he lacked. He spent a whole morning driving a harpoon again and again into a pig carcass to see what it was like. As a result he came to the right conclusion as to who killed Black Peter. This is to say, if Sherlock Holmes had been more careful he also would say that in following clues we cannot always rely on knowledge we already have; sometimes we need more; sometimes we even have to find that missing piece of knowledge for ourselves, doing so as a sidetrip to the main investigation.




(See also 'Is Sherlock Holmes a genius? and 'Yes Sherlock Holmes is a genius!'.)

How does one thief catch another?

Newspapers in Canada reported a few days ago that the RCMP had uncovered an identity-theft ring in Surrey, BC. Hundreds of stolen credit cards, licenses, records and pieces of mail were recovered, as well as Canada Post uniforms and Canada Post mailbox keys. Apparently, this ring is not a garden variety.

"While it is two weeks since the search warrant was executed..., we've got a long road ahead of us to try and figure out what is taking place," said Sergeant Roger Morrow of the Surrey RCMP.

--The Globe and Mail, February 29, 2008

How did these thieves amass such a quantity of stolen material? Did they steal all these many items themselves or did they buy them from other thieves? How did they come by these Canada Post uniforms? What did they intend to do with all these stolen goods? Use them themselves? Or sell them to other thieves? How in general does this particular group of thieves operate?

Laypeople reading this news item are not likely to be able to answer these questions and others like them. They may have guesses and suspicions but these will remain just guesses and suspicions--until they find out more in later news reports.

The RCMP may have the answers to some of these questions but, as they themselves admit, there is a lot they do not yet know.

Now suppose we are able to get hold of a thief who at one time was in the same line of business, who is willing to talk. Such a person is likely to have a better idea what the answers to our questions are. More particularly, he will be able to give us a general idea how groups like the one caught operate. Now general ideas of this sort are valuable when we are playing Sherlock Holmes: if we have a general idea, we know what details / clues to look for.

Will our source the former thief be right on every point?

Not necessarily since where this particular case is concerned he is as much an outsider as we are. Indeed, he could even be wrong about the general idea governing the way this group works. Things might have changed; this group could be cleverer than all their predeccessors!

Einstein says, God does not play dice. He is wrong in this particular case. In this particular case God is cleverer than Einstein!

Our source the former thief could be wrong on the general idea governing the way the Surrey ring works. Suppose he is in fact wrong; suppose the RCMP cannot find the clues this wrong general idea would lead us to find. So we go back to our source the former thief and this is what he says to us: 'I am sorry you did not find the clues you were looking for. I had warned you about this. Things could change, I said. In fact, I was toying with a new idea just before I left the business.' And he tells us his New Idea.

Could the Surrey ring be following the same New Idea? They might or they might not. But if they were not, we could go back to our source the former thief to see if he has any more new ideas. You never know, one day one of his new ideas might work!

Set a thief to catch a thief. Just to be safe, set an inventive thief, a thief that has new ideas when old ones have been found not to work.

(See also Yes Sherlock Holmes is a genius!)

Yes Sherlock Holmes is a genius!

There is a simple answer to our question from yesterday. The question was, does one have to be a genius to reconstruct a hidden structure, such as a crime? The question arises because it seems one does not. Detecting clues and finding out what they mean depend on knowledge we already have. If one has all the knowledge required to read the clues one should be able to recreate the structure from which the clues flow. From this point of view, following clues is nothing more than applying knowledge we already have to the solution of a practical problem. No genius is required in this kind of activity.

What is our answer to this question?

The answer is the following.

Let's start with ciphers first. It is clear that we do not have to be a genius to crack a cipher if we know already what clues to watch out for. But what about a new type of cipher? When we meet with a new type of cipher, will we know what clues to watch out for? Polyalphabetic substitution ciphers were invented in the 15th Century. It had a reputation of being unbreakable for some 400 years until a general solution was discovered by Kasiski in the 19th. Why did it take so long? Why is a new type of cipher sometimes so hard to break? Why do we still remember Kasiski?

As everyone knows, a cipher hides the characteristics of a language as it is ordinarily used. Sometimes a new cipher hides these characteristics so well that we do not know where the clues are, if any. Clues are the characteristics of structures, disguised. A disguise can be so heavy that, unless we have special knowledge, we will never know what is behind it. And this is where the work of a genius comes in. A cryptanalyst (codebreaker) who wants to break a new type of cipher will have to study the cipher to see where its weak points are. A weak point is where the disguise the cipher puts up is thin enough to be penetrated. Armed with this knowledge a cryptanalyst can then crack any cipher of the same or a similar type.

Cracking a new type of cipher sometimes requires knowledge we do not yet have. It may take a genius to provide us with this missing knowledge.

Now to crimes. Crimes of a known type do not need a Sherlock Holmes. Crimes of a new type can defeat a Sherlock Holmes ... unless he can figure out where its weak points are. But here Sherlock Holmes is at a disadvantage when compared to a cryptanalyst. Take Kasiski as example. Kasiski knows how polyalphabetic substitution works. He knows how to construct a cipher of this type and use it to encrypt messages. What he wants to find out is, when given a message encrypted by a polyalphabetic substitution cipher, how he could work out the cleartext on his own without having been given that particular cipher. But in the case of a novel kind of crime we don't know how the crime 'works': we have never committed it ourselves nor do we know of anyone who has. If we have committed it ourselves or know of someone who has, we can study the crime from the inside and find out where the weak points are. Lacking the knowledge that such a study will provide, how can we solve the crime? Here we need Sherlock Holmes. He is a consulting detective. He specializes in crimes that other people cannot solve.

How does Sherlock Holmes do it? How can we solve a new type of crime?

The answer is part of popular culture: Set a thief to catch a thief! Put yourself in the place of the criminal and see how you would carry out the same crime. It takes a clever criminal to catch another clever criminal: Sherlock Holmes and Professor Moriarty--they are of the same mind, just working on different sides of the law.

I have always had an idea that I would have made a highly efficient criminal.

--Sherlock Holmes in Charles Augustus Milverton

Einstein said, ‘God does not play dice.’ Bohr is reputed to have replied, ‘Don’t tell God what to do!’ Was Einstein in the habit of talking to God?

Is Sherlock Holmes a genius?

To decipher the messages from the Dancing Men Cipher Sherlock Holmes looked first for the most frequently occurring letter in these messages. He took this step because he knew that E is the most often used letter in the English alphabet.

When examining a crime scene one of the first things Sherlock Holmes does is to see if he could find any footprints. Footprints, he knows, can tell us a lot, including a person's height.

Detecting clues and finding out what they mean depends on knowledge already in our possession. If we did not know what footprints can tell us we would not look for footprints. If we did not know that E is the most frequently occurring letter we would not look for the most frequent Dancing Man. Now if we find enough clues we can reconstruct that structure which leave behind these clues, be that structure a crime, a secret message, or whatever. But if this is the case, Sherlock Holmes does not have to be a genius; anybody can do what he does; all we need do is make sure we know the same things as Sherlock Holmes. Take the Dancing Men Cipher again as example. Sherlock Holmes has shown us what we need to know to solve this cipher. Since he has already done this, next time we come across a similar cipher we should be able to solve it on our own. Is this not so?

All I want to know is who done it!

'Mr. Sherlock Holmes, your time is precious and my funds are limited. Instead of finding out what the butler was doing, and the maid, and the footman, and all the other people, why don't you simply concentrate on the murderer and find out who he is. That's all I want to know. For that I am willing to pay but not for anything having to do with the other people.'

Will Sherlock Holmes take on a commission like this?

Yet people often say, why don't scientists concentrate on discoveries that are practically useful and forget all the rest?

In solving the Dancing Men Cipher Sherlock Holmes was furnished with five messages from that cipher. Can we ask him, 'Don't bother with the other four; just tell me what the last one says.'?

(This is a follow-up on something I said in the last post.)

Why are some clues more specific than others?

In crime investigations fingerprints and DNA samples can provide important clues. Since no two people have the same set of fingerprints nor exactly the same DNA they identify individuals unambiguously. On the other hand, if a witness of a crime says the person running away has two arms and two legs, no one will marvel except perhaps at the naivete of this particular witness. Most people have two arms and two legs. Knowing that the suspect has two arms and two legs will not help us locate this person.

Why are some clues more specific than others? DNA is a specific clue; having two arms and two legs is not. (Indeed, having two arms and two legs is so useless as a clue that we might not want to call it a clue at all!) Now clues are the characteristics of structures, so one way to answer our question is to say, a clue is more specific if it is a characteristic found in a smaller number of structures. A particular DNA is the characteristic of just one person. Having two arms and two legs is a characteristic common to most human beings. DNA is a more specific clue than having two arms and two legs.

But this does not seem to be totally satisfactory. Consider the following. A murder investigation is nearing an end. Only two suspects are left: all the others have been ruled out. Of the two left only one is responsible for the crime; the other is innocent. So, who is the murderer? Which of the two?

Naturally, the thing to do at this point is look for more clues. Unfortunately in this case, none can be found. At the moment, everyone is still scratching their heads .... Then someone remembers: the person running away from the scene of the crime has two arms and two legs!

This wraps up the case. Why? Because of the two remaining suspects, one has lost a leg in an accident not so long ago.

Ordinarily, having two arms and two legs is not a specific clue but situations can arise in which it can become highly specific. In our example it is so specific that it points out who the guilty party is.

We have to re-think our question why some clues are more specific than others. We cannot say any more a clue is more specific if it is a characteristic shared by fewer structures. Most people have two arms and two legs; yet in our example having two arms and two legs became a significant clue!

It seems how specific a clue is depends on what stage of an investigation we have arrived at. In our example, at the beginning of the investigation having two arms and two legs is far from specific. Close to the end, it becomes highly specific. Why? What is going on?

I think what goes on is the following. We follow clues to reconstruct structures. Clues are the characteristics of structures but we do not gather all the characteristics first and then rebuild the structure. We do not ask, what are the characteristics of this crime? Then, after finding all of them, start to reconstruct the crime. Following clues is a narrowing-down process; we look for clues as we go along, to help us narrow down. In a murder investigation we do not start by suspecting everybody in the universe; we suspect only a tiny portion. As the investigation continues, as more clues are found, the number of suspects is whittled down until at the end the culprit is caught. But this is not yet the whole story. In order to pin down the murderer; in order to reconstruct the crime; we have to pin down the other players as well. Who are they? Where can we find them? What were they doing? What can they tell us? Are they telling the truth? In an investigation there are many things to pin down. A crime has many details (characteristics). The more of these details we can nail down, the clearer we will be as to who is responsible. Clues may not help us nail down all these details right away but they should at least help us converge on them.

What do we mean by 'converge on them'?

We mean having fewer interpretations ... We are converging on the things we want to know if our clues have fewer and fewer interpretations. If a clue tells us that the butler has to be in the room when the murder happened, we say this clue has nailed down the location of the butler: the clue has at this point only one interpretation. But sometimes a clue is not this specific. For example, it may only tell us that the butler is either in the room or the one next door. Clues are open to interpretations. The fewer interpretations a clue is open to, the closer it brings us to the truth. The more specific clues therefore are those that do not allow for much room in our interpretation. We have to remember, a clue is a characteristic of a structure, disguised. When the disguise is light it is easy to tell what it means. But when the disguise is heavy more than one interpretation of the clue will be possible. A characteristic under heavy disguise is a vague clue; a characteristic thinly disguised is more specific. But disguise is gradually removed as the investigation advances (things become clearer and clearer). At the beginning of an investigation knowing that the murderer has two arms and two legs would not help us but by the time the number of suspects is narrowed down to two this originally vague clue becomes highly specific.

Some clues are easy to notice ...

In A Study in Scarlet a man was killed in an unoccupied house. In the room in which the body was found there were five letters written in blood on the wall:

Rache

In this case we have a clue that is hard for anyone to miss. Of course, what it means is quite a different matter.

But not all clues jump out at us in this fashion. In cracking the Dancing Men Cipher, to find out which Dancing Man stands for the letter E Sherlock Holmes had to do a little statistical analysis.

The elliptical orbits of the planets around the sun was an important clue for Newton. It took astronomers centuries to extract this particular clue from the large body of observational data that had been collected. Part of the reason for this, as we all know, was that for the longest time they thought the data supported the view that the planets moved in circles around the earth.

Sometimes a clue is in open view and yet not noticed. Sherlock Holmes's professional colleagues from Scotland Yard often miss clues that Sherlock Holmes detects. The same clues are there for everybody to see but because they don't have the same background knowledge as Sherlock Holmes, they don't notice them. Sherlock Holmes can estimate the height of a person from the footprints that person leaves behind. Other detectives, not knowing there is a relation between a person's stride and that person's height, pay no attention to footprints.

Clues--what are they?

By following clues Sherlock Holmes can reconstruct a crime, a secret message, or even--if he works at it and is lucky enough--some law governing the workings of nature. What are they, these things we call clues? Why are they there? Why is it that they can lead us to things which initially we did not know?

Can we call just anything a clue? Criminals know that if they leave clues behind they run the risk of being caught. Can they, before they leave the crime scene, decree all clues to disappear?

Anyone who tries to play Sherlock Holmes will recognize the importance of clues. The word 'clue'--or some equivalent--they know is not a label we can apply to anything we please. Whether something is a clue in a particular case is a question with an objective answer. Indeed detectives distinguish between genuine clues and false clues (the practice at Sherlock Holmes's time was to call a false clue a blind). It is not a pleasant experience mistaking a false clue for genuine.

But what are they? What are clues? Why do they have this ability of leading us on, towards the truth?

Let's try to answer this question by going backwards; let's reason in the following way. Clues help us to reconstruct. To make things easier let us suppose the thing we want to reconstruct is not a crime but an ancient sailing ship. Some authors have mentioned this kind of sailing ship here and there but none has given a detailed account. In fact, so far as we know no one has come across even a single crude drawing. What should we do?

Naturally, the first thing we do is look for more information. We will scour books and museums and even archaeological sites to see what clues we can find, clues that will help us build this sailing ship.

What do we expect these clues to tell us if we should find any?
If we are to succeed in reconstructing this sailing ship, the clues we hope we will find will, eventually, have to tell us as many of the characteristics of this ancient sailing ship as we require. How big are these ships? How much can they hold? Can they go on the open sea or are they confined to shallow waters? We will have many questions like these and enough of them will have to be answered before we can go ahead and start building. This is of course a tall order, the reason why the chance of success in this kind of projects is seldom very high.

To build a sailing ship we need to know enough of its characteristics. A sailing ship is not a pile of timber with a piece of fabric thrown in. A sailing ship has a structure. Each kind of sailing ship has its own kind of structure with its own set characteristics. If we know enough of the characteristics of a sailing ship we can rebuild that kind of sailing ship. In the case of our sailing ship, there is no hope of our ever finding a blueprint. Because it is ancient we can find out its characteristics only through clues. Now the clues will not tell us directly what these characteristics are (because if they will, we will not call them clues) but eventually they will if they are to be useful at all.

So what are clues?

From our example we can see they are simply the characteristics of the structures we want to reconstruct, hidden behind some sort of disguise. We are using the word 'disguise' metaphorically to signify that clues do not tell us automatically what is behind them; we have to figure it out; we have to remove the disguise.

Which of course is usually the difficult part! But difficult or not, this is what clues are. They the characteristics of structures, disguised. This is the reason why by following them we can decipher secret messages. For those who send messages have to use a language. Now each language has its own structure with its own set of characteristics. For example, one of the characteristics of the English language is that of the twenty-six letters in the English alphabet E is the most frequently used. As we all know, this characteristic is often depended upon when deciphering English messages. In cracking the Dancing Men Cipher the first thing Sherlock Holmes did was to see which of the Dancing Men appeared most frequently in the intercepted messages. That Dancing Man is a clue. Behind it is the letter E.

Now just as a language has a structure, so does a crime. When a person is shot she falls. From the way she falls we can estimate the direction from which the shot comes. Thus it is that when Sherlock Holmes investigates a crime he will examine the crime scene carefully to see what clues he can find and basing on his knowledge of the characteristics of the various sorts of things he knows, figure out the meanings of these clues.

Scientists follow clues too. Why do the planets move around the sun in ellipses? Newton was able to work out mathematically that this is a characteristic of a universe governed by the theory of universal gravitation as he has formulated it, together with his three laws of motion.

In playing Sherlock Holmes we are trying to reconstruct structures we cannot see. To this end we have to follow clues because the clues are the characteristics of these structures, even though disguised.

No falsehoods in proofs!

In a mathematical proof we do not touch falsehoods at any point (see Note below for a minor exception). Any time a falsehood enters we are off the straight-and-narrow. In following clues, even when we are successful, the final product may not correspond exactly to the original.

---
Note: In a reductio ad absurdum proof we assume the opposite of the proposition to be proved. By showing that this assumption leads to a contradiction and therefore false, we conclude that the proposition to be proved must be true. In the case of the reductio then, we do touch a falsehood but only as an assumption. Moreover, the point of the reductio is to show how dangerous it is if we have any truckings with falsehoods: they lead to contradictions!

Truth and Impatience

People who think of themselves as serious about the truth are often impatient: they want to get at it right away because what is not true is false and no one in their right mind (they say) would want to touch falsehoods. But following clues is a tedious and even frustrating process: it requires a lot of patience. Even then the final product may not correspond exactly to the original.

Is sherlock Holmes not serious about the truth? Are we not serious about the truth when playing Sherlock Holmes?

Truth vs Partial Correspondence

A proposition is either true or false. A reconstruction obtained by following clues may not correspond exactly to the original, yet could be regarded as satisfactory or even valuable.

Proofs vs Reconstructions

In a mathematical proof we go from truth to truth to truth until we reach the conclusion. In reconstructing a crime, depending on how clues develop, we could go through many versions which differ in important ways from each other. In an earlier version the butler is the main suspect. In a later version he is totally innocent.

What does Sherlock Holmes aim at?

Sherlock Holmes has a method. What is he aiming at when he is exercising this method?

We all play detective sometimes. What should we be aiming at when we are playing detective?

I am asking this question because a method is employed to achieve some aim. If we know the aim we may be able to work out the method ourselves. So the question is, what are we aiming at when we are playing Sherlock Holmes?

Let's try out a few answers to see if any of them fits:

Solving a crime?
This can't be the right answer because, as we have pointed out in the last post, when we play detective we are not always trying to solve a crime: we could be trying to crack a cipher, for example.

Providing a solution to a mystery?
When we are playing Sherlock Holmes don't we always have some mystery that we want to solve? A mystery can be about anything, not necessarily a crime. A string of symbols that apparently doesn't make sense could be intriguing. Perhaps there is message behind it. If there is, can we find out? So we play Sherlock Holmes. Natural phenomena can be puzzling, so we engage in scientific research to find out what is really going on. Is this the right answer to our question then? When we play Sherlock Holmes are we aiming at solving some mystery?

I think the answer has to be no. Take the case of Watson's trip to the post-office: there is no mystery to it. If Sherlock Holmes wants to know where Watson went that morning all he has to do is ask.

The truth?
In solving crimes, in cracking ciphers, in scientific research--in all these endeavors we are aiming at the truth; are we not? In solving crimes we don't want to accuse the wrong person. In cracking ciphers we don't want to replace the true message with something we ourselves make up. In science we want to know what the world is really like; we don't want to deceive ourselves by adhering to superstitions.

Is this the right answer then? In playing Sherlock Holmes our aim is to get at the truth; is this not so?

Strange as it may sound, I don't think it is. Certainly Sherlock Holmes is interested in the truth but then most people are. Journalists get themselves into a habit of double checking their sources; why? To make sure they report the truth. So, yes Sherlock Holmes is interested in the truth but the kind of truth he aims at is more difficult to obtain than the kind that journalists report after double checking sources. Indeed, the kind of truth that Sherlock Holmes aims at is so difficult to obtain that usually no one would blame him if it were missing in some details. As Sherlock Holmes himself says, when he finds out what happens in a crime, usually he only knows in 'essentials', not in every single detail.

Truth of things hidden?
Why is it hard to obtain the kind of truth we seek when playing detective? Watson marvels at what Sherlock Holmes is able do; why? If Sherlock Holmes had accompanied Watson to the post-office, would Watson still marvel?

Clearly, the reason why the kind of truth detectives are interested in is difficult to obtain is that it is about things hidden, things that the detectives themselves cannot see. Now since this is the case, should we not say that our aim when playing Sherlock Holmes should be the truth of those things that we want to know but which are hidden?

This I think will not be an acceptable answer either. For when we play Sherlock Holmes we do not look for just anything that is hidden and try to see what we can find out about them; we look only for things for which we have clues. Not everything that is hidden can be known, only those that leave behind clues.

Perhaps then this is the answer to our question. Our question is, what should Sherlock Holmes aim at? It seems the right answer is, he should aim at truths of things hidden, which we want to know and for which there are clues.

Should we stop here? Should we take what has just been said as the answer?

We could but I suggest we do not. Detectives are practical people. The answer just given is not likely to mean much to them. It is very well to say we should get at the truth but how in practice do we do it when all that we have are clues? So instead, I suggest we adopt the following answer, which is slightly different but has the merit of being easier to understand. In playing Sherlock Holmes we should aim at the reconstruction of those things hidden, which we want to know, basing our reconstruction on clues. Put this way we can see that our answer conforms to actual practice. In practice, when we are trying to solve a crime we are in fact trying to recreate that crime (not by committing it again but in words or through play-acting). When we are cracking a cipher we are trying to recreate that cipher. And in the case of science, we are trying to recreate in symbolic form structures present in the universe. In all these instances, it is clear that we have to rely on clues. The more clues we have, and the more significant these clues, the more accurate and more complete the reconstruction. Once a reconstruction is in place we can tell what is true based on this reconstruction. Is it true that the butler fired the fatal shot? Our reconstruction tells us that it is. Did the butler intend to shoot the victim? We do not know because our reconstruction is incomplete (due to the lack of sufficient clues).

What Sherlock Holmes is able to do often surprises people. Sherlock Holmes wants to know about things hidden. But how can things hidden be known?! To say that they can seems to be a contradiction in terms. However, if we look at what Sherlock Holmes is aiming at slightly differently all the mystery disappears. The reconstruction of things whose originals no longer exist is done all time. It is just a matter of whether we have enough clues. The more clues we have the closer to success we are. Do we need complete success all the time? No, not when it is the truth we are looking for. Partial knowledge is better than complete ignorance.

Does Sherlock Holmes only solve crimes?

We all know that he does not. Using the same method that he applies in crime detection he does other things too. For example, there is this famous instance that Watson recounts about his visit to the post-office, something that he did on the spur of the moment and about which he had told no one. Exercising his Method Sherlock Holmes knew not only that Watson went to the post-office, but which post-office and what he was doing there.

"... you have been to the Wigmore Street Post-Office this morning, ... when there you dispatched a telegram."

"Right!" said I. "Right on both points! But I confess that I don't see how you arrived at it. It was a sudden impulse upon my part, and I have mentioned it to no one."

"It is simplicity itself," he remarked, chuckling at my surprise--"so absurdly simple that an explanation is superfluous; and yet it may serve to define the limits of observation and of deduction. Observation tells me that you have a little reddish mould adhering to your instep. Just opposite the Wigmore Street Office they have taken up the pavement and thrown up some earth, which lies in such a way that it is difficult to avoid treading in it in entering. The earth is of this peculiar reddish tint which is found, as far as I know, nowhere else in the neighbourhood. So much is observation. The rest is deduction." "How, then, did you deduce the telegram?"

"Why, of course I knew that you had not written a letter, since I sat opposite to you all morning. I see also in your open desk there that you have a sheet of stamps and a thick bundle of postcards. What could you go into the post-office for, then, but to send a wire? Eliminate all other factors, and the one which remains must be the truth."

--The Sign of Four

It is well known also that Sherlock Holmes solves ciphers. In The Dancing Men he was able to decipher a number of secret messages. How did he do it? He did it by using the same method that he used in solving crimes.

But Sherlock Holmes not only solves crimes and deciphers secret messages, he also has a strong interest in some of the sciences (including chemistry but not astronomy). What was he doing in his retirement besides tending bees? 'Of late I have been tempted to look into the problems furnished by nature rather than those more superficial ones for which our artificial state of society is responsible,' he says (The Final Problem). To Sherlock Holmes solving problems presented by nature is no different from solving problems presented by society: we use the same method.

From the above we see that solving crimes, tracking the whereabouts of Dr. Watson, cracking ciphers, engaging in scientific research--all these fall within the province of the detective inasmuch as the same method can be employed in all of them. When we are playing Sherlock Holmes it is not necessary that we be only interested in solving crimes. Indeed, whatever we may be interested in, so long as our interest requires that we employ the same method that Sherlock Holmes employs, we are playing Sherlock Holmes.

What this blog is about

Of all detectives Sherlock Holmes is the most famous. When we act like a detective we are often said to be playing Sherlock Holmes. What should we do if we want to play Sherlock Holmes? What should we do if we want to be a good detective? This is what this blog is about.