The Small Steps Principle

Clues allow us to reconstruct hidden structures. When a structure is complex we need many clues. For clues are the characteristics of structures. When a structure is complex, to mark it off from others like it--to mark it off, that is to say, from neighbouring structures--we will need as many of these characteristics / clues as we can. This is to say, in a complex investigation each clue can only give us a small piece of information about the structure which we want to reconstruct--small, that is to say, in relation to the whole. From footprints Sherlock Holmes can estimate a person's height. Now this is useful information but just one small item. He will need many more items, many more clues, if he is to identify the person.

In investigations our steps forward are guided by clues. When the structure we want to reconstruct is complex, to reconstruct this structure we will need many steps, each small. When playing Sherlock Holmes we therefore have to observe a Small Steps Principle: we take only small steps, never huge leaps. Failure to observe this principle will mean failure in the reconstruction.

The name the Small Steps Principle is my creation but its content is recognized in practice. In practice people know there is a limit to how much one can read into a clue. When an investigator exceeds the limit, we say he / she is taking too large a leap. In our cryptanalytic example used in earlier posts (reproduced below), to say SB might mean SH or TH or AR is permissible. But to say that SB alone tells us that the first two words are ARE AREAR is too large a leap.

SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI

The Small Steps Principle applies only to complex investigations, investigations in which there is a lot to be found out. But in the statement of the Principle given above, I said nothing about complex investigations. I will explain why later on.

The Small Steps Principle is important because it is easy to breach--not only by novices but even experienced investigators. Success can go to a person's head. An accomplished investigator, having met with success after success and perhaps too impatient for the next one, might let his / her guard down and, basing themselves on just one or two clues, start pronouncing how the investigation will turn out. This happens to Sherlock Holmes in The Yellow Face.

The events Sherlock Holmes were asked to investigate in The Yellow Face happened in a village called Norbury. Before he left London Sherlock Holmes already had a 'theory' as to what happened, a theory based on the interview he had with the client, a Mr. Munro. The surprising thing about this theory is that it is full of details but no explanation as to where these details come from.

“The facts, as I read them, are something like this [Sherlock Holmes says to Watson]: This woman was married in America. Her [first] husband developed some hateful qualities, or shall we say he contracted some loathsome disease and became a leper or an imbecile? She flies from him at last, returns to England, changes her name, and starts her life, as she thinks, afresh. She has been married three years [to Mr. Munro, her second husband] and believes that her position is quite secure, having shown her husband the death certificate of some man whose name she has assumed, when suddenly her whereabouts is discovered by her first husband, or, we may suppose, by some unscrupulous woman who has attached herself to the invalid. They write to the wife and threaten to come and expose her. She asks for a hundred pounds and endeavours to buy them off. They come in spite of it, and when the husband mentions casually to the wife that there are newcomers in the cottage, she knows in some way that they are her pursuers. She waits until her husband is asleep, and then she rushes down to endeavour to persuade them to leave her in peace. Having no success, she goes again next morning, and her husband meets her, as he has told us, as she comes out. She promises him then not to go there again, but two days afterwards the hope of getting rid of those dreadful neighbours was too strong for her, and she made another attempt, taking down with her the photograph which had probably been demanded from her. In the midst of this interview the maid rushed in to say that the master had come home, on which the wife, knowing that he would come straight down to the cottage, hurried the inmates out at the back door, into the grove of fir-trees, probably, which was mentioned as standing near. In this way he found the place deserted. I shall be very much surprised, however, if it is still so when he reconnoitres it this evening. What do you think of my theory?”

--The Yellow Face

Watson was surprised that his friend could have inferred so much from the interview and moreover be so certain about it. To the question, 'What do you think of my theory?' Watson had to answer the only way he could: “It is all surmise.”

Sherlock Holmes arrived at his theory right after the interview with his client. He had not yet set foot at Norbury; he had not yet examined 'the scene of the crime'. What did Norbury tell him when he got there? How did events turn out? Was blackmail involved?

At Norbury Sherlock Holmes discovered one of his biggest mistakes ever. In propounding his theory he breached the Small Steps Principle. Instead of small steps he was taking huge leaps.

Not another word did he say of the case until late that night, when he was turning away, with his lighted candle, for his bedroom.
“Watson,” said he, “if it should ever strike you that I am getting a little over-confident in my powers, or giving less pains to a case than it deserves, kindly whisper ‘Norbury’ in my ear, and I shall be infinitely obliged to you.”

--The Yellow Face

Let me now turn back to a point I raised earlier. The Small Steps Principle applies only to complex investigations but in my statement of the Small Steps Principle earlier, I said nothing about complex investigations. I think it is unnecessary because it can be taken as understood. For in simple investigations the only steps we can take are necessarily and in an absolute sense small. We need only two points to reconstruct a straight line. In a ten-letter word in which nine are known it takes one step to fill in the missing letter. In a simple investigation it is not likely that anyone who has any idea about clues would start on a long story after detecting one or two of them.

Even with complex investigations there is an exception to the Small Steps Principle. However, this exception is also well known and therefore can be taken as understood as well. The exception occurs in the case of a clue which represents a unique characteristic. Now the structure may be complex but once we recognize the clue to be the unique characteristic of this structure, we should be able to rattle off the rest of the characteristics / details of this structure if we know of them already. Alfred Hitchcock has a trademark silhouette. Give this to a cinema buff together with the question, who is this person? and the cinema buff will be able to tell you a long, long story.

Blinded-by-Progress Fallacy

In following clues we have to be careful with our claims. Because of the excitement of the chase, because of the surprising results clues sometimes afford, even experienced investigators will sometimes overstate. Now in an investigation when new clues appear it is an occasion for joy. But we should not say for this reason that ALL that we have done in the investigation up to that point must be correct. Such a claim is too broad: it overlooks the possibility of hidden mistakes, the kind we explained in the last post. Some old clues could have been incorrectly interpreted even though new clues appear. When new clues appear we can only say, what we have done so far in the investigation is largely correct.

I want to coin a new term here. If when new clues appear, we say ALL that we have done up to that point is correct, I shall say we have committed the blinded-by-progress fallacy. We are making progress when we manage to develop new clues from old but this progress does not entitle us to conclude that ALL that we have done in the investigation up to that point is correct.

Why largely?

In an investigation when new clues appear, we know we are moving in the right direction; that is, that what we have done so far in the investigation is largely correct. This is what the Right Direction Catechism tells us.

Some will ask, why the 'largely'? Are we hinting there could be mistakes in some of the early steps even though new clues appear?

This is indeed the case. Even though new clues appear, our interpretation of some of the old clues could be wrong. The point to notice here is, the formation of new clues does not always depend on all the old clues. If enough of the old clues have been correctly interpreted new clues could appear; it does not have to be the case that all the old clues have to be correctly interpreted. In the English language the letter X is not used all that often. Suppose in cryptanalyzing a message we make a mistake in deciphering X. Suppose X occurs only once in this message. Now when this is the case, the mistake could easily pass unnoticed. The one time that X occurs we make a mistake in interpreting it. But X does not occur again; it therefore cannot contribute to the formation of new clues. If the investigation nevertheless advances, it will be due to other clues. Even if X had been correctly interpreted, it still would not contribute to the formation of new clues since it occurs only once and does not occur a second time.

Some Comparisons

In making up a story, so long as our imagination does not run out, we could go on and on. In reasoning in mathematics, from a set of premises we could keep drawing conclusion one after another and never stop. But when following clues, if we make serious mistakes we hit a break wall. We cannot go through a brick wall.

The Brick Wall Catechism

The Brick Wall Catechism is a corollary of the Right Direction Catechism. The Right Direction Catechism tells us that when new clues appear, we must be moving in the right direction. Now ask the following: What happens if no new clues appear even though there is plenty of evidence? Answer: we must have made serious mistakes, mistakes serious enough to prevent us from interpreting the evidence properly, the reason why no new clues appear. New clues mean we have done things right. No new clues means we have made major mistakes.

In an investigation, when there are no new clues further gains in the investigation is impossible. When this happens investigators often say they have reached a dead end or hit a brick wall.

The Brick Wall Catechism:

What happens if we hit a Brick Wall?

We must have made serious mistakes.

In simple investigations Brick Walls are relatively easy to deal with: we simply turn back and correct the mistakes that cause them. In complex investigations a Brick Wall could be the occasion for a major crisis. In a complex investigation things can sometimes get murky. Now if we have plenty of evidence and we can make sense of it, new clues should appear. But none has! Could this be our fault? Maybe the new clues are right in front of our eyes except that for some reason we don't see them. True, it is possible that we might have made serious mistakes in the earlier stages but ... where are they?! Mistakes that escaped our attention at the time we made them could escape our attention again when we turn back and look for them.

In a complex investigation a Break Wall could stay with us for a long time, making life miserable. In The Man with the Twisted Lip things were so bad at one point for Sherlock Holmes that he had to ask Watson for help:

“You have a grand gift of silence, Watson .... It makes you quite invaluable as a companion. ‘Pon my word, it is a great thing for me to have someone to talk to, for my own thoughts are not over-pleasant.... It seems absurdly simple, and yet, somehow, I can get nothing to go upon. There’s plenty of thread, no doubt, but I can’t get the end of it into my hand. Now, I’ll state the case clearly and concisely to you, Watson, and maybe you can see a spark where all is dark to me.”

The Right Direction Catechism

In the last post I presented readers with the following cryptogram:

SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI

This cryptogram contains many clues, for example: SB in the first two words, GG in the fourth, THE in the second last (by 'clues' here I mean both genuine and false). Now these are clues we can see right from the beginning. If they are genuine they help us solve the cipher; if they are false they waste our time.

Suppose we follow up on some of these clues and have succeeded in fathoming their meaning (through trial and error). As a result, we are able to advance--that is, are able to decipher parts of the message. Now when this happens sometimes we see things which we did not see at the beginning of the investigation. We see, for example, a three-letter word the first two letters of which are TW, or a five-letter word beginning with WEE.

What are these?

They are NEW CLUES! clues that were not there at the beginning.

What do people say when they see new clues appearing this way?

They frequently say: we must be moving in the right direction!

What do they mean ... 'the right direction'?

They mean, what they have done so far, including their readings of the old clues must be largely right.

Why largely right?

This is easy to answer. If the old clues had been wrongly interpreted, the wrong interpretations could not come together to produce new clues! Notice that the new clues result from applying the interpretations of old clues to the cryptogram. How do we get to have TW at the beginning of a three-letter word? How else if it is not due to some old clues telling us where the Ts and Ws are. Clues are the characteristics of the hidden structure. A clue wrongly interpreted ascribes to the hidden structure a characteristic it does not possess. It is impossible for wrong characteristics to lead to right ones: a crocodile-head cannot suggest a horse-leg.

When we play Sherlock Holmes we are overjoyed, we are delighted, we are jubilant, when old clues lead to new ones. The appearance of new clues in this way tells us we are moving in the right direction.

The Right Direction Catechism:
How do you know when you are moving in the right direction?

When new clues appear.

Two ways to narrow down

SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI
What I have above is a message in cipher which even beginners can solve because there are so many clues. One they are likely to notice right away is furnished by the first two letters: SB. It should be easy to see there are a number of ways of interpreting these two letters. For example, we could think of SB as standing for SH, or TH, or AR. However, once we start making a few trials in deciphering the rest of this message we will soon discover that only one of these possibilities will work. Interestingly enough, this will happen long before we finish deciphering the whole message! This is to say, long before the end the possible interpretations of this particular clue will have been narrowed down to one.

Now this is a common phenomenon, as readers can attest from their own experience. In an investigation we do not have to wait till the end to have everything pegged down. As the investigation proceeds, it will become clearer and clearer what the earlier clues should mean. There is a simple reason for this. Our interpretation of any clue has to jibe with our interpretation of the other clues. If we start off the investigation with a wrong interpretation, after a few steps things will start looking awry because the wrong interpretation will not agree with the other clues, at which point we will have to backtrack and try a different interpretation.

If the structure we are trying to reconstruct is a horse we cannot start by giving it a crocodile's head.

In our example above SB at the beginning has many possible interpretations. However, if we make the effort to try to decipher this message, we will find that after a few steps the number of possible interpretations will be narrowed down to one. This is done through trial and error, as we have said. So here we have an example of the narrowing-down process which is common in investigations. In this case the narrowing down is achieved through trial and error.

But this is not the only way in which we narrow down. In an earlier post we mentioned Watson's trip to the post-office. Without Watson telling him Sherlock Holmes was able to determine that Watson went there to send a telegram. Understandably, Watson was surprised.

“How, then, did you deduce the telegram?”

“Why, of course I knew that you had not written a letter, since I sat opposite to you all morning. I see also in your open desk there that you have a sheet of stamps and a thick bundle of postcards. What could you go into the post-office for, then, but to send a wire? Eliminate all other factors, and the one which remains must be the truth.”

--The Sign of Four

In popular parlance what Sherlock Holmes is doing here is engage in a process of elimination. There are only three things Watson could do at the post-office: buy stamps, buy postcards, or send telegram. Sherlock Holmes narrows these three things down to one: Watson went to the post-office to send a telegram.

The process of elimination is a well-known technique among all those who follow clues but it is different from the SB example we explained earlier. In the SB example we land on the truth through trial and error: the true interpretation will fit in with the other clues and by doing so enable us to advance the investigation. In a process of elimination we have no direct support for the true answer; the only thing that tells us it is true is that all the other answers are false.

Two remarks I want to enter before we close, one for each kind of narrowing down.

When we plan on using trial and error to narrow down, sometimes we can be lucky enough to have picked up the right candidate at the very first trial so that there is no need to test the others. In this case there is only one trial and no error. Inasmuch as we would congratulate ourselves when this happens, I think I can assume it does not happen all that often.

Now the other remark, this one in connection with the process of elimination .... In employing this process we first list all the possibilities, then eliminate them one by one in the hope that only one will remain. When this goes as expected we have no complaints. But sometimes there are surprises. Sometimes we manage to eliminate all the possibilities!

We shall have more to say about both ways of narrowing down by and by.

Do criminals have to confess?

Some people think, unless criminals confess, we can have no idea what happened during a crime; we can have all sorts of theories but they will remain just theories; we can argue about these theories till the cows come home and still we would not know if any of them is true. The only way to know, according to these people, is if the criminals confess.

But people have been known to confess willingly to crimes they have not committed (as in the case of John Mark Karr confessing to the murder of Jonbenet Ramsey). When confessions are extracted through torture we have reason to doubt their veracity. But when they are offered voluntarily, are we to accept them at face value?

Crimes are sometimes complicated. More than one person could be involved. At the time the crime is occurring the participants could be at different places, unobservable by each other. In a case like this, can any of the criminals have a full picture of what happened? True, they will have a plan but will everyone be adhering to it? Criminals have a habit of doublecrossing each other. In the presence of such a tendency can we trust the confessions they provide?

How do we know whether a confession is true? How do we know what happened in a complicated case?

We can only find out by carrying out investigations--that is, by playing Sherlock Holmes.

Sherlock Holmes does not wait for criminals to confess; rather he sometimes uses what he has found out to induce criminals to confess:

“What do you wish me to do?”

“To give me a true account of all that happened at the Abbey Grange last night–-a true account, mind you, with nothing added and nothing taken off. I know so much already that if you go one inch off the straight, I’ll blow this police whistle from my window and the affair goes out of my hands forever.”

--The Abbey Grange

If we want to follow Sherlock Holmes in offering this kind of inducement we had better make sure what we have found out is largely true. Criminals are not all stupid; bluffing will work sometimes but not always.

A question arises. If Sherlock Holmes knows so much already, why does he ask for confessions? Just to tie up the loose ends? What if, in tidying up these loose ends, the criminal starts lying?

Good detectives welcome confessions and Sherlock Holmes is a good detective. Confessions voluntarily given are valuable not because they are invariably true but because sometimes they lead us to clues we have missed, thereby improving our reconstruction of what happened.

Take what criminals say always with a grain of salt, including confessions. They do not run our investigation; we do.

Correction

In an earlier post I said the following:

Clues are the characteristics of structures but we do not gather all the characteristics first and then rebuild the structure. We do not ask, what are the characteristics of this crime? Then, after finding all of them, start to reconstruct the crime. Following clues is a narrowing-down process; we look for clues as we go along, to help us narrow down.

What I have written above is not quite correct. Following clues is not always a narrowing-down process. Whether it is or not depends on the structures we are trying to reconstruct and the number and kinds of clues available. For example, suppose all parts of a straight line have faded away through the ravages of time except for two faint points. Do we need to go through a narrowing-down process to reconstruct the original line? No; all we need do is draw a line through those two points: the line drawn will correspond to the original line.

In crime detection, situations detectives face can sometimes come close to our example: all the clues needed are there and they are so obvious that it is easy to reconstruct what happened--no need to go through any narrowing-down process. They even have a term for it; they call a crime like this an open-and-shut case. (We have to be careful, however; sometimes an open-and-shut case could turn out to be more complicated than originally thought.)

In playing Sherlock Holmes sometimes we have to go through narrowing-down processes, sometimes we do not. Some investigations are complex; some, simple or simpler.

From our line example above we see also that sometimes all the clues we need can be collected first, before we carry out the reconstruction.

Unbreakable Ciphers

If we use ciphers and these ciphers leave behind clues they will be broken. Do we have ciphers that do not leave behind clues? Do we, in other words, have unbreakable ciphers?

Cryptologists have understood for a long time where clues come from, so yes there are unbreakable ciphers. These are ciphers that are not only practically unbreakable but theoretically unbreakable. When cryptologists talk about unbreakable ciphers without qualification they usually mean theoretically unbreakable.

A quick way to understand how we can construct an unbreakable cipher is the following. Encrypt a random string of alphabets using a simple substitution cipher (that is, anything resembling the Dancing Men Cipher). Examine the ciphertext thus obtained for clues. You will find that however hard you look, you will not discover any. Why? Because the ciphertext is just another random string. Clues are the characteristics of structures. A random string has no structure. A string without structure to begin with cannot acquire the structure of English or any other language after encipherment.

But we do not send each other random strings to communicate. When we send messages the cleartext is not a random string. However, if we can do something to the cipher so that it will always generate a random string as the ciphertext whether the cleartext is random or not, this cipher will be unbreakable--because there will be no clues.

How can we construct a cipher that will always generate random strings?

Simple! Just do the following. In encrypting your message change your cipher after every letter in a random way. If you proceed in this fashion your opponents, even if they know how you encrypt the first letter in your message, will have no way of knowing how you encrypt the second, and the third, and the fourth and so on. Needless to say, a cipher like this is very clumsy. For one thing, the intended receiver of your messages will have to know in advance the actual sequence of ciphers you use (which cipher for the first letter, which for the second, which for the third, and so on). Because they are clumsy unbreakable ciphers were not used all that often in the old days. But nowadays we have computers. What is clumsy for human beings is not clumsy for computers. Nowadays, theoretically unbreakable ciphers are more common than before.

Not so elementary, my dear Watson!

Sherlock Holmes sometimes speaks as though no genius is required in solving crimes, that it is just a matter of applying knowledge already in one's possession:

... I have a trade of my own. I suppose I am the only one in the world. I’m a consulting detective, if you can understand what that is. Here in London we have lots of government detectives and lots of private ones. When these fellows are at fault, they come to me, and I manage to put them on the right scent. They lay all the evidence before me, and I am generally able, by the help of my knowledge of the history of crime, to set them straight. There is a strong family resemblance about misdeeds, and if you have all the details of a thousand at your finger ends, it is odd if you can’t unravel the thousand and first ...

--A Study in Scarlet

The point Sherlock Holmes seems to be making here is that solving crimes is merely the application of knowledge already in our possession. He seems to have overlooked the fact that sometimes the knowledge required to locate clues and to make sense of them has yet to be discovered. Kasiski discovered the one weakness that all polyalphabetic substitution ciphers share. Before his discovery polyalphabetic substitution ciphers were thought to be unbreakable.

In Black Peter, to determine who the murderer was, Sherlock Holmes first had to make good a piece of knowledge he found he lacked. He spent a whole morning driving a harpoon again and again into a pig carcass to see what it was like. As a result he came to the right conclusion as to who killed Black Peter. This is to say, if Sherlock Holmes had been more careful he also would say that in following clues we cannot always rely on knowledge we already have; sometimes we need more; sometimes we even have to find that missing piece of knowledge for ourselves, doing so as a sidetrip to the main investigation.




(See also 'Is Sherlock Holmes a genius? and 'Yes Sherlock Holmes is a genius!'.)

How does one thief catch another?

Newspapers in Canada reported a few days ago that the RCMP had uncovered an identity-theft ring in Surrey, BC. Hundreds of stolen credit cards, licenses, records and pieces of mail were recovered, as well as Canada Post uniforms and Canada Post mailbox keys. Apparently, this ring is not a garden variety.

"While it is two weeks since the search warrant was executed..., we've got a long road ahead of us to try and figure out what is taking place," said Sergeant Roger Morrow of the Surrey RCMP.

--The Globe and Mail, February 29, 2008

How did these thieves amass such a quantity of stolen material? Did they steal all these many items themselves or did they buy them from other thieves? How did they come by these Canada Post uniforms? What did they intend to do with all these stolen goods? Use them themselves? Or sell them to other thieves? How in general does this particular group of thieves operate?

Laypeople reading this news item are not likely to be able to answer these questions and others like them. They may have guesses and suspicions but these will remain just guesses and suspicions--until they find out more in later news reports.

The RCMP may have the answers to some of these questions but, as they themselves admit, there is a lot they do not yet know.

Now suppose we are able to get hold of a thief who at one time was in the same line of business, who is willing to talk. Such a person is likely to have a better idea what the answers to our questions are. More particularly, he will be able to give us a general idea how groups like the one caught operate. Now general ideas of this sort are valuable when we are playing Sherlock Holmes: if we have a general idea, we know what details / clues to look for.

Will our source the former thief be right on every point?

Not necessarily since where this particular case is concerned he is as much an outsider as we are. Indeed, he could even be wrong about the general idea governing the way this group works. Things might have changed; this group could be cleverer than all their predeccessors!

Einstein says, God does not play dice. He is wrong in this particular case. In this particular case God is cleverer than Einstein!

Our source the former thief could be wrong on the general idea governing the way the Surrey ring works. Suppose he is in fact wrong; suppose the RCMP cannot find the clues this wrong general idea would lead us to find. So we go back to our source the former thief and this is what he says to us: 'I am sorry you did not find the clues you were looking for. I had warned you about this. Things could change, I said. In fact, I was toying with a new idea just before I left the business.' And he tells us his New Idea.

Could the Surrey ring be following the same New Idea? They might or they might not. But if they were not, we could go back to our source the former thief to see if he has any more new ideas. You never know, one day one of his new ideas might work!

Set a thief to catch a thief. Just to be safe, set an inventive thief, a thief that has new ideas when old ones have been found not to work.

(See also Yes Sherlock Holmes is a genius!)