DNA statistical evidence and the prosecutor's fallacy
47 But there are real dangers in misunderstanding the statistics in this area (for some empirical evidence of this, see Thompson and Schumann, "Interpretation of Statistical Evidence in Criminal Trials: The Prosecutor's Fallacy and the Defence Attorney's Fallacy" (1986) 10 Law & Human Behaviour 167).
48 In Doheny and Adams Phillips LJ (as he then was) said this (at 372-3):
"The Prosecutor's Fallacy"
It is easy, if one eschews rigorous analysis, to draw the following conclusion:
1. Only one person in a million will have a DNA profile which matches that of the crime stain.
2. The defendant has a DNA profile which matches the crime stain.
3. Ergo there is a million to one probability that the defendant left the crime stain and is guilty of the crime.
Such reasoning has been commended to juries in a number of cases by prosecuting counsel, by judges and sometimes by expert witnesses. It is fallacious and it has earned the title of "The Prosecutor's Fallacy". … The nature of that fallacy was elegantly exposed by Balding and Donnelly in "The Prosecutor's Fallacy and DNA Evidence" [1994] Crim LR 711. It should not, however, be thought that we endorse the calculations on pp715 and 716 of that article.
Taking our example, the prosecutor's fallacy can be simply demonstrated. If one person in a million has a DNA profile which matches that obtained from the crime stain, then the suspect will be 1 of perhaps 26 men in the United Kingdom who share that characteristic. If no fact is known about the Defendant, other than that he was in the United Kingdom at the time of the crime the DNA evidence tells us no more than that there is a statistical probability that he was the criminal of 1 in 26.
The significance of the DNA evidence will depend critically upon what else is known about the suspect. If he has a convincing alibi at the other end of England at the time of the crime, it will appear highly improbable that he can have been responsible for the crime, despite his matching DNA profile. If, however, he was near the scene of the crime when it was committed, or has been identified as a suspect because of other evidence which suggests that he may have been responsible for the crime, the DNA evidence becomes more significant. The possibility that two of the only 26 men in the United Kingdom with the matching DNA should have been in the vicinity of the crime will seem almost incredible and a comparatively slight nexus between the defendant and the crime, independent of the DNA, is likely to suffice to present an overall picture to the jury that satisfies them of the defendant's guilt.
The reality is that, provided there is no reason to doubt either the matching data or the statistical conclusion based upon it, the random occurrence ratio reduced from the DNA evidence, when combined with sufficient additional evidence to give it significance, is highly probative. As the art of analysis progresses, it is likely to become more so, and the stage may be reached when a match will be so comprehensive that it will be possible to construct a DNA profile that is unique and which proves the guilt of the defendant without any other evidence. So far as we are aware that stage has not yet been reached.
49 In Latcha (1998) 104 A Crim R 390 the Court of Appeal of the Northern Territory held that duly proved and scientifically valid statistical evidence could be given about the "likelihood ratio" in a relevant population of a DNA match. (The Court was addressing a case involving a match between sperm taken from the panties of a sexual assault victim and the accused man's blood.) However, the Court emphasised the need for care lest the jury confuse two questions:
What is the probability of obtaining a matching analysis of the crime scene sample if someone else (other than the accused) left it? (proper)
and
What is the likelihood that it was the defendant's DNA found at the crime scene? (improper).
50 See also Milat (1996) 87 A Crim R 446 at 451, Gibson v R [2001] TASSC 59 at [49].
51 As to the prosecutor's fallacy, see also Redmayne, "Doubts and Burdens: DNA Evidence, Probability and the Courts" [1995] Crim LR 464 at pp474-6, Thompson and Schumann, op cit. (The latter article describes the Defence Attorney's Fallacy as the suggestion that associative statistical evidence is irrelevant regardless of the rarity of the "matching" characteristics. What this reasoning fails to take into account is that the great majority of the pool of people with relevant matching are not suspects in the case against whom additional evidence is available linking them with the offence. On this, see also the passage from Doheny and Adams quoted above.)
52 I am not aware of any decision (apart from the trial rulings in the present case) where the risks inherent in this fallacious method of reasoning led to the complete rejection of (accurate and verified) statistical data of the "non-inclusionary" sort involved in the present case.
53 In Doheny and Adams, the English Court of Appeal considered statistical evidence based upon DNA testing to the effect that the chance of finding the profile in another unknown unrelated person was 1 in 40 million. The admissibility of the evidence was not in issue in that case, but nothing in the judgment suggests concern on that account. It was held that the verdict of the jury was unsafe because the forensic evidence was presented to the jury in a misleading and inaccurate manner, involving the Prosecutor's Fallacy. An expert witness, Mr Davie concluded his evidence in chief as follows:
Q. What is the combination, taking all those into account?
A. Taking them all into account, I calculated the chance of finding all of those bands and the conventional blood groups to be about 1 in 40 million.
Q . The likelihood of it being anybody other than Alan Doheny?
A. Is about 1 in 40 million.
Q. You deal habitually with these things, the jury have to say, of course, on the evidence, whether they are satisfied beyond doubt that it is he. You have done the analysis, are you sure that it is he?
A. Yes.
54 Giving the judgment of the Court, Phillips LJ said of these three questions and answers (at 378):
The second question, in leading form, and the affirmative given to it constituted a classic example of the "Prosecutor's Fallacy". The third question was one for the jury, not for Mr Davie. Mr Davie gave an affirmative answer to it. It is not clear to what evidence, if any, other than the DNA evidence, he had regard when giving that answer. For the reasons that we gave in our introduction to this judgment, this series of questions and answers was inappropriate and potentially misleading. The questions should not have been asked of this witness. If the random occurrence ratio was, indeed, 1 in 40 million this nonetheless meant that there was a statistical possibility of several individuals in the United Kingdom having the same DNA profile as the assailant, of whom one was the appellant. It was for the jury to decide on all the evidence whether they were sure that it was the appellant who left the crime stain, or whether it might have been one of a handful of other persons who might exist in the United Kingdom sharing the same DNA profile.
55 The fact that experienced witnesses, prosecutors and judges have themselves committed variants of the prosecutor's fallacy in dealing with DNA evidence (see Redmayne, op cit at p474 n68, Hassett [1999] EWCA Crim 1481) is reason for caution. Empirical research reinforces this (see Thompson and Schumann, op cit). Nevertheless, I cannot accept the correctness of Judge Moore's ruling that no arithmetical figures should be placed before the jury. I would adopt the following remarks of Judge Sievers, speaking for the Superior Court of Nebraska in State of Nebraska v Champ (20 March 2001, No A-00-617):
… it is worth observing that DNA matches mean little without the statistical probability component, which relies on the Hardy-Weinberg equilibrium and product rule. … "Without the probability assessment, the jury does not know what to make of the fact that the patterns match: the jury does not know whether the patterns are as common as pictures with two eyes, or as unique as the Mona Lisa." ( State v Carter 524 NW 2d 763 at 783)….
DNA science is accepted and admissible, there was no challenge to the methodology, and the evidence was that the DNA analysis was done in accordance with generally accepted scientific principles. Thus, while the evidence is obviously highly prejudicial to Champ, it is not unfairly so, and its probative value is great. There was no error in the use of the DNA evidence.
56 In Jarrett, Mullighan J declined to exclude expert evidence of DNA analyses and the statistical interpretation of those analyses. That was a stronger case than the present because the scientific experts were in conflict. His Honour cited well-known authorities about the jury's role in resolving conflicts in the evidence. These included the following remarks of King CJ in Duke (1979) 22 SASR at 48, 1 A Crim R at 41 where he said:
There may be unusual cases in which the judge has reason to fear that the jury will be overawed by the scientific garb in which the evidence is presented and will attach greater weight to it than it is capable of bearing. It is to be remembered, however, that under our system, whatever criticisms of it there might be, the assessment of the weight to be attached to expert evidence, as to other evidence, is the function of the jury. That being the system, a trial judge must assume, in my view, that the jury is capable of understanding that it is not bound to accept the expert evidence, that it is capable of resolving conflicts of opinion amongst the expert witnesses, and that it will not be overawed by the scientific garb in which the evidence is presented to it.
57 Later in his judgment in Jarrett, Mullighan J said (at 455):
It is not immediately clear to me how admissible expert testimony which bears upon an important fact in issue may have a prejudicial effect on the minds of jury which far outweighs its probative value in the relevant sense. As King CJ observed in Duke (at 47-48):
The prejudice … must, of course, be a prejudice additional to or distinct from the detriment to the accused's interests involved in the probative force of the evidence.
In order to be admissible, expert testimony must be relevant to a fact in issue and comply with well-known preconditions…. Once those preconditions are established, it is difficult to see how the evidence may have a prejudicial effect which far outweighs its probative value….
58 I agree.
59 It follows that the paternity index figures should not have been withheld from the jury pursuant to s137 of the Evidence Act. They should have gone to the jury accompanied by appropriate directions emphasising the need to avoid the prosecutor's fallacy. This appeal is not the proper vehicle to formulate a model direction in that regard. I content myself with the suggestion that it would be desirable if, at the time the judge tells the jury that evidence of paternity has been expressed as a high or very high probability, the jury should be reminded then and there that the evaluation of that evidence is a matter for them in light of the totality of the expert and non-expert evidence.
60 However, I agree with Sully J that the exclusion of the relative chance of paternity percentage was appropriate. I agree with his Honour's reasons and would add this consideration: Mr Goetz himself recognised that the relative chance of parentage percentage was "more complicated to explain". I shall not endeavour to explain why, lest I reveal my own misunderstanding of relative chance of paternity expressed as a percentage, beyond observing that part of the difficulty would be removed if 99.9993% was transposed to a statement that there was a 0.0007% probability or chance of C's father being anyone other than GK. Even such adjustment would leave me of the view that the evidence is unduly prejudicial in its impact.
61 The second question should therefore be answered, as proposed by Sully J: "Yes", as to the Paternity Index statistics; "No", as to the Relative Chance of Paternity statistics.
62 SULLY J: The New South Wales Director of Public Prosecutions has submitted for determination by this Court two stated questions of law. The right of the Director to do that derives from s.5A(2) of the Criminal Appeal Act 1912 NSW. It is there provided, relevantly:
"(2) (a) Where a person tried on indictment has been acquitted (whether in respect of the whole or part of the indictment) either before or after the commencement of the Criminal Appeal (Amendment) Act 1977 the Attorney-General or Director of Public Prosecutions may at any time after the conclusion of the trial submit for determination by the Court of Criminal Appeal any question of law arising at or in connection with the trial.
(b) The Attorney-General or Director of Public Prosecutions shall submit with the question to be determined a statement of the circumstances out of which the question arose and thereafter shall furnish such further statement as the Court of Criminal Appeal may require.
(c) The Court of Criminal Appeal shall hear and determine any question submitted to it under this subsection.
(d) The determination by the Court of Criminal Appeal of the question submitted shall not in any way affect or invalidate any verdict or decision given at the trial.
(e) Any person charged at the trial or affected by the decision shall be entitled to be heard before the Court of Criminal Appeal upon the determination of the question submitted, and if it appears that such person does not propose to be represented upon such determination, the Attorney-General or Director of Public Prosecutions shall instruct counsel to argue such question before the Court of Criminal Appeal on behalf of such person.
(f) The reasonable costs of legal representation of any person heard before the Court of Criminal Appeal as provided in this subsection shall be paid by the Crown.
(g) The hearing and determination of any question under this subsection shall be held in camera:
Provided that nothing in this paragraph shall preclude a barrister or solicitor from being present at the hearing and determination for the purpose of reporting the case for any lawful purpose of the Council of Law Reporting for New South Wales.
(h) No report of any submission made pursuant to paragraph (a) shall be published. No report of proceedings under this subsection shall be published which discloses the name or identity of the person charged at the trial or affected by the decision given at the trial. Any publication in contravention of the foregoing provision shall be punishable as contempt of the Supreme Court."
63 It is necessary to recapitulate, before turning to the precise questions which have been submitted by the Director, the relevant background which is as follows.
64 In September 1999 G.K. was presented for trial before Ford DCJ and a jury in the Campbelltown District Court. He was so presented upon an indictment containing 14 counts. The complainant in all counts was G.K's step-daughter. Some of the counts were counts of common assault; and the remainder of them were counts charging various offences of sexual impropriety. Some of the counts alleged sexual intercourse between G.K. and the complainant.
65 The upshot of that trial was that the jury found G.K. guilty of one charge of common assault; found him, by direction, not guilty of six other counts charged; and were unable to agree upon verdicts on the remaining seven counts in the indictment. By the time of this trial, the complainant had given birth to a baby boy, C. The Crown set out to prove, in aid of those charges involving an allegation of sexual intercourse between G.K. and the complainant, that G.K. was in fact the father of C. To that end the Crown tendered evidence of certain DNA testing. Objection was taken to the admission of that evidence. The transcript of the proceedings before Ford DCJ has not been included in the present appeal papers; but the materials that have been included in the present appeal papers suggest that Ford DCJ dealt with the objection upon the basis that the objection depended upon s 137 of the Evidence Act 1995 (NSW), ["the Evidence Act"]. It appears that Ford DCJ conducted a hearing on the voir dire in order to determine the admissibility of the challenged evidence. It appears, further, that Ford DCJ, at the conclusion of the submissions of counsel made in connection with the voir dire hearing, dealt briefly and ex tempore with the particular evidentiary issue. The relevant record discloses the following exchanges:
"COUNSEL ADDRESSED ON VOIR DIRE
HIS HONOUR: Yes very well thank you I think the evidence should be limited so that there is merely an expression of opinion as to the scientific view; that the testing does not exclude the possibility of paternity. So the evidence of likelihood and certainly the evidence as to percentage should be exluded.
CROWN PROSECUTOR: If your Honour pleases in that event --
HIS HONOUR: And I say that because I think there is a very considerable risk that the jury notwithstanding any blandishments on my part may take the practical position of a punter and say this is not just a mere certainty but an absolute certainty. Anyway sorry I interrupted you.
CROWN PROSECUTOR: No your Honour. I know what it is your Honour is saying. Your Honour that being the case I would not call both experts, there seems little point in doing that and your Honour subject to anything my friend might have to say I would not go through the intricacies of the testing involved in coming to that conclusion but simply have the expert give the opinion on the basis of testing.
HIS HONOUR: Yes. You will accept I suppose the proposition put by Mr. Pontello that DNA testing is very good when it is a question of exclusion, it can exclude the possibility that someone was the sire and that ought to be explained to the jury.
CROWN PROSECUTOR: Yes your Honour but it's certainly not finger printing. If your Honour would give me a moment to confer with my witnesses so that I can prepare them for that and then it will be very short evidence from that point onwards."
66 In February 2000 G.K. was re-presented for trial in the District Court at Campbelltown and before Moore DCJ and a jury. He was so presented on an indictment containing eight counts. Seven of the counts reproduced the charges as to which the jury in the first trial had been unable to agree upon verdicts. There was an additional charge alleging the intimidation of a witness.
67 This second trial aborted after two days of hearing. The reasons are of no present moment.
68 Prior to the commencement of this second trial, further DNA testing had been carried out by each of two independent laboratories. The Crown sought to lead this evidence, and Moore DCJ granted an application by the defence for a voir dire hearing to determine the admissibility of the evidence. At the conclusion of this voir dire hearing, and on 8 February 2000, Moore DCJ published two judgments. In the first judgment his Honour considered whether, and if so to what extent, he was bound by the earlier ruling of Ford DCJ. The conclusions to which his Honour came in that connection depended substantially upon a careful analysis by his Honour of the decision of the High Court of Australia in Rogers v The Queen (1994) 181 CLR 251; and of the decision of the Court of Appeal of Victoria in Edwards (1997) 94 A Crim R 204. His Honour's conclusions are expressed succinctly in paragraphs 26 and 27 of the published judgment. Those paragraphs read:
"26. The ratio in Rogers has two strands - (1) the ruling in the original trial must be conclusive and (2) the importance of avoiding the scandal of conflicting decisions. It is cloaked by the special shroud of the distinct nature of criminal proceedings, with its fundamental principles (see Rogers per Deane and Gaudron JJ quoted above)
27. My analysis of the application of Rogers , and as to the authority of Edwards in the present case, is to say that I should pay full deference to the ruling of his Honour Judge Ford unless (a) his Honour's finding was clearly wrong, or (b) there are different circumstances in the application of the discretion in the present case, or there is fresh evidence in the present case, in either of those two circumstances being such as would require a real re-determination of whether the discretion should be exercised."
69 Moore DCJ then heard further submissions from counsel; and thereafter published the second of the two judgments to which I earlier referred. His Honour described as follows his then task as his Honour perceived it:
"I have to determine whether the exercise of the discretion by his Honour was clearly wrong, or that the circumstances prevailing at this trial are sufficiently different, or that there would be an admission of fresh evidence of a sufficiently different nature, to warrant a fresh consideration of whether or not I should exercise the discretion."
70 His Honour then dealt with each of the elements identified in that definition of his task, and came ultimately to the following conclusion:
"I feel that the legal principles require me not to disturb the exercise of the discretion from that which was exercised by his Honour Judge Ford and I would therefore direct that the Crown is entitled to lead evidence that DNA testing was done which shows that the accused could be the father but that there should not be the arithmetical figures put before the jury."
71 In due course, and in connection with the third of the questions now submitted by the Director to this Court, it will be necessary to consider his Honour's reasoning and conclusions in the first of the two judgments. It will be, similarly, necessary to return in due course, and in connection with the second of the submitted questions, to the reasoning and conclusions expressed in his Honour's second judgment.
72 As I have earlier noted, this second trial of G.K. aborted. Subsequently, and on 20 March 2000, G.K. was indicted for the third time, and again before Moore DCJ, the indictment containing the same counts as had been contained in the indictment presented in connection with the second, and aborted, trial. The rulings which Moore DCJ had given during the course of the second trial were treated as continuing in connection with the third trial. On 22 March, and after his Honour had given the jury a direction of the kind described and explained by the South Australian Court of Criminal Appeal in Reg. v Prasad (1979) 23 SASR 161, the jury returned verdicts of not guilty upon all eight of the counts in the indictment.
73 Against the whole of the foregoing background, the Director has now submitted for determination by this Court the following two question:
"1. Where a trial Judge makes a discretionary ruling on the admissibility of certain evidence is a trial Judge at a subsequent trial following a failure by the first jury to agree on its verdict bound to follow that ruling with respect to the admissibility of the same evidence?
2. Was his Honour Judge Moore in error in refusing to admit evidence of the probability, in numerical terms derived from DNA testing, that ……..(G.K)…… was the father of the complainant's child, on the basis that there was a real risk of unfair prejudice to the accused?"
74 As to the first submitted question: I am of the opinion that the decisions, previously herein mentioned, of Rogers and of Edwards, support the following propositions: