The Roy Morgan Survey
310 Samsung sought to rely upon four related online surveys (the Survey) directed to the issue of whether the TVCs conveyed the pleaded representations to members of the public in Australia. The evidence concerning the Survey was received subject to a number of objections raised by LG. The objections raised by LG were as follows:
the evidence given by Mr Bardsley in relation to the Survey including, in particular, the opinions expressed by him in relation to it was not an expert opinion within s 79(1) of the Evidence Act;
if otherwise admissible, the Survey and Mr Bardsley's evidence in relation to it should be excluded under s 135 of the Evidence Act;
if admitted into evidence unconditionally the Survey and Mr Bardsley's evidence in relation to the results should be given no weight.
311 The matter of survey evidence was considered by the Full Court in Arnotts Ltd v Trade Practices Commission (1990) 24 FCR 313 (Arnotts). The Full referred to various authorities concerned with the admissibility of survey or market research evidence including, in particular, the decision of McLelland J in Ritz Hotel Ltd v Charles of the Ritz Ltd (1988) 15 NSWLR 158 (Ritz) in which his Honour had said of such evidence (at 178):
There is a substantial preponderance of authority in support of the proposition that survey evidence, and expert evidence in relation thereto, are admissible to prove the state of mind of the public or a section of the public on some particular matter, when that is an issue. However two distinct bases emerge from the cases in justification of the admissibility of such evidence. The first is that although out-of-court statements by persons interviewed in the course of a survey as to their impressions or opinions are of a hearsay nature, the admission of such statements as evidence of the existence of those impressions or opinions falls within a recognised exception to the hearsay rule. On this approach, the primary evidence is that of the individual responses of those interviewed, interpretative expert evidence being subsidiary. The second basis is that such statements are to be treated not as hearsay, but as original data providing a foundation for expert evidence as to the state of public opinion on the matter in question. On this approach, the primary evidence is that of the expert, and evidence of the individual responses is subsidiary.
312 The first basis (primary evidence) referred to by McLelland J is reflected in the exception to the hearsay rule found in s 66A of the Evidence Act. Section 66A provides:
The hearsay rule does not apply to evidence of a previous representation made by a person if the representation was a contemporaneous representation about the person's health, feelings, sensations, intention, knowledge or state of mind.
313 The second basis (interpretative expert evidence) is reflected in s 79(1) of the Evidence Act which establishes an exception to the "opinion rule".
314 The Full Court's subsequent discussion in Arnotts of Ritz and earlier Australian authorities suggests that it accepted that survey evidence might be received if (at 360) "… the survey was direct evidence of an issue in the case, namely the state of public perception." However, the Full Court did not find it necessary to express any final view as to whether or not such evidence is admissible because of O 33, r 3 of the then Federal Court Rules. The Full Court said at 360-361:
However, it is not very profitable - at least in this Court - to spend time in determining whether a particular survey is hearsay evidence. Even if it is, ordinarily the Court will have a discretion under O 33, r 3 to permit the evidence to be adduced. To call the persons who responded to the survey will almost always result in appreciable expense and delay. Given the existence of a discretion, it seems more sensible to concentrate attention upon the necessity for, and reliability of, the survey evidence; rather than to worry about its compliance with rules regarding hearsay evidence which were developed before this type of problem arose. This is not a situation, like that encountered in Pearce v Button (1986) 8 FCR 408, where the evidence sought to be adduced is the subject of "a real dispute about matters which go to the heart of the case": see per Lockhart J (at 422). See also Multi-Modal Ltd v Polakow (1987) 78 ALR 553 at 558.
We say nothing about criminal cases. But, in a civil case in which a market survey may cast light on relevant issues, it is desirable in principle to admit into evidence a report of a professionally conducted survey, upon proof that it has been satisfactorily conducted using relevant and unambiguous questions; and without requiring evidence from each of the participants.
There are two reasons for our view. In the first place, market survey techniques have now been refined to the point where, if undertaken by experienced, professional people, they are capable of providing answers which are highly likely to be accurate, subject only to a small sampling error. It is commonplace for polls to predict the results of elections within a couple of percentage points. Political parties and commercial organisations constantly use market research to ascertain public reactions upon a wide range of matters; and there is no reason to doubt that, by and large, they find the results reliable. As Hill J observed in Sterling Pharmaceuticals at 36,377, "... statistical analysis can confirm that to a specified degree of probability and subject to a specified error rate, the result can be projected to the whole or a defined section of the population".
315 The Federal Court Rules 2011 (Cth) do not include any rule equivalent to O 33, r 3. However, subss 190(1), (3) and (4) of the Evidence Act may enable the Court to make orders dispensing with certain provisions of the Evidence Act including, relevantly, those found in Part 3.2 - Hearsay.
316 If the Survey is admitted into evidence, it is necessary for the Court to determine what weight it should be given. This will usually call for some consideration of the quality and integrity of any questionnaire used for the purposes of the Survey and the methods of data collection and analysis employed. It may also require consideration of whether the respondents to the Survey are sufficiently representative of the population under investigation.
317 So far as the questionnaire is concerned, relevant considerations in determining what weight should be given to the Survey include the relevance of the answers given, and whether they were elicited, recorded and analysed in an accurate, reliable and unbiased manner. As to representativeness, it may be necessary to consider how the sample of respondents who participated in the Survey was selected and whether adequate steps were taken to ensure that the sample is sufficiently representative of the population being investigated to enable the drawing of reliable inferences.
318 Mr Bardsley's employer, Roy Morgan Research Pty Ltd (Roy Morgan) was engaged by Samsung's solicitors in mid-November 2011. The letter of retainer included the following:
We are instructed to retain you to provide independent expert assistance in this matter, including to conduct a survey and to prepare a report or an affidavit concerning that survey that will be read as evidence in the proceedings. You may be required to be available to give evidence at the hearing and be cross-examined.
The purposes of conducting the survey are to ascertain:
(a) what representations or messages are likely to be conveyed to consumers by the Advertisements;
(b) what consumers are likely to understand by the terms 'Conventional 3D TV' and 'Next Generation TV'; and
(c) whether consumers are likely to notice the disclaimers that appear in each of the Advertisements (other than the Modified Flickering Advertisement) and if so, to ascertain what they are likely to have understood those disclaimers to mean.
The survey questions that we propose to ask appear behind Tab 3.
Based on your expertise in market research and based on market research theory and principles, we would be grateful if you could:
(i) consider the appropriate methodology for the conduct of the survey, suggest any relevant controls to be used in undertaking the survey and advise of any other matters that are relevant to the conduct the survey [sic];
(ii) manage the conduct of the survey; and
(iii) once the survey has been conducted, consider and describe the results of the survey in your report or affidavit.
319 A draft survey questionnaire was included with Roy Morgan's letter of retainer. Who prepared the questionnaire is not clear, although Mr Bardsley accepted that it appeared to have been prepared by "somebody who knew something about surveys." I think it must have been obvious to him that it had been prepared by or with the assistance of someone who worked in the market research field. In any event, I do not think there is any reason to doubt that he considered it and approved it, and in doing so, drew on his expertise in the field of market research.
320 Mr Bardsley ultimately conducted four online surveys or survey "legs". The first three of these coupled the Brightness TVC with the Flickering TVC; the Battery TVC with the Weight TVC; and the Power Cable TVC with the Modified Flickering TVC. Each of the respondents in these survey legs viewed the two TVCs about which they were questioned. The fourth leg was directed to the Modified Flickering TVC alone (ie. the fourth leg was "uncoupled").
321 Each of the four questionnaires included five open-ended questions which the survey respondent was required to answer after viewing each of the two paired TVCs (except in the case of the "uncoupled" Modified Flickering TVC):
(1) What was the ad telling you? Please type a detailed response in the space below (QA2 & QA8)
(2) What was the ad telling you about LG Cinema 3D TV? Please tell us everything the ad was telling you about LG Cinema 3D TV. Please type a detailed response in the space below. (QA3 & QA9)
(3) What was the ad telling you about Conventional 3D TV? Please tell us everything the ad was telling you about Conventional 3D TV. Please type a detailed response in the space below. (QA4 & QA10)
(4) What was the ad telling you about the [feature] of LG Cinema 3D TV [glasses]? Please tell us everything the ad was telling you about the [feature] of LG Cinema 3D TV [glasses]. Please type a detailed response in the space below. (QA5 & QA11)
(5) What was the ad telling you about the [feature] of Conventional 3D TV glasses? Please tell us everything the ad was telling you about the [feature] of Conventional 3D TV [glasses.] Please type a detailed response in the space below. (QA6 & QA12)
322 The questionnaires included the following variations to take account of the particular TVCs shown to the survey respondents:
the feature referred to in the case of the Flickering TVC and the Modified Flickering TVC was "… the flickering of …" the glasses;
the feature referred to in the case of the Brightness TVC was "… the brightness of …" the TV;
the feature referred to in the case of the Battery TVC was "the Battery";
the feature referred to in the case of the Weight TVC was "… the weight of …" the glasses; and
the feature referred to in the case of the Power Cable TVC was the "… battery of …" the glasses.
323 Mr Briggs was critical of the decision to couple TVCs. His criticism of this approach has considerable force, but it is not a criticism I would give much weight in circumstances where LG's solicitors appear to have suggested this approach when the proposed survey methodology was first canvassed with them. Nevertheless, I do not accept Mr Bardsley's evidence (when rejecting Mr Briggs' criticism) that coupling did not produce responses that would not have been given if the advertisements had not been coupled. When the topline results obtained for the Modified Flickering TVC (coupled) (41%) and the Modified Flickering TVC (uncoupled) (31.2%) are compared, the difference is to my mind considerable (9.8%). Mr Briggs considered the difference significant, however, Mr Bardsley disagreed. I do not think there can be any doubt that the difference is significant. If the difference is not related to coupling then it would suggest that there is some other factor in play that has produced these divergent results.
324 Potential respondents were drawn from the Research Now Australian consumer panel (the Research Now Panel), an online panel consisting of approximately 170,000 members. Survey respondents who completed the Survey were provided with an incentive of $1.00 credited towards a $20.00 voucher they could earn by completing other surveys. The Survey commenced on 13 December 2011 and continued over the Christmas/New Year period. Apart from the Power Cable and Modified Flickering questionnaire, which closed on 24 December 2011, the questionnaires closed on 3 January 2012. A total of approximately 5,000 questionnaires were completed.
325 In order to analyse the detailed responses provided by survey respondents to the open-ended questions, it was necessary for Mr Bardsley to code those responses. Mr Bardsley was provided by the solicitors for LG with a list of representations (the Representations Document) which he used to code the verbatim responses. This list of representations generally corresponds with the representations pleaded in the SOC though there are some significant exceptions. In particular, the Representations Document does not include any representation corresponding to the pleaded "battery representation" 28(d). Further, while the Representations Document includes a representation that "conventional glasses are heavy", this does not accurately reflect "weight representation" 38(a) that "conventional glasses are heavy, too heavy or excessively heavy."
326 Mr Bardsley allocated a number to each representation in the Representations Document: eg. representations 23(a), 23(b), 23(c) and 23(d) were coded 1, 2, 3 and 4 respectively, with 8 being used as the code for "other representations" and to signify that, on Mr Bardsley's interpretation of a verbatim response to an open-ended question, the response did not correspond to any of the representations set out in the Representations Document.
327 Mr Bardsley:
formed a view as to what was meant by each of the coded representations;
read each of the verbatim responses to the open-ended questions and decided whether it conveyed a meaning which corresponded to one of the representations;
entered (in a spreadsheet) the code for a particular representation if he thought the response corresponded to that representation;
subsequently reviewed and checked his spreadsheet results against the verbatim responses to the open-ended questions and made such changes as he considered appropriate.
328 This last step involved reconsidering responses given, and the codes initially assigned to them, in light of other responses provided by the survey respondent. In this way the coding of the responses given to general questions (eg. QA2) has in some cases been re-interpreted and re-coded by Mr Bardsley based upon responses given by the respondent to more specific questions (eg. QA4).
329 It is not apparent from the evidence who prepared the Representations Document which was to become the basis for Mr Bardsley's coding of the Survey. The choice of language (eg. the use of the word "substantially" in the representation corresponding to 23(d)) and the complexity of some of the representations (eg. the representation corresponding to 23(b)) suggest to me that very little attention was given to the potential problems that could arise using what to my mind is a document that provided wide scope for variability and subjectivity in the evaluation of the verbatim responses. It appears that Mr Bardsley treated "substantially" as a synonymy for "much" with the consequence that "substantially brighter" became "much brighter". He also agreed that representation 23(b) was complex. Mr Bardsley also acknowledged that each category in the coding frame (which is what the list of representations served as) should be designated in the clearest possible way. I am satisfied that the coding frame used by Mr Bardsley fell well short of this standard.
330 Although Mr Bardsley made some changes to the structure of the questionnaires, the basic form of the open-ended questions was not changed by him (at least not in any way that either he or Mr Briggs considered material) except in one respect which involved removing emphasis given to particular words in these questions. As originally drafted, the word "everything" was printed in bold where it appeared in the contentious open-ended questions viz. what I will call the "follow-up" questions QA3 & QA9, QA4 & QA10, QA5 & QA11 and QA6 & QA12. Although in his affidavit evidence Mr Bardsley described the use of such emphasis as a standard market research technique that did not make or contribute to any of these questions being "leading" questions, he stated that he agreed to this change on the basis that the use of such emphasis "was not essential in order to meet the stated purpose of the survey."
331 The comments provided by LG's solicitors on the proposed questionnaire in their letter of 6 December 2011 raised objections to each of the follow-up questions on the basis that they were leading and, in particular, that they led the respondents towards a response that they were told multiple things about LG 3D TV, conventional 3D TV and (using the Flickering TVC as an example) flickering.
332 Mr Bardsley rejected this criticism. His evidence was that if the follow-up questions had not been asked the Survey would not have provided as much information about consumers' understandings of the TVCs. He also stated that he did not agree that the follow-up questions suggested to the respondents that each TVC conveyed "multiple messages … [or] that respondents would have understood themselves to be obliged to provide multiple responses in answer to a single question." I think the latter part of that statement misses the point that was being made by LG's solicitors, which was not that they would feel obligated to provide multiple answers to a single question, but that they would be encouraged to search for meanings or messages that might not have occurred to them had they not been led to do so by the follow-up questions.
333 Mr Bardsley accepted that the follow-up questions were repetitive, but said that when testing the understanding of an advertisement, they were standard practice because people generally will not give a full answer. He said that this was particularly so in the case of an online survey. Mr Briggs' evidence was that the follow-up questions were "leading" and that their use in this Survey raised considerable doubts about the value of the results. He conceded that, as Mr Bardsley had asserted, probing was a technique frequently used in market research.
334 Mr Briggs was cross-examined extensively on the basis that it would not be accurate to characterise the follow-up questions as leading because they did not suggest any particular answer. However, Mr Briggs' point was that the questions were leading in the sense that they "probed" the respondents for further information and, by doing so, encouraged them to focus on specific features of the relevant TVC in a way they may not when viewing the TVC in a different, and less artificial, context. I think Mr Briggs is correct.
335 In my view the follow-up questions, especially QA5 & QA6, are highly problematical. It is important to note that the follow-up questions are asked after the respondent has already been instructed to type a detailed response in answer to the first of the open-ended questions: "What was the ad telling you?". That same instruction is given for each successive follow-up question in which the respondents are told to "tell us everything the ad was telling you about" a specific matter (eg. the weight of conventional 3D glasses). The overall effect, in my view, was to require, or at least encourage, the respondents to provide detailed responses to a succession of questions about a 15 second TVC that may never have triggered any detailed response in the first place.
336 Another criticism raised by LG's solicitors in their letter of 6 December 2011 concerned the matter of humour. The point was expressed in this way:
[T]he advertisements are intended to be humorous and light-hearted. Yet the survey is very literal and [LG] is concerned that there is no attempt to ascertain whether humour is being conveyed by the television commercials.
337 The point was developed through cross-examination and in submissions. It was submitted by LG that the Survey is flawed because the TVCs can be interpreted in either a highly literal manner or in a less literal manner that makes allowance for the use of exaggeration and comedy, so as to convey to the viewer something less than, or different to, what is conveyed to a person interpreting the TVCs literally.
338 I think the form of the follow up questions was likely to prompt some respondents to provide a literal account of what they saw, even though it could not possibly reflect the actual message conveyed to them. The problem is illustrated by some well-chosen examples referred to in evidence in which some respondents responded that a person wearing conventional 3D glasses could fall though the floor. Mr Bardsley decided to code on the basis that this respondent was indicating that the Weight TVC was telling him or her that the glasses were too heavy. I think it is equally if not more likely that the Weight TVC was telling this respondent that LG 3D glasses were lighter than conventional 3D glasses and/or that conventional 3D glasses were heavier than LG 3D glasses.
339 Another example drawn directly from Mr Bardsley's affidavit has a respondent saying: "The glasses that one uses for conventional televisions are heavy. They could cause neck damage and make your chair fall through the floor." Another example quoted by Mr Bardsley has a respondent saying: "[T]he weight of the glasses are [sic] unbearable." Mr Bardsley interpreted these respondents as indicating that that they were being told by the Weight TVC "that conventional 3D glasses are too heavy to wear comfortably."
340 I am satisfied that the use of extensive probing, and the related problem of how to interpret and code highly literal responses, considerably undermines the reliability of the Survey. I am also satisfied that Mr Bardsley underestimated the difficulties involved in ascertaining what serious message (ie. one likely to lead ordinary and reasonable viewers into error as opposed to a "non-serious" message such as "you could fall through the floor") was conveyed to consumers by the TVCs if that is indeed what he (or whomever it was who designed the Survey) was actually seeking to do.
341 In the correspondence exchanged in late 2011 between the parties' solicitors, the solicitors for LG asked:
whether a pilot survey has been conducted by Samsung (whether using Roy Morgan or another research company) or your firm and if so, [we request] a copy of the questionnaire used in the pilot and the survey results.
The solicitors for Samsung responded that this was not a proper request, but that is clearly wrong. The request was a natural and obvious one to make in circumstances where LG was being asked to agree to a questionnaire that was to form the basis for the market research evidence that Samsung was seeking to adduce. In particular, LG was entitled to know whether the proposed survey instrument had been tested in any pilot survey or trial with a view to ensuring that it was suitable for the purposes stated in Roy Morgan's letter of retainer.
342 In the course of his evidence Mr Williams was asked whether his firm undertook a pilot survey, or caused a pilot survey to be undertaken, prior to Mr Bardsley conducting the Survey. He declined to answer that question and claimed privilege. Senior Counsel for LG did not contest the privilege claim.
343 The usual practice in the UK is to require a party seeking permission to carry out a survey to provide the Court (and a fortiori the opposite party) with a copy of the results of any pilot survey: Interflora Inc v Marks and Spencer plc [2012] EWCA Civ 1501; [2013] 2 All ER 663 at [151]. This Court's Practice Note CM13 "Survey Evidence" does not in terms require that this be done. However, at least where the form of the questionnaire is a matter of dispute then it seems to me appropriate that, as in UK, the party wishing to rely on the survey should produce any pilot study that was conducted for the purpose of designing or testing the questionnaire.
344 In the present case there is no direct evidence that there was any pilot study or trial conducted to test the suitability of the questionnaire. The purpose of a pilot study is to ensure that the questionnaire works. The absence of any direct evidence that the questionnaire was tested before it was put into the field is a matter that I have taken into account in deciding what weight to give to the Survey.
345 Whether a party seeking to tender a survey can successfully maintain a claim for privilege over any pilot survey it conducted beforehand and, if it can, what consequences follow in terms of the admissibility of the survey (particularly in light of s 135 of the Evidence Act) are questions for another case.
346 My conclusion is that the Survey results should not be given any weight in deciding which of the pleaded representations were likely to have been conveyed.
347 Other arguments were raised by LG as to the representativeness of the sample frame from which the respondents were drawn. These arguments centred on the "self-selecting" nature of the Research Now Panel and the absence of any cogent evidence (including from Mr Burge) as to its representativeness and, a fortiori, the sample drawn from it. It is not necessary to resolve these arguments.