10 Easy Ways to
Blow Away A Survey

By James T. Berger

The great Larry Bird was once asked by a sportswriter why he plays basketball. He replied – “I play to win.”

This article is about winning – about knowing what techniques that positively influence a case or to discredit an opponent. This is not intended as a scholarly article – it includes a “street smart ” information I have learned based on my own experiences and observations in the firing line. In the last 10 years, I have compiled dozens of intellectual property surveys and have been deposed almost as many times.

While a well planned and executed survey can create powerful evidence in trademark infringement case, such a survey can also be a major liability in the hands of a smart adversary. If designed or implemented improperly, your survey can boomerang into a weapon for your opponent.

My best advice is to program into your survey design those very criticisms you van expect during deposition and/or cross-examination. When designing your survey, be your own most severe critic. Every survey has flaws. Be objective of and mindful of your own survey’s flaws, so you can avoid being ambushed.

Before getting into specific flaws, it’s important to understand to crucially important aspects of survey research – Validity and Reliability. Validity refers to the survey’s ability to test those things that are relevant to the case; Reliability reflects the survey’s ability to replicate approximately the same results when given to other samples in other venues.

So, while working with your survey expert or working to discredit an adversary’s survey expert be mindful of the following 10 flaws that can help blow away a survey and render it meaningless..

FLAW# 1 – SELECTING THE WRONG TARGET MARKET

The purpose of the survey is to take a “frame,” a small portion of the target universe and draw conclusions about the entire universe from the knowledge you gain by surveying this frame. It is essential that the survey focus on the precise target market and not some related target market. If a target market is a dealer or distributor, don’t focus on t5he customer. Determine precise the market that is allegedly being confused or NOT being confused. Respondents should be selected from among persons who are potentially involved in the dispute being litigated.

In line with this flaw is the crucial need to recognize the “channels of trade.” How does the product proceed from manufacturer to distributor to retailer to end-user? Where is the product sold? How is it sold? How is it displayed?

FLAW #2 — INSTRUCTIONS TO INTERVIEWERS

Especially in a telephone or mall intercept survey, personnel working for research firms are generally neither the brightest nor most educated people. Therefore, create detailed, yet easily understood WRITTEN INSTRUCTIONS to interviewers and supervisors. Ideally, the research expert should deliver these instructions face-to-face to BOTH the research supervisor, venue supervisor and individual researchers. If not face-to-face then it should be done via conference call. The survey expert should be assured in his/her mind that the instructions are understood.

In the process of the research if any questions come up, the research expert should be available to answer them immediately. If it can be shown or proved that the research instructions are inadequate, this could seriously jeopardize the validity and reliability of the survey.

FLAW #3 — THE SCREENER QUESTIONNAIRE

In line with the rationale behind the interviewer instructions in terms of the mental capabilities and backgrounds of the interviewers, it is important that the survey expert produce a WRITTEN SCREENER. This is a separate document used in the selection process of interview subjects. The screener should identify who is to be interviewed with respect to age, gender, languages that are spoken and understood and the subject’s association with the products and/or services involved. Also in the SCREENER document should be questions geared to eliminate potential subjects. Such question deal with whether the subject or family member works for an employer falling within the channels of trade; whether the subject or a family member works for an advertising, marketing or marketing research firm, which also might be a point of bias. On the screener questionnaire should be information as to why potential responders were excluded.

 It’s important to understand that mall research generally involves an incentive. Most shopping center malls have a cadre of “regulars,” people who often stop by the research center in an effort to earn a few extra dollars by participating in surveys. These are NOT the kinds of people to participate in legal research studies and should be excluded. Make sure they have not participated in another survey for three months or six months. If they are not excluded, it can do serious damage to the validity and reliability of the survey. Finally, it’s important that interview subjects can see and hear. If a respondent regularly wears eyeglasses or contact lenses to see, make sure they are wearing them. The same goes for hearing aides.

FLAW #4 — THE QUESTIONNAIRE

There is no rule requiring a long or short questionnaire. What it takes to prove or disprove a hypothesis is sufficient, but there are some pitfalls:

• Avoid compound questions where similar variables are combined in the same question with words like AND, OR, NOR etc. Ask two or more separate questions even at the risk of making the survey seem unduly long or cumbersome.
• When a subject is asked a question, be sure to ask WHY if it is appropriate to ask WHY.
• Continue to PROBE an answer until you have exhausted every possible reason.

FLAW #5 — FAILURE TO TEST THE QUESTIONNAIRE

In virtually every case, a questionnaire should be tested. The rationale is that by testing, the research expert can be assured the survey results will have validity and reliability. It is especially important to test longer and more complex questionnaires. It is also important, when possible, for the survey expert to observe the test. Based on the results of the test, the survey expert should be prepared to make changes in the survey as well as the screener and instructions documents.

It is possible — usually in a short, uncomplicated survey — to roll the test results into the total findings of the survey. This occurs when the test shows that no changes or revisions are needed. Testing is an important survey protocol. Failure to test can subject the survey results to question.

FLAW #6 — NUMBERS OF INTERVIEWS AND VENUES; PERCENTAGES NEEDED FOR SIGNIFICANCE

If your survey doesn’t have a minimum of 200 interviews and four venues, your results are going to be suspect. If your client can’t spend what is necessary to do at least 200 interviews at four or more venues, don’t bother to do the research. There is a strong probability that anything less than that will be disregarded by the court.

There are no arbitrary numbers to prove significance, but generally a minimum of 20 percent is needed in likelihood of confusion cases and a minimum 50 percent is needed in secondary meaning cases.

FLAW #7 — BEWARE OF ANCILLARY DATA

In addition to the key data that supports or refutes likelihood of confusion or secondary meaning, the survey is likely to produce other data that can be damaging or uncomplimentary to the client doing the research. This data is often produced through asking WHY and PROBING in the survey. This data needs to be reported. But everybody involved with the survey should be aware of this data, and the survey expert should offer reasons or opinions to explain these findings. An adversary can take this ancillary data and embarrass a client to the extent the client may decide not to make use of perfectly valid and reliable findings from the survey.

FLAW #8 — MULTIPLE EXPERTS

Often a plaintiff or defendant will retain more than one survey expert. If this is done, do not allow the experts to review each other’s findings or opinions. In such cases, the adversary has the opportunity to play one expert off against another to the detriment of the client. In deposition or trial testimony, when one expert refutes the findings of another, everybody loses. Unfortunately, sometimes such contradictions are unavoidable. Opposing counsel can show the results of the other expert in deposition and ask for opinions on those findings. This can be embarrassing. Tread carefully with findings and opinions of different experts working the same side of the street. The best advice is not to use multiple experts offering opinions on the same areas.

FLAW #9 — ATTORNEY INVOLVEMENT

The involvement of an attorney in the survey design will always be viewed with suspicion. While the expert develops the survey, the attorney is needed to review it to make sure factual accuracies have been addressed. The attorney can even suggest re-wording questions in an effort to better reflect the wishes of the expert. The key is the intent and control of the survey. It’s a fine line. The attorney knows what facts are needed to help him prove or disprove a complaint. The expert’s role is draft a survey that fulfills that intent. Problems arise when attorneys start drafting questionnaires independent of the expert, and the attorney produces the survey and tells the expert to administer it. If this comes out in testimony, the results of the survey would most likely be disregarded.

The same goes for the survey report. This should be the work product of the expert and the expert will be responsible for defending it. The attorney may suggest changes in format, emphasis, order of presentation or nonmaterial factors, but the attorney’s material opinions should not find themselves in the report unless the expert is in agreement based on the findings of the study.

FLAW #10 — BEWARE OF BIAS

The survey expert through the entire process has to be a policeman against bias. It starts with the relationships with the researchers. Most litigation surveys should be “double blind,” in that neither the individual researchers nor the subject should be aware of the purpose of the survey, what is attempted to be proved or disproved, or the name of the client.

The questionnaire should be direct, clear and unambiguous. When products are displayed the order of the products should be varied if possible. Locations of research venues should be selected as not to prejudice results. Mall intercept interviewers should be instructed to avoid bias in the selection of interview candidates in the mall.  Every attempt throughout the planning and execution of the survey should be designed to eliminate bias. Obvious bias will jeopardize the value of the survey.

To Top