The Power and Perils of Internet Surveys

By James T. Berger

As one who is involved in trademark infringement litigation surveys, I have discovered the Internet as a means of probing the marketplace to learn if consumers are confused about various products or services.

CONVENTIONAL OFF-LINE TECHNIQUES

There are a number of ways to find this information. The most often used off-line (non-Internet) ways are mall intercept surveys and telephone surveys. Both have advantages and disadvantages.

Mall intercept surveys work best for consumer packaged goods and other branded products. This method allows for highly comprehensive screening. Costs are high but manageable and depending on the length of the survey, respondents can be recruited for an incentive fee as low as $5.00 per interview. Consumers can be shown: photos of actual products; the actual products themselves; products with labels redacted, and a shopping or shelf-space environment can be simulated.

Mall intercept research takes place in certain shopping malls where research centers are located. Respondents are screened and recruited from the mall and brought into the research center where they are asked the survey questions. Telephone surveys require no incentive.

Depending on the screening mechanism, the number of interviews per attempt can be low and this raises the costs.

Telephone surveys do not allow for any visual cues and thus trade dress (i.e. appearance of the product) is virtually impossible to ascertain via telephone. Since the telephone book is they key source for names and numbers, the possibilities are virtually unlimited.

BEST OF BOTH WORLDS

The Internet, in theory, combines the best of both worlds. Internet surveys not only permit the asking of verbal questions and recording verbatim answers, they also permit transmission of visual images such as products, labels, logos and packaging. Internet technology also permits sound transmission. Transmission costs are minimal with an e-mail blast of 5,000 names costing about $800 or $160 per thousand. (Typical mall costs are $30-$40 per interview). Unfortunately, there is no telephone book for e-mail addresses, and in order to use this medium you have to hook into a vendor that has large opt-in consumer panel data bases.

By using opt-in panels, you will bypass all the SPAM filers and anti-SPAM on-line watchdogs. Moreover, you have an instant, real-time tabulation process and, if you offer an incentive, you can request the names, addresses and phone numbers of respondents thus greatly facilitating the validation process. (Validation is a procedure when an independent research organization calls a sample of those who completed the interviews and verifies that they recall taking the survey.)

Another major plus with using on-line surveys is that, despite low response rates, research shows “conducting surveys on the web has proven to be just as effective as other methods, but if offers quicker turnaround, less biased responses, more representative sample and lower cost,” according to an April. 2005, Market Tool White Paper issued by San Francisco-based MarketTools, entitled “Why Online New Benefits and Possibilities”

The MarketTools White Paper goes on to report, “A number of studies have shown that data collected via the web very closely matches data collected through other media. In a study conducted for a MarketTools client in 2004, we compared the results of 90 concept tests. The results…indicate a very strong correlation between on-line and off-line scores.”

LOW-RESPONSE RATES

However, Internet surveys have their problems. For one thing, response rates are abysmal. This clearly is reflective of the high noise level in e-mail in-boxes. InfoUSA, a leading consumer panel name provider, reports the average consumer Internet survey produces a click-through rate of 0.5 to 1 percent and your average business-to-business survey produces a click-through rate of 0.5 to 2%. Also incentives are required. (Click through is the percentage of people who get the survey, open it and complete it.) Getting a sufficient number of responses is a pure “numbers game” based on your response rate. If your response rate is 2 percent and you need 200 completed interviews, you will have to send out 10,000 e-mails to obtain a valid number of responses. If your click-through rate is 1 percent it will require 20,000 e-mails.

Bounce-back rates are another problem. Consumers are fickle. They move and change e-mail address. Typical bounce rates for consumer e-mail surveys, according to InfoUSA, is 10 to 35 percent and typical bounce-back rates for Business-to- Business e-mail surveys are 2-8 percent.

NON-RESPONSE DILEMMA

A far bigger problem is the respondents versus non-respondents dilemma. If 2 percent of a universe responds, we know how that 2 percent feels. What about the other 98 percent? To rectify this problem, some kind of back-up or verification is required to make sure the non-respondents feel the same way as the respondents. Mall intercept is probably the most effective way to back-up a consumer study. Telephone won’t work if there is a visual element.

 However, the previously reported data from MarketTools, shows that despite the non-response problem, the tight correlation between on-line and off-line surveys gives comfort to the validity and reliability of the data despite the low response rates.

As any researcher will attest, there is no such thing as either the perfect survey or a totally useless one. Each method has its pluses and minuses. The key is to know and understand the limitations before you embark on the research effort.

To Top