Patrick Comer, CEO of Lucid, reviews how sampling practices have changed with the advent on online technology…
A lot has changed since the early days of online market research and sampling.
As with any industry, technology and user behavior have been the driving force. In this post, we take a look back at the evolution of online sampling and what we have learned as we move into a new era of programmatic.
In the beginning, there was panel, and it was good (2000 – 2005)
At the dawn of the new millennium, online surveying was starting to take hold and of course digital respondents were required to fill out those questionnaires. Very quickly, several e-mail driven panel companies were built to match this increase in global demand. The concept was simple – recruit online users to join your panel, register and profile them, they take lots of surveys, everyone wins. Most importantly all the sampling technology at that time was built to recruit and maintain panels. Sampling meant pulling user profiles from your database and sending out emails with a link to the market research survey. Simply, a one to one relationship between the panel and the survey.
The Rise of the Router (2005 – 2007)
Panels worked well but we started to notice a few challenges. First, as the volume of online surveys started to grow, the demands on a panel started to dramatically increase. One would think that just recruiting more panelists would solve the problem, but panels suffer from a diminishing returns problem: the next panelist is less valuable than the ones before it. Additionally, not all users will respond, qualify, and complete a specific client survey.
Thus, suppliers started focusing on sample optimization and the Router was born. If the respondent didn’t qualify for the first survey, then they would be screened against another survey optimizing respondents’ time and qualifying them against multiple surveys at once. As a result, routers improved sample capacity and efficiency. They also created a better experience for respondents by offering them more opportunities to complete a survey during their session.
However, as routers became more commonplace over time, researchers began to question their quality and efficiency.
Router concerns
- Bias: When Survey A is prioritized over Survey B, and these surveys are correlated, certain groups of respondents that may qualify for Survey B are excluded. This may lead to bias. To minimize this risk, most routers make use of random assignment in their design or a combination of both random and priority logic. The process is not as efficient as it could be, and respondents end up spending more time screening than necessary. This leads to a less than ideal experience.
- Control: One often hidden challenge of a router is that it placed all the control and survey decisioning in the hands of the manager of the router. Often times, the router manager isn’t the panel so the decisions on which survey a respondent might be matched to isn’t in the hands or best interest of the supplier.
Research has since shown that there is no evidence to suggest that the bias introduced by routers materially affects survey outcomes (Source). This risk is further reduced in routers with large and diverse survey inventories (Source).
Sample Aggregated: Friend or Foe? (2007-2013)
Even with routing increasing efficiency and capacity, sample companies and research agencies couldn’t get enough delivery out of panels. So new technologies and techniques were created to source respondents without incurring the cost of recruitment.
Will it blend?
To cope with increasing demand and lack of internal supply, sample companies and large sample buyers started down the path of massive sample aggregation. Previously, buyers would only use one panel company per project, but found that using several was more reliable from a delivery and pricing standpoint. Additionally, the sample companies themselves started buying sample from each other at large scale to replace the missing capacity that they were experiencing. Part of the challenge to the buyer was that this brokering was sometimes hidden from researcher view.
How loyal are you?
Seeing the impact of recruiting from brand name loyalty programs, sample companies started sourcing from a broader base of companies that offered some version of points, miles, or minutes to a user base. For most of loyalty-based companies, surveys start out a small part of their overall revenue but eventually become the majority. Sample companies start becoming heavily reliant on the long tail of loyalty programs.
High Tide on the River
River sample respondents, “are invited via the placement of ads, offers or invitations online. River sample may also be referred to as web intercept, real-time sampling, and dynamically sourced sampling.” (Source).
River sampling includes any individuals online who, upon visiting a website and clicking on an ad, are directed to take a survey. Rather than relying on a double opt-in process where respondents confirm their interest, are profiled and invited ahead of time, this method takes place dynamically and “in the moment.” In that moment, we don’t know anything about these respondents, and upon completing a survey we may never see them again.
While river sample opened up a new source of respondents to researchers, it has often been criticized for the following reasons:
River concerns
- Representation: Without a clear recruitment processes, river sampling lacks obvious census representation
- Quality: Respondents in river sample tend to be less engaged, and are more prone to straightlining behavior.
- Profiling: The lack of profiling information makes respondents difficult to track to engage in future surveys.
Within a few years, the combination of aggregation, loyalty, and river sources would become the majority of sample delivery by a number of sample companies with their own proprietary panel growing at a slower rate.
A New Era of Programmatic Sampling (2013 – present)
In recent years, we’ve seen yet another breakthrough in sampling technology. The rise of APIs has given way to a new era of sampling that is programmatic, automated, and marketplace driven.
When we began our API development in 2012, we couldn’t have predicted the tremendous impact that it would have on our business and the industry as a whole.
An API, which stands for application programming interface, allows our application (Fulcrum) to communicate with both supplier applications (e.g., panel databases) and buyer applications (e.g., survey platforms, reporting portals) in a common language that they both understand. APIs completely transform the way online sample is bought and sold.
Programmatic delivery vs. traditional routing
When suppliers take an API-driven approach, they are able to deliver sample more intelligently than routers have been able to do in the past. Suppliers have a deep understanding of their respondents and the surveys they are targeting enables them to accurately match qualified respondents with survey opportunities in real time. Programmatic delivery offers precision and efficiency that traditional routing could not. Essentially, the supplier can view all available survey inventory and recruit the best respondent from their panel. By providing information that suppliers need to decide for themselves which respondent goes into which survey, our API enables suppliers to earn revenue more efficiently while simultaneously providing a better respondent experience according to their own business prerogatives.
Similarly, for buyers, programmatic delivery makes it easier to get the sample required. Not only is the fielding process much easier on project managers, but the efficiency that suppliers get flows through to the researcher through less weary, more attentive respondents.
Everyone Wins in the Programmatic Sampling Era
In the programmatic sampling era, suppliers gain significant cost efficiencies, buyers get sample faster without the headache of manual bidding, and respondents spend less time trying to complete a survey. Everyone wins! We’ve seen LOIs go down and both conversion and incidence rates go up.
Lucid continuously explores and develops the technology to move the industry forward. We love taking new technologies and developing solutions that make online research easier, smarter, and faster than ever before. We are confident that the best days are yet to come.
Patrick Comer – CEO Lucid