Jan. 17-22, 2010
Summary/Action Points/Questions
1. Consider what web platforms to join or to create
2. Check and understand Google Analytics, Social Search, Google Goggles, location based services
3. Apply the workgroup model to MASMI, Product Movers and other new companies in the pipeline
4. Are we in or out of alignment (use workgroup model- see below), are we meeting the mission?
5. Ask young managers to create the workgroup model for the company and compare with what management thought. Do this frequently to know if out of alignment.
6. Take into account the workgroup model when defining strategy
7. Record over a period of one week all activities in a spreadsheet every time there is a change in task and compare to priorities/plan
8. Think of a good cause that can benefit from business experience and get involved
9. Which is our sandbox? We cannot be everything to everybody. How can we master saying no to non target customers?
10. Seek and find adjacencies to existing products, customers and capabilities to counter act economic crisis (slightly contradicting to point 7)
11. Books to read:
a. The greatest trade ever (Zuckermann)
b. Too big to fail
c. Presidential Decision making (R. Porter)
d. Different Youngme Moon
e. Disc chase retail services book
12. Judicious defense=focus on operating efficiencies not on employee reduction
13. Avoid kneejerk cost cutting reactions during recessions
14. 10% batting average in VC is very good, what is a good batting average for initiatives in a service business model
15. In economic crises: keep 4 initiatives, drop 4 and invest more in 2. Make big bets. 4-4-2
16. We want leaders who are optimistic
17. Apply Mediator tactics (see below) when negotiating
18. Visual illusion- Attention Experiment
Click on above link: This was a visual attention experiment conducted by Becklen and Cervone(1983) to show that the human mind has its limits. There is only so much the brain can process at a particular time and it must be selective in what it chooses to filter. Watch this short clip very closely...
This is just a summary, if you are interested in getting the full notes leave a comment
thanks
Friday, January 29, 2010
Monday, January 4, 2010
The Web Buzz
Web Buzz or Digital Buzz is a new way to gain insights about how people express themselves about your brand, company, cause, issue or you.
In addition to traditional ways of finding out what the consumer perceptions are companies like : www.thewebbuzz.com can now mine consumer insights from the social web.
Here is how it works:
- share of mentions
- positive, negative and neutral sentiment
- why the sentiment
This type of reporting is available globally in any language based on annual subscriptions with monthly or weekly reports.
Visit www.thewebbuzz.com and register for more information
In addition to traditional ways of finding out what the consumer perceptions are companies like : www.thewebbuzz.com can now mine consumer insights from the social web.
Here is how it works:
- an ontology/taxonomy is created for the product category or industry of interest
- with the help of a web-crawler all the sites are identified were relevant conversation "traffic" takes place
- the following information is harvested in automated way from selective web-sites:
- share of mentions
- positive, negative and neutral sentiment
- why the sentiment
This type of reporting is available globally in any language based on annual subscriptions with monthly or weekly reports.
Visit www.thewebbuzz.com and register for more information
Friday, April 11, 2008
Online Opinion Panels
One of the definitions for panels is: “Large numbers of people taken from a given universe of people, who agree to regularly participate in market research surveys. A more appropriate name for this kind of panels is: “Opinion Panels” or “Access Panels”. On a less serious note: not to be confused with instrument panels, or chipboard panels but very close to the function of (…back to being serious) an “expert panel”, the purpose of which, is also to generate information/knowledge/wisdom! The objective of this paper is to shed some light on the under-represented area of opinion panels (especially the ones utilizing online data collection methods) in the market research literature. It will attempt to take a holistic view on panels and discuss issues ranging from panel recruitment and management to the technology required and the relevant accounting and finance matters that revolve around panels. This will be attempted taking the global perspective of a current provider of global panels.
Chicago, February 2006
Table of Contents
Introduction
Recruitment of Panelists
Panel Representivity
The Issue with Professional Respondents
Ways to recruit panelists
Panel Management
Panel Types
The use of Panels
Treat them like Customers
Panelist Utilization
Sampling
“Friendly” Algorithms
Rates that matter
Quota Management
Technology
Infrastructure (load balancing, risk mitigation)
Panel Management System
Sampling System
Data Collection Platform and Tools
Broadcasting (ISP relations, multilingual)
Multimedia
Financials
The Size & Volume dilemma
Incentives
Pricing
Attrition
Data Collection Methods
Conclusions
Executive Summary
1. Introduction
People in the market research industry speak of the democratization of research. Creating and managing global panels is now theoretically possible, for any single-country research agency with enough financing to expand their platforms to cover more languages. Of course this is a naïve way of approaching global panels. Not only local knowledge but also local presence is still an absolute necessity in order to provide consistently high quality research.
Mail Panels exist in the US since 1941. When the internet became part of our lives it was natural to start moving to online data collection since it is cheaper and fieldwork can be carried out much faster. In most of the rest of the world however, mail panels were not as popular so things evolved the other way round. First came the internet and then panels as a good way to have access to email addresses and avoid spamming. This phenomenon can be seen as a “geographic paradox”!
Another reason why opinion panels are now so popular will be illustrated with a few statistics from the US again: Ten years ago to complete one telephone interview seven calls were needed, today we need to call 35 people to get one interview. If however we use a panel to select respondents for a telephone survey we return back to a seven-to-one ratio as it was 10 years ago using RDD[1].
2. Recruitment of Panelists
Panelists need to opt-in participating in a panel at least twice and typically they provide, demographic information for themselves and the members of their households. Based on this information and additional screening information collected in advance or just before taking a survey, panelists qualify to participate in individual surveys.
Figure 1. [2]
There is a minimum age limit that needs to be observed when recruiting panelists which defers by country. Minors can be interviewed after acquiring permission from at least one parent (panel member).
2.a. Panel Representivity
A panel is a sub-sample of the population of all households or all people. A sample drawn from a panel is thus only representative of the panel; unless the panel itself is randomly selected from the population, then the sample can be representative of the population[3].
Random Digit Dialing can be a way to recruit a random panel – provided all members of the population have equal chance to be selected and in addition they agree to participate. Realistically this is a) not achievable since some people only have mobile phones, some do not have any phone (still possible in emerging markets), and very few from those contacted will agree to become members of the panel, and b) in most developed countries it is much more expensive to recruit via phone versus online.
Consequently if a panel cannot be randomly selected a randomly selected sample drawn from it cannot be considered representative. However for all business decision making purposes it has been proven through parallel studies that a balanced sample according to demographics and other criteria combined with post stratification can be successfully used.
The bottom line is that it is not so important to understand if a survey is national representative, or just representative of the online population, or of any other sub-group of the population. What is much more important is to be comfortable to take a business decision. A data end-user should be confident that the business decision would not be different if the data collection method was e.g. Random Digit Dialing without the usage of a panel.
2.b. The Issue with Professional Respondents
One of the most important issues that access panel companies have to deal with is that of "professional respondents". It is key to avoid recruiting them as much as possible by not recruiting from sites that offer points or miles for different marketing related activities. Also it is important to avoid luring people to join a panel with the promise of money being the main attraction. Ideally people should join panels because they have an opinion and they want to be heard. "Incidentally" they will get some money or participate in sweepstakes but it should be clear that they cannot become rich from such participation. It is also key to have a process in place that helps identify patterns that would indicate or even prove that certain panelists are professional respondents in which case they should be purged from the panel. Another very important reason to purge a panelist from a panel is when it is discovered that they participate in multiple panels. This is a situation that exists today and makes it difficult for a panel provider to know how many times a panelist has participated in surveys, unless appropriate processes are in place that attempt to prohibit this.
Figure 2. [4]
2.c. Ways to Recruit Panelists
There is a number of appropriate ways to recruit panelists but of course like everything in life they have their pros and cons. Here is an attempt to provide a comprehensive list of double (or triple) opt-in recruitment methods:
Telephone (Random Digit Dialing – RDD)
In the absence of mobile phones and in markets where the telephone penetration is 100% this could have been the ideal way to recruit panelists. The reason being it is the closest to “everyone having equal chance to be selected”. But then again this is a very expensive way to recruit people to a panel – almost cost prohibitive in some countries. Another disadvantage of this approach to add to the cost, is the so called “catastrophic non-response rate”[5].
Email Campaigns
If one can get access to email lists that are not used for recruitment by other panels (and will not be used in the future for this purpose) then an email invitation can be sent that highlights the aspect of opinion sharing as opposed to earning money. Payment can be based on CPM (cost per thousand emails), CPC (cost per click on the URL provided in the email), CPA (cost per acquisition).
Web Banners
This is probably the most common way of recruitment. A panel owner has the option to contact web-sites directly, through affiliate networks to which those web-sites belong or through a media agency. Again here the banner content should focus more on making opinions known and impacting the world around us as opposed to “becoming rich”!! The higher the number of web-sites that participate in recruiting the more representative the panel will be. Payment can be based similarly to email campaigns to CPM, CPC or CPA.
When it comes to the actual recruitment deal (say of online panelists) between the panel owner and the web-site that has access to its visitors or -even better- subscribers there are various ways to go about it. The panel owner can pay cash for exclusive recruitment based on CPM, CPC or ideally CPA. Another possibility is to provide research services in return to recruited panelists – barter deal in other words. The recruitment could be for more than one purposes i.e. co-registration.
Face-to-Face
This recruitment method can be used at events, shopping malls, door-to-door (even though very expensive). Similarly like telephone the prospective panelists needs to take a note of the URL they need to manually type in their browser in order to go to the recruitment landing page.
Advertisements in Publications
An ad in a newspaper or targeted magazines, say a panel owner requires panelists that are very active with their garden, a gardening magazine might make sense in this case.
Offline syndicated research
A good way to collect email addresses to be used to invite people to participate to a panel is to ask at the end of each offline survey for the email address and a permission to contact them for this purpose.
With all the above methods an invitation is extended to click or insert a URL in ones browser and visit the recruitment web-site. On this web-site the expected panelist activity is explained once again and the panelist to be needs to go through a short questionnaire and provide information about herself and her household members (first opt-in). An email will follow from the panel company inviting the prospective panelists to create a user ID and a password to access the panel web-site (second opt-in).
Referrals (or snow-balling)
Generally referrals is not considered to be best practice unless it is kept in limits (say up to 3 referrals, per panel member, maximum) – and even then it is subject to the ratio of people recruited as a result of a referral compared to the total number of panelists. If for example one third of the panel came through referrals then the Representivity of the samples drawn from such a panel is questionable. An example of how not to do things is a specific panel company in the US that recruit only through referrals and the person who refers gets paid every time the panelists she recruited complete a survey. It goes even further – reminding us of a pyramid scheme – part of the payment to the recruits of the recruits also goes to the original panelist who invited them to join the panel.
There are 2 options: to recruit individuals only, or to recruit representatives of households and use them to also invite other members of the household to participate in surveys.
One real issue with online panels in specific is not knowing 100% that the registered person is who they say they are and their demographics and other details are reported truthfully. There are countries like South Korea where the reporting of the social security number is not a taboo like it is in the US. This way the true identity of a person is guaranteed. However even then there is the odd chance that the young son will be filling in surveys for the mother who is shown to be registered as a panelist. The only way to pick this up is through validation by using post study analysis of completion patterns.
3. Panel Management
3.a. Panel Types
A general population/consumer opinion panel is the most generic form of a panel and the one mostly used by CPG companies as well as other industries. As the title of this section suggests there are many different types of panels.
General Population / Consumer Panels
General opinion and Consumer panels are more or less the same and they can be recruited to represent the general online population, or they can be skewed towards demographics that are mostly used for CPG research such as females 25-55 or mothers with young children etc.
Sub-panels
These are panels within the general population panels for which specific attributes are available that place them in a certain homogeneous group of panelists available for specific types of research. An example here would be the Automotive Panel. These are people from the General population panel from which we have information on car ownership, intention to buy, preferred makes etc. This way low incidence research is made first of all possible and is not cost prohibitive. Other examples of sub-panels are Patients, Mothers, Teachers, Farmers, handicapped etc.
Specialty Panels
A specialty panel is one that has members who would not necessarily become members of the general population panel (GPP) or at least not enough of them can be found in a GPP. Special targeted (expensive) recruitment methods need to be applied in order to find and recruit these profiles of panelists. Some examples here are: Physicians, IT Professionals, CFOs, CEOs, HR Professionals etc.
Business to Business (B2B)
This is a sub-set of the Specialty Panels. Most of the SPs will be used for business to business research but not all of them. To be more specific about B2B, IT Professionals, CFOs, CEOs are used to provide information about their business to other businesses.
Proprietary Panels
These are panels that belong and are used by one company only for their own purposes. This can be managed by a research agency exclusively for the owner or the owner can opt to manage their panel themselves. Insurance companies, Banks, GSM providers are companies which own large transactional client databases. To turn those into active panels would make a lot of sense from their point of view.
3.b. The Use of Panels
Panel research and online research (or otherwise known as Internet Research or Web Research or CAWI) seem to be used in an interchangeable way by some. The thing is the former is a sample source and the latter a data collection method, both absolutely necessary in order to be able to talk of any market research! There is online panel research, offline panel research, and even online non-panel research. Consequently opinion panels can be used as a sample source with any data collection method known to mankind: face-to-face, CATI, CAPI, IVR, SMS, HAPI and CAWI. Panels can also be used to recruit participants for qualitative research (e.g. FGDs and IDIs). Mystery shoppers can also be recruited from panels provided special care is paid on product category exposure.
3.c. Treat them like Customers
A panel can literally be treated as a fixed asset in financial terms. Such a unique asset with a mind of its own needs to be nurtured and taken care of! It needs to be kept satisfied for at least 3 full years (asset amortization period!!). There are of course other reasons why a market research client would want a panel to be regularly refreshed. More details about panel attrition under paragraph 6.d.
In this paragraph let’s concentrate on the need for the panelists to be treated like customers. The goal is to achieve customer loyalty so as not to defect or become inactive, in which case the investment to recruit the panelists is lost. Another reason is that loyal and committed panelists are also more truthful and honest (will not want to harm a web-site/an institution they like).
What ways are there to create loyal panelists?
· Give them choices such as: how many invitations to receive in a year, data collection methods to be contacted with etc.
· Give them engaging surveys, questionnaires designed in a user friendly way
· Through the web-site provide a feeling of belonging through the panelist community
· Provide rewards for participation – ideally creative rewards (e.g. gift certificates, contribution to charities)
· Provide useful content on the web-site as well entertainment (games, Quiz) where possible
Figure 3. [6]
3.d. Panelist Utilization
This is quite a controversial subject. First of all there does not seem to exist common understanding on how many times a panelist can participate in surveys during a year without being considered an “expert respondent”.
This is a good point in this paper to make a strong distinction between Professional Respondents and Expert Respondents. It seems that in many publications these 2 types of panelists have been confused for one. Professional respondents participate in surveys just for the money and they tend to be in a hurry to fill in a questionnaire to cash the incentive and move on to the next one. Many of them are unemployed or they want to subsidize their income through participation in research and direct marketing activities. Through their urge to finish quickly they tend to click say …“just the middle box” thus providing untruthful responses. An expert respondent on the other hand is someone who participates in a great deal of surveys or is a market research professional who knows how questions are asked and for what purpose, but is not necessarily dishonest. The downside here is when they are exposed to brands that they would not know otherwise and their responses in subsequent surveys are biased.
Many agencies declare that they use their panelists only 12-14 times a year. Others quote number of invitations to participate as opposed to actual participation. One specific agency in Japan declares without hesitation that they send over 100 invitations to each panelist every year.
Newly recruited panels do not have this problem. Their revenue limitation is the ability of the panel owner to sell more surveys, whilst the revenue limitation of the scenario above is the number of available panelists and the times they are allowed to participate in surveys.
Figure 4. [7]
4. Sampling
Sampling is the process of drawing a sample from the panel that fulfills certain criteria, often referred to as quotas. Due to the large panel sizes required in order to provide a viable sample source even for low incidence populations a highly sophisticated sampling application is required. As mentioned under panel recruitment panelists can be selected based on demographics, but also brand and category usage, lifestyles and attitudes, ownership health conditions etc.
4.a. “Friendly Algorithms”
Even for basic samples an algorithm is required for balancing samples according to the given criteria. The question many research buyers ask is: can a representative sample be drawn by a panel that is not representative? If yes selecting a stratified sample according to demographics enough or is there a need to select a sample according to additional criteria in order to be as close to a randomly selected sample as possible? Criteria such as lifestyles, attitudes, brand or category usage etc.
Another opportunity to use an algorithm is in the area of probability sampling. This is a module of the sampling engine and it is used (by Synovate ViewsNet) to select panelists to be invited based on historical behavior in terms of survey participation.
A third example is an algorithm for balanced responses. See 4.c.
4.b. Rates that matter
A question that is often asked by clients of panel owners is : Give us an indication of the quality of your panel? What are good metrics in order to do this? Here is a selection of rates that are currently used:
Response rate: This is the number of people who took the first step in filling in a survey but did not necessarily complete divided by the total number of people who were invited to participate.
Cooperation Rate: This is the number of people who completed a survey divided by the number of people who qualified from those invited.
Completion Rate: Same as Cooperation rate
Drop-out Rate: This is the number of people who discontinued voluntarily filling in the survey divided by the total number of people who started the survey
Termination Rate: This is the number of people who were discontinued involuntarily as a result of a screener divided by the total number of people who started the survey
Incidence Rate: This is the number of people who fulfill the selection criteria to participate in a survey divided by the total number of available panelists. This rate requires some more attention since it is one of the biggest challenges to get right and is crucial at the proposal stage. Usually clients are much more optimistic than realistic in terms of incidence rate. Proposals tend to be based on higher incidence rates than what really is the case and then the result is cost disputes between the panel owner and the panel buyer.
The answer to the question asked at the beginning of this paragraph would have to be: higher than average Response and Cooperation rates are positive indicators for the quality of a panel.
Figure 5. [8]
4.c. Quota Management for online panel projects
Quota management is… as strange as it may sound… linked somewhat to client expectations management. It seems that researchers and clients alike have different expectations depending on
The geography they are in. In the US for many it is acceptable to balance the outgoing invitations (outgoes) to panelists according to what is required and hope that it is close to the ideal requirement – this is possibility 4-Leave as is in Figure 5 below. Some clients in the US and
elsewhere are OK to post weight the results in order to reflect the required quotas (possibility 3 in Figure 5). Some are not OK to do this.
In Japan for example and surely in some other countries in Asia and in Europe many clients expect that the quotas will be met exactly as required (possibility 1 in Figure 5). In other words if a quota cell is full we stop accepting responses(the guillotine approach) and if one is not full yet after we have send all the outgoes initially planned, we keep sending email invitations until each and every cell is full. This is obviously a very unpredictable and expensive way.
There is another way to do this and this is possibility 2- “Close to ideal” in Figure 5. The aim is to achieve balanced responses as opposed to just send balanced outgoes and see what happens. This can be achieved by sending the email invitations for participation in waves. Each wave of outgoes is adjusted according to the responses received up until the time of the next wave of outgoes. In other words each wave “learns” from the results of the previous one in terms of returns and adjusts in order to come as close to the quotas as possible by the time the last invitation is sent (according to an initial estimation based on incidence and cooperation rates). This approach works best when the fieldwork time is longer than 6 days since the average response time for all respondents invited drops if we keep sending emails until the last day. However if the multi-wave approach is utilized for the first half of the fieldwork time only the negative effect of including mainly fast responders is reduced.
5. Technology
The flow chart shown in Figure 6 shows the systems required in order to run research using panels. Most of the components belong to the online data collection method but some such as the databases for panel, the return system and the sampling system itself are used for all data collection methods.
Panel Management System
Sampling System
Internet Surveys
Panel DB Sampling DB
Returns DB
(Oracle)
Returns System:
Internet
Mail
IVR
Broadcast
Engine
Dimensions DB
(SQL Server)
Other Returns
Mail
Telephone
Email Management
Email undeliverable and bounce backs
Outgoes
(phases)
Emails
URL links
Postal mail
CATI
Figure 6. [9]
5.a. Infrastructure
Referring to the online data collection capability, if it is infrastructure for a single country, multiple countries or global the number of server locations is dictated by the need to mitigate risk (at least 2 locations required) and not so much by the internet speed (available bandwidth between countries and continents), or the need for every location to have their own servers to manage online surveys.
Even for offline research, servers and specialized applications are required for the sampling activity. A few conditions need to be fulfilled: there should be an automated load balancing mechanism for all available servers, fast scalability of each one of the 2 locations in order to cover the workload in case one facility is destroyed or is out of order.
5.b Panel Management System (PMS)
This is the web-site and the back-end application required to recruit and store panelist data of panelists who have access to the internet. The panelist web-site is used as a way to communicate, to do marketing, to reward, to entertain and to create loyalty amongst the panel members and their panel (or the panel owner).
It is advisable for the panel recruitment web-site to be totally different from the public web-site of the agency that owns the panel. There are a few good reasons for this advice: one does not want to make it easy for the competitors to register on one’s panel and have access to information that one would not necessarily want to share. Also competitors of clients might do the same in order to spy on the products their competitors test.
A world class Panel Management System should be able to handle extremely large and diversified global panels that can deliver global samples out of the available panel members. It should support the panel owner to attain highest retention rate and by extension lowest acceptable attrition rate, through highly satisfied panel members. Finally it should support the optimized usage of panelists and high quality of information from them.
5.c. Sampling System (SMS)
This is a system that could cost 20,000 US$ or 2,000,000 US$ (maybe this is slightly exaggerated!) to develop, depending on functionality (such as sample selection, elimination, user interface, returns, probability sampling, balanced responses etc.). The latter is more of a world class sampling system in terms of range of functionality, speed of service and quality of sample. It is a system that is able to draw a project-ready, consistent sample on the first attempt within minutes from submission into the sampling system.
5.d. Online data collection platform and tools
Since online is the most widely used data collection method with panels with growing importance, the technology platform used in conjunction with the other applications is of paramount importance.
Some examples of platforms widely used are MR Interview from the SPSS Dimensions suite of research applications, Confirmit, GMI.
SPSS Dimensions is the only truly open architecture platform that allows companies to connect applications other than the ones provided from SPPS. For example a company could use MR Interview and MR Tables for online data collection and reporting, but use NIPO for CATI and CAPI and a proprietary HAPI and IVR application, all connected through an interface : Data Source Components (DSCs) to the SPSS Dimensions platform.
Confirmit seems to be one of the most popular online data collection and reporting engines at least among the top 10 global market research agencies, and GMI is mostly used by smaller companies as an ASP only. GMI also offers panels in many countries whilst FIRM (the owner of Confirmit) is focused on technology. This is a limited selection of platforms used just as an example.
5.e. Broadcasting
An application is required in order to manage the vast amounts of email traffic from and to the online data collection servers. What is key here is for this application to be able to handle multiple languages when it comes to global panels (including double byte languages). Also the broadcasting system should be able to balance broadcasting in order to take into account own system load limitations and also ISP limitations. For every panel company it is of ultimate importance to stay on the white lists of the ISPs otherwise 2 things can happen:
a) certain ISPs might slow down the traffic coming from a web-site that is suspicious (because of the volumes) and
b) a panel URL might be moved to the black list if there is a suspicion that spamming is taking place.
It is advisable for panel owners to have direct relationships with all the ISPs they depend on and also to observe load/traffic limitations that are imposed by the ISPs or to negotiate separate treatment after proving that no spamming will ever incur through the panel owner’s URL/IP addresses.
5.f. Multimedia
With the internet being totally part of our lives new capabilities are made available to research agencies. All online questionnaires can take full advantage of the world of multimedia and use, photos, film and sound all together or separated. This was also possible with CAPI but the difference here is that it is real time and centrally managed – no need to worry about different versions of stimuli for example (the only worry is different versions of internet browsers). The downside is that for highly confidential materials there is no real guarantee that they cannot be copied. Even though there is special security software that would not allow to a panelist to download the photo of a new product or a new TV advertising spot, a panelist can always use a camera to film the screen or take a photo of the screen. A good piece of advice here is that if something is really confidential and should under no circumstances leak to competition then maybe it is better if it is not shown on the Internet.
6. Financials
6.a. The Size & Volume Dilemma
Owning panels is a great way to increase profitability for a full service market research agency. A precondition for this is that, a critical mass of business goes through those panels especially if we utilize them through online research. When converting offline research (say telephone) to online and panel research, clients have the expectation that the price will go down. This is a justified expectation as long as it is not over say 20%. Based on experiential data from one company no more than a 15% reduction in price is realistic over a wide range of study types with different deliverables ranging from just clean data (ASCII file) to a presentation of results with recommendations and action planning. This price reduction ideally should be coupled with a gross margin improvement of over 20% thus achieving an overall profitability increase for the full service agency.
This is a definitive win-win situation since from the client’s perspective the deliverable is faster and cheaper. Even though better is the third element of the clients’ wish list it is not always easy to convince clients that all three requirements are met when converting from offline to online. As a matter of fact a valid position of the MR industry or any other services industry for that matter could be that only 2 of the 3 requirements can realistically be offered. A client can whether expect a service to be provided faster and better at a higher price, or cheaper and better but it should take longer time to deliver, or lower quality service delivered faster and cheaper. One might go as far as claiming that delivering all 3 is rather a utopia!!
Recruiting a panel is a substantial investment for the panel owner. The good thing about this investment from an accounting point of view is that it can be viewed as capital expenditure and it can be amortized over a period of 3 years. The bad news is that only the incremental panel recruitment can be considered as capital expenditure, not the recruitment that aims to replace panelists that defect or are purged from the panel.
Now the dilemma referred to in the heading of this section is about how large the panel should be (at the beginning) in order to accommodate the volume of business and be profitable for the panel owner. There are a few issues here to be taken into account:
- What is the profile of the respondents required in the majority of the research projects to be sold initially? What is the incidence range of the samples used most often?
- What participation limitations are imposed to the panelists in terms of how many times per month/year they are allowed to participate in surveys.
- Is the limitation the ability of the MR agency to sell or will it be the number of available panelists
- At what percentage of the cases does it stop making sense to top-up the proprietary panel with 3rd party panels?
- What is the maximum possible revenue per panelist and how to make sure that this is met before a further increase of the panel size is initiated?
- In order to avoid this dilemma is it possible to recruit panelists without paying and then pay the web-site owners from where the panelists originated on a per usage basis?
- In how many months is it acceptable to return the investment in recruiting the panelists
Once all the above questions are answered then hopefully a formula can be created that can suggest the ideal panel size for a given revenue projection at a given margin.
6.b. Incentives
There are different ways to incentivise panelists to participate in surveys. The guiding principle should always be that whatever the remuneration, it should not be enough to make a panelist “rich” so that professional respondents will not be encouraged to join.
A very popular way is to use redeemable points that are earned every time a panelist fills in a survey. The number of points should depend on number of minutes and incidence of respondent.
The points can be redeemed in various ways: cash by check or bank transfer, gift certificates or other forms of vouchers, multiple participation in sweepstakes depending on number of points, purchase of products or services through specific web-sites. One issue here when it is left up to the panelists to decide when to redeem, is how to treat the unredeemed points at the end of a year for accounting purposes. Also when a panelist leaves the panel voluntarily or involuntary and points are unredeemed a way to deal with this “unexpected gain” should be identified in advance.
6.c. Pricing
The pricing of fieldwork using panels can be broken down in the following parts:
- Panel Usage: usually based on a combination of questionnaire length and respondent incidence.
- Labor: this is cost to draw the sample from the panel and when it comes to online research to execute the project
- Hosting: this is the cost for acquiring and maintaining the infrastructure
6.d. Attrition
Attrition is a double edged sword! If the attrition is low say below 20% per year the panel owner will be very pleased! The reason being, having to recruit less panelists in order to maintain the panel size. So from a pure financial point of view low attrition (ideally Zero) is sought after. However if you are a client of panel research you would want the panel you use to be refreshed frequently (say 40%-50% per year) so as to avoid the issue of the professional/expert respondents. Attrition over 40% could easily render a panel to be an uneconomical proposition. Again like most things in life the solution must be somewhere in the middle probably between 0-40%.
7. Data Collection Methods
This paper would not have been complete if we did not touch on the ways data can be collected from the panels. Of course the dominating data collection method is online and as time goes by its dominance will increase. However in markets like the US one cannot ignore the mail survey as still a major data collection method utilizing panels. Other methods are telephone (CATI to landline or mobile), face-to-face (PAPI, CAPI, HAPI) and text messaging. Panels can also be used to recruit participants to qualitative research such as Focus Group Discussions, In-Depth-Interviews, Mystery Shopping etc.
8. Conclusions
It is quite obvious that owning and operating panels is a multifaceted and complex business. It is challenging but at the same time it can be very rewarding if a panel owner manages to strike the optimum price Vs quality ratio. This ratio along with the ability of the panel owner to sell will define the revenues, profit and by extension the success of this business.
A constant debate with one’s own conscious will be about how high can one “afford” the quality to be. The sad truth is that this depends on the MR clients’ willingness to accept having to pay a premium for high quality standards. As long as clients will go after low price solutions they encourage low quality panel operators and make it less of a dilemma for the rest of the operators as to whether they should keep “crusading” for high quality panels.
Certain conditions need to be defined and accepted by the MR industry and related bodies and maybe even take it a step further and give to panel operators an accreditation after a thorough audit. This way a minimum quality standard will be set that will be accepted by both research agencies and research buyers.
The objective of this paper was to provide a holistic and honest view about today’s status of access panels globally. They will undoubtedly evolve further from today’s state to something bigger and different in the future. It is important for all concerned and involved to continue sharing views, debating issues and sharing best practice in order for the evolution to be guided towards the right direction. But then again who is willing to define “right”?
9.Executive Summary
As this is more of a “how to” (and “what not to”) approach to panels the practical applications lie in the implementation and management of panels for MR agencies but also from the MR users’ perspective it is a guide that can help select an appropriate panel research provider.
Representivity is always a question for everyone involved and a view is provided in this paper on how to approach information collected from online panels. It is very important for the users of panel research to be aware of the pitfalls of cheap research that involves professional respondents who are described diligently in this document.
Basically this paper can also be viewed as an expanded check-list (similar to the ESOMAR 25 questions) that can guide both panel owners when they build and maintain their panels but also panel users when they choose a panel provider. There are 3 additional questions that the author of this paper recommends to ESOMAR to include to the 25 questions (and become The 28 Questions about online and panel research). These are:
26. What procedures do you apply to ensure a balanced sample of respondents?
27. What are your procedures for supplementing your sample where it is necessary to meet the client's target?
28. What is your panel turnover rate and is it sufficient to ensure that “ internet newbies” are represented?
Finally the transparency provided in all opinion panel aspects might be a unique source of information that was never available before for students, Market Research trainees on the agency side and the client side alike, market research clients new to panels and online (e.g. in countries where internet penetration has not allowed this type of research up to now) or simply clients not as sophisticated and modern as some other.
References
[1] Thomas Mularz , SVP Synovate.
[2] Synovate Panel Operations Guidelines
[3] Jonathan Jephcott “Notes on Representativeness”
[4] Michalis Michael Presentation Europe Open Days 2005
[5] Jonathan Jephcott. In line with the example referenced as footnote 1 with 3% cooperation rate in the US we can hardly speak of random participation.
[6] Synovate Answers to 25 ESOMAR questions
[7] Synovate Answers to 25 ESOMAR questions
[8] Michalis Michael Presentation Europe Open Days 2005
[9] Synovate Business Specifications for Panel Management System
Chicago, February 2006
Table of Contents
Introduction
Recruitment of Panelists
Panel Representivity
The Issue with Professional Respondents
Ways to recruit panelists
Panel Management
Panel Types
The use of Panels
Treat them like Customers
Panelist Utilization
Sampling
“Friendly” Algorithms
Rates that matter
Quota Management
Technology
Infrastructure (load balancing, risk mitigation)
Panel Management System
Sampling System
Data Collection Platform and Tools
Broadcasting (ISP relations, multilingual)
Multimedia
Financials
The Size & Volume dilemma
Incentives
Pricing
Attrition
Data Collection Methods
Conclusions
Executive Summary
1. Introduction
People in the market research industry speak of the democratization of research. Creating and managing global panels is now theoretically possible, for any single-country research agency with enough financing to expand their platforms to cover more languages. Of course this is a naïve way of approaching global panels. Not only local knowledge but also local presence is still an absolute necessity in order to provide consistently high quality research.
Mail Panels exist in the US since 1941. When the internet became part of our lives it was natural to start moving to online data collection since it is cheaper and fieldwork can be carried out much faster. In most of the rest of the world however, mail panels were not as popular so things evolved the other way round. First came the internet and then panels as a good way to have access to email addresses and avoid spamming. This phenomenon can be seen as a “geographic paradox”!
Another reason why opinion panels are now so popular will be illustrated with a few statistics from the US again: Ten years ago to complete one telephone interview seven calls were needed, today we need to call 35 people to get one interview. If however we use a panel to select respondents for a telephone survey we return back to a seven-to-one ratio as it was 10 years ago using RDD[1].
2. Recruitment of Panelists
Panelists need to opt-in participating in a panel at least twice and typically they provide, demographic information for themselves and the members of their households. Based on this information and additional screening information collected in advance or just before taking a survey, panelists qualify to participate in individual surveys.
Figure 1. [2]
There is a minimum age limit that needs to be observed when recruiting panelists which defers by country. Minors can be interviewed after acquiring permission from at least one parent (panel member).
2.a. Panel Representivity
A panel is a sub-sample of the population of all households or all people. A sample drawn from a panel is thus only representative of the panel; unless the panel itself is randomly selected from the population, then the sample can be representative of the population[3].
Random Digit Dialing can be a way to recruit a random panel – provided all members of the population have equal chance to be selected and in addition they agree to participate. Realistically this is a) not achievable since some people only have mobile phones, some do not have any phone (still possible in emerging markets), and very few from those contacted will agree to become members of the panel, and b) in most developed countries it is much more expensive to recruit via phone versus online.
Consequently if a panel cannot be randomly selected a randomly selected sample drawn from it cannot be considered representative. However for all business decision making purposes it has been proven through parallel studies that a balanced sample according to demographics and other criteria combined with post stratification can be successfully used.
The bottom line is that it is not so important to understand if a survey is national representative, or just representative of the online population, or of any other sub-group of the population. What is much more important is to be comfortable to take a business decision. A data end-user should be confident that the business decision would not be different if the data collection method was e.g. Random Digit Dialing without the usage of a panel.
2.b. The Issue with Professional Respondents
One of the most important issues that access panel companies have to deal with is that of "professional respondents". It is key to avoid recruiting them as much as possible by not recruiting from sites that offer points or miles for different marketing related activities. Also it is important to avoid luring people to join a panel with the promise of money being the main attraction. Ideally people should join panels because they have an opinion and they want to be heard. "Incidentally" they will get some money or participate in sweepstakes but it should be clear that they cannot become rich from such participation. It is also key to have a process in place that helps identify patterns that would indicate or even prove that certain panelists are professional respondents in which case they should be purged from the panel. Another very important reason to purge a panelist from a panel is when it is discovered that they participate in multiple panels. This is a situation that exists today and makes it difficult for a panel provider to know how many times a panelist has participated in surveys, unless appropriate processes are in place that attempt to prohibit this.
Figure 2. [4]
2.c. Ways to Recruit Panelists
There is a number of appropriate ways to recruit panelists but of course like everything in life they have their pros and cons. Here is an attempt to provide a comprehensive list of double (or triple) opt-in recruitment methods:
Telephone (Random Digit Dialing – RDD)
In the absence of mobile phones and in markets where the telephone penetration is 100% this could have been the ideal way to recruit panelists. The reason being it is the closest to “everyone having equal chance to be selected”. But then again this is a very expensive way to recruit people to a panel – almost cost prohibitive in some countries. Another disadvantage of this approach to add to the cost, is the so called “catastrophic non-response rate”[5].
Email Campaigns
If one can get access to email lists that are not used for recruitment by other panels (and will not be used in the future for this purpose) then an email invitation can be sent that highlights the aspect of opinion sharing as opposed to earning money. Payment can be based on CPM (cost per thousand emails), CPC (cost per click on the URL provided in the email), CPA (cost per acquisition).
Web Banners
This is probably the most common way of recruitment. A panel owner has the option to contact web-sites directly, through affiliate networks to which those web-sites belong or through a media agency. Again here the banner content should focus more on making opinions known and impacting the world around us as opposed to “becoming rich”!! The higher the number of web-sites that participate in recruiting the more representative the panel will be. Payment can be based similarly to email campaigns to CPM, CPC or CPA.
When it comes to the actual recruitment deal (say of online panelists) between the panel owner and the web-site that has access to its visitors or -even better- subscribers there are various ways to go about it. The panel owner can pay cash for exclusive recruitment based on CPM, CPC or ideally CPA. Another possibility is to provide research services in return to recruited panelists – barter deal in other words. The recruitment could be for more than one purposes i.e. co-registration.
Face-to-Face
This recruitment method can be used at events, shopping malls, door-to-door (even though very expensive). Similarly like telephone the prospective panelists needs to take a note of the URL they need to manually type in their browser in order to go to the recruitment landing page.
Advertisements in Publications
An ad in a newspaper or targeted magazines, say a panel owner requires panelists that are very active with their garden, a gardening magazine might make sense in this case.
Offline syndicated research
A good way to collect email addresses to be used to invite people to participate to a panel is to ask at the end of each offline survey for the email address and a permission to contact them for this purpose.
With all the above methods an invitation is extended to click or insert a URL in ones browser and visit the recruitment web-site. On this web-site the expected panelist activity is explained once again and the panelist to be needs to go through a short questionnaire and provide information about herself and her household members (first opt-in). An email will follow from the panel company inviting the prospective panelists to create a user ID and a password to access the panel web-site (second opt-in).
Referrals (or snow-balling)
Generally referrals is not considered to be best practice unless it is kept in limits (say up to 3 referrals, per panel member, maximum) – and even then it is subject to the ratio of people recruited as a result of a referral compared to the total number of panelists. If for example one third of the panel came through referrals then the Representivity of the samples drawn from such a panel is questionable. An example of how not to do things is a specific panel company in the US that recruit only through referrals and the person who refers gets paid every time the panelists she recruited complete a survey. It goes even further – reminding us of a pyramid scheme – part of the payment to the recruits of the recruits also goes to the original panelist who invited them to join the panel.
There are 2 options: to recruit individuals only, or to recruit representatives of households and use them to also invite other members of the household to participate in surveys.
One real issue with online panels in specific is not knowing 100% that the registered person is who they say they are and their demographics and other details are reported truthfully. There are countries like South Korea where the reporting of the social security number is not a taboo like it is in the US. This way the true identity of a person is guaranteed. However even then there is the odd chance that the young son will be filling in surveys for the mother who is shown to be registered as a panelist. The only way to pick this up is through validation by using post study analysis of completion patterns.
3. Panel Management
3.a. Panel Types
A general population/consumer opinion panel is the most generic form of a panel and the one mostly used by CPG companies as well as other industries. As the title of this section suggests there are many different types of panels.
General Population / Consumer Panels
General opinion and Consumer panels are more or less the same and they can be recruited to represent the general online population, or they can be skewed towards demographics that are mostly used for CPG research such as females 25-55 or mothers with young children etc.
Sub-panels
These are panels within the general population panels for which specific attributes are available that place them in a certain homogeneous group of panelists available for specific types of research. An example here would be the Automotive Panel. These are people from the General population panel from which we have information on car ownership, intention to buy, preferred makes etc. This way low incidence research is made first of all possible and is not cost prohibitive. Other examples of sub-panels are Patients, Mothers, Teachers, Farmers, handicapped etc.
Specialty Panels
A specialty panel is one that has members who would not necessarily become members of the general population panel (GPP) or at least not enough of them can be found in a GPP. Special targeted (expensive) recruitment methods need to be applied in order to find and recruit these profiles of panelists. Some examples here are: Physicians, IT Professionals, CFOs, CEOs, HR Professionals etc.
Business to Business (B2B)
This is a sub-set of the Specialty Panels. Most of the SPs will be used for business to business research but not all of them. To be more specific about B2B, IT Professionals, CFOs, CEOs are used to provide information about their business to other businesses.
Proprietary Panels
These are panels that belong and are used by one company only for their own purposes. This can be managed by a research agency exclusively for the owner or the owner can opt to manage their panel themselves. Insurance companies, Banks, GSM providers are companies which own large transactional client databases. To turn those into active panels would make a lot of sense from their point of view.
3.b. The Use of Panels
Panel research and online research (or otherwise known as Internet Research or Web Research or CAWI) seem to be used in an interchangeable way by some. The thing is the former is a sample source and the latter a data collection method, both absolutely necessary in order to be able to talk of any market research! There is online panel research, offline panel research, and even online non-panel research. Consequently opinion panels can be used as a sample source with any data collection method known to mankind: face-to-face, CATI, CAPI, IVR, SMS, HAPI and CAWI. Panels can also be used to recruit participants for qualitative research (e.g. FGDs and IDIs). Mystery shoppers can also be recruited from panels provided special care is paid on product category exposure.
3.c. Treat them like Customers
A panel can literally be treated as a fixed asset in financial terms. Such a unique asset with a mind of its own needs to be nurtured and taken care of! It needs to be kept satisfied for at least 3 full years (asset amortization period!!). There are of course other reasons why a market research client would want a panel to be regularly refreshed. More details about panel attrition under paragraph 6.d.
In this paragraph let’s concentrate on the need for the panelists to be treated like customers. The goal is to achieve customer loyalty so as not to defect or become inactive, in which case the investment to recruit the panelists is lost. Another reason is that loyal and committed panelists are also more truthful and honest (will not want to harm a web-site/an institution they like).
What ways are there to create loyal panelists?
· Give them choices such as: how many invitations to receive in a year, data collection methods to be contacted with etc.
· Give them engaging surveys, questionnaires designed in a user friendly way
· Through the web-site provide a feeling of belonging through the panelist community
· Provide rewards for participation – ideally creative rewards (e.g. gift certificates, contribution to charities)
· Provide useful content on the web-site as well entertainment (games, Quiz) where possible
Figure 3. [6]
3.d. Panelist Utilization
This is quite a controversial subject. First of all there does not seem to exist common understanding on how many times a panelist can participate in surveys during a year without being considered an “expert respondent”.
This is a good point in this paper to make a strong distinction between Professional Respondents and Expert Respondents. It seems that in many publications these 2 types of panelists have been confused for one. Professional respondents participate in surveys just for the money and they tend to be in a hurry to fill in a questionnaire to cash the incentive and move on to the next one. Many of them are unemployed or they want to subsidize their income through participation in research and direct marketing activities. Through their urge to finish quickly they tend to click say …“just the middle box” thus providing untruthful responses. An expert respondent on the other hand is someone who participates in a great deal of surveys or is a market research professional who knows how questions are asked and for what purpose, but is not necessarily dishonest. The downside here is when they are exposed to brands that they would not know otherwise and their responses in subsequent surveys are biased.
Many agencies declare that they use their panelists only 12-14 times a year. Others quote number of invitations to participate as opposed to actual participation. One specific agency in Japan declares without hesitation that they send over 100 invitations to each panelist every year.
Newly recruited panels do not have this problem. Their revenue limitation is the ability of the panel owner to sell more surveys, whilst the revenue limitation of the scenario above is the number of available panelists and the times they are allowed to participate in surveys.
Figure 4. [7]
4. Sampling
Sampling is the process of drawing a sample from the panel that fulfills certain criteria, often referred to as quotas. Due to the large panel sizes required in order to provide a viable sample source even for low incidence populations a highly sophisticated sampling application is required. As mentioned under panel recruitment panelists can be selected based on demographics, but also brand and category usage, lifestyles and attitudes, ownership health conditions etc.
4.a. “Friendly Algorithms”
Even for basic samples an algorithm is required for balancing samples according to the given criteria. The question many research buyers ask is: can a representative sample be drawn by a panel that is not representative? If yes selecting a stratified sample according to demographics enough or is there a need to select a sample according to additional criteria in order to be as close to a randomly selected sample as possible? Criteria such as lifestyles, attitudes, brand or category usage etc.
Another opportunity to use an algorithm is in the area of probability sampling. This is a module of the sampling engine and it is used (by Synovate ViewsNet) to select panelists to be invited based on historical behavior in terms of survey participation.
A third example is an algorithm for balanced responses. See 4.c.
4.b. Rates that matter
A question that is often asked by clients of panel owners is : Give us an indication of the quality of your panel? What are good metrics in order to do this? Here is a selection of rates that are currently used:
Response rate: This is the number of people who took the first step in filling in a survey but did not necessarily complete divided by the total number of people who were invited to participate.
Cooperation Rate: This is the number of people who completed a survey divided by the number of people who qualified from those invited.
Completion Rate: Same as Cooperation rate
Drop-out Rate: This is the number of people who discontinued voluntarily filling in the survey divided by the total number of people who started the survey
Termination Rate: This is the number of people who were discontinued involuntarily as a result of a screener divided by the total number of people who started the survey
Incidence Rate: This is the number of people who fulfill the selection criteria to participate in a survey divided by the total number of available panelists. This rate requires some more attention since it is one of the biggest challenges to get right and is crucial at the proposal stage. Usually clients are much more optimistic than realistic in terms of incidence rate. Proposals tend to be based on higher incidence rates than what really is the case and then the result is cost disputes between the panel owner and the panel buyer.
The answer to the question asked at the beginning of this paragraph would have to be: higher than average Response and Cooperation rates are positive indicators for the quality of a panel.
Figure 5. [8]
4.c. Quota Management for online panel projects
Quota management is… as strange as it may sound… linked somewhat to client expectations management. It seems that researchers and clients alike have different expectations depending on
The geography they are in. In the US for many it is acceptable to balance the outgoing invitations (outgoes) to panelists according to what is required and hope that it is close to the ideal requirement – this is possibility 4-Leave as is in Figure 5 below. Some clients in the US and
elsewhere are OK to post weight the results in order to reflect the required quotas (possibility 3 in Figure 5). Some are not OK to do this.
In Japan for example and surely in some other countries in Asia and in Europe many clients expect that the quotas will be met exactly as required (possibility 1 in Figure 5). In other words if a quota cell is full we stop accepting responses(the guillotine approach) and if one is not full yet after we have send all the outgoes initially planned, we keep sending email invitations until each and every cell is full. This is obviously a very unpredictable and expensive way.
There is another way to do this and this is possibility 2- “Close to ideal” in Figure 5. The aim is to achieve balanced responses as opposed to just send balanced outgoes and see what happens. This can be achieved by sending the email invitations for participation in waves. Each wave of outgoes is adjusted according to the responses received up until the time of the next wave of outgoes. In other words each wave “learns” from the results of the previous one in terms of returns and adjusts in order to come as close to the quotas as possible by the time the last invitation is sent (according to an initial estimation based on incidence and cooperation rates). This approach works best when the fieldwork time is longer than 6 days since the average response time for all respondents invited drops if we keep sending emails until the last day. However if the multi-wave approach is utilized for the first half of the fieldwork time only the negative effect of including mainly fast responders is reduced.
5. Technology
The flow chart shown in Figure 6 shows the systems required in order to run research using panels. Most of the components belong to the online data collection method but some such as the databases for panel, the return system and the sampling system itself are used for all data collection methods.
Panel Management System
Sampling System
Internet Surveys
Panel DB Sampling DB
Returns DB
(Oracle)
Returns System:
Internet
IVR
Broadcast
Engine
Dimensions DB
(SQL Server)
Other Returns
Telephone
Email Management
Email undeliverable and bounce backs
Outgoes
(phases)
Emails
URL links
Postal mail
CATI
Figure 6. [9]
5.a. Infrastructure
Referring to the online data collection capability, if it is infrastructure for a single country, multiple countries or global the number of server locations is dictated by the need to mitigate risk (at least 2 locations required) and not so much by the internet speed (available bandwidth between countries and continents), or the need for every location to have their own servers to manage online surveys.
Even for offline research, servers and specialized applications are required for the sampling activity. A few conditions need to be fulfilled: there should be an automated load balancing mechanism for all available servers, fast scalability of each one of the 2 locations in order to cover the workload in case one facility is destroyed or is out of order.
5.b Panel Management System (PMS)
This is the web-site and the back-end application required to recruit and store panelist data of panelists who have access to the internet. The panelist web-site is used as a way to communicate, to do marketing, to reward, to entertain and to create loyalty amongst the panel members and their panel (or the panel owner).
It is advisable for the panel recruitment web-site to be totally different from the public web-site of the agency that owns the panel. There are a few good reasons for this advice: one does not want to make it easy for the competitors to register on one’s panel and have access to information that one would not necessarily want to share. Also competitors of clients might do the same in order to spy on the products their competitors test.
A world class Panel Management System should be able to handle extremely large and diversified global panels that can deliver global samples out of the available panel members. It should support the panel owner to attain highest retention rate and by extension lowest acceptable attrition rate, through highly satisfied panel members. Finally it should support the optimized usage of panelists and high quality of information from them.
5.c. Sampling System (SMS)
This is a system that could cost 20,000 US$ or 2,000,000 US$ (maybe this is slightly exaggerated!) to develop, depending on functionality (such as sample selection, elimination, user interface, returns, probability sampling, balanced responses etc.). The latter is more of a world class sampling system in terms of range of functionality, speed of service and quality of sample. It is a system that is able to draw a project-ready, consistent sample on the first attempt within minutes from submission into the sampling system.
5.d. Online data collection platform and tools
Since online is the most widely used data collection method with panels with growing importance, the technology platform used in conjunction with the other applications is of paramount importance.
Some examples of platforms widely used are MR Interview from the SPSS Dimensions suite of research applications, Confirmit, GMI.
SPSS Dimensions is the only truly open architecture platform that allows companies to connect applications other than the ones provided from SPPS. For example a company could use MR Interview and MR Tables for online data collection and reporting, but use NIPO for CATI and CAPI and a proprietary HAPI and IVR application, all connected through an interface : Data Source Components (DSCs) to the SPSS Dimensions platform.
Confirmit seems to be one of the most popular online data collection and reporting engines at least among the top 10 global market research agencies, and GMI is mostly used by smaller companies as an ASP only. GMI also offers panels in many countries whilst FIRM (the owner of Confirmit) is focused on technology. This is a limited selection of platforms used just as an example.
5.e. Broadcasting
An application is required in order to manage the vast amounts of email traffic from and to the online data collection servers. What is key here is for this application to be able to handle multiple languages when it comes to global panels (including double byte languages). Also the broadcasting system should be able to balance broadcasting in order to take into account own system load limitations and also ISP limitations. For every panel company it is of ultimate importance to stay on the white lists of the ISPs otherwise 2 things can happen:
a) certain ISPs might slow down the traffic coming from a web-site that is suspicious (because of the volumes) and
b) a panel URL might be moved to the black list if there is a suspicion that spamming is taking place.
It is advisable for panel owners to have direct relationships with all the ISPs they depend on and also to observe load/traffic limitations that are imposed by the ISPs or to negotiate separate treatment after proving that no spamming will ever incur through the panel owner’s URL/IP addresses.
5.f. Multimedia
With the internet being totally part of our lives new capabilities are made available to research agencies. All online questionnaires can take full advantage of the world of multimedia and use, photos, film and sound all together or separated. This was also possible with CAPI but the difference here is that it is real time and centrally managed – no need to worry about different versions of stimuli for example (the only worry is different versions of internet browsers). The downside is that for highly confidential materials there is no real guarantee that they cannot be copied. Even though there is special security software that would not allow to a panelist to download the photo of a new product or a new TV advertising spot, a panelist can always use a camera to film the screen or take a photo of the screen. A good piece of advice here is that if something is really confidential and should under no circumstances leak to competition then maybe it is better if it is not shown on the Internet.
6. Financials
6.a. The Size & Volume Dilemma
Owning panels is a great way to increase profitability for a full service market research agency. A precondition for this is that, a critical mass of business goes through those panels especially if we utilize them through online research. When converting offline research (say telephone) to online and panel research, clients have the expectation that the price will go down. This is a justified expectation as long as it is not over say 20%. Based on experiential data from one company no more than a 15% reduction in price is realistic over a wide range of study types with different deliverables ranging from just clean data (ASCII file) to a presentation of results with recommendations and action planning. This price reduction ideally should be coupled with a gross margin improvement of over 20% thus achieving an overall profitability increase for the full service agency.
This is a definitive win-win situation since from the client’s perspective the deliverable is faster and cheaper. Even though better is the third element of the clients’ wish list it is not always easy to convince clients that all three requirements are met when converting from offline to online. As a matter of fact a valid position of the MR industry or any other services industry for that matter could be that only 2 of the 3 requirements can realistically be offered. A client can whether expect a service to be provided faster and better at a higher price, or cheaper and better but it should take longer time to deliver, or lower quality service delivered faster and cheaper. One might go as far as claiming that delivering all 3 is rather a utopia!!
Recruiting a panel is a substantial investment for the panel owner. The good thing about this investment from an accounting point of view is that it can be viewed as capital expenditure and it can be amortized over a period of 3 years. The bad news is that only the incremental panel recruitment can be considered as capital expenditure, not the recruitment that aims to replace panelists that defect or are purged from the panel.
Now the dilemma referred to in the heading of this section is about how large the panel should be (at the beginning) in order to accommodate the volume of business and be profitable for the panel owner. There are a few issues here to be taken into account:
- What is the profile of the respondents required in the majority of the research projects to be sold initially? What is the incidence range of the samples used most often?
- What participation limitations are imposed to the panelists in terms of how many times per month/year they are allowed to participate in surveys.
- Is the limitation the ability of the MR agency to sell or will it be the number of available panelists
- At what percentage of the cases does it stop making sense to top-up the proprietary panel with 3rd party panels?
- What is the maximum possible revenue per panelist and how to make sure that this is met before a further increase of the panel size is initiated?
- In order to avoid this dilemma is it possible to recruit panelists without paying and then pay the web-site owners from where the panelists originated on a per usage basis?
- In how many months is it acceptable to return the investment in recruiting the panelists
Once all the above questions are answered then hopefully a formula can be created that can suggest the ideal panel size for a given revenue projection at a given margin.
6.b. Incentives
There are different ways to incentivise panelists to participate in surveys. The guiding principle should always be that whatever the remuneration, it should not be enough to make a panelist “rich” so that professional respondents will not be encouraged to join.
A very popular way is to use redeemable points that are earned every time a panelist fills in a survey. The number of points should depend on number of minutes and incidence of respondent.
The points can be redeemed in various ways: cash by check or bank transfer, gift certificates or other forms of vouchers, multiple participation in sweepstakes depending on number of points, purchase of products or services through specific web-sites. One issue here when it is left up to the panelists to decide when to redeem, is how to treat the unredeemed points at the end of a year for accounting purposes. Also when a panelist leaves the panel voluntarily or involuntary and points are unredeemed a way to deal with this “unexpected gain” should be identified in advance.
6.c. Pricing
The pricing of fieldwork using panels can be broken down in the following parts:
- Panel Usage: usually based on a combination of questionnaire length and respondent incidence.
- Labor: this is cost to draw the sample from the panel and when it comes to online research to execute the project
- Hosting: this is the cost for acquiring and maintaining the infrastructure
6.d. Attrition
Attrition is a double edged sword! If the attrition is low say below 20% per year the panel owner will be very pleased! The reason being, having to recruit less panelists in order to maintain the panel size. So from a pure financial point of view low attrition (ideally Zero) is sought after. However if you are a client of panel research you would want the panel you use to be refreshed frequently (say 40%-50% per year) so as to avoid the issue of the professional/expert respondents. Attrition over 40% could easily render a panel to be an uneconomical proposition. Again like most things in life the solution must be somewhere in the middle probably between 0-40%.
7. Data Collection Methods
This paper would not have been complete if we did not touch on the ways data can be collected from the panels. Of course the dominating data collection method is online and as time goes by its dominance will increase. However in markets like the US one cannot ignore the mail survey as still a major data collection method utilizing panels. Other methods are telephone (CATI to landline or mobile), face-to-face (PAPI, CAPI, HAPI) and text messaging. Panels can also be used to recruit participants to qualitative research such as Focus Group Discussions, In-Depth-Interviews, Mystery Shopping etc.
8. Conclusions
It is quite obvious that owning and operating panels is a multifaceted and complex business. It is challenging but at the same time it can be very rewarding if a panel owner manages to strike the optimum price Vs quality ratio. This ratio along with the ability of the panel owner to sell will define the revenues, profit and by extension the success of this business.
A constant debate with one’s own conscious will be about how high can one “afford” the quality to be. The sad truth is that this depends on the MR clients’ willingness to accept having to pay a premium for high quality standards. As long as clients will go after low price solutions they encourage low quality panel operators and make it less of a dilemma for the rest of the operators as to whether they should keep “crusading” for high quality panels.
Certain conditions need to be defined and accepted by the MR industry and related bodies and maybe even take it a step further and give to panel operators an accreditation after a thorough audit. This way a minimum quality standard will be set that will be accepted by both research agencies and research buyers.
The objective of this paper was to provide a holistic and honest view about today’s status of access panels globally. They will undoubtedly evolve further from today’s state to something bigger and different in the future. It is important for all concerned and involved to continue sharing views, debating issues and sharing best practice in order for the evolution to be guided towards the right direction. But then again who is willing to define “right”?
9.Executive Summary
As this is more of a “how to” (and “what not to”) approach to panels the practical applications lie in the implementation and management of panels for MR agencies but also from the MR users’ perspective it is a guide that can help select an appropriate panel research provider.
Representivity is always a question for everyone involved and a view is provided in this paper on how to approach information collected from online panels. It is very important for the users of panel research to be aware of the pitfalls of cheap research that involves professional respondents who are described diligently in this document.
Basically this paper can also be viewed as an expanded check-list (similar to the ESOMAR 25 questions) that can guide both panel owners when they build and maintain their panels but also panel users when they choose a panel provider. There are 3 additional questions that the author of this paper recommends to ESOMAR to include to the 25 questions (and become The 28 Questions about online and panel research). These are:
26. What procedures do you apply to ensure a balanced sample of respondents?
27. What are your procedures for supplementing your sample where it is necessary to meet the client's target?
28. What is your panel turnover rate and is it sufficient to ensure that “ internet newbies” are represented?
Finally the transparency provided in all opinion panel aspects might be a unique source of information that was never available before for students, Market Research trainees on the agency side and the client side alike, market research clients new to panels and online (e.g. in countries where internet penetration has not allowed this type of research up to now) or simply clients not as sophisticated and modern as some other.
References
[1] Thomas Mularz , SVP Synovate.
[2] Synovate Panel Operations Guidelines
[3] Jonathan Jephcott “Notes on Representativeness”
[4] Michalis Michael Presentation Europe Open Days 2005
[5] Jonathan Jephcott. In line with the example referenced as footnote 1 with 3% cooperation rate in the US we can hardly speak of random participation.
[6] Synovate Answers to 25 ESOMAR questions
[7] Synovate Answers to 25 ESOMAR questions
[8] Michalis Michael Presentation Europe Open Days 2005
[9] Synovate Business Specifications for Panel Management System
The 2 sides of Inventory Accuracy
With less than 200,000 hits on Google search “Inventory Accuracy” does not exactly qualify as one of the most popular business subjects. Especially if compared to buzz words or phrases such as “Customer Loyalty” (2 million hits), Innovation (134 million hits) etc. However if Inventory Accuracy (IA) is connected to 2 of the biggest unresolved (industry-wide) pains of the retail industry – and not only- then it follows that IA deserves more attention from the business community than it currently receives.
The 2 sides of IA which at the same time represent 2 of the biggest pains of the retail industry are:
1) Inventory Shrinkage: is the difference between the booked inventory a company should have as a result of its sales, purchasing and production and the actual inventory it has on hand. Reasons for shrinkage are: employee and customer theft, Accounting errors, inventory counting errors (IA), perishables, loss & damage, fraud.
2) Out of Stocks (OOS): or Out of Reach (OOR) which is probably a more accurate descriptor for the second side/pain is defined as missing products from the retailer shelves that were supposed to be available to the shoppers to buy. A store’s system might show a certain product in-stock (or on-hand) i.e. in the backroom or on the overhead shelves but as long as it is not available to the shoppers to buy this constitutes the pain in the form of lost opportunity to sell.
Quantification of the 2 pains is offered by different studies. According to the National Retail Security Survey conducted back in 2002 retailers in the US lost 31 Billion US$ due to Inventory Shrinkage. According to a GMA Logistics study CPG manufacturers in the US loose anywhere between 6-20 Billion US$ every year due to OOR.
This white paper discusses ways to improve on IA in order to provide sufficient information to understand the location, the time and the items involved in Shrinkage and OOR so that adequate preventive methods can be developed and implemented.
The 2 sides of IA which at the same time represent 2 of the biggest pains of the retail industry are:
1) Inventory Shrinkage: is the difference between the booked inventory a company should have as a result of its sales, purchasing and production and the actual inventory it has on hand. Reasons for shrinkage are: employee and customer theft, Accounting errors, inventory counting errors (IA), perishables, loss & damage, fraud.
2) Out of Stocks (OOS): or Out of Reach (OOR) which is probably a more accurate descriptor for the second side/pain is defined as missing products from the retailer shelves that were supposed to be available to the shoppers to buy. A store’s system might show a certain product in-stock (or on-hand) i.e. in the backroom or on the overhead shelves but as long as it is not available to the shoppers to buy this constitutes the pain in the form of lost opportunity to sell.
Quantification of the 2 pains is offered by different studies. According to the National Retail Security Survey conducted back in 2002 retailers in the US lost 31 Billion US$ due to Inventory Shrinkage. According to a GMA Logistics study CPG manufacturers in the US loose anywhere between 6-20 Billion US$ every year due to OOR.
This white paper discusses ways to improve on IA in order to provide sufficient information to understand the location, the time and the items involved in Shrinkage and OOR so that adequate preventive methods can be developed and implemented.
Retail Space Productivity
A new buzz word has emerged in the retail space relevant to both retailers and manufacturers: Space Productivity Optimization (SPO) also referred to as Store Mapping or Store Configuration Database or Space Measurement. Similarly to Category Management which was a buzz work for 2 decades before it became an established practice with a definition that everyone agreed to, SPO made a shy appearance about 3 years ago when RGIS (a then inventory services company) started experimenting with a few retailers a software company and a consulting group.
Even though Category Management as a discipline theoretically covers both the macro (product category level) and the micro (item level within product category) space in a retail store it has focused mainly in the micro space. Store mapping is here to address the missing link: the macro space. Retailers view the macro space as “real estate” that needs to return a given profit per square foot thus their continuous effort is to optimize product category space allocation. This need was not as acute as long as revenues and profits were satisfactory to shareholders. As competition in the retail space grows, optimization at the micro space (mainly driven by major manufacturers such as Procter & Gamble, Coca Cola, Kraft and Nestle) is not enough anymore. Retailers in Europe have started much earlier to focus on macro space productivity since they have higher space limitations than the US and they were not allowed to stay open during weekends. A proof for this is that the main software companies that have applications that can meet the macro space needs of large retailers are based out of the UK. It seems that 2007 has been the year that same store revenue increase has become a much more important and difficult objective than in previous years in the USA.
It is almost shocking that major retailers (among the top 20 in the world) do not really know where their products are in their individual stores. Most of them know where the individual stock keeping units (SKUs) should be within a product category and how many facings each one should have on the shelves but they do not know where the product categories are actually located in each store. There are 2 pieces of information that if known can have a tremendous impact in decision making when it comes to macro space allocation: Product Category Adjacencies and Linear Shelf or Square Feet occupied by each category. A simple example will illuminate the point: if we take 2 grocery stores that both sell beer and chips and one of them has chips placed to the right of beer and one has chips 2 aisles further next to other snacks, chances are, that chips sales will be higher in the store that has them after beer in the traffic flow. Now add another variable in the mix the former store has 8 linear ft of chips whilst the latter 12 ft and decision making between the buyers or category managers of beer and chips becomes even more complicated if the right data is not available.
We often hear that the Department VPs of major retailers meet once a month or once a quarter to discuss reallocation of space and the only data they have to base their decisions on is sales per product category per store. If they are lucky they might get space information by product category from a sample of stores, but other than that department space allocation decisions are based on assumptions, gut feeling and who can yell louder!
One way to fix what is wrong with the picture painted in the previous paragraph is as follows:
Decide what software and hardware to use for Initial Data Collection, Data Maintenance, Workflow Management or Interdepartmental Collaboration and the analytical tool that can connect POS data to the now updated floor plans for the creation of Heat Maps.
Decide who will collect the data to build the library of updated merchandised floor plans usually in AutoCAD format (DWG) and how and when this will be done.
Find a cost effective way to maintain the data fresh for acceptable intervals e.g. monthly, quarterly or every time a reset takes place in a store.
Define a process that will allow the constant optimization of space productivity through collaboration between the retailer HQ, the store and the people responsible for the execution of changes/resets.
Space Productivity Optimization with its 2 main deliverables : a merchandised floor plan (usually a CAD drawing) that shows the exact locations of the fixtures with the product categories on them and a query enabled database with all the adjacency and square footage information can be the answer to all these complex questions that retailers and manufacturers constantly debate about.
Even though Category Management as a discipline theoretically covers both the macro (product category level) and the micro (item level within product category) space in a retail store it has focused mainly in the micro space. Store mapping is here to address the missing link: the macro space. Retailers view the macro space as “real estate” that needs to return a given profit per square foot thus their continuous effort is to optimize product category space allocation. This need was not as acute as long as revenues and profits were satisfactory to shareholders. As competition in the retail space grows, optimization at the micro space (mainly driven by major manufacturers such as Procter & Gamble, Coca Cola, Kraft and Nestle) is not enough anymore. Retailers in Europe have started much earlier to focus on macro space productivity since they have higher space limitations than the US and they were not allowed to stay open during weekends. A proof for this is that the main software companies that have applications that can meet the macro space needs of large retailers are based out of the UK. It seems that 2007 has been the year that same store revenue increase has become a much more important and difficult objective than in previous years in the USA.
It is almost shocking that major retailers (among the top 20 in the world) do not really know where their products are in their individual stores. Most of them know where the individual stock keeping units (SKUs) should be within a product category and how many facings each one should have on the shelves but they do not know where the product categories are actually located in each store. There are 2 pieces of information that if known can have a tremendous impact in decision making when it comes to macro space allocation: Product Category Adjacencies and Linear Shelf or Square Feet occupied by each category. A simple example will illuminate the point: if we take 2 grocery stores that both sell beer and chips and one of them has chips placed to the right of beer and one has chips 2 aisles further next to other snacks, chances are, that chips sales will be higher in the store that has them after beer in the traffic flow. Now add another variable in the mix the former store has 8 linear ft of chips whilst the latter 12 ft and decision making between the buyers or category managers of beer and chips becomes even more complicated if the right data is not available.
We often hear that the Department VPs of major retailers meet once a month or once a quarter to discuss reallocation of space and the only data they have to base their decisions on is sales per product category per store. If they are lucky they might get space information by product category from a sample of stores, but other than that department space allocation decisions are based on assumptions, gut feeling and who can yell louder!
One way to fix what is wrong with the picture painted in the previous paragraph is as follows:
Decide what software and hardware to use for Initial Data Collection, Data Maintenance, Workflow Management or Interdepartmental Collaboration and the analytical tool that can connect POS data to the now updated floor plans for the creation of Heat Maps.
Decide who will collect the data to build the library of updated merchandised floor plans usually in AutoCAD format (DWG) and how and when this will be done.
Find a cost effective way to maintain the data fresh for acceptable intervals e.g. monthly, quarterly or every time a reset takes place in a store.
Define a process that will allow the constant optimization of space productivity through collaboration between the retailer HQ, the store and the people responsible for the execution of changes/resets.
Space Productivity Optimization with its 2 main deliverables : a merchandised floor plan (usually a CAD drawing) that shows the exact locations of the fixtures with the product categories on them and a query enabled database with all the adjacency and square footage information can be the answer to all these complex questions that retailers and manufacturers constantly debate about.
Company Meetings
The potential to create positive momentum during a company meeting is great. However it is not enough to choose a nice and expensive location, it is more important to have genuine objectives that are embraced by all participants. One of the objectives has to always be to create excitement and ethusiasm. The kind of enthusiasm that only participative management can create. You might be asking yourselves what's the news here...and what's with the photo.
I can explain more to those interested...you just need to write a comment!
Subscribe to:
Posts (Atom)
Posts
