Cookie versus Panel Based User Counting

comScore released the results of a study they did which claimed that cookie based measurement overstate Unique Users by as much as 150%.

I was not surprised by the results, not because I knew that cookies was over estimating because this is what I would have expected based on who was sponsoring the study. Since the study was conducted by comScore I wouldn't have expected them to come up saying panel data is worse than cookies based counting. If this study were done by Omniture or WebTrends we would have probably heard a different story. For example, if we were to compare wine and coffee and see which one is bad for health, wine company’s research will say wine is good for health while coffee is not. Coffee company will come with their research which will say coffee is good and wine is bad.

It is very likely that the users who participated in the study knew that they were being tracked (and they participated to get freebies) so they developed a tendency to clear cookies and hence skewed the results. There could be several reasons why comScore survey might be correct or not but comScore did not publish those so we can’t say for sure if panel based is really better than cookie based count.

It is likely that comScore is trying to solidify its position as a standard in audience calculation by releasing this study. In my article on Google and Behavioral Targeting I mentioned that Google is putting it’s cookie everywhere and could potentially get in the business that comScore is in currently (measuring audience size). This possibly could be a preemptive move by comScore. Don’t know for sure since I don’t have the details on how this study was done.

The crux of the matter is that no matter what system you use you will never get an accurate count of UU's. Even pages views are reported differently by different systems. Two panel based systems don't report the same numbers and two web analytics tool do not report the same number. To assume one system is not accurate or is better than other system is jumping to conclusions without basis. Both systems have their own advantages and disadvantages.

For individual site owners you need to pick one system, cookie based or panel based, and stick with it and accept the numbers. How does it matter if I told you that you actually get 3,000 users (or visits) and not 5000? What will change? What will you do differently? Have you ever moved from one web analytics system to another? You know they don't report same numbers. Say, your new system reports lower number than your old system than what are you going to do? Which one is correct, old or new? It is about trending and growth. You will probably use same growth and retention strategies no matter what the number is. If you goal is to increase the visitors by X% no matter what visitor count you use you will try to do X% of your current system weather that is cookie based or panel based.

If you are trying to compare sites then do the same thing, choose one system for comparison and stick with it. The same goes with advertising, if you are buying or selling ad inventory based on Unique Visitor count both buyer and seller will have to settle for one system and stick to stick to it.

However, one thing that surprised me in this study was a quote from Tacoda. Their whole business (Behavioral Targeting) is dependent on cookies and now they will validate with panel data because....(...they don't believe the size of their segment? Does that mean they will adjust the reach they claim they have?) Can someone from Tacoda please help me understand it?