Lies, Damned Lies, and Statistics

[A post from my blog (tysondeliguy.blogspot.com/). Other well-reasoned, insightful, and otherwise captivating posts located there as well.]

I suppose everyone is faniliar with the quote attributed to Mark Twain: there are three kinds of lies--lies, damned lies, and statistics. I wanted to take a moment today to talk about how retailers and suppliers alike get misled by some of the data that is supposed to guide them to informed business decisions.

Among the variety of errors that seem to be most prevalent is one related to the use of indexes. Let's use an example of a new product (but this could just as easily be the case with someone's consumer segmentation model, etc.). Some smart supplier shows up with some research telling you all the reasons why this product is a great fit for your store. There are a few errors your supplier can find himself in if he's not careful.

#1: When new product research is fielded, generally the research is designed to identify sub-segments of the consumer population who may prove to be targets. Often these segments are demographic (age, gender, income, region, race or national origin, etc.). An index is created to show the relative purchase intent of these various segments. Generally the index has 100 as par, and the range of 80 to 120 is considered on par. Someone will review the data and say that segment A has a purchase intent of 107, whereas segment B has a purchase intent of 95, so you should focus on segment A. Check with your research provider, but generally speaking, any values within 80 and 120 are not considered difference at all, or, to put it another way, they are not statistically significant. Why should you care? Because focusing on one segment rather than another when there is no valid reason for doing so could cause you to choose a strategy that does not optimize your sales. We segment in order to find out where we will have the biggest return--if your segments are not correctly defined, there is no reason to segment. Worse--your segmentation may be incorrect.

#2: Your well-prepared supplier points out that segment Q has an index of 247! You have GOT to target these guys and aim all of your messaging at this segment. Hold on, campers. There's something called incidence: the rate at which a given attribute occurs in a population. So, if segment Q has an incidence of 1.2 in the popluation, it means Qers are 1.2% of the population. The economic impact of a 247 index on 1.2% of the population may be diddly-squat (more marketing jargon). The 128 index for segment C might be far more relevant if segment C represents 24% of the population. So, temper your enthusiasm for a high index score with an appreciation for the size of the segment.

#3: This really shiny PowerPoint presentation shows a range of purchase intent indexes across age groups. All of the age groups except one are within the 80-120 range, indicating that these various groups are more or less equally likely to buy the product--which would mean that you shouldn't segment by age. But, wait a minute! Did you notice that the 18-24 age group indexes 141? So, this product is purchased by young people, and all our messaging needs to be directed at young people, right? Well, no, actually. In this case, you would see all age groups represented except you would see a slight over-representation of this young audience. In the case of a hot prepaed foods shopper, this age group would naturally represent about 12% of the population. A slight over index is not going to drive that % much higher, so you'd still have better than 80% of your population ignored if you exclusively target the 18-24 group. Never take an over index score and then believe that they are the only or most relevant segment.

#4: Sample size. I have seen businesses (suppliers and manufacturers alike) make major strategic investments on the basis of a "study" that had a sample size of 40. Forty people is somewhat better than 5 guys around a table, but not much. There are very good reasons for doing small sample studies, but strategic decisions will generally require studies with a sample size of 225 or more. We generally try to have something on the order of 1,000 or more. Always check the sample size to see if there are enough responses to make the results credible.

Why the lesson on research? Well, it's very self-serving in many ways (who would have guessed), but there is a larger issue. I am convinced that the greatest business results are achieved when a retailer and a supplier work together in partnership, and partnership must be founded on trust--trust in the integiry of both parties and trust in the competence of both parties. Now, the word "partnership" gets tossed around a whole bunch. As Col. Potter from MASH would say: Horse hockey! Real instances of partnership are all too rare. If, as an industry, we're to continue to build the economic potential of this channel, then retailers and suppliers must work in real partnership. Integrity will have to be established on a case by case basis, but competence is a great place to start. If we are going to commit to make fact-based decisions, we must hold ourselves to high standards in acquiring and analyzing data.

As it just so happens, in the Tyson deli group we do a pretty darn good job in this area. That might have motivated me somewhat in writing this post.

I hope today's entry hasn't been too dry, but I do think it's important to use data correctly to make great decisions. Retailers and manufacturers alike don't often have the luxury of making poor decisions.