Convenience Samples and YOU!

This post simply calls out the piece in an earlier post that specifically relates to convenience samples for non-profit leaders. Read here to see the original post and get a more in-depth look at the challenges and risks associated with convenience samples.

----

When should your organization use convenience samples? When your question is one of the three below. However, the real danger of collecting data in this manner is that we are always going to be tempted to use it. When we start writing the report, we will want to put numbers in it. If we do, people who do not know better are going to be quoting those numbers as fact and law.

You yourself may be pulled into thinking "roughly 1/4 of my constituents feel this way" because that was the number that came back in the convenience sample. This may lead you and your board to conclude that something is a problem when, in fact, only 10% of your constituents actually felt that way. Alternatively, you may decide something is not important when actually 'TRUTH' was closer to 50% for that issue. In either case, you will never know which it is unless you do a well planned random sample.

A more likely scenario is that you are trying to assess people's preferences and one group strongly over-responds compared to another group. As a consequence, you get the notion that one preference is far more common than the other (e.g. 60% vs 30%) when, in fact, 'TRUTH' was more like 50% and 40%. Certainly, the one preference was still greater than the other; but not by nearly as much as the convenience sample suggested. If you start making decisions based on the convenience sample data, however, you risk upsetting a far greater number of your constituents than you thought!

1. Does anybody, even just one person, do, think or feel something? e.g. 5% of respondents in the convenience sample said they need a vegetarian meal option, so we know at least some folks need a vegetarian option. But, just because no body said they needed a vegan option does not mean that there are not vegans in the program population.

2. Is something obviously really common or obviously really rare? e.g. 90% or more is likely very common, 10% or less is likely very rare, but we don't know the actual rates or even how bias our rates may be.

3. Is one thing clearly more common than another? e.g. 60% of people preferred apples while 20% of people preferred bananas in the convenience sample -> most likely more people prefer apples to bananas, but we don't know how many more.

If you are planning on doing a survey and have been considering a convenience sample, contact me first. Good data may be cheaper than you think!