How Do I Determine What Sample Size I Need? Part 1: Standard Error and Margin of Error
How many responses do you really need for statistically sound research? This is an important question every market researcher should ask. After all, sampling is a delicate balance between maximizing sample size while minimizing sample cost. In this article, we’ll review the central concepts market researchers should think about when selecting a sample size.
Setting Up An Example:
To better understand the concepts discussed in this article, imagine a client of yours just started making a new technology product for a smart home. While the technology might be ready for consumers, sometimes consumers aren’t ready for the technology. For example, while mobile payment is available in the U.S., the adoption and use of that technology is extremely low because consumers have yet to accept the claim that it is a faster and safer payment method. Imagine your client has the same fear about their own product—their concern is that consumers won’t adopt it. Consequently, they ask you to test how well their new smart home technology product will perform on a measure of “consumer acceptance.” Assume that the “consumer acceptance score” just represents the percent of people who will adopt the product. Also assume that this survey is administered the company’s current customers (N = 100).
What is a target population?
Every piece of research starts with a target population—the entire group of individuals we are talking about in the findings of our research (i.e., who do the results of the research apply to?). In our example, our target population is all of our client’s customers. In an ideal world we would survey all customers, for every piece of research we do. This way, we would always know definitively that the data represents everyone in our target population. Unfortunately, the cost burden on the client (i.e., paying all customers to take every survey we administer) and the time burden on the respondent (i.e., the time it takes to complete all those surveys) makes this impossible. What we are left with is taking a portion of the target population, collecting data from them, and then generalizing to everybody in the target population. This brings up two central concerns:
1. Who is to say that the sample of the target population should represent the total population? Does this generalization make sense?
2. If the generalization does make sense, how accurately does our sample data represent the data of the total population.
What is sample generalizability?
To help solve the first question, we turn to the concept of sample generalizability. Sample generalizability is the extent to which the results from one sample apply specifically or generally to another sample or to a larger target population. There are a few factors to consider when thinking about generalizability. However, the two most important factors are: (1) sample representativeness and (2) sample size.
Sample representativeness
Sample representativeness is the extent to which your sample “matches” your target population. For example, when we think about our example above, our target population is all of our client’s customers. If we sample only male customers, then the results of our research cannot be generalized to all customers because our data does not represent what the target population looks like.
Similarly, when researchers want to generalize their sample to the U.S. population, they sample their respondents by making sure the respondents in their sample are represented by the same proportion that is recorded in the U.S. population. Accordingly, if the 2018 U.S. census data reports that 18% of the U.S. is Hispanic/Latino, then a researcher would want 18% of the respondents in their sample to be Hispanic/Latino.
Incorrectly matching your sample to your population can happen in a number of different ways. The most frequent way researchers end up with a “mismatched” sample and population is by sampling the wrong people. This mistake often happens when recruitment procedures “select” for only a small subset of the population. For example, if in our example we recruited all customers but also included customer who purchased years ago but nothing since, then we’re making two errors in our sampling frame:
1. We are including individuals who might have owned smart home products, but didn’t like it and now are category rejecters.
2. We are not including those who have never purchased smart home products, but would like to—category intenders.
Be careful of how you design your sample recruitment. After all, sampling procedures are one of the main reasons the results of research come into question.
Sample Size
The second factor to consider when thinking about sample generalizability is sample size. In regards to sample size, researchers must ask themselves what is the size of their sample relative to the size of their target population. The closer your sample size is to the size of your target population, the more likely it is that your results of the sample are close to the results of the target population.
In most cases, the sample size is not close to the target population. These means there’s some uncertainty in just how precise your sample numbers are relative to the target population numbers. To quantify this uncertainty, we use standard error.
What is standard error?
Standard error is a measure of how precise a sample estimate is, such as a sample mean or sample proportion. If we think back to our example, imagine you tested the smart home product for your client and 72% of respondents in the sample say they will adopt the product. Calculating the standard error of the estimate (i.e., standard error of the proportion) tells us approximately how close the sample proportion is to the “true” consumer adoption proportion of the population (i.e., all of your client’s customers).
More specifically, if a researcher was to do the same study over again 4 more times, the proportion of consumer acceptance would be slightly different in each study. For example, consumer acceptance scores might look something like this over the course of five studies:
Study 1 – 72%
Study 2 – 80%
Study 3 – 70%
Study 4 – 71%
Study 5 – 76%
As we mentioned above, this variability exists because we are only collecting a sample of individuals. Since a sample is only a portion of the total population, there will always be some sample variability in the estimated proportion. Standard error is the measure of this variability (i.e., standard deviation) for the different samples (called a “sampling distribution”). The only way to get no sampling variability, and therefore the same estimate each time, is to survey the whole population.
To calculate standard error of the estimate (SE), you would use the following equations:
Notice that in the formula the sample size (n) is inversely related to standard error (SE). This means as sample size increases, standard error decreases.
To see this relationship more clearly, we’ve included three graphs below. In each of these graphs, the x-axis is sample size, and the y-axis is standard error. This will allow you to see the relationship between the two variables. It also can be a quick guide for you if you need a quick calculation of standard error for a proportion.
While it is useful to see how precise your estimated parameter is, it is conceptually hard to grasp the concrete implications of standard error on your sample estimate. To help make this more concrete, we can turn our standard error into a margin of error.
What is margin of error?
The margin of error (MOE) is calculated using the following formula:
MOE = critical value * standard error
Notice that this formula takes the standard error and multiplies it by a “critical value.” A critical value is a number on a test distribution associated with finding a significant difference. By multiplying standard error and the critical value, we are calculating the area in which there is not likely to be a significant difference relative to our estimated sample proportion. Said differently, if our sample proportion is 72% for consumer acceptance, the margin of error tells us just how many percentage points above 72% are likely to occur by random chance or random sampling variability if we were to collect another sample.
Similar to standard error, the bigger the sample size, the smaller the margin or error.
Feel free to use the charts as a quick reference when estimated margin of error for your own samples.
What is a confidence interval?
A confidence interval is the range of values within which the target population’s true score will likely be. That is, if you take your sample proportion and subtract the margin of error, this is your low end of the confidence interval. If you add the margin of error to your sample proportion, this is the upper end of your confidence interval. In our example, if our sample proportion is 72% for consumer acceptance and our margin of error is 6%, then our confidence interval would be between 66% and 78%.
Confidence intervals can be created at different confidence levels. While 99%, 95% and 90% are the three used most often, the 95% confidence interval is the one recommended across most industries. The confidence interval percentage, such as 95%, means that if we were to do 100 more studies, 95 of those studies would have sample means that fall within the confidence interval range.
This article only scratches the surface of what is required when thinking about sample size for your research. If you have questions and want to chat with one of our statistics consultants, please reach out to us today!