Home | Login | Register

Criteria for Assessing Research on the Effects of Marketing Communications

Paul N. Bloom, Julie Edell, and Richard Staelin, 1994 [94-123]

View/Order Item >>
Comments from members >>

The purpose of this report is to provide criteria for determining the validity and implications of empirical studies of marketing communications, especially those with public policy implications. We developed these criteria in consultation with potential users—regulatory agency officials, Congressional staffers, consumer advocates, business executives, marketing academics, and trade association representatives.

We begin by discussing the current uses of empirical marketing communications research in public policy making. We next present a model of how marketing communications and other factors influence consumers' beliefs, attitudes, and behavior. We then use this model to present and discuss the research guidelines. Finally, we compare these guidelines with existing research guidelines.

We recognize that there are different types of marketing communications studies—descriptive, associative, and causal—and that moving from descriptive research to causal research increases the requirements for validity. The criteria that follow are organized such that those that apply to all three types of studies are presented first. Those applicable to associative and causal research are presented second, followed by those applying only to causal studies. The final set of criteria addresses the issue of assessing the conduct of the researchers rather than the research study itself.

Research Criteria
Criteria Generally Applicable to All Communication Studies

Sample Selection Criteria

The sampling frame should reflect the population of interest, and the conclusions drawn and generalizations made should be limited to this sampling frame.
It is essential that the sample be drawn from the group that best matches the group to whom the researcher wants to generalize the findings.
The sample should be representative of the population from which it is drawn.
One way to ensure representativeness is to use random or probability sampling. Other sampling procedures should explain how representativeness was obtained.
The sample size should be large enough to provide reasonably precise estimates for all subgroups being studied.
The greater the variation in the constructs being measured and the greater the number of subsamples to analyze, the larger the overall sample must be. Increasing the sample size (along with probability sampling) allows calculation of, and decrease in, sampling error.
Nonresponse rates should be reported, and the effects of nonresponse should be estimated.
Those selected to be in the sample may not respond or may be unwilling to participate in the research. The higher the number of nonrespondents, the greater the danger that the results obtained do not accurately reflect the true results of the population of interest.
Measurement Criteria
Individuals gathering and coding the data should not know the purpose, hypotheses, or sponsor of the research study.
Because many aspects of research are subjective and open to interpretation and influence, those involved in data collection and coding should not be informed of the specifics of the research goals.
The data collection instrument or questionnaire should be pretested.
Pretests help eliminate mistakes. They check the procedures, provide training, and help ensure that the data collection instrument is appropriate for the target audience.
The measures, procedures, and questions should not suggest a response.
The data collection instrument is itself a form of marketing communication. It should be constructed in such a way that it will accurately reflect the beliefs, attitudes, or behavior of each respondent without suggesting a response or divulging the purpose of the research.
Wording and topics in the data collection instrument or questionnaire should be unambiguous and easily understood by the target population. Awareness of the language patterns in the target population as well as careful pretesting should allow the researcher to adapt the language to the target population and the topic under investigation.
Whenever possible, several methods and measures should be used to assess beliefs, attitudes, or behavior.
Using different data collection and measurement methods increases confidence in a study's findings.
Additional Criteria for Associative and Causal Research
Whenever possible, associative and causal studies should use different methods and scales to measure the constructs under investigation.
While there are always research benefits to be gained from using different approaches and measures (Criterion 9), it is especially important that the items tested for causal or associative relationships be measured using different scales to eliminate common method bias.
Associative and causal studies should report the test statistics used and their level of significance.
In order to assess the strength of a relationship, it is necessary to know how the relationship was tested and whether or not one would expect to see this level of association by chance.

Additional Criteria for Causal Research
Research designed to establish causality should rule out spurious associations caused by unobserved variables.
Just because constructs show a strong association does not mean they are causally linked. Constructs can also move together if they are linked to a third construct.
Research designed to establish causality should rule out reverse causality.
Even though constructs show a strong association and are causally linked, it is necessary to also show the direction of causation. For example, changes in consumer behavior may cause changes in a marketing communication rather than the other way around.
Research designed to establish causality should rule out simultaneous causality.
Two constructs can show a strong association because each construct has a causal effect on the other during the period of analysis.
Research designed to examine constructs, such as behavior, that are distant in time from the communication should give strong emphasis to controlling additional factors.
It is difficult to determine the effects of marketing communications that take place over long time intervals, since so many other changes also take place during these intervals. Greater controls are therefore needed to establish causal effects when there is a longer temporal distance between the cause and the effect.
Designs using a control group should ensure that the control and treatment groups are not different prior to the treatment.
Experimental control can be established by having the sample divided into identical control and treatment groups.
When different groups cannot be controlled for a priori, the study should control for differences statistically.
Control can be established by measuring other constructs thought to influence beliefs, attitudes, or behavior and removing their effects statistically.
Criteria for Conduct of Researcher
The researcher should publicly acknowledge all research sponsors and affiliations that might have had input into the research.
Many special interest groups fund or conduct research studies. While receipt of funds should not be a basis for judging the study, full disclosure by the researcher is necessary. The degree of sponsor involvement in the study design should also be noted.
The researcher should make available at no charge or at a nominal charge the research instruments, the sampling plan, and the analysis plan to any interested party.
As these criteria suggest, it is not possible to judge the quality of the conclusions of a research study without full knowledge of the data collection instrument and the sampling and analysis plans.
When generalizing findings to broader contexts, the researcher should temper the conclusions to reflect the study's limitations.
All studies are composed of many tradeoffs that the researcher must make. These tradeoffs can limit the generalizability of the results to populations or situations other than that of the drawn sample.

Paul N. Bloom is Professor of Marketing, the Kenan-Flagler Business School, University of North Carolina, Chapel Hill. Julie Edell is Associate Professor of Business Administration, the Fuqua School of Business, Duke University. Richard Staelin is the Edward and Rose Donnell Professor of Business Administration, the Fuqua School of Business, Duke University.

Included in the following collections:
1994 Working Paper Series Subscription

Comments from members

Please login to view and/or submit comments

View/Order Item

Online version
Your Price: not available
Hard copy
List Price: ~~$5.00~~ Your Price: $5.00

Please login for full access and discounted pricing.

Copyright 2013 MSI | Contact | Terms of Use
MSI and Marketing Science Institute are service marks of the Marketing Science Institute, Inc.