Dr. Joseph Cappella (Annenberg School for Communication, University of Pennsylvania) and I published two journal articles describing reliable, valid, and efficient message evaluation protocol.
The protocol includes:
(a) using a large pool of messages/stimuli to avoid case-category confound (Jackson, 1992; O’Keefe, 2015),
(b) recruiting evaluators from the target population,
(c) using valid and reliable effectiveness measures (e.g., perceived message effectiveness (PME), perceived argument strength (PAS)),
(d) exposing an evaluator to multiple messages,
and (e) ensuring enough evaluations per message.
The first article (Kim & Cappella, 2019, Journal of Communication Management) describes the theoretical and conceptual background of the recommended message evaluation protocol.
The second article (Kim & Cappella, 2019, Journal of Health Communication) provides two empirical analyses to support the aspects (d) and (e), to enhance the efficiency of message evaluation protocol while ensuring the validity and reliability of the evaluation results. The first analysis showed that the position in which a message appears has little or no impact on its evaluation, supporting the validity of multiple-exposure design. The second analysis suggested having 25 evaluations per message can achieve a fair balance between accuracy and efficiency.
Kim, M., & Cappella, J. N. (2019). Reliable, valid and efficient evaluation of media messages: Developing a message testing protocol. Journal of Communication Management, 23, 179-197. doi:10.1108/JCOM-12-2018-0132
Kim, M., & Cappella, J. N. (2019). An efficient message evaluation protocol: Two empirical analyses on positional effects and optimal sample size. Journal of Health Communication, 24, 761-769. doi:10.1080/10810730.2019.1668090
Minji Kim, PhD 