conjoint experiment political science

Using Conjoint Experiments to Analyze Election Outcomes: The Essential Role of the Average Marginal Component Effect (AMCE)

59 Pages Posted: 12 Jun 2020 Last revised: 22 Jan 2022

Kirk Bansak

University of California, Berkeley

Jens Hainmueller

Stanford University - Department of Political Science; Stanford Graduate School of Business; Stanford Immigration Policy Lab

Daniel J. Hopkins

University of Pennsylvania

Teppei Yamamoto

Massachusetts Institute of Technology (MIT) - Department of Political Science

Date Written: January 20, 2022

Political scientists have increasingly deployed conjoint survey experiments to understand multi-dimensional choices in various settings. In this paper, we show that the Average Marginal Component Effect (AMCE) constitutes an aggregation of individual-level preferences that is meaningful both theoretically and empirically. First, extending previous results to allow for arbitrary randomization distributions, we show how the AMCE represents a summary of voters' multidimensional preferences that combines directionality and intensity according to a probabilistic generalization of the Borda rule. We demonstrate why incorporating both the directionality and intensity of multi-attribute preferences is essential for analyzing real-world elections, in which ceteris paribus comparisons almost never occur. Second, and in further empirical support of this point, we show how this aggregation translates directly into a primary quantity of interest to election scholars: the effect of a change in an attribute on a candidate or party's expected vote share. These properties hold irrespective of the heterogeneity, strength, or interactivity of voters' preferences and regardless of how votes are aggregated into seats. Finally, we propose, formalize, and evaluate the feasibility of using conjoint data to estimate alternative quantities of interest to electoral studies, including the effect of an attribute on the probability of winning.

Keywords: conjoint, elections, causal inference, AMCE, survey experiments

JEL Classification: C25

Suggested Citation: Suggested Citation

University of California, Berkeley ( email )

310 Barrows Hall Berkeley, CA 94720 United States

Jens Hainmueller (Contact Author)

Stanford university - department of political science ( email ).

Stanford, CA 94305 United States

HOME PAGE: http://www.stanford.edu/~jhain/

Stanford Graduate School of Business ( email )

655 Knight Way Stanford, CA 94305-5015 United States

Stanford Immigration Policy Lab

30 Alta Road Stanford, CA 94305 United States

University of Pennsylvania ( email )

Stiteler Hall Philadelphia, PA 19104 United States

HOME PAGE: http://www.danhopkins.org

Massachusetts Institute of Technology (MIT) - Department of Political Science ( email )

77 Massachusetts Avenue Cambridge, MA 02139 United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics, related ejournals, econometrics: econometric & statistical methods - special topics ejournal.

Subscribe to this fee journal for more curated articles on this topic

Public Choice: Analysis of Collective Decision-Making eJournal

Political behavior: cognition, psychology, & behavior ejournal, political behavior: voting & public opinion ejournal, political methods: quantitative methods ejournal, political methods: experiments & experimental design ejournal.

• DOI: 10.1017/pan.2022.16
• Corpus ID: 250187327

Using Conjoint Experiments to Analyze Election Outcomes: The Essential Role of the Average Marginal Component Effect

• Kirk Bansak , Jens Hainmueller , +1 author Teppei Yamamoto
• Published in Political Analysis 30 June 2022
• Political Science

36 Citations

What do we learn about voter preferences from conjoint experiments, local preferences in candidate selection: evidence from a conjoint experiment among party leaders in germany, how does an economic shock affect environmental attitudes, preferences and issue importance evidence from switzerland, lexicographic preferences in candidate choice. how party affiliation dominates gender and race, multiple hypothesis testing in conjoint analysis, causal inference with ranking data: application to blame attribution in police violence and ballot order effects in ranked-choice voting.

• Highly Influenced

The Politics of Teachers’ Union Endorsements∗

Perceptions of electability: candidate (and voter) ideology, race, and gender, ideology, information, and social welfare preferences, criteria weights in hiring decisions—a conjoint approach, 26 references, measuring subgroup preferences in conjoint experiments, identification of preferences in forced-choice conjoint experiments: reassessing the quantity of interest, improving the external validity of conjoint analysis: the essential role of profile distribution, causal inference in conjoint analysis: understanding multidimensional choices via stated preference experiments.

• Highly Influential

More Important, but for What Exactly? The Insignificant Role of Subjective Issue Importance in Vote Decisions

Validating vignette and conjoint survey experiments against real-world behavior, causal interaction in factorial experiments: application to conjoint analysis, beyond the breaking point survey satisficing in conjoint experiments, what have we learned about gender from candidate choice experiments a meta-analysis of sixty-seven factorial survey experiments, related papers.

Showing 1 through 3 of 0 Related Papers

• DSpace@MIT Home
• MIT Open Access Articles

What Do We Learn about Voter Preferences from Conjoint Experiments?

Publisher with creative commons license.

Creative Commons Attribution

Terms of use

Date issued, collections.

Show Statistical Information

How to Use New Conjoint Analysis Tools

Paper in Political Methodologist on using conjoint analysis in contexts offline and with low literacy respondents.

The paper below published in The Political Methodologist outlines how to use new tools we developed, based on our experience in Tanzania, for implementing conjoint experiments in developing countries.

The first tool produces conjoint profiles in the Qualtrics offline app. The second app allows researchers to produce PDFs of conjoint profiles using images to represent attribute-levels. Read more on how to use these tools below.

Conjoint Analysis Tools for Developing Country Contexts by Alexander Meyer and Leah R. Rosenzweig (Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139. [email protected], corresponding author.)

1 Introduction

Conjoint analysis has long been used in marketing research, but has recently become popular in political science. Originally developed by Luce and Tukey in 1964, conjoint analysis serves as a useful tool for understanding preferences over multidimensional alternatives. This method presents respondents with profiles — for example of candidates (Carlson, 2015; Rosenzweig and Tsai, N.d.) or immigrants (Hainmueller and Hopkins, 2014; Berinsky et al., 2015) — that have randomly assigned attributes and asks respondents to evaluate and choose between them. The random assignment of profile characteristics allows researchers to identify the causal influence of attributes on a person’s decision to vote for a candidate or allow an immigrant into the country.

Conjoint analysis is advantageous for researchers interested in observing respondents’ choice-making behaviors and attitudes. Using this method, researchers can identify interaction effects as well as analyze particular aspects of treatments. For example, not only can it be used to identify the effect of a candidate’s past performance on the probability that respondents will vote for her, but we can also analyze the influence of past performance with respect to the candidate’s ethnic identity (Carlson, 2015). In addition, conjoint analysis allows us to investigate subgroup effects based on shared attributes between profiles and respondents, which can influence respondent attitudes (Berinsky et al., 2015). Thus, we are able to implement more realistic ‘bundled’ treatments, testing multiple hypotheses simultaneously (Hainmueller, Hopkins and Yamamoto, 2014).

As with all survey experiments, external validity is always a concern. Hainmueller, Hangartner and Yamamoto (2015) test the external validity of conjoint analysis by comparing results to a real-world behavioral benchmark in Switzerland. The authors find strong evidence that conjoint experiments can help to explain the preferences and behaviors of people in the real-world. From a paired conjoint design “estimates are within 2% percentage points of the effects in the behavioral benchmark” (Hainmueller, Hangartner and Yamamoto, 2015, p. 2395). Not only is conjoint analysis useful for investigating multiple hypotheses at once, but it can also achieve reliable results.

Until very recently, conjoint analysis had been relegated to online surveys. However, this method presents an excellent opportunity for researchers to understand preferences and behaviors across a host of different contexts. Researchers have begun to take advantage of this method in developing countries (Carlson, 2015; Hartman and Morse, 2015) but lack widely available resources for easy implementation and standardized best practices. Here we present the tools we developed to help researchers conduct conjoint experiments offline among respondents with little or no education.

Citation: Meyer, A., & Rosenzweig, L. (2016). Conjoint Tools for Developing Country Contexts. The Political Methodologist.

Related Work

Citizen engagement and voter behavior in tanzania.

Under what conditions do voters evaluate election candidates based on performance and programmatic considerations in dominant-party systems?

Voting for Change: Civic Engagement and Elections in Uganda

What do Ugandans most care about when selecting candidates for local office? How do citizen judge and act upon the legitimacy of elections and the quality of public goods provision?

New Tools for Conjoint Analysis in Developing Countries

Two new apps to help researchers design their own conjoint analysis.

Grad Life: Bringing the Lab to the Field

Leah Rosenzweig and her work for GOV/LAB is featured in MIT Alumni Association's "Slice of MIT."

Political Behavior of Development Conference at MIT

We are hosting a conference Friday, October 28, 2016 examining issues of racial and intergroup relations, determinants of political participation, partisanship and mobilization as well as the influence of informal institutions and elites on attitudes and behaviors.

“1, 2, 3...Vote”: Designing Voting Games in Uganda

MIT GOV/LAB and Twaweza interviewed 1,200 Ugandans after national elections to understand how citizens conceptualize politics and make voting decisions.

Understanding Citizen Preferences for Political Candidates — Learning Note 8

Why is the method big news? GOV/LAB team wrote new code and a manual for this type of experiment and made the code, manuals, and apps open source.

Politicians all Make the Same Promises — Learning Note 6

“Politics” is a word that has a very negative connotation for most citizens. Below we share research findings gathered from citizens around Tanzania.

Citizen Perspectives on Politics — Learning Note 5

How do ordinary Tanzanian citizens see politics and government? What do Tanzanian citizens think of as “engagement” and “participation” in politics? How do citizens interact with parties and political elites?

Funerals: A Communal Affair in Rural Tanzania

MIT GOV/LAB intern Neha Rajbhandary (Wellesley ‘20) explores the significance of funerals in Tanzania through an analysis of qualitative interviews.

More results...

• Help & FAQ

What Do We Learn about Voter Preferences from Conjoint Experiments?

• Political Science

Research output : Contribution to journal › Article › peer-review

Political scientists frequently interpret the results of conjoint experiments as reflective of majority preferences. In this article, we show that the target estimand of conjoint experiments, the average marginal component effect (AMCE), is not well defined in these terms. Even with individually rational experimental subjects, the AMCE can indicate the opposite of the true preference of the majority. To show this, we characterize the preference aggregation rule implied by the AMCE and demonstrate its several undesirable properties. With this result, we provide a method for placing bounds on the proportion of experimental subjects who prefer a given candidate feature. We describe conditions under which the AMCE corresponds in sign with the majority preference. Finally, we offer a structural interpretation of the AMCE and highlight that the problem we describe persists even when a model of voting is imposed.

ASJC Scopus subject areas

• Sociology and Political Science
• Political Science and International Relations

Access to Document

• 10.1111/ajps.12714

Other files and links

• Link to publication in Scopus
• Link to the citations in Scopus

Fingerprint

• Conjoint Experiment Psychology 100%
• Political Scientist Computer Science 100%
• Aggregation Rules Keyphrases 20%

T1 - What Do We Learn about Voter Preferences from Conjoint Experiments?

AU - Abramson, Scott F.

AU - Kocak, Korhan

AU - Magazinnik, Asya

N1 - Publisher Copyright: © 2022 The Authors. American Journal of Political Science published by Wiley Periodicals LLC on behalf of Midwest Political Science Association.

PY - 2022/10

Y1 - 2022/10

N2 - Political scientists frequently interpret the results of conjoint experiments as reflective of majority preferences. In this article, we show that the target estimand of conjoint experiments, the average marginal component effect (AMCE), is not well defined in these terms. Even with individually rational experimental subjects, the AMCE can indicate the opposite of the true preference of the majority. To show this, we characterize the preference aggregation rule implied by the AMCE and demonstrate its several undesirable properties. With this result, we provide a method for placing bounds on the proportion of experimental subjects who prefer a given candidate feature. We describe conditions under which the AMCE corresponds in sign with the majority preference. Finally, we offer a structural interpretation of the AMCE and highlight that the problem we describe persists even when a model of voting is imposed.

AB - Political scientists frequently interpret the results of conjoint experiments as reflective of majority preferences. In this article, we show that the target estimand of conjoint experiments, the average marginal component effect (AMCE), is not well defined in these terms. Even with individually rational experimental subjects, the AMCE can indicate the opposite of the true preference of the majority. To show this, we characterize the preference aggregation rule implied by the AMCE and demonstrate its several undesirable properties. With this result, we provide a method for placing bounds on the proportion of experimental subjects who prefer a given candidate feature. We describe conditions under which the AMCE corresponds in sign with the majority preference. Finally, we offer a structural interpretation of the AMCE and highlight that the problem we describe persists even when a model of voting is imposed.

UR - http://www.scopus.com/inward/record.url?scp=85136462892&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85136462892&partnerID=8YFLogxK

U2 - 10.1111/ajps.12714

DO - 10.1111/ajps.12714

M3 - Article

AN - SCOPUS:85136462892

SN - 0092-5853

JO - American Journal of Political Science

JF - American Journal of Political Science

Our systems are now restored following recent technical disruption, and we’re working hard to catch up on publishing. We apologise for the inconvenience caused. Find out more: https://www.cambridge.org/universitypress/about-us/news-and-blogs/cambridge-university-press-publishing-update-following-technical-disruption

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > Journals
• > Political Science Research and Methods
• > Volume 9 Issue 1
• > Beyond the breaking point? Survey satisficing in conjoint...

Article contents

Beyond the breaking point survey satisficing in conjoint experiments.

Published online by Cambridge University Press:  08 May 2019

• Supplementary materials

Recent years have seen a renaissance of conjoint survey designs within social science. To date, however, researchers have lacked guidance on how many attributes they can include within conjoint profiles before survey satisficing leads to unacceptable declines in response quality. This paper addresses that question using pre-registered, two-stage experiments examining choices among hypothetical candidates for US Senate or hotel rooms. In each experiment, we use the first stage to identify attributes which are perceived to be uncorrelated with the attribute of interest, so that their effects are not masked by those of the core attributes. In the second stage, we randomly assign respondents to conjoint designs with varying numbers of those filler attributes. We report the results of these experiments implemented via Amazon's Mechanical Turk and Survey Sampling International. They demonstrate that our core quantities of interest are generally stable, with relatively modest increases in survey satisficing when respondents face large numbers of attributes.

Access options

Bansak et al. dataset, bansak et al. supplementary material.

Bansak et al. supplementary material 1

This article has been cited by the following publications. This list is generated based on data provided by Crossref .

• Google Scholar

View all Google Scholar citations for this article.

Save article to Kindle

To save this article to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

• Volume 9, Issue 1
• Kirk Bansak (a1) , Jens Hainmueller (a2) , Daniel J. Hopkins (a3) and Teppei Yamamoto (a4)
• DOI: https://doi.org/10.1017/psrm.2019.13

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox .

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive .

Reply to: Submit a response

- No HTML tags allowed - Web page URLs will display as text only - Lines and paragraphs break automatically - Attachments, images or tables are not permitted

Your details

Your email address will be used in order to notify you when your comment has been reviewed by the moderator and in case the author(s) of the article or the moderator need to contact you directly.

You have entered the maximum number of contributors

Conflicting interests.

Please list any fees and grants from, employment by, consultancy for, shared ownership in or any close relationship with, at any time over the preceding 36 months, any organisation whose interests may be affected by the publication of the response. Please also list any non-financial associations or interests (personal, professional, political, institutional, religious or other) that a reasonable reader would want to know about in relation to the submitted work. This pertains to all the authors of the piece, their spouses or partners.