Wording, Polling, and Opinion:
Flaws in the CAS Survey of Faculty Attitudes on Affirmative Action

Professor Michael Hout
Director, Survey Research Center

Survey Research Center
University of California, Berkeley
2538 Channing Way
Berkeley, CA 94720-5100
PHONE: (510) 643-6874 FAX (510) 643-8292


Summary Faculty support for affirmative action has been called into question by the results of a Roper poll sponsored by the California Association of Scholars. The headline of a CAS press release claims that "faculty wants the UC to promote equal opportunity without the use of racial and gender preferences." The poll deviates from scientific practice in vital ways that make it inaccurate. Most importantly, the questions express the issues in a one-sided way favorable to affirmative action opponents (who prefer the word "preferences"). CAS compounds the scientific problems of their poll by focusing on the one question that favors their point of view.

Background The Roper Center - a national nonprofit research organization with offices on the campus of the University of Connecticut - conducted a telephone poll of UC faculty during the period between December 7 and 19, 1995. The poll was sponsored by the California Association of Scholars (CAS). Roughly 1,200 UC were contacted; 1,001 agreed to be interviewed.1 The survey consisted of a brief introduction and four questions on topics related to affirmative action.2 Data on gender and faculty were obtained from campus directories, and the campus was known in advance.

Framing the Issues in the Questions The CAS-sponsored poll ignores standard scientific practice of framing questions in a way that gives both sides of an issue a chance to voice their opinion. By using the word "preferences" commonly used by the opponents of affirmative action in the response alternative that is favorable to affirmative action they pose the question in a way that does not give affirmative action advocates a chance to express their views clearly in their own words.

Good practice in survey research includes framing the questions in a way that people can recognize their own point of view in the alternatives that they are given by the interviewer. Polling is, after all, the art of putting words into peoples' mouths. Objective practice demands that the words chosen for the questionnaire come close to the words that advocates of each point of view would use if they were given the chance to frame their opinion without prompting. In scientific or academic surveys, the phrasing of questions is usually drawn from published remarks by leaders of one point of view or another or from "focus groups' in which ordinary citizens are asked to discuss important issues in their own terms. At SRC we supplement these practices by "pretesting" questions in practice interviews, If the respondents to practice interviews have a hard time recognizing their point of view in the questions, then we rewrite them. We try not to have more than one respondent in 20 say they "don't know" how they feel on an issue unless we feel that the issue itself is so obscure that many people really have no opinion. With a widely discussed issue, a scientific poll should not have more than five percent of answers be "don't know" (Converse and Presser 1989).

Advocacy polls, on the other hand, frequently "slant" questions by raising questions in a way that are favorable to one point of view in a debate--and unfavorable to another--or by posing questions that create a dilemma for proponents of one side (Asher 1990). This practice makes it difficult for persons from one persuasion to answer the questions as stated. They are typically in a quandary because they cannot fully agree with any of the statements offered them, or they cannot choose between the alternatives that are posed because they agree with both or disagree with both. Some poll respondents then refuse to answer the question or say they don't know which alternative to choose. Others say "both?' or "neither" as their reply.

The CAS poll frames the affirmative action issue in a way that is favorable to affirmative action opponents and does not reflect the arguments of affirmative action proponents.3 In counterposing "preferences" and "equal opportunity" the poll gave voice to the opinions of affirmative action critics, but did not give proponents of affirmative action a chance to state their "true" opinion. These problems are compounded by the long list of potential criteria on which "preference" might be based (race, religion, sex, color, ethnicity, or national origin). Only race, color, and ethnicity are at issue with respect to admissions; sex or gender become an issue in hiring and contracting. The person who favors affirmative action for African Americans, let us say, but not other groups, would have difficulty endorsing "preferences" based on all of the criteria listed, even if she accepted the characterization of affirmative action as "preferences."

These kinds of difficulties are reflected in complaints that Roper interviewers heard during the interviews, in complaints that SRC received from irate respondents who called under the mistaken impression that SRC had something to do with the poll, and in public comments made by faculty who were interviewed. This is all anecdotal evidence. There is concrete evidence within the poll results themselves.

A significant fraction of faculty refused to register an opinion on all four of the questions. The "no opinion / don't know" rates for the two questions in "favor / oppose" format exceed the usual 5 percent by wide margins - 14 percent in each case. Answers of "both," "neither," or "don't know" are even more common in the two questions that pose two alternatives - 21 percent and 19 percent for questions #3 and #4, respectively.

The prevalence of "out of bounds" responses indicates that the poll failed to register a significant element of faculty opinion. We cannot say with certainty what that element is. Like many of the respondents, we don't know. In that way a poll with many out of bounds responses is a missed opportunity. Given the opportunity to measure opinion, it failed.

Forcing the Issue in Questions
The CAS press release warns against "the dangers of using phraseology in public opinion research that does not clearly force a decision about preferences." Public opinion research cannot "force" anything without running the risk of creating the phenomenon instead of measuring it. Remember that the respondents to a poll are representative of a larger group when the interviewer first encounters them. But if the interviewer forces choices on them that the people they represent do not face, then they cease to be representative. In measuring their opinion we would have changed it. That kind of manipulative practice is rejected by the public opinion research community (Converse and Presser 1989).

While the author of the CAS press release is correct that the public holds many inconsistent beliefs, structuring a questionnaire in a way that eliminates those inconsistencies for respondents does not eliminate them from the public consciousness or even from public discourse. That is the realm of persuasion and advocacy, not polling. Polling can be a powerful tool for advocates to track their progress in eliminating inconsistencies from the public's collective mind. But eliminating inconsistencies from respondents' minds via the questionnaire only affects the respondents, not the larger group they represent.

Misreading the Results at Hand
After leading the witness with slanted questions, the CAS-sponsored poll still failed to repudiate the practice of affirmative action. Of those who accepted the terms of the questions, 60 percent endorsed using race, religion, sex, color ethnicity, or national origin as a criterion for university admission, and 55 percent endorsed using them as criteria in hiring and contracting. Yet the press release claims that the faculty wants the university to drop race and gender as considerations. CAS's interpretation is just wrong on the face of it. Their claim ignores these two questions and focuses on the third question. A fair reading must attend to both questions.

The only interpretation consistent with all of the data is that the faculty is very sensitive to how the premise is phrased. That is why the division reflected in this poll contrasts so sharply with the high degree of consensus in the votes of the faculty's representative bodies. Those representative bodies deliberated the question for hours. They debated the wording until they found phrasing that most could agree to. The division reflected in the poll would have most likely existed prior to the meeting. But once the wording was agreed to, the resolutions passed overwhelmingly.

Conclusion
Faculty opinion on affirmative action cannot be reduced to simple "agree / disagree" propositions or even to the choice between "preferences" and "equality of opportunity." Many faculty members' opinions on the issue do not fit into the narrow confines of the CAS sponsored poll. In particular, the characterization of affirmative action as granting "preferences" is an issue that is part of the dispute. Using that term in a key question leaves affirmative action proponents without a response that reflects their views. As a consequence of its non-objective wording, the poll generated uncommonly high rates of "don't know," "both," and "neither" responses to the questions.

Even taken at face value the poll results do not support the proposition that faculty want the University of California to abandon the affirmative action practices that faculty and administrative committees have implemented over the past 20 years. Among those who accept the terms of the first two questions, 60 percent of faculty favor using race, religion, sex, color, ethnicity, or national origin as a criterion for admission to the University of California and 55 percent of faculty favor using these criteria in hiring and contracting. While survey and polls are the stock-in-trade of the Survey Research Center, I cannot endorse CAS's idea that polls substitute for dialogue and for voting in representative assemblies.

1 The exact number of contacts is not reported in the CAS press release. I estimate it to be 1,200 from the CAS claim that the response rate was 80 percent - very high for a political poll but typical of surveys drawn from lists of members of an organization or workplace.

2 Exact question wordings are appended.

3 The first two questions use the wording of Regent Connerly's resolutions (SP-1 and SP-2).


QUESTIONS IN THE CAS POLL

1. Do you favor or oppose using race, religion, sex, color, ethnicity, or national origin as a criterion for admission to the University of California?

Favor 52%
Oppose 34%
No opinion/don't know 14%


2. Do you favor or oppose using race, religion, sex, color, ethnicity, or national origin as a criterion in employment and contracting practices at the University of California?

Favor 47%
Oppose 39%
No opinion/don't know 14%

3. I would like to read two statements. Please tell me which one best describes the policy you believe the University of California should pursue:

First, the university should grant preferences to women and certain racial and ethnic groups in admissions, hiring, and promotions.

Second, the university should promote equal opportunities in these areas without regard to an individual's race, sex, or ethnicity.

First 31%
Second 48%
Both 3%
Neither 2%
No opinion/don't know 6%


4. The term "affirmative action" has different meanings to different people. I am going to read two definitions of the term "affirmative action." Please tell me which one best describes what you mean by the term.

First, affirmative action means granting preferences to women and certain racial and ethnic groups.

Second, affirmative action means promoting equal opportunities for all individuals without regard to their race, sex, or ethnicity. Which statement, the first or the second, comes closest to your own definition of affirmative action?

First 37%
Second 43%
Both 2%
Neither 14%
No opinion/don't know 4%

Carl Gutierrez-Jones,

Department of English

University of California

Santa Barbara, CA 93106

E-mail: carlgj@humanitas.ucsb.edu