Campus antisemitism report is based on fundamentally flawed methodology

The recently released Campus Antisemitism and Student Experiences (CASE) report and a follow-up published in University Affairs weaponize antisemitism while reaching deeply flawed, misleading conclusions.

As a group of social science researchers with training and experience in quantitative and qualitative approaches, it is simply not possible to accept the findings of this ‘study’ at face value because its methodology is fundamentally flawed. Independent Jewish Voices Canada, Jewish Faculty Network, and Joshua Sealy-Harrington have already pointed out a range of glaring issues. For one, the survey upon which the data is based relied on a self-selected, unweighted convenience sample recruited through established Zionist organizations. Progressive, anti-Zionist, and non-Zionist Jewish students were predominantly shut out of the data collection. As anyone who teaches methods for social science knows, you simply cannot make sweeping national claims about a diverse demographic when you intentionally survey a narrow faction. In essence, the report uses a qualitative survey methodology to make quantitative claims—a frankly irresponsible error in judgment for a professional social scientist.

To make definitively broad generalizations grounded in a small, biased selection sample is an unacceptable error, yet this is something the authors of this report repeatedly do, such as when they assert, “95.7% said they had experienced or witnessed at least one instance of antisemitism over the previous 12 months” without providing a caveat such as “of those surveyed” or a strong note on the limitations of the report’s methodology. This finding of 95 per cent has, predictably, been repeated ad nauseam in the media. In fact, the authors of the report have admitted that among the very small number of respondents who indicated Zionism was not important at all to them, 92 per cent said their campus was safe and inclusive for Jewish students, a finding that directly contradicts the thesis that it is Jewishness, rather than the state policy of Israel, that is the object of critique on campuses.

The 95 and 92 per cent figures, taken together, index a larger problem: the incentive for participants to articulate experiences that align with broader political projects in a moment of profound polarization. It is clear beyond doubt that students with Zionist proclivities reported discrimination and those with anti-Zionist leanings reported none. So, what the study seems to reveal above all is the political orientation of its participants. It is not really possible to falsify any of the findings beyond that. If Zionist students consider critique of Israel to be a form of harm, that is what they will report. But this is not an objective measure, as evinced by the testimony of anti-Zionist students.

There is also the issue of the survey instrument, which exacerbates this problem by baking the conclusion right into the premise. The definitions and terminology in the survey instrument repeatedly fail to distinguish Jews, Judaism, Israel, and Zionism when asking about experiences of antisemitism, thereby collapsing diverse experiences into a monolithic category. This design choice makes any rigorous analysis of antisemitism impossible. The researcher is wholly unable to separate actual anti-Jewish hatred from political criticism of a foreign state. In social science, we refer to this basic problem as ‘construct validity,’ and it means that a researcher is not properly measuring what it is they are trying to measure. In this case, antisemitism and anti-Zionism are being measured by all examples labelled as antisemitism.

In fact, many of the problems with the CASE report are issues qualitative researchers identify with surveys more broadly: they have a tendency to produce shallow knowledge. This is because it is not possible for the researcher to probe respondents in order to contextualize and interrogate the meaning of their responses or even to adequately vet the appropriateness of participants to the project at all. What results, as in the instance of the CASE report, is a chunk of data that may on the surface seem suggestive but ultimately loses salience upon closer scrutiny. This is precisely why qualitative researchers work with smaller samples—because while it is harder to (falsely) claim generalizability, it is much easier to be confident in the veracity of the findings.

An additional problem with the report is that it conflates criticism of Israel with antisemitism following the widely contested logic of the International Holocaust Remembrance Alliance (IHRA) working definition of antisemitism. That framework is notorious for chilling legitimate academic and political expression or opposition to Israeli state policy. By starting from the assumption that Jewish identity and Zionism are inseparable, the study actively erases a considerable proportion of Canada’s Jewish population—those who do not associate their religious identity with the state of Israel—before the research even begins. Put differently, this definition necessitates the exclusion of an entire constituency who must be included for the report to achieve its stated goal: to study the extent to which Jewish students experience antisemitism on Canadian university campuses.

Students deserve protection from bigotry. Antisemitism, like other forms of discrimination, does flourish in Canadian society and should be confronted. There are real examples of antisemitism presented in this study. However, the study’s claim to be studying antisemitism is fundamentally and irreparably flawed, as its quantitative findings merge critiques of Zionism with antisemitism in such a way that it is impossible to isolate these two different constructs.

At the same time, we must also note that it is puzzling to see federal government resources devoted to such a questionably executed project on antisemitism at a moment when another form of discrimination is clearly on the rise: anti-Palestinian racism. If the government is concerned specifically about discrimination on campus—rather than chilling critique of Israel—it is difficult to defend the decision to study one form over others. And yet here we are.

As university teachers, there is little we value more for our students than safety in classrooms and on campuses. But fundamentally flawed studies do not protect students or accomplish any of the objectives that the authors claim. We cannot allow the use of unrepresentative survey design to justify the censorship or surveillance of the right of students to freely express themselves. Student academic freedom is itself a core condition of university accreditation. It is not possible to satisfy this condition if students are prohibited from offering critique of state policy in Canada or elsewhere. Any research designed to shape university policy must be held to the most rigorous scholarly standards, which certainly includes an ability to account for the distinction between genuinely hateful acts and political disagreement.

Unfortunately, in that regard and others, this report merits a failing grade.

Nathan Kalman-Lamb is associate professor of sociology at the University of New Brunswick. Derek Silva is professor of sociology and criminology at King’s University College at Western University. Daniel Sailofsky is an assistant professor of sport management at York University. Priyansh is a doctoral candidate in the Faculty of Kinesiology & Physical Education at the University of Toronto.