What counts as empirical evidence when discussing whether a stereotype or other representation in a film reflects the actual world.
(This site addresses evidence for a particular course, but the basic rules apply elsewhere as well.)
(This site addresses evidence for a particular course, but the basic rules apply elsewhere as well.)
Evidence, Observables and Facts
- The evidence we are concerned with is whether some part of the world has a suggested characteristic.*
- Characteristics can be observed, that is, there is some way to detect that the characteristic exists.
- A true report of an observation is a fact.
- For this course, evidence is a reliable report of a fact.
- All evidence is highly dependent on framing: the way in which questions are asked and the assumptions made when observing.
*Facts can also refer to the occurrence of events and this discussion of evidence also relates to such facts.
Anecdotal evidence
Anecdotal evidence is evidence of a single observation (report of an event) or evidence of observations that occur incidentally or haphazardly. This evidence cannot be used to generalize beyond the specific observed instances.
Anecdotal reports or experiences of evidence include:
When you report an example, you are citing anecdotal evidence.
Anecdotal evidence is reliable when you have a good reason to believe the source:
Anecdotal reports or experiences of evidence include:
- What you observe yourself.
- What someone tells you (individually or as part of an audience) about an observation.
- A newspaper article or other media source report about any event or occurrence.
- A report of a single case study.
- A documentary.
When you report an example, you are citing anecdotal evidence.
Anecdotal evidence is reliable when you have a good reason to believe the source:
- is competent,
- is honest, and
- has had access to the observation or to reliable reports of the observations.
- The report is does not generalize beyond the observation.
Examples
You stand in line at the bursar’s office for a long time and ultimately have a rushed and impersonal interaction.
A reliable claim is: You (and others) had a long wait and you found your interaction rushed and impersonal.
An unreliable claim is: The staff of the bursar’s office and/or the administration of the college are rude and do not care about students.
You find a New York Times article reporting an incidence of police corruption.
You read a first-hand account of Lincoln delivering the Gettysburg Address.
A reliable claim is: You (and others) had a long wait and you found your interaction rushed and impersonal.
- You experienced this yourself and the report is limited to what you experienced.
An unreliable claim is: The staff of the bursar’s office and/or the administration of the college are rude and do not care about students.
- You have no evidence of the underlying causality (“do not care about students”) and the characterization of rude can, at most, be applied to the time period during which you were present.
You find a New York Times article reporting an incidence of police corruption.
- A correct interpretation of this article is that there has been an incidence of corruption.
- An incorrect interpretation of this article is that the police are corrupt.
You read a first-hand account of Lincoln delivering the Gettysburg Address.
- First hand accounts are anecdotes even when they are historically significant. If something is an anecdote, that does not mean it is incorrect or unimportant.
Some likely sources of anecdotal evidence
- Your own eyes.
- Newspapers, popular magazines, and other non-academic publications.
- Social media.
- Reference matter such as encyclopedias.
- Academic publications, when they report specific cases.
Some types of scientific evidence
Scientific evidence is evidence that has been collected following a systematic plan that permits the use of inductive logic to generalize to the population from which the observations were made.
Generalization is inferring that a characteristic observed in some instances is probably true for other unobserved instances. Generalization involves risk of error. Through the use of a systematic plan of observation this risk can be minimized and (partly) measured.*
Scientific evidence (with differing levels of possible risk)
*Measurement of risk results from the use of probability theory. The measurement of risk may be reported as a discussion of a level of confidence or a “p value.” When the assumptions of probability theory are violated, there will be some unmeasured risk. Measurement is possible because of appropriate sampling.
Generalization is inferring that a characteristic observed in some instances is probably true for other unobserved instances. Generalization involves risk of error. Through the use of a systematic plan of observation this risk can be minimized and (partly) measured.*
Scientific evidence (with differing levels of possible risk)
- A fully randomized experiment with control observations (fully generalizable to the relevant population with measured risk).
- Quasi-experiments and natural experiments (generalizable with some unmeasured risk).
- Surveys and other non-experimental observation studies (depending on the quality of sampling, the risk of error can be similar to experiments or quasi-experiments).
- Meta-analysis, which is typically a way to aggregate experiments (particularly when sample size is small or results differ), to correct for some unmeasured error in quasi-experiments, or aggregate both experiments and quasi-experiments.
- Multiple case studies (similar to meta-analysis, but possibly with more unmeasured error).
*Measurement of risk results from the use of probability theory. The measurement of risk may be reported as a discussion of a level of confidence or a “p value.” When the assumptions of probability theory are violated, there will be some unmeasured risk. Measurement is possible because of appropriate sampling.
Example
You watch a film in which police are portrayed as corrupt. (This is not evidence.)
After substantial search using the JSTOR database through the library website, you find a peer reviewed journal article that discusses police corruption, but does not report the details of the actual research (which is “primary” evidence or evidence from a “primary source”):
In the article you find a reference to four primary sources (that report actual research studies):
You go to the library and access these sources* (the first two of these sources are more similar to case studies than more substantial scientific evidence) and report the relevant factual data on corruption. Be sure to address the population to which they apply (the first two are focused on New York City alone).
You may also want to find additional primary sources.
*If a source is not at the library, check other system libraries and if it is not there, request it through Interlibrary loan.
After substantial search using the JSTOR database through the library website, you find a peer reviewed journal article that discusses police corruption, but does not report the details of the actual research (which is “primary” evidence or evidence from a “primary source”):
- Skogan, W., & Meares, T. (2004). Lawful Policing. The Annals of the American Academy of Political and Social Science, 593, 66-83. Retrieved from http://www.jstor.org/stable/4127667
In the article you find a reference to four primary sources (that report actual research studies):
- City of New York, Commission to Investigate Allegations of Police Corruption. 1973. The Knapp Commission report on police corruption. New York
- City of New York, Commission to Investigate Allegations of Police Corruption and the Anti-Corruption Procedures of the Police Department. 1994. Mollen Commission report. New York: Mollen Commission
- Sherman, Lawrence W. 1978. Scandal and reform: Controlling police corruption. Berkeley: University of California Press; and
- Klockars, Carl B., Sanja Kutnjak Ivkovich, William E. Harver, and Maria R. Haberfeld. 2000. The measurement of police integrity. Washington, DC: National Institute of Justice.
You go to the library and access these sources* (the first two of these sources are more similar to case studies than more substantial scientific evidence) and report the relevant factual data on corruption. Be sure to address the population to which they apply (the first two are focused on New York City alone).
You may also want to find additional primary sources.
*If a source is not at the library, check other system libraries and if it is not there, request it through Interlibrary loan.
Likely sources of scientific evidence
Peer reviewed journal articles or peer reviewed academic anthologies that report the results of a study such as an:
Program evaluations conducted by reputable organizations*
Reputable “Think Tanks”*
Evaluation sections of government websites
Governmental program auditors or legislative audit agencies
The GAO (General Accountability Office), IBO (New York City Independent Budget Office), or CBO (Congressional Budget Office).
Finding a source of these sorts does not guarantee that the item you have found is scientific evidence. All of these sources need to be examined for reliability (to the degree that you can) and most of these sources also contain other types of material that are not reports of empirical studies. They may also contain qualitative analyses which typically focus on meaning rather than characteristics.
* There is substantial risk that advocacy groups and highly ideological organizations will represent themselves as reputable organizations or think tanks, but the studies they report may be biased.
- Experiment,
- Survey,
- Meta-analysis, or
- Similar direct report of data collection and analysis.
Program evaluations conducted by reputable organizations*
Reputable “Think Tanks”*
Evaluation sections of government websites
Governmental program auditors or legislative audit agencies
The GAO (General Accountability Office), IBO (New York City Independent Budget Office), or CBO (Congressional Budget Office).
Finding a source of these sorts does not guarantee that the item you have found is scientific evidence. All of these sources need to be examined for reliability (to the degree that you can) and most of these sources also contain other types of material that are not reports of empirical studies. They may also contain qualitative analyses which typically focus on meaning rather than characteristics.
* There is substantial risk that advocacy groups and highly ideological organizations will represent themselves as reputable organizations or think tanks, but the studies they report may be biased.
Non-evidence and Non-facts
Non-evidence:
Non-facts:
Non-facts that may be meaningful:
Non-facts that are also not meaningful:
- The opinions expressed by anyone of any stature at any time.
- Fictional depiction of any sort including the films you are watching. Films do not demonstrate factual information, they depict or portray assumptions about facts.
- Things you know because you have heard about them all of your life.
Non-facts:
Non-facts that may be meaningful:
- Value assertions.
- Exploration of deep meanings.
- Rules, laws and similar material.
Non-facts that are also not meaningful:
- Baseless assertions.
- Deliberate falsehoods.
Reporting evidence
Whenever possible use the primary source, that is, the source that reports the study that found the evidence.
Quotes of conclusions from reports are not, themselves, evidence.
Report the relevant factual claims; where quantitative data is provided, report the relevant quantitative findings.
Be careful to appropriately interpret the findings. For example, when absolute numbers are reported, it is likely important to relate them to their sample or population base.
* Other guidance on this website advises you how to report quotes found in secondary sources.
- If you learn about a study from a secondary source (such as a newspaper or an article that refers to another article), use the information in the secondary source to find the primary source, then report what you find in the primary source.*
- Sometimes secondary sources confuse or misrepresent the findings of primary sources.
Quotes of conclusions from reports are not, themselves, evidence.
- Reporting the conclusion along with the substantiating evidence is preferred.
Report the relevant factual claims; where quantitative data is provided, report the relevant quantitative findings.
Be careful to appropriately interpret the findings. For example, when absolute numbers are reported, it is likely important to relate them to their sample or population base.
* Other guidance on this website advises you how to report quotes found in secondary sources.
The Relation Between Scientific Evidence and Causality
Williams and Calabrese summarize the criteria for causality as:
"Cause/Causal/Causal-Like – Hill (1965) asserts that two variables are causally related when a change in the variable labeled 'cause' is temporally prior or simultaneous with a change in the variable labeled 'effect,' where there is a plausible reason why the cause leads to the effect, the relationship is consistent, and there is a dose effect (the size of the change in cause is related to the size of change in effect). He includes four additional or alternative criteria (strength of relationship, specificity of relationship, subject to experimental modification, and reasoning by analogy) and one criterion (coherence) that is at the level of epistemology. Granger (1988a, 1988b) adds a complex test for causality when performing statistical modeling. With statistical models the change/change relationship can be established by correlation. When plausible causal variables are included in a statistical model, it is widely understood that plausible alternative causal variables – representing alternative hypotheses – should be excluded (Newton & Rudestam, 2012), which may be achieved through the relative strength of correlation diagnostics. Because correlation relates to only one of the conditions of causation, there is a widely known principle that 'correlation is not causation.' This principle can be too broadly applied in that sometimes correlation is disparaged as irrelevant to causation" (page 156).
Scientific evidence that is systematically collected and analyzed within the scope of a theory that provides a plausible reason for a causal relationship and adequately meets the other causal criteria above can lead to sound causal assertions. Nevertheless, there may also be scientific evidence that describes some element of the world without leading to a sound causal claim. Within statistical modeling two specific model features related to causal claims are comparison (some observations include the alleged causal element while others do not) and random assignment to treatment or comparison group. Some research problems do not lend themselves to random assignment and, consequently, causal claims. The best corrective for this problem is to control for as many elements as possible (include variables that relate to as many confounding factors as possible). Nevertheless any resulting causal claims must be treated as tentative. When there is no comparison group, causal claims should not be made.
"Cause/Causal/Causal-Like – Hill (1965) asserts that two variables are causally related when a change in the variable labeled 'cause' is temporally prior or simultaneous with a change in the variable labeled 'effect,' where there is a plausible reason why the cause leads to the effect, the relationship is consistent, and there is a dose effect (the size of the change in cause is related to the size of change in effect). He includes four additional or alternative criteria (strength of relationship, specificity of relationship, subject to experimental modification, and reasoning by analogy) and one criterion (coherence) that is at the level of epistemology. Granger (1988a, 1988b) adds a complex test for causality when performing statistical modeling. With statistical models the change/change relationship can be established by correlation. When plausible causal variables are included in a statistical model, it is widely understood that plausible alternative causal variables – representing alternative hypotheses – should be excluded (Newton & Rudestam, 2012), which may be achieved through the relative strength of correlation diagnostics. Because correlation relates to only one of the conditions of causation, there is a widely known principle that 'correlation is not causation.' This principle can be too broadly applied in that sometimes correlation is disparaged as irrelevant to causation" (page 156).
Scientific evidence that is systematically collected and analyzed within the scope of a theory that provides a plausible reason for a causal relationship and adequately meets the other causal criteria above can lead to sound causal assertions. Nevertheless, there may also be scientific evidence that describes some element of the world without leading to a sound causal claim. Within statistical modeling two specific model features related to causal claims are comparison (some observations include the alleged causal element while others do not) and random assignment to treatment or comparison group. Some research problems do not lend themselves to random assignment and, consequently, causal claims. The best corrective for this problem is to control for as many elements as possible (include variables that relate to as many confounding factors as possible). Nevertheless any resulting causal claims must be treated as tentative. When there is no comparison group, causal claims should not be made.