How (not) to do Science - Philosophy of Research in Cognitive Science (WS 21/22)
In the early 2010s, the scientific community became aware of a fundamental flaw in their work: Many cognitive and psychological phenomena could not be replicated. Today, this is known as the (still ongoing) replication crisis. During the course, we will explore pitfalls which led to the replication crisis and how to avoid them in future research. For this, we will use and evaluate selected papers on the issues of philosophy of science, experimental designs, (false) use of statistics, open science and others. While the replication crisis is usually attributed to the social and cognitive sciences, many of its pitfalls and solutions are important considerations for other fields as well. Among these are concepts like data management & sharing, researcher degrees of freedom and publication bias.
| Session Topic | Reading Material | Link to Paper |
|---|---|---|
| Karl Popper and demarcation | Dienes, Z. (2008). Karl Popper and demarcation (ch 1, pp. 1-32). Understanding psychology as a science: An introduction to scientific and statistical inference. New York: Palgrave Macmillan. | NA |
| Kuhn & Lakatos: Paradigms and Programs | Dienes, Z. (2008). Kuhn and Lakatos: Paradigms and programs (ch 2, pp. 33-54). Understanding psychology as a science: An introduction to scientific and statistical inference. New York: Palgrave Macmillan. | NA |
| Problem of p-Values | The Null Ritual What You Always Wanted to Know About Significance Testing but Were Afraid to Ask | http://library.mpib-berlin.mpg.de/ft/gg/GGNull2004.pdf |
| Researcher Degrees of Freedom | False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant - Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn, 2011 | https://journals.sagepub.com/doi/10.1177/0956797611417632 |
| Reproduction Crisis & Open Science | Estimating the reproducibility of psychological science ; Easing into open science | https://doi.org/10.1126/science.aac4716 ; https://online.ucpress.edu/collabra/article/7/1/18684/115927/Easing-Into-Open-Science-A-Guide-for-Graduate |
| Reproducibility in ML | A Step Toward Quantifying Independently Reproducible Machine Learning Research | https://arxiv.org/abs/1909.06674 |
| Replicability Crisis | Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences - Stefan Schmidt, 2009 | http://journals.sagepub.com/doi/10.1037/a0015108 |
| Effects of Low Power | Power failure: why small sample size undermines the reliability of neuroscience | https://www.nature.com/articles/nrn3475 |
| Solution for replicability: preregistrations | The Value of Preregistration for Psychological Science ; Is Preregistration Worthwile? | https://psyarxiv.com/jbh4w/ ; https://psyarxiv.com/x36pz/ |
| Theory Crisis | The empirical benefits of conceptual rigor ; Psychology’s Theory Crisis, and Why Formal Modelling Cannot Solve It | https://www.sciencedirect.com/science/article/pii/S0022103115001092 ; https://psyarxiv.com/puqvs/ |
| Generalizability Crisis | The Generalizability Crisis | https://psyarxiv.com/jqw35/ |
| Ethics | Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward | https://www.nature.com/articles/s41599-020-0501-9 |
Note: This course was taught at a University heavily influenced by computer science/ML/AI, thus two sessions (Reproducibility in ML & Ethics) are specifically tailored towards computer science students.
This course was taught together with Judith Schepers and Benedikt Ehinger.