Seminar SCALab 23/03/2023

Weak correlations in visual illusions, ageing, and disease: Do we really measure what we believe we are measuring?

Representative paradigms and tests are crucial in all empirical sciences. These tests are supposed to measure the characteristics of an underlying latent variable or scientific construct. For example, eye doctor’s tests are thought to be representative for visual acuity in general. Even though only, for example, Snellen E’s are used in the test, it is presupposed that good visual acuity predicts good acuity also in other tests and everyday life tasks. Similarly, many paradigms, e.g., visual illusions, are used in basic vision research to unearth the characteristics of the visual system in general. A paradigm would be useless when its performance does not correlate with the performance in similar paradigms.

However, this is exactly what we found in many areas such as basic vision research, healthy aging, and schizophrenia research. For example, we found that older people performed clearly worse than younger participants in a battery with many commonly used visual tests, including spatial acuity, motion acuity etc. We assumed that a participant, wo excels in a specific test, excels in all tests or at least in similar ones. However, this was not the case. Performance in the various tests was largely uncorrelated, even though test re-test reliability was good. We made similar findings with visual illusions and other paradigms in basic vision research and with schizophrenia patients, including an analysis with 192 EEG measures. Hence, tests are often less representative than previously thought or to put it this way: there is much more inter-observer variability than previously assumed.

Obviously, we need to rethink how we measure data and how we analyze them. It seems that standard simple methods need to be replaced by complex analyses in high dimensional spaces. The aim of this talk is to lay out the problem, present the empirical facts- and ask for help how these problems can be attacked from a theoretical point by colleagues from math, ML and data science.