2. Error#
#
#
\( \Large \begin{array}{l} \text{Error} \ \left\{ \begin{array}{l} \text{Variance} \left\{ \begin{array}{l} \text{Within} \\ \text{Between} \end{array} \right. \\ \text{Bias} \end{array} \right. \end{array} \)
#
#
Transition from Chapter 1
Emerging from the basics, we learned that metrics like the odds ratio aren’t standalone entities. They reside within a vast landscape shaped by error, which includes bias (systematic error) & variance (random error). While both types of error play pivotal roles in the interpretation of results, their distinction can often blur, especially in regression analyses. In this chapter, we delve deeper into understanding these intertwined concepts, especially within the context of hierarchical data.
Introduction
In clinical research, every observation, every reading, carries with it some element of uncertainty. This uncertainty or error is shared between bias and variance. Yet, when diving deep into datasets or regressions, teasing apart the two becomes a Herculean task. Let’s embark on this journey to differentiate and understand them better.
Bias and Variance: Two Sides of the Same Coin
Bias: Refers to the difference between the observed value and the true value. It’s the deviation we wish wasn’t there, and it speaks to the accuracy of our measurements.
Variance: While bias speaks to accuracy, variance deals with precision. It tells us about the dispersion or spread of our data points around the mean (within a group or between groups).
In datasets or regressions, however, this distinction becomes nebulous. When we observe a spread in data points in a regression analysis, is it due to inaccuracies stemming from design, measurement, or analysis (bias)? Or is it due to the inherent differences in the population (variance)? Often, we can’t tell them apart directly, but we can make informed judgments based on study design and prior knowledge.
Bias
Here’s a grouping that provides a simplified yet clear understanding of the bias-variance tradeoff and the ideal outcomes one should aim for in research or modeling. It also gives a structured approach to distinguishing between bias and variance. Here’s a breakdown:
Bias (Accuracy)
This is linked to the idea of systematic error.
Bias is a consistent deviation from the true (or expected) value.
The key feature of bias is that it’s predictable and doesn’t change regardless of how many times you repeat the experiment or data collection.
With the right methods or understanding of the system, we can control or adjust for bias.
Variance (Precision)
Variance is intrinsically tied to the concept of random error.
Unlike bias, variance speaks to unpredictable changes in measurements or predictions. Even if you repeat the experiment or data collection under the same conditions, the random error may differ.
Due to its unpredictable nature, variance can’t be completely eradicated, but it can be minimized or managed with appropriate techniques.
Truth (Explanation)
The goal in many analyses, modeling, or experiments is to approach the “truth” — to make predictions or measurements that are both accurate (low bias) and precise (low variance).
“Ideal methods” would achieve this balance, providing reliable and consistent results.
“Sloppy” methods might consistently miss the mark (high bias) but do so in a predictable manner (low variance).
“Overfit” refers to methods that fit the given data too closely, capturing not only the underlying patterns but also the noise or random error. This results in low bias but high variance, making the model less generalizable to new or unseen data.
A hallmark of all clinical science and epidemiology is to “Explain” variance. We want to understand the sources of variance and explain them. This is the essence of the scientific method.
This framework can serve as a useful reference for anyone trying to grasp these concepts, whether they’re seasoned researchers or newcomers to the field.
Hierarchical Data: Adding Layers of Complexity
Clinical data, with its nested structures, further complicates the bias-variance interplay:
Longitudinal Data: In repeated measures, within-person fluctuations might introduce both error (if measurements are inconsistent) and variance (if there’s a natural rhythm or pattern to the changes). The correlation between measurements, especially those taken closer in time, adds another layer to decipher.
Phenotypic Clustering: Grouping by phenotype or characteristics introduces variance at multiple levels. Within a cluster, measurements might show low variance, but across clusters, differences can be significant. However, errors in measurement or biases in data collection can permeate these clusters, further muddying the waters.
Decoding the Conundrum in Practical Scenarios
When you dive into regression analyses or hierarchical datasets:
Clinical Medicine: Focus on patterns. Is a patient’s response to treatment consistent across time? If fluctuations are beyond expected variance, consider the role of bias.
Public Health: While population-level trends provide insights, diving deeper into phenotypic clusters can reveal nuanced stories. Are observed variations typical of a phenotype, or could they result from systematic biases or errors in data collection?
Concluding Thoughts
Clinical research is both an art and a science. While the dance between bias and variance can be intricate and at times confusing, appreciating their roles and potential interplay ensures our interpretations are both grounded and discerning. As we continue our journey, keeping an eye on these subtle dynamics will ensure our path remains as clear as possible, even in the midst of data-driven complexities.
Netflix - Live to 100: Secrets of the Blue Zones
The ecological fallacy is a critical issue in epidemiology and social science, and it’s relevant to consider when interpreting findings such as those in the Netflix special on Blue Zones by Dan Buettner. Here’s how it relates to the topic at hand:
Variance Between Groups vs. Within Groups
Dan Buettner is discussing Blue Zones as areas where the life expectancy is notably higher than the average. However, there’s no rigorous discussion about the within-group variance in these Blue Zones (The special attempts to show homogeneity and thus low variance) compared to the between-group variance (Blue Zones vs. the United States, for example).
If the variance between groups (Blue Zones vs. United States) is much larger than the variance within groups (within Blue Zones or within the United States), then it would suggest that the factors that make Blue Zones unique are extremely potent and could be generalized to the rest of the population with a higher degree of confidence.
If the variance within groups is much larger than the variance between groups (if there is diversity in the blue zones), then the factors contributing to life expectancy in Blue Zones could be much more complicated and not easily generalizable. It would mean that there are many people in Blue Zones who do not live exceptionally long lives and many people in other parts of the world who do, for reasons not covered by the “secrets” of the Blue Zones.
ICC (Intra-Class Correlation)
The Intra-Class Correlation coefficient (ICC) here would give us a quantifiable measure of how much of the variance in life expectancy can be attributed to the “group” one is in (i.e., whether one lives in a Blue Zone or not) versus individual-level factors.
ICC > 0.5: If the ICC is greater than 0.5, it would suggest that the variance between groups is larger than the variance within groups, giving more credence to the idea that there are specific, generalizable factors in Blue Zones contributing to long life.
ICC < 0.5: If the ICC is less than 0.5, it implies that the variance within groups is larger, meaning that individual-level factors (genetics, personal lifestyle choices, etc.) are playing a more significant role than the “group-level” factors the Blue Zones represent.
Conclusions and Segue into Hierarchical Data
Does the Netflix special did delve into these nuances? If not, it runs the risk of overgeneralizing and simplifying a complex issue. Life expectancy is influenced by multiple factors, including genetics, lifestyle, healthcare systems, and even luck, to some extent. Hierarchical models would be the rigorous approach to these kind of data, accounting for individual-level factors nested within group-level factors, thereby providing a more nuanced understanding of life expectancy across different groups. Its noteworthy that the Netflix special interviews several academics and clinicians from respected institutions, but none of them discuss the ecological fallacy or the need for hierarchical models.
Thus, in your class on hierarchical data, you can discuss how a hierarchical model would be more appropriate for understanding such complex, nested phenomena, and why simply comparing averages between groups (as seems to be the case in the Netflix special), laced with anecdotes that bias the viewer to perceive low variance in the habits of blue zone inhabitants, can lead to misleading conclusions. How representative are these anecdotes of the entire population of Blue Zone inhabitants? How much variance is there within Blue Zones? How much variance is there between Blue Zones and the rest of the world? These are all questions that can be answered with hierarchical models.
A critique by GPT-4
To err is human.
\(\vdots\)
It appears you’ve delved into an exploration of narrative, statistics, and even philosophy, blending them together in an insightful meditation. You talk about how storytelling—from literature to film to documentaries—often presents protagonists (or subjects) who are more outliers than representatives of a population. This impacts our perceptions, biases our understanding, and shapes our inferences about reality. The use of the term ICC (Intraclass Correlation Coefficient) seems metaphorical here, representing how different or similar the protagonist is to the rest of the population.
Your mention of “careful casting” not only refers to the literal casting in films but also to the selection bias that happens in stories, documentaries, and data representation. This is a smart observation: in storytelling, we are often shown a specific perspective that might not be representative of the broader population, thereby affecting our inferences and interpretations. You could extend this metaphor to the way data is often “cast” in scientific studies, where outliers or non-representative samples can unduly influence findings and public perception.
As for the code snippet, it seems to fit a linear mixed-effects model to some hypothetical data representing ICC scores for various composers at different phases in their careers. It could metaphorically relate back to your main point: even when we use complex statistical methods to understand a population (in this case, the “composers”), we need to be aware of the biases that can come with our “casting choices” (the data points and variables we decide to include).
Your narrative certainly traverses an intriguing path—from the arts to statistical models to questions of the multiverse—each layer of complexity casting its own set of characters and contributing its own form of bias. It’s a multidimensional take on how we perceive and interpret the stories around us and the data we gather.