Recent Publications

Recent Publications

[In Press - British Journal of Mathematical and Statistical Psychology]

Effect sizes in ANCOVA and difference-in-differences designs

Larry V. Hedges, Elizabeth Tipton, Rrita Zejnullahi, Karina G. Diaz

It is common practice in both randomized and quasi-experiments to adjust for baseline characteristics when estimating the average effect of an intervention. The inclusion of a pre-test, for example, can reduce both the standard error of this estimate, and – in non-randomized designs – its bias. At the same time, it is also standard to report the effect of an intervention in standardized effect size units, thus making it comparable to other interventions and studies. Curiously, the estimation of this effect size including covariate adjustment has received little attention. In this article, we provide a framework for defining effect sizes in designs with a pre-test (e.g., difference-in-differences and analysis of covariance) and propose estimators of those effect sizes. The estimators and approximations to their sampling distributions are evaluated using a simulation study and then demonstrated using an example from published data.

[In Press - Cogent Education]

The Academy for Teachers Professional Development Program. A Model to Support Teacher Retention

Luesse, H. B., Luesse, J. E., Lawson, J. & Karina G. Diaz

This study tested The Academy for Teachers’ PD program, a content-focused intensive workshop with complementary events, for impact on teacher retention and other key teaching outcomes. We used an exploratory mixed methods approach, with interviews followed by a post-only quasi-experimental design of intervention effect. Qualitative data illustrated mechanisms underpinning program success. Significant positive effects on teacher retention (OR=3.71), integration of material into the classroom, perceived pride in the profession, and teacher efficacy with small to moderate effect sizes were found. This study illuminates how a holistic approach to PD can have measurable positive impacts on teacher retention and teaching outcomes.

[2022 - Research Synthesis Methods]

On the bias of complete- and shifting-case meta-regressions with missing covariates

Jacob M. Schauer, Jihyun Lee, Karina G. Diaz, Therese D. Pigott

Missing covariates is a common issue when fitting meta-regression models. Standard practice for handling missing covariates tends to involve one of two approaches. In a complete-case analysis, effect sizes for which relevant covariates are missing are omitted from model estimation. Alternatively, researchers have employed the so-called "shifting units of analysis" wherein complete-case analyses are conducted on only certain subsets of relevant covariates. In this article, we clarify conditions under which these approaches generate unbiased estimates of regression coefficients. We find that unbiased estimates are possible when the probability of observing a covariate is completely independent of effect sizes. When that does not hold, regression coefficient estimates may be biased. We study the potential magnitude of that bias assuming a log-linear model of missingness and find that the bias can be substantial, as large as Cohen's d = 0.4-0.8 depending on the missingness mechanism.

[2021 - Educational Studies in Mathematics]

A longitudinal study of the gender gap in mathematics achievement: evidence from Chile

Paulina Perez Mejias, Dora Elias McAllister, Karina G. Diaz, Javiera Andrea Ravest Tropa

Historic achievement gaps in mathematics favoring male students have recently started to narrow, close, or even shift in favor of female students. Still, in many countries, male students continue to outperform their female counterparts in international mathematics assessments. Chile has one of the highest mathematics achievement gaps in the world, as shown by international assessment tests, with males outperforming females. Using nationally representative longitudinal data and multigroup latent growth modeling (LGM), the purpose of this study was to track the gender scoring gap in mathematics from kindergarten to grade 12. Findings showed gender differences emerged during preschool and increasingly widened as students progressed through school. Although the gap subsided slightly between grades 10 and 12, the initial gap almost doubled by the end of high school, with important implications for access to higher education and choice of major.

[2021 - Alcohol and Alcoholism]

Exploratory Analyses for Missing Data in Meta-Analyses and Meta-Regression: A Tutorial

Jacob M. Schauer, Karina G. Diaz, Therese D. Pigott

INTRODUCTION: While systematic reviews of substance abuse interventions hold great promise for informing what works for whom and under what conditions, such reviews must contend with miss- ing data. Missing data can limit the accuracy of statistical analyses or the relevance of the evidence base. Current methods for analyzing missing data require assumptions about the reasons that the missing data occur.
OBJECTIVES: In this tutorial, we examine methods for exploring missingness in a dataset in ways that can help to identify the sources and extent of missingness, as well as clarify gaps in evidence.
METHODS: Using raw data from a meta-analysis of substance abuse interventions, we demonstrate the use of exploratory missingness analysis (EMA) including techniques for numerical summaries and visual displays of missing data.
RESULTS: These techniques examine the patterns of missing covariates in meta-analysis data and the relationships among variables with missing data and observed variables including the effect size. The case study shows complex relationships among missingness and other potential covariates in meta-regression, highlighting gaps in the evidence base.
CONCLUSION: Meta-analysts could often benefit by employing some form of EMA as they encounter missing data.

[2020 - Education Sciences]

Implementing Government Elementary Math Exercises Online: Positive Effects Found in RCT under Social Turmoil in Chile

Roberto Araya and Karina Diaz

The impact of online math programs depends on its implementation, especially in vulnerable populations from developing countries. An existing online platform was adapted, at the request of the Chilean Ministry of Education, to exclusively include exercises previously designed and tested by a paper-based government program for elementary school. We carried out a cluster-randomized controlled trial (RCT) with 50 fourth grade classrooms. Treatment classrooms used the platform in a weekly 90-min math session. Due to a social instability outbreak in the country, a large unexpected disruption with huge absenteeism occurred in the second half of the semester, which turned this study into a unique opportunity to explore the robustness of the platform’s effects on students’ learning. Using multiple imputation and multilevel models, we found a statistically significant effect size of 0.13, which corresponds to two extra months of learning. This effect is meaningful for four reasons. First, it has double the effect of the paper-based version. Second, it was achieved during one semester only. Third, is half that obtained with the platform for a complete year with its own set of exercises and with two sessions per week instead of one. Fourth, it was attained in a semester with a lot of absenteeism.

[2019 - Pensamiento Educativo]

Gender Gap in University Admission Test in Chile: What is Happening at the Top and Bottom of the Test Score Distribution?

Karina Gabriela Diaz Yanez, Javiera Andrea Ravest Tropa, Juan Pablo Queupil Quilamán

Transition to tertiary education is a key step for students, which usually involves the application of admission tests. In general, when contrasting different groups of people —mainly based on socio-economic variables—, analyzes of this type of evidence are mostly concentrated on averages test scores, where few studies incorporate a gender perspective. In this sense, this study focuses on the Chilean context, and concentrates the analysis in the upper tail of the scores —where the actual university admission is occurring—, and lower —where potential lack of learning opportunities is exposed— of the University Selection Test (PSU). At the same time, gender gaps are compared for the 2014-2018 period in the mentioned top and bottom zones of the distribution of test scores. Results indicate that there are persistent gaps that limit the possibilities for women to pursue university careers in areas related to science, technology, engineering and mathematics. Therefore, it is important that both public and university policies consider affirmative and complementary actions that may address these issues, considering the ongoing higher education reform and possible changes into the Chilean university admission system.