[2023 - British Journal of Mathematical and Statistical Psychology]
It is common practice in both randomized and quasi-experiments to adjust for baseline characteristics when estimating the average effect of an intervention. The inclusion of a pre-test, for example, can reduce both the standard error of this estimate, and – in non-randomized designs – its bias. At the same time, it is also standard to report the effect of an intervention in standardized effect size units, thus making it comparable to other interventions and studies. Curiously, the estimation of this effect size including covariate adjustment has received little attention. In this article, we provide a framework for defining effect sizes in designs with a pre-test (e.g., difference-in-differences and analysis of covariance) and propose estimators of those effect sizes. The estimators and approximations to their sampling distributions are evaluated using a simulation study and then demonstrated using an example from published data.
[2022 - Cogent Education]
This study tested The Academy for Teachers’ PD program, a content-focused intensive workshop with complementary events, for impact on teacher retention and other key teaching outcomes. We used an exploratory mixed methods approach, with interviews followed by a post-only quasi-experimental design of intervention effect. Qualitative data illustrated mechanisms underpinning program success. Significant positive effects on teacher retention (OR=3.71), integration of material into the classroom, perceived pride in the profession, and teacher efficacy with small to moderate effect sizes were found. This study illuminates how a holistic approach to PD can have measurable positive impacts on teacher retention and teaching outcomes.
[2022 - Research Synthesis Methods]
Missing covariates is a common issue when fitting meta-regression models. Standard practice for handling missing covariates tends to involve one of two approaches. In a complete-case analysis, effect sizes for which relevant covariates are missing are omitted from model estimation. Alternatively, researchers have employed the so-called "shifting units of analysis" wherein complete-case analyses are conducted on only certain subsets of relevant covariates. In this article, we clarify conditions under which these approaches generate unbiased estimates of regression coefficients. We find that unbiased estimates are possible when the probability of observing a covariate is completely independent of effect sizes. When that does not hold, regression coefficient estimates may be biased. We study the potential magnitude of that bias assuming a log-linear model of missingness and find that the bias can be substantial, as large as Cohen's d = 0.4-0.8 depending on the missingness mechanism.
[2021 - Educational Studies in Mathematics]
Historic achievement gaps in mathematics favoring male students have recently started to narrow, close, or even shift in favor of female students. Still, in many countries, male students continue to outperform their female counterparts in international mathematics assessments. Chile has one of the highest mathematics achievement gaps in the world, as shown by international assessment tests, with males outperforming females. Using nationally representative longitudinal data and multigroup latent growth modeling (LGM), the purpose of this study was to track the gender scoring gap in mathematics from kindergarten to grade 12. Findings showed gender differences emerged during preschool and increasingly widened as students progressed through school. Although the gap subsided slightly between grades 10 and 12, the initial gap almost doubled by the end of high school, with important implications for access to higher education and choice of major.
[2021 - Alcohol and Alcoholism]
INTRODUCTION: While systematic reviews of substance abuse interventions hold great promise for informing what works for
whom and under what conditions, such reviews must contend with miss- ing data. Missing data can limit the accuracy of
statistical analyses or the relevance of the evidence base. Current methods for analyzing missing data require assumptions
about the reasons that the missing data occur.
OBJECTIVES: In this tutorial, we examine methods for exploring missingness in a dataset in ways that can help to identify
the sources and extent of missingness, as well as clarify gaps in evidence.
METHODS: Using raw data from a meta-analysis of substance abuse interventions, we demonstrate the use of exploratory
missingness analysis (EMA) including techniques for numerical summaries and visual displays of missing data.
RESULTS: These techniques examine the patterns of missing covariates in meta-analysis data and the relationships among
variables with missing data and observed variables including the effect size. The case study shows complex relationships
among missingness and other potential covariates in meta-regression, highlighting gaps in the evidence base.
CONCLUSION: Meta-analysts could often benefit by employing some form of EMA as they encounter missing data.
[2020 - Education Sciences]
The impact of online math programs depends on its implementation, especially in vulnerable populations from developing countries. An existing online platform was adapted, at the request of the Chilean Ministry of Education, to exclusively include exercises previously designed and tested by a paper-based government program for elementary school. We carried out a cluster-randomized controlled trial (RCT) with 50 fourth grade classrooms. Treatment classrooms used the platform in a weekly 90-min math session. Due to a social instability outbreak in the country, a large unexpected disruption with huge absenteeism occurred in the second half of the semester, which turned this study into a unique opportunity to explore the robustness of the platform’s effects on students’ learning. Using multiple imputation and multilevel models, we found a statistically significant effect size of 0.13, which corresponds to two extra months of learning. This effect is meaningful for four reasons. First, it has double the effect of the paper-based version. Second, it was achieved during one semester only. Third, is half that obtained with the platform for a complete year with its own set of exercises and with two sessions per week instead of one. Fourth, it was attained in a semester with a lot of absenteeism.
[2019 - Pensamiento Educativo]
Transition to tertiary education is a key step for students, which usually involves the application of admission tests. In general, when contrasting different groups of people —mainly based on socio-economic variables—, analyzes of this type of evidence are mostly concentrated on averages test scores, where few studies incorporate a gender perspective. In this sense, this study focuses on the Chilean context, and concentrates the analysis in the upper tail of the scores —where the actual university admission is occurring—, and lower —where potential lack of learning opportunities is exposed— of the University Selection Test (PSU). At the same time, gender gaps are compared for the 2014-2018 period in the mentioned top and bottom zones of the distribution of test scores. Results indicate that there are persistent gaps that limit the possibilities for women to pursue university careers in areas related to science, technology, engineering and mathematics. Therefore, it is important that both public and university policies consider affirmative and complementary actions that may address these issues, considering the ongoing higher education reform and possible changes into the Chilean university admission system.