Elsevier

Journal of School Psychology

Volume 60, February 2017, Pages 65-82
Journal of School Psychology

Measurement equivalence: A non-technical primer on categorical multi-group confirmatory factor analysis in school psychology

https://doi.org/10.1016/j.jsp.2016.11.002Get rights and content

Abstract

Evidence-based interventions (EBIs) have become a central component of school psychology research and practice, but EBIs are dependent upon the availability and use of evidence-based assessments (EBAs) with diverse student populations. Multi-group confirmatory factor analysis (MG-CFA) is an analytical tool that can be used to examine the validity and measurement equivalence/invariance of scores across diverse groups. The objective of this article is to provide a conceptual and procedural overview of categorical MG-CFA, as well as an illustrated example based on data from the Social and Academic Behavior Risk Screener (SABRS) – a tool designed for use in school-based interventions. This article serves as a non-technical primer on the topic of MG-CFA with ordinal (rating scale) data and does so through the framework of examining equivalence of measures used for EBIs within multi-tiered models – an understudied topic. To go along with the illustrated example, we have provided supplementary files that include sample data, Mplus input code, and an annotated guide for understanding the input code (http://dx.doi.org/10.1016/j.jsp.2016.11.002). Data needed to reproduce analyses in this article are available as supplemental materials (online only) in the Appendix of this article.

Section snippets

Role of measurement

In psychology and education, researchers and practitioners regularly attempt to assess constructs that are believed to exist but cannot be observed directly (e.g., intelligence, mathematics achievement, depression, happiness). Because these constructs cannot be directly observed, the degree to which respondents possess an unobservable trait is typically inferred based on their responses to items (often responses to items on scales with Likert-type formats) that are believed to reflect the

What techniques are available to examine ME/I with ordinal data?

Rating scales with Likert-type formats are commonly used to assess the outcomes of social, emotional, and behavioral interventions in school settings, and these rating scales usually produce ordinal data. Ordinal variables consist of three or more mutually exclusive levels, which are presumed to be rank-ordered (Kline, 2010). Though commonly treated as interval variables within psychological and educational research, scales with Likert-type item formats typically lack the necessary

Necessary sample size

Prior to beginning analyses, researchers should ensure that they have enough participants to accurately evaluate ME/I. MG-CFA is based on CFA which is a large sample technique that generally requires a minimum of 200 participants (200 per group for MG-CFA). That said, a variety of factors should be considered in decisions about sample size including reliability of indicators, scaling, and strength of factor loadings. For more information about minimum sample sizes needed for CFA, see MacCallum,

Meaningful non-invariance: what next?

In this study, ME/I across race was supported for SABRS scores. However, in practice, it is common to encounter situations where invariance does not hold. For example, the SABRS (described previously) has been shown to meet criteria for configural, metric, and scalar/threshold invariance across gender (Kilgus & Pendergast, 2016). However, suppose that the scale developers added the following item, Gossips about classmates, to the social behavior factor. Gossiping is a form of relational

References (128)

  • A. Beauducel et al.

    On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA

    Structural Equation Modeling

    (2006)
  • A. Beck et al.

    Assessment of depression

  • P.M. Bentler et al.

    Significance tests and goodness of fit in the analysis of covariance structures

    Psychological Bulletin

    (1980)
  • K.A. Bollen

    Structural equations with latent variables

    (1989)
  • D. Borsboom

    When does measurement invariance matter?

    Medical Care

    (2006)
  • D. Borsboom et al.

    Measurement invariance versus selection invariance: Is fair selection possible?

    Psychological Methods

    (2008)
  • N.K. Bowen et al.

    Conducting measurement invariance tests with ordinal data: A guide for social work researchers

    Journal of the Society for Social Work and Research

    (2015)
  • T.A. Brown

    Confirmatory factor analysis for applied research

    (2006)
  • T.A. Brown et al.

    Confirmatory factor analysis

  • M.W. Browne et al.

    Alternative ways of assessing model fit

  • F.B. Bryant et al.

    Principles and practice of scaled difference chi-square testing

    Structural Equation Modeling: A Multidisciplinary Journal

    (2012)
  • B.M. Byrne

    Structural equation modeling with Mplus: Basic concepts, applications and programming

    (2011)
  • B.M. Byrne et al.

    Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance

    Psychological Bulletin

    (1989)
  • F.F. Chen

    Sensitivity of goodness of fit indexes to lack of measurement invariance

    Structural Equation Modeling

    (2007)
  • F.F. Chen

    What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research

    Journal of Personality and Social Psychology

    (2008)
  • F.F. Chen et al.

    Teacher's corner: Testing measurement invariance of second-order factor models

    Structural Equation Modeling

    (2005)
  • G.W. Cheung et al.

    Evaluating goodness-of-fit indexes for testing measurement invariance

    Structural Equation Modeling

    (2002)
  • L. Coertjens

    Change in learning strategies during and after the transition to higher education: The impact of statistical choices on growth trend estimates

    (2014)
  • D.A. Cole et al.

    The insidious effects of failing to include design-driven correlated residuals in latent-variable covariance structure analysis

    Psychological Methods

    (2007)
  • M. Comşa

    How to compare means of latent variables across countries and waves: Testing for invariance measurement. An application using eastern European societies

    Sociológia

    (2010)
  • C.R. Cook et al.

    Development and validation of the Student Internalizing Behavior Screener: Examination of reliability, validity, and classification accuracy

    Assessment for Effective Intervention

    (2011)
  • E. Davidov et al.

    Testing the invariance of values in the Benelux countries with the European Social Survey: Accounting for ordinality

  • R.M. Dennis

    Social Darwinism, scientific racism, and the metaphysics of race

    The Journal of Negro Education

    (1995)
  • D.M. Dimitrov

    Testing for factorial invariance in the context of construct validation

    Measurement and Evaluation in Counseling and Development

    (2010)
  • J.C. DiPerna

    Academic enablers and student achievement: Implications for assessment and intervention services in the schools

    Psychology in the Schools

    (2006)
  • C.V. Dolan et al.

    Testing measurement invariance in the target rotated multigroup exploratory factor model

    Structural Equation Modeling

    (2009)
  • N.J. Dorans et al.

    DIF detection and description: Mantel-Haenszel and standardization

  • E. Dowdy et al.

    School-based screening: A population-based approach to inform and monitor children's mental health needs

    School Mental Health

    (2010)
  • F. Drasgow et al.

    Equivalence of psychological measurement in heterogeneous populations

    Journal of Applied Psychology

    (1985)
  • T. Drummond

    The student risk screening scale (SRSS)

    (1994)
  • M.O. Edelen et al.

    Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement

    Quality of Life Research

    (2007)
  • L.R. Fabrigar et al.

    Evaluating the use of exploratory factor analysis in psychological research

    Psychological Methods

    (1999)
  • H.H. Fairchild

    Scientific racism: The cloak of objectivity

    Journal of Social Issues

    (1991)
  • E. Ferguson et al.

    Exploratory factor analysis: A users' guide

    International Journal of Selection and Assessment

    (1993)
  • D.B. Flora et al.

    An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data

    Psychological Methods

    (2004)
  • D.B. Flora et al.

    Old and new ideas for data screening and assumption testing for exploratory and confirmatory factor analysis

    Frontiers in Psychology

    (2012)
  • M. Fokkema et al.

    Response shifts in mental health interventions: An illustration of longitudinal measurement invariance

    Psychological Assessment

    (2013)
  • C.G. Forero et al.

    Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation

    Structural Equation Modeling

    (2009)
  • S.G. Forman et al.

    Implementation science and school psychology

    School Psychology Quarterly

    (2013)
  • T.A. Glover et al.

    Service delivery for response to intervention: Core components and directions for future research

    School Psychology Review

    (2007)
  • Cited by (151)

    View all citing articles on Scopus

    Data needed to reproduce analyses in this article are available as supplemental materials (online only) in the Appendix of this article.

    View full text