A primer on gathering and analysing multi-level quantitative evidence for differential student outcomes in higher education

Article Sidebar

HTML PDF Listen
Published: Mar 12, 2021
DOI: https://doi.org/10.14786/flr.v9i2.675

Main Article Content

Kieran Balloo
Surrey Institute of Education, University of Surrey, UK
https://orcid.org/0000-0002-1745-4653
Naomi E. Winstone
Surrey Institute of Education, University of Surrey, UK
https://orcid.org/0000-0001-8157-8274

Abstract

A significant challenge currently facing the higher education sector is how to address differential student outcomes in terms of attainment and continuation gaps at various stages of students’ transitions. Worryingly, there appears to be a ‘deficit’ discourse among some university staff in which differential outcomes are perceived to be due to student deficiencies. This may be exacerbated by institutional analyses placing an over-emphasis on the presence of the gaps rather than the causes. The purpose of this primer is to provide advice about how institutions can carry out far more nuanced analyses of their institutional data without requiring specialist software or expertise. Drawing on a multi-level framework for explaining differential outcomes, we begin with guidance for gathering quantitative data on explanatory factors for attainment and continuation gaps, largely by linking sources of internal data that have not previously been connected. Using illustrative examples, we then provide tutorials for how to model explanatory factors employing IBM SPSS Statistics (IBM Corp., Armonk, NY, USA) to perform and interpret regression and meta-regression analyses of individual- and group-level (aggregated) student data, combined with data on micro- and meso-level factors. We propose that university staff with strategic responsibilities could use these approaches with their institutional data, and the findings could then inform the design of context-specific interventions that focus on changing practices associated with gaps. In doing so, institutions could enhance the evidence-base, raise awareness, and further ‘embed the agenda’ when it comes to understanding potential reasons for differential student outcomes during educational transitions.

Article Details

How to Cite
Balloo, K., & Winstone, N. (2021). A primer on gathering and analysing multi-level quantitative evidence for differential student outcomes in higher education. Frontline Learning Research, 9(2), 121–144. https://doi.org/10.14786/flr.v9i2.675
Issue
Vol. 9 No. 2 (2021): Frontline Learning Research: Special Issue
Section
Articles

FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.

References

Alin, A. (2010). Multicollinearity. Wiley Interdisciplinary Reviews: Computational Statistics, 2(3), 370–374. https://doi.org/10.1002/wics.84

Boliver, V. (2015). Are there distinctive clusters of higher and lower status universities in the UK? Oxford Review of Education, 41(5), 608–627. https://doi.org/10.1080/03054985.2015.1082905

Broecke, S., & Nicholls, T. (2007). Ethnicity and degree attainment (DfES Research Report RW92). DfES.

Cheung, M. W. L. (2019). A guide to conducting a meta-analysis with non-independent effect sizes. Neuropsychology Review, 29(4), 387–396. https://doi.org/10.1007/s11065-019-09415-6

Courville, T., & Thompson, B. (2001). Use of structure coefficients in published multiple regression articles: β is not enough. Educational and Psychological Measurement, 61(2), 229–248. https://doi.org/10.1177/0013164401612006

Cousin, G., & Cuerton, D. (2012). Disparities in student attainment (DISA). Higher Education Academy (HEA). https://www.advance-he.ac.uk/knowledge-hub/disparities-student-attainment

Dunlap, W. P., & Landis, R. S. (1998). Interpretations of multiple regression borrowed from factor analysis and canonical correlation. The Journal of General Psychology, 125(4), 397–407. https://doi.org/10.1080/00221309809595345

Field, A. P. (2018). Discovering statistics using IBM SPSS Statistics (5th ed.). Sage.

Field, A. P., & Gillett, R. (2010). How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology, 63(3), 665–694. https://doi.org/10.1348/000711010X502733

Fisher, Z., Tipton, E., & Zhipeng, H. (2017). robumeta (Version 2.0) [Computer software]. https://cran.r-project.org/package=robumeta

France, M. K., Finney, S. J., & Swerdzewski, P. (2010). Students’ group and member attachment to their university: A construct validity study of the University Attachment Scale. Educational and Psychological Measurement, 70(3), 440–458. https://doi.org/10.1177/0013164409344510

Freedman, D. A. (1999). Ecological inference and the ecological fallacy (Technical Report 549). https://statistics.berkeley.edu/sites/default/files/tech-reports/549.pdf

Gravett, K. (2019). Troubling transitions and celebrating becomings: from pathway to rhizome. Studies in Higher Education, 1–12. https://doi.org/10.1080/03075079.2019.1691162

Hedges, L. V., Borenstein, M., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley & Sons, Ltd.

Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. https://doi.org/10.1002/jrsm.5

Jones, S. (2018). Expectation vs experience: might transition gaps predict undergraduate students’ outcome gaps? Journal of Further and Higher Education, 42(7), 908–921. https://doi.org/10.1080/0309877X.2017.1323195

Karpen, S. C. (2017). Misuses of regression and ANCOVA in educational research. American Journal of Pharmaceutical Education, 81(8), 84–85. https://doi.org/10.5688/ajpe6501

Lens, D., & Levrau, F. (2020). Can pre-entry characteristics account for the ethnic attainment gap? An analysis of a Flemish university. Research in Higher Education, 61(1), 26–50. https://doi.org/10.1007/s11162-019-09554-y

Miller, M. (2016). The ethnicity attainment gap: literature review. The University of Sheffield Widening Participation Research & Evaluation Unit. https://www.sheffield.ac.uk/polopoly_fs/1.661523!/file/BME_Attainment_Gap_Literature_Review_EXTERNAL_-_Miriam_Miller.pdf

Mood, C. (2010). Logistic regression: Why we cannot do what we think we can do, and what we can do about it. European Sociological Review, 26(1), 67–82. https://doi.org/10.1093/esr/jcp006

Mountford-Zimdars, A., Sabri, D., Moore, J., Sanders, J., Jones, S., & Hiagham, L. (2015). Causes of differences in student outcomes. Higher Education Funding Council for England (HEFCE). https://webarchive.nationalarchives.gov.uk/20180405123119/http://www.hefce.ac.uk/pubs/rereports/Year/2015/diffout/

Mountford-Zimdars, A., Sanders, J., Moore, J., Sabri, D., Jones, S., & Higham, L. (2017). What can universities do to support all their students to progress successfully throughout their time at university? Perspectives: Policy and Practice in Higher Education, 21(2–3), 101–110. https://doi.org/10.1080/13603108.2016.1203368

OfS. (n.d.). Continuation and attainment gaps. https://www.officeforstudents.org.uk/advice-and-guidance/promoting-equal-opportunities/evaluation-and-effective-practice/continuation-and-attainment-gaps/

Pigott, T. D., & Polanin, J. R. (2020). Methodological guidance paper: High-quality meta-analysis in a systematic review. Review of Educational Research, 90(1), 24–46. https://doi.org/10.3102/0034654319877153

Schwartz, S. (1994). The fallacy of the ecological fallacy: the potential misuse of a concept and the consequences. American Journal of Public Health, 84(5), 819–824. https://doi.org/10.2105/AJPH.84.5.819

Singh, G. (2011). Black and Minority Ethnic (BME) students participation in higher education: improving retention and success. Higher Education Academy (HEA). https://www.heacademy.ac.uk/system/files/bme_synthesis_final.pdf

Stevenson, J. (2012). Black and minority ethnic student degree retention and attainment. Higher Education Academy (HEA). https://www.heacademy.ac.uk/system/files/bme_summit_final_report.pdf

Tanner-Smith, E. E., & Tipton, E. (2014). Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 5(1), 13–30. https://doi.org/10.1002/jrsm.1091

Tanner-Smith, E. E., Tipton, E., & Polanin, J. R. (2016). Handling complex meta-analytic data structures using robust variance estimates: a tutorial in R. Journal of Developmental and Life-Course Criminology, 2(1), 85–112. https://doi.org/10.1007/s40865-016-0026-5

Thompson, C. G., Kim, R. S., Aloe, A. M., & Becker, B. J. (2017). Extracting the Variance Inflation Factor and other multicollinearity diagnostics from typical regression results. Basic and Applied Social Psychology, 39(2), 81–90. https://doi.org/10.1080/01973533.2016.1277529

Tieben, N. (2020). Non-completion, transfer, and dropout of traditional and non-traditional students in Germany. Research in Higher Education, 61(1), 117–141. https://doi.org/10.1007/s11162-019-09553-z

Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). University of Chicago Press.

Tipton, E., & Pustejovsky, J. E. (2015). Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. Journal of Educational and Behavioral Statistics, 40(6), 604–634. https://doi.org/10.3102/1076998615606099

Tonidandel, S., & LeBreton, J. M. (2015). RWA Web: A free, comprehensive, web-based, and user-friendly tool for relative weight analyses. Journal of Business and Psychology, 30(2), 207–216. https://doi.org/10.1007/s10869-014-9351-z

UUK & NUS. (2019). Black, Asian and minority ethnic student attainment at UK universities: #closingthegap. Universities UK & National Union of Students. https://www.universitiesuk.ac.uk/news/Pages/Universities-acting-to-close-BAME-student-attainment-gap.aspx

Woodfield, R. (2014). Undergraduate retention and attainment across the disciplines. Higher Education Academy (HEA). https://www.advance-he.ac.uk/knowledge-hub/undergraduate-retention-and-attainment-across-disciplines