Decentralization paradox in educational assessment: validity of the All-Russian School Olympiad
Keywords:
децентрализованные системы оценивания, справедливость оценивания, олимпиадыAbstract
The All-Russian School Olympiad encompasses over 6 million participants annually, determining their educational trajectories, but operates without systematic psychometric validation. This fact contradicts international standards for high-stakes assessment systems. The study aimed to comprehensively assess the validity of the All-Russian Olympiad through literature review and qualitative research of regional jury practices (37 experts from 12 regions). Semi-structured interviews with dual coding (κ=0.76) were employed. Five persistent problems were identified: absence of item piloting (89.2% of experts), lack of transparency in evaluation criteria (20% of regions), local interpretations of standards (70.3%), conflicts of interest within juries (54.1%), and unregulated appeals procedures (39.2%). The All-Russian Olympiad fails to meet any of the six components of Messick's unified validity framework. A hybrid reform model is proposed: 70% of items are developed at the federal level, while 30% of regional developments undergo external review with differential item functioning analysis.References
Messick S. Validity // Educational Measurement / Ed. by R.L.Linn. N.Y.: American Council on Education, 1989. Р. 13–103.
Kane M.T. Validating the interpretations and uses of test scores // Journal of Educational Measurement. 2013. № 50 (1). P. 1–73.
Benton T. (2024). How long should a highstakes test be? // Research Matters 38. Cambridge, 2024. P. 28–47.
Martinková P., Drabinová A., Zvára K., Potužníková E. Why differential item functioning analysis should be a routine part of developing conceptual assessments // CBE–Life Sciences Education. 2017. № 16 (2). [Электронный ресурс]. URL: https://doi.org/10.1187/cbe.16-10-0307.
Fleishman J.A., Spector W.D., Altman B.M. Impact of differential item functioning on age and gender comparisons of functional disability. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences. 2002. No. 57 (5). P. 275–284. [Электронный ресурс]. URL: https://doi.org/10.1093/geronb/57.5.S275.
Болотов В.А. Прошлое, настоящее и возможное будущее российской системы оценки качества образования // Вопросы образования. 2018. № 3. С. 287–297.
Федоров О.Д., Веринчук К.Д. ЕГЭ по истории и валидность его результатов: эксперты региональных предметных комиссий об открытых заданиях // Вопросы образования. 2021. № 3. С. 189–211. [Электронный ресурс]. URL: https://doi.org/10.17323/1814-9545-2021-3-189-211.
Иванова С.В., Малыгин А.А. Доверие как социально-дидактическая проблема при измерении образовательных результатов // Ценности и смыслы. 2024. № 3 (91). С. 53–71.
Приказ Министерства просвещения Российской Федерации от 27 ноября 2020 г. № 678 «Об утверждении Порядка проведения всероссийской олимпиады школьников». [Электронный ресурс]. URL: https://docs.cntd.ru/document/573339049 (дата обращения 14.01.2026).
Engelbrecht J., Mwambakana J. Validity and diagnostic attributes of a Mathematics Olympiad for Junior High School Contestants // Journal of Educational Measurement. 2016. № 53 (1). P. 18–37. URL: https://doi.org/10.1111/jedm.12098.
Florian L., Keßler C., Müller K., Schaper N. Assessing High Performers in the Life Sciences: Characteristics of Exams Used at the International Biology Olympiad (IBO) and Their Implications for Life Science Education // CBE–Life Sciences Education. 2019. URL: https://doi.org/10.1187/cbe.19-10-0215.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education // Standards for Educational and Psychological Testing. AERA, 2014.
Bond T.G., Fox C.M. Applying the Rasch Model: Fundamental Measurement in the Human Sciences (3rd ed.). Routledge, 2015.
Cohen J. Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates, 1988.
Holland P.W., Wainer H. (Eds.). Differential Item Functioning. Lawrence Erlbaum Associates, 1993.
Lipsky M. Street-level bureaucracy: Dilemmas of the individual in public services. Russell Sage Foundation, 1980.
