Earticle

초록

영어

The current study aims to design The current study aims to design an analytic writing scale with reference to the TOEFL iBT® Scoring rubric from the Educational Testing Service (ETS) and to examine the reliability of the newly designed analytic rubric using English argumentative writing collected from 26 Korean undergraduate and graduate students. There are three main findings in this study: First, intra-rater reliability was examined among three raters showing a high correlation between the holistic and the analytic scorings. Second, inter-rater reliability was explored in terms of correlation coefficients, agreement rates (adjacent and perfect), Cohen’s Kappa coefficients, and Fleiss Kappa coefficients. In addition, the intraclass correlation coefficient (ICC) revealed a high correlation between the score on a new analytic rubric and the original holistic TOEFL score. Among the four rubric dimensions, ‘unity-progression-coherence’ and ‘language use’ showed relatively lower correlations than the other two dimensions (topic and task, organization and development). Also, the dimension that showed the highest correlation with the holistic ETS score is ‘organization and development.’ Moreover, ‘unity-progression-coherence’ contributed notably to the analytic assessment. A survey on raters’ perceptions of rubrics and scoring, and interviews suggested that they had challenges scoring based on the descriptors of two rubrics (analytic and holistic). This study has pedagogical implications for scoring writing, designing rubrics, and rater training.

목차

Ⅰ. Introduction
Ⅱ. Literature review
A. Reliability between the analytic and the holistic rubric for independent writing
B. Rater perceptions of holistic and analytic scoring
Ⅲ. Research methods
A. Participants
B. Materials
C. Rater training
D. Data analysis
Ⅳ. Results and discussions
A. Intra-rater reliability between scores with the holistic and analytic rubrics
B. Inter-rater reliability between scores with the holistic rubric and with the analytic rubric
C. Correlations between the mean of four dimensions in the analytic score and the holistic score
D. Correlations between the three raters per four each dimension with the analytic score and the holistic score, and the agreement among three raters across four dimensions
E. Qualitative analysis of raters’ survey and interviews
Ⅴ. Conclusion
References
Appendix
Abstract

키워드

저자

• Jungyeon Koo [ Graduate Student, Seoul National University ]

참고문헌

발행기관

간행물

표지보기 관심저널 등록

• 간행물명

  영어영문학21 [English21]

• 간기

  계간

• pISSN

  1738-4052

• 수록기간

  1967~2025

• 등재여부

  KCI 등재

• 십진분류

  KDC 840 DDC 820