Development of a Multiple-Choice Writing Examination Validated with a Cognitive Diagnosis Model

The aim of this study was to explore the possibility of diagnosing second language (L2) learners’ writing skills through a multiple-choice examination by applying a CDM (cognitive diagnosis model). L2 learners’ writing abilities have mostly been assessed using direct methods such as essay tests. However, this method not only requires a large amount of human raters’ labor, but also faces several issues such as intra-rater and inter-rater reliabilities. Essay examinations are also hard to validate because they are limited in checking the internal structure (i.e., conducting factor analysis). To compensate for these limitations, this study developed a multiple-choice examination and validated it with factor analysis and a CDM. The examination consisted of 20 items on key L2 writing components: content, organization, grammar, vocabulary, and mechanics. It was conducted on 109 college students. The CDM analysis revealed that the test was valid based on response processes. It provided substantial information about each test-taker’s various skills. These results imply that L2 learners’ writing abilities can be assessed in an indirect way and that the test can be conducted with a large body of students to provide useful information regarding their writing abilities at the same time.