Which Linguistic Features Predict Writing Quality, and Which Change with Instruction?
Submitted by:
Charles A. MacArthur
Abstract:
Writing is complex process requiring discourse knowledge, linguistic skills, transcription skills, cognitive and metacognitive strategies, and social awareness. Research on linguistic development shows changes with time and expertise in syntactic complexity, vocabulary sophistication, content elaboration, and coherence. Research also identifies linguistic features that predict overall quality, including length, syntactic complexity, and vocabulary (e.g., Uccelli, Dobbs, & Scott, 2013). Automated linguistic analysis tools (McNamara, Graesser, McCarthy, & Cai, 2014) have made linguistic analysis feasible as an assessment tool for formative evaluation and assessment of the effects of instruction. Information on which linguistic features predict quality and how they change over time could have value for formative assessment and instructional design.
The current study used data from an experimental study of a curriculum for college basic writers (Authors, 2015) with 252 students, which found large positive effects on writing quality. The current analysis asked: 1) Which linguistic features predicted quality on the pretest and posttest? 2) Which of those features changed following instruction? Analysis was conducted using Coh-Metrix (McNamara et al., 2014) indices of productivity, syntactic complexity, vocabulary, and cohesion. Confirmatory factor analysis and structural equation modeling were used to develop models predicting quality of pretest and postest essays.
Length was highly predictive of quality on pretests (r=.71) but less so on the posttest (r=.56). Posttest quality was also predicted by cohesion measures of semantic overlap, syntactic complexity, and vocabulary, even after controlling for length. On the pretest, quality was predicted by vocabulary but not by cohesion or syntactic complexity. Following instruction, changes were found in semantic overlap, syntactic complexity, and vocabulary. Results for vocabulary and syntax are consistent with prior research, but prior studies have not found that cohesion measure predict quality (e.g., McNamara, Crossley, & McCarthy, 2010). Implications for further research and for instruction will be considered.
McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27, 57-86.
McNamara, D.S., Graesser, A.C., McCarthy, P., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge: Cambridge University Press.
Uccelli, P., Dobbs, C. L., & Scott, J. (2013). Mastering academic language: Organization and stance in the persuasive writing of high school students. Written Communication, 30, 36-62.