Writing Proficiency Assessment

Summary

The ULO Project on Writing Proficiency began in September 2008. The primary writing element assessed skill attainment at three course levels: first-year composition, writing-intensive GE, and senior discipline specific. By collecting a large sample of student essays, establishing a scoring rubric and inter-rating reliability, the results showed that writing skill improved after the freshman year (although sophomores, juniors, and seniors exhibited statistically equivalent levels of attainment). Other assessments involved the first-year composition course, the graduation writing requirement (GWR) , and employer surveys. These assessments led to two of the institutional action items listed below. There was strong support voiced during the Educational Effectiveness Review (EER) site visit that the University would continue to give priority attention to these plans.

Writing Assessment Planning

Committee Membership

Members on the Committee were composed of faculty and staff and were led by Brenda Helmbrecht (Writing Director, English). Other members included Kathyrn Rummell (English), Brian Self (Mechanical Engineering), Dawn Janke (English), Don Choi (Architecture), Clare Battista (Economics), Deborah Wilhelm (WPE and WINGED). 

Method

• The committee collected work from 56 class sections that either had a GE designation of “writing intensive” or were taught by faculty members who made writing a priority.
• In total, the committee collected 1,147 essays. From this pool, the committee randomly selected 272 essays for scoring: 88 from freshmen, 41 from sophomores, 54 from juniors, and 89 from seniors. 153 of the essays were from men (56%), and 119 were from women (44%), which approximates the university’s gender mix.
• There were three norming and scoring sessions. Once inter-rater reliability was established, two readers scored each essay from which all identifying information about student or class level had been removed. Because of time constraints, the two scores were averaged rather than using a third reader to resolve discrepancies. The average scores were used in the following analyses.

Results/Traits

• Seniors had higher scores across all rubric traits than freshmen.
• Juniors scored higher than freshmen on synthesis, mechanics, and support.
• Sophomores scored higher than freshmen on purpose, mechanics and support.
• Sophomores, juniors, and seniors exhibited statistically equivalent levels of attainment across all traits.

Committee Activities and Timeline

date	activity
2008-2009	Set up the Writing Assessment Committee Developed a University Writing Rubric University Writing Rubric (PDF) Collected and assessed student work from GE Area A1 (Expository Writing)
2009-2010	Collected and assessed student work from writing-intensive GE Areas C: Arts and Humanities: (lower-division C1-C2 and upper-division C4), and GE Area D: Society and the Individual (upper-division D5) , as well as from senior-level discipline-specific courses. The goal of this tiered assessment plan was to obtain a wider perspective of students' writing abilities than can be gained by looking at writing development at a single level (such as with the Writing Proficiency Exam, WPE, which was intended to focus on junior-level writing). In effect, this assessment method will track the nature of writing development across three stages in students' educational careers. A survey was created that asked students to reflect on their writing experiences at Cal Poly to understand:  - the kinds of instruction they received - the writing process they used - the kind of support they received on campus - their impressions of themselves as writers
2010-2011	Used the data to develop a plan for the following academic year by identifying ways in which Cal Poly can further support both student writers and faculty who teach writing

Graduation Writing Requirement

All CSU students must satisfy the Graduation Writing Requirement (GWR). Cal Poly students can meet this requirement in two ways:

1. Earn a C or better and successfully complete a timed essay in a GWR-designated, 300-level, writing-intensive GE course. Students who are unsuccessful receive feedback and at least one more opportunity to complete the essay. The pass rate was 84% for the academic year 2010-11.
2. Pass the Writing Proficiency Exam (WPE), a 350-500 word, timed, expository essay test scored by writing experts and other faculty members. The WPE pass rate was 70% for the academic year 2010-11. The essay and exam results likely constitute non-comparable samples for several reasons: students select the method of administration; the tests are administered in different environments; the content differs from test to test; the scoring differs across test types; and students taking the GWR course receive feedback and have a second opportunity to write the essay. In addition, each test may attract a different population, a factor that may interact with variables such as college, ethnicity, interest in writing, etc.

To date, this question has not been looked at in a systematic way because the data have not been readily available. Finally, the essays administered in a GWR course may not be suitable for drawing university-level conclusions because they are only assessed by the instructors of record. However, multiple readers score the WPE using the WPE scoring criteria, which differ from those of the expository writing rubric.

WPE readers assign a single score ranging from 1, ineffectual paper, to 6, exemplary paper, based on four traits: comprehension, organization, development, and expression. Stronger connections could be made between the WPE and expository writing rubrics. The expository writing rubric could be revised to function holistically, allowing readers to assign one score to an essay. Conversely, the WPE rubric could be revised to function analytically and thus provide more formative results. The latter approach seems appropriate as the WPE rubric was developed some time ago outside the framework of university-wide assessment.

Recommended Action Items

1. Ensure that Cal Poly juniors and seniors continue to improve their writing skills.

• Coordinate efforts with the University Writing and Rhetoric Center to develop and raise awareness of outreach programs that target upper-division students.
• Identify upper-division students who struggle with writing before their senior year, especially ESL students, and offer additional upper-division writing courses for these students.
• Coordinate efforts with the Center for Teaching, Learning and Technology (CTLT) and the Writing in Generally Every Discipline (WINGED) program to offer workshops and develop learning communities for faculty members who teach upper-division, writing-intensive courses in GE/majors.
• Emphasize the value of writing in every discipline by identifying non-GE, upper-division, writing-intensive courses in the majors and across colleges; if such courses are difficult to identify, work with departments to develop discipline-specific, advanced writing courses, possibly tied to the senior project.
• Actively support Cal Poly’s acquisition of an e-portfolio and assessment management system so that students can document and assess their own progress as writers.

2. Align learning experiences so that GE, the GWR, and the senior project form a coordinated assessment of writing skills at the beginning, developing, and mastery levels.

• Develop a single expository writing rubric for use by GE or GWR-designated courses, the WPE, and the senior project.
• Require Cal Poly undergraduates to satisfy the GWR as juniors, i.e., as soon as possible after completing ninety units, so that they can receive additional writing instruction if necessary before attempting the senior project.
• Make the WPE a formative assessment. The exam should be re purposed so that it becomes a formative tool for improvement rather than a summative gatekeeper to graduation.