In addition to using CAAP results to assess our general education Academic Outcomes Assessment Plan (AOAP), exit interviews, focus groups, student surveys, and employer surveys are also used. In the last DATAWave, students’ CAAP writing scores were analyzed. In this article, student responses from focus groups and exit interviews will be discussed. Students responses to exit interviews conducted by their college’s dean were reported last semester. It is anticipated, perhaps in the fall, that employer survey results will be reported.
Focus groups were held for the first time in the fall of 1994 for students graduating that semester. Of the number of students who signed up for graduation, and after the elimination of out-of-state addresses, approximately 100 students were solicited to participate in focus groups. Care was taken to ensure that an appropriate sample by college, gender, and ethnicity was obtained. In addition to sending a personalized letter, follow-up telephone calls were conducted to schedule the day and time students might participate in one of the four focus groups. While 21 students registered to participate in the focus groups, 10 students actually participated.
The AOAP for general education indicates “the majority of upper division students will articulate how they have learned to write and speak English effectively, including the ability to read and listen with understanding and critical discernment.” In an attempt to assess this, the following question prompts were used:
As this was an initial effort at focus groups, it quickly became apparent that to ask this level of detail in questions for all general education subject areas was excessive for the one hour time limit. The majority of students did state that the general education courses in English promoted their writing ability. However, when asked for more detail students generally were unable to explain how it helped. The best responses were those that indicated that writing skills were improved. Similarly, some students reported that communication classes helped improve speaking skills. Nearly half the students reported that they gained critical thinking abilities in the English areas; however, those skills only became apparent to the student later in their academic career, not while taking English 102 and 104. Overall, students indicated that they were pleased with the learning in the English and Communication classes, but there were some exceptions. A sampling of their comments follows:
These same students were asked to complete a short survey. When asked to respond to the question “I write and speak English effectively, including the ability to read and listen with understanding and critical discernment,” 8 of the students strongly agreed, 1 agreed, and 1 disagreed. When asked the question, “The general education program at ENMU improved my English skills,” 2 students strongly agreed, 6 students agreed, 1 student disagreed, and 1 student strongly disagreed.
In conducting the focus group with these students, it became clear that they were generally pleased with the learning that occurred, were supportive of the English and Communication departments, and felt prepared for employment or graduate school upon graduation from Eastern.
The CAAP Reading Test measures students’ comprehension as a product of referring, reasoning, and generalizing. The test items require the student to derive meaning from several texts by 1. referring to what is explicitly stated; 2. reasoning to determine implicit meaning; and 3. drawing conclusions, comparisons, and generalizations beyond the text. The test consists of four prose passages of about 900 words each that are representative of the level and kinds of writing encountered in college curriculum.
CAAP reading scale results have been reported in previous editions of the DATAWave. As reading is not a specific goal of Eastern New Mexico University’s general education curriculum, it is not possible to link reading test results to specific coursework. However, students’ reading abilities can be compared to incoming scores on the ACT. Also, score results can be examined for transfers versus natives, by motivation, by gender, and by ethnicity.
The national average for 4-year public college sophomores, the norm group most similar to Eastern’s rising juniors, on the reading scale is 62.7 (standard deviation 5.3). Of the 890 students who have taken the CAAP, from the fall of 1983 through the spring of 1995, the average is 62.2. However, more than 50% of the students (51.2%) did score above the national average. Eastern students compare favorably to the national norm.
Table 1 reports the ACT score of 321 Eastern students by decile in comparison to their CAAP reading scale results. When scores are compared by decile ranges, it can be observed that 141 (44%) of the students in the cohort improved their reading ability over their incoming ACT score, 63 students (20%) showed no change, and 117 (36%) showed a decline in scores. ACT encourages caution in generalizing results from the decile comparison to all students. However, for the students whose incoming ACT scores and CAAP scores are available, there is clear evidence that a sizable number of students demonstrate overall improvement in reading abilities.
| ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Approximately half (50.7%) of students tried their best on the self-report of motivation on the CAAP reading scale. For this group, the average score was 63.2, clearly above the national norm. The remaining half, who did not try as hard, all scored below (61.17) the national mean. Student motivation is considered to be a concern in the administration of the CAAP and analysis of the results. Scores differed significantly when classified by motivation (F(4,885)=12.50, p=.0000).
Of the 890 students who completed the CAAP, 523 indicated that they were first-time freshmen here at Eastern. These students scored slightly higher than the transfer students (62.37 versus 61.94), though these results were not statistically significant (1.24). Also, as it was evidenced in the writing scale scores, women scored statistically significantly higher than men (F(2,887)=9.24, p=.0001).
When CAAP scale scores are compared by ethnicity, students who preferred not to respond and students who responded as ‘other’ scored the highest, followed by those responding as white. The difference in comparing CAAP reading scale results by ethnicity was significant (F(8,865)=10.15, p=.00).
In conclusion, the CAAP reading test indicates that Eastern students perform well in comparison to national norms in their reading abilities. When students’ CAAP results are compared to incoming ACT scores, when highly motivated students are compared to the national norms, and when native students are compared to transfer students, Eastern students compared favorably.
