Eastern New Mexico University has assessed students’ learning in general education through objective testing since 1986. Results from the College Outcome Measures Program (COMP) was reported in the last edition of the DATAWave. The COMP was used from 1986 to 1992, and in 1993 was replaced by the Collegiate Assessment of Academic Proficiency (CAAP). The CAAP yields scale scores for writing, reading, mathematics, science reasoning, and critical thinking. Each of these scale scores will be examined in some depth through the DATAWave this semester—beginning with writing. To perhaps better understand students’ performance on the CAAP and learning which occurs here at Eastern, students’ CAAP performance on the writing scale will be compared to letter grades in general education courses, incoming ACT scores in English, the semester in which they participated in the CAAP, the term in which they completed their general education coursework, their self-reports of academic and writing abilities, their motivation in completing the CAAP, freshman students versus transfer students, and by gender.
The writing scales test of the CAAP consists of 72 items in a multiple choice format designed to be completed within 40 minutes. The test measures students’ understanding of standard written English in usage/mechanics (punctuation, grammar, sentence structure) and rhetorical skills (strategy, organization, style).
Students at Eastern complete the CAAP at the rising junior status (between 55 and 65 credit hours completed), and are compared to a national normative sample of college sophomores. The CAAP is designed to be given to students at the end of their sophomore year to measure their learning in general education.
Usage/mechanics and rhetorical skills form subscales of the writing scales score. Spelling, vocabulary, and rote recall of rules of grammar are not tested. The fall 1995 normative sample for 4-year public college sophomores was 64.1 for writing scales, 17.1 for usage/mechanics, and 17.0 for rhetorical skills. Fall 1995 norms are being used for purposes of simplicity. (Norms for each year of CAAP administration are available in the Assessment Resource Office.) For all three scales, Eastern students scored slightly below the national norms (writing scores-63.05, usage/mechanics-16.61, and rhetorical skills-16.57). In examining histograms of the distribution of scores, the scores for each of the scales appear to be normally distributed though somewhat leptokurtic (This indicates that the scores tend to mass towards the means). In comparing the standard deviation of scores for the normative sample to Eastern’s results, they appear to be similar. All normative information as well as a list of schools that make up the 4-year college sophomore cohort is available in the Assessment Resource Office.
The philosophy of general education of Eastern New Mexico University is stated on page 53 of the 1995-1997 Undergraduate Catalog. Most pertinent to this report is that the graduate of ENMU shall “use written and spoken English effectively, which includes the ability to read and to listen with understanding and critical discernment.” Courses in the general education requirements that must closely address this goal would be English 102 and 104. CAAP scale scores of reading and critical thinking may also be measures of appropriate learning in these classes to achieve the aforementioned goal. It is understood that objective testing alone would not measure this goal for the purposes of assessing learning. CAAP testing is augmented by the use of exit interviews, focus groups, and surveys. These results are summarized and provided to the general education committee on a regular basis. CAAP results are shared here as a means of informing the entire campus community.
English 102, English Composition, is a course which includes “sentence and paragraph grammar; literacy models; writing narrative, persuasive, and expository papers; enhanced critical thinking and writing skills.” English 104, English Compostion and Research, is “a continuation of English 102; applies critical thinking, reading, and writing skills to research paper writing.” To this researcher, it seems these classes are the best cases of the type of learning which should occur in a general education pattern to be measured by CAAP results. In analyzing the results for English 102 and 104, letter grades were recoded in numeric values with letter grades of A = 4, B = 3, C = 2, D = 1, F = 0, S = 2, and W = missing variables. [When reporting results, space would not permit the inclusion of a sum of squares means, standard deviations, degrees of freedom, and the number of cases for all analyses conducted. This report will focus on informing the reader the trends in the data, and when the results were statistically significant.]
Of the 892 CAAP results available, letter grades in English 102 were found for 447 students. For this group 168 earned a grade of A, 166 the grade of B, 99 the grade of C, 7 the grade of D, and 7 the grade of F. For the overall writing score the highest mean score was found for students with the letter grade of C; however, the mean scores were not statistically significant in their difference. Overall the mean score for students in the cohort was 63.00. Similarly on the subscales, higher scores were not associated with higher grades, and the distribution of means were not statistically significant. This indicates that the scores earned on the CAAP when examined by grades earned in English 102 do not represent different groups of students.
Five hundred sixty-three students who have participated in the CAAP have completed English 104 at Eastern. Two hundred six of these students have earned the letter grade of A, 185 the grade of B, 146 the grade of C, 20 the grade of D, and 6 the grade of F. The mean score of students who earned an A in English 104 was 64.65. This exceeds the national norm and ENMU average. Unlike the case with English 102, there appears to be a positive association between student performance on the CAAP writing scale and English 104. When the means are compared for the writing scale, there is a statistical significant difference. Similar results were found for the two subscales. It appears that the learning which occurs in English 104 is much more closely associated with that which is measured on the CAAP writing scale. Similarly, students who do well in English 104 perform better on the CAAP.
When regressing writing scale scores by letter grades earned in English 102, an insignificant amount of the variances is accounted for (R2=.00056). When a similar analysis is conducted using English 104, a statistically significant amount of the total variance is accounted for-8%.
Working with ACT, it was possible to examine students English subscale scores on the ACT test with the writing scale score on the CAAP. This analysis was conducted by distributing students scores by deciles. In preparing the ACT/CAAP decile cohort matrices, ACT first matches all students who have taken both the ACT and CAAP assessment. They next construct decile categories for both the ACT and the CAAP assessment. Decile categories are based on all students in the reference group and not just on Eastern local results. (Please see Figure 1.) By interpreting the decile matrix, it is possible to see how students performed on the CAAP relative to their incoming ACT score. For example, in review of row 1, 39 students originally scored at the first decile on the ACT score. Subsequently, 24 of them improved beyond the first decile. Similarly, looking at the bottom row, 23 students originally scored at the highest decile on the ACT, and subsequently 18 of them no longer score at the highest level. Students who score below the shaded diagonal in the matrix are students for whom there is an overall decline in achievement when ACT scores are compared to CAAP scores, students in the shaded diagonal have remained above the same, and students above the diagonal have performed better. Overall, 41% of students' (131) relative performance declined as measured by the ACT and CAAP, 23% (73) remained about the same, and 37% (118) improved. Interpretation of these results is cautioned as they do not necessarily represent the entire learning which occurred in the general education courses here at Eastern New Mexico University; they are representative only of the students for whom CAAP and ACT scores could be matched. Also for those students who scored below the diagonal, this may or may not be representative of abilities, motivation, or intentions to sabotage test results.
In a linear regression analysis, 61% of the total variance in CAAP writing scores is accounted for in students’ ACT scores. (Needless to say this was a statistically significant result.)
On the subject of writing motivation, students are asked to respond if they tried their best, gave moderate effort, gave little effort, or gave no effort in completion of the writing section of the CAAP. Needless to say, students performed better who tried their best. Of the 892 students who have completed the CAAP, 29 did not respond to this question, 579 indicated they tried their best, 251 indicated they gave moderate effort, 27 indicated they gave little effort, and 6 reported giving no effort. Students who tried their best scored on average 63.67 compared to 61.66 for those who gave no effort. This result was statistically significant. In a linear regression analysis writing motivation accounted for 3% of the total variance (R2=.03), and this was found to be a statistically significant result.
In the September 20, 1995 (Vol.2, No.2) edition of the DATAWave, CAAP results were reported by term. When the writing scale scores are analyzed by CAAP administration date, it is observed that fall scores have been higher than the spring scores. Results from spring 1995 are the lowest to date. It is difficult to speculate that these are a function of testing conditions or students' abilities. However, the difference in mean scores for the various test dates is statistically significant at the .05 level. Because this is not a continuous variable, a linear regression could only be conducted if the values were dummy coded. This analysis was not conducted.
Next, students CAAP results were reviewed by term in which they completed English 102 and 104 (Note: In all cases when CAAP results were compared to course grades, courses were completed prior to the administration of the CAAP.) Students' completion dates for English 102 and 104 range from the 1970’s to the present. The term in which students completed these courses were classified into 5 groups; 1973-1989, 1990, 1991, 1992, and 1993-1994. For English 102, 82 students were in the first cohort, 87 in the 1990 cohort, 149 in the 1991 cohort, 106 in the 1992 cohort, and 37 in the 1993-1994 cohort. For English 104, 48 students were in the first cohort, 55 in the year 1990, 156 in 1991, 184 in 1992, and 157 in 1993-1994. The mean scores were not statistically significantly different. The results of the 1993-1994 group may be associated with motivation and the relatively low performance of the spring 1994 cohort.
| ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CAAP writing scale scores were also assessed by students' self-reported academic and writing abilities. This data was collected by the Cooperative Institutional Research Project (CIRP) Freshman Survey. Students can respond to questions about their academic and writing ability by answering if they thought they were in the top 10%, above average, average, or below average. Of the 892 CAAP participants, data from the CIRP Freshman Survey was available for 204. Fifty-six percent of students that rated their academic ability as above average or in the top 10% had writing scale scores of 65.2 and 66.4, respectively; well above the local and national norms. Forty percent of transfer students in the sample rated their writing ability as above average or in the top 10%, and their scores on the CAAP writing scores were also well above average-65.65 and 66.46 respectively. The differences in mean scores were statistically significant. Students' self-report of academic ability accounts for 20% of the total variance in scores in a regression analysis, and writing ability accounted for 18% when entered alone. Both of these results were statistically significant.
When CAAP writing scale scores were analyzed by gender, women students consistently scored better than did men. This was discussed in more detail in the February 6, 1995 (Vol.1, No.4) edition of the DATAWave. Women scored on average 63.92 compared to 61.73 for men, and again this was statistically significant at the .05 level. In linear regression, with gender entered as a dummy variable, 5% of the total variance was accounted for, and this too was statistically significant.
Students who self-reported they began at Eastern New Mexico University as a first-time freshman (n=524) scored higher than those who self-reported they transferred to the University (n=367). This score was also statistically significant. When this variable was converted to a dummy variable and entered into a linear regression, 1% of the total variance was accounted for, which was found to be statistically significant at the .05 level.
CAAP writing scale results were analyzed by: letter grades earned in English 102 and 104, ACT English scale scores, CAAP administration dates which determined completed courses, self-reported academic and writing abilities, student motivation, gender, and freshman enrollment status. Mean scores, when classified by these variables, differed significantly. However, most interesting is the difference associated with letter grades and ACT scores.
To better interpret how these variables may account for the total variance, all variables were entered into a linear regression step-wide method with the writing scale as the dependent variable. ACT English scores and English 104 scores were first to enter when all other independent variables were blocked, no other variables entered the equation. ACT English alone accounted for 62% of the total variance (R2), and when English 104 entered the equation, the R2 was raised to 64%. Again, no other variables could enter the regression analysis. This demonstrates that English 104 grades are associated with CAAP scores and inseemingly English 104 has a positive impact on CAAP writing scale scores.
In these brief pages of the DATAWave, an attempt has been made to look at the many variables which might impact CAAP writing scores. The purpose is to document student learning, and ways in which students education might be enhanced here at Eastern. Parametric statistics suggest that the learning which occurs in English 104 is adding to students knowledge base; more so than the background variables. One possible recommendation would be to do more of what is being done in English 104.
Members of the campus community are invited to make inquiries of this and additional data sets. You may also wish to request data sets for continued analysis. To inquire about such data sets and/or support for assessment research, please contact the Assessment Resource Office.
