Eastern New Mexico University (ENMU) began outcomes assessment of its general education curriculum in the fall of 1986 through the administration of the College Outcome Measures Program (COMP) published by the American College Testing Service. The COMP yields a Total Score and Subscale Scores in: Functioning Within Social Institutions, Using Science, Using the Arts, Communicating, Solving Problems, and Clarifying Values. Descriptions of these scale scores and other psychometric properties can be found on the last page.
The purpose of this report is to provide some summary information on the more than 7,000 results obtained from 1986 to 1992. To interested parties, a more comprehensive review of the materials is available. For the years 1986 and 1987, freshmen only participated in the COMP administration. Beginning in 1988, juniors (students with more than 60 credit hours) were added to the cohort. The intention was to compare the scale scores of freshmen and juniors. Any change would then be attributed to student learning here at Eastern New Mexico University.
Many hours were spent exploring the various analytical scenarios for interpreting the data. Sometimes it’s best to tell people what happened, non-statistically, and allow conclusions to be made. For example, in analyzing scores, cohorts could be compared by year, class level, or longitudinally. However, without controlling for the effects of courses completed, transfer courses, incoming ACT, age, gender, etc., it would be difficult to form conclusions.
In 1986, 696 freshmen participated in the COMP administration. The hypothetical “average freshman” in the sample scored at the 43rd percentile on the Total Score. Subscale means ranged from the 35th to 55th percentile. (ENMU’s incoming non-enhanced ACT composite mean scores were 17.4 versus 20.5 for the reference group). Relative strengths were evidenced in Using the Arts and Solving Problems, while scores in Communicating were described as relatively weak. ACT alludes that the lower COMP composite scores might be influenced by lower ACT scores—that is, lower achievement as measured by the ACT score was associated with lower achievement as measured by the COMP (ACT, 1986).
In 1987, 568 freshmen participated in the COMP program. A hypothetical “average freshman” in the sample scored at the 32nd percentile on the Total Score compared to freshman norms. Subscale scores ranged from the 20th to the 37th percentile. Again, the non-enhanced ACT composite mean scores were lower than the national reference group (16.6 versus 20.1). Relative strengths for the year were found in Using the Arts and relative weaknesses were indicated in Functioning Within Social Institutions. Again, there appears to be an association with the lower mean ACT scores achieved by Eastern students and the lower COMP scores; however, it is observed that the general strengths and weaknesses from 1986 to 1987 do not overlap. Also it is observed that, in general, COMP scores from the 1987 sample were lower than the 1986 sample (ACT, 1987).
Six hundred seventy-six freshmen participated in the COMP. For the 1988 cohort, the Total Scores and all Subscale Scores were higher than the previous year; though in 1986, only the Using Science and Clarifying Values subscales were higher. For the 1988 freshman group, the cohort scored nearly 6 points below the reference group. ACT attributed this difference to the students relative weakness in two subscale areas, Functioning Within Social Institutions and Communicating. Again, there were significant differences in the incoming students ACT scores. Eastern students had a non-enhanced ACT composite of 17.8 while the 4-year reference group had a 20.2 average. For the first three years for the freshman cohort, there is no observable trend in the data with the exception that all Subscale Scores, the Total Score, and the ACT score remain below national averages.
More importantly, 1988 was the first year that rising juniors (students with between 60 and 89 credit hours completed) participated in the COMP. The intention was to provide an examination of “gain scores.” In 1988, the Office of Planning and Analysis found that the score of juniors and the total junior population was generally comparable to the reference groups. There is no specific identifiable parameter that counts for the difference in the scores (ENMU 177.3 versus national 181.1). One hundred forty-two scores were available for junior students who had participated in the COMP as freshmen. On the total scale, the changing range of scores was from -35 to +45 points. According to ACT, given pre- and post-test scores and mean ACT scores, Eastern’s reference groups’ estimated gain was 7.6 points.
There were significant differences between Eastern and the reference group in the area of Functioning Within Social Institutions (Eastern scored lower), yet, on the other hand, the largest increases were noted in Using Science and Clarifying Values.
In all areas, juniors scored higher than freshmen. Further, it can be observed that juniors who were natives at Eastern scored higher than non-native students (OPAS, 1989).
One thousand one hundred seventy-three students participated in the COMP in 1989, and of these students, 645 were entering freshmen and 498 were juniors. One hundred three of the rising juniors participated in the COMP as freshmen. The hypothetical “average freshman” in the sample scored at the 59th percentile on the Total Score. This year’s reference group was from 2-year institutions. The reason for the change in normative groups is unexplained. Also, means for the Subscale Scores ranged from the 53rd to 64th percentile. It is observed that the change to 2-year reference groups allowed for a more favorable comparison of ACT enhanced scores (17.7 for Eastern versus 17.0 for the national reference groups). Compared to the reference group, the 1989 sample of freshmen evidenced some relative strengths in Functioning Within Social Institutions and Communicating (compared to 1986 and 1987), but indicated relative weaknesses in Using the Arts, Solving Problems, and Clarifying Values (ACT, 1989).
Junior scores were compared to sophomore scores at 4-year institutions. Eastern students scored at the 39th percentile in comparison to this group, while Subscale Scores ranged from the 30th to 49th percentile. Among rising juniors, comparative strengths were seen in Functioning Within Social Institutions and Communicating, and scores for Clarifying Values were relatively weak. It should be noted that for both groups, ACT reports that Eastern students compared favorably or more than favorably for Total Scores and Subscale Scores. Among those students in the longitudinal analysis (those with freshman and junior scores), gain scores ranged from - 25 to 30 points. Of those students with valid scores, a total mean gain of 5.9 was measured in comparison to 7.6 for the national sample. Students at Eastern in the longitudinal sample did evidence a gain in their scores, however, not as much as schools in the national group.
One thousand one hundred fifty-one students participated in the ACT COMP in 1990 (651 freshmen, 510 juniors). The Total Score for freshmen was 167.1 and 177.6 for juniors. Eastern’s scores were favorable when compared to 2-year institutions, yet more than 5 points lower when compared to 4-year institutions in the normative samples. The score for the juniors was also lower by more than 5 points when compared to the national reference group. The gain score among students who participated in the COMP as both freshmen and juniors in 1990 was at 7.7; this represented a significant increase over 1989. The text of the report from ACT for the 1990 cohort was not available. However, the most salient points which can be observed are that the gain scores evidenced by the junior cohort were greater than in previous years. Mean scores for freshmen and juniors on Total and Subscale Scores remain relatively consistent. Through the first five years of the COMP administration at Eastern New Mexico University, the data clearly indicates that Eastern students are scoring lower than reference groups at 4-year schools and that the size of the gain scores appears to be growing (OPAS, 1991).
In 1991, 1230 students participated in the COMP; 704 of these were freshmen and 512 were rising juniors. In 1991, freshman students were again compared to students at 4-year institutions and the hypothetical “average freshman” in the sample, excluding questionable scores, was at the 48th percentile and the Total Score and Subscale Scores ranged from the 39th to the 65th percentile. Eastern students compared favorably to the national sample in ACT enhanced scores with a mean of 20.1 versus 19.6 for the national reference group. When compared to like institutions, Eastern compared most favorably in Functioning Within Social Institutions, Using Science, and Communicating, but less favorably in Using the Arts, Solving Problems, and Clarifying Values.
Juniors in the sample did not compare as favorably as freshmen. The hypothetical “average rising junior” within the sample scored at the 35th percentile on the Total Score with Subscale Scores ranging from the 35th percentile to the 45th percentile. ACT compared the results of Eastern students to students at seven other similar institutions, both in 1989 and 1991; and reported “in view of the results, both your current sample and your 1988-89 aggregate sample of rising juniors compared poorly with this reference group of second semester sophomore samples” (pp. 12, ACT, 1992). With regard to gain scores, Total Scores ranged from -56 to +62 points. The reported gain scores, when comparing freshman means to junior means, was again slightly above seven, which was below the national average.
In 1992, 1245 students completed the COMP, and of these, 665 were freshmen and 552 were juniors (in all cases, sums less than the total were due to students other than freshmen or juniors participating in the COMP). For frustrating and unanswerable reasons, the results from 1992 were again compared to 2-year institutions. When compared to this normative group, Eastern students scored at the 70th percentile with Subscale Scores ranging from the 64th to the 70th percentile. ACT reports that Eastern freshmen show strengths in all COMP areas. When compared to the 2-year cohort, the ACT scores of incoming students were comparable.
The junior cohort in 1992 was compared to seniors at 4-year institutions. When compared to seniors, our juniors scored at the 36th percentile on the Total Scores with Subscale Scores ranging from the 39th to 61st percentile. When compared to sophomores, our juniors compared in a ranking from like institutions, and Eastern students scored at the top. With regard to gain scores, the Total Scores range from -49 to +66. The estimated mean gain of 13.1 raw score points calculated by ACT was an amount of growth substantially above the average for the previous years.
In comparing the five years for which gain scores were available, ACT made the following observations:
The last year that the COMP was used by ENMU for the analysis of its general education curriculum was in 1992. Ironically, it was also in 1992 that substantial changes and improvements began to be recognized in the COMP. However, conclusively throughout, Eastern students seemed to score lower. This most likely is attributed to admission policy and student preparation. Reasons for eliminating the COMP exam included difficulty in interpretation of Subscales and Total Scores as being relevant to the general education curriculum, and difficulty in analysis of various cohorts.
Only small attempts were made by the OPAS in analyzing the reams of information that was made available by ACT. It most likely could be observed that in the beginning of Eastern’s assessment efforts, the intentions were either to show gain scores and/or to demonstrate comparison to national normative groups. As the outcomes assessment endeavors here at Eastern have matured, it has become more important to demonstrate how students have learned and how that can be tied back to the curriculum. The COMP in particular provided difficulties in tying student results back to student learning. In addition, personnel and resources were lacking to successfully compare students' transcripts and incoming variables to the eventual results.
So, finally, it can be observed that gain scores were measured by the COMP. We further can observe that Eastern students apparently had consistent trouble with the scales of Functioning Within Social Institutions and Communicating. In comparison, Eastern students scored relatively higher in the remaining areas. It appears that the replacement of the COMP with the Collegiate Assessment of Academic Proficiency (CAAP), also produced by ACT, was probably a prudent decision by the Assessment Committee in 1992. However, it appears that a good decision was also made by the General Education Committee to include in their outcomes assessment plan qualitative measures obtained through focus group, surveys, and exit interviews. It is the opinion of this author that perhaps the best measures of learning will occur when we can consistently articulate to students what it is that they should learn, and ask them to demonstrate that the learning has occurred.
Lastly, in considering the economic and personal resources outlayed for the administration of the COMP for a 7-year period, it should be observed that the institution did learn and mature through the process, and that, if required, the institution could have demonstrated to external constituencies its attempts to measure student learning and to improve the academic programs.
NOTE: Interpretation of all COMP results are spurious due to the number of results which were excluded from analysis ranging up to 40 percent for some cohorts. Results were eliminated because of apparent randomness in results and/or low scores or high scores which indicated that some efforts were sabotaged by students. Complete and full analysis of all COMP results have been conducted by ACT as well as the Office of Planning and Analysis Services. These are all on file in the Assessment Resource Office for individuals interested in more detail.
| Descriptions of COMP Scale Scores |
|---|
|
