|
|||||||
|
||||||||
District 211 PSAE AppealHigh School District 211 Superintendent Roger Thornton and Superintendent-Elect Nancy Robb delivered the District's appeal of its Prairie State Achievement Examination scores to Illinois State Superintendent Chris Koch on Tuesday, September 30, 2008. On October 2, 2008, High School District 211 amended its September 30, 2008 appeal of its PSAE results with the addition of If you wish to review exhibit materials, please visit the G. A. McElroy Administration Center or the Principal's office at any District 211 high school.
Letter to Illinois State Board of Education Amended letter to ISBE and Superintendent Koch Amended Exhibit List/Description APPEAL OF PSAE RESULTSTownship High School District 211
Historical Perspective of the Appeal Township High School District 211’s history includes the National Blue Ribbon of Excellence Award for each of its five (5) high schools. Palatine High School received the award on two occasions. A system-driven curriculum that includes end-of-course examinations has been present in the District for many years. The District’s feeder districts, Schaumburg District 54 and Palatine District 15, are noted for educational excellence and each has achieved awards of excellence in their own right. Beginning in 2004, Township High School District 211 began significant initiatives to increase student achievement. The goal was not focused solely on achieving Adequate Yearly Progress. Rather, acceleration of learning for students who were not succeeding through increased rigor in required core subjects and an increased mathematics requirement were coupled with an Incoming Freshman Academy summer class (for students who were not projected to meet PSAE standards based on their EXPLORE score). The State Board of Education may recall our request to permit seniors, who did not meet standards on the PSAE as juniors, to take the PSAE in their senior year. We were then willing, and we remain willing, to do all possible to make certain that all students graduate ready for higher education and/or the world of work. In 2007, our perspective of PSAE as a meaningful and reliable indicator of student achievement changed. The catalyst of our changed view was the PSAE scores for our District and the entire State of Illinois. Through no explainable reason, scores for the graduating class of 2008 (Spring 2007 PSAE test takers) dropped by 4%. District 211, along with several other school districts, met with ISBE staff and ACT representatives. In December 2007, District 211 leaders presented a request to the Illinois State Board of Education that a third-party review of the 2007 scores be completed and that District 211 representatives be permitted to work with the reviewers to express the concerns of high school districts. Though a commitment was made in the public Board Meeting to honor our request and to involve us in the discussions, no such involvement was permitted. The third-party review, completed by HumRRO in February 2008, confirmed our concerns but was unable to resolve the shortcomings inherent in the PSAE as the measurement instrument regarding a reliable indicator of student achievement relative to the State’s defined standards. (See exhibit 1) Page v of the report included the following statement: “It is possible that some methodological or random error might have contributed significantly to the apparent gain from 2004 to 2005. Then, when the 2006 administration is equated to 2005 and the 2007 administration to 2004 using common item equating, that same methodological or random equating error would result in an apparent decrease in mean scores. (emphasis added) HumRRO has no means of investigating the likelihood of this possibility from the data provided, but the overall data patterns indicate that this is a plausible explanation for the decline.”
Basis of the Appeal We formally make our appeal based on the following statistical and substantive reasons, consistent with the defined appeals process:
Examining the Differences and the Different Impacts of Reported Scores of Criterion-Referenced vs Norm-Referenced Examinations The HumRRO report corroborates much of the basis of our appeal. Please note the bold-type phrase in the previously cited quotation from the report where the writers of this appeal added emphasis. The referenced phrase is “decrease in mean scores.” Therein lays the issue: the ACT and WorkKeys, while set forth by the State of Illinois and the State Board of Education, as the means to assess student achievement of Illinois standards for 11th-graders, uses student score norms from significantly different segments of students across the United States to establish mean scores for the purposes of reporting student performance. It is critical that ISBE Board Members and staff pause to reflect on this issue.
Those who observe student accountability assessments across the United States have analyzed Illinois’ student assessments. The National Center for Education Statistics (see Exhibit 2) lists Illinois’ ISAT as a “criterion-referenced” assessment type. They list the Prairie State Achievement Examination as a “hybrid” assessment type. It is important to note that they describe West Virginia’s tests, the ACT Plan and the ACT Explore, as “norm-referenced” assessment type. Illinois recognizes the replicated assessment type connection among the ACT Explore, ACT Plan and the ACT test by its funding of all three examinations for the purpose of enabling high school students to predict their ACT score through the ACT Explore and the ACT Plan experience in the eight or ninth grade and the 10th grade, respectively. EXPLORE and PLAN are norm-referenced tests and can only be a predictor of ACT scores if ACT is also norm-referenced. The National Center for Education Statistics further defines the two types of tests with the following delineations:
A separate publication by FairTest (See Exhibit 3), the National Center for Fair and Open Testing, includes the following:
The ACT student information publication clearly describes the ACT as a norm-referenced examination. “THE 1973-74 ACT ASSESSMENT STUDENT’S BOOKLET” (See exhibit 4) states:
The more recent “2007/2008 USING YOUR ACT RESULTS” publication from ACT (See exhibit 5) includes the following statements: On page 5:
The contrasting impact of criterion-referenced tests and norm-referenced tests can be readily observed through Illinois ISAT and PSAE data. The Chicago Tribune noted the stark differences in its Thursday, September 18, 2008, Metro Section 2 article (See exhibit 6).
Prior ISBE Reviews of PSAE The McCabe, Miller, Lange presentation to the Illinois State Board of Education (See exhibit 7) entitled “Debunking the Myths of the PSAE” noted the differences as well. Though appearing to prove that the PSAE measures Illinois state standards, the report unintentionally proves that the PSAE is not an appropriate instrument for the reporting of student achievement of state standards. We quote from:
We accept the good intent of the authors of the report to give substantial support to a state mandate (the PSAE) that was then and remains under attack. However, the reporters and the report missed a very important environmental change. The assessment environment changed from 1999 when PSAE became a reality in Illinois as a compromise among legislators, the business community, and educators. In 1999, a norm-referenced test of national significance deemed to predict university readiness coupled with an examination that theoretically assessed workplace readiness made sense, at least to those making the decisions to put PSAE in place. The test’s shortcomings were evident but seemingly ignored, even then. Only a portion of students in most states take the ACT. Illinois, Colorado and Michigan, at 98% and 100% and 100%, respectively, led the way in 2008 in the percentage of high school graduates tested. Maine tested 9%, New Jersey tested 13%, and California tested 17% as did Massachusetts. Nationally, 43 % of seniors took the ACT at least once.
Summarizing the Impact on Illinois High School Students and Schools: Why Does it Matter? What is the impact of the variance described above between criterion-referenced results vs norm-referenced results? If ACT scores were reported on a criterion-reference basis, there would be no impact. The report would demonstrate which students achieved which standards and the report would reflect progress or lack of same. The same is true with WorkKeys. Reported in its norm-referenced basis, the “Average ACT Scores by State: 2008 ACT-Tested Graduates” (See exhibit 8) report compares all Illinois students to a varying depth of creaming across the country. When compared to Colorado and Michigan, Illinois’s composite score of 20.7 betters their composite scores of 20.5 and 19.6, respectively. But, none of the scores have meaning in terms of how many Illinois students met Illinois standards. The scores merely compare the entire range of graduates in three states, including Illinois, with sometimes very limited and likely highest performing graduates from other states. “ACT News” (See Exhibit 9) in its Facts about the ACT, confirms that comparison of students as the score reporting basis when it answers the question:
Combining norm-referenced ACT reports with norm-referenced WorkKeys reports does not result in a usable or valid PSAE report for AYP purposes. Thus, HumRRO, ISBE’s selected third-party reviewer, found that the 2007 issues with PSAE scores, even though the number of students scoring in the top three levels of WorkKeys declined by 10%, were not explainable though described as “ a dramatic and anomalous change.” Had the results been reported in a criterion-referenced basis as is the case with ISAT, an item analysis would have pointed to a solution. Equally troubling is the fact that annually Illinois students achieve high scores on the ACT, in a norm-referenced report, only to learn that they did not pass the PSAE. For example, 440 District 211 students scored a 19 or higher on the ACT during the 2007 PSAE administration (See exhibit 10), but did not pass the PSAE. Some of those students received scores in the mid-to-high 20s. In the 2008 administration (See exhibits 11 and 12), similar injustices occurred when 438 District 211 students scored a 19 or higher but failed the PSAE. Annually, approximately 90% of incoming District 211 freshman have met ISAT standards. The criteria of ISAT criterion-referenced examination are clear and the results report the percentage of students who have achieved standards. The clear and constant conflict and contrast between and among ISAT scores, a criterion-referenced test, and EXPLORE scores, a norm-referenced test, is evidenced in the scores of District 211 incoming freshmen of the graduating Class of 2011 (See exhibit 13). ISAT reports student performance in terms of mastery of standards. EXPLORE, a precursor to the ACT, reports student performance in terms of how students compare to other students. The historical trend of EXPLORE, PLAN and ACT scores among District 211 students demonstrates acceptable progress through the 11th grade (See exhibit 14). PSAE results are reported in a norm-referenced format, and while claims are made that Illinois standards are measured, no separate Illinois ACT exists. ACT results are reported as they always have been by comparing students with other students. That process seemingly met the goals of state policy-makers in 1999. Cemented in place by the passage of “No Child Left Behind” in 2001, and with retention of the examination held in place through the threat of the withholding of federal funds from the State of Illinois, it is imperative that the Illinois State Board of Education acknowledge and permit its staff to acknowledge that PSAE does not report Illinois students’ achievement of Illinois standards. Rather, in a general sense, the PSAE, through the ACT and WorkKeys, reports the performance of each and all Illinois students compared with the performance of an array of different proportions of creamed students across the United States and then reports the results in a “how you compare to others” format. Generally, by definition, no more than one-half of Illinois students (given that all Illinois students take the PSAE while far less than half of the students in most states take the exam) should be expected to be above the norm. But, PSAE scores are also irrelevant. Illinois students, many of whom fail the PSAE as demonstrated earlier with ACT scores of 19 or higher are prepared to gain entry and do gain entry to Illinois public universities (See exhibit 15). In fact, many of those who fail the PSAE achieve ACT scores that qualify them for entry into prestigious universities across the United States. It is our position that an exam that repeatedly, on an annual basis, fails students who, as part of the exam, test high enough to gain entry into highly ranked universities, and, yet, are deemed failing by the PSAE is just plain wrong. The test does not specifically test achievement of Illinois standards. The reports do not reflect to what degree a student or all students achieved Illinois standards. And, the reported scores hold no value for either universities or employers. Finally, we suggest that the PSAE is discriminatory toward students of poverty and disadvantage simply because the examination is reported in a norm-referenced basis. It is possible to accelerate learning for disadvantaged students. We have evidence of that success. Acceleration, added rigor, advanced courses, more time in school, mentoring caring and dedicated teachers and school leaders; all have an impact. However, at the same time disadvantaged students are progressing, so are those whose privileged life circumstances gave them a head start and that advantaged circumstance permits them the continued and ongoing opportunity to excel. In the end, disadvantaged students, though often learning at high levels and achieving standards, remain below the mean in a norm-referenced comparison, and thus fail the PSAE examination, handicapped solely because of their starting place.
Move Illinois High Schools to a Testing Model Similar to ISAT Under ISAT and similar criterion-referenced examinations, the results are not reported as a comparison to what others are achieving. Rather, the focus is how each student and the students in each school achieve in regard to standards. Under PSAE, the focus is how each student and each school achieves compared to all other who took the tests with no direct link to Illinois standards. The PSAE, as a norm-referenced examination, appears to tangentially measure student achievement on Illinois learning standards with results reported in a United States-wide-norm-referenced format. The results give no clue as to which Illinois students have met specific Illinois standards. Most students taking the ACT throughout the United States are not accountable for Illinois standards because they are not residents of Illinois and they do not attend Illinois schools. For school accountability purposes, this difference is the difference between success and failure for all Illinois high schools. With norm-referenced results, achieving 100% proficiency for all students is impossible. Achieving 90% or 80% or 70% proficiency statewide is impossible by definition. The Illinois State Board of Education must accept the reality of the definition of norm-referenced results: half of the students will be above the mean and half of the students will be below the mean. Illinois School Accountability requirements and No Child Left Behind requirements cannot be met in a norm-referenced reporting system. Further, it is our observation and position that the Illinois State Board of Education and its own staff are fully aware of the problems with the PSAE, but are unable, due to the financial penalties that would be imposed on the State of Illinois by the federal government, to enact the needed changes. In the meantime, high schools across Illinois are forced to allocate significant teacher and administrator time away from the classroom and the school to create school improvement plans solely on the basis of the school’s standing as prompted by its PSAE scores. And, for students, teachers, school leaders, and communities that care deeply about their schools, we urge the Illinois State Board of Education to move with haste to initiate a review of our appeal and to enact appropriate remedies in the immediate future. As a State, we can do better by our students and our schools. We must.
Requested Action We petition the Illinois State Board of Education to:
Amended Text Explanation (amended 10-2-08)
Township High School District Amended Appeal of PSAE Scores
Additional Amended Text of October 2, 2008. Exhibit 16 The issues of concern regarding PSAE scores are exemplified in the 2002 study of the relationship of ISAT and PSAE scores (Exhibit 16). The anticipated cross-validation coefficients of the tests were .76 for reading and .84 for mathematics. In actual performance, the actual scores on the PSAE compared to the predicted PSAE scores based on ISAT score predictions, had a reliability factor of 62% in reading and 70% in mathematics. Since, as we all know, a coefficient of 1 signifies alignment, Illinois has known for six years that the alignment of ISAT and PSAE earns a score that would merit a grade of “F” in most high school mathematics classes. On what basis is this acceptable to the State Board of Education as it grades its high schools’ performance? Results from the PSAE serve as the basis for schools to be required to remove approximately thirty teachers from our classrooms for multiple days to prepare school improvement plans with the threat of annually increasing penalties made nearly certain by the required ascending scale of proficiency increasing 7.5% per school year. Given the 2002 study, District 211 augments its appeal of the scores on the basis of unacceptable correlation between ISAT and PSAE. Exhibit 17 The August 26, 2008, communication from the State Superintendent set forth clear goals as determined by the State Board of Education. The first goal relates directly to PSAE: “Every student will demonstrate academic achievement and be prepared for success after high school.” Though adequate time has not elapsed for the State Board of Education to achieve this goal, progress toward the goal can be demonstrated by hearing and seriously considering the District 211 appeal of PSAE scores. Several factors are central to that hearing and consideration:
Exhibit 18 The State Superintendent’s comments regarding the scoring of PSAE in the September 30, 2008, Weekly Message are troubling. The District 211 PSAE appeal was delivered to ISBE at approximately 9:30 AM on the 30th. Perhaps pure coincidence, both in timing and substance, the comments reflect the position taken by certain leaders of ISBE in our previous meetings, rather than the position and comments of Dr. Koch and the State Board of Education. Regardless, the comments again deflect the focus from the real issue. Consider, please, our request for the following information:
Exhibit 19 This letter from the Illinois State Board of Education dated March 14, 2008, is either inaccurate or it impeaches the assertions made in Exhibit 18. We quote:
Several concerns are present with this exhibit. They are:
Exhibit 20 This exhibit confirms that the 60 items on the ACT all count toward the PSAE score. And, all 33 WorkKey items count. We cite the note at the bottom of the page:
District 211 again asserts that either Exhibit 20 is inaccurate or it impeaches the statements made in Exhibit 18. District 211 amends and affirms its appeal of the PSAE results based on the issues raised in the original filing and in these amended questions and concerns.
|
|
