The state’s testing vendor mis-scored 20 percent of English exams selected by the district for rescoring, and mistakes could cost LISD up to $51,475.
High school students in Texas must pass five required STAAR end-of-course exams in order to graduate. Not only are the standardized tests high-stakes for the kids, but they drive school ratings. For English I and II, those exams include multiple-choice and short-answer questions, and essays. The district believes incorrect scoring is responsible for some students failing the exams that were administered in December.
It was the scores on the short-answer portion of the tests that got the attention of Lewisville ISD administrators. According to a memo included with the LISD Board of Trustees meeting packet, the district became concerned when scores were received back from Educational Testing Service (ETS), the state’s new vendor for grading the tests. There was a significant increase in scores of zero on the short-answer portion.
In an April 13 letter from LISD Superintendent Dr. Kevin Rogers to Texas Commissioner of Education Mike Morath, Rogers said that 76 students who took the English I exam got a zero on the short answer section, yet scored 6, 7, or 8 on the composition section. The same was true for 72 students in English II.
Rogers wrote that 17 percent of the English I test-takers received a score of zero on short-answer when taking the test for the first time. That number rose to 47 percent for repeat test-takers.
The district reported its concerns to TEA, and received an unsatisfactory response, according to Rogers’ letter. “…Their response did nothing to alleviate our level of uneasiness with the reported scores,” wrote Rogers.
“In short, we were told the questions were difficult, and that the scores were accurate. This response was unsatisfactory to us, so the district made the decision to pay a fee to request rescoring of more than 150 documents.”
Administrators picked a set of 162 student tests where the short-answer score was significantly different from the essay score, and requested that ETS rescore them.
ETS charges $25 per test for re-scoring, but waives that fee only if the test score changes.
Of the 77 English I tests submitted for rescoring, 10 grades were changed. Fully 13 percent of the exams chosen for rescoring in English I had grading errors.
It gets worse.
Of the 85 English II tests submitted for rescoring, 23 scores increased, meaning that 27 percent of those scores had been incorrect.
The rescoring resulted in three additional students passing the exam, and six earning advanced designation.
Even though ETS graders had incorrectly scored 20% of the tests that LISD had chosen, they billed the district for $3,225, the full cost of rescoring for the tests that were not scored in error.
Rogers’ letter requests that the Texas Education Agency provide the district with all of the students’ written responses to the December English I and English II EOC assessments. It also asks TEA to arrange to regrade all of the English I and II written portions, and conduct a broader investigation.
Because of the lack of confidence in the scores, the district wants to have ETS rescore anywhere from 912 to 1,930 of the tests, at a cost ranging from $22,800 to $48,250 if LISD is unable to get the help it seeks from TEA. With copies of the student answers, LISD might be able to narrow its rescore request to those most likely to be incorrect.
The funding for rescoring is not currently available in the district’s testing budget, so the administration is asking the board of trustees to approve the expenditure at Monday night’s meeting.
Debbie Ratcliffe, a spokeswoman for TEA said Monday that the agency’s director of assessments said “there was an issue with interpretation of the scoring rubric.”
According to the website “Data Interaction for Texas Student Assessments”, 24% of Lewisville ISD students who took the English I STAAR EOC exam failed it. Statewide, 62% of students failed that exam. For English II, 22% of Lewisville ISD students failed, compared to 63% statewide.
Those statewide figures reflect the fact that most test takers in the fall are retaking the test after failing it in the spring. Lewisville ISD is one of the few districts that administers the EOC in December, since its high school schedules have students completing the course in one semester instead of an entire school year.
This is the first school year for ETS to score the STAAR tests for Texas. The vendor replaced Pearson for the multi-million dollar contract this year. In March, the company came under scrutiny when it was learned that the company’s servers lost student answers on over 14,000 of its computer-based STAAR tests.
LISD School Board member Kristi Hassett, who also serves on the board of directors for Texans Advocating for Meaningful Student Assessments (TAMSA) – a.k.a. “Mothers Against Drunk Testing”— said that she thought the issue was discovered initially because a student who took the test was the child of an employee in the district’s assessment department. She said the anomaly caught the employee’s eye.
“We are one of the only districts in the state who test in December,” noted Hassett. “Wait until we get the spring results; I bet the rest of the state could also have issues.”
Hassett said the bad scoring was not the only issue the district had with ETS. “We also received the wrong Braille tests right before they administered them,” said Hassett. “ETS had to overnight the correct tests.”
“I know the change was going to have issues— things like this rarely go smoothly,” she said. “However, they have had too many issues; it’s unacceptable.”
TEA was not able to provide many answers Monday afternoon. “We are working with the test contractor, ETS, to get a full explanation about what caused the unusually high number of zero scores on the December tests,” said Ratcliffe.
“This unusual situation only occurred on the short answer portion of the test and not on the essay portion. “
For its part, ETS would not provide The Lewisville Texan Journal with a response or make a representative available for an interview. ETS spokesman Tom Ewing deferred questions to the TEA.
We sent LISD a set of questions about the issue, but had not heard back by the end of the school day.
The English I and II end-of-course exams for the spring semester were given on Mar. 29 and 31 respectively— about 45 days prior to the end of the semester for Lewisville ISD. Those results are expected to arrive at the district by June 3, with students and parents being notified of results by June 10. The school year ends June 2.
Update #1 – Monday, 4/18/2016 – 11:15 p.m.
Upon a motion by outgoing trustee Brenda Latham to “approve these ridiculous costs”, LISD board members unanimously passed an authorization to spend up to $48,250 more to have additional tests regraded.
“I’m flat ticked off!” said Latham, who became visibly emotional while giving her thoughts on the problem.
“It’s totally unacceptable,” she said of the situation. “We have to at some point stand up and say ‘sorry, we’re not playing this game anymore.’”
Trustee Angie Cox thanked the district administration for their stance, and efforts to fix the problem.
Trustee Tracy Miller echoed Latham and Cox’s remarks and said he agreed with Dr. Rogers’ aggressive posture. “We need help with our legislature; we can no longer think that one party has the best consideration of our kids.”
Miller pointed out that the cost of regrading the tests is about the cost of a teacher salary.
Hassett said she expected that a new vendor would have hiccups initially, but that “ETS has found new ways in which to subject our students to errors.” She asked if staff could look into what the graders qualifications have to be. Director of assessments Sarah Fitzhugh said that she knows ETS hires graders from across the US, and has its own training process, but she was not sure what credentials or education that ETS required.
Fitzhugh also provided more information about the set of initial exams that were sent to ETS for regrading. She said that the group included all graduating seniors who stood to benefit from an increased grade.
Board president Trisha Sheffield said the situation infuriated her. “If even one test was misscored, it throws into question the entire system,” she said.
Assistant superintendent Dr. Beth Brockman assured the board that the administration had not given up on the idea of getting someone else (TEA or ETS) to take on the cost of regrading the tests.
Superintendent Dr. Rogers thanked the board for their support of efforts to end the testing in general. He said that not only are the “distorted tests” affecting students, but that they are used to judge campuses.
Rogers and Brockman were of the opinion that the time to act in getting the rescoring done is now, while it can do some good for the kids. They seemed determined to find a way to recover the cost, even though they’ve not gotten much traction on that in their discussions with TEA so far.
Update – Tuesday, 4/19/2016 – 5:40 p.m.:
From Lewisville ISD’s communications office:
Open in new window
It is important to note, Lewisville ISD continues to work with the Texas Education Agency to address our grave reservations about the accuracy and validity of English I and English II End of Course short answer scoring. Lewisville Superintendent of Schools Dr. Kevin Rogers has communicated directly with Commissioner of Education Mike Morath regarding the district’s concerns, and we hope to receive a response in the next week.
Lewisville ISD is unique in that the December administration of these EOCs includes students who are taking the test for the first time. Across the state, most students who take the exam in December are taking the exam for the second time. The fact that students who are enrolled in advanced courses were among those receiving zeros on the tests brought to light the obvious scoring issues.
We remain optimistic the agency will do what’s right by our students and for Texas public school students across the state by fully investigating the scoring discrepancies and ensuring students receive the scores they rightfully earn on the state assessments.
Beyond our specific concerns surrounding English I and II EOC short answer scoring, we have broader reservations about what this means for the integrity of the testing system in general, especially in light of the way campuses and districts are rated based solely on the results of the scores on these tests.
We believe that one student negatively impacted by flawed grading of a faulty state-mandated assessment required for graduation, is one too many. LISD first brought these concerns to the attention of TEA in late January. The agency sent the following response on February 9:
TEA and ETS have carefully reviewed the processes used for scoring the STAAR English I EOC assessment. Scoring outcomes across the December 2015 and prior administrations were carefully analyzed both in terms of the differences in test content and the differences in the characteristics of the students taking the assessments. In particular, the December 2015 English I EOC results were compared to the July 2014 English I EOC results, as the same test was used for these two administrations. These analyses indicate that the differences in scores may be attributed to differences in the characteristics of the students participating in the December 2015 administration and earlier administrations. Significant changes in testing requirements for graduation (e.g., students who are no longer testing because of Senate Bill 149 provisions) and the significant increase in students retesting for the third and fourth time have had an effect on the distribution of scores for the December 2015 administration.
LISD contacted TEA and expressed concerns over the integrity of grading. We were told to trust the process and submit scores to retest. LISD submitted 162 students’ tests for rescoring. On March 18th, the results of the rescore arrived. Three students, who had previously not met standard, met standard. Six students moved from meeting standard to advanced. In total, 33 student’s scores changed as a result of the rescoring.
Based on the egregious nature of these scoring irregularities, the district is now preparing to send more tests for rescoring, at a cost of nearly $50,000. In light of the obvious issues with scoring, it is Lewisville ISD’s hope in a show of good faith, TEA and ETS will waive the rescoring fees and demonstrate a commitment to providing outstanding service to our district as we work through these scoring errors.
Although we strongly believe our district should not shoulder the financial burden to ensure the scoring of these tests is accurate, we will do what we must to advocate for our students so issues beyond their control do not unfairly impact them.
Update – 4/20/2016 – 2:10 p.m.:
Tom Ewing, spokesman for ETS, clarified his company’s refusal to comment for our story:
“As a vendor to the State of Texas and to the TEA, ETS is contractually required to refer media questions to the TEA. We are not stonewalling anyone, especially the media, but rather following the specific guidelines of the STAAR contract. ”
TEA has still not responded to all of our questions either. Spokesperson Debbie Ratcliffe said that key testing staff (who could answer the questions) were off-site staffing the meeting.
The Lewisville Texan Journal will attempt to get a copy of that contract from TEA.