Tests Lacking in Reliability and Validity

Any discussion of standardized tests should note that K-12 curricula change. They’re flipped in and out, often at a huge cost in time, resources and money, but there are often no control groups against which new curricula are compared. Change is frequently complete and district wide.

Therefore, instead of testing the children’s learning – as they're intended to do – or the learning environment as they should do – perhaps the tests end up testing the effectiveness of the new curricula. Perhaps the math portions test the effectiveness of the entire reform math philosophy. (Hey, now, there’s a thought.)

When 40-60% of the students don’t pass the WASL, how do administrators know the fault isn’t with administration, the curricula or the tests themselves?

In this article, I want to talk about Washington’s standardized tests (the WASL), in terms of statistical reliability and validity. The following contains information from Brickell & Lyon (2003); and from James Dean Brown, University of Hawaii (used with permission).

Statistical Reliability

Statistical reliability is the degree to which a test’s results can be consistently replicated. Every change in procedure and every consequence to the testing environment can affect the level of test reliability. Here are three of my concerns:

1. The WASL has changed over time.

Since its inception, the WASL has evolved, with new questions, new emphases and new scoring techniques. This evolution might well be valid and necessary, but administrators continually compare scores of today with those of 8 or 10 years ago as if they represent trends in the same tests. They are not the same tests.

2. Lowered WASL passing scores.

In April 2008, I began asking the state about drops in cut scores (the point at which a test score is a passing score), trying to determine if the numbers represented the same expectations. (It’s like asking if a size 10 in women’s clothing is the same size in 2008 as it was in 1967, which of course it is not.)

On July 2, after repeated requests and a request for public information, the state confirmed that several cut scores had been lowered in 2004 and 2005. This likely lowered the level of achievement needed to reach a passing score. Again, even if the changes are valid and necessary, they still interrupt the trends.

3. Possibly flawed scoring.

My husband and I found questionable grading on our daughter’s 3rd- and 4th-grade math tests. On some questions requiring a written answer, her math, spelling and sentence structure were correct, her writing was legible, and she answered all that she was supposed to answer. Yet, points were docked. Had we been inclined to do so, we could not appeal the results. (Only scores for the 10th-grade WASL may be appealed.) Therefore, the questionable scores stand.

Statistical Validity

Statistical validity is the degree to which a test measures what it’s designed to measure. Validity can be calculated in several ways, including whether the test matches the testing objectives and whether it correlates with a comparable outside measure. Here are two of my concerns:

1. No comparisons against comparable outside measures.

Perhaps scores that went up did so because the WASL became easier. Perhaps the students aren’t progressing but rather going backward with higher numbers and less knowledge. Without an outside measure to compare against, how would we know? (Other than by noticing, for example, that students are having trouble with basic skills.)

2. Poor alignment with testing objectives.

The math portion includes many exercises in literacy. Students must write short and long answers that use little math and lots of words. When students get answers wrong, how do we know if they didn’t understand the math, didn’t read the question properly, didn’t understand the question, didn’t write legibly, weren’t able to put together a coherent answer, or just ran out of time? No grading marks were made on our daughter’s tests and no explanations were given.

Mixing variables like this can cause unfortunate consequences in the scoring (which can also affect the test’s reliability). Teachers and parents have told me that correct math answers can get fewer points than wrong answers that include expected written words. The math WASL doesn’t appear to be a good indicator of the quality of the math instruction. A math teacher can be brilliant – the best teacher ever – but if the test is in literacy and not math, the math teacher must bow to the English teacher.

The No. 1 thing the WASL appears to be for is to instigate macro machinations at the state level, some flutter of frenzied activity that will eventually filter down to the district level, which will eventually flutter down to the school level, which will have an effect – possibly positive – on a certain subset of the students. Besides being clunky, costly and inefficient, this supposed effort to be publicly accountable is just pretense.

If we are to retain the WASL or some other standardized measure, there must be a stronger, more direct connection between the subject and the test - and between the test and the teacher, parent and student.



Please note: The information in this post is copyrighted. The proper citation is: Rogers, L. (September, 2008). "Tests lacking in reliability and validity." Retrieved (date) from the Betrayed Web site: http://betrayed-whyeducationisfailing.blogspot.com/

WASL is the "floor" of expectations

People have complained that Washington’s standardized tests (the WASL) have suffered “mission creep.” The original intent, they say, was to test students as a way to “assess teachers and other educators and hold them accountable.” But now, the goal is to prove administrator competence to the federal government.

In a March 2008, radio interview, former Spokane Public Schools Superintendent Dr. Brian Benzel said the WASL was always designed to test the students. Benzel was a member of the Commission on Student Learning (which created the WASL). He said a “link to the high school diploma” was always part of the deal, and that the WASL was a “pre-diploma standard.” It was never intended to be a college-readiness or work-readiness indicator, he said. It was always designed to test reading, writing and math skills “at a basic level.”

Karin Short, former curriculum director for Spokane Public Schools, agrees. She told me in January 2007 that the WASL is “just the floor” of expectations. “We want our kids to exceed that,” she said.

Well, that just makes the whole thing worse. Almost half of the students can’t pass the math portion of this “floor” of expectations, and few can pass science. Interestingly, those who do pass the math, reading and writing portions can earn a high-school diploma and a special “Certificate of Academic Achievement.” If I correctly understand Benzel and Short, then, this “Certificate of Academic Achievement” is an award for being able to reach the floor.

The math and science portions of the WASL were to be graduation requirements by now, but consistently disappointing scores across the state inspired the legislature in 2007 to delay that until 2013. Students must keep taking the math test, but to graduate, they can pass it, pass an alternative or just keep taking math courses until they graduate.

In March 2008, Washington Gov. Christine Gregoire signed a bill to reintroduce end-of-course exams and do away with the math portion of the 10th-grade WASL altogether. But the students have to keep taking it. In 2013, the 10th-grade math WASL will become an option, and in 2014, it will be history. Until then, like some incredibly expensive headless chicken that doesn’t know it’s dead, it will keep running around the farmyard bleeding money.

At this time, 10th-grade students in Washington have five separate opportunities to pass the WASL. Those who can’t pass the math portion can apply to show competency through alternative testing. The alternatives and regulations have been tinkered with and modified over time, driving some students halfway to distraction. One of the alternatives is called a WASL-Grades comparison.

Are you curious about the WASL-Grades comparison? So was I. That option works like this (let’s see if I can write it with a straight face): A student who has at least a 3.2 GPA can compare class grades against a group of fellow students who passed the WASL, and if the student’s grades are above the mean of the grades of the rest of the group, then that can count as an alternative to the WASL.

(Ha, ha, ha, ha… No, sorry. I couldn’t do it.) They can’t be serious. One starts to wonder if they just don’t want students to achieve.

It’s too bad these alternative ways don’t include learning core academic content with a more direct approach. It’s too bad, but it’s understandable because that would admit to the world that the current approach to math isn’t working well. And that, my friends, administrators must never do lest they all turn into pillars of salt.



Please note: The information in this post is copyrighted. The proper citation is:
Rogers, L. (September, 2008). "WASL is the 'floor' of expectations." Retrieved (date) from the Betrayed Web site: http://betrayed-whyeducationisfailing.blogspot.com/

Washington State superintendent "spins" the data on education

When people “gloss over” bad news and “spin the data,” are they being deceitful? Is there a point at which “data spinning” turns into a “lie”?

In July, The Spokesman-Review noted the “tendency” of Washington’s Superintendent of Public Instruction Terry Bergeson to “gloss over bad news" and "spin" the data. This comment was tucked inside its endorsement of her re-election bid.

As a voter, you’re probably wondering about the details of that “glossing over and spinning.” Well, I’m up for the conversation. I’ll show you what Bergeson says versus what the data say, and you can decide for yourself. It’s democracy in action.

In February 2007, Washington’s Office of Superintendent of Public Instruction (OSPI) said the state earned “high marks” from the U.S. Chamber of Commerce (CoC) for “very strong” academics, as indicated on the National Assessment of Educational Progress (NAEP).
Data: You’re expecting excellent skills, low dropout rates and low rates of remediation? Sorry, but no. The specific CoC data referred to Washington’s 8th graders having scored 8 percentage points above the national average for achieving “proficiency” or better in mathematics (in 2005, the national average was 29%), and 27% of 8th-grade black students having achieved “proficiency” or better in reading (in 2007, the national average was just 12%).

In February 2007, OSPI said Washington scored among the top four states for boosting the number of students scoring a 3 or higher on Advanced Placement exams. Bergeson said Washington was “among the national leaders in creating a public education system that strives for excellence and equity.”
Data: By OSPI’s own accounting, the percentage of students in AP classes who scored a “3” or higher on the exams actually dropped from 2001 to 2006. What puts Washington in the top four was its increase from 2000 to 2006 in the percentage of students in the entire Class of 2006 who took an AP test and who scored a 3 or higher. In 2006, however, the Class of 2006 ranked behind at least 15 other states.

In March 2007, Bergeson said NAEP results showed black 8th-graders “going from 19th place to second.”
Data: The gains shown on the 2005 NAEP were lost in 2007. Additionally, when OSPI said 36% of black 4th graders and 26% of black 8th graders were “proficient” in math, NAEP said only 17% of black 4th graders and 16% of black 8th graders were proficient. (Similar gaps between OSPI and NAEP were found for all 4th and 8th graders in reading, for 4th- and 8th-grade Hispanic students in math, and for low-income 4th graders in reading.)

In March 2007, Bergeson said NAEP scores put Washington at “fourth in the nation.” Additionally, math achievement had supposedly “tripled,” reading had “doubled,” and Washington had the highest average SAT scores in America for four years straight. “No matter where you look, people across the country are saying ‘What is Washington doing?’” Bergeson said. “We have a revolution happening in our schools, but you don't hear the positive stuff. The yammerers out there complaining about what's not happening rule the day.”
Data: I’ve looked through NAEP scores from 2003, 2005 and 2007. I have no idea of how or when Washington was 4th in the nation. What does it mean to say that math achievement tripled when 40-60% of students can’t pass the WASL and up to half of all of the state's high-school graduates need remediation in math? Washington’s SAT scores aren’t the highest in the nation; they’re the highest among those states “in which more than half of the eligible students” took the test. And I am not a “yammerer.”

In her 2007 address, Bergeson said: “Our math and science standards and our curriculum and teaching approaches have brought us to new levels of excellence.”
Facts: Pass rates for the math and science WASL are consistently dismal. In 2007, an outside review found the math standards to be inadequate; they were rewritten at a cost of more than $1.65 million. The science standards are being reviewed. Legislators voted to not make the 10th-grade math WASL a graduation requirement; it will eventually be kicked out in favor of end-of-course tests.

In April 2008, Bergeson was “very excited” about 8th-grade writing scores on the 2007 NAEP: “Washington’s young writers are outpacing the nation. These students are not only prepared to meet the rigor of a high school curriculum, they are prepared to communicate well no matter what career they pursue.”
Data: Washington’s 8th-graders scored an average of 158, in the middle of the NAEP scale of 0-300. As a group, they scored at a “basic level of ability,” which is considered to be a “partial mastery of the skills needed for proficient work.”

In June 2008, Bergeson “celebrated” 91.4% of students in the Class of 2008 achieving the “new, more rigorous” graduation requirement: “I truly believe when we stop fighting about this and say we’re doing it, we’re capable of doing it.”
Data: The “new, more rigorous” requirement was solely in reading and writing and included WASL alternatives. The figure is cumulative, gathering data over several attempts. It doesn’t include students who dropped out or who fell way behind in credits.

In August 2008, OSPI said that for the fifth straight year, Washington students scored “far above” the national average on the ACT. Bergeson said, “It shows our students are more college ready than their peers around the nation.”
Data: Washington’s composite ACT score of 23.1 constitutes just 64.17% of the total possible score of 36. In terms of academic achievement, that’s a D grade.

There’s more to tell you, but this gives you the general idea. What say you? Are you feeling deceived?

Please note: The information in this post is copyrighted. The proper citation is: Rogers, L. (September, 2008). "Washington State superintendent 'spins' the data on education." Retrieved (date) from the Betrayed Web site: http://betrayed-whyeducationisfailing.blogspot.com/

Washington's Math Standards Failing the Students

Educators talk constantly about the “standards.” You can barely toss an adjective in a school hallway without hearing about them. Learning standards are supposed to drive the curricula and the tests. If the standards aren’t effective or don’t make sense, nothing else will be effective or make sense.

(So it’s best to get them right the first time.)

Logically, K-12 learning standards should be structured so that by graduation, college is a conceivable option for all capable students. Standards should be clear, concrete, reasonable, rigorous, achievable and measurable.

But Washington’s math standards were based on reform philosophy. For years, students have labored ineffectively under reform curricula. Despite heavy long-term criticism of reform curricula and the recent standards revision, reform hasn’t gone away. I can tell you why.

The development of Washington’s first math standards was guided by several publications, including three from the National Council of Teachers of Mathematics. When you know that “reform” curricula typically cite the NCTM Standards as their inspiration, you understand why Washington’s standards and curricula tended to be “reform” in nature.

In 2007, legislators questioned the effectiveness of the standards and ordered an outside review. Strategic Teaching’s subsequent assessment was painfully frank:

“There is insufficient emphasis on core mathematical content. Some math should be taught earlier … and some crucial math is missing completely. … Washington standards do not ensure that students learn the critical algorithms of arithmetic … it ends in secondary school with minimal expectations that are missing most of the algebra, geometry, and trigonometry found in other places.”

Forced into it, the Office of Superintendent of Public Instruction (OSPI) solicited bids to coordinate a math standards rewrite, ultimately contracting with The Charles A. Dana Center at the University of Texas.

The Dana Center’s Executive Director Dr. Uri Treisman wouldn’t tell me the value of the contract, but seven weeks after requesting a release of public records from OSPI, I was told the Dana Center bid was $769,943. Another $110,000 extended the contract through March 15, 2009. OSPI also contracted with Education Service District (ESD) 113 to handle logistics. The combined funding was “approximately $1,279,943.”

(StandardsWork’s unsuccessful bid was $129,403 plus expenses. StandardsWork previously assisted Indiana and California with aspects of their learning standards.)

I asked Treisman about reports that he and fellow Dana Center employee Susan Hudson Hull were on the advisory board of reform math curriculum “Connected Mathematics.” Treisman said he’s never served on an advisory board for any textbook company, that he’d been asked to serve and had declined.

However, Treisman didn’t answer my question about Susan Hudson Hull. “Connected Mathematics” lists her as a member of its advisory board. Additionally, Dana Center senior fellow Cathy Seeley is a past president of the National Council of Teachers of Mathematics. She helped write the 1989 NCTM Standards that many cite as the basis for reform math curricula.

Treisman, Seeley and Hull were the Dana Center “facilitators” of Washington’s “Standards Revision Team.”

Of the 16 other “national” members, five had connections or former connections to The Dana Center, 10 had ties or past ties to the NCTM. One was on the 1989 NCTM Standards writing team and also on the advisory board for reform math curriculum “Core-Plus Mathematics.” Two were on the writing team for the NCTM’s 2000 publication “Principles and Standards for School Mathematics.” Another indicated that “short division” and calculators could replace long division in Washington’s standards.

In December 2007, Seeley reportedly said “the problem with the old standards was not so much the content, but how difficult they were to use by parents and teachers…The old standards left everyone in the dark about the learning priorities for each year, so teachers had to do some guessing about what to emphasize and most parents didn’t have a clue.”

(Strategic Teaching’s report indicated that Washington’s math problems were very much about content.)

The Dana Center team’s first draft, released in December 2007, came under heavy attack from math advocates, who said it was still unclear and lacking in core content. According to Bob Dean, chair of Evergreen High School’s math department and member of the revision team, subsequent failed Dana Center/OSPI drafts resulted in the State Board of Education rehiring Strategic Teaching to oversee the work. It wasn’t until July that the board finally approved the high-school portion.

Strategic Teaching’s original contract was for $194,400, plus $180,600 to monitor the rewrite. Total costs to assess and rewrite the K-12 math standards were “approximately” $1,654,943.
(An additional $108,000 was recently approved for Strategic Teaching to handle issues relative to math curricula.)

After OSPI’s August 15 “Preliminary Curricula Review” placed reform curricula fairly high on the list of curricula meeting the revised standards, I asked Dean how that could be. “The standards revision team was selected by (Superintendent) Terry Bergeson,” he replied, “and out of the roughly 30 people involved, only about three of us were not pro-reformers.”

Although Dean feels there were “small victories,” he says the revised standards are inadequate. The standards were “basically designed by reformers,” he said, “and favor reform curricula over more traditional curricula.”

Still.



Please note: The information in this post is copyrighted. The proper citation is: Rogers, L. (September, 2008). "Washington's math standards failing the students." Retrieved (date) from the Betrayed Web site: http://betrayed-whyeducationisfailing.blogspot.com/

Birth of "reform" = demise of math skills

Curiosity, questions and a tape recorder: That's what I had in January 2007 when I met with the superintendent and the curriculum director of Spokane Public Schools. I thought I'd write an article about why my daughter's 4th-grade class wasn't working. I brought my recorder because I'm a former journalist, and that's what journalists do.

The responses I got that day drove me to ask more questions. By October, my curiosity had become a calling, and in January, it became a book. This book wasn't what I'd planned to do; it became something I had to do.

My book is called "Betrayed." It articulates the lies, ego and blatant opportunism that have turned public education into a public disgrace. School districts across America have betrayed millions of families. Self-serving, self-important "educators" have tortured the process to death with off-the-wall theories as they grasp for billions of taxpayer dollars. Despite being filled with the nicest, most caring teachers and principals you'd ever want to meet, Spokane Public Schools is nearly a "perfect storm" of what's wrong.

Based as it is on lies, ego and greed, public education is failing, and our children pay the price with their futures.

There isn't room here to tell you everything I've discovered, so for now, I'll focus on mathematics. Mathematics is straightforward, and its mishandling is crystal clear. Indeed, it's the proverbial "canary" in the mine.

Spokane, like many districts across the country, uses three reform curricula: "Investigations in Number, Data and Space"; "Connected Mathematics"; and "Core-Plus Mathematics." "Reform mathematics" is the current education fad. It's less about mathematics and more about how educators want to teach math. Reform is heavy on problem solving, estimation, calculators, computers, group work and constructivist approaches (where children figure out things for themselves). It's typically light on basic arithmetic, practice and direct teaching.

Sadly, students whose teachers depend on reform curricula are less likely to know how to multiply vertically, do long division, manage fractions and exponents, or handle much algebra beyond the basics. They're likely to add on their fingers, become dependent on calculators and be confounded by the simplest arithmetic. They're likely to estimate and "think outside of the box," but less likely to know whether their estimations are in the right galaxy. The most capable are likely to get As in school, pass honors classes and standardized tests, yet require remediation before they begin college, learn a trade or enter the workforce.

Reform curricula are popular, but not because they work best or because scientific research supports their efficacy. (They don't, and it doesn't.) They're popular because they've been promoted around the country as being "exemplary."

Here's how that happened.

In 1989, the National Council of Teachers of Mathematics published new standards that focused on problem solving, estimation and calculators, and downplayed "rote use of symbols and operations," "rote practice" and "teaching by telling."

Several of those same authors then developed (or helped develop) curricula based on the standards.

The National Science Foundation financed several of the curricula, including "Investigations," "Connected Mathematics" and "Core-Plus," and it paid to disseminate them around the country.
(The NCTM has denied to me that it advises school districts, uses the word "reform" or is supported by government funding.)

In 1999, the U.S. Department of Education solicited commercial mathematics curricula. Proposals were to be based on NCTM Standards. Despite public protestations from 200+ professionals, the final list of 10 (which included "Connected Mathematics" and "Core-Plus") were endorsed by Secretary Richard Riley and widely promoted as being "promising" or "exemplary."

Later, the NSF called 13 NSF-supported reform curricula "exemplary," including at least five of the DoE's list of 10. It continued to promote and financially support them.

Which came first: the NCTM Standards or colleges of education that promote reform philosophy? I don't know, but they're in sync now.

As this money, ego and flawed philosophy scuttled around the country, the children's math skills fell through the floor.

Ironically, the more children struggled, the harder reform advocates pushed. If it didn't work, it wasn't because "reform" is flawed; it was because teachers needed more "professional development." Calculators and computers were pushed on schools to take the place of basic skills. Standards were revised, pass scores were lowered and assessments became easier. Critical math skills, including long division and multiplication, were deemed "unnecessary." Ninth-grade math class became a game with molding clay and pipe cleaners.

In 2007, Washington's K-12 math standards (which were guided in part by the NCTM Standards) were found to be inadequate and unclear.

No one will tell you that billions of dollars and children's futures were willfully sacrificed on ineffective philosophies.

Meanwhile, despite pitifully low WASL scores (especially in math and science), struggling schools in our district have won state awards for how well they're doing.

That's known as "spin."

In July, The Spokesman-Review commented on the state superintendent's "tendency to gloss over bad news, such as spinning the high school dropout rate." Then it glossed over that bad news by endorsing her re-election bid.

At what point does "spinning" become deceit? I say it's when the intent is to deceive. Estimates of costs for Washington's standardized tests are typically partial figures, covering just state costs and just the actual tests. Estimated dropout figures and graduation rates typically exclude groups of students whose performance would depress the numbers. Statistics that place Washington at first in this or at the top of that are typically extracted from reports that put Washington in the middle of a nation whose performance is generally abysmal.

Administrators can pluck, parse and work the statistics until they say something positive, but the actual data tell a gloomy tale. Ethnic groups and lower-income families suffer, it's true, but don't kid yourself. The gifted and talented are some of the most neglected students in the state and country.



Please note: The information in this post is copyrighted. The proper citation is: Rogers, L. (August, 2008). "Birth of 'reform' = demise of math skills." Retrieved (date) from the Betrayed Web site: http://betrayed-whyeducationisfailing.blogspot.com/

Portions of this article were published August 28, 2008, in The Inlander at http://www.inlander.com/commentary/305115792188570.php
The entire article was published September 6, 2008, in EducationNews.org at http://ednews.org/articles/28780/1/My-book-is-called-Betrayed/Page1.html