Can We Assess Learning?

Chad Aldermna at The Quick and Ed linked to a piece by Bernard Fryshman called Do We Assess Learning? Pull Up a Chair... in which Fryshman argues about the assessment of colleges and college learning. After a lengthy and hypothetical series of questions, Fryshman writes:

Come to think of it, we may not be able to answer your question. Because assessment is not a number, assessment is a conversation. Learning outcomes are not numbers. We simply can’t assimilate all the variables — those noted above and so many others — without papering over most of the things that really matter. There are so many permutations, even a statistical approach is not viable. Because each person’s interaction with higher education is unique, the sample size is always one, no matter how large the population.

There are no clear answers — and not because we wouldn’t like to give you an answer, nor is it because we think “you won’t understand.” There is little that can be measured with any degree of certainty. A shoemaker can perhaps be judged by the quality of the shoes he turns out. He and his colleagues start with identical pieces of leather, nails, shoelaces and thread. The products have few enough variations that there can be some comparison between the two.

Not so, colleges and universities. Every individual coming in has so complex a series of characteristics, and emerges after so many different activities and variables, that any comparison or generalization is meaningless. Sometimes, when large enough numbers of students from similar enough backgrounds travel through a narrow program that is relatively unchanging, one can reach some general conclusions. But only on a discipline by discipline basis — anything broader brings to bear so many different variables as to make assigning a numerical value to student learning outcomes an exercise in futility.

First of all, I don't think anyone who insists on some sort of assessment ignores the concept that each student in any school is an individual, with a complex background, different skils, interests and ambitions. Having said that, I don't understand why anyone would insist that it is impossible to assess learning at any level.

I don't understand when and where the term "assessment" came to be equated with "numbers." Assessment is not all about a number and no person with a goal of determining the quality of any institution is going to be satisfied simply with "the numbers." Are the numbers instructive? No doubt. When talking about colleges and high schools, numbers like graduation rates, cumulative GPA, absolute numbers of graduates, type and number of degrees, enrollment, remedial educaiton numbers, etc. All of these numbers matter, but there is more and there we get in the qualitative measurements which Fryshman seems to think cannot be assessed, but is that really the case.

There is and has to be a systematic manner in which qualitative measurements be taken. Fryshman his ilk seem to think that because they can't envision (or don't like the measurement) then it doesn't or can't exist.

This is bollocks, to put it bluntly.

We have to assess whether or not high schools and colleges are providing a valuable service and worthy of the money spent and the education supposedly offered.

Denver ProComp Negotiations

The Quick and the Ed has a good summary about the economic principles, including incentives that go into designing a compensation system for teachers. The summary does a good job, but fails to account for one big mover--what is the market, i.e. the voters, willing to bear in terms of salaries.

But what the story does reflect is the tension between old-school union ideas of compensation and the package put together with teachers (as opposed to union officials or negotiators) providing imput.

That is why the political maneuvering in pay negotiations is far more important than the actual terms of the negotiation itself. Threatening a strike when thousands of Democratic activists are in town (many of them teachers or union supporters) makes the possiblity of a strike more palatable for the Denver teachers and less so for the city.

However, the City can pull an ace out of their hat--the voters--which supported the ProComp plan to the tune of cedeing $25 million to make it work. The City should not shy away from breaking out the big gun of public opinion.

This is where too many cities and school systems lose. They don't play the big guns. ProComp was negotiated and endorsed by the union, the school system and the voters. The City should just call the union on it and actually dare them to strike and shut the schools down.

The primary effect of the strike will be to put the union into the position of having to admit that their primary goal is protecting teachers. There is nothing wrong with this position, but the unions often couch their spin in terms of "good for the students." Striking is good for the teachers, not the students and such a position is clearly exposed when teachers strike.

Of course, you will also see a number of teachers cross the picket lines and the size of the line crossing contingent surely indicates the relative strength of the union.

Dumbing Down: Then And Now (Forum)

This 8th Grade test from Kansas (hat tip the Instapundit)has been largely debunked as a hoax. But here is a more a deeper question hinted at by the guys at Classical Values a couple of years ago--why not take the test at face value and upgrade it a little. How many Kansas teenagers, say high school seniors, could pass this test?

You could update the test a little and maybe take out some of the more specific agricultural references, but still at the core it is not an unreasonable test in some ways. The history section, the language arts section and what not still apply and one wonders how many students would pass such a test.

Some of these items are mere factual questions, i.e. dates etc. But while one need not know in depth every event of American history, there are some key dates and those dates should be readily known. Additionally, basics of grammar, orthography (spelling to you and I), georgraphy, history and math without a calculator are important skills. Why should we simply rely upon the usefulness of the internet to find out answers? Why can't we have students armed with basic information in their heads who can then supplement it with research skills as necessary?

Assessment 101

As I look toward a teaching career at some point in higher ed, this post by Right Wing Nation is one to take heart.

Not Every Education Problem Begins and Ends at NCLB

Over the weekend, Naomi Wolf has an op-ed in the Washington Post talking about how the young people of America seem aloof or indifferent about politics these days.

When I speak on college campuses, I find that students are either baffled by democracy's workings or that they don't see any point in engaging in the democratic process. Sometimes both.

Not long ago, I gave a talk at a major university in the Midwest. "They're going to raze our meadows and put in a shopping mall!" a young woman in the audience wailed. "And there's nothing we can do!" she said, to the nods of young and old alike.

I stared at her in amazement and asked how old she was. When she said 26, I suggested that she run for city council. Then she stared at me-- with complete incomprehension. It took me a long time to convince her and her peers in the audience that what I'd suggested was possible, even if she didn't have money, a major media outlet of her own or a political "machine" behind her.

That young people are not involved in politics is not all that new or surprising. The one group of people with the greatest untapped potential for political activism is and always has been voters age 18-25, which historically have the lowest turnout of any age cohort.

Wolf points out, not incorrectly, that many American high school students are no longer required to take a course in American government or civics and that such a dearth of civic instruction is damaging our young people's perception of politics. But it was this statement that bothered me most:

In recent years, the trend away from teaching democracy to young Americans has been at least partly a consequence of the trend of teaching to the standardized tests introduced by the Bush administration. Mandated by the federal No Child Left Behind Act, the tests assess chiefly math and reading comprehension. Basic civics and history have suffered.

I would like to point out to Ms. Wolf that the Bush Administration did not invent standardized tests and while they may be used differently, standardized tests are not necessarily an evil to be dispised at the mere mention of the term. Attacking NCLB as the strawman for all that is wrong with current American curricula ignores the reality that, just as the hapless college student had the power to change her political circumstances, we have the power through local activism to change the curricula in our schools.

I will admit that NCLB may have accelerated the trend away from teaching civics and American government, but it is not the cause of the decreased number of civically illiterate in America. Wolf was speaking to a 26-year-old person and on a college campus. These are students who came of age before NCLB, and so that crutch, that NCLB has undermined civic education, is not available to them or their educators.

The root cause of the lack of civic education in America's students lies in changes in curricula and what was and is deemed important. Curricula change, ebbing and flowing, with what is deemed important by our communities. Certainly, math skills and reading comprehension are important and should be stressed. But one of the things we are starting to learn is that there are many subjects that are important to the development of a liberally (in the classic sense) educated child. It takes math, reading, and science skills, but a well-educated child must also have an exposure to the arts, history, literature, and yes, government and civics, to have that well-rounded, well-prepared status of citizenship before exiting high school. Curricula have evolved away from history, the arts and literature, and civics in favor of an emphasis on reading, math and science. This is not, in itself, a bad thing since the decisions have been made politically. While the consequences of the decisions at all levels of the government may not have been fully considered, they are political decisions.

NCLB did not cause the curricular change, but the emphasis on math and reading has caused America to realize what gets lost when basic skills are not learned in elementary school. Schools have two scare resources to contend with--time and talent. There are only so many hours in a school day and so many schools days in the calendar. While adding hours or days may expand the opportunity to teach reading, math, science and all the other necessary subjects, in the end, there are political and policy decisions to be made. There are only so many teachers available and those teachers have specialitis that may not lend themselves to the teaching of civics or government (although that may be a problem to consider as well).

In Maryland, at least, students must take an exam in Government in order to graduate. Students must take a class before they take an exam. Maryland has made the policy decision that goverment is an important subject to learn and enforced that decision. If civic education is important (and I whole-heartedly agree that it is), then we must treat it as such and make the decision to include it in the curriculum.

We can't blame a law (a political document) for our failings and it is ironic to say the least, that Wolf doesn't understand the politics of education so much as she misunderstands the education of politics.

The Politics of Exit Exams

The Baltimore Sun looks at the politics and perils of high school exit exams--the chance that some kids won't pass.

Maryland isn't the first state to have second thoughts about denying diplomas to thousands of high school kids who can't pass state tests.

As the graduation deadline grew near, Washington state delayed requiring its math exam. Utah dropped the testing requirement altogether. In Massachusetts, the teachers union mounted an ad campaign against the tests - though the state held firm.

And in California, parents got angry and filed a class action suit. Students still have to pass the tests to get a diploma, but they can stay in high school for up to six years if that's what it takes.

Across the nation, state leaders have gotten nervous as the date approaches to deny high school diplomas. They have agonized over what to do, though in the end, only Utah has decided to do away with the requirement.

"States realize that they have set standards that kids simply aren't able to meet ... or the failure rates are simply too high politically," said Jay Heubert, a professor at the Teachers College, Columbia University.

So the question that is asked by policy makers is not "What should we do to help each kid pass the test as written?" but the question becomes "What can we do to get around the exit exam requirement so that I can look tough on standards without actually holding everyone accountable?"

These are two very different questions. One is a question about education and pedagogy, the other is about political CYA and unfortunately the latter does nothing to help kids.

In Maryland,

State data released last month showed that 68 percent of students in the Class of 2009 have passed the English exam. In Baltimore, 41 percent have passed. Statewide, there was a 77 percent pass rate on the algebra, 71 percent on government and 62 percent on biology.

In concept, making sure that all high school graduates are at least competent in a few basic subjects is easy for politicians to embrace. Maryland began planning more than a decade ago for high school assessments in many subjects. The testing requirement was supposed to take effect for the Class of 2007 but was delayed until 2009.

Then last month, after hearing complaints, state schools Superintendent Nancy S. Grasmick proposed that students who fail the tests three times be allowed to do a senior project instead. Grasmick said it would be a rigorous assignment designed for only a few thousand students a year.

Under Maryland's current rule, students also can get a diploma by getting a minimum composite score on the four tests. And for special-education students and those with testing anxiety, a different but "comparable" test is now being tried out in three districts.

Under the Maryland program, the exams are given to students after they take the relevant class. English is at the 10th grade level.

The problem of course is that people are making all kinds of excuses for the kids. Everything for English as as Second Language (ESL) students to kids with "test anxiety" are going to get a pass from the exam requirement. While federal law requires states make accomodations for students with verified disabilities, my fear is that test anxiety will become a disability and that means they will probably get a pass.

No one seems to be asking an important but relevant question. What is the reason why kids are failing the tests? I find it difficult to believe that an average of 30% of students haven't passed the tests (statewide) without a reason. Do we even know what kids are missing on the tests? Assuming such an analysis has been done as to which questions are being missed, what is the cause. With 66,000 students in the class of 2009, surely there must be some statistical pointers.

Another relevant question might be how well does the test match up with the curriculum? One could easily assume that the two match, but that may not be the case at all.

The answers to these questions should lead to either a better curriculum or a better test. But we cannot and should not allow ourselves to be led down the path of softening the requirement. Sure, it is politically scary to be looking at 2,000 to 3,000 students and telling them they won't graduate because they haven't passed the requisite tests. Even scarier is the prospect of looking their parents in the eye and saying the same thing.

Make no mistake, the finger pointing is going to be ugly. But the blame rests not only on teh students, but on their schools, their teachers and their politicians. The parents are not without their share of the blame, either. But no one should escape accountability and letting the students off the hook allows them and everyone else to escape the consequences of their failure and no one has learned a thing.

For all the states with an exit exam, when it was developed, it was easy as there were no human faces associated with the failure of the exam or the failure of the policy. Now we as citizens are looking at both squarely. If you are going to have an exit exam and spend millions to develop it, you can't simply scrap it because the price is suddenly too high. Either it was a good policy to begin with or it is not. If state policy makers think it is a bad idea, they should say so instead of trying to end run around the problem by granting some dispensation because a kid can't pass the test. In either case, we must take responsibility.

We will do better in the long run by failing these kids now and figuring why they are passing rather than to give them a pass.

Civic Literacy Report

Take this interesting quiz. I did well, missing some questions related to political philosophy and economic policy, but still socred a 55 out of 60 for a 91.6% score. The average for the test is 74.3, so I am ahead of the curve.

the quiz is somewhat hard and is apparently designed to be taken by college graduates. The problem is that many college graduates don't have the same background in political science and history that I do. That is the point behind the group who posted the test.

Harder Courses Produce Gains

A pilot project in Hartford and Frederick Counties has led to some gains among studentsenrolling in more difficult courses reports the Baltimore Sun.

From 2003 to this year, the number of African-American high school students completing algebra I by their freshmen year rose 158 percent, the number of children from lower-income households finishing a chemistry course increased 115 percent, and the number of students overall taking a fourth science course went up 54 percent.

"We have seen significant gain in all student subgroups and the percentage of students taking rigorous coursework," Haas said.

To become Maryland Scholars, students must complete the prescribed additional courses - algebra II, chemistry, physics and foreign languages - and maintain at least a 2.5 grade point average.

Maryland Scholars, who are eligible for the federal Pell grant in college, receive additional aid for their tuition. In their freshmen year, scholars get an extra $750, and if they maintain a 3.0 GPA, they earn an additional $1,300 in their sophomore year. Students who major in math, science or specific foreign languages could receive an additional $4,000 through the federal Academic Competitiveness Grant in their junior and senior years.

The number of Harford County graduates who met the program's criteria increased from 931 in 2003 to 1,481 this year. More than half of Harford's 2007 graduating class were Maryland Scholars.

Since 2003, there has been a 33 percent increase in the number of students receiving free and reduced-price meals who qualified as Maryland Scholars.

The Maryland Business Roundtable for Education, a statewide nonprofit coalition of employers and businesses, partners with the governor's office and county school systems to run the Maryland Scholars program.

Color me shocked--if you challenge kids, they will rise to the challenge. Self-esteem is not built through touchy-feely help, but through success at something that is hard to do.

It is not enough, of course to simply, encourage kids to take the classes, but to succeed in them. Having a GPA requirement helps. Now that half of one county's graduating class qualifies for a program, it is time to up the ante a little. Let's make the coursework a little more challenging, including adding a significant writing requirement, i.e. a "senior thesis" type effort and a higher GPA requirement, say 2.75 on a 4.0 scale.

I have spoken before to classrooms on behalf of the Maryland Business Roundtable for Education and intend to do so again this year. If you live or work in Maryland, I strongly encourage you to check out this program.

Saving Money by Making Tests Easier

Yesterday, the Washington Post carried a story where Maryland will dispose of written response to its high school assessment tests (required for graduation for the class of 2009 and beyond). Oh, the stated reason is not to save money, but that is the real reason.

Maryland plans to eliminate written-response questions from its high school exit exams to address long-standing complaints about how slowly test results are processed, state education officials said yesterday.

Beginning in May 2009, the Maryland school system will phase out "brief constructed responses" and "extended constructed responses" -- questions requiring a short or long written answer -- from its four tests covering algebra, English, biology and government, said Ronald A. Peiffer, the state's deputy superintendent for academic policy.

Eliminating those questions will allow the state to process test results up to four weeks faster than before, Peiffer said....

The move to a pure multiple-choice format addresses complaints from school systems about how long the tests take to be processed.

Written-response questions take much longer to grade than multiple-choice questions because they have to be evaluated by humans, not computers.

Some jurisdictions did not find out how their students had done on the High School Assessments until after the school year had begun, making it difficult to home in on students who need extra help to pass the tests.

Written responses have to be graded by humans, which means you have to pay humans, which means more money.

Yes, hand grading takes more time, and given that the exams are usually administered late in the academic year, there is a rational concern about the fact that grades are delivered later, which does present a problem. But here is a thought, why not administer the tests a little earlier in the year?

The material in the HSAs is geared to a 10th grade level. Students can take the test multiple times in order to pass. According to this flyer regarding the HSA, the tests for each subject (English, government, biology and algebra) are given at the end of each class, not in the senior year. So why not given the exam earlier in the year, or I don't know, require biology, government and algebra of all ninth graders. English could be given at the end of 10th grade. Sounds like a smarter idea, then students have three years (or two in the case of English) to pass the exam. Schedule retests to occur in say October and February of each year and first time tests in the normal course of time.

Of course, retests cost money and that is the real problem here. Maryland wants its kids to graduate, of course, they just don't want to spend the necessary amount of money to make sure they actually know what they should and have the basic skills necessary to succeed later, whether in college or in the workplace. Written communications skills are just one such skill area that is necessary.

Testing Disconnect

Joanne Jacobs links to a Center on Education Policy study that says most of the states with an exit exam requirement for graduation are not making students

(Of 23 states surveyed), only six say that the purpose of the test is to measure the knowledge and skills needed for college-readiness, while nine indicate work-readiness as a purpose.

In contrast, 18 states say that the tests — which are generally aligned to the 10th-grade level — are intended to determine mastery of the state curriculum (e.g. standards, curriculum framework). And 18 states say that the exams are used to provide data to state policymakers on student progress toward state education goals to inform policy decisions.

. While this may seem shocking to, it is not all that surprising to me.

One of the fundamental precepts of testing is knowing what you are measuring and then designing a test to measure that standard. For most school systems and states, what the exit exams are measuring is not college preparedness or workforce preparedness, but "how well we have done our jobs." That is schools are measuring to students to see how well the schools have done educating studetns.

Such a testing arrangement is very different from testing for college or workforce readiness. The bias itself is not all that surprising, given that it is the Department of Education that is building the exam, largely as a means to demonstrating to the public and more importantly to the legislative appropriators that they are not only earning their keep but need more money to earn their keep.

If exits exams are supposed to measure college/workforce readiness, they would have to be built on the standards colleges and employers want. To be fair to the states, that is not really easy since the two groups may be looking for different, if not divergent skill sets. However, that is the only way in which exit exams will measure readiness for higher education or employment.

Social Promotion To A Diploma In Maryland

Beginning in with the class of 2009, the State of Maryland will be requiring graduating seniors pass a series of subject matter exams in order to receive their diploma. The exit exams are designed to be on the 10th grade level and students can begin taking the tests at that time and may take each subject matter test a number of times in order to pass.

Of course, a whole group of people, namely those who think that requiring a standardized test is bad, objected and today, State Superintendent Nancy Grasmick has outlined a plan that would allow students to still graduate

but can't pass the High School Assessment exams in one or more subjects because of test-taking anxiety or other factors...Education officials believe the alternative to the test will essentially prevent the state from denying diplomas to students who would have graduated otherwise.(emphasis added)

For some reason the words "social promotion" are popping into my head.

If these kids can take tests in school without "anxiety" why are they now anxious about taking an exit exam that they can take again and again over the course of three years in order to pass? Presumably these kids did not suffer test anxiety on other standardized tests, for say their driver's license or the other state wide tests given to them earlier in their academic career. It is not like the format of the test is going to change all that much and one would think that with familiarization, the anxiety level is going to go down. Here is my prediction, in the fall of 2008 and early 2009, you will see a large number of students who have not passed the HSAs being diagnosed with "test anxiety," a new form of disability that will allow Maryland high school students to graduate without having to meet all the requirements, namely passing the HSAs.

Under the Individuals with Disabilities Education Act, the state is required to make all reasonable acomoodations for test taking, particularly the HSAs. So, for example, those with diagnosed reading problems like dyslexia, are given extra time. But with "test anxiety" diagnoses are accommodated by not having to take and pass the test--at all!! So by merely showing up to school (admittedly an important requirement) and getting barely passing grades, a student can graduate from high school without having to demonstrate any competency on skills the state of Maryland has deemed important. The value of a Maryland High school diploma is going to plummet.

What makes this practice any different than promoting some seventh grader who has not masterd the necessary skills to 8th grade? How is this going to prepare our kids for the workforce or college, where "tests" are common and employers/professors don't give a hoot about your "test anxiety" condition?

Can someone please explain to me why this is not just social promotion to a diploma?

Education Measurements

Kevin Carey talks about the NCLB reauthorization, and the proposal by Rep. George Miller (D-CA), chairman of the House Education & Workforce Committee and principal guide for the bill, to include multiple measures as part of the accountability package.

As this gets discussed in the coming weeks, months, and possibly years, it's important to understand what's at stake. Nearly all the back-and-forth will be about what gets measured. But equally important--perhaps more important--is who does the measuring.

The multiple measures idea stems from one the most common--and correct--criticisms of NCLB: schools are rated almost exclusively based on state assessments in reading and math. This system can be inaccurate and reductive--not only are we limited to one way of measurement, via standardized tests, but we're also limited in what's measured. Subjects like art, music, social studies, etc. are left out, along with the non-academic skills and character traits that schools are charged with teaching students. By expanding school measures beyond once-a-year tests, the thinking goes, we can get a broader, more nuanced, more accurate sense of what schools are really doing for their students.

A worthy goal, to be sure. But here's the problem: in many multiple measure scenarios, it's the schools themselves that will be doing the measuring. And that undermines one of the great virtues of NCLB: the separation of those being held accountable from the process by which they're judged. That independence is based on a rock-solid understanding of human nature: people can't be wholly accountable to themselves.

Carey is completely right, you cannot allow the persons being held accountable to do the measuring. The measurements must be as "third" party as possible. But even now, we are not at a true independent assessment, no matter what model of assessment you choose.

I talked about multiple test format accoutability schemes and the benefits that can be garnered by relying on more than one test as a measurement of education success. But even in my scheme, the idea is that the testing must be done by an outside group. The fact is that even now we don't have a true third party evaluation. The states generally choose the test and administer the test while still having a pretty massive stake in the outcome (states with better schools attract more business and therefore more residents and consequently more tax dollars).

If the who of assessment is so important to objective testing, why are we still allowing the states, let alone the local school systems or even the schools themelves, to do the assessments? If as Carey points out, independent evaluators like outside auditors or tennis judges are important to maintain independence, then the states are not independent enough of an arbiter when it comes to education.

Multiple Test Format Accountability Schemes

EdWeeks NCLB: Act II blog is looking and following closely the reauthorization of the No Child Left Behind Act. One of the big questions they are now asking is whether the accountability provisions will survive the reauthorization.

In my comments to that post, I noted that the political environment is far too heated for any sort of drastic changes, so my prediction is that Congress will punt until after the 2008 elections. But one of the deeper questions about the accountability provisions is one that will have to be addressed. Do we put too much emphasis on a single, snap shot in time measurements?

I will agree that once a year testing produces incentives that may not actually raise the academic achievement of students. Too much time is spent preparing for the tests, too much emphasis is given to "bubble kids" and there are far too many opportunities for the state and school districts to game the system.

I believe that accountability is important and that data doesn't lie. We cannot scrap the accountability provisions all together, but we do need to look for alternatives. One of the alternative methods that is being tested is the so-called growth model, whereby states can show that their students are learning, thus meeting the definition of adequate yearly progress.

So if there are already alternatives for measuring AYP, what about different models for testing. Well some of those ideas include the use of so-called multiple measures for gauging AYP. I am not sure what Congress means by that, but Education Trust and other groups worry that such multiple measures will gut the accountability provisions and confuse parents.

Yet, I am not sure. Multiple measures can mean several things and what I would like to see the term used in a way that would help the growth model people--multiple state wide tests used to measure progress. At a minium we would be talking about three or four tests a year. The first test of each year would give a baseline starting point for the students. The middle test or tests would provide data to schools and teachers, allowing them to modify teaching content and the final test would provide year end measures.

The wealth of test data that is possible with this model is far superior to any single test format currently available and can help alleviate some of the worst features. Additionally, it can help advance many of the teacher quality issues that I and other have spoken about.

As a starting point the multiple test format ("MTF") allows for mid-year corrections to curricula and teahcing methods. The test data would allow for schools to address areas that show weaknesses on the tests, helping students achieve the necessary standards on later tests. MTF also enable experts who develop tests and curricula to get a better match between the test standards and the curricula so that the disconnect between what is taught and what and how it is tested shrinks.

One of hte most common complaints about the current single test format is the "high stakes" testing atmosphere that is roundly decired by teachers and testing critics. A MTF lowers that stress level as multiple tests breeds a familiarity with the format and provides a sort of leveling of scores. One bad day can be overcome with regular performance on the remaining test days.

The MTF is not all about helping students, but can also help school districts and administrator to identify exceptional teachers and teachers needing some professional development. Growth models and MTF would allow teachers to be measured against their peers in a number of different factors and with far more data available to study, the evaluation becomes much more accurate. Issues such as merit pay can be more readily accepted because the primary means of measuring educational skill is based on multiple tests, not on one test.


Of course, a MTF evaluation would cost more and that is a real hurdle for such a scheme. However, a series of four tests can provide so much more data on what is working, what is not working, the matching of curricula to tested items, the impact of a teacher upon student learning, etc, that the investment might be appealing. Instead of guessing what is happening in classrooms, principals, policymakers and parents would have real data, available in time so that corrections can be made instead of weeks or months after the fact.

When the Adults In Education Cheat

Joanne Jacobs has the disburbing story of a charter school in Oakland that is not only changing greades for report cards, but changing transcripts it sends to colleges.

Truly disturbing.

Standardized Tests Actually Inflate Scores

That is exactly what Mike Antonucci claims, but without supporting data. Of course, he has no supporting data because most politicians rail against standardizd "fill in the bubbles" testing without thinking about the implication if they went with another form. Writes Antonucci:

Every one of the eight presidential candidates to address the National Education Association Representative Assembly made a derogatory reference to standardized tests. At least five - possibly more - railed at "fill in the bubble" tests instead of more comprehensive assessments.

There is no question about it. A "fill-in-the-bubble" test is an insufficient method of assessing a student's knowledge and skills. But to whose advantage is that? One would think that the Educational Testing Service alone benefits from standardized tests.

NEA and the presidential candidates assume that alternative forms of assessment would demonstrate students, teachers and schools are performing better than standardized tests would indicate. That assumption is a statement of faith and not of empiricism.

Any question on a fill-in-the-bubble test provides all the data necessary to come up with the correct answer. Students are then supplied with four or five possible responses. By their very nature, standardized tests inflate the scores of students on the low end of the scale. The only students who score lower than 20 percent on a fill-in-the-bubble test are victims of bad luck, since entirely random responses should raise you at least that high.

Just the appearance of a correct answer printed on a test booklet should increase scores across-the-board. Some percentage of students who cannot correctly answer the question "Who was President of the United States during the Civil War?" with no further prompting, will no doubt choose the correct answer when it is placed next to George Washington, Woodrow Wilson and Bill Clinton. There is good reason to believe that scores would plummet if tests were "fill in the blank" instead of "fill in the bubble."

The logic of that last paragraph is undeniable. Unless a test has a "guessing" deduction, that is, you receive a greater penalty for a wrong answer than for leaving a question blank, a random guess on a four distractor multiple choice exam has a 25 percent chance to be a correct choice. Given that most test have obviously wrong distractors, a moderately skilled test taker can improve their chances. Of course a guessing deduction type test will not sit well with the educational "elite."

Antonucci goes on to talk about the benefit of standardized testing when it comes to scoring:

So why would educators and their political allies criticize measurements that cast them in a better light than the alternatives? It seems silly to put it this way, but it's because standardized tests are standardized. A machine checks the answers. The correct answer in Utah is the correct answer in Alabama. Not much interpretation is needed for parents, journalists and the public. There is little avenue to appeal to subjectivity.

All alternative assessments require an evaluator. Evaluators are not, and cannot be, standardized. They may be influenced, properly or improperly, by whether they taught the student, the student's performance on previous tests, the student's background and circumstances, the evaluator's view of the curriculum, the test, and the specific question being asked, and hundreds of other factors.

But here is another issue--money.

A machine is expensive, but you only need one or two machines to grade tens of thousands of tests in a single day. Program the machine, put a stack of tests into it and let it run. You don't need particularly skilled labor to do it. The machine doesn't care if it is 3:00am or 3:00pm, it doesn't care if it must work 24/7 for three weeks. A machine doesn't make mistakes because it is fatigued or because their spouse yelled at them this morning or their son or daughter is hurt, sick or something.

Human evaluators have all of those distractions, plus they want to get paid. Human evaluators would be astromonically more expensive than anyone could even predict. Ask a teacher how long they spend grading tests for 150 students. Now imagine having to do that for tens of thousands of students.

Evaluation is always a touchy subject and unfortunately, it must be done. The schools demand it, the parents demand it and the taxpayers demand it.

Combatting Cheating

Last month, a story our of Severna Park, Maryland caught my eye since it dealt with cheating on an Advance placement exam. Students at Severna Park High discussed the culture of cheating that runs rampant in that high school and possibly many others. Today, the Washington Post reports on the other side of the cheating world, the efforts by teachers to prevent cheating. Teachers use some common, and common sensical, methods to deter cheating such as putting backpacks, cell phones, Ipods and anything else other than pens and pencils at the front of the room, different test forms, chair spacing and things of that nature to prevent cheating. Some teachers do a "hand check," making students show their hands and arms to prove they don't have notes written on flesh.

All of these are good steps and logical. But one of the most effective methods to combat cheating is to foster an ethical environment where cheating is not only frowned upon by teachers and administrators but frowned upon by the students. The phrase "cheaters never win" should be a mantra with peer enforcement as much as administration enforcement. Strong honor codes, rapidly and publicly enforced, would reduce cheating.

Most students do not like to be branded in negative ways and being branded with a scarlet C is going to be much more an effective deterrant than anything teachers can do.

State School Standards Vary Widely

Well, call me surprised--State School Standards Vary Widely according to the NY Times. This is not news nor is it any big secret.

The National Center for Education Statistis has issued a study in which it attempts to apply a

methodology for mapping state proficiency standards in reading and mathematics onto the appropriate NAEP scale, employing data from the 2004–05 academic year. The mapping exercise was carried out for both grades 4 and 8. For each of the four subject and grade combinations, the NAEP score equivalents to the states’ proficiency standards vary widely, spanning a range of 60 to 80 NAEP score points. Although there is an essential ambiguity in any attempt to place state standards on a common scale, the ranking of the NAEP score equivalents to the states’ proficiency standards offers an indicator of the relative stringency of those standards.

There is a strong negative correlation between the proportions of students meeting the states’ proficiency standards and the NAEP score equivalents to those standards, suggesting that the observed heterogeneity in states’ reported percents proficient can be largely attributed to differences in the stringency of their standards.

In English that means that it appears that as differences in the results among the states can be largely the result of how tough their state standards are.

The NY Times paints a debate between those who think there should be national standards and those, like Secretary of Education Margaret Spellings, who think that states should set standards, but the story--and the bureaucrats--are confusing a couple of things. Take this quote from Spellings:

The new report, she said, will make it possible for parents and legislators to compare national and state test results meaningfully, and to improve school performance as they think best.

“For us to dictate one curriculum and one level of rigor would be very imprudent,” she said.

Spelllings herself, the Secretary of Education, confuses two important concepts--standards and curriculum.

A standard is a level of desired proficiency or skill. It is a goal than can be laid out and measured. Achievement against the standard can likewise be measured. A curriculum on the other hand is a course of study, ideally designed to get the student to achieve the standard, but it is a series of topics that are to be presented in the classroom--a curriculum is not a standard and a standard is not a curricula.

But what this story does cover is the real problem in the manner in which NCLB currently works. What NCLB does is establish a "quasi-standard" of proficiency--without defining proficiency nor ordering teh Department of Education to establish a standard. So the task of setting the standard is left to the states, who want to be in control and avoid penalities, so they set their definition of proficiency at various levels. Then the localities get involved since localities generally set the curricula at the various levels. The problem is that you have a federal mandate left undefined, states setting varying standards and localities implementing a curriculum which may nor may not achieve the state standard.

But what Spellings and Congress don't understand is that they can set a national standard (a goal if you will) and then say that this is the national definition of "proficient" in reading, or math, or science. So long as that standard is measurable and consistent, then you have a standard that states cannot game and you get a better picture of how states and schools are doing.

With a national standard, states can still pick their own tests, assuming the tests are designed to test for the standard--which they can be guaranteed to do so. The states and localities can still set the curriculum with a clear understanding of what is to be achieved with the curriculum.

The problem with NCLB is in an attempt to appease the federalism concerns of states, Congress punted instead of holding firm to the notion that the federal govenrment can set goals and standards without unduly interfereing in how states operate.

This map gives a graphical representation of the study results.

Test Scores Up Since NCLB

According to a "major independent study," the Washington Post is reporting that test scores in reading and math are up.

The study's authors warned that it is difficult to say whether or how much the No Child Left Behind law is driving the achievement gains. But Republican and Democratic supporters of the law said the findings indicate that it has been a success. Some said the findings bolster the odds that Congress will renew the controversial law this year.

The spin is in and getting furious. From Secretary of Education Margaret Spellings:

"This study confirms that No Child Left Behind has struck a chord of success with our nation's schools and students," U.S. Education Secretary Margaret Spellings said in a statement. "We know the law is working, so now is the time to reauthorize."

From the AFT:

"There are a lot of problems with No Child Left Behind that we need to fix because they work against some of the progress that is being noted in this study," said Edward J. McElroy, president of the American Federation of Teachers, a 1.3 million-member union.

The study can be found here and the group behind the study is the Center on Education Policy.

More once I read the study.

Serverna Park High School Cheating Scandal

Yesterday I noted a story out of Anne Arundel County Maryland featured in the Baltimore Sun about Severna Park High School, one of the top schools in the state and nation, and the cheating that appears to be plaguing the school. Today, the Sun is reporting that the College Board blames the school for the AP exam cheaing that was mentioned prominently:

The College Board chided an acclaimed Anne Arundel County high school yesterday for lax test monitoring that students say permitted cheating on the Advanced Placement U.S. history exam.

The AP program's parent company said an inexperienced proctor failed to follow rules banning cell phones and talking in the May 11 incident, which some top-ranked students at Severna Park High School have said is only the latest example of rampant cheating that has gone unchecked.

Such serious proctoring mistakes happen once or twice a year out of the more than 2.5 million AP tests administered nationwide, said the College Board, which also found "inappropriate behavior" among some of the students.

It and school officials, however, carefully steered clear of using the word "cheating."

Three unidentified girls will not be allowed to retake the test but can keep their history class grades, the school system said yesterday. The other 42 students in the "compromised" classroom must take the test again tomorrow.

If the reports about what happened in teh AP, detailed in this Sun article from yesterday, are even remotely true, the cheating is the right word.

These three young (I hate to use the word "ladies") women are be barred from taking the exam and I would hope that their registration fees are retained by the College Board.

School officials, including the Superintendent and the school's principal are being quite circumspect, but now that the story is out, it is going to be hard for these two men and many other adults at Severna Park to keep their jobs--and rightfully so.

"A Culture of Cheating"

That is how one student at Severna Park High School in Maryland calls the atmosphere, and there is little enforcement from the administration:

Peter Thompson calls it his "baptism by fire" at Severna Park High School: In ninth grade, a classmate snatched a test paper from his desk and passed it around the classroom. Teachers and administrators didn't punish anyone after he and his family complained, Thompson said.

The junior, ranked first in the school of 1,700 students, said he has seen plenty of other incidents of cheating since then, but nothing like one earlier this month: At least one student allegedly stole away to a bathroom with a sealed booklet of essay questions for the Advanced Placement American history exam and, with two friends, scoured a review manual for answers.

Now, the College Board, which developed the test, is investigating, and all 45 students in that testing group have to retake the exam Thursday.

But that wasn't enough for Thompson and a group of other top-ranked students. They're so outraged at what they view as rampant cheating at the school that they surveyed their peers last week and plan to press the principal today to discipline the cheaters and adopt stricter test proctoring rules.

"Anyone who doesn't think there's a culture of cheating here is either oblivious or blind," said Thompson. "Ranks and GPAs here are meaningless."

Reports of this single incident - and students' outcry that it is part of a widespread problem - have rocked this exclusive community just north of Annapolis, where homes command higher prices because of the lauded high school, which consistently leads the county in state test scores.

snip

Lax oversight during tests and classwork, students say, has fueled cheating in a school that vows with banners over the front door and over public address announcements to become "the No. 1 School in Maryland."

In results to be shared with Principal James Hamilton today, 69 percent of the students who responded to the survey Friday said they felt that a culture of cheating exists at the school. About 50 percent of the roughly 300 who responded estimated that up to half the student body cheats and felt that teachers didn't hold enough power to stop cheaters.

"Some teachers do not allow any form of cheating and offer extreme punishments for anyone caught cheating, but on the other hand, some teachers do not say ANYTHING or pay attention at all to students when they are taking tests or to monitor students during tests to prohibit cheating," wrote Kathryn Muha, a junior at the school who said she saw the May 11 cheating. "Students are rarely punished and if anything happens to them, they are simply warned not to cheat again, or else."

The College Board is investigating, but their impact will be limited to the AP test. The fact that the school's leadership has permitted this perception, let alone reality, says a great deal. At the very least, the entire school leadership and teaching corps needs to be examined throughly and very harsh punishments handed down for allowing this culture to grow and flourish.

Read the whole article for an example of what is wrong in Severna Park.