Tuesday, March 27, 2007

Moneyball and Money Teachers

A few years ago, I had a boss who was constantly hammering into those of us working for him that numbers matter, statistics matter and we can make more money if we pay attention to metrics and past performance. Because we were in a political business, that of managing software and political action committees, I thought he was nuts. But, trying to keep an open mind, I went along with a couple of his experiments, examining costs to achieve certain results, time spent on activities, and other ideas that led to some hard numbers about our performance. The results were impressive, we were able to cut costs, cut time on common tasks by spending the time and money to build tools and processes to cut time and reduce error. As a result, we were able to take more clients and make more money.

I have been revisiting the idea by reading Moneyball, the fabulous book by Michael Lewis about how the Oakland A's baseball team that was able to win so many games with a payroll that was a fraction of the big teams' salaries. Billy Beane, the general manager of the A's instituted a program where he studied stats and examined the data behind the most successful players and games. With his system of hard data examination, he was able to achieve practical miracles that countered everything "baseball experts," including his own staff, thought would happen.

Moneyball, this post by a new blog, The Common School, and this post by Brett Pawlowski at the DeHaviland Blog have gotten me thinking about what metrics we can use in schools to measure teacher effectiveness.

Right now we have lots of metrics and measurements. The NCLB proficiency ratings, state school assessments, individual test scores, and on and on and on. But do we really have stats on teachers? If we as a society do, we certainly don't talk about them and we certainly use them to determine what works and what does not.

In Moneyball, Billy Beane and his team learned that among the most important factors in winning baseball games was on-base percentage (that is how likely a person is going to get on base every time they come to the plate--this includes walks) and slugging percentage, (that is the number of bases a person is likely to get for each hit). Teams and players that had high percentages were more likely to win games because they would score runs, you can't score runs if you don't have players on base and if players are on base they are not adding to the out count.

So if the goal is the successful education of children, then we need to determine what factors increase the likelihood of a well-educated child. As the Common Room recently noted (and many others) the single most important factor is the quality of the teacher.
First of all, teacher experience and licensure rank near the top of the list while class size reduction and teacher's having master's degrees rank at the bottom. But more important than the ranking is how much smaller of an impact class-size has than some of these other factors. For example, having a teacher who is not a novice (7.2% - 9.1% SD) exerts an influence 3½ to 7 times greater than the impact of reducing class size by 5 students (1.0% -2.5% SD).
So that is one factor, but experience is hardly the only determinant of quality teachers. So what are some of the other factors, i.e. statistics, that can be determined and applied.

Well, some obvious ones would be proficiency on state/federally mandated tests, i.e. are the teachers students below, at or above grade level. Another would be the difference in these numbers relative to their peers with the same task/experience level. Thus you would be comparing 6th grade Math teachers with 4 years of experience with other 6th grade math teachers with 4 years of experience.

Keeping with that theme, how many students changed their proficience level relative to last year. For example, how many (what percentage of students) went from basic to proficient or vice versa? Now these may be a little broad, so if the criteria were narrowed to look at test scores themselve, i.e. the raw scale change in scores.

Of course, of the the most common complaints from teachers is that they have little control over the assingment of students to their classroom. So measurements like those above may be skewed unfairly by poor student assignment luck. But such concerns can be alleviated by pre-and post testing, that is test given at teh start and end of the year on matters of curricula. You test students at teh start of the year to see what they know and the same thing at the end of the year. Good teachers should be able to increase the knowledge base better than their peers. These kinds of stats would allow for each teacher to be judged on what they accomplished with what they had. Plus as an added bonus the teacher can use the results to help tailor curricula.

There have been a number of times where I discussed the need for treating teachers as professional and having teachers act like professionals. One of the most important things for us to do is find a way to empiracally study teachers, i.e. come up with a way of measuring their effeciveness. Teachers who are effective should be examined to see what can be replicated and I refuse to believe that successful teachers are "magical" in some indefinable way that cannot be replicated.

But we don't have any data.

Any other suggestions for measurement?


Mrs.K said...

Is it possible to be offended by a statistic? I bristled at seeing "novice" equated with "[lowered] teacher quality." I know I can't fight the numbers but still...

Matt Johnston said...

Mrs. K,

Sure, you can take offense at a statistic, I do all the time in education.

But in reality, experience does lend itself to a certain level of quality. All things being the same in education, training, temprament, etc. is not a teacher with five year of experience worth more (comparatively) than a teacher with one year.

I don't think Common School was arguing that a novice teacher lacks worth, merely that a novice teacher on a quality scale has less of a given quality (in this case, experience) than others. That lack of a quality impacts the ability of that teacher to make an impact on their students.

Of course, we all have to gain experience in teaching, but generally teachers gain that experience at the expense of students who need better quality (read more experienced at least) teachers.

Jen said...

Pre and post-testing does get rid of some of the problems of what children a teacher is assigned -- but would a teacher with say 75% below basic students be likely to show as much change over the year as a teacher with 75% proficient or advanced?

If some children learn more slowly (not that they can't learn, but that it will take more exposure, more repetition, more *time*) the teacher who works with more of those kids will likely have lower scores than another teacher with fewer of those kids.

On a related note, rather than these yearly state tests, I'd much rather see some sort of ongoing assessment -- that is, an enormous test wherein a child starts just below where they'd finished the year before and keeps going until they are getting too many wrong/can't go on. Then you could actually look at changes in higher performing students as well. A 5th grader scoring advanced -- might have tested that way as a 4th grader or even a 3rd grader taking the same test.

Brett said...

Great post. Regarding your statement/question:

"But we don't have any data. Any other suggestions for measurement?"

You might be interested to know that Tennessee does have specific and relevant data on teacher effectiveness. Tennessee uses a value-added assessment system, which is a sophisticated method for measuring only the education system's contribution to student learning - it can actually remove factors that influence general learning levels (socioeconomic variables and the like) to identify whether students are gaining one year's worth of knowledge in one year's time. If a student goes beyond their individual expectations, the school is clearly the reason; the inverse is true as well.

The TVAAS system tracks student, teacher, and school performance, so that data is available on teachers. Unfortunately, only school-level data is made available to the public.

The good news is that the data is being captured, and hopefully used by schools (the schools do have access to teacher-level and student-level data). And more good news is that other states are adopting the TVAAS model, including PA, OH, and NC.

The Education Consumers Foundation (a client of mine) has been doing some very interesting work to highlight the value-of value-added data in Tennessee. They launched an awards program last year to recognize the principals of the highest-achieving schools in Tennesee; you can see information here: http://www.education-consumers.com/awards/06awards.html. A little research, by the way, showed that eight of their nine elementary-level award winners were Title I schools, which breaks with the myth that high-poverty schools can't offer an effective education.

Anonymous said...

How will you compare 3rd grade at an underperforming school at 30-to-1 with an experienced teacher but 75% turnover, vs. a novice teacher, "average" school, 20-to-1? You can't collect data to evaluate a teacher without at least a minimum statistical percentage of the classroom, what happens when turnover exceeds that?

Are you going to incorporate ELL test scores to equalize that status? What if tests like the CELDT are testing everyday language, not academic language, and what happens when some students are say, a 3 across the board and others are like 2-5-1?

Better question- what is this data FOR? Considering NCLB wants to use it, if it's created, to classify the bottom 25% of teachers based on such scores as a quota to allow them to be fired?