There seems to be quite a bit of confusion surrounding the various levy options under consideration by the Worthington School District administration and the Board of Education. This post is an attempt to demystify the options.
Before doing so, some background might be helpful. There are two distinct sets of "needs" that are going to be included in any funding request eventually. These two needs have two distinct checking accounts and a law that says that you can't commingle the funds. Let's first identify the needs.
First is the need for additional operating money. The vast majority of operating funds go to pay employee salaries and benefits, but these funds are also used to pay other day to day operating costs like electricity, natural gas, diesel fuel, paper and some larger ticket items like the tuition that Worthington pays to charter schools or community schools to educate kids in our community. To see where the money is being spent today, click here.
Second is the need for additional capital improvement money. This money is typically called "bond money" and can be used to pay for items (things) that will be in use for more than 5 years like technology, buses, roofs, painting, repaving, lockers, alarm systems and so forth. By law, this money cannot be used to pay for salaries and benefits. The administration has recommended that we borrow 40 million dollars over a period of 15-18 years. The 40 million dollar figure comes from the maximum amount that can be borrowed without increasing the current 3.8 mill burden on residents for capital improvements. Note that if we didn't borrow this money, the 3.8 mill burden would be 3.0 mills next year and eventually, as previous bonds are paid off, go to zero. I mention this because the levy campaigns like to highlight (correctly) that the bond component is "no additional millage", however, that doesn't mean that it's free, it means that the property tax decreases that would otherwise occur as principal is paid off won't occur. A writeup of what the district wants to spend $40,000,000 on can be found here.
The question, and the confusion, come about because the law allows you to ask for both of these things with a single ballot question or with two ballot questions. Further complicating the issue is the fact that the law allows for the phasing in of operating money over a period of time (This is known as the incremental levy) but you cannot combine this with a bond request on a single ballot.
As a result, on June 5, 2012, the Board and the administration will meet to consider the following options.
1) An operating levy in combination with a bond levy on a single ballot question.
This option allows the administration to combine both requests in a single ballot question. Voters can either vote YES to both requests or NO to both requests and you only get one vote. The bond component will be 40 million dollars. The operating component is still unknown. The Treasurer has recommended 6.9 mills, or about $494/year on the average Worthington home of $234,000 dollars. You can calculate the amount for your own home here
2) An incremental levy and a bond issue on two ballot questions.
Many residents have contacted us asking for a phased in levy similar to what we did in 2009. The incremental levy in 2009 was 3.9 mills for 2010, 5.4 mills for 2011 and 6.9 mills for 2012 and each year thereafter. The Treasurer has indicated that an incremental levy of 4.9 mills in 2013, 5.9 mills in 2014 and 6.9 mills in 2015 would meet the operating needs of the district. The law does not permit you to combine these two issues on a single ballot question, therefore, two ballot questions would be required. Voters could vote YES on neither issue, one issue or the other issue, or both issues. You can calculate the tax for your own home, by year, here.
3) A "promised incremental levy (PIL)" and a bond issue on a single ballot question.
Some administrators, including the treasurer, do not believe the district should offer the opportunity to vote on each of the two needs separately. They are concerned that you might vote for one issue or the other issue and not both issues, but they also recognize that in this economy, phasing in the levy is also desirable. Again, the law does not allow the combination of an incremental levy and a bond issue on a single ballot. To get around this, we could offer a "Promised Incremental Levy" which takes advantage of the fact that the school district does not have to collect all of the millage it is allowed to collect. In fact, Worthington Schools has been doing this for 5 years now, collecting less millage than we are otherwise entitled. The Promised Incremental Levy would simply tell the community that in 2013, we would collect only a fraction, perhaps 70% of the 6.9 mills that would appear on the ballot. The next year, 2014, we would collect perhaps 85% and the final year and each year thereafter, we would collect the entire 100% of the 6.9 mills. Note that the increments still need to be discussed - the above is just for illustration. The PIL and the Bond issue could be combined in a single ballot question as in Option #1. One concern with the PIL is whether the community would believe that the Board and the Administration would keep this promise. Once again, Worthington has been collecting less millage than we are entitled to for 5 years now. In addition, the Treasurer said in a community conversation that he would resign before breaking the promise. Frankly, there is no way that the Board or any of its members, or our Superintendent or Treasurer would renege on a promise like this.
4) Run an Incremental Levy in November of 2012 and a Bond Levy in 2013.
The desire to run both the operating levy and the bond levy in November is a function of the difficulty of the campaign and the time and resources required, not the needs of the district. Nothing bad is going to happen if the bond money is not approved until the next election in May of 2013. I can certainly understand why the residents who volunteer for levy campaigns don't want to do this twice, however, it is an option and should be considered.
It is possible to run an incremental levy in November of 2012 (Option 2) and a bond issue in May of 2013. Obviously, voters could approve neither issue, one issue or both issues. The treasurer will point out that a bond issue in May of 2013 would not be "No additional millage" and he is correct, but only because the millage you are already paying for bonds is scheduled to decrease by around 0.8 mills. A bond issue passed in May of 2013 would restore the millage you are paying for capital improvements to 3.8 mills, just as if you passed the issue in 2012. There is no economic difference. Please note that the district does have a considerable cash reserve to handle any emergencies that may come up. In fact, there is about 7 million dollars that is earmarked for contingencies and/or emergencies.
So those are the four options. Please note that the amounts (6.9 mills operating, 40 million dollars bond) are the treasurer's recommendation. They may or may not be the final numbers. My thoughts on that in a future blog post.
Thursday, May 31, 2012
Tuesday, March 27, 2012
The New State Report Card proposal.
In the waiver request filed with the federal government to excuse the state from the provisions of No Child Left Behind legislation, the state provided the roadmap for its current thinking with regard to the state report card. What the state of Ohio has decided to do, at least in the short term, is to maintain most of the existing metrics but simply grade downward in an effort to give less "Excellent" ratings on the report card. Whether their motivation was to combat grade inflation or to make their waiver request more appealing to the federal government, a new report card is coming to Ohio.
The new report card would be made up of 4 components and they are all graded separately and given equal weight in determining the final grade. The components are:
1) The scores on the proficiencies
2) The performance index
3) The value-add assessment
4) A new metric designed to measure the success a district has at eliminating the achievement gap
The overall grade is determined by averaging the grades on the 4 metrics.
Let's look at each in turn.
PROFICIENCIES
At least through 2015, the proficiency tests will remain largely as-is, cut scores and all, so, for example, a district will continue to receive credit for passing the 6th grade reading proficiency if 75% of the students get a 35% on the test. If you pass 90% of the available proficiencies, your district receive an "A" on that particular metric with 80% required to get a "B".
PERFORMANCE INDEX
The performance index metric attempts to give the district credit for those students scoring in the "advanced" and "accelerated" range on the tests, but as we see from the charts in the previous posts, the cut off scores for advanced and accelerated remain at the same low levels that they were before, therefore, the performance index metric should be similarly unchanged. The difference is in the score required to receive an "A". If you receive a performance index of 90% of the maximum, or 108 out of a possible 120 points, you get your "A". Only one school in Worthington, Evening Street Elementary, received a performance index in excess of 108. It's score was 108.8, however, we must remember that the performance index is only concerned with how many students clear the proficient, accelerated and advanced bar and not necessarily by how much those students passed the bar. Let's take an example from a school that would narrowly miss getting that magic 108 to receive the "A" on the performance index component.
Value-Added: While performance scores demonstrate a student’s level of proficiency, Value-Added
measures the effects of schools on their students’ growth. It is calculated only for schools with students in any Grades 4-8. Ohio, using the SAS® at EVAAS® model computes a Value-Added measure for each school and district in English language arts and mathematics and reports whether the expected growth has been met (a year’s growth in a year’s time), exceeded (more than a year’s growth in a year’s time) or not met (less than a year’s growth in a year’s time).
Interested readers can learn more about value-add here.
For purposes of this discussion, value-add was not changed in the new state report card. What was changed was the impact that value-add could have on the overall grade. Prior to the new state report card methodology, value-add was used to modify the overall grade. This was the algorithm used:
Thus, under the old system, it was possible to be "excellent" with a performance index of 90 or if you met 20 of the 26 standards as long as your value-add scores indicated "above expected" growth for two years running. Under the new system, that will no longer be possible.
ANNUAL MEASURABLE OBJECTIVES
The latest edition to the eduspeak vocabulary is "Annual Measurable Objectives" or AMO's.
The AMO metric is designed to replace the previous level of proficiency that was required by No Child Left Behind. NCLB required, by 2014, that 100% of kids achieve proficiency. Scholars can argue whether that was ever a realistic goal, but states eagerly took the money offered by the federal government back in 2001 and then back loaded the proficiency requirements. Fast forward 10 years, and as the mandatory levels of proficiency started to increase as we got closer to 2014, panic set it. It is fair to say that the entire purpose of the waiver request is to relieve Ohio of the burden of a 100% proficiency requirement and the associated penalties when they are not met. The new goals, the "AMO's" are derived by taking the percentage of Ohio's kids that are currently deemed as not proficient in reading (18.1%), establishing a somewhat arbitrary goal of getting half of them to proficient within 6 years (e.g. 9% additional proficiency in 6 years) and using equal increments, so the bar is moved by 1.5% each year. This would then apply to all subgroups. A similar methodology is employed for the graduation percentage component of the annual measurable objectives. A somewhat complex calculation than goes into arriving at a final letter grade for the achievement gap component of the state report card.
The big difference between AYP and AMO's is this. In order for AYP to negatively impact a final rating on the existing report card, you would have to miss AYP in the SAME subgroup for three (3) consecutive years. Of course, given the requirement to be 100% proficient, absent the waiver, that would happen to every district sooner or later, but not for several years. With the new methodology, AMO's are a full partner in the calculation and the failure to meet the annual objectives could impact your rating in the current year.
This new system will be in place, assuming the legislature goes along, for the current year (2011-2012), so it is important to understand the differences between the old methodology and the new methodology, but it is also important to understand that the State Report Card will be changing fairly dramatically a few years from now to meet the demands of the assessment component of the common core.
The new methodology may therefore have a life span of a few years until the Common Core assessments kick in. Where this leaves Worthington is a bit unknown but as with most things, knowledge is power so understanding the methodology behind the new report card should give us a leg up on figuring out whether Worthington will be "A" rated or, if not, why not.
The new report card would be made up of 4 components and they are all graded separately and given equal weight in determining the final grade. The components are:
1) The scores on the proficiencies
2) The performance index
3) The value-add assessment
4) A new metric designed to measure the success a district has at eliminating the achievement gap
The overall grade is determined by averaging the grades on the 4 metrics.
Let's look at each in turn.
PROFICIENCIES
At least through 2015, the proficiency tests will remain largely as-is, cut scores and all, so, for example, a district will continue to receive credit for passing the 6th grade reading proficiency if 75% of the students get a 35% on the test. If you pass 90% of the available proficiencies, your district receive an "A" on that particular metric with 80% required to get a "B".
PERFORMANCE INDEX
The performance index metric attempts to give the district credit for those students scoring in the "advanced" and "accelerated" range on the tests, but as we see from the charts in the previous posts, the cut off scores for advanced and accelerated remain at the same low levels that they were before, therefore, the performance index metric should be similarly unchanged. The difference is in the score required to receive an "A". If you receive a performance index of 90% of the maximum, or 108 out of a possible 120 points, you get your "A". Only one school in Worthington, Evening Street Elementary, received a performance index in excess of 108. It's score was 108.8, however, we must remember that the performance index is only concerned with how many students clear the proficient, accelerated and advanced bar and not necessarily by how much those students passed the bar. Let's take an example from a school that would narrowly miss getting that magic 108 to receive the "A" on the performance index component.
 |
| Performance Index Calculation for Phoenix Middle School, Worthington OH |
The example demonstrates that even with 95% of the scores being proficient and fully 2/3 being accelerated or advanced, it still isn't enough to earn that "A", apparently because the state realizes that with the cut scores being as low as they are, "Accelerated" or "Advanced" simply isn't good enough any more.
VALUE ADD
The next component is "value add" and this is the description from the waiver request:
Value-Added: While performance scores demonstrate a student’s level of proficiency, Value-Added
measures the effects of schools on their students’ growth. It is calculated only for schools with students in any Grades 4-8. Ohio, using the SAS® at EVAAS® model computes a Value-Added measure for each school and district in English language arts and mathematics and reports whether the expected growth has been met (a year’s growth in a year’s time), exceeded (more than a year’s growth in a year’s time) or not met (less than a year’s growth in a year’s time).
Interested readers can learn more about value-add here.
For purposes of this discussion, value-add was not changed in the new state report card. What was changed was the impact that value-add could have on the overall grade. Prior to the new state report card methodology, value-add was used to modify the overall grade. This was the algorithm used:
 |
| Value Add impact on final report card grade |
With the new methodology, value-add is a full partner in the calculation. If you exceed Value-Add, you get an "A" for that component and the "A" will get averaged in to the final grade. This chart shows the new algorithm:
Thus, under the old system, it was possible to be "excellent" with a performance index of 90 or if you met 20 of the 26 standards as long as your value-add scores indicated "above expected" growth for two years running. Under the new system, that will no longer be possible.
ANNUAL MEASURABLE OBJECTIVES
The latest edition to the eduspeak vocabulary is "Annual Measurable Objectives" or AMO's.
The AMO metric is designed to replace the previous level of proficiency that was required by No Child Left Behind. NCLB required, by 2014, that 100% of kids achieve proficiency. Scholars can argue whether that was ever a realistic goal, but states eagerly took the money offered by the federal government back in 2001 and then back loaded the proficiency requirements. Fast forward 10 years, and as the mandatory levels of proficiency started to increase as we got closer to 2014, panic set it. It is fair to say that the entire purpose of the waiver request is to relieve Ohio of the burden of a 100% proficiency requirement and the associated penalties when they are not met. The new goals, the "AMO's" are derived by taking the percentage of Ohio's kids that are currently deemed as not proficient in reading (18.1%), establishing a somewhat arbitrary goal of getting half of them to proficient within 6 years (e.g. 9% additional proficiency in 6 years) and using equal increments, so the bar is moved by 1.5% each year. This would then apply to all subgroups. A similar methodology is employed for the graduation percentage component of the annual measurable objectives. A somewhat complex calculation than goes into arriving at a final letter grade for the achievement gap component of the state report card.
The big difference between AYP and AMO's is this. In order for AYP to negatively impact a final rating on the existing report card, you would have to miss AYP in the SAME subgroup for three (3) consecutive years. Of course, given the requirement to be 100% proficient, absent the waiver, that would happen to every district sooner or later, but not for several years. With the new methodology, AMO's are a full partner in the calculation and the failure to meet the annual objectives could impact your rating in the current year.
This new system will be in place, assuming the legislature goes along, for the current year (2011-2012), so it is important to understand the differences between the old methodology and the new methodology, but it is also important to understand that the State Report Card will be changing fairly dramatically a few years from now to meet the demands of the assessment component of the common core.
The new methodology may therefore have a life span of a few years until the Common Core assessments kick in. Where this leaves Worthington is a bit unknown but as with most things, knowledge is power so understanding the methodology behind the new report card should give us a leg up on figuring out whether Worthington will be "A" rated or, if not, why not.
Saturday, March 17, 2012
Grade inflation in the state of Ohio.
This post is about grade inflation in the state and the broader question of whether the rankings that districts receive on the state report card is truly reflective of the "quality" of the school district. In this context, grade inflation refers to the elevation, over time, of Ohio's School District Rankings and the number of districts receiving top marks from the state. The first thing we need to determine is whether grade inflation is real. Over the last year or so, an increasing number of articles, bloggers and reports seem to think so. Let's start with some statistics.
This was the breakdown in the 2004-2005 school year:
Where 18% of school districts were rated excellent and a combination of 67% were rated as excellent of effective.
Fast forward to 2011 where the chart looks like this:
Where 58% of school districts are now rated excellent or higher and an astounding 93% of school districts are considered effective.
Did Ohio really make all these academic gains or did the state manipulate the report card such that it now provides a distorted view of reality?
Let's look at a few other data points. The first comes to us from the Ohio Board of Regents where they track the success of Ohio graduates who go on to Ohio colleges and see how many of them require remediation - essentially, to retake high school courses when they get to college. If 93% of Ohio's School Districts were effective, you would anticipate this to be a relatively low number.
Unfortunately, it is not. NPR is reporting that 41% of freshman from Ohio High Schools have to take at least one remediation class once they get to college. The report goes on to say that an analysis of Ohio Board of Regents data shows that in nearly 80 high schools rated A or A+, at least half of students who enroll in Ohio public colleges must take remedial math and/or English.
That is an astounding statistic! Your local high school is rated "Excellent" or even "Excellent with Distinction" and yet, half of its graduates have to essentially retake high school classes in order to be successful in college, yet, this applies to 20% of those districts with that excellent rating.
The Ohio Association of Gifted Children produced a report that was a less subtle about this problem. The report, entitled "Grading on a Curve, the illusion of excellence in Ohio Schools" made the case that grade inflation is taking place and that the state was essentially lying to its citizens about the quality of its schools. Charts such as this one:
compares the number of children scoring "advanced" on the Ohio Achievement Test vs. the number of children scoring advanced on the National Assessment of Education Progress (NAEP).
Obviously, Ohio is reporting a much higher percentage of children as "Advanced" then what you find on the national test. Of course, while Ohio isn't any better or worse than many other states, the state is using terminology that would seem to imply that it is.
One of the more egregious examples of how Ohio has created an illusion of excellence according to the OAGC report is through the use of "cut scores". A cut score is defined as the passing grade a student must achieve on one of the proficiency tests. The number that gets reported is that percentage of children that have achieved the passing grade, but up until this article, the grade that was considered passing never got any attention. The cut scores for the Ohio Achievement Tests in 2011 are documented in this chart:
So, for example, to receive a passing grade for 6th grade reading, a student must answer 17 out of 49 questions correctly for a total of 35%. If 75% of the students accomplish that goal, the school or the district will be designated at having met that state standard. To receive a designation of "Advanced" in 5th grade math, a student must receive a score of 73%. The categories, proficient, accelerated and advanced all go into the calculation for the districts performance index.
The OAGC report and the Dispatch reporting has numerous other examples of why the State Report Card is not a meaningful indicator of school district quality, so much so that the state has decided to take action. The next blog post will cover the new state report card and whether what was done will adequately address the problem.
This was the breakdown in the 2004-2005 school year:
Fast forward to 2011 where the chart looks like this:
Where 58% of school districts are now rated excellent or higher and an astounding 93% of school districts are considered effective.
Did Ohio really make all these academic gains or did the state manipulate the report card such that it now provides a distorted view of reality?
Let's look at a few other data points. The first comes to us from the Ohio Board of Regents where they track the success of Ohio graduates who go on to Ohio colleges and see how many of them require remediation - essentially, to retake high school courses when they get to college. If 93% of Ohio's School Districts were effective, you would anticipate this to be a relatively low number.
Unfortunately, it is not. NPR is reporting that 41% of freshman from Ohio High Schools have to take at least one remediation class once they get to college. The report goes on to say that an analysis of Ohio Board of Regents data shows that in nearly 80 high schools rated A or A+, at least half of students who enroll in Ohio public colleges must take remedial math and/or English.
That is an astounding statistic! Your local high school is rated "Excellent" or even "Excellent with Distinction" and yet, half of its graduates have to essentially retake high school classes in order to be successful in college, yet, this applies to 20% of those districts with that excellent rating.
The Ohio Association of Gifted Children produced a report that was a less subtle about this problem. The report, entitled "Grading on a Curve, the illusion of excellence in Ohio Schools" made the case that grade inflation is taking place and that the state was essentially lying to its citizens about the quality of its schools. Charts such as this one:
Obviously, Ohio is reporting a much higher percentage of children as "Advanced" then what you find on the national test. Of course, while Ohio isn't any better or worse than many other states, the state is using terminology that would seem to imply that it is.
One of the more egregious examples of how Ohio has created an illusion of excellence according to the OAGC report is through the use of "cut scores". A cut score is defined as the passing grade a student must achieve on one of the proficiency tests. The number that gets reported is that percentage of children that have achieved the passing grade, but up until this article, the grade that was considered passing never got any attention. The cut scores for the Ohio Achievement Tests in 2011 are documented in this chart:
So, for example, to receive a passing grade for 6th grade reading, a student must answer 17 out of 49 questions correctly for a total of 35%. If 75% of the students accomplish that goal, the school or the district will be designated at having met that state standard. To receive a designation of "Advanced" in 5th grade math, a student must receive a score of 73%. The categories, proficient, accelerated and advanced all go into the calculation for the districts performance index.
The OAGC report and the Dispatch reporting has numerous other examples of why the State Report Card is not a meaningful indicator of school district quality, so much so that the state has decided to take action. The next blog post will cover the new state report card and whether what was done will adequately address the problem.
Monday, February 6, 2012
Value-Add based Teacher Evaluations
This is an interesting and unprecedented time for reform minded school board members and district administrators. One of the reforms being pushed by education policy specialists is to have meaningful teacher evaluations. These evaluations will (eventually) inform everything from personnel decisions to hiring decisions to compensation decisions. It is clearly an idea whose time has come. In Ohio, House Bill 153 mandates teachers to be evaluated according to state-ordered criteria and further indicates that 50% of that evaluation has to be from the results of student growth measures meaning, I suppose, standardized tests where they exist or other assessments where they do not. Student growth measures is eduspeak for "Value Add" which is itself eduspeak for the concept of trying to determine the impact that a school district, a school or a teacher had on student growth when you filter out all other contributing factors such as socio-economic condition.
In Worthington, work on eventual compliance with this mandate is being done through the "Race to the Top (RTTT) committee" on which I've represented the Worthington Board of Education for 18 months now. I've concluded that to some extent, compliance with the mandate will be difficult because filtering out all contributing factors other than the impact of a specific educator on student growth will prove to be quite elusive for a multitude of reasons. I wanted to post a blog entry to talk about some of the problems with using student growth measures (AKA Value Add) but someone beat me to it. The following is a blog post from David B. Cohen. I believe it captures the elusive nature of fairly evaluating teachers using Value Add. Note that the specific examples are from California, meaning California law and California standards, but Ohio would have similar, though not identical concerns. None of this is to say that Value Add cannot one day be used for teacher evaluations, just that we need to firm up the process of filtering out extraneous material to get at the "real" value-add.
The source material can be found here:
http://accomplishedcaliforniateachers.wordpress.com/2011/11/02/vam-on-trial/
Turning the Tables: VAM on Trial
November 2, 2011
Los Angeles Unified School District is embroiled in negotiations over teacher evaluations, and will now face pressure from outside the district intended to force counter-productive teacher evaluation methods into use. Yesterday, I read this Los Angeles Times article about a lawsuit to be filed by an unnamed “group of parents and education advocates.” The article notes that, “The lawsuit was drafted in consultation with EdVoice, a Sacramento-based group. Its board includes arts and education philanthropist Eli Broad, former ambassador Frank Baxter and healthcare company executive Richard Merkin.” While the defendant in the suit is technically LAUSD, the real reason a lawsuit is necessary according to the article is that “United Teachers Los Angeles leaders say tests scores are too unreliable and narrowly focused to use for high-stakes personnel decisions.” Note that, once again, we see a journalist telling us what the unions say and think, without ever, ever bothering to mention why, offering no acknowledgment that the bulk of the research and the three leading organizations for education research and measurement (AERA, NCME, and APA) say the same thing as the union (or rather, the union is saying the same thing as the testing expert). Upon what research does the other side base arguments in favor of using test scores and “value-added” measurement (VAM) as a legitimate measurement of teacher effectiveness? They never answer, but the debate somehow continues ad nauseum.
It’s not that the plaintiffs in this case are wrong about the need to improve teacher evaluations. Accomplished California Teachers has published a teacher evaluation report that has concrete suggestions for improving evaluations as well, and we are similarly disappointed in the implementation of the Stull Act, which has been allowed to become an empty exercise in too many schools and districts.
Over at EdWeek, Stephen Sawchuk picked up the story and wondered if this action is a sign of things to come – litigating education policy. On one hand, I hate to think that we would resort to the courts to settle matters that can and should be addressed by professional educators based on an understanding of research and best professional practices. On the other hand, I figured a good defense attorney would shred these plaintiffs and the credibility of their cause. It will be interesting to see the actual language in their lawsuit, but the broader concept is already familiar. It’s just never been given a courtroom treatment that I know of. So, I’ve taken the liberty of dreaming up the court transcript ahead of time (using Q: for the defense attorney’s questions and A: for the plaintiff’s answers). Enjoy this cross-examination.
Q: You are demanding that LAUSD use measures of student growth in teacher evaluations, is that correct?
A: Yes.
Q: And you believe that student test scores are a measure of growth that would reflect teaching quality, correct?
A: Yes.
Q: If LAUSD were to adopt a policy that attributes the growth or lack of growth in student test scores to the student’s teacher, and uses the scores of all students to evaluate the teacher’s effectiveness, you would drop this lawsuit, is that correct?
A: Yes.
Q: How often are these tests administered?
A: Once per year.
Q: And the district has no way of knowing if the student’s performance on that day reflects the student’s ability or perhaps reflects some trauma, distress, boredom, distraction, or rebelliousness?
A: No.
Q: And for students who have changed schools, or changed teachers during the year, there’s no way to factor that into the analysis of data when a student simply shows up on one roster or another, right?
A: That could be adjusted.
Q: There’s no study that would guide you in how to do that with any accuracy, is there?
A: I don’t know.
Q: No evidence that a move at the mid-point of the year gives each teacher half the responsibility for the student’s learning, or that each week has a proportionate effect?
A: None that I know of.
Q: And would the degree of change in a certain classroom affect students in that classroom who had not been part of any change?
A: I don’t know.
Q: Does it seem likely that changing the students in a class would change the class itself and affect some of the students who had been there all along?
A: I guess so.
Q: But you would have no way of knowing which students were affected or how they were affected?
A: Not really, no.
Q: Now, if I were a high school English teacher, I would be responsible for teaching in four standards areas, but would the test cover all four of those areas?
A: No.
Q: How many does it cover?
A: Two.
Q: You’re including writing when you say “two” but in fact there’s no writing on the tests currently used, is there?
A: No.
Q: So more accurately, the test covers one out of the four standards areas?
A: Yes.
Q: Does the test cover every standard in reading?
A: No.
Q: So, you’re proposing basing a significant part of an English teacher’s evaluation, for example, on a test result that covers a small fraction of the standards?
A: It’s the only objective way.
Q: So your answer is yes?
A: Yes.
Q: By objective, you mean it’s the same for every student and teacher?
A: Yes.
Q: Does every teacher have an equal assignment, equal students, classes, and resources?
A: No.
Q: So, you do not concern yourself with objectivity in all of the factors affecting the teacher’s work, but you figure you can evaluate different teachers working with different students and different classes using the same test that covers only a fraction of their standards?
A: Yes.
Q: So is that an objective process for evaluation, or an arbitrary process with an objective element in it?
[Plaintiffs' counsel objects to argumentative question. Judge upholds the objection.]
Q: Do the words “objective” and “fair” have the same definition?
A: I couldn’t say.
Q: I could give an objective geometry test to every student in an algebra class, but would that be fair?
A: Okay, I see. They have different meanings.
Q: So your claim that the test is objective doesn’t cover the question of fairness, does it?
A: But it is fair!
Q: Please answer the question. A claim of objectivity is different from a claim of fairness, correct?
A: Yes.
Q: So an objective test may be inappropriate for certain students and therefore unfair, no matter how objective?
A: I would say that the test is fair to everyone.
Q: Like a geometry test for algebra students?
A: Well, no.
Q: Does a student’s linguistic skill relate to their success in a test that requires use of language?
A: Of course.
Q: So a test given in an unfamiliar language might yield a result that reflects linguistic confusion rather than conceptual confusion, or poor teaching?
A: We could adjust for language in a teacher’s evaluation.
Q: In what way?
A: If the student is still learning English their scores could be separated out.
Q: What if a student did well on the test despite being new to the language?
A: Well, we can’t just use the scores that help the teacher. We have to be fair.
Q: You mean objective?
A: Yes.
Q: Because actually, it would be fair to use the results that are valid and exclude the results that are invalid. Are you suggesting that such a determination could be made for each student, or that we should come up with a single formula and stick to it?
A: Just use a single formula.
Q: So regardless of the student’s actual linguistic knowledge, you would suggest making assumptions based on a certain number of years for students to learn enough academic English.
A: That would be logical.
Q: No matter the variables in the student’s instruction in English or the amount of time it actually takes them to learn English?
A: It’s the only fair way.
Q: Fair, or objective?
A: Objective.
Q: Objective regarding the student’s knowledge and skill, or objective regarding only measures of time?
A: Time.
Q: Is it fair to use value-added measurements to rank teachers even when numerous studies show that it is a volatile measure with error rates exceeding 25%?
A: It would only be one of multiple measures.
Q: That wasn’t my question. Is it fair to use an error-prone measure?
A: It’s not fair to exclude student performance from evaluations.
Q: Your Honor, would you instruct the witness to answer the question?
A: I’ll answer. It may not always be fair in every case, but no method is perfect.
Q: You’re suing the Los Angeles Unified School District to compel them to use a teacher evaluation method that is prone to errors and unfair to perhaps a quarter of the teachers evaluated in this manner, is that correct?
A: Yes! The alternative is the status quo, which is intolerable.
Q: But there are thriving, high-quality schools around the U.S. and around the world that are not using value-added measures. Doesn’t that prove that there are alternatives to the LAUSD status quo that are something other than the remedy you seek to impose?
[Plaintiffs' counsel objects to argumentative question. Judge upholds the objection.]
Q: Have you heard of the National Council for Measurement in Education, the American Psychology Association, the American Education Research Association?
A: Yes.
Q: Are you aware of their position on the lack of validity in using tests designed for one purpose and then used for another purpose?
A: More or less.
Q: I’m quoting from their joint position statement on this topic: “Tests valid for one use may be invalid for another. Each separate use of a high-stakes test, for individual certification, for school evaluation, for curricular improvement, for increasing student motivation, or for other uses requires a separate evaluation of the strengths and limitations of both the testing program and the test itself.” Does that sound familiar to you?
A: More or less.
Q: In other words, you’ve heard this argument before?
A: Yes.
Q: Is it fair to say that these are the three leading organizations for educational measurement and research?
A: I suppose so.
Q: Are you a professional organization for educational research and measurement?
A: No.
Q: Do you think it’s advisable, or even responsible, to ignore the policy position of these leading organizations?
A: But we know that teachers are the most important in-school factor on student performance!
Q: Okay, no argument there. But you have no basis upon which to argue against the validity issues raised in that quote, do you?
A: No.
Q: Now, taking up your contention that the teacher is the most important in-school factor, could you say most important out of how many factors?
A: No.
Q: You don’t know how many factors influence student performance?
A: No.
Q: If I threw out a number, like five, would you guess that it’s too low, too high, or about right?
A: That sounds too low.
Q: How about ten?
A: I don’t know, that might be right.
Q: Fifteen?
A: Maybe.
Q: Just hypothetically, could we proceed on the assumption there are ten factors in schools, other than teachers, that affect student performance?
A: Okay, yes.
Q: Would you expect every factor to have the same influence on every student, or would some factors have strong influences on one student and almost no influence on another student?
A: It would vary.
Q: If you wanted to design a fair formula, you would take those ten factors into account?
A: Yes.
Q: Even though you can’t say for sure how much each factor affects the student?
A: Yes.
Q: You can’t even say with certainty that a specific factor has any effect on a certain student or group of students?
A: No.
Q: So, let’s assume that each of those ten factors could play out in only two different ways: how many possible combinations do we have for each student?
A: Twenty.
Q: I’m sorry to correct your math, but actually, that would be ten-squared, or one-hundred possibilities.
A: Oh, yes, one hundred, I see.
Q: But we don’t know for sure how many factors to consider and what they are. And if we could actually identify fifteen variables instead of ten, and if each variable could play out in three different ways, would it surprise you to know that there would be 3,375 possible combinations?
A: That sounds like a lot, but you’re just playing with numbers.
Q: “Just playing with numbers.” I see. So just because something is true mathematically or statistically, it doesn’t necessarily translate into an actionable policy?
A: That’s not what I said.
Q: Of course you wouldn’t say that. Your case is predicated on the idea that because you can make value-added calculations that show some teachers are less effective than others, it therefore makes sense to use the numbers in policy that leads to the outcomes you want. Though again, the actual experts in educational measurement would warn against that, correct?
[Plaintiffs' counsel objects to argumentative question. Judge upholds the objection.]
Q: That’s what you need to do if you use test scores and value-added measures in teacher evaluation, isn’t it? Play with the numbers? You would need to come up with a formula that makes certain assumptions about the effect of each factor, even though you can’t test your assumptions?
A: They’ve been researched!
Q: But you just said that we can’t assume factors are the same for each student – or did you mean that these students in this hypothetical school will have been researched before any formulas are applied to them?
A: No.
Q: Okay, to be fair, let’s assume that we can come up with a formula for each of these individual factors. Wouldn’t it also be necessary to know about the interactions of the variables?
A: What do you mean?
Q: Well, perhaps we can apply a statistical control for homelessness, another to control for the time of day that the student studies a certain subject, and another to control for the change from last year’s 50-minute class periods to this year’s 90-minute class periods. Is it likely that there is any research on the effects for homeless students in longer classes at different times of day?
A: No.
Q: So when we combine factors, we not only make assumptions about each one, but also assume that these factors do not influence each other in any way, is that right?
A: You can’t study every little thing.
Q: So, if this were a medicine, you’d be comfortable saying that we have plenty of science about the ingredients and we don’t need to study them in this particular combination in order to assume the effects the medicine will have?
A: I don’t know anything about medicine.
Q: Have you ever been a teacher?
A: No.
Q: Thank you. No further questions.
Advertisement
In Worthington, work on eventual compliance with this mandate is being done through the "Race to the Top (RTTT) committee" on which I've represented the Worthington Board of Education for 18 months now. I've concluded that to some extent, compliance with the mandate will be difficult because filtering out all contributing factors other than the impact of a specific educator on student growth will prove to be quite elusive for a multitude of reasons. I wanted to post a blog entry to talk about some of the problems with using student growth measures (AKA Value Add) but someone beat me to it. The following is a blog post from David B. Cohen. I believe it captures the elusive nature of fairly evaluating teachers using Value Add. Note that the specific examples are from California, meaning California law and California standards, but Ohio would have similar, though not identical concerns. None of this is to say that Value Add cannot one day be used for teacher evaluations, just that we need to firm up the process of filtering out extraneous material to get at the "real" value-add.
The source material can be found here:
http://accomplishedcaliforniateachers.wordpress.com/2011/11/02/vam-on-trial/
Turning the Tables: VAM on Trial
November 2, 2011
Los Angeles Unified School District is embroiled in negotiations over teacher evaluations, and will now face pressure from outside the district intended to force counter-productive teacher evaluation methods into use. Yesterday, I read this Los Angeles Times article about a lawsuit to be filed by an unnamed “group of parents and education advocates.” The article notes that, “The lawsuit was drafted in consultation with EdVoice, a Sacramento-based group. Its board includes arts and education philanthropist Eli Broad, former ambassador Frank Baxter and healthcare company executive Richard Merkin.” While the defendant in the suit is technically LAUSD, the real reason a lawsuit is necessary according to the article is that “United Teachers Los Angeles leaders say tests scores are too unreliable and narrowly focused to use for high-stakes personnel decisions.” Note that, once again, we see a journalist telling us what the unions say and think, without ever, ever bothering to mention why, offering no acknowledgment that the bulk of the research and the three leading organizations for education research and measurement (AERA, NCME, and APA) say the same thing as the union (or rather, the union is saying the same thing as the testing expert). Upon what research does the other side base arguments in favor of using test scores and “value-added” measurement (VAM) as a legitimate measurement of teacher effectiveness? They never answer, but the debate somehow continues ad nauseum.
It’s not that the plaintiffs in this case are wrong about the need to improve teacher evaluations. Accomplished California Teachers has published a teacher evaluation report that has concrete suggestions for improving evaluations as well, and we are similarly disappointed in the implementation of the Stull Act, which has been allowed to become an empty exercise in too many schools and districts.
Over at EdWeek, Stephen Sawchuk picked up the story and wondered if this action is a sign of things to come – litigating education policy. On one hand, I hate to think that we would resort to the courts to settle matters that can and should be addressed by professional educators based on an understanding of research and best professional practices. On the other hand, I figured a good defense attorney would shred these plaintiffs and the credibility of their cause. It will be interesting to see the actual language in their lawsuit, but the broader concept is already familiar. It’s just never been given a courtroom treatment that I know of. So, I’ve taken the liberty of dreaming up the court transcript ahead of time (using Q: for the defense attorney’s questions and A: for the plaintiff’s answers). Enjoy this cross-examination.
Q: You are demanding that LAUSD use measures of student growth in teacher evaluations, is that correct?
A: Yes.
Q: And you believe that student test scores are a measure of growth that would reflect teaching quality, correct?
A: Yes.
Q: If LAUSD were to adopt a policy that attributes the growth or lack of growth in student test scores to the student’s teacher, and uses the scores of all students to evaluate the teacher’s effectiveness, you would drop this lawsuit, is that correct?
A: Yes.
Q: How often are these tests administered?
A: Once per year.
Q: And the district has no way of knowing if the student’s performance on that day reflects the student’s ability or perhaps reflects some trauma, distress, boredom, distraction, or rebelliousness?
A: No.
Q: And for students who have changed schools, or changed teachers during the year, there’s no way to factor that into the analysis of data when a student simply shows up on one roster or another, right?
A: That could be adjusted.
Q: There’s no study that would guide you in how to do that with any accuracy, is there?
A: I don’t know.
Q: No evidence that a move at the mid-point of the year gives each teacher half the responsibility for the student’s learning, or that each week has a proportionate effect?
A: None that I know of.
Q: And would the degree of change in a certain classroom affect students in that classroom who had not been part of any change?
A: I don’t know.
Q: Does it seem likely that changing the students in a class would change the class itself and affect some of the students who had been there all along?
A: I guess so.
Q: But you would have no way of knowing which students were affected or how they were affected?
A: Not really, no.
Q: Now, if I were a high school English teacher, I would be responsible for teaching in four standards areas, but would the test cover all four of those areas?
A: No.
Q: How many does it cover?
A: Two.
Q: You’re including writing when you say “two” but in fact there’s no writing on the tests currently used, is there?
A: No.
Q: So more accurately, the test covers one out of the four standards areas?
A: Yes.
Q: Does the test cover every standard in reading?
A: No.
Q: So, you’re proposing basing a significant part of an English teacher’s evaluation, for example, on a test result that covers a small fraction of the standards?
A: It’s the only objective way.
Q: So your answer is yes?
A: Yes.
Q: By objective, you mean it’s the same for every student and teacher?
A: Yes.
Q: Does every teacher have an equal assignment, equal students, classes, and resources?
A: No.
Q: So, you do not concern yourself with objectivity in all of the factors affecting the teacher’s work, but you figure you can evaluate different teachers working with different students and different classes using the same test that covers only a fraction of their standards?
A: Yes.
Q: So is that an objective process for evaluation, or an arbitrary process with an objective element in it?
[Plaintiffs' counsel objects to argumentative question. Judge upholds the objection.]
Q: Do the words “objective” and “fair” have the same definition?
A: I couldn’t say.
Q: I could give an objective geometry test to every student in an algebra class, but would that be fair?
A: Okay, I see. They have different meanings.
Q: So your claim that the test is objective doesn’t cover the question of fairness, does it?
A: But it is fair!
Q: Please answer the question. A claim of objectivity is different from a claim of fairness, correct?
A: Yes.
Q: So an objective test may be inappropriate for certain students and therefore unfair, no matter how objective?
A: I would say that the test is fair to everyone.
Q: Like a geometry test for algebra students?
A: Well, no.
Q: Does a student’s linguistic skill relate to their success in a test that requires use of language?
A: Of course.
Q: So a test given in an unfamiliar language might yield a result that reflects linguistic confusion rather than conceptual confusion, or poor teaching?
A: We could adjust for language in a teacher’s evaluation.
Q: In what way?
A: If the student is still learning English their scores could be separated out.
Q: What if a student did well on the test despite being new to the language?
A: Well, we can’t just use the scores that help the teacher. We have to be fair.
Q: You mean objective?
A: Yes.
Q: Because actually, it would be fair to use the results that are valid and exclude the results that are invalid. Are you suggesting that such a determination could be made for each student, or that we should come up with a single formula and stick to it?
A: Just use a single formula.
Q: So regardless of the student’s actual linguistic knowledge, you would suggest making assumptions based on a certain number of years for students to learn enough academic English.
A: That would be logical.
Q: No matter the variables in the student’s instruction in English or the amount of time it actually takes them to learn English?
A: It’s the only fair way.
Q: Fair, or objective?
A: Objective.
Q: Objective regarding the student’s knowledge and skill, or objective regarding only measures of time?
A: Time.
Q: Is it fair to use value-added measurements to rank teachers even when numerous studies show that it is a volatile measure with error rates exceeding 25%?
A: It would only be one of multiple measures.
Q: That wasn’t my question. Is it fair to use an error-prone measure?
A: It’s not fair to exclude student performance from evaluations.
Q: Your Honor, would you instruct the witness to answer the question?
A: I’ll answer. It may not always be fair in every case, but no method is perfect.
Q: You’re suing the Los Angeles Unified School District to compel them to use a teacher evaluation method that is prone to errors and unfair to perhaps a quarter of the teachers evaluated in this manner, is that correct?
A: Yes! The alternative is the status quo, which is intolerable.
Q: But there are thriving, high-quality schools around the U.S. and around the world that are not using value-added measures. Doesn’t that prove that there are alternatives to the LAUSD status quo that are something other than the remedy you seek to impose?
[Plaintiffs' counsel objects to argumentative question. Judge upholds the objection.]
Q: Have you heard of the National Council for Measurement in Education, the American Psychology Association, the American Education Research Association?
A: Yes.
Q: Are you aware of their position on the lack of validity in using tests designed for one purpose and then used for another purpose?
A: More or less.
Q: I’m quoting from their joint position statement on this topic: “Tests valid for one use may be invalid for another. Each separate use of a high-stakes test, for individual certification, for school evaluation, for curricular improvement, for increasing student motivation, or for other uses requires a separate evaluation of the strengths and limitations of both the testing program and the test itself.” Does that sound familiar to you?
A: More or less.
Q: In other words, you’ve heard this argument before?
A: Yes.
Q: Is it fair to say that these are the three leading organizations for educational measurement and research?
A: I suppose so.
Q: Are you a professional organization for educational research and measurement?
A: No.
Q: Do you think it’s advisable, or even responsible, to ignore the policy position of these leading organizations?
A: But we know that teachers are the most important in-school factor on student performance!
Q: Okay, no argument there. But you have no basis upon which to argue against the validity issues raised in that quote, do you?
A: No.
Q: Now, taking up your contention that the teacher is the most important in-school factor, could you say most important out of how many factors?
A: No.
Q: You don’t know how many factors influence student performance?
A: No.
Q: If I threw out a number, like five, would you guess that it’s too low, too high, or about right?
A: That sounds too low.
Q: How about ten?
A: I don’t know, that might be right.
Q: Fifteen?
A: Maybe.
Q: Just hypothetically, could we proceed on the assumption there are ten factors in schools, other than teachers, that affect student performance?
A: Okay, yes.
Q: Would you expect every factor to have the same influence on every student, or would some factors have strong influences on one student and almost no influence on another student?
A: It would vary.
Q: If you wanted to design a fair formula, you would take those ten factors into account?
A: Yes.
Q: Even though you can’t say for sure how much each factor affects the student?
A: Yes.
Q: You can’t even say with certainty that a specific factor has any effect on a certain student or group of students?
A: No.
Q: So, let’s assume that each of those ten factors could play out in only two different ways: how many possible combinations do we have for each student?
A: Twenty.
Q: I’m sorry to correct your math, but actually, that would be ten-squared, or one-hundred possibilities.
A: Oh, yes, one hundred, I see.
Q: But we don’t know for sure how many factors to consider and what they are. And if we could actually identify fifteen variables instead of ten, and if each variable could play out in three different ways, would it surprise you to know that there would be 3,375 possible combinations?
A: That sounds like a lot, but you’re just playing with numbers.
Q: “Just playing with numbers.” I see. So just because something is true mathematically or statistically, it doesn’t necessarily translate into an actionable policy?
A: That’s not what I said.
Q: Of course you wouldn’t say that. Your case is predicated on the idea that because you can make value-added calculations that show some teachers are less effective than others, it therefore makes sense to use the numbers in policy that leads to the outcomes you want. Though again, the actual experts in educational measurement would warn against that, correct?
[Plaintiffs' counsel objects to argumentative question. Judge upholds the objection.]
Q: That’s what you need to do if you use test scores and value-added measures in teacher evaluation, isn’t it? Play with the numbers? You would need to come up with a formula that makes certain assumptions about the effect of each factor, even though you can’t test your assumptions?
A: They’ve been researched!
Q: But you just said that we can’t assume factors are the same for each student – or did you mean that these students in this hypothetical school will have been researched before any formulas are applied to them?
A: No.
Q: Okay, to be fair, let’s assume that we can come up with a formula for each of these individual factors. Wouldn’t it also be necessary to know about the interactions of the variables?
A: What do you mean?
Q: Well, perhaps we can apply a statistical control for homelessness, another to control for the time of day that the student studies a certain subject, and another to control for the change from last year’s 50-minute class periods to this year’s 90-minute class periods. Is it likely that there is any research on the effects for homeless students in longer classes at different times of day?
A: No.
Q: So when we combine factors, we not only make assumptions about each one, but also assume that these factors do not influence each other in any way, is that right?
A: You can’t study every little thing.
Q: So, if this were a medicine, you’d be comfortable saying that we have plenty of science about the ingredients and we don’t need to study them in this particular combination in order to assume the effects the medicine will have?
A: I don’t know anything about medicine.
Q: Have you ever been a teacher?
A: No.
Q: Thank you. No further questions.
Advertisement
Sunday, January 29, 2012
District Boundaries
Periodically, the subject of district boundaries comes up. Why are any properties in the city of Columbus zoned for the Worthington School District. Most people think that Worthington is part of the "Win-Win" agreement but this is not correct. Instead, the boundaries between Worthington and Columbus were first drawn up by an agreement between the two school boards back in 1968. This is the documentation for that agreement.
As a result, Worthington does not have to periodically renew the "Win-Win" agreement as other districts do.
As a result, Worthington does not have to periodically renew the "Win-Win" agreement as other districts do.
Monday, November 21, 2011
Merit Pay
Last week, I was contacted by a government student at Worthington Kilbourne High School who was doing a paper on Merit Pay. He wanted to interview me as his teacher told him I was a proponent of Merit Pay. Because of the difficulty in scheduling, I told him to send me a list of questions and I thought the responses might be an interesting blog post. Here are the questions (and answers) from the "interview". The questions are in normal type and the responses are in bold.
1. How do you feel about the current, mostly seniority based system for determining teachers pay and why?
Before answering this question and some of the others, I think we need to reach some understanding of why people get paid what they get paid. Why does Mark Sanchez, QB for the New York Jets, make 15 million dollars per year while the guy selling beer in the stands makes $10/hour. How and why do corporations pay what they pay.
This is a complicated question but the short answer is that in the United States and in most capitalist societies, there is a law of supply and demand. If the New York Jets could hire a similar quality quarterback for less than they are paying Sanchez, they would do it. If you couldn't find workers willing to sell beer in the stands for less than $20/hour, you would have to pay that much to attract workers.
Now let's look at teachers. How much do we have to pay to attract quality teachers into the profession and retain them once they are employed? There answer is - no one knows because a free market does not exist in the education profession. In a perfect world, since we are using public taxpayer dollars, we would pay exactly what we must to attract the quality we desire, no more and no less. Lacking a free market, there is absolutely no correlation between what you spend and what you get. You can spend $90,000/year and get a horrible teacher who makes that much money because they have been there 30 years, and you can spend $40,000 to get a fantastic teacher who works 5 times as hard but makes that much money because they just started teaching. Furthermore, because there is no free market, experienced, quality teachers cannot go to other districts and negotiate their own raises. In fact, after about 10 years in the business, teachers are forced to stay in their current districts or take a pay cut.
There is also the reality that some teaching jobs are harder to fill than others. In Worthington, we needed to hire a high school physics teacher. We received 3 applications of which we thought 1 was qualified. Each year, we also need to hire elementary school classroom teachers. We get literally hundreds, perhaps thousands of applications for each one of these positions. Clearly, this is because the person qualified to teach high school physics is also qualified to work in the private sector, and those jobs pay more so few qualified individuals want to go into teaching. I want to pay high school physics teachers more so I can attract them to Worthington but I am not permitted to do so by the seniority system. Likewise, I can attract quality elementary school teachers for much less than I am paying now. If I was permitted to do so, I would be able to employ more teachers, lower class sizes and increase offerings for students.
With that as background, the answer to your first question is that I think the seniority system for determining pay for teachers does a disservice to the taxpayers, the students and the teachers. We should be able to pay great teachers more, we should let market forces work for determining how much taxpayers must pay to attract quality and we should be able to provide the best education experience for the students without the artificial constraints imposed by the seniority system. Please note, however, that this first question did not address "Merit Pay". There are two concepts which are sometimes confused. The first is "Merit Pay", the ability to pay teachers extra for better teaching or for a better quantifiable result, the second is "Differentiated Compensation" which is the ability to pay teachers based on supply and demand or for quality.
2. Can teachers performance be effectively evaluated by testing? Why or why not?
Not completely and sometimes, not at all, for a variety of different reasons. Whole books have been written on this subject. Standardized tests will provide some insight into how well a teacher performs their task, but there are a multitude of other factors that one must consider when determining teacher quality. I'll offer two (fictitious) examples to illustrate the point. Let's say that you have an Algebra 2 teacher at WKHS. In her class are kids who took Algebra 1 from a variety of teachers at McCord, some good and some not-so-good. The kids who did not have quality instruction in Algebra 1 will not do well in Algebra 2 through no fault of the Algebra 2 teacher. The second example is even more obvious. Some kids are not prepared to go to school each morning. Their parents are struggling to make a living, they don't have time to help the kids with homework, they don't have internet access at home and they can barely afford breakfast. Other kids have the best of everything, including tutors if they need extra help. Any attempt at evaluating teachers must take these factors into consideration.
In addition, there are some areas where testing is subjective. For example, how would you judge a theatre teacher or a music teacher lacking standardized tests. In these situations, other metrics would need to be used. Finally, there are a range of categories where some teachers excel in ways that don't show up on tests. For example, a student may reach out to a teacher in a time of crisis in their life and the teacher's actions might have kept the student from making a serious mistake, getting on drugs or worse. Some teachers get that connection with their students and others go through the motions. All of that must be part of a teacher's evaluation.
3. Many say that paying teachers based on years of teaching experience is unfair, and often results in lower pay for the best teachers. Is this a common problem?
To the extent that quality can be measured (a prerequisite for determining who the "best" teachers are), most studies show no correlation between quality and seniority, therefore, there is also no correlation between pay and quality. Since there is no correlation between pay and quality, it is fair to say that the "best" teachers will sometimes receive lower pay. In your own high school experience, did you have really good teachers who were just starting out, or really bad teachers who were older? How often this occurs would be difficult to measure but to the extent that it does happen, the pay differential would be unfair.
This is one such study that you can use for background information:
http://www.cedr.us/papers/working/CEDR%20WP%202011-1.2%20Teacher%20Layoffs%20(6-15-2011).pdf
4. The goal of merit pay is to reward the best teachers. Is this ideal and is this possible?
First, I do not agree with the premise of the question. The purpose of strategic compensation is to advance the goals of the organization. School districts are best served with quality teachers, therefore, the purpose of strategic compensation in a school district is to attract and retain quality teachers. Merit Pay is a component, a small component, of strategic compensation. Arguably, a district with merit pay might be more attractive to a quality teacher than a district without merit pay. The other thing to keep in mind is that as we look at teaching as a profession, we need to fix the fact that there is no career path for teachers that does not lead to administration. A teacher essentially has the same job in year 30 as they did in year 1. We need to define that career path and recognize that there are other ways to reward teachers than with money. For example, a "Master Teacher" designation, in addition to carrying a higher salary, might receive greater autonomy, the ability to serve on state committees, the ability to write and teach new courses and so forth. If we are just talking about merit pay as a small incremental for achieving good test scores in a given year than no, I don't think you accomplish much of anything. We need to look hard at the teaching profession and start treating teachers as the professionals that they are.
5. Could merit pay create hostile competition between teachers?
Sure, but that's no different than what some teachers feel today. If you are a great teacher working 70 hours a week doing everything possible for your students, and you are making half as much as the teacher next door who does the bare minimum, that might also foster negative feelings. It is human nature. As long as compensation is administered fairly and always with an eye towards advancing the goals of the organization, strategic compensation would yield better results than the status quo.
6. Some opponents of merit pay say that teachers don t become teachers for the money. Does this mean incentives for higher performance do not work?
Choice of career is a function of a large number of variables. I didn't become a software developer for the money either, I did it because I enjoyed working with computers and I had a passion to learn everything about them. Did that mean that as my career advanced, I didn't take advantage of opportunities to increase my pay? Of course not. The notion that teaching is the only profession whose practitioners are never motivated by money is silly.
There are volumes of research on the question of whether incentives for higher performance work and I assume as part of your study, you will cite work on either side of the argument. I think you are asking the wrong question. What you need to ask yourself is whether a system of strategic compensation works better than the existing system. With the existing system, there is no correlation between quality and pay. Don't take my word for it - try to find such a study. Research shows that we are pulling teachers from the bottom 33% of college classes. That would seem to indicate that we are not paying enough to attract the best possible candidates to the profession, or, that the best candidates for the profession have many options for career choice and the thought of going through 35 years of a career with no opportunity for advancement is not appealing. Either way, we can do better. One thing is for certain. With the existing system, we are surely limiting the pool of possible teachers to those who are NOT motivated by money and excluding those who are, and that can't be healthy for the profession.
7. Private schools are often cited as successful examples of merit pay. Could this success be translated to public schools?
I don't agree with the premise of the question, that private schools are often cited as successful examples of merit pay. There are many merit pay experiments occurring in both public and private schools, however, I am aware of only a few that use strategic compensation to attract and retain teachers. For example, 60 minutes did a report earlier this year on a public school in Manhattan that paid all of their teachers $125,000/year. They wanted to hire the absolute best. This would not be merit pay, this would be strategic compensation. This was the report:
http://www.cbsnews.com/stories/2011/03/10/60minutes/main20041733.shtml
8. Would merit pay distort the goals of teachers for better or for worse?
Sure, but no more so that the incentives and disincentives built into the current system. For example, to pass a state standard might require 75% of the kids to achieve a score of 60 on an achievement test. A teacher might spend more time with kids bordering on 60 to pull them over the top and spend less time on kids at the top or kids that have no chance to pass the test. That's why you have to be really careful when designing the goals. If I were to ask you what the goal of the Worthington School District should be, how would you answer? It is not an easy question because there would be many goals - selecting only one would be quite a challenge. Is it to produce National Merit Scholars? What about kids at the bottom? Is it to see that everyone graduates? How does that serve kids that are most gifted? See what I mean?
Bottom line is that merit pay, as an adjunct to strategic compensation can certainly further the goals of the organization but the organization itself needs to understand those goals first. If the organization understands what it is trying to accomplish, formulating a merit pay scheme to move everyone in that direction is possible.
9. Obviously there are a lot of compromises that must be made to create an effective system. Please describe what you view to be the best method for determining teacher salaries.
I'd need a bit more time to answer this one, but you get a sense of the alternatives from my earlier answers. Market forces must be allowed to prevail. We cannot not continue to treat teachers, one of the most important professions, the same as assembly line workers at GM. We need to use strategic compensation to attract and retain people in the profession. There is one question you should have asked that you did not. If the market says that to attract quality people, we need to pay more, can we afford to do so. My answer is yes, because if we can fix public education, I believe that a significant percentage of money currently being spent on social services would no longer be required. We are going to invest money anyway, either in education or in social services - I think society is best served by investing in education at the front end.
1. How do you feel about the current, mostly seniority based system for determining teachers pay and why?
Before answering this question and some of the others, I think we need to reach some understanding of why people get paid what they get paid. Why does Mark Sanchez, QB for the New York Jets, make 15 million dollars per year while the guy selling beer in the stands makes $10/hour. How and why do corporations pay what they pay.
This is a complicated question but the short answer is that in the United States and in most capitalist societies, there is a law of supply and demand. If the New York Jets could hire a similar quality quarterback for less than they are paying Sanchez, they would do it. If you couldn't find workers willing to sell beer in the stands for less than $20/hour, you would have to pay that much to attract workers.
Now let's look at teachers. How much do we have to pay to attract quality teachers into the profession and retain them once they are employed? There answer is - no one knows because a free market does not exist in the education profession. In a perfect world, since we are using public taxpayer dollars, we would pay exactly what we must to attract the quality we desire, no more and no less. Lacking a free market, there is absolutely no correlation between what you spend and what you get. You can spend $90,000/year and get a horrible teacher who makes that much money because they have been there 30 years, and you can spend $40,000 to get a fantastic teacher who works 5 times as hard but makes that much money because they just started teaching. Furthermore, because there is no free market, experienced, quality teachers cannot go to other districts and negotiate their own raises. In fact, after about 10 years in the business, teachers are forced to stay in their current districts or take a pay cut.
There is also the reality that some teaching jobs are harder to fill than others. In Worthington, we needed to hire a high school physics teacher. We received 3 applications of which we thought 1 was qualified. Each year, we also need to hire elementary school classroom teachers. We get literally hundreds, perhaps thousands of applications for each one of these positions. Clearly, this is because the person qualified to teach high school physics is also qualified to work in the private sector, and those jobs pay more so few qualified individuals want to go into teaching. I want to pay high school physics teachers more so I can attract them to Worthington but I am not permitted to do so by the seniority system. Likewise, I can attract quality elementary school teachers for much less than I am paying now. If I was permitted to do so, I would be able to employ more teachers, lower class sizes and increase offerings for students.
With that as background, the answer to your first question is that I think the seniority system for determining pay for teachers does a disservice to the taxpayers, the students and the teachers. We should be able to pay great teachers more, we should let market forces work for determining how much taxpayers must pay to attract quality and we should be able to provide the best education experience for the students without the artificial constraints imposed by the seniority system. Please note, however, that this first question did not address "Merit Pay". There are two concepts which are sometimes confused. The first is "Merit Pay", the ability to pay teachers extra for better teaching or for a better quantifiable result, the second is "Differentiated Compensation" which is the ability to pay teachers based on supply and demand or for quality.
2. Can teachers performance be effectively evaluated by testing? Why or why not?
Not completely and sometimes, not at all, for a variety of different reasons. Whole books have been written on this subject. Standardized tests will provide some insight into how well a teacher performs their task, but there are a multitude of other factors that one must consider when determining teacher quality. I'll offer two (fictitious) examples to illustrate the point. Let's say that you have an Algebra 2 teacher at WKHS. In her class are kids who took Algebra 1 from a variety of teachers at McCord, some good and some not-so-good. The kids who did not have quality instruction in Algebra 1 will not do well in Algebra 2 through no fault of the Algebra 2 teacher. The second example is even more obvious. Some kids are not prepared to go to school each morning. Their parents are struggling to make a living, they don't have time to help the kids with homework, they don't have internet access at home and they can barely afford breakfast. Other kids have the best of everything, including tutors if they need extra help. Any attempt at evaluating teachers must take these factors into consideration.
In addition, there are some areas where testing is subjective. For example, how would you judge a theatre teacher or a music teacher lacking standardized tests. In these situations, other metrics would need to be used. Finally, there are a range of categories where some teachers excel in ways that don't show up on tests. For example, a student may reach out to a teacher in a time of crisis in their life and the teacher's actions might have kept the student from making a serious mistake, getting on drugs or worse. Some teachers get that connection with their students and others go through the motions. All of that must be part of a teacher's evaluation.
3. Many say that paying teachers based on years of teaching experience is unfair, and often results in lower pay for the best teachers. Is this a common problem?
To the extent that quality can be measured (a prerequisite for determining who the "best" teachers are), most studies show no correlation between quality and seniority, therefore, there is also no correlation between pay and quality. Since there is no correlation between pay and quality, it is fair to say that the "best" teachers will sometimes receive lower pay. In your own high school experience, did you have really good teachers who were just starting out, or really bad teachers who were older? How often this occurs would be difficult to measure but to the extent that it does happen, the pay differential would be unfair.
This is one such study that you can use for background information:
http://www.cedr.us/papers/working/CEDR%20WP%202011-1.2%20Teacher%20Layoffs%20(6-15-2011).pdf
4. The goal of merit pay is to reward the best teachers. Is this ideal and is this possible?
First, I do not agree with the premise of the question. The purpose of strategic compensation is to advance the goals of the organization. School districts are best served with quality teachers, therefore, the purpose of strategic compensation in a school district is to attract and retain quality teachers. Merit Pay is a component, a small component, of strategic compensation. Arguably, a district with merit pay might be more attractive to a quality teacher than a district without merit pay. The other thing to keep in mind is that as we look at teaching as a profession, we need to fix the fact that there is no career path for teachers that does not lead to administration. A teacher essentially has the same job in year 30 as they did in year 1. We need to define that career path and recognize that there are other ways to reward teachers than with money. For example, a "Master Teacher" designation, in addition to carrying a higher salary, might receive greater autonomy, the ability to serve on state committees, the ability to write and teach new courses and so forth. If we are just talking about merit pay as a small incremental for achieving good test scores in a given year than no, I don't think you accomplish much of anything. We need to look hard at the teaching profession and start treating teachers as the professionals that they are.
5. Could merit pay create hostile competition between teachers?
Sure, but that's no different than what some teachers feel today. If you are a great teacher working 70 hours a week doing everything possible for your students, and you are making half as much as the teacher next door who does the bare minimum, that might also foster negative feelings. It is human nature. As long as compensation is administered fairly and always with an eye towards advancing the goals of the organization, strategic compensation would yield better results than the status quo.
6. Some opponents of merit pay say that teachers don t become teachers for the money. Does this mean incentives for higher performance do not work?
Choice of career is a function of a large number of variables. I didn't become a software developer for the money either, I did it because I enjoyed working with computers and I had a passion to learn everything about them. Did that mean that as my career advanced, I didn't take advantage of opportunities to increase my pay? Of course not. The notion that teaching is the only profession whose practitioners are never motivated by money is silly.
There are volumes of research on the question of whether incentives for higher performance work and I assume as part of your study, you will cite work on either side of the argument. I think you are asking the wrong question. What you need to ask yourself is whether a system of strategic compensation works better than the existing system. With the existing system, there is no correlation between quality and pay. Don't take my word for it - try to find such a study. Research shows that we are pulling teachers from the bottom 33% of college classes. That would seem to indicate that we are not paying enough to attract the best possible candidates to the profession, or, that the best candidates for the profession have many options for career choice and the thought of going through 35 years of a career with no opportunity for advancement is not appealing. Either way, we can do better. One thing is for certain. With the existing system, we are surely limiting the pool of possible teachers to those who are NOT motivated by money and excluding those who are, and that can't be healthy for the profession.
7. Private schools are often cited as successful examples of merit pay. Could this success be translated to public schools?
I don't agree with the premise of the question, that private schools are often cited as successful examples of merit pay. There are many merit pay experiments occurring in both public and private schools, however, I am aware of only a few that use strategic compensation to attract and retain teachers. For example, 60 minutes did a report earlier this year on a public school in Manhattan that paid all of their teachers $125,000/year. They wanted to hire the absolute best. This would not be merit pay, this would be strategic compensation. This was the report:
http://www.cbsnews.com/stories/2011/03/10/60minutes/main20041733.shtml
8. Would merit pay distort the goals of teachers for better or for worse?
Sure, but no more so that the incentives and disincentives built into the current system. For example, to pass a state standard might require 75% of the kids to achieve a score of 60 on an achievement test. A teacher might spend more time with kids bordering on 60 to pull them over the top and spend less time on kids at the top or kids that have no chance to pass the test. That's why you have to be really careful when designing the goals. If I were to ask you what the goal of the Worthington School District should be, how would you answer? It is not an easy question because there would be many goals - selecting only one would be quite a challenge. Is it to produce National Merit Scholars? What about kids at the bottom? Is it to see that everyone graduates? How does that serve kids that are most gifted? See what I mean?
Bottom line is that merit pay, as an adjunct to strategic compensation can certainly further the goals of the organization but the organization itself needs to understand those goals first. If the organization understands what it is trying to accomplish, formulating a merit pay scheme to move everyone in that direction is possible.
9. Obviously there are a lot of compromises that must be made to create an effective system. Please describe what you view to be the best method for determining teacher salaries.
I'd need a bit more time to answer this one, but you get a sense of the alternatives from my earlier answers. Market forces must be allowed to prevail. We cannot not continue to treat teachers, one of the most important professions, the same as assembly line workers at GM. We need to use strategic compensation to attract and retain people in the profession. There is one question you should have asked that you did not. If the market says that to attract quality people, we need to pay more, can we afford to do so. My answer is yes, because if we can fix public education, I believe that a significant percentage of money currently being spent on social services would no longer be required. We are going to invest money anyway, either in education or in social services - I think society is best served by investing in education at the front end.
Thursday, October 20, 2011
Wireless Communication Device Policy.
A few weeks ago, I asked Superintendent Tucker to look into the issue of whether our board policy (Page 73) that disallows the use of Wireless Communication Devices during the school day should be looked at. A WCD could be anything from an internet enable cellphone to an IPAD or other tablet device to a small netbook. I made the request because it makes no sense for kids to have to power down when they enter our buildings. We've spent a lot of money enabling WIFI in the buildings so perhaps it is time to maximize that investment.
As with most policies, there are pros and cons. The biggest objection is that the WCD will be misused, taking inappropriate photographs, texting (or worse) with friends, spending too much time on Facebook and perhaps even reading board member blog posts.
My response is that we should punish the behavior if inappropriate, but not ban the tool. Within a few years, WCD will be ubiquitous, spanning all age groups and income levels. It is the nature of technology. Furthermore, the world that our kids inherit will be filled with this kind of technology - isn't it best to learn how to use it in a controlled environment.
Superintendent Tucker commissioned a task force to look at more permissive policies employed by other school districts around the country. Several look promising. The consensus is that Worthington's policy, assuming we change it at all, should look something like this.
I see great value to the use of WCD in educational settings but I also understand that there are risks. There is also an equity issue as our district cannot afford to provide devices to all students, but many students have them already. We will simply not require them to be turned off when they enter our buildings. Over time, as the technology matures, the equity issue will disappear.
As always, our entire board values constituent input and any new policy will go through the normal process of a first reading at a board meeting (probably in December) and a vote in January. In the meantime, please feel free to share any thoughts on this topic on the blog or by contacting me directly.
As with most policies, there are pros and cons. The biggest objection is that the WCD will be misused, taking inappropriate photographs, texting (or worse) with friends, spending too much time on Facebook and perhaps even reading board member blog posts.
My response is that we should punish the behavior if inappropriate, but not ban the tool. Within a few years, WCD will be ubiquitous, spanning all age groups and income levels. It is the nature of technology. Furthermore, the world that our kids inherit will be filled with this kind of technology - isn't it best to learn how to use it in a controlled environment.
Superintendent Tucker commissioned a task force to look at more permissive policies employed by other school districts around the country. Several look promising. The consensus is that Worthington's policy, assuming we change it at all, should look something like this.
I see great value to the use of WCD in educational settings but I also understand that there are risks. There is also an equity issue as our district cannot afford to provide devices to all students, but many students have them already. We will simply not require them to be turned off when they enter our buildings. Over time, as the technology matures, the equity issue will disappear.
As always, our entire board values constituent input and any new policy will go through the normal process of a first reading at a board meeting (probably in December) and a vote in January. In the meantime, please feel free to share any thoughts on this topic on the blog or by contacting me directly.
Subscribe to:
Posts (Atom)