The case for a 5.9 mill levy.

On June 5, 2012, The Worthington Board of Education voted to send 3 levy options to the Franklin County Auditor for certification. The three options are:

1) A 5.9 mill levy with a 40 million dollar bond issue on 2 separate ballots costing $423/year
2) A 6.3 mill levy with a 40 million dollar bond issue on a single ballot costing $451/year
3) A 7.4 mill levy with a 40 million dollar bond issue on a single ballot cost  $530/year

All costs are calculated based on the average house value in Worthington of $234,000.

The first topic in any levy discussion is whether or not a levy is justified or whether it is possible to live within the confines of current revenue. Our treasurer will point out, correctly, that total revenues in FY13, the last year which can be forecasted with any accuracy given the vagaries of the state budget, are all of 3% higher than total revenues in FY08. The primary reason why is the loss of TPP reimbursements which, while anticipated, are still sizable. Whether the phase out continues past FY13 is the subject of some debate, but the phase out that has already occurred does partially justify a levy in 2012. Again turning to the forecast, we see an increase in expenditures of approximately 10.4% from FY08 to FY12. This works out to a 2.6% annualized increase during a period where inflation, while benign, was still around 1.75 and given the rapid increase in health care costs, that difference is certainly understandable. In the period from FY12 to FY16, the treasurer forecasts a total expenditure increase of 10.9%, an average of 2.72% while forecasting a revenue decrease at the same time. The revenue decrease is by no means certain, but I agree with the treasurer that it is more likely to happen than not. Given this dynamic, there is clearly a mathematical case that can be made for the levy.

By what process should a board of education come up with a levy amount? As a student of this process in Worthington and throughout the state, the methodology varies depending on local circumstances. In Worthington, for at least the last 11 years and inclusive of the levy cycles of 2001, 2004, 2006 and 2009, the methodology employed would be to calculate how long you wanted the levy to last (FY16 in this case), take the deficit at the end of that period (17,000,000 in this case), add a fudge factor, divide by how much a mill raises (1.7 million in this case) and finally, divide by the number of years of collection (3.5 in this case). The only real variable in this calculation has been the size of the fudge factor, so let’s talk about that. In math terms:

L = (D+F) / P / M

Where:

L is the Levy Amount
D is the deficit at the end of the year you are trying to balance (inclusive of existing reserve funds)
F is the Fudge Factor
P is the Years of Collection for the new levy
M is how much is raised by 1 mill


The fudge factor, known in financial circles as the cash balance reserve is the amount of cash you want to have available at all times for emergencies and to carry over into the next cycle. The policies vary widely, even in Central Ohio. Our friends in Hilliard, for example, have a board policy indicating a minimum of 10%. The Ohio Association of School Business Officials says a minimum of 60 days is justified. The GFOA concurs, saying 5 to 15% or 1 to 2 months should be sufficient. Dublin City Schools, in their recent decision to run a 6.94 mill operating levy is apparently content, according to the Columbus Dispatch, with a cash reserve of 3.4 million dollars at the end of the forecast period, or a little under 2%. Westerville’s stated policy is 30 days, or 8.2% (Page 32 of the link).


As we analyze the levy recommendations from our administration, it soon becomes clear that the only thing really under discussion, at least as far as the operating levy goes, is the size of the cash balance at the end of the levy period, which is FY16. The calculation is not difficult:


As the spreadsheet demonstrates, a 6.9 mill levy will leave us with around 24.2 million dollars in the bank at the end of the levy period whereas a 5.9 mill levy will leave us at 18.2 million dollars at the end of FY16 (Upper Chart), however, that does not tell the complete story because the district still maintains a contingency reserve of 3.118 million dollars and could therefore count as cash in the bank should the need arise to balance a budget, therefore, a 5.9 mill levy will leave us with 21.3 million dollars, a few dollars short of the official GFOA recommendation of 60 days (Lower Chart).

The GFOA recommendation is inclusive of a number of risk factors that wouldn’t necessarily apply to us. For example, a successful levy would include bond money, therefore, we would be protected from such exogenous events as roof repairs, blown boilers, buses failing and so forth. There are only a few events that would materially and negatively change the forecast.  The three events that can be game changers are – a really bad state budget that phases out TPP and/or foundation at a rate exceeding that projected in the forecast, a negotiated agreement with significantly higher increases than suggested by the forecast or an outflow of kids to charter schools should the legislature implement something like the original version of HB136. All of these events are possible, none are likely.

So let’s apply all this to the levy decision. While we haven't received a lot of input, the input we have received suggests taking as little as possible. Absent some kind of incremental levy, a 5.9 mill levy is the lowest amount that keeps us within GFOA guidelines in FY16, the end of the levy cycle. Plugging those numbers into our formula, we have:

L = (D+F) / P / M

Where D is the deficit of 17,046,595 - the existing contigency of 3,118,000
and F is 60 days of FY16 estimated expenses (60/365)*130,132,835


or L = (13,928,595+ 21,432,679) / 3.5 / 1707933 or   5.915 mills. I rounded that down to 5.9 mills.


The Treasurer's case for 6.9 mills has to do with our policy of "Reasonable Levys at Reasonable Intervals", the concept that our school district should be on a planned levy cycle. A planned levy cycle makes sense when you consider that school districts are flat funded (e.g. no incremental revenue) absent a new levy, unlike cities and states whose revenue is partially dependent on the income tax which will be a function of the economy. His argument is that if we go with 6.9 mills now, the next levy, in 2015, won't have to be higher than what Worthington is generally comfortable with, below 7 mills. I respectfully disagree. To take money from people who are living paycheck to paycheck (or don't have a paycheck) so that Worthington Schools can build up cash balances in excess of even what the GFOA would recommend and with a goal towards softening the next levy in 2015 seems like bad public policy to me. History demonstrates that the last 2 years of the five year forecast are difficult to project with accuracy, so to treat those assumptions as gospel and levy our residents based on that seems wrong. A 5.9 mill levy should leave our district with an appropriate safety net.

So that's it. The case for the 5.9 mill levy. Thoughts?

Levy options.

There seems to be quite a bit of confusion surrounding the various levy options under consideration by the Worthington School District administration and the Board of Education. This post is an attempt to demystify the options.

Before doing so, some background might be helpful. There are two distinct sets of "needs" that are going to be included in any funding request eventually. These two needs have two distinct checking accounts and a law that says that you can't commingle the funds. Let's first identify the needs.

First is the need for additional operating money. The vast majority of operating funds go to pay employee salaries and benefits, but these funds are also used to pay other day to day operating costs like electricity, natural gas, diesel fuel, paper and some larger ticket items like the tuition that Worthington pays to charter schools or community schools to educate kids in our community. To see where the money is being spent today, click here.

Second is the need for additional capital improvement money. This money is typically called "bond money" and can be used to pay for items (things) that will be in use for more than 5 years like technology, buses, roofs, painting, repaving, lockers, alarm systems and so forth. By law, this money cannot be used to pay for salaries and benefits. The administration has recommended that we borrow 40 million dollars over a period of 15-18 years. The 40 million dollar figure comes from the maximum amount that can be borrowed without increasing the current 3.8 mill burden on residents for capital improvements. Note that if we didn't borrow this money, the 3.8 mill burden would be 3.0 mills next year and eventually, as previous bonds are paid off, go to zero. I mention this because the levy campaigns like to highlight (correctly) that the bond component is "no additional millage", however, that doesn't mean that it's free, it means that the property tax decreases that would otherwise occur as principal is paid off won't occur. A writeup of what the district wants to spend  $40,000,000 on can be found here.

The question, and the confusion, come about because the law allows you to ask for both of these things with a single ballot question or with two ballot questions. Further complicating the issue is the fact that the law allows for the phasing in of operating money over a period of time (This is known as the incremental levy) but you cannot combine this with a bond request on a single ballot.

As a result, on June 5, 2012, the Board and the administration will meet to consider the following options.

1) An operating levy in combination with a bond levy on a single ballot question.

This option allows the administration to combine both requests in a single ballot question. Voters can either vote YES to both requests or NO to both requests and you only get one vote. The bond component will be 40 million dollars. The operating component is still unknown. The Treasurer has recommended 6.9 mills, or about $494/year on the average Worthington home of $234,000 dollars. You can calculate the amount for your own home here

2) An incremental levy and a bond issue on two ballot questions.

Many residents have contacted us asking for a phased in levy similar to what we did in 2009. The incremental levy in 2009 was 3.9 mills for 2010, 5.4 mills for 2011 and 6.9 mills for 2012 and each year thereafter. The Treasurer has indicated that an incremental levy of 4.9 mills in 2013, 5.9 mills in 2014 and 6.9 mills in 2015 would meet the operating needs of the district. The law does not permit you to combine these two issues on a single ballot question, therefore, two ballot questions would be required. Voters could vote YES on neither issue, one issue or the other issue, or both issues. You can calculate the tax for your own home, by year, here.

3) A "promised incremental levy (PIL)" and a bond issue on a single ballot question.

Some administrators, including the treasurer, do not believe the district should offer the opportunity to vote on each of the two needs separately. They are concerned that you might vote for one issue or the other issue and not both issues, but they also recognize that in this economy, phasing in the levy is also desirable. Again, the law does not allow the combination of an incremental levy and a bond issue on a single ballot. To get around this, we could offer a "Promised Incremental Levy" which takes advantage of the fact that the school district does not have to collect all of the millage it is allowed to collect. In fact, Worthington Schools has been doing this for 5 years now, collecting less millage than we are otherwise entitled. The Promised Incremental Levy would simply tell the community that in 2013, we would collect only a fraction, perhaps 70% of the 6.9 mills that would appear on the ballot. The next year, 2014, we would collect perhaps 85% and the final year and each year thereafter, we would collect the entire 100% of the 6.9 mills. Note that the increments still need to be discussed - the above is just for illustration. The PIL and the Bond issue could be combined in a single ballot question as in Option #1. One concern with the PIL is whether the community would believe that the Board and the Administration would keep this promise. Once again, Worthington has been collecting less millage than we are entitled to for 5 years now. In addition, the Treasurer said in a community conversation that he would resign before breaking the promise. Frankly, there is no way that the Board or any of its members, or our Superintendent or Treasurer would renege on a promise like this.

4) Run an Incremental Levy in November of 2012 and a Bond Levy in 2013.

The desire to run both the operating levy and the bond levy in November is a function of the difficulty of the campaign and the time and resources required, not the needs of the district. Nothing bad is going to happen if the bond money is not approved until the next election in May of 2013. I can certainly understand why the residents who volunteer for levy campaigns don't want to do this twice, however, it is an option and should be considered.

It is possible to run an incremental levy in November of 2012 (Option 2) and a bond issue in May of 2013. Obviously, voters could approve neither issue, one issue or both issues. The treasurer will point out that a bond issue in May of 2013 would not be "No additional millage" and he is correct, but only because the millage you are already paying for bonds is scheduled to decrease by around 0.8 mills. A bond issue passed in May of 2013 would restore the millage you are paying for capital improvements to 3.8 mills, just as if you passed the issue in 2012. There is no economic difference. Please note that the district does have a considerable cash reserve to handle any emergencies that may come up. In fact, there is about 7 million dollars that is earmarked for contingencies and/or emergencies.

So those are the four options. Please note that the amounts (6.9 mills operating, 40 million dollars bond) are the treasurer's recommendation. They may or may not be the final numbers. My thoughts on that in a future blog post.

The New State Report Card proposal.

In the waiver request filed with the federal government to excuse the state from the provisions of No Child Left Behind legislation, the state provided the roadmap for its current thinking with regard to the state report card. What the state of Ohio has decided to do, at least in the short term, is to maintain most of the existing metrics but simply grade downward in an effort to give less "Excellent" ratings on the report card. Whether their motivation was to combat grade inflation or to make their waiver request more appealing to the federal government, a new report card is coming to Ohio.

The new report card would be made up of 4 components and they are all graded separately and given equal weight in determining the final grade. The components are:

1) The scores on the proficiencies

2) The performance index

3) The value-add assessment

4) A new metric designed to measure the success a district has at eliminating the achievement gap

The overall grade is determined by averaging the grades on the 4 metrics.

Let's look at each in turn.

PROFICIENCIES

At least through 2015, the proficiency tests will remain largely as-is, cut scores and all,  so, for example, a district will continue to receive credit for passing the 6th grade reading proficiency if 75% of the students get a 35% on the test. If you pass 90% of the available proficiencies, your district receive an "A" on that particular metric with 80% required to get a "B".

PERFORMANCE INDEX

The performance index metric attempts to give the district credit for those students scoring in the "advanced" and "accelerated" range on the tests, but as we see from the charts in the previous posts, the cut off scores for advanced and accelerated remain at the same low levels that they were before, therefore, the performance index metric should be similarly unchanged. The difference is in the score required to receive an "A". If you receive a performance index of 90% of the maximum, or 108 out of a possible 120 points, you get your "A".  Only one school in Worthington, Evening Street Elementary, received a performance index in excess of 108. It's score was 108.8, however, we must remember that the performance index is only concerned with how many students clear the proficient, accelerated and advanced bar and not necessarily by how much those students passed the bar. Let's take an example from a school that would narrowly miss getting that magic 108 to receive the "A" on the performance index component.

Performance Index Calculation for Phoenix Middle School, Worthington OH

The example demonstrates that even with 95% of the scores being proficient and fully 2/3 being accelerated or advanced, it still isn't enough to earn that "A", apparently  because the state realizes that with the cut scores being as low as they are, "Accelerated" or "Advanced" simply isn't good enough any more.

VALUE ADD

The next component is "value add" and this is the description from the waiver request:

Value-Added: While performance scores demonstrate a student’s level of proficiency, Value-Added
measures the effects of schools on their students’ growth. It is calculated only for schools with students in any Grades 4-8. Ohio, using the SAS® at EVAAS® model computes a Value-Added measure for each school and district in English language arts and mathematics and reports whether the expected growth has been met (a year’s growth in a year’s time), exceeded (more than a year’s growth in a year’s time) or not met (less than a year’s growth in a year’s time).

Interested readers can learn more about value-add here.

For purposes of this discussion, value-add was not changed in the new state report card. What was changed was the impact that value-add could have on the overall grade. Prior to the new state report card methodology, value-add was used to modify the overall grade. This was the algorithm used:

Value Add impact on final report card grade

With the new methodology, value-add is a full partner in the calculation. If you exceed Value-Add, you get an "A" for that component and the "A" will get averaged in to the final grade. This chart shows the new algorithm:

Thus, under the old system, it was possible to be "excellent" with a performance index of 90 or if you met 20 of the 26 standards as long as your value-add scores indicated "above expected" growth for two years running. Under the new system, that will no longer be possible.

ANNUAL MEASURABLE OBJECTIVES

The latest edition to the eduspeak vocabulary is "Annual Measurable Objectives" or AMO's.

The AMO metric is designed to replace the previous level of proficiency that was required by No Child Left Behind. NCLB required, by 2014, that 100% of kids achieve proficiency. Scholars can argue whether that was ever a realistic goal, but states eagerly took the money offered by the federal government back in 2001 and then back loaded the proficiency requirements. Fast forward 10 years, and as the mandatory levels of proficiency started to increase as we got closer to 2014, panic set it.  It is fair to say that the entire purpose of the waiver request is to relieve Ohio of the burden of a 100% proficiency requirement and the associated penalties when they are not met. The new goals, the "AMO's" are derived by taking the percentage of Ohio's kids that are currently deemed as not proficient in reading (18.1%), establishing a somewhat arbitrary goal of getting half of them to proficient within 6 years (e.g. 9% additional proficiency in 6 years) and using equal increments, so the bar is moved by 1.5% each year. This would then apply to all subgroups. A similar methodology is employed for the graduation percentage component of the annual measurable objectives. A somewhat complex calculation than goes into arriving at a final letter grade for the achievement gap component of the state report card.

The big difference between AYP and AMO's is this.  In order for AYP to negatively impact a final rating on the existing report card, you would have to miss AYP in the SAME subgroup for three (3) consecutive years. Of course, given the requirement to be 100% proficient, absent the waiver, that would happen to every district sooner or later, but not for several years. With the new methodology, AMO's are a full partner in the calculation and the failure to meet the annual objectives could impact your rating in the current year.

This new system will be in place, assuming the legislature goes along,  for the current year (2011-2012), so it is important to understand the differences between the old methodology and the new methodology, but it is also important to understand that the State Report Card will be changing fairly dramatically a few years from now to meet the demands of the assessment component of the common core.

The new methodology may therefore have a life span of a few years until the Common Core assessments kick in. Where this leaves Worthington is a bit unknown but as with most things, knowledge is power so understanding the methodology behind the new report card should give us a leg up on figuring out whether Worthington will be "A" rated or, if not, why not.

Grade inflation in the state of Ohio.

This post is about grade inflation in the state and the broader question of whether the rankings that districts receive on the state report card is truly reflective of the "quality" of the school district. In this context, grade inflation refers to the elevation, over time, of Ohio's School District Rankings and the number of districts receiving top marks from the state. The first thing we need to determine is whether grade inflation is real. Over the last year or so, an increasing number of articles, bloggers and reports seem to think so. Let's start with some statistics.

This was the breakdown in the 2004-2005 school year:

Where 18% of school districts were rated excellent and a combination of 67% were rated as excellent of effective.

Fast forward to 2011 where the chart looks like this:

Where 58% of school districts are now rated excellent or higher and an astounding 93% of school districts are considered effective.

Did Ohio really make all these academic gains or did the state manipulate the report card such that it now provides a distorted view of reality?

Let's look at a few other data points. The first comes to us from the Ohio Board of Regents where they track the success of Ohio graduates who go on to Ohio colleges and see how many of them require remediation - essentially, to retake high school courses when they get to college. If 93% of Ohio's School Districts were effective, you would anticipate this to be a relatively low number.

Unfortunately, it is not. NPR is reporting that 41% of freshman from Ohio High Schools have to take at least one remediation class once they get to college. The report goes on to say that an analysis of Ohio Board of Regents data shows that in nearly 80 high schools rated A or A+, at least half of students who enroll in Ohio public colleges must take remedial math and/or English.

That is an astounding statistic!  Your local high school is rated "Excellent" or even "Excellent with Distinction" and yet, half of its graduates have to essentially retake high school classes in order to be successful in college, yet, this applies to 20% of those districts with that excellent rating.

The Ohio Association of Gifted Children produced a report that was a less subtle about this problem. The report, entitled "Grading on a Curve, the illusion of excellence in Ohio Schools" made the case that grade inflation is taking place and that the state was essentially lying to its citizens about the quality of its schools. Charts such as this one:

compares the number of children scoring "advanced" on the Ohio Achievement Test vs. the number of children scoring advanced on the National Assessment of Education Progress (NAEP).

Obviously, Ohio is reporting a much higher percentage of children as "Advanced" then what you find on the national test. Of course, while Ohio isn't any better or worse than many other states, the state is using terminology that would seem to imply that it is.

One of the more egregious examples of how Ohio has created an illusion of excellence according to the OAGC report is through the use of "cut scores". A cut score is defined as the passing grade a student must achieve on one of the proficiency tests. The number that gets reported is that percentage of children that have achieved the passing grade, but up until this article, the grade that was considered passing never got any attention. The cut scores for the Ohio Achievement Tests in 2011 are documented in this chart:

So, for example, to receive a passing grade for 6th grade reading, a student must answer 17 out of 49 questions correctly for a total of 35%.  If 75% of the students accomplish that goal, the school or the district will be designated at having met that state standard. To receive a designation of "Advanced" in 5th grade math, a student must receive a score of 73%. The categories, proficient, accelerated and advanced all go into the calculation for the districts performance index.

The OAGC report and the Dispatch reporting  has numerous other examples of why the State Report Card is not a meaningful indicator of school district quality, so much so that the state has decided to take action. The next blog post will cover the new state report card and whether what was done will adequately address the problem.

Value-Add based Teacher Evaluations

This is an interesting and unprecedented time for reform minded school board members and district administrators. One of the reforms being pushed by education policy specialists is to have meaningful teacher evaluations. These evaluations will (eventually)  inform everything from personnel decisions to hiring decisions to compensation decisions. It is clearly an idea whose time has come. In Ohio, House Bill 153 mandates teachers to be evaluated according to state-ordered criteria and further indicates that 50% of that evaluation has to be from the results of student growth measures meaning, I suppose, standardized tests where they exist or other assessments where they do not. Student growth measures is eduspeak for "Value Add" which is itself eduspeak for the concept of trying to determine the impact that a school district, a school or a teacher had on student growth when you filter out all other contributing factors such as socio-economic condition.

In Worthington, work on eventual compliance with this mandate is being done through the "Race to the Top (RTTT) committee" on which I've represented the Worthington Board of Education for 18 months now.  I've concluded that to some extent, compliance with the mandate will be difficult because filtering out all contributing factors other than the impact of a specific educator on student growth will prove to be quite elusive for a multitude of reasons. I wanted to post a blog entry to talk about some of the problems with using student growth measures (AKA Value Add) but someone beat me to it. The following is a blog post from David B. Cohen. I believe it captures the elusive nature of fairly evaluating teachers using Value Add. Note that the specific examples are from California, meaning California law and California standards, but Ohio would have similar, though not identical concerns. None of this is to say that Value Add cannot one day be used for teacher evaluations, just that we need to firm up the process of filtering out extraneous material to get at the "real" value-add.

The source material can be found here:

http://accomplishedcaliforniateachers.wordpress.com/2011/11/02/vam-on-trial/

Turning the Tables: VAM on Trial


November 2, 2011

Los Angeles Unified School District is embroiled in negotiations over teacher evaluations, and will now face pressure from outside the district intended to force counter-productive teacher evaluation methods into use. Yesterday, I read this Los Angeles Times article about a lawsuit to be filed by an unnamed “group of parents and education advocates.” The article notes that, “The lawsuit was drafted in consultation with EdVoice, a Sacramento-based group. Its board includes arts and education philanthropist Eli Broad, former ambassador Frank Baxter and healthcare company executive Richard Merkin.” While the defendant in the suit is technically LAUSD, the real reason a lawsuit is necessary according to the article is that “United Teachers Los Angeles leaders say tests scores are too unreliable and narrowly focused to use for high-stakes personnel decisions.” Note that, once again, we see a journalist telling us what the unions say and think, without ever, ever bothering to mention why, offering no acknowledgment that the bulk of the research and the three leading organizations for education research and measurement (AERA, NCME, and APA) say the same thing as the union (or rather, the union is saying the same thing as the testing expert). Upon what research does the other side base arguments in favor of using test scores and “value-added” measurement (VAM) as a legitimate measurement of teacher effectiveness? They never answer, but the debate somehow continues ad nauseum.

It’s not that the plaintiffs in this case are wrong about the need to improve teacher evaluations. Accomplished California Teachers has published a teacher evaluation report that has concrete suggestions for improving evaluations as well, and we are similarly disappointed in the implementation of the Stull Act, which has been allowed to become an empty exercise in too many schools and districts.

Over at EdWeek, Stephen Sawchuk picked up the story and wondered if this action is a sign of things to come – litigating education policy. On one hand, I hate to think that we would resort to the courts to settle matters that can and should be addressed by professional educators based on an understanding of research and best professional practices. On the other hand, I figured a good defense attorney would shred these plaintiffs and the credibility of their cause. It will be interesting to see the actual language in their lawsuit, but the broader concept is already familiar. It’s just never been given a courtroom treatment that I know of. So, I’ve taken the liberty of dreaming up the court transcript ahead of time (using Q: for the defense attorney’s questions and A: for the plaintiff’s answers). Enjoy this cross-examination.

Q: You are demanding that LAUSD use measures of student growth in teacher evaluations, is that correct?

A: Yes.

Q: And you believe that student test scores are a measure of growth that would reflect teaching quality, correct?

A: Yes.

Q: If LAUSD were to adopt a policy that attributes the growth or lack of growth in student test scores to the student’s teacher, and uses the scores of all students to evaluate the teacher’s effectiveness, you would drop this lawsuit, is that correct?

A: Yes.

Q: How often are these tests administered?

A: Once per year.

Q: And the district has no way of knowing if the student’s performance on that day reflects the student’s ability or perhaps reflects some trauma, distress, boredom, distraction, or rebelliousness?

A: No.

Q: And for students who have changed schools, or changed teachers during the year, there’s no way to factor that into the analysis of data when a student simply shows up on one roster or another, right?

A: That could be adjusted.

Q: There’s no study that would guide you in how to do that with any accuracy, is there?

A: I don’t know.

Q: No evidence that a move at the mid-point of the year gives each teacher half the responsibility for the student’s learning, or that each week has a proportionate effect?

A: None that I know of.

Q: And would the degree of change in a certain classroom affect students in that classroom who had not been part of any change?

A: I don’t know.

Q: Does it seem likely that changing the students in a class would change the class itself and affect some of the students who had been there all along?

A: I guess so.

Q: But you would have no way of knowing which students were affected or how they were affected?

A: Not really, no.

Q: Now, if I were a high school English teacher, I would be responsible for teaching in four standards areas, but would the test cover all four of those areas?

A: No.

Q: How many does it cover?

A: Two.

Q: You’re including writing when you say “two” but in fact there’s no writing on the tests currently used, is there?

A: No.

Q: So more accurately, the test covers one out of the four standards areas?

A: Yes.

Q: Does the test cover every standard in reading?

A: No.

Q: So, you’re proposing basing a significant part of an English teacher’s evaluation, for example, on a test result that covers a small fraction of the standards?

A: It’s the only objective way.

Q: So your answer is yes?

A: Yes.

Q: By objective, you mean it’s the same for every student and teacher?

A: Yes.

Q: Does every teacher have an equal assignment, equal students, classes, and resources?

A: No.

Q: So, you do not concern yourself with objectivity in all of the factors affecting the teacher’s work, but you figure you can evaluate different teachers working with different students and different classes using the same test that covers only a fraction of their standards?

A: Yes.

Q: So is that an objective process for evaluation, or an arbitrary process with an objective element in it?

[Plaintiffs' counsel objects to argumentative question. Judge upholds the objection.]

Q: Do the words “objective” and “fair” have the same definition?

A: I couldn’t say.

Q: I could give an objective geometry test to every student in an algebra class, but would that be fair?

A: Okay, I see. They have different meanings.

Q: So your claim that the test is objective doesn’t cover the question of fairness, does it?

A: But it is fair!

Q: Please answer the question. A claim of objectivity is different from a claim of fairness, correct?

A: Yes.

Q: So an objective test may be inappropriate for certain students and therefore unfair, no matter how objective?

A: I would say that the test is fair to everyone.

Q: Like a geometry test for algebra students?

A: Well, no.

Q: Does a student’s linguistic skill relate to their success in a test that requires use of language?

A: Of course.

Q: So a test given in an unfamiliar language might yield a result that reflects linguistic confusion rather than conceptual confusion, or poor teaching?

A: We could adjust for language in a teacher’s evaluation.

Q: In what way?

A: If the student is still learning English their scores could be separated out.

Q: What if a student did well on the test despite being new to the language?

A: Well, we can’t just use the scores that help the teacher. We have to be fair.

Q: You mean objective?

A: Yes.

Q: Because actually, it would be fair to use the results that are valid and exclude the results that are invalid. Are you suggesting that such a determination could be made for each student, or that we should come up with a single formula and stick to it?

A: Just use a single formula.

Q: So regardless of the student’s actual linguistic knowledge, you would suggest making assumptions based on a certain number of years for students to learn enough academic English.

A: That would be logical.

Q: No matter the variables in the student’s instruction in English or the amount of time it actually takes them to learn English?

A: It’s the only fair way.

Q: Fair, or objective?

A: Objective.

Q: Objective regarding the student’s knowledge and skill, or objective regarding only measures of time?

A: Time.

Q: Is it fair to use value-added measurements to rank teachers even when numerous studies show that it is a volatile measure with error rates exceeding 25%?

A: It would only be one of multiple measures.

Q: That wasn’t my question. Is it fair to use an error-prone measure?

A: It’s not fair to exclude student performance from evaluations.

Q: Your Honor, would you instruct the witness to answer the question?

A: I’ll answer. It may not always be fair in every case, but no method is perfect.

Q: You’re suing the Los Angeles Unified School District to compel them to use a teacher evaluation method that is prone to errors and unfair to perhaps a quarter of the teachers evaluated in this manner, is that correct?

A: Yes! The alternative is the status quo, which is intolerable.

Q: But there are thriving, high-quality schools around the U.S. and around the world that are not using value-added measures. Doesn’t that prove that there are alternatives to the LAUSD status quo that are something other than the remedy you seek to impose?

[Plaintiffs' counsel objects to argumentative question. Judge upholds the objection.]

Q: Have you heard of the National Council for Measurement in Education, the American Psychology Association, the American Education Research Association?

A: Yes.

Q: Are you aware of their position on the lack of validity in using tests designed for one purpose and then used for another purpose?

A: More or less.

Q: I’m quoting from their joint position statement on this topic: “Tests valid for one use may be invalid for another. Each separate use of a high-stakes test, for individual certification, for school evaluation, for curricular improvement, for increasing student motivation, or for other uses requires a separate evaluation of the strengths and limitations of both the testing program and the test itself.” Does that sound familiar to you?

A: More or less.

Q: In other words, you’ve heard this argument before?

A: Yes.

Q: Is it fair to say that these are the three leading organizations for educational measurement and research?

A: I suppose so.

Q: Are you a professional organization for educational research and measurement?

A: No.

Q: Do you think it’s advisable, or even responsible, to ignore the policy position of these leading organizations?

A: But we know that teachers are the most important in-school factor on student performance!

Q: Okay, no argument there. But you have no basis upon which to argue against the validity issues raised in that quote, do you?

A: No.

Q: Now, taking up your contention that the teacher is the most important in-school factor, could you say most important out of how many factors?

A: No.

Q: You don’t know how many factors influence student performance?

A: No.

Q: If I threw out a number, like five, would you guess that it’s too low, too high, or about right?

A: That sounds too low.

Q: How about ten?

A: I don’t know, that might be right.

Q: Fifteen?

A: Maybe.

Q: Just hypothetically, could we proceed on the assumption there are ten factors in schools, other than teachers, that affect student performance?

A: Okay, yes.

Q: Would you expect every factor to have the same influence on every student, or would some factors have strong influences on one student and almost no influence on another student?

A: It would vary.

Q: If you wanted to design a fair formula, you would take those ten factors into account?

A: Yes.

Q: Even though you can’t say for sure how much each factor affects the student?

A: Yes.

Q: You can’t even say with certainty that a specific factor has any effect on a certain student or group of students?

A: No.

Q: So, let’s assume that each of those ten factors could play out in only two different ways: how many possible combinations do we have for each student?

A: Twenty.

Q: I’m sorry to correct your math, but actually, that would be ten-squared, or one-hundred possibilities.

A: Oh, yes, one hundred, I see.

Q: But we don’t know for sure how many factors to consider and what they are. And if we could actually identify fifteen variables instead of ten, and if each variable could play out in three different ways, would it surprise you to know that there would be 3,375 possible combinations?

A: That sounds like a lot, but you’re just playing with numbers.

Q: “Just playing with numbers.” I see. So just because something is true mathematically or statistically, it doesn’t necessarily translate into an actionable policy?

A: That’s not what I said.

Q: Of course you wouldn’t say that. Your case is predicated on the idea that because you can make value-added calculations that show some teachers are less effective than others, it therefore makes sense to use the numbers in policy that leads to the outcomes you want. Though again, the actual experts in educational measurement would warn against that, correct?

[Plaintiffs' counsel objects to argumentative question. Judge upholds the objection.]

Q: That’s what you need to do if you use test scores and value-added measures in teacher evaluation, isn’t it? Play with the numbers? You would need to come up with a formula that makes certain assumptions about the effect of each factor, even though you can’t test your assumptions?

A: They’ve been researched!

Q: But you just said that we can’t assume factors are the same for each student – or did you mean that these students in this hypothetical school will have been researched before any formulas are applied to them?

A: No.

Q: Okay, to be fair, let’s assume that we can come up with a formula for each of these individual factors. Wouldn’t it also be necessary to know about the interactions of the variables?

A: What do you mean?

Q: Well, perhaps we can apply a statistical control for homelessness, another to control for the time of day that the student studies a certain subject, and another to control for the change from last year’s 50-minute class periods to this year’s 90-minute class periods. Is it likely that there is any research on the effects for homeless students in longer classes at different times of day?

A: No.

Q: So when we combine factors, we not only make assumptions about each one, but also assume that these factors do not influence each other in any way, is that right?

A: You can’t study every little thing.

Q: So, if this were a medicine, you’d be comfortable saying that we have plenty of science about the ingredients and we don’t need to study them in this particular combination in order to assume the effects the medicine will have?

A: I don’t know anything about medicine.

Q: Have you ever been a teacher?

A: No.

Q: Thank you. No further questions.



Advertisement

District Boundaries

Periodically, the subject of district boundaries comes up. Why are any properties in the city of Columbus zoned for the Worthington School District. Most people think that Worthington is part of the "Win-Win" agreement but this is not correct. Instead, the boundaries between Worthington and Columbus were first drawn up by an agreement between the two school boards back in 1968. This is the documentation for that agreement.

As a result, Worthington does not have to periodically renew the "Win-Win" agreement as other districts do.