Tangotiger Blog

This is a terrific paragraph:

We can think of this as a regulatory problem from the league’s vantage point. A fixed zone seems obviously fair, but it creates an arbitrage opportunity in some counts for one side to exploit umpires’ inevitable mistakes. At 0-2 the pitcher knows the hitter has to swing at anything close, even bad pitches, and he’s earned that advantage by getting the hitter to 0-2. But then a neutral zone gives the pitcher an extra, unfair advantage: By throwing a large majority of borderline pitches outside the zone, pitchers would benefit from more than their fair share of wrong calls. At 0-2, about 90 percent of umpire errors would be in the pitcher’s favor using a neutral zone. By shifting their decision rule on close pitches, umpires prevent either side from systematically exploiting their mistakes.

And another great one on the unintended consequences of a robo-ump:

Of course, there is a way to remove this arbitrage opportunity while also maintaining a fixed zone: use robot umpires. I don’t know whether that would be a good change or not. But based on our analysis here, we can see that the resulting changes could be substantial. Without the “counter-cyclical” adjustment umpires make at each count, the impact of the count will be even more pronounced than in today’s game. An 0-2 count will truly be a death sentence, as hitters are forced to swing even more frequently and pitchers take advantage of this by throwing even more toward the edges of the zone. With three balls we will see the reverse: pitchers forced to groove the ball even more than they do now, and hitters teeing off with greater ferocity. Knowledge of these enhanced count effects may also change behavior earlier in the count, with unknown effects. My own guess is that a constant zone across counts would mainly work to further suppress scoring, since the main change in size is a shrinking zone in pitchers’ counts. But it’s very hard to forecast all the consequences here, other than to say they could be profound.

Great read.  It’s somewhat comforting to know that umpires use a system for making calls that is part observation and part informed guess to minimize the error inherent in those observations and that their apparent biases are motivated by striving for accuracy as opposed to an attempt to favor the underdog.  But the fact remains that the limits of umpire accuracy appear to be in the range of 90%.  Ultimately, for me, the question of whether robo-umps are a good idea is less about whether human umps are earnestly trying to make accurate calls and more about simple levels accuracy.  That even highly trained, apparently well-intentioned umpires miss 20+ calls a game is not a source of comfort for me.

I wrote this in the comments section:

Not really buying this. If umpires are being Bayesian and actually improving their accuracy as the count progresses (or at least maintaining accuracy), then the size of the zone should not change much, if at all. Changes in the size of the zone are a proxy for poor umpiring (assuming that good umpiring means calling a consistent zone).

That assumes that the neutral or overall zone is de facto the correct one. If we assume it is not and that borderline pitches outside of the zone are often called strikes and borderline pitches inside of the zone are often called balls (I’m not sure those two things together are possible), then an umpire can “improve” the zone by changing it at hitter’s and pitcher’s counts. However, if they do that, we are back to the “compassionate” umpire paradigm.

I’m not sure I can explain it any better than I did at B-Pro (where I had 4,000 words!), so I’ll mainly let others respond.  But two quick points:

1) yes, I am in some sense taking 0-0 to be the “correct” zone, at least in the sense that it can’t be biased by compassion (by definition, neither side has an advantage at 0-0).  I’m not passing judgement on the inherent validity of the effective 0-0 zone, I’m just accepting it as the current MLB norm and addressing why umps depart from that in other counts.

2) Yes, of course IZ borderline pitches are often called balls, and OZ borderline pitchers are often called strikes. The perimeter of what we all call the effective zone is a line on which 50% of pitches are called a strike. You don’t really think any human umpire could lower that called strike rate to 0% one inch beyond that line, and lift it to 100% one inch inside the line—do you?

Don’t get me wrong, it is a very good article. Sure, umpires can get more calls right by “assuming” that a borderline pitch is a strike at a 3-0 count and a ball at a 0-2 count. That is almost tautological. I can do that without looking at the pitch, right?

Does that mean that umpires are NOT being compassionate? I don’t know that it changes anything. We still have no idea of the motivation.

We still have no idea of the motivation.

I think we do. If compassion is the motivation, zone size should be as responsive to the number of balls in the count as strikes, but it isn’t. At 3-1, the zone is *smaller* than at 0-0. The changes in zone size essentially fit the accuracy story perfectly, but fit the compassion story less well.  And even if the fit were the same, we should prefer the explanation that umps are trying to get as many calls right as possible, because that’s their job. 

Plus, if umps tried to help underdogs, that should show up in many other places.  But it doesn’t.

There may be other things going on that I have missed, but honestly there is zero reason to believe the compassion story at this point.

Re: robo umps
My intuition, based in part on my experience with softball, is that the less reliable the umpire, the more it is to the (modest) advantage of the pitcher, especially when a pitcher with good control faces a hitter with a good eye.

Hitters chase borderline pitches they know are just out-of-zone, especially with 2-strikes, when they don’t trust the umpire to call them balls.  And pitchers know this and test the outer-boundaries of the zone accordingly.

Of course, the opposite effect—where a pitcher grooves a pitch because they don’t trust the umpire to call a strike near the boundary—is possible too.  But my premise is that good pitchers are capable of throwing fairly clear strikes to the outer-thirds of the zone without completely grooving very many.

For what it’s worth, we do have the ridiculously small sample of 2 Pacific Association games played with a robo ump last year.

In a league that averaged 11.1 runs, 15.5 K’s and 7.4 BB’s per game, the 2 robo ump games were remarkably “normal”—10 runs, 16.5 K’s and 7 BB’s per game.

http://www.pacificsbaseball.com/index.php/press-submenu-box-submenu/game-recaps-2015/685-conroy-solid-as-pacifics-win-final-roboump-night

http://www.pointstreak.com/baseball/gamelive/?gameid=290983

http://www.pointstreak.com/baseball/gamelive/?gameid=290984

I don’t think “compassion” is the right word. Part of the motivation is inertia and adherence to the unwritten rule, although I can’t say (and I don’t think anyone can) what the origin is.

For example, at a 3-0 count it is well known that an umpire is going to call a strike on anything close. The pitcher knows that, the batter knows that and more importantly, the umpire knows that. I really don’t think it is a result of the umpire trying to get the borderline pitch right. I don’t think anyone who plays or watches baseball thinks that either. Is that “compassion?” I have no idea. Somewhere along the line, it developed and umpires simply keep the inertia going. At other counts, it is not so clear to me what the impetus might be…

In regards to motivation, I think a reasonable explanation is quite the opposite of compassion.  Rather, it is same judgment avoidance behavior we often ascribe to managers and coaches who make wrote decisions.  Given a debatable circumstance, they don’t want to be responsible for making a mistake that directly produces a bad (or unfair) outcome.  By extending the plate appearance, the umpire gives the batter & pitcher another opportunity to produce an outcome that does not require the umpire’s judgment and thus does not subject him to criticism.  Sure, he can be criticized for the poor call earlier in the PA, but that criticism is less salient, fades quickly, and is somewhat obviated by the unwritten rule expectations.

I’m also a bit confused as to how an umpire can be “more accurate” on 3-0 and 0-2 counts when the sizes of the zones are vastly different in those counts.

#10, I agree 100% with that. No one cares that an umpire made a bad call on a ball outside the zone on a 3-0 count. Similarly with an 0-2 count. I think you hit the nail on the head. Again, not buying the “umpires are trying to be more accurate” theory.

MGL/12,

I’ve thought about this, too (zone actually changing sizes), and it’s largely to do with the quantity of pitches in each.

Ultimately, I agree it is consistent with both the impact aversion and the Bayes idea. But looking at other unequal call rates might help figure out which it is (Guy and I have talked about outs, but I didn’t have that in my data at the time).

Alternatively, there’s evidence for a status/age effect: pitchers with higher status get the benefit of the doubt on calls. This could also be a Bayes rule driving the umpires to increase accuracy. This sort of thing has interesting implications in the context of profiling, etc.

Ultimately, the size of the zone can change and accuracy go up overall due to the relative frequency of in and out of zone.

An extreme example is 99 pitches out of the zone, 1 pitch in the zone, and calling all the out of zone pitches correctly while calling the 1 pitch in the zone correctly. I think that case, the area of the called strike zone is 0, and the accuracy rate is 99%.

This is actually easy to simulate using the actual data in R, I think, and let me see if I can put something together today and share the code.

On another note, you say that a good umpire is a consistent one, but I’m not sure that’s right. It’s the general rhetoric in the game, and I do get the idea (at least having information even if wrong is better than no information on the zone). But the catch is that umpires are explicitly graded on accuracy, not consistency, through ZE. So even if we were to agree on your consistency definition, umpires are responding to different incentives in their behavior and performance and we would expect them to respond accordingly.

Oops. I think I meant to say MGL/11

Ultimately, I agree it is consistent with both the impact aversion and the Bayes idea.

I really don’t see any evidence at all for the impact aversion idea. If umps are afraid to make decisions, or fear criticism for determining outcomes either way, then the aversion to calling a strike should be much larger with two strikes. But strike 2 does no more to shrink the zone than strike 1.  Similarly, ball 3 does not expand the zone any more than ball 1 or 2.

Also, if it’s outcome aversion, why do strikes have so much more impact on zone size than balls?  Both have an effect on outcomes.

Now, if the hypothesis is just that umpires want to prolong plate appearances, what is the intuition behind that—that umpires are afraid to go back to an empty hotel room at the end of the night? That they prefer to work longer hours for the same pay?  Seems like a weird notion to me, and again, what’s the evidence for this?

I was using “impact aversion” a bit generally to encompass these other situations where I mean more generally inequality aversion. Should have been more explicit.

To your point about the hotel, umpires actually call more strikes later in the game. So we could see this as 1) Wanting to get home earlier, or 2) Having Mariano in the game means the prior is that almost all the pitches will be strikes.

As we talked about before, there is probably some usefulness from allowing an at bat to proceed where fans prefer to see a swing and miss, out in play, or hit. That incentive would be very small from the umpire perspective unless we think there is significant social pressure from crowd size (I don’t find the crowd size to be influential).

I find it plausible that both these things take place, largely because I have had umpires explicitly tell me this is what they do. I suspect the differences in the impact on the zone size is again the relative change in the frequency of strikes vs. balls across the counts. But again that’s testable and I could be totally wrong.

Again, I think you’re right regarding the Bayes decision making here. I also think there’s room to provide more evidence for it. And I also think we are talking about humans that we know are susceptible to cognitive biases. And it is plausible that these biases are justified and improve performance under uncertainty (which I find to be a fascinating result).

I suspect the differences in the impact on the zone size *COULD BE* again the relative change in the frequency of strikes vs. balls across the counts. But again that’s testable and I could be totally wrong.

“...then the aversion to calling a strike should be much larger with two strikes.”

I’m not getting that. It is, according to your chart.

Count % correct calls in zone
0-2 0.63
1_2 0.66

0-1 0.74
1_1 0.77

According to this, umpires call way fewer strike in the zone when there are 2 strikes already, consistent with “not wanting to ring the batter up.” In other words, with 2 strikes, the umpire is much more likely to call a pitch in the zone a ball, consistent with the umpire “not wanting to end an AB with a called pitch.”

“Similarly, ball 3 does not expand the zone any more than ball 1 or 2.”

Same with this. At 3-0 count, umpire has outside zone correct call 77% of the time. At 2-0 it is 82% and at 1-0, it is 85%. So at 3-0, he is much more likely to call a strike on a pitch outside the zone, again, consistent with not wanting to walk the batter on a 3-0 count, i.e., not wanting to end an AB on a called pitch.

So, either I’m not understanding your charts or the data is perfectly consistent withe the umpire avoiding ending the AB with 2 strikes and with 3 balls.

BTW, I don’t think the umpire gives a hoot about how long an AB takes.

#18:  look at Table 4, which shows the changing side of the zone by count. You can see that for any given number of balls, the impact of adding strike 1 is about the same as adding strike 2.
For example, here is the marginal impact on zone size (sq ft) of strike 1 and strike 2 for each number of balls:
  Strike 1, Strike 2
0 balls: -.60, -.50
1 ball:  -.45, -.64
2 balls: -.42, -.33
3 balls: -.37, -.39

I don’t see any indication that getting strike 2 has any more impact on umps than strike 1.

Similarly, there is no indication that umps suddenly get nervous about calling a ball at a 3-ball count. Starting at 0-1, for example, the imact on zone size of ball 1, 2, 3 is .18, .14, and .15.

I am not interested in strike 1 versus strike 2 for my thesis and that of #10. Only what happens with 2 strikes, especially 0-2.

As far as calls with 3 balls, again, mostly interested in 3-0 but the pattern is clear at 3-1 as well.

Also not interested in size of zone. Only interested in this, in order to test/verify the thesis:

What happens on marginal pitches OUT OF the zone at 3-0 compared to other counts. The hypothesis is that an umpire is reluctant to call a marginal pitch a ball at 3-0 counts because he does not want to affect the AB (by issuing a walk) when he could be wrong. If he is wrong and calls a ball a strike, no one cares, especially since it is accepted that a marginal call gets called a strike.

Same thing with 0-2 and to a lesser extent 1-2 and 2-2. If he makes a mistake calling a strike a ball, no one cares, but if he makes a mistake calling a ball a strike, the batter (and his team) is angry. So the umpire is more likely to call a marginal pitch a ball.

Both of those things are supported by the data, regardless of the size of the zone.

You are arguing with a strawman in #19, since I am saying nothing about the impact of strike one versus strike 2 and I am saying nothing about the size of the zone.

I am ONLY saying that with 2 strikes (especially 0-2) and with 3 balls (especially 3-0) the umpire is motivated not to unduly influence the AB (by ending it with a walk or called strike 3). Period. The data clearly support that hypothesis.

To Guy’s point in #19, one way to think about this is to look at the relative change in run expectancy in 0-1, 1-1, and 2-1 counts (or whatever you want). The changes are not equivalent for the two, which means we might not expect the changes in the size of the zone to be equivalent either.

Based on Jim Albert’s work in JQAS in 2010, I did a (very) rough calculation of these in my paper in MDE in 2014.

In an 0-1 count, for example, the change in run expectancy for calling a strike is -.052, while the change run expectancy for calling a ball is + .025.

So I’m curious: how do these correlate with changes in the size of the zone (also: where are you getting the zone size measurements just so we’re all on the same page?)? I don’t know which way this would go, just providing some information that might tell us more about the relative changes expected in strike/ball call rates under the two hypotheses.

The stuff I put together for run expectancy changes is below:

Count RE Change w/Strike Change w/Ball
0-0 -0.038 -0.043 0.038
0-1 -0.081 -0.052 0.025
0-2 -0.133 -0.300 0.013
1_0 0.000 -0.056 0.060
1_1 -0.056 -0.064 0.058
1_2 -0.120 -0.300 0.041
2_0 0.060 -0.058 0.107
2_1 0.002 -0.081 0.100
2_2 -0.079 -0.300 0.097
3_0 0.167 -0.065 0.330
3_1 0.102 -0.084 0.330
3_2 0.018 -0.300 0.330

The logic of #20 is entirely circular. Your argument is that 2-strike counts are special, in terms of influencing umpire behavior, because the next strike will make a hitter angry. I provide evidence that strike 2 has no more impact on umpire behavior than strike 1, and you say “thou must not speak to me of these first strikes, I carest only about strike 2!” The claim that strike two is special is non-falsifiable if you refuse to consider evidence from other counts! (It’s also impossible to verify your theory this way, but I guess that’s not a problem since you already “know” you are right.)

Look, there is just no evidence that umpires are specifically trying to avoid taking actions that end a plate appearance (ball four or strike three). None. Zero. Nada. If you don’t see that, you need to look harder at the data. Walsh reached the same conclusion, which is why he developed the “compassion” theory.

MGL/20,

I think Guy is getting at: why would we see changes in other counts, if your impact aversion (3-0 and 0-2) are the only counts that should matter?

Clearly things change throughout. This would lead us to thinking along the lines of some aversion to inequality in the ball/strike count, rather than specific to ending the AB. The data seem more consistent with this latter explanation (and the Bayes explanation) than the impact aversion one.

And from there, untying the inequality aversion with the Bayes decision making is much more difficult. Though the run expectancy relationship might be helpful.

Ultimately, Guy lays out some neat evidence, and it can be tested in other situations.

In particular, Guy and I have talked over email about number of outs, runners on base, etc. as possibly influencing umpire calls. In my paper I find that runners on base actually decrease the probability of a strike, so that’s not consistent with inequality aversion (but, it is consistent with Bayes if there is an implicit assumption that in those scenarios, pitchers have more difficultly throwing strikes—an empirical question, of course). I don’t have the number of current outs in my data, but that is another test of this, as is the current score.

There is also evidence of status-based effects. Here, umpires call more strikes for good/older pitchers. We know these guys are more likely to throw strikes, so the bias may be perfectly explained under the “maximizing accuracy” hypothesis, rather than an explicit bias against rookies. The status-based effects are also there for batters.

We might also expect this could be a reasonable explanation for HFA in ball-strike calls. If home pitchers are more likely to throw strikes, and umpires have that prior, then they will get that larger zone.

I’ve also found some evidence that catchers get slightly more favorable calls at the plate. I suspect umpires respect their lack of swings as a useful signal of whether it was a strike or not. We could also see this in the same light.

#21:  Brian, the strike zone estimates come from the Caruth article listed in my sources.

BTW I ran the same calculations, using your coefficients for the change in likelihood of a strike call for each count instead of zone size.  The story seems to be the same:
* your coefficients align somewhat better with IZ% than with wOBA (though highly correlated with both);
* your coefficients correlate much more with the number of strikes in the count than the number of balls (despite the fact that balls and strikes have roughly equal weight in determining whether the hitter or pitcher has the upper hand).

Hi Guy,

Cool. Not sure what you’re saying with #2 here, though. When you say “correlate…”, is this with the IZCC vs. OZCC?

25:  no, just correlating your coefficients separately with number of strikes in count (0, 1, 2) and number of balls in count. You could obviously do a more sophisticated measure, but it seems clear that zone changes are driven mainly by the number of strikes in the count, while balls have much less impact.  I was shocked to discover that the 3-1 zone is a bit smaller than on the first pitch.  The story isn’t really about a “changing” zone as much as a *shrinking* zone as number of strikes increases.

I’d like to remind everyone that Pitch Fx doesn’t really know where the ball is when it crosses home plate.  The nearest measurements that it makes are 8 to 10 feet in front of the plate.  From that and the other measurements it makes from 40 t0 45 feet to 8 to ten feet it then infers the balls flight path and extrapolates the position at the front of the plate.  During a phone call in December I specifically asked Marv White what he thought the Pitch Fx in-game accuracy was for home plate locations and he responded that is best guess was a standard deviation of about 1 inch. 

Consider the implications of this Pitch Fx measurement error on batter behavior and the article’s statistics on umpire pitch calling.  If pitchers are more likely to work the edges of the strike zone when the count is in their favor then if Pitch Fx “sees” a pitch as just outside the zone that is actually just inside the zone then the batter is more likely to swing at that pitch because he sees that pitch as a strike so it is a no call for the umpire and the batter appears to be acting aggressively because he swung at a ball that Pitch Px determined was out of the zone.  If the batter decides not to swing then the umpire calls the ball correctly as a strike then the Pitch Fx watchers say the umpire is incompetent for calling a strike on a ball that Pitch Fx determined was out of the zone.  If Pitch Fx “sees” a pitch as just inside the zone that is actually just outside the zone the batter is less likely to swing at the pitch and if the umpire calls the pitch correctly as a ball then the Pitch Fx watchers say that the umpire is incompetent for calling a ball on a pitch that Pitch Fx has documented as being inside the zone.  Since the batter will swing at far more of the first type of error than the second type the umpire is more opportunities for making strike calls on pitches Pitch Fx sees as outside the zone than calls of ball on pitches that Pitch Fx sees as inside the zone and therefore the mistaken conclusion that the compassionate umpire has “expanded the strike zone” on pitcher’s counts.

And just the opposite reasoning on close calls on batter’s counts.

Trackman follows the flight all the way, to at least the point of contact with the bat.  So, you can evaluate FX against Trackman.

And I’m sure you’d get the same kind of results.

Peter you make a good point, but unfortunately umpires apparently SHRINK the zone on pitcher’s counts and EXPAND it on hitter’s counts, which is the opposite of what your theory would result in.