Evaluating Managers, Again
For those of you interested in research in trying to quantify a manager’s effect on a team’s won-loss record, you’ll enjoy this article by Chris Jaffe, the third part in a series of extremely long articles he’s written on the subject. Building off work by Phil Birnbaum, he’s taken steps to try to evaluate managers in historical context, and today’s piece takes a shot at attempting to evaluate active managers.
His methods are a bit crude, but probably in the ballpark of being useful. Essentially, he’s taken a huge sample of statistical simulations from three different projection systems and compared expected team performance to actual team performance. Now, there’s a lot of reasons a team can underachieve or overachieve a statistical projection, with injuries and flaws in the projection systems being two of the big ones. But managerial impact is at least part of the difference, and Chris is right that the noise should begin to be filtered out when looking at a decent sample. He’s using six years of data for this piece, which may not be as much as we’d like, but is enough to avoid cries of small sample size.
Anyways, that’s enough talk about the methods. What you guys care about are the results, right? Thirty four managers worked in at least three seasons, qualifying them for the study. I won’t put the whole list here (you should read Chris’ piece anyways), but here are the parts Mariner fans care about:
#1: Lou Piniella, +49
#28: Bob Melvin, -13
#34: Mike Hargrove, -28
This data needs to be taken with a grain of salt or ten, but according to the data, the Mariners replaced the best manager over the last six years with the 7th worst, then replaced him with the worst of all. In addition to the numbers, Chris wrote a little blurb on each manager. Again, the relevant parts for Mariner fans:
Mike Hargrove.
Well, someone’s got to come in last. His only really bad year was 2005 when the Mariners were 13 games under their projection. From 2000-6, exactly 60% have scored at least a –2 in their projections. Hargrove’s been –3 or worse four times in six years. Only one of his six clubs have exceeded expectations. That was 1992 when the O’s scored a big +2. He was –12 with the Orioles, and –16 with the Mariners. I’m glad he’s not managing my favorite team. The Birnbaum database likes him through 2001 putting him around +175 runs, which was almost entirely built up by his Pythagoras W/L record.
Bob Melvin:
He actually scores well in three of his four seasons, but the 2004 Mariners were 21 games under projection. All four of their main starting pitchers fell apart. Most disturbingly, both their young studs, Gil Meche and Joel Pineiro, collapsed. I don’t know if he overworked them the year before of if there’s something else going on. Looking at it, Meche wore down considerably in the second half of 2003, and Pineiro couldn’t get anyone out that August. Now in Arizona, history’s not repeating itself with Brandon Webb. In fact, ChadBradfordWannabe thinks Webb’s mechanics have improved since 2004.
Lou Piniella:
I can’t believe he’s as good as this system claims he is if for no other reason than I have too much respect for some of the other managers to think Piniella could dominate them so much. Then again, I think enough of this system that I have to think he’s done a real wing-dinger of a job over the last couple years. As mentioned earlier, he’s the best at orchestrating immediate dramatic improvement since the death of Billy Martin. His score of +49 wins here is off the charts. How the hell did he do that good? Well, the 2001 Mariners didn’t hurt his mark. They’re the biggest overachieving team under study here, at +29 games. They’re also the biggest overachievers in the Birnbaum database. Take that away from him and he’s “only” sixth best. That’s not a fair comparison, however. Let’s take everyone’s best season away from them and see how they compare. Here’s the top five without their best season:
1. Lou Piniella +20
2. Ron Gardenhire +17
3. Mike Scioscia +13
4. Ozzie Guilen +11
5. Bobby Cox +10He’s still the king, and only one man is within two-thirds of him. Piniella’s aided by only having one negative year, a mere –3 in 2005. Guillen has no negative years, but he’s only been around 3 years. Gardy has two. Granted, they’re only –2, and –1, but he hasn’t managed as much.
The Birnbaum database didn’t like Piniella at all. In part two I explained why I disagreed. Anyone who can capture a ring with the 1990 Reds and win 116 wins in Seattle has to have something going for him. I’m amazed how much this system likes him, though.Piniella scores +36 with Seattle and +13 with Tampa Bay. Maybe it’s easier to exceed projections when you’re supposed to win as few games as the Rays are always calculated at, but the other Tampa managers are a combined –19 wins. Perhaps he’s not as good as he once was. The aging patterns info works against him, but he’s earned the right to show if he still has it.
Anyways, like I said, the idea is interesting, but the methods are so blunt that you can’t take the results with any kind of precision, and even the general conclusions reached have to be tempered a bit. Lou Piniella comes out looking great, but 2001 was obviously a huge part of his big positive addition, and while he certainly deserves some credit for that season, I’d imagine this system is overestimating his abilities quite a bit.
So, yea, even the projection systems think Mike Hargrove is pretty lousy.
What is Bill Bavasi’s email address? I think I have a link to send him.
I can’t imagine how any credibile analyst or projection system could give Hargrove a favorable evaluation. Unless his players really like him (which hasn’t translated to performance), and with the exception of his apparently solid front office relations, what does he bring to the table?
His unique ability to work with young players. Didn’t you read the team’s email?
Dave,
Has there been any similar research compiled on GM’s? I would be very curious to see where Bavasi falls.
You’d have to do something entirely different to evaluate a General Manager.
Because of the limited amount of roster decisions a GM makes in any given year (usually, less than 6 or 7 of any consequence), I’m actually not a big fan of evaluating them by their results. I believe that we have a pretty decent grasp of what constitutes solid organizational philosophies, and if you have a GM who is putting those in place, you should be relatively happy. I don’t think any less of Mark Shapiro now than I did a year ago, for instance. The Indians are doing things better than everyone else, but things just didn’t break their way this year.
Essentially, I’m of the school of thought that gives Bavasi thumbs up for things like the Beltre signing and the Garcia trade, even though in hindsight, they haven’t worked out as well as we had hoped.
But, if your goal is to find out where Bavasi falls on the GM scale, I’d imagine most analysis would place him in the 20-25 range. He’s probably a little below average.
Or, here is Sparky Anderson’s take on the whole thing,
“Sparky Anderson, 72, when asked what makes a great manager: “You’re not going to hit the ball. You’re not going to pitch it. You’re not going to catch it. You’re not going to throw it. So you basically are sitting on your big, fat fanny on the bench. You’re not going to do anything except you must know the moment when you have to pull the trigger. If you don’t do that, you’re going to be signing a lot of contracts that say you’re fired.”
Dave, May I ask what your take on the Kotsay vs Kielty thing reported in ESPN is? Beane wants Kielty to play, Macha plays Kotsay against lefties.
I think we knew Grover was a lousy manager by this measure. No team is actually bad enough to finish 4-32 (featuring 2 distinct 12-game losing streaks) like Hargrove’s Orioles did.
He never should have found another job in baseball after that.
This is just all too much.
Dave, May I ask what your take on the Kotsay vs Kielty thing reported in ESPN is? Beane wants Kielty to play, Macha plays Kotsay against lefties.
Macha and Beane couldn’t co-exist any longer, personality wise. Their shelf life expired. The Kielty/Kotsay thing was just a weed born of a deeper root.
Does this study take into account the year he managed the Indians against the Mariners and beat the M’s in the ALCS?
Macha and Beane couldn’t co-exist any longer, personality wise. Their shelf life expired. The Kielty/Kotsay thing was just a weed born of a deeper root.
Beane had similar trouble with Art Howe, too.
I wonder if Beane should just petition MLB to let him both manage and GM the club.
I wonder if Beane should just petition MLB to let him both manage and GM the club.
Hmmm. If this were to happen (didn’t know he’d have to petition MLB to do so), he’d probably remind people that Zito’s amp only went to 11 and then spontaneously combust.
#10, only if it happened since 2000.
A few things jumped out at me in the full list.
Dusty Baker is 6th, at +23, but no way would I want him managing my team. That also doesn’t bode well for Lou in Chicago. Except the author says the Cubs underachieved, so that is the bright spot for Lou.
Joe Torre is a very middle of the road -3, tied for 20th with Bobby Valentine.
13- Yes, but as the article goes on to say, Torre is hampered by expectations. Since the evaluation method is judging managers by how their teams vary from predictions, and the Yankees are predicted to win approx. 95 games every year during the years being examined, it’s pretty hard to get on the positive side of things.
>weed born of a deeper root
very nice
You would have to petition because the GM isn’t allowed in the dugout.
If he was the GM and the manager, he would be.
Does Billy even watch the games? He didn’t used to.
It’s a nice attempt at analysis but it’s really limited. For instance, Bob Melvin ends up losing points because the Mariners starting rotation fell apart in 2004. How the heck was that Bob Melvin’s fault? I’m not a Bob Melvin fan by any means but this seems to be a still very blunt tool for trying to make some fine discriminating judgements.
Lou seemed to spend half his time in Tampa dissociating himself from the team because management hadn’t spent as much on the team as he was supposedly promised.
And Hargrove’s Orioles quit on him but I always remember him doing an expert job wheeling his relievers in and out of games to protect wins against my favorites, the Red Sox.
From the outside, it always seemed like Lou was making the late 90’s Mariners bullpen situation a little worse than it had to be. Maybe that’s a false perception based on superficial exposure to the M’s from 3000 miles away.
I think I remember Bill James trying to do something like this and ending up not very satisfied with his own result.
Dusty Baker is 6th, at +23, but no way would I want him managing my team. That also doesn’t bode well for Lou in Chicago. Except the author says the Cubs underachieved, so that is the bright spot for Lou.
If the Cubs underachieved according to that perspective, then (not considering other changes) they would be expected to do better next year. Of course things could change to make the expectation drop down, but you get my point.
In that case, it would be tougher for Piniella to equal or beat the expectations next year, unless the truth is that they really did underachieve this year, meaning that they really would be likely to do better next year.
I’m more or less thinking that Baker screwed the pitching staff again. I don’t see that as something easily rebounded from.
If a manager was terrible at bringing along young pitching, it would never develop and create an expectation in the predictions. He’d have hurt his organization but wouldn’t suffer much under this study, would he?
Almost forgot. Scioscia’s Angels totally collapsed in the second half of 2003. Not quite to the degree of the Hargrove collapse cited but badly enough that one might’ve cashiered him for good, too.
I’ve always thought Lou was good at using all 25 guys on the roster and assembling a solid bench, and his biggest flaw (abusing starting pitchers) can be fixed fairly easily by giving him a good pitching coach he develops a rapport with.
He wasn’t going to win in Tampa any more than Casey Stengel was going to win in Boston, Brooklyn or with the Mets- not in a division where two of the competitors have 5-10 times the financial resources, and not anywhere where the ownership has clearly decided the path of least resistance is being a farm team to the rest of the league they play in while cashing the revenue-sharing checks.
His chances in the Central are somewhat better, because while Chicago’s not a great franchise, nobody owns that division like Boston and NY own the AL East, and the resources in Wrigley are better than the ones in Tampa. That being said, I won’t be surprised if he doesn’t win out there, and ends up gone ala Baker, Baylor and so on.
For instance, Bob Melvin ends up losing points because the Mariners starting rotation fell apart in 2004. How the heck was that Bob Melvin’s fault?
What, seriously? You can go back and read our 2004 posts, but Melvin did a pretty crappy job with those guys.
Hargrove’s Orioles quit on him
Since his primary job is to motivate the players, I call that pretty damning.
What, seriously? You can go back and read our 2004 posts, but Melvin did a pretty crappy job with those guys
Wasn’t that the Year Of Only Five Starter? No, wait, that was 2003. In which Melvin also did a pretty crappy job with those guys.
Wasn’t that the Year Of Only Five Starter? No, wait, that was 2003. In which Melvin also did a pretty crappy job with those guys.
But… But… He tied a record! That’s pretty cool!
If he’d gone through the whole year wih only 4 starters, then I’d be impressed.
What he actually did was really just slavish devotion to an inefficient model.
And, I would argue, the “Year of Five Starters” ™ in 2003 was a direct cause of the rotation falling apart in 2004. Thus, Melvin does deserve the blame for that, even if it shows up in the following year.
Hmmm…as a Cubs fan, this makes cautionsly, very cautiously, slightly optimistic. I still would have preferred to see what Girardi could do.
As a Cubs fan, you should fight that optimism.