Shutdowns and Meltdowns

Tigers closer Jose Valverde had a spectacular season closing games in 2011 accumulating 49 saves in 49 opportunities.  His ability to close out games was so impressive that he finished fifth in the American League Cy Young balloting.  Was he really that good though?

There are a few limitations of the Saves statistic in evaluating closers.  First, not all save situations are especially challenging.  For example, if a team has a three-run lead with nobody out and the bases empty in the ninth inning, there is about a 98% expectancy that the team will win the game with an average reliever.  So getting a save in that scenario is not necessarily a big deal, yet closers are routinely used in those situations and pile up rather easy saves.

An additional issue is that closers are often brought into games where there is no save opportunity.  They may come into a tie game and can have either a strong positive or negative impact on the outcome, but they get no credit for either a save or a blown save.  Valverde was particularly bad in non-save situations last year, but none of those games went into the Blown Saves column.  Moreover, many set-up men contribute substantially to their team's chances of winning games, but register no saves at all because are never called upon to do so.
      
What would be useful are some numbers which tell us how often a reliever added significantly to his team's chance of winning games and how often he significantly hurt his team's chances. Two relatively new statistics at FanGraphs - Shutdowns (SD) and Meltdowns (MD) do just that.    
 
The SD and MD statistics are based on the concept of Win Probability Added (WPA).  The WPA system gives relievers credit based on the effect each batter faced has has on the team's probability of winning.  These probabilities vary depending on the game state before and after each play.

Suppose Tigers reliever Joaquin Benoit comes into the game in the top of the eighth with a two run lead, no outs and a runner on first. There is an 83.7% expectancy that an average team will win a game given that situation. Now, assume that Benoit strikes out the first batter. There is now one out and the probability of winning increases to 88.4%. Subtracting the win expectation before the strikeout from the win expectation after the strikeout gives us the value of the play in terms of the probability of winning added by the strikeout, that is, 4.7% (88.4% - 83.7%).

Now, suppose the next batter, facing Benoit, doubles home a run. That gives the Tigers a one run lead with a runner on second and one out. The probability of winning goes down to 77.4% so Benoit loses 11%  (77.4 – 88.4) on that batter.  Summing all the gains and subtracting all the losses for all the batters Benoit faces during a game yields his WPA for that game

An SD is defined as any game where a pitcher had a WPA of 6% or more, that is, he greatly helped his team's chances of winning the game.  Similarly, an MD is any game where a pitcher has a WPA of -6% or less, that is, he significantly hampered his team's chances.  The 6% cutoff was chosen so that SD would be on a similar scale as saves.  For example, 40 SDs for a pitcher is about as rare as 40 saves is for a pitcher.  The SDs and MDs of all pitchers can be found on FanGraphs.

The SD and MD metrics are counting statistics which don't take numbers of opportunities into consideration.  So, the following rate statistics are also useful:

(1) The Shutdown Percentage (SD%) for a pitcher is the percentage of total games pitched which resulted in an SD.

(2) The Meltdown Percentage (MD%) is the percentage of games which resulted in a MD.

(3) Shutdown Meltdown Ratio (SD/MD) is Shutdowns divided by Meltdowns.

The AL Leaders for SD% are shown in Table 1 below. You can see that the best relievers had SD%'s above 50% and that Indians reliever Chris Perez and Yankees standout Mariano Rivera tied for the league lead at 57.8%.  Valverde was tenth in the league at 51.7%.  Benoit was 15th with 43.9%. 

Table 1: AL Shutdown Percentage Leaders, 2011

Player	Team	G	SD	SD%
Chris Perez	CLE	64	37	57.8
Mariano Rivera	NYY	64	37	57.8
Rafael Soriano	NYY	42	24	57.1
David Robertson	NYY	70	37	52.9
Andrew Bailey	OAK	42	22	52.4
Greg Holland	KCR	46	24	52.2
Joakim Soria	KCR	60	31	51.7
Grant Balfour	OAK	62	32	51.6
Jonathan Papelbon	BOS	63	32	50.8
Jose Valverde	DET	75	38	50.7
Jordan Walden	LAA	62	31	50.0
Daniel Bard	BOS	70	35	50.0
Chris Sale	CHW	58	28	48.3
Jim Johnson	BAL	69	31	44.9
Joaquin Benoit	DET	66	29	43.9

 Table 2 ranks AL pitchers according to fewest meltdowns.  Greg Holland of the Royals had just one meltdown all season giving him an SD% of 2.2%.  Valverde was fourth in that category at 6.7%.  Somewhat surprisingly, Tigers lefty Daniel Schlereth finished 11th with 10.2%.  That might have something to do with how he was used, but the fact that he had so few meltdowns is still a positive.      


Table 2: AL Meltdown Percentage Leaders, 2011

Player	Team	G	MD	MD%
Greg Holland	KCR	46	1	2.2
Frank Herrmann	CLE	40	2	5.0
Jonathan Papelbon	BOS	63	4	6.3
Jose Valverde	DET	75	5	6.7
Casey Janssen	TOR	55	4	7.3
Koji Uehara	AL	65	5	7.7
Jason Frasor	AL	64	5	7.8
Grant Balfour	OAK	62	6	9.7
Brandon Gomes	TBR	40	4	10.0
Cesar Ramos	TBR	59	6	10.2
Daniel Schlereth	DET	49	5	10.2
Dan Wheeler	BOS	47	5	10.6
Mike Gonzalez	AL	56	6	10.7
Mariano Rivera	NYY	64	7	10.9
Neftali Feliz	TEX	64	7	10.9

Finally, SDs and MDs are combined in Table 3.  Holland was the run away leader with an SD/MD Ratio of 24.0 (24 SD, 1 MD).  Valverde came in third at 7.6.  While his season may not have been quite as amazing as his perfect save percentage suggests, this final number confirms that Papa Grande was indeed one of the best at consistently shutting down opponents while keeping meltdowns to a minimum.  

Table 3: AL Shutdown Meltdown Ratio Leaders, 2011

Player	Team	G	SD	MD	SD/MD
Greg Holland	KCR	46	24	1	24.0
Jonathan Papelbon	BOS	63	32	4	8.0
Jose Valverde	DET	75	38	5	7.6
Grant Balfour	OAK	62	32	6	5.3
Mariano Rivera	NYY	64	37	7	5.3
David Robertson	NYY	70	37	8	4.6
Koji Uehara	AL	65	23	5	4.6
Casey Janssen	TOR	55	18	4	4.5
Andrew Bailey	OAK	42	22	5	4.4
Chris Sale	CHW	58	28	7	4.0
Joakim Soria	KCR	60	31	8	3.9
Jason Frasor	AL	64	19	5	3.8
Chris Perez	CLE	64	37	10	3.7
Vinnie Pestano	CLE	68	29	8	3.6
Neftali Feliz	TEX	64	25	7	3.6