Umpiring Quality

James Rodrigues

Director of Test Excellence

Microsoft Corporation

December 2007

Summary: In this article, the author explores the similarities between baseball umpiring and quality assurance. (6 printed pages)

Contents

Introduction

The Keys to Umpiring

Applying the Keys to Quality Assurance

Judgment Calls and Logistics

The Importance of a Good Umpiring Crew

Conclusion

Introduction

Over the last year, I have acted as a volunteer umpire for the Redmond (Washington) Little League, which is a baseball league for children between the ages of 8 and 12 (mainly). I had played for years and previously helped coach, but umpiring wasn't something that I looked to do, as it seemed to be a thankless job. You do well, and it's expected. You miss a call, and you become everyone's scapegoat. My son's team didn't need an assistant coach, so I found myself enlisted in the parent umpire corps. What I found was some very similar parallels between umpiring and the quality-assurance (QA) work that I have done previously in the role of Test Manager.

The Keys to Umpiring

What makes for a good umpire? A good umpire should:

• Know the rules.
• Make good calls.
• Be decisive.

Know the Rules

Let's start with knowing the rules. Baseball has a lot of rules. They are written down in a rules book. Additionally, little league has adopted variations to the rules of baseball that they publish in a Little League Rules book. Also, each league publishes its own additions or modifications to the rules—"local rules"—that are to be used locally.

Here's what I've learned about rules. Rules must be:

• Written down. If it isn't written down in an official place, it is not a rule.
• Communicated broadly. The more people who know the rules, the less arguing that will happen. Notice that I said "less," as umpiring still involves a large amount of arguing over what I call judgment and interpretation of the rules; but we'll get to that, later on.
• Reviewed and updated periodically. Experience tells us that rules must change. I want to emphasize a few things here. Periodically implies a cadence. This might mean that the rules of baseball are talked about (and changes are made) every year, based on experience from the previous year.
  Experience implies that we know something that we did not before, or that we recognize now that something is different from before, and this is usually what precipitates a rules change. Changes occur during the off-season. The off-season refers to the time after the end of the World Series and before the start of spring training. Rules do not change in the middle of a season, as that would cause chaos. We live with the rules for this season, and then do something about them before the next season.

Make Good Calls

Let's jump to the second key to umpiring, which is to make good calls. As an umpire, it is important to position yourself to be able to see what is happening. If something blocks your vision, you can't see what has happened. Umpires recognize where they should be to give themselves a good perspective into the event. This allows them an unobstructed view of what is occurring.

Whenever something happens in baseball, from an umpire's mindset, one of three things is going to result:

• Bad—You don't know the rules, and, although you see what is happening, you make the wrong call.
• Bad—You know the rules, but you were in a bad position to see what has happened, so that you make a bad call.
• Good—You know the rules; you see what has happened, so that you make a good call.

Now, let's get to the arguing part. I find that arguments happen for two main reasons: Either (a) somebody doesn't know the rule and is arguing with you out of ignorance or (b) they know the rules, but they question your judgment with regard to the sequence of events that has occurred and how the rules should have been applied. Argument type (a) is easy: Pull out the rule book, and show them the rule. (You see why rules must be written down?) Argument (b) is more difficult, and it leads us to the third key to umpiring.

Be Decisive

Let's say that there is a close play and that the umpire hesitates a bit, (weakly) shows the "safe" sign, and then (weakly) vocalizes, Safe. If you think that the runner was out, will you argue with this umpire? Now, let's take that same situation; but, this time the umpire quickly and authoritatively flashes his arms to give the "safe" signal, and—with a loud and assertive voice—yells, Safe! How likely are you to argue with this umpire?

In real life, arguments might still happen. But my experience is that people argue with umpires who act decisively much less than they do with the indecisive ones. The reason is simple: Indecisive umps aren't sure of themselves, so that arguers think that they can change the minds of those umpires and get the calls reversed in their favor. You know that you are not going to change a decisive call that was made by an assertive umpire, so that an argument there is simply venting.

Applying the Keys to Quality Assurance

Now, let's talk about software quality, and allow me to ask a few questions.

Rules

• What are your product's quality-assurance rules?
  • Do you have release criteria?
  • Do you have a customer's "Bill of Rights"?
  • Who makes your rules?
  • Are you clear about any "local rules" for certain areas of your product?
• Are your rules written down?
• Are your rules communicated broadly?
• Do you review and update your rules periodically, based on experience?
• Do you make sure that rules change only between ship cycles, and not in the middle of one?

Calls

• Do you have metrics and other mechanisms to give you the perspective and unobstructed view of the state of your product?
• Do you recognize the QA rules and apply them in making your product decisions (calls)?
  • Do the rules dictate the priority that is given to bugs?
  • Do the rules hold when release criteria are not met and product slips are imminent?
  • Do the rules hold when you are deciding whether to fix or postpone a bug?

Decisiveness

• Do you know who the umpire is on your team?
  • Is it the test manager? PUM? Committee? AI?
• Does your umpire make the calls decisively?
• Are you confident that the decisions are made based on knowledge of the rules and good perspective/unobstructed view of what is happening?
• When there is no rule, does your team make decisive decisions and live with them?

Judgment Calls and Logistics

Now, let's spend a little more time talking about judgment calls. Learning the rules is as simple as dedicating the time and attention that are needed to read and memorize the rule book. Being decisive is simply a conditioning mechanism, in that you must train yourself to react in a decisive manner. Judgments are a different beast, altogether. You get better at judgments via experience. With nearly every game that I umpire, something happens that is new and not covered by the rule book. As you manage these situations, you learn from them; and, the next time that (or something similar) happens, you are more prepared.

The problem is that you could spend 50 years of umpiring before experiencing it all; and, by then, you might have lost too much of your eyesight to be effective. Umpires have a very simple and very effective mechanism for speeding this up: They tell each other a lot of stories. Umpires love to find another umpire and tell them about something new that happened in their last game, or repeat a story that they have recently heard from another umpire. Umpires just love to communicate to each other and swap stories. They argue, they suggest alternatives, they recall similar situations. Often, they call in a very senior umpire to lend an opinion (somebody who has umpired for 50 years). They also make use of chat boards and e-mail aliases, asking questions and getting opinions.

Each local league has an Umpire Rules Committee that is led by the league's Head Umpire, and this group makes final decisions on the rulings in question, which they then communicate out. I liken this to an umpire's version of the State Supreme Court. They sometimes ask the Little League Baseball Rules Committee to make a global ruling. I liken this to the U.S. Supreme Court, and it is essentially the highest ruling body available. The net result of all this is that experiences are readily shared, which greatly accelerates the learning curve.

Logistics are worth discussing a bit, too. Baseball games need at least one umpire, the home-plate umpire, who calls the balls and strikes, amongst other duties. Ideally, there are field umpires, who watch for plays at the bases and in the outfield. If two field umpires show up, one handles the left side of the field (third base), and one handles the right side (first base). They communicate who handles calls at second base, depending on scenarios into which I won't go. The goal is to have umpires who are in positions to make good calls by being able to see the play clearly, with unobstructed vision, from a close proximity.

Note  The closer that you are to a play, the easier it is to see the details. You can see if the fielder has his foot on the base or is not on the base by a fraction of an inch; but you must use your ears to listen to when the fielder catches the ball, to know if the ball gets caught before the runner reaches the base. This all happens very quickly, and it involves developing skills with multiple sensors.

Together, the home-plate and field umpires are referred to as the umpiring crew. During the game, the members of the umpiring crew help each other. If one umpire gets in the wrong position, that umpire might ask the others to huddle up to decide a call. Also, if there is a ruling in question, the umpires will often get together to ensure that they are recalling the rule correctly.

Sometimes, this involves a judgment call, and one of the umpires has experience with or has learned about a similar ruling to be applied. The crew works as a team, with the home-plate umpire being the lead umpire (who has final say). The league makes sure to schedule umpiring crews so as to ensure that they have at least one experienced umpire in the game. Often, this is the home-plate umpire.

With little league, it has been my experience that sometimes only one umpire shows up for the game. This person must become the home-plate umpire; from there, this person must make all of the calls, albeit from a difficult perspective on some faraway plays. I have learned some interesting umpire math, too. Sometimes, 1 + 1 = 0. Let me explain.

I was alone for a game, so that both teams volunteered a parent to umpire in the field. I was pleased, because now I had what I thought were two field umpires, which should have made the game easier to manage. As the game progressed (and I began to realize how inexperienced these umpires were and what a difficult situation I was having, trying to fix mistakes), I recognized that I would have been better off doing the game alone. "1+1 = 0" means that adding a couple of inexperienced umpires will not help productivity. I daresay that, on occasion, 1+1 = -1, and that it causes more harm than good.

The Importance of a Good Umpiring Crew

Are you relating this to software quality assurance, yet? (I'll bet that you are; but, just in case, here goes.) Too often, we think that we can get quality fixed by hiring a bunch of inexperienced testers. When given a choice between two inexperienced testers and one experienced tester, many shortsighted managers opt for the numbers—failing to understand that, just like in umpiring, 1+1=0 sometimes in quality assurance. I want to ensure that I have the right mix of senior folks to guide a team that has an acceptable number of inexperienced folks. Here are some thoughts:

1. Create your QA team by ensuring that the QA crew that you put together has the experience that it needs to get the job done.
   • Additionally, you should recognize when one or two senior folks can be way more productive than an army of inexperienced folks. Don't fall into a Dev/Test ratio trap.
   • Also, you should critically analyze what is the right mix that gets the QA job done right and develops future talent.
2. Ensure that your QA folks work together and help each other make good decisions.
3. Empower your senior folks to mentor and to share their experience with others.
   • Ensure that your senior folks are visibly recognized and rewarded for helping others.
   • Encourage active communication, allowing folks to learn from each other's experiences. This will expedite the learning curve.
4. Communicate who is doing what, and ensure that each person understands how the work that they do fits into the system, so that, together, your QA efforts act as a finely tuned QA machine.
   • Establish the senior QA committee that can be escalated to, for making difficult decisions.

Conclusion

Is umpiring a thankless job? Well, yes and no. You are expected to be perfect, and you will hear it from the crowd if not; so, in that regard, it is thankless. But, actually, it is a very rewarding job, for several reasons. Firstly, you know that you are helping make the baseball game a better experience for both the players and the fans, which gives you a sense of accomplishment. Secondly, the kids, coaches, and parents (from both the winning and losing teams) often thank you for your effort after the game, which shows appreciation. Thirdly, it is neat to get that close to the game and have the umpire stories to tell—which, unlike the talks of your child's first day at school, you find to be met by attentive folks who are interested in hearing your tale (which they, in turn, will likely tell others about later). I think that I'll keep doing it, and I would encourage others to try it out. If you stick with it, I guarantee your decision-making and assertiveness will readily improve.

Is quality assurance a thankless job? Well, yes and no. You are expected to be perfect, and you will be the first person looked to when a problem makes its way to the consumer. Actually, it is a very rewarding job, for several reasons. Firstly, you know that you are helping to make the product a better experience for the customer, which gives you sense of accomplishment. Secondly, the product team members often thank you for your effort, which shows appreciation. Thirdly, it is neat to get that close to the product and have the experience to share with other, less experienced folks. I think that I'll keep doing it, and I would encourage others to try it out. You'll likely find it as rewarding as any other work that you have done—especially, as you become great at it and start to see the impact that you really have on the lives of so many users.

About the author

James Rodrigues is the director of Test Excellence for Microsoft. He and his team are responsible for improving the people, processes, and practices of software test engineering across Microsoft through learning strategies, strategic projects, and partnerships with companywide Test Leadership Teams. Jim is a Microsoft veteran, having managed various test teams and shipped many products across Microsoft. He also holds software patents for areas of performance and test automation. Prior to working for Microsoft, Jim spent five years as a software developer at Loral Infrared & Optics division.