If you are a responsible person at all, sooner or later in your career, you'll be asked to take a leadership position. In this role, you may find yourself evaluating the work of others. When that happens, I hope that you will remember the first rule of evaluating their work: measure results, not effort.
This goes for any manager or supervisor or project manager: results first. Then ask if the work was delivered on time and under budget. But at the end of the day, you should care about the work produced, not the effort that it took to get there.
Of course, if the project is at the mid-point of its development when it comes time to evaluate the staff, you might not have easy access to results to measure. Even without this factor, it will be tempting to measure effort. Effort is very easy to see. If someone is working 60 hour weeks, week after week, that's easy to measure. If they explain to you what they're doing is really hard, that's also easy to understand. If the entire team working on a project is stressed trying to meet a deadline, their hard work and effort will be obvious.
It will even be easy and tempting to value the work based on the effort that went into it. I can't tell you the number of poor managers, supervisors, and project leaders that I've seen that reward long hours of overtime put into a project instead of the results that were delivered with that overtime.
This is flawed thinking. As a supervisor or project manager, you need to be thinking exactly the opposite of this.
If I show you a completed house, and tell you that it was completed on time and within its budget with the requisite materials and following all applicable laws, as a supervisor, you should not care how it was done. It's the results that matter, not the effort that went into the building of the house. You shouldn't care if it was done with pre-fabricated sections or raw lumber hauled to the site. All you should care about is that the house is done. You should congratulate the team that did it and the foreman that led the team on a job well done.
Now suppose I show you a second house being built right next door. It had the same budget and schedule, the same materials provided, the same laws expected to be observed. Only, this house is half-complete. It's obviously well behind schedule. You, as the manager, ask the construction foreman what went wrong, and he says "Nothing went wrong! This is going to be a type of house that has never seen before! A huge amount of work is going into producing it. My team is working very long hours, and we're using innovative house-building methods. It's really, really cool!"
As a manager, you should rightly be extremely skeptical of such "results." The supervisor may even ask you to reward a team in this situation for all of the hard work and extra effort they're putting into a project, even though it's incomplete. But put into the terms of two houses under construction, even a layman can see that this is a ridiculous expectation. It's even sillier to value the second house over the first house just because more work is going into it.
Which brings us to software development in general, and CCP's software development in particular.
As I said above, as a supervisor or project manager, you can't and you shouldn't reward effort, if that effort does not produce results. Extra effort, in fact, is counter-productive to your aims as a manager. You want the work that you set your teams to do to be completed with as little effort as possible. A team that has exhausted itself producing the first work task is going to be of very little use to you when it's time to assign them another. A team that easily accomplishes work with less effort, on the other hand, is the team that should be rewarded.
It's counter-intuitive, I'll grant you. But reward results, not the effort that it took to get them.
Too often in CCP, we see effort being valued and rewarded, not results. CCP will congratulate themselves on doing things that have never been done before. That's exciting, sure. But -- and let's be honest here -- if that innovation doesn't also produce results, the effort that goes into the work doesn't matter. You and I as players don't much care how the work is done as long as it is done.
I covered some of this back in April when Incarna first appeared on Singularity and I had a weekend to play around with it. As I put it then...
The CQ comes off as a fourth-year CompSci student's attempt to recreate the Unreal engine. It works. It would even get a good grade on the final. But it would leave a true pro tsk'ing at the student's many mistakes and foibles.Too often, CCP falls into this trap. They get very excited about working on "cool stuff", and then spend a tremendous amount of effort to produce that cool stuff, even when it's unnecessary. This was driven home particularly with the new font that's going to appear in-game. CCP's UI team worked on this for months. I was reminded in the comments of my recent font post that the new font was announced at Fanfest in March, and we've seen occasional glimpses into this new font in the intervening months since then. Everyone, right up to CCP's uppermost management, gets very excited about the huge amount of effort that goes into these tasks.
And it would leave the other CompSci students grinning and wondering why the nerdy kid spent 650 hours on his project to get an "A" when they spent 120 hours on theirs to get the same "A".
And then they become upset when we don't share their enthusiasm for their effort.
But the funniest thing about measuring results, not effort, is that gamers do it instinctively.
Take game reviews. Let's look at Duke Nukem Forever, for instance. This is a game that went through an enormous development cycle, taking years. The developers switched engines at least three times during its development. But the reviews -- and rightly so! -- gave them no credit for all of the effort that went into producing this game. Only the results -- how good was the game? -- counted, and they weren't that great.
I've mentioned CCP's NIH disease before, several times, and this is part of the problem, of course. But the main problem is that CCP management needs to retrain themselves to value results over effort. If the results aren't there, the amount of effort that went into producing them shouldn't matter.
Remember it if you're ever put into a similar position.