Welcome to Jester's Trek.
I'm your host, Jester. I've been an EVE Online player for about six years. One of my four mains is Ripard Teg, pictured at left. Sadly, I've succumbed to "bittervet" disease, but I'm wandering the New Eden landscape (and from time to time, the MMO landscape) in search of a cure.
You can follow along, if you want...

Thursday, September 29, 2011

Measure results, not effort

Warning: geek philosophy ahead.

If you are a responsible person at all, sooner or later in your career, you'll be asked to take a leadership position.  In this role, you may find yourself evaluating the work of others.  When that happens, I hope that you will remember the first rule of evaluating their work: measure results, not effort.

This goes for any manager or supervisor or project manager: results first.  Then ask if the work was delivered on time and under budget.  But at the end of the day, you should care about the work produced, not the effort that it took to get there.

Of course, if the project is at the mid-point of its development when it comes time to evaluate the staff, you might not have easy access to results to measure.  Even without this factor, it will be tempting to measure effort.  Effort is very easy to see.  If someone is working 60 hour weeks, week after week, that's easy to measure.  If they explain to you what they're doing is really hard, that's also easy to understand.  If the entire team working on a project is stressed trying to meet a deadline, their hard work and effort will be obvious.

It will even be easy and tempting to value the work based on the effort that went into it.  I can't tell you the number of poor managers, supervisors, and project leaders that I've seen that reward long hours of overtime put into a project instead of the results that were delivered with that overtime.

This is flawed thinking.  As a supervisor or project manager, you need to be thinking exactly the opposite of this.

If I show you a completed house, and tell you that it was completed on time and within its budget with the requisite materials and following all applicable laws, as a supervisor, you should not care how it was done.  It's the results that matter, not the effort that went into the building of the house.  You shouldn't care if it was done with pre-fabricated sections or raw lumber hauled to the site.  All you should care about is that the house is done.  You should congratulate the team that did it and the foreman that led the team on a job well done.

Now suppose I show you a second house being built right next door.  It had the same budget and schedule, the same materials provided, the same laws expected to be observed.  Only, this house is half-complete.  It's obviously well behind schedule.  You, as the manager, ask the construction foreman what went wrong, and he says "Nothing went wrong!  This is going to be a type of house that has never seen before!  A huge amount of work is going into producing it.  My team is working very long hours, and we're using innovative house-building methods.  It's really, really cool!"

As a manager, you should rightly be extremely skeptical of such "results."  The supervisor may even ask you to reward a team in this situation for all of the hard work and extra effort they're putting into a project, even though it's incomplete.  But put into the terms of two houses under construction, even a layman can see that this is a ridiculous expectation.  It's even sillier to value the second house over the first house just because more work is going into it.

Which brings us to software development in general, and CCP's software development in particular.

As I said above, as a supervisor or project manager, you can't and you shouldn't reward effort, if that effort does not produce results.  Extra effort, in fact, is counter-productive to your aims as a manager.  You want the work that you set your teams to do to be completed with as little effort as possible.  A team that has exhausted itself producing the first work task is going to be of very little use to you when it's time to assign them another.  A team that easily accomplishes work with less effort, on the other hand, is the team that should be rewarded.

It's counter-intuitive, I'll grant you.  But reward results, not the effort that it took to get them.

Too often in CCP, we see effort being valued and rewarded, not results.  CCP will congratulate themselves on doing things that have never been done before.  That's exciting, sure.  But -- and let's be honest here -- if that innovation doesn't also produce results, the effort that goes into the work doesn't matter.  You and I as players don't much care how the work is done as long as it is done.

I covered some of this back in April when Incarna first appeared on Singularity and I had a weekend to play around with it.  As I put it then...
The CQ comes off as a fourth-year CompSci student's attempt to recreate the Unreal engine.  It works.  It would even get a good grade on the final.  But it would leave a true pro tsk'ing at the student's many mistakes and foibles.

And it would leave the other CompSci students grinning and wondering why the nerdy kid spent 650 hours on his project to get an "A" when they spent 120 hours on theirs to get the same "A".
Too often, CCP falls into this trap.  They get very excited about working on "cool stuff", and then spend a tremendous amount of effort to produce that cool stuff, even when it's unnecessary.  This was driven home particularly with the new font that's going to appear in-game.  CCP's UI team worked on this for months.  I was reminded in the comments of my recent font post that the new font was announced at Fanfest in March, and we've seen occasional glimpses into this new font in the intervening months since then.  Everyone, right up to CCP's uppermost management, gets very excited about the huge amount of effort that goes into these tasks.

And then they become upset when we don't share their enthusiasm for their effort.

But the funniest thing about measuring results, not effort, is that gamers do it instinctively.

Take game reviews.  Let's look at Duke Nukem Forever, for instance.  This is a game that went through an enormous development cycle, taking years.  The developers switched engines at least three times during its development.  But the reviews -- and rightly so! -- gave them no credit for all of the effort that went into producing this game.  Only the results -- how good was the game? -- counted, and they weren't that great.

I've mentioned CCP's NIH disease before, several times, and this is part of the problem, of course.  But the main problem is that CCP management needs to retrain themselves to value results over effort.  If the results aren't there, the amount of effort that went into producing them shouldn't matter.

Remember it if you're ever put into a similar position.

5 comments:

  1. Conveniently, there's a name already in place and widely accepted for CCP's apparent mindset, though I've always heard it applied and mentioned in the context of we Americans far more than Euros, but hey, the tides they are a changing. Anyways, that term? Narcissism. Why else would they put SO very much energy into taking a game (that, of course, embraces highly narcissistic concepts -- you don't necessarily have to "play well" with others, just be able to manipulate them), turning it into Barbies In Spaaaaaaaace, and then balking when the playerbase has the unmitigated gall to question their all-wisdom and knowing-ness about THEIR video game.
    I'm sure if we handed CCP's senior (hell, even junior) management an unlabeled copy of the Narcissistic Personality Inventory, they'd score very highly ... and concurrently pat themselves on the back for scoring so highly on a test, even if it measures rampant douchbebaggery (see also: Mittens?).

    The good news? Many books and articles are available on the subject of narcissism and how it affects [fucks up] businesses, governments, leaders as individuals, etc. Of course, CCP would never bother with reading that information, since they didn't write it. ;-)

    ReplyDelete
  2. Kinda makes me think the "What you do, not what you say" meme would be better phrased as "What you produce, not what you promise." Really good post.

    By the way, if you look back in the CSM Minutes from June 2010, there was a discussion of the font way back then, and CCP said they were having a font house design a custom font for them. I'm guessing that--and all the case it cost--went out the window? Not really quite sure what they're ending up with but from the little bit I've seen talked about it among players, it's certainly not something that warranted 15 months of effort if not more.

    ReplyDelete
  3. Perhaps the "custom font house" was some Icelandic kid who wanted a leg up in the world of typographic design?

    But regardless, here's a thread over on the forums where I've mentioned this post. https://forums.eveonline.com/default.aspx?g=posts&m=119939

    That thread is about maintaining effective relationships ("how to stay married and not kill someone"). But it's meaningful in this context ("how to please your customer") too, IMHO.

    ReplyDelete
  4. Its not rocket science making a font, maybe they should have just bought one of the many many fonts available for commerical use...
    Surely someone has made a sci fi based font thats readable at various resolutions required...

    CCP seems to waste an amazing amount of resources trying to do things that have already been done.

    Incarna could have been fully functional by now, just purchasing the Unreal engine, and rolling it out. Still waiting to see how they are going to manage to render 1500 people docked in Jita, its going to have to be full of "instances" the shop, the corridor, the CQ's.

    Sorry Concord Station control has locked you into the Nex store as there is too many people in the corridor outside.

    ReplyDelete
  5. Personally I think some (or a lot) of the problem is that its easy to say "lets buy a package that does the job" but its quite rare these days to find something that will do the job that doesn't have too many strings attached.

    Something like the unreal or hero MMO engine are quite likely to have some sort of royalty cost associated with them that will eat away at whatever profit they get and since CCP intends to use this technology in WoD as well your doubling the money that your essentially giving away to another company.

    While to us (the customer) we want good results in a short time frame the long term objective of any company is to make money and its a gamble that CCP is willing to make that a short term loss (in terms of time and development cost) will turn into a long term gain (in terms of saved revenue from not having to pay out royalties) - especially if this engine will be deployed in multiple games.

    Now would we be complaining if they'd been working on this behind the scenes for years and got it all polished, working and efficient? Personally I can see its potential but its far from ready and its CCPs fault for taking the roast out of the oven while its still cooking; but even then most players would agree that they are wasting time building content that people just aren't interested in to try and gain new players while losing the ones they've already got.

    Sounds like the inmates are running the asylum at the moment; its take a while but I think CCP are beginning to see the errors of their ways.

    ReplyDelete

Note: Only a member of this blog may post a comment.