Welcome to Jester's Trek.
I'm your host, Jester. I've been an EVE Online player for about six years. One of my four mains is Ripard Teg, pictured at left. Sadly, I've succumbed to "bittervet" disease, but I'm wandering the New Eden landscape (and from time to time, the MMO landscape) in search of a cure.
You can follow along, if you want...

Monday, March 5, 2012

Garbage in, garbage out

Let's play a little game.  I say "The most important person associated with a baseball team is the manager."  Do you strongly agree, agree, disagree, or strongly disagree with that statement?  Take a second and think about it, then answer.

Have an answer?

Good.  It was a trick question.  No matter what you chose, you're wrong.

Let's say you agree: the most important person associated with a baseball team is the manager.  Obviously, you disregard the importance of the owner, the pitching staff, the star center fielder, et cetera.  Well, let's say you disagree instead: the most important person is not the manager.  Well, in that case, you obviously disregard the importance of the manager.  Obviously.

At this point, you can say, "that's not what I meant" and sure, sure, I believe you.  Know why?  Because every answer to that question is the wrong one.  And every answer to that question is the right one.  All at the same time.  It's the question itself that's flawed, because it states the question in absolutes: who is the most important person.  But everyone looks at an absolute differently, and a lot of people don't see absolutes at all.  That's because absolute phrasing of this kind of question is flawed and meaningless.  Let's rephrase the question to illustrate:

"The most important part of your car is the engine."

Whether you say yes or no, you're both right and wrong at the same time because an engine without a steering wheel, tires, a chassis, and... oh yeah... a driver... is an object that ain't gonna go anywhere.

Now suppose you're asked 39 other questions framed the same way at the same time.  And let's suppose you're asked the same 40 questions a week later.  Do you think it's possible that your answers would change?  I think it's not only likely, it's inevitable.  Because of the way the questions are framed in absolutes, you might lean toward nuances in one regard one day and nuances in another regard another day.  But the very nature of a survey with 40 questions dictates that your "agrees" and "disagrees" will be folded into a numeric matrix where all nuance is completely lost.

It's another absolute piled on top of 40 smaller absolutes.  The result is going to be a meaningless hash.

Which brings us to Vote Match, Dierdra Vaal's attempt to create a survey to match you with your preferred CSM candidate.(1)  The theory is simple enough: each CSM candidate answers 40 "agree/disagree" statements.  You then answer the same 40 agree/disagree statements.  And a little bit of weighting aside, the system matches you with the CSM candidate that feels the same way you do on the issues described in the 40 questions.  In theory, it's a fine idea.

In practice, it's executed sloppily and with no regard for nuance.  Computer programmers have a name for the results of this kind of algorithm: garbage in, garbage out.  Since the answers of the CSM candidates are stripped of nuance and the questions are phrased in absolutes, their answers are meaningless.  That's the "garbage in" part.  Since you fill out the same survey they do, your opinions are likewise stripped of nuance.  That's the "garbage out" part.

Seriously, you may as well assign the CSM candidates random numbers and roll some dice.  The results will be about as meaningful.

Let's look at some specific examples, shall we?  And to reduce accusations of bias, I'll concentrate on four candidates that will almost certainly be elected no matter what I say: Seleene, Trebor Daehdoow, Two Step, and The Mittani.

Seleene works the system.  He tagged the following three items as "strongly agree" and flagged all three as important issues:
  • The industry side of Eve needs more game design attention and should be a priority over all other areas of the game that need improvements.
  • Improving low sec is the single most important thing for CCP to do and should be a priority over all other areas of the game that need improvements.
  • 0.0 is most in need of further development, and should be a priority over all other areas of the game that need improvements.
Wow, three things that should be given a priority over all other areas of the game.  And Seleene strongly agreed with all three statements.  Did he misunderstand the questions?  After all, there surely can't be three areas that receive priority over each other, all at the same time.  There can only be one, right?  Obviously not.  Seleene understands that CCP can (usually) walk and chew gum at the same time and there's room for several priority areas of development even if Vote Match doesn't.  This is very, very smart.

And coincidentally, he increases his match percentage with people who feel the same about all three areas.

The Mittani, meanwhile, strongly disagreed with two of those statements and only strongly agreed with one of them (guess which one).  As a result, The Mittani clearly disregards the importance of industry and low-sec.  Obviously.

Trebor takes nuanced positions, then tries to explain them in the comments.  In particular, he disagrees (not strongly) that 0.0 is the highest priority issue, then tries to explain "It's not that 0.0 doesn't need work.  It does.  But other areas of the game affect more players."  He also didn't flag 0.0 as important.  Is anyone who lives in 0.0 (and therefore puts at least "agree" on this question) going to see that?  Nope.  They're going to see that Trebor disagrees with them, doesn't see their play-style as important and therefore will be much less likely to match with them.  Trebor also disagrees with the statement "The CSM's primary role is to identify the good ideas generated by the player base and suggest them to CCP.", explains it in more or less the same way, and in the same fashion again trips over nuance.

And coincidentally, he decreases his match percentage with people who feel differently, because nuance is bad.

The Mittani ignores the system.  He answers the questions the way you'd expect him to, but doesn't attach comments to anything.  Only a few other candidates did that.  It's commendable in a "you know what you're getting" sense.  He also selects "no opinion" more than any other major candidate.  Notably, he does what Trebor should have done: he selects "no opinion" on every single question that even mentions "CSM" except one: that the CSM should stand in the way of micro-transactions.  That's also very, very smart.

And coincidentally, he increases his match percentage with people who ignore the questions about what the CSM is about and sticks to the issues themselves.

Two Step makes rookie mistakes, and makes them again and again.  That surprised me because he should have encountered this mess last year and learned from it, and he apparently didn't.  First and most amusingly, he tags the 0.0 issues as his important ones, even though he's not a 0.0 candidate and he's unlikely to get many 0.0 votes.  That immediately reduces his match percentage with anyone likely to vote for him.  He makes the same mistakes as Trebor in terms of taking nuanced positions and then trying to explain them in comments.  For instance, for the three "priority" items, he actually ranks them in order.  What he should have done was matched Seleene's move and ignored nuance.

He also makes the most fun mistake of all: he takes positions that are in opposition to his comments.  For instance, on the question "0.0 Logistics (moving, managing and creating supplies) is too safe at the moment.", he selected disagree.  But in the comments, he wrote, "It may be a little safe, but it is also still a huge pain point, and something that makes it hard for smaller groups to live in 0.0."  So... wait.  Wait.  Do you disagree or not?  ;-)  Similarly, for "Jump bridges and Titan bridges should be nerfed to reduce power projection.", he selected disagree, but his comment makes it clear that he agrees.

And coincidentally, he pisses off people who feel either way on these issues...

So yeah... Vote Match?  Skip it.  Seriously.  Take the time to actually learn what the candidates think.  You might even... gasp... ask them questions yourself!  And in the process, take this particular means of choosing a candidate and chuck it in the garbage where it came from.

(1) I'm not going to link it, because I'm not going to encourage you to use it.  Matter of fact, I'm going to encourage you not to use it.  Keep reading.


  1. Do you feel it is impossible to construct a useful set of questions that would actually help a voter identify the candidates most aligned with their own positions on the issues and concerns about the game?

    1. Well i wouldn't call it impossible but it will be hard.

      IMO you need to "know" the guy/girl you are voting for and this "hard stated" answers don't reflect the personality behind the candidate. I would like to the something more like a psychological profile to determine if the candidate recognize the game in the same way i do.

      I strongly encourage everyone who takes the time to read about it to talk about it too. Your friends may not be interested in reading it but maybe they accept you as reliable extension for csm matters. Since you already play together chances are that you are like-minded and whom you think is good for eve is good in your friends opinion too.

  2. Karbox DelacroixMarch 5, 2012 at 7:29 PM

    The Mittani, he feels your pain.

  3. I guess it comes down to how you read the question. I don't think 0.0 logistics is *too* safe (I think it is pretty close to the right level of risk), but I still think it is a pain and something that keeps small groups from 0.0.

    For the jump bridge question, I think the question was worded incredibly poorly. I don't think the solution to power projection is to simply nerf jump and titan bridges, but that doesn't mean I don't think *titan* bridges need some looking at.

  4. My answer to the most important part of baseball was : Steroids, the money you make, news writers writing nice things about you as a player, and of course last but not least the trading cards of baseball players and getting them signed.

  5. And this news just in: it's just as easy to game the questions as it is to game the answers.

    My argument about logistics is that it's too easy: there is no thinking required when you're faced with the decision of "mine minerals locally" versus "import compressed minerals from Jita". I don't give a hoot about whether it's too safe at this point in time. Which way do I choose on that statement then? Whether the solution to "too easy" ends up making logistics "safer" or "more dangerous" in the future is irrelevant right now.

    The "Vote Match" quiz is good for people who are prepared to think about their answers and analyse the answers provided by the candidates. The way it's presented means that incompetent voters are going to be treating "Vote Match" as advice. In fact Diedre Vaal advertises the service as an advisory tool.

    Democracy is based on the assumption that all participants are rational players with perfect information, just like that other game we love to play called "Economics". Both are fundamentally flawed, but both are just a little bit better than any alternatives that have been tried in the past. Another of the invalid assumptions of the democratic system is that all participants are interested in the greater good (because anthropologists tell us that social creatures tend to make decisions for the good of the herd, not the good of the self), when in many cases that is patently untrue.

    Vote Match will ensure that the players get the representatives they deserve, regardless of which representatives they might need.

    Rather than a series of "agree/disagree" questions, vote match could provide some meaningful information if the questions required participants to make choices between sets of things: "Of these three facets of the game, which requires the most attention from CCP for the next 12 months: Sovereignty, Wardecs, Industry".

    Even better, pull out the obvious hot issues and replace them with something else: "Wardecs, Industry, Ship Balancing".

    This would set up a "personality inventory" style of questionnaire where internal priorities can be discovered, and the service could match punters to the candidate who most accurately reflects their internal priorities. One service the Vote Match system does provide is to help identify candidates that I previously wouldn't have paid any attention to.

    In the meantime, for the punters like me the best option is to answer the questions honestly (and brutally), then cut out the obvious gamers such as people suggesting that three mutually exclusive decisions should be made at the same time. But that means you behave as a rational participant attempting to gain perfect information. Which we know doesn't happen.

  6. The questions were somewhat crap, there was no questions about wspace, there is one about FW but nothing about there? I guess the under 6% of characters (over 5mil SP) that live in wormhoeles and some other daytrippers are really not as important as a dead part of the game. no I am not dissing FW, I am saying that unknown space must have been also represented in questions.

  7. An interesting analysis, Jester. It brings up a lot of things I wouldn't have otherwise thought about. Believe it or not I just did the survey, tagged the stuff I thought should be tagged and that was that. Your 'walk and chew gum' comment is pretty accurate. I don't really see the reason to pick one 'small' issue as important. v0v :)

  8. Pretty much the reason I didn't filled out the poll. "Contrived" was the term I used when I discussed with Bagehi and the powers that be.

    I have to agree with Seleene's comment, he pretty much nailed it.

    Still I do praise the effort put into the whole vote match thing, but shamefully there is no algorithm to predict voting, charisma and spreading a sound message is what really sells politics.


    1. That's just the thing - I appreciate the effort Val put into Vote Match; I certainly couldn't do something like it and see it as just one more tool in the voter's box. They can take away what they want to. Saying that I was "very, very smart" for the way I did it... heh, well, I spent maybe 30 minutes on the whole thing. I answered the stuff and didn't think much about how the whole thing 'worked' which is why I find Jester's analysis interesting.

  9. The garbage-out part is the CSM candidate that is recommended. Your answers to the questions are also part of the garbage-in. Think of it as a function with two sets of input and one output.

  10. I hope that CSM crap is soon over, so we can find some interesting stuff on your blog again :)

    1. Well, there's an interesting thing below called Serfdom that I kinda like, and tomorrow or the next day, a long post called Snapcount that you'll like. ;-)

    2. Would that be re Project Snapshot?

      Hope so, I did 2 runs of 10 systems each. Interested to get a broader picture of whether it was worth it...

  11. Or you can do what I did:

    1: do the survey
    2: select the questions you find interesting
    3: look up the candidates that you know have opinions or you find interesting yourself
    4: read their comments.
    5: use that information

    In that way votematch works well. I know personally I found the comments on the questions posed quite interesting and enlightening.

  12. Speaking of "CSM Crap" there was a debate, of sorts, between The Mittani and Riverini recently on EVE Radio, you might be able to listen to it on the Rewind thing, or whatever the archive is for.
    Would be interested in your opinions on it.

  13. ^^^ awesome!! All the new stuff I learned about CSM and politics is cool and all, but there's only so many times I can read the same names without losing interest in the CSM entirely. Loved the Serfdom article, very interesting, if a bit pessimistic.

  14. Hear that? It was the sound programmers' heads everywhere hitting their desks as you defined garbage in, garbage out incorrectly.

    1. Actually you could argue that GIGO applies in that the Garbage In is the "Best Candidate Selected" (which I think everyone agrees is bad) and the Garbage Out is a corresponding vote cast for a CSM rep.

      Although that woud require CSM voters to be unthinking automotons.

      (also it's not how he defines it, which, you're right, is incorrect.)

  15. I think it could be done in two parts. The first is the questions, and those could be given some more though to minimize how much these nuances affect the result.

    The second part could be a resource allocation simulation. it starts with "Put yourself in the role of a CCP manager. You have 100 developers and can put up to 20 working in the same issue without affecting too much the overall performance. How would you allocate your resources over the following areas of gameplay: highsec, lowsec, nullsec, WH, FW, PvE content, ships/mods balancing, industry, WiS"


Note: Only a member of this blog may post a comment.