Welcome to Jester's Trek.
I'm your host, Jester. I've been an EVE Online player for about six years. One of my four mains is Ripard Teg, pictured at left. Sadly, I've succumbed to "bittervet" disease, but I'm wandering the New Eden landscape (and from time to time, the MMO landscape) in search of a cure.
You can follow along, if you want...

Monday, July 25, 2011

Key non-conformance indicators

I'm going to delve a bit into the technical aspects of metrics.  Hang on tight.

In the business community, Key Performance Indicators (KPIs) are the important metrics that you use to measure how well your business is doing if your piece of the business is not directly financial in nature.  Let's say we're building iPhones for Apple to sell.  In manufacturing, three important KPIs are the availability, performance, and quality of your manufacturing lines.  The first measures how many hours per day your manufacturing line can run.  The second measures how many iPhones your manufacturing line can produce when it is running.  The third measures what percentage of those iPhones meet quality standards once their manufacturing is complete.  Simple enough so far?

KPIs are important, because they make it easy to see the "levers" that can be used to improve your business.  If you can manufacture 5000 quality iPhones per day, it's easy to see where you can make changes to make that 6000 per day.  You can increase availability (unless it's already at 24 hours/day), or you can increase quality (unless it's already at 100%), or you can increase performance.

Or you can open a second factory.

Once your metrics are in place, changing the underlying assumptions underneath them can make it a lot harder to compare historical data.  If you open a second factory, do you combine the metrics from your second factory with your first?  This will create a large change in the data that you will have to explain every time you show your metrics.  Or do you measure separately?  And if you do that, what happens if one factory (because it is newer, say) has significantly higher performance than the other?  How do you explain the difference month after month?

Things really get tricky if the data start showing things that you don't want to share.  To get to 6000 iPhones manufactured per day, suppose you greatly upgrade your factory's performance at the cost of its quality.  You're producing more sale-ready iPhones, but you're also producing many more defective iPhones that will need rework before they can be shipped.  That's data you might not want to share with Apple.  So one month, you might not publish your quality data.  You publish and highlight your performance data instead, showing the large increase in output, explain the historical difference, and keep the quality data to yourself.

And hopefully, Apple doesn't ask.  And if they don't, it's easier to keep that quality data to yourself next month, too.

Which brings us to today's EVE Online devblog.

Dr EyjoG, CCP's economist in charge of EVE, has traditionally published Quarterly Economic Newsletters about 45 days after the end of the quarter.  I've spoken of them in this blog many times, and the 4Q 2010 was great fodder for two or three blog posts looking at different aspects of the EVE economy.  I've been waiting impatiently for the 1Q 2011 QEN, because I thought it would contain excellent baseline data to compare to the 2Q QEN.  That was important because the biggest economic change to EVE Online since Dominion was made the first week in April, with the sanctum/haven nerf.  That presumably hugely reduced the amount of ISK moving in the game and I was really looking forward to seeing the numbers.

Only, it looks like we're not going to get QENs in the future.

The devblog itself is a complete waste of your time... a little piece of nothing-fluff comparing the four established EVE price indices.  Each index supposedly measures a different aspect of the EVE consumer economy, but as I've written before, the quarterly slices make this data practically useless.  It doesn't help that EVE's Consumer Price Index contains more than 4000 items which range from ships to POSs to ammo to PLEXes: in short, items from durable goods to consumables and back.  It's worthless as an economic indicator.  The graph that accompanies the devblog confirms it showing (as it does) the CPI being completely flat for the last four years.

Despite massive changes to EVE's consumer goods in that time, the CPI graph is flat because there are so many items in the CPI, that big changes to one item are damped down by stagnation in other items.  It's one of the reasons that I wish EVE had a "durable goods index" containing items that are rarely destroyed like Hulks and capital ships.  Taking these items out of the CPI would reduce this stagnating factor.

So, as I said, the devblog is practically worthless.  An EVE player was quick to ask: "is this blog a replacement for the QEN?"  Dr EyjoG was good enough to answer, and his answer isn't comforting:
We will change the format of the QEN but this devblog and the data that comes along with it is an addition to other economic information available to the players. This is a part of a larger plan to give more data directly to players so that they can do their own analysis. And yes, this will continue in the future.

We have not published QENs for 2010 (sic) and will most likely not publish them in the previous format but rather moving the Economic Newsletters to more of an annual report. The frequency of the publication is though still being discussed.
In other words: "yes", this useless crap is replacing the QENs, with some other "annual report" data and "more data directly to the players" being discussed for the future.  In short, I think we can safely assume that the Q1 QEN metrics showed some data that CCP is not in a hurry to share.

Fearless.

7 comments:

  1. My thought was the same as yours when I saw Dev blog. "useless" Most of this is data that players could develop on their own if they thought it was useful enough. However, I didn't think about aspect that reason QEN didn't come out is CCP is hiding data because it's ugly. Good catch.

    ReplyDelete
  2. Another hard hitting and insightful blog post to chew on. I came to the same conclusions reading this and the Doc's subsequent forum post.

    Wow...just...wow. I'm a big fanboi, but this move is really shaking my confidence. Don't be afraid of the truth CCP!

    ReplyDelete
  3. Sigh. I really hope you're wrong, and that EyjoG just decided he was sick of writing QENs or was overworked and the deadline just kept slipping. (See also: what happens to me for every other project ever.)

    The last thing CCP needs is for this to blow up into another PR disaster. :P

    ReplyDelete
  4. Having seen each QEN slip later and later, I think they pulled them as impractical to deliver on time. All the database mining and an analysis, on top of often being a bit dubious, was taking a lot of man-hours from a team that, in theory, is also meant to be working on the game itself, advising on economic issues, like ganglink 'spawn' rates in storyline missions and changing officer spawns to reflect their value.

    ReplyDelete
  5. killfalcon - once a metric is established, the database queries are basically done. They only have to be updated if the database schema is changed.

    Once a metric such as "number of Ravens killed per day" is established, I see no reason to not include that in all future reports, unless the metric is shown to be useless (and it is announced that this metric will no longer be included).

    The CSM's lack of interest in economics is not going unnoticed either. They want to "fix" null sec with their stupid "fields and farms" ideas, but they have no idea what is "broken" about null sec in the first place, and no idea how to measure whether what was "broken" has indeed been "fixed". The same people complaining about "no good fights" are the ones holding back from fights because they aren't guaranteed a win.

    The CSM needs to get their heads around the economics of this MMO. CCP needs to take the economics seriously. We need to have identifiable metrics of what CCP and the players consider to be the "health" of the game.

    I'm convinced that the reduction in the number of ships killed in PvP is directly connected to the plunging value of datacores and the loss of Sanctums/Havens in the bulk of null sec, for example. People aren't going to risk ships in PvP if the ships cost too much time to replace. On the other hand, a braindead metric such as "number of ships destroyed per player" will simply encourage ships-for-aurum. Let people buy more ships to get them blown up, the splodeys-per-player metric will go though the roof while the rest of the game dies.

    I guess that we need to start thinking about what "metrics" make sense to measure on a continuing basis. The number of hulk-hours and super-capital-hours is one metric, but what do they show? I'd like to see some indication of the average-ISK-profit-per-week so that we can get some kind of idea of the time-to-replace-splodeys.

    What metrics actually matter in EVE?

    ReplyDelete
  6. The most apt comparison (for the sake of a broader understand through mild analogy) I can make of the level of ignorant sycophancy here and in the eve-online forum threads about this subject is this: you're fundamentalists. This is like some kind of mild religion for you. Mitigating facts are discredited and discarded off-hand. The idea planted in your heads by The Mittani becomes a mantra of sorts.

    What does this mantra include?
    - The true-sec nerf was implemented because there was too much isk in the game.
    (not an exhaustive list)

    CCP posted several devblogs about this change, but one of them addressed it directly and IN DETAIL. Within that blog, they detailed the reasons for the change. Nowhere within that blog is any mention of "isk sinks" or "too much isk in the game". Did that matter to GSF? hell no. Within hours every goon sycophant was railing in the forums about how it was all about "isk sinks". Bloggers like yourself are repeating the same propaganda.

    Consider the following:
    - All of the data you're looking for can be mined via the game/api by yourself. A deal struck with someone like eve-markets.net to collect throughput (instead of focusing solely on pricing trends) could give you a head start in this direction if you want the hard data.

    - It's at least as plausible (as plausible as your conspiracy-laced theory) that CCP doesn't want to devote the man-hours needed to produce and release this report quarterly. It's just as plausible that someone at CCP asked why an in-game economy should automatically adopt the same reporting customs as the United States. (The main impetus behind having your accounting department produce quarterly reports is taxation, not KPI.) additionally, I'd wager some discussion of the amount of corporate resources used to compile and publish these reports vs. the percentage of their customer base who would actually read and appreciate said data had to be taken into account. A splendid example of the obvious result of such a discussion has already been provided by the first comment to this article. (TL;DR?)

    - EVE is not, by any stretch of imagination, a real economy. It isn't. Transactions occur. Things of virtual value are traded for other virtual goods, services, or virtual currency, but these are all discretionary purchases. No one is going to lose a job, a home, or starve if no transactions occurred tomorrow. It may feel like a full economy as a player, but this is a closed system.

    In closing, I'm sure you had fun putting on your tinfoil hat and trying to expose the big evil corporation hiding things from us, but this is not hard-hitting investigative journalism. Your first here already showed those of us who like to think for ourselves that you're just another sycophant with a blog, when you made the laughable assertion that 5,000 NPC kills in a constellation over 24 hours was irrefutable proof that the new residents were the same as the old. 5 minutes spent testing your theory on evemaps.dotlan.net would have made all the difference. You didn't bother.

    ReplyDelete
  7. @PanicGeddon: OK, you bring up a lot of points. I'll address them one by one.

    The sense that the sanctum/haven nerf was about ISK wasn't something that was pulled out of thin air. As I covered in this blog, the 4Q QEN made a *large* issue of it, as did the Economics session in the CSM December summit. In both places, CCP made it clear that they were extremely concerned about the amount of ISK entering the game, and communicated their intention to do something about it, and soon.

    Let's say your position is correct, and the sanctum/haven nerf wasn't it. The sanctum/haven nerf had nothing to do with reducing ISK faucets. Fine. I invite you then to tell me what *was* it. What *did* CCP do to fulfill their promise of doing something about the amount of ISK in the economy, and soon?

    Second: you specify that we can somehow see the amount of ISK entering the game using player-available tools. The only problem with this is that you're dead wrong. The only source we had for that data was the QENs. eve-markets uses data provided by EVE Central, and is market and price data. It doesn't say a thing about bounty ISK. If you have an actual source for *that* data, tell me and I'll happily pull it and use it.

    Third: you're operating under the assumption that QENs are produced for the use of the players, which is just silly. CCP produces that data internally for their own use. QENs also include position papers on various aspects of the EVE economy, most recently supercap proliferation and mineral compression. CCP produces that data internally for their own use as well.

    These aren't guesses, by the way. They're facts. Know how I know they're facts? Because I have a copy of the first QEN, from 3Q 2007. It includes dozens of metrics going back to summer 2006! Hell, the very devblog you're beating me over the head with has the four price indices going back to 2003! CCP was producing this data for their own use *before* they got the cool idea to share it with the players. I guarantee you they're not just gonna stop producing this data.

    I *will* grant you that there's some measurable amount of additional work done to make the raw data and the position papers "ready for prime time" viewing by the players, but the data itself is *not* something that CCP is producing for the benefit of the players alone. You're deluding yourself if you think that's the case. The fact that they can present this data to players is a bonus, not the end result.

    Finally, you can't take the CCP apologist position in the rest of your comment, and then say the EVE economy isn't real at the same time. You have to pick one of those two. The player-driven EVE economy is one of only a very few competitive advantages that EVE has compared to other MMOs. If "EVE is real", then its economy is real as well. Certainly, most players operate under the assumption that it is.

    ReplyDelete

Note: Only a member of this blog may post a comment.