Welcome to Jester's Trek.
I'm your host, Jester. I've been an EVE Online player for about six years. One of my four mains is Ripard Teg, pictured at left. Sadly, I've succumbed to "bittervet" disease, but I'm wandering the New Eden landscape (and from time to time, the MMO landscape) in search of a cure.
You can follow along, if you want...

Wednesday, September 14, 2011

Tin-foil hat: 2000 AUR, or 163 million ISK

It's time for another episode of "Jester connects the dots between a lot of apparently unrelated items, and in so doing, adds another layer to his tin-foil hat."

Quick.  You want to check the price of an item available for sale on the EVE market without having to log in.  What do you do?  If you said, "visit EVE Central or EVE Marketdata", you get one point.

Bet very few of you said "wait for market data provided by CCP's statistics team", but sure, you have that option, too.  A little-noticed devblog posted a couple of weeks ago gives you this option: CSV- or SQL-formatted data of several months of market data.  CCP's Research and Statistics unit is talking about producing similar data periodically after a "reasonable delay", "one month, two weeks, several days?" being the listed options.

It's not real-time data, of course, but neither is EVE Central, whose data is produced through filtering cache file data from the game and then using an automatic uploader installed on a player's machine to upload that data to the website.  As a result, the data is only as good as the players that use the uploaders, but quite a few of them do.  A number of players have market alts who sit in the markets all day long, and the auto-uploders cheerfully pull the market cache data for thousands of items on the market to fill EVE Central's database.

Next, let's examine one of the reasons for the recent $99 commercial license debacle.  CCP indicated that some sort of fee was reasonable to ask because many "commercial" applications of EVE data -- kill boards, skill point trackers, and the like -- use data pulled from the EVE server APIs to operate.  CCP felt that it was fair that producers of such data-pulling applications should pay a nominal fee to CCP because of the extra load and bandwidth need these applications place on their servers.

Finally, let's look at a recent post to the older EVE Online forum by CCP Sreegs in CCP's security team.  Sreegs is leading the fight against bots in EVE Online.  He says:
We're going to be discussing what constitutes a bot and what doesn't in an upcoming devblog to clear up any ambiguity. I will note as a word of caution that we provide the API as a gateway to obtain the information we want you to be obtaining and that this is not the purpose of any of the files which are delivered as a part of the client.
(Scroll way down to the bottom of the page if you're not taken directly to his post.)

What do these three things have in common?  Interesting question.

It's totally paranoid, of course, but suppose CCP banned the use of auto-uploaders to pull cached data?  A literal interpretation of Sreegs's post above could easily be used to call these auto-uploaders "bots".  This would put EVE Central, EVE Marketdata, and all similar sites instantly out of business.  But of course, such sites would no longer be needed because CCP would be providing the needed market data... right?


Or am I connecting the dots to reach a totally paranoid conclusion?  What do you think?


  1. I can see how you may have come to that paranoid conclusion, but I think the comment from Sreegs should be taken in context as a response to a question about something that most people would consider to be a bot (the Eve Mentat thing) which does something automatically that a user would normally do manually.

    Granted, the comment was rather vague and had a sort of veiled threatening tone.

    And, at the end of the day, if CCP actually provided the market data (with a minimal delay) via the API wouldn't it be better than what we get through EVE Central or EVE marketdata without allowing the sort of automation that EVE Mentat does?

  2. not an unreasonable paranoia on your part given CCP's actions in the past, ie using a cannon to kill a fly.

  3. I'm not sure if it really would put those sites out of business, simply because I think that CCP would simply provide the APIs to gain access to the data but the user would still need to provide the front end and a way to display the data.

    Yes if they had a nominal fee I guess it wouldn't keep everyone happy but most of these sites are ad run anyway and a small fee wouldn't impact the bottom line that much, and some would possibly be even willing to pay if it could provide much more accurate and useful information.

    I guess its a fair comment of CCP to ensure that their hardware bills are being paid and I could also imagine that CCP could even compete themselves and open a website showing the information and try to advertise its own products but we would hope that with all thats going on at the moment that CCP should be putting its effort into FIS than building more failsites :P

  4. Since CCP are going to have to wait a few months before non vanity ingame items hit the NEX store it would only make sense that they would kill third party developers so they may put their own apps into the store.

  5. No idea if you are paranoid or not but all of that makes sense. However, I believe the real truth behind all of that is much darker than you suspect. Why do you think DRF is so powerful at the moment? Botting much and CCP is not batting an eye at them.

  6. http://www.eveonline.com/ingameboard.asp?a=topic&threadID=1571803&page=1#23

    CCP Stillman, comment #23:

    I just want to address this one already.

    From a technical point of view, this is completely impossible I'm afraid. The amount of data it takes to generate this sort of data is absolutely huge. There has to be a delay, because of that. And it has to be measured in days.

    From a design point of view, there's also the concern about people getting a huge advantage from getting this sort of information.

    So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?

  7. I honestly always figured CCP that considered the market bots to be an exploit.

  8. The comment you quote is in relation to EVE Mentat, which uses Javascript in the IGB to automate the acquisition of market data while the player is AFK (i.e.: it's a bot).

    The statement has been made in the past that collecting market data from the cache is acceptable since that's just replicating the behaviour already available via "Export" in the Market window.

    CCP Stillman claims that presenting an API to present market data is "completely impossible" which to me sounds like he's gotten himself mixed up in the brain. Getting live market data via an API is completely possible: it's part of the existing game client.

    Now consider a future where there is an API for collecting market data, and CCP states that cache parsing is considered violating the EULA: in this situation the outcome for sites such as EVE Central is improved, since they will have access to reliable data.

    The only question is whether CCP will allow non-commercial use of the API without the $99 licence.

    "Extra load" is bunkum since many players already produce market data by browsing the entire market - thus there will be multiple players in one market region at the same time. Using an API will reduce the load since sites like EVE Central will only be polling the API once per unit time, rather than multiple instances per unit time.

    I have about 200 items on my quick bar, which I'm continually polling as I fly around. Open market, click first item on Quickbar, hold down the "down" button. A few minutes later, the last item is selected, so I hold down the "up" button. No automation involved, but a lot of load on the servers.

    So in short, no you are not paranoid. I fully expect CCP to release a live market data API and declare cache parsing a EULA violation at some point in the future. But that hasn't happened yet.

    The question still remains about whether the $99 licence will only be charged to commercial users of the API.

  9. Yeah I don't buy it either. CCP says that automated market data and transaction history is too big a load on the servers, when thousands of players do exactly the same thing in smaller, redundant batches every day?

  10. CCP cutting off the ability to upload market data has always been a threat, and there's a number of ways they could do it. I don't think there's necessarily anything new to worry about here.

    In some situations I would trust a CCP feed less than I would EVE Central (MarketData, Marketeer et al). There are numerous occasions where, to cover up exploits or their own ignorance, they've zeroed-out historical data in-game - and presumably the doctored feed would be the one served...


Note: Only a member of this blog may post a comment.