Welcome to Jester's Trek.
I'm your host, Jester. I've been an EVE Online player for about six years. One of my four mains is Ripard Teg, pictured at left. Sadly, I've succumbed to "bittervet" disease, but I'm wandering the New Eden landscape (and from time to time, the MMO landscape) in search of a cure.
You can follow along, if you want...

Monday, May 2, 2011

White-out paper

It's rather amazing the amount of nothing that CCP can put into 26 paragraphs sometimes.

On Friday, April 8, the EVE forum debacle occurred.  Late on April 11, CCP Sreegs wrote an 11-paragraph devblog on the subject.  That devblog included two paragraphs about what the forum exploit could be used to do, plus two paragraphs about what it couldn't be used to do.  It then contained several paragraphs about how "cool dudes" would report vulnerabilities that they discovered in CCP code.  It contained zero paragraphs describing how such an embarrassing mistake could be made in the first place.  It also contained zero paragraphs about the testing methodology that was used that could miss such a simple exploit.

This morning, CCP Rhayger of the web team wrote a 15-paragraph devblog on this same subject.  It spends ten of those 15 paragraphs detailing some of the back-story, including why and how YAF was chosen to be the foundation for the new forums.  One of the amusing memes on the subject of the new forums was the claim that 30 man-years of development time had been spent "re-skinning YAF".  Rhayger tries to dispel this belief, saying "there was an awful lot more work necessary" and spending one of his ten paragraphs listing some of the tasks that were done.  The phrase "lot of" is used four times in this one paragraph to describe the amount of work involved.  ;-)

However, the new devblog contains zero paragraphs describing how such an embarrassing mistake could be made in the first place.  It also contains zero paragraphs about the testing methodology that was used that could miss such a simple exploit.  The new devblog didn't even address the very real question EVE players had about why CCP claimed to have built the new forums from scratch when this wasn't the case.  It does say this:
I didn't address the significant flaws that made it into release and how that came to be, that will be for another dev blog to detail. We are doing a post mortem right now and doing some serious soul searching. We don't like making mistakes, let alone obvious ones we should have caught at various stages but much more importantly we do not want to repeat mistakes or gloss over flaws in process or skillsets that caused it.
Given that the new forums have been down for nearly a month now, and we still haven't been told how these "obvious mistakes" happened, it's fairly easy to argue that "glossing over flaws" is exactly what's happening.

So instead of talking about what isn't in this new devblog, let's talk about what is.

Let's start with feedback.  The new devblog uses this word six times.  Of the 26 paragraphs in these two devblogs, 11 have to do with giving feedback to CCP.  There's only one minor problem here: ignoring player feedback is practically a component part of CCP's DNA.  The "CCP commit to excellence" thread in the Assembly Hall, which was the first call from players for CCP to iterate and improve on existing EVE Online features after their release, has been read 100,000 times since its posting last May.  It has 2372 "supports"(1) and 2862 replies.  None of them are from CCP employees.

During the testing of the new forums, the number of people who said CCP was ignoring player feedback about problems with the new forums was legion, and went all the way up to Estel Arador, a reasonably famous EVE player.  CCP didn't respond in that thread, either, except to move it from "EVE General Discussion", where it would be seen, to (I still do not get this) "Out of Pod Experience", where it would not.  Not only were these concerns ignored, but CCP Sreegs went on a number of EVE forum threads and claimed that such feedback had never been received.  Estel has quit EVE over the experience.

The other problem with this new devblog that really, really bugs me is this:
Now some have worried that by choosing an Open Source solution we have to reveal the source code thus making your accounts and activities in EVE Gate vulnerable. This is not the case here as we have purchased a commercial license to YAF so we can properly protect our efforts. That said, we are big fans of Open Source initiatives and if we note issues in YAF that we come up with improvements for we will communicate that back to their project team to benefit the YAF community.
I'm not even a software developer, and I can tell you this is a ridiculous reversal of the tenets of Open Source development.  Releasing your source code does not make your code less secure.  It makes it more secure.  I'll grant you this is counter-intuitive, but it is true nonetheless.  In any case, EVE players and non-players alike are going to find holes in CCP's code whether the source code is released or not.  And bragging that they will share "issues in YAF that we come up with" with the community is rather hysterical in context.

Keith Neilson aka Mandrill, another well-known EVE player, wrote a rather long post to the EVE forums, Failheap, and evereport.com on this issue called "Loss of Faith".  The post is (sorry, dude, but it is) extremely melodramatic and wildly unrealistic.  The full piece is probably not worth your time.  I debated whether I should link it at all.  But it does call CCP out in one key area: leadership.  There's a serious break-down in leadership at CCP that is becoming increasingly obvious.  Information from the line programmers does not get through middle management to upper management.  Information from upper management -- including stuff directly from Hilmar -- does not get through middle management to the line developers.

James Harrison wrote an excellent piece examining some of the underpinnings of CCP's hiring practices that points to some reasons why.  Lack of leadership is practically being built into CCP's DNA, too.  That piece is worth your time.

Without strong leadership and good internal communication, it's not at all surprising that CCP customer feedback is being ignored, and CCP external communication -- while frequent lately -- is empty.

(1) By the way, this is -- far and away -- the highest number of "supports" ever garnered for an Assembly Hall proposal.  And it still is not enough to require this issue to be brought before the CSM according to the official CSM White Paper.  This is one of the many amusing holes in the CSM White Paper.  It was brought to the CSM (because the proposal was written by a CSM member), but it wasn't required to be.


  1. While I believe you are correct in the fact that this Dev Blog brings nothing to the table as to how such a horrible thing has happened, it continues to ignore the fact that we as players failed to do our part.

    Many (and I mean MANY) players knew of these exploits in the YAF software and did nothing about it until the forums were released to the public. They simply waited until they were released and then pounced on CCP like an Orca on auto-pilot in low-sec.

    CCP's development has always included the community in its testing and development. In this case, we didn't just stand by and wait for them to fall on their face. We waited until they were on the edge of a cliff and then PUSHED them off of it.

    Again, I agree, the Dev Blog is almost offensive in its lack of actual content but to continue flaming CCP for this is irritating at the least and shows blatant disrespect at best.

  2. Even if it's true that EVE players wanted CCP to fall on their collective faces here -- and that's highly debatable at best -- it's no excuse.

    It's the responsibility of any company that wants to do business on the Internet to shield themselves from this kind of embarrassment, even from (especially from!) their own users.

    Are you this forgiving of, say, Sony's foible over the weekend?

    No, until CCP actually answers some of the actual freakin' questions about this debacle, I'm going to continue to call them out on it, disrespectful or not.

  3. It does seem like its 90% technobable designed to sound like some form of justification of 'we didn't do this half cocked'. Its either that or its written by a techy who thinks that people need to know that stuff and doesn't know when to stop talking (we all know someone like that :P)

    As for the open source bit I can see their point; releasing code CAN make it more secure ONLY if people are willing to HELP. If people use the code to exploit the system then it'll only make things worse; and yes any who wanted to try to hack it probably will but their job is significantly harder without a document which tells them where all the backdoors are located.

    Feedback is something that is essential for developers but speaking as one sometimes its a hard pill to swallow. Its a creative job and you put a lot of effort into something for someone to basically shit all over it and feelings do get hurt. That being said its something that all developers need to learn that they need to grow out of this as problems are infinatly easier to fix BEFORE release than afterwards.

    Personally this sounds like the developers got this package thinking that all this stuff was built in already and all they had to do was fancy up the front end, connect to the DB at the backend and jobs a good 'un. Maybe this is a result of CCPs recent growth and new recruits being employed who aren't experianced enough to be fully aware of this type of thing.

    It also sounds as if the entire project wasn't thought through well enough in regards to security, almost to the point of it feeling like some sort of University project rather than a commerical website.

    The only thing we can hope for is that they'll actually learn from their mistakes.

  4. Jester, you don't need to state that you are not a developer as your comments make that very evident. Your primary complaint is that CCP doesn't lay bare its soul about how this problem occurred. Whatever industry you work in, I'm sure when a breach of security happens, it is NOT common to give details about the methods the attackers used, yet you expect CCP to divulge sensitive security information to placate people like you who couldn't do anything positive with the information anyway. Call it "glossing over flaws" or cover-up or whatever conspiracy theory that you want to make it sound like their non-disclosure is bad, but it's a standard practice for any responsible business. Companies will release ONLY what is absolutely necessary to inform its customers of what private information has been compromised.

    Second, use of open source code doesn't mean anything derived from it will be equally open. Again, if you were familiar with software development you would know that the majority of software companies--even the strongest open source proponents--build proprietary code around open source and contribute some knowledge back to the source. They may release that proprietary content at a much later time, but most aren't in the business of funding software development to give away. Don't expect CCP to be any different.

    "The post is (sorry, dude, but it is) extremely melodramatic and wildly unrealistic." This fits this blog entry. You Monday morning quarterback this issue with the most unrealistic expectations. And then for some supporting drama, you say Estel quit playing because the devs won't listen to his complaints about the forums. Really? Was he playing the game or the forums?

    You do good at calling CCP over the fact that a breach occurred. Too bad all the melodramatic drivel that followed detracts from that message.

  5. Open source licenses are intended to preserve to openness of the original. That applies to derived works. It is a violation of the fundamental principles of open source to wrap proprietary code around open source code and treat the resulting combination as a proprietary product. It is generally not permitted to modify open source and claim the modifications as proprietary. Some licenses permit development of proprietary add-on works that interact with, but are not modifications of, the open source software.

    YAF is released under GPLv2, which specifically prevents any derived work from being proprietary and all derived works must also be released under the terms of the GPLv2. Any derivations or modifications of the YAF open source code must also be open source and must be freely redistributable as open source software under the GPLv2.

    YAF is also available under a commercial license which permits private use of the product and local modifications that are _not_ for distribution outside of the licensed organization. The commercial license permits the use of the resulting forums on a private or public domain. Under this license, "open source" is no longer applicable, which is why further distribution of the resulting software is not permitted.


Note: Only a member of this blog may post a comment.