Welcome to Jester's Trek.
I'm your host, Jester. I've been an EVE Online player for about six years. One of my four mains is Ripard Teg, pictured at left. Sadly, I've succumbed to "bittervet" disease, but I'm wandering the New Eden landscape (and from time to time, the MMO landscape) in search of a cure.
You can follow along, if you want...

Saturday, April 9, 2011

All lines... are currently down

For those not keeping up over the weekend, the new EVE Online forums, recently announced with much fanfare, are now down... probably for the foreseeable future.

Here's what happened, as I understand events.  Friday morning, an EVE player named Catari Taga claimed on Scrapheap Challenge that he could impersonate any toon that he liked on the EVE Online official forums.  He also claimed to have found a way to add as many "likes" -- the reputation system in the new forums -- as he wanted to his own posts.  He implied that the methodology for this involved the signature code that the forums had used, and indicated that there were a number of exploits that this signature code could be used for.

Needless to say, virtually all of SHC rose up and said with one voice: "Proof, or STFU."  So Catari proved it... by posting on the new EVE Online forums under the name of one of the people -- a player named Durzel -- who had been demanding proof.  I have a screen-shot of the post, but I won't link it because the posting is rather rude and not particularly politically correct.  However, the post was quite convincing.  It appeared on the forum at 08:20 EVE time on April 8.

Catari claimed -- and I believe him -- that he attempted multiple means of making CCP aware of this issue before creating the fake post, including both filing a petition and a bug report, as well as other means.  Meanwhile, he further claimed that not only could he impersonate others on the forums, he could access parts of the forums that should have been inaccessible by him, including the private CSM and developer sections of the new forums.  Some SHC forum posters began agitating for him to do a full public "forum dump," something that's happened to a number of EVE alliances, including Pandemic Legion, but never to the official forums themselves.

A few hours later, two step, one of the alternate members of CSM6, apparently posted a notification of this to the CSM section of the forums.  In addition, Helicity Boson, a well-known EVE player, posted exact details on Catari's methods and claimed to be trying to reach someone in CCP through every means at his disposal.  Either because of this, because another member of the CSM had more direct access to CCP, or some other reason, CCP finally, finally took notice.  The new forums were taken off-line, with an info-graphic and a small bit of text explaining that the amount of time they'd be down was not known:
The EVE Online forums have been taken down for further maintenance. At this time we do not have an ETA on when the forums will be back online.

Additional information will be published both on our Facebook page and Twitter feed.
It was 21:00 EVE time.  In total, the CCP forums were wide open to Catari for more than 12 hours.

Sure enough, the EVE Online Twitter feed became quite active with status updates.  By about 00:00 EVE time on 9 April, it was posted that...
A forum update: We have updated our forum software and the fixes are currently being tested.
The forums were down for an additional three hours (between five and six hours total).  Meanwhile, Catari reported on SHC that he'd been banned from EVE Online -- lock, stock, and barrel.  His accounts were banned, even his IP address was banned.  Helicity worried publicly that he'd be next.

At 02:52 on 9 April, CCP Fallout posted the following to the new forums and to a devblog:
At approximately 21:00 UTC on Friday, April 8 we were made aware of some security issues with the new EVE forums which needed to be addressed.  The issues were as follows:
  • We discovered that it was possible to access some forums which certain users should not have been able to access
  • Users could make and edit posts as another user's character
  • It was possible to inject some HTML code into signatures
This resulted in us disabling the new forums temporarily while we investigated and addressed the situation.  We can assure players that none of their personal details, login credentials or billing information were compromised as that information is maintained on a separate encrypted server.

We have since identified and fixed these security issues, and have patched and tested the software.  We believe that we have resolved the issue successfully.  We will continue to investigate the causes of these problems, and hope to have more information available to you soon.

We would like to reiterate that your personal details and billing information have not been compromised, and your EVE Online account was not at risk.
Three minutes later, at 02:55 and despite being banned, Catari Taga posted a response to Fallout's post on the forums!

Needless to say, Fallout's post was quickly deleted, as well as Catari's response.  The forums were shut down in very short order afterward, and a follow-up message was posted indicating that it would be "later Saturday morning" before the forums were fully repaired.  Fallout's message to EVE players was also updated to include the following:
At 03:30 UTC on April 9, we took the forums down again for a second time to apply a hotfix that would restore several moderator features. Unfortunately, this hotfix did not resolve the issues with moderator functionality, and as a result, we are keeping the forums down until the full team is available Saturday morning to resolve the issue.

We know how important it is for you to communicate with other EVE Online players, and sincerely apologize for the upheavals of April 8 and 9. We hope to return the forums to you shortly, so that you may once again engage in your epic forum battles and troll posting. We extend an invitation to those of you who would like to talk to others of like minds by joining the #tweetfleet and following @eveonline on Twitter.

In the meantime...

Fly safely!
The new forums were back in play on Saturday, but never for more than a few hours at a time.  By 22:30 EVE time or so, the new forums were down for good.  Not long afterward, the link to the new forums was being redirected to the old forums and the old forums were brought back on-line.

At 23:21 on 9 April, CCP Navigator posted the following to the old forums:
Over the last few days we rolled out new forums which, following some security issues, have resulted in us taking them offline for further investigation.

In the meantime we have reopened the old forums for your posting pleasue (sic).

We will use this thread to keep everyone updated about the status of the new forums and to answer your questions.
I've looked at the code that Catari used here, and my first impulse is that we're not going to see the new forums back for a while.  The issue strikes me as architectural and occurred in the design phase of the system.  It's now clear that the new forums were never subjected to even the most basic forms of professional security screening pre-launch.

What's worse: I have it on very good authority that CCP wanted to launch these new forums right in the middle of the CSM6 elections, and were only convinced otherwise through some active lobbying by CSM5.  I can only imagine the chaos had CCP proceeded with their original plan.

So, expect this one to be very big news in the gaming websites by Monday morning, and a point of discussion for a few weeks.  Will Catari be unbanned?  Not a freakin' chance.  One of CCP's well-known and less endearing qualities is their enthusiasm for perma-bans on anyone who embarrasses them in public.  And Catari not only did that, but he became actively adversarial toward CCP with his second forum post.  He's out for good, and will no doubt be very motivated to sternly "test" the new forums if they do ever come back up.

Will we see a forum dump?  Very possible!  As I said, the forums were wide open for about 12 hours, so there's no telling if such a dump was taken.  I'd be surprised if it were not.

Stay tuned...

5 comments:

  1. Glad Catari did it.
    Attention whoring is far better than some other nefarious uses it could be put to use.
    Having a CEO of a big corp post something, or look at the dev forums and corner the market on something...

    ReplyDelete
  2. The design issues with the new forums were pointed out several times during the beta testing, but CCP ignored those messages. The forums disobeyed the basic rule of security on the Internet, which is "do not trust the client".

    ReplyDelete
  3. I'm a dev. A substantial part of my job is to do security analysis and training for my product team. It is shocking that CCP released their forums with bugs like this. These are obvious. These are terrible. It's a good case study for my "intro to security" presentations, though.

    CCP needs to recognize and admit they screwed up, and hire/train someone in standard security practices. I kind of wish I wanted to move to Iceland.

    ReplyDelete
  4. Jonathan McarthurApril 9, 2011 at 11:01 PM

    I have mixed feelings about this, one hand CCP got their asses handed to them in the forums this friday by Scrapheap Challenge community however this same community is now down for the immediate future due to "funding" issues. This smells like a CCP counter attack (black mail) against Calmdown thus causing him too quit hosting the servers for either legal or career related reasons.

    ReplyDelete
  5. I, too, believe in conspiracy theories.

    Fact is, shutting down SHC just moves people to Kugu, or to new sites like FHC (i.e. does nothing).

    ReplyDelete

Note: Only a member of this blog may post a comment.