Welcome to Jester's Trek.
I'm your host, Jester. I've been an EVE Online player for about six years. One of my four mains is Ripard Teg, pictured at left. Sadly, I've succumbed to "bittervet" disease, but I'm wandering the New Eden landscape (and from time to time, the MMO landscape) in search of a cure.
You can follow along, if you want...

Saturday, January 18, 2014

Soul-crushing lag

And the final bookmark for today, 18 January 2014, is a short one. Specifically, it's a reference to the bad old days of the introduction of Time Dilation two years ago this month. Just a couple of months ago, I wrote a longer piece about TiDi in which I said...
TiDi is a brilliant band-aid and for now, it's still holding the wound closed. But yeah, we can now see it's not going to hold forever.
As of January 2014, it's no longer holding the wound closed. The wound is reopening and it's starting to bleed. Yeah, I want to talk about HED-GP a little.

Thousands of people piled into HED-GP for the timer today and every report I've heard from the fight indicated that the bad old days of 2012 have returned: ships trapped in warp tunnels, ships subjected to random effects, ships unable to cycle mods or launch drones, ships apparently operating in space while the pilots of these ships received notification that their ship was dead, fleets ordered not to jump in to prevent them from suffering these ill effects.

And of course, ships that already held the field having a significant advantage over ships trying to enter the field... one of the major things that TiDi was introduced to the game to prevent.

The fight itself was a demonstration of where sov fights are in EVE these days: dozens of titans, hundreds of super-carriers, hundreds and hundreds of Archons and Chimeras, and a number probably not all that short of a thousand dreadnoughts either in the system or trying to jump into the system.

Welcome to sov fights, circa the first month of 2014.

So yeah, as I've mentioned, the CSM requested a session on "The Future of Big Fights in EVE." CCP bought themselves two years with the last band-aid, and I know there have been plans to address this problem further. But I don't see the fights in EVE getting any smaller, do you? I mean after all, even a relatively small event in EVE now features some 1200 pilots (and TiDi, and lag!) in a single system... a fight size that would have been a major event in 2012. Now this size fight is relatively routine. And yet it still causes problems.

What will a "big fight" in EVE look like two years from now?

EDIT (19/Jan/2014): CSM8 Chairman Trebor Daehdoow wrote a long piece on this topic. Go read it! It's worth your time. I'm trying to talk him into a second piece with some of his ideas about how to address this long-standing problem.

45 comments:

  1. CCP poured all the EVE cash into other pursuits and guess what? Dust was no miracle cash cow to have gazillions to spend on EVE - we just have a series of sub-par "expansions". It's the new world of the perpetual :18months:

    CCP are trapped. When a company is laying people off I don't get a sense they could spend the blood and treasure required to re-architect the game even if they wanted to.

    ReplyDelete
  2. Q: "What will a "big fight" in EVE look like two years from now?"

    A: The same as today once EVE departs the PCU plateau in the easiest direction... ;)

    ReplyDelete
  3. ...dozens of titans, hundreds of super-carriers...

    and yet none of them died.

    ReplyDelete
  4. One of the best suggestions I've heard is basically centered around preventing large fights from happening, rather than adjusting server architecture in order to make them play okay.

    It's in regards to making sov fights more like FW. Instead of a timer revolving around a single point in single system, make the timer spawn a number of FW like complexes around the constellation. You could even have size restrictions so some parts of the sov fight would be frigate fights.

    Obviously the whole sov process and structure would have to change, but I feel like it is the correct direction.

    Maybe a not-too-far-away expansion should center completely around a sov re-write to make it more like FW?

    Those are my thoughts, anyway.

    ReplyDelete
  5. Just as "the work expands to fill the time" so does "the number of ships increases to the limits of the server(s) can bear". CCP's fight against the Lag Monster was heavily relying on someone named CCP Warlock, i.e. Jacky Mallet. She worked at CCP from 2009 through 2011 and used her considerable expertise (PhD, MIT, ) to tackle the problem of distributed systems engineering and technology. She was forced out by office politics if the various blogs and boards of two years ago are to be believed. The whole brain-in-a-box initiative was probably her baby and that's why we haven't heard anything more about it.

    Magical Unicorns from Uranus could land tomorrow and give CCP terabit fiber connections to every subscriber's home, 7.5 GHz octo-CPU processors, a new Python that could take advantage of all those cores, 1024 petabyes of memory on each.... and before long the number of people involved in fights would exceed the ability of these unicorn servers to keep up. That's just the way it is. The amount of data crunching is that high when you figure in 4000 ships, 10,000 drones and FB (conservatively) and everything else that goes along with a huge fleet vs fleet fight.

    It's very similar to expecting average (or even above-average) PCs to be able to handle realtime rendering of tens of lifelike fullbody avatars with each one controlled by human players who are liable to do anything and everything from playing poker to bushwacking their enemies to having a drink to ogling the drag queens in Jita 4-4. It's just not going to happen any time soon.

    The post-Incarna "walking in stations" mess is not fixable any time soon. The Lag Monster is, but it will require that someone take the courageous decision (hah, imagine CCP doing that!) that the number of pilots in space in any given system has to be limited. Tough decisions need to be made.

    CCP doesn't have what it takes to make those hard decisions. IMO the recent departures of long-time CCPers validates this hypothesis. Sure, people change jobs and pursue other opportunities. But not if your present company is on the brink of breakthrough developments that will change the face of MMO gaming. Those days are long gone for our friends in Iceland.

    ReplyDelete
    Replies
    1. See also this interesting PPT from Jacky Mallet , "Scaling the Single Universe" http://www.gdcvault.com/play/1014710/Fighting-Real

      Delete
  6. I was in that fight for nine hours and it’s by far the worst experience I’ve had since playing in TIDI.

    The gameplay mechanics have exceeded the physical limits of the game. We’ve now got to the situation where the on paper counter to caps, that being more caps – isn’t actually possible on the servers.

    If players can’t get their assets on the field to apply their tactics and are at the mercy of the servers; that’s not a game of skill, it’s a game of chance.

    One of the big ticket marketing tools of this game, the big fights, is pretty much off the table – because if CCP dared to brag about this fight a large proportion of the Eve player base wouldn’t be too kind to them.

    To quote eve online ‘physics says no’

    ReplyDelete
  7. You know, I believe this sandbox needs some new rules. There should not be limitations on how many players can be in a system at one time, but there should be rules that limit the effectiveness of how the players can be controlled by a single "master". There should not be rules that limit a group of players ability to controll space mora than it is today, but there should be rules that limit what goups of groups of players (alliances) can control.
    I guess my point is this; over the last couple of years CCP has developed and delivered lots of improvements to the large entities "quality of life" and effectiveness. I think that is a large part of the current problem. It is "too easy" to controll the alliances and coalitions of todays EvE. It used to be, and should again become, much harder and thus limiting their effectiveness. Alliance bookmaks to make it easier for alliances to coordinate their players to get all the hordes in line with (almost) a single click? Certainly not. In fact, perhaps the game needs to have some of its "quality of life" improvements revoked or nerfed so that they only benefit small entities, not the empires. Looking at what CCP worked out for Incursions, they created a system where you would need to comply to a optimum group size to get optimum results. Such a guideline should go into future work where the superlarge entities should have deiminishing returns as a result of their size. It a cold, hard universe and lets get that back by making it harder to be the master puppeters that we see today and more attractive to be smaller, more agile and controll a smaller part of space.

    ReplyDelete
  8. About time to implement stacking penalty to remote reps i say....

    ReplyDelete
  9. If CCP is going to enhance server stability the answer from the power blocks will be: "pile more people in".
    Force projection (the ability to transfer large number of ships/pilots to any spot in New Eden bar whormholes) is the fundamental problem.

    Make New Eden larger, not in terms of number of systems but in distance and accessibility.

    ReplyDelete
  10. Fights will scale to the capacity of the servers. TiDi artificially buys additional capacity by slowing things down 10 fold. It has taken 2 years for the nullsec alliances to catch up with this new size but catch it they have - and they have been on a steady march towards it with each passing month.

    CCP have a couple of choices here - either they roll out new performance enhancements which buy them a similar leap in server capacity to what TiDi offered and they continue to do that as fleet sizes inevitably increase and increase. There probably are a few more large performance leaps that can be made but ultimately there is a limit to what can be achieved with optimization.

    The second option is to address fleet sizes themselves and drive the meta towards smaller fleets. I'm not entirely sure how they could achieve this as any mechanic which works well for a small number of ships is going to work even better for a large number of ships. Any grid caps will favour whoever can get the most number of ships to the cap first. Any fleet limitation will be ineffective as we already have one and it has been overcome.

    hard one to answer - but what will a big fight look like in eve in 2 years? If CCP manage to conjure another quantum leap in capacity through some form of trickery I wouldn't be suprised if we start seeing 4-5000 ships aside..we know the null coalitions certainly have the manpower now to make this kind of thing a reality..

    ReplyDelete
  11. I'm surprised that you didn't make what seems to me an obvious connection to another of your recent articles: if the game had "terrain" in any meaningful sense, if there were some incentive to split forces into multiple fleets with different objectives, or if there were some in-game penalty to clustering too many ships too close together, then it would be much easier to manage the server load. But as it is with everyone piling onto one grid for a single brawl, it will always be an arms race between the growth of computing power vs. player organizations. It is clear now that computing power is never going to win that race.

    ReplyDelete
    Replies
    1. Completely agree with this. The lack of "Terrain" contributes to creating an oversimplified battlefield. The aforementioned post seemed like it was hinting at ways to potentially improve wars and large scale conflicts in New Eden. Adding terrain would improve combat in EVE across all forms and scale. Hoping that it's included in EVE 2.0. Really looking forward to that game. :(

      Delete
    2. This was actually done in that fight. Several Subcap fleets had different purposes and tasks prior to the Capitals "coming" in to try and take down the opposing Caps and Supers.

      The game has terrain, things need to be accomplished prior to fielding caps and supers. Unless you are fielding Sentry Carriers. Then you don't need subcap to extract.

      Drone Assist isn't broken, RR Carriers using Sentries are.

      Delete
  12. Ccp could help themselves a lot by creating weapon systems that make it dangerous to cram lots of people into a small space. This would work in much the same way that machine guns ended the tradition of standing soldiers shoulder to shoulder and three ranks deep. It could be as somple as making ships damage other ships around them when destroyed (warp core breach.) Alternatively they could add missles or shells with warheads that explode like bombs.) Yes you could try to make tge server withstand the activity but it might be easier to just encourage different behavior.

    The desire to be there so that you can be on the mail could be corrected by includind the supportung cast meaning everyone in either participating fleet on all mails regardless of what system tgey are in and thus including scouts, logi, planners, the tackle that is on the other side of the gate who heretofor have contributed to the win but have never been recognized. Human behavior not hardware is the easier solution.

    ReplyDelete
  13. I was there, sort of. I jumped in with a small reinforcement group of 25 or so. Upon hitting jump, I saw the "Jumping to Beacon" for two hours. Them the TIDI circle appeared. About a half hour later I saw the 15 minute aggression timer appear, all still looking at the titan I was jumping from. An hour later, my wallet and mail icons flashed, notifying me that I had received my insurance payment and a few bounty payments. I received that "Soul-Crushing Lag" pop-up about every half hour during the process. I can imagine this happened hundreds of times yesterday. If the CSM does not make this issue the #1 topic of the winter meeting, it won't matter what CCP does to improve the new player experience as their base of multi-account null sec players will quietly un-sub until a time the game becomes playable for them.

    ReplyDelete
    Replies
    1. Which solves the problem nicely ! =Þ

      Delete
    2. Totally agree the CSM NEEDS TO PUSH THIS to top or near top of their list to discuss with CCP. I spent 11 hours in that fight that I really wish I could get back. Instead of "I was there" it's I wish I wasn't there.

      https://www.youtube.com/watch?v=HJ_jOS-cvq0 Need an update on what Veritas talked about here.

      Delete
  14. Negative incentives against piling in big battles... everyone can come up with some.

    The simplest solution is to create an artificial sweet spot. Fleets too small get massive resistance nerf (so you better bring enough ships). Fleets too large get massive resistance nerf (so you better split them). Too many fleets in a grid get massive resistance nerf (so you better don't bring too many fleets). Too few fleets in a grid get massive resistance nerf (so you better split them).

    ReplyDelete
  15. CCP needs to drop python, and they need to do it as soon as possible.
    Sure it's good enough to be running ui code on the clients but it's performance is way to low to be running combat calculations on the servers. CCP needs to understand that interpreted scripting languages cannot compete with native code.

    ReplyDelete
  16. The main fight was fine. Tidi isnt fun but better than perma black screens. If the cfc had dropped their caps in at a safe then warp them in it might have gone better for them but dropping into the middle of 3k people already loaded brought most of that mess on themselves. I was in the main fight, yeah not the greatest. Fortunately got killed and reshipped into another fleet that probed/killed a lot of dreads afterwards. Overall I had a blast yesterday.

    ReplyDelete
  17. Heh, that reminds me of the old days. I read some guides back in 2009: What to do in lag. A lot of instructions how you can circumvent a few of the problems. Yes, there were field manuals to fight lag. The difference between lag today and back then is just that people aren't used to it anymore. Oh, and there is about a magnitude of ships more involved ;-)

    Mr. Magical Unicorn Server guy up there really hits the nail. Remove the incentive to pile up people or live with the lag.

    ReplyDelete
  18. I am suddenly reminded of the quote "640K ought to be enough for anybody". Maybe its an opportunity to change the game from yet another dogpile into an alliance tournament style contest. So I will state it now: 250 ought to be enough. Intelligent play is always a lot more interesting from any point of view. It wormholes can control mass limit, then the mechanics exist within the game influence a fleet size.

    There so much in this game were players initiate actions and find any excuse to blame the result on some else. Knowing that the system has a limit.

    This recent load of tripe. There is a known large force of ships in HED, So why not attack undefended valuable assets, system sovereignty or just about anywhere else. Send in a token force to tie then down,

    If I really wanted to tinfoil this, then this was a pseudo contest. The known result of just flooding in the same system is either a direct crash or beyond playable lag. Its then a fixed contest to minimalise any loss and uphold the new b0tlord rubbish

    ReplyDelete
    Replies
    1. Since I was there and saying it - it was 48K, not 640K. And even the context is educational (though a bit of a digression).

      48K was plenty for the businesses running computers at that time. The thing is that 'toy' the Commodore 64 came out and it had 25% more; 64K. See, the rest of the quote, paraphrased, is that 48K is enough for businesses, 'It's gamers who need more.'

      My bet is that CCP finds work-arounds and solutions. My further bet is that the cycle will continue. My recommendation for CCP, then, is to consider making 'beating lag' a permanent team assignment while recognizing it's a moving target.

      Delete
  19. make trebor run another csm cycle to accomplish his first smart goals he set up like 4 years ago. it wassomething like instead of increasingthe performance, change game design in a way that stacking up as many people as possible in one single system is no longer the best way to win the game.

    ReplyDelete
  20. Jester, what you are asking for is impossible without throwing out the entire game and starting from scratch. Time and again you make these cute posts as if the situation can be remedied in some other way.

    It's time to start facing the facts, in no uncertain terms can the current core code of Eve support anything further than short term band-aids.

    ReplyDelete
    Replies
    1. No it can't and Veritas talks about it in detail here https://www.youtube.com/watch?v=HJ_jOS-cvq0 He's also got some ideas that haven't been heard from since like "Brain in a Box" and Dispatcher etc.

      Would be nice if someone interviewed him both about the HED-GP fight and to get an update on what he's working on.

      Delete
  21. Who else is left to join the only two coalitions left in null sec? Proviblock, brave newbies, RvB... With Eve's population stagnating the coalitions are rapidly running out of people to ally with and are having to resort to multi-boxing to increase fleet sizes. Doesn't the ability to find more pilots to bring start breaking down at some point and aren't we pretty close to that point now?

    If actual engaging game play existed (is multi-boxing DOTA a big problem?) then mullti-boxing would become much more difficult and potentially suicidal - reducing the number of pilots in system. Easier said than done without crushing the servers handling the interesting game-play of course.

    Drones per ship could easily be cut by 25% (4 per dominix/8 per carrier and boosted hp/damage wise to compensate) without really hurting anything. That's pretty low-hanging fruit imho.

    I'm not trying to say that fixing lag is an easy problem but it doesn't seem like it's been much of a priority at CCP recently. Another casualty of diverting so many resources into Dust/WoD and now Valkyrie?

    ReplyDelete
  22. I wouldn't go so far as to make this issue the #1 topic of the winter meeting. It's important, yes, but only 20% of players take part in these huge nullsec fights. Maybe the issue gets a little more attention than it deserves because the huge fleet fight has always been CCP's bling feature.

    If they're looking for something to try there are some good ideas expressed above. I'd add in a DPS limiter. Link it to whatever metrics they use for TiDi. When the first stages of TiDi are kicking in, shave 10% off all damage. As the system load goes up, DPS goes down to the point where the equivalent of 10% TiDi is a 90% reduction in DPS. That will provide a huge incentive to not overload a system for a fight.

    ReplyDelete
    Replies
    1. Are you including the miners, manufacturers, haulers and traders in your 20% guestimate?

      Delete
    2. If you put in some sort of DPS limiter, you should also at least at a limiter to reppers (local and remote) and also a limiter to shield regen. You would also probably want to add a limiter to cap transfer and neuts.

      I guess you would also need to add some sort of acceleration and warp speed limiter else you'll just have everyone using the "low dps safe period" to safely move their fleets around and get into position.

      Once you limit all the things that need to be limited to create a fair experience under severe load, you just end up with TiDi again.

      Delete
    3. Not on everything, just DPS. If it got put on everything it would defeat the point. Losing more and more DPS makes the fight more and more pointless... so the meta would become trying not to induce tidi at all. I like this idea, makes blobs try to find a sweet spot of how many people to bring and how many they think the enemy will bring.

      Only downside I can see, defensive fleets can put a node into tidi before the enemy even shows up, in an attempt to keep their structures from being killed by reducing incoming DPS.

      Delete
  23. "There have been plans..."

    What were those? That 40-minute video link is not very useful. I'm interested, but not THAT interested.

    ReplyDelete
    Replies
    1. At about 10 minutes, there's a quote from Mister Vee of Goonswarm, claiming that he used to have to use tactics to avoid black screening fleets that cyno'd in, and now, after the introduction of Time Dilation, he doesn't. (There are similar quotes from Elise Randolph and Shadoo.)

      I bet that they'd like to walk those statements back now. All Time Dilation did was to up the limit before it became a problem again.

      Delete
  24. Not sure how this affect your reasoning in this post, but after reading the BRs the only sane explanation for the CFC/RUS FCs' tactical decisions is that they were doing everything in their power to actually crash the node, trapping PL/NC. cap and supercap fleet and allowing a clean jump for their cap/supercap fleet on top of the (then logged of) opposing fleet.

    Brilliant move (and not unheard of), except that this time the server held up (badly, but did) in spite of record-breaking number of players. If they were to accomplish that intent, you'd be hearing a lot less people QQ'ing the whole event for some reason, even though the technical problems would still be there.

    ReplyDelete
  25. I was in the RvB Ganked event you mentioned. It was loads of fun, but also very, ah, challenging.

    There was stuff that had more to do with the difficulties of having two full fleets made of Red, Blues, and numerous various allies and participants (all were welcome) with various ship types, along with pilots who are more used to small gang warfare than this. Those challenges can be overcome. But what was very frustrating was the lag (Tidi wents from 10-30%) and the overview not being able to handle so many ships. Frustrating when you are told to achor towards someone you can't even see on your overview... I also had a hard time activating my modules at times.

    I had a blast, and I'm glad I did it, but if this was my regular game play, I wouldn't play EVE. I'm sure the novelty and excitement of being in such a huge fleet with huge ship would wear off, leaving only frustration at the slow pace, lack of responsiveness and overview problems. I don't know how the null warriors do it.

    LR

    ReplyDelete
  26. Re-introduce aoe doomsdays, for capitals only. If they field a large enough fleet and maintain a cluster of carriers/supercaps within remote rep range, then they can be hit by the area of the the doomsday. If they split their fleet so it's not just one ball, the damage will be limited to one cluster of carriers.

    ReplyDelete
    Replies
    1. If the problem is the number of ships in system, as opposed to number of ships on grid, will AOE weapons help? Note that bombers already provide AOE. I agree about the need to split people up more, though.

      Delete
  27. Part of the problem is that players are too rich. One way to prevent such big fights would be logistics. If bridges were mass-limited, the way wormholes are, the cost of moving a zillion ships around would increase.

    We might get more, smaller fights. Smaller fights also tend to be deadlier (more ships are lost as a percentage of the total fleets on field), which will go some way toward reducing the blob.

    ReplyDelete
  28. Tons of things that could be done, but what can be implemented within a development cycle, and not bankrupt CCP?

    terrain was basically the keyword, but the devil is in the details. What would provide benefit for splitting up fleets instead of blobbing?

    lower HP on large nul structures? if you only needed one sieged dread to take out an IHUB, makes it better to put one on each system instead of dogpiling. even making siege mode ignore resists on the things might work, so it doesn't just make it BS time again

    larger bubbles around ships? since you can fit thousands in the same area, why not just increase the bubbles around ships? you try bringing 1k ships intoa space, they just bump each other off grid. physically being unable to fit that many on a single grid will space them out, though unsure if seperate grids solve the problem

    attack windows. normally i hate adding things to solve problems, consider a constellation having a structure that, when shot, makes other structures in the constellation vulnerable, but only until downtime etc. makes it pretty incentivising to spread your fleet out to take advantage of.

    then it could just be as simple as allowing players their own nul gates, but i havent fleshed that one out nearly enough to understand the implications.

    but yeah, they can handle a thousand in system, assume that is the limit, and change the game until more than that doesn't help your cause. you can even start to just blow ships up randomly, say it's the radiation from all the ships causing damage or something.

    would have gone a lot differently if a quick wave of damage hit (enough to take out a cruiser) once the system hit 10%, if brain in a box was in place, would have cleared a lot of shaff out of system. would have probably caused smaller subcaps to have a fight in a secondary target, if they knew anything smaller than a bs couldn't survive long enough to shoot anyways

    ReplyDelete
  29. i so wished CCP implemented a gravity collapse outcome...

    Bring too many ships (specially supercaps) and eventually you end up with a gravity collapsing singularity swallowing your ships.

    ReplyDelete
  30. I brought this up in one of your earlier posts - I welcome any arguments of why it wouldn't work:

    1) Introduce Line of sight requirements.
    That's it - require that two ships have a clear firing path between each other. If they don't, the weapon just blinks until a firing solution is available. Bigger blobs would no longer be better and player behavior would change.
    Of course, the monster battles are a selling point for CCP so maybe they don't really want to fix it.

    -Mark

    ReplyDelete
  31. Eve needs size-based corruption mechanics, like Communism in Civ2 had. It's really depressing to read blog after blog of how the rich get richer, the poor stay poor, and some players use "sandbox" to excuse poor game design. It makes me wonder if I'll every come back to Eve, which is disappointing.
    And the repeated, absolute stupidity/near-sightedness of CCP employees is frustrating to no end.
    -Bantara

    ReplyDelete
  32. Favorite ideas so far: DPS goes down as TiDi goes up, and FW-like, multiple concurrent objectives for Sov warfare.

    Overall, I'm more in favor of honey than vinegar (gotta attract the bees).

    ReplyDelete
  33. My solution: when TIDI gets bad, have the server automatically create damaging explosions that will serve to remove ships from space. (This is like Alain's singularity, but not as harsh.) Thus all large concentrations of pilots become self-correcting; either they disperse themselves willingly or it will be done to them unwillingly.

    ReplyDelete