Comment by jeroenhd
24 days ago
By their own industry data (https://store.steampowered.com/news/app/553850/view/49158394...), deduplication causes a 5x performance increase loading data from HDD. There's a reason so many games are huge, and it's not because they're mining your HDD for HDDCoin.
The "problem" is a feature. The "so it remains unfixed until someone with pride in their work finally carves out the time to do it" mindset suggests that they were simply too lazy to ever run fdupes over their install directory, which is simply not the case. The duplication was intentional, and is still intentional in many other games that could but likely won't apply the same data minimization.
I'll gladly take this update because considerable effort was spent on measuring the impact, but not one of those "everyone around me is so lazy, I'll just be the noble hero to sacrifice my time to deduplicate the game files" updates.
> In the worst cases, a 5x difference was reported between instances that used duplication and those that did not. We were being very conservative and doubled that projection again to account for unknown unknowns.
That makes no goddamn sense. I’ve read it three times and to paraphrase Babbage, I cannot apprehend the confusion of thought that would lead to such a conclusion.
5x gets resources to investigate, not assumed to be correct and then doubled. Orders of magnitude change implementations, as we see here. And it sounds like they just manufactured one out of thin air.
Perhaps this is a place where developers can offer two builds.
HDD and SSD, where SSD is deduplicated.
Im.sure some gamers will develop funny opinions, but for the last 8 years I have not had a HDD in sight inside my gaming or work machines. I'd very much rather save space if the load time is about the same.on an SSD. A 150gb install profile is absolute insanity.
I mean when you optimize assets for a single read on mechanical drives size blow up pretty quickly, but the single IO read reduces latency greatly. That said it only makes sense on drives with high IO latency.
It's not a "feature" for the 89% of users whose SSD capacity was being wasted.
Seems to me that most of these situations have an 80/20 rule and it would be worth someone’s time to figure out what that is.
Get rid of 80% of that duplication for a 2x instead of a 5x slowdown would be something.
I expect better from HN, where most of us are engineers or engineer-adjacent. It's fair to question Arrowhead's priorities but...
Really? I think the PC install size probably should have been addressed sooner too, but... which do you think is more likely?
Arrowhead is a whole company full of "lazy" developers who just don't like to work very hard?
Or do you think they had their hands full with other optimizations, bug fixes, and a large amount of new content while running a complex multiplatform live service game for millions of players? (Also consider that management was probably deciding priorities there and not the developers)
I put hundreds of hours into HD2 and had a tremendous amount of fun. It's not the product of "lazy" people...
An 85% disk size reduction at minimal performance impact is negligent by the standard of professional excellence.
But that's also par for the course with AA+ games these days, where shoving content into an engine is paramount and everything else is 'as long as it works.' Thanks, Bethesda.
Evidenced by the litany of quality of life bug fixes and performance improvements modders hack into EOL games.
The only sane way to judge "professional excellence" would be holistically, not by asking "has this person or team ever shipped a bug?" If you disagree, then I hope you are judged the same way during your next review.
In the case of HD2 I'd say the team has done well enough. The game has maintained player base after nearly two years, including on PC. This is rare in the world of live service games, and we should ask ourselves what this tells us about the overall technical quality of the game - is the game so amazing that people keep playing despite abysmal technical quality?
The technical quality of the game itself has been somewhat inconsistent, but I put hundreds of hours into it (over 1K, I think) and most of the time it was trouble-free (and fun).
I would also note that the PC install size issue has only become egregious somewhat recently. The issue was always there, but initially the PC install size was small enough that it wasn't a major issue for most players. I actually never noticed the install size bug because I have a $75 1TB drive for games and even at its worst, HD2 consumed only a bit over 10% of that.
It certainly must have been challenging for the developers. There has been a constant stream of new content, and an entirely new platform (Xbox) added since release. Perhaps more frustratingly for the development team, there has also been a neverending parade of rebalancing work which has consumed a lot of cycles. Some of this rebalancing work was unavoidable (in a complex game, millions of players will find bugs and meta strategies that could never be uncovered by testing alone) and some was the result of perhaps-avoidable internal discord regarding the game's creative direction.
The game is also somewhat difficult to balance and test by design. There are 10 difficulty levels and 3 enemy factions. It's almost like 30 separate games. This is an excellent feature of the game, but it would be fair to Arrowhead for perhaps biting off more than any team can chew.