PSA: Memory Leak found in TF2 and most Source Engine games

[VforVel0city]

[QUOTE=chemo;50804921]Wow, Source 1 has become a goddamn mess.[/QUOTE]

It has always been a mess. A bunch of thrown together code, both old stuff from the GoldSrc (and even Quake) days and new stuff. You can't expect a game engine to last you for 12 years. Maybe nowadays you can, but when Source was created the computer industry looked vastly different. Nowadays we have powerful GPUs and multi-core processors that change their frequency depending on load. Back then you had single-core CPUs clocked insanely high and the GPUs of the time were hardly anything to write home about.

Source 2 is about 5 years late. It should've been released with Portal 2, with every game out before then being ported to it. But no, Valve didn't do that, and now we're stuck with a broken engine that's also horribly optimized, if at all, and this is the result.

[Galactic S.O.U.L]

I'm not sure if it's related but I've noticed that TF2 uses quite a lot of VRAM, on High textures it uses from 600-800MB and on Very High it goes to 700-1100MB, compared to L4D2 which uses a mere 300-400MB

[sigsegv]

[QUOTE=Fox Powers;50806560]I'm not sure if it's related but I've noticed that TF2 uses quite a lot of VRAM, on High textures it uses from 600-800MB and on Very High it goes to 700-1100MB, compared to L4D2 which uses a mere 300-400MB[/QUOTE]

All those hats have to get loaded somewhere, my friend.

[artacus]

[QUOTE=Fox Powers;50806560]I'm not sure if it's related but I've noticed that TF2 uses quite a lot of VRAM, on High textures it uses from 600-800MB and on Very High it goes to 700-1100MB, compared to L4D2 which uses a mere 300-400MB[/QUOTE]

It seems all the extra hat/misc textures TF2 has to render now seems to have a dramatic affect on it's performance and vram usage. I used to struggle getting above 40 fps in tf2 in it's current state but after I downloaded a "no hat" mod the game runs smoothly & uses less memory.

[sigsegv]

Yeah. The hat thing is completely serious, as much as people like to joke about it. The additional hundreds/thousands of models and textures that the game has to load (mostly on-demand but some preloaded) just puts a big strain on everything.

Plus many of those models aren't ideally well-optimized (workshop contributor: "what are these 'LODs' I keep hearing about?").

[VforVel0city]

No wonder it TF2 runs out of RAM and then crashes at like 2.3GB, that's just system RAM. Add the 1+GB of VRAM TF2 needs to address and you're at that magical 3GB limit of 32-bit programs.

[superbro11]

[QUOTE=VforVel0city;50807215]No wonder it TF2 runs out of RAM and then crashes at like 2.3GB, that's just system RAM. Add the 1+GB of VRAM TF2 needs to address and you're at that magical 3GB limit of 32-bit programs.[/QUOTE]

I have 8 gb ram and a few times tf2 ran out of ram

[Galactic S.O.U.L]

[QUOTE=superbro11;50808875]I have 8 gb ram and a few times tf2 ran out of ram[/QUOTE]

how the hell does that happen

i have 8gb too and TF2 has not crashed a single time, were you running stuff on the background?

[superbro11]

[QUOTE=Fox Powers;50808953]how the hell does that happen

i have 8gb too and TF2 has not crashed a single time, were you running stuff on the background?[/QUOTE]

Nope, just tf2 in the background. It kept on saying out of memory randomly and crashing
I managed to fix it by reinstalling tf2 and putting it on "fatory settings"

[SageJFox]

I tried it in Black Mesa retail version and it came up with "Failed to load sound "x.mp3", file probably missing from disk/repository" with no gradual increase to memory usage. Though based on the GMod discrepancy, would like a secondary confirmation before we say it's okay.

[VforVel0city]

I played Black Mesa yesterday. Basically, the console was being spammed by this error:

BlockingGetDataPointer: Async I/O Force weapons\machete_swing.wav ( 0.64 msec / 0.64 msec total )
346.874711
BlockingGetDataPointer: Async I/O Force weapons\machete_swing.wav ( 0.49 msec / 0.49 msec total )
346.874711
BlockingGetDataPointer: Async I/O Force weapons\machete_swing.wav ( 0.66 msec / 0.64 msec total )

Now these are actually from the TF2 console, but they're exactly the same in BM. However, sometimes the sounds don't take 0.xx milliseconds, they take anything up to 40 milliseconds. And not jus one, no, every sound takes 40 milliseconds, causing fps to basically drop below 1.

Don't know if it's related to this memory leak, but it's something pretty recent. Haven't seen this in TF2 so far though.

Here's a console dump from the actual game, on the map bm_c3a2h.

187.760879 BlockingGetDataPointer:  Async I/O Force ambient\atmosphere\city_rumble_loop1.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\strider\charging.wav (    1.78 msec /     1.79 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force bms_scripted\lc\lc-mb_fire.wav (    0.14 msec /     0.14 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force bms_scripted\lc\lc-mainbeam.wav (    0.27 msec /     0.27 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force ambient\levels\citadel\weapon_disintegrate3.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force ambient\machines\teleport3.wav (    0.14 msec /     0.14 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force bms_scripted\lc\lc_airlock_open.wav (    1.93 msec /     1.94 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force bms_ambience\machine_ambience\m_ambience_lc22.wav (    2.22 msec /     2.23 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force bms_scripted\lc\lc-roboarm.wav (    4.45 msec /     4.46 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\energyorb_exp.wav (    0.13 msec /     0.14 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\energyorb_loop.wav (    0.37 msec /     0.38 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_idle1.wav (    2.23 msec /     2.24 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_idle2.wav (    1.80 msec /     1.80 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_alert1.wav (    4.77 msec /     4.78 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_alert2.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_pain1.wav (    0.16 msec /     0.17 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_pain2.wav (    0.03 msec /     0.03 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_pain3.wav (    0.04 msec /     0.03 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_die1.wav (    0.10 msec /     0.11 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_lift1.wav (    0.19 msec /     0.18 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_lift2.wav (    0.16 msec /     0.17 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_throw1.wav (    0.03 msec /     0.03 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\alien_controller\con_hover.wav (    0.30 msec /     0.31 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force ambient\explosions\citadel_end_explosion2.wav (   19.63 msec /    19.64 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci10_finalaccoutrement02.wav (    0.07 msec /     0.09 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci10_finalaccoutrement01.wav (    0.06 msec /     0.08 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci10_thisistheljm02.wav (    0.08 msec /     0.09 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci10_thisistheljm01.wav (    0.11 msec /     0.12 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci10_onceinxen02.wav (    6.17 msec /     6.18 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci10_onceinxen01.wav (    0.13 msec /     0.14 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci10_onceinxen03.wav (    0.05 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_goingthroughwithit01.wav (    0.41 msec /     0.41 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_goingthroughwithit02.wav (    0.09 msec /     0.09 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_personalcollection01.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_personalcollection03.wav (    0.13 msec /     0.14 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_haveatit01.wav (    0.03 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_haveatit02.wav (    1.30 msec /     1.30 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_finallyfoundus01.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_finallyfoundus02.wav (    5.60 msec /     5.60 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_nonsensehunter01.wav (    0.10 msec /     0.11 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_nonsensehunter02.wav (    0.16 msec /     0.17 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_wevebeenexpectingyou01.wav (    0.37 msec /     0.38 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_wevebeenexpectingyou02.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_wevebeenexpectingyou03.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force bms_objects\clickbeep\beep10.wav (    0.03 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_thisistheguy01.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_thisistheguy03.wav (    0.07 msec /     0.08 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_iguess01.wav (    0.09 msec /     0.09 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_thisisthesupplydepot01.wav (    5.84 msec /     5.84 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_handsomespecimens01.wav (    4.99 msec /     4.99 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_handsomespecimens02.wav (    0.47 msec /     0.47 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_reasontosuspect01.wav (    2.28 msec /     2.29 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_reasontosuspect02.wav (    0.18 msec /     0.20 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_knowitwhenyouseeit01.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_mustkillit01.wav (    0.05 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_mustkillit02.wav (    0.03 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_oweusnothing01.wav (    0.07 msec /     0.08 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_oweusnothing02.wav (    0.05 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_oweusnothing03.wav (    0.05 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_taskahead01.wav (    0.06 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_taskahead02.wav (    0.03 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_brashattitude01.wav (    0.06 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_brashattitude02.wav (    0.05 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_ifyouarewilling01.wav (    0.05 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_ifyouarewilling02.wav (    0.11 msec /     0.12 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci07_ifyouarewilling03.wav (    0.10 msec /     0.12 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_yeahyoubetterkillit01.wav (    0.05 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_yeahyoubetterkillit02.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_ifyoudontwipeitout01.wav (    0.10 msec /     0.11 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_ifyoudontwipeitout02.wav (    0.05 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\grd04_ifyoudontwipeitout03.wav (    0.05 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force bms_scripted\lc\lc_airlock_close.wav (    0.15 msec /     0.15 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci11_hellofrm01.wav (    0.10 msec /     0.11 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci11_hellofrm02.wav (    0.12 msec /     0.12 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci11_canopenportal01.wav (    0.30 msec /     0.29 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci11_canopenportal02.wav (    0.06 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci11_canopenportal03.wav (    0.05 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci11_getintoposition01.wav (    0.04 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci11_getintoposition02.wav (    0.03 msec /     0.05 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci11_breach01alt.wav (    0.05 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force vo\c3a2h\sci11_portalopen_youmustgo01.wav (    0.05 msec /     0.06 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force ambient\levels\labs\equipment_printer_loop1.wav (    1.78 msec /     1.79 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force ambient\machines\keyboard_fast2_1second.wav (    0.03 msec /     0.03 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force bms_scripted\lc\lc_test_chamber.wav (   34.07 msec /    34.07 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force music\bms - end credits.ogg (   17.71 msec /    17.73 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\nihilanth\breath1.wav (    1.90 msec /     1.91 msec total )
187.760879 BlockingGetDataPointer:  Async I/O Force npc\nihilanth\breath2.wav (    1.89 msec /     1.89 msec total )

Now imagine the same amount of sounds taking 40+ milliseconds, and you have sub-1 fps.

No wait, no need to imagine, here it is:

1228.309563 BlockingGetDataPointer:  Async I/O Force physics\concrete\concrete_impact_bullet3.wav (    0.58 msec /     0.61 msec total )
1228.309563 BlockingGetDataPointer:  Async I/O Force npc\alien_slave\vort_bm_alert04.wav (   15.91 msec /    15.99 msec total )
1233.349376 BlockingGetDataPointer:  Async I/O Force weapons\mp5\single_npc.wav (   26.68 msec /    26.61 msec total )
[...]
1238.545319 BlockingGetDataPointer:  Async I/O Force weapons\hivehand\buzz.wav (   12.21 msec /    12.21 msec total )
1238.820829 BlockingGetDataPointer:  Async I/O Force weapons\hivehand\buzz.wav (   12.25 msec /    12.21 msec total )

In total, 43 of these messages appeared. fps during this time? 0. Yes, [B]0[/B]​.

[Doom64hunter]

That has most likely nothing to do with the Memory Leak itself, and I believe the effects you're describing are actually something you'd expect to happen when you see the message being spammed.

If I understand this correctly, then what the message is saying is that the normally asynchronously executed Input/Output operation to load the sound files was instead forced to execute using a blocking I/O operation.
This means that, instead of loading the sounds in parallel to the game running, the game's execution was halted to make sure the files would be properly loaded into memory. The time measurements you're seeing afterwards describe for how long the game was halted, so if the program does nothing but try and do more synchronous I/O operations, it's obvious that the framerate would go down to 0. The execution of the game has come to a complete stop.

I have seen this happen on a TF2 map once before, but then it was connected to something going horribly, horribly wrong in how the map is designed.
No idea what causes it, but it's definitely unrelated to the memory leak.

[FlyingScotsman]

I was wondering why I kept getting "out of memory" errors after a couple of matches with custom sounds (I've got 16GB of RAM, so it's certainly not running out of memory) and this seems to match what I saw when it happened.

Thanks for posting this. I was at my wits end trying to find out what was causing it!

[quake84]

Please note that this bug CAN be used to crash/flood memories of clients on servers... All you need is for them to be on a server you have control over and that's running SourceMod or any other adminmod/valveplugin hooked into /addons/ - from there it can just issue SDK calls and you know what's next.
(happily this only decreases performance slightly and can lead to crashes only - there are more efficient ways of booting people off the game (crashing it on demand) so this is a minor problem)

[LeoMozoloa]

Hello, I recently made a Half life 2 deathmatch server and some users are having this exact problem, DMPs even say out of memory, then crash do desktop. What can I do? What would be the fix? It has Sourcemod, but it seemed to crash with no sound installed.

[highlight](User was banned for this post ("Dumb Bump" - Kiwi))[/highlight]