Most common reason of long fullscreen alt-tabbing in DirectX 9 games (along with Garry’s Mod) is device removal failure described in this article. This causes the game to recreate and reload all active resources which mostly takes a while. Starting with DirectX 11 that is not a problem anymore as device removal can happen under rare circumstances.
Update: the game should also handle window messages in time, otherwise OS is unable to switch display modes for application.
Windowed mode has its own flaws too, for example it increases input lag as image should be drawn to window buffer at first, only then it can be drawn to display.