Any need for cache folder for sv_downloadurl

Is there any point in transferring my cache to my fastdl server?

yes. you need the dua files

For?

So the clients can download the clientside lua.

Which contains the stuff clients need for your gamemode.