Logging in to my Server takes 1-3 Minutes?

Hey everyone. So, I decided to do a server wipe on my TTT server. I installed 3 addons-Payday 2 Styled HUD, Battlefield 4 Scoreboard, and a controllable drone addon. I added all of the appropriate files to FAST DL. (There are very few files to download since there are only 3 addons.) However, two people tried connecting to my server for the first time. They reported that it took around 1-3 minutes for them to join. After that, it seems like it takes around 10-20 seconds for them to join. I was wondering if it is normal for the first login to be long? Either way, there are literally 10 files and I don’t think it should be taking this long. :confused:

Looking for any advice or thoughts on my issue.