Okay, so I didn't know I could do this but opening up my script after it executed I found that nothing was actually causing that much of a delay but for some reason there is about 100 ms being added to my loop for no reason I'll include my script below
The issue occurs way at the bottom at mousemove, I've found it to related to the problem because setting fl:=A_TickCount above the mousemove event returns 32m delay while having it just under returns 110m delay, I have no idea why though.
Edit:
SetDefaultMouseSpeed was the issue, I thought 2 was like 20ms but turning it down to 1.5 fixed it. =)