Turns out I was drawing the scene way more times than I needed, probably causing excess v-blank waiting. When the draw rate is clamped to my target framerate, instead of drawing after every input/logic update, the latency is practically gone. All I need now is a way to calculate the screen refresh rate so I can clamp the draw rate to that instead.