Explicitly check if we actually timed out

Explicitly check if we actually timed out #4050