There should be a time period before the votes are removed. A witness might disable their witness for things like maintenance or upgrades and don't want to miss blocks during that time.

_{Posted using Steeve, an AI-powered Steem interface}