The display reader count only increments when wl_display_prepare_read
succeeds. Calling wl_display_cancel_read when wl_display_prepare_read
has not succeeded results in a negative reader count, which does not
have well-defined behavior.
Calling wl_display_cancel_read immediately ensures that other readers
waiting on us will have a chance to wake up and discover the error in a
timely manner.