Storing flushes in the refs vector doesn't let us recognize when the
remote is empty, as we'd always introduce at least one element into
it. These flushes aren't necessary, so we can simply ignore them.