An error here will typically mean that the directory was removed between
the time we iterated the parent and the time we wanted to visit it in
which case we should ignore it.

Other kinds of errors such as permissions (or transient errors) also
better dealt with by pretending we didn't see it.