The checkout code didn't iterate into a subdir if it didn't match the
pathspec, but since the pathspec might match files in the subdir we
should recurse into it (In contrast to gitignore handling).

Fixes #5089