In the workdir iterator we do some tricky things to step down into
directories to look for things that are in our pathlist.  Make sure
that we don't confuse between folders that we're definitely going to
return everything in and folders that we're only stepping down into
to keep looking for matches.