Tighten up which references we consider for renaming so we don't try to
rename unrelated ones and end up with unexplained references.

If there is a reference on the target namespace, git overwrites it, so
let's do the same.