Since references are not owned by the repository anymore we have to free
them manually now.

Signed-off-by: schu <schu-github@schulog.org>