Store the ref information in a private struct so we can free it
together with the rest.

Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>