Don't use the filter's free callback to free the actual data structure
holding the filter, as we may not always actually initialize it (the
test may be skipped).