Rename detection slow

odb_pack: handle duplicate objects from different packs

This is based on 24634c6f.

This also corrects an issue with error codes being mixed up with the
number of found objects.

Parse config headers with quoted quotes

Parse config headers that have the last quote on the
line quoted instead of walking off the end.

More tests for ambiguous OIDs across packs

The test coverage for ambiguous OIDs was pretty thin.  This adds
a bunch of new objects both in packs, across packs, and loose that
match to 8 characters so that we can test various cases of
ambiguous lookups.

submodule: check alloc and name presence

Fixes #1762

Create a new section of clar tests "stress" that will default to
being off where we can put slow tests that push the library for
performance testing purposes.

This restores the commented out tests (even though they're slow)
and fixes some trailing whitespace.

A rename test that illustrates a rename from a rewrite.

When using a rename source that is actually a to-be-split record,
we have to update the best-fit mapping data in both the case where
the target is also a split record and the case where the target
is a simple added record.  Before this commit, we were only doing
the update when the target was itself a split record (and even in
that case, the test was slightly wrong).

A rename test that illustrates a source matching multiple targets.

After doing further profiling, I found that a lot of time was
being spent attempting to insert hashes into the file hash
signature when using the rolling hash because the rolling hash
approach generates a hash per byte of the file instead of one
per run/line of data.

To optimize this, I decided to convert back to a run-based file
signature algorithm which would be more like core Git.

After changing this, a number of the existing tests started to
fail.  In some cases, this appears to have been because the test
was coded to be too specific to the particular results of the file
similarity metric and in some cases there appear to have been bugs
in the core rename detection code where only by the coincidence
of the file similarity scoring were the expected results being
generated.

This renames all the variables in the core rename detection code
to be more consistent and hopefully easier to follow which made it
a bit easier to reason about the behavior of that code and fix the
problems that I was seeing.  I think it's in better shape now.

There are a couple of tests now that attempt to stress test the
rename detection code and they are quite slow.  Most of the time
is spent setting up the test data on disk and in the index.  When
we roll out performance improvements for index insertion, it
should also speed up these tests I hope.

Allow Makefile.embed to be used when cross-compiling

This option is already present in the CMake config, but was missing from `Makefile.embed` and would cause all kinds of weird failures when compiling rugged on windows with the ruby devkit.

Fix possible double close

Fix -n bug; default to all ancestors

Signed-off-by: Sven Strickroth <email@cs-ware.de>

The size data in the index may not reflect the actual size of the
blob data from the ODB when content filtering comes into play.
This commit fixes rename detection to use the actual blob size when
calculating data signatures instead of the value from the index.

Because of a misunderstanding on my part, I first converted the
git_index_add_bypath API to use the post-filtered blob data size
in creating the index entry.  I backed that change out, but I
kept the overall refactoring of that routine and the new internal
git_blob__create_from_paths API because it eliminates an extra
stat() call from the code that adds a file to the index.

The existing tests actually cover this code path, at least when
running on Windows, so at this point I'm not adding new tests to
cover the changes.

Close p->mwf.fd only if necessary

This fixes a regression introduced in revision 9d2f841a.

Signed-off-by: Sven Strickroth <email@cs-ware.de>

The previous fix for checking file sizes with rename detection
always loads the blob.  In this version, if the odb backend can
get the object header without loading the whole thing into memory,
then we'll just use that, so that we can eliminate possible rename
sources & targets without loading them.

The performance improvements I introduced for rename detection
were not able to run successfully for tree-to-tree diffs because
the blob size was not known early enough and so the file signature
always had to be calculated nonetheless.

This change separates loading blobs into memory from calculating
the signature.  I can't avoid having to load the large blobs into
memory, but by moving it forward, I'm able to avoid the signature
calculation if the blob won't come into play for renames.

This allows libgit2 to be cross-compiled (e.g. when building native rugged binaries for windows from Linux or OS X).

```
CROSS_COMPILE=i686-w64-mingw32 make -f Makefile.embed
```

Before the optimization commits, this test used to take about 20
seconds to run on my machine.  Afterwards, there is still a couple
seconds of data setup, but the actual diff and rename detection
runs in a fraction of a second.

Doc fixes

The description of what the function does hasn't been true for quite a
while. Change it to reflect the way it currently works.

While here, remove an even older comment about missing features that
have been implemented.

clang's docparser highlighted these.

Invalid refs on disk cause revwalk globbing to fail

The new tests don't always want to use the same fixture data as
the old ones so this makes it configurable on a per-test basis.