Use the sorted input in the tree updater

Concurrency fixes for the reference db

We look at whether we're trying to replace a blob with a tree during the
update phase, but we fail to look at whether we've just inserted a blob
where we're now trying to insert a tree.

Update the check to look at both places. The test for this was
previously succeeding due to the bu where we did not look at the sorted
output.

The loop is made with the assumption that the inputs are sorted and not
using it leads to bad outputs.

We do not currently use the sorted version of this input in the
function, which means we produce bad results.

On Windows we can find locked files even when reading a reference or the
packed-refs file. Bubble up the error in this case as well to allow
callers on Windows to retry more intelligently.

At times we may try to delete a reference which a different thread has
already taken care of.

It does not help us to check whether the file exists before trying to
unlink it since it might be gone by the time unlink is called.

Instead try to remove it and handle the resulting error if it did not
exist.

Checking the size before we open the file descriptor can lead to the
file being replaced from under us when renames aren't quite atomic, so
we can end up reading too little of the file, leading to us thinking the
file is corrupted.

The logic simply consists of retrying for as long as the library says
the data is locked, but it eventually gets through.

This allows the caller to know the errors was e.g. due to the
packed-refs file being already locked and they can try again later.

We can reduce the duplication by cleaning up at the beginning of the
loop, since it's something we want to do every time we continue.

There might be a few threads or processes working with references
concurrently, so fortify the code to ignore errors which come from
concurrent access which do not stop us from continuing the work.

This includes ignoring an unlinking error. Either someone else removed
it or we leave the file around. In the former case the job is done, and
in the latter case, the ref is still in a valid state.

We need to save the errno, lest we clobber it in the giterr_set()
call. Also add code for reporting that a path component is missing,
which is a distinct failure mode.

In order not to undo concurrent modifications to references, we must
make sure that we only delete a loose reference if it still has the same
value as when we packed it.

This means we need to lock it and then compare the value with the one we
put in the packed file.

We can get useful information like GIT_ELOCKED out of this instead of
just -1.

We say it's going to work if you use a different repository in each
thread. Let's do precisely that in our code instead of hoping re-using
the refdb is going to work.

This test does fail currently, surfacing existing bugs.

giterr format

transports: smart: abort on early end of stream

Update THREADING for OpenSSL 1.1

git_repository_open_ext: fix handling of $GIT_NAMESPACE

fileops: fix typos in `git_futils_creat_locked{,with_path}`

The existing code would set a namespace of "" (empty string) with
GIT_NAMESPACE unset.  In a repository where refs/heads/namespaces/
exists, that can produce incorrect results.  Detect that case and avoid
setting the namespace at all.

Since that makes the last assignment to error conditional, and the
previous assignment can potentially get GIT_ENOTFOUND, set error to 0
explicitly to prevent the call from incorrectly failing with
GIT_ENOTFOUND.

We're recently trying to upgrade to the current master of libgit2
in Cargo but we're unfortunately hitting a segfault in one of our
tests. This particular test is just a small smoke test that https
works (e.g. it's configured in libgit2). It attempts to clone
from a URL which simply immediately drops connections after
they're accepted (e.g. terminate abnormally). We expect to see a
standard error from libgit2 but unfortunately we're seeing a
segfault.

This segfault is happening inside of the `wait_for` function of
`curl_stream.c` at the line `FD_SET(fd, &errfd)` because `fd` is
-1. This ends up doing an out-of-bounds array access that faults
the program. I tracked back to where this -1 came from to the
line here (returned by `CURLINFO_LASTSOCKET`) and added a check
to return an error.

global: synchronize initialization and shutdown with pthreads

The code correctly detects that forced creation of a branch on a
nonbare repo should not be able to overwrite a branch which is
the HEAD reference.  But there's no reason to prevent this on
a bare repo, and in fact, git allows this.  I.e.,

   git branch -f master new_sha

works on a bare repo with HEAD set to master.  This change fixes
that problem, and updates tests so that, for this case, both the
bare and nonbare cases are checked for correct behavior.

add support for OpenSSL 1.1.0 for BIO filter

We need to include the initialisation and construction functions in all
backend, so we include this header when building against SecureTransport
and WinHTTP as well.

pack: fix race in pack_entry_find_offset