Ever since commit 823c0e9c (Fix broken logic for attr cache invalidation,
2014-04-17), we have completely disabled warnings for unused functions. The only
comment that was added back then is about "annoying extra warnings" from Clang,
but in fact we shouldn't just ignore warnings about functions which aren't used
at all. Instead, the right thing would be to either only conditionally compile
functions that aren't used in all configurations or, alternatively, to remove
functions that aren't required at all.

As remaining instances of unused functions have been removed in the last two
commits, re-enable the warning.

The function `tree_iterator_entry_cmp` has been introduced in commit be30387e
(iterators: refactored tree iterator, 2016-02-25), but in fact it has never been
used at all. Remove it to avoid unused function warnings as soon as we re-enable
"-Wunused-functions".

The function `test_canonicalize` is only used on Win32 platforms. It will thus
result in an unused function warning if these warnings are enabled and one is on
a platform different than Win32.

Fix the issue by only compiling in the function on Win32 platforms.

Currently, the "apply_helper" functions used for testing the apply logic are all
statically defined in the "apply_helpers.h" header file. This may lead to
warnings from the compiler in case where this header file is included, but not
all functions it brings along are used in the compilation unit where it has been
included into.

Fix these potential warnings by moving the implementation into its own
compilation unit "apply_helpers.c".

Fix warning C4133 incompatible types in MSVC

Introduced in commit b433a22a.

Signed-off-by: Sven Strickroth <email@cs-ware.de>

index: introduce git_index_iterator

commit: fix out-of-bound reads when parsing truncated author fields

While commit objects usually should have only one author field, our commit
parser actually handles the case where a commit has multiple author fields
because some tools that exist in the wild actually write them. Detection of
those additional author fields is done by using a simple `git__prefixcmp`,
checking whether the current line starts with the string "author ". In case
where we are handed a non-NUL-terminated string that ends directly after the
space, though, we may have an out-of-bounds read of one byte when trying to
compare the expected final NUL byte.

Fix the issue by using `git__prefixncmp` instead of `git_prefixcmp`.
Unfortunately, a test cannot be easily written to catch this case. While we
could test the last error message and verify that it didn't in fact fail parsing
a signature (because that would indicate that it has in fact tried to parse the
additional "author " field, which it shouldn't be able to detect in the first
place), this doesn't work as the next line needs to be the "committer" field,
which would error out with the same error message even if we hadn't done an
out-of-bounds read.

As objects read from the object database are always NUL terminated, this issue
cannot be triggered in normal code and thus it's not security critical.

Since we were not expecting this config entry to contain a string, we
would fail as soon as its (cached) value would be accessed. Hence,
provide some constants for the 4 states we use, and account for "always"
when we decide to reflog changes.

tests: 🌀 address two null argument instances

Some OpenSSL issues

worktree: Expose git_worktree_add_init_options

transport/http: Include non-default ports in Host header

Support symlinks on Windows when core.symlinks=true

Provide a public git_index_iterator API that is backed by an index
snapshot.  This allows consumers to provide a stable iteration even
while manipulating the index during iteration.

strntol: fix out-of-bounds reads when parsing numbers with leading sign

The function `parse_number` was replaced by `git_parse_advance_digit`
which is provided by the parser interface in commit 252f2eee (parse:
implement and use `git_parse_advance_digit`, 2017-07-14). As there are
no remaining callers, remove it.

When parsing a number, we accept a leading plus or minus sign to return
a positive or negative number. When the parsed string has such a leading
sign, we set up a flag indicating that the number is negative and
advance the pointer to the next character in that string. This misses
updating the number of bytes in the string, though, which is why the
parser may later on do an out-of-bounds read.

Fix the issue by correctly updating both the pointer and the number of
remaining bytes. Furthermore, we need to check whether we actually have
any bytes left after having advanced the pointer, as otherwise the
auto-detection of the base may do an out-of-bonuds access. Add a test
that detects the out-of-bound read.

Note that this is not actually security critical. While there are a lot
of places where the function is called, all of these places are guarded
or irrelevant:

- commit list: this operates on objects from the ODB, which are always
  NUL terminated any may thus not trigger the off-by-one OOB read.

- config: the configuration is NUL terminated.

- curl stream: user input is being parsed that is always NUL terminated

- index: the index is read via `git_futils_readbuffer`, which always NUL
  terminates it.

- loose objects: used to parse the length from the object's header. As
  we check previously that the buffer contains a NUL byte, this is safe.

- rebase: this parses numbers from the rebase instruction sheet. As the
  rebase code uses `git_futils_readbuffer`, the buffer is always NUL
  terminated.

- revparse: this parses a user provided buffer that is NUL terminated.

- signature: this parser the header information of objects. As objects
  read from the ODB are always NUL terminated, this is a non-issue. The
  constructor `git_signature_from_buffer` does not accept a length
  parameter for the buffer, so the buffer needs to be NUL terminated, as
  well.

- smart transport: the buffer that is parsed is NUL terminated

- tree cache: this parses the tree cache from the index extension. The
  index itself is read via `git_futils_readbuffer`, which always NUL
  terminates it.

- winhttp transport: user input is being parsed that is always NUL
  terminated

apply: small fixups in the test suite

signature: fix out-of-bounds read when parsing timezone offset

Handle two null argument cases that occur in the unit tests.
One is in library code, the other is in test code.

Detected by running unit tests with undefined behavior sanitizer:
```bash
 # build
mkdir build && cd build
cmake -DBUILD_CLAR=ON -DCMAKE_C_FLAGS="-fsanitize=address \
-fsanitize=undefined -fstack-usage -static-libasan" ..
cmake --build .

 # run with asan
ASAN_OPTIONS="allocator_may_return_null=1" ./libgit2_clar
...
............../libgit2/src/apply.c:316:3: runtime error: null pointer \
passed as argument 1, which is declared to never be null
...................../libgit2/tests/apply/fromfile.c:46:3: runtime \
error: null pointer passed as argument 1, which is declared to never be null
```

Since commit 56ffdfc6 (buffer: deprecate `git_buf_free` in favor of
`git_buf_dispose`, 2018-02-08), the function `git_buf_free` is
deprecated and shall not be used anymore. As part of the new apply
framework that has been cooking for quite some time some new references
have been introduced to that deprecated function. Replace them with
calls to `git_buf_dispose`.

Some function calls in the new "apply" test suite were missing the
checks whether they succeeded as expected. Fix this by adding the
missing `cl_git_pass` wrappers.

Remote creation API

Index collision fixes

Patch (diff) application

smart transport: only clear url on hard reset (regression)

When the port is omitted, the server assumes the default port for the
service is used (see
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Host). In
cases where the client provided a non-default port, it should be passed
along.

This hasn't been an issue so far as the git protocol doesn't include
server-generated URIs. I encountered this when implementing Rust
registry support for Sonatype Nexus. Rust's registry uses a git
repository for the package index. Clients look at a file in the root of
the package index to find the base URL for downloading the packages.
Sonatype Nexus looks at the incoming HTTP request (Host header and URL)
to determine the client-facing URL base as it may be running behind a
load balancer or reverse proxy. This client-facing URL base is then used
to construct the package download base URL. When libgit2 fetches the
index from Nexus on a non-default port, Nexus trusts the incorrect Host
header and generates an incorrect package download base URL.

Constant strings and logic for HTTP(S) default ports were starting to be
spread throughout netops.c.  Instead of duplicating this again to
determine if a Host header should include the port, move the default
port constants and logic into an internal method in netops.{c,h}.

When parsing a signature's timezone offset, we first check whether there
is a timezone at all by verifying that there are still bytes left to
read following the time itself. The check thus looks like `time_end + 1
< buffer_end`, which is actually correct in this case. After setting the
timezone's start pointer to that location, we compute the remaining
bytes by using the formula `buffer_end - tz_start + 1`, re-using the
previous `time_end + 1`. But this is in fact missing the braces around
`(tz_start + 1)`, thus leading to an overestimation of the remaining
bytes by a length of two. In case of a non-NUL terminated buffer, this
will result in an overflow.

The function `git_signature__parse` is only used in two locations. First
is `git_signature_from_buffer`, which only accepts a string without a
length. The string thus necessarily has to be NUL terminated and cannot
trigger the issue.

The other function is `git_commit__parse_raw`, which can in fact trigger
the error as it may receive non-NUL terminated commit data. But as
objects read from the ODB are always NUL-terminated by us as a
cautionary measure, it cannot trigger the issue either.

In other words, this error does not have any impact on security.

After creating a transport for a server, we expect to be able to call
`connect`, then invoke subsequent `action` calls.  We provide the URL to
these `action` calls, although our built-in transports happen to ignore
it since they've already parsed it into an internal format that they
intend to use (`gitno_connection_data`).

In ca2eb460, we began clearing the URL
field after a connection, meaning that subsequent calls to transport
`action` callbacks would get a NULL URL, which went undetected since the
builtin transports ignore the URL when they're already connected
(instead of re-parsing it into an internal format).

Downstream custom transport implementations (eg, LibGit2Sharp) did
notice this change, however.

Since `reset_stream` is called even when we're not closing the
subtransport, update to only clear the URL when we're closing the
subtransport.  This ensures that `action` calls will get the correct URL
information even after a connection.

Tree parsing fixes

Ensure that we can add a file back after it's been removed.  Update the
renamed/deleted validation in application to not apply to deltas that
are adding files to support this.

Ensure that we cannot modify a file after it's been renamed out of the
way.  If multiple deltas exist for a single path, ensure that we do not
attempt to modify a file after it's been renamed out of the way.

To support this, we must track the paths that have been removed or
renamed; add to a string map when we remove a path and remove from the
string map if we recreate a path.  Validate that we are not applying to
a path that is in this map, unless the delta is a rename, since git
supports renaming one file to two different places in two different
deltas.

Further, test that we cannot apply a modification delta to a path that
will be created in the future by a rename (a path that does not yet
exist.)

Multiple deltas can exist in a diff, and can be applied in-order.
If there exists a delta that modifies a file followed by a delta that
renames that file, then both will be captured.  The modification delta
will be applied and the resulting file will be staged with the original
filename.  The rename delta will be independently applied - to the
original file (not the modified file from the original delta) and staged
independently.

Multiple deltas can exist in a diff, and can be applied in-order.
However if there exists a delta that renames a file, it must be first,
so that other deltas can reference the resulting target file.

git enforces this (`error: already exists in index`), so ensure that we
do, too.