Commits · ba9725a28f351e52c8054e543298f78d4d5f2cf0 · lvzhengyang / git2

11 Jul, 2019 9 commits

Merge pull request #5132 from pks-t/pks/config-stat-cache · ba9725a2
```
config_file: implement stat cache to avoid repeated rehashing
```
Patrick Steinhardt committed Jul 11, 2019
ba9725a2 Browse Files

config_file: avoid re-reading files on write · 2ba7020f

When we rewrite the configuration file due to any of its values
being modified, we call `config_refresh` to update the in-memory
representation of our config file backend. This is needlessly
wasteful though, as `config_refresh` will always open the on-disk
representation to reads the file contents while we already know
the complete file contents at this point in time as we have just
written it to disk.

Implement a new function `config_refresh_from_buffer` that will
refresh the backend's config entries from a buffer instead of
from the config file itself. Note that this will thus _not_
update the backend's timestamp, which will cause us to re-read
the buffer when performing a read operation on it. But this is
still an improvement as we now lazily re-read the contents, and
most importantly we will avoid constantly re-reading the contents
if we perform multiple write operations.

The following strace demonstrates this if we're re-writing a key
multiple times. It uses our config example with `config_set`
changed to update the file 10 times with different keys:

And now with the optimization of `config_refresh_from_buffer`:

As can be seen, this is quite a lot of `open` calls less.

committed Jul 11, 2019

2ba7020f Browse Files

config_file: split out function that sets config entries · a0dc3027

Updating a config file backend's config entries is a bit more
involved, as it requires clearing of the old config entries as
well as handling locking correctly. As we will need this
functionality in a future patch to refresh config entries from a
buffer, let's extract this into its own function
`config_set_entries`.

committed Jul 11, 2019

a0dc3027 Browse Files

config_file: split out function that reads entries from a buffer · 985f5cdf

The `config_read` function currently performs both reading the
on-disk config file as well as parsing the retrieved buffer
contents. To optimize how we refresh our config entries from an
in-memory buffer, we need to be able to directly parse buffers,
though, without involving any on-disk files at all.

Extract a new function `config_read_buffer` that sets up the
parsing logic and then parses config entries from a buffer, only.
Have `config_read` use it to avoid duplicated logic.

committed Jul 11, 2019

985f5cdf Browse Files

config_file: move refresh into `write` function · 3e1c137a

We are quite lazy in how we refresh our config file backend when
updating any of its keys: instead of just updating our in-memory
representation of the keys, we just discard the old set of keys
and then re-read the config file contents from disk. This refresh
currently happens separately at every callsite of `config_write`,
but it is clear that we _always_ want to refresh if we have
written the config file to disk. If we didn't, then we'd run
around with an outdated config file backend that does not
represent what we have on disk.

By moving the refresh into `config_write`, we are also able to
optimize the case where the config file is currently locked.
Before, we would've tried to re-read the file even if we have
only updated its cached contents without touching the on-disk
file. Thus we'd have unnecessarily stat'd the file, even though
we know that it shouldn't have been modified in the meantime due
to its lock.

committed Jul 11, 2019

3e1c137a Browse Files

config_file: implement stat cache to avoid repeated rehashing · d7f58eab

To decide whether a config file has changed, we always hash its
complete contents. This is unnecessarily expensive, as
well-behaved filesystems will always update stat information for
files which have changed. So before computing the hash, we should
first check whether the stat info has actually changed for either
the configuration file or any of its includes. This avoids having
to re-read the configuration file and its includes every time
when we check whether it's been modified.

Tracing the for-each-ref example previous to this commit, one can
see that we repeatedly re-open both the repo configuration as
well as the global configuration:

	$ strace lg2 for-each-ref |& grep config
	access("/home/pks/.gitconfig", F_OK)    = -1 ENOENT (No such file or directory)
	access("/home/pks/.config/git/config", F_OK) = 0
	access("/etc/gitconfig", F_OK)          = -1 ENOENT (No such file or directory)
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	access("/tmp/repo/.git/config", F_OK)   = 0
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	open("/tmp/repo/.git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/home/pks/.gitconfig", 0x7ffd15c05290) = -1 ENOENT (No such file or directory)
	access("/home/pks/.gitconfig", F_OK)    = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	access("/home/pks/.config/git/config", F_OK) = 0
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	open("/home/pks/.config/git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	open("/tmp/repo/.git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/home/pks/.gitconfig", 0x7ffd15c051f0) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	open("/home/pks/.config/git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	open("/tmp/repo/.git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/home/pks/.gitconfig", 0x7ffd15c05090) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	open("/home/pks/.config/git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	open("/tmp/repo/.git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/home/pks/.gitconfig", 0x7ffd15c05090) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	open("/home/pks/.config/git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	open("/tmp/repo/.git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/home/pks/.gitconfig", 0x7ffd15c05090) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	open("/home/pks/.config/git/config", O_RDONLY|O_CLOEXEC) = 3

With the change, we only do stats for those files and open them a
single time, only:

	$ strace lg2 for-each-ref |& grep config
	access("/home/pks/.gitconfig", F_OK)    = -1 ENOENT (No such file or directory)
	access("/home/pks/.config/git/config", F_OK) = 0
	access("/etc/gitconfig", F_OK)          = -1 ENOENT (No such file or directory)
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	access("/tmp/repo/.git/config", F_OK)   = 0
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	open("/tmp/repo/.git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/home/pks/.gitconfig", 0x7ffe70540d20) = -1 ENOENT (No such file or directory)
	access("/home/pks/.gitconfig", F_OK)    = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	access("/home/pks/.config/git/config", F_OK) = 0
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	open("/home/pks/.config/git/config", O_RDONLY|O_CLOEXEC) = 3
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	stat("/home/pks/.gitconfig", 0x7ffe70540ca0) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.gitconfig", 0x7ffe70540c80) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	stat("/home/pks/.gitconfig", 0x7ffe70540b40) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.gitconfig", 0x7ffe70540b20) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	stat("/home/pks/.gitconfig", 0x7ffe70540b40) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.gitconfig", 0x7ffe70540b20) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0
	stat("/tmp/repo/.git/config", {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
	stat("/home/pks/.gitconfig", 0x7ffe70540b40) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.gitconfig", 0x7ffe70540b20) = -1 ENOENT (No such file or directory)
	stat("/home/pks/.config/git/config", {st_mode=S_IFREG|0644, st_size=1154, ...}) = 0

The following benchmark has been performed with and without the
stat cache in a best-of-ten run:

```

int lg2_repro(git_repository *repo, int argc, char **argv)
{
	git_config *cfg;
	int32_t dummy;
	int i;

	UNUSED(argc);
	UNUSED(argv);

	check_lg2(git_repository_config(&cfg, repo),
			"Could not obtain config", NULL);

	for (i = 1; i < 100000; ++i)
		git_config_get_int32(&dummy, cfg, "foo.bar");

	git_config_free(cfg);
	return 0;
}
```

Without stat cache:

	$ time lg2 repro
	real    0m1.528s
	user    0m0.568s
	sys     0m0.944s

With stat cache:

	$ time lg2 repro
	real    0m0.526s
	user    0m0.268s
	sys     0m0.258s

This benchmark shows a nearly three-fold performance improvement.

This change requires that we check our configuration stress tests
as we're now in fact becoming more racy. If somebody is writing a
configuration file at nearly the same time (there is a window of
100ns on Windows-based systems), then it might be that we realize
that this file has actually changed and thus may not re-read it.
This will only happen if either an external process is rewriting
the configuration file or if the same process has multiple
`git_config` structures pointing to the same time, where one of
both is being used to write and the other one is used to read
values.

committed Jul 11, 2019

d7f58eab Browse Files

config: use `git_config_file` in favor of `struct config_file` · d0868646
Patrick Steinhardt committed Jul 11, 2019

d0868646 Browse Files

examples: implement config example · 8ee3d39a

Implement a new example that resembles git-config(1). Right now,
this example can both read and set configuration keys, only.

committed Jul 11, 2019

8ee3d39a Browse Files

cmake: report whether we are using sub-second stat information · df54c7fb

Depending on the platform and on build options, we may or may not
build libgit2 with support for nanoseconds when using `stat`
calls. It's currently unclear though whether sub-second stat
information is used at all.

Add feature info for this to tell at configure time whether it's
being used or not.

committed Jul 11, 2019

df54c7fb Browse Files

05 Jul, 2019 8 commits

Merge pull request #5143 from libgit2/ethomson/warnings · 398412cc
```
ci: build with ENABLE_WERROR on Windows
```
Patrick Steinhardt committed Jul 05, 2019
398412cc Browse Files

tests: trace: fix parameter type of aux callback · a3afda9f

The function `git_win32__stack__set_aux_cb` expects the second parameter
to be a function callback of type `git_win32__stack__aux_cb_lookup`,
which expects a `size_t` parameter. In our test suite
trace::windows::stacktrace, we declare the callback with `unsigned int`
as parameter, though, causing a compiler warning.

Correct the parameter type to silence the warning.

committed Jul 05, 2019

a3afda9f Browse Files

w32_stack: convert buffer length param to `size_t` · 2f14c4fc

In both `git_win32__stack_format` and `git_win32__stack`, we handle
buffer lengths via an integer variable. As we only ever pass buffer
sizes to it, this should be a `size_t` though to avoid loss of
precision. As we also use it to compare with other `size_t` variables,
this also silences signed/unsigned comparison warnings.

committed Jul 05, 2019

2f14c4fc Browse Files

clar: use `size_t` to keep track of current line number · 77d7e5eb

We use the `__LINE__` macro in several places throughout clar to allow
easier traceability when e.g. a test fails. While `__LINE__` is of type
`size_t`, the clar functions all accept an integer and thus may loose
precision. While unlikely that any file in our codebase will exceed a
linecount of `INT_MAX`, let's convert it anyway to silence any compiler
warnings.

committed Jul 05, 2019

77d7e5eb Browse Files

examples: avoid warning when iterating over index entries · 2dea4736

When iterating over index entries, we store the indices in an unsigned
int. As the index entrycount is a `size_t` though, this may be a loss of
precision which a compiler might rightfully complain about.

Use `size_t` instead to fix any warnings.

committed Jul 05, 2019

2dea4736 Browse Files

examples: avoid conversion warnings when calculating progress · abf24a30

When computing the progress, we perform some arithmetics that are
implicitly converting from `size_t` to `int`. In one case we're
calclulating a percentage, so we know that it should always be in the
range of [0,100] and thus we're fine. In the other case we convert from
bytes to kilobytes -- this should be stored in a `size_t` to avoid loss
of precision, even though it probably won't matter due to limited
download rates.

committed Jul 05, 2019

abf24a30 Browse Files

examples: avoid passing signed integer to `memchr` · e7bb1fe8

The memchr(3P) function expects a `size_t` as its last parameter, but we
do pass it an object size, which is of signed type `git_off_t`. As we
can be sure that the result will be non-negative, let's just cast the
parameter to a `size_t`.

committed Jul 05, 2019

e7bb1fe8 Browse Files

examples: cast away constness for reallocating head arrays · 976eed80

When reallocating commit arrays in `opts_add_commit` and
`opts_add_refish`, respectively, we simply pass the const pointer to
`xrealloc`. As `xrealloc` expects a non-const pointer, though, this will
generate a warning with some compilers.

Cast away the constness to silence compilers.

committed Jul 05, 2019

976eed80 Browse Files

04 Jul, 2019 4 commits

Merge pull request #5152 from csware/attr-system-attr-file · 5c87b5a8
```
Fix Regression: attr: Correctly load system attr file (on Windows)
```
Patrick Steinhardt committed Jul 04, 2019
5c87b5a8 Browse Files

tests: attr: add tests for system-level attributes · c87abeca

There were no tests that verified that system-level gitattributes files
get handled correctly. In fact, we have recently introduced a regression
that caused us to abort if there was a system-level gitattributes file
available.

Add two tests that verify that we're able to handle system-level
gitattributes files. The test attr::repo::sysdir_with_session would've
failed without the fix to the described regression.

committed Jul 04, 2019

c87abeca Browse Files

attr_file: completely initialize attribute sessions · 1bbec26d

The function `git_attr_session__init` is currently only initializing
setting up the attribute's session key by incrementing the repo-global
key by one. Most notably, all other members of the `git_attr_session`
struct are not getting initialized at all. So if one is to allocate a
session on the stack and then calls `git_attr_session__init`, the
session will still not be fully initialized. We have fared just fine
with that until now as all users of the function have allocated the
session structure as part of bigger structs with `calloc`, and thus its
contents have been zero-initialized implicitly already.

Fix this by explicitly zeroing out the session to enable allocation of
sessions on the stack.

committed Jul 04, 2019

1bbec26d Browse Files

attr: Don't fail in attr_setup if there exists a system attributes file · 18a6d9f3
```
Regression introduced in commit 5452e49f on PR #4967.

Signed-off-by: Sven Strickroth <email@cs-ware.de>
```
Sven Strickroth committed Jul 04, 2019
18a6d9f3 Browse Files

27 Jun, 2019 7 commits
- Merge pull request #5145 from pks-t/pks/hash-algo-uninit-return · c4c1500a
```
hash: fix missing error return on production builds
```
  Patrick Steinhardt committed Jun 27, 2019
  c4c1500a Browse Files
- hash: fix missing error return on production builds · 7fd3f32b
```
When no hash algorithm has been initialized in a given hash context,
then we will simply `assert` and not return a value at all. This works
just fine in debug builds, but on non-debug builds the assert will be
converted to a no-op and thus we do not have a proper return value.

Fix this by returning an error code in addition to the asserts.
```
  Patrick Steinhardt committed Jun 27, 2019
  7fd3f32b Browse Files
- Merge pull request #5142 from scottfurry/StaticChkFixExamples · 73427b94
```
Resolve static check warnings in example code
```
  Patrick Steinhardt committed Jun 27, 2019
  73427b94 Browse Files
- Resolve static check warnings in example code · 2ba7dbbe
```
Using cppcheck on libgit2 sources indicated two warnings in
example code.

merge.c was reported as having a memory leak. Fix applied
was to `free()` memory pointed to by `parents`.

init.c was reported as having a null pointer dereference
on variable arg. Function 'usage' was being called with
a null variable. Changed supplied parameter to empty string.
```
  Scott Furry committed Jun 27, 2019
  2ba7dbbe Browse Files
- Merge pull request #4438 from pks-t/pks/hash-algorithm · e9102def
```
Multiple hash algorithms
```
  Patrick Steinhardt committed Jun 27, 2019
  e9102def Browse Files
- Merge pull request #5128 from tiennou/fix/docs · b6625a3b
```
More documentation
```
  Patrick Steinhardt committed Jun 27, 2019
  b6625a3b Browse Files
- Merge pull request #4967 from tiennou/fix/4671 · 3d22394a
```
Incomplete commondir support
```
  Patrick Steinhardt committed Jun 27, 2019
  3d22394a Browse Files
26 Jun, 2019 6 commits

docs: More of it · 33448b45
Etienne Samson committed Jun 26, 2019

33448b45 Browse Files

repo: commondir resolution can sometimes fallback to the repodir · 501c51b2

For example, https://git-scm.com/docs/gitrepository-layout says:

    info
    Additional information about the repository is recorded in this directory.
    This directory is ignored if $GIT_COMMON_DIR is set
    and "$GIT_COMMON_DIR/info" will be used instead.

So when looking for `info/attributes`, we need to check the commondir first,
or fallback to "our" `info/attributes`.

committed Jun 26, 2019

501c51b2 Browse Files

docs: fixups · 9f723c97
Etienne Samson committed Jun 26, 2019

9f723c97 Browse Files

ignore: fix a missing commondir causing failures · b883d370

As with the preceding commit, the ignore code tries to load code from
info/exclude, and we fail to ignore a non-existent file here.

committed Jun 26, 2019

b883d370 Browse Files

attr: fix attribute lookup if repo has no common directory · 82c7a9bc

If creating a repository without a common directory (e.g. by
using `git_repository_new`), then `git_repository_item_path` will
return `GIT_ENOTFOUND` for every file that's usually located in
this directory. While we do not care for this case when looking
up the "info/attributes" file, we fail to properly ignore these
errors when setting up or collecting attributes files. Thus, the
gitattributes lookup is broken and will only ever return
`GIT_ENOTFOUND`.

Fix this issue by properly ignoring `GIT_ENOTFOUND` returned by
`git_repository_item_path`.

committed Jun 26, 2019

82c7a9bc Browse Files

attr: refactor setup to match current coding style · 5452e49f

The code in the `attr_setup` function is not really matching our
current coding style. Besides alignment issues, it's also hard to
see what functions calls depend on one another because they're
split up over multiple conditional statements.

Fix these issues by grouping together dependent function calls
and adjusting the alignment.

committed Jun 26, 2019

5452e49f Browse Files

25 Jun, 2019 6 commits
- examples: ssize_t is signed, not unsigned · b6b2d9d7
  Edward Thomson committed Jun 25, 2019
  
  b6b2d9d7 Browse Files
- examples: cast away const-ness · cd67a903
  Edward Thomson committed Jun 25, 2019
  
  cd67a903 Browse Files
- examples: don't lose `const` · 1118dd9a
  Edward Thomson committed Jun 25, 2019
  
  1118dd9a Browse Files
- example: use `git_off_t` for the object size · ede458b4
  Edward Thomson committed Jun 25, 2019
  
  ede458b4 Browse Files
- w32_stack: treat a len as an size_t · f48cf5b3
  Edward Thomson committed Jun 25, 2019
  
  f48cf5b3 Browse Files
- ci: build with ENABLE_WERROR on Windows · 6b224054
```
Build with -Werror's equivalent (/WX) on MSVC
```
  Edward Thomson committed Jun 25, 2019
  6b224054 Browse Files