repository: make check if repo is a worktree more strict

To determine if a repository is a worktree or not, we currently check
for the existence of a "gitdir" file inside of the repository's gitdir.
While this is sufficient for non-broken repositories, we have at least
one case of a subtly broken repository where there exists a gitdir file
inside of a gitmodule. This will cause us to misidentify the submodule
as a worktree.

While this is not really a fault of ours, we can do better here by
observing that a repository can only ever be a worktree iff its common
directory and dotgit directory are different. This allows us to make our
check whether a repo is a worktree or not more strict by doing a simple
string comparison of these two directories. This will also allow us to
do the right thing in the above case of a broken repository, as for
submodules these directories will be the same. At the same time, this
allows us to skip the `stat` check for the "gitdir" file for most
repositories.

src/repository.c

@@ -763,6 +763,13 @@ static int repo_is_worktree(unsigned *out, const git_repository *repo)
 	git_buf gitdir_link = GIT_BUF_INIT;
 	int error;
 
+	/* Worktrees cannot have the same commondir and gitdir */
+	if (repo->commondir && repo->gitdir
+	    && !strcmp(repo->commondir, repo->gitdir)) {
+		*out = 0;
+		return 0;
+	}
+
 	if ((error = git_buf_joinpath(&gitdir_link, repo->gitdir, "gitdir")) < 0)
 		return -1;
 

tests/submodule/open.c

@@ -69,3 +69,22 @@ void test_submodule_open__direct_open_succeeds(void)
 
 	git_buf_free(&path);
 }
+
+void test_submodule_open__direct_open_succeeds_for_broken_sm_with_gitdir(void)
+{
+	git_buf path = GIT_BUF_INIT;
+
+	/*
+	 * This is actually not a valid submodule, but we
+	 * encountered at least one occasion where the gitdir
+	 * file existed inside of a submodule's gitdir. As we are
+	 * now able to open these submodules correctly, we still
+	 * add a test for this.
+	 */
+	cl_git_mkfile("submod2/.git/modules/sm_unchanged/gitdir", ".git");
+	cl_git_pass(git_buf_joinpath(&path, git_repository_workdir(g_parent), "sm_unchanged"));
+	cl_git_pass(git_repository_open(&g_child, path.ptr));
+	assert_sm_valid(g_parent, g_child, "sm_unchanged");
+
+	git_buf_free(&path);
+}