Skip to content

fix(gitextractor): save state after full clone - #9401

Open
fxgagnon wants to merge 1 commit into
apache:mainfrom
fxgagnon:fix/gitextractor-full-clone-state
Open

fxgagnon wants to merge 1 commit into
apache:mainfrom
fxgagnon:fix/gitextractor-full-clone-state

Conversation

@fxgagnon

@fxgagnon fxgagnon commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Pre Checklist

Please complete ALL items in this checklist, and remove before submitting

  • I have read through the Contributing Documentation.
  • I have added relevant tests.
  • I have added relevant documentation (no documentation changes are needed for this internal state-persistence fix).
  • I will add labels to the PR, such as pr-type/bug-fix, pr-type/feature-development, etc.

Summary

Save subtask state after a successful full clone so blueprints without timeAfter can run incrementally on subsequent executions. Adds a regression test for state persistence after a full clone.

Does this close any open issues?

Does not close an open issue. But we've observed that:

What happened

For blueprints whose timeAfter is null (“collect all data”), gitextractor never switches to incremental mode. Every scheduled pipeline performs a full git clone --bare and recomputes commit stats for the entire history of every branch.

On one of our repositories, the gitextractor task takes about an hour on every daily run. Processing a day’s new commits should take seconds; the GitHub GraphQL collectors in the same pipeline finish in about a minute.

Root cause

Since a26791b (#7941 / #7944, “double clone against AzDo”), GitcliCloner.CloneRepo() sets g.success = true only in the since != nil (shallow/double-clone) branch. CloseRepo() persists the subtask state only when g.success is true.

See clone_gitcli.go:

func (g *GitcliCloner) CloneRepo() errors.Error {
	if g.since == nil {
		// Full sync
		if err := g.fullClone(); err != nil {
			return err
		}
	} else {
		// ...
		g.deepenUntilSince()
		g.success = true // Only reached when since != nil
	}

	return nil
}

func (g *GitcliCloner) CloseRepo() errors.Error {
	if g.success {
		g.logger.Info("save state")
		return g.stateManager.Close()
	}

	return nil
}

When the blueprint has no timeAfter, these steps repeat on every run:

  1. There is no row in _devlake_subtask_states, so PrevStartedAt == nil. calculateStateManagerIncrementalMode returns (false, syncPolicy.TimeAfter), which is (false, nil) (subtask_state_manager.go).
  2. Since since == nil, fullClone() runs and g.success remains false.
  3. CloseRepo() skips stateManager.Close(), so PrevStartedAt is never saved (subtask_state_manager.go).

The next run starts over at step 1. The bootstrapStateFromCollectorStateIfNeeded fallback does not help because the cloner does not pass a Table.

Before a26791b, CloneRepo called g.stateManager.Close() after any successful clone, including a full clone (clone_gitcli.go at 7a805c1). After the first full run, subsequent runs were incremental. Since #7941 describes adding the AzDo double clone and a readability refactor without mentioning this behavior change, it appears to be unintentional.

What do you expect to happen?

After a successful full clone, the subtask state should be saved. The next run should then be incremental (since = PrevStartedAt), using a shallow fetch and collecting only new commits, as introduced by #7319.

How to reproduce

  1. Create a project/blueprint with a GitHub or GitLab repository scope and leave the data time range empty (timeAfter: null).
  2. Run the blueprint twice.
  3. Observe that both runs perform a full clone and recompute commit stats for the repository’s entire history.
  4. Set any timeAfter value and run the blueprint twice. The second run is incremental.

Anything else

The suggested fix is to mark the clone successful after either branch completes:

func (g *GitcliCloner) CloneRepo() errors.Error {
	if g.since == nil {
		if err := g.fullClone(); err != nil {
			return err
		}
	} else {
		// ... unchanged ...
		g.deepenUntilSince()
	}

	g.success = true
	return nil
}

A related case occurs when running “Collect Data in Full Refresh Mode” with no timeAfter: state is not saved, so the stored PrevStartedAt remains at its pre-refresh value.

Workaround: Set a timeAfter value on the blueprint.

Screenshots

Not applicable.

Other Information

Validation: git diff --check passed. go test ./plugins/gitextractor/parser was not run because Go and gofmt are not installed in the environment.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant