Skip to content

[Feature][GitExtractor] Support incremental sync when collecting Git data #6853

Closed

Description

@Startrekzky

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Use case

As a user who has large repos with more than 100,000 commits, I'd like to have incremental sync when collecting Git data.

Currently, every pipeline takes more than 5 hours to collect data. That makes it difficult to utilize DevLake in my org.

Description

Support incremental sync in the GitExtractor plugin. Specifically,

Entity Sync Mode Cursor Field
repos Full refresh. There's no need to be incremental N/A
refs Full refresh. There's no create/update date of the ref as far as I know N/A
commits Incremental committed_date. It seems to make more sense than commits.authored_date
commit_files Incremental committed_date. Update the commit_files of the new commits.

Related issues

#6138

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. added this to the v1.0 milestone on Jan 19, 2024
  2. d4x1 commented on Feb 6, 2024

    @d4x1
    Contributor

    GitExtractor now collects repo's all commits every time when it's executed. I'll update it and make GitExtractor only collect new commits after last run with go-git.
    But GitExtractor will ignore project's sync policy such FullSync or TimeAfter. There are some reasons:

    1. When components config is updated, all data in commit_file_components should be recalculated. But there is no entry point updating it in Config UI. So supporting FullSync is unecessary so far.
    2. In a repository, commit id may change when rebase happens. After rebase operation, old commits will be dangling comits, and if these commits still exist in database, they have no side effect. And new commit ids will be collected in the next run.

    In a summary, GitExtractor will not support FullSync so far.

  3. d4x1 commented on Feb 21, 2024

    @d4x1
    Contributor

    I wan to implement this feature in package go-git, after merging #6701.
    But in #6701, I do some benchmarks with go-git, the result is pessimistic. When collecting commits' detail, go-git is slower than libgit2 about 5-6 times(46min vs 6hour on clickhouse), which is unacceptable.
    So I have to deply the development until we can get a similar performance with go-git.

    cc @Startrekzky @klesh

  4. removed this from the v1.0 milestone on Feb 21, 2024
  5. github-actions commented on Apr 22, 2024

    @github-actions
    Contributor

    This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.

  6. klesh commented on Apr 24, 2024

    @klesh
    Contributor

    Closed by #7319

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions