You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature][GitExtractor] Support incremental sync when collecting Git data #6853
GitExtractor now collects repo's all commits every time when it's executed. I'll update it and make GitExtractor only collect new commits after last run with go-git.
But GitExtractor will ignore project's sync policy such FullSync or TimeAfter. There are some reasons:
When components config is updated, all data in commit_file_components should be recalculated. But there is no entry point updating it in Config UI. So supporting FullSync is unecessary so far.
In a repository, commit id may change when rebase happens. After rebase operation, old commits will be dangling comits, and if these commits still exist in database, they have no side effect. And new commit ids will be collected in the next run.
In a summary, GitExtractor will not support FullSync so far.
I wan to implement this feature in package go-git, after merging #6701.
But in #6701, I do some benchmarks with go-git, the result is pessimistic. When collecting commits' detail, go-git is slower than libgit2 about 5-6 times(46min vs 6hour on clickhouse), which is unacceptable.
So I have to deply the development until we can get a similar performance with go-git.
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
Activity
d4x1 commentedon Feb 6, 2024
GitExtractor now collects repo's all commits every time when it's executed. I'll update it and make GitExtractor only collect new commits after last run with go-git.
But GitExtractor will ignore project's sync policy such
FullSyncorTimeAfter. There are some reasons:componentsconfig is updated, all data incommit_file_componentsshould be recalculated. But there is no entry point updating it in Config UI. So supportingFullSyncis unecessary so far.rebasehappens. After rebase operation, old commits will be dangling comits, and if these commits still exist in database, they have no side effect. And new commit ids will be collected in the next run.In a summary, GitExtractor will not support
FullSyncso far.d4x1 commentedon Feb 21, 2024
I wan to implement this feature in package go-git, after merging #6701.
But in #6701, I do some benchmarks with go-git, the result is pessimistic. When collecting commits' detail, go-git is slower than libgit2 about 5-6 times(46min vs 6hour on clickhouse), which is unacceptable.
So I have to deply the development until we can get a similar performance with go-git.
cc @Startrekzky @klesh
github-actions commentedon Apr 22, 2024
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
klesh commentedon Apr 24, 2024
Closed by #7319