incubator-devlake: [Bug][Gitex] Unable to cancel gitextractor when preparing data

Search before asking

I had searched in the issues and found no similar issues.

What happened

When gitextractor is preparing data, we cannot cancel the pipeline because libgit2 does not support cancel

What do you expect to happen

Even if gitextractor is preparing data, we should be able to cancel the whole pipeline

How to reproduce

Use normal mode to collect more than 2 github/gitlab repos, and when the pipeline is running, click cancel

Anything else

No response

Version

main

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project’s Code of Conduct

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 18 (13 by maintainers)

Most upvoted comments

@stultus was it stuck at the clone stage? if that was true, the only solution for now is to restart the devlake container. 🥲

klesh on Jul 4, 2023

Keep this issue.

d4x1 on Jun 26, 2023

I am still working on it and it will be updated.

d4x1 on May 17, 2023

Got it.

d4x1 on Mar 27, 2023

Well, it has been a while since we tested it. Maybe we should do another round, there were two things that impacted the performance IIRC:
1. Iterates all commits

2. Calculates diff between 2 commits.
Can we write Benchmark PoC with those actions covered to test on a couple of medium size repositories, like clickhouse and pingcap. So we can assess how slow it will be in the real-world.
Benchmark is necessary. +1 But I am doubt about whether pingcap or clikckhouse is appropriate. Does devlake team has any statistics about the pct90 size of repos it will analyse? I think this data varies greatly from scenario to scenario. But I have no idea on how to get a appropriate test repository.

Nah, not really. There are not enough resources to conduct such a massive investigation and gitext has never been a bottleneck compared to other departments, such as collecting data from API, Chart loading speed, etc.

I would say randomly picking a couple of famous and long-lived open-source projects is sufficient at this point.

klesh on Mar 27, 2023

@d4x1 I want to express my gratitude for the confirmation - it is very thoughtful and much appreciated! ❤️

klesh on Mar 24, 2023