swift: utils/update-checkout error from git checkout icu maint/maint-69

Description

https://github.com/apple/swift/blob/main/docs/HowToGuides/GettingStarted.md#cloning-the-project

Hi! I just tried following the cloning-the-project steps on a new (vanilla) swift checkout (from latest main). I’m seeing some errors on console:

Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/Users/rick/Developer/swift-project/swift/utils/update_checkout/update_checkout/update_checkout.py", line 376, in obtain_additional_swift_sources
    shell.run(['git', '--git-dir',
  File "/Users/rick/Developer/swift-project/swift/utils/swift_build_support/swift_build_support/shell.py", line 257, in run
    raise eout
Exception: ['git', '--git-dir', '/Users/rick/Developer/swift-project/icu/.git', '--work-tree', '/Users/rick/Developer/swift-project/icu', 'checkout', 'maint/maint-69']

I tried running cloning-the-project again and see similar errors:

[icu]                                   + git checkout maint/maint-69
[icu]                                   error: Your local changes to the following files would be overwritten by checkout:
[icu]                                   	.ci-builds/.azure-exhaustive-tests.yml
[icu]                                   	.ci-builds/.azure-pipelines.yml
[icu]                                   	.ci-builds/data-filter.json
...
[icu]                                   error: The following untracked working tree files would be overwritten by checkout:
[icu]                                   	.appveyor.yml
[icu]                                   	.ci-builds/.azure-valgrind.yml
[icu]                                   	.github/workflows/icu_ci.yml
...
[icu]                                   	ic

Anyone seen these errors before or know the steps to work around and test building swift locally? I’m on Apple Silicon BTW. Thanks!

Reproduction

mkdir swift-project
cd swift-project
git clone https://github.com/apple/swift.git swift
cd swift
utils/update-checkout --clone-with-ssh

Expected behavior

utils/update-checkout succeeds and installs deps with no errors.

Environment

uname -a
Darwin C97RX2X127.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020 arm64
sw_vers
ProductName:		macOS
ProductVersion:		14.2.1
BuildVersion:		23C71
git --version
git version 2.39.3 (Apple Git-145)
git remote -v
origin	git@github.com:apple/swift.git (fetch)
origin	git@github.com:apple/swift.git (push)
git branch
* main
git rev-parse HEAD
e7475c89edb66a26d762c6e9a24c9bdb1a80b87a

Additional information

No response

About this issue

  • Original URL
  • State: open
  • Created 5 months ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

Exception: ['git', '--git-dir', '/Users/rick/Developer/swift-project/icu/.git', '--work-tree', '/Users/rick/Developer/swift-project/icu', 'checkout', 'maint/maint-69']

Is this all it prints—no further details on the exception?


Try utils/update-checkout --clone-with-ssh --skip-repository icu as a workaround. I think the ICU checkout is for non-Darwin platforms.

Workaround: --skip-repository icu

it will fail the initial sync but should succeed subsequently

@compnerd Here is what I saw. The second attempt also failed:

error: The following untracked working tree files would be overwritten by checkout

ICU is not windows specific, it is needed for Linux, Android, PS4, Windows, and WASI off the top of my head. Potentially for Haiku as well. It is not needed very specifically for Darwin because it is required for Foundation, which you do not build on Darwin and instead use the system provided copy. I think that introducing a --do-not-clone-foundation-dependencies would be a better name for skipping ICU, CURL, and libxml2.

I think that it is better to have the uniform repository checkouts. I think it was less of a concern when Windows was less reliant on update-checkout, but since there was a push to homogenise the CI environment, I think that uniformly checking out the repositories is a reasonable expectation. It is technically possible to build for Windows from macOS, and that would require that the sources are checked out.

I’m happy to accept whatever the platform support folks think is best, but we need to be careful not to forget to update project dependencies.

Giving those engineers the option to skip out through just one flag (something like skip-windows-deps?) could improve the DevX… what do you think about that?

Yes, having tailored checkouts for specific workflows would be nice. This has been on my radar for a while. An average compiler development workflow needs only half, if not less, of what update-checkout clones.

@AnthonyLatsis I think that it is better to have the uniform repository checkouts. I think it was less of a concern when Windows was less reliant on update-checkout, but since there was a push to homogenise the CI environment, I think that uniformly checking out the repositories is a reasonable expectation. It is technically possible to build for Windows from macOS, and that would require that the sources are checked out.

Thanks for checking!

it looks like the CI jobs are running from Intel?

Yes.

Has this error never thrown on CI before?

No idea. This was irrelevant to macOS until recently because icu had been specified as a Linux-only dependency. macOS CI nodes are running fine, so they must have git-lfs installed, purposefully or not.

@compnerd Was making icu part of the macOS checkout intentional? Also, I have never seen anyone run into this on Linux either, despite git-lfs not being immediately among the Linux build dependencies listed in the Docker files. Is it a transitive dependency?

Hm. I guess we’re not getting any output from Git on why the checkout subcommand failed because of the echo=False here:

https://github.com/apple/swift/blob/1da24ece36a799ce23bf7c2f2e8f24db5a0fa62a/utils/update_checkout/update_checkout/update_checkout.py#L376-L381

Cloning dependencies from scratch with echo=True might help us identify the problem it you’re up for it.