go: proxy.golang.org: does not work in China

While having a mirror is fine, maybe some brave Chinese citizens could contact their government and let them know that they are blocking what is basically a free and open source handout of technology and knowledge to their own detriment. IIRC, the block on Github in china was lifted after similar complaints.

I know that GOPROXY=direct can prevents the go command from accessing the default proxy.golang.org. And I also know the importance of an official sumdb that can be accessed in China without using any proxies. What I’m trying to do is to highlight the current issue, that is, we’re eager for the Go team to announce an OFFICIAL Go module proxy that can be accessed in China. You may not know, the Go module proxy feature is extremely important to our Chinese Gophers. I’m pretty sure this feature is much more acceptable in China than in all other countries in the world, because if we don’t use this feature, then we’ll need a lot of other tools for getting the usual packages like the famous golang.org/x/text.

But, of course, I also know that Go team can’t build a proxy.golang.google.cn because of the no UGC policy. So, I was thinking, if Google can’t build it, then why can’t Go team work with a trusted Chinese company to build one and then officially announce it? Is Go official stuff can only run by Google? I don’t think so. So, building another official Go module proxy running in China and using a domain name that has been ICP filed in the MIIT of China, by doing so, this Go module proxy will be fully legal in China and can serve Chinese Gophers without any interference. If the Go team is willing to do this, then I think I can help you to promote this. In fact, I have donated the goproxy.cn to a Chinese company (Qiniu Cloud), and this domain name has been ICP filed by them, not me (see the picture below), so it’s now fully legal in China, and it’s quite fast now (CDNed). I’m currently working with Qiniu Cloud’s employees to make the goproxy.cn as stable as the proxy.golang.org. Our plan is to officially advertise it on the day that Go 1.13 is released. So I think the goproxy.cn is eligible to be the official Go module proxy for Go in China. If we can do this, then I’ll be happy to help the Go team and Qiniu Cloud to achieve this cooperation. And, of course, I don’t think they need to get any money from the Go team, so this cooperation will be more easier. 😊

@williamzion Really? It’s already 9102, still so racist? And, what exactly is the “FREE WORLD” you talked about? A world without China? A world without 1.42 billion population? Should we give you some time for you to snap your fingers? Well, I think I’ll just ignore your existence, my “FREE WORLD” friend.

Hi again @beoran, it’s okay. In fact, I admit that our country may not be so good. But all of us are working hard for a better future, and I believe that there will be some changes in this kind of things in our country in that future. So, yeah, let’s skip this “China” topic and move on. I also realize that it’s too late to talk about things like “There should be no default GOPROXY and GOSUMDB”. So I think your concerns should be answered directly by the Go team in #25530.

I suggest that we should treat this issue as a broader issue. I mean, what if a proxy endpoint is not blocked but is down? Should we fall back to direct?

And @bradfitz, wow boss, you’re always so efficient. I think the “heuristic check” you mentioned may be a viable option. I plan to try it after the holiday. Thank you. 😃

Okay, as you wish, boss. @bradfitz

As I said in #31757. I started the goproxy/goproxy.cn with Qiniu Cloud mainly to make everyone better use Go modules (or just Go) in the mainland of China. Note that it’s not like the goproxy.io created by @oiooj (thank you for doing this of course). The goproxy.cn will be a business-supported project rather than a personal project. This means that the goproxy.cn will be fully legal in China and we will try our best to spread it. However, when GOPROXY has a default value (a blocked endpoint), not only will our goproxy.cn become more difficult to spread, but we must even say “You must first change your GOPROXY to something else before you start Go” in every single Go beginner’s teaching article. As I said, imagine how disappointing the Chinese (and other countries with the same problems) Go beginners would be if they couldn’t run a simple “hello, world” with the gin framework or something else.

And I know disabling GOPROXY by default will not help the rest of the world. Of course, we are not selfish like that. So let’s find a more perfect way.

What about reminding users to modify their GOPROXY when the go tool gets a full timeout while getting modules?

@googollee, it’s not that simple. A requirement for Google getting Chinese approval for golang.google.cn was that we didn’t serve any user-generated content (UGC). That’s why the playground is disabled on the Chinese domain, as play.golang.org’s “Share” functionality lets people share user-generated content (code).

It’s hard to run a caching module proxy mirror without user generated content. All the code in the modules is user-generated.

It might be possible, however, to run https://sum.golang.org/ on a *.cn domain (requiring approval) since that doesn’t serve any UGC… only cryptographic sums. Unless the GitHub module names themselves could be considered UGC, which they might. Then people can run a proxy inside China, but at least everybody can be on the same checksum database. (See @FiloSottile’s comments an https://go-review.googlesource.com/c/go/+/174338/4#message-3b3b14410b157d8ebb620dd6c08268efb94f8763)

@beoran It’s hard to do that because golang.org is hosted by Google servers, and government blocked all Google servers (or trying to do that). That’s why Go team created a mirror site golang.google.cn for Chinese access. It’s reasonable that Go team could bring proxy and sum services to golang.google.cn to resolve this problem.

Hi folks, Go 1.13 has been released and https://sum.golang.google.cn is now generally available. If you’re using Go 1.13, you can set your GOSUMDB environment variable using go env -w GOSUMDB=sum.golang.google.cn. If proxy.golang.org is not available for you, you must set GOPROXY=direct or to a service that is available (some are mentioned above).

Thanks everyone for your patience on this. Closing as there is not much more we can do. If there are any issues with the sumdb in mainland China please feel free to file a separate issue.

Andy for the Go Team

Thanks everyone. We do care deeply about this issue and are doing everything we can to ensure everyone has a good experience with Go 1.13 and beyond.

As @aofei mentioned, we are able to to have Go’s sumdb available in mainland China and are currently working on this. However, the Go project is currently unable to offer proxy.golang.org inside mainland China. Fortunately, many similar services are already being hosted by Go community members in mainland China.

The Go project will not endorse any 3rd party service as “official”. The tooling is designed to make it easy to choose any of the many compatible proxies. In our blog post announcing the new changes, we will provide clear instructions on how to configure your environment to any proxy.

Thanks again for you continued patience on this issue.

Andy for the Go team

@williamzion @aofei @beoran Please, let’s keep this thread on track. It’s okay to discuss technical issues with the caching and checksum servers; and possible solutions that would make the module system usable from mainland China. Discussions about the free world or general remarks about internet censorship do not belong on the Go github issue tracker. Thanks.

So let’s head to #31757 and continue the discussion, what do you think, boss? @bradfitz

As I said there, let’s use this bug. It predates your bug and is mostly overlapping. Both are about China & the experience for users there.

Disabling GOPROXY might help China, but doesn’t help the rest of the world.

We should find a solution that works for both those firewalled and those not.

Understood. Then I think there is only one thing we can do on our side now: after the release of Go 1.13, we spread the GOPROXY configuration instructions as widely as possible in our Go communities, forums, blogs, and books etc. That’s all.

And, of course, @andybons, thank you very much for your response and Go team’s efforts. We really appreciate the Go team for caring about this issue. 🙏

I just sorted out my thoughts and wrote a more substantial snippet to better express them.

const (
	GOPROXY                        = "https://proxy.golang.org"
	GOPROXY_REACHABILITY_CHECK_URL = GOPROXY + "/foo/bar"
)

req, err := http.NewRequest(http.MethodGet, GOPROXY+"/golang.org/x/text/@latest", nil)
if err != nil {
	return err
}

ctx, cancel := context.WithCancel(context.Background())
req = req.WithContext(ctx)

isProxyRequestReturned := false

go func() { // Check in parallel whether GOPROXY can be reached
	time.Sleep(time.Second) // I think this may be what @bradfitz said
	if isProxyRequestReturned { // Minimize unnecessary checks
		return
	}

	reachabilityCheckClient := &http.Client{
		Timeout: 3 * time.Second,
	}

	res, err := reachabilityCheckClient.Get(GOPROXY_REACHABILITY_CHECK_URL)
	if err == nil { // Everything is fine
		res.Body.Close() // No body is needed, headers is more than enough
		return
	}

	if !err.(*url.Error).Timeout() { // Everything is fine too, even if there is an error here
		return
	}

	cancel() // Cancel the proxy request
	// Fall back and get the module directly from the origin
}()

res, err := http.DefaultClient.Do(req) // Do the proxy request
isProxyRequestReturned = true
if err != nil {
	return err
}
defer res.Body.Close()

// Get the module from the proxy

Are there security implications for disabling GOPROXY and GOSUMDB?

As I mentioned on Gerrit, the key and tree of the checksum database are disjoint from who serves them.

There is no need to request trust in a new entity in order to provide the service to a client that can’t reach sum.golang.org. The design of the checksum database is such that any untrusted proxy can mirror the main log. Even better, proxies can act as auditors, and ensure the log is run consistently and honestly.

The core advantage of a single checksum database is that it ensures everyone agrees on the same content for each version, and that everyone is seeing the same comprehensive list of versions. You wouldn’t want an attacker to be able to single out gosum.io users with different contents, or to show them a version that the author (who monitors sum.golang.org) is not aware of.

Proxying and auditing sum.golang.org makes everyone safer, while splitting off a separate log makes everyone less safe.

It’s important that checksum database checks are enforced precisely because they allow third-party proxies or direct fallbacks without losing most of the security properties.

The checksum database is designed as it is to make it easier for users to select alternative proxies without having to make a trust judgement call.

Is sum checking a must for projects using modules? Can we skip it?

It’s not that we are not brave, @beoran. This is an extremely complicated matter in our country. It’s because we can’t do anything about it, so some of us need help from the Go team.

And no UGC is impossible at all. So, since Go team can not set up proxies on golang.google.cn, we should talk about the #31757. At least we can set up proxies ourselves. But once the proxy.golang.org becomes the default, teaching beginners to learn Go will be a very difficult or even illegal thing (we need to teach them how to break through the network blockade) in China and countries that @davecheney mentioned in https://github.com/golang/go/issues/25530#issuecomment-469583362, right?

So let’s head to #31757 and continue the discussion, what do you think, boss? @bradfitz

For the first solution, I don’t think it solves the problem of not being able to access the proxy.golang.org in the mainland of China. And just a Chinese variant of the sum.golang.org is not that helpful, IMO.

Can you elaborate on this? GOPROXY=direct should make it unnecessary to access proxy.golang.org, while sum.golang.google.cn will provide all the sumdb security guarantees. It should be at least as functioning as Go has been so far, and more secure.

@ankushchadha

If we can add other defaults for GOPROXY, then I’d of course strongly recommend making the goproxy.cn one of them. After all, it’s as fast as the proxy.golang.org now and it can be accessed in the mainland of China (tested by https://tools.ipip.net/newping.php).

However, I think we can’t do this, at least I think the Go team will never agree to do this. In fact, a lot of Gophers may not agree, think about how hard it is for us to accept the proxy.golang.org.

According to the email sent by the Go team to @astaxie, the Go team can now offer two solutions for Chinese Gophers:

1. Using go env -w to configure the go command before go get:

go env -w GOPROXY=direct
go env -w GOSUMDB=sum.golang.google.cn # Yeah, they're willing to build a Chinese variant of the sum.golang.org

1. Making the Go distribution downloaded from the golang.google.cn/dl different from the golang.org/dl. As I discussed with @astaxie, in this solution, we can make the go command default like this:

export GOPROXY=https://goproxy.cn
export GOSUMDB=sum.golang.org # Since the goproxy.cn already supports proxying checksum databases, so no need to build the sum.golang.google.cn

For the first solution, I don’t think it solves the problem of not being able to access the proxy.golang.org in the mainland of China. And just a Chinese variant of the sum.golang.org is not that helpful, IMO.

For the last solution, I don’t know how many people know the golang.google.cn, at least I rarely hear people around me mention it. And I believe most people are using something like Homebrew or Snapd to install and manage the Go distribution. So, I think this solution is not that perfect too.

I think the best solution at the moment is the “heuristic check and fallback” mentioned by @bradfitz and @ianlancetaylor.

However, I’m still grateful to the Go team for being concerned about this issue, which means a lot to us. 🙏😊

Thanks, @aofei. The proposal here is to set https://gocenter.io as one of the default GOPROXY since it’s accessible to the golang developers all over the world.

What I’m trying to say is that we only need to check if we can reach the proxy machine through the provided address, not whether it can handle GOPROXY requests (or just any HTTP requests). But of course, a more comprehensive check is better.

So let’s say we will send a GET request. But to where? The proxy’s root path may be an option. But some proxies may use their root path for other purposes, such as showing some guidance, those may slow down our checks. So I think we should take the initiative to trigger some errors, such as 404 Not Found (in fact, no matter what the error is). I think this way may improve TTFB more or less. It’s just another unthinking view of mine.

client := &http.Client{
	Timeout: 3 * time.Second,
}

res, err := client.Get("https://proxy.golang.org/foo/bar") // 400 Bad Request
if err != nil {
	// Do something here...
}

// No body is needed, headers is more than enough.
// I think this should be almost the same as sending a HEAD request.
res.Body.Close()

Moreover, if we’re trying to implement this idea, then I think we should use net.Dial instead of net/http ways. Maybe something like:

I think we’d want to get past the (valid) TLS handshake, though.

Is sum checking a must for projects using modules? Can we skip it?

@go101 You can set environment GOSUMDB="off" to disable it now.

From docs:

GOSUMDB defaults to "sum.golang.org" when GOPROXY="https://proxy.golang.org"
and otherwise defaults to "off". NOTE: The GOSUMDB will later default to
"sum.golang.org" unconditionally.

If GOSUMDB is set to "off", or if "go get" is invoked with the -insecure flag,
the checksum database is never consulted, but at the cost of giving up the
security guarantee of verified repeatable downloads for all modules.
A better way to bypass the checksum database for specific modules is
to use the GONOSUMDB environment variable.

@ipfans You can try gosum.io first, but as @FiloSottile said, we recommend that you use sum.golang.org if you can.