containerized-data-importer: HTTP Datasource can't handle s3 presigned URL for ISO file

What happened: Create a DataVolume with HTTP Datasource. The HTTP URL is a s3 presigned URL, which is only signed with HTTP Method ‘GET’ (which means any HTTP Method other than ‘GET’ is forbidden by s3 policy)

I got error log from cdi-importer, looks like nbdkit try to do HTTP ‘HEAD’ on the s3 presigned URL, which is forbidden.

E0601 12:44:17.844208       1 importer.go:177] qemu-img: Could not open 'nbd+unix:///?socket=/tmp/nbdkit.sock': Requested export not available
, qemu-img execution failed: exit status 1 Log line from nbdkit: nbdkit: curl[1]: error: problem doing HEAD request to fetch size of URL [http://xxx/]: HTTP response code said error: The requested URL returned error: 403
Unable to convert source data to target format

kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func6
	pkg/importer/data-processor.go:242
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:275
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:184
main.handleImport
	cmd/cdi-importer/importer.go:174
main.main
	cmd/cdi-importer/importer.go:140
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1571

What you expected to happen: A clear and concise description of what you expected to happen.

How to reproduce it (as minimally and precisely as possible): Steps to reproduce the behavior.

Additional context: I can’t use s3 datasource, because I think it’s not safe to use s3 AK/SK just to download something.

Environment:

CDI version: 1.56.0
Others: nbdkit-1.30.1-2.el9

About this issue

Original URL
State: closed
Created a year ago
Comments: 23 (6 by maintainers)

Commits related to this issue

tests/web-server.c: Add ability to selectively fail HEAD requests For a future commit we will need to be able to simulate an Amazon AWS S3 server, which in some circumstances will fail HEAD requests ... — committed to asomers/nbdkit by rwmjones a year ago
curl: Fallback to GET if HEAD not supported Some servers do not support HEAD for requesting the headers. If the HEAD request fails, fallback to using the GET method, abandoning the transfer as soon ... — committed to asomers/nbdkit by rwmjones a year ago

Most upvoted comments

My test:

$ ./nbdkit curl '[VERY LONG URL]' --run 'nbdinfo $uri'
nbdkit: curl[1]: error: problem doing HEAD request to fetch size of URL [VERY LONG URL]: HTTP response code said error: The requested URL returned error: 403
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 51380224 (49M)
	content: DOS/MBR boot sector; partition 2 : ID=0xef, start-CHS (0x3ff,254,63), end-CHS (0x3ff,254,63), startsector 300, 2880 sectors
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: false
	can_df: true
	can_fast_zero: true
	can_flush: false
	can_fua: false
	can_multi_conn: false
	can_trim: false
	can_zero: true

So it still prints the error on the fallback path, but at least it doesn’t fail.

rwmjones on Jun 6, 2023

Hey @lxs137, this should’ve been temporarily fixed with https://github.com/kubevirt/containerized-data-importer/pull/2841. We plan to release this version today, so feel free to try it when ready and confirm if your issue persists.

Edit: v1.57 is already out https://github.com/kubevirt/containerized-data-importer/releases/tag/v1.57.0

alromeros on Aug 16, 2023