spider: Operation not permitted from tokio-uring-0.4.0 in docker container

Hey there. I started to get following error while crawling;

thread '<unnamed>' panicked at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-uring-0.4.0/src/lib.rs:230:46:
called `Result::unwrap()` on an `Err` value: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }

I started to get this error after recently updated spider package to latest(2.24.6). Running in docker container. If I set container to run in privileged mode, then it works fine. This is after v2.23.7 which introduces tokio-uring. spider = { version = “2.22.16”, features = [“reqwest_native_tls_vendored”] } works fine. spider = { version = “2.24.6”, features = [“reqwest_native_tls_vendored”] } gives the error.

I tried to disable io_uring flag in cargo.toml as mentioned in 2.23.7 release notes but I guess there is no easy way to only disable one feature flag keeping default ones(sorry, rust noob here).

Is there anything more to follow in a container environment to work with tokio-uring?


docker version
Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.23.1
 Git commit:        ce1223035a
 Built:             Wed Sep 25 14:43:43 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.23.1
  Git commit:       41ca978a0a
  Built:            Wed Sep 25 14:43:43 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.0.1
  GitCommit:        88aa2f531d6c2922003cc7929e51daf1c14caa0a.m
 runc:
  Version:          1.2.3
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker image: rustlang/rust:nightly

And installed these packages in container build;

libssl-dev openssl pkg-config ca-certificates

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (7 by maintainers)

Most upvoted comments

You need to compile the container on Linux for it to work since it depends on io_uring on the system with a version of Linux above 4.1 or so.

What do you mean by “compile the container on linux”? Compile the docker?

Hey there. I started to get following error while crawling;

thread ‘’ panicked at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-uring-0.4.0/src/lib.rs:230:46:

called Result::unwrap() on an Err value: Os { code: 1, kind: PermissionDenied, message: “Operation not permitted” }

I started to get this error after recently updated spider package to latest(2.24.6). Running in docker container. If I set container to run in privileged mode, then it works fine. This is after v2.23.7 which introduces tokio-uring. spider = { version = “2.22.16”, features = [“reqwest_native_tls_vendored”] } works fine. spider = { version = “2.24.6”, features = [“reqwest_native_tls_vendored”] } gives the error.

I tried to disable io_uring flag in cargo.toml as mentioned in 2.23.7 release notes but I guess there is no easy way to only disable one feature flag keeping default ones(sorry, rust noob here).

Is there anything more to follow in a container environment to work with tokio-uring?

docker version

Client:

Version: 27.3.1

API version: 1.47

Go version: go1.23.1

Git commit: ce1223035a

Built: Wed Sep 25 14:43:43 2024

OS/Arch: linux/amd64

Context: default

Server:

Engine:

Version: 27.3.1

API version: 1.47 (minimum version 1.24)

Go version: go1.23.1

Git commit: 41ca978a0a

Built: Wed Sep 25 14:43:43 2024

OS/Arch: linux/amd64

Experimental: false

containerd:

Version: v2.0.1

GitCommit: 88aa2f531d6c2922003cc7929e51daf1c14caa0a.m

runc:

Version: 1.2.3

GitCommit:

docker-init:

Version: 0.19.0

GitCommit: de40ad0

docker image: rustlang/rust:nightly

And installed these packages in container build;

libssl-dev openssl pkg-config ca-certificates

Hi there, v2.24.7 now you can use the basic feature flag to get all the default prior features without tokio-uring. For the docker container it needs a new version of linux. If you compile on linux rust:1.83-alpine3.21 should work etc.

Updated to v2.24.7 with basic feature flag spider = { version = "2.24.7", features = ["basic"] }. still getting Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" } Attaching cargo tree -p spider result; spider.txt Looking at the cargo tree output, I still can see tokio-uring in the deps btw.

Close! Add this “default-features = false”.

When I put spider = { version = "2.24.7", features = ["basic"], default-features = false} it gives following error with libc;

error[E0433]: failed to resolve: use of undeclared crate or module `libc`
  --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spider-2.24.7/src/utils/connect.rs:63:28
   |
63 |                         if libc::nice(10) == -1 && *libc::__errno_location() != 0 {
   |                            ^^^^ use of undeclared crate or module `libc`

error[E0433]: failed to resolve: use of undeclared crate or module `libc`
  --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spider-2.24.7/src/utils/connect.rs:63:53
   |
63 |                         if libc::nice(10) == -1 && *libc::__errno_location() != 0 {
   |                                                     ^^^^ use of undeclared crate or module `libc`

For more information about this error, try `rustc --explain E0433`.
error: could not compile `spider` (lib) due to 2 previous errors
error: command `/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo test --no-run --message-format json-render-diagnostics` exited with code 101
make: *** [Makefile:24: test] Error 101

Btw, when I test with ubuntu 24.10(latest stable ubuntu) and spider = { version = "2.24.7" } it still gives me Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }. And searching the support for io_uring in container environment, found this issue in moby: moby/moby#47532 shows that io_uring is blocked by default in docker. Also found out that node maintainers reverted a change that makes io_uring default for the same reason: nodejs/node@686da19abb So, using spider-rs in a container environment with io_uring default is currently not possilbe.

Ty for the details, the fix for the missing libc crate is now in v2.24.8.

I can confirm that with spider = { version = "2.24.8", features = ["basic"], default-features = false } it works without failing on io_uring/tokio_uring. Thanks for the fast communication/operation.

For the your comment on:

You need to compile the container on Linux for it to work since it depends on io_uring on the system with a version of Linux above 4.1 or so.

I am on archlinux with 6.12 kernel. Compiling my app with spider in docker:rolling(vscode devcontainer). This is a RAG app that utilizes crawling capability of spider and does a lot of other stuff. And deploying to kubernetes. But currently io_uring is blocked by builtin seccomp profile in docker by default. I haven’t had a deep search on if I can enable aws eks to allow enabling io_uring, but I doubt it.

Again, thanks for fast communication and operation!

Yes, even though the docker container is using a linux image the machine compiling the image also needs to be linux with the correct kernel for it. No problem, ask any questions anytime!

Disabling seccomp seccomp=unconfined allows you to use io_uring in containers https://github.com/containers/podman/issues/16796


podman run -it -v $(pwd):/backend -w /backend --rm --security-opt seccomp=unconfined docker.io/rustlang/rust:nightly-alpine sh

Ok, I will try and let you know, thanks 🙏

On Wed, Jan 8, 2025, 16:14 Jeff Mendez @.***> wrote:

You need to compile the container on Linux for it to work since it depends on io_uring on the system with a version of Linux above 4.1 or so.

What do you mean by “compile the container on linux”? Compile the docker?

Hey there. I started to get following error while crawling;

thread ‘’ panicked at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-uring-0.4.0/src/lib.rs:230 :46:

called Result::unwrap() on an Err value: Os { code: 1, kind: PermissionDenied, message: “Operation not permitted” }

I started to get this error after recently updated spider package to latest(2.24.6). Running in docker container. If I set container to run in privileged mode, then it works fine. This is after v2.23.7 which introduces tokio-uring. spider = { version = “2.22.16”, features = [“reqwest_native_tls_vendored”] } works fine. spider = { version = “2.24.6”, features = [“reqwest_native_tls_vendored”] } gives the error.

I tried to disable io_uring flag in cargo.toml as mentioned in 2.23.7 release notes but I guess there is no easy way to only disable one feature flag keeping default ones(sorry, rust noob here).

Is there anything more to follow in a container environment to work with tokio-uring?

docker version

Client:

Version: 27.3.1

API version: 1.47

Go version: go1.23.1

Git commit: ce1223035a

Built: Wed Sep 25 14:43:43 2024

OS/Arch: linux/amd64

Context: default

Server:

Engine:

Version: 27.3.1

API version: 1.47 (minimum version 1.24)

Go version: go1.23.1

Git commit: 41ca978a0a

Built: Wed Sep 25 14:43:43 2024

OS/Arch: linux/amd64

Experimental: false

containerd:

Version: v2.0.1

GitCommit: 88aa2f531d6c2922003cc7929e51daf1c14caa0a.m

runc:

Version: 1.2.3

GitCommit:

docker-init:

Version: 0.19.0

GitCommit: de40ad0

docker image: rustlang/rust:nightly

And installed these packages in container build;

libssl-dev openssl pkg-config ca-certificates

Hi there, v2.24.7 now you can use the basic feature flag to get all the default prior features without tokio-uring. For the docker container it needs a new version of linux. If you compile on linux rust:1.83-alpine3.21 should work etc.

Updated to v2.24.7 with basic feature flag spider = { version = “2.24.7”, features = [“basic”] }. still getting Os { code: 1, kind: PermissionDenied, message: “Operation not permitted” } Attaching cargo tree -p spider result; spider.txt https://github.com/user-attachments/files/18317993/spider.txt Looking at the cargo tree output, I still can see tokio-uring in the deps btw.

Close! Add this “default-features = false”.

When I put spider = { version = “2.24.7”, features = [“basic”], default-features = false} it gives following error with libc;

error[E0433]: failed to resolve: use of undeclared crate or module libc –> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spider-2.24.7/src/utils/connect.rs:63:28 | 63 | if libc::nice(10) == -1 && *libc::__errno_location() != 0 { | ^^^^ use of undeclared crate or module libc

error[E0433]: failed to resolve: use of undeclared crate or module libc –> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spider-2.24.7/src/utils/connect.rs:63:53 | 63 | if libc::nice(10) == -1 && *libc::__errno_location() != 0 { | ^^^^ use of undeclared crate or module libc

For more information about this error, try rustc --explain E0433. error: could not compile spider (lib) due to 2 previous errors error: command /usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo test --no-run --message-format json-render-diagnostics exited with code 101 make: *** [Makefile:24: test] Error 101

Btw, when I test with ubuntu 24.10(latest stable ubuntu) and spider = { version = “2.24.7” } it still gives me Os { code: 1, kind: PermissionDenied, message: “Operation not permitted” }. And searching the support for io_uring in container environment, found this issue in moby: moby/moby#47532 https://github.com/moby/moby/issues/47532 shows that io_uring is blocked by default in docker. Also found out that node maintainers reverted a change that makes io_uring default for the same reason: @.*** https://github.com/nodejs/node/commit/686da19abb So, using spider-rs in a container environment with io_uring default is currently not possilbe.

Ty for the details, the fix for the missing libc crate is now in v2.24.8.

I can confirm that with spider = { version = “2.24.8”, features = [“basic”], default-features = false } it works without failing on io_uring/tokio_uring. Thanks for the fast communication/operation. For the your comment on:

You need to compile the container on Linux for it to work since it depends on io_uring on the system with a version of Linux above 4.1 or so.

I am on archlinux with 6.12 kernel. Compiling my app with spider in docker:rolling(vscode devcontainer). This is a RAG app that utilizes crawling capability of spider and does a lot of other stuff. And deploying to kubernetes. But currently io_uring is blocked by builtin seccomp profile in docker by default. I haven’t had a deep search on if I can enable aws eks to allow enabling io_uring, but I doubt it. Again, thanks for fast communication and operation!

Yes, even though the docker container is using a linux image the machine compiling the image also needs to be linux with the correct kernel for it. No problem, ask any questions anytime!

Just to be helpful, I wanted to clarify building the docker image part. When testing locally, I hit the same problem(io_uring operation not permitted). And my setup was;

  • archlinux with latest kernel (linux kernel 6.12)
  • with latest docker version installed
  • and using latest stable ubuntu:rolling(24.10) as container base image

docker info Client: Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: 0.19.3 Path: /usr/lib/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: 2.32.1 Path: /usr/lib/docker/cli-plugins/docker-compose

Server: Containers: 20 Running: 6 Paused: 0 Stopped: 14 Images: 178 Server Version: 27.3.1 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Using metacopy: true Native Overlay Diff: false userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 88aa2f531d6c2922003cc7929e51daf1c14caa0a.m runc version: init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 6.12.8-arch1-1 Operating System: Arch Linux OSType: linux Architecture: x86_64 CPUs: 32 Total Memory: 62.5GiB Name: gua ID: d545ec53-803e-49dc-953e-37cddb6b70c8 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled

forgive me if I misunderstood some part of your message. just wanted to clarify if it helps you to debug easier.

This is what I currently use to compile the io-uring.

AWS EC2 instance

@.*** ~]$ lscpu Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Vendor ID: ARM Model name: Neoverse-N1 Model: 1 Thread(s) per core: 1 Core(s) per socket: 64 Socket(s): 1 Stepping: r3p1 BogoMIPS: 243.75 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp Caches (sum of all): L1d: 4 MiB (64 instances) L1i: 4 MiB (64 instances) L2: 64 MiB (64 instances) L3: 32 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-63 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Reg file data sampling: Not affected Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Mitigation; CSV2, BHB Srbds: Not affected Tsx async abort: Not affected

The docker image base I use is rust:1.83-alpine3.21. You may need to run some installation setup at the start:

RUN apk add --update
build-base cmake make libc6-compat gcc g++ make pkgconfig libffi-dev zlib-dev musl-dev libressl-dev &&
rm -rf /var/cache/apk/*

RUN update-ca-certificates

— Reply to this email directly, view it on GitHub https://github.com/spider-rs/spider/issues/243#issuecomment-2577645283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACJOIL73BRMUA52LWYH4A32JUQETAVCNFSM6AAAAABUUOVCRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZXGY2DKMRYGM . You are receiving this because you authored the thread.Message ID: @.***>

You need to compile the container on Linux for it to work since it depends on io_uring on the system with a version of Linux above 4.1 or so.

What do you mean by “compile the container on linux”? Compile the docker?

Hey there. I started to get following error while crawling;

thread ‘’ panicked at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-uring-0.4.0/src/lib.rs:230:46:

called Result::unwrap() on an Err value: Os { code: 1, kind: PermissionDenied, message: “Operation not permitted” }

I started to get this error after recently updated spider package to latest(2.24.6). Running in docker container. If I set container to run in privileged mode, then it works fine. This is after v2.23.7 which introduces tokio-uring. spider = { version = “2.22.16”, features = [“reqwest_native_tls_vendored”] } works fine. spider = { version = “2.24.6”, features = [“reqwest_native_tls_vendored”] } gives the error.

I tried to disable io_uring flag in cargo.toml as mentioned in 2.23.7 release notes but I guess there is no easy way to only disable one feature flag keeping default ones(sorry, rust noob here).

Is there anything more to follow in a container environment to work with tokio-uring?

docker version

Client:

Version: 27.3.1

API version: 1.47

Go version: go1.23.1

Git commit: ce1223035a

Built: Wed Sep 25 14:43:43 2024

OS/Arch: linux/amd64

Context: default

Server:

Engine:

Version: 27.3.1

API version: 1.47 (minimum version 1.24)

Go version: go1.23.1

Git commit: 41ca978a0a

Built: Wed Sep 25 14:43:43 2024

OS/Arch: linux/amd64

Experimental: false

containerd:

Version: v2.0.1

GitCommit: 88aa2f531d6c2922003cc7929e51daf1c14caa0a.m

runc:

Version: 1.2.3

GitCommit:

docker-init:

Version: 0.19.0

GitCommit: de40ad0

docker image: rustlang/rust:nightly

And installed these packages in container build;

libssl-dev openssl pkg-config ca-certificates

Hi there, v2.24.7 now you can use the basic feature flag to get all the default prior features without tokio-uring. For the docker container it needs a new version of linux. If you compile on linux rust:1.83-alpine3.21 should work etc.

Updated to v2.24.7 with basic feature flag spider = { version = "2.24.7", features = ["basic"] }. still getting Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" } Attaching cargo tree -p spider result; spider.txt Looking at the cargo tree output, I still can see tokio-uring in the deps btw.

Close! Add this “default-features = false”.

When I put spider = { version = "2.24.7", features = ["basic"], default-features = false} it gives following error with libc;

error[E0433]: failed to resolve: use of undeclared crate or module `libc`
  --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spider-2.24.7/src/utils/connect.rs:63:28
   |
63 |                         if libc::nice(10) == -1 && *libc::__errno_location() != 0 {
   |                            ^^^^ use of undeclared crate or module `libc`

error[E0433]: failed to resolve: use of undeclared crate or module `libc`
  --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spider-2.24.7/src/utils/connect.rs:63:53
   |
63 |                         if libc::nice(10) == -1 && *libc::__errno_location() != 0 {
   |                                                     ^^^^ use of undeclared crate or module `libc`

For more information about this error, try `rustc --explain E0433`.
error: could not compile `spider` (lib) due to 2 previous errors
error: command `/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo test --no-run --message-format json-render-diagnostics` exited with code 101
make: *** [Makefile:24: test] Error 101

Btw, when I test with ubuntu 24.10(latest stable ubuntu) and spider = { version = "2.24.7" } it still gives me Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }. And searching the support for io_uring in container environment, found this issue in moby: moby/moby#47532 shows that io_uring is blocked by default in docker. Also found out that node maintainers reverted a change that makes io_uring default for the same reason: nodejs/node@686da19abb So, using spider-rs in a container environment with io_uring default is currently not possilbe.

Ty for the details, the fix for the missing libc crate is now in v2.24.8.

I can confirm that with spider = { version = "2.24.8", features = ["basic"], default-features = false } it works without failing on io_uring/tokio_uring. Thanks for the fast communication/operation.

For the your comment on:

You need to compile the container on Linux for it to work since it depends on io_uring on the system with a version of Linux above 4.1 or so.

I am on archlinux with 6.12 kernel. Compiling my app with spider in docker:rolling(vscode devcontainer). This is a RAG app that utilizes crawling capability of spider and does a lot of other stuff. And deploying to kubernetes. But currently io_uring is blocked by builtin seccomp profile in docker by default. I haven’t had a deep search on if I can enable aws eks to allow enabling io_uring, but I doubt it.

Again, thanks for fast communication and operation!