gcsfuse: IO Error: interrupt out of nowhere?

I have spun up a new GPU instance on Google Cloud that I am using to train my model. The instance reads the data from the persistent disk and logs the checkpoints and log files into a Google Cloud storage bucket that is mounted with gcsfuse. Unfortunately I keep getting an IO error (see below)

INFO:training.training:Found new best! Metric: 0.39190895331317616
INFO:training.training:Saving checkpoint at global step 176
Traceback (most recent call last):
  File "src/train_script.py", line 118, in <module>
    main()
  File "src/train_script.py", line 112, in main
    args.train_path, args.val_path, logdir, args.checkpoint, hparams
  File "/home/gorjanradevski/prosiebensat1/src/training/training.py", line 228, in train_and_evaluate
    checkpoint=logdir,
  File "/home/gorjanradevski/prosiebensat1/src/training/utils.py", line 51, in save_checkpoint
    torch.save(state, filepath)
  File "/home/gorjanradevski/.local/share/virtualenvs/prosiebensat1-YKpXIpQK/lib/python3.6/site-packages/torch/serialization.py", line 161, in save
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/home/gorjanradevski/.local/share/virtualenvs/prosiebensat1-YKpXIpQK/lib/python3.6/site-packages/torch/serialization.py", line 116, in _with_file_like
    f = open(f, mode)
OSError: [Errno 5] Input/output error: 'log2/logging4/last.pth.tar'

After looking at the output from gcsfuse --foreground --debug_gcs --debug_fuse it seems like the problem is an interrupt coming from somewhere. TBH I have no idea where this interrupt might come from. If I run the train script on a local directory (not a bucket mounted with gcsfuse) everything works fine.

gcs: Req             0x2e: <- StatObject("logging4/last.pth.tar/")
gcs: Req             0x2f: <- StatObject("logging4/last.pth.tar")
gcs: Req             0x2d: -> Read("logging4/events.out.tfevents.1532356845.gorjan-k80-30gbs", <nil>) (84.454985ms): OK
fuse_debug: Op 0x0000005b        connection.go:479] -> OK ()
fuse_debug: Op 0x0000005c        connection.go:479] -> OK ()
fuse_debug: Op 0x00000060        connection.go:396] <- FlushFile (inode 7)
fuse_debug: Op 0x00000061        connection.go:396] <- WriteFile (inode 7, handle 3, offset 8192, 3701 bytes)
fuse_debug: Op 0x00000062        connection.go:396] <- SetInodeAttributes (inode 7, mtime 2018-07-23 14:41:01.192127484 +0000 UTC)
gcs: Req             0x30: <- CreateObject("logging4/events.out.tfevents.1532356845.gorjan-k80-30gbs")
gcs: Req             0x2f: -> StatObject("logging4/last.pth.tar") (182.243631ms): gcs.NotFoundError: googleapi: Error 404: No such object: luminovo-prosiebensat1-logs/logging4/last.pth.tar, notFound
gcs: Req             0x2e: -> StatObject("logging4/last.pth.tar/") (182.952289ms): gcs.NotFoundError: googleapi: Error 404: No such object: luminovo-prosiebensat1-logs/logging4/last.pth.tar/, notFound
fuse_debug: Op 0x0000005f        connection.go:481] -> Error: "no such file or directory"
fuse_debug: Op 0x00000063        connection.go:396] <- CreateFile (parent 5, name "last.pth.tar")
gcs: Req             0x31: <- CreateObject("logging4/last.pth.tar")
gcs: Req             0x30: -> CreateObject("logging4/events.out.tfevents.1532356845.gorjan-k80-30gbs") (265.8374ms): OK
fuse_debug: Op 0x00000060        connection.go:479] -> OK ()
gcs: Req             0x32: <- Read("logging4/events.out.tfevents.1532356845.gorjan-k80-30gbs", <nil>)
fuse_debug: Op 0x00000064        connection.go:396] <- interrupt (fuseid 0x00000063)
gcs: Req             0x31: -> CreateObject("logging4/last.pth.tar") (158.469322ms): Put https://www.googleapis.com/upload/storage/v1/b/luminovo-prosiebensat1-logs/o?ifGenerationMatch=0&projection=full&uploadType=resumable&upload_id=AEnB2UplIquptC6zgoZrfSBzQNAwsBZZSu8fOXIK0wR9ZN_U4eyRgSRORQLQivVhuk1PGiyUaVfW4HiF2mvWpg0Tg1h1zn2swh8ukbuwNfDKKZjj0ywqksk: net/http: request canceled
fuse_debug: Op 0x00000063        connection.go:481] -> Error: "CreateChildFile: Put https://www.googleapis.com/upload/storage/v1/b/luminovo-prosiebensat1-logs/o?ifGenerationMatch=0&projection=full&uploadType=resumable&upload_id=AEnB2UplIquptC6zgoZrfSBzQNAwsBZZSu8fOXIK0wR9ZN_U4eyRgSRORQLQivVhuk1PGiyUaVfW4HiF2mvWpg0Tg1h1zn2swh8ukbuwNfDKKZjj0ywqksk: net/http: request canceled"
fuse: 2018/07/23 14:41:01.543811 *fuseops.CreateFileOp error: CreateChildFile: Put https://www.googleapis.com/upload/storage/v1/b/luminovo-prosiebensat1-logs/o?ifGenerationMatch=0&projection=full&uploadType=resumable&upload_id=AEnB2UplIquptC6zgoZrfSBzQNAwsBZZSu8fOXIK0wR9ZN_U4eyRgSRORQLQivVhuk1PGiyUaVfW4HiF2mvWpg0Tg1h1zn2swh8ukbuwNfDKKZjj0ywqksk: net/http: request canceled
gcs: Req             0x32: -> Read("logging4/events.out.tfevents.1532356845.gorjan-k80-30gbs", <nil>) (95.750685ms): OK
fuse_debug: Op 0x00000061        connection.go:479] -> OK ()
fuse_debug: Op 0x00000062        connection.go:479] -> OK ()

Any help would be greatly appreciated 😃

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 4
  • Comments: 15 (4 by maintainers)

Commits related to this issue

Most upvoted comments

Hi there. I have the exactly same issue also.

Use go v.1.17.0 build code with copy files from disk to Google Storage bucker mount by gcsfuse.

The “operation canceled” occurs often and randomly… (seems no reason?!). Process still running not kill.

machine: GCP no GPU instance; ubuntu 18.04;

kernel version:

5.4.0-1040-gcp

context logs:

{"level":"ERROR","ts":"2021-08-25T01:26:57.370+0800","logger":"migrator","caller":"migrator.go:65","msg":"transfer files failed","pid":1,"err":"open xxx: operation canceled"}
{"level":"ERROR","ts":"2021-08-25T01:27:08.225+0800","logger":"migrator","caller":"migrator.go:65","msg":"transfer files failed","pid":1,"err":"open xxx: operation canceled"}