fn: API error (500): invalid mode: /tmp/iofs

Description Unable to execute any functions 😦

Steps to reproduce the issue:

Downloaded latest version of fn.exe CLI
Ran fn start
Created new function as per quickstart guide (tried both Node.js and Python)
Tried executing function using fn invoke

Describe the results you received:

λ fn invoke helloapp hellopy
Error invoking function. status: 500 message: internal server error

Error:

time="2019-03-30T15:49:51Z" level=info msg="Fn serving on `:8080`" type=full
time="2019-03-30T15:50:19Z" level=error msg="Could not create container" app_id=01D77KZNGHNG8G00GZJ0000001 cpus= error="API error (500): invalid mode: /tmp/iofs" fn_id=01D77MHK5YNG8G00GZJ0000003 id=01D77MPSYDNG8G00GZJ0000002 idle_timeout=30 image="hellopy:0.0.2" memory=256 stack=CreateContainer
time="2019-03-30T15:50:19Z" level=error msg="internal server error" action="server.handleFnInvokeCall)-fm" error="API error (500): invalid mode: /tmp/iofs" fn_id=01D77MHK5YNG8G00GZJ0000003 stack="goroutine 127 [running]:
runtime/debug.Stack(0xc42047fce0, 0x1297ca0, 0xc4204ab960)
	/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
github.com/fnproject/fn/api/server.HandleErrorResponse(0x12a3fc0, 0xc4201a96e0, 0x7f5dad00e7c0, 0xc4200e82c0, 0x1297ca0, 0xc4204ab960)
	/go/src/github.com/fnproject/fn/api/server/error_response.go:57 +0x57f
github.com/fnproject/fn/api/server.handleErrorResponse(0xc4200e82c0, 0x1297ca0, 0xc4204ab960)
	/go/src/github.com/fnproject/fn/api/server/error_response.go:25 +0xa9
github.com/fnproject/fn/api/server.(*Server).handleFnInvokeCall(0xc42040eb40, 0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/runner_fninvoke.go:53 +0x1da
github.com/fnproject/fn/api/server.(*Server).(github.com/fnproject/fn/api/server.handleFnInvokeCall)-fm(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/server.go:1198 +0x34
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.(*Server).runMiddleware(0xc42040eb40, 0xc4200e82c0, 0x0, 0x0, 0x0)
	/go/src/github.com/fnproject/fn/api/server/middleware.go:42 +0x29f
github.com/fnproject/fn/api/server.(*Server).rootMiddlewareWrapper.func1(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/middleware.go:32 +0x58
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.panicWrap(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:190 +0x51
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.panicWrap(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:190 +0x51
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.apiMetricsWrap.func1.1(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:154 +0x2f8
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.traceWrap(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:97 +0x507
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.loggerWrap(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:212 +0x1ea
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc42040ea20, 0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/gin.go:361 +0x586
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc42040ea20, 0x12a3940, 0xc420495630, 0xc42042ed00)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/gin.go:326 +0x153
github.com/fnproject/fn/vendor/go.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP(0xc4200adf40, 0x12a3940, 0xc420495630, 0xc420152c00)
	/go/src/github.com/fnproject/fn/vendor/go.opencensus.io/plugin/ochttp/server.go:86 +0x1c8
net/http.serverHandler.ServeHTTP(0xc42040ab60, 0x12a25c0, 0xc420242460, 0xc420152c00)
	/usr/local/go/src/net/http/server.go:2697 +0xbc
net/http.(*conn).serve(0xc4200d20a0, 0x12a3f00, 0xc420503240)
	/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2798 +0x27b
"

Describe the results you expected: Wanted it to work 😦

Additional information you deem important (e.g. issue happens only occasionally): Happens 100% of the time

Output of fn version (CLI command):

Client version is latest version: 0.5.65
Server version:  0.3.682

Additional environment details (OSX, Linux, flags, etc.): Windows 10, 1809

About this issue

Original URL
State: open
Created 5 years ago
Comments: 16 (7 by maintainers)

Most upvoted comments

Ok - confirmed a few things are going on here:

We need to make sure directories exist prior to mounting on windows (d4w won’t create a source dir as part of a bind mount as for linux)
unix sockets do_not work when mounted via host volumes on d4m - there is some magic for /var/run/docker.sock which does not translate over the standard (cifs) host volume shares
paths needed to be treated carefully - all forward slashes etc.

For unix sockets we end up in a similar situation as we had for mac except we could create them on xyhve shared volume mounts but can’t on windows-

There is a gnarly solution which is insecure :

fn cli ensures that a docker volume “fniofs” exists,
starts fn with this mounted with some flags
fn starts containers with this mounted and tells them where to put their unix sockets.

the downside here is that each function can see each other functions sockets - v bad but it would work I think.

Docker doesn’t support mounting paths of volumes, and unless I’m mistaken this is the only way to get a pure-blood linux dir that supports unix sockets and can be shared between containers in d4w

zootalures on Mar 31, 2019