fsnotify: Does not work in powerpc64 or arm64

Hi,

I package prometheus for Debian. And I am getting build errors in ppc64el, which seem to originate in fsnotify. If I try to run fsnotify’s tests there I get all kind of failures, and a hang:

github.com/go-fsnotify/fsnotify
   dh_auto_test -O--buildsystem=golang
    go test -v github.com/go-fsnotify/fsnotify
=== RUN   TestPollerWithBadFd
--- PASS: TestPollerWithBadFd (0.00s)
=== RUN   TestPollerWithData
--- FAIL: TestPollerWithData (0.00s)
    inotify_poller_test.go:85: expected poller to return true
=== RUN   TestPollerWithWakeup
--- PASS: TestPollerWithWakeup (0.00s)
=== RUN   TestPollerWithClose
--- FAIL: TestPollerWithClose (0.00s)
    inotify_poller_test.go:119: expected poller to return true
=== RUN   TestPollerWithWakeupAndData
--- FAIL: TestPollerWithWakeupAndData (0.00s)
    inotify_poller_test.go:140: expected poller to return true
=== RUN   TestPollerConcurrent
--- FAIL: TestPollerConcurrent (0.05s)
    inotify_poller_test.go:197: expected true
=== RUN   TestInotifyCloseRightAway
--- PASS: TestInotifyCloseRightAway (0.05s)
=== RUN   TestInotifyCloseSlightlyLater
--- PASS: TestInotifyCloseSlightlyLater (0.10s)
=== RUN   TestInotifyCloseSlightlyLaterWithWatch
--- PASS: TestInotifyCloseSlightlyLaterWithWatch (0.10s)
=== RUN   TestInotifyCloseAfterRead
--- PASS: TestInotifyCloseAfterRead (0.10s)
=== RUN   TestInotifyCloseCreate
--- FAIL: TestInotifyCloseCreate (0.05s)
    inotify_test.go:136: Took too long to wait for event
=== RUN   TestInotifyStress
--- FAIL: TestInotifyStress (5.00s)
    inotify_test.go:238: Expected at least 50 creates, got 0
=== RUN   TestInotifyRemoveTwice
--- PASS: TestInotifyRemoveTwice (0.00s)
=== RUN   TestInotifyInnerMapLength

signal: terminated
FAIL    github.com/go-fsnotify/fsnotify 348.362s