pgfutter: pgfutter dies with a large stacktrace randomly while importing a 150,000 line file
I have about 300,000 lines of data to load. pgfutter always dies unless I split the file into 2 and then is dies sometimes. Usually 3 tries and it takes.
Here’s the stack trace
failed MSpanList_Insert 0x5e4f70 0x6c6f6e0f0130 0x0
fatal error: MSpanList_Insert
runtime stack:
runtime.MSpanList_Insert(0x578850, 0x5e4f70)
/goroot/src/runtime/mheap.c:692 +0x8f
runtime.MHeap_Free(0x5787e0, 0x5e4f70, 0x0)
/goroot/src/runtime/mheap.c:500 +0x5b
runtime.MCentral_FreeSpan(0x57fb38, 0x5e4f70, 0x40, 0xc20809c000, 0xc20809df80, 0x0, 0x64)
/goroot/src/runtime/mcentral.c:181 +0x1bb
runtime.MSpan_Sweep(0x5e4f70, 0xff00000000, 0x1)
/goroot/src/runtime/mgc0.c:1099 +0x477
runtime.MHeap_Alloc(0x5787e0, 0x1, 0x1000000000d, 0x15159)
/goroot/src/runtime/mheap.c:240 +0x66
runtime.MCentral_CacheSpan(0x57ffb8, 0xc2080f8708)
/goroot/src/runtime/mcentral.c:85 +0x167
runtime.MCache_Refill(0x5e0000, 0xd, 0x10005e4820)
/goroot/src/runtime/mcache.c:90 +0xa0
goroutine 1 [running]:
runtime.switchtoM()
/goroot/src/runtime/asm_amd64.s:198 fp=0xc2080f87c8 sp=0xc2080f87c0
runtime.mallocgc(0xc0, 0x0, 0x3, 0x0)
/goroot/src/runtime/malloc.go:178 +0x849 fp=0xc2080f8878 sp=0xc2080f87c8
runtime.rawstring(0xbe, 0x0, 0x0, 0x0, 0x0, 0x0)
/goroot/src/runtime/string.go:195 +0x93 fp=0xc2080f88a8 sp=0xc2080f8878
runtime.slicebytetostring(0xc208082480, 0xbe, 0x207, 0x0, 0x0)
/goroot/src/runtime/string.go:69 +0x4b fp=0xc2080f8938 sp=0xc2080f88a8
encoding/csv.(*Reader).parseRecord(0xc20805c320, 0xc208141580, 0x7, 0x8, 0x0, 0x0)
/goroot/src/encoding/csv/reader.go:235 +0x449 fp=0xc2080f89f8 sp=0xc2080f8938
encoding/csv.(*Reader).Read(0xc20805c320, 0x0, 0x0, 0x0, 0x0, 0x0)
/goroot/src/encoding/csv/reader.go:136 +0x5e fp=0xc2080f8a60 sp=0xc2080f89f8
main.copyCSVRows(0xc208033aa0, 0xc20805c320, 0x0, 0xc2080337f5, 0x1, 0xc2080c8100, 0x8, 0x8, 0x0, 0x0, ...)
/usr/src/pgfutter/csv.go:57 +0xa9 fp=0xc2080f8c08 sp=0xc2080f8a60
main.importCSV(0x7fff5fbffb01, 0x13, 0xc2080f22a0, 0x6b, 0xc208033606, 0x6, 0xc208033630, 0xc, 0x0, 0xc2080f2230, ...)
/usr/src/pgfutter/csv.go:144 +0x530 fp=0xc2080f8e40 sp=0xc2080f8c08
main.func·003(0xc208080a00)
/usr/src/pgfutter/pgfutter.go:172 +0x332 fp=0xc2080f8f60 sp=0xc2080f8e40
runtime.call16(0x440268, 0xc2080580b0, 0x800000008)
/goroot/src/runtime/asm_amd64.s:401 +0x45 fp=0xc2080f8f78 sp=0xc2080f8f60
reflect.Value.call(0x29e880, 0x440268, 0x13, 0x382790, 0x4, 0xc2080f93d0, 0x1, 0x1, 0x0, 0x0, ...)
/goroot/src/reflect/value.go:419 +0x10e5 fp=0xc2080f9290 sp=0xc2080f8f78
reflect.Value.Call(0x29e880, 0x440268, 0x13, 0xc2080f93d0, 0x1, 0x1, 0x0, 0x0, 0x0)
/goroot/src/reflect/value.go:296 +0xbc fp=0xc2080f92f0 sp=0xc2080f9290
github.com/codegangsta/cli.HandleAction(0x29e880, 0x440268, 0xc208080a00, 0x0, 0x0)
/gopath/src/github.com/codegangsta/cli/app.go:487 +0x2b1 fp=0xc2080f9420 sp=0xc2080f92f0
github.com/codegangsta/cli.Command.Run(0x38eb70, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3dfcf0, 0x18, 0x0, ...)
/gopath/src/github.com/codegangsta/cli/command.go:191 +0x10a5 fp=0xc2080f9998 sp=0xc2080f9420
github.com/codegangsta/cli.(*App).Run(0xc2080f4000, 0xc20800a000, 0xf, 0xf, 0x0, 0x0)
/gopath/src/github.com/codegangsta/cli/app.go:240 +0xa76 fp=0xc2080f9c10 sp=0xc2080f9998
main.main()
/usr/src/pgfutter/pgfutter.go:178 +0xcd3 fp=0xc2080f9f98 sp=0xc2080f9c10
runtime.main()
/goroot/src/runtime/proc.go:63 +0xf3 fp=0xc2080f9fe0 sp=0xc2080f9f98
runtime.goexit()
/goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080f9fe8 sp=0xc2080f9fe0
goroutine 5 [syscall]:
os/signal.loop()
/goroot/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/goroot/src/os/signal/signal_unix.go:27 +0x35
goroutine 6 [chan receive]:
database/sql.(*DB).connectionOpener(0xc20805d720)
/goroot/src/database/sql/sql.go:589 +0x4c
created by database/sql.Open
/goroot/src/database/sql/sql.go:452 +0x31c
goroutine 9 [IO wait]:
net.(*pollDesc).Wait(0xc2080100d0, 0x72, 0x0, 0x0)
/goroot/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080100d0, 0x0, 0x0)
/goroot/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208010070, 0xc2080cc000, 0x1000, 0x1000, 0x0, 0x5f1da0, 0xc208157408)
/goroot/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208058030, 0xc2080cc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/goroot/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc2080961e0)
/goroot/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Read(0xc2080961e0, 0xc208034020, 0x5, 0x200, 0x0, 0x0, 0x0)
/goroot/src/bufio/bufio.go:174 +0x26c
io.ReadAtLeast(0x5f3398, 0xc2080961e0, 0xc208034020, 0x5, 0x200, 0x5, 0x0, 0x0, 0x0)
/goroot/src/io/io.go:298 +0xf1
io.ReadFull(0x5f3398, 0xc2080961e0, 0xc208034020, 0x5, 0x200, 0x0, 0x0, 0x0)
/goroot/src/io/io.go:316 +0x6d
github.com/lib/pq.(*conn).recvMessage(0xc208034000, 0xc208153b00, 0x0, 0x0, 0x0)
/gopath/src/github.com/lib/pq/conn.go:892 +0x17a
github.com/lib/pq.(*copyin).resploop(0xc20805a3c0)
/gopath/src/github.com/lib/pq/copy.go:143 +0x55
created by github.com/lib/pq.(*conn).prepareCopyIn
/gopath/src/github.com/lib/pq/copy.go:90 +0xae5
goroutine 10 [select]:
github.com/cheggaaa/pb.(*ProgressBar).refresher(0xc208080000)
/gopath/src/github.com/cheggaaa/pb/pb.go:423 +0xff
created by github.com/cheggaaa/pb.(*ProgressBar).Start
/gopath/src/github.com/cheggaaa/pb/pb.go:123 +0xcd
About this issue
- Original URL
- State: open
- Created 8 years ago
- Reactions: 2
- Comments: 15 (2 by maintainers)
With the provided binary package for OSX I also ran into the issue of random failures while importing a JSON file into PostgreSQL (
fatal error: MSpanList_Insert
) on macOS Sierra 10.12.5 with PostgreSQL 9.6.3.However, when building the binary from source it works. Steps:
Let me know if you need any more info with this issue. Can confirm it only happened on Mac. I’m having millions rows in one CSV, so it’s pretty unusable on Mac. Had to use Linux to test it out