flatbuffers: [go]Flatbuffer serialization performance is slow compared to protobuf
Env: Golang
Version : go version go1.13.3 linux/amd64
With following IDL files my intention is to measure the serialization speed of Flatbuffer . I am using golang for my analysis
namespace MyFlat;
struct Vertices {
x : double;
y :double;
}
table Polygon {
polygons : [Vertices];
}
table Layer {
polygons : [Polygon];
}
root_type Layer;
Here is the code I have written for calculation
package main
import (
"MyFlat"
"fmt"
"io/ioutil"
"log"
"strconv"
"time"
flatbuffers "github.com/google/flatbuffers/go"
)
func calculation(size int, vertices int) {
b := flatbuffers.NewBuilder(0)
var polyoffset []flatbuffers.UOffsetT
rawSize := ((16 * vertices) * size) / 1024
var vec1 flatbuffers.UOffsetT
var StartedAtMarshal time.Time
var EndedAtMarshal time.Time
StartedAtMarshal = time.Now()
for k := 0; k < size; k++ {
MyFlat.PolygonStartPolygonsVector(b, vertices)
for i := 0; i < vertices; i++ {
MyFlat.CreateVertices(b, 2.0, 2.4)
}
vec1 = b.EndVector(vertices)
MyFlat.PolygonStart(b)
MyFlat.PolygonAddPolygons(b, vec1)
polyoffset = append(polyoffset, MyFlat.PolygonEnd(b))
}
MyFlat.LayerStartPolygonsVector(b, size)
for _, offset := range polyoffset {
b.PrependUOffsetT(offset)
}
vec := b.EndVector(size)
MyFlat.LayerStart(b)
MyFlat.LayerAddPolygons(b, vec)
finalOffset := MyFlat.LayerEnd(b)
b.Finish(finalOffset)
EndedAtMarshal = time.Now()
SeElaprseTime := EndedAtMarshal.Sub(StartedAtMarshal).String()
mybyte := b.FinishedBytes()
` file := "/tmp/myflat_" + strconv.Itoa(size) + ".txt"`
if err := ioutil.WriteFile(file, mybyte, 0644); err != nil {
log.Fatalln("Failed to write address book:", err)
}
StartedAt := time.Now()
layer := MyFlat.GetRootAsLayer(mybyte, 0)
size = layer.PolygonsLength()
obj := &MyFlat.Polygon{}
layer.Polygons(obj, 1)
for i := 0; i < obj.PolygonsLength(); i++ {
objVertices := &MyFlat.Vertices{}
obj.Polygons(objVertices, i)
fmt.Println(objVertices.X(), objVertices.Y())
}
EndedAt := time.Now()
DeElapseTime := EndedAt.Sub(StartedAt).String()
fmt.Println(size, ",", vertices, ", ", SeElaprseTime, ",", DeElapseTime, ",", (len(mybyte) / 1024), ",", rawSize)
}
func main() {
data := []int{500000, 1000000, 1500000, 3000000, 8000000}
for _, size := range data {
//calculation(size, 5)
//calculation(size, 10)
calculation(size, 20)
}
}
Problem is I find it serialization is quite slow compared to protobuff with similar idl.
For 3M polygons serialization its taking almost 4.1167037s. Where in protobuf its taking half. Deserilization time for flatbuf is very less (in micro sec). In protobuf its quite high. But still if I add both flatbuf performance is lower.
Do you see any optimized way to serialize it. Flatbuffer is having a method createBinaryVector for byte vector but there is no direct way to serialize vector of polygon from a existing a user defined type vector.
Thanks!
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 26 (5 by maintainers)
Quick thoughts:
Have you tried re-using the FlatBufferBuilder using its
Resetfunction? That will re-use the scratch space, which makes it “warm”. Right now, you are using a builder that has no scratch space, so it must grow it each time.Perhaps more importantly, I’ll note that the codes in this discussion provided by @tsingson use the newer “object” FlatBuffers API (which is more ergonomic, but requires allocations). In my experience with golang, the many small allocations you are making, one for each of your “Vertices” type, probably puts a lot of memory pressure on golang’s GC. Here is the code I’m referring to:
My understanding is that this code allocates 8 * 1024 * 1024 VerticesT objects on the heap. It may be better to allocate them using the flat (traditional) FlatBuffers types used in the initial post: “Vertices”, not “VerticesT”.