flatbuffers: [go]Flatbuffer serialization performance is slow compared to protobuf

Env: Golang
Version : go version go1.13.3 linux/amd64

With following IDL files my intention is to measure the serialization speed of Flatbuffer . I am using golang for my analysis

namespace MyFlat;

struct Vertices {
    x : double;
    y  :double;

}
table Polygon  {

    polygons : [Vertices];
}

table Layer {

    polygons : [Polygon];
}
root_type Layer;

Here is the code I have written for calculation

package main

import (
    "MyFlat"
    "fmt"
    "io/ioutil"
    "log"
    "strconv"
    "time"

    flatbuffers "github.com/google/flatbuffers/go"
)

func calculation(size int, vertices int) {
    b := flatbuffers.NewBuilder(0)
    var polyoffset []flatbuffers.UOffsetT

    rawSize := ((16 * vertices) * size) / 1024
    var vec1 flatbuffers.UOffsetT
    var StartedAtMarshal time.Time
    var EndedAtMarshal time.Time
    StartedAtMarshal = time.Now()
    for k := 0; k < size; k++ {

        MyFlat.PolygonStartPolygonsVector(b, vertices)

        for i := 0; i < vertices; i++ {
            MyFlat.CreateVertices(b, 2.0, 2.4)

        }

        vec1 = b.EndVector(vertices)
        MyFlat.PolygonStart(b)
        MyFlat.PolygonAddPolygons(b, vec1)
        polyoffset = append(polyoffset, MyFlat.PolygonEnd(b))

    }

    MyFlat.LayerStartPolygonsVector(b, size)
    for _, offset := range polyoffset {
        b.PrependUOffsetT(offset)
    }
    vec := b.EndVector(size)
    MyFlat.LayerStart(b)
    MyFlat.LayerAddPolygons(b, vec)
    finalOffset := MyFlat.LayerEnd(b)

    b.Finish(finalOffset)
    EndedAtMarshal = time.Now()

SeElaprseTime := EndedAtMarshal.Sub(StartedAtMarshal).String()
    mybyte := b.FinishedBytes()

`   file := "/tmp/myflat_" + strconv.Itoa(size) + ".txt"`
    if err := ioutil.WriteFile(file, mybyte, 0644); err != nil {
        log.Fatalln("Failed to write address book:", err)
      }

     StartedAt := time.Now()

    layer := MyFlat.GetRootAsLayer(mybyte, 0)

     size = layer.PolygonsLength()
    obj := &MyFlat.Polygon{}
    layer.Polygons(obj, 1)

    for i := 0; i < obj.PolygonsLength(); i++ {
        objVertices := &MyFlat.Vertices{}
        obj.Polygons(objVertices, i)
        fmt.Println(objVertices.X(), objVertices.Y())
    }

    EndedAt := time.Now()
    DeElapseTime := EndedAt.Sub(StartedAt).String()
    fmt.Println(size, ",", vertices, ", ", SeElaprseTime, ",", DeElapseTime, ",", (len(mybyte) / 1024), ",", rawSize)
}

func main() {

    data := []int{500000, 1000000, 1500000, 3000000, 8000000}

    for _, size := range data {
        //calculation(size, 5)
        //calculation(size, 10)
        calculation(size, 20)

    }
}
Problem is I find it serialization is quite slow compared to protobuff with similar idl.

For 3M polygons serialization its taking almost 4.1167037s. Where in protobuf its taking half. Deserilization time for flatbuf is very less (in micro sec). In protobuf its quite high. But still if I add both flatbuf performance is lower.

Do you see any optimized way to serialize it. Flatbuffer is having a method createBinaryVector for byte vector but there is no direct way to serialize vector of polygon from a existing a user defined type vector.







Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 26 (5 by maintainers)

Most upvoted comments

Quick thoughts:

  1. Have you tried re-using the FlatBufferBuilder using its Reset function? That will re-use the scratch space, which makes it “warm”. Right now, you are using a builder that has no scratch space, so it must grow it each time.

  2. Perhaps more importantly, I’ll note that the codes in this discussion provided by @tsingson use the newer “object” FlatBuffers API (which is more ergonomic, but requires allocations). In my experience with golang, the many small allocations you are making, one for each of your “Vertices” type, probably puts a lot of memory pressure on golang’s GC. Here is the code I’m referring to:

	vlist := make([]*flat.VerticesT, 0)
	for _, v := range in {
		vl := &flat.VerticesT{X: v.X, Y: v.Y}
		vlist = append(vlist, vl)
	}

My understanding is that this code allocates 8 * 1024 * 1024 VerticesT objects on the heap. It may be better to allocate them using the flat (traditional) FlatBuffers types used in the initial post: “Vertices”, not “VerticesT”.

  1. Finally, given that @debabratadebroy is serializing three million Polygon objects in the same FlatBuffer, and those objects are Tables (in the FlatBuffers nomenclature), it’s possible that our vtable deduplicator algorithm is causing you problems. Therefore, the next experiment for one of you to run is to see how long serialization takes with increasing numbers of Polygons being serialized: 1, 2, 4, 8, 16, …, 8 * 1024 * 1024. If the serialization takes much more than linear time, then this is the culprit and we can suggest ways to work around it. (In Rust, we’ve had interest in providing a pluggable deduplicator implementation, and it might make sense to add that to Go, too).