influxdb: Backup/restore fails with a lot of databases

The fix from PR #17495 has been merged to master-1.x on 31 Mar. If I rebuild influxd from master-1.x it works.

So the fix was not released neither with 1.8.0 (Jun), nor 1.8.1 (Jul), nor 1.8.2 (Aug) 😐 Why? People can’t restore backups… However I have to say some backups do restore ok, but some don’t due to this issue.

This seems pretty critical. Is this prioritized?

I submitted the patch above. Anyone interested, please review so we can get this merged as quickly as possible.

my influxdb version is: 1.7.6 I found the problem in file ”influxdata/influxdb/services/snapshotter/service.go“: `func (s *Service) readRequest(conn net.Conn) (Request, []byte, error) { var r Request d := json.NewDecoder(conn)

if err := d.Decode(&r); err != nil {
	return r, nil, err
}

bits := make([]byte, r.UploadSize+1)

if r.UploadSize > 0 {

	remainder := d.Buffered()

	n, err := remainder.Read(bits)
	if err != nil && err != io.EOF {
		return r, bits, err
	}

	// it is a bit random but sometimes the Json decoder will consume all the bytes and sometimes
	// it will leave a few behind.
	if err != io.EOF && n < int(r.UploadSize+1) {
		_, err = conn.Read(bits[n:])
	}

	if err != nil && err != io.EOF {
		return r, bits, err
	}
	// the JSON encoder on the client side seems to write an extra byte, so trim that off the front.
	return r, bits[1:], nil
}

return r, bits, nil

}` This func is to read the contents of the metadata file from the TCP connection. Because the file is too large, it needs to be sent several times, but it only receives two times at most when it is received. That is to say, when the metadata file is too large, it can not receive the complete content. So it should be improved here to fully receive the data sent by the client.

I have modified and compiled this part of code in my environment. After modification, there will be no such problem as “proto: meta.data: illegal tag 0 (wire type 0)”. The restore command is executed successfully.

And my solution is： ` func (s *Service) readRequest(conn net.Conn) (Request, []byte, error) { var r Request d := json.NewDecoder(conn)

if err := d.Decode(&r); err != nil {
    return r, nil, err
}
var buffer bytes.Buffer
if r.UploadSize > 0 {
    bits := make([]byte, r.UploadSize+1)
    remainder := d.Buffered()
    n, err := remainder.Read(bits)
    if err != nil && err != io.EOF {
        return r, bits, err
    }
    fmt.Println("remainder num: ", n)
    buffer.Write(bits[0:n])
    // Set the timeout according to the actual situation
    _ = conn.SetReadDeadline(time.Now().Add(20 * time.Second))
    for {
        //bs := make([]byte, r.UploadSize-int64(n+rn))
        nu, err := conn.Read(bits)
        if err != nil && err != io.EOF {
            return r, buffer.Bytes(), err
        }
        if err != io.EOF && n < int(r.UploadSize+1) {
            buffer.Write(bits[0:nu])
            n += nu
            if n >= int(r.UploadSize) {
                // upStream receiving completed
                break
            }
            continue
        }
    }
    // the JSON encoder on the client side seems to write an extra byte, so trim that off the front.
    return r, buffer.Bytes()[1:], nil
}
return r, buffer.Bytes(), nil

} `

didn’t this commit solve the problem?

Yes, it helps.

It seems #17495 will be merged in next release v1.9.0.

For us not being able to restore databases reliably was the final straw to decide to move away from influx. Lots of valuable business data could have been lost because of this

Same here. We are moving away from influx which is much behind any other competitor.

my influxdb version is: 1.7.6 I found the problem in file ”influxdata/influxdb/services/snapshotter/service.go“: `func (s *Service) readRequest(conn net.Conn) (Request, []byte, error) { var r Request d := json.NewDecoder(conn)

if err := d.Decode(&r); err != nil {
	return r, nil, err
}

bits := make([]byte, r.UploadSize+1)

if r.UploadSize > 0 {

	remainder := d.Buffered()

	n, err := remainder.Read(bits)
	if err != nil && err != io.EOF {
		return r, bits, err
	}

	// it is a bit random but sometimes the Json decoder will consume all the bytes and sometimes
	// it will leave a few behind.
	if err != io.EOF && n < int(r.UploadSize+1) {
		_, err = conn.Read(bits[n:])
	}

	if err != nil && err != io.EOF {
		return r, bits, err
	}
	// the JSON encoder on the client side seems to write an extra byte, so trim that off the front.
	return r, bits[1:], nil
}

return r, bits, nil

}` This func is to read the contents of the metadata file from the TCP connection. Because the file is too large, it needs to be sent several times, but it only receives two times at most when it is received. That is to say, when the metadata file is too large, it can not receive the complete content. So it should be improved here to fully receive the data sent by the client.

I have modified and compiled this part of code in my environment. After modification, there will be no such problem as “proto: meta.data: illegal tag 0 (wire type 0)”. The restore command is executed successfully.

This should be fixed by a combination of #21991 (in 1.8.9) and #17495 (in 1.8.10).

I was able to duplicate with https://github.com/influxdata/influxdb/issues/9968#issue-331908486 on some tries with v1.8.0

I was not able to duplicate on latest 1.8 including https://github.com/influxdata/influxdb/pull/22427 (coming in 1.8.10). Ran a script to run the repro 20x. Will close this when the 1.8 backport for #22427 closes.

It seems #17495 will be merged in next release v1.9.0.

https://github.com/influxdata/influxdb/blob/b26a2f7a0e41349938cec592a2abac4d93c9ab1c/CHANGELOG.md #17495: fix(snapshotter): properly read payload

I found that, restore on real machine (laptop) works, but not on any server with virtual disk. I tried with VPS from Digital Ocean, Azure and some Vietnamese provider (vHost, Vinahost, VCCloud, Vietnix). I also tried bare metal server from Scaleway, which comes with network disk. All failed to restore InfluxDB database (portable mode).

Log from client:

$ influxd restore -portable -db daothanh /tmp/db/ts_daothanh
2019/03/08 08:12:44 error updating meta: updating metadata on influxd service failed: err=read tcp 127.0.0.1:54174->127.0.0.1:8088: read: connection reset by peer, n=16
restore: updating metadata on influxd service failed: err=read tcp 127.0.0.1:54174->127.0.0.1:8088: read: connection reset by peer, n=16

Log from server:

$ journalctl -u influxdb

...
Mar 08 08:12:44 db-ams influxd[6119]: ts=2019-03-08T08:12:44.044335Z lvl=info msg="failed to decode meta: proto: meta.ShardGroupInfo: illegal ta
...

We just ran into this issues. It would be nice if the fix #17495 gets backported to 1.8.x

Again the fix is not included in 1.8.3 (Sep).

While influxdata doesn’t care, here is all-in-one Dockerfile to build a new release from master-1.x:

FROM golang:1.15.5-alpine3.12 as builder

RUN set -ex && \
    apk update && \
    apk add ca-certificates git bash gcc musl-dev && \
    git config --global http.https://gopkg.in.followRedirects true && \
    git clone --depth 1 --branch master-1.x https://github.com/influxdata/influxdb.git /opt/ && \
    cd /opt && \
    go build ./cmd/influxd && \
    chmod +x influxd

RUN git clone --depth 1 --branch master https://github.com/influxdata/influxdata-docker.git /opt2


FROM alpine:3.12

RUN echo 'hosts: files dns' >> /etc/nsswitch.conf
RUN set -ex && \
    apk add --no-cache tzdata bash ca-certificates && \
    update-ca-certificates

COPY --from=builder /opt/influxd /usr/bin/influxd
COPY --from=builder /opt2/influxdb/1.8/alpine/influxdb.conf /etc/influxdb/influxdb.conf

EXPOSE 8086

VOLUME /var/lib/influxdb

COPY --from=builder /opt2/influxdb/1.8/alpine/entrypoint.sh /entrypoint.sh
COPY --from=builder /opt2/influxdb/1.8/alpine/init-influxdb.sh /init-influxdb.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["influxd"]

docker build -t yourimage:1.8.x .