balena-cli: balena os download hangs on 83%, errors out with unexpected end of file at Zlib.zlibOnError

I am running balenaCLI v12.37.0 (Installed globally through NPM) inside a container with fincm3-alpine-node as the base image. On our testing runs, we see the balena os download command flaking out about 8 out of 10 times.

Expected Behavior

The image requested should download without errors.

Actual Behavior

On running the command inside the container, balena os download raspberrypi3 -o /tmp/raspberrypi3.img --version v2.58.3+rev1.prod the download proceeds normally and gets stuck strangely at 83% (Every time). The ETA time does keep increasing after eventually failing with the error mentioned below.

bash-5.0# balena os download raspberrypi3 -o /tmp/raspberrypi3.img --version v2.58.3+rev1.prod
Getting device operating system for raspberrypi3
Downloading Device OS 2.58.3+rev1.prod [========================] 100% eta 5s  
Error: unexpected end of file
    at Zlib.zlibOnError [as onerror] (zlib.js:180:17)

We did some troubleshooting of our own. Tried different connections, cli versions (previously v12.32), and base images (previously it was debian) balena os download works perfectly on x86 (my system, popOS 18.04) but not sure why it is having troubles inside a container. Anything you suggest, I be most happy to try out. We have tried using --debug flag as well but no change in output.

Steps to Reproduce the Problem

This is the most important and helpful part of a bug report. If we cannot reproduce the problem, it is difficult to tell what the fix should be, or whether code changes have fixed it.

Dockerfile.template and docker-compose.yml here for the container
Deployed on a provisioned balenaFin v1.1.1 with balena push <app_name>
Run the command on the container, balena os download raspberrypi3 -o /tmp/raspberrypi3.img --version v2.58.3+rev1.prod

Specifications

balena CLI version: balena-cli version “12.37.0”, Node.js version “12.19.1”
**Cloud backend: balenaCloud
Operating system version: Alpine: latest
32/64 bit OS and processor: armv7hf (balenaFin)
Install method: npm
If npm install, Node.js and npm version: Node v12.19.1 and NPM v6.14.8

Please do comment if other folks face this problem or are able to reproduce this issue for the team to start working on diagnosing this issue. Thanks!

About this issue

Original URL
State: open
Created 3 years ago
Comments: 15 (14 by maintainers)

Commits related to this issue

Update balena sdk Fixes [EOF zlib issue](https://github.com/balena-io/balena-cli/issues/2152 ) per [@klutchell 's balena-request fix](https://github.com/balena-io-modules/balena-request/pull/156). ... — committed to balena-io-modules/balena-image-manager by zwhitchcox 3 years ago
Update balena sdk Fixes [EOF zlib issue](https://github.com/balena-io/balena-cli/issues/2152 ) per [@klutchell 's balena-request fix](https://github.com/balena-io-modules/balena-request/pull/156). S... — committed to balena-io-modules/balena-image-manager by zwhitchcox 3 years ago

Most upvoted comments

I can reproduce this with balena-image-manager but not with balena-sdk directly, so at least I’ve narrowed it down some more.

const { getSdk } = require('balena-sdk');

const balena = getSdk({
    apiUrl: "https://api.balena-cloud.com/",
    dataDirectory: "/opt/local/balena"
});

const fs = require('fs');
const manager = require('balena-image-manager');

// works
// balena.models.os.download('raspberrypi3').then(function(stream) {
//     stream.pipe(fs.createWriteStream('balena.img'));
// });

// fails
manager.get('raspberrypi3', 'default').then(function(stream) {
    stream.pipe(fs.createWriteStream('balena.img'));
});

Error: unexpected end of file
    at Zlib.zlibOnError [as onerror] (zlib.js:182:17)
Emitted 'error' event on Gunzip instance at:
    at Gunzip.onerror (/home/pi/sdk-test/node_modules/readable-stream/lib/_stream_readable.js:640:52)
    at Gunzip.emit (events.js:314:20)
    at Zlib.zlibOnError [as onerror] (zlib.js:185:8) {
  errno: -5,
  code: 'Z_BUF_ERROR'
}

klutchell on Jul 13, 2021

I’m running into this with a fresh install of CLI v12.44.23 on Raspberry Pi OS (no container). Will see if I can find a root cause.

pi@raspberrypi:~ $ node -v
v12.22.3
pi@raspberrypi:~ $ npm -v
6.14.13
pi@raspberrypi:~ $ docker version
Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:57:27 2021
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:55:25 2021
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux
pi@raspberrypi:~ $ balena os download raspberry-pi -o balena.img --debug
[debug] new argv=[/home/pi/.nvm/versions/node/v12.22.3/bin/node,/home/pi/.nvm/versions/node/v12.22.3/bin/balena,os:download,raspberry-pi,-o,balena.img] length=6
Getting device operating system for raspberry-pi
OS version not specified: using latest stable version
Downloading Device OS [========================] 100% eta 26s 
Error: unexpected end of file
    at Zlib.zlibOnError [as onerror] (zlib.js:182:17)

klutchell on Jul 6, 2021