adb_shell: ADB Push fails: "Unable to unpack ADB command"

Hi there,

I was trying to push some files to the newest Android Emulator (stable v.30.2.5 x86_64) but the push command fails after sending ‘983040’ bytes with a “Unable to unpack ADB command” (see Log 1). In another attempt I tried to send a file which is smaller than the ‘983040’ bytes but the same error occurs (see Log 2) and from the progress log it seems like too many bytes were sent.

From the exception I can see that the error is raised in adb_message.py on Line 122 in the unpack method. Could it be that there is some new adb message which is not yet supported or something like that? Or has anyone an idea what’s causing this error?

Thx for any help

Tom

Log 1

worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: b'CNXN\x00\x00\x00\x01\x00\x00\x10\x00\x13\x00\x00\x00a\x05\x00\x00\xbc\xb1\xa7\xb1'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: b'host::eb22059493bd\x00'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_read(24): b'CNXN\x01\x00\x00\x01\x00\x00\x10\x00\x10\x01\x00\x00\xa1f\x00\x00\xbc\xb1\xa7\xb1'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_read(272): b'device::ro.product.name=sdk_gphone_x86_64_arm64;ro.product.model=sdk_gphone_x86_64_arm64;ro.product.device=generic_x86_64_arm64;features=sendrecv_v2_brotli,remount_shell,sendrecv_v2,abb_exec,fixed_push_mkdir,fixed_push_symlink_timestamp,abb,shell_v2,cmd,ls_v2,apex,stat_v2'
worker-low           | 2020-11-16 07:06:51 INFO     ADB device xxx:5555 is connected: True
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: b'OPEN\x01\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\xfe\x01\x00\x00\xb0\xaf\xba\xb1'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: b'root:\x00'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_read(24): b'OKAYj\x06\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_read(24): b'WRTEj\x06\x00\x00\x01\x00\x00\x00 \x00\x00\x00\x8c\x0b\x00\x00\xa8\xad\xab\xba'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_read(32): b'adbd is already running as root\n'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: b'OKAY\x01\x00\x00\x00j\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_read(24): b'CLSEj\x06\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: b'CLSE\x01\x00\x00\x00j\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
worker-low           | 2020-11-16 07:06:51 INFO     local_path: ./file_store/00_file_storage/libs/frida-server-14.0.8-android-x86_64
worker-low           | 2020-11-16 07:06:51 INFO     device_path: /data/local/tmp/frida-server-14.0.8-android-x86_64
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: b'OPEN\x01\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\xf7\x01\x00\x00\xb0\xaf\xba\xb1'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: b'sync:\x00'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_read(24): b'CLSEj\x06\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_read(24): b'OKAYk\x06\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:65536 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:131072 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:196608 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:262144 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:327680 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:393216 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:458752 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:524288 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:589824 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:655360 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:720896 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:786432 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:851968 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:917504 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-x86_64 bytes_written:983040 total_bytes:29902472
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: b'WRTE\x01\x00\x00\x00k\x06\x00\x00\xb8\x00\x0f\x00\xba\xa5x\x05\xa8\xad\xab\xba'
worker-low           | 2020-11-16 07:06:51 DEBUG    bulk_write: bytearray(b'SEND8\x00\x00\x00/data/local/tmp/frida-server-14.0.8-android-x86_64,33272DAT...<ENDLESS NUMBER OF HEX BYTES>...\x00\x00\x00\x89t$0PW\xe8\xf1\xef\xff\xff\x83\xc4\x0c\x8b|$\x08W\xb8$\x00\x00\x00PP\xe8\xdd\xef\xff')

And then after some time this error occurres:

worker-low           | 2020-11-16 07:11:54 DEBUG    bulk_read(24): b''
worker-low           | 07:11:54 Traceback (most recent call last):
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_message.py", line 120, in unpack
worker-low           |     cmd, arg0, arg1, data_length, data_checksum, _ = struct.unpack(constants.MESSAGE_FORMAT, message)
worker-low           | struct.error: unpack requires a buffer of 24 bytes
worker-low           | 
worker-low           | During handling of the above exception, another exception occurred:
worker-low           | 
worker-low           | Traceback (most recent call last):
worker-low           |   File "/usr/local/lib/python3.7/site-packages/rq/worker.py", line 936, in perform_job
worker-low           |     rv = job.perform()
worker-low           |   File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 684, in perform
worker-low           |     self._result = self._execute()
worker-low           |   File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 690, in _execute
worker-low           |     return self.func(*self.args, **self.kwargs)
worker-low           |   File "./scripts/dynamic_analysis/emulator_control/emulator_runner.py", line 21, in start_dynamic_analysis
worker-low           |     install_frida_server(adb_device)
worker-low           |   File "./scripts/dynamic_analysis/frida/frida_server_installer.py", line 17, in install_frida_server
worker-low           |     push_file(adb_device, local_path, device_path)
worker-low           |   File "./scripts/dynamic_analysis/emulator_control/adb/adb_wrapper.py", line 64, in push_file
worker-low           |     read_timeout_s=600)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 551, in push
worker-low           |     self._push(stream, _device_path, st_mode, mtime, progress_callback, adb_info, filesync_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 591, in _push
worker-low           |     self._filesync_send(constants.DATA, adb_info, filesync_info, data=data)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 1111, in _filesync_send
worker-low           |     self._filesync_flush(adb_info, filesync_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 963, in _filesync_flush
worker-low           |     self._write(filesync_info.send_buffer[:filesync_info.send_idx], adb_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 945, in _write
worker-low           |     self._read_until([constants.OKAY], adb_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 819, in _read_until
worker-low           |     cmd, remote_id2, local_id2, data = self._read(expected_cmds, adb_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 754, in _read
worker-low           |     cmd, arg0, arg1, data_length, data_checksum = unpack(msg)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_message.py", line 122, in unpack
worker-low           |     raise ValueError('Unable to unpack ADB command. (length={})'.format(len(message)), constants.MESSAGE_FORMAT, message, e)
worker-low           | ValueError: ('Unable to unpack ADB command. (length=0)', b'<6I', b'', error('unpack requires a buffer of 24 bytes'))
worker-low           | Traceback (most recent call last):
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_message.py", line 120, in unpack
worker-low           |     cmd, arg0, arg1, data_length, data_checksum, _ = struct.unpack(constants.MESSAGE_FORMAT, message)
worker-low           | struct.error: unpack requires a buffer of 24 bytes

worker-low           | Traceback (most recent call last):
worker-low           |   File "/usr/local/lib/python3.7/site-packages/rq/worker.py", line 936, in perform_job
worker-low           |     rv = job.perform()
worker-low           |   File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 684, in perform
worker-low           |     self._result = self._execute()
worker-low           |   File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 690, in _execute
worker-low           |     return self.func(*self.args, **self.kwargs)
worker-low           |   File "./scripts/dynamic_analysis/emulator_control/emulator_runner.py", line 21, in start_dynamic_analysis
worker-low           |     install_frida_server(adb_device)
worker-low           |   File "./scripts/dynamic_analysis/frida/frida_server_installer.py", line 17, in install_frida_server
worker-low           |     push_file(adb_device, local_path, device_path)
worker-low           |   File "./scripts/dynamic_analysis/emulator_control/adb/adb_wrapper.py", line 64, in push_file
worker-low           |     read_timeout_s=600)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 551, in push
worker-low           |     self._push(stream, _device_path, st_mode, mtime, progress_callback, adb_info, filesync_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 591, in _push
worker-low           |     self._filesync_send(constants.DATA, adb_info, filesync_info, data=data)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 1111, in _filesync_send
worker-low           |     self._filesync_flush(adb_info, filesync_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 963, in _filesync_flush
worker-low           |     self._write(filesync_info.send_buffer[:filesync_info.send_idx], adb_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 945, in _write
worker-low           |     self._read_until([constants.OKAY], adb_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 819, in _read_until
worker-low           |     cmd, remote_id2, local_id2, data = self._read(expected_cmds, adb_info)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_device.py", line 754, in _read
worker-low           |     cmd, arg0, arg1, data_length, data_checksum = unpack(msg)
worker-low           |   File "/usr/local/lib/python3.7/site-packages/adb_shell/adb_message.py", line 122, in unpack
worker-low           |     raise ValueError('Unable to unpack ADB command. (length={})'.format(len(message)), constants.MESSAGE_FORMAT, message, e)
worker-low           | ValueError: ('Unable to unpack ADB command. (length=0)', b'<6I', b'', error('unpack requires a buffer of 24 bytes'))

Log 2

worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_write: b'CNXN\x00\x00\x00\x01\x00\x00\x10\x00\x13\x00\x00\x00l\x05\x00\x00\xbc\xb1\xa7\xb1'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_write: b'host::ff918e63b615\x00'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_read(24): b'CNXN\x01\x00\x00\x01\x00\x00\x10\x00\x10\x01\x00\x00\xa1f\x00\x00\xbc\xb1\xa7\xb1'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_read(272): b'device::ro.product.name=sdk_gphone_x86_64_arm64;ro.product.model=sdk_gphone_x86_64_arm64;ro.product.device=generic_x86_64_arm64;features=sendrecv_v2_brotli,remount_shell,sendrecv_v2,abb_exec,fixed_push_mkdir,fixed_push_symlink_timestamp,abb,shell_v2,cmd,ls_v2,apex,stat_v2'
worker-low           | 2020-11-16 07:19:44 INFO     ADB device xxxx:5555 is connected: True
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_write: b'OPEN\x01\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\xfe\x01\x00\x00\xb0\xaf\xba\xb1'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_write: b'root:\x00'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_read(24): b'OKAY\x86\x06\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_read(24): b'WRTE\x86\x06\x00\x00\x01\x00\x00\x00 \x00\x00\x00\x8c\x0b\x00\x00\xa8\xad\xab\xba'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_read(32): b'adbd is already running as root\n'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_write: b'OKAY\x01\x00\x00\x00\x86\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_read(24): b'CLSE\x86\x06\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_write: b'CLSE\x01\x00\x00\x00\x86\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
worker-low           | 2020-11-16 07:19:44 INFO     local_path: ./file_store/00_file_storage/libs/frida-server-14.0.8-android-arm
worker-low           | 2020-11-16 07:19:44 INFO     device_path: /data/local/tmp/frida-server-14.0.8-android-arm
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_write: b'OPEN\x01\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\xf7\x01\x00\x00\xb0\xaf\xba\xb1'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_write: b'sync:\x00'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_read(24): b'CLSE\x86\x06\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_read(24): b'OKAY\x87\x06\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:65536 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:131072 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:196608 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:262144 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:327680 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:393216 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:458752 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:524288 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:589824 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:655360 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:720896 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:786432 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:851968 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:917504 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 INFO     ADB Push-Progress: /data/local/tmp/frida-server-14.0.8-android-arm bytes_written:983040 total_bytes:8814864
worker-low           | 2020-11-16 07:19:44 DEBUG    bulk_write: b'WRTE\x01\x00\x00\x00\x87\x06\x00\x00\xb5\x00\x0f\x00\xdd\x15l\x05\xa8\xad\xab\xba'

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (12 by maintainers)

Most upvoted comments

@atti92 thanks for looking into this!

There do seem to be some issues with Android emulators (see https://github.com/JeffLIrion/adb_shell/issues/81). If someone wants to contribute fixes that would be welcome, but emulators are not the intended use case for this package.

That said, I’m glad that we found a simple workaround for this issue. I don’t know why the emulator says the max data is 1048576 = 1 MB, but that doesn’t work. And I don’t know why using 256 KB doesn’t work, either.

JeffLIrion on Dec 1, 2020

I created a branch and a draft pull request for investigating this issue. Here’s a comment that I left: https://github.com/JeffLIrion/adb_shell/pull/134#discussion_r525769414

JeffLIrion on Nov 18, 2020