om: upload-product is not efficient

Noticed this yesterday, when I ran upload-product --product pas.pivotal, om spend a long time saying processing product. Eventually found that it writes the content payload to the temp disk. This makes sense because we never want to load a 9G file into the memory, but it is not efficient. And after the upload process finishes, it will keep the payload in the /tmp folder. Although eventually it will be cleaned up by the operating system, it occupies a lot of the space.

So I tried to find is there a way to multipart upload big file without consuming much disk or memory, but failed to find a good library to do that. Then I just tried to write a library, with the aim to plugin into om as easy as possible.

Then I come up with this: https://github.com/fredwangwang/formcontent/

It loads the content of the file from the disk on demand, so it doesn’t eat additional disk or consume much memory.

And I write there are two versions in the repository, one is threaded (master branch), another is non-threaded (non-threaded branch). The non-threaded branch works better, but the code clearity is worse, as it basically implement a cheap state machine in a custom Read method.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 22 (22 by maintainers)

Commits related to this issue

move formcontent to om related to issue #196, I created the formcontent in my repo, now there is a request to move the code into a pivotal repo to remove the dependency on a personal repo. [#1598354... — committed to pivotal-cf/om by fredwangwang 6 years ago
move formcontent to om related to issue #196, I created the formcontent in my repo, now there is a request to move the code into a pivotal repo to remove the dependency on a personal repo. [#1598354... — committed to pivotal-cf/om by fredwangwang 6 years ago

Most upvoted comments

And dig /tmp folder, you would find several copy of ~1.9G file, which are generated by the old command executions

On Wed, Jul 25, 2018 at 12:56 David McClure notifications@github.com wrote:

Get ts:

dm $ brew install moreutils

How big is this thing? (1.9G)

dm $ ls -lh ~/Downloads/p-isolation-segment-2.3.0-build.182.pivotal -rw-r–r–@ 1 pivotal staff 1.9G Jul 25 11:32 /Users/pivotal/Downloads/p-isolation-segment-2.3.0-build.182.pivotal

Before (8s)

dm $ om -k upload-product --product ~/Downloads/p-isolation-segment-2.3.0-build.182.pivotal | ts Jul 25 11:49:04 processing product Jul 25 11:49:12 beginning product upload to Ops Manager …

After (0s)

dm $ om -k upload-product --product ~/Downloads/p-isolation-segment-2.3.0-build.182.pivotal | ts Jul 25 11:51:38 processing product Jul 25 11:51:38 beginning product upload to Ops Manager …

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pivotal-cf/om/issues/196#issuecomment-407858893, or mute the thread https://github.com/notifications/unsubscribe-auth/AUr56ZooYMdH-q4VUZbZZGVRnIbDrWHBks5uKL9rgaJpZM4U70P9 .

fredwangwang on Jul 25, 2018

yep, the improvement would not speed up network connection (I hope I can…), its about reducing the processing product time and the temp disk occupied by the command

fredwangwang on Jul 25, 2018

Merged #211 to fix this. Thanks for working through this tricky issue @fredwangwang!

ljfranklin on Jul 23, 2018