warpgate: Ansible connections using warpgate are slow

I just noticed that when I use Ansible with Warpgate the playbook execution time is slower.

Test setup

I run a playbook against the host where warpgate is installed via systemd. For the authentication with Ansible I used the password login for warpgate. For the test without warpgate I used a ssh private key to connect to the target host directly by Ansible. I then run the Ansible playbook with the time command to measure the time it took to complete.

Results:

warpgate (ssh with password auth):

real    0m44.063s
user    0m3.355s
sys     0m0.567s

warpgate (ssh with key auth):

real    0m28.975s
user    0m2.655s
sys     0m0.414s

direct:

real    0m4.923s
user    0m1.212s
sys     0m0.338s

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 30 (12 by maintainers)

Commits related to this issue

ssh: remove auth delay after initial "none" auth - #459 — committed to warp-tech/warpgate by Eugeny 2 years ago
fixed missing CHANNEL_CLOSE messages - #459 — committed to warp-tech/warpgate by Eugeny 2 years ago

Most upvoted comments

This is great - thank you so much for testing!

Eugeny on Nov 21, 2022

@Eugeny I just tested the fix and it works perfectly, I changed back the MaxSessions back to 10 and let a playbook run and one warpgate sessions was used with more then 10 recordings 👍

ntimo on Nov 20, 2022

I’ve just found out that Warpgate didn’t correctly close SSH channels - leading to MaxSessions getting exhausted - now the whole playbook should run through the same connection

Eugeny on Nov 20, 2022

@Eugeny I just tested it with the test script I had written and the times for direct ssh and warpgate are basically identical.

Just a few samples: SSH:

00:02.887 **
00:02.292 **
00:02.257 **
00:02.181 **
00:02.163 **
00:02.234 **
00:02.226 **
00:02.047 **

Warpgate:

00:03.016 **
00:02.559 **
00:02.678 **
00:02.517 **
00:02.550 **
00:02.594 **
00:02.567 **

ntimo on Nov 21, 2022

… but I’ve been experiencing an issue with slowness as well. … Should I post my findings …

Providing facts is indeed a good thing. IMNSHO: Go for a fresh issue with a title like “Propagation delay in comparison with SSH jumphost”.

I went to actually collect the data and discovered that speed is not the issue (at least not a primary issue). It seems like the connection is being dropped for some reason. I give details here: https://github.com/warp-tech/warpgate/issues/479 but I believe the issue can be reproduced simply by transferring a file of non-trivial size using scp (unless the issue is somehow unique to my setup). Funnily enough, rsync works just fine and is exactly as fast between a direct connection, a jumphost, and a Warpgate host.

Abraxos on Nov 14, 2022

I just run a test by running the playbook 50 times each direct / using a jumphost / using warpgate.

Nice!

afbeelding

stappersg on Nov 13, 2022

I actually found something out, I tested ansible in combination with mitogen https://mitogen.networkgenomics.com/ansible_detailed.html which only uses 1 persistent ssh connection for the entire playbook run against a given host (this is also visable in warpate since only one sessions is logged with one ssh recording). Where Ansible opens a new connection per task which is probably what makes it so slow. Using mitogen I had really really fast ansible playbook times. But I will post more details later, first need to do more tests. I also want to try to use the ssh ControlMaster feature. The only problem is that mitogen is kind of a patch for Ansible so it always takes time for them to catch up with upstream.

ntimo on Nov 7, 2022