aws-google-auth: id-challenge HTML element does not always contain a JSON value

Sometimes aws-google-auth fails with the following exception:

  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/__init__.py", line 197, in process_auth
    google_client.do_login()
  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/google.py", line 259, in do_login
    sess = self.handle_sk(sess)
  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/google.py", line 371, in handle_sk
    raise e
  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/google.py", line 366, in handle_sk
    challenges = json.loads(challenges_txt)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I think the issue is because of the challenge HTML form having changed. During a failed run, it looks like this:

            <div class="LJtPoc" jsname="Ki8mld" jscontroller="qNel6e" jsaction="rcuQ6b:WYd">
                <form method="POST" id="challenge" action="/signin/challenge/sk/6" jsname="rzWj5" jscontroller="HNBfvc" jsaction="submit:zbvklb" jsshadow>
                    <content>
                        <input name="challengeId" type="hidden" id="challengeId" value="6">
                        <input name="challengeType" type="hidden" id="challengeType" value="2">
                        <input name="continue" type="hidden" value="https://accounts.google.com/o/saml2/initsso?idpid=REDACTED&amp;spid=REDACTED&amp;forceauthn=false&amp;from_login=1&amp;as=REDACTED">
                        <input name="scc" type="hidden" value="1">
                        <input name="sarp" type="hidden" value="1">
                        <input name="checkedDomains" type="hidden" value="youtube">
                        <input name="pstMsg" type="hidden" value="0">
                        <input name="TL" type="hidden" value="REDACTED">
                        <input type="hidden" name="gxf" id="gxf" value="REDACTED">
                        <div jsname="KrwUDc"><img jsname="TqVmm" class="JC07Dd" src="//ssl.gstatic.com/accounts/marc/gnubby_icon.png" alt="">
                            <div jsname="kwHene"></div>
                            <div class="EGmPD" jsname="BCqkPb">Insert your security key</div>
                            <div class="VnJmLc" jsname="NhJ5Dd">If your security key has a button, tap it.
                                <br>If it doesn't, remove and re-insert it.</div>
                            <div>
                                <input type="hidden" name="id-challenge" jsname="wCVnAe" value="REDACTED_STRING_NOT_JSON">
                                <input type="hidden" name="id-assertion" jsname="n6knUb">
                            </div>
                            <div jsname="C0oDBd" data-challenge-ui="%.@.null,null,&quot;TWO_STEP_VERIFICATION&quot;,6,null,&quot;SEND_SUCCESS&quot;,null,null,2,1,true,true,true,null,null,null,&quot;REDACTED&quot;,&quot;https://lh3.googleusercontent.com/-XdUIqdMkCWA/AAAAAAAAAAI/AAAAAAAAAAA/ACHi3re6XM1OFpqpdEsscOf-CRg8KCNC_g/mo/photo.jpg&quot;,null,null,null,null,[]
,{&quot;1010&quot;:[2,false]
,&quot;5010&quot;:[null,null,null,&quot;https://accounts.google.com/signin/challenge/sk/6&quot;,null,[&quot;google.com&quot;,&quot;REDACTED_STRING&quot;,[[2,&quot;REDACTED_STRING&quot;]
]
,&quot;{\&quot;appid\&quot;:\&quot;https://www.gstatic.com/securitykey/origins.json\&quot;}&quot;]
]
}]
"></div>
                            <div class="ARshqb">
                                <input type="checkbox" name="TrustDevice" id="trustDevice" class="aCOJmf" checked><span>Don&#39;t ask again on this computer</span>
                                <div class="Bfmfyc" role="tooltip">
                                    <div class="x7qQqf"></div>
                                    <div class="hzC8Lb">For your convenience, keep this checked. On shared devices, additional precautions are recommended. <a href="https://support.google.com/accounts/?p=securesignin&amp;hl=en" target="_blank">Learn more</a></div>
                                </div>
                            </div>
                        </div>
                    </content>
                </form>
            </div>

Based on this it looks like it comes down to the value string of id-challenge no longer being a stringified JSON object, but instead a “random” string that looks like "...hUTYdnvUG6M25UzFVz...". I poked around but it wasn’t clear to me what the right way is to fix this, especially because

<div jsname="C0oDBd" data-challenge-ui="%.@.null,null,&quot;TWO_STEP_VERIFICATION&quot;,6,null,&quot;SEND_SUCCESS&quot;,null,null,2,1,true,true,true,null,null,null,&quot;fabian@ruist.com&quot;,&quot;https://lh3.googleusercontent.com/-XdUIqdMkCWA/AAAAAAAAAAI/AAAAAAAAAAA/ACHi3re6XM1OFpqpdEsscOf-CRg8KCNC_g/mo/photo.jpg&quot;,null,null,null,null,[]
,{&quot;1010&quot;:[2,false]
,&quot;5010&quot;:[null,null,null,&quot;https://accounts.google.com/signin/challenge/sk/6&quot;,null,[&quot;google.com&quot;,&quot;REDACTED_STRING&quot;,[[2,&quot;REDACTED_STRING&quot;]
]
,&quot;{\&quot;appid\&quot;:\&quot;https://www.gstatic.com/securitykey/origins.json\&quot;}&quot;]
]
}]
"></div>

looks like it has the relevant data but also looks like it isn’t simply JSON…

This has started happening approximately 5 days ago.

In case it is relevant, saving the html and opening it in a browser (Chrome) renders this page: Screen Shot 2019-03-11 at 1 59 24 PM

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 22
Comments: 31 (4 by maintainers)

Most upvoted comments

Set up some kind of campaign / funding avenue and I’m sure a bunch of us will have no trouble donating the money required to buy a YubiKey.

sinkr on Mar 28, 2019

same here, yubikey as well.

maxleonca on Mar 14, 2019

On Mar 28, 2019, at 3:26 PM, Robert Sink notifications@github.com wrote:

Set up some kind of campaign / funding avenue and I’m sure a bunch of us will have no trouble donating the money required to buy a YubiKey.

If https://www.yubico.com/product/security-key-by-yubico/#security-key will work and you think you can do it, make an amazon wishlist with it and I’ll buy it for you right now. $20 is definitely worth it to me.

davidrjonas on Mar 28, 2019

It looks repeatable enough that we should be able to work something out.

The challenge with this whole tool is that it is reverse engineered from behavior in the wild, and google can (and has) change the format of these pages as they see fit.

It looks like this is another one of those cases.

If someone with access to one of the hardware keys that are having problems has the capability to help troubleshoot and reverse-engineer what is happening, I’d willingly accept (and help with) updates to the tool.

I don’t have one of these keys at the moment, so not something I can simulate, and hence resolve.

If you have ways in which I can replicate this on my account I’m willing to take on the task of trying to work out a way around this.

Just note, we can’t promise anything - there have been a few things - like Captcha problems of the past - that required considerable time to try and work out, and as quickly as they were issues they seemed to go away again. There is no real view into the complexity that is the Google SSO machine from the outside.

stevemac007 on Mar 28, 2019