zaproxy: HTTP/NTLM reauthentication no longer works in Active Scan

Describe the bug

After upgrading from ZAP 2.11.1 to 2.12.0, running an Active Scan against a node in the Site tree that requires Digest or NTLM authentication stopped working. The console log shows “java.lang.IllegalStateException: AuthScheme is null”:

java.lang.IllegalStateException: AuthScheme is null
        at org.apache.hc.core5.util.Asserts.notNull(Asserts.java:56) ~[?:?]
        at org.apache.hc.client5.http.impl.auth.HttpAuthenticator.updateAuthState(HttpAuthenticator.java:216) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.needAuthentication(ProtocolExec.java:294) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:207) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec.execute(ZapHttpRequestRetryExec.java:81) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ZapInternalHttpClient.doExecute(ZapInternalHttpClient.java:158) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl0(HttpSenderApache.java:394) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:297) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:103) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAuthenticated(BaseHttpSender.java:298) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendNoRedirections(BaseHttpSender.java:266) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.send(BaseHttpSender.java:222) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:193) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:57) ~[?:?]
        at org.parosproxy.paros.network.HttpSender.sendAndReceive(HttpSender.java:478) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.sendAndReceive(Analyser.java:528) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.analyse(Analyser.java:190) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.inOrderAnalyse(Analyser.java:381) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.inOrderAnalyse(Analyser.java:384) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.start(Analyser.java:136) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.HostProcess.run(HostProcess.java:372) ~[zap-2.12.0.jar:2.12.0]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Steps to reproduce the behavior

Install ZAP “Core Cross Platform Package”.
Start zap.sh. If absent, install the “Spider” add-on then restart.
Double-click “Default Context”
Under “Include in Context”, click “Add…”, type “.*httpbin.*”, then click “Add”.
Under “Authentication”, change “Manual Authentication” to “HTTP/NTLM Authentication”. Set “Hostname” to “httpbin.org”.
Under “Users”, click “Add…”, set Username to “user” and password to “passwd” then click Add.
Under “Forced User”, make sure “user” is selected then click OK to close the dialog.
Click the “Forced User Mode” toolbar button to enable it.
On the Quick Start tab, click Automated Scan. Set “URL to attack” to “http://httpbin.org/digest-auth/auth/user/passwd”. Make sure “Use traditional spider” is selected then click Attack. Wait for the spider to complete.
In the Site tree, click the “GET:passwd” node. Verify the Request and Response tabs show the authentication having succeeded.
In the Site tree, right-click the “GET:passwd” node then click “Attack > Active Scan…”. Select “user” then click “Start Scan”.

Expected behavior

At least one successful 200 OK traffic message should be shown on the Active Scan tab. This worked fine in 2.11.1. In 2.12.0, active scan rules can’t get past the “AuthScheme is null” IllegalStateException.

Software versions

OWASP ZAP Version: 2.12.0

Installed Add-ons: [[id=bruteforce, version=12.0.0], [id=callhome, version=0.5.0], [id=commonlib, version=1.11.0], [id=database, version=0.1.0], [id=diff, version=12.0.0], [id=gettingStarted, version=14.0.0], [id=help, version=15.0.0], [id=invoke, version=12.0.0], [id=network, version=0.3.0], [id=onlineMenu, version=10.0.0], [id=pscanrules, version=44.0.0], [id=quickstart, version=35.0.0], [id=reports, version=0.16.0], [id=reveal, version=5.0.0], [id=spider, version=0.1.0], [id=tips, version=10.0.0]]

Screenshots

No response

Errors from the zap.log file

java.lang.IllegalStateException: AuthScheme is null
        at org.apache.hc.core5.util.Asserts.notNull(Asserts.java:56) ~[?:?]
        at org.apache.hc.client5.http.impl.auth.HttpAuthenticator.updateAuthState(HttpAuthenticator.java:216) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.needAuthentication(ProtocolExec.java:294) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:207) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec.execute(ZapHttpRequestRetryExec.java:81) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ZapInternalHttpClient.doExecute(ZapInternalHttpClient.java:158) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl0(HttpSenderApache.java:394) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:297) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:103) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAuthenticated(BaseHttpSender.java:298) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendNoRedirections(BaseHttpSender.java:266) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.send(BaseHttpSender.java:222) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:193) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:57) ~[?:?]
        at org.parosproxy.paros.network.HttpSender.sendAndReceive(HttpSender.java:478) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.sendAndReceive(Analyser.java:528) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.analyse(Analyser.java:190) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.inOrderAnalyse(Analyser.java:381) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.inOrderAnalyse(Analyser.java:384) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.start(Analyser.java:136) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.HostProcess.run(HostProcess.java:372) ~[zap-2.12.0.jar:2.12.0]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Additional context

In the above reproduction steps, I demonstrate the issue with some public HTTP Digest endpoint. However, I am running into this same “AuthScheme is null” issue with various internal service endpoints within my organization including those that require NTLM authentication. Both Digest and NTLM authentication worked fine in ZAP 2.11.1.

Would you like to help fix this issue?

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 1
Comments: 22 (10 by maintainers)

Commits related to this issue

network: correct HTTP/NTLM reauthentication Try first authenticate solely with the Authorization header present, only if it fails retry with the credentials available. Fix zaproxy/zaproxy#7566. Sig... — committed to thc202/zap-extensions by thc202 2 years ago
network: correct HTTP/NTLM proxy reauthentication Retry the proxy authentication using the credentials available if it failed with the header already present. Part of zaproxy/zaproxy#7566. Signed-o... — committed to thc202/zap-extensions by thc202 2 years ago
network: correct HTTP/NTLM proxy reauthentication Retry the proxy authentication using the credentials available if it failed with the header already present. Part of zaproxy/zaproxy#7566. Signed-o... — committed to thc202/zap-extensions by thc202 2 years ago
network: fix (re)authentication Correct proxy reauthentication. Attempt authentication of TRACE requests. Part of zaproxy/zaproxy#7566. Signed-off-by: thc202 <thc202@gmail.com> — committed to thc202/zap-extensions by thc202 2 years ago
network: fix (re)authentication Correct proxy reauthentication. Attempt authentication of TRACE requests. Part of zaproxy/zaproxy#7566. Signed-off-by: thc202 <thc202@gmail.com> — committed to thc202/zap-extensions by thc202 2 years ago

Most upvoted comments

The referenced PR should address the issues reported

Yes, it does! Your commit solves the remainder of the problems I was running into. I verified after building a new network add-on from thc202:network/reauth-fixes. I now see all the expected traffic and alerts. I also verified that the ZAP log shows no more runtime exceptions for any of my scans.

benken-parasoft on Nov 8, 2022

Configuring a proxy server with NTLM authentication against Active Directory (for example) can be tricky to setup. So, if you have a patch or pull request that are you need help verifying then I am happy to pull it in and build a new network add-on locally and retest using the proxy server in my environment.

benken-parasoft on Nov 8, 2022

You can also build it yourself, just clone the zap-extensions repo and run ./gradlew :aO:network:assemble the add-on will be under zap-extensions/addOns/network/build/zapAddOn/bin/

I did this. The authentication problems went away! I also didn’t see any proxy-authentication issues after configuring an HTTP proxy. Thanks again for resolving the authentication issue so quickly!

However, I now ran into a different runtime exception from the network add-on that seems to be causing some rules to abort and not report any alerts. I just filed #7578. So, this new issue is now blocking me because it seems to have broken some rules.

benken-parasoft on Nov 4, 2022