DependencyCheck: NVD API returns transient 403 response with API key
Describe the bug When using an API key, the NVD API has started returning a transient 403 response. It occurs in the middle of a database update, so not a key configuration issue (and when retrying the key is used successfully). Some ODC database updates do complete, but this has occurred about half of the time since upgrading to v9.0.2.
This is related to https://github.com/jeremylong/DependencyCheck/issues/6180 and https://github.com/jeremylong/DependencyCheck/issues/6149, but is still occurring with the 9.0.2 CLI.
[INFO] Running: [/bin/sh -c /usr/share/dependency-check/bin/dependency-check.sh --updateonly --nvdApiKey "$(cat /kaniko/NVD_API_KEY)" --retireJsForceUpdate --hostedSuppressionsForceUpdate]
[INFO] Checking for updates
[INFO] NVD API has 231,966 records in this update
[INFO] Downloaded 10,000/231,966 (4%)
[INFO] Downloaded 20,000/231,966 (9%)
[INFO] Downloaded 30,000/231,966 (13%)
[INFO] Downloaded 40,000/231,966 (17%)
[INFO] Downloaded 50,000/231,966 (22%)
[INFO] Downloaded 60,000/231,966 (26%)
[INFO] Downloaded 70,000/231,966 (30%)
[INFO] Downloaded 80,000/231,966 (34%)
[INFO] Downloaded 90,000/231,966 (39%)
[INFO] Downloaded 100,000/231,966 (43%)
[INFO] Downloaded 110,000/231,966 (47%)
[INFO] Downloaded 120,000/231,966 (52%)
[INFO] Downloaded 130,000/231,966 (56%)
[INFO] Downloaded 140,000/231,966 (60%)
[INFO] Downloaded 150,000/231,966 (65%)
[INFO] Downloaded 160,000/231,966 (69%)
[INFO] Downloaded 170,000/231,966 (73%)
[INFO] Downloaded 180,000/231,966 (78%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:340)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:110)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427)
at org.owasp.dependencycheck.App.run(App.java:172)
at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:346)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:319)
... 6 common frames omitted
[ERROR] Failed to process CVE-2011-0074
org.owasp.dependencycheck.data.nvdcve.DatabaseException: Unable to retrieve id for new vulnerability for 'CVE-2011-0074'
at org.owasp.dependencycheck.data.nvdcve.CveDB.updateOrInsertVulnerability(CveDB.java:1054)
at org.owasp.dependencycheck.data.nvdcve.CveDB.updateVulnerability(CveDB.java:866)
at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call(NvdApiProcessor.java:87)
at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call(NvdApiProcessor.java:33)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "org.h2.mvstore.MVStoreException: Reading from file sun.nio.ch.FileChannelImpl@fec3929 failed at 43616274 (length -1), read 0, remaining 1024 [2.1.214/1]"; SQL statement:
DELETE FROM reference WHERE cveid = ? [50000-214]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:554)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:477)
at org.h2.message.DbException.get(DbException.java:212)
at org.h2.message.DbException.convert(DbException.java:395)
at org.h2.command.Command.executeUpdate(Command.java:264)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:209)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:169)
at org.owasp.dependencycheck.data.nvdcve.H2Functions.updateVulnerability(H2Functions.java:223)
at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.h2.schema.FunctionAlias$JavaMethod.execute(FunctionAlias.java:495)
at org.h2.schema.FunctionAlias$JavaMethod.getTableValue(FunctionAlias.java:363)
at org.h2.expression.function.table.JavaTableFunction.getValue(JavaTableFunction.java:34)
at org.h2.table.FunctionTable.getResult(FunctionTable.java:51)
at org.h2.index.VirtualConstructedTableIndex.find(VirtualConstructedTableIndex.java:38)
at org.h2.index.IndexCursor.find(IndexCursor.java:161)
at org.h2.table.TableFilter.next(TableFilter.java:394)
at org.h2.command.query.Select$LazyResultQueryFlat.fetchNextRow(Select.java:1832)
at org.h2.result.LazyResult.hasNext(LazyResult.java:78)
at org.h2.result.FetchedResult.next(FetchedResult.java:34)
at org.h2.command.query.Select.queryFlat(Select.java:728)
at org.h2.command.query.Select.queryWithoutCache(Select.java:833)
at org.h2.command.query.Query.queryWithoutCacheLazyCheck(Query.java:197)
at org.h2.command.query.Query.query(Query.java:512)
at org.h2.command.query.Query.query(Query.java:475)
at org.h2.command.CommandContainer.query(CommandContainer.java:251)
at org.h2.command.Command.executeQuery(Command.java:190)
at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:128)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:123)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:123)
at org.owasp.dependencycheck.data.nvdcve.CveDB.updateOrInsertVulnerability(CveDB.java:1049)
... 7 common frames omitted
Caused by: org.h2.mvstore.MVStoreException: Reading from file sun.nio.ch.FileChannelImpl@fec3929 failed at 43616274 (length -1), read 0, remaining 1024 [2.1.214/1]
at org.h2.mvstore.DataUtils.newMVStoreException(DataUtils.java:1004)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:470)
at org.h2.mvstore.FileStore.readFully(FileStore.java:98)
at org.h2.mvstore.Chunk.readBufferForPage(Chunk.java:422)
at org.h2.mvstore.MVStore.readPage(MVStore.java:2569)
at org.h2.mvstore.MVMap.readPage(MVMap.java:633)
at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1125)
at org.h2.mvstore.Page.get(Page.java:243)
at org.h2.mvstore.MVMap.get(MVMap.java:436)
at org.h2.mvstore.tx.TransactionMap.getFromSnapshot(TransactionMap.java:472)
at org.h2.mvstore.tx.TransactionMap.getFromSnapshot(TransactionMap.java:467)
at org.h2.mvstore.db.MVPrimaryIndex.getRow(MVPrimaryIndex.java:263)
at org.h2.mvstore.db.MVTable.getRow(MVTable.java:331)
at org.h2.mvstore.db.MVSecondaryIndex$MVStoreCursor.get(MVSecondaryIndex.java:421)
at org.h2.index.IndexCursor.get(IndexCursor.java:270)
at org.h2.table.TableFilter.get(TableFilter.java:515)
at org.h2.command.dml.Delete.update(Delete.java:59)
at org.h2.command.dml.DataChangeStatement.update(DataChangeStatement.java:74)
at org.h2.command.CommandContainer.update(CommandContainer.java:169)
at org.h2.command.Command.executeUpdate(Command.java:252)
... 34 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null
at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:159)
at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:814)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:456)
... 52 common frames omitted
Version of dependency-check used
The problem occurs using version 9.0.2 of the cli (from the owasp/dependency-check image)
To Reproduce
Run /usr/share/dependency-check/bin/dependency-check.sh --updateonly, but since transient it’s hard to reproduce reliably.
Expected behavior
Database update should occur without error.
About this issue
- Original URL
- State: closed
- Created 7 months ago
- Comments: 20 (6 by maintainers)
Commits related to this issue
- fix: reduce chance NVD API block updates due to rate limit - increase default delay from 2000 to 3500 - resolves #6195 — committed to jeremylong/DependencyCheck by jeremylong 6 months ago
Setting
-DnvdApiDelay=6000when using Maven worked for me as per recommendations here https://nvd.nist.gov/general/news/API-Key-Announcement It is also recommended that users “sleep” their scripts for six seconds between requests.(The default delay in the code is currently 2000ms, perhaps this should be changed).
I can confirm the same issue with gradle. During initial NVD download, I get transient 403 errors. Here’s the error:
I have the same issue with the version 9.0.7, with the maven plugin. the execution with the API-Key starts download but runs into a 403 error: [INFO] — dependency-check:9.0.7:check (default-cli) @ spielprojekt — [INFO] Checking for updates [INFO] NVD API has 233.797 records in this update [INFO] Downloaded 10.000/233.797 (4%) [INFO] Downloaded 20.000/233.797 (9%) [INFO] Downloaded 30.000/233.797 (13%) [INFO] Downloaded 40.000/233.797 (17%) [INFO] Downloaded 50.000/233.797 (21%) [INFO] Downloaded 60.000/233.797 (26%) [INFO] Downloaded 70.000/233.797 (30%) [INFO] Downloaded 80.000/233.797 (34%) [INFO] Downloaded 90.000/233.797 (38%) [INFO] Downloaded 100.000/233.797 (43%) [ERROR] Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@375cb85d[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@7dad6591[Wrapped tas k = org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$ScheduledRequestExecution@784c5717]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@6ad823 9a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@375cb85d[Not completed, task = java.util.concurrent.Execu tors$RunnableAdapter@7dad6591[Wrapped task = org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$ScheduledRequestExecution@784c5717]] rejected from java.util.conc urrent.ScheduledThreadPoolExecutor@6ad8239a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution (ThreadPoolExecutor.java:2055) at java.util.concurrent.ThreadPoolExecutor.reject (ThreadPoolExecutor.java:825) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute (ScheduledThreadPoolExecutor.java:340) at java.util.concurrent.ScheduledThreadPoolExecutor.schedule (ScheduledThreadPoolExecutor.java:562) at java.util.concurrent.ScheduledThreadPoolExecutor.execute (ScheduledThreadPoolExecutor.java:705) at java.util.concurrent.Executors$DelegatedExecutorService.execute (Executors.java:687) at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.executeScheduled (InternalAbstractHttpAsyncClient.java:361) at org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec$1.failed (AsyncHttpRequestRetryExec.java:164) at org.apache.hc.client5.http.impl.async.AsyncProtocolExec$1.failed (AsyncProtocolExec.java:295) at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.failed (HttpAsyncMainClientExec.java:131) at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.failed (ClientHttp1StreamHandler.java:285) at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.disconnected (ClientHttp1StreamDuplexer.java:220) at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onDisconnect (AbstractHttp1StreamDuplexer.java:409) at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.disconnected (AbstractHttp1IOEventHandler.java:95) … [ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:375) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:115) at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906) … Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403 at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:357) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:348) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:115) at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637) … [ERROR] Failed to execute goal org.owasp:dependency-check-maven:9.0.7:check (default-cli) on project spielprojekt: Fatal exception(s) analyzing Spielprojekt: One or more exceptions occurred during analysis: [ERROR] UpdateException: Error updating the NVD Data [ERROR] caused by NvdApiException: NVD Returned Status Code: 403 [ERROR] NoDataException: No documents exist [ERROR] -> [Help 1]
if i run the same check without the API-Key it succeeds.
If you have multiple builds happening at the same time - using the same API key you could hit the NVD rate limiting threshold. Ideally, in an environment with multiple builds you would implement some sort of caching strategy. The documentation on this continues to evolve/improve: https://github.com/jeremylong/DependencyCheck/pull/6220