DependencyCheck: NVD API update is failing

Describe the bug Running dependency-check --enableExperimental --nvdApiKey <my-nvd-api-key> --out ~/file-dependency-check.html --scan Package.resolved on 9.0.5 fails during the records update

Version of dependency-check used The problem occurs using version 9.0.5 of the CLI

Log file gist

To Reproduce Steps to reproduce the behavior: Run the dependency check

Expected behavior The update should be successful, and the dependency check report should be generated.

Additional context This is running on a Circleci job, I’m downloading the latest version of the dependency check. After the update it started failing.

About this issue

Original URL
State: closed
Created 7 months ago
Comments: 24 (9 by maintainers)

Commits related to this issue

fix: favor stability over performance - increase delay between NVD API calls - reduces chance of hitting rate limit imposed by the NVD - resolves #6294 — committed to jeremylong/DependencyCheck by jeremylong 6 months ago

Most upvoted comments

@jeremylong

I’ve been getting a multitude of errors on 9.0.7, from 403s (even though I’m running the command on my local machine with a fresh API Key) to database errors like

org.owasp.dependencycheck.data.nvdcve.DatabaseException: Unable to retrieve id for new vulnerability for ‘CVE-2015-8482’
Caused by: org.h2.jdbc.JdbcSQLNonTransientConnectionException: The database has been closed [90098-214]
Writing to sun.nio.ch.FileChannelImpl@22ec811e failed; length 204800 at 9560064 [2.1.214/2]
java.lang.NullPointerException: Cannot invoke “org.apache.commons.dbcp2.BasicDataSource.getConnection()” because “this.connectionPool” is null

    at org.owasp.dependencycheck.data.nvdcve.DatabaseManager.getConnection (DatabaseManager.java:572)
    at org.owasp.dependencycheck.data.nvdcve.CveDB.updateOrInsertVulnerability (CveDB.java:941)
    at org.owasp.dependencycheck.data.nvdcve.CveDB.updateVulnerability (CveDB.java:866)
    at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call (NvdApiProcessor.java:98)
    at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call (NvdApiProcessor.java:33)

I’m downgrading back to 8.4.2 because 9.x.x has been nothing but trouble for me 👎

TheMorganHub on Dec 26, 2023

Curl works w/ a new key, but I can’t seem to get the key passed in to the gradle tasks.

Tried :
* `./gradlew dependencyCheckAnalyze -DnvdApiKey=<redacted>`

* `./gradlew dependencyCheckAnalyze -Dnvd.ApiKey=<redacted>`
and in build.gradle.kts
dependencyCheck {
    nvd.apiKey = System.getenv("NVD_API_KEY")
}
edit: now curl is failing w/ the same api key message: Invalid apiKey

edit2: got another key, curl is failing w/ that one too. Just feels like they were simply not ready for this change to go live.

FWIW: You’re passing the key as a system property. So you might want to try System.getProperty("nvdApiKey")

el-dom on Dec 19, 2023

Weirdly my key works when I run the command in my local machine. I even purged the nvd file to check. The issue only happens when I run it on CircleCI with the API key. 🤦

Have the same issue. API key works locally but on GitHub Actions the NVD update is (sometimes) interrupted by an exception:

[INFO] Downloaded 90,000/233,564 (39%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
...
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403`.

Detailed maven log excerpt:

2023-12-18T04:40:04.2168058Z [INFO] Checking for updates
2023-12-18T04:40:06.0015022Z [INFO] NVD API has 233,564 records in this update
2023-12-18T04:40:12.4805299Z [INFO] Downloaded 10,000/233,564 (4%)
2023-12-18T04:40:21.0776143Z [INFO] Downloaded 20,000/233,564 (9%)
2023-12-18T04:40:26.4392808Z [INFO] Downloaded 30,000/233,564 (13%)
2023-12-18T04:40:31.0227923Z [INFO] Downloaded 40,000/233,564 (17%)
2023-12-18T04:40:35.7817056Z [INFO] Downloaded 50,000/233,564 (21%)
2023-12-18T04:40:41.4263361Z [INFO] Downloaded 60,000/233,564 (26%)
2023-12-18T04:40:46.7154836Z [INFO] Downloaded 70,000/233,564 (30%)
2023-12-18T04:40:50.8825364Z [INFO] Downloaded 80,000/233,564 (34%)
2023-12-18T04:40:50.8826214Z [INFO] Downloaded 90,000/233,564 (39%)
2023-12-18T04:40:57.1447264Z [ERROR] Error updating the NVD Data
2023-12-18T04:40:57.1559725Z org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
2023-12-18T04:40:57.1571805Z     at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:375)
2023-12-18T04:40:57.1574019Z     at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:115)
2023-12-18T04:40:57.1575563Z     at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
2023-12-18T04:40:57.1576943Z     at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
2023-12-18T04:40:57.1578431Z     at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
2023-12-18T04:40:57.1580127Z     at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1943)
2023-12-18T04:40:57.1582075Z     at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1126)
2023-12-18T04:40:57.1584303Z     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
2023-12-18T04:40:57.1587704Z     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
2023-12-18T04:40:57.1589515Z     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
2023-12-18T04:40:57.1591232Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
2023-12-18T04:40:57.1592922Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
2023-12-18T04:40:57.1594928Z     at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
2023-12-18T04:40:57.1596603Z     at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
2023-12-18T04:40:57.1598475Z     at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
2023-12-18T04:40:57.1600380Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
2023-12-18T04:40:57.1602375Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
2023-12-18T04:40:57.1604867Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
2023-12-18T04:40:57.1607724Z     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
2023-12-18T04:40:57.1609961Z     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
2023-12-18T04:40:57.1611578Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
2023-12-18T04:40:57.1612909Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
2023-12-18T04:40:57.1614194Z     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
2023-12-18T04:40:57.1615714Z     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:910)
2023-12-18T04:40:57.1616923Z     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
2023-12-18T04:40:57.1618076Z     at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
2023-12-18T04:40:57.1619632Z     at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
2023-12-18T04:40:57.1621150Z     at java.lang.reflect.Method.invoke (Method.java:580)
2023-12-18T04:40:57.1622581Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
2023-12-18T04:40:57.1624278Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
2023-12-18T04:40:57.1626359Z     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
2023-12-18T04:40:57.1628282Z     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
2023-12-18T04:40:57.1629982Z     at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
2023-12-18T04:40:57.1631352Z     at java.lang.reflect.Method.invoke (Method.java:580)
2023-12-18T04:40:57.1632618Z     at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:52)
2023-12-18T04:40:57.1634147Z     at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:161)
2023-12-18T04:40:57.1635657Z     at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:73)
2023-12-18T04:40:57.1637386Z Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403
2023-12-18T04:40:57.1639316Z     at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:357)
2023-12-18T04:40:57.1641216Z     at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:348)
2023-12-18T04:40:57.1643073Z     at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:115)
2023-12-18T04:40:57.1644534Z     at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
2023-12-18T04:40:57.1645969Z     at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
2023-12-18T04:40:57.1647462Z     at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
2023-12-18T04:40:57.1701778Z     at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1943)
2023-12-18T04:40:57.1704162Z     at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1126)
2023-12-18T04:40:57.1706263Z     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
2023-12-18T04:40:57.1708388Z     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
2023-12-18T04:40:57.1710139Z     at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
2023-12-18T04:40:57.1711794Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
2023-12-18T04:40:57.1713411Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
2023-12-18T04:40:57.1715056Z     at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
2023-12-18T04:40:57.1716755Z     at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
2023-12-18T04:40:57.1718568Z     at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
2023-12-18T04:40:57.1720530Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
2023-12-18T04:40:57.1722322Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
2023-12-18T04:40:57.1724646Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
2023-12-18T04:40:57.1727014Z     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
2023-12-18T04:40:57.1729180Z     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
2023-12-18T04:40:57.1730753Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
2023-12-18T04:40:57.1732009Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
2023-12-18T04:40:57.1733240Z     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
2023-12-18T04:40:57.1734404Z     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:910)
2023-12-18T04:40:57.1735532Z     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
2023-12-18T04:40:57.1736632Z     at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
2023-12-18T04:40:57.1738209Z     at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
2023-12-18T04:40:57.1739616Z     at java.lang.reflect.Method.invoke (Method.java:580)
2023-12-18T04:40:57.1740977Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
2023-12-18T04:40:57.1742632Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
2023-12-18T04:40:57.1744400Z     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
2023-12-18T04:40:57.1745917Z     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
2023-12-18T04:40:57.1747730Z     at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
2023-12-18T04:40:57.1749153Z     at java.lang.reflect.Method.invoke (Method.java:580)
2023-12-18T04:40:57.1750476Z     at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:52)
2023-12-18T04:40:57.1752050Z     at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:161)
2023-12-18T04:40:57.1753548Z     at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:73)
2023-12-18T04:40:57.1755138Z [ERROR] Failed to process CVE-2011-1485
2023-12-18T04:40:57.1756498Z org.owasp.dependencycheck.data.nvdcve.DatabaseException: Error updating 'CVE-2011-1485'
2023-12-18T04:40:57.1758196Z     at org.owasp.dependencycheck.data.nvdcve.CveDB.updateVulnerability (CveDB.java:877)
2023-12-18T04:40:57.1760006Z     at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call (NvdApiProcessor.java:98)
2023-12-18T04:40:57.1761924Z     at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call (NvdApiProcessor.java:33)
2023-12-18T04:40:57.1763432Z     at java.util.concurrent.FutureTask.run (FutureTask.java:317)
2023-12-18T04:40:57.1764799Z     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144)
2023-12-18T04:40:57.1766474Z     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
2023-12-18T04:40:57.1767836Z     at java.lang.Thread.run (Thread.java:1583)
2023-12-18T04:40:57.1770011Z Caused by: org.h2.jdbc.JdbcBatchUpdateException: IO Exception: "C:/Users/runneradmin/.m2/repository/org/owasp/dependency-check-data/9.0/odc.mv.db"; SQL statement:
2023-12-18T04:40:57.1773106Z INSERT INTO cpeEntry (part, vendor, product, version, update_version, edition, lang, sw_edition, target_sw, target_hw, other, ecosystem) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [90028-214]
2023-12-18T04:40:57.1775415Z     at org.h2.jdbc.JdbcPreparedStatement.executeBatch (JdbcPreparedStatement.java:1269)
2023-12-18T04:40:57.1777170Z     at org.apache.commons.dbcp2.DelegatingStatement.executeBatch (DelegatingStatement.java:230)
2023-12-18T04:40:57.1788683Z     at org.apache.commons.dbcp2.DelegatingStatement.executeBatch (DelegatingStatement.java:230)
2023-12-18T04:40:57.1790451Z     at org.owasp.dependencycheck.data.nvdcve.CveDB.executeBatch (CveDB.java:1381)
2023-12-18T04:40:57.1792269Z     at org.owasp.dependencycheck.data.nvdcve.CveDB.updateVulnerabilityInsertSoftware (CveDB.java:1221)
2023-12-18T04:40:57.1794140Z     at org.owasp.dependencycheck.data.nvdcve.CveDB.updateVulnerability (CveDB.java:871)
2023-12-18T04:40:57.1795970Z     at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call (NvdApiProcessor.java:98)
2023-12-18T04:40:57.1797869Z     at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call (NvdApiProcessor.java:33)
2023-12-18T04:40:57.1799387Z     at java.util.concurrent.FutureTask.run (FutureTask.java:317)
2023-12-18T04:40:57.1800747Z     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144)
2023-12-18T04:40:57.1802310Z     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
2023-12-18T04:40:57.1803491Z     at java.lang.Thread.run (Thread.java:1583)
2023-12-18T04:40:57.1804522Z [ERROR] Failed to process CVE-2011-1486
2023-12-18T04:40:57.1807897Z org.owasp.dependencycheck.data.nvdcve.DatabaseException: Unable to retrieve id for new vulnerability for 'CVE-2011-1486'

gorankarlic on Dec 18, 2023

I would highly recommend reading https://jeremylong.github.io/DependencyCheck/data/index.html - specifically “The NVD Database” section. It would be ideal if you persisted the data directory between builds.

jeremylong on Dec 15, 2023

Weirdly my key works when I run the command in my local machine. I even purged the nvd file to check. The issue only happens when I run it on CircleCI with the API key. 🤦

rouse-andre-rios on Dec 14, 2023

If you are using an NVD API key and you get an immediate 403 - it is likely your API Key is not valid.

See https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

jeremylong on Dec 14, 2023

I request for new nvd api key and now it works

sysmat on Dec 14, 2023