zaproxy: Error Persisting OpenAPI definition

Describe the bug

Hi,

when I try to “Import an OpenAPI definition from the local file system” from Zap UI, I obtain an error.

At first, message is:

OpenAPI definition parsed successfully

When I close pop up, I obtain the following message:

Job Load Openapi Definition target: null error: Failed to parse OpenAPI definition.
org.parosproxy.paros.db.DatabaseException: java.sql.SQLDataException: data exception: text data, right truncated ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
    at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:86)
    at org.zaproxy.zap.extension.openapi.ExtensionOpenApi$1.run(ExtensionOpenApi.java:449)
Caused by: java.sql.SQLDataException: data exception: text data, right truncated ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source)
    at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:84)

The problem is in zap UI, and zap Automation Framework, because plan has been stopped when error is present.

File tested is 280KB

Steps to reproduce the behavior

Go to Import Options
Select: “Import an OpenAPI definition from the local file system”
Close pop up: “OpenAPI definition parsed successfully”
Process is stopped with error

Expected behavior

Zap Automation framework should´t receive error and plan continue as expected

Software versions

ZAP 2.12.0 on Windows

ZAP 2.12.0 on Mac

Screenshots

Errors from the zap.log file

Log info

2023-05-24 10:14:41,084 [ZAP-BootstrapGUI] INFO  BaseDatabaseType - Database: jdbc:hsqldb:file:C:\Users\Username\OWASP ZAP\db\permanent (HSQL Database Engine 2.7)
2023-05-24 10:14:41,099 [ZAP-BootstrapGUI] WARN  Database - Flyway upgrade recommended: HSQLDB 2.7 is newer than this version of Flyway and support has not been tested. The latest supported version of HSQLDB is 2.6.
2023-05-24 10:14:41,153 [ZAP-BootstrapGUI] INFO  DbValidate - Successfully validated 1 migration (execution time 00:00.030s)
2023-05-24 10:14:41,168 [ZAP-BootstrapGUI] INFO  DbMigrate - Current version of schema "PUBLIC": 1
2023-05-24 10:14:41,168 [ZAP-BootstrapGUI] INFO  DbMigrate - Schema "PUBLIC" is up to date. No migration necessary.
2023-05-24 10:14:41,184 [ZAP-BootstrapGUI] INFO  CallbackService - Started callback service on 0.0.0.0:1959
2023-05-24 10:14:45,888 [AWT-EventQueue-0] INFO  Control - New Session
2023-05-24 10:14:45,919 [AWT-EventQueue-0] INFO  Control - Create and Open Untitled Db
2023-05-24 10:14:48,309 [ZAP-cfu] WARN  AddOnCollection - Failed to create add-on for openapi
java.util.zip.ZipException: zip END header not found
	at java.util.zip.ZipFile$Source.findEND(ZipFile.java:1474) ~[?:?]
	at java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1482) ~[?:?]
	at java.util.zip.ZipFile$Source.<init>(ZipFile.java:1320) ~[?:?]
	at java.util.zip.ZipFile$Source.get(ZipFile.java:1282) ~[?:?]
	at java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:709) ~[?:?]
	at java.util.zip.ZipFile.<init>(ZipFile.java:243) ~[?:?]
	at java.util.zip.ZipFile.<init>(ZipFile.java:172) ~[?:?]
	at java.util.zip.ZipFile.<init>(ZipFile.java:186) ~[?:?]
	at org.zaproxy.zap.control.AddOn.loadManifestFile(AddOn.java:490) ~[zap-2.12.0.jar:2.12.0]
	at org.zaproxy.zap.control.AddOn.<init>(AddOn.java:578) ~[zap-2.12.0.jar:2.12.0]
	at org.zaproxy.zap.control.AddOnCollection.load(AddOnCollection.java:140) ~[zap-2.12.0.jar:2.12.0]
	at org.zaproxy.zap.control.AddOnCollection.<init>(AddOnCollection.java:59) ~[zap-2.12.0.jar:2.12.0]
	at org.zaproxy.zap.extension.autoupdate.ExtensionAutoUpdate$4.run(ExtensionAutoUpdate.java:1122) ~[zap-2.12.0.jar:2.12.0]
2023-05-24 10:14:48,340 [ZAP-cfu] WARN  AddOnCollection - Failed to create add-on for reports
java.util.zip.ZipException: zip END header not found
	at java.util.zip.ZipFile$Source.findEND(ZipFile.java:1474) ~[?:?]
	at java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1482) ~[?:?]
	at java.util.zip.ZipFile$Source.<init>(ZipFile.java:1320) ~[?:?]
	at java.util.zip.ZipFile$Source.get(ZipFile.java:1282) ~[?:?]
	at java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:709) ~[?:?]
	at java.util.zip.ZipFile.<init>(ZipFile.java:243) ~[?:?]
	at java.util.zip.ZipFile.<init>(ZipFile.java:172) ~[?:?]
	at java.util.zip.ZipFile.<init>(ZipFile.java:186) ~[?:?]
	at org.zaproxy.zap.control.AddOn.loadManifestFile(AddOn.java:490) ~[zap-2.12.0.jar:2.12.0]
	at org.zaproxy.zap.control.AddOn.<init>(AddOn.java:578) ~[zap-2.12.0.jar:2.12.0]
	at org.zaproxy.zap.control.AddOnCollection.load(AddOnCollection.java:140) ~[zap-2.12.0.jar:2.12.0]
	at org.zaproxy.zap.control.AddOnCollection.<init>(AddOnCollection.java:59) ~[zap-2.12.0.jar:2.12.0]
	at org.zaproxy.zap.extension.autoupdate.ExtensionAutoUpdate$4.run(ExtensionAutoUpdate.java:1122) ~[zap-2.12.0.jar:2.12.0]
2023-05-24 10:14:49,386 [ZAP-cfu] INFO  ExtensionAutoUpdate - There is/are 3 newer addons
2023-05-24 10:14:59,378 [ZAP-DownloadInstaller] INFO  ExtensionAutoUpdate - Installing new addon domxss v15.0.0
2023-05-24 10:14:59,378 [ZAP-DownloadInstaller] INFO  ExtensionAutoUpdate - Finished installing new addon domxss v15.0.0
2023-05-24 10:15:00,212 [ZAP-DownloadInstaller] INFO  ExtensionAutoUpdate - Installing new addon selenium v15.12.0
2023-05-24 10:15:00,539 [ZAP-DownloadInstaller] INFO  ExtensionPassiveScan - loaded passive scan rule: Authentication Request Identified
2023-05-24 10:15:00,539 [ZAP-DownloadInstaller] INFO  ExtensionPassiveScan - loaded passive scan rule: Session Management Response Identified
2023-05-24 10:15:00,539 [ZAP-DownloadInstaller] INFO  ExtensionPassiveScan - loaded passive scan rule: Verification Request Identified
2023-05-24 10:15:01,194 [ZAP-DownloadInstaller] INFO  ExtensionAutoUpdate - Finished installing new addon selenium v15.12.0
2023-05-24 10:15:01,279 [ZAP-DownloadInstaller] INFO  ExtensionAutoUpdate - Installing new addon authhelper v0.7.0
2023-05-24 10:15:01,324 [ZAP-DownloadInstaller] INFO  ExtensionPassiveScan - loaded passive scan rule: Authentication Request Identified
2023-05-24 10:15:01,324 [ZAP-DownloadInstaller] INFO  ExtensionPassiveScan - loaded passive scan rule: Session Management Response Identified
2023-05-24 10:15:01,324 [ZAP-DownloadInstaller] INFO  ExtensionPassiveScan - loaded passive scan rule: Verification Request Identified
2023-05-24 10:15:01,339 [ZAP-DownloadInstaller] INFO  ExtensionAutoUpdate - Finished installing new addon authhelper v0.7.0
2023-05-24 10:21:20,570 [ZAP-Import-OpenAPI-1] WARN  ExtensionOpenApi - java.sql.SQLDataException: excepción de datos: dato de tipo texto, truncado por la derecha ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
org.parosproxy.paros.db.DatabaseException: java.sql.SQLDataException: excepción de datos: dato de tipo texto, truncado por la derecha ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
	at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:86) ~[?:?]
	at org.zaproxy.zap.extension.openapi.ExtensionOpenApi$1.run(ExtensionOpenApi.java:449) ~[?:?]
Caused by: java.sql.SQLDataException: excepción de datos: dato de tipo texto, truncado por la derecha ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
	at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:84) ~[?:?]
	... 1 more
Caused by: org.hsqldb.HsqlException: excepción de datos: dato de tipo texto, truncado por la derecha ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
	at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.enforceTypeLimits(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.generateAndCheckData(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.insertSingleRow(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementDML.insertSingleRow(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementInsert.getResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementDMQL.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Session.executeCompiledStatement(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Session.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:84) ~[?:?]
	... 1 more
Caused by: org.hsqldb.HsqlException: excepción de datos: dato de tipo texto, truncado por la derecha
	at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.types.CharacterType.convertToTypeLimits(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.enforceTypeLimits(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.generateAndCheckData(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.insertSingleRow(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementDML.insertSingleRow(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementInsert.getResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementDMQL.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Session.executeCompiledStatement(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Session.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:84) ~[?:?]
	... 1 more
2023-05-24 10:31:24,243 [ZAP-Import-OpenAPI-2] WARN  ExtensionOpenApi - java.sql.SQLDataException: excepción de datos: dato de tipo texto, truncado por la derecha ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
org.parosproxy.paros.db.DatabaseException: java.sql.SQLDataException: excepción de datos: dato de tipo texto, truncado por la derecha ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
	at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:86) ~[?:?]
	at org.zaproxy.zap.extension.openapi.ExtensionOpenApi$1.run(ExtensionOpenApi.java:449) ~[?:?]
Caused by: java.sql.SQLDataException: excepción de datos: dato de tipo texto, truncado por la derecha ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
	at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:84) ~[?:?]
	... 1 more
Caused by: org.hsqldb.HsqlException: excepción de datos: dato de tipo texto, truncado por la derecha ; size limit: 16777216 table: OPENAPI_SPECS column: DEFINITION
	at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.enforceTypeLimits(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.generateAndCheckData(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.insertSingleRow(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementDML.insertSingleRow(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementInsert.getResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementDMQL.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Session.executeCompiledStatement(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Session.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:84) ~[?:?]
	... 1 more
Caused by: org.hsqldb.HsqlException: excepción de datos: dato de tipo texto, truncado por la derecha
	at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.types.CharacterType.convertToTypeLimits(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.enforceTypeLimits(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.generateAndCheckData(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Table.insertSingleRow(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementDML.insertSingleRow(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementInsert.getResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.StatementDMQL.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Session.executeCompiledStatement(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.Session.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source) ~[hsqldb-2.7.1.jar:2.7.1]
	at org.zaproxy.zap.extension.openapi.TableOpenApi.insertOpenApiSpec(TableOpenApi.java:84) ~[?:?]
	... 1 more

Additional context

No response

Would you like to help fix this issue?

Done/Todo:

Increase the column size. Done in: https://github.com/zaproxy/zap-extensions/pull/4697
Warn but don’t fail if the column is too small. Done in: https://github.com/zaproxy/zap-extensions/pull/4697
Consider the timing/conditions for the def’n being persisted (parsed/resolved? complete? etc)

About this issue

Original URL
State: open
Created a year ago
Comments: 15 (8 by maintainers)

Most upvoted comments

Version 34 of the OpenAPI add-on is being released right now. You should be able to test the fixes shortly.

kingthorin on Jun 27, 2023

Thank you!

thc202 on Jun 23, 2023

https://github.com/zaproxy/zaproxy/issues/7876#issuecomment-1599722925

What’s the difference between the two? Is the definition originally split between several files?

thc202 on Jun 21, 2023

I dumped the definition to a file just before it went to persist it to the db:

pdicarlo@Pauls-MacBook-Pro /tmp % ls -ltrh file.txt
-rw-r--r--@ 1 pdicarlo  wheel    19M Jun 20 18:26 file.txt
pdicarlo@Pauls-MacBook-Pro /tmp % wc -l file.txt
  373126 file.txt

pauldicarlo on Jun 20, 2023

Looks like 4meg to me. (Tip: You can add h to your ls switches for human readable sizes.)

kingthorin on Jun 20, 2023

The import file ends up 395 tasks being run.

pdicarlo@Pauls-MacBook-Pro vpc-not-public % ls -ltr vpc-scoped.json              
-rw-r--r--  1 pdicarlo  staff  4034889 Jun  5 09:05 vpc-scoped.json

pdicarlo@Pauls-MacBook-Pro vpc-not-public % wc -l vpc-scoped.json  
  105060 vpc-scoped.json

pdicarlo@Pauls-MacBook-Pro vpc-not-public % grep responses vpc-scoped.json |wc -l
     395

I’ll see if I can modify the code to get the size of the definition string that is created from the import. I don’t think I can provide my employers json file

pauldicarlo on Jun 20, 2023

In my local environment, I increased the size for definition to 64M and that seemed to fix the problem for our rather large json file we were seeking to import. Would it be possible to expose this to a configurable setting? (apologies, I’m not an expert in Zap). Thank you.

pauldicarlo on Jun 20, 2023