kestra: Logs page freeze / lag when dbt plugin generates a lot of models results

Expected Behavior

We expect the logs section of dbt cloud trigger run to load in a reasonable amount of time.

Actual Behaviour

A lot of timeout and laggy page / blocked / crashing.

Steps To Reproduce

Run large project dbt cloud trigger run from Kestra

Environment Information

Kestra Version: 0.11.0-SNAPSHOT
Operating System (OS / Docker / Kubernetes): Kubernetes
Java Version (If not docker):

Example flow

No response

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 16 (16 by maintainers)

Most upvoted comments

Those 3 issues will also provide a smoother log explore experience:

brian-mulier-p on Sep 26, 2023

@aballiet can you try setting parseRunResult to false? (you need the latest plugin with this https://github.com/kestra-io/plugin-dbt/commit/1ba6aa721d7e758caa4fea37343e8cf675469451)

anna-geller on Sep 26, 2023

Hello @aballiet can you try again with the latest SNAPSHOT version which includes this ? I think it will fix the issue 👍

brian-mulier-p on Sep 25, 2023

closing since there is more detailed issues open by @brian-mulier-p ⏩

tchiotludo on Sep 26, 2023

@aballiet can you try setting parseRunResult to false? (you need the latest plugin with this https://github.com/kestra-io/plugin-dbt/commit/1ba6aa721d7e758caa4fea37343e8cf675469451)

Nice! Now we bundle our custom docker image to have more control on upgrades. I can try by EOW 😊

aballiet on Sep 26, 2023

aballiet on Sep 25, 2023

Also maybe it’s because you have the “expand all” settings for logs which make the page display everything at once, might help with FAILED only or collapse all at first. Still you can try it as-is with the new version I made a lot of improvement on the page

brian-mulier-p on Sep 25, 2023

Hi @aballiet ! Could you show us your browser console? That could leverage if the issue is coming from the front or the back

edit: You may need to open the console before the crash occurred

Skraye on Sep 6, 2023

Did a quick Loom video to show the bug : https://www.loom.com/share/0556984bd9564b328ea4cf413954a514

A fix is required because currently clicking on “logs” freezes the web browser 😦

Yes I think displaying just the logs like we have in the Gantt is fine. No need to display one block per model. Might be nice though to list all warning models / tests and in case of error, highlight the model responsible of error.

They have this system on dbt cloud :

aballiet on Sep 6, 2023