if: 👍🏼 Update IF to dump logs and errors to yaml file

Sub of: #629 -> #650

What

Update IF so that errors and logs are saved to file as well as being displayed in the console.

Why

As a core developer I want to be able to run negative tests, which requires errors and logs to be captured in a file we can snapshot against. As a user I want to be able to provide a log file as part of a bug report to help the core team guide me through debugging.

Context

Testing negative flows is important for us to make sure IF behaves as intended across a large set of scenarios. It is hard to do this without dumping errors and logs to files because the alternative is for IF to panic without any persistent result to test against. This can be resolved by capturing the errors and logs to a file.

SoW

  • This output manifest format can be used to test negative test cases as per #615
  • Documentation has been added to the if-docs repo to explain this change.

Acceptance criteria

  • Manifests that causes critical exceptions (causing the tool to fail) output status and error detail in an execution node.

GIVEN the user is in a folder WHEN they run this manifest file:

name: basic-error-demo
description:
tags:
initialize:
  plugins:
    teads-curve: 
      path: '@grnsft/if-unofficial-plugins'
      method: TeadsCurve
      global-config:
        interpolation: spline
  outputs:
    - yaml
tree:
  children:
    child-0:
      defaults:
        cpu/thermal-design-power: 100
      pipeline:
        - teads-curve
      inputs:
        - timestamp: 2023-07-06T00:00
          cpu/utilization: 20

THEN the tool logs the error, exits and outputs a manifest file which contains these additional nodes:

  • execution
    • status: (success/fail)
    • error: The exception that caused the failure and contents of the exception (to string())
name: basic-error-demo
description:
tags:
initialize:
  plugins:
    teads-curve: 
      path: '@grnsft/if-unofficial-plugins'
      method: TeadsCurve
      global-config:
        interpolation: spline
  outputs:
    - yaml
execution:
  status: fail
  error: 'InputValidationError:   "duration" parameter is required. Error code: invalid_type'.
tree:
  children:
    child-0:
      defaults:
        cpu/thermal-design-power: 100
      pipeline:
        - teads-curve
      inputs:
        - timestamp: 2023-07-06T00:00
          cpu/utilization: 20

The errors and logs should still be displayed in the console as usual, because the version int he manifest file will not contain the full stack trace.

About this issue

  • Original URL
  • State: open
  • Created 3 months ago
  • Reactions: 1
  • Comments: 16 (13 by maintainers)

Most upvoted comments

@zanete FYI I untagged @MariamKhalatova and added a failing manifest and expected output manifest to the ticket description

@jmcook1186 this one is good to go from me.