powertools-lambda-python: Bug: S3Model not compatible with ObjectRemoved events

Expected Behaviour

After deleting an object from S3, if a Lambda catches this event, they should be able to parse it. This event name from S3 is ObjectDeleted:*.

Current Behaviour

A ValidationError is thrown because of the missing size and eTag attributes.

Records -> 0 -> s3 -> object -> size
  field required (type=value_error.missing)
Records -> 0 -> s3 -> object -> eTag
  field required (type=value_error.missing)

The ObjectDeleted event does not include those attributes.

Code snippet

parse(model=S3Model, event=json.loads(s3_message))

Payload Example: https://github.com/awslabs/aws-lambda-powertools-python/issues/1637#issuecomment-1288170059

Possible Solution

Update S3Object so that both size and eTag are optional:

class S3Object(BaseModel):
    key: str
    size: Optional[NonNegativeFloat]
    eTag: Optional[str]
    sequencer: str
    versionId: Optional[str]

Steps to Reproduce

Create a Lambda function and an S3 bucket
Trigger the Lambda function through S3 events, and enable the ObjectDeleted type
Delete an object from S3
S3Model will raise a ValidationError

AWS Lambda Powertools for Python version

latest

AWS Lambda function runtime

3.8

Packaging format used

PyPi

Debugging logs

No response

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 20 (19 by maintainers)

Commits related to this issue

FIX: Mark size and eTag as optional attributes Fixes ObjectRemoved API event, as they don't include these attributes. https://github.com/awslabs/aws-lambda-powertools-python/issues/1637 Signed-off-... — committed to barreeeiroo/aws-lambda-powertools-python by barreeeiroo 2 years ago
FIX: Mark size and eTag as optional attributes Fixes ObjectRemoved API event, as they don't include these attributes. https://github.com/awslabs/aws-lambda-powertools-python/issues/1637 Signed-off-... — committed to barreeeiroo/aws-lambda-powertools-python by barreeeiroo 2 years ago

Most upvoted comments

@heitorlessa done, the PR should be available now: PR#1638. I couldn’t really test the change right now, but if strictly needed I can fully test it tomorrow.

barreeeiroo on Oct 23, 2022

@barreeeiroo yeah that makes sense! nice one. I’ll open a PR with tests in a few min.

ran-isenberg on Oct 23, 2022

For now, let’s make them optional - it’s a two-way door (we can always add the validators back as it’s a non-breaking change).

We need a better strategy for Parser. It’s starting to break customers more frequently.

Maybe a rule for now could be to only add any additional Parser Models until we have either E2E or regular runs on its various configurations — e.g., MSK takes 30m to setup which isn’t suitable for 30m as it adds up quickly.

@Ran - if you’ve got ideas on how we can make Parser more accurate, despite not having official schemas from Lambda (actual root cause), could you kick off a RFC on this, please?

Thanks a lot you both, truly.

On Sun, 23 Oct 2022 at 20:44, Ran Isenberg @.***> wrote:

Thanks @barreeeiroo https://github.com/barreeeiroo for finding and solving this!

@heitorlessa https://github.com/heitorlessa maybe we can add a validator that checks that they are indeed optional only if the event name is ObjectRemoved:Delete? For other S3object events, these fields are not optional, right?

— Reply to this email directly, view it on GitHub https://github.com/awslabs/aws-lambda-powertools-python/issues/1637#issuecomment-1288173796, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZPQBH2YPWWK5ZAVUXEZCLWEWBRHANCNFSM6AAAAAARMK6GCQ . You are receiving this because you were mentioned.Message ID: @.***>

heitorlessa on Oct 23, 2022