matano: Transform error with client.geo.location

I’m creating a log source for Okta logs and am struggling to transform log data to the ECS fields client.geo.location.lat and client.geo.location.lon. With the VRL below, I consistently get the error “USER_ERROR: Failed at FindUnionVariant, likely schema issue.” in the transformer Lambda. I have pretty much every other Okta log field working.

Looking at the ECS schema JSON, both lat and lon are defined as floats, so this should work.

Relevant VRL transform: .client.geo.location.lat = to_float(del(.json.client.geographicalContext.geolocation.lat)) ?? null .client.geo.location.lon = to_float(del(.json.client.geographicalContext.geolocation.lon)) ?? null

Relevant log data:

{
    "json": {
        "client": {
            "geographicalContext": {
                "city": "Ashburn",
                "country": "United States",
                "geolocation": {
                    "lat": 39.0469,
                    "lon": -77.4903
                },
                "postalCode": "20149",
                "state": "Virginia"
            }
        }
}

Any assistance identifying the issue or bug would be appreciated.

Thanks.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 19 (11 by maintainers)

Most upvoted comments

Yeah, my internet is stable. I’ll continue to troubleshoot the CDK issue on my own and open another issue if necessary. I’ll close this issue because I was able to get the Okta float data through the Matano data pipieline, so I think we’re good.

I’d be happy to contribute my Okta work to the Matano project if you’re interested in making it a managed source, just let me know.

Thanks for your help and responsiveness solving this!

gdrapp on Dec 18, 2022

The stack deploy debug no longer showed null etags for the multipart uploads, which is good, but it still failed to deploy the first two timed I tried, with this error:

⠹ Deploying Matano...[91%] fail: One or more of the specified parts could not be found.  The part may not have been uploaded, or the specified entity tag may not match the part's entity tag.

The third time I ran it, the deployment was successful.

It seems like it’s having trouble with the multipart uploads but if you run it a few times CDK eventually gets everything uploaded and it’s happy. September/October I was experimenting with Matano and didn’t have these issues, so it’s strange that this just started happening with later builds (didn’t have time to play with it in November).

Matano version - matano/0.0.0 darwin-x64 node-v14.18.1

Someone in the AWS CDK issue linked earlier mentioned they were running 16.7 and didn’t see any issues, so it might be worth trying that version if 16.3 is giving you problems.

gdrapp on Dec 14, 2022

Not seeing any nightlies available for download right now. Can you check the build?

gdrapp on Dec 13, 2022