prowler: [Bug]: Unable to complete scans for Chinese AWS accounts

What happened? I’ve tried to run all-checks scan using Prowler v3, however, prowler suddenly stops execution and is only able to show me the report for the 3 services:

Date: 2022-12-21 16:47:49


This report is being generated using credentials below:

AWS-CLI Profile: [default] AWS Filter Region: [all]
AWS Account: [xxxxxxxxxx] UserId: [XXXXXXXXXXX:xxxxxx]
Caller Identity ARN: [arn:aws-cn:sts::xxxxxxxxxx:assumed-role/xxx/xxx]

Executing 244 checks, please wait...

-> Scan completed! |▉▉▉▉▉▉▉⚠︎                                | (!) 43/244 [18%] in 30.5s

Overview Results:
╭───────────────────┬───────────────────╮
│ 26.0% (13) Failed │ 68.0% (34) Passed │
╰───────────────────┴───────────────────╯

Account xxxxxxxxxx Scan Results (severity columns are for fails only):
╭────────────┬────────────┬───────────┬────────────┬────────┬──────────┬───────╮
│ Provider   │ Service    │ Status    │   Critical │   High │   Medium │   Low │
├────────────┼────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ account    │ PASS (3)  │          0 │      0 │        0 │     0 │
├────────────┼────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ cloudwatch │ PASS (1)  │          0 │      0 │        0 │     0 │
├────────────┼────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ iam        │ FAIL (13) │          2 │      3 │        6 │     2 │
╰────────────┴────────────┴───────────┴────────────┴────────┴──────────┴───────╯
* You only see here those services that contains resources.

How to reproduce it Steps to reproduce the behavior:

What command are you running? prowler aws -M HTML
Environment you have, like single account, multi-account, organizations, etc. Single AWS account in the Chinese partition
See error After I enabled log-level=DEBUG I can see the following errors for each set of checks:

2022-12-21 16:52:24,271 [File: check.py:343]    [Module: check]  ERROR: KeyError[325]: 'aws-cn'
2022-12-21 16:52:24,274 [File: check.py:343]    [Module: check]  ERROR: KeyError[325]: 'aws-cn'
2022-12-21 16:52:24,276 [File: check.py:343]    [Module: check]  ERROR: KeyError[325]: 'aws-cn'

Moreover, I can see some errors related to hitting the Global AWS endpoints instead of Chinese ones:

2022-12-21 16:52:23,503 [File: connectionpool.py:1001]  [Module: connectionpool]         DEBUG: Starting new HTTPS connection (1): globalaccelerator.us-west-2.amazonaws.com:443
2022-12-21 16:52:24,232 [File: connectionpool.py:456]   [Module: connectionpool]         DEBUG: https://globalaccelerator.us-west-2.amazonaws.com:443 "POST / HTTP/1.1" 400 106
2022-12-21 16:52:24,232 [File: parsers.py:240]  [Module: parsers]        DEBUG: Response headers: {'Date': 'Wed, 21 Dec 2022 14:52:24 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '106', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'ab20be38-cf2a-4207-a87a-5bcad84bf168'}
2022-12-21 16:52:24,233 [File: parsers.py:241]  [Module: parsers]        DEBUG: Response body:
b'{"__type":"UnrecognizedClientException","message":"The security token included in the request is invalid"}'
2022-12-21 16:52:24,234 [File: hooks.py:238]    [Module: hooks]  DEBUG: Event needs-retry.global-accelerator.ListAccelerators: calling handler <botocore.retryhandler.RetryHandler object at 0x111af2370>
2022-12-21 16:52:24,234 [File: retryhandler.py:211]     [Module: retryhandler]   DEBUG: No retry needed.
2022-12-21 16:52:24,235 [File: globalaccelerator_service.py:40]         [Module: globalaccelerator_service]      ERROR: us-west-2 -- ClientError[27]: An error occurred (UnrecognizedClientException) when calling the ListAccelerators operation: The security token included in the request is invalid

It says token is invalid because the region is the wrong one for some reason.

Expected behavior Prowler v3 is able to scan Chinese AWS accounts

Screenshots or Logs

From where are you running Prowler? Please, complete the following information:

Resource: workstation
OS: MacOS Monterey
AWS-CLI Version [aws --version]: aws-cli/2.7.2 Python/3.9.12 Darwin/21.6.0 source/x86_64 prompt/off (I suppose it’s not needed anymore as you guys are using boto3 as an AWS client)
Prowler Version [./prowler -V]: Prowler 3.0.0
Shell and version: zsh 5.8.1 (x86_64-apple-darwin21.0)
Others:

Additional context Add any other context about the problem here.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 17 (17 by maintainers)

Most upvoted comments

Awesome, thank you for all the help @daftkid ❤️ I close the issue since it is solved.

sergargar on Dec 23, 2022

@sergargar please see below

{"timestamp": "2022-12-22 18:08:18,088", "filename": "awslambda_service.py:124", "level": "ERROR", "module": "awslambda_service", "message": "cn-northwest-1 -- ClientError[112]: An error occurred (AccessDeniedException) when calling the GetFunctionUrlConfig operation: Unable to determine service/operation name to be authorized"}
{"timestamp": "2022-12-22 18:09:14,054", "filename": "ec2_elastic_ip_shodan.py:38", "level": "ERROR", "module": "ec2_elastic_ip_shodan", "message": "ERROR: No Shodan API Key -- Please input a Shodan API Key with -N/--shodan or in config.yaml"}
{"timestamp": "2022-12-22 18:10:02,638", "filename": "kms_service.py:75", "level": "ERROR", "module": "kms_service", "message": "cn-northwest-1 -- ClientError:71 -- An error occurred (AccessDeniedException) when calling the GetKeyRotationStatus operation: User: arn:aws-cn:iam::xxxxxxxxxxxxxxx:user/oleksandr.mykytenko is not authorized to perform: kms:GetKeyRotationStatus on resource: arn:aws-cn:kms:cn-northwest-1:xxxxxxxxxxxxxxxxx:key/key_id because no resource-based policy allows the kms:GetKeyRotationStatus action"}
{"timestamp": "2022-12-22 18:10:14,164", "filename": "route53_service.py:115", "level": "ERROR", "module": "route53_service", "message": "us-east-1 -- ClientError[106]: An error occurred (UnrecognizedClientException) when calling the ListDomains operation: The security token included in the request is invalid."}
{"timestamp": "2022-12-22 18:10:29,871", "filename": "globalaccelerator_service.py:40", "level": "ERROR", "module": "globalaccelerator_service", "message": "us-west-2 -- ClientError[27]: An error occurred (UnrecognizedClientException) when calling the ListAccelerators operation: The security token included in the request is invalid."}
{"timestamp": "2022-12-22 18:10:42,050", "filename": "trustedadvisor_service.py:47", "level": "ERROR", "module": "trustedadvisor_service", "message": "cn-northwest-1 -- EndpointConnectionError[31]: Could not connect to the endpoint URL: "https://support.cn-northwest-1.amazonaws.com.cn/""}

daftkid on Dec 22, 2022

Hi @sergargar Yep, I can run all of the checks, some of them are still failing but that will be fixed eventually I believe as almost all errors are related to the fact that some services are not available in the Chinese AWS partition.

One of the recent reports:

-> Scan completed! |▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉| 244/244 [100%] in 2:42.3

Overview Results:
╭────────────────────┬────────────────────╮
│ 38.51% (57) Failed │ 59.46% (88) Passed │
╰────────────────────┴────────────────────╯

Account xxxxxxxxxxxx Scan Results (severity columns are for fails only):
╭────────────┬────────────────┬───────────┬────────────┬────────┬──────────┬───────╮
│ Provider   │ Service        │ Status    │   Critical │   High │   Medium │   Low │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ accessanalyzer │ FAIL (2)  │          0 │      0 │        0 │     2 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ account        │ PASS (3)  │          0 │      0 │        0 │     0 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ lambda         │ FAIL (1)  │          0 │      0 │        0 │     1 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ cloudtrail     │ FAIL (6)  │          0 │      0 │        2 │     4 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ cloudwatch     │ FAIL (17) │          0 │      0 │       17 │     0 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ config         │ PASS (2)  │          0 │      0 │        0 │     0 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ ec2            │ FAIL (14) │          0 │      9 │        2 │     3 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ emr            │ PASS (2)  │          0 │      0 │        0 │     0 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ glue           │ FAIL (4)  │          0 │      0 │        4 │     0 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ guardduty      │ PASS (4)  │          0 │      0 │        0 │     0 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ iam            │ FAIL (9)  │          1 │      0 │        8 │     0 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ s3             │ FAIL (1)  │          0 │      1 │        0 │     0 │
├────────────┼────────────────┼───────────┼────────────┼────────┼──────────┼───────┤
│ aws        │ vpc            │ FAIL (3)  │          0 │      0 │        3 │     0 │
╰────────────┴────────────────┴───────────┴────────────┴────────┴──────────┴───────╯
* You only see here those services that contains resources.

And I really like this great performance improvement 😄 (~3 mins instead of 25 with version 2)

daftkid on Dec 22, 2022

@sergargar please see it here: https://gist.github.com/daftkid/52a5a9aefbaa2393245df02faec5d0c6 I executed it via ./prowler.py instead of just prowler

daftkid on Dec 21, 2022