cartography: Error while enabling GSuite integration
Issue: The documentation for setting up the GSuite integration is incomplete and leads towards a 400 error
Description: Following the README, I:
- Enabled Google API access (instructions)
- Created a new G Suite user account and accept the Terms of Service (this account will be used as the domain-wide delegated access).
- Performed G Suite Domain-Wide Delegation of Authority, as explained here
- Downloaded the service account’s credentials
- Setup env vars for cartography
GSUITE_GOOGLE_APPLICATION_CREDENTIALS - location of the credentials file.
GSUITE_DELEGATED_ADMIN - email address that you created in step 2
After this, Cartography crashes with the following output:
cartography --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password-env-var NEO4J_PASSWORD_ENV_VAR
INFO:cartography.sync:Starting sync with update tag '1580915718'
INFO:cartography.sync:Starting sync stage 'create-indexes'
INFO:cartography.intel.create_indexes:Creating indexes for cartography node types.
INFO:cartography.sync:Finishing sync stage 'create-indexes'
INFO:cartography.sync:Starting sync stage 'aws'
...
INFO:cartography.sync:Finishing sync stage 'aws'
INFO:cartography.sync:Starting sync stage 'gcp'
...
INFO:cartography.sync:Finishing sync stage 'gcp'
INFO:cartography.sync:Starting sync stage 'gsuite'
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
ERROR:cartography.sync:Unhandled exception during sync stage 'gsuite'
Traceback (most recent call last):
File "/app/cartography/sync.py", line 69, in run
stage_func(neo4j_session, config)
File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
resp_objs = get_all_users(admin)
File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=my_customer&maxResults=500&orderBy=email&alt=json returned "Invalid Input">
Traceback (most recent call last):
File "/usr/local/bin/cartography", line 11, in <module>
load_entry_point('cartography', 'console_scripts', 'cartography')()
File "/app/cartography/cli.py", line 240, in main
return CLI(default_sync, prog='cartography').main(argv)
File "/app/cartography/cli.py", line 220, in main
return cartography.sync.run_with_config(self.sync, config)
File "/app/cartography/sync.py", line 135, in run_with_config
return sync.run(neo4j_driver, config)
File "/app/cartography/sync.py", line 69, in run
stage_func(neo4j_session, config)
File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
resp_objs = get_all_users(admin)
File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=my_customer&maxResults=500&orderBy=email&alt=json returned "Invalid Input">
I started troubleshooting this and I found the string my_customer is hardcoded in cartography/intel/gsuite/api.py:
request = admin.users().list(customer='my_customer', maxResults=500, orderBy='email')
I decided to replace it with the customerId of my GSuite org, and then I faced a 403 - Not Authorized error:
cartography --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password-env-var NEO4J_PASSWORD_ENV_VAR
INFO:cartography.sync:Starting sync with update tag '1580916464'
INFO:cartography.sync:Starting sync stage 'create-indexes'
INFO:cartography.intel.create_indexes:Creating indexes for cartography node types.
INFO:cartography.sync:Finishing sync stage 'create-indexes'
INFO:cartography.sync:Starting sync stage 'aws'
...
INFO:cartography.sync:Finishing sync stage 'aws'
INFO:cartography.sync:Starting sync stage 'gcp'
...
INFO:cartography.sync:Finishing sync stage 'gcp'
INFO:cartography.sync:Starting sync stage 'gsuite'
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
WARNING:googleapiclient.http:Encountered 403 Forbidden with reason "forbidden"
ERROR:cartography.sync:Unhandled exception during sync stage 'gsuite'
Traceback (most recent call last):
File "/app/cartography/sync.py", line 69, in run
stage_func(neo4j_session, config)
File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
resp_objs = get_all_users(admin)
File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=<REDACTED>&maxResults=500&orderBy=email&alt=json returned "Not Authorized to access this resource/api">
Traceback (most recent call last):
File "/usr/local/bin/cartography", line 11, in <module>
load_entry_point('cartography', 'console_scripts', 'cartography')()
File "/app/cartography/cli.py", line 240, in main
return CLI(default_sync, prog='cartography').main(argv)
File "/app/cartography/cli.py", line 220, in main
return cartography.sync.run_with_config(self.sync, config)
File "/app/cartography/sync.py", line 135, in run_with_config
return sync.run(neo4j_driver, config)
File "/app/cartography/sync.py", line 69, in run
stage_func(neo4j_session, config)
File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
resp_objs = get_all_users(admin)
File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=<REDACTED>&maxResults=500&orderBy=email&alt=json returned "Not Authorized to access this resource/api">
I tried to add more scopes to the service account (listed below), but I still get Not Authorized:
- https://www.googleapis.com/auth/admin.directory.domain.readonly
- https://www.googleapis.com/auth/admin.directory.group.member.readonly
- https://www.googleapis.com/auth/admin.directory.user.readonly
Please complete the following information::
- Cartography release version or commit hash:
801bb58 - Python version:
3.7.5 - OS:
python:3.7docker image
About this issue
- Original URL
- State: open
- Created 4 years ago
- Comments: 17 (12 by maintainers)
Hi @achantavy I do get the same error as @marco-lancini, I’ve also tried to build the GSuite section by myself according to this guide: https://developers.google.com/admin-sdk/directory/v1/guides/delegation same result 😦