cartography: Error while enabling GSuite integration

Issue: The documentation for setting up the GSuite integration is incomplete and leads towards a 400 error

Description: Following the README, I:

  1. Enabled Google API access (instructions)
  2. Created a new G Suite user account and accept the Terms of Service (this account will be used as the domain-wide delegated access).
  3. Performed G Suite Domain-Wide Delegation of Authority, as explained here
  4. Downloaded the service account’s credentials
  5. Setup env vars for cartography
GSUITE_GOOGLE_APPLICATION_CREDENTIALS - location of the credentials file.
GSUITE_DELEGATED_ADMIN - email address that you created in step 2

After this, Cartography crashes with the following output:

cartography --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password-env-var NEO4J_PASSWORD_ENV_VAR
INFO:cartography.sync:Starting sync with update tag '1580915718'
INFO:cartography.sync:Starting sync stage 'create-indexes'
INFO:cartography.intel.create_indexes:Creating indexes for cartography node types.
INFO:cartography.sync:Finishing sync stage 'create-indexes'
INFO:cartography.sync:Starting sync stage 'aws'
...
INFO:cartography.sync:Finishing sync stage 'aws'
INFO:cartography.sync:Starting sync stage 'gcp'
...
INFO:cartography.sync:Finishing sync stage 'gcp'
INFO:cartography.sync:Starting sync stage 'gsuite'
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
ERROR:cartography.sync:Unhandled exception during sync stage 'gsuite'
Traceback (most recent call last):
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=my_customer&maxResults=500&orderBy=email&alt=json returned "Invalid Input">
Traceback (most recent call last):
  File "/usr/local/bin/cartography", line 11, in <module>
    load_entry_point('cartography', 'console_scripts', 'cartography')()
  File "/app/cartography/cli.py", line 240, in main
    return CLI(default_sync, prog='cartography').main(argv)
  File "/app/cartography/cli.py", line 220, in main
    return cartography.sync.run_with_config(self.sync, config)
  File "/app/cartography/sync.py", line 135, in run_with_config
    return sync.run(neo4j_driver, config)
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=my_customer&maxResults=500&orderBy=email&alt=json returned "Invalid Input">

I started troubleshooting this and I found the string my_customer is hardcoded in cartography/intel/gsuite/api.py:

request = admin.users().list(customer='my_customer', maxResults=500, orderBy='email')

I decided to replace it with the customerId of my GSuite org, and then I faced a 403 - Not Authorized error:

cartography --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password-env-var NEO4J_PASSWORD_ENV_VAR
INFO:cartography.sync:Starting sync with update tag '1580916464'
INFO:cartography.sync:Starting sync stage 'create-indexes'
INFO:cartography.intel.create_indexes:Creating indexes for cartography node types.
INFO:cartography.sync:Finishing sync stage 'create-indexes'
INFO:cartography.sync:Starting sync stage 'aws'
...
INFO:cartography.sync:Finishing sync stage 'aws'
INFO:cartography.sync:Starting sync stage 'gcp'
...
INFO:cartography.sync:Finishing sync stage 'gcp'
INFO:cartography.sync:Starting sync stage 'gsuite'
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
WARNING:googleapiclient.http:Encountered 403 Forbidden with reason "forbidden"
ERROR:cartography.sync:Unhandled exception during sync stage 'gsuite'
Traceback (most recent call last):
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=<REDACTED>&maxResults=500&orderBy=email&alt=json returned "Not Authorized to access this resource/api">
Traceback (most recent call last):
  File "/usr/local/bin/cartography", line 11, in <module>
    load_entry_point('cartography', 'console_scripts', 'cartography')()
  File "/app/cartography/cli.py", line 240, in main
    return CLI(default_sync, prog='cartography').main(argv)
  File "/app/cartography/cli.py", line 220, in main
    return cartography.sync.run_with_config(self.sync, config)
  File "/app/cartography/sync.py", line 135, in run_with_config
    return sync.run(neo4j_driver, config)
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=<REDACTED>&maxResults=500&orderBy=email&alt=json returned "Not Authorized to access this resource/api">

I tried to add more scopes to the service account (listed below), but I still get Not Authorized:

  1. https://www.googleapis.com/auth/admin.directory.domain.readonly
  2. https://www.googleapis.com/auth/admin.directory.group.member.readonly
  3. https://www.googleapis.com/auth/admin.directory.user.readonly

Please complete the following information::

  • Cartography release version or commit hash: 801bb58
  • Python version: 3.7.5
  • OS: python:3.7 docker image

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Comments: 17 (12 by maintainers)

Most upvoted comments

Hi @achantavy I do get the same error as @marco-lancini, I’ve also tried to build the GSuite section by myself according to this guide: https://developers.google.com/admin-sdk/directory/v1/guides/delegation same result 😦