google-play-scraper: Error: Cannot read property 'replace' of undefined

Operating System: Linux Node version: 14.19.2 google-play-scraper version: 8.1.0

Description:

We are using this scraper on a cron once an hour as part of a version control system. Occasionally the below Http Exception is thrown and we are emailed about the failure.

Example code:

This is the code we’re running.

private static async getLatestAppVersionAndroid (): Promise<string> {
  const scrape = await gplay.app({
    appId: config.appIds.android
  });
  return scrape.version;
}

This is the code in the package that appears to be throwing the error:

function descriptionText (description) {
  // preserve the line breaks when converting to text
  const html = cheerio.load('<div>' + description.replace(/<br>/g, '\r\n') + '</div>');
  return cheerio.text(html('div'));
}

It’s as if the scraper fails every now and then and the description field is therefore undefined?

Error message:

"stack": [
    "TypeError: Cannot read property 'replace' of undefined",
    "    at Object.descriptionText [as fun] (/var/www/node2/ms-version-control/node_modules/google-play-scraper/lib/mappers/details.js:125:51)",
    "    at /var/www/node2/ms-version-control/node_modules/google-play-scraper/lib/utils/scriptData.js:52:19",
    "    at XWrap.f (/var/www/node2/ms-version-control/node_modules/ramda/dist/ramda.js:5955:28)",
    "    at XWrap.@@transducer/step (/var/www/node2/ms-version-control/node_modules/ramda/dist/ramda.js:392:25)",
    "    at _arrayReduce (/var/www/node2/ms-version-control/node_modules/ramda/dist/ramda.js:5137:46)",
    "    at _reduce (/var/www/node2/ms-version-control/node_modules/ramda/dist/ramda.js:5167:24)",
    "    at map (/var/www/node2/ms-version-control/node_modules/ramda/dist/ramda.js:5954:20)",
    "    at /var/www/node2/ms-version-control/node_modules/ramda/dist/ramda.js:586:23",
    "    at Object.f2 [as map] (/var/www/node2/ms-version-control/node_modules/ramda/dist/ramda.js:473:22)",
    "    at extractFields (/var/www/node2/ms-version-control/node_modules/google-play-scraper/lib/utils/scriptData.js:40:14)",
    "    at runMicrotasks (<anonymous>)",
    etc. drilling into our files
]

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 16
Comments: 20 (2 by maintainers)

Most upvoted comments

can you check if this still happens with the latest version?

+10

facundoolano on Jun 23, 2022

So i fetched this big weird json from Play Store for the old version and the new version. Then i basically searched the value that was outputted by the scraper in the new json of the Play Store and copied the path. 🙂

My biggest problem was actually that Google didnt fully roll out the changes yet so it kept breaking depending which version the scraper got. Therefore i added a detection if it’s the new design or the old design. The scraper automatically switches then between the different mappings.

pvomhoff on May 26, 2022

Same thing happening to me. I was able to fix the mapping by completely redo it. Unfortunaly Google doesnt provide all properties as before.

I created a PR with my changes. Hope this helps.

pvomhoff on May 26, 2022

Any update? It’s been a week.

lau1944 on Jun 4, 2022

@pvomhoff I would like to help with the mapping of search and lists…

I wrote a pair of scripts which helped me to build my mappings json-inspect.zip

xshagg on Jun 2, 2022

@facundoolano @icarcal Anything you can do about this? I tried taking a look at the mappings myself but I can’t make any sense of it.

Davies-Owen on May 24, 2022