serpbear: All keywords dropped to 0 - serpbear unable to query google results?

Environment

  • Serpbear: v2.0.6 in Docker
  • Scraper: I tried my usual scraping robot config as well as a proxy configuration for testing

Problem Description

Since about 2 days, my serpbear instance is unable to crawl any keywords, which results in the position dropping to 0.

The log shows no issues on keyword update:

[0] POST /api/refresh?id=5
[0] keywordIDs:  [ 5 ]
[0] START SCRAPE:  mykeyword
[0] GET /api/keywords?domain=mydomain.de
[0] GET /api/keyword?id=5
[0] [SERP]:  mykeyword 0 
[0] [SUCCESS] Updating the Keyword:  mykeyword
[0] time taken: 1929.1866679999512ms

Nevertheless, the search results stay empty:

Image

Issues with that

  • If no search results are gathered at all, serpbear should probably return an error instead of changing position to 0
  • It seems to me, that this is a parsing issue in serpbear as the domain clearly shows up in the search results when manually searching with a browser or curl using the same proxy I tried with serpbear.

Any help would be greatly appreciated 😊

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (3 by maintainers)

Most upvoted comments

i have the Same issue too. @phoehnel thanks a lot.

A quick note for those who may have a similar problem to mine: before Google changed the results, I updated Serpbear to version 2.0.6. Using Scraping Robot until yesterday, the keyword ranking no longer worked for me. After several tests, I discovered that before version 2.0.6, my domains were configured as “domainname.com” (but appeared in Google SERPs as www.domainname.com). After upgrading to version 2.0.6 (and therefore to version 2.0.7), Serpbear could no longer detect the keyword position with any of the scraping services I tested. I solved the problem by adding a new domain (and importing the keywords), this time with the ‘www’ as it appears in Google SERPs. Using Scraping Robot, everything worked again.

This did the trick.

A quick note for those who may have a similar problem to mine: before Google changed the results, I updated Serpbear to version 2.0.6. Using Scraping Robot until yesterday, the keyword ranking no longer worked for me. After several tests, I discovered that before version 2.0.6, my domains were configured as “domainname.com” (but appeared in Google SERPs as www.domainname.com). After upgrading to version 2.0.6 (and therefore to version 2.0.7), Serpbear could no longer detect the keyword position with any of the scraping services I tested. I solved the problem by adding a new domain (and importing the keywords), this time with the ‘www’ as it appears in Google SERPs. Using Scraping Robot, everything worked again.

For me, scraping robot now does mostly work again.

From my around 70 monitored keywords, around 10 different ones fail each day, which is still usable for me.

I set the “Keyword Scrape Delay” in the Settings to 2 Minutes, which made it better. Also, I am still using my fork of serpbear (for which PR #273 is there) which prevents the keywords from dropping to 0 on failures to not mess with the history and prevent me waking up to “70 keywords declined to 0” mails from serpbear 😆

If you want to do use the fork, you can e.g. in docker-compose:

  serpbear:
    # image: towfiqi/serpbear
    build: https://github.com/phoehnel/serpbear.git
    container_name: serpbear
    ports:
      - "3000:3000"
    environment:
      - USER=admin
      - PASSWORD=CHANGEME
      - SECRET=CHANGEME
      - APIKEY=CHANGEME
      - NEXT_PUBLIC_APP_URL=http://192.168.0.3:800
      - SCREENSHOT_API=CHANGEME
    volumes:
      - "/my_data/serpbear_data/:/app/data"

It looks like Google made scraping search results harder and more expensive. https://searchengineland.com/is-google-rank-tracking-dead-451606

Are you checking Desktop keywords or Mobile keywords? In my case I tried both, and none are working. This is the message I’m getting: scrapingrobot: Unknown Error. Just in case I have checked if the API key is not correct but it is

@iwywnsb Desktop and Mobile keywords are working. i know it shouldn’t matter but try to add keyword Scrape Delay and restart the container if you are sure from your API key is correct.

In my case, I have restarted the machine and I have configured delay of 2 minutes and retry when they fail and as I said it fails randomly in the searches. I am going to leave several tests on what usually happens, which as I said is more frequent in mobile version.

1Âș It has worked more or less, but it has only extracted the first 11 results.

Image

2Âș In spite of giving error (scrapingrobot: Unknown Error), it has extracted the 100 results and has extracted the result well.

Image

3Âș It has extracted all the html as a same block, and only counts the first url. If you are not in first position it does not find you and says >100.

Image

I hope it helps Thanks

Scraping Robot is working for me now

I looked into it and it seems like there is a new bot-prevention popup that serpbear is unable to overcome or detect. As SerpAPI directly returns the parsed positions and serpbear does no HTML parsing itself, it works with that. For the same reason it should also work with SearchAPI, Serper, ValueSerp, hasdata, serply and spaceserp.

I filed #273, which checks if the returned HTML contains valid results and returns an error if not. This way, the position in serpbear will at least not drop to 0 but show an error instead.

Image

Same issue here. I switched the scraping method from Scraping Robot to SerpApi, and it worked again.