app-store-scraper: Apps position is not the same showed in the iTunes Store

Hello there,

I’m using the module to search for some apps, but the result returned is not the same that I get on the iTunes app (Yes, I had set the country correctly) which works perfectly in the google-play-scraper module. As I’m using it for ASO purposes, it’s critical to know the real apps positions in the iTunes. (For example, I want to know if the app which i’m optimizing in ranking better, if it’s in the first position for a given word…)

Talking to a friend of mine who works with ASO daily, he told me that the iTunes API is not reliable when it comes to the apps positioning (maybe for a delay to update, idk, but it isn’t the same list) and another solution would be necessary.

Had you guys have the same problems or notice something like this? Or am I the only one?

Actually, if it’s really a problem, I’m considering a solution: I’ve found out using a Sniffer (like Wireshark), to monitor the network when I’m using the iTunes app, and the app do requests to another url, setting different headers to fetch the content to be presented on it…

Please, let me know what you guys think about it.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 23 (13 by maintainers)

Most upvoted comments

Ops, I’m sorry. Got the wrong endpoint.

Check it:

curl -X GET \
  'https://itunes.apple.com/WebObjects/MZStore.woa/wa/search?submit=edit&mt=1&term=pandas' \
  -H 'accept-language: en-us' \
  -H 'x-apple-i-md: AAAABQAAABASYMVHFwrAt3wPSUwbTCqGAAAAAg==' \
  -H 'x-apple-i-md-m: hxpW6pCpiMIBZ+ch0fA6Hjke7zRLyqe5KdegwA9TzU2aDGPWCL1CX85fi+s4dFX4yDC58Owm+DJImUbI' \
  -H 'x-apple-store-front: 143441-1,32 ab:IMP0' \
  -H 'x-apple-tz: -7200'

It’ll return a HTML file. A bit confusing.

But, search a for a <script> tag that has its.serverData. You’ll notice that it’s a JSON. Get it.

Access its property pageData.searchPageData.bubbles. It’ll bring an array that aggregates data by its kind. The index 3 bubbles[3], whose name proprety is ‘software’, has a property results which contains 116 results.

Each item of array has an id property. its the app id.

The most important here: these results array in are “ordered correctly”, which means that it brings the same apps order that I get when a search for something in my iPhone or iTunes APP, which is not true when using iTunes API.

Hope it’ll help.

icaroponce on May 4, 2017

Sure,

here an example of url the app hits:

https://search.itunes.apple.com/WebObjects/MZSearchHints.woa/wa/hints?q=panda

Just sending a get request to it, you’ll get a XML file.

You can set params to it through the header:

{"Accept-Language":"pt-br"}
{"X-Apple-Store-Front":"143503-15,32"}
{"X-Apple-I-MD-RINFO":"17106176"}
{"X-Apple-I-MD-M":"5qf7Ex9VvAxuH98/nDvbIHqW76fYyHTpVEhcWC2tBT8FKc7zqErTS9/kmJlqWXGSfPgcdl831PNkbddc"}
{"X-Apple-I-MD":"AAAABQAAABCDXFWHMZGdT6C1K2IFjB00AAAAAg=="}

These last two tokens represent language and country.

I’m quite busy this week, but if you can’t check it yourself during the week, I could dig it deeper on the weekend.

But, all we need is to turn on the sniffer and watch the requests behavior to understand all the params sent on header.

I know it’s a more verbose way to do it, but if really there’s difference in the api, this way seems to be the most accurate way to retrieve data for some methods, like search for example.

icaroponce on May 3, 2017

@facundoolano did you find a way to filter by device (ipad/iphone)?

DataGreed on Apr 23, 2019