TwitchDropsMiner: Application doesn't work anymore

Apparently it could be because the Client-Version header changes very frequently. Not sure how frequent it is, but right now, it’s only fetched once and stored for the entire application run duration. If it’s more often than 24 hours (which it probably is), I’ll need to add a way to refresh this value every so often.

I’ll try to look at this later today.

@josephisticated The problem is, that’s not the only thing that’s broken - every GQL operation is. One of the most basic and important GQL operations is fetching the current drops inventory, which powers the entire channel fetching, selection and watching logic. Without that, you won’t get the channel list populated, and thus nothing will work.

I’m working on a solution which I’m 80-90% into finishing by now, tired to binge it yesterday. Here’s everything I’ve done so far: https://github.com/DevilXD/TwitchDropsMiner/commit/01729761d8efb3d6824bec7f334fa5ad86dc9b04

I’ve managed to track down this: https://k.twitchcdn.net/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/p.js

Not sure how long is this URL going to remain valid, but it looks to me like some kind of obfuscated, self-modifying JavaScript code. Here’s some non-obfuscated keywords from it:

'workTime'
_unhandledRejectionFn
encodedClientToken
"fantasy-land/empty"
FrontendErrorCode.JsSdkInterceptXhrError
interceptForms
FrontendErrorCode
defineProperty
prototype
formHasTag
tagForm
createHiddenInputElement(..., "submit")
addEventListener
getSolvedChallengeIfNeeded
getSolvedChallengeIfNeededSync
requestChallengeSync
getInterceptDataSync
interceptConfig
JS_SHA256_NO_NODE_JS
JS_SHA256_NO_COMMON_JS
requestMatchesEndpointDefinition
iframeManager
cryptoChallengeEnabled
checkAllDomainsReady
document.dispatchEvent
managedIFrames
addServerOffset
parseSDKMessage
platformInputs
decorateFormSyncIfPossible
getActionAndMethodFromForm
interceptSubmitFunctionSync
addInterceptDataToForm
FIELD_NAME_CHALLENGE_DATA
"@@transducer/init"
"__esModule"

These mentions of “challenge” point out to me that this is probably the new “entry point” when it comes to both GQL and passport authentication. The result is a new header that needs to be added when requesting an integrity token:

x-kpsdk-cd: {"workTime":1663952254845,"id":"...","answers":[5,4],"duration":11,"d":-4692,"st":1663962234862,"rst":1663962230179}

Trying to request an integrity token without this header, or even something as simple as re-sending the same request a second time, causes the returned token to contain the is_bad_bot flag set to "true", which can’t be used for mining.

Since I’m not a JS programmer, going anywhere with this is quite a lost cause. Simpler obfuscated code can usually be bypassed by just running it and recording the output, but this one seems to use iFrames and hidden forms, which require a typical browser environment. De-obfuscation is out of question since it’d take a really long time and everything would be rendered worthless the moment Twitch would decide to use a different p.js file, which the URL appears to heavily suggest.

I’ll need to think heavily about what to do in this situation, but it may mean the end of this project. There’s nothing much any of us can do, besides waiting and playing this game by Twitch’s rules for now.

I have a semi-working version that you should be able to use for the time being - please do note that I want to release a proper v13 as soon as I will be able to iron out some bugs with the current solution, as it still seems to end up with this error sometimes. Currently waiting for a couple of drop-claim cases, with GQL debug and log enabled, to see what exactly is returned that causes the error. If you would encounter it while testing too, please leave a comment with the traceback and information if you had any successful claims on that particular application run, or did it break on the first claim it tried.

Download moved: https://github.com/DevilXD/TwitchDropsMiner/issues/40#issuecomment-1250123813

EDIT: Oh, and it’s possible you can get a KeyError: unique_id of some sorts, which will happen if your cookie file is missing one. Had this happen to me during testing. If so, just delete the cookie file and login again - the login flow has been rearranged a little so that it’ll fill up that value in the cookie file before it’s used, which should prevent this error from happening. A note about this behavior should find it’s way into the eventual v13 release too.

@josephisticated I did mention above that the mining operation itself has not been impaired, because it doesn’t interact with GQL - however the points claiming and drop claiming does. I don’t see much reason to run this application vs just open a stream in your browser, if it’s not going to claim anything automatically anyway. There is at least two other “miners” out there, that can do what this application does just as well, but it’s going to keep trying to redeem points and drops, spamming Twitch with errors - and when that happens, usually developers bring out the ban hammer and block your IP from bothering their website.

These are only my words of precaution. I could just remove the autoclaiming feature in literally 10 minutes, but it’d just stop mining and get stuck on a single channel without being able to claim a drop, doing literally nothing. You’re free to make a local copy and edit it to your own needs, if you know how - but again, I see no point if you can just open a web page to have the same functionality.

Besides, with the passport endpoint being protected with the same system as GQL, getting some auth information so that the watching would be even possible, is impossible now as well. There would need to be a separate system added for handling that instead. like hooking up to your browser and stealing the cookie from there - however, I always found that solution quite messy and easy to break, hence why I never even considered implementing it in my application.

I’m still in the process of re-thinking where exactly I can go from here, so please have patience. If you really cannot wait, you can try using other miners out there for now - not like I could stop you anyway. Just keep in mind what I said above, you’re doing so at your own risk.

Got one! 👍 Ran almost 24 hours.

[...]
No available channels to watch. Waiting for an ONLINE channel...
Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 418, in run
  File "twitch.py", line 465, in _run
  File "twitch.py", line 1093, in fetch_inventory
  File "twitch.py", line 1039, in gql_request
  File "twitch.py", line 981, in get_auth
  File "twitch.py", line 198, in validate
  File "twitch.py", line 279, in _validate
exceptions.MinerException: Twitch considers this miner as a "Bad Bot". Try deleting the cookie file and try again.

Exiting...
2022-09-21 22:32:44.794:	   INFO:	Websocket[0] stopped.

Application Terminated.
Close the window to exit the application.

Deleting the cookies.jar and a fresh login will show the samce traceback.

Not working anymore for me 🤔

I knew this was probably gonna happen eventually. The integrity token is actually some base64-encoded JSON with some trash at the end, which since I knew about it, I added some double-checking code as a precausion.

Code: https://github.com/DevilXD/TwitchDropsMiner/blob/53d232daa98177313fd8f6f2c9eead58207e89bf/twitch.py#L269-L282

The decoded token:

{
    "client_id":"kimne78kx3ncx6brgo4mv6wki5h1ko",
    "client_ip":"...",
    "device_id":"...",
    "exp":"2022-09-22T13:03:01Z",
    "iat":"2022-09-22T05:03:01Z",
    "is_bad_bot":"false",
    "iss":"Twitch Client Integrity",
    "nbf":"2022-09-22T05:03:01Z",
    "user_id":"94275183"
}

As you can see, there’s this "is_bad_bot":"false" flag there, which ends up being "true" when the miner requests an integrity token (this one is from a browser), which means that Twitch actively knows that it’s an automated script accessing the API, not a browser. The point is that, to decode this information, an integrity token has to be returned - so it’s technically possible to still make it mine using this token, you’ll just be actively letting Twitch know that there’s a bot accessing your account, which may end up badly. The current logic throws an exception if this flag is detected (per the code above), to prevent you from mining with it as an account safety precaution.

This “grounds” this project yet again, until I can figure out how to make it "false". If you’re brave enough, you can find the above code, and comment it out to disable the check, but you’re doing so at your own risk. I’m not even sure if Twitch won’t reject GQL requests when that flag is set, and I don’t really want to try and find that out myself.

Version v13 has been released, which fixes this issue. Enjoy =)

If the mining is still working, you could think about deactivating the Claimer for now and just let it do the mining. The user needs to claim it himself, right, but doesn’t need to keep full track of whether a stream started (for i.e. a 1day / few hours drop).

Just “deactivating” the claiming isn’t trivial, as I’ve already mentioned before. I could do it as a “for now” dev release just like I did earlier, in an attempt to go anywhere with this project. The thing is, the authorization endpoint is still kinda dead, so even that might refuse to work properly.

I’ve tried launching the miner today and it seemed to have passed the auth token verification - which might mean the passport endpoint isn’t “quite dead” after all. I might actually try doing this and see how far I can salvage the current functionality - but testing anything at this point is quite risky, even for myself. This “kasada” thing, or whatever it’s called, appears to be on-par (or greater) with the CAPTCHA solving challenge, which is why the CAPTCHA exception exists in the application. Trying to defeat this is really hard, but theorethically possible given enough time, dedication and JS knowledge, none of which I really have. I kept maintaining this project with small steps along the way, because I liked doing so to keep it fresh and going. Having to rewrite the majority of it because of something like this is quite painful in itself, I really liked what I’ve made so far.

Anyway, you can expect some kind of a new dev version in the nearest future. I’m currently quite busy IRL and what Twitch did has turned me off for this project quite a bit, but I’ll still try to go somewhere with it, rather than simply abandoning it.

Then I doubt you’ll be able to get the 6th dev version working, even after these changes are made and I’m testing. And this has nothing to do with the application itself, because as I said, it stopped happening to me. Twitch can just randomly flag your account for an “extra security check” and redirect your login to the /protected_login endpoint, that this application cannot handle, and never will, because it’d be equivalent to trying to beat CAPTCHA (which is also unhandled, by design).

Besides, all v13 dev versions, from the 1st to the 5th one, shouldn’t be used - otherwise you’re just asking Twitch to flag your account. My recommendation is: don’t run it, and then complain it doesn’t work. It doesn’t work for anyone, even after you’d somehow get past the login phase. I have a 6th version that appears to be working for now, but I give exactly zero guarantee it’ll let you through the login phase once you get it. It’s not up to me to decide, as I said.

@DevilXD Thanks for your reply, I totally missed the answer (because a few days prior it was working fine). But it seems twitch seems to actively fight bots (additionally to adblockers) now. There is a repo just trying to solve the integrity and the dev too advises not do use anything like that right now https://github.com/Kappador/twitch-integrity and one user guessed there that they implemented kasada antibot protection (mentioned elsewhere, too)

Twitch Channel Point Miner doesn’t work either and this is one of the applications spamming the twitch API. But you can disable that.

If the mining is still working, you could think about deactivating the Claimer for now and just let it do the mining. The user needs to claim it himself, right, but doesn’t need to keep full track of whether a stream started (for i.e. a 1day / few hours drop).

@hrmlsr Your v12 is going to stop working soon, since the validation endpoint keeps clearing everyone’s cookies, and the login endpoint no longer exists. Without the login endpoint, there’s no way for me to create or recreate the cookie file, and without that, you can’t authenticate with Twitch or use the miner. Surprisingly, the mere “watching” a streamer action remains untouched, since that interacts with Spade and not GQL, but claiming bonus points or claiming drops does, and is thus no longer possible.

This isn’t some kind of an “issue” with v13, it’s just how Twitch decided to change things. v12 does not have any of the additional checks v13 does, so it’s going to try claiming bonus points or claiming a drop, fail miserably, and then Twitch will invalidate your authentication token, leading to you getting the same error as the guy above next time you start the miner.

I know where the issue is, but now I need time. Lots of it. There’s at least two ways this project can go forward with, but it’s going to be a heavy decision to make, and thus I want to try some things before I do so. For now, you’ll have to go back to mining the usual way. I’m sorry but I can’t change this.

v12 no

v13.dev.5 - done like I said so. The error is now turned into a non-fatal output message, which is also easy to miss - but I’m hopeful you’re still gonna help me by providing whatever it outputs from time to time. It’d be really cool to have a couple instances of it, before proper v13 release.

Download has been removed, because this version doesn’t work anymore. See the releases page for the latest official release, or check out the comments below for any additional dev version.

Last commit: https://github.com/DevilXD/TwitchDropsMiner/commit/53d232daa98177313fd8f6f2c9eead58207e89bf

There’s something funky going on. It can be either Twitch being unhappy, or yet another bug (which I doubt cos there’s already so many checks in place), but it can also be something silly, like I describe below.

request =  2022-09-20 03:13:16.397231
response = 2022-09-20 03:13:16.687202
now =      2022-09-20 03:57:15.416353
expires =  2022-09-20 10:25:16.711320

request and response are when the integrity token got requested, received and stored. now was the time at which the error happened. expires is the time the stored token was supposed to expire at, yet it’s already invalid.

No available channels to watch. Waiting for an ONLINE channel…
Traceback (most recent call last): …

This reminds me of a possibility of a system I saw at another API, where a special session token would expire prematurely, if there was no request using it in the past 15 minutes. Essentially, you could start a session, get a token, and then keep using it for literal days, but as soon as you stopped using it for more than 15 minutes, it’d expire and any request trying to use it from then on would error out. The solution was to just start another session when this happens, of course.

There actually might be a system like this here. When the application is waiting for something to happen, it doesn’t do any GQL requests until the maintenance task reloads the inventory (up to once per hour). This would explain many people getting the error on their first claim, if they started the application way beforehand, where the expiration time wasn’t over, but the token wasn’t used so it just expired shortly after, and the next claim trying to use it caused an error. It’s weird though, because the maintenance task also ensures no unclaimed points and does a GQL request to do so, which would still use the token every ~15-30 minutes at most. It seems like that could already be too long and the token may expire anyway.

I have an ultimate way of dealing with this though, already setup in the application - the GQL request can just obtain a new token once the current one is detected to be expired. It’s also already in the code, just needs to be uncommented: https://github.com/DevilXD/TwitchDropsMiner/blob/f93da9349393b37e4836313d1657d2ab48c9cc53/twitch.py#L1052-L1068 I changed the logic around so that the entire application won’t get reloaded for a silly reason like this. This should deal with the issue for good. I’m making this a thing for the upcoming v13.dev.5 to test.

I just saw I got an error with v13.dev.4

[...]
No available channels to watch. Waiting for an ONLINE channel...
Earned points for watching:  10, total: 10
2022-09-20 05:57:15.416:	  ERROR:	Exception in task
Traceback (most recent call last):
  File "utils.py", line 79, in wrapper
  File "twitch.py", line 978, in process_points
  File "twitch.py", line 1195, in claim_points
  File "twitch.py", line 1061, in gql_request
exceptions.MinerException: GQL integrity error: now=2022-09-20 03:57:15.416353, request=2022-09-20 03:13:16.397231, response=2022-09-20 03:13:16.687202, expires=2022-09-20 10:25:16.711320
Earned points for watching:  10, total: 40
Claimed drop: Ouroboros Crown Crate (1/1)
No available channels to watch. Waiting for an ONLINE channel...

But it works, it stil idles without crash 😃 The app now ran easily 24h

Thanks 👍

@DevilXD my fault,my test account didnt link to any platform so that no any stream i can join to get drops

@ThisIsCyreX Also, if you’d be interested, regarding this:

Nothing in the log except Response 200, keep-alive

Try it with a --debug-gql argument to see the GQL errors returned as responses.

It appears that Twitch has made a change to their web client:

The claiming operation now requires a bunch of new headers:

There’s a new GQL endpoint associated to obtaining the Client-Integrity value, which I’m assuming is a part of the application state now (just like the authorization token): https://gql.twitch.tv/integrity It probably comes with it’s own set of headers that all need to be send in order to pass the integrity check, like the X-Device-ID: cookies.unique_id header I just found in one of the JS files.

Unfortunately, all of this means that there needs to be a bunch of more additional logic designed and implemented, for which I have to find the time for. There is also a possibility that during the design process, I’ll discover a captcha-like mechanism that could kill this project - let’s hope it’s just a bunch of new hops one needs to go through and that’s it.

For now, I recommend mining the drops the usual way. I’ll look into this as soon as I’ll be able to.

EDIT: More information about the /integrity endpoint: https://github.com/mauricew/twitch-graphql-api#integrity