node-twitter-api-v2: Looking for a proper way to handle connection errors.
Hello folks, thanks for developing and maintaining this library. I have a question, or two questions actually if you don’t mind.
Right now I am running a bot that is connected to a Twitter stream, which listens for tweets from an array of users, and then I do some fancy stuff with it. The problem is, every once in a while, Twitter throws a 429 error at me, maybe once every 24-48 hrs, and as a result of that I get an unhandledRejection error, and then the Stream no longer works without manually restarting the bot.
app[worker.1]: Error: Request failed with code 429
app[worker.1]: at RequestHandlerHelper.createResponseError (/app/node_modules/twitter-api-v2/dist/client-mixins/request-handler.helper.js:73:16)
app[worker.1]: at IncomingMessage.<anonymous> (/app/node_modules/twitter-api-v2/dist/client-mixins/request-handler.helper.js:114:33)
app[worker.1]: at IncomingMessage.emit (node:events:402:35)
app[worker.1]: at endReadableNT (node:internal/streams/readable:1343:12)
app[worker.1]: at processTicksAndRejections (node:internal/process/task_queues:83:21)
Here is a simplified version of my code:
class _Twitter {
constructor({ streamName }) {
this.streamName = streamName;
}
async initSources() {
const sourceList = [
'Barca_Buzz', 'barcacentre', 'BarcaTimes', 'BarcaUniversal', 'FCBarcelona',
'FCBarcelonaFl', 'GSpanishFN', 'infosfcb', 'LaSenyera', 'ReshadRahman_',
];
const streamRules = await this.client.v2.streamRules();
// Cleanup all existing rules upon initiating
if (streamRules?.data?.length > 0) {
await this.client.v2.updateStreamRules({
delete: {
ids: streamRules.data.map(rule => rule.id),
},
});
}
await this.client.v2.updateStreamRules({
add: sourceList.map(source => ({
value: `(from:${source}) -is:retweet -is:reply`,
tag: source.name
}))
});
}
async stream() {
try {
this.client = new TwitterApi(TwitterBearerToken);
await this.initSources();
this.stream = await this.client.v2.searchStream(
{
"tweet.fields": ['id', 'text', 'created_at', 'entities'],
"user.fields": ['username', 'name', 'profile_image_url'],
'media.fields': ['preview_image_url', 'url', 'width', 'height', 'type'],
'expansions': ['author_id', 'attachments.media_keys']
}
);
this.stream.autoReconnect = true;
this.stream.autoReconnectRetries = 999;
// Listen for all possible events (for debugging purposes);
this.stream.on(ETwitterStreamEvent.Data, async (data) => {
console.log(`Event: Data (${data.includes.users[0].username})\n---`);
// do something with the tweet here
});
this.stream.on(ETwitterStreamEvent.ConnectionClosed, async (data) => {
console.log(`Event: ConnectionClosed${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
this.stream.on(ETwitterStreamEvent.ConnectionError, async (data) => {
console.log(`Event: ConnectionError${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
this.stream.on(ETwitterStreamEvent.ConnectionLost, async (data) => {
console.log(`Event: ConnectionLost${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
this.stream.on(ETwitterStreamEvent.DataError, async (data) => {
console.log(`Event: DataError${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
this.stream.on(ETwitterStreamEvent.DataKeepAlive, async (data) => {
console.log(`Event: DataKeepAlive${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
this.stream.on(ETwitterStreamEvent.Error, async (data) => {
console.log(`Event: Error${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
this.stream.on(ETwitterStreamEvent.ReconnectAttempt, async (data) => {
console.log(`Event: ReconnectAttempt${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
this.stream.on(ETwitterStreamEvent.ReconnectLimitExceeded, async (data) => {
console.log(`Event: ReconnectLimitExceeded${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
this.stream.on(ETwitterStreamEvent.Reconnected, async (data) => {
console.log(`Event: Reconnected${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
this.stream.on(ETwitterStreamEvent.TweetParseError, async (data) => {
console.log(`Event: TweetParseError${data ? `\n${JSON.stringify(data, null, 4)}\n` : '\n'}---`);
});
} catch (error) {
console.log(this.streamName, 'error');
console.log(error);
if (error.code = 429) {
await this.stream.reconnect();
}
}
}
}
const Twitter1 = new _Twitter({ streamName: 'Stream 1' });
(async () => {
await Twitter1.stream();
})();
So as you can see, I am creating a _Twitter class, and then creating an instance of that class to connect to the Streaming API. This all works fine, except that only the Data and DataKeepAlive events are emitting properly, and the rest goes into my catch block.
It also appears that autoReconnect and autoReconnectRetries are not a properties of this.stream, and reconnect() is not a valid method either. I think the problem is that I have to use and implement the TweetStream into my code, but I am not sure how to exactly, and I can’t find documentation for this either.
In any case, I don’t want to make this too long, so I would just like to ask if you can point me in the right direction, such as providing a very quick example if that is possible.
Thank you for your time.
EDIT: It appears that this.stream is in fact a TweetStream object, there was just an error with my test script.
I would still like to ask, is this the proper way to implement autoReconnect? The actual code on my server does not have this try catch block, because I just assumed it would reconnect automatically on any kind of connection error. I’m not sure how to test this properly because Twitter only allows for 1 concurrent stream.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 34
Hey guys,
I forgot to also check back in, I have also stopped having reconnect rate limits also.
I have a funny feeling this was fixed quietly by twitter when they released API v2 from Early Access
I feel I was at most part trying to diagnose an issue that wasn’t caused by us 😅 Sorry for fuelling that rabbit hole!
Wish you guys all the best with what you create And again @alkihis excellent work again, Thank you both for the chat!
Hey @alkihis, I just wanted to let you know that I haven’t had any more 15 minute reconnect loops since the 15th of December last year, so that’s around 20 days ago. I really haven’t done or changed anything since my last comment in November, because I ran out of time and patience to deal with this issue and just decided to live with it.
But in any case, this just confirms what I suspected, which is that it was a problem with Twitter (or less likely with Heroku) that they have eventually fixed, and not a problem with this library.
So that’s all, and finally I also wanted to say thank you again for providing help and even going lengths trying to reproduce the problem on your end, it is appreciated.
I’m closing this issue now because it’s resolved (and hopefully it stays that way). Cheers!
Hello again,
1.6.1 update that contains the fix is now available on npm 😃 It comes with another tiny new features that has been raised before: you can now create a stream instance that isn’t auto-connected to Twitter immediately. Please consider this code sample, inspired by your example:
If you want to configure the retry timeout computed between each reconnect attempt, you can now overwrite
nextRetryTimeoutproperty of theTweetStreaminstance. By default, this is set totryOccurence => Math.min((tryOccurence ** 2) * 1000, 25000).Keep me updated if the new version meets your expectations. 😃
Hello once again @alkihis, thank you for your reply.
This was indeed a copy-paste from the code running on the server. And you have correctly pointed out that the class method of
stream()was a duplicate of the propertythis.stream. Huge oversight on my part, I have no idea how I managed to miss something so simple.So as mentioned, because Twitter does not allow you to run two or more concurrent connections (stream), I temporarily turned off my bot, then ran a modified version of the code on my local machine, and I can confirm that everything works as intended.
Upon launching the script and then turning off my network connection to artificially simulate a disconnect, I’ve got the following output:
The event handlers are emitting properly, and the connection automatically recovered upon turning on my network again.
Thanks so much once again for helping me, I appreciate it!
No issue whatsoever, you are contributing your free time to develop this, I should be thankful more than anything. It’s a free product with no warranty.
There is a free tier on Heroku which you can run 24/7 and the setup is quite trivial. They only require a valid credit card, but no payments necessary. This is in fact the same server I am using for this community bot.
As for the rest of your comment, I really don’t feel competent enough to comment about it because I have limited experience with Node’s
reqand I haven’t really studied the source code here, was just taking a look to see what is going on.In any case, this same error occurred again just around 30 min ago and I was around and being able to observe it. Not exactly sure if this is of any help to you, but here are all the events as logged out in my database by the
logToDb()method. I have cleaned up the output a little bit for clarity, and removed theclientState(this.client) key because it didn’t seem to contain any useful information.The events are sorted first to last, or in other words the first event in the list below is the first error that occurred.
From here on
Error,ReconnectAttemptandReconnectErrorkeep emitting forever.Hi again, Many thanks for the detailed explanation of what’s happening.
I don’t really know what’s happening and why the reconnect timeout isn’t awaited. I will look into it as quick as possible following your specific test case and I’ll come back here with more infos.
EDIT: After a quick review, I know what’s happening and it’s all my fault 😅 I’d tell you that you can use
InfinityasautoReconnectRetriessetting, but… it causes a bug that hasn’t fixed inmasterbranch yet. UsingInfinitycauses the try occurence count resolved asNaN; as reconnect timeout is computed using current try occurence number,setTimeoutis called withNaNas timeout milliseconds parameter (that must be resolved as0, so delayed to next event loop turn). This bug will be quickly resolved, but for now you can just re-switch to a value forautoReconnectRetrieslike9999.Very sorry for the inconvenience.
I’ll update this issue when bugfix is available on npm.