algoliasearch-client-go: AWS Lambda: socket: too many open files

I am consuming events from a kinesis queue in a lambda function and calling

algoliaIndex := algoliaClient.InitIndex(lambdaEvent.IndexName)
_, algoliaOperationError = algoliaIndex.AddObjects(objectsToSync(lambdaEvent))

After 6,930 successful calls I get the following error:

Cannot perform request [POST] /1/indexes/SyncItem/batch (8OVNLPT6JS-1.algolianet.com): Post https://8OVNLPT6JS-1.algolianet.com/1/indexes/SyncItem/batch: dial tcp: lookup 8OVNLPT6JS-1.algolianet.com on 10.15.0.2:53: dial udp 10.15.0.2:53: socket: too many open files

Is it possible that the Algolia go library doesn’t correctly close old connections or am I missing something?

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 16 (6 by maintainers)

Commits related to this issue

Fix unlimited socket opening problem transport.go#L171 tries to make a request to each host, until one succeed and ignore errors unless there is no host left. But transport.go#L249 can return with an... — committed to algolia/algoliasearch-client-go by fsenart 7 years ago
Fix unlimited socket opening problem transport.go#L171 tries to make a request to each host, until one succeed and ignore errors unless there is no host left. But transport.go#L249 can return with an... — committed to algolia/algoliasearch-client-go by fsenart 7 years ago
chore(ci): use correct variable to get the latest commit message in js repo (#287) — committed to algolia/algoliasearch-client-go by eunjae-lee 2 years ago

Most upvoted comments

@sorenbs Merged and version bumped to 2.7.1 on master.

@fsenart Thank you again for the fix.

aseure on Jan 25, 2017

@fsenart - you are amazing 😃

@aseure - I am using latest master

sorenbs on Jan 25, 2017

Comment from the https://github.com/eawsy/aws-lambda-go maintainer

@sorenbs AWS Lambda is not freshly initialized each time you invoke it. One Lambda can persist over the time (nearly 8 hours) before being recycled. So when I see your error, it seems that you orthe algolia lib have forgotten something like defer foo.Close() somewhere. Thus, over the time the number of opened sockets grows and the error occur. But without more info, this is just an intuition.

sorenbs on Jan 25, 2017