google-cloud-php: [Spanner] Query occasionally returns nothing when there should be data

I’ve been trying to debug this since December last year and haven’t had much luck so I’d like to ask for some help.

Environment details

• OS: Debian
• PHP version: 7.2.11
• Package name and version: google/cloud-spanner v1.11.5

When we execute the following query and try to get its rows, sometimes we would get nothing back when it should be returning data.

SELECT * FROM `User` WHERE `userId` = @userId LIMIT 1

We checked GCP Console -> Spanner ->query stats to see if there was some useful info and we found the following. (see arrow)

Total CPU (sec)          1.13
Execution count          8371
Avg latency (sec)	0.001
Avg CPU (sec)           0.000
Avg rows scanned         1.00 <-- 
Avg rows returned        1.00 <--
Avg bytes returned        382

If there was something wrong with the data/query this should show a number lower than 1.00 so I’m thinking this is a bug on the client side.

Since we know the data exists, we added some code that will retry the query if it returns nothing. After retrying the query, we get data back.

This is really hard for me to debug since it only happens once in a while (like once or twice a day with around 5000 executions) but I’m guessing it has something to do with the retry mechanism.

Specifically, I think the code below might have something to do with it. https://github.com/googleapis/google-cloud-php/blob/master/Spanner/src/Result.php#L175

Please let me know what you think. Thanks.