geotiff.js: AggregateError: Request failed / Block is not in the block requests

Has there been any progress on this? This is a pretty basic use case that seems to fail most of the time.

I fix this problem with using GeoTIFFSourceOptions, like this:

const sourceOptions: GeoTIFFSourceOptions = {
    allowFullFile: true,
}

const source = new GeoTIFF({
    sources: [
        {
            url: url,
        },
    ],
    sourceOptions: sourceOptions
});

It’s worked for me !

I’ve been looking at this a little recently as I was working on requesting tiles from a COG. It seems like there is an issue with the blockCache. I think what’s happening is that an id is present in the cache at the start of the fetch function, but by the end when it’s time to retrieve the block from the blockCache it’s been pushed out of the cache by more recent arrivals, and so when it comes time to grab all the relevant blocks from the cache occasionally something is no longer available. https://github.com/geotiffjs/geotiff.js/blob/4de2c3deb2e751809afada6546330462b9406ae6/src/source/blockedsource.js#L141.

As an easy fix I bumped up the cacheSize from 100 to 1000 and all my errors went away, however there is probably a smarter way to manage things rather than relying on a magic number.

@constantinius I was able to get the test page working except for 404’s for some tiff files I was not able to generate due to some GDAL issue. If you could post some of the not-working TIFFs, that’d be great.

I believe that these lines: https://github.com/geotiffjs/geotiff.js/blob/a0153dc0d4956fca78e84bd9ce26d1995c89d32d/src/source/blockedsource.js#L104-L118 contain a race condition. The scenario goes:

• fetch-1 is initiated and adds blockA to this.blockIdsToFetch and missingBlockIds
• fetch-2 is initiated and only finds blockA in this.blockIdsToFetch but not this.blockRequests
• fetch-2 needs that block for its own request (or fetches it anyway, as is now possible) calling this.blockIdsToFetch.clear() and this.blockRequests.delete(blockId) at the end of the request
• fetch-1 comes looking for the block in either the this.blockCache and this.blockRequests but never finds it because fetch-2 deleted it after completing its request so quickly and the block was pushed out by others

I think this is very unlikely to happen but theoretically possible without the abort signal (and it becomes very clear if you start using abort controllers because fetch-2 can complete its request very quickly if it just aborts).

Neither fetch-1 nor fetch-2 in this scenario ever actually adds blockA to blockRequests so simply deleting the error at the end of the if...else if...else in the above will not help since let results = await Promise.allSettled(blockRequests.values()); will not have that block. Thus its error cannot be checked by the subsequent code or refetched.

I started a new branch to resolve this using the mechanism in #182 and it seems to work well (likely because it looks very similar to the old code which seemed to work) except that the LRU cache doesn’t work with it, which seems understandable since the cache is expected to be consistent for so long throughout the whole fetch call. So setting that to have infinite size (or equivalently going back to a Map) should work. I would be in favor of going back to a Map because I think the LRUCache is fairly prone to race conditions if you have requests coming in and out quick enough (for example calling has doesn’t guarantee that get will actually return anything from what I understand). However, I don’t know if people have complained about memory issues so I would understand keeping it.

Let me know if this makes sense or not. Would love to understand the use case more for changing the code so much and make progress on this. Thanks and sorry for the delay!

Okay, I fixed the test page. Its true, sometimes I still get the Block is not in the block requests error, but to a far lesser extent than before. I think there is still a race condition somewhere.