pyproj: Instantiating many proj objects much slower than 1.9.6

I have a tool in pyproj 1.9.6 that I use for doing a “reverse lookup” of projections. Given an x/y in unknown CRS and a known longitude/latitude, this finds the projections which place that x/y closest to the known longitude/latitude. This is helpful when trying to track down an unknown coordinate system.

This requires instantiating thousand of proj/Proj objects, but it only takes a few seconds in pyproj 1.9.6. Recently I wanted to upgrade to more recent versions of PROJ and GDAL, but this tool is now taking a few minutes, about 50 times longer in pyproj 2.x:

This gist takes 2.32 seconds to run with pyproj 1.9.6.
This nearly identical gist takes 168 seconds to run with pyproj 2.6.1

I know that a lot changed in the underlying PROJ C++ library between pyproj 1.9.6 and 2.x. But is there any way to restore the fast instantiation of the proj/Proj objects? The projections don’t have to be exact - just close enough to this reverse lookup tool. Also, I am willing to serialize/pickle the proj objects if that would help, though my understanding was that that didn’t work with Python C extensions.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (17 by maintainers)

Most upvoted comments

@snowman2 I definitely thought about using area of interest to filter projections. And that’s a really cool new helper function for querying them! But there’s nothing stopping a novice GIS user from (wrongly) using a coordinate system for points outside of the area of interest, right? In that case I think it’s better to be thorough and just check everything. Each request to the cloud function only takes about 500 ms 😃

dmahr1 on Oct 27, 2020

But it looks like the global context is leaving open a persistent connection to the database via the PROJ C API?

That was one of the settings that needed to be tweaked to get this to work. Also, adding the settings to the context beforehand and not updating them each time shaved off time.

snowman2 on Jul 3, 2020

In #675 it seems like I have achieved speedups. This example is how I got the best speedup:

import pyproj, datetime

test_codes = pyproj.get_codes("EPSG", pyproj.enums.PJType.PROJECTED_CRS, False)
start = datetime.datetime.now()
projs = []
for code in test_codes:
    try:
        projs.append(pyproj.Proj(f'EPSG:{code}'))
    except pyproj.exceptions.ProjError as err:
	pass
print(f'Instantiating {len(projs)} projs took {(datetime.datetime.now() - start).total_seconds()} seconds')

In this example, I am assuming that since you were using the Proj objects, only the Projected CRS objects would provide much value. But, not 100% sure.

Using pyproj 3.0.dev0 with PYPROJ_GLOBAL_CONTEXT=ON, the output was:

Instantiating 4664 projs took 1.592608 seconds

Using pyproj 2.6.1post1

Instantiating 4659 projs took 116.082649 seconds

Using the second gist you linked above, it still took ~100 seconds to initialize everything using the global context. I am guessing it is due to some of the EPSG codes causing errors that slowed things down.

snowman2 on Jul 3, 2020