inaturalist: Fix MushroomObserver Import (504 Gateway Timeouts)

Users are reporting a 504 Gateway Timeout when using the MushroomObserver import tool. There is an aggregated list of user reports concerning this issue here.

MushroomObserver maintainer @JoeCohen has kindly pointed out the potenial source of these issues. I’ll try and condense them here but you can see the full explanation at the iNat forum link above.

1. The MushroomObserver Observation XML API Response is returning invalid observation urls

This bug has been reported in MushroomObserver/mushroom-observer/issues/815.

Once fixed, the MO import tool can undo it’s workaround, trust the response urls, and stop calling HEAD on each url which may be why the page times out for people with lot’s of observations.

2. The MushroomObserver import code is using the wrong url format for resolving image sources in https://github.com/inaturalist/inaturalist/blob/88c24041fe0df7f81fe1d04914ecdba29c319f43/app/models/mushroom_observer_import_flow_task.rb#L303

It is currently in the format:

image_url = "[https://images.mushroomobserver.org/orig/#{image[:id]}.jpg

The correct way to reference image urls is:

https://mushroomobserver.org/images/<*size*>/<*id*>.<*ext*>

@JoeCohen It probably wouldn’t be a bad idea to add the image urls to the api2 response so that clients don’t need to worry about url construction.

3. MushroomObserver recently moved from DigitalOcean to Google Cloud Platform.

This means this url needs to be updated as well: https://github.com/inaturalist/inaturalist/blob/88c24041fe0df7f81fe1d04914ecdba29c319f43/app/models/mushroom_observer_import_flow_task.rb#L293

Although it doesn’t appear iNats importer looks at anything specific in potential error response bodies, future additions might incorporate parsing the response body and providing a better error to the end-user.

The import code assumes source images are all jpg and this is not always the case and may cause 404’s or other errors when downloading images.

Proposed Fix

Get MO to fix urls in their xml API response
Update the import urls to meet the new MO url schema (Preferably, MO would start returning the image urls)
Don’t hardcode the jpg image extension and instead construct it from the list of returned images in: https://github.com/inaturalist/inaturalist/blob/88c24041fe0df7f81fe1d04914ecdba29c319f43/app/models/mushroom_observer_import_flow_task.rb#L291

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 21 (8 by maintainers)

Most upvoted comments

Looks like Mushroom Observer changed their API and now our tool isn’t working again. Not much we can do on our end without some additional information from them, since I’m not finding any documentation of this change. In brief, we were doing something like this to verify that your API key works and to get your MO user ID to retrieve on your observations:

curl --user-agent iNaturalist \
  -XGET \
  "https://mushroomobserver.org/api2/api_keys.xml?api_key=YOUR_API_KEY"

That used to return info about the user who owns that key, but now it returns

<error id="1">
  <code>API2::NoMethodForAction</code>
  <details>Invalid request method "GET" for "api_keys".</details>
  <fatal>true</fatal>
</error>

If someone familiar with the MO API wants to suggest another way to get the user ID associated with an API key, let me know.

kueda on Feb 14, 2022

I agree with @joshuamcginnis here. I’m a PhD student working on fungal systematics research and I’ve been using this tool for the past couple years to duplicate my records. I appreciate that this tool was developed to facilitate users moving from MO to iNat, but this is not how it was perceived by MO users at the time, or by myself or other uses using the tool for FunDiS (formerly North American Mycoflora Project) related projects. Some BioBlitz projects, permits or grants have required observations to be uploaded to iNat. I use MO primarily to hold my collection data and images (original size/quality), as well as for printing collection labels for herbarium submission (a feature not yet implemented in iNat). My desktop collection image / notes database is also organized by MO number. This feature has allowed me and others to comply with the aforementioned requirements while not sacrificing the personal herbarium management functionality of MO. It also acts as a failsafe in case MO goes down permanently, which seems more of a possibility than the reverse. I do hope you reopen this @kueda. It’s a very useful tool even if it isn’t being used the way it was originally intended.

heelsplitter on Dec 8, 2021