siegfried: error running `roy harvest -wikidata`
I’m trying out the instructions here and am getting the following error/output when trying to run $ roy harvest -wikidata to start off:
2022/04/27 09:23:21 Roy (Wikidata): Harvesting Wikidata definitions: lang ‘en’ 2022/04/27 09:23:21 Roy (Wikidata): Harvesting definitions from: ‘https://query.wikidata.org/sparql’ 2022/04/27 09:23:21 Roy (Wikidata): Harvesting revision history from: ‘https://www.wikidata.org/’ 2022/04/27 09:24:55 Error trying to retrieve SPARQL with revision history: warning: there were errors retrieving provenance from Wikibase API: wikiprov: unexpected response from server: 429
I’m on Ubuntu 20.04 with the latest siegfried release (1.9.2), is there something obvious I’m doing wrong? (@ross-spencer?)
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 24 (5 by maintainers)
thanks @ross-spencer , @richardlehane - wasn’t sure if it was expected to be able to run entirely in user space or not, but in any case, running with
sudoharvests both with and without provenance info completed smoothly! (the first nearly exactly 2 minutes on the dot 😄 )thanks for the work on this (and the write-up, Ross!), looking forward to digging back into the documentation and trying things out here now that I can sail past that step
v1.11.2 is out today with these fixes, closing
I am sooo excited for this, all have worked well in my testing! All wikidata harvests have worked and added to signature.
Thanks @EG-tech. You can definitely run it with sudo in wsl if you like. It might also be interesting in the long run to understand why it is trying to install to
/usr/shareon that platform instead of a more accessible user folder.If you want to just run the tests above to confirm it is possible to download definitions then the following two commands should work as well, providing wsl has
/tmpbut this could also be your $HOME directory too if not.I think I have it.
I have a utility for running wikidata queries like shell scripts using golang here: https://github.com/ross-spencer/spargo
I found out the standard query is very quick.
It looks like the hang-up in roy is retrieving user provenance, e.g.
e.g.
Via this Stack Overflow post there’s some basic web-scraping good behaviour I could be implementing: https://stackoverflow.com/a/62396996
And if I add more logging to wikiprov which is doing this work, I see that the responses do have
retry-afterset after a number of attempts:Here it is set to
1which is a small number for the number of requests we’re making. We’re also not honouring this.I think there are more than a few things we can do here, as well as a simple “no provenance” option.
If we add a sleep and retry based on the retry-after value we can make this pass pretty quickly:
If we’re a bit nicer to the Wikidata’s Wikibase API it’s really nice to us so we should be able to do this pretty easily.
I’ll probably:
And I think this will fix things.
Apologies for the time wasted on this, I wasn’t even looking in the Wikibase’s API direction and didn’t think for a second it would be interfering. I’m not sure why I was unable to recreate the issue in my regional virtual machines but maybe the internet connection does still have some impact, e.g. a slower connection maybe has fewer attempts and so doesn’t trigger the issue?
Will be good to fix this. Hopefully have something up soon!
thanks @ross-spencer!! that all makes sense, thanks for confirming and I’ll play with your suggestion when I get the chance. amazing work!
that’s exactly it @EG-tech. Tyler had the same a while back. (Via email request so not on Github). We might need to put it into the FAQ.
Some notes on what I wrote to Tyler:
Long-term something approaching rate limiting may work. Right now it’s just a single request asking for a lot of data.
In the short-to-medium term, this pull request should mean you can grab an identifier from Richard’s itforarchivists server and it will let you get up and running: https://github.com/richardlehane/siegfried/pull/178 (PR just needs review and (and fixes) and merging).
EDIT: NB. For Tyler, he just tried it later in the day or next morning and it worked.