siegfried: error running `roy harvest -wikidata`

I’m trying out the instructions here and am getting the following error/output when trying to run $ roy harvest -wikidata to start off:

2022/04/27 09:23:21 Roy (Wikidata): Harvesting Wikidata definitions: lang ‘en’ 2022/04/27 09:23:21 Roy (Wikidata): Harvesting definitions from: ‘https://query.wikidata.org/sparql’ 2022/04/27 09:23:21 Roy (Wikidata): Harvesting revision history from: ‘https://www.wikidata.org/’ 2022/04/27 09:24:55 Error trying to retrieve SPARQL with revision history: warning: there were errors retrieving provenance from Wikibase API: wikiprov: unexpected response from server: 429

I’m on Ubuntu 20.04 with the latest siegfried release (1.9.2), is there something obvious I’m doing wrong? (@ross-spencer?)

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 24 (5 by maintainers)

Most upvoted comments

thanks @ross-spencer , @richardlehane - wasn’t sure if it was expected to be able to run entirely in user space or not, but in any case, running with sudo harvests both with and without provenance info completed smoothly! (the first nearly exactly 2 minutes on the dot 😄 )

thanks for the work on this (and the write-up, Ross!), looking forward to digging back into the documentation and trying things out here now that I can sail past that step

v1.11.2 is out today with these fixes, closing

I am sooo excited for this, all have worked well in my testing! All wikidata harvests have worked and added to signature.

Thanks @EG-tech. You can definitely run it with sudo in wsl if you like. It might also be interesting in the long run to understand why it is trying to install to /usr/share on that platform instead of a more accessible user folder.

If you want to just run the tests above to confirm it is possible to download definitions then the following two commands should work as well, providing wsl has /tmp but this could also be your $HOME directory too if not.

./roy harvest -wikidata -home /tmp/ 
./roy harvest -wikidata -home /tmp/ -noprov

NB. the first command might take ~2 mins but shouldn’t fail 😃

I think I have it.

I have a utility for running wikidata queries like shell scripts using golang here: https://github.com/ross-spencer/spargo

I found out the standard query is very quick.

$ time ./wd.sparql | grep entity | sort | uniq | wc -l
Connecting to: https://query.wikidata.org/sparql

Query: # Return all file format records from Wikidata.
SELECT DISTINCT ?uri ?uriLabel ?puid ?extension ?mimetype ?encoding ?referenceLabel ?date ?relativity ?offset ?sig WHERE {
  { ?uri (wdt:P31/(wdt:P279*)) wd:Q235557. }
  UNION
  { ?uri (wdt:P31/(wdt:P279*)) wd:Q26085352. }
  FILTER(EXISTS { ?uri (wdt:P2748|wdt:P1195|wdt:P1163|ps:P4152) _:b2. })
  FILTER((STRLEN(?sig)) >= 6 )
  OPTIONAL { ?uri wdt:P2748 ?puid. }
  OPTIONAL { ?uri wdt:P1195 ?extension. }
  OPTIONAL { ?uri wdt:P1163 ?mimetype. }
  OPTIONAL {
    ?uri p:P4152 ?object.
    OPTIONAL { ?object pq:P3294 ?encoding. }
    OPTIONAL { ?object ps:P4152 ?sig. }
    OPTIONAL { ?object pq:P2210 ?relativity. }
    OPTIONAL { ?object pq:P4153 ?offset. }
    OPTIONAL {
      ?object prov:wasDerivedFrom ?provenance.
      OPTIONAL {
        ?provenance pr:P248 ?reference;
          pr:P813 ?date.
      }
    }
  }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE], en". }
}
ORDER BY (?uri)


8482

real    0m1.535s
user    0m1.222s
sys     0m0.455s

It looks like the hang-up in roy is retrieving user provenance, e.g.

https://www.wikidata.org/w/api.php?action=query&format=json&prop=revisions&rvlimit=5&rvprop=ids%7Cuser%7Ccomment%7Ctimestamp%7Csha1&titles=item%3AQ35221946

e.g.

───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: inspect.example
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ Q35221946
   2   │ Name: 'RAR, version 5'
   3   │ MIMEType: 'application/x-rar-compressed; application/vnd.rar'
   4   │ Sources: 'Gary Kessler's File Signature Table (source date: 2017-08-07) PRONOM (Official (fmt/613))' 
   5   │ Revision History: {
   6   │   "Title": "Q35221946",
   7   │   "Revision": 2209073726,
   8   │   "Modified": "2024-07-20T23:13:41Z",
   9   │   "Permalink": "https://www.wikidata.org/w/index.php?oldid=2209073726&title=Q35221946",
  10   │   "History": [
  11   │     "2024-07-20T23:13:41Z (oldid: 2209073726): 'Vitplister' edited: '/* wbremoveclaims-remove:1| */ [[Property:P4152]]: 526172211A070100'",
  12   │     "2024-07-20T23:13:34Z (oldid: 2209073671): 'Vitplister' edited: '/* wbsetclaim-update:2||1|1 */ [[Property:P4152]]: 526172211A070100'",
  13   │     "2024-07-20T23:13:07Z (oldid: 2209073493): 'Vitplister' edited: '/* wbsetclaim-update:2||1|1 */ [[Property:P4152]]: 526172211A070100'",
  14   │     "2023-04-02T15:05:39Z (oldid: 1866796586): 'Renamerr' edited: '/* wbsetdescription-add:1|uk */ формат файлу, [[:toollabs:quickstatements/#/batch/151018|batch #151018]]'",
  15   │     "2021-03-10T14:16:29Z (oldid: 1379273425): 'YULdigitalpreservation' edited: '/* wbeditentity-update:0| */ WikiDP Trid Signatures Single([[:toollabs:editgroups/b/CB/a6d7c6
       │ d0301|details]])'"
  16   │   ]
  17   │ }
  18   │ ---
  19   │ Signatures:
  20   │ globs: *.rar
  21   │ sigs: (B:0 seq "Rar!\x1a\a\x01\x00")
  22   │       (B:0 seq "Rar!\x1a\a\x01\x00")
  23   │ superiors: none
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

NB. The history array on line 10 is generated by Wikibase’s native API not Wikidata.

Via this Stack Overflow post there’s some basic web-scraping good behaviour I could be implementing: https://stackoverflow.com/a/62396996

And if I add more logging to wikiprov which is doing this work, I see that the responses do have retry-after set after a number of attempts:

2025/01/01 17:11:41 request &{GET https://www.wikidata.org/w/api.php?action=query&format=json&prop=revisions&rvlimit=5&rvprop=ids%7Cuser%7Ccomment%7Ctimestamp%7Csha1&titles=item%3AQ105851857 HTTP/1.1 1 1 map[User-Agent:[wikiprov/0.2.0 (https://github.com/ross-spencer/wikiprov/; all.along.the.watchtower+github@gmail.com)]] <nil> <nil> 0 [] false www.wikidata.org map[] map[] <nil> map[]   <nil> <nil> <nil> {{}} <nil> [] map[]}
2025/01/01 17:11:41 header: map[Content-Length:[1854] Content-Type:[text/html; charset=utf-8] Date:[Wed, 01 Jan 2025 22:11:42 GMT] Nel:[{ "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}] Report-To:[{ "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }] Retry-After:[1] Server:[Varnish] Server-Timing:[cache;desc="int-front", host;desc="cp1114"] Set-Cookie:[WMF-Last-Access=01-Jan-2025;Path=/;HttpOnly;secure;Expires=Sun, 02 Feb 2025 12:00:00 GMT WMF-Last-Access-Global=01-Jan-2025;Path=/;Domain=.wikidata.org;HttpOnly;secure;Expires=Sun, 02 Feb 2025 12:00:00 GMT] Strict-Transport-Security:[max-age=106384710; includeSubDomains; preload] X-Cache:[cp1114 int] X-Cache-Status:[int-front] X-Client-Ip:[64.92.67.130]]
2025/01/01 17:11:41 header: [1]

Here it is set to 1 which is a small number for the number of requests we’re making. We’re also not honouring this.

I think there are more than a few things we can do here, as well as a simple “no provenance” option.

If we add a sleep and retry based on the retry-after value we can make this pass pretty quickly:

2025/01/01 17:31:59 ------- retrying ---------
2025/01/01 17:31:59 1
2025/01/01 17:31:59 ------- retrying ---------
2025/01/01 17:31:59 1
2025/01/01 17:32:00 Roy (Wikidata): Harvesting Wikidata definitions '/home/ross/.local/share/siegfried/wikidata/wikidata-definitions-3.0.0' complete

real    1m46.684s
user    0m23.893s
sys     0m17.254s
diff --git a/pkg/wikiprov/wikiprov.go b/pkg/wikiprov/wikiprov.go
index 85d6ab8..9c27f26 100644
--- a/pkg/wikiprov/wikiprov.go
+++ b/pkg/wikiprov/wikiprov.go
@@ -24,6 +24,10 @@ import (
        "io/ioutil"
        "net/http"
        "strings"
+
+       "log"
+       "strconv"
+       "time"
 )
 
 func getRevisionProperties() string {
@@ -91,6 +95,16 @@ func GetWikidataProvenance(id string, history int) (Provenance, error) {
                return Provenance{}, err
        }
 
+       if len(resp.Header["Retry-After"]) > 0 {
+               retry, _ := strconv.Atoi(resp.Header["Retry-After"][0])
+               if retry > 0 {
+                       log.Println("------- retrying ---------")
+                       log.Println(retry)
+                       time.Sleep(time.Duration(retry))
+                       return GetWikidataProvenance(id, history)
+               }
+       }
+
        const expectedCode int = 200
        if resp.StatusCode != expectedCode {
                responseError := ResponseError{}

If we’re a bit nicer to the Wikidata’s Wikibase API it’s really nice to us so we should be able to do this pretty easily.

I’ll probably:

  • implement no prov as this is an as-yet underused function.
  • implement a retry-after approach like above.
  • add some better return information to delineate when an error is with Wikidata and the Wikibase API.
  • Have a look at anything else that might help along the way. Suggestions welcome.

And I think this will fix things.

Apologies for the time wasted on this, I wasn’t even looking in the Wikibase’s API direction and didn’t think for a second it would be interfering. I’m not sure why I was unable to recreate the issue in my regional virtual machines but maybe the internet connection does still have some impact, e.g. a slower connection maybe has fewer attempts and so doesn’t trigger the issue?

Will be good to fix this. Hopefully have something up soon!

thanks @ross-spencer!! that all makes sense, thanks for confirming and I’ll play with your suggestion when I get the chance. amazing work!

that’s exactly it @EG-tech. Tyler had the same a while back. (Via email request so not on Github). We might need to put it into the FAQ.

Some notes on what I wrote to Tyler:

The Wikidata documentation on the query service (WDQS) is here but it’s not very clear, i.e. it talks about processing time, not how that translates to some large queries:

https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Query_limits

I have known that it is a risk that this might happen, though I can’t quantify for how many and when. There are times for example when I have been testing where I have run the query in-upwards of 30 times in a day.

We set a custom header for the request which should be recognized by WDQS and prevent this issue somewhat - it is more friendly to known user-agents for example than unknown ones.

Long-term something approaching rate limiting may work. Right now it’s just a single request asking for a lot of data.

In the short-to-medium term, this pull request should mean you can grab an identifier from Richard’s itforarchivists server and it will let you get up and running: https://github.com/richardlehane/siegfried/pull/178 (PR just needs review and (and fixes) and merging).

EDIT: NB. For Tyler, he just tried it later in the day or next morning and it worked.