imposm3: Error while updating planet import

Hi, I’ve had a cron job running everyday updating a planet osm import and it has been working great. However, I checked my logs this morning and found that the update has crashed for the past two days (October 5th and 6th). Error below. I’ve tried to re run a few times, but it seems to get the same error while parsing the diff. Seems to be something while parsing a feature’s tags.

OS is Ubuntu 14.04 Build is 0.2.0dev-20160615-d495ca4

Thanks!

[Oct  6 17:17:33] Processing /home/ubuntu/osm_data/changes.osc.gz
[Oct  6 17:17:33] Parsing changes, updating cache and removing elements
[Oct  6 17:18:33] [INFO] [  1m0s] C:   13000/s (815821) N:     400/s (29836) W:       0/s (0) R:      0/s (0)
[Oct  6 17:19:33] [INFO] [  2m0s] C:   22000/s (2667879) N:     600/s (74916) W:       0/s (0) R:      0/s (0)
[Oct  6 17:19:47] [INFO] [diff] Processing /home/ubuntu/osm_data/changes.osc.gz took: 2m13.958160413s
panic: runtime error: index out of range

goroutine 1 [running]:
panic(0x990960, 0xc82000e0b0)
        /root/imposm/go/src/runtime/panic.go:464 +0x3e6
github.com/omniscale/imposm3/cache/binary.tagsFromArray(0xc82249b420, 0x2, 0x2, 0x7fe452fcfc40)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/cache/binary/tags.go:85 +0x43e
github.com/omniscale/imposm3/cache/binary.UnmarshalNode(0xc82244bbc0, 0x21, 0x21, 0xc82249b440, 0x0, 0x0)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/cache/binary/serialize.go:43 +0x1d4
github.com/omniscale/imposm3/cache.(*NodesCache).GetNode(0xc82011ab70, 0x1081266ff, 0x0, 0x0, 0x0)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/cache/nodes.go:70 +0x291
github.com/omniscale/imposm3/diff.(*Deleter).deleteNode(0xc8211bf960, 0x1081266ff, 0x0, 0x0)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/diff/deleter.go:188 +0x57
github.com/omniscale/imposm3/diff.(*Deleter).Delete(0xc8211bf960, 0x1, 0xc82249b0a0, 0x0, 0x0, 0x0, 0x0)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/diff/deleter.go:235 +0x2af
github.com/omniscale/imposm3/diff.Update(0x7fff361c4930, 0x24, 0x0, 0x0, 0x0, 0xc820019000, 0xc82011ac30, 0x0, 0x0, 0x0)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/diff/process.go:209 +0x3434
github.com/omniscale/imposm3/diff.Diff()
        /root/imposm/gopath/src/github.com/omniscale/imposm3/diff/process.go:60 +0x835
github.com/omniscale/imposm3/cmd.Main(0xb4bea8)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/cmd/main.go:53 +0x1f1
main.main()
        /root/imposm/gopath/src/github.com/omniscale/imposm3/imposm3.go:8 +0x23

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 19 (13 by maintainers)

Commits related to this issue

Partial fix for cache problem : #122 This is only handle the indexing problem, but not clean ! — committed to ImreSamu/imposm3 by ImreSamu 8 years ago
Check & Drops if the OSM keys starting with codepoints fully fix #122 If the OSM key start with a codepoints - we don't write to the cache * codepoint(1) - codepoint(31) * codepoint('\uE000') -... — committed to ImreSamu/imposm3 by ImreSamu 8 years ago
fix encoding of empty keys (#122) — committed to omniscale/imposm3 by olt 8 years ago

Most upvoted comments

my quick fix / workaround : dropping/excluding all keys - starting with the 31 “code points for common keys” ; ( "\u0001*" … "\u001F*" ) // this is solved the error message, but need more test !!

{
    "tags": {
        "load_all": true,
        "exclude": [
            "created_by",
            "source",
            "\u0001*", 
            "\u0002*",
            "\u0003*",
            "\u0004*",
            "\u0005*",
            "\u0006*",
            "\u0007*", 
            "\u0008*",
            "\u0009*",                                                                                                      
            "\u000A*",
            "\u000B*", 
            "\u000C*",
            "\u000D*",
            "\u000E*",
            "\u000F*",
            "\u0010*",
            "\u0011*", 
            "\u0012*",
            "\u0013*",                                                                                                      
            "\u0014*",
            "\u0015*",
            "\u0016*",
            "\u0017*", 
            "\u0018*",
            "\u0019*",                                                                                                      
            "\u001A*",
            "\u001B*", 
            "\u001C*",
            "\u001D*",
            "\u001E*",
            "\u001F*"      
            ]
    },
    "use_single_id_space": true,

...

ImreSamu on Oct 8, 2016

Uh. I’ve fixed this and extended the tests. Thanks!

olt on Oct 12, 2016

Imposm reduces the cache size by encoding common keys and tag combinations (like highway=residential) with “ASCII” control characters and private Unicode symbols. There was no escaping of these symbols as the user defined which tags are cached and other tags were filtered out anyway. This condition does not apply anymore with the load_all feature and it now crashed when a user added a backspace character in the key (Thanks @ImreSamu for finding this one).

All encoding symbols are now properly escaped.

All users who use load_all with diff imports should upgrade to 4a472f5 from 2016-10-10 or higher and do a full re-import.

There is a new binary at https://imposm.org/static/rel/

Thanks for reporting this issue and sorry for the trouble it might have caused you.

olt on Oct 10, 2016

in the 2125774.osc.gz file ( thanks @bdon ) I have found a strange Key
<tag k="
bulding" v="warehouse"/>

probably the first char ( &#10 ) detected as a “code points for common keys” ;

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.5.4 (26385 thorn-03.openstreetmap.org)" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
 <node id="4430391039" visible="true" version="1" changeset="42622374" timestamp="2016-10-04T08:13:53Z" user="ROGERRE" uid="4569166" lat="13.9160490" lon="121.8154030">
  <tag k="&#10;bulding" v="warehouse"/>
 </node>
</osm>

or in OPL osmium cat 2125774.osc.gz -f opl | grep 4430391039

n4430391039 v1 dV c42622374 t2016-10-04T08:13:53Z i4569166 uROGERRE T%0a%bulding=warehouse x121.8154030 y13.9160490

ImreSamu on Oct 8, 2016