homebrew-osgeo4mac: QGIS3/GRASS: error while executing a script

@luispuerto @nickrobison I’ve solved v.dissolve and v.to.lines 🎉

With all the changes applied, GRASS should be completely compatible with Python 3.

While we wait for version 7.8/8.0, which apparently (https://gis.stackexchange.com/a/304598) will be compatible with Python 3.

Files to which I had to apply changes:

/etc/python/grass/script/core.py

--- a/etc/python/grass/script/core.py
+++ b/etc/python/grass/script/core.py
@@ -25,9 +25,11 @@
 import subprocess
 import shutil
 import codecs
+import string
+import random
 import types as python_types
 
-from .utils import KeyValue, parse_key_val, basename, encode
+from .utils import KeyValue, parse_key_val, basename, encode, decode
 from grass.exceptions import ScriptError, CalledModuleError
 
 # i18N
@@ -38,13 +40,13 @@
     # python2
     import __builtin__
     from os import environ
+    __builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.ugettext
 except ImportError:
     # python3
     import builtins as __builtin__
     from os import environb as environ
     unicode = str
-__builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.lgettext
-
+    __builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.gettext
 
 # subprocess wrapper that uses shell on Windows
 
@@ -104,6 +106,21 @@
     except TypeError:
         pass
     return bytes(val)
+
+
+def _make_unicode(val, enc):
+    """Convert value to unicode with given encoding
+
+    :param val: value to be converted
+    :param enc: encoding to be used
+    """
+    if val is None or enc is None:
+        return val
+    else:
+        if enc == 'default':
+            return decode(val)
+        else:
+            return decode(val, encoding=enc)
 
 
 def get_commands():
@@ -301,7 +318,8 @@
             continue
         # convert string to bytes
         opt = encode(opt)
-        if val != None:
+        prog = encode(prog)
+        if val is not None:
             if opt.startswith(b'_'):
                 opt = opt[1:]
                 warning(_("To run the module <%s> add underscore at the end"
@@ -328,7 +346,8 @@
     else:
         # TODO: construction of the whole command is far from perfect
         args = make_command(*args, **kwargs)
-        raise CalledModuleError(module=None, code=repr(args),
+        code = ''.join([decode(each) for each in args])
+        raise CalledModuleError(module=None, code=code,
                                 returncode=returncode)
 
 def start_command(prog, flags=b"", overwrite=False, quiet=False,
@@ -360,6 +379,9 @@
 
     :return: Popen object
     """
+    if 'encoding' in kwargs.keys():
+        encoding = kwargs.pop('encoding')
+
     options = {}
     popts = {}
     for opt, val in kwargs.items():
@@ -379,7 +401,6 @@
         sys.stderr.flush()
     return Popen(args, **popts)
 
-
 def run_command(*args, **kwargs):
     """Execute a module synchronously
 
@@ -408,11 +429,18 @@
 
     :raises: ``CalledModuleError`` when module returns non-zero return code
     """
+    encoding = 'default'
+    if 'encoding' in kwargs:
+        encoding = kwargs['encoding']
+
     if _capture_stderr and 'stderr' not in kwargs.keys():
         kwargs['stderr'] = PIPE
     ps = start_command(*args, **kwargs)
     if _capture_stderr:
         stdout, stderr = ps.communicate()
+        if encoding is not None:
+            stdout = _make_unicode(stdout, encoding)
+            stderr = _make_unicode(stderr, encoding)
         returncode = ps.poll()
         if returncode:
             sys.stderr.write(stderr)
@@ -466,10 +494,17 @@
 
     :return: stdout
     """
+    encoding = 'default'
+    if 'encoding' in kwargs:
+        encoding = kwargs['encoding']
+
     if _capture_stderr and 'stderr' not in kwargs.keys():
         kwargs['stderr'] = PIPE
     process = pipe_command(*args, **kwargs)
     stdout, stderr = process.communicate()
+    if encoding is not None:
+        stdout = _make_unicode(stdout, encoding)
+        stderr = _make_unicode(stderr, encoding)
     returncode = process.poll()
     if _capture_stderr and returncode:
         sys.stderr.write(stderr)
@@ -539,12 +574,22 @@
 
     :raises: ``CalledModuleError`` when module returns non-zero return code
     """
+    encoding = 'default'
+    if 'encoding' in kwargs:
+        encoding = kwargs['encoding']
     # TODO: should we delete it from kwargs?
     stdin = kwargs['stdin']
+    if encoding is None or encoding == 'default':
+        stdin = encode(stdin)
+    else:
+        stdin = encode(stdin, encoding=encoding)
     if _capture_stderr and 'stderr' not in kwargs.keys():
         kwargs['stderr'] = PIPE
     process = feed_command(*args, **kwargs)
     unused, stderr = process.communicate(stdin)
+    if encoding is not None:
+        unused = _make_unicode(unused, encoding)
+        stderr = _make_unicode(stderr, encoding)
     returncode = process.poll()
     if _capture_stderr and returncode:
         sys.stderr.write(stderr)
@@ -738,14 +783,15 @@
             break
         try:
             [var, val] = line.split(b'=', 1)
+            [var, val] = [decode(var), decode(val)]
         except:
             raise SyntaxError("invalid output from g.parser: %s" % line)
 
-        if var.startswith(b'flag_'):
+        if var.startswith('flag_'):
             flags[var[5:]] = bool(int(val))
-        elif var.startswith(b'opt_'):
+        elif var.startswith('opt_'):
             options[var[4:]] = val
-        elif var in [b'GRASS_OVERWRITE', b'GRASS_VERBOSE']:
+        elif var in ['GRASS_OVERWRITE', 'GRASS_VERBOSE']:
             os.environ[var] = val
         else:
             raise SyntaxError("invalid output from g.parser: %s" % line)
@@ -768,7 +814,7 @@
     "flags" are Python booleans.
 
     Overview table of parser standard options:
-    https://grass.osgeo.org/grass74/manuals/parser_standard_options.html
+    https://grass.osgeo.org/grass77/manuals/parser_standard_options.html
     """
     if not os.getenv("GISBASE"):
         print("You must be in GRASS GIS to run this program.", file=sys.stderr)
@@ -820,6 +866,30 @@
     os.mkdir(tmp)
 
     return tmp
+
+
+def tempname(length, lowercase=False):
+    """Generate a GRASS and SQL compliant random name starting with tmp_
+    followed by a random part of length "length"
+
+    :param int length: length of the random part of the name to generate
+    :param bool lowercase: use only lowercase characters to generate name
+    :returns: String with a random name of length "length" starting with a letter
+    :rtype: str
+
+    :Example:
+
+    >>> tempname(12)
+    'tmp_MxMa1kAS13s9'
+    """
+
+    chars = string.ascii_lowercase + string.digits
+    if not lowercase:
+        chars += string.ascii_uppercase
+    random_part = ''.join(random.choice(chars) for _ in range(length))
+    randomname = 'tmp_' + random_part
+
+    return randomname
 
 
 def _compare_projection(dic):
@@ -1270,7 +1340,7 @@
 
     :return: directory of mapsets/elements
     """
-    if isinstance(type, python_types.StringTypes) or len(type) == 1:
+    if isinstance(type, str) or len(type) == 1:
         types = [type]
         store_types = False
     else:

and to the remaining files just change file by open

/grass-7.4.3/scripts/db.py
/grass-7.4.3/scripts/v.db.addtable
/grass-7.4.3/scripts/v.dissolve
/grass-7.4.3/scripts/v.lines

note: file() is not supported in Python 3. Using open()

I will apply the changes now!

p/d: If a similar error arises, you already know where to start. Now I want that beer 😄 Ha!

@fjperini @luispuerto I would be more than happy to buy you each a beer this Christmas, if you let me know where to send the money! One for @luispuerto for finding all the errors, and one for @fjperini for fixing them!

I just tested it v.dissolve in QGIS3&2 and it works perfectly in both. 👍

Again, great work!

@nickrobison As there are several changes for Python 3, it might be best to create a grass7-dev formula for QGIS 3 and use it momentarily until GRASS 7.8 / 8.0 arrives.

@luispuerto I’ll give you the formula with the changes and the tests in QGIS 2 and QGIS3.

If something comes up in QGIS 2, we’ll have to use grass7-dev.

Here is a solution mentioned:

https://stackoverflow.com/questions/52269281/fix-import-error-on-using-environb-in-python

# 1. at the top, add an import
import pipes
# 2. remove the `from os import environb as environ` line altogether
# 3. in def parse(), use
cmdline = [basename(sys.argv[0])]
cmdline += (pipes.quote(a) for a in sys.argv[1:])
os.environ['CMDLINE'] = ' '.join(cmdline)

We are very close to solving it. This must be reported to the GRASS developers.

@fjperini you are right… what an eye have! I swear that v.dissolve want the second script I tried after v.buffer. I’ve been trying to use some other scripts —from r.* or i.* toolboxes— but I don’t have anything at hand right now to test them —that I can say the script it’s working properly.

@luispuerto I checked that with v.buffer everything works fine, doing v.dissolve the error TypeError exits again.

It seems that now we have to correct the file: /usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/etc/python/grass/script/core.py. Since it is the last (of GRASS) to run.

@luispuerto We are far, I to a sea. Ha! But soon I will be walking through the old continent.

The integration of R and ORFEO is completed. It will only be necessary to enable them from Manage and Install Plugins and and ready.

I was workining to configure the default folders and activate the plugin when it is enabled:

Manage and Install Plugins -> Installed -> Plugin name (click its checkbox)

I correct the --with-globe integration and apply the changes. With that would be all the corrections to supports that were by default.

Hey!!

Now it work 🎉 but two things.

that error was a config error, probably due to previous tweaking trying to make it work.

I delete the config:

$ rm -r ~/.grass*

Then I also have to disable the v.external option from qgis3 processing options → grass

I can’t see that option in QGIS2… but in qgis2 the same script is faster than in qgis3. I don’t know if everything is exactly the same, since I haven’t thoroughly checked.

BUT…

🎉 IT WORKS! 🎉

(I owe you a beer @fjperini !)

Great! @fjperini are the changes related to the last 3 pull request? I can download them and merge locally in a branch and test them

@luispuerto I’m about to upload the other changes!

As mentioned @msandifo (see), a solution:

a temp fix is to replace tabs with spaces by editing line 1693 in /usr/local/Cellar/grass7/7.4.1/libexec/bin/grass74

which can be applied by @nickrobison #549 :

# Patch grass.py to remove bad tab at line 1693 (and insert 12 spaces)
# Needs to be pushed upstream.
inreplace "lib/init/grass.py", "\t\t\t", " 

or, with the patch that was to apply:

--- a/lib/init/grass.py
+++ b/lib/init/grass.py
@@ -1668,7 +1668,7 @@ def print_params():
             filerev.close()
             sys.stdout.write("%s\n" % val[0].split(':')[1].rstrip('$"\n').strip())
         elif arg == 'version':
-			sys.stdout.write("%s\n" % grass_version)
+            sys.stdout.write("%s\n" % grass_version)
         else:
             message(_("Parameter <%s> not supported") % arg)

Once merged #549, I will apply the other changes.

@luispuerto When the bottle for qt5-webkit is ready, I to apply changes in qgis3 and improve integration with GRASS, R and ORFEO.

@luispuerto This commented @nickrobison in #228

@bolliger32 That’s a really strange error, I can see the incorrect indenting in my grass74 file, but no error in thrown.

Which version of python are you using?

If you remove the tabs at the beginning on the line, and ensure that the indentation is lined up correctly, does the error still occur?

Regarding the problem with PYTHON, I noticed that in src/plugins/grass/qgsgrassshell.cpp:

env << QStringLiteral( "GRASS_PYTHON=python" );

It is very likely that you are calling python3 by default when grass is configured to use python2

Another thing you can do to make sure:

export GRASS_PYTHON=$(brew --prefix)/opt/python2/bin/python2

GRASS addons may fail due to python3 incompatibility. If that is tha case you can change the shebang a the beginning of the script to enforce python2 usage.

#!/usr/bin/env python

Should be changed into

#!/usr/bin/env python2

Try this and then tell me how it turned out!

p/d: Yes, I am working on the implementation of orfeo6, I applied several changes with respect to #461 #462.

@luisspuerto, the problem is in GRASS, @nickrobison will create a patch to solve it.

@luisspuerto The tab/space issue is an upstream problem in grass, I think it was first reported in #228. I’ll try and create a patch and upgrade the grass formula.