fastlane: Running xcrun extractLocStrings as part of Fastlane causes crash for duplicated high-codepoint unicode localisable strings

New Issue Checklist

Issue Description

We run xcrun extractLocStrings as part of our fastlane workflow and we have a project where some of the localisable strings contain high-codepoint unicode characters such as “µm” where the µ is the high-codepoint character. This string is localisable in multiple parts of the codebase and by default xcrun extractLocStrings outputs standard output warning you that you have used multiple comments for those strings.

When fastlane outputs to screen the first instance of one of the warnings where the localisable string contains a high-codepoint character, it crashes with the stacktrace shown further below.

I was able to supress the output of the warnings from xcrun extractLocStrings by using the flag -q but this is not the default and fastlane should be able to handle the output generated by any tool.

Command executed

bin/fastlane test

Complete output when running fastlane, including the stack trace and command used
/Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane_core/lib/fastlane_core/ui/implementations/shell.rb:71:in `split': [!] invalid byte sequence in UTF-8 (ArgumentError)
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane_core/lib/fastlane_core/ui/implementations/shell.rb:71:in `command_output'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane_core/lib/fastlane_core/ui/ui.rb:17:in `method_missing'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:57:in `block (2 levels) in sh_control_output'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:56:in `each'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:56:in `block in sh_control_output'
	from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/open3.rb:205:in `popen_run'
	from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/open3.rb:190:in `popen2e'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:54:in `sh_control_output'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:16:in `sh_no_action'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/fast_file.rb:213:in `block in sh'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/actions/actions_helper.rb:50:in `execute_action'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/fast_file.rb:212:in `sh'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/fast_file.rb:206:in `sh'
	from Fastfile:145:in `block (2 levels) in parsing_binding'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/lane.rb:33:in `call'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:204:in `try_switch_to_lane'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:146:in `trigger_action_by_name'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/fast_file.rb:154:in `method_missing'
	from Fastfile:176:in `block (2 levels) in parsing_binding'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/lane.rb:33:in `call'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:49:in `block in execute'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:45:in `chdir'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:45:in `execute'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/lane_manager.rb:56:in `cruise_lane'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/command_line_handler.rb:36:in `handle'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/commands_generator.rb:108:in `block (2 levels) in run'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in `call'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in `run'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in `run_active_command'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in `run!'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/commands_generator.rb:333:in `run'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/commands_generator.rb:41:in `start'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/cli_tools_distributor.rb:119:in `take_off'
	from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/bin/fastlane:23:in `<top (required)>'
	from /Users/my_user/buildAgent/work/1234567890abcdef/bin/fastlane:29:in `load'
	from /Users/my_user/buildAgent/work/1234567890abcdef/bin/fastlane:29:in `<main>'

Environment

✅ fastlane environment ✅

Stack






Key
Value






OS
10.14.6




Ruby
2.3.7




Bundler?
true




Git
git version 2.20.1 (Apple Git-117)




Installation Source
bin/fastlane




Host
Mac OS X 10.14.6 (18G87)




Ruby Lib Dir
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib




OpenSSL Version
LibreSSL 2.6.5




Is contained
false




Is homebrew
false




Is installed via Fabric.app
false




Xcode Path
/Applications/Xcode1030.app/Contents/Developer/




Xcode Version
10.3






System Locale






Variable
Value







LANG
en_GB.UTF-8
✅




LC_ALL






LANGUAGE








fastlane gems






Gem
Version
Update-Status






fastlane
2.128.1
✅ Up-To-Date






Loaded fastlane plugins:






Plugin
Version
Update-Status






fastlane-plugin-appcenter
1.2.1
✅ Up-To-Date






Loaded gems





Gem
Version






bundler
1.16.6




io-console
0.4.5




rake
12.3.2




CFPropertyList
3.0.0




concurrent-ruby
1.1.5




i18n
0.9.5




minitest
5.11.3




thread_safe
0.3.6




tzinfo
1.2.5




activesupport
4.2.11.1




public_suffix
2.0.5




addressable
2.6.0




eventmachine
1.2.7




websocket-extensions
0.1.4




websocket-driver
0.7.1




faye-websocket
0.10.9




childprocess
1.0.1




rubyzip
1.2.3




selenium-webdriver
3.142.3




appium_lib_core
3.2.0




mini_portile2
2.4.0




nokogiri
1.10.3




tomlrb
1.2.8




appium_lib
10.3.1




atomos
0.1.3




babosa
1.0.2




claide
1.0.3




colored
1.2




colored2
3.1.2




colorize
0.8.1




highline
1.7.10




commander-fastlane
4.4.6




declarative
0.0.10




declarative-option
0.1.0




diff-lcs
1.3




digest-crc
0.4.1




unf_ext
0.0.7.6




unf
0.1.4




domain_name
0.5.20190701




dotenv
2.7.5




emoji_regex
1.0.1




excon
0.66.0




multipart-post
2.0.0




faraday
0.15.4




http-cookie
1.0.3




faraday-cookie_jar
0.0.6




faraday_middleware
0.13.1




fastimage
2.1.5




gh_inspector
1.1.3




jwt
2.1.0




memoist
0.16.0




multi_json
1.13.1




os
1.0.1




signet
0.11.0




googleauth
0.6.7




httpclient
2.8.3




mime-types-data
3.2019.0331




mime-types
3.2.2




uber
0.1.0




representable
3.0.4




retriable
3.1.2




google-api-client
0.23.9




google-cloud-env
1.2.0




google-cloud-core
1.3.0




google-cloud-storage
1.16.0




json
2.2.0




mini_magick
4.9.5




multi_xml
0.6.0




plist
3.5.0




security
0.1.3




naturally
2.2.0




simctl
1.6.5




slack-notifier
2.3.2




terminal-notifier
2.0.0




unicode-display_width
1.6.0




terminal-table
1.8.0




tty-screen
0.7.0




tty-cursor
0.7.0




tty-spinner
0.9.1




word_wrap
1.0.0




nanaimo
0.2.6




xcodeproj
1.12.0




rouge
2.0.7




xcpretty
0.3.0




xcpretty-travis-formatter
1.0.0




fastlane-plugin-appcenter
1.2.1




handlebars-source
4.0.14




libv8
3.16.14.19




ref
2.0.0




therubyracer
0.12.3




handlebars
0.8.0




netrc
0.11.0




rest-client
2.0.2




rspec-support
3.8.2




rspec-core
3.8.2




rspec-expectations
3.8.4




rspec-mocks
3.8.1




rspec
3.8.0




rubyXL
3.4.5




ruby_rncryptor
3.0.2








generated on: 2019-08-07

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 43 (28 by maintainers)

Most upvoted comments

Thanks for the feedback @rogerluan I will make the branch tidier and add some tests in the coming couple of weeks and submit a pull request. It would be good to see a partial fix – obviously it relies on the encodings of inputs being in the list in the code, but at least the encoding of the strings is being considered. I did do some manual testing of this, but not as a complete run through. The original reason for submitting this bug isn’t relevant to my current work any more, but I still want to get it fixed.

+2
sdjmchattie on Jan 10, 2021

I’ve been meaning to look at this myself for a very long time but haven’t yet gotten around to it. I have my suspicions though that this might be to do with unicode normalisation. As you’ve already noticed, there are different ways to represent the micro sign in unicode. I suspect the one in my text.txt file is one kind of them and split doesn’t like it. It may be possible to use unicode normalisation before using split to make sure split only receives unicode that it can split. https://www.honeybadger.io/blog/ruby-unicode-normalization/ looks like a good summary of unicode normalisation in Ruby.

+2
sdjmchattie on Nov 10, 2020

I can confirm the bug still exists in Fastlane version 2.134.0

+2
sdjmchattie on Oct 18, 2019

This file reproduces the error by including sh('cat text.txt') in the Fastfile.

text.txt

+2
sdjmchattie on Aug 8, 2019

This is being fixed in a pull request I have open.

+1
sdjmchattie on Feb 10, 2021

It would be good to get your feedback on my attempt at fixing this issue though @rogerluan . If you think it’s good, then I would be delighted to put in a pull request for it.

https://github.com/fastlane/fastlane/compare/master...sdjmchattie:shell-stdout-fix-15153

+1
sdjmchattie on Jan 3, 2021

Hi @rogerluan thanks for spending so much time on this issue. I took some time today to fiddle around with this myself and I can see where the error occurs and I can see that it is indeed an issue with encodings. Specifically, like you said genstrings is outputting UTF-16 and then Ruby tries to split it using UTF-8 which causes code points to be encountered that aren’t valid in UTF-8. I thought I had a fix, but have now realised it was a spurious fix. I’m not 100% sure, but I think each line of output from PTY could be in any encoding and its hard to work out which encoding a line has from just the bytes of the string. I don’t think there is a good fix for this in fastlane after all.

I think your suggestion to update the documentation would be good. And probably some way of getting the thrown error to tell the user that this might be the issue so they will actually notice and read the documentation about it. My only concern with doing this is that we aren’t reading the output of genstrings from a file before sending it to fastlane. It is what genstrings has generated as standard output, but I guess/hope you can redirect the standard output with a | to be converted to UTF-8 before sending to fastlane which circumvents the issue.

Interestingly, it seems like maybe the test file I supplied is actually encoded as ISO-8859-1 and not UTF-16, which may be my mistake when I tried to create a file that would recreate the issue. When I tell Ruby to convert the output from ISO-8859-1 to UTF-8 with message.encode('UTF-8', 'ISO-8859-1', invalid: :replace) then the standard output is correctly split and reported. This only works for this one case though, and wouldn’t necessarily work with something encoded as UTF-16.

+1
sdjmchattie on Jan 3, 2021

I can confirm the bug still exists in Fastlane version 2.130.0

+1
sdjmchattie on Sep 9, 2019
Read more comments on GitHub
← fastlane: Resign error, when using match xcode 8
fastlane: Runtime error archiving  →