fastlane: Crashlytics hidden stack trace
New Issue Checklist
- Updated fastlane to the latest version
- I read the Contribution Guidelines
- I read docs.fastlane.tools
- I searched for existing GitHub issues
Issue Description
We are having a problem with Crashlytics not resolving symbols in our builds created and uploaded using Fastlane. I think it started after upgrading to XCode 9 because I know it used to work before. I have read all docs around this extensively and tried everything I could think of but haven’t found a solution. I even had a long running support issue with Fabric about it. They say that it must be a problem with Fastlane and Gym since the symbols resolve fine when we build the same configuration (Adhoc) using XCode and submit with the Fabric app.
I have examined the dSYM files produced during the build and as far as I can see (using dwarfdump) the method and class names used in our code are there.
I really need some ideas of what settings in gym/fastlane or the project file that could affect this and cause these problems.
Here are some more details:
- Symbol files have been uploaded successfully and Crashlytics does not state that any symbol files are missing
- We are using bitcode
- We have set debug format to “DWARF with dSYM File”
- When we build and upload to Testflight using Fastlane, and then re-upload symbols from Apple servers to Crashlytics, it works fine
- I tried adding “include_symbols: true” in the Fastfile but it seems this is the default and it did not make a difference
- Also tried changing “Symbols Hidden by Default” to NO in the project file but that didn’t help
I’m reluctant to post our complete Fastfile because it is large, but here is a snippet with the parts relevant for the Adhoc build.
Fastfile
desc "Build AdHoc version"
lane :adhoc do
sh("rm -f ../build/*")
match(type: "adhoc", readonly: true)
build(configuration: "AdHoc", export_method: "ad-hoc", internal_beta: true)
end
private_lane :build do |options|
gcc_flags = options[:gcc_flags]
if options[:internal_beta]
gcc_flags = "#{gcc_flags} INTERNAL_BETA=1"
end
gym(
scheme: "xxxx",
configuration: options.fetch(:configuration),
clean: true,
silent: true,
# suppress_xcode_output: true,
xcargs: "GCC_PREPROCESSOR_DEFINITIONS='$(inherited) #{gcc_flags}'",
output_directory: "./build",
export_method: options.fetch(:export_method),
include_bitcode: true,
)
end
desc "Upload build and symbols to Crashlytics"
lane :fabric do |options|
crashlytics(api_token: 'xxxx',
build_secret: 'xxxx',
groups: ['xxx'],
ipa_path: options.fetch(:ipa_path),
notifications: true)
upload_symbols_to_crashlytics(
api_token: 'xxxxx',
dsym_path: options[:dsym_path],
binary_path: options.fetch(:binary_path),
dsym_worker_threads: 5)
end
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 1
- Comments: 31 (4 by maintainers)
I was looking into this with @alexpersian and the Crashlytics team earlier today. We’re still trying to figure out the root cause, but as a short-term fix, it’s possible to update the original dSYMs using the BCSymbolMaps.
Running:
dsymutil --symbol-map <bcSymbol-file> <dsym-file>will re-map the symbols for the original dSYM, removing the __hidden frames. You can then upload the re-mapped dSYM and future crashes will show symbolicated traces.
Apparently you can provide the option inline:
When the option is enabled, on top of your original dSYM files it produces a dSYM for every library + a dSYM for every libswift*.dylib. Actually multiply that number by 2 if you are still building for 2 architectures (arm64 and armv7) as each UUID dSYM file contains only one architecture. So that may explain the difference in their number.
If you are only distributing a natively compiled app yourself though Ad-Hoc/Enterprise, what is the benefit you get from enabling bitcode in the first place? Your compilation produced both native arm64 and bitcode, then when extracting you drop the native and recompile bitcode into arm64 again.
From my logs, it appears that I have the same issue as @cyupa and @nebiros the
dsymutildoesn’t find the correspondingsymbolmapfiles. All the files inside theBCSymbolMapsdirectory have 36 character alphanumeric names. As an example:6D054CF4-4DD7-3047-A061-99EA02D2ACB5.bcsymbolmapwhich is nowhere near the expected:AlamofireNetworkActivityLogger-armv7.bcsymbolmapAlamofireNetworkActivityLogger-arm64.bcsymbolmapI’m having the same issue except I don’t get any status code. It seems that it doesn’t manage to map the dSYM files to the BCSymbolMaps. I also ran gym with
"compileBitcode": false. I uploaded them manually in the end but I still end up with a couple of missing dSMYs.Hope this helps 😃
Fastlane env:
✅ fastlane environment ✅
Stack
System Locale
fastlane files:
`./fastlane/Fastfile`
`./fastlane/Appfile`
fastlane gems
Loaded fastlane plugins:
Loaded gems
generated on: 2018-03-05
Wanted to drop in with an update on my search. I have been able to narrow it down to the swift from Xcode 8.3.3 to Xcode 9.0. Going back to the commit before our Swift 4 conversion and running the same
gymcommand using each version produces different results.Xcode 8.3.3: No
.bcsymbolmapfiles included in the compressed dSYM package. No__hiddensymbols Xcode 9.0:.bcsymbolmapfiles included, and symbols are obfuscated with__hiddenAgain the only thing changing in this comparison is the Xcode version. After inspecting the build settings for each the differences are a couple of added build settings from Xcode 9.0.
The problem seems to happen only in the ad-hoc export method in XCode9. When it is used, all the components are being recompiled from bitcode, and swift symbols are being removed (effectively rebuilding all the runtime libswift*.dylib files and generating new dSYMs for them as well). These are new defaults: https://www.dropbox.com/s/8zkuy5awk3tkw9w/Screenshot 2017-11-22 13.06.47.png?dl=0
The xcodebuild flags that should be put into the generated plist (-exportOptionsPlist) to prevent that from happening are:
Luckily, it is possible to override the plist options altogether for now. That may be a good interim solution until the two options are added to gym (sound like they should be added):
The contents of ExportOptions.plist (copy the existing one and add the 2 flags there):
no need to remap dSYM files back, everything is intact.
Good news everyone! The short-term fix that @alexsinger mentioned above worked. It’s a slow, manual process at this point but it does work and we are able to upload the non-obfuscated symbol files to Fabric for our future builds.
In order to symbolicate manually:
Resourcesfolder within that contains a plist file named the same as the dSYM. This has a key in it calledDBGOriginalUUIDwhich holds the name of the BCSymbolMap file needed to symbolicateThis should symbolicate the file so that it can be uploaded to Fabric. I haven’t found an automated solution for this work yet, but when I do I’ll post it here.
To add some further information I’ve tested with the fastlane gym build in two states, Bitcode ON and Bitcode OFF.
__hiddensymbols in UUID files__hiddensymbols since they are only contained in the UUID filesThis is something new with Xcode 9 as we’ve had Bitcode enabled forever. Unfortunately I can’t regress and gather information from Xcode 8.3 since our project was converted to Swift 4.
About the use of bitcode… In our case we have to use it because we have an Apple Watch app and it requires bitcode compilation.
@alexpersian my bad, I thought we were talking about a testflight or app store build.
Seeing the same issues with our application and have been chasing down potential causes for the last week. The “more details” section of the original post matches all of the things we’ve tried so far to resolve this.
This seems to be exclusive to the UUID based dSYM files, and the
.app.dSYM/.framework.dSYMfiles are properly symbolicated.__hidden@taquitos this is also unrelated to the
download_dsymsaction AFAIK since the files are never uploaded to iTC during the Fabric based build flows.