mas: 🐛 `mas account` doesn't work on macOS 12 and later

Hi, all!

Any news if this issue is being addressed?

You’re in the right place. Efforts toward resolving this issue will be tracked here. @phatblat and I are both volunteer maintainers with obligations apart from mas maintenance. We are not paid (though I do welcome sponsorship). We do this because we love open source and want to give back to the community. We are also mas users.

Is there a workarround?

We’re not aware of a workaround at this time, other than using the App Store app, of course.

How can we help?

As you’re probably aware, Apple has made some changes to the private frameworks mas uses to manage App Store apps on your Mac. Complicating things, the class-dump tool that was used by the original mas author to generate Objective-C headers from those private frameworks does not support extracting dylibs from a cache, and does not appear to support the x86_64h, arm64, or arm64e architectures. The original mas author is no longer involved in the mas or class-dump projects, and class-dump appears to have been abandoned.

A few things have to happen in order to resolve this issue:

• We need tooling to generate Objective-C headers from current CommerceKit.framework and StoreFoundation.framework. @phatblat has already done some investigation toward this effort, but there’s more to do.
• This will probably require tooling to extract the binary modules of those frameworks from the dylib cache that was introduced in macOS 11 Big Sur. There is some sample dsc_extractor code available from Apple that seems to work well enough for this, but it isn’t (yet?) maintained as a standalone tool. This snippet may have already been built into some third-party tools, so it might not be necessary to publish it as a standalone tool.
• We may or may not need to find or build tooling to support extracting modern arm64/arm64e binaries or generating headers from them.
• We then need to figure out what has changed in these private frameworks in order to determine whether there might be a new path to mas functionality. This may involve making new calls into these frameworks, or dumping headers for some additional framework(s). This one is the trickiest to estimate effort involved. It could be a one-liner, or it could be very involved, or it could be infeasible.

tl;dr: This is going to take time and effort. We welcome any helpful contributions from subject matter experts!

That said, additional reports of mas errors on Monterey aren’t necessary or helpful right now. Please do try to minimize the noise on this thread while we work to solve the problem. We appreciate it!

#428 has been merged, v1.8.5 released, and bottles have been published. This resolves the issue many of us were having on Monterey with install, lucky, and upgrade.

For folks using brew bundle with mas dependencies (as I do), you may need to wait for corresponding changes to Homebrew-bundle.

I will leave this issue open, as the symptom of mas account not working on Monterey remains.

As Monterey has been released, it would be nice if mas supports it (fully).

(I understand it takes a lot of work to play around with private frameworks and fix code for macOS upgrades. Keep up the good work — mas is a very useful tool!)

I think I have a workaround for the issue preventing install, lucky, and I think upgrade from working on Monterey in #428. It doesn’t exactly fix account, because macOS 12 appears to no longer expose this information, or at least not via the same API. However, I suspect most people following this issue are here for redownloads and upgrades. 🤗

Getting new headers generated will still be useful to see what’s changed and to see if we can re-enable account on macOS 12, or possibly even purchase on macOS 10.15 and signin on macOS 10.13. I’m hopeful others with more expertise in that area will be able to meet that goal! 🍀

I ran across Extracting libraries from dyld_shared_cache and was impressed with his/her understanding of the process. Using https://github.com/zhuowei/dsc_extractor_badly I was able to get a CommerceKit x86_64 binary. Neither class-dump nor Dumper were able to work with it, but IDA Free is able to understand this file.

I’ll try updating the headers manually and then see which new methods might return the active account.

Still not fixed?

Just a quick update on my progress:

I’m giving this project a try: https://github.com/ChiChou/IDA-ObjCExplorer

It was a bit broken on Monterey’s dyld_shared_cache and the latest IDA, but I was able to hack on it a bit and git it going.

Unfortunately, the first thing it tries to resolve, it failed on. It parses __objc_protolist and for each entry, tries to resolve name, types, and impl. It wasn’t able to resolve any names. The name pointers look reasonable, but are slightly outside of CommerceKit’s mapped addresses. I’m guessing they point into another library or framework’s segments? Not sure how protocols work really.

Anyway, if my theory is correct, then creating an IDB with all of CommerceKit’s dependencies loaded may help. My original IDB only had CommerceKit itself. So, yesterday afternoon I created a new IDB and had IDA load all dependencies. For anyone unfamiliar with loading dyld_shared_cache in IDA, parsing a library and all its dependencies is non-trivial. The resulting IDB loaded about 500 dylibs and is about 17GB. I let IDA analyze all afternoon and through the night, and it’s still going this morning. Hopefully analysis will finish soon and I can try the above plugin again.

In the mean time if anyone comes up with an easier/faster way, like a class-dump implementation that runs directly on the shared cache, my feelings won’t be hurt. Happy to be preempted by better ideas.

Cheers, Zach

Same issue on macOS 12.0.1, 13" MacBook Pro 2020 M1

Managed to get the function list out of IDA free using the “Copy All” context menu entry.

• CommerceKit.csv
• StoreKit.csv

This still needs additional manual processing to generate usable Objective-C headers.

Shouldn’t it be marked as “not working” on Monterey on brew.sh?

Maybe file that bug with homebrew-core then? https://github.com/Homebrew/homebrew-core

Unfortunately I can’t share the tool since it’s work-related

As for disassembly, I’m using IDA to load the reconstructed dyld_shared_cache, and then disassemble individual dylibs. So, happy to run any IDA python scripts against my IDB and share the results if that helps.

And yeah I filed that bug on Ghidra 😃

Same problem with Release Candidate - macOS 12.0 Beta (21A5552a)

Someone maybe take a look at let me know if we’re getting close

I haven’t found time to dive into this gist yet, but I’m hoping to do so this week. Hopefully there are enough breadcrumbs here to piece together a plan for the original issue reported here, among others.

It’s supposed to be one .h per protocol or class, is that right?

Yep!

My proposals are:

1. Dump out JSON and let others work on a tooling to ingest it (either by modifying class-dump or some other way), or
2. Dump header files as best I can, and let others refine those header files and discard what they don’t need

Option 1 would probably be best in the long run if you were able to publish your IDA scripts as open source. That way, we could try to integrate the functionality directly into class-dump. If you aren’t able to do so, let’s go for the quicker, stop-gap progress of option 2.

Great work on this, @zcutlip. I really appreciate the time and effort you’re putting into this, and I’m sure others here do too.

Yeah that’s really cool. This gets us really close. I’ll see if I can clean up the output to be a proper header file.

Very nice work on ipsw

I’m familiar with your tool, but wasn’t aware of it’s dsc/obj-c capabilities. Giving it at try now. Thanks!

If you have bat installed (brew install bat) you can pipe the output of my tool to: bat -l m --tabs 0 -p --theme Nord --wrap=never --pager "less -S" to make it look amazing.

Okay, think I figured something out. The dyld_shared_cache has table of “preoptimized” strings for all the libraries bundled in the cache. This is located in the libobjc.A:__OBJC_RO segment. It starts out with a table of what appear to be class names, from all across the shared cache. Following that there appears to be a table of selector strings.

ibobjc.A:__OBJC_RO:00000001CA535943 unk_1CA535943   DCB 0xF0                ; DATA XREF: getMethodNoSuper_nolock(objc_class *,objc_selector *)+8C↑o
libobjc.A:__OBJC_RO:00000001CA535943                                         ; fixupMethodList(method_list_t *,bool,bool)+A8↑o ...
libobjc.A:__OBJC_RO:00000001CA535944                 DCB 0x9F
libobjc.A:__OBJC_RO:00000001CA535945                 DCB 0xA4
libobjc.A:__OBJC_RO:00000001CA535946                 DCB 0xAF
libobjc.A:__OBJC_RO:00000001CA535947                 DCB    0
libobjc.A:__OBJC_RO:00000001CA535948 aRetain         DCB "retain",0          ; DATA XREF: _objc_retain_0:loc_18019836C↑o
libobjc.A:__OBJC_RO:00000001CA535948                                         ; _objc_retain_0+70↑o ...
libobjc.A:__OBJC_RO:00000001CA53594F aRelease        DCB "release",0         ; DATA XREF: _objc_release_0:loc_18019A7C8↑o
libobjc.A:__OBJC_RO:00000001CA53594F                                         ; _objc_release_0+8C↑o ...
libobjc.A:__OBJC_RO:00000001CA535957 aRetaincount    DCB "retainCount",0     ; DATA XREF: -[OS_xpc_int64 retainCount]+28↑o
libobjc.A:__OBJC_RO:00000001CA535957                                         ; -[OS_xpc_int64 retainCount]+2C↑o ...
libobjc.A:__OBJC_RO:00000001CA535963 aAutorelease_0  DCB "autorelease",0     ; DATA XREF: _objc_autorelease:loc_18019DC30↑o
libobjc.A:__OBJC_RO:00000001CA535963                                         ; _objc_autorelease+54↑o ...
libobjc.A:__OBJC_RO:00000001CA53596F aTryretain      DCB "_tryRetain",0      ; DATA XREF: -[NSObject retainWeakReference]↑o
libobjc.A:__OBJC_RO:00000001CA53596F                                         ; -[NSObject retainWeakReference]+4↑o ...
libobjc.A:__OBJC_RO:00000001CA53597A aIsdeallocating DCB "_isDeallocating",0 ; DATA XREF: -[NSObject allowsWeakReference]+C↑o
libobjc.A:__OBJC_RO:00000001CA53597A                                         ; -[NSObject allowsWeakReference]+10↑o ...
libobjc.A:__OBJC_RO:00000001CA53598A aCopy_0         DCB "copy",0            ; DATA XREF: _objc_setAssociatedObject:loc_1801A5B34↑o
libobjc.A:__OBJC_RO:00000001CA53598A                                         ; _objc_setAssociatedObject+B4↑o ...
libobjc.A:__OBJC_RO:00000001CA53598F aCopywithzone   DCB "copyWithZone:",0   ; DATA XREF: -[NSObject copy]↑o
libobjc.A:__OBJC_RO:00000001CA53598F                                         ; -[NSObject copy]+4↑o ...

I believe the 0xE10DC5 offset is from the start of that selector string table, which gets us to 0x1cb346708:

ibobjc.A:__OBJC_RO:00000001CB346708 aShareddialogco DCB "sharedDialogController",0
libobjc.A:__OBJC_RO:00000001CB346708                                         ; DATA XREF: CommerceKit:__objc_selrefs:00000001D6F78788↓o

We’ll see if that theory holds true for all the selector strings

I haven’t started StoreFoundation yet. I have a separate IDB for it. I wanted to get the overall process working first,

I’ll check on CKDownloadDirectory & CKDownloadQueueObserver in my CommerceKit IDB and figure out why they aren’t being dumped.

Got ivars going. Here’s a sample

/*
 * CKAccountStore.h
 * ./classdump.py ../build/Release/formatType ./CommerceKit/ ../../IDA-ObjCExplorer/CommerceKit.json
 */


@interface CKAccountStore
{
    CKStoreClient *_storeClient _storeClient;
}

// Class methods
+ (id)accountStoreForStoreClient:(id)arg1;
+ (id)sharedAccountStore;

// Instance methods
- (void).cxx_destruct;
- (id)accounts;
- (id)primaryStoreAccount;
- (void)addAccount:(id)arg1;
- (id)primaryAccount;
- (id)accountWithAppleID:(id)arg1;
- (_Bool)isDemoModeEnabled;
- (id)initWithStoreClient:(id)arg1;
- (id)demoAccount;
- (id)storeClient;
- (id)storeAccountForDSID:(id)arg1;
- (id)addAccountObserver:(id)arg1;
- (id)_initWithStoreClient:(id)arg1;
- (id)knownAccounts;
- (void)signOutWithCompletionHandler:(id /* CDUnknownBlockType */)arg1;
- (void)removeAccountObserver:(id)arg1;
- (id)storeAccountForAppleID:(id)arg1;
- (void)getEligibilityForService:(long long)arg1 completionBlock:(id /* CDUnknownBlockType */)arg2;
- (_Bool)primaryAccountIsPresentAndSignedIn;
- (void)signOut;
- (id)addPrimaryAccountObserverWithBlock:(id /* CDUnknownBlockType */)arg1;
- (void)removePrimaryAccountObserver:(id)arg1;
- (id)accountForDSID:(id)arg1;
- (void)signIn;
- (void)signInWithSuggestedAppleID:(id)arg1 allowChangeOfAppleID:(_Bool)arg2 completionHandler:(id /* CDUnknownBlockType */)arg3;
- (void)viewAccount;
- (id)eligibilityForService:(long long)arg1;
- (void)getPasswordSettingsWithCompletionBlock:(id /* CDUnknownBlockType */)arg1;
- (void)updatePasswordSettings:(id)arg1 completionBlock:(id /* CDUnknownBlockType */)arg2;
- (void)setTouchIDStateForAccount:(id)arg1 state:(long long)arg2 completionBlock:(id /* CDUnknownBlockType */)arg3;
- (void)getTouchIDStateForAccount:(id)arg1 completionBlock:(id /* CDUnknownBlockType */)arg2;


@end  /* CKAccountStore */

I haven’t looked at properties yet, but ivars weren’t too bad. Hopefully properties will be equally straightforward.

I’m attaching a zip file of all the headers I’ve spit out so far, along with the JSON I spit out from IDA.

Cheers

EDIT: Note, many/most of the headers in the zip file are superfluous, but as mentioned previously, I’m not sure of a way to programmatically filter out the ones that don’t need to be generated. So for now they’re all in there, and we can work on a better way next.

commercekit.zip

I’m now writing out individual .h files per protocol & class, but I’m now realizing there’s a lot more to it. I’m currently dumping about 90-ish header files from CommerceKit alone, because there are lots of class and protocol structures in the binary that don’t need to be dumped

I studied what class-dump does, and it’s quite sophisticated. It actually builds a dependency graph and then is able to do things like add forward declarations and imports as well as work out what protocols & classes it doesn’t need to dump. I don’t think I’ll be able to get to that level of refinement.

My proposals are:

1. Dump out JSON and let others work on a tooling to ingest it (either by modifying class-dump or some other way), or
2. Dump header files as best I can, and let others refine those header files and discard what they don’t need

Here’s an example of one of the header files I dumped, CKAuthenticationSettings.h:

@interface CKAuthenticationSettings : NSObject
{
}

// Class methods
+ (_Bool)supportsSecureCoding

// Instance methods
- (id)copyWithZone:(struct _NSZone *)arg1
- (id)init
- (void)encodeWithCoder:(id)arg1
- (id)initWithCoder:(id)arg1
- (void).cxx_destruct
- (id)_password
- (_Bool)showHelp
- (_Bool)createSession
- (void)setShowHelp:(_Bool)arg1
- (id)suggestedUsername
- (void)setSuggestedUsername:(id)arg1
- (id)authenticateArguments
- (void)setAuthenticateArguments:(id)arg1
- (id)createAccountArguments
- (void)setCreateAccountArguments:(id)arg1
- (_Bool)forceAccount
- (void)setForceAccount:(_Bool)arg1
- (void)setCreateSession:(_Bool)arg1
- (void)set_password:(id)arg1
- (_Bool)_loginToiCloud
- (void)set_loginToiCloud:(_Bool)arg1


@end  /* CKAuthenticationSettings */

I still need to do ivars and a couple other things, but that shouldn’t be too bad.

Cheers

Oh, I think I get the organization. It’s supposed to be one .h per protocol or class, is that right?

No need to apologize. Your expertise, efforts, and time are appreciate.

Slower progress than I’d hoped, but getting closer. I was able to get the IDA plugin I’m working on to mostly resolve all the protocol and class structures.

Where it’s currently choking is class & instance method names. It’s able to find the type strings as well as the actual method IMPL

IDA can clearly work these out, as @phatblat as demonstrated. I’ll see if there’s a way to either ask IDA “what’s the Obj-C method name for this IMPL?” or somehow parse @phatblat’s CSV data.

Regarding typing, this plugin isn’t currently rendering that, but I think the type encoding is well understood, and we’re finding the type strings. So shouldn’t be hard to fix up the output to look like a real objc header.

IDA output attached below ida-output.txt

cheers

@zcutlip That’s a good question. I haven’t ever looked for these private frameworks on iOS, but that did have an app store first so these low-level frameworks may be used on both platforms.

I haven’t ever used class-dump on iOS frameworks. Work I do on that platform is to ship to the App Store so we steer clear of Apple’s private stuff.

Just a thought: do these frameworks exist on iOS? DyldExtractor (linked above) does a pretty good job of extracting dylibs from iOS’s shared cache. They’re not actually linkable/usable at run-time but they’re very complete for static analysis purposes. If class-dump generally works on iOS libraries, and the frameworks you need exist on iOS, would that be an angle worth looking at? Happy to give it at try.

In other news: IDA FINALLY finished analyzing my 17GB IDB tonight. Just manually poking around, it seems like obj-c xrefs that weren’t resolving in the “single-module” IDB are now resolving in the “I pulled on the thread until the whole sweater unravelled” IDB. It’s a bit late for me to go back down the rabbit hole, but I’ll have a look in the morning, and see what there is to see.

Cheers

Shouldn’t it be marked as “not working” on Monterey on brew.sh? It’s confusing and creates unnecessary noise. Sorry for my snark earlier.

I’ll see what I can do. I’m not super familiar with the class-dumping side of it. If there’s something already out there, particularly in python, I’ll see if I can adapt it.

Regarding dyld_shared_cache, here are a couple resources for informational purposes. Neither of these help move the ball down the field, but hopefully they offer a bit of insight into the problem space.

There’s a project called DyldExtractor which unfortunately doesn’t work on macOS shared caches; it’s iOS only. But looking through it can give you some insight into how hard a problem it is to reconstruct the dylibs. It’s a pure python project that does not use dsc_extractor. It parses the shared cache and attempts to reconstruct the dylibs as faithfully as possible. https://github.com/arandomdev/DyldExtractor

And here’s a project that essentially dlopen()s dsc_extractor from Xcode to extract individual dylibs from the shared cache. Since it relies on Xcode’s dsc_extractor it does support the new split format, but that also means it spits out broken dylibs. This doesn’t get you any farther that what @phatblat probably has done, but just linking here as a generally useful tool. https://github.com/keith/dyld-shared-cache-extractor

cheers, zach

When you use dsc_extractor to dump a dylib from the shared cache, it doesn’t really give you a proper dylib. Lots of things get rewritten during the linking process that generates the shared cache, and dsc_extractor doesn’t undo most of that. So, you mostly get a mach-o dylib that, at best, superficially resembles the original, but is mostly broken internally. @phatblat, you probably found lots of broken xrefs in IDA.

I’m not sure what the specific issues are with class-dump but I wouldn’t be surprised if that’s part of it.

Afaik, IDA still doesn’t support the new dyld shard cache format in Monterey (split across several slices), but I have tooling that allows me to re-join the slices. I can then load it up and disassemble individual dylibs cleanly, for the Apple Silicon version (not the intel version, yet).

I don’t know much about class-dump but if there exists (or someone wants to write) an IDA python script that replicates what it does, I can give it a try and share its output. Or really any IDA script that might give you useful info.

Same here (with same macOS Monterey version). I noticed to it stopped working after it updated to Beta 7, mas had been working on previous betas without issue.