So you need to manually sign and notarize a MacOS application, and it isn’t working out of the box. Here’s the steps that worked for me, in particular for a Gamemaker VM build of a project. Obviously this could be automated via a shell script, if you so desire.
First, manually sign the application. Using --force and --deep make it replace and be recursive, both of which we want here. Having --timestamp is required for notarization, and was not present in Gamemaker's default signing. Using options=runtime will set it to use hardened runtime, which is also required for notarization. Note that the certificate you sign with has to be a Developer ID Application and you MUST have the private key locally. If you can't open the key listing and see someone's name underneath in the keychain, don’t expect it to work because that’s only a public key.
You’ll want to have an entitlements file. This needs to contain at least the com.apple.security.automation.apple-events if you're doing a hardened runtime, and past experience has shown me that bluetooth and USB access are generally necessary for gamepad. An example entitlements file is here, though those bottom few may not all be needed depending on your game and engine.
codesign --sign "Developer ID Application: COMPANY NAME, LLC" "Your Game.app" --force --deep --timestamp --options=runtime --entitlements game.entitlements
After this is done, zip the app. I read in more than one place to use this command-line rather than the right-click menu’s “compress” feature, though I’m not sure that’s really true. Still, easy enough.
/usr/bin/ditto -c -k --keepParent "Your Game.app" YourGameBuildMacSteam_Resigned.zip
Now that you’ve signed the build and made a new archive of that, you’ll need to submit it for notarization. Note that asc-provider should match the organization of the Developer ID application in the first step, and I’ve obfuscated the one I used. The password is for an app-specific password, which you probably have around but you’ll need to create one if not. Finally, you can add the password to your keychain with an alias so it doesn't end up in the command-line directly, which is what I did here. That’s important if you want to write a shell script for these steps and check it in.
xcrun altool --notarize-app --primary-bundle-id "com.company.yourgame" --username flast@company.com --password "@keychain:APP_PASSWORD" --asc-provider J93FFYZ67E --file YourGameMacSteam_Resigned.zip
This command will output a randomly generated ID, keep that around. Now you wait.
If things go well, you’ll get an e-mail from Apple after a while saying notarization is a success. If notarization succeeds, you'll need to staple the result to the app. Note that this is the app, not the zip. I assume it figures out what to do based on the signature.
xcrun stapler staple "Your Game.app"
At this point, you should be able to upload your application to Steam, or zip it (remember, your existing zip is now out of date) and distribute it.
If notarization failed, the e-mail from Apple will say as much. You can use altool to ask for reasons of the failure, using that randomly generated ID that you hopefully kept around.
xcrun altool --notarization-info fdeb1abc-25ae-43f1-8bca-97b89078196c --username flast@company.com --password "@keychain:APP_PASSWORD”
Assuming the process is completed, this will give you a URL you can paste into a web browser. Sometimes the listed reasons are clear and other times not. If the process isn’t yet complete, it’ll say so.
