Bug 29180 - Rogue iTunesMetadata.plist causes issues with StoreKit on iOS7
Summary: Rogue iTunesMetadata.plist causes issues with StoreKit on iOS7
Status: RESOLVED FIXED
Alias: None
Product: iOS
Classification: Xamarin
Component: MSBuild ()
Version: XI 8.9.x (iOS 8.3)
Hardware: PC Mac OS
: High critical
Target Milestone: 8.10.1 (C5SR1)
Assignee: Jeffrey Stedfast
URL:
: 27843 ()
Depends on:
Blocks:
 
Reported: 2015-04-17 05:34 UTC by Johannes Rudolph
Modified: 2015-09-30 09:20 UTC (History)
12 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Johannes Rudolph 2015-04-17 05:34:51 UTC
Exec Summary: Apps build with Xamarin.iOS of the specified version are not able to retrieve data from the AppStore about InAppPurchases via SKProductsRequest. 

* In the Sandbox/Test environment we can query product ids just fine and receive proper responses, on iOS 7 and iOS8 devices
* The App passed review, we're already successfully selling items, so it's working for some users.
* Yet, we have customers reporting that they are unable to see anything in our In App Store. They all have in common that they're running iOS 7 devices.

Please see the details in the following thread: https://forums.xamarin.com/discussion/35500/storekit-returns-invalid-product-identifiers-only-on-the-real-app-store-only-on-ios7

The underlying cause is that Xamarin creates an iTunesMetadata.plist file for each App bundle (see thread above for details). 

Reproduction: 
- Create new Xamarin.iOS App, select iPhone deployment target and build archive. Examine the AppBundle in the generated *.xcarchive, you will see it contains an iTunesMetadata.plist file. 

Desired behaviour: 
- An iTunesMetadata.plist file should not be generated

Possible Fix: 
_CompileITunesMetadata target in Xamarin.iOS.Common.targets should check whether an iTunesMetadata.plist needs to be generated (I think this should only be the case if $(BuildIpa) is true)

Here's my workarond:

<Import Project="$(MSBuildExtensionsPath)\Xamarin\iOS\Xamarin.iOS.CSharp.targets" />
  <!-- NOP out CompileITunesMetada task, which creates a rouge metadata plist file that can break In App Purchases on iOS7 -->
  <Target Name="_CompileITunesMetadata" DependsOnTargets="_DetectSdkLocations;_DetectAppManifest;_GenerateBundleName;_CompileAppManifest">
     <Message Text="Skipping CompileITunesMetada task, which creates a rouge metadata plist file that can break In App Purchases on iOS7" />
  </Target>
Comment 1 Sebastien Pouliot 2015-04-17 09:58:29 UTC
Thanks for the extra details.

-> Jeff

-> Paola, please add this case to the test suite
Comment 2 Jeffrey Stedfast 2015-04-17 10:30:54 UTC
The problem here is that you can build an IPA from the xcarchive, at which point it needs the iTunesMetadata.plist file... so we can't rely on the BuildIpa variable.
Comment 3 Jesper 2015-04-20 13:43:18 UTC
We have had exactly the same problem for about 2-3 months. Apple has helped us and found that there Were 4 cases with the same problem. All were Xamarin developed apps. Moving forward Apple has decided not to approve apps with the itunesmetadata.plist anymore. So this is now even more critical.
Comment 4 Johannes Rudolph 2015-04-20 14:29:10 UTC
I know things like this can and will happen, but I'm seriously pissed about it though and hope this is understandable.

This problem has cost us significant developer time (about 10 man-days) and considerable revenue so far which we may not be able to make up for now that we worked around the problem. 

I'm happy to see that Apple is soon able to make developers of the issue before they learn about it the hard way.
Comment 5 Jesper 2015-04-20 16:03:11 UTC
Yes. I agree. As you say these things will happen. In our case we have probably lost 1-2 man days + perhaps 20% revenue for 2-3 months. Good thing this will be solved. In the past I have been underwhelmed by Apples support. But this time it was really amazing. Hope Xamarin will solve it soon. In the mentioned I will use your work around (have you released a version of your app that works with that ?)
Comment 6 Jesper 2015-04-20 17:16:47 UTC
BTW. I saw another Bugzilla bug (27843) which sounds similar. In that bug report they blame the unified API. Just to be clear. We do not yet use the unified API. I don't know if you do in your App Johannes ?
Comment 7 Rolf Bjarne Kvinge [MSFT] 2015-04-21 03:35:40 UTC
*** Bug 27843 has been marked as a duplicate of this bug. ***
Comment 8 Johannes Rudolph 2015-04-21 03:49:37 UTC
I'm using the unified API. We spent some time digging down into the unified bindings as well and eventually ended up releasing a version of our App that had a custom binding to StoreKit. This didn't fix the issue.

We've created a version with the workaround mentioned above that prevents adding an iTunesMetadata.plist to the App bundle. This is currently in review, but I'm positive this will resolve the issue for us. 

Our experience with Apple was horrible at first (iTunes provider support, these guys aren't technical at all) with multiple > 30min phone calls until we finally got fed up and opened a DTS incident which was professionally adressed (but it was a back and forth of about 6 weeks). 

Our DTS engineer has been fantastic, but of course he needed to wait for the internal engineering teams to triage this bug, which took quite long. We've been on this issue for about 8 weeks now total.
Comment 9 Jeffrey Stedfast 2015-04-21 16:05:31 UTC
I've fixed the Publishing Workflow logic in Xamarin Studio to remove the iTunesArtwork and iTunesMetadata.plist files when creating .ipa files for AppStore distribution.

I'm not sure if we need to change the logic in the MSBuild targets as well? This will get REALLY tricky seeing as how we have no idea how the builds will get used. For example, if the user archives the build and then later decides to create an .ipa from the archived build, we need the iTunes* files to be in the archive.
Comment 10 Johannes Rudolph 2015-04-21 16:07:57 UTC
@Jeffrey: Our usual CI script is to build an xcarchive and deploy from there to the AppStore. So what you propose would be quite dangerous. Also, you should get the same behavior when building from xbuild (cmdlline) and Xamarin Studio. 

How does Apple do it with Xcode?
Comment 11 Jeffrey Stedfast 2015-04-21 16:13:05 UTC
I'm not talking about *builds* from Xamarin Studio, I'm talking about Xamarin Studio's Publishing Workflow (which is a set of dialogs that you use from the Archives View window to publish your app to the AppStore, etc).

How exactly does your CI script work? Does it set the BuildIpa configuration option to true and deploy the .ipa generated by the MSBuild targets? Or does it do something with the xcarchive to create the .ipa?
Comment 12 Johannes Rudolph 2015-04-21 16:20:52 UTC
We do it like this: 

    # generate archives
    # can only build and copy last generated archive (not the best way, but works), so doing this twice
    $(MDTOOL) archive -p:Project -c:"Release|iPhone" MySolution.sln
    sh copy-archive.sh $(ARTEFACTS) Project

And our build server preserves the xcarchive in $(ARTEFACTS) in its build history. When it's time to push to the AppStore, I download the xcarchive from Teamcity and open it on the mac that has our AppStore signing keys. This imports the xcarchive file into xcode organizer. I then use xcode organizer to resign and upload to the AppStore. 

The BuildIpa option is set to false (I believe, it's actually not set at all, i.e. left at default value) in all our release builds, and we build using an AdHoc provisioning profile. In the workflow above, xcode takes care of resigning for AppStore. 

We do also occassionally deploy IPAs from the same xcarchive via http://www.diawi.com/ so we'd definitely want Xamarin to support both AppStore and AdHoc deployments from the same xcarchive via xcode. It should be possible for native Objective-C Apps too in some way. 

Does that make sense?
Comment 13 Jeffrey Stedfast 2015-04-21 16:27:17 UTC
Okay, so you are using Xcode's "Publishing Workflow" (it's not called that, but it's the same idea).

So unfortunately we have no control over the way Xcode creates the .ipa's from the archive, so I think the only way to get this to work will be to use Xamarin Studio's Publishing Workflow instead.

I'll make sure that Xamarin Studio 5.9's .ipa creation logic is correct (I've only fixed it in git master so far).
Comment 14 Jeffrey Stedfast 2015-04-21 16:28:37 UTC
FWIW, Xcode's Archive publishing logic somehow pulls the iTunesArtwork and iTunesMetadata.plist files from the project and not from anywhere in the archive afaict, which is why it can't possibly work for Xamarin projects in the same way.
Comment 15 Johannes Rudolph 2015-04-21 16:34:39 UTC
Hm, I remember having issues with icons in Ipa builds but never worried too much about since everything passed AppStore verification. 

Good to now that we'll have to switch to publishing everything from Xamarin Studio. It's very convenient though to be able to create xcarchive files and download them from your buildservers for deployment, so I'd be very pleased if you can preserve this feature with the Xamarin Studio workflow.
Comment 16 Jeffrey Stedfast 2015-04-21 16:38:50 UTC
You will be able to keep your xcarchive logic (Xamarin Studio uses the .xcarchive in the same directory as Xcode uses it).

I've reverse engineered most of their xcarchive logic (as you can probably tell from using Xcode's organizer to publish apps so far), but it's not 100% compatible (e.g. I never figured out exactly how they create .ipa's and pull in iTunesArtwork and iTunesMetadata.plist files when making ad-hoc IPA's since Xcode's archives do not include those files).
Comment 17 Jeffrey Stedfast 2015-04-21 16:53:38 UTC
Note to QA:

You guys can test the publishing workflow fix with monodevelop-lion-master commit 1ef1197257d9b20e4834516a41e739ec4d67fbda

Once that's approved, we can merge the fix to Xamarin Studio 5.9.
Comment 18 Jesper 2015-04-22 17:41:51 UTC
Sorry to Johannes for using this same thread (but Xamarin told me to do so).

I am not really sure what to do.

First. The Workaround provided by Johannes did not work for me (perhaps because I do not use the unified API ?)

I do not use a build server because there is only one developer (me) on our team. So I build from Visual Studio.

I was wondering if I can just open up the app package generated and just delete the iTunesMetadata.plist manually as a workaround while waiting for the Xamarin update ?
Comment 19 Jeffrey Stedfast 2015-04-22 17:54:30 UTC
Hi Jesper,

Yes, that should work as long as the iTunesMetadata.plist file that you remove is at the top-level of the .ipa (if it's within the .app directory, it will break the code signature and will require a resign).

Hopefully the ipa is created correctly in 5.8 and puts the iTunesMetadata.plist file at the root of the ipa file, so it should be a very trivial manual process of just unzipping, deleting that file and then re-zipping.
Comment 20 Jeffrey Stedfast 2015-04-27 14:09:24 UTC
This will be fixed in Xamarin Studio 5.9.1
Comment 21 Jesper 2015-04-27 17:40:24 UTC
Hi Jeffrey,

When I look into the .ipa file there is a payload folder under that a AppName.app folder and under that there is the iTunesMetadata.plist

So of course it is not in the top-level... I have build it with Visual Studio. Not Xamarin Studio.

When I delete the iTunesMetadata.plist I cannot test it via Testfairy so I assume this means I need to resign ?

Or did you mean something else ?


Regards Jesper
Comment 22 Jeffrey Stedfast 2015-04-27 18:33:34 UTC
Correct, if you remove anything from the .app, you will need to resign.
Comment 23 Jesper 2015-04-28 16:12:56 UTC
Ok. I have no idea how to do that. But I guess google will help :) But then I am not sure what you meant by your previous comment (the one starting with "Yes, that should work ..." )
Comment 24 Jeffrey Stedfast 2015-04-30 12:22:33 UTC
You can find the command to sign an app bundle in your build log, just look for the "Codesign" task output in the log and copy & paste the codesign command-line that it prints out.
Comment 25 Jesper 2015-05-01 17:18:29 UTC
There is no such thing in my build log (perhaps because it is Visual Studio or else you have enabled a more explicit log ?) Anyway I found it by googling, so I have managed to generate a new IPA. Only thing is I do not have a matching app.dsym package.
Comment 26 Loan 2015-05-12 20:53:57 UTC
I can reproduce this error in Xamarin 5.9.1.3 and the Alpha build of 5.9.1.4.  The .ipa still contains an iTunesMetadata.plist file.  I have tried different build configurations, AdHoc, AppStore and Release.  All yield the same result.
Comment 27 Jeffrey Stedfast 2015-05-12 21:00:59 UTC
You need to use the publishing workflow, not the IPA's produced by the build.
Comment 28 Jesper 2015-05-20 17:35:18 UTC
I am still struggling with this. I have managed to remove the iTunesMetadata.plist file manually and have resigned the IPA file using this. https://github.com/RichardBronosky/ota-tools but the resulting IPA file can only be installed on iOS6 devices, not on iOS 8.3 devices. Why ? As mentioned I use Visual Studio - how do I generate the correct IPA file using Visual Studio instead of having to resort to the extremely time consuming manual process ? It has now been 5 months with unsatisfied users...
Comment 29 Loan 2015-05-20 17:38:33 UTC
@Jesper - I Manages to get around this in Visual Studio by upgrading to the alpha updates.  Visual Studio 2013 > Tools > Options > Xamarin > this fixed it for me and produced the correct IPA which was accepted.
Comment 30 Jesper 2015-05-20 17:46:21 UTC
Thank you very much @Loan :) Would be nice if someone from Xamarin would just give us a version number that solves this (they did for Xamarin Studio, but not Visual Studio), then I could have done this weeks ago. I do not normally install from the Alpha channel. I am currently installing Xamarin 3.11.446 from the stable channel. It is apparently new today does that mean I can just use that version ? and not having to install from the Alpha channel ?
Comment 31 Loan 2015-05-20 17:49:10 UTC
Not sure @Jesper I am on Alpha 3.11.576.  But agree Xamarin need improve the stability of the releases.
Comment 32 Brendan Zagaeski (Xamarin Team, assistant) 2015-05-20 18:00:05 UTC
In case any users find this bug first when searching for solutions to this problem, more details about this issue (including the latest available fixes and workarounds) can now be found in the corresponding forum thread:

http://forums.xamarin.com/discussion/40388/disallowed-paths-itunesmetadata-plist-found-at-when-submitting-to-app-store/p1
Comment 33 Jesper 2015-05-20 18:11:58 UTC
Thank you, Brendan wish I had seen that thread earlier. I will look into it tomorrow (it is past my bedtime :) )  @Loan - 3.11.446 still generates an iTunesMetadata.plist file. Yes I have always been impressed by Xamarin's quality, but it seems to have decreased dramatically in 2015.  I have had multiple issues among others with backwards compatibility.

I still use the Classic API does anyone know if this fix is only for the Unified API ? I am desperatly trying to make a last upgrade of my Classic API app before june 1st when they will no longer be allowed. Been trying to generate for months now, but each time one Xamarin problem is solved another arises.
Comment 34 Jesper 2015-05-21 17:05:22 UTC
I just installed 3.11.576 (which is now Beta) however cannot check if it fixes the bug since I now get Error	16	Failed to show the IPA file in Finder on build server	Xamarin.iOS Extension. Sigh... This is an uphill battle :) This has never happened with any of the stable releases.
Comment 35 Rolf Bjarne Kvinge [MSFT] 2015-05-25 12:41:00 UTC
*** Bug 30410 has been marked as a duplicate of this bug. ***