Bug 41963 - [Cycle 7] The fix for Bug 35667 removes the "version" string from `.ipa` package as requested, but also moves the package _into_ a date-stamped directory, possibly breaking convention compared to other C# project types
Summary: [Cycle 7] The fix for Bug 35667 removes the "version" string from `.ipa` pack...
Status: CLOSED FIXED
Alias: None
Product: iOS
Classification: Xamarin
Component: MSBuild ()
Version: XI 9.8 (tvOS / C7)
Hardware: PC Mac OS
: --- normal
Target Milestone: C7SR1
Assignee: Jeffrey Stedfast
URL:
: 41871 42821 ()
Depends on: 42387
Blocks:
  Show dependency tree
 
Reported: 2016-06-17 22:29 UTC by Brendan Zagaeski (Xamarin Team, assistant)
Modified: 2017-01-07 01:22 UTC (History)
15 users (show)

Tags: BZRC7S1_C6SR4S1
Is this bug a regression?: Yes
Last known good build: Cycle 6 – Service Release 4

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
CLOSED FIXED

Description Brendan Zagaeski (Xamarin Team, assistant) 2016-06-17 22:29:35 UTC
[Cycle 7] The fix for Bug 35667 removes the "version" string from `.ipa` package as requested, but also moves the package _into_ a date-stamped directory, possibly breaking convention compared to other C# project types




## Regression status: explicit change in Cycle 7

maccore/master 72ad4abd0f7f78f90276551ccd23c110af193b0f

(Which corresponds to the fix for non-public Bug 35667: "Created IPA uses Build number in stead of Version number for created file")




## Steps to replicate

1. Use any existing iOS app project or create a new "Single View" iPhone app.

2. Enable "Project Options > iOS IPA Options > Build iTunes Package Archive (IPA)" for the "Release|iPhone" configuration, or set `<BuildIpa>true</BuildIpa>` by hand for that configuration in the `.csproj` file.

3. Build the project in the "Release|iPhone" configuration.




## "Breaking change" Results

The `.ipa` file is now located in a sub-folder that is named dynamically according to the date and time of the build:

> $ find bin/iPhone/Release -name "*.ipa"
> bin/iPhone/Release/UnifiedSingleViewIphone1 2016-06-17 16-42-51/UnifiedSingleViewIphone1.ipa




## OLD Results (Cycle 6)

The `.ipa` file was located in the top directory of the "Release" folder.

> $ find bin/iPhone/Release -name "*.ipa"
> bin/iPhone/Release/UnifiedSingleViewIphone1-1.0.ipa




## The complication for user CI builds

This "breaking change" (for users who were treating the output path of the `.ipa` file as an "API stable" feature) is not currently called out as a "breaking change" in the release notes [1].  That would be one good place to guide users on how to adjust their CI build scripts to account for the change.

[1] https://developer.xamarin.com/releases/ios/xamarin.ios_9/xamarin.ios_9.8/


This has led to quite a few user questions about what to do:

2016/06/16 - https://forums.xamarin.com/discussion/comment/203647/#Comment_203647
2016/06/16 - https://forums.xamarin.com/discussion/comment/203606/#Comment_203606
2016/06/14 - https://forums.xamarin.com/discussion/comment/203217/#Comment_203217
2016/06/10 - https://forums.xamarin.com/discussion/comment/202478/#Comment_202478
2016/06/10 - https://forums.xamarin.com/discussion/comment/202377/#Comment_202377
2016/06/10 - Desk #353437
2016/06/09 - http://stackoverflow.com/q/37726925
2016/06/08 - Desk #352958
2016/05/18 - https://forums.xamarin.com/discussion/67044/ipa-output-location/p1




## Is there any precedent among other C# project types for placing build output items within folders that change names on each build?

Before discussion of other possible enhancements, perhaps it is worth asking whether there is any precedent among other C# project types for placing completed build artifacts in folder names that change on each successive build.

Many of the common C# projects like console apps, Windows Phone apps, Xamarin.Android apps, and Xamarin.iOS apps (until now) all place the final build artifacts in a fixed directory.  For all of those projects, each successive build will leave each final packaged item in precisely the same location as each previous build.

If there is _not_ a precedent for breaking that convention, then it might be worth considering changing the `.ipa` package output path again to take back out the date-stamped sub-folder folder.  If there _is_ a precedent that matches this scenario, then I can help document the breaking change on the release notes, and the "other possible improvements" below will be more important.




## Other possible improvements

These improvements could apply even if we end up removing the date-stamped directory from the `.ipa` package output path, but they will be a higher priority if we do _not_ remove that part of the path.


a. Change the logic that sets the `<IpaPackagePath>` [2] to make it easier for users to customize the `$(_IpaOutputDir)` portion of that property.

[2] https://github.com/xamarin/xamarin-macios/blob/ef0b8867b9cfb14af9bc9f83e1ee74d3e340d89c/msbuild/Xamarin.iOS.Tasks.Core/Xamarin.iOS.Common.targets#L1622


b. In coordination with (a), also provide a new corresponding text field on the "iOS IPA Options" property page in the IDE, similar to the existing "Package name" field (that corresponds to the `IpaPackageName` property).  Since this new field would correspond (at least roughly) to the `_IpaOutputDir` property, this change might mean that `_IpaOutputDir` should be renamed to remove the underscore prefix and treated as API-stable moving forward. 



Many thanks!
Comment 2 Brendan Zagaeski (Xamarin Team, assistant) 2016-06-17 22:58:19 UTC
## Possible temporary workarounds for CI builds

A. The simplest workaround I've come across so far is probably to use a relative path under "Project Options > iOS IPA Options > Package Name".  For example, if you set that field to:

../UnifiedSingleViewIphone1.ipa

Then the IPA file will be placed directly in `bin/iPhone/Release/` rather than in the date-stamped sub-directory.  This is admittedly a little bit of a fragile approach because it depends on the precise way the logic that sets the `<IpaPackagePath>` currently works.  (Also, as a caution for any users who are accustomed to using the "Show IPA File on Build Server" button in XamarinVS, this approach does break that button.)  But as a _temporary_ workaround, it is quick and fairly easy.




B. Depending on the CI system, there would be various options for how to use the final generated path (as provided by the `IpaPackagePath` property any time _after_ the `_CoreCreateIpa` target has run) to take the `.ipa` package from its new sub-folder location.  This would be the most elegant approach at the moment, but it would require updating the local CI build scripts themselves, which I believe in some environments can be a somewhat fragile process.  And deciding on the best way to share the `IpaPackagePath` with the CI system could be a stumbling block.




C. There are of course also various possible ways to customize the MSBuild properties and targets (from [2]) locally (including manually reverting the change), but those approaches are at best a bit "inelegant" at the moment because the provided logic is only really set up to allow easy customization of the `IpaPackageName`, and not the `_IpaOutputPath`.

[2] https://github.com/xamarin/xamarin-macios/blob/ef0b8867b9cfb14af9bc9f83e1ee74d3e340d89c/msbuild/Xamarin.iOS.Tasks.Core/Xamarin.iOS.Common.targets#L1622
Comment 3 Sebastien Pouliot 2016-06-20 13:34:10 UTC
@Jeff can you clarify the change, thanks.
Comment 4 Jeffrey Stedfast 2016-06-20 13:49:40 UTC
The change was made because no one could agree what version string to use in the generated IPA package.

We decided that since Xcode does not add a version string to their IPA files, we shouldn't either and so we followed the Xcode pattern.

FWIW, the $(IpaPackagePath) MSBuild variable can be used to locate the .ipa file after the build if customers want to write custom build steps.
Comment 6 Brendan Zagaeski (Xamarin Team, assistant) 2016-06-21 20:25:15 UTC
I have now added this change in behavior to the release notes for Xamarin.iOS 9.8.




## Continuing the discussion

As a small follow-up question for Comment 4, just to keep the conversation alive, would it break a documented supported worfklow if the `.ipa` file were placed directly in `bin/iPhone/Release/` without a version number?  At my first (quite possibly naive) glance, it seems like it could be OK to overwrite that file on each build, regardless of any of the settings in the `Info.plist`:

- For CI builds, I would expect users to have their build scripts copy the `.ipa` package for each CI build to a location outside of the `bin` folder, so there wouldn't be a need to preserve previous versions of the `.ipa` file in the `bin` folder itself.

- For non-CI builds (or for CI builds that use `.xcarchive` bundles rather then `.ipa` files), the correlation to Xcode's style of timestamped naming would still be preserved because the `.xcarchive` bundles would still have timestamped names.


From some quick searching about Xcode, it seems that the most likely way a user would create a `.ipa` package in Xcode without creating an intermediate `.xcarchive` would be from the command line [1].  And in that case, it looks like the default output path for the `.ipa` file would be the same directory as the `.app` bundle.  It looks like the output path for the Xcode command line build can also be customized by adding the `-o` command line switch.  That could be one data point in favor of implementing "possible improvement (a)" from Comment 0 (regardless of what we settle on for the _default_ output path of the `.ipa` package).

[1] http://stackoverflow.com/questions/2664885/xcode-build-and-archive-from-command-line
Comment 7 Alek Slater 2016-06-22 13:17:12 UTC
I personally think this doesn't make much sense, because the top level folder (Release/ etc) contains the dsym, the .app etc, so why should the ipa go in a separate subfolder.
Comment 8 Jeffrey Stedfast 2016-06-22 13:55:22 UTC
The reason people wanted the IPA to have the build # in the file name is so that they could do build after build and not have the IPA's be removed/replaced.

This is why we had to put it in another directory if we did not use the build #.
Comment 9 Alek Slater 2016-06-22 16:47:38 UTC
Maybe offer a few different modes for this ? Because this is going to affect a lot of build scripts etc. I genuinely feel like this is a step in the wrong direction.

Personal feelings aside, is this change going to stay ? Should I update all the build scripts, or wait for a fix to undo it / change it ?
Comment 10 Jeffrey Stedfast 2016-06-22 16:51:30 UTC
I would recommend updating your build scripts to make use of the $(IpaPackagePath) variable for locating your IPA package.

Whether we change this or not, I will make sure that variable continues to be accurate.
Comment 11 Alek Slater 2016-06-22 16:52:31 UTC
Sure but $(IpaPackagePath) includes me also adding some extra stuff to the csproj files? Doesn't this mean I have to keep doing this for all new projects ?
Comment 12 Brendan Zagaeski (Xamarin Team, assistant) 2016-06-22 17:27:10 UTC
> Should I update all the build scripts, or wait for a fix to undo it / change it ?

While this bug report is still under active discussion, it is tricky to make a final recommendation.  One temporary option (mentioned in passing in Comment 2) that would let you hold off on modifying your individual project files for now would be to "manually revert the change" in the `Xamarin.iOS.Common.targets` file itself.  The specifics of that option have been filled out by some users on the forums:

https://forums.xamarin.com/discussion/comment/204894/#Comment_204894




## Further discussion of workarounds on the forums

If possible, I'd like to avoid expanding the discussion on this bug report too much further about recommended workarounds so that we can keep the discussion tightly focused on the potential outcomes for this bug.

I'll recommend directing further discussion of possible workarounds to:

https://forums.xamarin.com/discussion/67044/ipa-output-location
https://stackoverflow.com/q/37970028/2561894

Thanks!




## Roughly 3 possible outcomes for this bug

I would recommend that the Xamarin team consider at least 3 possible options for how to move forward with this bug:


1. Precisely revert the change introduced by the fix for Bug 35667 for now.  Additional consideration can then be given to re-implementing the change in some way in a future release, taking extra care to make the reimplemented change as easy-to-use across a variety of CI systems as possible and being sure to "loudly" document the change in the release notes.


2. Change the behavior to use a "static" name for the `.ipa` output file that includes neither a version string nor a timestamp.  This option would need to be researched to some degree to get some sense of (a) how many users are still depending on a worfklow where previous `.ipa` files are retained directly in the `bin/` folder, and (b) how many users' CI build scripts currently depend on the inclusion of the version string in the final `.ipa` package name (otherwise this change would cause the same kinds of breakages as the current behavior, though probably noticeably easier for users to fix on their own).


3. Keep the behavior unchanged (and hopefully add "possible improvements" (a) and (b) from Comment 0).  In this case it will be important to carefully document (in both the release notes and in the documentation guides) the reasons this option is superior to either option (1) or option (2).
Comment 16 Brendan Zagaeski (Xamarin Team, assistant) 2016-06-22 19:02:48 UTC
## Correction for the "manually revert the change" forum link in Comment 12:

A better link for "manually revert the change" is:

https://forums.xamarin.com/discussion/comment/203296/#Comment_203296

(Since that points to the particular comment in the thread that describes the changes to the `Xamarin.iOS.Common.targets` file)
Comment 17 david 2016-06-23 00:53:23 UTC
Having raised this bug in the forums I'd like to add $0.02 here as well.

From my perspective it doesn't matter over-much what the eventual solution ends up being. What matters much more is that

a) it doesn't change unless the change is really needed
b) the reason for the change and the change itself are both heavily documented and signalled as breaking changes for anyone doing automated builds

I made a fuss at the time because the change slipped through the net and wasn't documented as part of the upgrade notes. I puzzled over my broken build for maybe 2 hours before I remembered reading an obscure bug report that talked about putting .ipa files into a folder rather than having them loose in the /bin folder. Once I realised what was going on, fixing my build script was not very hard. Now it's working again I'd really rather not have to "fix" it any further. :)

I think it's quite common for automated builds to copy all relevant files away from the /bin folder to a known "safe" location so they can be preserved. 

Our solution file contains (at present) 21 .csproj files that can produce a .ipa file on building. I explicitly do not use the .csproj file to set the location/name of my .ipa file since I refuse to try and keep all those configurations in sync. That's a fool's errand.

In addition, since we find that Fody throws exceptions during the build process if the previous build's artefacts are not removed first, we always do a full "clean" before every "build". This means that we _have_ to copy the .ipa file for one .csproj file away from its build location to somewhere safe before we trigger the build for the next one. Anyone who uses Fody will have this problem.

Therefore the actual location of the .ipa file is almost irrelevant. What matters more is that it can be found easily by the automated tooling. I'm not a big fan of the timestamped folder approach since it changes with every build. Because it's the only thing about the build that changes, we have to have a script that's adaptable to that change. However, now that I have such a script, it works well enough that I'm content to let sleeping dogs lie.

If someone wants to revisit the whole process later, the build process could usefully be enhanced to parameterise:

* Bundle Id / Package Name + Matching Watch App / Extension Bundle Ids
* App Name
* Version / Build number
* Insights / HockeyApp tracking id
* Resources folder / Assets folder
* Other custom "environment variables" like $PACKAGE_NAME$ that the build process can substitute in code (e.g. API keys for use with GCM, or other API keys that shouldn't be checked in to code repos)
* Provisioning profile name / KeyStore file
* Type of output: signed or unsigned (or both)
* Include / exclude symbolication output file
* Location of output file(s)

These all need to be options in the xbuild and mdtool apps.
Comment 18 Ruben Macias 2016-06-24 21:31:42 UTC
Might I also add that the Jenkins CI Documentation should also be updated to reflect this change.  

https://developer.xamarin.com/guides/cross-platform/ci/jenkins_walkthrough/#Building_a_Xamarin.iOS_Project

My Jenkins build relied on Environment Variables for the Path of the IPA file, now this is broken.  I see that there is an MSBuild variable, but how do I access this from a bash script?  Is there a way to write an MSBuild Variable to an Environment Variable so I can access it from a bash script?
Comment 19 Brendan Zagaeski (Xamarin Team, assistant) 2016-06-25 00:31:57 UTC
I'll make sure that I review the release notes and request appropriate changes to various documentation for completeness once the Xamarin team has settled on what's happening with this bug.  For example, even if the Xamarin team keeps the current mechanism that writes `.ipa` packages into timestamped folders, if the team also implements the possible improvements from Comment 0 those might be the more straightforward approach for integrating across a range of CI tools that might be invoked in various ways after the Xamarin MSBuild targets run.  So the Xamarin team would likely want to hold off on updating the documentation until those improvements are in place.


To reiterate Comment 12:

For now, if possible, I'd like to avoid expanding the discussion on this bug report too much further about recommended workarounds (including how to use IpaPackagePath) so that we can keep the discussion tightly focused on the potential outcomes for this bug.

I'll recommend directing further discussion of possible workarounds to:

https://forums.xamarin.com/discussion/67044/ipa-output-location
https://stackoverflow.com/q/37970028/2561894

Thanks!
Comment 20 Brendan Zagaeski (Xamarin Team, assistant) 2016-06-25 00:56:54 UTC
(I also know it's very tempting to ask all kinds of questions on this bug report since it's one specific conversation about the change.  If Bugzilla had a bit richer conversation threading model and the community visibility of Stack Overflow, that could be quite nice, but with Bugzilla the way it is, keeping the conversations tightly focused is quite helpful.)

> Is there a way to write an MSBuild Variable to an Environment Variable

My first try at a Google for "write environment variable msbuild" turned up some promising links:

- https://social.msdn.microsoft.com/Forums/vstudio/en-US/0fb8d97d-513e-4d37-8528-52973f65a034/how-to-set-envrionment-variables-in-msbuild-file?forum=msbuild
- https://www.nuget.org/packages/MSBuild.SetEnvVariable/
- http://blog.jdhardy.ca/2011/12/setting-environment-variables-for.html
Comment 21 Jeffrey Stedfast 2016-06-27 16:27:23 UTC
What if, instead of reverting this patch, we simply make the code use the $(IpaPackagePath) variable if set (instead of calculating it)?

And only calculate it if it isn't set...

If I can find an easy way of importing this MSBuild variable via an environment variable, I can look into that as well.
Comment 22 Brendan Zagaeski (Xamarin Team, assistant) 2016-06-27 16:44:05 UTC
I'd be mostly on board with that.  That would correspond roughly to part (a) of the "possible improvements" from Comment 0.  The one little complication hinted at in the earlier discussion of that approach is that $(IpaPackagePath) "overrides" $(IpaPackageName).  That is, if you explicitly set the $(IpaPackagePath), then the value you set for $(IpaPackageName) will be ignored.  An alternative would be add customization of the $(_IpaOutputDir) instead.  Then you'd have 2 separate adjustments you could make:

1. The old familiar $(IpaPackageName).

2. The new $(_IpaOutputDir) that would allow you to change just the output directory.

(But as also hinted at earlier, surfacing $(_IpaOutputDir) would mean it would probably be best to drop the underscore _ prefix and ensure API stability of that _additional_ MSBuild property moving forward.)



It wouldn't be the worst thing to use $(IpaPackagePath), but thinking another step ahead (part (b) of the "possible improvements" from Comment 0), it would be nice to have a setting in the Project Properties for adjusting the IPA output path, and if the implementation of that UI simply adds a text field for $(IpaPackagePath), then that text field would "conflict" with the existing text field for $(IpaPackageName) (because it would "override" it).



If we do decide to go with this option (option 3 from Comment 12) of adding the possible improvements, then the Xamarin will need to make sure to update the documentation consistently across all of the documentation with the new best practice for accessing the `.ipa` package from build scripts.  In that case, I will aim to coordinate with the documentation team to ensure that the documentation is updated on precisely the same timeline as the release of the "possible improvements."
Comment 23 Jeffrey Stedfast 2016-06-27 18:03:34 UTC
$(IpaPackagePath) is now only computed if it hasn't already been set elsewhere.

Also introduced $(IpaPackageDir) so you can set that instead of the full path and have msbuild auto-compute $(IpaPackagePath) based on the $(IpaPackageDir) and $(IpaPackageName) vars.
Comment 24 Sebastien Pouliot 2016-06-27 19:42:17 UTC
*** Bug 41871 has been marked as a duplicate of this bug. ***
Comment 25 david 2016-06-27 22:58:10 UTC
So how do we set $(IpaPackagePath) from a build script?
Comment 26 Brendan Zagaeski (Xamarin Team, assistant) 2016-06-27 23:13:56 UTC
Comment 23 was referring to a patch that has only been _committed_.  That change has not yet been released, so the property $(IpaPackagePath) doesn't yet exist in any published build.  As mentioned in Comment 22:

> I will ... coordinate with the documentation team to ensure that the
> documentation is updated on precisely the same timeline as the release
> of the "possible improvements."

So I'll push to make sure the primary documentation updates (at least in the release notes) are ready for the initial Beta version of Cycle 7 – Service Release 1 (tentatively scheduled for later this week), and then push on the CI guide docs updates for Cycle 7 – Service Release 1 Stable release (or earlier).

I'll recommend to keep an eye out for those Cycle 7 – Service Release 1 release notes, and then you can follow-up in the corresponding announcement on the forums if there's anything unclear in the release notes.


In short, until we have that Beta version published, I'll request that we continue avoid getting wrapped up in discussing "how to" questions about on this bug report.  Thanks!
Comment 27 Mohit Kheterpal 2016-06-29 17:50:58 UTC
Blocked to verify this issue due to issue https://bugzilla.xamarin.com/show_bug.cgi?id=42225
Comment 28 Akhilesh kumar 2016-07-01 17:59:58 UTC
I have checked this issue with C7 build xamarin.ios-9.8.2.10_843a527d90a5ade61fed45cc05db3a1b2e2c2879, I have used command line instead of IDE to verify this issue, since CI users will most probably be using command line option.

On running below command, the ipa is get generate at given the IpaPackageDir.

Xamarin77Macmini:~ xamarin77$ xbuild /p:Configuration=Release /p:Platform=iPhone /p:ArchiveOnBuild=true /p:IpaPackageDir="/Users/xamarin77/Projects/Single3e3/" /t:"Build" Projects/Single3e3/Single3e3/Single3e3.csproj

However, by default it is generated in date/time stamp at location, in my case at  location "/Users/xamarin77/Projects/Single3e3/Single3e3/bin/iPhone/Release/Single3e3 2016-07-01 12-19-40/Single3e3.ipa" 

So we can override the IpaPackageDir path.

So I am closing this issue.
Comment 29 Alek Slater 2016-07-02 09:40:29 UTC
What about people who use mdtool ? How do you achieve this with that?
Comment 30 Arpit Jha 2016-07-04 09:00:40 UTC
I have checked this issue C7SR1 build XI 9.8.2.10 (Xamarin Enterprise) Hash: 843a527 and its working fine as mentioned in comment28.

iPA generated on given iPAPackageDir path using command line

Environment Info: https://gist.github.com/Arpit360/9a7238e892957599875f9f54a3f9e352
Comment 31 Brendan Zagaeski (Xamarin Team, assistant) 2016-07-07 01:20:07 UTC
## Status updates

An initial version of the candidate change discussed in Comment 23 is now available in the Beta channel.  A preliminary version of the corresponding release notes has also now been published:

https://developer.xamarin.com/releases/ios/xamarin.ios_9/xamarin.ios_9.8/#Xamarin.iOS_9.8.2_Service_Release_1

(Note that as mentioned in these release notes, for this early Beta version the change is currently only available on Mac.)


The forum announcement for the new Beta channels builds has been posted as well:

http://forums.xamarin.com/discussion/70675/beta-release-cycle-7-service-release-1/p1
Comment 32 GouriKumari 2016-07-27 11:33:07 UTC
*** Bug 42821 has been marked as a duplicate of this bug. ***
Comment 33 James Clancey 2017-01-04 07:36:44 UTC
There is no way to disable this inside the IDE. We need a way to turn off the nesting inside the IPA options.
Comment 34 Brendan Zagaeski (Xamarin Team, assistant) 2017-01-04 18:49:29 UTC
This bug has been marked as verified fixed for several months and across several releases.  It can be considered closed at this time.  New appearances of similar issues, or requests for additional capabilities will need to be filed in their own new bug reports.  Be sure to include as many details [1] as possible.  Thanks in advance!

[1] https://developer.xamarin.com/guides/cross-platform/troubleshooting/questions/howto-file-bug/