Bug 60393 - Implementing ARSessionDelegate.DidUpdateFrame freezes ARSCNView rendering
Summary: Implementing ARSessionDelegate.DidUpdateFrame freezes ARSCNView rendering
Status: RESOLVED FIXED
Alias: None
Product: iOS
Classification: Xamarin
Component: Documentation ()
Version: XI 11.2 (d15-4)
Hardware: PC Windows
: --- normal
Target Milestone: Future Cycle (TBD)
Assignee: Larry O'Brien
URL:
Depends on:
Blocks:
 
Reported: 2017-10-26 14:09 UTC by Rich Zwaap
Modified: 2017-11-06 21:02 UTC (History)
4 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Visual Studio solution containing repro case and Xcode project demonstrating that the same works in native iOS (37.42 KB, application/x-zip-compressed)
2017-10-26 14:09 UTC, Rich Zwaap
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Rich Zwaap 2017-10-26 14:09:17 UTC
Created attachment 25469 [details]
Visual Studio solution containing repro case and Xcode project demonstrating that the same works in native iOS

Implementing the ARSessionDelegate.DidUpdateFrame protocol method causes rendering of the associated ARSCNView to freeze.  The following code shows a trivial ViewController with an ARSCNView named "_sceneView" that demonstrates the issue:

    public partial class ViewController : UIViewController, IARSessionDelegate
    {
        public override void ViewDidAppear(bool animated)
        {
            base.ViewDidAppear(animated);

            var configuration = new ARWorldTrackingConfiguration();
            _sceneView.Session.Run(configuration, ARSessionRunOptions.ResetTracking);

            _sceneView.Session.Delegate = this;
        }

        // Having this implemented freezes ARSCNView rendering
        [Export("session:didUpdateFrame:")]
        public void DidUpdateFrame(ARSession session, ARFrame frame)
        {
        }
    }

Note that implementing the protocol method by sub-classing ARSessionDelegate and overriding DidUpdateFrame results in the same behavior.

This protocol method can be implemented in native iOS without any impact on rendering.  However, the Xamarin documentation states that "developers who override this method must take complete responsibility for rendering the frame" (see here: https://developer.xamarin.com/api/member/ARKit.ARSessionDelegate.DidUpdateFrame/p/ARKit.ARSession/ARKit.ARFrame/#Remarks).  Additionally, key-value observation does not appear to be applicable for the ARSession's CurrentFrame (and sub-properties), so there does not appear to be any alternative for subscribing to frame updates.

I've attached a Visual Studio solution that contains a trivial reproducer, as well as an Xcode project that demonstrates the same functionality in Swift.  Both applications emit debug messages pertaining to the frame camera's transform in DidUpdateFrame, but the code contained in that method has no bearing on the issue.  The Visual Studio solution demonstrates the protocol method implemented using both the ExportAttribute and sub-classing of the ARSessionDelegate.
Comment 1 Sebastien Pouliot 2017-10-26 16:25:42 UTC
It's not really frozen, but it does not hit the condition for being updated very often.

Short story: iOS does not update the current frame as long as it is referenced. It's an optimization (from Apple) to avoid processing unneeded frames, e.g. if you're still working on frame X then processing X+1 is not needed.

However in a GC world the native ARFrame is kept referenced by the managed ARFrame until:

a. the GC frees it (default, but it might take a while); or 
b. until it's disposed.

So iOS _thinks_ an update is not required. Simply adding

> frame.Dispose ();

at the end of your `DidUpdateFrame ` method will fix the issue, i.e. the native reference will be dropped immediately, not waiting for the GC, and iOS will process a new frame.


@Larry, we discussed this for the sample but it seems this was not added to the documentation.
Comment 2 Rich Zwaap 2017-10-26 16:47:22 UTC
@Sebastien thanks so much for the rapid response.  I can confirm that this works like a champ.
Comment 3 Sebastien Pouliot 2017-10-26 17:30:20 UTC
actually add @Larry for doc update
Comment 4 Larry O'Brien 2017-11-06 21:02:26 UTC
ios-api-docs updated in 5512a53c