Bug 42456 - Custom OpenTK: lot OpenGL symbols missing while linking
Summary: Custom OpenTK: lot OpenGL symbols missing while linking
Status: RESOLVED NOT_ON_ROADMAP
Alias: None
Product: iOS
Classification: Xamarin
Component: OpenTK ()
Version: XI 9.10 (C8)
Hardware: PC Windows
: --- enhancement
Target Milestone: Future Cycle (TBD)
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2016-07-11 05:33 UTC by Virgile Bello
Modified: 2017-01-05 15:57 UTC (History)
5 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NOT_ON_ROADMAP

Description Virgile Bello 2016-07-11 05:33:15 UTC
Due to the lack of OpenTK upgrade, we went ahead and built our own iOS version on top of official OpenTK.
The Android and iOS work is merged there: https://github.com/opentk/opentk/tree/develop

This helped a lot in our most recent version of Xenko to unify our OpenGL stacks with all platforms (PC, Linux, Mac, iOS and Android).

Everything was working well with Xamarin iOS stable (9.8) but it started to complain about a lot of missing symbols in iOS alpha (9.9), mostly OpenGL extensions (that Apple probably don't support).

My question are:
- why was it working in 9.8 (if those are not supported by Apple), and how come it broke in 9.9 while 9.8 was OK?
- any way to have "weak" symbols linking for PInvoke symbols when doing AOT? -- at least the non-executing code won't bother, since we use OpenGL extensions to check if they actually exists; the problem is AOT related
- would you be open to merging your effort with official OpenTK? (I think you already filter out the calls that don't work on OpenTK) to have an up to date API, as was promised in http://forums.xamarin.com/discussion/comment/25681/

For now, we manually do a Mono Linker phase (which is not so easy to do in the context of our own custom engine/framework), but this is definitely not a long term plan.

Thanks!
Comment 1 Rolf Bjarne Kvinge [MSFT] 2016-07-13 22:01:20 UTC
(In reply to Virgile Bello from comment #0)
> Due to the lack of OpenTK upgrade, we went ahead and built our own iOS
> version on top of official OpenTK.
> The Android and iOS work is merged there:
> https://github.com/opentk/opentk/tree/develop
> 
> This helped a lot in our most recent version of Xenko to unify our OpenGL
> stacks with all platforms (PC, Linux, Mac, iOS and Android).
> 
> Everything was working well with Xamarin iOS stable (9.8) but it started to
> complain about a lot of missing symbols in iOS alpha (9.9), mostly OpenGL
> extensions (that Apple probably don't support).
> 
> My question are:
> - why was it working in 9.8 (if those are not supported by Apple), and how
> come it broke in 9.9 while 9.8 was OK?

We changed how we treat P/Invokes: previously we used dlsym by
default, which hurts performance and executable size for most projects
(since most projects don't have P/Invokes to functions that
don't exist).

Also note that using dlsym this way won't work for static libraries on
platforms that use bitcode (tvOS/watchOS), since the native linker
will remove unreferenced symbols from those static libraries (although
I don't think this is an issue for you, since you're referencing a platform
library).

> - any way to have "weak" symbols linking for PInvoke symbols when doing AOT?
> -- at least the non-executing code won't bother, since we use OpenGL
> extensions to check if they actually exists; the problem is AOT related

You can add "--dlsym:true" to the additional mtouch arguments in the project's iOS Build options.
Comment 2 Rolf Bjarne Kvinge [MSFT] 2016-07-13 22:02:00 UTC
CC'ing Radek: do you know this?

> - would you be open to merging your effort with official OpenTK? (I think
> you already filter out the calls that don't work on OpenTK) to have an up to
> date API, as was promised in
> http://forums.xamarin.com/discussion/comment/25681/
Comment 3 Virgile Bello 2016-07-14 00:33:12 UTC
Thanks for the explanation, it makes sense and seems like a good way forward (before I was quite puzzled since nothing was mentionned in https://developer.xamarin.com/releases/ios/xamarin.ios_9/xamarin.ios_9.9/ ).

Considering what you said, it would probably be much better to filter out the non-existing symbols from the OpenTK iOS project directly.

I suppose that's what you already do on your custom OpenTK, so ideally if that could be done on OpenTK upstream too, that would be the best.

If you are still open to unify with upstream OpenTK, I suppose it might be worth discussing and organize how to proceed.

Thanks,
Virgile
Comment 4 Virgile Bello 2016-07-21 05:13:33 UTC
Checking, is there any way to have static "weak" linking too?

It sounds like "--dlsym:true" would go back to previous behavior which wouldn't work on tvOS/watchOS.

However, weak static linking would simply set a null value for non-existing symbols (which wouldn't be called in practice because we know what platform we are currently on and/or which GL extensions are availables).
Comment 5 Virgile Bello 2016-07-21 06:02:57 UTC
Additional note related to previous comment, allowing static "weak" linking would allow assemblies to be compatible with other .NET implementation (having compilation errors for missing symbols only on iOS means we have to compile specific assembly just for iOS).
Comment 6 Rolf Bjarne Kvinge [MSFT] 2016-07-21 16:57:06 UTC
There's no way to have static weak linking, it's either one or the other.

The problem with tvOS/watchOS is that without the static linking, any symbols in libraries statically linked into the app would end up getting removed by the native linker (because the native linker doesn't see that those symbols are in fact used at runtime).

However in your case I believe this is not a problem, because you're referencing a dynamic library provided by the platform, so the symbols will always be available.

Note that it's possible to enable dlsym per assembly: --dlsym:+OpenTK,+AssemblyX

We realize that changing the behavior with regards to P/Invokes to inexistent native functions is making things more complicated to library vendors, but consider this from an app developer or app user perspective: having unused code in an app is wasted space, and due a cascading effect when removing (or not removing) unused code [1] by the managed linker, a single type you use in your library (for instance in a P/Invoke declaration) can add megabytes to the final app size (and for instance prevent an app developer from having an app that can be downloaded over the cellular network, because it surpasses Apple's app size limit for cellular downloads).

[1] Unused code is not removed from user assemblies by default, only from SDK assemblies.
Comment 7 Radek Doulik 2016-07-28 19:57:07 UTC
(In reply to Rolf Bjarne Kvinge from comment #2)
> CC'ing Radek: do you know this?
> 
> > - would you be open to merging your effort with official OpenTK? (I think
> > you already filter out the calls that don't work on OpenTK) to have an up to
> > date API, as was promised in
> > http://forums.xamarin.com/discussion/comment/25681/

Back in 2013 we incorporated changes from upstream OpenTK, but later the upstream introduced backward incompatible changes and the code diverged too much.

Do you need some specific fix/feature from upstream? Maybe we can backport that?
Comment 8 Sebastien Pouliot 2017-01-05 15:57:54 UTC
Like Radek mentioned, we need to maintain API compatibility for our existing code base. As such a direct replacement/merge is not possible.

However feel free to open specific bug reports if you want some additional API (or fixes) from upstream.