Bug 16687 - `make get-monolite-latest` fails because http://storage.bos.xamarin.com doesn't work
Summary: `make get-monolite-latest` fails because http://storage.bos.xamarin.com doesn...
Status: RESOLVED FIXED
Alias: None
Product: Tools
Classification: Mono
Component: other ()
Version: unspecified
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Duncan Mak
URL:
: 25634 ()
Depends on:
Blocks:
 
Reported: 2013-12-10 00:01 UTC by Michael Friis
Modified: 2016-08-01 16:17 UTC (History)
13 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Michael Friis 2013-12-10 00:01:03 UTC
I'm building Mono on a Linux system without a working Mono installation. I run `make get-monolite-latest` as part of the build to get a working mcs. This doesn't work because the monolite archive cannot be pulled from http://storage.bos.xamarin.com. Here's the relevant command with output:

cd /tmp/compile_W4i9p/mono-3.2.5/mcs/class/lib && { (wget -O- http://storage.bos.xamarin.com/mono-dist-master/latest/monolite-110-latest.tar.gz || curl http://storage.bos.xamarin.com/mono-dist-master/latest/monolite-110-latest.tar.gz) | gzip -d | tar xf - ; }
/bin/bash: wget: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                              Dload  Upload   Total   Spent    Left  Speed
112   337  112   337    0     0   4543      0 --:--:-- --:--:-- --:--:-- 15318

gzip: stdin: not in gzip format
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
make: *** [get-monolite-latest] Error 2

Note that the system doesn't have wget, but that's immaterial.

I tried to find other ways to get monolite so that I might fix the makefile. This page talks about Wrench: http://www.mono-project.com/Monolite

But I cannot find it there.

I can reproduce with both 3.2.3 and 3.2.0 source tarballs.
Comment 1 Duncan Mak 2013-12-10 02:25:46 UTC
The latest URL is http://storage.bos.xamarin.com/mono-dist-master/latest/monolite-111-latest.tar.gz and not 110.
Comment 2 Michael Friis 2013-12-10 11:09:34 UTC
I see, splendid.

Given that there's lots of source tarballs and what-not around that reference version, 110 should that not still be available? Or should the 110 url maybe redirect to 111 if that works?
Comment 3 Brandon White 2014-01-08 23:08:51 UTC
I've run into this as well with the latest git master.  Seems to me like they should keep older versions around.
Comment 4 Alex Lennon 2014-10-24 07:10:40 UTC
I am trying to build Mono 3.2.3 for some testing.

This fails due to the missing archive.
Comment 6 Alex Lennon 2014-10-24 15:43:40 UTC
Thanks for coming back to me Zoltan. I did manage to find a copy but was just thinking that if monolite is needed to bootstrap a build of mono (which I think it is?) then couldn't these older archives be left available?

I guess I'm not understanding the reasoning behind removing them? Perhaps there's a good reason but wouldn't be be easy just to leave them in place and then older releases of Mono could still be built?

Cheers, A/
Comment 7 Alex Lennon 2014-10-24 15:45:49 UTC
(Also I'm not really understanding why a bootstrap is needed? Apologies if this is obvious stuff but I've been working through trying to get on top of the Win32 build and I can't really see why the Mono build just build mono.exe instead of needing gmcs/basic or whatever?)
Comment 8 Michael Friis 2014-10-24 16:26:51 UTC
@Alex A minimal bootstrap compiler is needed because the Mono C# compiler is itself implemented in C#. So to build it, you need some minimal self-contained C# compiler.

Otherwise I agree that:

1) It's bad that Xamarin removed the bootstrap-compiler-download at the location used in previous Mono version build files
2) That Xamarin kept shipping Mono versions that couldn't self-build because of these messed-up build scripts
Comment 9 Alex Lennon 2014-10-24 16:31:55 UTC
@Michael

Thanks for expanding :)

I get the idea of self-hosting and have been wondering what comes before the chicken and the egg in terms of building Mono...

I mean, at some point the runtime must have had to be built, so is it just that this was lost back in the mists of time and would be incredibly painful/time-consuming to bring up to date.

As a more esoteric question, if one of the licenses for Mono is GPL then doesn't there have to be a way to re-generate the supplied binaries from source? (or is this covered under the "there is if you have a year to spend working out how to do it" clause?)
Comment 10 Zoltan Varga 2014-10-24 17:53:52 UTC
The files are still there, they just for moved around a bit so the build can't pick them up any more.
Comment 11 Alex Lennon 2014-10-25 04:32:04 UTC
Is there a reason not to move them back so the build can pick them up?
Comment 12 Alex Lennon 2014-10-25 04:34:48 UTC
Also where to I get the source to the mono binaries that are downloaded so I can build them (or are they not licensed under GPL?)
Comment 13 Alex Lennon 2014-10-25 05:13:54 UTC
I had a little play to follow this thought through...

Start with mono-3.10.0-branch which gives Mono JIT compiler 3.10.0

- get-monolite-latest pulls monolite-111-latest so corlib version 111.

- basic.exe pulled down is Mono C# compiler 3.6.1.

This doesn't seem to be available as an explicit branch in github or as a tarball (?) No doubt there's something in the repo somewhere.

So download 3.6.0 tarball as closest available release

- get-monolite-latest pulls monolite-111-latest so corlib version 111.

- basic.exe pulled down is Mono C# compiler 3.6.1.

So I am downloading  basic.exe 3.6.1 to build Mono 3.6.x

Any ideas/advice on how I build that basic.exe 3.6.1 ?

Thanks :)
Comment 14 Zoltan Varga 2014-10-25 11:02:31 UTC
The sources are in various branches like mono-3.6.0-branch. You don't need an exact version of monolite to build, only the mscorlib version needs to match.
Comment 15 Dale 2014-12-16 10:08:45 UTC
Our internal builds of Mono are now broken due to this bad URL to monolite. 

We count on this package to bootstrap our OpenEmbedded build. Both 3.6.0 and the latest 3.10.0 are broken. This is bad as there doesn't seem to be a work around that doesn't require having a full mono (which version?) installed on the build machine rather than adapting to the targeted version.

I would think the official build, as a minimum goal, should always build.

Could you please restore these builds to functionality?

Dale Larson
Johnson Controls, Inc
Comment 16 Alex Lennon 2014-12-16 10:15:10 UTC
Hi Dale,

Are you using meta-mono or something else?

Thanks, Alex
Comment 17 Dale 2014-12-16 10:23:01 UTC
Hi Alex,

I'm running a much modified version of the original recipes from the meta-mono layer I grabbed over a year ago. I've added a lot of packaging to allow installing subsets of the full mono (e.g., san debug stuff or docs.)

I also have update XSP recipes as well. I haven't checked the "official" layer for a long time.

It's all busted now though.

Dale
Comment 18 Alex Lennon 2014-12-16 10:28:28 UTC
Hi Dale,

Thanks for the heads up. I agree with what you say. It's difficult when the monolite packages go missing. I'll have to find time have a look and see if my current master still builds. Presumably it won't :-/

Cheers, Alex
Comment 19 Dale 2014-12-16 10:47:34 UTC
Hi Alex,

I found an old build on our build server that had the unpacked version of the v111 monolite binaries. I'm trying to repackage those bits and redirect the recipe's make to use them as a stop-gap.

Dale
JCI
Comment 20 Zoltan Varga 2014-12-16 17:39:49 UTC
There is now a 111 monolite at:
http://storage.bos.xamarin.com/mono-dist-3.12.0-release/latest/monolite-111-latest.tar.gz
Comment 21 Alex Lennon 2014-12-17 08:20:32 UTC
Dale,

fwiw I have successfully rebuilt 3.10.0 from current meta-mono master and I don't have anything in my logs about the get-monolite-latest step.

Instead I set EXTERNAL_MCS to use ~/mcs/class/lib/monolite/basic.exe

Double-checking I see this monolite comes with the release tarball download I am using for the build.

Would be interested if you could confirm for me.

Thanks,

Alex
Comment 22 Anthony Tarlano 2014-12-18 19:12:36 UTC
@Zoltan,

The current monolite is 113 and you have removed the 111 version. Unfortunately this breaks the ASP.NET vNext instructions for installing on OS X at https://github.com/aspnet/home#aspnet-vnext-home

The is because "brew install kvm" uses the 111 in the brew forumla for Mono. 

Can you put the 111 version of the tar ball back or update the mono brew formula?

Thanks,

Anthony
Comment 23 Dale 2014-12-26 11:55:52 UTC
Alex,

>Would be interested if you could confirm for me.

Confirmed. This is a workable approach for the tarball versions. You'll still be hosed when building from GIT but at least the monolite binaries are available from somewhere.

Dale
Comment 24 Alex Lennon 2014-12-29 13:20:36 UTC
Thanks for coming back to me Dale. Better than nothing then, but I completely agree it would be wonderful to not have this external dependency when building from source

Cheers, Alex
Comment 25 Ales Ruda 2014-12-30 04:42:39 UTC
Hi,
I try build mono 3.8 using get-monolite-latest switch. Because http://storage.bos.xamarin.com/mono-dist-master/latest/monolite-111-latest.tar.gz not found I use 
http://storage.bos.xamarin.com/mono-dist-3.12.0-release/latest/monolite-111-latest.tar.gz.

Compilation crash on 

cd /usr/src/mono-3.8/mono/mcs && make --no-print-directory -s NO_DIR_CHECK=1 PROFILES='net_2_0 net_3_5 net_4_0 net_4_5 xbuild_12   ' CC='gcc' all-profiles
make[6]: gmcs: Command not found
make[6]: *** [build/deps/basic-profile-check.exe] Error 127
*** The compiler 'gmcs' doesn't appear to be usable.
*** Trying the 'monolite' directory.
cant resolve internal call to "System.String::InternalSetLength(int)" (tested without signature also)

Your mono runtime and class libraries are out of sync.
The out of sync library is: /usr/src/mono-3.8/mono/mcs/class/lib/monolite/mscorlib.dll

What is right url for monolite?

Thanks


Ales
Comment 26 Zoltan Varga 2015-01-02 19:01:26 UTC
*** Bug 25634 has been marked as a duplicate of this bug. ***
Comment 27 Zoltan Varga 2015-01-02 19:06:01 UTC
So the situation is the following: we have a lane on wrench which builds the monolite packages:

https://wrench.mono-project.com/Wrench/ViewLane.aspx?lane_id=4&host_id=21&revision_id=82730

There are two problems here:
- this was broken for some time, so it didn't build anything for some mono versions
- get-monolite-latest always download from http://storage.bos.xamarin.com/mono-dist-master/latest
   which contains the monolite version corresponding to master, and not any released mono version.
Comment 28 Marek Safar 2015-01-03 03:30:55 UTC
Zoltan,

- get-monolite-latest gets correct monolite version based on 

mono_corlib_version = $(shell sed -n "s/\#define MONO_CORLIB_VERSION //p" $(srcdir)/mono/metadata/appdomain.c)
monolite_url = http://storage.bos.xamarin.com/mono-dist-master/latest/monolite-$(mono_corlib_version)-latest.tar.gz

however, due to wrench unreliability we don't have monolite for all previous versions we need it.
Comment 29 Zoltan Varga 2015-01-03 03:33:45 UTC
The problem is that 'latest' is a symlink to the directory containing the build artifacts for the latest build on the mono-dist-master lane, it won't contain any older monolites, just the files generated by the last build.
Comment 30 Alex Lennon 2015-01-04 09:05:11 UTC
Zoltan,

I've been thinking about this and I'm unclear on why the older monolite archives go missing. e.g. If there was a symlink for corlib 111 -

http://storage.bos.xamarin.com/mono-dist-master/latest/monolite-111-latest.tar.gz

What causes this to disappear? Surely the symlink shouldn't change and its target archive shouldn't change unless there is a successful build and upload of a newer 111 archive to that lane?

Thanks, Alex
Comment 31 delcypher 2015-01-07 12:28:52 UTC
I just ran into this issue. I agree with Alex. I don't see why you would remove the old versions of monolite if they are necessary to bootstrap a build of mono.
Comment 32 Zoltan Varga 2015-01-07 12:37:12 UTC
The old versions are still there in different directories, but 'latest' is a symlink to the latest build from the mono-dist-master lane which only contains the latest monolite.
Comment 33 Alex Lennon 2015-01-07 12:38:34 UTC
But that's my question. At some point the symlink works, and then at another point the target goes missing. Surely the symlink for a specific version of corlib should not be updated unless the target is valid?
Comment 34 delcypher 2015-01-07 12:45:42 UTC
@Zoltan could you me what the URL for monolite-111-latest.tar.gz is? I tried using

http://storage.bos.xamarin.com/mono-dist-master/latest/monolite-113-latest.tar.gz

and that doesn't work when building mono-3.10.0 because that needs mscorlib version 111. I'm currently stuck because I'm on a system where I can't install a working mono installation so I have to bootstrap but can't because I can't get hold off the right monolite version.

I think a reasonable temporary solution would be to

* Put up a list of URLs to the different monolite versions (better yet have the directory index automatically generated)
* Note in the build instructions for mono that you can force the URL with

$ make get-monolite-latest monolite_url=http://someurl/blah.tar.gz

and where a list of the monolite versions can be found.
Comment 36 delcypher 2015-01-08 12:41:24 UTC
@Zoltan - Thanks a lot, I've managed to build mono-3.10.0 now :)

What do you think of the idea of having an automatically generated list of monolite tarballs? Could you just have the web server generate the directory listing (e.g. like Apache does by default)?
Comment 37 peter 2015-01-13 11:33:06 UTC
Anyone seeing this error by using "brew install kvm" on mac, can execute "brew update"
Comment 38 Zoltan Varga 2015-01-19 17:45:46 UTC
This should be fixed now, the monolite file should be at the location expected by mono for versions 109 to current.