Bug 57516 - Sporadic Mono.Btls.MonoBtlsException: Ssl error:1000007d CERTIFICATE_VERIFY_FAILED
Summary: Sporadic Mono.Btls.MonoBtlsException: Ssl error:1000007d CERTIFICATE_VERIFY_F...
Status: NEEDINFO
Alias: None
Product: Class Libraries
Classification: Mono
Component: Mono.Security ()
Version: 5.0 (2017-02)
Hardware: PC Linux
: --- normal
Target Milestone: Future Release
Assignee: Bugzilla
URL:
Depends on: 58141
Blocks:
  Show dependency tree
 
Reported: 2017-06-15 09:09 UTC by paulo.borges
Modified: 2018-01-22 15:47 UTC (History)
7 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Source Code + Assemblies (826.75 KB, application/x-7z-compressed)
2017-06-15 09:09 UTC, paulo.borges
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report for Bug 57516 on GitHub or Developer Community if you have new information to add and do not yet see a matching new report.

If the latest results still closely match this report, you can use the original description:

  • Export the original title and description: GitHub Markdown or Developer Community HTML
  • Copy the title and description into the new report. Adjust them to be up-to-date if needed.
  • Add your new information.

In special cases on GitHub you might also want the comments: GitHub Markdown with public comments

Related Links:
Status:
NEEDINFO

Description paulo.borges 2017-06-15 09:09:34 UTC
Created attachment 22913 [details]
Source Code + Assemblies

I developed a .NET tool to query Azure Table Storage. Under Mono only, the tool is failing sporadically (4.3% of the times) trying to establish an SSL connection.


### Details of Runtime ###
Mono version 5.0.1.1, installed as per Mono's documentation in http://www.mono-project.com/download/#download-lin and fully up to date as far as "apt update" goes.

Vanilla Ubuntu Linux 16.06 LTS up to date, running as an Azure instance of size DS2_V2 on datacentre EUW.
Note of potential interest, every 5 seconds I'm "time changed" events into /var/log/syslog:
Jun 15 09:00:47 PerfTeam-MnA-LinuxWorker1 systemd[1]: Time has been changed
Jun 15 09:00:47 PerfTeam-MnA-LinuxWorker1 systemd[1]: snapd.refresh.timer: Adding 4h 41min 9.715078s random time.
Jun 15 09:00:47 PerfTeam-MnA-LinuxWorker1 systemd[1]: apt-daily.timer: Adding 7h 48min 24.261667s random time.
Jun 15 09:00:52 PerfTeam-MnA-LinuxWorker1 systemd[1]: Time has been changed
Jun 15 09:00:52 PerfTeam-MnA-LinuxWorker1 systemd[1]: snapd.refresh.timer: Adding 1h 24min 13.757854s random time.
Jun 15 09:00:52 PerfTeam-MnA-LinuxWorker1 systemd[1]: apt-daily.timer: Adding 18min 45.335117s random time.

Executables compiled with VS 2013 targeting .NET 4.5.
Source code of relevant file + linked non-default assemblies, attached.
Let me know if you need anything else to investigate this.


### Details of failure ###
Around 4.3% of the times, the tool just fails to connect to all data sources. It is always failing either none or all of the attempts to connect to the Azure Table Storage (it tries to pull data from 24 different connection strings).

The exceptions logged are:

##############################
# Azure_PerfMon_Querier starting work now 2017-06-14 16:17:02
Preparing to query PartitionKey 0636330536400000000
Microsoft.WindowsAzure.Storage.StorageException: Error: TrustFailure (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /tmp/buildd/mono-5.0.1.1/external/boringssl/ssl/handshake_client.c:1132) ---> System.Net.WebException: Error: TrustFailure (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /tmp/buildd/mono-5.0.1.1/external/boringssl/ssl/handshake_client.c:1132) ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /tmp/buildd/mono-5.0.1.1/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00038] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncProtocolRequest asyncRequest, Mono.Net.Security.AsyncOperationStatus status) [0x0002a] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (Mono.Net.Security.AsyncOperationStatus status) [0x0006b] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation () [0x0000d] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Mono.Net.Security.AsyncProtocolRequest.StartOperation () [0x0003c] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Mono.Net.Security.AsyncProtocolRequest.StartOperation (Mono.Net.Security.AsyncOperation operation) [0x00024] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Net.LazyAsyncResult lazyResult) [0x00057] in <2b0d86369d72459baed0cee98a8e578a>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <a07d6bf484a54da2861691df910339b1>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Net.LazyAsyncResult lazyResult) [0x00078] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x0000c] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Mono.Net.Security.Private.MonoSslStreamWrapper.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x00006] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Byte[] buffer) [0x0007b] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at System.Net.WebConnection.CreateStream (System.Net.HttpWebRequest request) [0x00073] in <2b0d86369d72459baed0cee98a8e578a>:0 
   --- End of inner exception stack trace ---
  at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00058] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x0000e] in <2b0d86369d72459baed0cee98a8e578a>:0 
  at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T] (Microsoft.WindowsAzure.Storage.Core.Executor.RESTCommand`1[T] cmd, Microsoft.WindowsAzure.Storage.RetryPolicies.IRetryPolicy policy, Microsoft.WindowsAzure.Storage.OperationContext operationContext) [0x00242] in <1a6da444611441028237826045bb8126>:0 
   --- End of inner exception stack trace ---
  at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T] (Microsoft.WindowsAzure.Storage.Core.Executor.RESTCommand`1[T] cmd, Microsoft.WindowsAzure.Storage.RetryPolicies.IRetryPolicy policy, Microsoft.WindowsAzure.Storage.OperationContext operationContext) [0x006b5] in <1a6da444611441028237826045bb8126>:0 
  at Microsoft.WindowsAzure.Storage.Table.TableQuery`1[TElement].ExecuteQuerySegmentedInternal (Microsoft.WindowsAzure.Storage.Table.TableContinuationToken token, Microsoft.WindowsAzure.Storage.Table.CloudTableClient client, Microsoft.WindowsAzure.Storage.Table.CloudTable table, Microsoft.WindowsAzure.Storage.Table.TableRequestOptions requestOptions, Microsoft.WindowsAzure.Storage.OperationContext operationContext) [0x00044] in <1a6da444611441028237826045bb8126>:0 
  at Microsoft.WindowsAzure.Storage.Table.TableQuery`1+<>c__DisplayClass7[TElement].<ExecuteInternal>b__6 (Microsoft.WindowsAzure.Storage.IContinuationToken continuationToken) [0x00000] in <1a6da444611441028237826045bb8126>:0 
  at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility+<LazyEnumerable>d__0`1[T].MoveNext () [0x00025] in <1a6da444611441028237826045bb8126>:0 
  at System.Collections.Generic.List`1[T].InsertRange (System.Int32 index, System.Collections.Generic.IEnumerable`1[T] collection) [0x000ea] in <a07d6bf484a54da2861691df910339b1>:0 
  at System.Collections.Generic.List`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] collection) [0x00000] in <a07d6bf484a54da2861691df910339b1>:0 
  at Azure_PerfMon_Querier.NET.Extraction.ExtractionWorker.DoUnitOfWork_unsafe_unwrapped (Azure_PerfMon_Querier.NET.DataModel.UnitOfWork unitOfWork) [0x000a6] in <c3cc1e9188da466e85153cf1de42e594>:0 
  at Azure_PerfMon_Querier.NET.Extraction.ExtractionWorker.DoUnitOfWork_unsafe (Azure_PerfMon_Querier.NET.DataModel.UnitOfWork unitOfWork) [0x00007] in <c3cc1e9188da466e85153cf1de42e594>:0 
  at Azure_PerfMon_Querier.NET.Extraction.ExtractionWorker.DoUnitOfWork (Azure_PerfMon_Querier.NET.DataModel.UnitOfWork unitOfWork) [0x00002] in <c3cc1e9188da466e85153cf1de42e594>:0 
Request Information
RequestID:
RequestDate:
StatusMessage:

Microsoft.WindowsAzure.Storage.StorageException: Error: TrustFailure (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /tmp/buildd/mono-5.0.1.1/external/boringssl/ssl/handshake_client.c:1132) ---> System.Net.WebException: Error: TrustFailure (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /tmp/buildd/mono-5.0.1.1/external/boringssl/ssl/handshake_client.c:1132) ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
...
Comment 1 Marek Safar 2017-09-12 13:51:48 UTC
Could you please attach full project. I cannot compile it using files you provided.

Thank you!
Comment 2 paulo.borges 2017-09-12 14:25:13 UTC
Hi Marek,

That project is my company's private intellectual property and these bug reports are public I believe?

I'll ask my manager for permission to share the complete source code in here.
Comment 3 Marek Safar 2017-09-12 14:39:40 UTC
You can mark the bug private if you want or share only bits which reproduce the issue
Comment 4 paulo.borges 2017-09-14 13:20:46 UTC
Hi Marek Safar,
Sorry it took this long.

I've gotten permission to share the code with you but only in private.
As you've mentioned, I looked for a way to mark the bug private or to upload something in private but was unable to discover how to do it.

I've emailed you directly the source code. I'm hoping it reaches you...
Comment 5 Enrico Sada 2018-01-22 15:47:28 UTC
@marek i can repro too.

Happen during test suite running in travis (https://travis-ci.org/fsprojects/Paket/jobs/331845176#L7399 all the environment info on top) 

Test execution just run paket.exe multiple tims to download from nuget.org. test who fail is random.

the BTLS certificates are already imported correctly at start (https://travis-ci.org/fsprojects/Paket/jobs/331845176#L1588 )

Same code compile and execute correctly in .net core (see jobs from same build matrix)

How to repro: clone + `build RunIntegrationTests`


$ mono --version
Mono JIT compiler version 5.4.1.7 (tarball Wed Jan 17 20:08:37 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen (concurrent by default)