Bug 35064 - certmgr -importKey returns Invalid MAC
Summary: certmgr -importKey returns Invalid MAC
Status: RESOLVED FIXED
Alias: None
Product: Tools
Classification: Mono
Component: other ()
Version: unspecified
Hardware: PC Linux
: Normal normal
Target Milestone: ---
Assignee: Sebastien Pouliot
URL:
Depends on:
Blocks:
 
Reported: 2015-10-19 15:58 UTC by Neale Ferguson
Modified: 2016-01-15 17:53 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Fix handling of unicode null password (1.24 KB, patch)
2016-01-07 17:15 UTC, Neale Ferguson
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Neale Ferguson 2015-10-19 15:58:39 UTC
I used these commands to create certificates and private keys etc. on Windows:

makecert.exe -n "CN=MonoTestCA" -cy authority -a sha1 -len 2048 -pe -r -sv MonoTestCA.pvk MonoTestCA.cer
makecert.exe -n "CN=MonoTestCert" -b 01/01/2000 -e 12/31/2039 -eku 1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2,1.3.6.1.5.5.7.3.3,1.3.6.1.5.5.7.3.4,1.3.6.1.5.5.7.3.5,1.3.6.1.5.5.7.3.6,1.3.6.1.5.5.7.3.7,1.3.6.1.5.5.7.3.8,1.3.6.1.5.5.7.3.9 -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12 -ic MonoTestCA.cer -iv MonoTestCA.pvk -a sha1 -len 2048 -pe -sky exchange -sv MonoTestCert.pvk MonoTestCert.cer
pvk2pfx.exe -pvk MonoTestCert.pvk -spc MonoTestCert.cer -pfx MonoTestCert.pfx

If I the run the certMgr application on Windows I am able to import the private key without error. However, if I issue this command on Linux:

certmgr.exe --importKey -c -v -m My MonoTestCert.pfx

I get:

[neale@lneale3 - ~] MONO_PATH=$MCS/class/lib/net_4_x:$HOME/Mono/xsp/src/lib $MONO/mono/mini/mono-sgen $MCS/tools/security/certmgr.exe --importKey -c -v -m My MonoTestCert.pfx
Mono Certificate Manager - version 4.3.0.0
Manage X.509 certificates and CRL from stores.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Unhandled Exception:
System.Security.Cryptography.CryptographicException: Invalid MAC - file may have been tampered!
  at Mono.Security.X509.PKCS12.Decode (System.Byte[] data) <0x3fffcca5378 + 0x00fd0> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12..ctor (System.Byte[] data) <0x3fffcca4cb0 + 0x0005e> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.LoadFromFile (System.String filename) <0x3fffcca47e8 + 0x0006e> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.LoadCertificates (System.String filename, System.String password, Boolean verbose) <0x3fffcca2fa8 + 0x0046e> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.ImportKey (ObjectType type, Boolean machine, System.String file, System.String password, Boolean verbose) <0x3fffcca2488 + 0x000b2> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.Main (System.String[] args) <0x3fffac5d120 + 0x007d4> in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.Security.Cryptography.CryptographicException: Invalid MAC - file may have been tampered!
  at Mono.Security.X509.PKCS12.Decode (System.Byte[] data) <0x3fffcca5378 + 0x00fd0> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12..ctor (System.Byte[] data) <0x3fffcca4cb0 + 0x0005e> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.LoadFromFile (System.String filename) <0x3fffcca47e8 + 0x0006e> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.LoadCertificates (System.String filename, System.String password, Boolean verbose) <0x3fffcca2fa8 + 0x0046e> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.ImportKey (ObjectType type, Boolean machine, System.String file, System.String password, Boolean verbose) <0x3fffcca2488 + 0x000b2> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.Main (System.String[] args) <0x3fffac5d120 + 0x007d4> in <filename unknown>:0

This happens either 3.10.1 or head (as at 19 Oct 2015). I added some debug code just to see what was happening:

password:  iterations: 2000
MAC does not match calculated MAC
	Lengths: 20 20
47 0D 4F D5 B3 2B D6 62 45 C7 53 8C 33 89 C8 6C B1 DB C5 DA <--- MAC from key
8B 3D 08 E3 F5 CE 37 AB 64 D1 A6 CE F6 B8 86 75 C1 07 4F BE <--- Calculated MAC

There are no passwords on keys/certificates.
Comment 1 Neale Ferguson 2015-11-17 21:48:35 UTC
Just for the hell of it, I bypassed the MAC verification by ignoring the throw when it was found to be invalid. I wanted to see if it would then be imported. It was not:

Unhandled Exception:
System.Security.Cryptography.CryptographicException: Bad PKCS7 padding. Invalid length 140.
  at Mono.Security.Cryptography.SymmetricTransform.ThrowBadPaddingException (PaddingMode padding, Int32 length, Int32 position) <0x3fffa962dd8 + 0x00182> in <filename unknown>:0 
  at Mono.Security.Cryptography.SymmetricTransform.FinalDecrypt (System.Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) <0x3fffa960980 + 0x004a8> in <filename unknown>:0 
  at Mono.Security.Cryptography.SymmetricTransform.TransformFinalBlock (System.Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) <0x3fffa960610 + 0x000d4> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.Decrypt (System.String algorithmOid, System.Byte[] salt, Int32 iterationCount, System.Byte[] encryptedData) <0x3fffa95b1d0 + 0x000ca> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.ReadSafeBag (Mono.Security.ASN1 safeBag) <0x3fffa959930 + 0x00414> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.Decode (System.Byte[] data) <0x3fffcc0b4d8 + 0x0093c> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12..ctor (System.Byte[] data) <0x3fffcc0ae10 + 0x0005e> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.LoadFromFile (System.String filename) <0x3fffcc0a948 + 0x0006e> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.LoadCertificates (System.String filename, System.String password, Boolean verbose) <0x3fffcc09108 + 0x0046e> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.ImportKey (ObjectType type, Boolean machine, System.String file, System.String password, Boolean verbose) <0x3fffcc085e8 + 0x000b2> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.Main (System.String[] args) <0x3fffac5b120 + 0x007d4> in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.Security.Cryptography.CryptographicException: Bad PKCS7 padding. Invalid length 140.
  at Mono.Security.Cryptography.SymmetricTransform.ThrowBadPaddingException (PaddingMode padding, Int32 length, Int32 position) <0x3fffa962dd8 + 0x00182> in <filename unknown>:0 
  at Mono.Security.Cryptography.SymmetricTransform.FinalDecrypt (System.Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) <0x3fffa960980 + 0x004a8> in <filename unknown>:0 
  at Mono.Security.Cryptography.SymmetricTransform.TransformFinalBlock (System.Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) <0x3fffa960610 + 0x000d4> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.Decrypt (System.String algorithmOid, System.Byte[] salt, Int32 iterationCount, System.Byte[] encryptedData) <0x3fffa95b1d0 + 0x000ca> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.ReadSafeBag (Mono.Security.ASN1 safeBag) <0x3fffa959930 + 0x00414> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.Decode (System.Byte[] data) <0x3fffcc0b4d8 + 0x0093c> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12..ctor (System.Byte[] data) <0x3fffcc0ae10 + 0x0005e> in <filename unknown>:0 
  at Mono.Security.X509.PKCS12.LoadFromFile (System.String filename) <0x3fffcc0a948 + 0x0006e> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.LoadCertificates (System.String filename, System.String password, Boolean verbose) <0x3fffcc09108 + 0x0046e> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.ImportKey (ObjectType type, Boolean machine, System.String file, System.String password, Boolean verbose) <0x3fffcc085e8 + 0x000b2> in <filename unknown>:0 
  at Mono.Tools.CertificateManager.Main (System.String[] args) <0x3fffac5b120 + 0x007d4> in <filename unknown>:0
Comment 2 Neale Ferguson 2015-11-18 16:00:06 UTC
I used the openssl command to verify that the pfx file is valid:

> openssl pkcs12 -info -in /tmp/MonoTestCert.pfx 
Enter Import Password:
MAC Iteration 2000
MAC verified OK
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2000
Bag Attributes
    localKeyID: 01 00 00 00 
    Microsoft CSP Name: Microsoft Strong Cryptographic Provider
    friendlyName: PvkTmp:030eaac8-de5d-49cb-875b-0d38c3081ecd
Key Attributes
    X509v3 Key Usage: 10
Comment 3 Neale Ferguson 2015-11-18 16:25:08 UTC
I converted the PFX from DER to PEM and then retried the import:

Mono Certificate Manager - version 4.3.0.0
Manage X.509 certificates and CRL from stores.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

0 keys(s) imported to KeyPair LocalMachine persister.
Comment 4 Neale Ferguson 2015-11-25 12:58:15 UTC
I instrumented openssl pkcs12 to show what it determined the MAC and Key values to be:

Key: 04 68 e0 f7 92 f7 7f d4 22 d4 1b 65 d8 10 c6 0b 24 33 38 1a 99 cf ae a0 00 00 00 00 00 00 00 00 00 00 00 00 80 08 59 f8 00 00 00 4d 60 6e 30 00 00 00 00 4d 60 52 50 08 00 00 00 4d 60 51 95 40 

MAC: c6 40 7c fb 8d 9d 2f 94 8b 2e 43 12 31 8e c3 08 a4 11 23 a7 

Key: 29 04 24 e8 47 43 95 fd 5d 8d c7 9e 9d d8 f6 b4 b7 58 ea 48 99 cf ae a0 00 00 00 00 00 00 00 00 00 00 00 00 80 08 59 f8 00 00 00 4d 60 6e 30 00 00 00 00 4d 60 52 50 08 00 00 00 4d 60 51 95 40 

MAC: 1d 4a 44 58 e9 6d ce ed aa 5f d2 e8 08 ab ea bc ff 8f 70 55 

Derived MAC: 1d 4a 44 58 e9 6d ce ed aa 5f d2 e8 08 ab ea bc ff 8f 70 55 

Compare this to certmgr:

Key: 04 68 e0 f7 92 f7 7f d4 22 d4 1b 65 d8 10 c6 0b 24 33 38 1a 
MAC: 1d 4a 44 58 e9 6d ce ed aa 5f d2 e8 08 ab ea bc ff 8f 70 55 
CLC: c6 40 7c fb 8d 9d 2f 94 8b 2e 43 12 31 8e c3 08 a4 11 23 a7 

Note, the calculated value is that of the MAC printed first by the openssl command and the extracted MAC is that of the 2nd MAC  printed by that command. 

The two calculations done by openssl are
1. Password = "" Passlen = 0
2. Password = "" Passlen = 2 (for unicode)

So, it appears the MAC in the PFX file is Unicode but PKCS12 only tries ASCII.
Comment 5 Neale Ferguson 2015-11-25 17:19:49 UTC
This seems to do the trick for the MAC calculation. It doesn't solve the overall problem of loading the key though.

--- a/mcs/class/Mono.Security/Mono.Security.X509/PKCS12.cs
+++ b/mcs/class/Mono.Security/Mono.Security.X509/PKCS12.cs
@@ -383,8 +383,12 @@ namespace Mono.Security.X509 {
 
                                byte[] authSafeData = authSafe.Content [0].Value;
                                byte[] calculatedMac = MAC (_password, macSalt.Value, _iterations, authSafeData);
-                               if (!Compare (macValue, calculatedMac))
-                                       throw new CryptographicException ("Invalid MAC - file may have been tampered!");
+                               if (!Compare (macValue, calculatedMac)) {
+                                       byte[] nullPassword = {0, 0};
+                                       calculatedMac = MAC(nullPassword, macSalt.Value, _iterations, authSafeData);
+                                       if (!Compare (macValue, calculatedMac))
+                                               throw new CryptographicException ("Invalid MAC - file may have been tampe
red!");
+                               }
                        }
 
                        // we now returns to our original presentation - PFX
Comment 6 Miguel de Icaza [MSFT] 2015-12-07 15:18:11 UTC
CCing Sebastien for reviewing that patch
Comment 7 Sebastien Pouliot 2015-12-10 20:39:34 UTC
The patch is fine (even if it does not fix the whole issue). 

I'm pretty sure that the same issue was fixed elsewhere (years ago) and it does need to be checked twice as different implementations uses different defaults (leading to different null lengths).
Comment 8 Neale Ferguson 2015-12-10 20:47:49 UTC
Thanks. Any idea why once that issue is fixed we get the "Bad PKCS7 padding. Invalid length 140." exception?
Comment 9 Sebastien Pouliot 2015-12-10 21:11:51 UTC
Invalid or incomplete (e.g. truncated) data.

PKCS#7 padding uses the number of bytes to pad, so if you need 5 bytes of padding (to match a block) then the last fives bytes would be 0x5, 0x5, 0x5, 0x5 and 0x5.

IOW it's not possible to get 140 as most blocks will be between 64 bits to 256 bits, so max 32 bytes (way less than the 140 value).
Comment 10 Neale Ferguson 2015-12-10 21:15:57 UTC
Using the openssl command set I've verified the key (see comment #2). So I'm not sure where to look next.
Comment 11 Neale Ferguson 2016-01-05 20:47:37 UTC
The problem exists on 3.12 which doesn't use the Microsoft reference source so we can rule that out. 

I compared the operation using the PFX encoded in PCKS12Test which works when the non-password version of the encoded pfx was used as input to certmgr. The difference between this and the pfx that's failing is that the failure uses System.Security.Cryptography.TripleDESCryptoServiceProvider/System.Security.Cryptography.TripleDESTransform whereas the test pfx uses System.Security.Cryptography.RC2CryptoServiceProvider/System.Security.Cryptography.RC2Transform. So the problem appears to be restricted to 3DES.
Comment 12 Neale Ferguson 2016-01-06 18:06:48 UTC
When I ran the decryption through openssl I get a different derived key and IV than with mono:

Mono Key - 0x82 0x8a 0x51 0xf4 0x11 0x23 0xf4 0x4c 0x85 0x2b 0x96 0xf5 0x60 0x18 0x9f 0xd1 0x4a 0xb3 0x98 0xbf 0x0a 0xad 0x46 0x7c 
OpenSSL Key - 0xee 0xbc 0x52 0x41 0x01 0x01 0x19 0x0c 0xe3 0x60 0x87 0x83 0x63 0xc2 0xa5 0x3d 0x91 0x6f 0x12 0xb3 0xaf 0xcc 0x7c 0x06 
Mono IV - 0x8d 0x7f 0x6d 0x7d 0x7f 0x11 0x8b 0xbb 
OpenSSL IV - 0xdd 0xc7 0x10 0xa0 0x8d 0xeb 0x5d 0xef 

Using the openssl key/IV I no longer get the invalid padding value but now get:

Unhandled Exception:
System.Security.Cryptography.CryptographicException: Specified key is not a valid size for this algorithm.
  at System.Security.Cryptography.SymmetricAlgorithm.set_Key (System.Byte[] value) [0x00042] in /home/neale/Mono/mono/external/referencesource/mscorlib/system/security/cryptography/symmetricalgorithm.cs:146
Comment 13 Neale Ferguson 2016-01-06 23:09:27 UTC
I think I've found it. It's related to the first problem (unicode null password). If the MAC verifies using the unicode null password (byte[] = {0, 0} then this needs to replace the value in _password (which is a 0 length array as it's a null non-unicode password). So just adding this to the code seems to make everything work as it should:

@@ -383,8 +399,14 @@ namespace Mono.Security.X509 {
 
                                byte[] authSafeData = authSafe.Content [0].Value;
                                byte[] calculatedMac = MAC (_password, macSalt.Value, _iterations, authSafeData);
-                               if (!Compare (macValue, calculatedMac))
-                                       throw new CryptographicException ("Invalid MAC - file may have been tampered!");
+                               if (!Compare (macValue, calculatedMac)) {
+                                       byte[] nullPassword = {0, 0};
+                                       calculatedMac = MAC(nullPassword, macSalt.Value, _iterations, authSafeData);
+                                       if (!Compare (macValue, calculatedMac))
+                                               Console.Error.WriteLine("Invalid MAC - file may have been tampered!");
+                                               // throw new CryptographicException ("Invalid MAC - file may have been ta
mpered!");
+                                       _password = (byte[]) nullPassword.Clone();
+                               }
                        }
 
                        // we now returns to our original presentation - PFX
Comment 14 Neale Ferguson 2016-01-06 23:12:04 UTC
I've cleaned it up:

--- a/mcs/class/Mono.Security/Mono.Security.X509/PKCS12.cs
+++ b/mcs/class/Mono.Security/Mono.Security.X509/PKCS12.cs
@@ -383,8 +383,13 @@ namespace Mono.Security.X509 {
 
                                byte[] authSafeData = authSafe.Content [0].Value;
                                byte[] calculatedMac = MAC (_password, macSalt.Value, _iterations, authSafeData);
-                               if (!Compare (macValue, calculatedMac))
-                                       throw new CryptographicException ("Invalid MAC - file may have been tampered!");
+                               if (!Compare (macValue, calculatedMac)) {
+                                       byte[] nullPassword = {0, 0};
+                                       calculatedMac = MAC(nullPassword, macSalt.Value, _iterations, authSafeData);
+                                       if (!Compare (macValue, calculatedMac))
+                                               throw new CryptographicException ("Invalid MAC - file may have been tampe
red!");
+                                       _password = (byte[]) nullPassword.Clone();
+                               }
                        }
 
                        // we now returns to our original presentation - PFX
Comment 15 Sebastien Pouliot 2016-01-07 03:56:29 UTC
Hello Neale,

> _password = (byte[]) nullPassword.Clone();

is the Clone needed for some reason (that I don't recall) ? or could you just do:

> _password = nullPassword;

I'm fine with the patch landing in Mono and a huge thanks for tracking this down, I know that simple line must have been hell to figure out :) Happy New Year!
Comment 16 Neale Ferguson 2016-01-07 17:15:23 UTC
Created attachment 14481 [details]
Fix handling of unicode null password
Comment 17 Neale Ferguson 2016-01-07 17:15:48 UTC
Updated patch attached. Will this be sufficient for getting in pushed to master?
Comment 18 Neale Ferguson 2016-01-15 16:53:26 UTC
Pull request #2445 generated.
Comment 19 Alexander Köplinger [MSFT] 2016-01-15 17:53:12 UTC
Merged the PR. Thanks!