Problem using FIPS with PKI

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem using FIPS with PKI

Christophe Webb

I am running keycloak in a docker container. We are using PKI as one of the authentication methods for our applications. I followed the instructions for keycloak(X.509 Client Certificate User Authentication) to set this up, and everything seems to work. Next, we needed to update keycloak to be FIPS compliant. For this, we are using the bouncy castle FIPS provider(bc-fips-1.0.1.jar). I have set up the java.security file to make the bouncy castle fips provider the default. This all works correctly as well.

However, once I update the java.security file to use "com.sun.net.ssl.internal.ssl.Provider BCFIPS", PKI no longer works. Regular SSL with out a client certificate provided works just fine, and we can log in with username and password, but we need PKI.
I have updated the keycloak standalone.xml with the following:
<server-identities>
<ssl>
<keystore provider="BCFKS" path="keystore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>" alias="keycloak" key-password="<password>"/>
</ssl>
</server-identities>
<authentication>
<truststore provider="BCFKS" path="truststore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>"/>
<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>

I took the JKS files for the keystore and truststore that I was using before and imported them to BCFKS files using this basic command:
keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

I also updated the JAVA_OPTS to include -Djavax.net.debug=ssl. In the output, I can see that my certificate is provided, and it looks correct.

In the log output after the client certificate is logged, I see the the following log statements.

14:38:30,927 INFO [stdout] (default task-1) default task-1, fatal error: 46: General SSLEngine problem
14:38:30,927 INFO [stdout] (default task-1) sun.security.validator.ValidatorException: No trusted certificate found
14:38:30,927 INFO [stdout] (default task-1) %% Invalidated: [Session-2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
14:38:30,927 INFO [stdout] (default task-1) default task-1, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
14:38:30,927 INFO [stdout] (default task-1) default task-1, WRITE: TLSv1.2 Alert, length = 2
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeInbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeOutbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, closeOutboundInternal()


Is it possible that I created the BCFKS keystore and truststore incorrectly? If I switch back to the jks files, PKI works without issue.

Reply | Threaded
Open this post in threaded view
|

Re: Problem using FIPS with PKI

Arshad Noor-2

If you did not get an error with the keytool command, then the files were created correctly.

There are usually three problems with keytool many people stumble on:

  • The keystore and truststore are two different files.  You've shown the command to import the keys into the keystore in your e-mail, but not the corresponding command to create the truststore.  The trusted certificate the logs are complaining about, is in the truststore.  I trust (sorry for the pun) you created the truststore similarly;
  • keytool silently creates a new keystore/truststore in the local directory without complaining, as long as the command parameters are correct.  Application servers use their keystores/truststores only from certain configuration folders.  If the BCFKS file you created is not in the folder your application server expects, its most likely using the original truststore  - which probably does not have the trusted CA certificate you needed to import for TLS ClientAuth to work.  confirm that the truststore your application server is using has a timestamp equivalent to the time you created the truststore; to be absolutely certain, list the certificates from your truststore to confirm that your trusted self-signed CA's certificate is in there;
  • The last problem is that the truststore does not have the precise self-signed CA certificate, or intermediate CA certificates, for the specific Client certificate you're using to perform client authentication.  Verify that you have the chain in the keystore (at least up to the self-signed Root CA certificate) and the self-signed Root CA certificate in the truststore.

Arshad Noor
StrongKey

On 3/27/19 12:06 PM, Christophe Webb wrote:

I am running keycloak in a docker container. We are using PKI as one of the authentication methods for our applications. I followed the instructions for keycloak(X.509 Client Certificate User Authentication) to set this up, and everything seems to work. Next, we needed to update keycloak to be FIPS compliant. For this, we are using the bouncy castle FIPS provider(bc-fips-1.0.1.jar). I have set up the java.security file to make the bouncy castle fips provider the default. This all works correctly as well.

However, once I update the java.security file to use "com.sun.net.ssl.internal.ssl.Provider BCFIPS", PKI no longer works. Regular SSL with out a client certificate provided works just fine, and we can log in with username and password, but we need PKI.
I have updated the keycloak standalone.xml with the following:
<server-identities>
<ssl>
<keystore provider="BCFKS" path="keystore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>" alias="keycloak" key-password="<password>"/>
</ssl>
</server-identities>
<authentication>
<truststore provider="BCFKS" path="truststore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>"/>
<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>

I took the JKS files for the keystore and truststore that I was using before and imported them to BCFKS files using this basic command:
keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

I also updated the JAVA_OPTS to include -Djavax.net.debug=ssl. In the output, I can see that my certificate is provided, and it looks correct.

In the log output after the client certificate is logged, I see the the following log statements.

14:38:30,927 INFO [stdout] (default task-1) default task-1, fatal error: 46: General SSLEngine problem
14:38:30,927 INFO [stdout] (default task-1) sun.security.validator.ValidatorException: No trusted certificate found
14:38:30,927 INFO [stdout] (default task-1) %% Invalidated: [Session-2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
14:38:30,927 INFO [stdout] (default task-1) default task-1, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
14:38:30,927 INFO [stdout] (default task-1) default task-1, WRITE: TLSv1.2 Alert, length = 2
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeInbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeOutbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, closeOutboundInternal()


Is it possible that I created the BCFKS keystore and truststore incorrectly? If I switch back to the jks files, PKI works without issue.

Reply | Threaded
Open this post in threaded view
|

RE: Problem using FIPS with PKI

Christophe Webb
In reply to this post by Christophe Webb

Sorry, I should have been more clear with my initial question.

I did create both a keystore and a truststore. If I reference a truststore file that doesn’t exist keycloak presents a different error. The only difference between the command used to create the truststore vs the keystore is the “-srckeystore” and “-destkeystore” arguments in the keytool command from my initial post. So, I did the following (minus the paths and passwords):

keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

keytool -importkeystore -srckeystore truststore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore truststore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

 

As I mentioned before, if I switch it back to using the original keystore.jks and truststore.jks files I had created before, and update the java.security file to remove “BCFIPS”, PKI will work as expected.

 

v/r

 

Chris Webb

 

From: Arshad Noor <[hidden email]>
Sent: Friday, March 29, 2019 8:09 AM
To: Christophe Webb <[hidden email]>; [hidden email]
Subject: Re: [dev-crypto] Problem using FIPS with PKI

 

If you did not get an error with the keytool command, then the files were created correctly.

There are usually three problems with keytool many people stumble on:

  • The keystore and truststore are two different files.  You've shown the command to import the keys into the keystore in your e-mail, but not the corresponding command to create the truststore.  The trusted certificate the logs are complaining about, is in the truststore.  I trust (sorry for the pun) you created the truststore similarly;
  • keytool silently creates a new keystore/truststore in the local directory without complaining, as long as the command parameters are correct.  Application servers use their keystores/truststores only from certain configuration folders.  If the BCFKS file you created is not in the folder your application server expects, its most likely using the original truststore  - which probably does not have the trusted CA certificate you needed to import for TLS ClientAuth to work.  confirm that the truststore your application server is using has a timestamp equivalent to the time you created the truststore; to be absolutely certain, list the certificates from your truststore to confirm that your trusted self-signed CA's certificate is in there;
  • The last problem is that the truststore does not have the precise self-signed CA certificate, or intermediate CA certificates, for the specific Client certificate you're using to perform client authentication.  Verify that you have the chain in the keystore (at least up to the self-signed Root CA certificate) and the self-signed Root CA certificate in the truststore.

Arshad Noor
StrongKey

On 3/27/19 12:06 PM, Christophe Webb wrote:

I am running keycloak in a docker container. We are using PKI as one of the authentication methods for our applications. I followed the instructions for keycloak(X.509 Client Certificate User Authentication) to set this up, and everything seems to work. Next, we needed to update keycloak to be FIPS compliant. For this, we are using the bouncy castle FIPS provider(bc-fips-1.0.1.jar). I have set up the java.security file to make the bouncy castle fips provider the default. This all works correctly as well.

However, once I update the java.security file to use "com.sun.net.ssl.internal.ssl.Provider BCFIPS", PKI no longer works. Regular SSL with out a client certificate provided works just fine, and we can log in with username and password, but we need PKI.
I have updated the keycloak standalone.xml with the following:
<server-identities>
<ssl>
<keystore provider="BCFKS" path="keystore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>" alias="keycloak" key-password="<password>"/>
</ssl>
</server-identities>
<authentication>
<truststore provider="BCFKS" path="truststore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>"/>
<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>

I took the JKS files for the keystore and truststore that I was using before and imported them to BCFKS files using this basic command:
keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

I also updated the JAVA_OPTS to include -Djavax.net.debug=ssl. In the output, I can see that my certificate is provided, and it looks correct.

In the log output after the client certificate is logged, I see the the following log statements.

14:38:30,927 INFO [stdout] (default task-1) default task-1, fatal error: 46: General SSLEngine problem
14:38:30,927 INFO [stdout] (default task-1) sun.security.validator.ValidatorException: No trusted certificate found
14:38:30,927 INFO [stdout] (default task-1) %% Invalidated: [Session-2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
14:38:30,927 INFO [stdout] (default task-1) default task-1, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
14:38:30,927 INFO [stdout] (default task-1) default task-1, WRITE: TLSv1.2 Alert, length = 2
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeInbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeOutbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, closeOutboundInternal()

 

Is it possible that I created the BCFKS keystore and truststore incorrectly? If I switch back to the jks files, PKI works without issue.

 

Image removed by sender.

Virus-free. www.avg.com

 

Reply | Threaded
Open this post in threaded view
|

Re: Problem using FIPS with PKI

Arshad Noor-2

Have you verified that the truststore.bcfks file not only contains the self-signed Root CA certificate of the chain you need to verify the client certificate, and that the truststore.bcfks file is in the precise directory JBoss needs to establish the TLS session?

I would troubleshoot this as follows:

  1.  Move the keystore.jks/truststore.jks and keystore.bcfks/truststore.bcfks files to other names temporarily from the folder you think is being used by JBoss to start TLS;
  2. Start JBoss.  If it complains of starting TLS because it cannot find the JKS and BCFKS files, then you're in the right directory;
  3. Rename the keystore.bcfks and truststore.bcfks files to their original names;
  4. Restart JBoss.  If TLS starts and you're still failing ClientAuth (because of a missing trusted certificate, then you're missing either the self-signed or an intermediate certificate of the chain in your truststore.bcfks;
  5. Add them and confirm that they do exist by listing out the certs using keytool;
  6. Restart JBoss.  At this point TLS ClientAuth should work.  If it doesn't then you may need to turn on JBoss debugging at more granular levels to see what the TLS library is doing, as it performs certificate validation, to learn what it is complaining about. I'm assuming, at this point, that all the CA certs you have in the truststore are the correct ones that are part of the chain.  Be careful with assuming that the certificates are correct based on the Subject DN/IssuerDN - unless the SubjectKeyIdentifier (SKI) of the Issuer and the AuthorityKeyIdenfier (AKI) of the subject match (along with Subject DN of the issuer the Issuer DN of the subject), the chain will not be identical.  This problem can arise if you've issued multiple certificates with the same DNs, or if there are expired certificates in the chain.

Arshad Noor
StrongKey


On 3/29/19 10:07 AM, Chris Webb wrote:

Sorry, I should have been more clear with my initial question.

I did create both a keystore and a truststore. If I reference a truststore file that doesn’t exist keycloak presents a different error. The only difference between the command used to create the truststore vs the keystore is the “-srckeystore” and “-destkeystore” arguments in the keytool command from my initial post. So, I did the following (minus the paths and passwords):

keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

keytool -importkeystore -srckeystore truststore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore truststore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

 

As I mentioned before, if I switch it back to using the original keystore.jks and truststore.jks files I had created before, and update the java.security file to remove “BCFIPS”, PKI will work as expected.

 

v/r

 

Chris Webb

 

From: Arshad Noor [hidden email]
Sent: Friday, March 29, 2019 8:09 AM
To: Christophe Webb [hidden email]; [hidden email]
Subject: Re: [dev-crypto] Problem using FIPS with PKI

 

If you did not get an error with the keytool command, then the files were created correctly.

There are usually three problems with keytool many people stumble on:

  • The keystore and truststore are two different files.  You've shown the command to import the keys into the keystore in your e-mail, but not the corresponding command to create the truststore.  The trusted certificate the logs are complaining about, is in the truststore.  I trust (sorry for the pun) you created the truststore similarly;
  • keytool silently creates a new keystore/truststore in the local directory without complaining, as long as the command parameters are correct.  Application servers use their keystores/truststores only from certain configuration folders.  If the BCFKS file you created is not in the folder your application server expects, its most likely using the original truststore  - which probably does not have the trusted CA certificate you needed to import for TLS ClientAuth to work.  confirm that the truststore your application server is using has a timestamp equivalent to the time you created the truststore; to be absolutely certain, list the certificates from your truststore to confirm that your trusted self-signed CA's certificate is in there;
  • The last problem is that the truststore does not have the precise self-signed CA certificate, or intermediate CA certificates, for the specific Client certificate you're using to perform client authentication.  Verify that you have the chain in the keystore (at least up to the self-signed Root CA certificate) and the self-signed Root CA certificate in the truststore.

Arshad Noor
StrongKey

On 3/27/19 12:06 PM, Christophe Webb wrote:

I am running keycloak in a docker container. We are using PKI as one of the authentication methods for our applications. I followed the instructions for keycloak(X.509 Client Certificate User Authentication) to set this up, and everything seems to work. Next, we needed to update keycloak to be FIPS compliant. For this, we are using the bouncy castle FIPS provider(bc-fips-1.0.1.jar). I have set up the java.security file to make the bouncy castle fips provider the default. This all works correctly as well.

However, once I update the java.security file to use "com.sun.net.ssl.internal.ssl.Provider BCFIPS", PKI no longer works. Regular SSL with out a client certificate provided works just fine, and we can log in with username and password, but we need PKI.
I have updated the keycloak standalone.xml with the following:
<server-identities>
<ssl>
<keystore provider="BCFKS" path="keystore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>" alias="keycloak" key-password="<password>"/>
</ssl>
</server-identities>
<authentication>
<truststore provider="BCFKS" path="truststore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>"/>
<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>

I took the JKS files for the keystore and truststore that I was using before and imported them to BCFKS files using this basic command:
keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

I also updated the JAVA_OPTS to include -Djavax.net.debug=ssl. In the output, I can see that my certificate is provided, and it looks correct.

In the log output after the client certificate is logged, I see the the following log statements.

14:38:30,927 INFO [stdout] (default task-1) default task-1, fatal error: 46: General SSLEngine problem
14:38:30,927 INFO [stdout] (default task-1) sun.security.validator.ValidatorException: No trusted certificate found
14:38:30,927 INFO [stdout] (default task-1) %% Invalidated: [Session-2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
14:38:30,927 INFO [stdout] (default task-1) default task-1, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
14:38:30,927 INFO [stdout] (default task-1) default task-1, WRITE: TLSv1.2 Alert, length = 2
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeInbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeOutbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, closeOutboundInternal()

 

Is it possible that I created the BCFKS keystore and truststore incorrectly? If I switch back to the jks files, PKI works without issue.

 

Image removed by sender.

Virus-free. www.avg.com

 

Reply | Threaded
Open this post in threaded view
|

Re: Problem using FIPS with PKI

David Hook-3

I'd run through Arshad's list first, but one other thing to check as well - it may be assuming that the certs keystore can have it's certificates loaded without a password (true in JKS, not true in BCFKS). I haven't heard of this before in connection with JBoss before but you may need to specify a password where you did not originally. We ran into something like this a while ago in another application, although it turned out the certs keystore had to be JKS - BC-FJA 1.0.2 has a new FIPS type key store which will read both BCFKS and JKS for this reason.

Regards,

David

On 31/3/19 3:57 am, Arshad Noor wrote:

Have you verified that the truststore.bcfks file not only contains the self-signed Root CA certificate of the chain you need to verify the client certificate, and that the truststore.bcfks file is in the precise directory JBoss needs to establish the TLS session?

I would troubleshoot this as follows:

  1.  Move the keystore.jks/truststore.jks and keystore.bcfks/truststore.bcfks files to other names temporarily from the folder you think is being used by JBoss to start TLS;
  2. Start JBoss.  If it complains of starting TLS because it cannot find the JKS and BCFKS files, then you're in the right directory;
  3. Rename the keystore.bcfks and truststore.bcfks files to their original names;
  4. Restart JBoss.  If TLS starts and you're still failing ClientAuth (because of a missing trusted certificate, then you're missing either the self-signed or an intermediate certificate of the chain in your truststore.bcfks;
  5. Add them and confirm that they do exist by listing out the certs using keytool;
  6. Restart JBoss.  At this point TLS ClientAuth should work.  If it doesn't then you may need to turn on JBoss debugging at more granular levels to see what the TLS library is doing, as it performs certificate validation, to learn what it is complaining about. I'm assuming, at this point, that all the CA certs you have in the truststore are the correct ones that are part of the chain.  Be careful with assuming that the certificates are correct based on the Subject DN/IssuerDN - unless the SubjectKeyIdentifier (SKI) of the Issuer and the AuthorityKeyIdenfier (AKI) of the subject match (along with Subject DN of the issuer the Issuer DN of the subject), the chain will not be identical.  This problem can arise if you've issued multiple certificates with the same DNs, or if there are expired certificates in the chain.

Arshad Noor
StrongKey


On 3/29/19 10:07 AM, Chris Webb wrote:

Sorry, I should have been more clear with my initial question.

I did create both a keystore and a truststore. If I reference a truststore file that doesn’t exist keycloak presents a different error. The only difference between the command used to create the truststore vs the keystore is the “-srckeystore” and “-destkeystore” arguments in the keytool command from my initial post. So, I did the following (minus the paths and passwords):

keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

keytool -importkeystore -srckeystore truststore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore truststore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

 

As I mentioned before, if I switch it back to using the original keystore.jks and truststore.jks files I had created before, and update the java.security file to remove “BCFIPS”, PKI will work as expected.

 

v/r

 

Chris Webb

 

From: Arshad Noor [hidden email]
Sent: Friday, March 29, 2019 8:09 AM
To: Christophe Webb [hidden email]; [hidden email]
Subject: Re: [dev-crypto] Problem using FIPS with PKI

 

If you did not get an error with the keytool command, then the files were created correctly.

There are usually three problems with keytool many people stumble on:

  • The keystore and truststore are two different files.  You've shown the command to import the keys into the keystore in your e-mail, but not the corresponding command to create the truststore.  The trusted certificate the logs are complaining about, is in the truststore.  I trust (sorry for the pun) you created the truststore similarly;
  • keytool silently creates a new keystore/truststore in the local directory without complaining, as long as the command parameters are correct.  Application servers use their keystores/truststores only from certain configuration folders.  If the BCFKS file you created is not in the folder your application server expects, its most likely using the original truststore  - which probably does not have the trusted CA certificate you needed to import for TLS ClientAuth to work.  confirm that the truststore your application server is using has a timestamp equivalent to the time you created the truststore; to be absolutely certain, list the certificates from your truststore to confirm that your trusted self-signed CA's certificate is in there;
  • The last problem is that the truststore does not have the precise self-signed CA certificate, or intermediate CA certificates, for the specific Client certificate you're using to perform client authentication.  Verify that you have the chain in the keystore (at least up to the self-signed Root CA certificate) and the self-signed Root CA certificate in the truststore.

Arshad Noor
StrongKey

On 3/27/19 12:06 PM, Christophe Webb wrote:

I am running keycloak in a docker container. We are using PKI as one of the authentication methods for our applications. I followed the instructions for keycloak(X.509 Client Certificate User Authentication) to set this up, and everything seems to work. Next, we needed to update keycloak to be FIPS compliant. For this, we are using the bouncy castle FIPS provider(bc-fips-1.0.1.jar). I have set up the java.security file to make the bouncy castle fips provider the default. This all works correctly as well.

However, once I update the java.security file to use "com.sun.net.ssl.internal.ssl.Provider BCFIPS", PKI no longer works. Regular SSL with out a client certificate provided works just fine, and we can log in with username and password, but we need PKI.
I have updated the keycloak standalone.xml with the following:
<server-identities>
<ssl>
<keystore provider="BCFKS" path="keystore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>" alias="keycloak" key-password="<password>"/>
</ssl>
</server-identities>
<authentication>
<truststore provider="BCFKS" path="truststore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>"/>
<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>

I took the JKS files for the keystore and truststore that I was using before and imported them to BCFKS files using this basic command:
keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

I also updated the JAVA_OPTS to include -Djavax.net.debug=ssl. In the output, I can see that my certificate is provided, and it looks correct.

In the log output after the client certificate is logged, I see the the following log statements.

14:38:30,927 INFO [stdout] (default task-1) default task-1, fatal error: 46: General SSLEngine problem
14:38:30,927 INFO [stdout] (default task-1) sun.security.validator.ValidatorException: No trusted certificate found
14:38:30,927 INFO [stdout] (default task-1) %% Invalidated: [Session-2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
14:38:30,927 INFO [stdout] (default task-1) default task-1, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
14:38:30,927 INFO [stdout] (default task-1) default task-1, WRITE: TLSv1.2 Alert, length = 2
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeInbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeOutbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, closeOutboundInternal()

 

Is it possible that I created the BCFKS keystore and truststore incorrectly? If I switch back to the jks files, PKI works without issue.

 

Image removed by sender.

Virus-free. www.avg.com

 


Reply | Threaded
Open this post in threaded view
|

Re: Problem using FIPS with PKI

Christophe Webb

I would be interested in trying out 1.0.2 if that allows me to use the JKS files I was using previously. How would I go about getting early access to it?


v/r


Chris Webb

On March 30, 2019 at 7:52 PM David Hook <[hidden email]> wrote:


I'd run through Arshad's list first, but one other thing to check as well - it may be assuming that the certs keystore can have it's certificates loaded without a password (true in JKS, not true in BCFKS). I haven't heard of this before in connection with JBoss before but you may need to specify a password where you did not originally. We ran into something like this a while ago in another application, although it turned out the certs keystore had to be JKS - BC-FJA 1.0.2 has a new FIPS type key store which will read both BCFKS and JKS for this reason.

Regards,

David

On 31/3/19 3:57 am, Arshad Noor wrote:

Have you verified that the truststore.bcfks file not only contains the self-signed Root CA certificate of the chain you need to verify the client certificate, and that the truststore.bcfks file is in the precise directory JBoss needs to establish the TLS session?

I would troubleshoot this as follows:

  1.  Move the keystore.jks/truststore.jks and keystore.bcfks/truststore.bcfks files to other names temporarily from the folder you think is being used by JBoss to start TLS;
  2. Start JBoss.  If it complains of starting TLS because it cannot find the JKS and BCFKS files, then you're in the right directory;
  3. Rename the keystore.bcfks and truststore.bcfks files to their original names;
  4. Restart JBoss.  If TLS starts and you're still failing ClientAuth (because of a missing trusted certificate, then you're missing either the self-signed or an intermediate certificate of the chain in your truststore.bcfks;
  5. Add them and confirm that they do exist by listing out the certs using keytool;
  6. Restart JBoss.  At this point TLS ClientAuth should work.  If it doesn't then you may need to turn on JBoss debugging at more granular levels to see what the TLS library is doing, as it performs certificate validation, to learn what it is complaining about. I'm assuming, at this point, that all the CA certs you have in the truststore are the correct ones that are part of the chain.  Be careful with assuming that the certificates are correct based on the Subject DN/IssuerDN - unless the SubjectKeyIdentifier (SKI) of the Issuer and the AuthorityKeyIdenfier (AKI) of the subject match (along with Subject DN of the issuer the Issuer DN of the subject), the chain will not be identical.  This problem can arise if you've issued multiple certificates with the same DNs, or if there are expired certificates in the chain.

Arshad Noor
StrongKey


On 3/29/19 10:07 AM, Chris Webb wrote:

Sorry, I should have been more clear with my initial question.

I did create both a keystore and a truststore. If I reference a truststore file that doesn’t exist keycloak presents a different error. The only difference between the command used to create the truststore vs the keystore is the “-srckeystore” and “-destkeystore” arguments in the keytool command from my initial post. So, I did the following (minus the paths and passwords):

keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

keytool -importkeystore -srckeystore truststore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore truststore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

 

As I mentioned before, if I switch it back to using the original keystore.jks and truststore.jks files I had created before, and update the java.security file to remove “BCFIPS”, PKI will work as expected.

 

v/r

 

Chris Webb

 

From: Arshad Noor [hidden email]
Sent: Friday, March 29, 2019 8:09 AM
To: Christophe Webb [hidden email]; [hidden email]
Subject: Re: [dev-crypto] Problem using FIPS with PKI

 

If you did not get an error with the keytool command, then the files were created correctly.

There are usually three problems with keytool many people stumble on:

  • The keystore and truststore are two different files.  You've shown the command to import the keys into the keystore in your e-mail, but not the corresponding command to create the truststore.  The trusted certificate the logs are complaining about, is in the truststore.  I trust (sorry for the pun) you created the truststore similarly;
  • keytool silently creates a new keystore/truststore in the local directory without complaining, as long as the command parameters are correct.  Application servers use their keystores/truststores only from certain configuration folders.  If the BCFKS file you created is not in the folder your application server expects, its most likely using the original truststore  - which probably does not have the trusted CA certificate you needed to import for TLS ClientAuth to work.  confirm that the truststore your application server is using has a timestamp equivalent to the time you created the truststore; to be absolutely certain, list the certificates from your truststore to confirm that your trusted self-signed CA's certificate is in there;
  • The last problem is that the truststore does not have the precise self-signed CA certificate, or intermediate CA certificates, for the specific Client certificate you're using to perform client authentication.  Verify that you have the chain in the keystore (at least up to the self-signed Root CA certificate) and the self-signed Root CA certificate in the truststore.

Arshad Noor
StrongKey

On 3/27/19 12:06 PM, Christophe Webb wrote:

I am running keycloak in a docker container. We are using PKI as one of the authentication methods for our applications. I followed the instructions for keycloak(X.509 Client Certificate User Authentication) to set this up, and everything seems to work. Next, we needed to update keycloak to be FIPS compliant. For this, we are using the bouncy castle FIPS provider(bc-fips-1.0.1.jar). I have set up the java.security file to make the bouncy castle fips provider the default. This all works correctly as well.

However, once I update the java.security file to use "com.sun.net.ssl.internal.ssl.Provider BCFIPS", PKI no longer works. Regular SSL with out a client certificate provided works just fine, and we can log in with username and password, but we need PKI.
I have updated the keycloak standalone.xml with the following:
<server-identities>
<ssl>
<keystore provider="BCFKS" path="keystore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>" alias="keycloak" key-password="<password>"/>
</ssl>
</server-identities>
<authentication>
<truststore provider="BCFKS" path="truststore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>"/>
<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>

I took the JKS files for the keystore and truststore that I was using before and imported them to BCFKS files using this basic command:
keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

I also updated the JAVA_OPTS to include -Djavax.net.debug=ssl. In the output, I can see that my certificate is provided, and it looks correct.

In the log output after the client certificate is logged, I see the the following log statements.

14:38:30,927 INFO [stdout] (default task-1) default task-1, fatal error: 46: General SSLEngine problem
14:38:30,927 INFO [stdout] (default task-1) sun.security.validator.ValidatorException: No trusted certificate found
14:38:30,927 INFO [stdout] (default task-1) %% Invalidated: [Session-2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
14:38:30,927 INFO [stdout] (default task-1) default task-1, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
14:38:30,927 INFO [stdout] (default task-1) default task-1, WRITE: TLSv1.2 Alert, length = 2
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeInbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeOutbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, closeOutboundInternal()

 

Is it possible that I created the BCFKS keystore and truststore incorrectly? If I switch back to the jks files, PKI works without issue.

 

Image removed by sender.

Virus-free. www.avg.com

 



 

Reply | Threaded
Open this post in threaded view
|

Re: Problem using FIPS with PKI

David Hook-3
Hi Chris,

Early access to the ongoing FIPS work is available to people who have a BC support contract Bronze level or above (at this point you can probably work out how we pay for this).

If you would like further information about support, drop us a line at [hidden email]

Regards,

David

On 2/4/19 11:59 pm, Christophe Webb wrote:

I would be interested in trying out 1.0.2 if that allows me to use the JKS files I was using previously. How would I go about getting early access to it?


v/r


Chris Webb

On March 30, 2019 at 7:52 PM David Hook [hidden email] wrote:


I'd run through Arshad's list first, but one other thing to check as well - it may be assuming that the certs keystore can have it's certificates loaded without a password (true in JKS, not true in BCFKS). I haven't heard of this before in connection with JBoss before but you may need to specify a password where you did not originally. We ran into something like this a while ago in another application, although it turned out the certs keystore had to be JKS - BC-FJA 1.0.2 has a new FIPS type key store which will read both BCFKS and JKS for this reason.

Regards,

David

On 31/3/19 3:57 am, Arshad Noor wrote:

Have you verified that the truststore.bcfks file not only contains the self-signed Root CA certificate of the chain you need to verify the client certificate, and that the truststore.bcfks file is in the precise directory JBoss needs to establish the TLS session?

I would troubleshoot this as follows:

  1.  Move the keystore.jks/truststore.jks and keystore.bcfks/truststore.bcfks files to other names temporarily from the folder you think is being used by JBoss to start TLS;
  2. Start JBoss.  If it complains of starting TLS because it cannot find the JKS and BCFKS files, then you're in the right directory;
  3. Rename the keystore.bcfks and truststore.bcfks files to their original names;
  4. Restart JBoss.  If TLS starts and you're still failing ClientAuth (because of a missing trusted certificate, then you're missing either the self-signed or an intermediate certificate of the chain in your truststore.bcfks;
  5. Add them and confirm that they do exist by listing out the certs using keytool;
  6. Restart JBoss.  At this point TLS ClientAuth should work.  If it doesn't then you may need to turn on JBoss debugging at more granular levels to see what the TLS library is doing, as it performs certificate validation, to learn what it is complaining about. I'm assuming, at this point, that all the CA certs you have in the truststore are the correct ones that are part of the chain.  Be careful with assuming that the certificates are correct based on the Subject DN/IssuerDN - unless the SubjectKeyIdentifier (SKI) of the Issuer and the AuthorityKeyIdenfier (AKI) of the subject match (along with Subject DN of the issuer the Issuer DN of the subject), the chain will not be identical.  This problem can arise if you've issued multiple certificates with the same DNs, or if there are expired certificates in the chain.

Arshad Noor
StrongKey


On 3/29/19 10:07 AM, Chris Webb wrote:

Sorry, I should have been more clear with my initial question.

I did create both a keystore and a truststore. If I reference a truststore file that doesn’t exist keycloak presents a different error. The only difference between the command used to create the truststore vs the keystore is the “-srckeystore” and “-destkeystore” arguments in the keytool command from my initial post. So, I did the following (minus the paths and passwords):

keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

keytool -importkeystore -srckeystore truststore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore truststore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

 

As I mentioned before, if I switch it back to using the original keystore.jks and truststore.jks files I had created before, and update the java.security file to remove “BCFIPS”, PKI will work as expected.

 

v/r

 

Chris Webb

 

From: Arshad Noor [hidden email]
Sent: Friday, March 29, 2019 8:09 AM
To: Christophe Webb [hidden email]; [hidden email]
Subject: Re: [dev-crypto] Problem using FIPS with PKI

 

If you did not get an error with the keytool command, then the files were created correctly.

There are usually three problems with keytool many people stumble on:

  • The keystore and truststore are two different files.  You've shown the command to import the keys into the keystore in your e-mail, but not the corresponding command to create the truststore.  The trusted certificate the logs are complaining about, is in the truststore.  I trust (sorry for the pun) you created the truststore similarly;
  • keytool silently creates a new keystore/truststore in the local directory without complaining, as long as the command parameters are correct.  Application servers use their keystores/truststores only from certain configuration folders.  If the BCFKS file you created is not in the folder your application server expects, its most likely using the original truststore  - which probably does not have the trusted CA certificate you needed to import for TLS ClientAuth to work.  confirm that the truststore your application server is using has a timestamp equivalent to the time you created the truststore; to be absolutely certain, list the certificates from your truststore to confirm that your trusted self-signed CA's certificate is in there;
  • The last problem is that the truststore does not have the precise self-signed CA certificate, or intermediate CA certificates, for the specific Client certificate you're using to perform client authentication.  Verify that you have the chain in the keystore (at least up to the self-signed Root CA certificate) and the self-signed Root CA certificate in the truststore.

Arshad Noor
StrongKey

On 3/27/19 12:06 PM, Christophe Webb wrote:

I am running keycloak in a docker container. We are using PKI as one of the authentication methods for our applications. I followed the instructions for keycloak(X.509 Client Certificate User Authentication) to set this up, and everything seems to work. Next, we needed to update keycloak to be FIPS compliant. For this, we are using the bouncy castle FIPS provider(bc-fips-1.0.1.jar). I have set up the java.security file to make the bouncy castle fips provider the default. This all works correctly as well.

However, once I update the java.security file to use "com.sun.net.ssl.internal.ssl.Provider BCFIPS", PKI no longer works. Regular SSL with out a client certificate provided works just fine, and we can log in with username and password, but we need PKI.
I have updated the keycloak standalone.xml with the following:
<server-identities>
<ssl>
<keystore provider="BCFKS" path="keystore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>" alias="keycloak" key-password="<password>"/>
</ssl>
</server-identities>
<authentication>
<truststore provider="BCFKS" path="truststore.bcfks" relative-to="jboss.server.config.dir" keystore-password="<password>"/>
<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>

I took the JKS files for the keystore and truststore that I was using before and imported them to BCFKS files using this basic command:
keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS -srcstorepass <password> -destkeystore keystore.bcfks -deststoretype BCFKS -deststorepass <password> -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /path/to/jar/bc-fips-1.0.1.jar

I also updated the JAVA_OPTS to include -Djavax.net.debug=ssl. In the output, I can see that my certificate is provided, and it looks correct.

In the log output after the client certificate is logged, I see the the following log statements.

14:38:30,927 INFO [stdout] (default task-1) default task-1, fatal error: 46: General SSLEngine problem
14:38:30,927 INFO [stdout] (default task-1) sun.security.validator.ValidatorException: No trusted certificate found
14:38:30,927 INFO [stdout] (default task-1) %% Invalidated: [Session-2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
14:38:30,927 INFO [stdout] (default task-1) default task-1, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
14:38:30,927 INFO [stdout] (default task-1) default task-1, WRITE: TLSv1.2 Alert, length = 2
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeInbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, called closeOutbound()
14:38:30,928 INFO [stdout] (default I/O-2) default I/O-2, closeOutboundInternal()

 

Is it possible that I created the BCFKS keystore and truststore incorrectly? If I switch back to the jks files, PKI works without issue.

 

Image removed by sender.

Virus-free. www.avg.com