TLS Session resumption works on Windows but fails on Linux (BC 1.68)

[Lothar Kimmeringer-4]

Hi,

as described in a previous mail on this list, I try to get rid off horrible
code fiddling with the internals of JDK-classes to get TLS session resumption
working with FTP control and data channels.

I've successfully implemented it using BCJSSE on Windows, i.e. my test cases
go through, at least when doing passive transfers where both connections are
created by the client. At least for my tests on my Windows based development
machine. The same test fails on Jenkins that runs on Linux. This is still
the case with 1.68 where TLS session resumption is explicitly mentioned.

Here in short is my code on the client side (this is the result of a decade
long iteration of test code, so please be gentle ;-)

             assertTrue("check that service is sending passive-mode-confirmation (" + line + ")",
                 line.startsWith("227 Entering Passive Mode ("));
             [more checks concerning the answer and calculate host and port to connect]
             
             bw.write("list\r\n");
             bw.flush();

             passive = factory.createSocket(TextTools.join(address, "."), port);
             FtpConnection.prepareSocketForSessionReuse((SSLSocket) s, (SSLSocket) passive);
             
             assertEquals("check passive-socket being connected", true, passive.isConnected());
             passive.setSoTimeout(10000);
             assertTrue("check type of socket (" + passive.getClass() + ")", passive instanceof SSLSocket);
             BufferedReader passiveBr = new BufferedReader(new InputStreamReader(passive.getInputStream(), "8859_1"));
             
             line = br.readLine();
             assertEquals("check that service is sending port ready to read",
                 "150 ASCII data connection for / (" + TextTools.join(address, ".") + "," + port + ").", line);
             
             int readDataLines = readDataLines(passiveBr);
             assertTrue("check that lines have been read", readDataLines != 0);

factory is a factory created with BCJSSE as provider and the sockets being
returned are BCSSLSocket. That becomes clear with FtpConnection.prepareSocketForSessionReuse
where I added some debug messages:

     public static void prepareSocketForSessionReuse(final SSLSocket oldSocket, final SSLSocket newSocket) throws FtpException {
      System.out.println("prepare socket for reuse");
      if (oldSocket instanceof BCSSLSocket || newSocket instanceof BCSSLSocket) {
             try {
                 BCSSLSocket oldS = (BCSSLSocket) oldSocket;
                 BCSSLSocket newS = (BCSSLSocket) newSocket;
                 BCExtendedSSLSession session = oldS.getBCSession();
              System.out.println("setting old session as session for new socket: " + TextTools.join(session.getId()));
                 newS.setBCSessionToResume(session);
                 System.out.println("new session was set");
              System.out.println("new session after setting: " + TextTools.join(newS.getBCSession().getId()));
                 return;
             }
             catch(ClassCastException cce) {
                 throw new FtpException("incompatible socket types, both sockets must be BC-sockets or non-BC-sockets");
             }
         }
         [I omit the horrors that are here where the session resumption is
          enforced on the original JSSE-TLS-sockets]
     }

The debug messages can be seen together with the verbose messages of my FTP-server:

21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 Reply: 227 Entering Passive Mode (127,0,0,1,188,241)',
  user='remoteID(ftpchannel)' for session TLS34062@127.0.0.1/1609186851319
21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 Received command 'LIST' with resource 'null',
  user='remoteID(ftpchannel)' for session TLS34062@127.0.0.1/1609186851319
prepare socket for reuse
setting old session as session for new socket:
  5f ea 3e 23 40 3f aa 17 12 c6 63 2f f0 8f db 6e 19 18 d4 e8 83 17 a4 80 8b c1 ae ff d9 d7 95 05
new session was set
21:20:51 [MESSAGE] SYSTEM:FTP:FTPCONNECTION Listing: /
21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 Reply: 150 ASCII data connection for / (127.0.0.1,48369).',
  user='remoteID(ftpchannel)' for session TLS34062@127.0.0.1/1609186851319
21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 TLS connection established with protocol TLSv1.2
  and cipher TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 control channel ssl session id:
  5f ea 3e 23 40 3f aa 17 12 c6 63 2f f0 8f db 6e 19 18 d4 e8 83 17 a4 80 8b c1 ae ff d9 d7 95 05
21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 control channel creation time: 1609186851325
21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 data channel ssl session id   :
  5f ea 3e 23 01 e8 a2 3a 2c f0 6c 89 94 da ba 3c 0e f7 84 15 19 6a 6d 19 fd c1 49 09 66 22 4a ad
21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 data channel creation time: 1609186851519
21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 check that session used TLS session resumption
21:20:51 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:44467 Reply: 550 Connection received without reused SSL session',
  user='remoteID(ftpchannel)' for session TLS34062@127.0.0.1/1609186851319
new session after setting: 5f ea 3e 23 01 e8 a2 3a 2c f0 6c 89 94 da ba 3c 0e f7 84 15 19 6a 6d 19 fd c1 49 09 66 22 4a ad

So TLS session resumption failed here.

The same test under Windows:

21:38:20 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:22195 Reply: 227 Entering Passive Mode (127,0,0,1,86,180)', user='remoteID(ftpchannel)' for session TLS22196@127.0.0.1/1609187898494
21:38:20 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:22195 Received command 'LIST' with resource 'null', user='remoteID(ftpchannel)' for session TLS22196@127.0.0.1/1609187898494
21:38:20 [MESSAGE] SYSTEM:FTP:FTPCONNECTION Listing: /
prepare socket for reuse
setting old session as session for new socket: 5f ea 42 3a be d8 13 2f 2f 5d 73 a3 48 21 2c 07 b2 97 c3 9a 7c 8d d7 b7 07 6e 66 e5 9d 4b fd ed
new session was set
21:38:20 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:22195 Reply: 150 ASCII data connection for / (127.0.0.1,22196).', user='remoteID(ftpchannel)' for session TLS22196@127.0.0.1/1609187898494
new session after setting: 5f ea 42 3a be d8 13 2f 2f 5d 73 a3 48 21 2c 07 b2 97 c3 9a 7c 8d d7 b7 07 6e 66 e5 9d 4b fd ed
21:38:20 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:22195 TLS connection established with protocol TLSv1.2 and cipher TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
21:38:20 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:22195 control channel ssl session id: 5f ea 42 3a be d8 13 2f 2f 5d 73 a3 48 21 2c 07 b2 97 c3 9a 7c 8d d7 b7 07 6e 66 e5 9d 4b fd ed
21:38:20 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:22195 control channel creation time: 1609187898498
21:38:20 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:22195 data channel ssl session id   : 5f ea 42 3a be d8 13 2f 2f 5d 73 a3 48 21 2c 07 b2 97 c3 9a 7c 8d d7 b7 07 6e 66 e5 9d 4b fd ed
21:38:20 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:22195 data channel creation time: 1609187898498
21:38:20 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:22195 check that session used TLS session resumption

I'm not sure what's happening here and how this can be solved (which is why I send this mail).
Both test were done with Java 1.8.0 (_181 on Windows, _121 on Linux). I'll download a more
current version of Java 1.8 for Linux to see if that changes anything.


Thanks, a happy new year and cheers

Lothar

[Peter Dettman-3]

Hi Lothar,

If the client-side code is the same in each case, then we should look to
the server and/or the specifics of the initial connection.

Can you comment more on the server used in each case (SecureFtpDaemon):
is it using the same underlying TLS stack or is it platform-specific?

I suggest watching your client-side code in a debugger, including during
the initial ('control') connection. Specific code of interest is in
ProvTlsClient (org.bouncycastle.jsse.provider). The relevant sequence
for a handshake is:

- getSessionToResume to find a resumable session.
- notifySessionID to report the sessionID the server actually chose.
- notifyHandshakeComplete once the handshake succeeds.

For the first connection you would mainly be interested in
notifyHandshakeComplete: is connectionTlsSession.resumable true? Does
sslSessionContext.reportSession end up actually storing the session?

Then for the second connection you want to check that getSessionToResume
returns the intended session object, and that the server actually
selects this session ID (notifySessionID).

Regards,
Pete Dettman


On 29/12/20 3:49 am, Lothar Kimmeringer wrote:

> Hi,
>
> as described in a previous mail on this list, I try to get rid off horrible
> code fiddling with the internals of JDK-classes to get TLS session
> resumption
> working with FTP control and data channels.
>
> I've successfully implemented it using BCJSSE on Windows, i.e. my test
> cases
> go through, at least when doing passive transfers where both connections
> are
> created by the client. At least for my tests on my Windows based
> development
> machine. The same test fails on Jenkins that runs on Linux. This is still
> the case with 1.68 where TLS session resumption is explicitly mentioned.

[Lothar Kimmeringer-4]

Am 29.12.2020 um 09:42 schrieb Peter Dettman:

> If the client-side code is the same in each case, then we should look to
> the server and/or the specifics of the initial connection.

Both are the same in each case. That particular test is a JUnit test
testing the FTP server by opening "hand made" TLS sockets, i.e. I'm
not using any "off the shelf" FTP client for this.

> Can you comment more on the server used in each case (SecureFtpDaemon):

It's mine ;-)

> is it using the same underlying TLS stack or is it platform-specific?

No platform dependency here.
Up until yesterday I was using JSSE but changed it to BCJSSE. The
behavior was the same independent from that, though as long as the
client used BCJSSE instead of JSSE. I've avoided using BC-specific
classes but rely on the JSSE-API by creating an SSLContext by calling
SSLContext.getInstance("SSL", "BCJSSE").

> I suggest watching your client-side code in a debugger, including during
> the initial ('control') connection. Specific code of interest is in
> ProvTlsClient (org.bouncycastle.jsse.provider).

I can try that but it's worth mentioning that it seems that there
is a dependency on the Java version in use. As I've said yesterday,
the Linux test was done using Java 1.8.0_121 while on Windows I
was using 1.8.0_181. I've downloaded the most recent version of
Java 1.8.0 for Linux and the test now works. There seem to be some
dependency in BC-TLS that leads to the different behavior.

For me that's a perfect solution of "good enough".

I can write a Unit test for BC for the usage in FTPS-scenarios
(without the FTP-protocol stuff, the whole thing is ugly enough)
which might make things easier to communicate and debug if needed.


Thanks and cheers, Lothar

[Lothar Kimmeringer-4]

Am 29.12.2020 um 11:36 schrieb Lothar Kimmeringer:
>> is it using the same underlying TLS stack or is it platform-specific?
>
> No platform dependency here.
> Up until yesterday I was using JSSE but changed it to BCJSSE.

Have to take that back, by accident the test still used SunJSSE
as provider for the server side of the test. When changing that
to BCJSSE the test fails again because no session IDs are returned:

13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 TLS connection established with protocol TLSv1.2 and cipher TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 control channel ssl session id:
13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 control channel creation time: 1609244382940
13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 data channel ssl session id   :
13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 data channel creation time: 1609244382982

I've seen this behavior with TLSv1.3 when using SunJSSE which is why
I've restricted the server to TLSv1.2 to be able to provide that
security feature commonly used with FTPS-transfers.

Is this a bug or an intended behavior on the server side (i.e.
sockets coming from a ServerSocket.accept()? Is there another
way to check if the session of tls-connection A is the same
that is used for connection B?


Thanks and cheers, Lothar

[Lothar Kimmeringer-4]

Hi,

happy new year and it starts as last year ended: I'm answering to my own mails ;-)
German Usenet once called it (translated) "doing the Ingrid".

Fist of all: My original problem really seemed to be a Java version related
thing. After updating to the most current version of Java 1.8 that particular
test succeeded on Linux as well (and up to Java 15). But there are other (platform
independent problems).

Am 29.12.2020 um 13:27 schrieb Lothar Kimmeringer:

>
> Am 29.12.2020 um 11:36 schrieb Lothar Kimmeringer:
>>> is it using the same underlying TLS stack or is it platform-specific?
>>
>> No platform dependency here.
>> Up until yesterday I was using JSSE but changed it to BCJSSE.
>
> Have to take that back, by accident the test still used SunJSSE
> as provider for the server side of the test. When changing that
> to BCJSSE the test fails again because no session IDs are returned:
>
> 13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 TLS connection established with protocol TLSv1.2 and cipher TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
> 13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 control channel ssl session id:
> 13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 control channel creation time: 1609244382940
> 13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 data channel ssl session id   :
> 13:19:42 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:41029 data channel creation time: 1609244382982

I found the reason for that as well. BC's server side has session resumption diabled
by default, so by switching the provider default-behaviors are changing. Setting
the system property org.bouncycastle.jsse.server.enableSessionResumption to true,
that test case also succeeds.

But still I'm ending up with problems if the server uses BC-sockets and a
client using SunJSSE tries to connect to it:

%% No cached client session
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1592985036 bytes = { 67, 169, ... }
Session ID:  {}
Cipher Suites: [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, ..., TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA
13:15:08 [MESSAGE] SYSTEM:FTP:FTPDAEMON SecureFtpDaemon@0.0.0.0:18610 received ftp-connection from 127.0.0.1, concurrent connections=0
Extension extended_master_secret
***
main, WRITE: TLSv1.2 Handshake, length = 193
Jan 04, 2021 1:15:08 PM org.bouncycastle.jsse.provider.PropertyUtils getBooleanSystemProperty
INFORMATION: Found boolean system property [org.bouncycastle.jsse.server.enableSessionResumption]: true
Jan 04, 2021 1:15:08 PM org.bouncycastle.jsse.provider.ProvTlsServer notifyAlertRaised
INFORMATION: Server raised fatal(2) handshake_failure(40) alert: Failed to read record
org.bouncycastle.tls.TlsFatalAlert: handshake_failure(40)
        at org.bouncycastle.tls.AbstractTlsServer.getSelectedCipherSuite(Unknown Source)
        at org.bouncycastle.jsse.provider.ProvTlsServer.getSelectedCipherSuite(Unknown Source)
        at org.bouncycastle.tls.TlsServerProtocol.generateServerHello(Unknown Source)
        at org.bouncycastle.tls.TlsServerProtocol.handleHandshakeMessage(Unknown Source)

So the BC-based server doesn't accept the SunJSSE's ClientHello. To rule out
side effects of the server application this part is running in (which isn't much
in the context of a JUnit-Testcase but still), I've created a TestCase that only
contains the stuff to do TLS connections and resumptions. It's attached to this
mail and also fails if a SunJSSE provider based SSL-socket is trying to connect
to a BCJSSE based server socket.

So I ses three issues here (both I can create on GitHub but I like to get some
feedback first if this doesn't end up as as PEBKAC):

  - There seem to be a dependency on the update-level of Java 1.8 somewhere.
    The minimum update level should be mentioned somewhere (if it's already,
    I haven't found it) or - even better - checked by the provider and if
    the requirement isn't met, an exception should be raised.
  - When using the JSSE-API, changing the provider from SunJSSE to BCJSSE
    changes the server's behavior on TLS session resumption. With SunJSSE
    session resumption is enabled, with BCJSSE it's disabled. IMHO, behaviors
    shouldn't change on that level
  - A SunJSSE SSLSocket on the client side and a BCJSSE SSLSocket on the server
    side are incompatible with TLSv1.2 and TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
    (not that I've searched for this combination, it's the one being agreed on
    during my tests). I haven't checked other ciphers and switching to one of
    them as a workaround wouldn't help because in real life I have no control
    over the client.

BTW: I've created the test to make it quite easy to test different combinations
of TLS protocol, used providers and ciphers. If you see it as useful for the
test suite in BouncyCastle, feel free to add it there (it's using Java 8 features,
though).


Thanks and cheers, Lothar

[Lothar Kimmeringer-4]

Hi,

and another Ingrid ;-)

Am 04.01.2021 um 13:39 schrieb Lothar Kimmeringer:
> But still I'm ending up with problems if the server uses BC-sockets and a
> client using SunJSSE tries to connect to it:

This is also a problem, that goes away if a newer version of Java 1.8 is used.
As mentioned before, I've used _181 on Windows, I've downloaded _275 and
the test goes through. There are differences in ClientHello between these
two version:

_181:

Transport Layer Security
     TLSv1.2 Record Layer: Handshake Protocol: Client Hello
         Content Type: Handshake (22)
         Version: TLS 1.2 (0x0303)
         Length: 111
         Handshake Protocol: Client Hello
             Handshake Type: Client Hello (1)
             Length: 107
             Version: TLS 1.2 (0x0303)
             Random: 5ff32dbafa05ad759ad7ee0df12a12ae0854afeacf02ed76…
             Session ID Length: 0
             Cipher Suites Length: 4
             Cipher Suites (2 suites)
             Compression Methods Length: 1
             Compression Methods (1 method)
             Extensions Length: 62
             Extension: supported_groups (len=22)
             Extension: ec_point_formats (len=2)
             Extension: signature_algorithms (len=22)
             Extension: extended_master_secret (len=0)

_275:

Transport Layer Security
     TLSv1.2 Record Layer: Handshake Protocol: Client Hello
         Content Type: Handshake (22)
         Version: TLS 1.2 (0x0303)
         Length: 154
         Handshake Protocol: Client Hello
             Handshake Type: Client Hello (1)
             Length: 150
             Version: TLS 1.2 (0x0303)
             Random: 52f09ad3cac72f3034238e142c1dc73fdd19f2de42900fce…
             Session ID Length: 0
             Cipher Suites Length: 4
             Cipher Suites (2 suites)
             Compression Methods Length: 1
             Compression Methods (1 method)
             Extensions Length: 105
             Extension: supported_groups (len=8)
             Extension: ec_point_formats (len=2)
             Extension: signature_algorithms (len=34)
             Extension: signature_algorithms_cert (len=34)
             Extension: extended_master_secret (len=0)
             Extension: supported_versions (len=3)

I'm guessing that the missing extension supported_version is the reason
for the failing handshake. Can we consider this a bug? Shall I open
an issue for this on GitHub?


Thanks and cheers, Lothar

[Lothar Kimmeringer-4]

Hi again,

more tests lead to another Ingrid ;-)

I've created yet another test, where I manually create a ClientHello where
I remove and set extensions to see when the server raises an Alert instead
of answering with a ServerHello.

Reason is not the missing extension supported_versions but the list of
signature algorithms. This list is longer with newer versions of Java.
If BCJSSE is the client and SunJSSE is server, the signature algorithm
rsa_pkcs1_sha256 (0x0401) is used, so it's offered by the BC-client but
seems to be ignored by the BC-server ending in the state that no suitable
cipher can be found.

Is that algorithm considered insecure or why is the server ignoring it
while it seems to be perfectly OK for the client to be used?


Thanks and cheers, Lothar