This can be a root issue for ongoing efforts for Jetty to support JPMS features in Java.

Jetty has had OSGi support for nigh on a decade and it is not something we can simply abandon in lieu of java adding on a module management system. While that community has made strides in integrating well with the JPMS mechanics at the point of this thread creation it has largely been up in the air as to what that should mean for the Jetty project itself.

Jetty 9.4.x supports the usage of Java 9 with the module system disabled and it remains to be seen what Jetty 10 will do with it. Lets have some discussion on this thread about that and maybe start getting this broken down a bit.

@guw We would welcome your input here as well as a historical user of jetty and osgi, your plans regarding jpms and what you might expect from jetty regarding it.

We also have @PashaTurok with #2189 as the first user requesting this support and there is some info in that issue.

Additionally, to set expectations this is not something that will happen overnight, especially since it seems that it will involve a fairly substantial review of the overall artifact layout of Jetty.

Hi,

it was suggested to me by @joakime to open a new issue for #272 as it still occurs on 9.4.8 - with a slight tendency to occur more often now (which might be just bad luck on our end).

Unfortunately, I can't say anything new about the issue. It still appears to be random (regardless of load for example) that threads end up in WAITING state and only a server restart helps to solve the issue.

"clientOutboundChannel-23" #144 prio=5 os_prio=0 tid=0x00007fceb0001000 nid=0x28f7 waiting on condition [0x00007fce59acc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006ca773c88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:210)
        at org.eclipse.jetty.websocket.common.BlockingWriteCallback$WriteBlocker.block(BlockingWriteCallback.java:90)
        at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:107)
        at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:393)
        at org.springframework.web.socket.adapter.jetty.JettyWebSocketSession.sendTextMessage(JettyWebSocketSession.java:273)
        at org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:101)
        at org.springframework.web.socket.handler.ConcurrentWebSocketSessionDecorator.tryFlushMessageBuffer(ConcurrentWebSocketSessionDecorator.java:132)
        at org.springframework.web.socket.handler.ConcurrentWebSocketSessionDecorator.sendMessage(ConcurrentWebSocketSessionDecorator.java:104)
        at org.springframework.web.socket.messaging.StompSubProtocolHandler.sendToClient(StompSubProtocolHandler.java:470)
        at org.springframework.web.socket.messaging.StompSubProtocolHandler.handleMessageToClient(StompSubProtocolHandler.java:457)
        at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:338)
        at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:135)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

As this is affecting our production servers, I'd appreciate if this is investigated again. I'd also appreciate any workaround that doesn't suggest a server restart.

Cheers, Christoph

I am running embedded jetty 9.4.8.v20171121. My app runs fine for a period of time, usually about 2 hours, with very little CPU usage. Then, 2 threads, both named WebSocketContainer@1861866092-, start to consume 100% CPU when idle. A stack trace at this point yields the following:

 Thread id32:WebSocketContainer@1861866092-32     RUNNABLE     55.84375s
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
 sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
 sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
 sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
 sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
 org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
 org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
 org.eclipse.jetty.io.ManagedSelector$$Lambda$23/663060787.run(Unknown Source)
 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
 org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
 java.lang.Thread.run(Thread.java:748)

This seems very similar to: https://github.com/eclipse/jetty.project/issues/1446

but I am running a newer version of jetty.

I can only get this problem to occur on servers running Windows 2012R2. I am running java jdk1.8.0_162 although I have also seen the problem with jdk1.8.0_131.

Does anyone have any ideas what could be causing this?

Frequency of issue - Intermittent but frequent (50% success rate) Below is what we see

-- Jetty receives the request (version - 9.3.14.v20161028) -- Gets the data -- Pass the data (chunked response ) -- Connection is reset by jetty (per wireshark) after passing first chunk and before receiving the ack -- Below exception is seen

2018-08-29 08:27:12.453 GMT+0000 <@> ERROR <@> [:qtp1586600255-18] <@> <@> <@> <@> <@> <@> <@> Aug 29, 2018 9:27:12 AM org.glassfish.jersey.message.internal.OutboundMessageContext close FINE: Closed org.eclipse.jetty.io.EofException: Closed at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:437) at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:325) at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:167) at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:307) at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:261) at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:276) at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:877) at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:412) at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:784) at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:837) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:745)

-- It's not clear under what circumstances the connection is getting closed.

-- Jetty is listening on port 9001 -- Web server does receive the half chunk

Working wireshark

Non working

migrated from Bugzilla #484698 status NEW severity normal in component websocket for 9.3.x Reported in version 9.3.5 on platform PC Assigned to: Joakim Erdfelt

On 2015-12-18 10:48:14 -0500, Ben Tatham wrote:

My application hangs in the following state. This is usually triggered by some underlying network change (like ip address change of the host running jetty), but it might happen in other cases too.

java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Native Method)

• parking to wait for <1edddee> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:210) at org.eclipse.jetty.websocket.common.BlockingWriteCallback$WriteBlocker.block(BlockingWriteCallback.java:82) at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:107) at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendBytes(WebSocketRemoteEndpoint.java:252)

Based on the code, it looks like the m_complete.await() call should be signalled on close, failure, or success, but never is.

Jetty version 9.4.33.v20201020

Java version 1.8

OS type/version Redhat 7.4

Description I am seeing intermittent occurrences of connection loss from the server It is not often but we haven't seen this issue when using the 9.2.x version before the upgrade

From the client we see KeepAliveFailure The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

With the jetty debug logs enabled, it looks like jetty is aborting the connection due to a IOException: unconsumed input It is not always reproducible.

Wanted insight from the jetty team to help determine the cause and how to avoid Below is a snippet of the debug logs which captures the failing request Any help is appreciated

21380: Selector sun.nio.ch.EPollSelectorImpl@70e457b woken with none selected
2020-11-05 22:26:54.151:DBUG:oeji.ManagedSelector:qtp1378318626-21380: Selector sun.nio.ch.EPollSelectorImpl@70e457b woken up from select, 0/0/1 selected
2020-11-05 22:26:54.151:DBUG:oejs.HttpConnection:qtp1378318626-16923: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=FI,flush=-,to=0/30000}{io=0/1,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0} onFillable exit HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0} null
2020-11-05 22:26:54.151:DBUG:oeji.ManagedSelector:qtp1378318626-21380: Selector sun.nio.ch.EPollSelectorImpl@70e457b processing 0 keys, 1 updates
2020-11-05 22:26:54.151:DBUG:oeji.ManagedSelector:qtp1378318626-21380: updateable 1
2020-11-05 22:26:54.151:DBUG:oejut.ReservedThreadExecutor:qtp1378318626-16923: ReservedThreadExecutor@629db94d{s=3/20,p=0}@5ddf9d34 waiting
2020-11-05 22:26:54.151:DBUG:oeji.ManagedSelector:qtp1378318626-21380: update org.eclipse.jetty.io.ChannelEndPoint$$Lambda$updateKeyAction$637863116/0x0000000000026032@eaa5af3
2020-11-05 22:26:54.151:DBUG:oeji.ChannelEndPoint:qtp1378318626-21380: Key interests updated 0 -> 1 on SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=FI,flush=-,to=0/30000}{io=1/1,kio=1,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0}
2020-11-05 22:26:54.151:DBUG:oeji.ManagedSelector:qtp1378318626-21380: updates 0
2020-11-05 22:26:54.151:DBUG:oeji.ManagedSelector:qtp1378318626-21380: Selector sun.nio.ch.EPollSelectorImpl@70e457b waiting with 1 keys
2020-11-05 22:26:54.172:DBUG:oeji.ManagedSelector:qtp1378318626-21380: Selector sun.nio.ch.EPollSelectorImpl@70e457b woken up from select, 1/1/1 selected
2020-11-05 22:26:54.172:DBUG:oeji.ManagedSelector:qtp1378318626-21380: Selector sun.nio.ch.EPollSelectorImpl@70e457b processing 1 keys, 0 updates
2020-11-05 22:26:54.172:DBUG:oeji.ManagedSelector:qtp1378318626-21380: selected 1 sun.nio.ch.SelectionKeyImpl@749cb8f1 SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=FI,flush=-,to=21/30000}{io=1/1,kio=1,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0} 
2020-11-05 22:26:54.172:DBUG:oeji.ChannelEndPoint:qtp1378318626-21380: onSelected 1->0 r=true w=false for SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=FI,flush=-,to=21/30000}{io=1/0,kio=1,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0}
2020-11-05 22:26:54.172:DBUG:oeji.ChannelEndPoint:qtp1378318626-21380: task CEP:SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=FI,flush=-,to=21/30000}{io=1/0,kio=1,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
2020-11-05 22:26:54.172:DBUG:oejut.ReservedThreadExecutor:qtp1378318626-21380: ReservedThreadExecutor@629db94d{s=3/20,p=0} tryExecute EatWhatYouKill@3e0218ef/SelectorProducer@3e0218eb/PRODUCING/p=false/QueuedThreadPool[qtp1378318626]@52277922{STARTED,8<=23<=200,i=4,r=20,q=0}[ReservedThreadExecutor@629db94d{s=3/20,p=0}][pc=235,pic=0,pec=6,epc=775]@2020-11-05T22:26:54.172996062Z
2020-11-05 22:26:54.173:DBUG:oejut.ReservedThreadExecutor:qtp1378318626-21380: ReservedThreadExecutor@629db94d{s=2/20,p=0}@5ddf9d34 offer EatWhatYouKill@3e0218ef/SelectorProducer@3e0218eb/PRODUCING/p=false/QueuedThreadPool[qtp1378318626]@52277922{STARTED,8<=23<=200,i=4,r=20,q=0}[ReservedThreadExecutor@629db94d{s=2/20,p=0}][pc=235,pic=0,pec=6,epc=775]@2020-11-05T22:26:54.173049559Z
2020-11-05 22:26:54.173:DBUG:oejut.ReservedThreadExecutor:qtp1378318626-16923: ReservedThreadExecutor@629db94d{s=2/20,p=0}@5ddf9d34 task=EatWhatYouKill@3e0218ef/SelectorProducer@3e0218eb/IDLE/p=true/QueuedThreadPool[qtp1378318626]@52277922{STARTED,8<=23<=200,i=4,r=20,q=0}[ReservedThreadExecutor@629db94d{s=2/20,p=0}][pc=235,pic=0,pec=6,epc=775]@2020-11-05T22:26:54.173155782Z
2020-11-05 22:26:54.173:DBUG:oejuts.EatWhatYouKill:qtp1378318626-21380: EatWhatYouKill@3e0218ef/SelectorProducer@3e0218eb/IDLE/p=true/QueuedThreadPool[qtp1378318626]@52277922{STARTED,8<=23<=200,i=4,r=20,q=0}[ReservedThreadExecutor@629db94d{s=2/20,p=0}][pc=235,pic=0,pec=6,epc=775]@2020-11-05T22:26:54.173103891Z m=EXECUTE_PRODUCE_CONSUME t=CEP:SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=FI,flush=-,to=22/30000}{io=1/0,kio=1,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING/BLOCKING
2020-11-05 22:26:54.173:DBUG:oejuts.EatWhatYouKill:qtp1378318626-16923: EatWhatYouKill@3e0218ef/SelectorProducer@3e0218eb/IDLE/p=true/QueuedThreadPool[qtp1378318626]@52277922{STARTED,8<=23<=200,i=4,r=20,q=0}[ReservedThreadExecutor@629db94d{s=2/20,p=0}][pc=235,pic=0,pec=6,epc=776]@2020-11-05T22:26:54.17321245Z tryProduce true
2020-11-05 22:26:54.173:DBUG:oeji.ManagedSelector:qtp1378318626-16923: updateable 0
2020-11-05 22:26:54.173:DBUG:oeji.ManagedSelector:qtp1378318626-16923: updates 0
2020-11-05 22:26:54.173:DBUG:oeji.FillInterest:qtp1378318626-21380: fillable FillInterest@eaa5afe{AC.ReadCB@eaa5362{HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=FI,flush=-,to=22/30000}{io=1/0,kio=1,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0}}}
2020-11-05 22:26:54.173:DBUG:oeji.ChannelEndPoint:qtp1378318626-16923: Key interests updated 1 -> 0 on SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=FI,flush=-,to=22/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0}
2020-11-05 22:26:54.173:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=22/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0} onFillable enter HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0} null
2020-11-05 22:26:54.173:DBUG:oeji.ManagedSelector:qtp1378318626-16923: Selector sun.nio.ch.EPollSelectorImpl@70e457b waiting with 1 keys
2020-11-05 22:26:54.173:DBUG:oeji.ChannelEndPoint:qtp1378318626-21380: filled 8192 HeapByteBuffer@6b49cc31[p=0,l=8192,c=8192,r=8192]={<<<POST /application/webservices...1:2187630],\n[Hierarchie>>>}
2020-11-05 22:26:54.173:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0} filled 8192 HeapByteBuffer@6b49cc31[p=0,l=8192,c=8192,r=8192]={<<<POST /application/webservices...1:2187630],\n[Hierarchie>>>}
2020-11-05 22:26:54.173:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=START,0 of -1},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=4,c=false/false,a=IDLE,uri=null,age=0} parse HeapByteBuffer@6b49cc31[p=0,l=8192,c=8192,r=8192]={<<<POST /application/webservices...1:2187630],\n[Hierarchie>>>} {}
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: parseNext s=START HeapByteBuffer@6b49cc31[p=0,l=8192,c=8192,r=8192]={<<<POST /application/webservices...1:2187630],\n[Hierarchie>>>}
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: START --> SPACE1
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: SPACE1 --> URI
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: URI --> SPACE2
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: SPACE2 --> REQUEST_VERSION
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: REQUEST_VERSION --> HEADER
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.36627) --> IN_VALUE
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.36627) --> FIELD
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:Content-Type: text/xml; charset=UTF-8 --> IN_VALUE
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:Content-Type: text/xml; charset=UTF-8 --> FIELD
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER: --> IN_NAME
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:SOAPAction --> VALUE
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:SOAPAction --> IN_VALUE
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:SOAPAction --> FIELD
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:Host: hostnameA:8082 --> IN_VALUE
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:Host: hostnameA:8082 --> FIELD
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:Content-Length --> VALUE
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:Content-Length --> IN_VALUE
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:Content-Length --> FIELD
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:Accept-Encoding: gzip --> IN_VALUE
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER:Accept-Encoding: gzip --> FIELD
2020-11-05 22:26:54.173:DBUG:oejh.HttpParser:qtp1378318626-21380: HEADER --> CONTENT
2020-11-05 22:26:54.173:DBUG:oejs.HttpChannel:qtp1378318626-21380: REQUEST for //hostnameA:8082/application/webservices/Queries on HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=5,c=false/false,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=0}
POST //hostnameA:8082/application/webservices/Queries HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.36627)
Content-Type: text/xml; charset=UTF-8
SOAPAction: ""
Host: hostnameA:8082
Content-Length: 14781
Accept-Encoding: gzip


2020-11-05 22:26:54.173:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,0 of 14781},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=5,c=false/false,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=0} parsed true HttpParser{s=CONTENT,0 of 14781}
2020-11-05 22:26:54.173:DBUG:oejs.HttpChannel:qtp1378318626-21380: handle //hostnameA:8082/application/webservices/Queries HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=5,c=false/false,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=0} 
2020-11-05 22:26:54.173:DBUG:oejs.HttpChannelState:qtp1378318626-21380: handling HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0}
2020-11-05 22:26:54.173:DBUG:oejs.HttpChannel:qtp1378318626-21380: action DISPATCH HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=5,c=false/false,a=HANDLING,uri=//hostnameA:8082/application/webservices/Queries,age=0}
2020-11-05 22:26:54.173:DBUG:oejs.Server:qtp1378318626-21380: REQUEST POST /application/webservices/Queries on HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=5,c=false/false,a=HANDLING,uri=//hostnameA:8082/application/webservices/Queries,age=0}
2020-11-05 22:26:54.173:DBUG:oejsh.ContextHandler:qtp1378318626-21380: scope ||/application/webservices/Queries @ o.e.j.w.WebAppContext@52262d08{/application,file:///home/monjuu-g/dev/application-app/temp/jetty-0_0_0_0-8082-application-app-1_2_1-20201023_205439_war-_application-any-1718594855883126077/webapp/,AVAILABLE}{/home/monjuu-g/dev/application-app/lib/application-app-1.2.1-20201023_205439.war}
2020-11-05 22:26:54.173:DBUG:oejsh.ContextHandler:qtp1378318626-21380: context=/application||/webservices/Queries @ o.e.j.w.WebAppContext@52262d08{/application,file:///home/monjuu-g/dev/application-app/temp/jetty-0_0_0_0-8082-application-app-1_2_1-20201023_205439_war-_application-any-1718594855883126077/webapp/,AVAILABLE}{/home/monjuu-g/dev/application-app/lib/application-app-1.2.1-20201023_205439.war}
2020-11-05 22:26:54.173:DBUG:oejs.session:qtp1378318626-21380: Entering scope org.eclipse.jetty.server.session.SessionHandler1654434902==dftMaxIdleSec=1800, dispatch=REQUEST asyncstarted=false
2020-11-05 22:26:54.173:DBUG:oejs.session:qtp1378318626-21380: sessionHandler=org.eclipse.jetty.server.session.SessionHandler1654434902==dftMaxIdleSec=1800 session=
2020-11-05 22:26:54.174:DBUG:oejs.ServletHandler:qtp1378318626-21380: servlet /application||/webservices/Queries -> springDispatcherServlet@abe94f11==org.springframework.web.servlet.DispatcherServlet,jsp=null,order=1,inst=true,async=false
2020-11-05 22:26:54.174:DBUG:oejs.ServletHandler:qtp1378318626-21380: chain=Chain@4ed594f6(springSecurityFilterChain@5d5577f3==org.springframework.web.filter.DelegatingFilterProxy,inst=true,async=false)->ChainEnd@4ed594f4(springDispatcherServlet@abe94f11==org.springframework.web.servlet.DispatcherServlet,jsp=null,order=1,inst=true,async=false)
2020-11-05 22:26:54.174:DBUG:oejs.session:qtp1378318626-21380: Creating new session id=node0val0bgye9f3z1ex6wtbo2fpo010270
2020-11-05 22:26:54.174:DBUG:oejs.session:qtp1378318626-21380: Session node0val0bgye9f3z1ex6wtbo2fpo010270 in use, stopping timer, active requests=1
2020-11-05 22:26:54.174:DBUG:oejs.session:qtp1378318626-21380: Cancelled timer for session node0val0bgye9f3z1ex6wtbo2fpo010270
2020-11-05 22:26:54.174:DBUG:oejs.Request:qtp1378318626-21380: Request Request(POST //hostnameA:8082/application/webservices/Queries)@eaa53e6 entering session=Session@2677f78e{id=node0val0bgye9f3z1ex6wtbo2fpo010270,x=node0val0bgye9f3z1ex6wtbo2fpo010270.node0,req=1,res=true}
2020-11-05 22:26:54.174:DBUG:oejh.HttpCookie:qtp1378318626-21380: No default value for SameSite
2020-11-05 22:26:54.174:DBUG:oejs.HttpChannelState:qtp1378318626-21380: sendError HttpChannelState@eaa53ec{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0}
2020-11-05 22:26:54.174:DBUG:oejs.session:qtp1378318626-21380: Leaving scope org.eclipse.jetty.server.session.SessionHandler1654434902==dftMaxIdleSec=1800 dispatch=REQUEST, async=false, session=Session@2677f78e{id=node0val0bgye9f3z1ex6wtbo2fpo010270,x=node0val0bgye9f3z1ex6wtbo2fpo010270.node0,req=1,res=true}, oldsession=, oldsessionhandler=
2020-11-05 22:26:54.174:DBUG:oejs.Server:qtp1378318626-21380: handled=true async=false committed=true on HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=true i=true al=0},r=5,c=false/false,a=HANDLING,uri=//hostnameA:8082/application/webservices/Queries,age=1}
2020-11-05 22:26:54.174:DBUG:oejs.HttpChannelState:qtp1378318626-21380: unhandle HttpChannelState@eaa53ec{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=true i=true al=0}
2020-11-05 22:26:54.174:DBUG:oejs.HttpChannelState:qtp1378318626-21380: nextAction(false) SEND_ERROR HttpChannelState@eaa53ec{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=false al=0}
2020-11-05 22:26:54.174:DBUG:oejs.HttpChannel:qtp1378318626-21380: action SEND_ERROR HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=false al=0},r=5,c=false/false,a=HANDLING,uri=//hostnameA:8082/application/webservices/Queries,age=1}
2020-11-05 22:26:54.174:DBUG:oejs.ErrorPageErrorHandler:qtp1378318626-21380: getErrorPage(POST /application/webservices/Queries) => error_page=null (from global default)
2020-11-05 22:26:54.174:DBUG:oejs.HttpChannelState:qtp1378318626-21380: completing HttpChannelState@eaa53ec{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=false al=0}
2020-11-05 22:26:54.174:DBUG:oejs.HttpChannel:qtp1378318626-21380: sendResponse info= content=DirectByteBuffer@fbcfc13[p=0,l=496,c=32768,r=496]={<<<<html>\n<head>\n<meta http-...<hr/>\n\n</body>\n</html>\n>>>evelPerHi...alue xs} complete=true committing=true callback=org.eclipse.jetty.util.Callback$3@267627b7
2020-11-05 22:26:54.174:DBUG:oejs.HttpChannel:qtp1378318626-21380: COMMIT for /application/webservices/Queries on HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=HANDLING rs=COMPLETING os=COMMITTED is=IDLE awp=false se=false i=false al=0},r=5,c=false/false,a=HANDLING,uri=//hostnameA:8082/application/webservices/Queries,age=1}
401  HTTP/1.1
Set-Cookie: JSESSIONID=node0val0bgye9f3z1ex6wtbo2fpo010270.node0; Path=/application
WWW-Authenticate: Basic realm="Realm"
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: DENY
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1


2020-11-05 22:26:54.174:DBUG:oejs.Request:qtp1378318626-21380: Response Request[POST //hostnameA:8082/application/webservices/Queries]@eaa53e6 committing for session Session@2677f78e{id=node0val0bgye9f3z1ex6wtbo2fpo010270,x=node0val0bgye9f3z1ex6wtbo2fpo010270.node0,req=1,res=true}
2020-11-05 22:26:54.174:DBUG:oejs.HttpConnection:qtp1378318626-21380: generate: NEED_HEADER for org.eclipse.jetty.server.HttpConnection$SendCallback@eaa5394[PROCESSING][i=HTTP/1.1{s=401,h=7,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$SendCallback@26763053] (null,[p=0,l=496,c=32768,r=496],true)@START
2020-11-05 22:26:54.174:DBUG:oejh.HttpGenerator:qtp1378318626-21380: generateHeaders HTTP/1.1{s=401,h=7,cl=-1} last=true content=DirectByteBuffer@fbcfc13[p=0,l=496,c=32768,r=496]={<<<<html>\n<head>\n<meta http-...<hr/>\n\n</body>\n</html>\n>>>evelPerHi...alue xs}
2020-11-05 22:26:54.174:DBUG:oejh.HttpGenerator:qtp1378318626-21380: Set-Cookie: JSESSIONID=node0val0bgye9f3z1ex6wtbo2fpo010270.node0; Path=/application<|WWW-Authenticate: Basic realm="Realm"<|X-Content-Type-Options: nosniff<|X-XSS-Protection: 1; mode=block<|X-Frame-Options: DENY<|Cache-Control: must-revalidate,no-cache,no-store<|Content-Type: text/html;charset=iso-8859-1<|<|
2020-11-05 22:26:54.174:DBUG:oejh.HttpGenerator:qtp1378318626-21380: CONTENT_LENGTH
2020-11-05 22:26:54.174:DBUG:oejs.HttpConnection:qtp1378318626-21380: generate: FLUSH for org.eclipse.jetty.server.HttpConnection$SendCallback@eaa5394[PROCESSING][i=HTTP/1.1{s=401,h=7,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$SendCallback@26763053] ([p=0,l=386,c=8192,r=386],[p=0,l=496,c=32768,r=496],true)@COMPLETING
2020-11-05 22:26:54.174:DBUG:oeji.WriteFlusher:qtp1378318626-21380: write: WriteFlusher@eaa5af5{IDLE}->null [HeapByteBuffer@711c6a74[p=0,l=386,c=8192,r=386]={<<<HTTP/1.1 401 Unauthorized...y(9.4.33.v20201020)\r\n\r\n>>>\nl versio...; EMERG},DirectByteBuffer@fbcfc13[p=0,l=496,c=32768,r=496]={<<<<html>\n<head>\n<meta http-...<hr/>\n\n</body>\n</html>\n>>>evelPerHi...alue xs}]
2020-11-05 22:26:54.174:DBUG:oeji.WriteFlusher:qtp1378318626-21380: update WriteFlusher@eaa5af5{WRITING}->null:IDLE-->WRITING
2020-11-05 22:26:54.174:DBUG:oeji.ChannelEndPoint:qtp1378318626-21380: flushed 882 SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=W,to=1/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,0 of 14781},g=HttpGenerator@eaa53b5{s=COMPLETING}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=HANDLING rs=COMPLETING os=COMMITTED is=IDLE awp=false se=false i=false al=0},r=5,c=false/false,a=HANDLING,uri=//hostnameA:8082/application/webservices/Queries,age=1}
2020-11-05 22:26:54.174:DBUG:oeji.WriteFlusher:qtp1378318626-21380: Flushed=true written=882 remaining=0 WriteFlusher@eaa5af5{WRITING}->null
2020-11-05 22:26:54.174:DBUG:oeji.WriteFlusher:qtp1378318626-21380: update WriteFlusher@eaa5af5{IDLE}->null:WRITING-->IDLE
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: generate: DONE for org.eclipse.jetty.server.HttpConnection$SendCallback@eaa5394[PROCESSING][i=HTTP/1.1{s=401,h=7,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$SendCallback@26763053] ([p=386,l=386,c=8192,r=0],[p=496,l=496,c=32768,r=0],true)@END
2020-11-05 22:26:54.175:DBUG:oejs.HttpChannelState:qtp1378318626-21380: completed HttpChannelState@eaa53ec{s=HANDLING rs=COMPLETING os=COMPLETED is=IDLE awp=false se=false i=false al=0}
2020-11-05 22:26:54.175:DBUG:oejs.HttpChannelState:qtp1378318626-21380: unhandle HttpChannelState@eaa53ec{s=HANDLING rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0}
2020-11-05 22:26:54.175:DBUG:oejs.HttpChannelState:qtp1378318626-21380: nextAction(false) TERMINATED HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0}
2020-11-05 22:26:54.175:DBUG:oejs.HttpChannel:qtp1378318626-21380: action TERMINATED HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2}
2020-11-05 22:26:54.175:DBUG:oejs.HttpChannel:qtp1378318626-21380: onCompleted for /application/webservices/Queries written=496
2020-11-05 22:26:54.175:DBUG:oejs.Request:qtp1378318626-21380: Request Request[POST //hostnameA:8082/application/webservices/Queries]@eaa53e6 leaving session Session@2677f78e{id=node0val0bgye9f3z1ex6wtbo2fpo010270,x=node0val0bgye9f3z1ex6wtbo2fpo010270.node0,req=1,res=true}
2020-11-05 22:26:54.175:DBUG:oejs.session:qtp1378318626-21380: Complete called with session Session@2677f78e{id=node0val0bgye9f3z1ex6wtbo2fpo010270,x=node0val0bgye9f3z1ex6wtbo2fpo010270.node0,req=1,res=true}
2020-11-05 22:26:54.175:DBUG:oejs.session:qtp1378318626-21380: Session node0val0bgye9f3z1ex6wtbo2fpo010270 complete, active requests=0
2020-11-05 22:26:54.175:DBUG:oejs.session:qtp1378318626-21380: Session node0val0bgye9f3z1ex6wtbo2fpo010270 no eviction
2020-11-05 22:26:54.175:DBUG:oejs.session:qtp1378318626-21380: (Re)starting timer for session node0val0bgye9f3z1ex6wtbo2fpo010270 at 1800000ms
2020-11-05 22:26:54.175:DBUG:oeji.CyclicTimeout:qtp1378318626-21380: Installed timeout in 1800000 ms, waking up in 1800000 ms
2020-11-05 22:26:54.175:DBUG:oejs.session:qtp1378318626-21380: Store: id=node0val0bgye9f3z1ex6wtbo2fpo010270, mdirty=true, dirty=true, lsave=0, period=0, elapsed=1604615214175
2020-11-05 22:26:54.175:DBUG:oejs.session:qtp1378318626-21380: Non passivating SessionDataStore, session in SessionCache only id=node0val0bgye9f3z1ex6wtbo2fpo010270
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,0 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} parse HeapByteBuffer@6b49cc31[p=279,l=8192,c=8192,r=7913]={POST /pro...zip\r\n\r\n<<<<?xml version="1.0" encod...1:2187630],\n[Hierarchie>>>} {}
2020-11-05 22:26:54.175:DBUG:oejh.HttpParser:qtp1378318626-21380: parseNext s=CONTENT HeapByteBuffer@6b49cc31[p=279,l=8192,c=8192,r=7913]={POST /pro...zip\r\n\r\n<<<<?xml version="1.0" encod...1:2187630],\n[Hierarchie>>>}
2020-11-05 22:26:54.175:DBUG:oejs.HttpChannel:qtp1378318626-21380: onContent HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} Content@2676aaaf{HeapByteBufferR@2676aaa9[p=279,l=8192,c=8192,r=7913]={POST /pro...zip\r\n\r\n<<<<?xml version="1.0" encod...1:2187630],\n[Hierarchie>>>}}
2020-11-05 22:26:54.175:DBUG:oejs.HttpInput:qtp1378318626-21380: HttpInputOverHTTP@eaa5203[c=0,q=0,[0]=null,s=STREAM] addContent Content@2676aaaf{HeapByteBufferR@2676aaa9[p=279,l=8192,c=8192,r=7913]={POST /pro...zip\r\n\r\n<<<<?xml version="1.0" encod...1:2187630],\n[Hierarchie>>>}}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,7913 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} parsed false HttpParser{s=CONTENT,7913 of 14781}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: unconsumed input HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,7913 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: releaseRequestBuffer HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,7913 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2}
2020-11-05 22:26:54.175:DBUG:oeji.ChannelEndPoint:qtp1378318626-21380: filled 5227 HeapByteBuffer@6b49cc31[p=0,l=5227,c=8192,r=5227]={<<<s].[Global FX].[ALL].[All...NodeA>>>AllMember...rarchie}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,7913 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} filled 5227 HeapByteBuffer@6b49cc31[p=0,l=5227,c=8192,r=5227]={<<<s].[Global FX].[ALL].[All...NodeA>>>AllMember...rarchie}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,7913 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} parse HeapByteBuffer@6b49cc31[p=0,l=5227,c=8192,r=5227]={<<<s].[Global FX].[ALL].[All...NodeA>>>AllMember...rarchie} {}
2020-11-05 22:26:54.175:DBUG:oejh.HttpParser:qtp1378318626-21380: parseNext s=CONTENT HeapByteBuffer@6b49cc31[p=0,l=5227,c=8192,r=5227]={<<<s].[Global FX].[ALL].[All...NodeA>>>AllMember...rarchie}
2020-11-05 22:26:54.175:DBUG:oejs.HttpChannel:qtp1378318626-21380: onContent HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} Content@2676d903{HeapByteBufferR@2676d90d[p=0,l=5227,c=8192,r=5227]={<<<s].[Global FX].[ALL].[All...NodeA>>>AllMember...rarchie}}
2020-11-05 22:26:54.175:DBUG:oejs.HttpInput:qtp1378318626-21380: HttpInputOverHTTP@eaa5203[c=7913,q=0,[0]=null,s=STREAM] addContent Content@2676d903{HeapByteBufferR@2676d90d[p=0,l=5227,c=8192,r=5227]={<<<s].[Global FX].[ALL].[All...NodeA>>>AllMember...rarchie}}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} parsed false HttpParser{s=CONTENT,13140 of 14781}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: releaseRequestBuffer HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2}
2020-11-05 22:26:54.175:DBUG:oeji.ChannelEndPoint:qtp1378318626-21380: filled 0 HeapByteBuffer@6b49cc31[p=0,l=0,c=8192,r=0]={<<<>>>s].[Globa...rarchie}
2020-11-05 22:26:54.175:DBUG:oeji.ChannelEndPoint:qtp1378318626-21380: filled 0 HeapByteBuffer@6b49cc31[p=0,l=0,c=8192,r=0]={<<<>>>s].[Globa...rarchie}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} filled 0 HeapByteBuffer@6b49cc31[p=0,l=0,c=8192,r=0]={<<<>>>s].[Globa...rarchie}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} parse HeapByteBuffer@6b49cc31[p=0,l=0,c=8192,r=0]={<<<>>>s].[Globa...rarchie} {}
2020-11-05 22:26:54.175:DBUG:oejh.HttpParser:qtp1378318626-21380: parseNext s=CONTENT HeapByteBuffer@6b49cc31[p=0,l=0,c=8192,r=0]={<<<>>>s].[Globa...rarchie}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2} parsed false HttpParser{s=CONTENT,13140 of 14781}
2020-11-05 22:26:54.175:DBUG:oejs.HttpConnection:qtp1378318626-21380: releaseRequestBuffer HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=5,c=true/true,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=2}
2020-11-05 22:26:54.176:DBUG:oejs.HttpConnection:qtp1378318626-21380: abort HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0},r=5,c=true/false,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=3} {}
java.io.IOException: unconsumed input
	at org.eclipse.jetty.server.HttpConnection.onCompleted(HttpConnection.java:430)
	at org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:743)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:363)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.lang.Thread.run(Thread.java:748)
2020-11-05 22:26:54.176:DBUG:oeji.AbstractEndPoint:qtp1378318626-21380: close SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0},r=5,c=true/false,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=3}
2020-11-05 22:26:54.176:DBUG:oeji.AbstractEndPoint:qtp1378318626-21380: close() SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0},r=5,c=true/false,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=3}
2020-11-05 22:26:54.176:DBUG:oeji.ChannelEndPoint:qtp1378318626-21380: doClose SocketChannelEndPoint@eaa5ad1{l=/hostnameA_IP:8082,r=/clientA_IP:51419,CLOSED,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@eaa5362[p=HttpParser{s=CONTENT,13140 of 14781},g=HttpGenerator@eaa53b5{s=END}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0},r=5,c=true/false,a=IDLE,uri=//hostnameA:8082/application/webservices/Queries,age=3}
2020-11-05 22:26:54.176:DBUG:oeji.FillInterest:qtp1378318626-21380: onClose FillInterest@eaa5afe{null}
2020-11-05 22:26:54.176:DBUG:oeji.ManagedSelector:qtp1378318626-21380: Wakeup ManagedSelector@3e021b44{STARTED} id=3 keys=1 selected=0 updates=0
2020-11-05 22:26:54.176:DBUG:oejut.QueuedThreadPool:qtp1378318626-21380: queue org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint@2e7e33f1 startThread=0
2020-11-05 22:26:54.176:DBUG:oejs.HttpChannelState:qtp1378318626-21380: recycle HttpChannelState@eaa53ec{s=IDLE rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0}
2020-11-05 22:26:54.176:DBUG:oeji.ManagedSelector:qtp1378318626-16923: Selector sun.nio.ch.EPollSelectorImpl@70e457b woken with none selected
2020-11-05 22:26:54.176:DBUG:oejh.HttpParser:qtp1378318626-21380: close HttpParser{s=CONTENT,13140 of 14781}
2020-11-05 22:26:54.176:DBUG:oeji.ManagedSelector:qtp1378318626-16923: Selector sun.nio.ch.EPollSelectorImpl@70e457b woken up from select, 0/0/0 selected
2020-11-05 22:26:54.176:DBUG:oejh.HttpParser:qtp1378318626-21380: CONTENT --> CLOSE
2020-11-05 22:26:54.176:DBUG:oeji.ManagedSelector:qtp1378318626-16923: Selector sun.nio.ch.EPollSelectorImpl@70e457b processing 0 keys, 0 updates
2020-11-05 22:26:54.176:DBUG:oeji.ManagedSelector:qtp1378318626-16923: updateable 0
2020-11-05 22:26:54.176:DBUG:oeji.ManagedSelector:qtp1378318626-16923: updates 0
2020-11-05 22:26:54.176:DBUG:oejs.HttpChannel:qtp1378318626-21380: !handle TERMINATED HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=5,c=false/false,a=IDLE,uri=null,age=0}
2020-11-05 22:26:54.176:DBUG:oejut.QueuedThreadPool:qtp1378318626-20348: run org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint@2e7e33f1 in QueuedThreadPool[qtp1378318626]@52277922{STARTED,8<=23<=200,i=3,r=20,q=0}[ReservedThreadExecutor@629db94d{s=2/20,p=0}]
2020-11-05 22:26:54.176:DBUG:oeji.ManagedSelector:qtp1378318626-16923: Selector sun.nio.ch.EPollSelectorImpl@70e457b waiting with 0 keys
2020-11-05 22:26:54.176:DBUG:oejs.HttpConnection:qtp1378318626-21380: HttpConnection@eaa5362::SocketChannelEndPoint@eaa5ad1{l=0.0.0.0/0.0.0.0:8082,r=null,CLOSED,fill=-,flush=-,to=1/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@eaa5362[p=HttpParser{s=CLOSE,13140 of 14781},g=HttpGenerator@eaa53b5{s=START}]=>HttpChannelOverHttp@eaa53b2{s=HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=5,c=false/false,a=IDLE,uri=null,age=0} onFillable exit HttpChannelState@eaa53ec{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0} null
2020-11-05 22:26:54.176:DBUG:oejut.ReservedThreadExecutor:qtp1378318626-21380: ReservedThreadExecutor@629db94d{s=3/20,p=0}@5db4c856 waiting

If an HTTP client instance is used in a multithreading environment (e.g. multiple requests are sent via the same object in parallel) we get occasionally the following error:

java.io.EOFException: HttpConnectionOverHTTP@43f11127::DecryptedEndPoint@58c1c452{localhost/127.0.0.1:8080<->/127.0.0.1:50734,OPEN,fill=-,flush=P,to=1/0}

Code to reproduce can be found here: https://github.com/Andremoniy/jetty.httpclient.sslissue

The full stacktrace:

java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@43f11127::DecryptedEndPoint@58c1c452{localhost/127.0.0.1:8080<->/127.0.0.1:50734,OPEN,fill=-,flush=P,to=1/0}
	at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
	at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:683)
	at com.github.andremoniy.jetty.httpclient.sslissue.Bootstrap.lambda$main$1(Bootstrap.java:60)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: 
java.io.EOFException: HttpConnectionOverHTTP@43f11127::DecryptedEndPoint@58c1c452{localhost/127.0.0.1:8080<->/127.0.0.1:50734,OPEN,fill=-,flush=P,to=1/0}
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:338)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1552)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:209)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:147)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73)
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint$IncompleteWriteCallback.succeeded(SslConnection.java:1309)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:293)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onIncompleteFlush(SslConnection.java:1068)
	at org.eclipse.jetty.io.AbstractEndPoint$2.onIncompleteFlush(AbstractEndPoint.java:54)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:285)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380)
	at org.eclipse.jetty.client.http.HttpSenderOverHTTP$HeadersCallback.process(HttpSenderOverHTTP.java:268)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
	at org.eclipse.jetty.client.http.HttpSenderOverHTTP.sendHeaders(HttpSenderOverHTTP.java:62)
	at org.eclipse.jetty.client.HttpSender.send(HttpSender.java:214)
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.send(HttpChannelOverHTTP.java:85)
	at org.eclipse.jetty.client.HttpChannel.send(HttpChannel.java:128)
	at org.eclipse.jetty.client.HttpConnection.send(HttpConnection.java:201)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP$Delegate.send(HttpConnectionOverHTTP.java:255)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.send(HttpConnectionOverHTTP.java:122)
	at org.eclipse.jetty.client.http.HttpDestinationOverHTTP.send(HttpDestinationOverHTTP.java:38)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:346)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:304)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:294)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:269)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:246)
	at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:576)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:726)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:679)
	at com.github.andremoniy.jetty.httpclient.sslissue.Bootstrap.lambda$main$1(Bootstrap.java:60)
	at java.base/java.lang.Thread.run(Thread.java:834)

Hi guys, I tried to ask a question on StackOverflow here seems better format according to sbordet, so posting here. http://stackoverflow.com/questions/40837801/jetty-http-2-client-problems?noredirect=1#comment68912667_40837801

I have been trying different Jetty client versions for my http client and I have been hitting some problems with high load (several thousands requests per minutes). I use 2 client selectors (setSelectors), 16384 as setInputBufferSize. Same number as setRequestBufferSize, setResponseBufferSize. 2024 as setMaxConnectionsPerDestination. setConnectBlocking set to true. For response I use BufferingResponseListener with 8388608 as param in constructor (8 MB).

The problems are:

9.3.9.v20160517:

AsynchronousCloseException

2016-11-26 23:20:06,638 [asyncCallback11] WARN  http.HttpDataAccess - Exception details: 
java.nio.channels. AsynchronousCloseException: null
at org.eclipse.jetty.http2. client.http. HttpConnectionOverHTTP2.close( HttpConnectionOverHTTP2.java: 90) ~[http2-http-client-transport- 9.3.14.v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. client.http. HttpClientTransportOverHTTP2. onClose( HttpClientTransportOverHTTP2. java:160) ~[http2-http-client-transport- 9.3.14.v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. client.http. HttpClientTransportOverHTTP2$ SessionListenerPromise. onClose( HttpClientTransportOverHTTP2. java:216) ~[http2-http-client-transport- 9.3.14.v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. HTTP2Session.notifyClose( HTTP2Session.java:1061) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. HTTP2Session.onGoAway( HTTP2Session.java:424) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.BodyParser. notifyGoAway(BodyParser.java: 187) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.GoAwayBodyParser. onGoAway(GoAwayBodyParser. java:169) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.GoAwayBodyParser.parse( GoAwayBodyParser.java:139) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.Parser.parseBody( Parser.java:182) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.Parser.parse(Parser. java:110) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. HTTP2Connection$HTTP2Producer. produce(HTTP2Connection.java: 199) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.util.thread. strategy.ProduceConsume. execute(ProduceConsume.java: 69) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.http2. HTTP2Connection.onFillable( HTTP2Connection.java:110) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. HTTP2Connection$FillCallback. succeeded(HTTP2Connection. java:251) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.io. FillInterest.fillable( FillInterest.java:95) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.io.ssl. SslConnection.onFillable( SslConnection.java:202) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.io. AbstractConnection$ ReadCallback.succeeded( AbstractConnection.java:273) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.io. FillInterest.fillable( FillInterest.java:95) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.io. SelectChannelEndPoint$2.run( SelectChannelEndPoint.java:93) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. strategy. ExecuteProduceConsume. executeProduceConsume( ExecuteProduceConsume.java: 303) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. strategy. ExecuteProduceConsume. produceConsume( ExecuteProduceConsume.java: 148) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. strategy. ExecuteProduceConsume.run( ExecuteProduceConsume.java: 136) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. QueuedThreadPool.runJob( QueuedThreadPool.java:671) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. QueuedThreadPool$2.run( QueuedThreadPool.java:589) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]

9.3.10.v20160621 , 9.3.11.v20160721 - same

9.3.12.v20160915 - onComplete callback of BufferingResponseListener is called more than once.

9.3.13.v20161014 - AsynchronousCloseException, as previous versions. And in addition something new

2016-11-26 23:21:50,672 [asyncCallback10] WARN  http.HttpDataAccess -     Exception details: 
java.lang. ArrayIndexOutOfBoundsException : 16384
at org.eclipse.jetty.http2.hpack. Huffman.encode(Huffman.java: 466) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2.hpack. Huffman.encode(Huffman.java: 414) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2.hpack. HpackEncoder.encodeValue( HpackEncoder.java:370) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2.hpack. HpackEncoder.encode( HpackEncoder.java:325) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2.hpack. HpackEncoder.encode( HpackEncoder.java:165) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. generator.HeadersGenerator. generateHeaders( HeadersGenerator.java:72) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. generator.HeadersGenerator. generate(HeadersGenerator. java:56) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. generator.Generator.control( Generator.java:80) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. HTTP2Session$ControlEntry. generate(HTTP2Session.java: 1122) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. HTTP2Flusher.process( HTTP2Flusher.java:231) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.util. IteratingCallback.processing( IteratingCallback.java:241) ~[jetty-util-9.3.13.v20161014. jar:9.3.13.v20161014]
at org.eclipse.jetty.util. IteratingCallback.iterate( IteratingCallback.java:224) ~[jetty-util-9.3.13.v20161014. jar:9.3.13.v20161014]
at org.eclipse.jetty.http2. HTTP2Session.newStream( HTTP2Session.java:496) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpSenderOverHTTP2. sendHeaders( HttpSenderOverHTTP2.java:87) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpSender.send(HttpSender. java:204) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpChannelOverHTTP2.send( HttpChannelOverHTTP2.java:87) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpConnection.send( HttpConnection.java:207) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpConnectionOverHTTP2.send( HttpConnectionOverHTTP2.java: 64) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpDestinationOverHTTP2.send( HttpDestinationOverHTTP2.java: 37) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpDestinationOverHTTP2.send( HttpDestinationOverHTTP2.java: 27) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. MultiplexHttpDestination. process( MultiplexHttpDestination.java: 143) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. MultiplexHttpDestination.send( MultiplexHttpDestination.java: 72) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpDestination.send( HttpDestination.java:202) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpClient.send(HttpClient. java:540) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpRequest.send(HttpRequest. java:693) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpRequest.send(HttpRequest. java:677) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]

9.3.14.v20161028 - AsynchronousCloseException

Do you have any advice why my clients gets those GO_AWAY message leading to AsynchronousCloseException? This is the main issue. It doesn't happen all the time but still a substantial number of requests which prevents me from progressing with the production deployment.

Maxim

jetty.sh looks broken if used under systemd.

While starting jetty seems to work via sudo service jetty start, stopping it does not seem to work, via sudo service jetty stop.

Also, systemd may need additional metadata to make jetty "known" to systemd so that systemd tools work with jetty too.

We noticed a significant problem in increasing of memory usage of Jetty when we updated from Jetty 9.4.11 to Jetty 9.4.12. The benchmarking setup here is quite basic, a leader node receive update requests from client and distribute the update requests to another replica node using Http2Client which is setup as below

HTTP2Client http2client = new HTTP2Client();
transport = new HttpClientTransportOverHTTP2(http2client);
httpClient = new HttpClient(transport, sslContextFactory);

httpClient.setExecutor(new QueuedThreadPool(128, 4, 60000, new BlockingArrayQueue<>(128, 128)));
httpClient.setStrictEventOrdering(false);
httpClient.setConnectBlocking(true);
httpClient.setFollowRedirects(false);
httpClient.setMaxRequestsQueuedPerDestination(3000);
httpClient.setUserAgentField(new HttpField(HttpHeader.USER_AGENT, AGENT));
httpClient.setMaxConnectionsPerDestination(4);
httpClient.setIdleTimeout(600000);
httpClient.setConnectTimeout(60000);

Here is the benchmark result when nodes run Jetty 9.4.11

09:59:48 Time taken (total): 1397.6 sec
09:59:48 Printing Stats for replica node
09:59:48 	Time in Old Generation GC: 0.2 ms
09:59:48 	Time in Young Generation GC: 29.0 ms
09:59:48 	Time in JIT compilation: 107.1 ms
09:59:48 	Garbage Generated in Old Generation: 4226.6 MiB
09:59:48 	Garbage Generated in Young Generation: 496606.6 MiB
09:59:48 	Garbage Generated in Survivor Generation: 9870.3 MiB
09:59:48 	Peak memory usage in Old Generation: 1553.7 MiB
09:59:48 	Peak memory usage in Young Generation: 682.8 MiB
09:59:48 	Peak memory usage in Survivor Generation: 170.6 MiB
09:59:48 	Average System Load: 10.288
09:59:48 	Average CPU Time: 192.492/800
09:59:48 	Average CPU Load: 24.542
09:59:48 Printing Stats for leader node
09:59:48 	Time in Old Generation GC: 0.8 ms
09:59:48 	Time in Young Generation GC: 51.4 ms
09:59:48 	Time in JIT compilation: 83.9 ms
09:59:48 	Garbage Generated in Old Generation: 19462.2 MiB
09:59:48 	Garbage Generated in Young Generation: 802795.1 MiB
09:59:48 	Garbage Generated in Survivor Generation: 13183.9 MiB
09:59:48 	Peak memory usage in Old Generation: 1581.0 MiB
09:59:48 	Peak memory usage in Young Generation: 682.8 MiB
09:59:48 	Peak memory usage in Survivor Generation: 170.6 MiB
09:59:48 	Average System Load: 10.288
09:59:48 	Average CPU Time: 443.473/800
09:59:48 	Average CPU Load: 54.502
09:59:48 

Here is benchmark result when nodes run Jetty 9.4.12

20:11:25 Time taken (total): 2236.1 sec
20:11:25 Printing Stats for replica node
20:11:25 	Time in Old Generation GC: 0.8 ms
20:11:25 	Time in Young Generation GC: 75.9 ms
20:11:25 	Time in JIT compilation: 97.2 ms
20:11:25 	Garbage Generated in Old Generation: 14976.7 MiB
20:11:25 	Garbage Generated in Young Generation: 1067270.1 MiB
20:11:25 	Garbage Generated in Survivor Generation: 21313.5 MiB
20:11:25 	Peak memory usage in Old Generation: 1562.7 MiB
20:11:25 	Peak memory usage in Young Generation: 682.8 MiB
20:11:25 	Peak memory usage in Survivor Generation: 170.6 MiB
20:11:25 	Average System Load: 15.403
20:11:25 	Average CPU Time: 320.363/800
20:11:25 	Average CPU Load: 39.998
20:11:25 Printing Stats for leader node
20:11:25 	Time in Old Generation GC: 0.7 ms
20:11:25 	Time in Young Generation GC: 68.6 ms
20:11:25 	Time in JIT compilation: 95.7 ms
20:11:25 	Garbage Generated in Old Generation: 18060.5 MiB
20:11:25 	Garbage Generated in Young Generation: 1113447.8 MiB
20:11:25 	Garbage Generated in Survivor Generation: 20980.7 MiB
20:11:25 	Peak memory usage in Old Generation: 1562.0 MiB
20:11:25 	Peak memory usage in Young Generation: 682.8 MiB
20:11:25 	Peak memory usage in Survivor Generation: 170.6 MiB
20:11:25 	Average System Load: 15.403
20:11:25 	Average CPU Time: 310.151/800
20:11:25 	Average CPU Load: 37.968
20:11:25 

As we can see here the time for complete the benchmark increased significantly after upgrade to Jetty 9.4.12 from 1397.6s to 2236.1s. One other point that seems interested here is the memory usage on the replica side (replica only response to the leader, it never query the leader) is almost doubled.

migrated from Bugzilla #470848 status ASSIGNED severity enhancement in component http2 for 9.3.x Reported in version unspecified on platform All Assigned to: Greg Wilkins

On 2015-06-23 17:47:12 -0400, Louis Ryan wrote:

Full disclosure: I work on GRPC for Google and we have been using the ALPN/NPN bootclasspath mechanism for a while but are planning to switch to tcnative (specifically http://netty.io/wiki/forked-tomcat-native.html)

This change is motivated by a number of factors:

• Its proven hard to get client developers to properly set their bootclasspath based on their VM. It's been a pain with Maven and other harness-like environments too.
• SSLEngine has poor performance for many of the ciphers mandated by HTTP2 E.g. (https://bugzilla.redhat.com/show_bug.cgi?id=1135504)
• Our benchmarking has shown that a properly compiled pairing of tcnative and openssl outperforms SSLEngine on all ciphers.
• We need to be able to support a transparent fallback from ALPN to NPN when talking to servers that are restricted to using OpenSSL 1.0.1 or earlier (all current releases of Debian and RedHat do not yet have 1.0.2 available on stable). NPN bootclasspath is not supported for JDK1.8 and you cant have both Jetty-ALPN and Jetty-NPN installed at the same time.

Not sure if this is something you've looked into or talked about so I just wanted to make sure it was on your radar

Cheers

• Louis

On 2016-01-26 14:57:03 -0500, Louis Ryan wrote:

Just an update on work occurring in this area

netty-tcnative now statically links boringssl and will likely soon also provide options for a statically linked libressl & openssl for linux/mac/windows

https://github.com/netty/netty-tcnative/pull/89#issuecomment-174346457

This should fix its remaining installation overheads (e.g. dependency on modern shared openssl to be present in the environment)

This will also address Java7s lack of GCM ciphers (http2 required) and Java8s GCM performance issues which seem unlikely to be fixed until Java9.

There's also some ongoing discussion about doing the same with Androids JCE/JSSE provider Conscrypt which is licensed under Apache 2.

https://android.googlesource.com/platform/external/conscrypt/

On 2016-01-27 15:12:25 -0500, Jesse McConnell wrote:

Greg, circle up with me on this one!

On 2016-02-03 06:24:56 -0500, Greg Wilkins wrote:

We implemented Unix domain socket support for similar reasons. I did look at tcnative at the time but it was a little entangled.

The forks linked in this issue look a lot more usable and the drop in replacement of ssl engine should make this almost trivial to integrate with most work going into the pom and modules.

So worthwhile giving this a try in 9.4.

Jetty version(s) 12+

Enhancement Description Many other modules (server, websocket, etc.) have standardized using an internal package that is not JPMS publicly exported to hide implementation classes.

Module jetty-client should do the same.

• Fixes #9119 - uses proper context path that satisfies the root context rules of the servlet spec

Signed-off-by: Joakim Erdfelt [email protected]

Jetty version(s) 12+

Description In Jetty 11, writes from a Handler via response.getOutputStream() are blocking, but internally that blocking call is converted into a non-blocking call typically via:

try (Blocker blocker = _writeBlocker.acquire())
{
    channelWrite(content, complete, blocker);
    blocker.block();
}

where blocker.invocationType==NON_BLOCKING.

For large writes that TCP congest, this means that task SelectableChannelEndPoint._runCompleteWrite is non-blocking; in case all threads are taken, unblocking a write is possible because the task is non-blocking and will be run by the selector thread when it wakes up being write-ready.

In Jetty 12, writes from a Handler receive a Callback whose invocationType depends on the HttpStream implementation. All of them do not override getInvocationType() so they are all blocking. This means that task SelectableChannelEndPoint._runCompleteWrite is blocking; in case all threads are taken the task will be queued, rather than executed by the selector thread, therefore locking up the server as it is the task that would have unblocked a write, but there are no threads to execute it.

This is the reason of the flakyness of this test: sometimes there always is a thread that can run the completeWrite task, but in some cases there is not and the test fails.

Jetty version(s) Jetty 12

Java version/vendor (use: java -version) Java 19

OS type/version Linux

Description

Error:

Expected: an instance of java.net.ConnectException
     but: <java.io.IOException: frame_size_error/invalid_frame_length> is a java.io.IOException

Stacktrace:

java.lang.AssertionError: 
Expected: an instance of java.net.ConnectException
     but: <java.io.IOException: frame_size_error/invalid_frame_length> is a java.io.IOException
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at org.eclipse.jetty.ee9.websocket.tests.WebSocketOverHTTP2Test.testWebSocketConnectPortDoesNotExist(WebSocketOverHTTP2Test.java:256)

Output:

Running org.eclipse.jetty.ee9.websocket.tests.WebSocketOverHTTP2Test.testWebSocketConnectPortDoesNotExist()
2023-01-02 21:48:32.631:INFO :oejs.Server:main: jetty-12.0.0-SNAPSHOT; built: 2023-01-02T20:54:23.235Z; git: 3498fe4e8134b70fb709e7e603daf87e1980f116; jvm 19+36-2238
2023-01-02 21:48:32.633:INFO :oejsh.ContextHandler:main: Started oeje9n.ContextHandler$CoreContextHandler@3e05586b{ROOT,/,b=null,a=AVAILABLE,h=oeje9n.ContextHandler$CoreContextHandler$CoreToNestedHandler@35b17c06{STARTED}}
2023-01-02 21:48:32.633:INFO :oejs.AbstractConnector:main: Started ServerConnector@343e225a{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:39111}
2023-01-02 21:48:32.643:INFO :oejus.SslContextFactory:main: x509=X509@7e62cfa3(1,h=[localhost],a=[],w=[]) for Server@46e190ed[provider=null,keyStore=file:///home/jenkins/agent/workspace/jetty.project_PR-9118/jetty-ee9/jetty-ee9-websocket/jetty-ee9-websocket-jetty-tests/target/test-classes/keystore.p12,trustStore=null]
2023-01-02 21:48:32.649:INFO :oejs.AbstractConnector:main: Started ServerConnector@36f7d7b{SSL, (ssl, alpn, http/1.1, h2)}{0.0.0.0:46339}
2023-01-02 21:48:32.649:INFO :oejs.Server:main: Started oejs.Server@17dad32f{STARTING}[12.0.0-SNAPSHOT,sto=0] @12934ms
2023-01-02 21:48:32.651:WARN :oejusS.config:main: Trusting all certificates configured for Client@37348491[provider=null,keyStore=null,trustStore=null]
2023-01-02 21:48:32.651:WARN :oejusS.config:main: No Client EndPointIdentificationAlgorithm configured for Client@37348491[provider=null,keyStore=null,trustStore=null]
2023-01-02 21:48:32.657:INFO :oejs.Server:main: Stopped oejs.Server@17dad32f{STOPPING}[12.0.0-SNAPSHOT,sto=0]
2023-01-02 21:48:32.657:INFO :oejs.AbstractConnector:main: Stopped ServerConnector@343e225a{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:0}
2023-01-02 21:48:32.658:INFO :oejs.AbstractConnector:main: Stopped ServerConnector@36f7d7b{SSL, (ssl, alpn, http/1.1, h2)}{0.0.0.0:0}
2023-01-02 21:48:32.658:INFO :oejewc.WebSocketClient:main: Shutdown WebSocketClient@cca2a347[coreClient=oejwcc.WebSocketCoreClient@4dac121d{STARTED},openSessions.size=0]

Jetty version(s) 10.0.13

Java version/vendor (use: java -version) 11.0.7

OS type/version Linux/Fedora 37

Description I work on Pax Web project and it unifies 3 embedded web servers (Jetty, Tomcat and Undertow) in OSGi runtime. I try hard to make the tests consistent and ensure similar behavior between the runtimes.

According to Chapter 9.4.2 Forwarded Request Parameters of the Servlets 4 specification:

The following request attributes must be set: javax.servlet.forward.mapping javax.servlet.forward.request_uri javax.servlet.forward.context_path javax.servlet.forward.servlet_path javax.servlet.forward.path_info javax.servlet.forward.query_string The values of these attributes must be equal to the return values of the HttpServletRequest methods getRequestURI, getContextPath, getServletPath, getPathInfo, getQueryString respectively, invoked on the request object passed to the first servlet object in the call chain that received the request from the client.

When using Jetty 9.4.50, I see correct (in org.eclipse.jetty.server.Dispatcher#forward()):

final String old_context_path = baseRequest.getContextPath();
...
ForwardAttributes attr = new ForwardAttributes(old_attr);
...
attr._contextPath = old_context_path;

However in Jetty 10, it's:

if (old_attr.getAttribute(FORWARD_REQUEST_URI) == null)
    baseRequest.setAttributes(new ForwardAttributes(old_attr,
        old_uri.getPath(),
        old_context == null ? null : old_context.getContextHandler().getContextPathEncoded(),
        baseRequest.getPathInContext(),
        source_mapping,
        old_uri.getQuery()));

And the forwarded context path is "/" instead of "" (empty string).

How to reproduce?

I have a "/gateway/*" mapped servlet with:

protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
	String what = req.getParameter("what");
	String where = req.getParameter("where");
	switch (what) {
		case "redirect":
			resp.sendRedirect(where);
			return;
		case "forward":
			// we can't send anything when forwarding
			req.getRequestDispatcher(where).forward(req, resp);
			return;
		case "include":
			resp.getWriter().print(">>>");
			req.getRequestDispatcher(where).include(req, resp);
			resp.getWriter().print("<<<");
			return;
		default:
	}
}

And I call it with this URI:

"/gateway/x?what=forward&where=/"

The test is https://github.com/ops4j/org.ops4j.pax.web/blob/web-9.0.4/pax-web-jetty/src/test/java/org/ops4j/pax/web/service/jetty/internal/UnifiedJettyTest.java#L505-L516

Jetty version(s) 12.0.0.alpha3

Java version/vendor (use: java -version) openjdk version "19.0.1" 2022-10-18 OpenJDK Runtime Environment GraalVM CE 22.3.0 (build 19.0.1+10-jvmci-22.3-b08) OpenJDK 64-Bit Server VM GraalVM CE 22.3.0 (build 19.0.1+10-jvmci-22.3-b08, mixed mode, sharing)

OS type/version macOS

Description When using Resources from the resource class path, GraalVM turns them into URLs of the abstract "resource:" scheme (e.g., resource:/com/example/server/webapp/). This resource scheme appears to be currently unsupported by Jetty:

java.lang.IllegalArgumentException: URI scheme not supported: resource:/com/example/server/webapp/
	at org.eclipse.jetty.util.resource.ResourceFactoryInternals$CompositeResourceFactory.newResource(ResourceFactoryInternals.java:145)
	at org.eclipse.jetty.util.resource.ResourceFactory.newResource(ResourceFactory.java:229)

How to reproduce?

Place a file named "test.txt" in the resource class path relative to some class (e.g., foo.bar.SomeClass -> src/main/resources/foo/bar/test.txt). Build GraalVM native image.

Compare the output of

Resource res = ResourceFactory.root().newResource(SomeClass.class.getResource("test.txt"));
System.out.println(res);    

between Hotspot and native-image.

With Hotspot, the URL will be something like jar:file:///path/to/classpath.jar!/foo/bar/test.txt or file:///path/to/resource/foo/bar/test.txt).

With GraalVM native-image, it's resource:/foo/bar/test.txt.

Note that this also means that tricks to enumerate/list the contents of a resource directory (by traversing the file hierarchy or jar archive) will fail.