Hi we are noticing a lot of activity in some threads related to Restlet. The problem starts when the servers are on production for a while. Then we take out all requests but still get a lot of CPU usage sporadically.
This is the top with threads option:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
36774 apalabra 20 0 44.5g 11g 51m R 99.9 19.1 187:27.85 java
36780 apalabra 20 0 44.5g 11g 51m R 99.9 19.1 186:33.75 java
50351 apalabra 20 0 44.5g 11g 51m R 99.9 19.1 89:22.97 java
2853 apalabra 20 0 44.5g 11g 51m R 99.9 19.1 82:24.84 java
When I do a jstack to the process and look for that specific threads I find these two stack traces:
Thread 50351: (state = IN_JAVA)
- org.restlet.engine.io.ReaderInputStream.read(byte[], int, int) @bci=213, line=166 (Compiled frame; information may be imprecise)
- java.io.InputStream.skip(long) @bci=44, line=222 (Compiled frame)
- java.io.FilterInputStream.skip(long) @bci=5, line=151 (Interpreted frame)
- org.restlet.engine.application.RangeInputStream.read(byte[], int, int) @bci=22, line=159 (Interpreted frame)
- java.io.FilterInputStream.read(byte[]) @bci=5, line=107 (Compiled frame)
- org.restlet.engine.io.BioUtils.copy(java.io.InputStream, java.io.OutputStream) @bci=8, line=75 (Compiled frame)
- org.restlet.engine.application.RangeRepresentation.write(java.io.OutputStream) @bci=5, line=143 (Interpreted frame)
- org.restlet.engine.http.ServerCall.writeResponseBody(org.restlet.representation.Representation, java.nio.channels.WritableByteChannel, java.io.OuputStream) @bci=18, line=553 (Compiled frame)
- org.restlet.engine.http.ServerCall.sendResponse(org.restlet.Response) @bci=49, line=491 (Compiled frame [deoptimized])
- org.restlet.ext.servlet.internal.ServletCall.sendResponse(org.restlet.Response) @bci=218, line=462 (Compiled frame [deoptimized])
- org.restlet.engine.http.adapter.ServerAdapter.commit(org.restlet.engine.http.HttpResponse) @bci=442, line=200 (Compiled frame)
- org.restlet.engine.http.HttpServerHelper.handle(org.restlet.engine.http.ServerCall) @bci=30, line=154 (Compiled frame)
- org.restlet.ext.servlet.ServerServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=24, line=1031 (ompiled frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=366, line=790 (Compiled frame)
- org.eclipse.jetty.servlet.ServletHolder.handle(org.eclipse.jetty.server.Request, javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bc=89, line=808 (Compiled frame)
- org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=211, line=1669 (Cmpiled frame)
- org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletRespnse, javax.servlet.FilterChain) @bci=53, line=88 (Compiled frame)
- org.springframework.web.filter.OncePerRequestFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChan) @bci=383, line=76 (Compiled frame)
- org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=103, line=1652 (Cmpiled frame)
- org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, java.servlet.http.HttpServletResponse) @bci=203, line=585 (Compiled frame)
- org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, avax.servlet.http.HttpServletResponse) @bci=32, line=143 (Compiled frame)
- org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, java.servlet.http.HttpServletResponse) @bci=553, line=577 (Compiled frame)
Thread 2853: (state = IN_JAVA)
- sun.nio.cs.Surrogate$Parser.parse(char, char[], int, int) @bci=104, line=242 (Compiled frame; information may be imprecise)
- sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(java.nio.CharBuffer, java.nio.ByteBuffer) @bci=264, line=542 (Compiled frame)
- sun.nio.cs.UTF_8$Encoder.encodeLoop(java.nio.CharBuffer, java.nio.ByteBuffer) @bci=17, line=619 (Compiled frame)
- java.nio.charset.CharsetEncoder.encode(java.nio.CharBuffer, java.nio.ByteBuffer, boolean) @bci=57, line=561 (Compiled frame)
- org.restlet.engine.io.ReaderInputStream.read(byte[], int, int) @bci=116, line=146 (Compiled frame)
- java.io.InputStream.skip(long) @bci=44, line=222 (Compiled frame)
- java.io.FilterInputStream.skip(long) @bci=5, line=151 (Interpreted frame)
- org.restlet.engine.application.RangeInputStream.read(byte[], int, int) @bci=22, line=159 (Interpreted frame)
- java.io.FilterInputStream.read(byte[]) @bci=5, line=107 (Compiled frame)
- org.restlet.engine.io.BioUtils.copy(java.io.InputStream, java.io.OutputStream) @bci=8, line=75 (Compiled frame)
- org.restlet.engine.application.RangeRepresentation.write(java.io.OutputStream) @bci=5, line=143 (Interpreted frame)
- org.restlet.engine.http.ServerCall.writeResponseBody(org.restlet.representation.Representation, java.nio.channels.WritableByteChannel, java.io.Ou
- org.restlet.engine.http.ServerCall.sendResponse(org.restlet.Response) @bci=49, line=491 (Compiled frame [deoptimized])
- org.restlet.ext.servlet.internal.ServletCall.sendResponse(org.restlet.Response) @bci=218, line=462 (Compiled frame [deoptimized])
- org.restlet.engine.http.adapter.ServerAdapter.commit(org.restlet.engine.http.HttpResponse) @bci=442, line=200 (Compiled frame)
- org.restlet.engine.http.HttpServerHelper.handle(org.restlet.engine.http.ServerCall) @bci=30, line=154 (Compiled frame)
- org.restlet.ext.servlet.ServerServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=24, line=1031 (
- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=366, line=790 (Compiled frame)
Do you have any clues what could this be? I know we are using an outdated version of Restlet, but if we find that this issue is related to the project, we could update and fix it.
System specs
Jetty version: 9.2.8
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: /usr/java/jdk1.7.0_67/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.35-43.137.amzn1.x86_64", arch: "amd64", family: "unix"
Thanks!
Module: Restlet Engine Edition: Java SE State: analysis Priority: high