I have setup a eureka server and a handful of web services as eureka clients, all deployed as aws elastic beanstalk apps. Everything appears to be working correctly- the apps register with eureka and are visible on the system status page, and I can hit them via a zuul server that gets routing info from eureka.
However, the clients are reporting that they are not able to send heartbeats, despite the fact that the server is receiving them just fine. The server lists the services as UP and the access log for the server shows that it is receiving the heartbeats and replying with a 200 status code.
But the clients are are throwing 400/Bad Request exceptions:
2019-03-08 15:38:11,884 [DiscoveryClient-HeartbeatExecutor-0] DEBUG c.n.d.s.t.j.AbstractJerseyEurekaHttpClient - Jersey HTTP PUT <eureka server host>/eureka/apps/HOOPLA-RECOMMENDATIONS/<hostname>:80:3aaeab0f-8899-4890-9323-584cf416aec2; statusCode=200
2019-03-08 15:38:11,884 [DiscoveryClient-HeartbeatExecutor-0] ERROR c.n.d.s.t.d.RedirectingEurekaHttpClient - Request execution error. endpoint=DefaultEndpoint{ serviceUrl='<eureka server host>/eureka}
javax.ws.rs.WebApplicationException: HTTP 400 Bad Request
at com.netflix.discovery.provider.DiscoveryJerseyProvider.readFrom(DiscoveryJerseyProvider.java:110)
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:634)
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:586)
at com.netflix.discovery.shared.transport.jersey.AbstractJerseyEurekaHttpClient.sendHeartBeat(AbstractJerseyEurekaHttpClient.java:105)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.MetricsCollectingEurekaHttpClient.execute(MetricsCollectingEurekaHttpClient.java:73)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.executeOnNewServer(RedirectingEurekaHttpClient.java:118)
at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.execute(RedirectingEurekaHttpClient.java:79)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:120)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:846)
at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1405)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.thoughtworks.xstream.io.StreamException:
at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:115)
at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:84)
at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1445)
at com.netflix.discovery.converters.wrappers.CodecWrappers$XStreamXml.decode(CodecWrappers.java:387)
at com.netflix.discovery.provider.DiscoveryJerseyProvider.readFrom(DiscoveryJerseyProvider.java:103)
... 23 common frames omitted
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:108)
... 27 common frames omitted
2019-03-08 15:38:11,884 [DiscoveryClient-HeartbeatExecutor-0] WARN c.n.d.s.t.d.RetryableEurekaHttpClient - Request execution failed with message: HTTP 400 Bad Request
2019-03-08 15:38:11,884 [DiscoveryClient-HeartbeatExecutor-0] ERROR c.netflix.discovery.DiscoveryClient - DiscoveryClient_HOOPLA-RECOMMENDATIONS/<hostname>:80:3aaeab0f-8899-4890-9323-584cf416aec2 - was unable to send heartbeat!
The client acknowledges the 200 response from the server, but then throws an error when trying to parse the response body. Stepping through the client code, it looks like the client is expecting an InstanceInfo object in the body of the response it gets from the server. But the heartbeat endpoint's response body is empty.
The server and client are the same eureka version (1.9.3).
Is this the expected behavior? Is this a bug, or could there be a problem with my configuration?