we successfully run lv1 example, about 25.6G approxiamately, including calculating

but when we turn to lv2 example, the calculation of which is bigger, we find the following problem:

After client session is started, and we run eqh.runEventLoop(1, -1), it gets no established signal and stop. After this session, all the following session shares the same problem.

actually what they get is a SESSION_CLOSED! why!

i am using structure like JxioConnection now, now it can reuse msgs

but i see [WARN] - kref_get:45 资源暂时不可用(resource is temporarily no available)

what may be a reason for that?

actually we are trying to make JXIO run smoothly on spark.

the class name with 'Rdma' is created by us.

now we have one problem, that is: why there is a XIO_E_NO_BUFS

the XIO_E_NO_BUFS is 1247689302 actually, I replace them with their names.

and in this one: 14/10/07 06:20:25 INFO ConnectionRdma: Sending a message.488

the 448 reveals that this message has a length of 488. it should not be a null message

14/10/07 06:19:57 DEBUG MsgPool: MP CTOR entry 14/10/07 06:19:57 DEBUG MsgPool: jxio.MsgPool(2b6c18076620)[count=74, capacity=74, inSize=1000, outSize=1000]: MP CTOR done 14/10/07 06:19:57 DEBUG EventQueueHandler: EQH CTOR entry 14/10/07 06:19:57 DEBUG EventQueueHandler: jxio.EQH[2b6c180b6f60]: EQH CTOR done 14/10/07 06:19:57 DEBUG ServerPortal: SP CTOR entry 14/10/07 06:19:57 DEBUG ServerPortal: jxio.SP[2b6c180c73d0]: listening to rdma://12.12.12.7:0/ 14/10/07 06:19:57 DEBUG ServerPortal: jxio.SP[2b6c180c73d0]: uri with port 0 is rdma://12.12.12.7:0/ 14/10/07 06:19:57 DEBUG ServerPortal: jxio.SP[2b6c180c73d0]: uri with port 56242 is rdma://12.12.12.7:56242/ 14/10/07 06:19:57 DEBUG EventQueueHandler: jxio.EQH[2b6c180b6f60]: adding 2b6c180c73d0 to map 14/10/07 06:19:57 DEBUG ServerPortal: jxio.SP[2b6c180c73d0]: SP CTOR done 14/10/07 06:19:57 INFO ConnectionManagerRdma: RDMA: waiting for JXIO incoming connections 14/10/07 06:19:57 INFO HttpFileServer: HTTP File server directory is /tmp/spark-b9c3e6b4-661e-47dd-90bb-40cb31d98d1d 14/10/07 06:19:57 INFO HttpServer: Starting HTTP Server 14/10/07 06:20:18 INFO HttpBroadcast: Started reading broadcast variable 0 14/10/07 06:20:19 INFO HttpBroadcast: Reading broadcast variable 0 took 0.490863155 s 14/10/07 06:20:25 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache 14/10/07 06:20:25 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them 14/10/07 06:20:25 INFO MapOutputTrackerWorker: Doing the fetch; tracker actor = Actor[akka.tcp://spark@node19:37687/user/MapOutputTracker#1240346833] 14/10/07 06:20:25 INFO MapOutputTrackerWorker: Got the output locations 14/10/07 06:20:25 DEBUG EventQueueHandler: EQH CTOR entry 14/10/07 06:20:25 DEBUG EventQueueHandler: jxio.EQH[2b6cb004ac60]: EQH CTOR done 14/10/07 06:20:25 DEBUG MsgPool: MP CTOR entry 14/10/07 06:20:25 DEBUG MsgPool: jxio.MsgPool(2b6cb004aa50)[count=10, capacity=10, inSize=1000, outSize=1000]: MP CTOR done 14/10/07 06:20:25 DEBUG ClientSession: CS CTOR entry 14/10/07 06:20:25 DEBUG ClientSession: jxio.CS[2b6cb00651b0]: connecting to rdma://12.12.12.19:0/?cacheId=47745809558624 14/10/07 06:20:25 DEBUG EventQueueHandler: jxio.EQH[2b6cb004ac60]: adding 2b6cb00651b0 to map 14/10/07 06:20:25 DEBUG ClientSession: jxio.CS[2b6cb00651b0]: CS CTOR done 14/10/07 06:20:25 INFO ConnectionRdma: Try to establish a new session to 'rdma://12.12.12.19:0/' 14/10/07 06:20:25 INFO ConnectionRdma: Sending a message.488 14/10/07 06:20:25 INFO ConnectionRdma: My due is okay! 14/10/07 06:20:25 DEBUG ClientSession: jxio.CS[2b6cb00651b0]: Received Session Event: Type=7, Reason=XIO_E_NO_BUFS 14/10/07 06:20:25 DEBUG ClientSession: jxio.CS[2b6cb00651b0]: Received Msg Error Event: Msg=jxio.Msg(2b6c1807b970)[msgIn=[pos=0 lim=0 cap=1000], msgOut=[pos=4 lim=1000 cap=1000], msgPool=jxio.MsgPool(2b6c18076620)[count=73, capacity=74, inSize=1000, outSize=1000]], Reason=XIO_E_MSG_FLUSHED 14/10/07 06:20:25 ERROR ConnectionRdma: [JXIO] ERROR. MSG_FLUSHED 14/10/07 06:20:25 DEBUG ClientSession: jxio.CS[2b6cb00651b0]: Received Session Event: Type=4, Reason=XIO_E_NO_BUFS 14/10/07 06:20:25 INFO ConnectionRdma: [JXIO] Session event occured. 14/10/07 06:20:26 DEBUG EventQueueHandler: jxio.EQH[2b6cb004ac60]: closing EQH 14/10/07 06:20:26 DEBUG EventQueueHandler: jxio.EQH[2b6cb004ac60]: attempting to close EQH while objects [47745809666480] are still listening 14/10/07 06:20:26 DEBUG EventQueueHandler: jxio.EQH[2b6cb004ac60]: attempting to close EQH while objects [47745809666480] are still listening 14/10/07 06:20:26 DEBUG ClientSession: jxio.CS[2b6cb00651b0]: Received Session Event: Type=1, Reason=XIO_E_NO_BUFS 14/10/07 06:20:26 DEBUG EventQueueHandler: jxio.EQH[2b6cb004ac60]: removing 2b6cb00651b0 from map 14/10/07 06:20:26 DEBUG EventQueueHandler: jxio.EQH[2b6cb004ac60]: no more objects listening 14/10/07 06:20:26 DEBUG EventQueueHandler: jxio.EQH[2b6cb004ac60]: closing EQH is finished

when there is a job, everything seems to run smoothly in the first

but later there is a MSG_FLUSHED what does it mean?

14/10/11 20:09:33 INFO ConnectionRdma: Message Contentjxio.Msg(2aaf0c0e8e90)[msgIn=[pos=0 lim=0 cap=65536], msgOut=[pos=8618 lim=65536 cap=65536], msgPool=jxio.MsgPool(2aaf0c0c5700)[count=1964, capacity=2064, inSize=65536, outSize=65536]] 14/10/11 20:09:33 INFO ConnectionRdma: My due is okay! 14/10/11 20:09:33 INFO ConnectionManagerRdma: Handling delay is 609 ms 14/10/11 20:09:33 INFO ConnectionRdma: Try to establish a new session to 'rdma://ibnode7:57869/' 14/10/11 20:09:33 INFO ConnectionRdma: Sending a message.9275 14/10/11 20:09:33 INFO ConnectionRdma: Message Contentjxio.Msg(2aaf0c0e91f0)[msgIn=[pos=0 lim=0 cap=65536], msgOut=[pos=63198 lim=65536 cap=65536], msgPool=jxio.MsgPool(2aaf0c0c5700)[count=1963, capacity=2064, inSize=65536, outSize=65536]] 14/10/11 20:09:33 INFO ConnectionRdma: My due is okay! 14/10/11 20:09:33 INFO ConnectionRdma: Sending a message.9275 14/10/11 20:09:33 INFO ConnectionRdma: Message Contentjxio.Msg(2aaf0c0e9550)[msgIn=[pos=0 lim=0 cap=65536], msgOut=[pos=65146 lim=65536 cap=65536], msgPool=jxio.MsgPool(2aaf0c0c5700)[count=1962, capacity=2064, inSize=65536, outSize=65536]] 14/10/11 20:09:33 INFO ConnectionRdma: My due is okay! 14/10/11 20:09:33 INFO ConnectionRdma: Sending a message.9275 14/10/11 20:09:33 INFO ConnectionRdma: Message Contentjxio.Msg(2aaf0c0e98b0)[msgIn=[pos=0 lim=0 cap=65536], msgOut=[pos=62427 lim=65536 cap=65536], msgPool=jxio.MsgPool(2aaf0c0c5700)[count=1961, capacity=2064, inSize=65536, outSize=65536]] 14/10/11 20:09:33 INFO ConnectionRdma: My due is okay! 14/10/11 20:09:33 INFO ConnectionRdma: Sending a message.9275 14/10/11 20:09:33 INFO ConnectionRdma: Message Contentjxio.Msg(2aaf0c0e9c10)[msgIn=[pos=0 lim=0 cap=65536], msgOut=[pos=63728 lim=65536 cap=65536], msgPool=jxio.MsgPool(2aaf0c0c5700)[count=1960, capacity=2064, inSize=65536, outSize=65536]] 14/10/11 20:09:33 ERROR ConnectionRdma: [JXIO] ERROR. MSG_FLUSHED 14/10/11 20:09:33 ERROR ConnectionRdma: [JXIO] ERROR. MSG_FLUSHED 14/10/11 20:09:33 ERROR ConnectionRdma: [JXIO] ERROR. MSG_FLUSHED 14/10/11 20:09:33 INFO ConnectionRdma: [JXIO] Session event occured. 14/10/11 20:09:33 ERROR ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[pool-2-thread-2,5,main] java.lang.Error: Session is already closed. sendRequest failed at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: Session is already closed. sendRequest failed at com.mellanox.jxio.ClientSession.sendRequest(ClientSession.java:181) at org.apache.spark.network.ConnectionRdma.send(ConnectionRdma.scala:46) at org.apache.spark.network.ConnectionManagerRdma.sendMessage(ConnectionManagerRdma.scala:270) at org.apache.spark.network.ConnectionManagerRdma.handleMessage(ConnectionManagerRdma.scala:318) at org.apache.spark.network.ConnectionManagerRdma$$anon$3.run(ConnectionManagerRdma.scala:335) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ... 2 more 14/10/11 20:09:33 INFO ConnectionManagerRdma: Client is releasing JXIO resources and exiting 14/10/11 20:09:33 INFO ConnectionRdma: [JXIO] Got response. 14/10/11 20:09:33 INFO ConnectionRdma: [JXIO] Got response. 14/10/11 20:09:33 INFO ConnectionRdma: [JXIO] Got response.

I am using SPARK with hive and it is using RDMA now.

In order to co-operate with SPARK session system, we will send a null response to each of the request in RDMA sessions. The ACK message will be later in another way.

and, will the system at one time receive several session?

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00002b527697c379, pid=5523, tid=47633195259648

JRE version: Java(TM) SE Runtime Environment (8.0_20-b26) (build 1.8.0_20-b26) Java VM: Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode linux-amd64 compressed oops) Problematic frame: C [libxio.so+0x3a379] xio_send_response+0x29

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

An error report file with more information is saved as: /public/rdma14/hivetest/spark-rdma/work/app-20141011180557-0001/0/hs_err_pid5523.log [thread 47642922776320 also had an error] [thread 47633310287616 also had an error] [thread 47633562380032 also had an error] [thread 47633329235712 also had an error] [thread 47633322919680 also had an error] [thread 47633539278592 also had an error] [thread 47633306498816 also had an error] [thread 47633330288384 also had an error] [thread 47633316603648 also had an error][thread 47633332393728 also had an error][thread 47633333446400 also had an error][thread 47633544541952 also had an error] [thread 47633318708992 also had an error] [thread 47633326077696 also had an error][thread 47633325025024 also had an error]

[thread 47633315550976 also had an error] [thread 47633323972352 also had an error] [thread 47633540331264 also had an error]

[thread 47633312392960 also had an error]

If you would like to submit a bug report, please visit: http://bugreport.sun.com/bugreport/crash.jsp The crash happened outside the Java Virtual Machine in native code. See problematic frame for where to report the bug.

I find it difficult to find a free rdma port, in java.nio.ServerChannel, user can write port=0 to let java find a free port to use.

Can JXIO provides some apis to find free port? or it would be difficult to find one.

Is this ServerPortal acting much more like a Apache Server? I mean that he can receive messages from clients, and respond, but the ServerPortal cannot send a message to the Client directly?

Therefore, if considers this to be used in a Master/Worker computation structure, is it true that

A. Worker is much more like to use ServerPortal or B. Master should have ServerPortal and ClientSession together

Hello, I'm trying to reimplement Hadoop RPC with JXIO. When the connection is setting up, client sends a header message to server. This message does not expect any responses. Normally, msg can return to its pool, when the response arrives and processed. However, in my case, the msg won't get response(ie. no onResponse() callback happens to this msg), thus I wonder when I should call returnToParentPool() on the msg.

warning: there were 1 deprecation warning(s); re-run with -deprecation for details -->> Error getting JNIEnv when trying to log message: '../common/xio_idr.c:138:xio_idr_destroy() user object leaked: 0x2aca84d107e0 ' -->> Error getting JNIEnv when trying to log message: '../common/xio_idr.c:138:xio_idr_destroy() user object leaked: 0x2aca84d261d0

I always see sometimes like that, and i am worrying what should i do to solve this.

this should be my bad. I use jxio to support a hadoop like application.

and the com.mellanox.jxio.helloworld run smoothly. but in my app, there is always this errors

it does not bother the running, but memory leaking is not good for such an application.

and there is: 14/10/04 17:20:52 DEBUG MsgPool: jxio.MsgPool(2aca84cd03f0)[count=74, capacity=74, inSize=100, outSize=65536]: MP CTOR done 14/10/04 17:20:52 DEBUG EventQueueHandler: EQH CTOR entry 14/10/04 17:20:52 DEBUG EventQueueHandler: jxio.EQH[2aca84d10340]: EQH CTOR done 14/10/04 17:20:52 DEBUG ServerPortal: SP CTOR entry 14/10/04 17:20:52 DEBUG ServerPortal: jxio.SP[2aca84d26060]: listening to rdma://12.12.12.19:10000/ 14/10/04 17:20:52 DEBUG ServerPortal: jxio.SP[2aca84d26060]: uri with port 0 is rdma://12.12.12.19:0/ 14/10/04 17:20:52 DEBUG ServerPortal: jxio.SP[2aca84d26060]: uri with port 10000 is rdma://12.12.12.19:10000/ 14/10/04 17:20:52 DEBUG EventQueueHandler: jxio.EQH[2aca84d10340]: adding 2aca84d26060 to map

While EventQueueHandler thread is running， asynchronized ClientSession.sendRequest() in another thread cannot be handled correctly, if this method is issued too fast. I modified the 'JXIO/examples/org/accelio/jxio/SimpleClient.java' to reproduce this issue：

import java.net.URI;
import java.net.URISyntaxException;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import java.io.IOException;
import java.io.UnsupportedEncodingException;

import org.accelio.jxio.Msg;
import org.accelio.jxio.MsgPool;
import org.accelio.jxio.EventName;
import org.accelio.jxio.EventReason;
import org.accelio.jxio.ClientSession;
import org.accelio.jxio.EventQueueHandler;

public class SimpleClient {

    private final static Log LOG = LogFactory.getLog(SimpleClient.class.getCanonicalName());
    private final MsgPool mp;
    private final EventQueueHandler eqh;
    private ClientSession client;
    public int exitStatus = 1;

    public static long numberofReqs = 0;
    public static long numberofRsps = 0;
    public static long maxNumberofReqs = 1000;
    public static long PRINT_COUNTER = 10;
    class EventLoop extends Thread {        
        EventLoop(){}
        public void run() {
        eqh.run();
        }
    }
    public static void main(String[] args) {        
        final String serverhostname = "172.18.0.14";
        final int port = 9900;

        URI uri = null;
        try {
            uri = new URI("rdma://" + serverhostname + ":" + port + "/");
        } catch (URISyntaxException e) {
            e.printStackTrace();
            return;
        }

        SimpleClient client = new SimpleClient();
        // Connect
        client.connect(uri);
        // Wait for a single connection response
        client.run();       
        // Send
        for (numberofReqs = 1; numberofReqs <= maxNumberofReqs; ++numberofReqs){
            LOG.trace("sending req " + numberofReqs);
            client.send();
            /* XXX if uncomment sleep the program would pop up random error message */
            Thread.sleep(10);      
        }
        // Finish
        LOG.info("Closing the session...");
        client.close();
        LOG.info("Client is releasing JXIO resources and exiting");
        client.releaseResources();
        System.exit(client.exitStatus);
    }

    SimpleClient() {
        this.eqh = new EventQueueHandler(null);
        this.mp = new MsgPool(1024, 100, 100);
    }

    public void connect(URI uri) {
        LOG.info("Try to establish a new session to '" + uri + "'");
        this.client = new ClientSession(eqh, uri, new MyClientCallbacks(this));
    }

    public void send(){
        Msg msg = this.mp.getMsg();
        try {
            msg.getOut().put("Simple request".getBytes("UTF-8"));
        } catch (UnsupportedEncodingException e) {
            // Just suppress the exception handling in this demo code
        }
        try {
            client.sendRequest(msg);
        } catch (IOException e) {
            //all exceptions thrown extend IOException
            LOG.error(e.toString());
        }
    }

    public void run() {
        EventLoop loop = new EventLoop();
        loop.start();
    }

    public void close(){
        client.close(); 
    }

    public void releaseResources() {
        mp.deleteMsgPool();
        eqh.close();
    }

    class MyClientCallbacks implements ClientSession.Callbacks {
        private final SimpleClient client;

        MyClientCallbacks(SimpleClient client) {
            this.client = client;
        }

        public void onSessionEstablished() {
            LOG.info("[SUCCESS] Session established! Hurray !");
        }

        public void onResponse(Msg msg) {
            ++numberofRsps;

            if (numberofRsps % PRINT_COUNTER == 0){
                // Read reply message String
                byte ch;
                StringBuffer buffer = new StringBuffer();
                while (msg.getIn().hasRemaining() && ((ch = msg.getIn().get()) > -1)) {
                    buffer.append((char) ch);
                }
                LOG.info("Got message response " + numberofRsps + ": '" + buffer.toString() + "'");
            }

            msg.returnToParentPool();
            exitStatus = 0; // Success, we got our message response back
        }

        public void onSessionEvent(EventName event, EventReason reason) {
            String str = "[EVENT] Got event " + event + " because of " + reason;
            if (event == EventName.SESSION_CLOSED) { // normal exit
                LOG.info(str);
            } else {
                this.client.exitStatus = 1; // Failure on any kind of error
                LOG.error(str);
            }
            this.client.eqh.stop();
        }

        public void onMsgError(Msg msg, EventReason reason) {
            LOG.info("[ERROR] onMsgErrorCallback. reason=" + reason);
            if (reason == EventReason.MSG_FLUSHED) {
                LOG.info("[STATUS] getIsClosing() = " + this.client.client.getIsClosing());
            }
            msg.returnToParentPool();
            this.client.exitStatus = 1; // Failure on any kind of error
            System.exit(exitStatus);
        }
    }
}

According to JXIO API doc for ServerSession.sendResponse():

org.accelio.jxio.exceptions.JxioSessionClosedException - if session already closed. In case exception is thrown, discardRequest must be called for this message after SESSION_CLOSED event is officially recieved.

Does it mean that the message that failed to send due to this exception can only be discarded in onSessionEvent()? My problem is, in my program, ServerSession might closes before another thread calls sendResponse(), which means discardRequest() might has no chance to be called in onSessionEvent(). Can I just discardRequest(msg) in JxioSessionClosedException handler?

Creating a MsgPool with a total size greater than 2GB causes the allocation to fail.

Ex: new MsgPool(256, 8096000, 8096000)

Is this an expected limitation of accelio/JXIO?

It seems like there is a long to int conversion somewhere, but I haven't been able to track down the exact spot yet.

Hi i have clone the git ,and execute ./bulid.sh under the JXIO-master/JXIO$,and there are some errors:

hadoop@namenode:~/RDMA/JXIO-master/JXIO$ ./build.sh

The JXIO top directory is /home/hadoop/RDMA/JXIO-master/JXIO

JXIO git version: v1.3-16-gca7f25c Build Accelio... libxio C code Submodule 'src/accelio' (https://github.com/accelio/accelio.git) registered for path 'src/accelio' Cloning into 'src/accelio'... remote: Counting objects: 12318, done. remote: Total 12318 (delta 0), reused 0 (delta 0), pack-reused 12317 Receiving objects: 100% (12318/12318), 7.82 MiB | 1.35 MiB/s, done. Resolving deltas: 100% (8876/8876), done. Submodule path 'src/accelio': checked out 'bc0d930ccb033d754d9720e660491edb60d8d52e' AccelIO git version: v1.3-rc3-70-gbc0d930 configure.ac:18: installing ./compile' configure.ac:13: installing./config.guess' configure.ac:13: installing ./config.sub' configure.ac:7: installing./install-sh' configure.ac:7: installing ./missing' benchmarks/usr/xio_perftest/Makefile.am: installing./depcomp' configure: WARNING: Unable to find the infiniband header files, rdma transport will not be available. configure: error: Unable to find the numactl-devel header files FAILURE! stopped JXIO build

What's the problem? thank u. xuxc