This is an attempt to fix #133 and #164 at the same time, without having to set a code page manually.

Further investigation of #164 has shown: Even when the UTF-8 code page is supported just fine (e.g. on Windows 10), it seems to be partially broken. As an example, with the system locale set to Chinese, the UTF-8 code page will produce incorrect characters to be displayed.

Since we avoid all encoding problems for the input mechanism by using the Windows API to read a Unicode char directly, I've been looking for solutions to apply the same for console output. This pull request changes the Windows terminal implementation to use WriteConsoleW to write Unicode text directly to the console window - bypassing all problems caused by the code pages.

Since the Windows API accepts characters directly, the primary output for the Windows terminal now operates as Writer: the encoding step is skipped entirely. For compatibility reasons, we still need to support a OutputStream: some applications may use it directly through Terminal.output(). The output stream now emulates the old behavior: it decodes the written bytes using the configured code page. Consequently, Unicode works only properly when using the Writer directly (through Terminal.writer()). This is the case for JLine's LineReader implementation and probably most other applications.

I've tested this on Windows 10 with the system locale set to English, German and Chinese and various code pages and it now handles both #133 and #164 automatically, without having to change the code page. We still need to test if it works properly on older Windows versions.

jline version 3.18.0 works, but 3.19.0 and 3.20.0 prints out raw unprocessed ANSI sequences on Git Bash from Git For Windows version 2.33.0.2-64-bit.

Version 3.19.0 and higher

Version 3.18.0

Trying to use the JLine as library for a console launching OS or perl scripts, we found the following problem on Windows: Below the source code of small application, launching a subprocess from the input of JLine readline. When trying to execute a small perl script (see below), from the small application, the std input is blocked (see the image included). Calling this small application (from this small application) is also blocking. So, it is not a perl problem (;o).

Tests made on Windows 7 and Windows 2016 server.

import org.jline.reader.*; 
import org.jline.terminal.Terminal; 
import org.jline.terminal.TerminalBuilder; 
import java.util.concurrent.TimeUnit;

public class Main {

	public static void main(String[] args) {

		String prompt = "prompt> "; 
		String rightPrompt = null; 
		TerminalBuilder builder = null; 
		
		try {
			builder = TerminalBuilder.builder(); 

			Terminal terminal = builder
					.system(true)
					.build(); 

			LineReader reader = LineReaderBuilder.builder() 
					.terminal(terminal) 
					.build(); 
			
			while (true) { 

				String line = null; 
				try { 
					line = reader.readLine(prompt, rightPrompt, null, null); 
				} catch (UserInterruptException e) { 
					terminal.writer().println("User Interruption !");
				} catch (EndOfFileException e) { 
					return; 
				} 

				if (line == null) { 
					continue; 
				} 

				line = line.trim(); 

				terminal.writer().println("======>\"" + line + "\""); 

				terminal.flush(); 

				if (line.equalsIgnoreCase("quit") || line.equalsIgnoreCase("exit")) { 
					break; 
				} 

				runCommand (line);
			}
		}
		catch (Throwable t) {
			t.printStackTrace();
		}
	}

	public static int runCommand (String command) {

		int returnCode = -1;

		try {
			ProcessBuilder pb = new ProcessBuilder(command.split(" "));
			pb.inheritIO();
			Process p = pb.start ();  

			if (!p.waitFor(60,TimeUnit.SECONDS)) 
				System.out.println ("Timeout");
			else {
				returnCode = p.exitValue();
				System.out.println ("return code :" + returnCode);
			}

			return returnCode;

		} catch (Throwable t) { 
			t.printStackTrace();
			returnCode = -1;
		}
		return returnCode;
	}
}

Perl Script:

print  "Do you Wish To continue: Yes/No \n";
	my $contExec = <STDIN>;
	chomp($contExec);
	if ( uc($contExec) ne "YES" )
	{
        	print "Exiting\n";
		exit -1;
	}

When running a java program from cygwin, jline3 prints the following stack trace:

WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)                                            
Jul 21, 2017 1:59:03 AM org.jline.utils.Log logr                                                                                                             
WARNING: Unable to retrieve infocmp for type cygwin                                                                                                          
java.io.IOException: Cannot run program "infocmp": CreateProcess error=2, The system cannot find the file specified                                          
        at java.lang.ProcessBuilder.start(Unknown Source)                                                                                                    
        at org.jline.utils.InfoCmp.getInfoCmp(InfoCmp.java:547)                                                                                              
        at org.jline.terminal.impl.AbstractTerminal.parseInfoCmp(AbstractTerminal.java:160)                                                                  
        at org.jline.terminal.impl.DumbTerminal.<init>(DumbTerminal.java:102)                                                                                
        at org.jline.terminal.TerminalBuilder.doBuild(TerminalBuilder.java:296)                                                                              
        at org.jline.terminal.TerminalBuilder.build(TerminalBuilder.java:172)                                                                                
        at org.jline.terminal.TerminalBuilder.terminal(TerminalBuilder.java:57)                                                                              
        at org.jline.reader.LineReaderBuilder.build(LineReaderBuilder.java:96)                                                                                                                                                                                
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified                                                             
        at java.lang.ProcessImpl.create(Native Method)                                                                                                       
        at java.lang.ProcessImpl.<init>(Unknown Source)                                                                                                      
        at java.lang.ProcessImpl.start(Unknown Source)                                                                                                       
        ... 11 more

When using a custom version of sbt on windows that uses the latest jline, none of the delete, home or insert keys work. I was able to fix it with a workaround (https://github.com/sbt/sbt/pull/5874/commits/036fef33832ae672f13c394f39a0ed1b5040d4ec) but I'm wondering if maybe these keys should be added to the ansi.caps file.

I'm currently trying to update from jLine2 to jLine3 but I'm having problems creating a command prompt which always "stays as the bottom". This is want I'm currently testing with.

public class Main {

    public static void main( String[] args ) {
        new Main();
    }

    private Terminal terminal;
    private LineReader reader;

    private Main() {
        try {
            this.terminal = TerminalBuilder.terminal();
            this.reader = LineReaderBuilder.builder().terminal( terminal ).build();

            Thread thread = new Thread( new Runnable() {
                @Override
                public void run() {
                    while ( !Thread.interrupted() ) {
                        try {
                            Thread.sleep( 1000 );
                        } catch ( InterruptedException e ) {
                            break;
                        }

                        terminal.writer().println( "test" );
                        terminal.flush();
                    }
                }
            } );
            thread.start();

            String line;
            while ( !Thread.interrupted() ) {
                line = reader.readLine( "> " );

                if ( line == null ) {
                    continue;
                }

                if ( line.equalsIgnoreCase( "quit" ) ) {
                    thread.interrupt();
                    System.exit( 0 );
                    break;
                }
            }
        } catch ( IOException e ) {
            e.printStackTrace();
        }
    }

}

Link to google group topic https://groups.google.com/forum/#!topic/jline-users/VBLZ6Wn15LY

Problem description:

When running the REPL application in cygwin, the terminal's behavior is unpredictable. E.g. the prompt name is all over the place when I execute a command, additional whitespaces or line breaks are inserted randomly, tab completion and command history are not working.

Steps to reproduce:

1. Download and install cygwin: https://cygwin.com/install.html
2. Build the demo application in the project
3. Execute jline-repl.bat

Log during startup: JLine terminal: xterm-256color

In the terminal try using tab completion, get the previous command with the up arrow key, start typing a command then move the cursor within the command with the left and right arrow keys, execute the help command or just simply hit enter.

Setup:

• Windows 10 x64
• JLine3 3.14.0

CommandBox allows you to pipe a file of commands into it and readLine() will read the lines one at a time off the standard input just as though a human had typed them.

box.exe < commands.txt

If there's no trailing line break at the end of the file, then the last command is read in but the OEF fires before the last line is handed back for execution. I get that this makes some deal of sense since I can press Ctrl-D while typing at the shell to exit immediately, but in the case of reading an actual text file input, I'd like to be able to execute that last line even if there's no line break at the end.

The org.jline.reader.UserInterruptException exception provides getPartialLine() to see if the user typed something. Is the correct fix to add the partial line to the org.jline.reader.EndOfFileException as well?

Related to #138, it would be interesting to have a way to stop reading the main terminal input stream in advance for signal processing. While adding a couple of methods pause() / resume() / isPaused() to the Terminal interface is easy, the problem is that the implementation can not easily interrupt its reading. So this would require a smarter control of the underlying input stream. On Windows, it may be achievable using WaitForSingleObject on the console handle to actually wait until there are some events to read. However, this method is missing in the jansi library. On Unix, we may 2 possibilities that would require investigation: using a FileChannel on top of the InputStream (when it's based on a real unix pipe) and check if the reading thread can actually be interrupted or when using a Pty, it may be possible to set the VTIME flag to set the timeout (in tenth of seconds) so that the read will unblock automatically.

I'm trying to implement a persistent input line while having asynchronous (logging) output with a LineReader. To do this, I call some widgets as suggested in https://github.com/jline/jline3/issues/75#issuecomment-268496777:

reader.callWidget(LineReader.CLEAR);
terminal.writer().println(line);
reader.callWidget(LineReader.REDRAW_LINE);
reader.callWidget(LineReader.REDISPLAY);

This works fine, but since the output is asynchronous, there is a potential race condition:

OutputThread: works
InputThread:  readLine() returns with a new input line
OutputThread: callWidget() results in
    java.lang.IllegalStateException: Widgets can only be called during a `readLine` call
        at org.jline.reader.impl.LineReaderImpl.callWidget(LineReaderImpl.java:671)
InputThread:  readLine() is called to start reading another line
OutputThread: works, but may result into duplicate input prompts (e.g. > > ...)
OutputThread: works

Here is a very simple example that makes this reproducible: https://gist.github.com/Minecrell/374c4d3a0239a8c8a9c50849acf0b3f8

It reads lines from input, and schedules them to be printed a few times asynchronously directly after they were read. In reality, this situation would only occur occasionally if there is new output at the same time. With this example, I always get a duplicate input prompt (> > instead of >), and when typing quickly enough it will also run into the IllegalStateException.

All this brings me to the conclusion that I need some sort of locking mechanism to prevent the two threads from interfering with each other. However, I've been unable to think of any solution that doesn't require changes in JLine.

One way I was thinking about would be to synchronize parts of LineReader.readLine (e.g. everything except actual input I/O) to prevent the input/output thread from modifying the input line at the same time. E.g. https://github.com/Minecrell/jline3/commit/c4c0560a1472d2d10c39daf8cc03f9f2285b8ebb together with this change in the output thread (full code):

synchronized (reader) {
    if (reader.isReading()) {
        reader.callWidget(LineReader.CLEAR);
        terminal.writer().println(line);
        reader.callWidget(LineReader.REDRAW_LINE);
        reader.callWidget(LineReader.REDISPLAY);
    } else {
        // No need to redraw prompt
        terminal.writer().println(line);
    }
}

These changes fix both issues I mentioned above. There is always only a single input prompt (>) and I wasn't able to observe the IllegalStateException anymore.

I'm not sure if this should be default behavior or if some kind of optional, external lock would be more appropriate here. Or maybe you have any other suggestions to avoid the problems mentioned above?

Emacs comint announces it self as DUMP terminal, but it still support ANSI colors.

JLine Terminal should detect emacs based on the INSIDE_EMACS environment variable and turn on ANSI support for it.

I could make a PR and test it if somebody gave me a few pointers where this would be best implemented in JLine.

Bumps sshd-common from 2.7.0 to 2.9.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

AttributedString.fromAnsi(
  new AttributedStringBuilder()
    .append("─")
    .toAttributedString()
    .toAnsi(terminal)
).toAnsi(terminal)

This prints garbage instead of "─".

Culprit appears to be https://github.com/jline/jline3/commit/adb1d94643725b28686cd215e8f6a40e79c19d26: toAnsi converts to alternate charset, but fromAnsi does not convert back. Setting DISABLE_ALTERNATE_CHARSET fixes the issue.

The implication of this is that it is not possible to use AttributedString to set prompt, because there is always round-trip conversion involved.

During integration of jline3 into our projects we noticed that copy-and-pasting leads to an unresponsive shell, requiring us to kill the ssh-client process afterwards.

The issue is reproducible on macOS 12.6 on Intel Macs. After some debugging we found out that only the JNA Pty implementation (JnaNativePty) shows the above problem. Setting org.jline.terminal.jna=fase, i.e. using the JANSI implementation can be used as a workaround in our case. Neither the simple ExternalTerminal nor the JANSI variant (JansiNativePty) have the described issue.

Currently we suspect that issue only affects the macOS implementation of JnaNativePty, however this still needs to be verified.

There 's a potentially related issue: #757

I was looking why cursor show/hide doesn't work in conemu but works fine in cmd and powershell. Looks like windows-conemu.caps is quite old and conemu has matured quite a bit since caps were updated. Essentially in repl demo tput cursor_invisible and tput cursor_visible doesn't work if type is windows-conemu.

If I add below to windows-conemu.caps those those start to work.

civis=\E[?25l, cvvis=\E[?25h,

Looking at https://conemu.github.io/en/AnsiEscapeCodes.html#List_of_supported_codes I'm just wondering what else this caps file might be missing?

This issue can be tested with the jline-repl.bat jansi in the demo folder. If the output of of the command is too long, it's impossible to scroll to it. In order to test it run the above command in the Windows Terminal (either in Powershell or in Command Prompt). Then press enter several times so that the scroll is enabled. Press Alt-S to enable tailtip widgets. Run help command (make sure that the windows is small enough so that the output is longer that the height of the terminal window). Now try to scroll up - the first lines of the output are cropped. It seems to me that the Windows Terminal treats the Capability.change_scroll_region in such a way that it crops the text which is outside the region? Most probably it's a Windows Terminal bug but since I encountered it in the jline, reporting it here.