Reported by jeningar on 2014-09-23 10:37 UTC Hi Steve, hi Gerwin,

using the skeleton.nested, the resulting program showed the following behaviour:

echo "list sessions;" | sdmsh

hangs forever. But

echo "list sessions;" > /tmp/x sdmsh < /tmp/x

teminates as should be.

After merging the skeleton.default and skeleton.nested this erroneous behaviour vanished.

Diffing the old and new skeleton.nested shows (< == new; > == old):

[ronald@cheetah shell]$ diff skeleton.nested* | more
39c39
< 

---
>   
169c169
<     while (numRead == 0) { // bug #130 discussion; while is better than if

---
>     if (numRead == 0) {
403,409d402
<     // cached fields:
<     int zzCurrentPosL;
<     int zzMarkedPosL;
<     int zzEndReadL = zzEndRead;
<     char [] zzBufferL = zzBuffer;
<     char [] zzCMapL = ZZ_CMAP;
< 
413c406,411
<       zzMarkedPosL = zzMarkedPos;

---
>       // cached fields:
>       int zzCurrentPosL;
>       int zzMarkedPosL = zzMarkedPos;
>       int zzEndReadL = zzEndRead;
>       char [] zzBufferL = zzBuffer;
>       char [] zzCMapL = ZZ_CMAP;

The HUGE difference is that the variables (notably zzEndReadL) aren't initialized by skeleton.default in every iteration of the while loop.

During the merge I wondered why there are two skeletons in the first place. Those who don't want to read from multiple streams just don't call yypushStream() and friends. One skeleton would be more than enough (and a link for backward compatibility).

for the sake of completeness I attached the new skeleton.nested.

Regards,

Ronald

java_cup/javacup needs jflex to build. jflex needs java_cup/javacup to build. Both have circular dependencies on each other requiring bootstrap. No clean way to build either from source without using a pre-made binary. Ideally this should change and allow pieces to be built. Such that either can be built fully from source without requiring a pre-built jar. This must have been the case initially before either was made. Chicken and egg situation. Java seems to have lots of issues with such, jaxen/jdom, antlr/stringtemplate, and jflex/javacup. Of course the JDK itself, though open jdk can technically be built from source going back to 1.5 gcj and gnu-classpath.

Hopefully that is possible here, some old version of either javacup or jflex does not need the other and can be the first step in a building from source solution. At the present time each is having to use binaries which is not preferred. Thank you for your consideration in addressing this circular dependency issue.

Reported by jeningar on 2014-08-05 13:14 UTC Hi,

We have an application that receives commands over a TCP/IP socket. After each command a reply is sent. Normally the communication is interactive, which means a strict order of questions and answers. This works perfectly with jflex 1.4.x. (The grammar is easy from the jflex point of view. Every command is terminated by a semicolon, no read ahead necessary; this follows your FAQ answer on "I want my scanner to read from a network byte stream or from interactive stdin. Can I do this with JFlex?").

Now we have some problems with Jflex 1.6.0.

Jflex generates following code in zzRefill():

...
    int requested = zzBuffer.length - zzEndRead;
    int totalRead = 0;
    while (totalRead < requested) {
      int numRead = zzReader.read(zzBuffer, zzEndRead + totalRead, requested - totalRead);
      if (numRead == -1) {
        break;
      }
      totalRead += numRead;
    }
...

This code reads bytes from the input as long as the buffer isn't full or until EOF is reached. This is nice in case of reading a file, but leads to deadlocks if using an interactive scanner.

The easiest way to avoid the deadlock is to eliminate the while loop. If the assumption that a read() returns at least one character (or EOF) isn't valid, the while loop must be executed as long as nothing is read. (I tested this; it seems to work flawlessly).

Since I assume there was a good reason for this piece of code, I'd like to have a command line option to eliminate the while loop here. I'd be delighted if someone would explain me why a repeated call of zzRefill() is worse than the while loop. It can't really be a performance issue. The costs of an I/O exceed by far the costs of a function call and a bit of basic arithmetic.

Regards,

Ronald

This pull request increases coverage on property tests by improving generators and updating test cases. Improvements made to the generators to ensure cased characters are chosen with more frequency. Tests with caseless options have been added as a boolean with updates to allow caseless to operate correctly. For State Set updates to ensure resize code is called using the new OffsetGen generator. Also, property and additional code added to State Set to improve coverage.

I am a part of a research group at the University of Illinois Chicago that works towards novel methods to detect bugs in software projects. This project is identified as being a good candidate for application of our research method to improve coverage using property tests. The above issue outlines the findings with a pull request of the enhanced property test to document the observed behavior.

Default %unicode means 10x times static memory footprint (see zzUnpackCMap). Switching to 2.0 leaves Character.codePointAt() generation intact leading to:

java.lang.ArrayIndexOutOfBoundsException: 65536

in advance() method.

The jflex-unicode-maven-plugin is slow and fetches data from unicode.org which is also not very reliable. As a result, the generated code has been checked in. This is bad practice, and we have been modifying these generated files.

This is the first step in an attempt to replace the jflex-unicode-maven-plugin by Bazel:

1. Bazel fetches and caches all resources. It can use mirrors. See #522
2. With, this change, only UnicodeProperties is re-generated, using versions given in the command-line.
3. Instead of using a custom "skeleton", I'm using Apache Velocity template engine. See #520
4. I've rewritten more than I wanted because the URL was too much part of the previous model. See DataFileType.java

Effective changes in generated file:

• Set default version to 9.0.0 instead of 9.0. This should be a no-op.
• Use switch/case rather than chain of ifs
• Bump unicode 3.1.0 to 3.1.1

Reported by thierryblind on 2015-01-13 23:42 UTC Hello, I'm a committer for the eclipse PDT project (https://eclipse.org/pdt/) and I'm actually trying to migrate all the lexers (using JFlex 1.4) to use JFlex >= 1.5 Sadly I'm now facing infinite loops because some lexers don't seem to behave the same anymore. I added as attachment a test project that contains some of the problematic lexers (in folder "parserTools"), the generated JFlex 14.3 and 1.5.1 classes, and a test case (src/launch/Tests.java). Could you please have a look and tell me what's happening? Thank you very much for your help,

Thierry.

Reported by kneunert on 2010-03-01 20:56 UTC I'm using a bit of a trick in the Lexer like this:

<YYINITIAL> {identifier}[ { yybegin(ARRAY1); return someSymbol; } <ARRAY1> $? { yybegin(ARRAY2); return someSymbol; } <ARRAY2> $? { yybegin(ARRAY3); return someSymbol; } <ARRAY3> $? { yybegin(YYINITIAL); return someSymbol; } <YYINITIAL> ] { yybegin(YYINITIAL); return someSymbol; }

The trick here is, that i use an unconvential optional character. This Character is not there, so no character gets consumed however a series of symbols are returned. This used to work in JLex and it does not seem to work in JFlex anymore. I get this:

Symbol: [ Exception in thread "main" java.lang.Error: Error: could not match input at struktor.processor.Yylex.zzScanError(Yylex.java:439) at struktor.processor.Yylex.next_token(Yylex.java:590) at struktor.processor.MyMain.main(MyMain.java:19)

I have a simple testcase for this. If needed, i can attach it to this ticket.

Thanks

Kim ( https://sourceforge.net/projects/struktor/ )

Reported by smagoun on 2004-01-30 20:35 UTC yy_refill() in skeleton.default and skeleton.nested seems to have a problem expanding the buffer correctly. The bug manifests itself when reading a lot of data at once. I ran into this using the Piccolo XML parser, which uses JFlex to parse XML. Piccolo died while reading a very long CDATA element in the XML. I tracked it to yy_refill(), which seems to have been copied from one of the skeleton files JFlex ships with.

The problem is that the buffer never expands properly when reading long input, which results in an ArrayIndexOutOfBoundsException. The following patch fixes Piccolo; I'm not sure if it applies to JFlex, but I'm guessing it might.

(I'm not convinced that the if() should check yy_currentPos>=buffer.length at all, but it seems harmless)

--- PiccoloLexer.java   Sun Jul  7 14:21:18 2002
+++ PiccoloLexer copy.java      Fri Jan 30 15:07:44 2004
@@ -3291,9 +3291,10 @@
}

/* is the buffer big enough? */
-    if (yy_currentPos >= yy_buffer.length) {
+    if (yy_currentPos >= yy_buffer.length)
+        || yy_markedPos >= yy_buffer.length) {
/* if not: blow it up */
-      char newBuffer[] = new char[yy_currentPos*2];
+      char newBuffer[] = new char[yy_buffer.length*2];
System.arraycopy(yy_buffer, 0, newBuffer, 0, 
yy_buffer.length);
yy_buffer = newBuffer;
}

Hello,

After trying an upgrade of OpenGrok to JFlex v1.8.1, I found that code specified in %eof{ is not included anymore in the generated lexers. I browsed through the JFlex manual, but I didn't see anything to indicate the handling would have changed.

(I did check but didn't see any other %eof open issues here.)

Please any advice?

Thank you.

Odd generated output with error: orphaned default and error: 'else' without 'if'.

qdox 1.12.1 builds with 1.4.3, but not 1.6.1 bootstrapped qdox 2 builds with 1.4.3 and 1.6.1 bootstrapped jflex 1.6.1 builds with 1.6.1 bootstrapped but not 1.4.3

The errors are the same when jflex 1.6.1 bootstrapped fails on qdox 1.12.1, and when jflex 1.4.3 fails on jflex 1.6.1. That is odd and I cannot explain.

qdox 1.12.1 under 1.6.1 bootstrapped via binary fails

 * Compiling ...
src/java/JFlexLexer.java:1357: error: orphaned default
        default:
        ^
src/java/JFlexLexer.java:2051: error: 'else' without 'if'
          else {
          ^
2 errors
 * ERROR: dev-java/qdox-1.12.1-r10::os-xtoo failed (compile phase):

jflex 1.6.1 under 1.4.3 fails, under 1.6.1 bootstrapped via binary it builds fine

Writing code to "java/LexScan.java"
 * Compiling ...
src/main/java/LexScan.java:3697: error: orphaned default
          default:
          ^
src/main/java/LexScan.java:3626: error: 'else' without 'if'
      else {
      ^
2 errors

I have 1.4.3 built and running under Java 9. I do not believe the jdk version has anything to do with the generated output with errors. I think that has more to do with syntax or something.

The template uses \uXXXX for all code points, but this syntax is incorrect for code points > 0xFFFF (needs \U instead of \u).

• [ ] investigate why this does not lead to test failures
• [ ] fix it

Bumps guava from 30.1.1-jre to 31.1-jre.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Since we now allow macro uses in char class content, we may encounter arbitrary regular expressions. #996 turns these into properly reported errors, but we could actually allow all expressions that have a fixed length of 1. Some examples that would make sense:

%%
M = a
%%
[ {M} ]	 {  }

%%
M = "a"
%%
[ {M} ]	 {  }

%%
M = [a] | [b]
%%
[ {M} ]	 {  }

(This is a follow-on from #888)

They might also be a bit easier to read..

Tracking issue for:

• [ ] https://github.com/jflex-de/jflex/security/code-scanning/12
• [ ] https://github.com/jflex-de/jflex/security/code-scanning/13

This is really annoying when you generate on different machines/oses and your resulting class is under VCS. Path is already changing.

I'm talking about

 * This class is a scanner generated by 
 * <a href="http://www.jflex.de/">JFlex</a> 1.7.0-1
....