Write parsers for arbitrary text inputs, entirely in Java, with no preprocessing phase

TracingParseRunner.getLogs() was deprecated, however, the functionality is essential for debugging (as outlined here: https://github.com/sirthias/parboiled/wiki/Grammar-and-Parser-Debugging).

This is to suggest that an alternative for getLogs() or a refactoring of getLogs() is provided in order to make the functionality available in non-deprecated form again.

(Version grappa 1.0.0-beta.6)

Thanks!

A NPE is thrown if the value of a boolean action variable is directly used to guard a set of rules like in the following example:

public Rule someRule() {
    Var<Boolean> guard = new Var<>(false);
    return sequence(
            optional('@', guard.set(true)),
            "some text",
            optional(guard.get(), // fail if this does not start with @
                    "some other text");
}

The stack trace for the NPE:

java.lang.NullPointerException
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
    at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.add(ImmutableCollection.java:339)
    at com.google.common.collect.ImmutableList$Builder.add(ImmutableList.java:652)
...

The problem can be avoided by using !!guard.get() or guard.get() ? true : false. I think that grappa does not correctly detect guard.get() as an action invocation.

I'm updating some code from 1.0.4 to 2.0.0-beta.1

Everything works correctly, with the sole exception of the value stack retaining its contents from previous runs using the same ParseRunner instance.

Is this an intended change, or a regression?

The following rules do work fine with Grappa 1.0.4 but cause an infinite recursion with Grappa 2.0.0-beta.4.

    @Cached
    public Rule rule2(Var<Integer> count) {
        return firstOf(sequence("c", count.set(0)), sequence("b", rule2(count), count.set(count.getNonnull()+1), "b"));
    }

    @Cached
    public Rule rule3(Var<AtomicInteger> count) {
        return firstOf(sequence("c", set(count, 0)), sequence("b", rule3(count), set(count, count.getNonnull().get()+1), "b"));
    }

    public boolean set(Var<AtomicInteger> var, final int value) {
        var.getNonnull().set(value);
        return true;
    }

In neither case an infinite recursion should occur because both methods are annotated with @Cached and because in each recursion the same Var instance is provided.

The stack overflow exception is:

Exception in thread "main" java.lang.StackOverflowError
    at java.util.Arrays.deepHashCode(Arrays.java:4199)
    at org.parboiled.transform.CacheArguments.hashCode(CacheArguments.java:43)
    at java.util.HashMap.hash(HashMap.java:338)
    at java.util.HashMap.get(HashMap.java:556)
    at grappa1.app.Parser$$parboiled.string(Unknown Source)
    at grappa1.app.Parser$$parboiled.fromCharArray(Unknown Source)
    at grappa1.app.Parser$$parboiled.fromStringLiteral(Unknown Source)
    at grappa1.app.Parser$$parboiled.toRule(Unknown Source)
    at org.parboiled.BaseParser.toRules(BaseParser.java:1845)
    at grappa1.app.Parser$$parboiled.sequence(Unknown Source)
    at grappa1.app.Parser$$parboiled.sequence(Unknown Source)
    at grappa1.app.Parser$$parboiled.rule2(Unknown Source)
    at grappa1.app.Parser$$parboiled.rule2(Unknown Source)
    at grappa1.app.Parser$$parboiled.rule2(Unknown Source)
    at grappa1.app.Parser$$parboiled.rule2(Unknown Source)
[...over and over again...]

In ParsingResult there are two different indicators whether a parsing error has occurred: matched and parseErrors. Is it possible that !matched but parseErrors.isEmpty()? If so: which error message should the application using the parser print? If not: why do both indicators exist at all?

I'd suggest that matched is removed. A parse should be successful if and only if no parse errors did occur.

Creating a new parser crashes 1.0.0-beta.4. The same code did work with Parboiled 1.1.6.

Exception in thread "main" java.lang.RuntimeException: Error creating extended parser class: null
       at org.parboiled.Parboiled.createParser(Parboiled.java:60)
       at bugfinder.app.CParser.<init>(CParser.java:63)
       at bugfinder.app.Main.main(Main.java:70)
Caused by: java.nio.BufferOverflowException
       at java.nio.Buffer.nextPutIndex(Buffer.java:519)
       at java.nio.HeapByteBuffer.putChar(HeapByteBuffer.java:264)
       at org.parboiled.transform.process.InstructionGroupPreparer$MD5Digester.update(InstructionGroupPreparer.java:369)
       at org.parboiled.transform.process.InstructionGroupPreparer$MD5Digester.visitTypeInsn(InstructionGroupPreparer.java:241)
       at org.objectweb.asm.tree.TypeInsnNode.accept(TypeInsnNode.java:83)
       at org.objectweb.asm.tree.InsnList.accept(InsnList.java:162)
       at org.parboiled.transform.process.InstructionGroupPreparer.name(InstructionGroupPreparer.java:176)
       at org.parboiled.transform.process.InstructionGroupPreparer.process(InstructionGroupPreparer.java:116)
       at org.parboiled.transform.ParserTransformer.runMethodTransformers(ParserTransformer.java:98)
       at org.parboiled.transform.ParserTransformer.extendParserClass(ParserTransformer.java:81)
       at org.parboiled.transform.ParserTransformer.transformParser(ParserTransformer.java:55)
       at org.parboiled.Parboiled.createParser(Parboiled.java:55)
       ... 2 more

If a private rule is called during parser generation, the following exception is thrown:

Exception in thread "main" java.lang.RuntimeException: Error creating extended parser class: Rule method 'rule1'must not be private.
[...]
Caused by: org.parboiled.errors.GrammarException: Rule method 'rule1'must not be private.
Mark the method protected or package-private if you want to prevent public access!
    at org.parboiled.support.Checks.ensure(Checks.java:41)
    at org.parboiled.transform.ClassNodeInitializer.visitMethod(ClassNodeInitializer.java:187)
    at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:911)
    at org.objectweb.asm.ClassReader.accept(ClassReader.java:693)
    at org.objectweb.asm.ClassReader.accept(ClassReader.java:506)
    at org.parboiled.transform.ClassNodeInitializer.process(ClassNodeInitializer.java:97)
    at org.parboiled.transform.ParserTransformer.extendParserClass(ParserTransformer.java:97)
    at org.parboiled.transform.ParserTransformer.transformParser(ParserTransformer.java:66)
    at org.parboiled.Parboiled.createParser(Parboiled.java:67)

There should be a space in "[...]'rule1'must[...]".

enforceVersion mandates 1.7, exactly, even with source version set to 1.7; this means that compilation with gradle fails if the user has 1.8 installed. Should be more lenient IMO.

From IRC:

<Ragnor> Hmmm... That is not a good change -- now my test just hangs. No CPU usage: it just doesn't finish for some reason. <idletask> Can you obtain a stack trace? <idletask> Or maybe it will eventually finish with some StackOverflowError or something... <idletask> (I think I know where this can come from; but I'd need a stack trace) <idletask> Ragnor: that's important -- can I have the input text, please? <idletask> I'll see if I can reproduce the bug here <Ragnor> No stack trace. No nothing. Currently trying to find out the problem in a debugger -- so far no results. <Ragnor> When I halt the program, it doesn't show anything at all. At least nothing related to my program; only Threads within java.*. <idletask> If you can provide the input text of the hanging text I'll try here <Ragnor> Hah! Do you start some Threads maybe? <idletask> Hanging test, I mean <idletask> Yes <Ragnor> Seems my application is exiting from main() but does not terminate. Some non-deamon thread maybe? <idletask> No, not non-daemon <idletask> That's curious, none of my tests hang <idletask> Do you have veeery large inputs? <Ragnor> Not really. It's just a tiny input. Regression tests ;) <Ragnor> (Trying to get a stack dump to show...) <idletask> It's the LineCounter which is launched in an ExecutorService, I think <idletask> And this ExecutorService is a Executors.newCachedThreadPool(); I guess I have to change that <idletask> Oh wait <Ragnor> This is the output of CTRL-\ : http://pastebin.mozilla.org/8263309 <idletask> My threads are non-daemon <idletask> Meh <idletask> I misinterpreted your question <Ragnor> (Maybe I should just System.exit(0) at end of main(). I'd guess this should circumvent the problem for now...) <idletask> Or just use the DefaultInputBuffer... <idletask> Curious however that it hangs like that <Ragnor> idletask: From the stack dump I just pasted: ""pool-1-thread-1" prio=10 tid=0x00007ffcf03e3000 nid=0x1221 waiting on condition [0x00007ffcfcc53000]" -- this line is missing the "daemon" flag. Therefore this thread prevents the JVM from terminating. <idletask> Ragnor: yep, that's a bug in CharSequenceInputBuffer <idletask> Ragnor: I'll fix it for 1.0.1 but in the meantime you'll have to revert to DefaultInputBuffer

Some function names in BaseParser have been renamed and now start with a lower case character. Error messages still refer to the now deprecated function names: The inner rule of ZeroOrMore rule 'file/ZeroOrMore/FirstOf/Sequence/ZeroOrMore' must not allow empty matches This should be changed so that error messages use the exact same names as the (non-deprecated) function names.

This piece of code is one of the more complex in the current code; understanding what it does is all the harder that the test code for this parser consists of... String test outputs.

Which is not a good thing, of course. This means the mechanics are not tested, and they need to be.

Also, a lot of code in matchers themselves, or dedicated visitors, or even input buffers (see MutableInputBuffer), exist solely for the purpose of this runner.

The only benefit that I can see is that it allows to not stop at the first parsing error...

Do you use it at all? If yes, how? Another problem I have is figuring out actual use cases for it!

The following grammar contains three rules. Each of these rules should match inputs "a" or "b", optionally followed by spaces.

rule1 explicitly matches the trailing whitespace with zeroOrMore(sp()). This rule matches the input "a".

rule2 implicitly matches the trailing whitespace by overriding fromStringLiteral(String) and adding zeroOrMore(sp()). That is: "x" matches this string; "x " matches sequence("x", zeroOrMore(sp())). This rule does NOT match the input "a". This is incorrect.

rule3 is the same as rule2 but also matches cr(). This slight change causes the rule to match the input "a".

Note: it seems that the actual change in rule3 is not important: If at least one non-String argument is present, the rule does match the input "a".

The issue seems to be that if all arguments of firstOf() are string literals, the overloaded fromStringLiteral(String) is ignored.

$ gradle
:compileJava
:processResources UP-TO-DATE
:classes
:run
rule1: true
rule2: false
rule3: true

BUILD SUCCESSFUL

Total time: 5.954 secs
$ cat build.gradle
defaultTasks 'run'

apply plugin: 'application'
apply plugin: 'java'

sourceCompatibility = 1.8
mainClassName = 'app.Main'

dependencies {
    compile 'com.github.fge:grappa:2.0.0'
}

repositories {
    mavenCentral()
}
$ cat src/main/java/app/Main.java
package app;

import com.github.fge.grappa.Grappa;
import com.github.fge.grappa.buffers.CharSequenceInputBuffer;
import com.github.fge.grappa.parsers.BaseParser;
import com.github.fge.grappa.rules.Rule;
import com.github.fge.grappa.run.ListeningParseRunner;

public class Main {

    public static void main(final String[] args) {
        final Parser1 parser = Grappa.createParser(Parser1.class);
        final CharSequenceInputBuffer buffer = new CharSequenceInputBuffer("a");
        System.out.println("rule1: "+new ListeningParseRunner<>(parser.rule1()).run(buffer).isSuccess());
        System.out.println("rule2: "+new ListeningParseRunner<>(parser.rule2()).run(buffer).isSuccess());
        System.out.println("rule3: "+new ListeningParseRunner<>(parser.rule3()).run(buffer).isSuccess());
    }

    public static class Parser1 extends BaseParser<Object> {

        public Rule rule1() {
            return firstOf(sequence(fromStringLiteral("a"), zeroOrMore(sp())), sequence(fromStringLiteral("b"), zeroOrMore(sp())));
        }

        public Rule rule2() {
            return firstOf("a ", "b ");
        }

        public Rule rule3() {
            return firstOf("a ", "b ", cr());
        }

        @Override
        protected Rule fromStringLiteral(final String string) {
            return string.endsWith(" ") ? sequence(super.fromStringLiteral(string.substring(0, string.length()-1)), zeroOrMore(sp())) : super.fromStringLiteral(string);
        }

    }

}

anyOf(Characters.ALL) does match even if no more input is available. It should not match because no input character is available and thus no input character could have been consumed.

The behavior of ANY is correct: it does not match if no more input is available.

$ gradle
:compileJava
:processResources UP-TO-DATE
:classes
:run
Parser1: false
Parser2: true

BUILD SUCCESSFUL

Total time: 6.269 secs
$ cat build.gradle
defaultTasks 'run'

apply plugin: 'application'
apply plugin: 'java'

sourceCompatibility = 1.8
mainClassName = 'app.Main'

dependencies {
    compile 'com.github.fge:grappa:2.0.0'
}

repositories {
    mavenCentral()
}
$ cat src/main/java/app/Main.java
package app;

import com.github.fge.grappa.Grappa;
import com.github.fge.grappa.buffers.CharSequenceInputBuffer;
import com.github.fge.grappa.parsers.BaseParser;
import com.github.fge.grappa.rules.Rule;
import com.github.fge.grappa.run.ListeningParseRunner;
import com.github.fge.grappa.support.Characters;

public class Main {

    public static void main(final String[] args) {
        final Parser parser = Grappa.createParser(Parser.class);
        final CharSequenceInputBuffer buffer = new CharSequenceInputBuffer("");
        System.out.println("Parser1: "+new ListeningParseRunner<>(parser.rule1()).run(buffer).isSuccess());
        System.out.println("Parser2: "+new ListeningParseRunner<>(parser.rule2()).run(buffer).isSuccess());
    }

    public static class Parser extends BaseParser<Object> {

        public Rule rule1() {
            return ANY;
        }

        public Rule rule2() {
            return anyOf(Characters.ALL);
        }

    }

}

A rule that is declared "static" causes a "java.lang.VerifyError" during parser generation. Instead Grappa should either support this or else should generate a proper error message.

$ gradle
:compileJava
Download http://repo1.maven.org/maven2/com/github/fge/grappa/2.0.0/grappa-2.0.0.jar
:processResources UP-TO-DATE
:classes
:run
Exception in thread "main" java.lang.VerifyError: Bad local variable type
Exception Details:
  Location:
    app/Main$Parser$$grappa.rule()Lcom/github/fge/grappa/rules/Rule; @0: aload_0
  Reason:
    Type top (current frame, locals[0]) is not assignable to reference type
  Current Frame:
    bci: @0
    flags: { }
    locals: { }
    stack: { }
  Bytecode:
    0x0000000: 2ab4 01a1 59c6 0004 b057 bb00 3059 b700
    0x0000010: 3459 2a5f b501 a12a b701 a359 c600 0b13
    0x0000020: 01a4 b900 4702 005a c000 49b6 004d 592a
    0x0000030: 5fb5 01a1 b0
  Stackmap Table:
    same_locals_1_stack_item_frame(@9,Object[#67])
    full_frame(@39,{},{Object[#48],Object[#67]})

        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2663)
        at java.lang.Class.getConstructors(Class.java:1643)
        at com.github.fge.grappa.Grappa.findConstructor(Grappa.java:111)
        at com.github.fge.grappa.Grappa.createParser(Grappa.java:67)
        at app.Main.main(Main.java:12)
:run FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':run'.
> Process 'command '/usr/local/java/64/jdk1.8.0_31/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 7.967 secs
$ cat build.gradle
defaultTasks 'run'

apply plugin: 'application'
apply plugin: 'java'

sourceCompatibility = 1.8
mainClassName = 'app.Main'

dependencies {
    compile 'com.github.fge:grappa:2.0.0'
}

repositories {
    mavenCentral()
}
$ cat src/main/java/app/Main.java
package app;

import com.github.fge.grappa.Grappa;
import com.github.fge.grappa.buffers.CharSequenceInputBuffer;
import com.github.fge.grappa.parsers.BaseParser;
import com.github.fge.grappa.rules.Rule;
import com.github.fge.grappa.run.ListeningParseRunner;

public class Main {

    public static void main(final String[] args) {
        new ListeningParseRunner<>(Grappa.createParser(Parser.class).rule()).run(new CharSequenceInputBuffer("a"));
    }

    public static class Parser extends BaseParser<Object> {

        public static Rule rule() {
            return ANY;
        }

    }

}

When using firstOf() with only String arguments and if one String has length 1, then parser generation fails with a reference to trie(). This is incorrect for two reasons:

I didn't use trie() but firstOf(). The error message should use the function name that I actually did use, even if Grappa internally maps it.

I cannot think of any reason why one-character strings should be forbidden. Therefore this restriction seems so arbitrary to me that consider it a bug.

/tmp/bug-firstof$ cat build.gradle
defaultTasks 'run'
apply plugin: 'application'
sourceCompatibility = 1.8
mainClassName = 'app.Main'
dependencies { compile 'com.github.fge:grappa:2.0.0-beta.4' }
repositories { mavenCentral() }
/tmp/bug-firstof$ cat src/main/java/app/Main.java
package app;

import com.github.fge.grappa.Grappa;
import com.github.fge.grappa.parsers.BaseParser;
import com.github.fge.grappa.rules.Rule;

public class Main extends BaseParser<Object> {

    public static void main(final String[] args) {
        Grappa.createParser(Main.class).rule();
    }

    public Rule rule() {
        return firstOf("//", "#");
    }

}
/tmp/bug-firstof$ gradle
:compileJava
:processResources UP-TO-DATE
:classes
:run
Exception in thread "main" java.lang.IllegalArgumentException: strings in a trie must be two characters long or greater
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
        at com.github.fge.grappa.matchers.trie.TrieBuilder.addWord(TrieBuilder.java:52)
        at app.Main$$grappa.trie(Unknown Source)
        at app.Main$$grappa.firstOf(Unknown Source)
        at app.Main$$grappa.firstOf(Unknown Source)
        at app.Main.rule(Main.java:14)
        at app.Main$$grappa.rule(Unknown Source)
        at app.Main.main(Main.java:10)
:run FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':run'.
> Process 'command '/usr/local/java/64/jdk1.8.0_31/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 5.13 secs