Code that depends upon Parboiled2 no longer works with Java 16, because encapsulation is now enforced. The following exception is thrown:

Exception in thread "main" java.lang.RuntimeException: Error creating extended parser class: Could not determine whether class 'mypackage.MyParser$$parboiled' has already been loaded
	at org.parboiled.Parboiled.createParser(Parboiled.java:58)
	at javaparse.JavaParsers.benchmarkParboiled1_java(JavaParsers.java:25)
	at squirrel.TestUtils.findMinTime(TestUtils.java:46)
	at squirrel.TestJavaParsing.main(TestJavaParsing.java:17)
Caused by: java.lang.RuntimeException: Could not determine whether class 'mypackage.MyParser$$parboiled' has already been loaded
	at org.parboiled.transform.AsmUtils.findLoadedClass(AsmUtils.java:217)
	at org.parboiled.transform.ParserTransformer.transformParser(ParserTransformer.java:35)
	at org.parboiled.Parboiled.createParser(Parboiled.java:54)
	... 3 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.findLoadedClass(java.lang.String) accessible: module java.base does not "opens java.lang" to unnamed module @3c5a99da
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
	at java.base/java.lang.reflect.Method.setAccessible(Method.java:193)
	at org.parboiled.transform.AsmUtils.findLoadedClass(AsmUtils.java:210)
	... 5 more

The problematic code in org.parboiled.transform.AsmUtils.findLoadedClass is the call to findLoadedClassMethod.setAccessible(true) here:

    public static Class<?> findLoadedClass(String className, ClassLoader classLoader) {
        checkArgNotNull(className, "className");
        checkArgNotNull(classLoader, "classLoader");
        try {
            Class<?> classLoaderBaseClass = Class.forName("java.lang.ClassLoader");
            Method findLoadedClassMethod = classLoaderBaseClass.getDeclaredMethod("findLoadedClass", String.class);

            // protected method invocation
            findLoadedClassMethod.setAccessible(true);
            try {
                return (Class<?>) findLoadedClassMethod.invoke(classLoader, className);
            } finally {
                findLoadedClassMethod.setAccessible(false);
            }
        } catch (Exception e) {
            throw new RuntimeException("Could not determine whether class '" + className +
                    "' has already been loaded", e);
        }
    }

Note that to even get this far you have to override the version of ASM depended upon by Parboiled to one that works with JDK 16:

    <dependency>
      <groupId>org.parboiled</groupId>
      <artifactId>parboiled-java</artifactId>
      <version>1.3.1</version>
      <scope>test</scope>
      <exclusions>
        <exclusion>
          <artifactId>asm</artifactId>
          <groupId>org.ow2.asm</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.ow2.asm</groupId>
      <artifactId>asm</artifactId>
      <version>9.1</version>
      <scope>test</scope>
    </dependency>

Java 15 is dramatically faster than previous JVMs, and Java 16 has record types, so Java 16 will fast become a new baseline for many Java projects. Parboiled is still a dependency for a lot of old code. It would be nice if Parboiled could be updated to not use introspection in this way so that it works with Java 16+.

I've managed to "hack" this using some @lukasraska insights. I've wanted to preserve old behavior so I've split it into two (before JDK16 and after). People with JDK16+ will need to add "unused" methods to their parser classes but I hope we could add info about it to wiki or maybe even throw a runtime exception that those methods are missing.

    public static Class<?> findLoadedClass(String className) throws IllegalAccessException {
        try {
            return MethodHandles.lookup().findClass(className);
        } catch (ClassNotFoundException e) {
            return null;
        }
    }

    public static Class<?> loadClass(byte[] code) throws IllegalAccessException {
        return MethodHandles.lookup().defineClass(code);
    }

This way we can buy as some time to think about better fix ;) What do you think?

I've tested this using my parser and snapshot of this PR. It works well on JDK17. I'll attach jar for people to test :)

https://ufile.io/2snwwjao

WARNING: Illegal reflective access by org.parboiled.transform.AsmUtils (file:/home/aaime/.m2/repository/org/parboiled/parboiled-java/1.1.7/parboiled-java-1.1.7.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
WARNING: Illegal reflective access by org.parboiled.transform.AsmUtils (file:/home/aaime/.m2/repository/org/parboiled/parboiled-java/1.1.7/parboiled-java-1.1.7.jar) to method java.lang.ClassLoader.findLoadedClass(java.lang.String)

I'm going to upgrade to 1.2.0 soon, but checked the code, the access seems to be still there:

• https://github.com/sirthias/parboiled/blob/master/parboiled-java/src/main/java/org/parboiled/transform/AsmUtils.java#L240
• https://github.com/sirthias/parboiled/blob/master/parboiled-java/src/main/java/org/parboiled/transform/AsmUtils.java#L207

I’ve updated the master branch to fully utilize the latest version of ASM.

This mostly entails properly configuring ASM apis and removing calls to and implementations of deprecated apis.

I do seem to be having challenges with later versions of the JDK I’m hoping to resolve, but wanted to make my current progress available for comment.

Note I bumped the version to 1.2 to simplify things, it’s not a comment on the intended release version.

Hi there,

We have a problem in our build caused by duplication of classes in parboiled-core and parboiled-java (see http://jira.xwiki.org/browse/XRENDERING-297?focusedCommentId=77508&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-77508 ).

Specifically:

[INFO] --- maven-duplicate-finder-plugin:1.0.3:check (default) @ xwiki-enterprise-web ---
[INFO] Checking compile classpath
[INFO] Checking runtime classpath
[WARNING] Found duplicate classes in [org.parboiled:parboiled-core:1.1.5,org.parboiled:parboiled-java:1.1.5] :
[WARNING]   org.parboiled.Action
[WARNING]   org.parboiled.Context
[WARNING]   org.parboiled.ContextAware
[WARNING]   org.parboiled.MatchHandler
[WARNING]   org.parboiled.MatcherContext
[WARNING]   org.parboiled.Node
[WARNING]   org.parboiled.NodeImpl
[WARNING]   org.parboiled.ParserStatistics
[WARNING]   org.parboiled.Rule
[WARNING]   org.parboiled.SkippableAction
[INFO] ------------------------------------------------------------------------

Parboiled-java draws parboiled-core:

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ xwiki-rendering-syntax-markdown10 ---
[INFO] org.xwiki.rendering:xwiki-rendering-syntax-markdown10:jar:5.2-SNAPSHOT
[INFO] +- org.pegdown:pegdown:jar:1.4.1:compile
[INFO] |  \- org.parboiled:parboiled-java:jar:1.1.5:compile
[INFO] |     +- org.parboiled:parboiled-core:jar:1.1.5:compile

Would it be possible that you fix this issue? :)

Thanks!

The current implementation of MemoMismatchesMatcher incorrectly stores the state of the match as a mutable field of the Matcher. This means that consecutive (or concurrent) parse runs using the same matchers will possibly fail to match at positions where they should.

This patch corrects this by storing the memo in the MatcherContext.

Hallo Mathias,

this change represents a cleaned up version of the code that allows passing of action arguments by marking them with @Var rather than wrapping each variable by an instance of org.parboiled.support.Var<T>.

Using the @Var annotation the code of VarFramingTest looks like

        public Rule Clause() {
            Integer a;
            return Sequence(
                    DO(a = -1),
                    Digits(), DO(a = peek()),
                    SomeRule(a, 2),
                    Optional(
                            Sequence(
                                    '+',
                                    Clause(), push(a)
                            )
                    )
            );
        }

        @SuppressNode
        public Rule Digits() {
            return Sequence(
                    OneOrMore(CharRange('0', '9')),
                    push(Integer.parseInt(match()))
            );
        }
        
        public Rule SomeRule(@Var Integer var, int ruleArg) {
            return toRule(DO(var == count++));
        }

instead of

        public Rule Clause() {
            Var<Integer> a = new Var<Integer>(-1);
            return Sequence(
                    Digits(), a.set(peek()),
                    SomeRule(a),
                    Optional(
                            '+',
                            Clause(), push(a.get())
                    )
            );
        }

        @SuppressNode
        public Rule Digits() {
            return Sequence(
                    OneOrMore(CharRange('0', '9')),
                    push(Integer.parseInt(match()))
            );
        }

        public Rule SomeRule(Var<Integer> var) {
            return toRule(var.get() == count++);
        }

The semantics of annotated action parameters are aligned with the semantics of "normal" method parameters in Java. Hence they do not suffer from restrictions imposed by using generic types in Java:

SomeRule(Var<Integer> var) -> can only be called with type Var<Integer>
SomeRule(Var<Number> var) -> can only be called with type Var<Number>
SomeRule(Var<? extends Number> var) -> can be called with Var<T> where T extends Number, but var.set(...) is prohibited

Maybe - based on this exemplary implementation - your are interested in discussing the incorporation of annotated action parameters into parboiled with me?

Best regards,

Ken

Hi, I wrote the missing POM sections to have this project eligible for deployment to Maven Central repository. I tried to respect as much as possible what is done by Buildr's buildfile. To build the project, just run:

mvn clean install

Now this should make releases a lot simpler for you, as all you need to do to release the project is:

mvn release:prepare
mvn release:perform
mvn release:clean

This will do the git tag and push it, build the main artifacts, source and javadoc archives and deploy them to scala-tools repository.

All you need to do is to add your scala-tools credentials in your Maven settings.xml this way:

<servers>
  <server>
    <id>nexus.scala-tools.org</id>
    <username>your username</username>
    <password>your password</password>
  </server>
</servers>

(See corresponding distribution management section here)

• ensures compatibility with JDK 16 and newer
• gets lookup instance through static lookup() method on parser class
• uses sun.misc.Unsafe as another option to create trusted lookup (requires no changes to existing parser classes)

Fixes #175

Seems to be a copy/paste error in the documentation on page

https://github.com/sirthias/parboiled/wiki/Action-Variables

in the example. Seems

Rule Verbatim() {
    StringVar text = new StringVar("");
    StringVar temp = new StringVar("");
    return Sequence(
        StringVar text = new StringVar("");
        StringVar temp = new StringVar("");
        return Sequence(
            OneOrMore(
                ZeroOrMore(BlankLine(), temp.append("\n")),
                NonblankIndentedLine(), text.append(temp.getAndSet(""), pop().getText())
            ),
            push(new VerbatimNode(text.get()))
        );
    );
}

just has doubled copy of the first few lines.

Hi Sirthias,

Thanks a lot for the wonderful framework. I am trying it out for one of my upcoming project.

I am writing the following code in a Action class.

public boolean run(Context cntxt) { System.out.println("okfsokfokfsdkf"); System.out.println("Get first match="+cntxt.getFirstMatchChar()); System.out.println("Get match="+cntxt.getMatch()); //System.out.println("Get sub node 0="cntxt.getSubNodes()); List<Node> nodeList = cntxt.getSubNodes(); for(Node node : nodeList){ // System.out.println("Value:"+node.getChildren().get(0)); for(int i=0;i< node.getChildren().size();i++){ Node cnode = (Node)node.getChildren().get(i); System.out.println("Cnode Value:"+cnode.getValue()); }

    }
    return true;      
}

when I try to print the value of node using cnode.getValue() it prints null. cnode.getLabel() is working fine. Here is my parse tree:

okfsokfokfsdk Get first match=i Get match=i="3" Cnode Value:null Cnode Value:null [VariableDeclarator] 'i="3"' [Sequence] 'i="3"' [Identifier] 'i' [StringLiteral] '"3"' [ZeroOrMore] '3'

Thanks & Regards, Abhishek

Hello,

I use parboiled since a long time to parse some routers configuration files. I've got a lot of test cases to test the parsers results. With version 1.4.1 few test cases fail when, in the parser rules, i use string concatenation wih the '+' operator. If I replace the '+' operator with the String.concat method this works fine.

Exemple:

public Rule AceTcpOldFlag() {
return
	Sequence(
		FirstOf(
			String("ack"),
			String("fin"),
			String("psh"),
			String("rst"),
			String("syn"),
			String("urg")
		),
		_ace.getTcpFlags().add("+" + match()), // ace.getTcpflags is a String list
		_ace.setTcpKeyword("match-any"),
		SkipSpaces()
	);
}

In a test case for this rule like "ack rst psh fin syn urg", flags are set with the match() value, (ie ack, fin, psh etc), but without the "+" at the start as if the string + operator does nothing. If I replace the + operator with String.concat it works fine : _ace.getTcpFlags().add("+".concat(match()))

I tried with jdk11, jdk16 with the same problem. Also tried to use asm 9.3 instead asm 9.2. All works fine if I specify 8 in the maven compiler plugin.

I think there is something wrong with the String + operator and ASM but I cannot reproduce this problem with a simple test parser Any clue?

Thanks regards,

Hi, base on the CalculatorParser3 example I create a parser to evaluate expressions like

key=='abc'

It works fin in java. However, when I converted the code to kotlin, then I see an error:

Exception in thread "main" java.lang.RuntimeException: Error creating extended parser class: An ACTION or Var initializer in rule method 'Identifier' contains illegal writes to a local variable or parameter
	at org.parboiled.Parboiled.createParser(Parboiled.java:58)
	at pl.tfij.copy.CalculatorParser3copy$Companion.main(CalculatorParser3copy.kt:144)
	at pl.tfij.copy.CalculatorParser3copy.main(CalculatorParser3copy.kt)
Caused by: org.parboiled.errors.GrammarException: An ACTION or Var initializer in rule method 'Identifier' contains illegal writes to a local variable or parameter
	at org.parboiled.support.Checks.ensure(Checks.java:37)
	at org.parboiled.transform.InstructionGroupCreator.verify(InstructionGroupCreator.java:135)
	at org.parboiled.transform.InstructionGroupCreator.process(InstructionGroupCreator.java:58)
	at org.parboiled.transform.ParserTransformer.runMethodTransformers(ParserTransformer.java:62)
	at org.parboiled.transform.ParserTransformer.extendParserClass(ParserTransformer.java:45)
	at org.parboiled.transform.ParserTransformer.transformParser(ParserTransformer.java:39)
	at org.parboiled.Parboiled.createParser(Parboiled.java:54)
	... 2 more

Any idea why?

parboiled-java version : 1.4.1

My java code:

package pl.tfij;

import org.parboiled.BaseParser;
import org.parboiled.Parboiled;
import org.parboiled.Rule;
import org.parboiled.annotations.BuildParseTree;
import org.parboiled.parserunners.RecoveringParseRunner;
import org.parboiled.support.ParsingResult;
import org.parboiled.trees.ImmutableBinaryTreeNode;

import static org.parboiled.errors.ErrorUtils.printParseErrors;
import static pl.tfij.CalculatorParser3.CalcNode;

@BuildParseTree
public class CalculatorParser3 extends BaseParser<CalcNode> {

    public Rule InputLine() {
        return Sequence(Expression(), EOI);
    }

    Rule Expression() {
        return EqualityExpression();
    }

    Rule EqualityExpression() {
        return FirstOf(
                Sequence(
                        Identifier(),
                        "== ",
                        StringLiteral(),
                        push(new CalcNode("==", pop(1), pop()))
                ),
                Sequence(
                        Identifier(),
                        "!= ",
                        StringLiteral(),
                        push(new CalcNode("!=", pop(1), pop()))
                )
        );
    }

    Rule Identifier() {
        return Sequence(
            Sequence(
                OneOrMore(
                        Letter(),
                        ZeroOrMore(FirstOf(Letter(), Digit()))
                ),
                WhiteSpace()
            ),
            push(new CalcNode("Identifier", match().trim()))
        );
    }

    Rule Letter() {
        return FirstOf(CharRange('a', 'z'), CharRange('A', 'Z'), '_', '$');
    }

    Rule StringLiteral() {
        return Sequence("'", StringContent(), "'", WhiteSpace());
    }

    Rule StringContent() {
        return Sequence(
                ZeroOrMore(Sequence(TestNot(AnyOf("\r\n'")), FirstOf(EscapedChar(), ANY))),
                push(new CalcNode("String", escapeString(matchOrDefault(""))))
        );
    }

    protected String escapeString(String string) {
        StringBuilder result = new StringBuilder();
        var i = 0;
        while (i < string.length()) {
            if (string.charAt(i) == '\\') {
                i++;
            }
            result.append(string.charAt(i));
            i++;
        }
        return result.toString();
    }

    Rule EscapedChar() {
        return Sequence("\\", ANY);
    }

    Rule Digit() {
        return CharRange('0', '9');
    }

    Rule WhiteSpace() {
        return ZeroOrMore(AnyOf(" \t\f"));
    }

    // we redefine the rule creation for string literals to automatically match trailing whitespace if the string
    // literal ends with a space character, this way we don't have to insert extra whitespace() rules after each
    // character or string literal

    @Override
    protected Rule fromStringLiteral(String string) {
        return string.endsWith(" ") ?
                Sequence(String(string.substring(0, string.length() - 1)), WhiteSpace()) :
                String(string);
    }

    //****************************************************************

    /**
     * The AST node for the calculators. The type of the node is carried as a Character that can either contain
     * an operator char or be null. In the latter case the AST node is a leaf directly containing a value.
     */
    public static class CalcNode extends ImmutableBinaryTreeNode<CalcNode> {
        private Object value;
        private String type;

        public CalcNode(String type, Object value) {
            super(null, null);
            this.type = type;
            this.value = value;
        }

        public CalcNode(String type, CalcNode left, CalcNode right) {
            super(left, right);
            this.type = type;
        }

        public Object getValue() {
            switch (type) {
                case "==":
                    return left().getValue().equals(right().getValue());
                case "!=":
                    return !left().getValue().equals(right().getValue());
                case  "Identifier":
                    return "abc"; //TODO
                case "String":
                    return value;
                default:
                    throw new IllegalStateException(type);
            }
        }

        @Override
        public String toString() {
            return (type == null ? "Value " + value : "Operator '" + type + '\'') + " | " + getValue();
        }
    }

    //**************** MAIN ****************

    public static void main(String[] args) {
        CalculatorParser3 parser = Parboiled.createParser(CalculatorParser3.class);
        String input = "key=='abc'";
        ParsingResult<CalcNode> result = new RecoveringParseRunner<CalcNode>(parser.InputLine()).run(input);

        if (result.hasErrors()) {
            System.out.println("\nParse Errors:\n" + printParseErrors(result));
        }

        CalcNode value = result.resultValue;

        System.out.println("value: " + value.getValue());
    }
}

and Kotlin version:

package pl.tfij

import org.parboiled.BaseParser
import org.parboiled.Parboiled
import org.parboiled.Rule
import org.parboiled.annotations.BuildParseTree
import org.parboiled.errors.ErrorUtils
import org.parboiled.parserunners.RecoveringParseRunner
import org.parboiled.trees.ImmutableBinaryTreeNode
import pl.tfij.copy.CalculatorParser3copy

@BuildParseTree
open class CalculatorParser3copy : BaseParser<CalculatorParser3copy.CalcNode?>() {
    open fun InputLine(): Rule {
        return Sequence(Expression(), EOI)
    }

    open fun Expression(): Rule {
        return EqualityExpression()
    }

    open fun EqualityExpression(): Rule {
        return FirstOf(
            Sequence(
                Identifier(),
                "== ",
                StringLiteral(),
                push(CalcNode("==", pop(1), pop()))
            ),
            Sequence(
                Identifier(),
                "!= ",
                StringLiteral(),
                push(CalcNode("!=", pop(1), pop()))
            )
        )
    }

    open fun Identifier(): Rule {
        return Sequence(
            Sequence(
                OneOrMore(
                    Letter(),
                    ZeroOrMore(FirstOf(Letter(), Digit()))
                ),
                WhiteSpace()
            ),
            push(CalcNode("Identifier", match().trim()))
        )
    }

    open fun Letter(): Rule {
        return FirstOf(CharRange('a', 'z'), CharRange('A', 'Z'), '_', '$')
    }

    open fun StringLiteral(): Rule {
        return Sequence("'", StringContent(), "'", WhiteSpace())
    }

    open fun StringContent(): Rule {
        return Sequence(
            ZeroOrMore(Sequence(TestNot(AnyOf("\r\n'")), FirstOf(EscapedChar(), ANY))),
            push(CalcNode("String", escapeString(matchOrDefault(""))))
        )
    }

    protected fun escapeString(string: String): String {
        val result = StringBuilder()
        var i = 0
        while (i < string.length) {
            if (string[i] == '\\') {
                i++
            }
            result.append(string[i])
            i++
        }
        return result.toString()
    }

    open fun EscapedChar(): Rule {
        return Sequence("\\", ANY)
    }

    open fun Digit(): Rule {
        return CharRange('0', '9')
    }

    open fun WhiteSpace(): Rule {
        return ZeroOrMore(AnyOf(" \t\u000c"))
    }

    // we redefine the rule creation for string literals to automatically match trailing whitespace if the string
    // literal ends with a space character, this way we don't have to insert extra whitespace() rules after each
    // character or string literal
    override fun fromStringLiteral(string: String): Rule {
        return if (string.endsWith(" ")) Sequence(
            String(string.substring(0, string.length - 1)),
            WhiteSpace()
        ) else String(string)
    }
    //****************************************************************
    /**
     * The AST node for the calculators. The type of the node is carried as a Character that can either contain
     * an operator char or be null. In the latter case the AST node is a leaf directly containing a value.
     */
    class CalcNode : ImmutableBinaryTreeNode<CalcNode?> {
        private var value: Any? = null
        private var type: String?

        constructor(type: String?, value: Any?) : super(null, null) {
            this.type = type
            this.value = value
        }

        constructor(type: String?, left: CalcNode?, right: CalcNode?) : super(left, right) {
            this.type = type
        }

        fun getValue(): Any? {
            return when (type) {
                "==" -> left()!!.getValue() == right()!!.getValue()
                "!=" -> left()!!.getValue() != right()!!.getValue()
                "Identifier" -> "abc" //TODO
                "String" -> value
                else -> throw IllegalStateException(type)
            }
        }

        override fun toString(): String {
            return (if (type == null) "Value $value" else "Operator '$type'") + " | " + getValue()
        }
    }

    companion object {
        //**************** MAIN ****************
        @JvmStatic
        fun main(args: Array<String>) {
            val parser = Parboiled.createParser(
                CalculatorParser3copy::class.java
            )
            val input = "key=='abc'"
            val result = RecoveringParseRunner<CalcNode>(parser.InputLine()).run(input)
            if (result.hasErrors()) {
                println(
                    """
    Parse Errors:
    ${ErrorUtils.printParseErrors(result)}
    """.trimIndent()
                )
            }
            val value = result.resultValue
            println("value: " + value.getValue())
        }
    }
}

Was reading today about parser error handling, and noticed many of the links to decodified.com were failing to resolve on my browser.

https://github.com/sirthias/parboiled/wiki/Parse-Error-Handling

Thanks Bruno

Hi, I don't know if the project is still active.

I am an user of Parboiled and I would like to give an hand in maintaining or upgrading the framework.

Sylvain

A simplified example of my grammar:

OfferResponse    <- SerialNumber OfferIndex{1, 10}
OfferIndex       <- [0-9]{2}

How can we represent :

1. [0-9]{2}
2. OfferIndex{1, 10}

PS: A Discord, Slack, Gitter, or Mattermost channel for the community would be great.