Major change here is a set of MCTS enhancements to implement various algorithmic variants from the literature. This has involved some significant 'refactoring' of SingleTreeNode in particular. The major design point is that child nodes are no longer kept in an array with a fixed order, but in a Map keyed by the AbstractAction that leads to the node.

MCTSParams has also had a fair amount added to it:

1. SelectionPolicy can be ROBUST or SIMPLE. ROBUST (the old default) picks the final action based on the number of visits, while SIMPLE picks the action with the highest mean value.

2. RolloutsEnabled has been subsumed into RolloutLength. RolloutLength ==0 implies no rollouts.

3. OpenLoop can be FALSE or TRUE. FALSE is the old default, which assumes deterministic environments and does not advance the GameState as the Tree is descended. TRUE now supports OpenLoop MCTS, in which the game state is advanced on every iteration as we extend the tree. (For this reason I have added a copyCount as well as fmCallsCount, and for the moment set the fmCallsBudget to apply to the sum of the two; the old code was advancing fmCallsCount as the tree was descended, even though the FM was never called.)

4. OpponentTreePolicy has 3 values SelfOnly, Paranoid, Paranoid2. SelfOnly means that we ignore our opponents in the Tree, and only construct this using our actions. Paranoid2 is the original version - assuming that opponents are attempting to minimise our score. Paranoid is my version of this - I suspect, subject to feedback/review - that the Exploration bonus was misapplied in Paranoid2.

5. I have split RolloutLength into two parts: MaxTreeDepth and RolloutLength. MaxTreeDepth limits the depth of the tree (you'd never have guessed!), while RolloutLength now applies purely to the rollout outside the tree. This does mean that currently you cannot restrain the sum of the two to a specific value - but it does mean that Monte Carlo Search is now supported with MaxTreeDepth=1, RolloutLength = High.

6. Redeterminise can be TRUE or FALSE. (FALSE is the old functionality). TRUE is a very simple form of Information Set MCTS that redeterminised the root game state used for each iteration. We should expect this to be significantly better on stochastic games.

7. TreePolicy can be any of UCB, AlphaGo, EXP3, RegretMatching. UCB is the standard logic, AlphaGo replaces the form of the UCB exploration term with that used by AlphaGo (instead of sqrt(logN / n) it uses sqrt(N) / (1 + n)). EXP3 and Regret Matching implement the bandit algorithms of the same names instead of UCB. These have an exploration term, which is configured using exploreEpsilon as a new parameter (this is not used outside of these variants).

Added back the heuristics for OSLA, MCTS and RHEA. To use a heuristic simply just pass it in the agent's constructor: players.add(new OSLAPlayer(new PandemicHeuristics()); In the rollouts there are checks if the agent has a heuristic assigned or not. If there is one it uses that for evaluating the game state if not it uses the game's getScore function.

We intend to participate in the COG2022 competition: Pandemic. We want to know whether the configuration of this competition has been determined. For example, how many players are there in one game and whether their roles are fixed or random assigned. Also, is there a evaluation program to give score to the entry? We think this is important as it will help the training of AIs if we can know the true scores.

Hi, the project has a dependency in SolutionEvaluator that needs Java 11 at least. Trying to build it with Java 8 got me this error:

java: cannot access evodef.SolutionEvaluator bad class file: .../.m2/repository/com/github/hopshackle/NTBEA/0.2/NTBEA-0.2.jar!/evodef/SolutionEvaluator.class class file has wrong version 55.0, should be 52.0 Please remove or make sure it appears in the correct subdirectory of the classpath.

Please update that info on the setting up section.

Otherwise really great framework. Congrats!

Currently a heuristic state evaluation function can be provided as par of the MCTSPlayer constructor.

It may be useful to incorporate this into MCTSParams so that the heuristic used can be optimised via NTBEA?

Implementation of Event actions, but:

TODO:

• [x] force action for Resilient Population (PandemicForwardModel.java)
• [ ] GameState.java Forecast in partial observability should leave first N cards visible to player
• [x] actions spawned from Event cards should be free
• [x] should be able to play Event cards any time

I've added another line for each player to the round robin tournament output. This indicates the win rate of each agent in the games it played. Previous shown percentage referred to the number of victories as a percentage of games played in the tournament (independently if the agent played in it or not).

I'm calculating the number of games played per agent in a tournament as:

 int gameCounter = (gamesPerMatchUp * matchUpsRun);
 int gamesPerPlayer = gameCounter * playersPerGame.get(g) / agents.size();

but it'd be good a double-check on this.

⚠️ Dependabot is rebasing this PR ⚠️

Rebasing might not happen immediately, so don't worry if this takes some time.

Note: if you make any changes to this PR yourself, they will take precedence over the rebase.

Bumps junit from 4.12 to 4.13.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Adds the game Stratego to the framework (implemented by Jonny Betts). Some adjustments made (game runs), but it needs further testing on functionality (and potentially some more simplifications are possible).

1. Added ParameterFactory to support Game Parameter initisalisation from json files

• a game parameter file can now be passed to GameReportII

2. Added MultiNTBEA in ParameterSearch

• GameMultiPlayerEvaluator.java; plus see updated --help documentation in ParameterSearch.java, which now defaults to use MulitNTBEA unless an opponent is specified (MultiNTBEA optimises all agents in a game in parallel, potentially avoiding issues with optimising to a trivial, random player)

3. First version of Dice Monastery (v1.8 of rules)

4. Added some standard 'action-based' heuristics: These rely on having a fixed hash code for each Action as a key to look up a simple Advantage (numerical value of 'good' that action is). Think of it as a MAST-style heuristic.

• ActionAdvantageHeuristic - this is initialised from a file of hashCode -> double values (so, fixed)
• LearnedAdvantage - this can be used to learn from one game to the next when using MCTS, learning from the Advantages calculated from the MCTS Tree after each decision is made. This can then be used in Rollout, to decide Node Expansion Order, or as an Opponent Model in later games.

5. Various MCTS variants and parameters added:

• Added 'ExpertIteration' facility to record state information and the associated statistics for each node in the tree - to be used for learning useful stats and policies (currently only used for the ActionAdvantageHeuristic mentioned above...but much space for expansion)
• Added MAST options to MCTS (as in Bjornnson & Finnson 2008). These Action-specific values are keyed by the Action hash code, and can be learned either from the Tree actions, Rollout actions, or both.
• Added AdvantageFunction support that enables a class to be specified to split the order of node expansion
• Added the ability to specify classes for both Opponent Models and Rollout Policies. These must be classes with either a non-arg constructore, and/or a single Stinrg constructor
• Added Progressive Widening to MCTS parameters - which then uses the specified Advantage Function

6. RoundRobinTournament updated to have the same ability as GameReportII to add in Listeners to report on game progress and statistics

This is still work in progress! The game is far along now to request playtesting to hunt for bugs (as it contains a lot of actions and variations of action types and parameters)! Happy to receive reports of any issues, but specifically looking for issues with:

• Crashes when trying to do a specific action
• Being allowed to do an action when you shouldn't be
• Not being allowed to do an action when you should be
• Visual bugs (something not correctly drawn / out of bounds etc.)
• Possible visual improvements (anything that could be better drawn / better interactions possible to make the game more intuitive / easier to play / more accessible)

Any reports along these lines with as much detail as possible to allow reproduction or intuition of the problem would be highly appreciated!

A summary of what's been done follows. Boardgamegeek page for rules etc: https://boardgamegeek.com/boardgame/167791/terraforming-mars

Implemented:

• [x] Base game board, corporations and project cards + corporate era corporations and project cards - using resources from https://ssimeonoff.github.io/ an amazing website for TM!
• [x] Most actions on cards implemented as variations and parameterized options of several action classes
• [x] (hopefully) intuitive GUI for human play
• [x] Winning conditions (though not yet displayed in GUI)
• [x] Solo play (set just 1 player): - start with 14 TR + 0 production (Corporate Era) - game ends after generation 14 (regardless of global parameters) - player wins if Mars is terraformed by the end of the game - setup: X cities + 1 adjacent greenery each are placed randomly on the map on legal positions by neutral player (gives X money production if Tharsis Republic corporation is chosen) - cards: if a card reduces other player's resources/production, they can always be played as if a 2nd player with infinite resources exists (therefore these can still be played for the benefits, and can ignore the negative effects). - not fully play-tested
• [x] Corporate Era 0 starting production
• [ ] Some actions are not yet implemented:
  • [x] Selling projects for 1MC/project
  • [ ] TITANIUM may be used as if... :
  • [ ] STEEL MAY BE USED as if.. :
  • [ ] temporary discounts or effects (next card / this generation only)
  • [ ] but (do something else) if you have ...
• [ ] If the game goes on long enough to use up all the project cards, the deck does not yet reshuffle with the discards.
• [ ] Final Greenery placement

Known issues:

• [x] Bugs in state copies, random player recommended for human testing
• [x] A strange city bug where one player can place infinite cities for free (unkown cause) [Edit: caused by Tharsis Republic corporation, first actions and active actions were not flagged properly as done - now fixed, state saved on the card instead of actions]
• [x] Sometimes resources to pay are not calculated correctly when multiple options are available; reason for this yet unknown, as it only happens in some cases, and this will cause the game to crash as the player will end up in a situation where they can't pay for the action with remaining resources. Similar issue: sometimes paying for a card does not actually play the card, but you lose the resources.
• [x] Actions that would cause specific card choices (e.g. add resource to another card) are technically implemented, but don't work.
• [x] Actions to look at top cards and buy/take them not working
• [x] Action fail: place X>1 tiles (only places the first)
• [x] Effects for money back and discounts on cards sometime not working (corporations ok)
• [x] Bonuses from global parameters don't apply anymore
• [x] Winner and final scoring not diplayed in GUI
• [ ] Actions menu flickering

This includes functionality to automatically learn heuristics from game trajectories using:

• Simple Ordinary Least Squares Regression (OLS)
• Support Vector Machines (SVM)
• Logistic Regression

The entry point to use this is evaluation.ProgressiveLearner.

• This starts with a specified starting heuristic (the default being NULL, which gives random play)
• This runs N games, and then learns a state evaluation heuristic from the trajectories of those games.
• On the next iteration an agent is added in to the growing set that uses the new heuristic.
• This stops after X iterations, and a round robin tournament is run between all the agents to estimate fitness

See documentation in the ProgressiveLearner class for further details.

This requires a State feature space to be defined (one of the inputs to ProgressiveLearner) that implements IStateFeatureVector. See examples in Dots and Boxes, Love Letter and Dominion.

The regression target can be any of WIN, ORDINAL, SCORE. (ORD_SCALE is also possible, which allows logistic regression to be used on the ORDINAL by converting it to a 0..1 range); otherwise Logistic Regression can only be used on a WIN target.

The new players.learners package contains the three OLS/SVM/Logistic learners that can be plugged in,

Well, we intend to participate in COG 2022 competition of Pandemic so we execute the code. However, we discover some bugs and report them here.

1. The diseases can not be cured in one action. To be specific, if we use the antidote or the skill of medic to cure the disease of one colour in a city, only one disease can be removed. However, according to the rules, all these diseases should be removed. We think it is a serious bug as it makes the role of medic useless.
2. The exchange of cards between players is unclear. If the players try to exchange the cards, we don't know who gives the card and who receives the card.
3. The turn order of the games may be disrupted after using the event cards, for example, one player may be given another action step while someone's turn may be skipped.

• Parameter search generalised to any EvoAlg, not only NTBEA
• Adds an enumeration for search frameworks in ParameterSearch class, and a method to construct these appropriately.
• Skeleton for MapElites included, with some methods left open on TODOs. Can specify arbitrary number of behaviours, where each has a name, and a list of possible values (if comparable, the closest one is used for indexing; if not, equality is checked to used for indexing in the map). MapElites implements the EvoAlg interface, with some adjustments.
• A new method added in GameEvaluator which returns not only a value, but also a map from behaviour to value. These behaviour names and value types should match to those defined in ParameterSearch (though no explicit checks for this are included yet).

Better full integration of other optimisers could be expanded in the future (this still assumes model-based approaches and makes some other assumptions MapElites ignores, NTBEA has quite a rigid framework with some odd decisions behind it too to base this on).

Player turn sometimes gets stuck in loop after infecting cities step. After infecting cities the turn goes back to perform player actions and lets one more action be executed before going to infecting cities again. The loop continues until the game is lost.

• The nextNode/childNode of InfectCities node (somehow) gets set to null, causing the AbstractRuleBasedForwardModel to reset the current node to the root node.

The issue appears when playing with different setups of MCTS players. Haven't tested with other types of players. I'm looking into this problem but if anyone got any suggestions I would appriciate it.

This has been bugging me for too long and it should be fixed. Rename TurnOrder to AbstractTurnOrder (consistency with all other abstract core classes) and move into core package.