• Add Eclipse Collections as a dependency
• Add a flag to generate asEcList() method
• Made Selectors extend EC Function so that uses have a choice to use Reladomo selectors with Eclipse Collections

Hi, Sorry for posting my question in the issue section, but i had an issue that i am stuck at for some time now so wanted to check if anybody could kindly provide any help here. We are working on creating a purge application that runs throughout the day. We fetch objects using the listCursor and for each x rows, check if they meet certain criteria and delete. We stop purge and also stop fetching data using the cursor(but do not close the cursor ,effectively pausing the cursor )if the database is busy or if other processes are running. I was looking to find a way such that the read from the listCursor would be in readOnly mode so it does not hold on to any locks(this is currently blocking other processes even thought the cursor is paused and not fetching any data) and the delete's can happen in the same JVM. Since we declare txParticpation at the connection level, it would not let me do it currently. Could someone please suggest if there is a way i can specify txParticpation at query level or if could have 2 connection managers for an application so i could use the listCursor in readOnly mode and run the delete's using the second connection manager or if there is any other way to tackle this problem. Please let me know if you need any other details.

Thanks

I want to completely disable caching - I know this will be a performance hit but our table sizes are small and we'll re-enable the caching once we configure the TCP Notification Server and have tested it in our environment/ with redundancy etc.

So, to disable the caching I set cacheType = none as follows and sure enough I see all application finds going to the db (sql logs and profiler show that)

 <MithraObjectConfiguration
  className="com.greenhedges.Account" cacheType="none"/>

 <MithraObjectConfiguration
  className="com.greenhedges.AccountAgreement" cacheType="none"/>

My issue is the one-to-many relationship are not refetched/queried from the db each time - e.g. say an Account can have n number of AccountAgreements:

<MithraObject objectType="transactional">
<PackageName>com.greenhedges</PackageName>
    <ClassName>Account</ClassName>
    <DefaultTable>account</DefaultTable>
...
    <Relationship name="aggreements" relatedObject="AccountAgreement" cardinality="one-to-many" relatedIsDependent="true">
         AccountAgreement.id = this.id
    </Relationship>
</MithraObject>

Is there a way to force the relatedObjects to be reloaded always?

Getting below exception when tried to join table with equalsEdgePoint().

Sample Reladomo Query: DocumentFinder.authors().processingDate().equalsEdgePoint() .and(DocumentFinder.authors().processingDateTo().greaterThan())

Exception : can't join with non-standard as-operation com.gs.fw.common.mithra.finder.asofop.AsOfEdgePointOperation processingDate

There are couple of open issues I want to consult with @gs-rezaem . Please kindly advise.

• In #30 , Moh suggested to try getting rid of GSC impl dependency, but there are too many dependencies for FastList/UnifiedSet/UnifiedMap as internal storage, so I left them as is for now.
• In #30, Moh suggested below for ByteArraySet, but this caused conflict in get(Object) and get(K), also signature conflict in collect(). So I didn't touch this class for now. Please advise how you want to handle this class.

For ByteArraySet, extend the EC UnifiedSetWithHS and implement GSC MutableMap.

• MithraFastList and AdhocFastList extend FastList. Please advise how you want to handle these classes.
• AbstractDatedCache.MatchAllAsOfDatesProcedureForMany#getResult() returns FastList. Please advise how you want to handle this.
• MasterSyncResult#getBuffers() returns FastList. Please advise how you want to handle this.
• MithraCompositeList#getLists() returns FastList. Please advise how you want to handle this.
• AsOfEqualityChecker#getMsiPool() returns UnifiedMap. Please advise how you want to handle this.
• ConcurrentIntObjectHashMap#parallelForEachValue() takes List<IntObjectProcedure<V>> in its signature. Introducing EC equivalent method would cause same erasure issue. Please advise how you want to handle this.
• ListFactory has couple of create() methods that return FixedSizeList. We can either have GSC version or EC version for this class. Making it EC version may cause wide impact as it's used widely in reladomo. Please advise how you want to handle this class.

Hi,

I've been reviewing Reladomo for use in a project but see nothing about multi-tenancy.

I plan on implementing shared schema tenancy by using row level security in PostgreSQL. My original plan before coming across Reloadomo was to set a connection parameter when the connection was grabbed for use. This was to be used in PostgreSQL via a table policy that checked the value compared the the column value stored essentially creating a virtual database.

I think I can do the PostgreSQL part still and the connection parameter by overriding the getConnection in say the SourcelessConnectionManager.

Where I get confused is with the Caching in Reladomo.

I've read what I can on the caching and It would appear that there is a collection of indices and that matching an object in the cache must be done via a match with "=" or "in". So if the tenant id was part of an index this may work?

Is there a way I can do what I want to do? How would one use Reladomo for multi-tenancy?

Any help would be greatly appreciated.

Hi,

https://goldmansachs.github.io/reladomo/objectxmlgenerator/Generator.html describes that the Reladomo Mithra XML generator (reverse generation from an existing database) is capable to connect "sybase" and "udb82" databases. Any "hard" reasons why this is the case ? Or asked differently - what is required to be able to connect e.g. postgresql ?

Thank you. Best Regards, jj

Tabs are there in entity source generated by reladomo. I found the indentation is hardcoded here, is it good to raise a pull request to change to spaces? or is there any other approach to the matter? Thanks.

case1: codeFormat = default value (fast)

here is the source of build.gradle:

task genReladomo {
    ant.taskdef(name: "genReladomo",
            classpath: configurations.reladomoGenTask.asPath,
            classname: "com.gs.fw.common.mithra.generator.MithraGenerator")
    ant.genReladomo(xml: "$projectDir/src/main/resources/reladomo/model/ReladomoClassList.xml",
            generateEcListMethod: "true",
            generatedDir: "$buildDir/generated-sources/reladomo",
            nonGeneratedDir: "$projectDir/src/main/java")
}

and here is one of the sources generated, with tabs as indentation.

public class ChannelList extends ChannelListAbstract
{
	public ChannelList()
	{
		super();
	}

	public ChannelList(int initialSize)
	{
		super(initialSize);
	}

	public ChannelList(Collection c)
	{
		super(c);
	}

	public ChannelList(Operation operation)
	{
		super(operation);
	}
}

case2: codeFormat = none

here is the source of build.gradle, with parameter codeFormat specified as "none".

task genReladomo {
    ant.taskdef(name: "genReladomo",
            classpath: configurations.reladomoGenTask.asPath,
            classname: "com.gs.fw.common.mithra.generator.MithraGenerator")
    ant.genReladomo(xml: "$projectDir/src/main/resources/reladomo/model/ReladomoClassList.xml",
            generateEcListMethod: "true",
            codeFormat: "none",
            generatedDir: "$buildDir/generated-sources/reladomo",
            nonGeneratedDir: "$projectDir/src/main/java")
}

indentation are gone, but 1 tab is there in another file generated, in the next line of "{".

public class ChannelDatabaseObject extends ChannelDatabaseObjectAbstract
{
	

}

Hi,

I would be grateful if you could please review and consider accepting this bug fix, or let me know if you have any questions / feedback / changes required.

This fix is for a race condition in deep fetch which causes inconsistent related objects to be returned in certain cases when concurrent updates arrive via notification events.

This happens due to stale relationship query results in the partial cache that do not expire as they should, because the cache entries (CachedQuery objects) are stamped with update counters that were captured at the wrong time (i.e. after the database query completed, instead of before).

I've added test cases which reproduced the issue and made these pass. An earlier version of this fix has also been tested in a real world application for over a month without any repeat of the original issue, which had previously been happening on a daily basis.

The code changes should largely be self-explanatory but note that some additional classes had to be refactored slightly just to make it possible to inject a test harness around the deep fetch strategy under the test. I've also included a detailed step by step walkthrough below in case this helps to explain the mode of failure, since it was quite subtle and depended upon a chain of events. I also explained a bit more about the rationale behind the fix.

Looking forward to your feedback.

Thanks in advance, Andrew

Detailed walkthrough of the issue and the approach to fix it:

As an example to provide some context, suppose we have a table A which has a relationship to table B. We retrieve Table A using findMany, and we request a deep fetch of the related objects in table B. Both tables are partial-cached in Reladomo. We have two Reladomo JVMs running concurrently which are linked via notification events. One of these JVMs is performing read queries including deep fetch, and the other JVM is performing writes to the same tables and sending corresponding notification events.

The race condition can occur if concurrent updates happen which bring additional parent objects (in table A) into scope of the finder query at the same time as the related objects (in table B) are being deep fetched from the database. The results of this database query on table B may not reflect the concurrent update to table A - in some cases because the query was optimised using an in-clause that was computed based on the original contents of table A, or because of a database join that got executed just prior to table A being updated. Note that the results of this deep fetch are actually correct, but the cache entry that is created for the related objects is tagged with the wrong update counter values and this causes the next deep fetch that happens later to return the wrong results.

Specifically, the deep fetch strategy submits these results for the related objects (of table B) into the partial cache as a CachedQuery which it incorrectly stamps with the latest update counter values that were captured after the query was executed. This causes the CachedQuery to report that it is not expired because the update counter incorporates the latest notification events, even though the cached results do not reflect the latest state of the database.

If the same finder query and deep fetch is subsequently executed again by the application at a later time, the additional parent objects (in table A) which came into scope will get returned by the findMany as expected, because it will requery table A in the database, but the corresponding related objects (in table B) will be missing from the deep fetch results because it will return the stale results from the cache.

The fix is relatively straightforward. The deep fetch strategy should capture the update counters before the related object query is executed, rather than afterwards. This will cause the CachedQuery to correctly detect that it has expired, so there will be no cache hit for the related objects query, and the deep fetch strategy will correctly re-retrieve the related objects from the database, using the correct join (or in-clause) based on the latest set of parent objects, and therefore the deep fetch will return the complete set of related objects as expected.

There is one other related type of race condition. In certain parts of the deep fetch logic we also need the related objects query to expire if the parent query has invalidated, since the change to the parent table may bring new objects into scope of the deep fetch, e.g. if the deep fetch strategy executed the related objects query as a 'simplified' query that uses an in-clause instead of a join. The fix for this issue is to enhance CachedQuery to be able to merge the update counters of the parent query with the update counters of the related objects query, taking account of the fact that these are two different operations which depend on different tables and hence depend on different update counter holders. This fix is also included as part of this pull request.

There is no way to provide sql schema name for MithraObjectXmlGenerator (because of initializeSchemaOrCatalog method logic, "schemaName" variable is always null for SqlServer).

SQL server database always contains system views and tables, so running MithraObjectXmlGenerator leads to processing all those system views and tables. In my sample DB there are 400 system views (INFORMATION_SCHEMA.* and sys.*) for 20 user tables, so the output of generator is nearly unusable (and on SqlServer 2012 generator actually throws exception trying to process sys.trace_xe_action_map table)

The work around is to provide "includeTables" parameter, but for real db that list would be very long.

Bumps h2 from 1.4.192 to 2.1.210.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

After upgrading to the latest version of Reladomo, I see the new attribute quoting behavior. I'm generating ddl files to disk using the postgres database type, and then I load them into h2 inside tests. I realize this might not be a valid thing to do, but it's always worked in the past and CoreMithraDbDefinitionGenerator#setDatabaseType doesn't have an option for h2. Should it?

The quoting is happening a little differently in ddl files, idx files, and fk files.

In ddl files, the quotes appear escaped with slashes. H2 doesn't seem to like this syntax.

drop table if exists MYTYPE;

create table MYTYPE
(
    \"name\" varchar(256) not null,
    data varchar(100000) not null,
);

In fk files, there's no quoting. H2 is fine with this, at least for the identifier "name".

references MYTYPE(
    name
);

In idx files, there's quoting with no slashes. H2 is fine with this too.

alter table MYTYPE add constraint MYTYPE_PK primary key ("name");

I stepped through AbstractGeneratorDatabaseType a bit and see the places where getColumnNameWithEscapedQuote() and getPlainColumnName() are called, and I'm happy to help with a fix, but I'm not confident I understand the issue.

I cloned reladomo code and imported the project into IntelliJ/IDEA.

See the build errors below.

missing libraries.

import org.eclipse.collections.api.list.FixedSizeList; import org.eclipse.collections.impl.factory.Lists;

Where is the JAR for above 2 Java classes located? Could you someone provide instructions on how to build the reladomo project in IntelliJ/IDEA?

in the reladomo-sample-simple project, there is code like below

public class ObjectSequence extends ObjectSequenceAbstract implements MithraSequence {

where is the java class ObjectSequenceAbstract come from? I do not see it in master branch?

If I update the existing XML schema file. The newly generated Finder class will generate a class with a new serialVersionId and thus breaking the Backward compatibility. https://github.com/goldmansachs/reladomo/blob/master/reladomogen/src/main/templates/readonly/Finder.jsp#L783

Reladomo's sql logging is so useful that I sometimes leave it turned on in production.

DEBUG com.gs.fw.common.mithra.sqllogs.MyClass: connection:1030722971 find with: select t0.key,t0.number from MY_CLASS t0 where  t0.key = 'abcd' and t0.system_to = '9999-12-01 23:59:00.000'
DEBUG com.gs.fw.common.mithra.sqllogs.MyClass: retrieved 1 objects, 2.0 ms per

I have some ideas on how to make this logging even more useful.

It's currently hard to log only sql, or only timings, without the other. That's because both log statements using the same logger, com.gs.fw.common.mithra.sqllogs.MyClass in this example. I propose adding markers for each.

private static final Marker MARKER_SQL = MarkerFactory.getMarker("Reladomo SQL");
private static final Marker MARKER_TIMING = MarkerFactory.getMarker("Reladomo Timing");

And adding the markers to the relevant log statements. For example, change this:

logger.debug("source '" + source + "': " + text);

to

logger.debug(MARKER_SQL, "source '" + source + "': " + text);

In addition, the timing information could be even more useful if I didn't have to parse it out of the logged message. I propose adding the timing information to MDC. That way I more easily create dashboards of the amount of time spent on queries over time.

For example, change this:

this.sqlLogger.debug("retrieved " + this.rowCount + " objects, " +
        ((this.rowCount > 0) ? ((totalTime / this.rowCount) + " ms per") : (totalTime +
                " ms")));

to

MDC.put("rowCount", String.valueOf(this.rowCount));
MDC.put("totalTime", String.valueOf(totalTime));
try
{
    this.sqlLogger.debug(MARKER_TIMING, "retrieved " + this.rowCount + " objects, " + ((this.rowCount > 0) ? (totalTime / this.rowCount + " ms per") : (totalTime + " ms")));
}
finally
{
    MDC.remove("rowCount");
    MDC.remove("totalTime");
}

I'm happy to contribute these changes but I wanted to discuss here before working on the whole thing.

I'm starting with the assumption that changing the existing logging is undesirable because users already rely on the current behavior. So this proposal adds markers and MDC but doesn't change, remove, or combine any messages. What do you think?