The simple, stupid batch framework for Java

Last update: Dec 29, 2022

Overview

Easy Batch
The simple, stupid batch framework for Java™

Project status

As of November 18, 2020, Easy Batch is in maintenance mode. This means only bug fixes will be addressed from now on. Version 7.0.x is the only supported version.

Latest news

18/12/2020: Version 7.0.1 is out with a minor bug fix and some dependency updates. Check the release notes here.
24/09/2020: Version 7.0.0 is out with several enhancements towards type safe APIs. See the change log here.

What is Easy Batch?

Easy Batch is a framework that aims to simplify batch processing with Java. It was specifically designed for simple ETL jobs. Writing batch applications requires a lot of boilerplate code: reading, writing, filtering, parsing and validating data, logging, reporting to name a few.. The idea is to free you from these tedious tasks and let you focus on your batch application's logic.

How does it work?

Easy Batch jobs are simple processing pipelines. Records are read in sequence from a data source, processed in pipeline and written in batches to a data sink:

The framework provides the Record and Batch APIs to abstract data format and process records in a consistent way regardless of the data source type.

Let's see a quick example. Suppose you have the following tweets.csv file:

id,user,message
1,foo,hello
2,bar,@foo hi!

and you want to transform these tweets to XML format. Here is how you can do that with Easy Batch:

Path inputFile = Paths.get("tweets.csv");
Path outputFile = Paths.get("tweets.xml");
Job job = new JobBuilder<String, String>()
         .reader(new FlatFileRecordReader(inputFile))
         .filter(new HeaderRecordFilter<>())
         .mapper(new DelimitedRecordMapper<>(Tweet.class, "id", "user", "message"))
         .marshaller(new XmlRecordMarshaller<>(Tweet.class))
         .writer(new FileRecordWriter(outputFile))
         .batchSize(10)
         .build();

JobExecutor jobExecutor = new JobExecutor();
JobReport report = jobExecutor.execute(job);
jobExecutor.shutdown();

This example creates a job that:

reads records one by one from the input file tweets.csv
filters the header record
maps each record to an instance of the Tweet bean
marshals the tweet to XML format
and finally writes XML records in batches of 10 to the output file tweets.xml

At the end of execution, you get a report with statistics and metrics about the job run (Execution time, number of errors, etc). All the boilerplate code of resources I/O, iterating through the data source, filtering and parsing records, mapping data to the domain object Tweet, writing output and reporting is handled by Easy Batch. Your code becomes declarative, intuitive, easy to read, understand, test and maintain.

Quick start

Add the following dependency to your project and you are ready to go:

<dependency>
    <groupId>org.jeasy</groupId>
    <artifactId>easy-batch-core</artifactId>
    <version>7.0.1</version>
</dependency>

You can also generate a quick start project with the following command:

$>mvn archetype:generate \
      -DarchetypeGroupId=org.jeasy \
      -DarchetypeArtifactId=easy-batch-archetype \
      -DarchetypeVersion=7.0.1

For more details, please check the Getting started guide.

Presentations, articles & blog posts

Current versions

Stable:

The current stable version is v7.0.1 | documentation | tutorials | javadoc

Development:

The current development version is 7.0.2-SNAPSHOT:

If you want to import a snapshot version, please check the Getting started guide.

Contribution

You are welcome to contribute to the project with pull requests on GitHub. Please note that Easy Batch is in maintenance mode, which means only pull requests for bug fixes will be considered.

If you believe you found a bug or have any question, please use the issue tracker.

Awesome contributors

Thank you all for your contributions!

Who is using Easy Batch?

Easy Batch has been successfully used in production in a number of companies which I (@benas) am not allowed to mention. That said, there are some companies which publicly mention that they use Easy Batch:

You can also find some feedback from the community about the project in the Testimonials section.

Credits

Many thanks to YourKit, LLC for providing a free license of YourKit Java Profiler to support the development of Easy Batch.

License

Easy Batch is released under the terms of the .

The MIT License (MIT)

Copyright (c) 2021 Mahmoud Ben Hassine ([email protected])

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

Comments

Job Chaining

This is a very common job scenario where we want to run jobs in sequence/chain. Each job would define a set of requirements for it to execute eg, status of previous job in the chain. The report of the chain would contain the status of each job, no of jobs processed successfully, overall status of the chain, etc. This concept could easily be extended to create a chain of chain jobs.
feature

opened by tusharbhasme 33
Add MultiFileRecordReader

Hi,

I saw there is a CompositeRecordWriter, but I couldn't find the corresponding Reader. In my case I need to process csv files in a directory, so I've started with the FileRecordReader to find out that it sends the file names only to the mapper :-(. I couldn't find any examples regarding this - is is possible to accomplish it without to extend the currently available Readers?

TIA
feature

opened by ilkomiliev 24

Multi-Threaded Job PR Suggestion

Hi,

in reference to #299

May I suggest an alternative to the fork-join method? These are the issues I have been having with the fork-join method:

It breaks up your job into multiple jobs.
It does not have the constant memory guarantee of a single batch job. A single batch job will only process a batch size of N, so you will only have N records in memory at any given time. With the fork join model, the memory grows somewhat unexpectedly. If your fork threads are much slower than your file reading thread (as is the case for me) then the memory can grow very fast. I ran the fork-join tutorial with a million line file, changing the following code:

private static Job buildForkJob(String jobName, File tweets, List<BlockingQueue<Record>> workQueues)
            throws FileNotFoundException {
        return aNewJob()
                .named(jobName)
                .reader(new FlatFileRecordReader(tweets))
                .writer(new RoundRobinBlockingQueueRecordWriter(workQueues))
                .jobListener(new PoisonRecordBroadcaster(workQueues))
                .build();
    }

    private static Job buildWorkerJob(String jobName, BlockingQueue<Record> workQueue, BlockingQueue<Record> joinQueue) {
        return aNewJob()
                .named(jobName)
                .reader(new BlockingQueueRecordReader(workQueue))
                .processor(new TweetProcessor(jobName))
                .processor(x -> {
                    Thread.sleep(1000);
                    return x;
                })
                .writer(new BlockingQueueRecordWriter(joinQueue))
                .build();
    }

    private static Job buildJoinJob(String jobName, File out, BlockingQueue<Record> joinQueue) throws IOException {
        return aNewJob()
                .named(jobName)
                .reader(new BlockingQueueRecordReader(joinQueue, NB_WORKERS))
                .filter(new PoisonRecordFilter())
                .writer(new FileRecordWriter(out))
                .build();
    }

I stopped the process early, but as you can see the memory kept growing. I would not expect a 100 size batch to use 1.5 GB of data!

To solve this issue, I would like to either create a new job type or update the existing BatchJob class.

The current BatchJob code reads as follows:

while (moreRecords() && !isInterrupted()) {
                Batch batch = readAndProcessBatch();
                writeBatch(batch);
            }

and I would like to implement something more like this (pseudo code):

 while (moreRecords() && !isInterrupted()) {
                Batch batch = readBatch();
                for(RecordProcessor processor : recordProcessorList){
                    if(BatchProcessor.class.isAssignableFrom(processor.class)){
                        batch = (BatchProcessor) processor.processBatch(batch);
                    }
                    else {
                        Batch newBatch = new Batch();
                        for(Record record : batch){
                            newBatch.addRecord(processor.processRecord(record));
                        }
                        batch = newBatch;
                    }
                }
                writeBatch(batch);
            }

The BatchProcessor class allows a processor to run on a batch of records. With this I could create a multiThreadedProccessor that can run on batches. I think this code could provide the following:

allow for a new type of processor that can process batches instead of individual records
keep constant memory
keep all multi threaded code within the same job
keep records in a consistent order

Do you think this is a bad idea? Are there any major issues that I am not addressing? If you think this is a good idea, may I attempt a PR?

feature

opened by ipropper 15

asynchronous between pipelines

Hi Mahmoud,

Not sure if here is the right place to ask, is there any way we can run pipeline processors asynchronously? Such as adding buffers in between processors.

Thanks, Erick
enhancement feature waiting for feedback

opened by echang 15
FlatFileRecordReader is not available in the Reader Class
Hi, The Hello World example displays the below method:

Job job = new JobBuilder() .named("hello world job") .reader(new FlatFileRecordReader(dataSource))

The *FlatFileRecordReader * is not available. Can you please confirm?
question
opened by anilkulkarni87 14
java 8 stream support

Hello,

These classes add support for java 8 streams and path streams.

I've read the framework is compatible with java 7, so i set java.version to 1.8 only for the added module. Build and tests OK for 1.8.

I hope that it will be useful. Best regards.

Charles Fleury
feature

opened by chsfleury 12
Custom post/pre data processing - General question

While processing my sources I need to apply some pre/post processes over source/target data. My real example scenario is: zip the file - custom CompressedFileWriter encrypt the file - I will probably implement by custom EncryptStream or I am not sure in the moment call Oracle PLSQL stored procedure put file on FTP server etc.

In general I would like to have some interface to implement custom processes whih will be considered(its results) as part of the job, so they will be reflected in job execution reports.

Thanks again for any hints if such feature exists already, looking into api.
question

opened by archenroot 10
unable to determine number of filtered record from batch

While performing BatchRecord Processing, even if some individual records are filtered out, the job report shows 0% filtered. Also, currently the BatchRecord is treated as one record, there must be a way to show individual record count and individual filtered count in the job report.
bug enhancement

opened by chazo 10
Throttle throughput

Hi there,

is it possible to throttle the throughput of a batch job?

I saw that there is the batch size which is at least limiting the size per Batch. But this is not what I am looking for.

Let's assume you need to call a 3rd party within your batchjob and you don't want to put too much stress on it. Therefore you want to make sure that you don't process more then 10 records per second.

I hope you got what I mean?

opened by MALPI 9
easybatch-flatfile bug reading file

Hello,

i try to read csv file(with 5000 lines). i use flatfile reader.
i get a correct total number but he did'nt process any line . in Engine.caller the hasnext() method get false even we are on the firt line. however, when i use a file with only 300 lines i didn't get this bug.

Thank for this useful framework.

I use the 3.0.0 version

opened by arnauldSerge 9
When Reader fails JobListeners are not being invoked

This issue can be replicated by simply modifying the file name in ParallelTutorialWithRecordDispatching to one that does not exist. The test will not end gracefully, instead, it will hang waiting for the poison record which is set as job listener, and in this case it is never invoked
bug

opened by chazo 9

Dependency org.apache.activemq:activemq-core, leading to CVE problem

Hi, in easy-batch-tutorials/, there is a dependency org.apache.activemq:activemq-core:5.7.0 that calls the risk method.

CVE-2015-7559

The scope of this CVE affected version is [4.1.1,)

After further analysis, in this project, the main Api called is org.apache.activemq.ActiveMQConnection: onControlCommand(org.apache.activemq.command.ControlCommand)

Risk method repair link : GitHub

CVE Bug Invocation Path--

Path Length : 8

org.jeasy.batch.tutorials.advanced.jms.JmsBrokerLauncher: main(java.lang.String[]) .m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.broker.BrokerService: stop() .m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.util.ServiceStopper: stop(org.apache.activemq.Service).m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.transport.vm.VMTransport: stop() .m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.ActiveMQConnection: onCommand(java.lang.Object) .m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.command.ControlCommand: visit(org.apache.activemq.state.CommandVisitor)Lorg.apache.activemq.command.Response; .m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.ActiveMQConnection$3: processControlCommand(org.apache.activemq.command.ControlCommand)Lorg.apache.activemq.command.Response; .m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.ActiveMQConnection: onControlCommand(org.apache.activemq.command.ControlCommand)

Dependency tree--

[INFO] org.jeasy:easy-batch-tutorials:jar:7.0.3-SNAPSHOT
[INFO] +- org.jeasy:easy-batch-core:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] +- org.jeasy:easy-batch-flatfile:jar:7.0.3-SNAPSHOT:compile
[INFO] +- org.jeasy:easy-batch-xml:jar:7.0.3-SNAPSHOT:compile
[INFO] |  +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.3:compile
[INFO] |  |  \- jakarta.activation:jakarta.activation-api:jar:1.2.2:compile
[INFO] |  \- org.glassfish.jaxb:jaxb-runtime:jar:2.3.3:compile
[INFO] |     +- org.glassfish.jaxb:txw2:jar:2.3.3:compile
[INFO] |     +- com.sun.istack:istack-commons-runtime:jar:3.0.11:compile
[INFO] |     \- com.sun.activation:jakarta.activation:jar:1.2.2:runtime
[INFO] +- org.jeasy:easy-batch-jdbc:jar:7.0.3-SNAPSHOT:compile
[INFO] +- org.jeasy:easy-batch-hibernate:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- org.hibernate:hibernate-core:jar:5.4.29.Final:compile
[INFO] |     +- org.jboss.logging:jboss-logging:jar:3.4.1.Final:compile
[INFO] |     +- javax.persistence:javax.persistence-api:jar:2.2:compile
[INFO] |     +- org.javassist:javassist:jar:3.27.0-GA:compile
[INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.21:compile
[INFO] |     +- antlr:antlr:jar:2.7.7:compile
[INFO] |     +- org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:jar:1.1.1.Final:compile
[INFO] |     +- org.jboss:jandex:jar:2.2.3.Final:compile
[INFO] |     +- com.fasterxml:classmate:jar:1.5.1:compile
[INFO] |     +- javax.activation:javax.activation-api:jar:1.2.0:compile
[INFO] |     +- org.dom4j:dom4j:jar:2.1.3:compile
[INFO] |     +- org.hibernate.common:hibernate-commons-annotations:jar:5.1.2.Final:compile
[INFO] |     \- javax.xml.bind:jaxb-api:jar:2.3.1:compile
[INFO] +- org.jeasy:easy-batch-validation:jar:7.0.3-SNAPSHOT:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.1.7.Final:compile
[INFO] |  |  \- jakarta.validation:jakarta.validation-api:jar:2.0.2:compile
[INFO] |  +- javax.el:javax.el-api:jar:3.0.0:compile
[INFO] |  \- org.glassfish:javax.el:jar:3.0.0:compile
[INFO] +- org.jeasy:easy-batch-spring:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- org.springframework:spring-context:jar:5.3.4:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.3.4:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.3.4:compile
[INFO] +- org.jeasy:easy-batch-jms:jar:7.0.3-SNAPSHOT:compile
[INFO] +- org.jeasy:easy-batch-json:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- org.eclipse:yasson:jar:1.0.8:compile
[INFO] |     +- jakarta.json.bind:jakarta.json.bind-api:jar:1.0.2:compile
[INFO] |     +- jakarta.json:jakarta.json-api:jar:1.1.6:compile
[INFO] |     \- org.glassfish:jakarta.json:jar:module:1.1.6:compile
[INFO] +- org.jeasy:easy-batch-xstream:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- com.thoughtworks.xstream:xstream:jar:1.4.16:compile
[INFO] |     \- io.github.x-stream:mxparser:jar:1.2.1:compile
[INFO] |        \- xmlpull:xmlpull:jar:1.1.3.1:compile
[INFO] +- org.jeasy:easy-batch-integration:jar:7.0.3-SNAPSHOT:compile
[INFO] +- org.hsqldb:hsqldb:jar:2.5.1:compile
[INFO] +- org.glassfish:javax.json:jar:1.1.4:compile
[INFO] +- com.google.code.gson:gson:jar:2.8.6:compile
[INFO] +- org.apache.activemq:activemq-core:jar:5.7.0:compile
[INFO] |  +- org.apache.geronimo.specs:geronimo-jms_1.1_spec:jar:1.1.1:compile
[INFO] |  +- org.apache.activemq:kahadb:jar:5.7.0:compile
[INFO] |  +- org.apache.activemq.protobuf:activemq-protobuf:jar:1.1:compile
[INFO] |  +- org.fusesource.mqtt-client:mqtt-client:jar:1.3:compile
[INFO] |  |  +- org.fusesource.hawtdispatch:hawtdispatch-transport:jar:1.11:compile
[INFO] |  |  |  \- org.fusesource.hawtdispatch:hawtdispatch:jar:1.11:compile
[INFO] |  |  \- org.fusesource.hawtbuf:hawtbuf:jar:1.9:compile
[INFO] |  +- org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec:jar:1.0.1:compile
[INFO] |  +- commons-net:commons-net:jar:3.1:compile
[INFO] |  \- org.jasypt:jasypt:jar:1.9.0:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.3.4:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.3.4:compile
[INFO] |  +- org.springframework:spring-core:jar:5.3.4:compile
[INFO] |  |  \- org.springframework:spring-jcl:jar:5.3.4:compile
[INFO] |  \- org.springframework:spring-tx:jar:5.3.4:compile
[INFO] +- org.slf4j:slf4j-simple:jar:1.7.30:compile
[INFO] \- org.quartz-scheduler:quartz:jar:2.3.2:compile
[INFO]    +- com.mchange:c3p0:jar:0.9.5.4:compile
[INFO]    +- com.mchange:mchange-commons-java:jar:0.2.15:compile
[INFO]    \- com.zaxxer:HikariCP-java7:jar:2.4.13:compile

Thank you very much.

opened by CVEDetect 0

Dependency org.apache.activemq:activemq-core, leading to CVE problem

Hi, in easy-batch-tutorials/, there is a dependency org.apache.activemq:activemq-core:5.7.0 that calls the risk method.

CVE-2014-3576

The scope of this CVE affected version is [0,]

After further analysis, in this project, the main Api called is org.apache.activemq.broker.TransportConnection: processControlCommand(org.apache.activemq.command.ControlCommand)Lorg.apache.activemq.command.Response

Risk method repair link : GitHub

CVE Bug Invocation Path--

Path Length : 7

org.jeasy.batch.tutorials.advanced.jms.JmsBrokerLauncher: main(java.lang.String[]) .m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.broker.BrokerService: stop() .m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.util.ServiceStopper: stop(org.apache.activemq.Service).m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.transport.vm.VMTransport: stop().m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.ActiveMQConnection: onCommand(java.lang.Object).m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.command.ControlCommand: visit(org.apache.activemq.state.CommandVisitor)Lorg.apache.activemq.command.Response; .m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
org.apache.activemq.broker.TransportConnection: processControlCommand(org.apache.activemq.command.ControlCommand)Lorg.apache.activemq.command.Response;

Dependency tree--

[INFO] org.jeasy:easy-batch-tutorials:jar:7.0.3-SNAPSHOT
[INFO] +- org.jeasy:easy-batch-core:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] +- org.jeasy:easy-batch-flatfile:jar:7.0.3-SNAPSHOT:compile
[INFO] +- org.jeasy:easy-batch-xml:jar:7.0.3-SNAPSHOT:compile
[INFO] |  +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.3:compile
[INFO] |  |  \- jakarta.activation:jakarta.activation-api:jar:1.2.2:compile
[INFO] |  \- org.glassfish.jaxb:jaxb-runtime:jar:2.3.3:compile
[INFO] |     +- org.glassfish.jaxb:txw2:jar:2.3.3:compile
[INFO] |     +- com.sun.istack:istack-commons-runtime:jar:3.0.11:compile
[INFO] |     \- com.sun.activation:jakarta.activation:jar:1.2.2:runtime
[INFO] +- org.jeasy:easy-batch-jdbc:jar:7.0.3-SNAPSHOT:compile
[INFO] +- org.jeasy:easy-batch-hibernate:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- org.hibernate:hibernate-core:jar:5.4.29.Final:compile
[INFO] |     +- org.jboss.logging:jboss-logging:jar:3.4.1.Final:compile
[INFO] |     +- javax.persistence:javax.persistence-api:jar:2.2:compile
[INFO] |     +- org.javassist:javassist:jar:3.27.0-GA:compile
[INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.21:compile
[INFO] |     +- antlr:antlr:jar:2.7.7:compile
[INFO] |     +- org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:jar:1.1.1.Final:compile
[INFO] |     +- org.jboss:jandex:jar:2.2.3.Final:compile
[INFO] |     +- com.fasterxml:classmate:jar:1.5.1:compile
[INFO] |     +- javax.activation:javax.activation-api:jar:1.2.0:compile
[INFO] |     +- org.dom4j:dom4j:jar:2.1.3:compile
[INFO] |     +- org.hibernate.common:hibernate-commons-annotations:jar:5.1.2.Final:compile
[INFO] |     \- javax.xml.bind:jaxb-api:jar:2.3.1:compile
[INFO] +- org.jeasy:easy-batch-validation:jar:7.0.3-SNAPSHOT:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.1.7.Final:compile
[INFO] |  |  \- jakarta.validation:jakarta.validation-api:jar:2.0.2:compile
[INFO] |  +- javax.el:javax.el-api:jar:3.0.0:compile
[INFO] |  \- org.glassfish:javax.el:jar:3.0.0:compile
[INFO] +- org.jeasy:easy-batch-spring:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- org.springframework:spring-context:jar:5.3.4:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.3.4:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.3.4:compile
[INFO] +- org.jeasy:easy-batch-jms:jar:7.0.3-SNAPSHOT:compile
[INFO] +- org.jeasy:easy-batch-json:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- org.eclipse:yasson:jar:1.0.8:compile
[INFO] |     +- jakarta.json.bind:jakarta.json.bind-api:jar:1.0.2:compile
[INFO] |     +- jakarta.json:jakarta.json-api:jar:1.1.6:compile
[INFO] |     \- org.glassfish:jakarta.json:jar:module:1.1.6:compile
[INFO] +- org.jeasy:easy-batch-xstream:jar:7.0.3-SNAPSHOT:compile
[INFO] |  \- com.thoughtworks.xstream:xstream:jar:1.4.16:compile
[INFO] |     \- io.github.x-stream:mxparser:jar:1.2.1:compile
[INFO] |        \- xmlpull:xmlpull:jar:1.1.3.1:compile
[INFO] +- org.jeasy:easy-batch-integration:jar:7.0.3-SNAPSHOT:compile
[INFO] +- org.hsqldb:hsqldb:jar:2.5.1:compile
[INFO] +- org.glassfish:javax.json:jar:1.1.4:compile
[INFO] +- com.google.code.gson:gson:jar:2.8.6:compile
[INFO] +- org.apache.activemq:activemq-core:jar:5.7.0:compile
[INFO] |  +- org.apache.geronimo.specs:geronimo-jms_1.1_spec:jar:1.1.1:compile
[INFO] |  +- org.apache.activemq:kahadb:jar:5.7.0:compile
[INFO] |  +- org.apache.activemq.protobuf:activemq-protobuf:jar:1.1:compile
[INFO] |  +- org.fusesource.mqtt-client:mqtt-client:jar:1.3:compile
[INFO] |  |  +- org.fusesource.hawtdispatch:hawtdispatch-transport:jar:1.11:compile
[INFO] |  |  |  \- org.fusesource.hawtdispatch:hawtdispatch:jar:1.11:compile
[INFO] |  |  \- org.fusesource.hawtbuf:hawtbuf:jar:1.9:compile
[INFO] |  +- org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec:jar:1.0.1:compile
[INFO] |  +- commons-net:commons-net:jar:3.1:compile
[INFO] |  \- org.jasypt:jasypt:jar:1.9.0:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.3.4:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.3.4:compile
[INFO] |  +- org.springframework:spring-core:jar:5.3.4:compile
[INFO] |  |  \- org.springframework:spring-jcl:jar:5.3.4:compile
[INFO] |  \- org.springframework:spring-tx:jar:5.3.4:compile
[INFO] +- org.slf4j:slf4j-simple:jar:1.7.30:compile
[INFO] \- org.quartz-scheduler:quartz:jar:2.3.2:compile
[INFO]    +- com.mchange:c3p0:jar:0.9.5.4:compile
[INFO]    +- com.mchange:mchange-commons-java:jar:0.2.15:compile
[INFO]    \- com.zaxxer:HikariCP-java7:jar:2.4.13:compile

Thank you very much.

opened by CVEDetect 0

Bump jackson-databind from 2.12.2 to 2.12.6.1 in /easy-batch-jms
Bumps jackson-databind from 2.12.2 to 2.12.6.1.

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
Bump jackson-databind from 2.12.2 to 2.12.6.1 in /easy-batch-extensions/easy-batch-jackson
Bumps jackson-databind from 2.12.2 to 2.12.6.1.

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
Hello! We found a vulnerable dependency in your project. Are you aware of it?
Hi! We spot a vulnerable dependency in your project, which might threaten your software. And we found that the vulnerable function of this CVE can be easily accessed from your software.

CVE_ID: CVE-2021-39154

Vulnerable dependency: com.thoughtworks.xstream:xstream

Your invocation path to the vulnerable method:

org.jeasy.batch.extensions.xstream.XstreamRecordMarshaller:<init>() ⬇️ com.thoughtworks.xstream.XStream:<init>() ⬇️ ... ⬇️ com.thoughtworks.xstream.XStream:setupSecurity()

Therefore, maybe you need to upgrade this dependency. Hope this can help you! 😄
opened by HelloMavenEco 0

Releases(easy-batch-7.0.2)

easy-batch-7.0.2(Mar 14, 2021)
This is a patch release that fixes the following issue:

Issue #394 : Resource leak in FileRecordReader

This release also includes some minor backward-compatible dependency updates:

jackson-databind: 2.12.0 -> 2.12.2

yasson: 1.0.7 -> 1.0.8

univocity-parsers: 2.9.0 ->2.9.1

xstream: 1.4.15 -> 1.4.16

spring: 5.3.2 -> 5.3.4

hibernate: 5.4.26.Final -> 5.4.29.Final

Source code(tar.gz)
Source code(zip)
easy-batch-7.0.1(Dec 18, 2020)
This is a patch release that fixes the following issue:

Issue #393 : Errors in reader/writer open/close methods are not handled

This release also includes some minor backward-compatible dependency updates:

jackson-databind: 2.11.2 -> 2.12.0

univocity-parsers: 2.8.4 ->2.9.0

xstream: 1.4.12 -> 1.4.15

yamlbeans: 1.13 -> 1.15

spring: 5.2.9.RELEASE -> 5.3.2

hibernate: 5.4.21.Final -> 5.4.26.Final

hibernate-validator: 6.1.5.Final -> 6.1.7.Final

Source code(tar.gz)
Source code(zip)
easy-batch-7.0.0(Sep 24, 2020)
This release is a major version as it requires Java 11 or above. The main theme for this release is API type safety. There are no new features in this version except updating all APIs to be generic:

RecordReader (and its listener): 7ed62a5d3c0a3ddfaa89fbc0757fddd6c5591288

Batch (and its listener): ba60977cd2135e3f91968ad2f95744d474e77bdc

RecordWriter (and its listener): d437aa550f0c9654fde355f68261b3efd98c0aa4

RecordProcessor (and its listener / sub interfaces): 279ed297d5eb60d590958a11d8971f70199d6e5f

Predicate: 063fcc573d5783f48c01d3375e6d6a6459bf5d4e

JobBuilder: 69299364256fade74a219553540bf04222ef62d8

These changes require updating job definitions to use generics for input/output types in order to enforce the correctness and coherence of expected types between the record reader and writer (More details about this in issue #388).

Migration guide from v6.1 to v7.0

1. Job definition updates

The most notable change required by this release is regarding job definitions:

--Job job = new JobBuilder() // v6 ++Job job = new JobBuilder<String, Tweet>() // v7 // define batch components .build();

Specifying input/output types is not mandatory, but not doing so will lead to a raw type usage warning.

2. RecordProcessor generic types updates

Another important change has been introduced in this release about the generic type of RecordProcessor. In v6, the generic types of RecordProcessor were the types of input/output records. In v7, generic types now represent the type of the payload of input/output records:

--public interface RecordProcessor<I extends Record, O extends Record> { // v6 -- O processRecord(I record) throws Exception; --} ++public interface RecordProcessor<I, O> { // v7 ++ Record<O> processRecord(Record<I> record) throws Exception; ++}

This change has a direct impact on all interfaces extending RecordProcessor (like RecordFilter, RecordMapper, RecordMarshaller and RecordValidator) and their corresponding implementations.

3. Other API updates

Any usage of the aforementioned APIs (RecordReader, RecordWriter, Batch, etc) and their associated listeners should be updated to use generics where appropriate.

4. Deprecated APIs removal

All deprecated APIs in v6.0 and v6.1 have been removed in this major release. The suggested replacement (if any) should be found in the Javdoc of the deprecated API.
Source code(tar.gz)
Source code(zip)
easy-batch-6.1.0(Jul 14, 2020)
This is a minor release which can be used as a drop-in replacement for v6.0.0. Here are the major changes:

New Features and enhancements

Issue #368: Add RetryableRecordProcessor implementation

Issue #381: Add default methods in record reader/writer interfaces

Bug fixes

Issue #372: BeanPropertiesPreparedStatementProvider is failing to prepare statement if bean properties has null value

Deprecations

Issue #380: Deprecate the static factory method JobBuilder#aNewJob

Issue #379: Deprecate ContentBasedBlockingQueueRecordWriterBuilder

Source code(tar.gz)
Source code(zip)
easy-batch-6.0.0(Feb 7, 2020)
This major release marks a new generation of the framework which is now based on Java 8. It has been an opportunity to improve the framework internals as well as some public APIs. All deprecated APIs in v5.3 and before have been removed. Some APIs have changed in a non-backward compatible way to fix a couple of minor design inconsistencies introduced in the v5 line.

The root package has been updated from org.easybatch to org.jeasy.batch for consistency with other Jeasy projects. Artifact IDs have also been changed to match the same naming pattern as other projects. Please refer to the migration guide below for more details.

As you will see in the migration guide, many APIs have been deprecated in previous versions and are now completely removed. As Antoine de Saint-Exupery said:

"Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away"

I believe with this release, there is no much more to take away from Easy Batch, but this does not make it perfect! That said, I am very happy and proud to see some serious companies using it in production as well as encouraging feedback from the community. So I would like to thank all contributors who helped making this release possible! Thank you all for your time and efforts! You are awesome 👍

In addition to dependencies and documentation updates, here are the major highlights for v6.0:

New features

Issue #261: Add support for JSON mapping with Yasson (JSON-B RI)

Issue #311: Add support for Java 8 date/time classes

Issue #353: Add converters for Java 8 date/time types

Issue #361: Make JobExecutor implement java.lang.AutoCloseable

Issue #363: Add batch scanning feature

Issue #366: Add formatting option in BeanFieldExtractor

Enhancements

Issue #351: FileRecordWriter should not throw an Exception at construction time

Issue #355: Remove system properties from JobReport#toString

Issue #356: Improve job duration formatting

Issue #357: Show job duration in the logs

Issue #364: Add default methods in listener interfaces

Bug fixes

Issue #352: XmlRecordReader should close the input stream

Issue #365: Incorrect fixed length record marshalling

New tutorials

Reading data from a REST endpoint

Writing a dynamic header/footer to an output file

Skipping bad records with batch scanning on write failure

Breaking changes

All usages of java.io.File (namely in constructors of all file based record readers/writers) were deprecated in v5.3 and have been replaced with java.nio.file.Path in v6

Quartz support was deprecated in v5.3 and has been removed in v6. The entire module easybatch-quartz has been removed in v6. This module used to provide two classes that act as a bridge between Easy Batch APIs and Quartz APIs. Those two classes (EasyBatchJob and EasyBatchJobFactory) can now be copied from the Quartz tutorial.

MongoDB support has been dropped (Issue #354). Please note that this decision was made after releasing v5.3, hence you will not find any @Deprecated annotations on classes of this module in the v5.3 release. Sorry for any inconvenience!

JobListener#beforeJobStart and JobListener#afterJobEnd have been renamed to JobListener#beforeJob and JobListener#afterJob (Issue #362)

Jms reader/writer now work with a JMS Destination instead of Queue to support Topics as well (Issue #359)

The constructor of JmsMessageTransformer now accepts a javax.jms.Session instead of javax.jms.QueueSession to support both queue and topic sessions (Issue #360)

The JmsRecordReader now returns null based on a timeout and not a JmsPoisonMessage anymore (because JmsPoisonMessage has been removed, see "Removed APIs" section of the migration guide)

The BlockingQueueRecordReader now returns null based on a timeout and not a PoisonRecord anymore (because PoisonRecord has been removed, see "Removed APIs" section of the migration guide)

All EIP related APIs (ContentBasedBlockingQueueRecordWriter, ContentBasedBlockingQueueRecordWriterBuilder, DefaultPredicate, Predicate, RandomBlockingQueueRecordWriter, RoundRobinBlockingQueueRecordWriter) and their JMS equivalent were moved to a new extension module called easy-batch-integration

Migration guide from v5.3 to v6.0

Although care has been taken to document all changes in details, a thing or two could have been missed (in which case, apologies upfront).

Maven coordinates

The group id has changed from org.easybatch to org.jeasy

Artifact IDs have been updated like follows: easybatch-core -> easy-batch-core (same pattern for other artifacts)

Here is the new maven dependency for the core module:

<dependency> <groupId>org.jeasy</groupId> <artifactId>easy-batch-core</artifactId> <version>6.0.0</version> </dependency>

Removed APIs

org.easybatch.xml.XmlWrapperTagWriter was deprecated in v5.3 and has been removed in v6

org.easybatch.tools.monitoring.CliJobMonitoringListener, org.easybatch.tools.reporting.HtmlJobReportFormatter andorg.easybatch.tools.reporting.JobReportEmailSender were deprecated in v5.3 and have been removed in v6. The entire easybatch-tools module has been removed

Quartz support was deprecated in v5.3 and has been removed in v6. The entire module easybatch-quartz has been removed in v6

MongoDB support has been be removed in v6 (See issue #354). Please note that this decision has been made after releasing v5.3, hence you will not find any @Deprecated annotation on classes of this module in the v5.3 release. Sorry for any inconvenience!

org.easybatch.jms.JmsPoisonMessage, org.easybatch.jms.JmsPoisonRecord, org.easybatch.jms.JmsPoisonRecordBroadcaster and org.easybatch.jms.JmsPoisonRecordFilter were deprecated in v5.3 and have been removed in v6.

org.easybatch.core.record.PoisonRecord, org.easybatch.core.listener.PoisonRecordBroadcaster and org.easybatch.core.filter.PoisonRecordFilter were deprecated in v5.3 and have been removed in v6.

The constructor org.jeasy.batch.core.reader.BlockingQueueRecordReader#BlockingQueueRecordReader(java.util.concurrent.BlockingQueue<org.jeasy.batch.core.record.Record>, int) has been removed in v6. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.

The constructor BlockingQueueRecordWriter(final List<BlockingQueue<Record>> has been be removed in v6 as this writer now operates on a single blocking queue instead of multiple queues.

org.easybatch.core.listener.RecordProcessingTimeListener was deprecated in v5.3 and has been removed in v6

org.easybatch.core.filter.RecordNumberBetweenFilter, org.easybatch.core.filter.RecordNumberEqualToFilter, org.easybatch.core.filter.RecordNumberGreaterThanFilter and org.easybatch.core.filter.RecordNumberLowerThanFilter were deprecated in v5.3 and have been removed in v6

JobReportFormatter and DefaultJobReportFormatter were deprecated in v5.3 and have been removed in v6

Constructors that take a delimiter and qualifier in DelimitedRecordMarshaller and ApacheCommonCsvRecordMarshaller have been removed. Use the new setters for these parameters instead.

Replaced/Renamed/Moved APIs

org.easybatch.core.filter.FilteredRecordsSavingRecordFilter was deprecated in v5.3 and has been removed in v6. Use the new org.easybatch.core.filter.FilteredRecordsCollector instead

org.easybatch.core.filter.StartWithStringRecordFilter was deprecated in v5.3 and has been removed in v6. Use org.easybatch.core.filter.StartsWithStringRecordFilter instead

org.easybatch.core.filter.EmptyRecordFilter was deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.EmptyStringRecordFilter instead

org.easybatch.core.filter.EndWithStringRecordFilter was deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.EndsWithStringRecordFilter instead

Usages of java.util.date (namely in Header class) were deprecated in v5.3 and have been replaced with java.time.LocalDateTime in v6

Usages of java.io.File (namely in constructors of all file based record readers/writers) were deprecated in v5.3 and have been replaced with java.nio.file.Path in v6

PayloadExtractor was deprecated in v5.3 and has been removed in v6. Use Utils#extractPayloads instead

The constructor org.jeasy.batch.extensions.yaml.YamlRecordReader#YamlRecordReader(java.io.InputStream, java.lang.String) has been replaced with one that takes a java.nio.charset.Charset instead of a String for the charset name. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.

The constructor org.jeasy.batch.json.JsonRecordReader#JsonRecordReader(java.io.InputStream, java.lang.String) has been replaced with one that takes a java.nio.charset.Charset instead of a String for the charset name. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.

org.easybatch.extensions.stream.StreamRecordReader was moved to the core module under org.easybatch.core.reader

Monitoring APIs (namely JobMonitor and JobMonitorMBean) have been moved from org.jeasy.batch.core.job to org.jeasy.batch.core.jmx

JpaRecordReader#setFetchSize has been renamed to JpaRecordReader#setMaxResults. Note that this has not been deprecated in v5.3.

Source code(tar.gz)
Source code(zip)
easybatch-5.3.0(Jan 14, 2020)
This is a minor release which is the last release of the v5.x line. Here are the major changes:

New features

Issue #333: Add Header/Footer callbacks in FileRecordWriter

Issue #350: Add configurable JobReportFormatter in JobReportEmailSender

Enhancements

Issue #315: Replace Java Util Logging with SLF4J

Issue #320: Error Threshold - Zero Errors

Issue #338: Inefficient JobMetrics merge loops in DefaultJobReportMerger

Issue #317: Unused XMLStreamException

Bug fixes

Issue #349: Incorrect writeCount when the transaction in HibernateRecordWriter is rolled back

Issue #348: Incorrect writeCount when the transaction in JpaRecordWriter is rolled back

Issue #347: Incorrect writeCount when the transaction in JdbcRecordWriter is rolled back

Issue #314: Error while setting a date (java.util.Date) field in JdbcRecordWriter

Issue #345: HtmlJobReportFormatter should not generate apache velocity logs

Issue #337: XmlRecordReader throws a ClassCastException when reading a comment line

Issue #319: Excel reader Skipping Empty Columns

Deprecations

For removal

org.easybatch.xml.XmlWrapperTagWriter is deprecated in v5.3 and will be removed in v6

org.easybatch.tools.monitoring.CliJobMonitoringListener, org.easybatch.tools.reporting.HtmlJobReportFormatter andorg.easybatch.tools.reporting.JobReportEmailSender are deprecated in v5.3 and will be removed in v6. The entire easybatch-tools module will be removed

Quartz support is deprecated in v5.3 and will be removed in v6. All classes in easybatch-quartz module are deprecated and the entire module will be removed in v6

MongoDB support will be removed in v6. Please note that this decision has been made after releasing v5.3, hence you will not find any @Deprecated annotation on classes of this module in the v5.3 release.

org.easybatch.jms.JmsPoisonMessage, org.easybatch.jms.JmsPoisonRecord, org.easybatch.jms.JmsPoisonRecordBroadcaster and org.easybatch.jms.JmsPoisonRecordFilter are deprecated in v5.3 and will be removed in v6. The JmsRecordReader will return null based on a timeout and not a JmsPoisonMessage.

org.easybatch.core.record.PoisonRecord, org.easybatch.core.listener.PoisonRecordBroadcaster and org.easybatch.core.filter.PoisonRecordFilter are deprecated in v5.3 and will be removed in v6. The org.jeasy.batch.core.reader.BlockingQueueRecordReader will return null based on a timeout and not a PoisonRecord

The constructor org.jeasy.batch.core.reader.BlockingQueueRecordReader#BlockingQueueRecordReader(java.util.concurrent.BlockingQueue<org.jeasy.batch.core.record.Record>, int) will be removed in v6. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.

The constructor BlockingQueueRecordWriter(final List<BlockingQueue<Record>> will be removed in v6 as this writer will operate on a single blocking queue.

org.easybatch.core.listener.RecordProcessingTimeListener is deprecated in v5.3 and will be removed in v6

org.easybatch.core.filter.RecordNumberBetweenFilter, org.easybatch.core.filter.RecordNumberEqualToFilter, org.easybatch.core.filter.RecordNumberGreaterThanFilter and org.easybatch.core.filter.RecordNumberLowerThanFilter are deprecated in v5.3 and will be removed in v6

JobReportFormatter and DefaultJobReportFormatter are deprecated in v5.3 and will be removed in v6

With replacement

org.easybatch.core.filter.FilteredRecordsSavingRecordFilter is deprecated in v5.3 and will be removed in v6. Use the new org.easybatch.core.filter.FilteredRecordsCollector instead

org.easybatch.core.filter.StartWithStringRecordFilter is deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.StartsWithStringRecordFilter instead

org.easybatch.core.filter.EmptyRecordFilter is deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.EmptyStringRecordFilter instead

org.easybatch.core.filter.EndWithStringRecordFilter is deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.EndsWithStringRecordFilter instead

Usages of java.util.date (namely in Header class) are deprecated and will be replaced with java.time.LocalDateTime starting from v6

Usages of java.io.File (namely in constrcutors of all file based record readers/writers) are deprecated and will be replaced with java.nio.file.Path starting from v6

PayloadExtractor is deprecated in v5.3 and will be removed in v6. Use Utils#extractPayloads instead

The constructor org.jeasy.batch.extensions.yaml.YamlRecordReader#YamlRecordReader(java.io.InputStream, java.lang.String) will be replaced with one that takes a java.nio.charset.Charset instead of a String for the charset name. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.

The constructor org.jeasy.batch.json.JsonRecordReader#JsonRecordReader(java.io.InputStream, java.lang.String) will be replaced with one that takes a java.nio.charset.Charset instead of a String for the charset name. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.

For more details about deprecations and replacements, please refer to the Javadocs.

What's next?

The next version will be v6 and will be based on Java 8. The root package name will be updated from org.easybatch to org.jeasy.batch for consistency with other Jeasy projects. Artifact IDs will also change to match the same naming pattern as other projects.

I would like to thank all contributors (@anarayn, @MALPI , @jcamiloradamesa , @IvanAtanasov, @sheikhu, @vian42, @verdi8 and @psoares-resilient) for submitting bug reports, testing fixes and contributing to the project with awesome PRs!
Source code(tar.gz)
Source code(zip)
easybatch-5.2.0(Nov 18, 2017)
This is the second maintenance release of the v5.x line. It brings some features and bug fixes:

Features

issue #280 : Invert PipelineListener call order

issue #283 : Add OSGi headers to manifest

issue #303 : Add FilteredRecordsSavingRecordFilter

Bug fixes

issue #291 : XmlRecordReader does not escape gt and lt

issue #292 : XmlRecordReader does not call getLocalPart() on QName

issue #293 : DefaultJobReportFormatter.formatReport duration rolls over after 24 Hours

issue #296 : FilteredCount metric should be renamed to FilterCount

Enhancements

issue #301 : Add getters for file and charset in AbstractFileRecordReader

issue #304 : otherwise method in content based queue record writer builders should be optional

API deprecation

The following methods are deprecated due to renaming FilteredCount to FilterCount in issue #296:

Method org.easybatch.core.job.JobMetrics#getFilteredCount

Method org.easybatch.core.job.JobMetrics#incrementFilteredCount

Method org.easybatch.core.job.JobMonitor#getFilteredCount

Method org.easybatch.core.job.JobMonitorMBean#getFilteredCount

These methods will be removed in v5.3.

I would like to thank @DanieleS82, @ipropper and @tobias-- for their time and effort to make this release happen!
Source code(tar.gz)
Source code(zip)
easybatch-5.1.0(Jun 5, 2017)
This is minor release adding some new features and bug fixes. Here are the most important changes:

New features

issue #92 : Add email sending job listener

issue #121 : Add support for univocity-parsers

issue #122 : Add support for Yaml format

issue #254 : Add MultiFileRecordReader

issue #255 : Add XmlFileRecordReader

issue #256 : Add JsonFileRecordReader

issue #260 : Add parameter to control how quartz should act when certain job is delayed for a long time

issue #269 : Add encoding parameter in XmlRecordReader

issue #270 : Add encoding parameter in JsonRecordReader

issue #278 : Add the ability to interrupt a running job

issue #279 : Add the ability to wait for jobs to terminate

Bug fixes

issue #266 : Incorrect data source name in text file records

issue #267 : Incorrect data source name in JmsRecord header

issue #268 : System properties absent from JobReport

issue #276 : PreProcessed records are ignored

issue #277 : XmlRecordReader should not escape special characters in tag's body

Enhancements

issue #271 : Improve performance of the JdbcRecordReader

issue #263 : JsonFileRecordReader should not throw a FileNotFoundException at construction time

issue #264 : XmlFileRecordReader should not throw a FileNotFoundException at construction time

issue #178 : Improve the QueueRecordWriter builders to guide the user through predicate/queue mapping

Deprecations

issue #265 : XmlFileRecordReader constructor parameters should be swapped

issue #274 : Deprecate JmsQueueSessionListener

issue #275 : Deprecate JmsQueueConnectionListener

Many thanks to all contributors for reporting bugs and testing fixes. A special thank to @AussieGuy0 for adding support for uniVocity parsers!
Source code(tar.gz)
Source code(zip)
easybatch-4.2.1(Feb 25, 2017)
This is a bug fix release for the following issue:

issue #237 : MsExcel file not closed by the MsExcelRecordReader

Many thanks to @packley1 for reporting the issue and testing the fix.
Source code(tar.gz)
Source code(zip)
easybatch-5.0.0(Oct 23, 2016)
The main theme of this new major release is less is more :exclamation: With this new version, Easy Batch has never been easier! Many APIs have been simplified and improved for more consistency and ease of use. Some APIs have been removed in order for you to do less work and delegate more to the framework :smile:

After one year of active development, version 5.0.0 is finally here. This would not have been possible without the amazing open source community !! Many thanks to all contributors who suggested features, filed bugs and reported inconsistencies. I'm really grateful and thankful to all of you!

Version 5 is not 100% backward compatible with version 4. It introduces fundamental changes in the core framework in order to fix some inconsistent APIs. The processing workflow has been modified to improve performance.

This is the final release of v5.0 after many snapshot versions and 2 release candidates. Please find below the complete change log and a migration guide to help you move from v4 to v5.

Major changes

issue #211 : Inconsistent Batch API

issue #233 : Inconsistent RecordReader API

issue #223 : Inconsistent RecordWriter API

issue #227 : Inconsistent RecordDispatcher API

issue #221 : Resource management should be handled by the framework

issue #222 : Transaction management should be handled by the framework

issue #230 : JobExecutor should use an ExecutorService

issue #220 : Adapt listeners to batch processing workflow

New features

issue #253 : Add MongoDBRecordMarshaller

issue #251 : Add support for custom metrics

issue #250 : Add java 8 stream support

issue #248 : Add onConnectionOpened method in JobMonitoringListener

issue #243 : add OpenCsvRecordMarshaller

issue #242 : add XmlRecordValidator

issue #241 : add option to activate/deactivate recursivity in FileRecordReader

issue #224 : Add RetryableRecordReader

issue #225 : Add RetryableRecordWriter

issue #226 : Add CompositeRecordWriter

Bug fixes

issue #247 : onConnectionClosed in JobMonitoringListener is inconsistent

issue #238 : FlatFileRecordReader should not throw a FileNotFoundException at construction time

issue #170 : Unable to determine number of filtered record from batch

issue #204 : Unnecessary generic type in BeanValidationRecordValidator

issue #209 : Unnecessary generic type in BlockingQueueRecordReader / BlockingQueueRecordWriter

issue #228 : Job executionId is not coherent

issue #229 : Job name is not a parameter

issue #231 : JobStatus.ABORTED is redundant

issue #232 : Possible resource leak in XmlWrapperTagWriter

Enhancements and API changes

issue #252 : MongoDBRecordWriter should use bulk writes

issue #249 : Update example in maven quick start archetype

issue #236 : add constructor with java.nio.file.Path in FileRecordReader

issue #239 : add constructor with java.nio.file.Path in FileRecordWriter

issue #240 : add constructor with java.nio.file.Path in FlatFileRecordReader

issue #244 : ApacheCommonCsvRecord is redundant

issue #245 : ApacheCommonCsvRecordReader is redundant

issue #246 : ApacheCommonCsvRecordMapper should be configurable with field names

issue #234 : RecordFieldExtractor is poorly named

issue #235 : BeanRecordFieldExtractor is poorly named

Migration guide

JobBuilder

Methods deprecated in v4.2 have been removed in v5:

JobBuilder#skip(long)

JobBuilder#limit(long)

JobBuilder#timeout(long) and JobBuilder#timeout(long, TimeUnit)

JobBuilder#silentMode(boolean)

JobBuilder#strictMode(boolean)

JobBuilder#jmxMode(boolean)

JobBuilder#call()

The keepAlive parameter has been removed since resource handling is now done by the framework ( issue #221 ). Hence the following methods have been removed from JobBuilder:

JobBuilder#reader(RecordReader recordReader, boolean keepAlive)

JobBuilder#reader(RecordReader recordReader, boolean keepAlive, RetryPolicy retryPolicy)

The "Retry Read" feature is now done with a decorator of RecordReader (issue #224 ). Hence the method JobBuilder#reader(RecordReader recordReader, RetryPolicy retryPolicy) has been removed

The RecordDispatcher API has been removed. Hence, the method JobBuilder#dispatcher(RecordDispatcher recordDispatcher) has been removed

JobExecutor

The method org.easybatch.core.job.JobExecutor.execute is no more static, you need to create a JobExecutor instance to execute jobs

Job executors must be shutdown explicitly

Resource handling & transaction management

The constructors of JdbcRecordReader/JdbcRecordWriter now take a JDBC DataSource as argument instead of a Connection and the JdbcConnectionListener/JdbcTransactionListener have been removed

The constructors of JpaRecordReader/JpaRecordWriter now take a EntityManagerFactory as argument instead of a EntityManager and the JpaEntityManagerListener/JpaTransactionListener have been removed

The constructors of HibernateRecordReader/HibernateRecordWriter now take a SessionFactory as argument instead of a Session and the HibernateSessionListener/HibernateTransactionListener have been removed

Moved APIs

org.easybatch.core.dispatcher.Predicate has moved to org.easybatch.core.writer.Predicate

org.easybatch.core.dispatcher.DefaultPredicate has moved to org.easybatch.core.writer.DefaultPredicate

org.easybatch.core.dispatcher.PoisonRecordBroadcaster has moved to org.easybatch.core.listener.PoisonRecordBroadcaster

Removed APIs

The Batch class is no more a Record. Instead, it now contains a list of records. Hence, all interfaces and implementations related to the previous inconsistent Batch API have been removed: BatchReader, BatchFilter, BatchMapper, BatchProcessor, BatchMarshaller, BatchValidator and BatchWriter.

The "record dispatcher" concept has been removed. All dispatchers have been transformed into writers:

BroadcastRecordDispatcher -> BlockingQueueRecordWriter

ContentBasedRecordDispatcher -> ContentBasedBlockingQueueRecordWriter

ContentBasedRecordDispatcherBuilder -> ContentBasedBlockingQueueRecordWriterBuilder

RoundRobinRecordDispatcher -> RoundRobinBlockingQueueRecordWriter

RandomRecordDispatcher -> RandomBlockingQueueRecordWriter

BroadcastJmsRecordDispatcher -> BroadcastJmsQueueRecordWriter

ContentBasedJmsRecordDispatcher -> ContentBasedJmsQueueRecordWriter

ContentBasedJmsRecordDispatcherBuilder -> ContentBasedJmsQueueRecordWriterBuilder

RandomJmsRecordDispatcher -> RandomJmsQueueRecordWriter

RoundRobinJmsRecordDispatcher -> RoundRobinJmsQueueRecordWriter

Exception handling have been simplified, the following exceptions have been removed:

org.easybatch.core.reader.RecordReaderOpeningException

org.easybatch.core.reader.RecordReaderClosingException

org.easybatch.core.reader.RecordReadingException

org.easybatch.core.processor.RecordProcessingException

org.easybatch.core.dispatcher.RecordDispatchingException

org.easybatch.core.field.RecordFieldExtractionException

org.easybatch.core.mapper.RecordMappingException

org.easybatch.core.marshaller.RecordMarshallingException

org.easybatch.core.validator.RecordValidationException

org.easybatch.core.writer.RecordWritingException

ApacheCommonCsvRecord was removed. This is redundant with StringRecord. ApacheCommonCsvRecordReader was removed. This is redundant with FlatFileRecordReader.

Other changes

org.easybatch.core.processor.ComputationalRecordProcessor deprecated in v4.2 has been removed

JobResult deprecated in v4.2 has been removed from JobReport, this is related to the removal of ComputationalRecordProcessor

All constructors in XmlWrapperTagWriter now take a File instead of FileWriter, this is due to a resource leak

BeanValidationRecordValidator is no more parametrized type

Predicate is no more a parametrized type

BlockingQueueRecordReader is no more a parametrized type

BlockingQueueRecordWriter is no more a parametrized type

RecordFieldExtractor interface has been renamed to FieldExtractor

BeanRecordFieldExtractor class has been renamed to BeanFieldExtractor

Source code(tar.gz)
Source code(zip)
easybatch-5.0.0-RC2(Oct 14, 2016)
New features

issue #243 : add OpenCsvRecordMarshaller

issue #242 : add XmlRecordValidator

issue #241 : add option to activate/deactivate recursivity in FileRecordReader

Bug fixes

issue #238 : FlatFileRecordReader should not throw a FileNotFoundException at construction time

Enhancements and API changes

issue #236 : add constructor with java.nio.file.Path in FileRecordReader

issue #239 : add constructor with java.nio.file.Path in FileRecordWriter

issue #240 : add constructor with java.nio.file.Path in FlatFileRecordReader

issue #244 : ApacheCommonCsvRecord is redundant

issue #245 : ApacheCommonCsvRecordReader is redundant

issue #246 : ApacheCommonCsvRecordMapper should be configurable with field names

Source code(tar.gz)
Source code(zip)
easybatch-5.0.0-RC1(Sep 30, 2016)
Major changes

issue #211 : Inconsistent Batch API

issue #233 : Inconsistent RecordReader API

issue #223 : Inconsistent RecordWriter API

issue #227 : Inconsistent RecordDispatcher API

issue #221 : Resource management should be handled by the framework

issue #222 : Transaction management should be handled by the framework

issue #230 : JobExecutor should use an ExecutorService

issue #220 : Adapt listeners to batch processing workflow

New features

issue #224 : Add RetryableRecordReader

issue #225 : Add RetryableRecordWriter

issue #226 : Add CompositeRecordWriter

Enhancements

issue #234 : RecordFieldExtractor is poorly named

issue #235 : BeanRecordFieldExtractor is poorly named

Fixed bugs

issue #170 : Unable to determine number of filtered record from batch

issue #204 : Unnecessary generic type in BeanValidationRecordValidator

issue #209 : Unnecessary generic type in BlockingQueueRecordReader / BlockingQueueRecordWriter

issue #228 : Job executionId is not coherent

issue #229 : Job name is not a parameter

issue #231 : JobStatus.ABORTED is redundant

issue #232 : Possible resource leak in XmlWrapperTagWriter

Source code(tar.gz)
Source code(zip)
easybatch-4.2.0(Sep 24, 2016)
New Features

issue #205 : Add BeanPropertiesPreparedStatementProvider

issue #208 : add StandardErrorRecordWriter

issue #214 : Replace "strict mode" parameter with "error threshold"

Bug Fixes

issue #202 : Incorrect duration when unable to open the record reader

issue #203 : Incorrect data source when unable to open the record reader

issue #206 : Incoherent log message when filtering a batch of records

issue #207 : Incorrect log message when unable to process a batch of records

issue #210 : java.rmi.UnmarshalException when using the JobMonitorProxy

Deprecations

issue #212 : "skip" parameter is redundant

issue #213 : "silent mode" parameter is redundant

issue #215 : "limit" parameter is redundant

issue #216 : Job timeout should be handled by the executor

issue #217 : Inappropriate "call" method in JobBuilder API

issue #218 : Rename "jmx mode" parameter into "jmx monitoring" deprecation

issue #219 : deprecate the ComputationalRecordProcessor API

Source code(tar.gz)
Source code(zip)
easybatch-4.1.0(Jul 31, 2016)
New Features

issue #123: Add support for MS Excel format using Apache POI

issue #168: Add last error in job report

issue #171: Add support to trim whitespace for FixedLengthRecordMapper

issue #175: Add retry on failure for the record reader

issue #187: Add host name to job report

issue #196: Add JMX monitoring push notifications

issue #198: BlockingQueueRecordReader should be configurable with the number of poison records received

Fixed bugs

Issue #169: When Reader fails Job Listeners are not being invoked

Issue #191: NumberFormatException in IntegerTypeConverter

Enhancements and API changes

issue #186: inconsistent BlockingQueue record reader/writer APIs

issue #190: JobMonitor should not be public

issue #193: Data source name of blocking queue is too long

issue #194: Improve JUL log record format

issue #195: Not appropriate JobReportMerger and DefaultJobReportMerger API package

issue #199: Add utility method to check if a record is a poison record

Source code(tar.gz)
Source code(zip)
easybatch-4.0.0(Nov 7, 2015)
Major changes

issue #136 : Redesign the workflow into a consistent pipeline

issue #160 : Improve workflow consistency

issue #137 : Unify Handlers and Listeners

issue #142 : Introduce JobParameters and JobMetrics

issue #141 : Rename Engine to Job

issue #154 : Rename MultiRecord to Batch

issue #148 : Migrate to Java 7

Features

issue #162 : Add PayloadExtractor

issue #157 : Add BatchProcessor

issue #156 : Add BatchValidator

issue #153 : Add BatchFilter

issue #152 : Add BatchMapper

issue #151 : Add BatchMarshaller

issue #131 : Add JpaBatchWriter

issue #130 : Add HibernateBatchWriter

issue #161 : Add JmsPoisonMessageBroadcaster

issue #158 : Add JmsRecord dispatchers

issue #147 : Add JMS queue/connection listeners

issue #144 : Add FAILED job status

issue #143 : Add JobExecutor

issue #133 : Add RecordProcessingTimeListener

issue #132 : Add "timeout" parameter

issue #129 : Add "keep alive" parameter

issue #120 : Add EmptyRecordFilter

issue #159 : Add method to register a record dispatcher in the job builder API

issue #138 : Add method to register a record marshaller in the job builder API

issue #135 : Add BlockingQueueRecordWriter

Enhancements

issue #150 : Use varargs instead of arrays in mapper constructor

issue #155 : Rename JmsRecordWriter to JmsQueueRecordWriter

issue #149 : Rename JmsRecordReader to JmsQueueRecordReader

issue #146 : Rename CliRecordReader to StandardInputRecordReader

issue #145 : Rename QueueRecordReader to BlockingQueueRecordReader

issue #140 : Remove ListRecordReader, deprecated in V3.0

issue #139 : Remove commit-interval from transaction listeners

issue #134 : Remove redundant generic type declaration in ObjectMapper

Bug fixes

issue #126 : ApacheCommonsCsv prints a header for each line

issue #125 : Skipped records are not calculated in the merged report

Source code(tar.gz)
Source code(zip)
easybatch-3.1.0(Oct 25, 2015)
Added the "skip" parameter (issue #77)

Added the "limit" parameter (issue #78)

Added record writers: OutputStream (issue #79), JPA (issue #80), MongoDB (issue #82), JDBC (issue #83), JMS (issue #85), Collection (issue #106), String (issue #109)

Added Hibernate support: new HibernateRecordReader and HibernateRecordWriter APIs (issue #81)

Added XmlRecordCompactor and JsonRecordCompactor to flatten hierarchical data (issue #86)

Added IterableRecordReader and deprecated ListRecordReader (issue #88)

Added RecordCollector (issue #89)

Added record marshallers: Xstream (issue #94), Xml (issue #95), Delimited (issue #96), FixedLength (issue #97), ApacheCommonCsv (issue #98), Gson (issue #99), Jackson (issue #100)

Added multi-record readers to support chunk processing (issue #87): Flat files, Iterable, String, File, Xml, Json, JPA, Apache Common Csv, Hibernate, MongoDB

Added multi-record writers: Collection (issue #110), File (issue #111), OutputStream (issue #112), StandardOutput (issue #113), String (issue #115), JDBC (issue #116)

Added GenericMultiRecordMapper (issue #101)

Implemented improvement #91: Add system properties to the execution report

Implemented improvement #93: Make the JpaRecordReader stream records

Implemented improvement #103: Make the ObjectMapper properly handle empty values

Implemented improvement #119: Use CSVFormat null recordSeparator to skip new line creation

Merged PR #117: Add RecordFieldExtractor

Merged PR #118: Add lineSeparator parameter to OutputStreamRecordWriter

Source code(tar.gz)
Source code(zip)