Persistent cluster-friendly scheduler for Java

Last update: Dec 31, 2022

Overview

db-scheduler

Task-scheduler for Java that was inspired by the need for a clustered java.util.concurrent.ScheduledExecutorService simpler than Quartz.

As such, also appreciated by users (cbarbosa2, rafaelhofmann):

Your lib rocks! I'm so glad I got rid of Quartz and replaced it by yours which is way easier to handle!

cbarbosa2

Features

Cluster-friendly. Guarantees execution by single scheduler instance.
Persistent tasks. Requires single database-table for persistence.
Embeddable. Built to be embedded in existing applications.
High throughput. Tested to handle 2k - 10k executions / second. Link.
Simple.
Minimal dependencies. (slf4j)

Getting started

Add maven dependency

<dependency>
    <groupId>com.github.kagkarlsson</groupId>
    <artifactId>db-scheduler</artifactId>
    <version>9.4</version>
</dependency>

Create the scheduled_tasks table in your database-schema. See table definition for postgresql, oracle, mssql or mysql.
Instantiate and start the scheduler, which then will start any defined recurring tasks.

RecurringTask<Void> hourlyTask = Tasks.recurring("my-hourly-task", FixedDelay.ofHours(1))
        .execute((inst, ctx) -> {
            System.out.println("Executed!");
        });

final Scheduler scheduler = Scheduler
        .create(dataSource)
        .startTasks(hourlyTask)
        .threads(5)
        .build();

// hourlyTask is automatically scheduled on startup if not already started (i.e. exists in the db)
scheduler.start();

For more examples, continue reading. For details on the inner workings, see How it works. If you have a Spring Boot application, have a look at Spring Boot Usage.

Who uses db-scheduler?

List of organizations known to be running db-scheduler in production:

Company	Description
Digipost	Provider of digital mailboxes in Norway
Vy Group	One of the largest transport groups in the Nordic countries.
TransferWise	A cheap, fast way to send money abroad.
Becker Professional Education
Monitoria	Website monitoring service.

Feel free to open a PR to add your organization to the list.

Examples

Recurring task

Define a recurring task and schedule the task's first execution on start-up using the startTasks builder-method. Upon completion, the task will be re-scheduled according to the defined schedule (see pre-defined schedule-types).

RecurringTask<Void> hourlyTask = Tasks.recurring("my-hourly-task", FixedDelay.ofHours(1))
        .execute((inst, ctx) -> {
            System.out.println("Executed!");
        });

final Scheduler scheduler = Scheduler
        .create(dataSource)
        .startTasks(hourlyTask)
        .registerShutdownHook()
        .build();

// hourlyTask is automatically scheduled on startup if not already started (i.e. exists in the db)
scheduler.start();

One-time tasks

An instance of a one-time task has a single execution-time some time in the future (i.e. non-recurring). The instance-id must be unique within this task, and may be used to encode some metadata (e.g. an id). For more complex state, custom serializable java objects are supported (as used in the example).

Define a one-time task and start the scheduler:

OneTimeTask<MyTaskData> myAdhocTask = Tasks.oneTime("my-typed-adhoc-task", MyTaskData.class)
        .execute((inst, ctx) -> {
            System.out.println("Executed! Custom data, Id: " + inst.getData().id);
        });

final Scheduler scheduler = Scheduler
        .create(dataSource, myAdhocTask)
        .registerShutdownHook()
        .build();

scheduler.start();

... and then at some point (at runtime), an execution is scheduled using the SchedulerClient:

// Schedule the task for execution a certain time in the future and optionally provide custom data for the execution
scheduler.schedule(myAdhocTask.instance("1045", new MyTaskData(1001L)), Instant.now().plusSeconds(5));

More examples

Configuration

Scheduler configuration

The scheduler is created using the Scheduler.create(...) builder. The builder has sensible defaults, but the following options are configurable.

Option	Description
`.threads(int)`	Number of threads. Default `10`.
`.pollingInterval(Duration)`	How often the scheduler checks the database for due executions. Default `30s`.
`.pollUsingFetchAndLockOnExecute(double, double)`	Use default polling strategy `fetch-and-lock-on-execute`. `lowerLimitFractionOfThreads`: threshold for when new executions are fetched from the database (given that last batch was full). Default `0.5`. `executionsPerBatchFractionOfThreads`: how many executions to fetch in each batch. Defualt `3.0`. These executions will not be pre-locked, so the scheduler will compete with other instances for the lock when it is executed. Supported by all databases.
`.pollUsingLockAndFetch(double, double)`	Use polling strategy `lock-and-fetch` which uses `select for update .. skip locked` for less overhead. `lowerLimitFractionOfThreads`: threshold for when new executions are fetched from the database (given that last batch was full). `upperLimitFractionOfThreads`: how many executions to lock and fetch. For high throughput (i.e. keep threads busy), set to for example `1.0, 4.0`. Currently hearbeats are not updated for picked executions in queue. If they stay there for more than 4 * , they will be marked as dead and likely be unlocked again (determined by `DeadExecutionHandler`). Currently supported by postgres.
`.heartbeatInterval(Duration)`	How often to update the heartbeat timestamp for running executions. Default `5m`.
`.schedulerName(SchedulerName)`	Name of this scheduler-instance. The name is stored in the database when an execution is picked by a scheduler. Default `<hostname>`.
`.tableName(String)`	Name of the table used to track task-executions. Change name in the table definitions accordingly when creating the table. Default `scheduled_tasks`.
`.serializer(Serializer)`	Serializer implementation to use when serializing task data. Default standard Java serialization.
`.enableImmediateExecution()`	If this is enabled, the scheduler will attempt to directly execute tasks that are scheduled to `now()`, or a time in the past. For this to work, the call to `schedule(..)` must not occur from within a transaction, because the record will not yet be visible to the scheduler (if this is a requirement, see the method `scheduler.triggerCheckForDueExecutions()`). Default `false`.
`.executorService(ExecutorService)`	If specified, use this externally managed executor service to run executions. Ideally the number of threads it will use should still be supplied (for scheduler polling optimizations). Default `null`.
`shutdownMaxWait(Duration)`	How long the scheduler will wait before interrupting executor-service threads. If you find yourself using this, consider if it is possible to instead regularly check `executionContext.getSchedulerState().isShuttingDown()` in the ExecutionHandler and abort long-running task. Default `30min`.
`.deleteUnresolvedAfter(Duration)`	The time after which executions with unknown tasks are automatically deleted. These can typically be old recurring tasks that are not in use anymore. This is non-zero to prevent accidental removal of tasks through a configuration error (missing known-tasks) and problems during rolling upgrades. Default `14d`.
`.jdbcCustomization(JdbcCustomization)`	db-scheduler tries to auto-detect the database used to see if any jdbc-interactions need to be customized. This method is an escape-hatch to allow for setting `JdbcCustomizations` explicitly. Default auto-detect.
`.commitWhenAutocommitDisabled(boolean)`	By default no commit is issued on DataSource Connections. If auto-commit is disabled, it is assumed that transactions are handled by an external transaction-manager. Set this property to `true` to override this behavior and have the Scheduler always issue commits. Default `false`.
`.failureLogging(Level, boolean)`	Configures how to log task failures, i.e. `Throwable`s thrown from a task execution handler. Use log level `OFF` to disable this kind of logging completely. Default `WARN, true`.
`.registerShutdownHook()`	Registers a shutdown-hook that will call `Scheduler.stop()` on shutdown. Stop should always be called for a graceful shutdown and to avoid dead executions.

Task configuration

Tasks are created using one of the builder-classes in Tasks. The builders have sensible defaults, but the following options can be overridden.

Option	Default	Description
`.onFailure(FailureHandler)`	see desc.	What to do when a `ExecutionHandler` throws an exception. By default, Recurring tasks are rescheduled according to their `Schedule` one-time tasks are retried again in 5m.
`.onDeadExecution(DeadExecutionHandler)`	`ReviveDeadExecution`	What to do when a dead executions is detected, i.e. an execution with a stale heartbeat timestamp. By default dead executions are rescheduled to `now()`.
`.initialData(T initialData)`	`null`	The data to use the first time a recurring task is scheduled.

Schedules

The library contains a number of Schedule-implementations for recurring tasks. See class Schedules.

Schedule	Description
`.daily(LocalTime ...)`	Runs every day at specified times. Optionally a time zone can be specified.
`.fixedDelay(Duration)`	Next execution-time is `Duration` after last completed execution. Note: This `Schedule` schedules the initial execution to `Instant.now()` when used in `startTasks(...)`
`.cron(String)`	Spring-style cron-expression.

Another option to configure schedules is reading string patterns with Schedules.parse(String).

The currently available patterns are:

Pattern	Description
`FIXED_DELAY\|Ns`	Same as `.fixedDelay(Duration)` with duration set to N seconds.
`DAILY\|12:30,15:30...(\|time_zone)`	Same as `.daily(LocalTime)` with optional time zone (e.g. Europe/Rome, UTC)

More details on the time zone formats can be found here.

Spring Boot usage

For Spring Boot applications, there is a starter db-scheduler-spring-boot-starter making the scheduler-wiring very simple. (See full example project).

Prerequisites

An existing Spring Boot application
A working DataSource with schema initialized. (In the example HSQLDB is used and schema is automatically applied.)

Getting started

Add the following Maven dependency

<dependency>
    <groupId>com.github.kagkarlsson</groupId>
    <artifactId>db-scheduler-spring-boot-starter</artifactId>
    <version>9.4</version>
</dependency>

NOTE: This includes the db-scheduler dependency itself.

In your configuration, expose your Task's as Spring beans. If they are recurring, they will automatically be picked up and started.
If you want to expose Scheduler state into actuator health information you need to enable db-scheduler health indicator. Spring Health Information.
Run the app.

Configuration options

Configuration is mainly done via application.properties. Configuration of scheduler-name, serializer and executor-service is done by adding a bean of type DbSchedulerCustomizer to your Spring context.

# application.properties example showing default values

db-scheduler.enabled=true
db-scheduler.heartbeat-interval=5m
db-scheduler.polling-interval=10s
db-scheduler.polling-limit=
db-scheduler.table-name=scheduled_tasks
db-scheduler.immediate-execution-enabled=false
db-scheduler.scheduler-name=
db-scheduler.threads=10
# Ignored if a custom DbSchedulerStarter bean is defined
db-scheduler.delay-startup-until-context-ready=false

Interacting with scheduled executions using the SchedulerClient

TODO

How it works

A single database table is used to track future task-executions. When a task-execution is due, db-scheduler picks it and executes it. When the execution is done, the Task is consulted to see what should be done. For example, a RecurringTask is typically rescheduled in the future based on its Schedule.

Optimistic locking is used to guarantee that a one and only one scheduler-instance gets to pick a task-execution.

Recurring tasks

The term recurring task is used for tasks that should be run regularly, according to some schedule (see Tasks.recurring(..)).

When the execution of a recurring task has finished, a Schedule is consulted to determine what the next time for execution should be, and a future task-execution is created for that time (i.e. it is rescheduled). The time chosen will be the nearest time according to the Schedule, but still in the future.

To create the initial execution for a RecurringTask, the scheduler has a method startTasks(...) that takes a list of tasks that should be "started" if they do not already have an existing execution. The initial execution-time is determined by the Schedule. If the task already has a future execution (i.e. has been started at least once before), but an updated Schedule now indicates another execution-time, the existing execution will be rescheduled to the new execution-time (with the exception of non-deterministic schedules such as FixedDelay where new execution-time is further into the future).

One-time tasks

The term one-time task is used for tasks that have a single execution-time (see Tasks.oneTime(..)). In addition to encode data into the instanceIdof a task-execution, it is possible to store arbitrary binary data in a separate field for use at execution-time. By default, Java serialization is used to marshal/unmarshal the data.

Custom tasks

For tasks not fitting the above categories, it is possible to fully customize the behavior of the tasks using Tasks.custom(..).

Use-cases might be:

Recurring tasks that needs to update its data
Tasks that should be either rescheduled or removed based on output from the actual execution

Dead executions

During execution, the scheduler regularly updates a heartbeat-time for the task-execution. If an execution is marked as executing, but is not receiving updates to the heartbeat-time, it will be considered a dead execution after time X. That may for example happen if the JVM running the scheduler suddenly exits.

When a dead execution is found, the Taskis consulted to see what should be done. A dead RecurringTask is typically rescheduled to now().

Performance

While db-scheduler initially was targeted at low-to-medium throughput use-cases, it handles high-throughput use-cases (1000+ executions/second) quite well due to the fact that its data-model is very simple, consisting of a single table of executions. To understand how it will perform, it is useful to consider the SQL statements it runs per batch of executions.

Polling strategy fetch-and-lock-on-execute

The original and default polling strategy, fetch-and-lock-on-execute, will do the following:

select a batch of due executions
For every execution, on execute, try to update the execution to picked=true for this scheduler-instance. May miss due to competing schedulers.
If execution was picked, when execution is done, update or delete the record according to handlers.

In sum per batch: 1 select, 2 * batch-size updates (excluding misses)

Polling strategy lock-and-fetch

In v10, a new polling strategy (lock-and-fetch) was added. It utilizes the fact that most databases now have support for SKIP LOCKED in SELECT FOR UPDATE statements (see 2ndquadrant blog). Using such a strategy, it is possible to fetch executions pre-locked, and thus getting one statement less:

select for update .. skip locked a batch of due executions. These will already be picked by the scheduler-instance.
When execution is done, update or delete the record according to handlers.

In sum per batch: 1 select-and-update, 1 * batch-size updates (no misses)

Benchmark test

To get an idea of what to expect from db-scheduler, see results from the tests run in GCP below. Tests were run with a few different configurations, but each using 4 competing scheduler-instances running on separate VMs. TPS is the approx. transactions per second as shown in GCP.

	Throughput fetch (ex/s)	TPS fetch (estimates)	Throughput lock-and-fetch (ex/s)	TPS lock-and-fetch (estimates)
Postgres 4core 25gb ram, 4xVMs(2-core)
20 threads, lower 4.0, upper 20.0	2000	9000	10600	11500
100 threads, lower 2.0, upper 6.0	2560	11000	11200	11200

Postgres 8core 50gb ram, 4xVMs(4-core)
50 threads, lower: 0.5, upper: 4.0	4000	22000	11840	10300

Observations for these tests:

For fetch-and-lock-on-execute
- TPS ≈ 4-5 * execution-throughput. A bit higher than the best-case 2 * execution-throughput, likely due the inefficiency of missed executions.
- throughput did scale with postgres instance-size, from 2000 executions/s on 4core to 4000 executions/s on 8core
For lock-and-fetch
- TPS ≈ 1 * execution-throughput. As expected.
- seem to consistently handle 10k executions/s for these configurations
- throughput did not scale with postgres instance-size (4-8 core), so bottleneck is somewhere else

Currently, polling strategy lock-and-fetch is implemented only for Postgres. Contributions adding support for more databases are welcome.

Things to note / gotchas

There are no guarantees that all instants in a schedule for a RecurringTask will be executed. The Schedule is consulted after the previous task-execution finishes, and the closest time in the future will be selected for next execution-time. A new type of task may be added in the future to provide such functionality.
The methods on SchedulerClient (schedule, cancel, reschedule) and the CompletionHandler will run using a new Connectionfrom the DataSourceprovided. To have the action be a part of a transaction, it must be taken care of by the DataSourceprovided, for example using something like Spring's TransactionAwareDataSourceProxy.
Currently, the precision of db-scheduler is depending on the pollingInterval (default 10s) which specifies how often to look in the table for due executions. If you know what you are doing, the scheduler may be instructed at runtime to "look early" via scheduler.triggerCheckForDueExecutions(). (See also enableImmediateExecution() on the Builder)

Versions / upgrading

See releases for release-notes.

Upgrading to 8.x

Custom Schedules must implement a method boolean isDeterministic() to indicate whether they will always produce the same instants or not.

Upgrading to 4.x

Add column consecutive_failures to the database schema. See table definitions for postgresql, oracle or mysql. null is handled as 0, so no need to update existing records.

Upgrading to 3.x

No schema changes
Task creation are preferrably done through builders in Tasks class

Upgrading to 2.x

Add column task_data to the database schema. See table definitions for postgresql, oracle or mysql.

FAQ

Why `db-scheduler` when there is `Quartz`?

The goal of db-scheduler is to be non-invasive and simple to use, but still solve the persistence problem, and the cluster-coordination problem. It was originally targeted at applications with modest database schemas, to which adding 11 tables would feel a bit overkill..

Why use a RDBMS for persistence and coordination?

KISS. It's the most common type of shared state applications have.

I am missing feature X?

Please create an issue with the feature request and we can discuss it there. If you are impatient (or feel like contributing), pull requests are most welcome :)

Is anybody using it?

Yes. It is used in production at a number of companies, and have so far run smoothly.

Comments

Question about one time tasks not getting picked

We are using the scheduler for a huge implementation of one time tasks (typically more than 50 one time tasks every second). After a certain time the db scheduler is not picking up any tasks but seems to work intermittently without consistency. Can you shed some light on the issue we are facing and where we might be going wrong? Also today we observed more than 2500 jobs fail sequential with multiple reattempts and then db scheduler stopped again picking up any tasks
question

opened by gowrinathj-kr 42
Spring / SpringBoot integration
Hi Gustav,

first of all nice work!

Did you consider to prepare separate module (ie. db-scheduler-spring) which provide seamless integration with Spring/Spring Boot?

I was thinking about that a little bit and I think that kind of option could help library gain more popularity. From my perspective what needs to be done to do that:

prepare Spring Boot autostarter which creates the table and initialize all required beans

provide the option to to declare task to execute as Spring managed beans so executed task will be retrieved from context and run (in order to that task should implement specific task interface)

the cron jobs support would be also good for that (but I've seen task for it)

Are you open to that integration? What do you think about that?
enhancement help wanted
opened by pnowy 22

Task completion sometimes fails inside onCompleteReschedule

We're running db-scheduler on 2 servers, with 1s polling interval and immediate execution enabled. I have about 10 jobs and from time to time some of them fail in the onCompleteReschedule step (5-10 times a day).

Failed while completing execution Execution: task=foo, id=recurring, executionTime=2019-11-14T09:00:00.454264Z, picked=true, pickedBy=bar-567f94488b-jmq9v, lastHeartbeat=2019-11-14T09:00:00Z, version=67910. Execution will likely remain scheduled and locked/picked. The execution should be detected as dead in PT4M, and handled according to the tasks DeadExecutionHandler.

java.lang.RuntimeException: Expected one execution to be updated, but updated 0. Indicates a bug.
	at com.github.kagkarlsson.scheduler.JdbcTaskRepository.rescheduleInternal(JdbcTaskRepository.java:196)
	at com.github.kagkarlsson.scheduler.JdbcTaskRepository.reschedule(JdbcTaskRepository.java:154)
	at com.github.kagkarlsson.scheduler.task.ExecutionOperations.reschedule(ExecutionOperations.java:38)
	at com.github.kagkarlsson.scheduler.task.CompletionHandler$OnCompleteReschedule.complete(CompletionHandler.java:50)
	at com.github.kagkarlsson.scheduler.Scheduler$PickAndExecute.complete(Scheduler.java:327)
	at com.github.kagkarlsson.scheduler.Scheduler$PickAndExecute.executePickedExecution(Scheduler.java:309)
	at com.github.kagkarlsson.scheduler.Scheduler$PickAndExecute.run(Scheduler.java:286)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:844)

opened by tarmo-lehtpuu 20

Support timezone in daily schedule

Hello! First of all thank you very much for this project, very useful!

Similarly to #68, it would be also useful to support time zones in Daily schedules.

opened by alex859 19
Logging: Found execution with unknown task-name

Hello,

The use case scenario is that I have a single scheduler with 5 scheduler threads, 10s heartbeat interval and polling interval of 5s. This scheduler has to start several recurring tasks both on a FixedDelay & Daily. I'm still not sure if the issue happens for the Daily task, but for the FixedDelay tasks I get a huge amount of logging, for each task, saying: "internal-execute-due-pool-7-thread-1] com.github.kagkarlsson.scheduler.TaskResolver.resolve : Found execution with unknown task-name <TASK_NAME>". It seems that task execution itself is not affected.

I'm convinced it might be related to threading, but can't be sure. Is this an issue for anyone else? Could I be missing something?

Thanks, Rui
bug

opened by ghost 19
Recurring Task getting stuck

I have a recurring task that's supposed to run more or less every 30 seconds. It's a polling service that handles tasks as they come in which can take longer to run if there's anything to process.

I keep having an issue though where it gets "stuck". The DB record shows it was picked and which machine it was picked by indefinitely and doesn't keep executing. The last_heartbeat column gets updated regularly. But the last_success column stays stale. The only way I've found to get it back running again is to set the picked_by column to null and set picked to 0.

Do you have any advice for how I can make this more robust? Some way of just resuming the usual schedule if it ever gets stuck for more than 10 minutes or something like that? I am also trying to figure out how/why it's getting stuck to begin with, obviously, but I'm assuming that's some issue on my end and not related to db-scheduler.

I'm currently using db-scheduler version 6.7.
question

opened by RLHawk1 18

Unable to deserialize task data: Failed to deserialize object exception

First, some particulars:

db-scheduler 6.9 Spring Boot 2.1.5.RELEASE PostgresQL 11.6

I have been wrestling with this for a few hours now, but I am stumped. Following the Spring Boot idiom, I have a OneTimeTask defined as a Bean on a TasksConfiguration class (marked with @Configuration) with the signature Task<CountdownTimerTask> countdownTimerTask().

I am able to successfully schedule this task. The task data I supply as an instance of CountdownTimerTask, which implements Serializable and carries a serialVersionUID, serializes just fine.

The task fires as it should and my handler executes, but I get a deserialization exception (as described in the subject line above).

Here are the relevant bits of code (exception-handling has been elided):

(I tried eliminating Lombok and the toString override, just to be pure, but it had no effect).

CountdownTimerTask.java

import lombok.Data;
import java.io.Serializable;

@Data
public class CountdownTimerTask implements Serializable {
    private static final long serialVersionUID = 1L;
    private String projectId;

    @Override
    public String toString() {
        return projectId;
    }
}

TasksConfiguration.java

import com.github.kagkarlsson.scheduler.task.Task;
import com.github.kagkarlsson.scheduler.task.helper.Tasks;
import ***************************.ITwilioService;
import ***************************.CountdownTimerTask;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class TasksConfiguration {
    private final ITwilioService twilioService;

    public TasksConfiguration(ITwilioService twilioService) {
        this.twilioService = twilioService;
    }
   
    @Bean
    Task<CountdownTimerTask> countdownTimerTask() {
        return Tasks.oneTime("countdown-timer-task", CountdownTimerTask.class)
                .execute((instance, ctx) -> {
                    this.twilioService.sendSms(new String[] { "**********"}, "Project ID: " + instance.getData().getProjectId());
                });
    }
}

Below is from my service class, which is also the site of the dynamic scheduling of the CountdownTimerTask instance. The transaction event listener fires post-commit, and the handler code executes within the same transaction of the method that published the ProjectCreatedOrUpdatedEvent.

ProjectsService.java

import com.github.kagkarlsson.scheduler.Scheduler;
import com.github.kagkarlsson.scheduler.task.Task;
import ***************************.service.interfaces.IProjectsService;
import ***************************.model.project.*;
import ***************************.model.project.dtos.*;
import ***************************.service.event.ProjectCreatedOrUpdatedEvent;
import ***************************.task.CountdownTimerTask;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.event.TransactionalEventListener;

@TransactionalEventListener
public void processProjectCreatedOrUpdatedEvent(ProjectCreatedOrUpdatedEvent event) 
    throws JsonProcessingException {
        ...

        if (event.getProjectDto().getEarliestBidDeadline() == null) return;

        CountdownTimerTask task = new CountdownTimerTask();
        task.setProjectId(event.getProjectDto().getId());

        scheduler.schedule(countdownTimerTask.instance(
                "countdown-timer-task", task), event.getProjectDto().getEarliestBidDeadline().toInstant());
    }

I've got to be missing something here. Also, everything works fine if I don't involve task data.

enhancement pri1

opened by estaylorco 17

Failed to find implementation for task with name, Execution will be excluded from due. Either delete the execution from the database, or add an implementation for it
I am seeing below error when I schedule a list of recurring tasks ( around 20 recurring tasks scheduled). The scheduler randomly throws below warning and exclude one or two of the tasks from execution and delete after the default period. I am using the below code to get recurring task and pass it to the scheduler. From the code, I see the startTasks method is adding all the tasks to knownTasks, but I am not sure why its complaining the tasks as unknown.

"Failed to find implementation for task with name 'testTask'. Execution will be excluded from due. Either delete the execution from the database, or add an implementation for it. The scheduler may be configured to automatically delete unresolved tasks after a certain period of time."

final Scheduler scheduler = Scheduler.create(dataSource).startTasks(recurringTasks).threads(recurringTasks.size()) .pollingInterval(Duration.ofSeconds(10)) .heartbeatInterval(Duration.ofSeconds(10)) .registerShutdownHook() .build(); taskScheduler = scheduler; scheduler.start(); private RecurringTask<Void> getRecurringTask(Task task, TaskConfig config) { return Tasks.recurring(task.getType().name(), toSchedule(config)) .onFailure(new OnFailureReschedule(FixedDelay.ofSeconds(60))) .execute((VoidExecutionHandler<Void>) task); }
question
opened by ashishkattamuri 15
Evaluation considerations for db-scheduler

Hi db-scheduler team, My team and I are evaluating available open source scheduler projects to incorporate for one of our high throughput use cases. We're looking at roughly around 50 million jobs per day, where each job is a short task execution (less than 10s). The design of db-scheduler (coupled with horizontal scaling) seems to fit the use case for us.

While evaluating db-scheduler, is there something that we need to explicitly keep in mind? If there are pointers which would help us start off on the right track, could you please let me know? Thanks in advance, any help is appreciated.
question

opened by rajki 15
Task removal if there is no more implementation

In our spring based project we are using conditional task registration. Right now we are facing the problem when we are disabling the Tasks.

In other words in database we still have record for the task and but in runtime there is no more registered implementation.

From my perspective there will be useful property or explicit function for removing / marking unsable tasks. Because currently it is generating a lot of redundant log messages. When you need, for instance, disable specific task due to unsuccessful rollout.

I'm happy to submit PR, but first would like to know your vision on this particular problem and if you want to give instruments in the library.
enhancement

opened by ystarikovich 15
Scheduler.exists(TaskInstanceId)
I'm using your library to fulfill a timeout-watchdog. So basicly I use OneTimeTasks that timeout unless reset (canceled and rescheduled) before. Currently I'm catching the RuntimeException of scheduler.cancel and schedule the task again as follows:

try { scheduler.cancel(TaskInstanceId.of(tasktype.getTaskName(), watchdogId.getId())); } catch (RuntimeException e) { if (e.getMessage().startsWith("Could not cancel schedule")) { LOG.debug("Exception:", e); return; } throw e; } // and afterwards schedule the task again

What I'm looking for is an exists-Method to get rid of this ugly exception handling, e.g.:

if(scheduler.exists(instanceId)){ scheduler.cancel(instanceId); } scheduler.schedule(...);
enhancement
opened by zeitwesen 15

Database Deadlocks with MSSQL 2019

Prerequisites

[ ] I am running the latest version (We are running 11.5)
[x] I checked the documentation and found no answer
[x] I checked to make sure that this issue has not already been filed

For bug reports

When we execute around 300 - 400 tasks per minute, we experience database deadlocks on the table scheduled_tasks. The deadlock report from the MSSQL database shows a deadlock between either one of the indexes execution_time_idx or last_heartbeat_idx and the clustered primary key of the table.

The deadlock happens when a Custom-Task tries to reschedule itself for another execution (PollingTaskConfiguration:42 in the stacktrace below) with executionOperations.reschedule.

Workaround

For now we have deleted both indexes to stop the deadlocks from happening (so far successfully), but probably at the cost of reduced throughput.

Steps to Reproduce the bug

Currently unclear

Context

DB-Scheduler Version : 11.5
Java Version : 1.8
Spring Boot (check for Yes) : [x]
Database and Version : Microsoft SQL Server 2019
Database Transaction Isolation Level: Read Committed Snapshot

Logs


Failed while completing execution Execution: task=polling-task, id=polling-task-155175, executionTime=2022-12-17T23:28:53.936Z, picked=true, pickedBy=workflow-engine-36-v6vm9, lastHeartbeat=2022-12-17T23:29:03.339Z, version=642. Execution will likely remain scheduled and locked/picked. The execution should be detected as dead after a while, and handled according to the tasks DeadExecutionHandler.
                com.github.kagkarlsson.jdbc.SQLRuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 71) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
                at com.github.kagkarlsson.jdbc.JdbcRunner.translateException(JdbcRunner.java:126)
                at com.github.kagkarlsson.jdbc.JdbcRunner.lambda$execute$2(JdbcRunner.java:93)
                at com.github.kagkarlsson.jdbc.JdbcRunner.withConnection(JdbcRunner.java:140)
                at com.github.kagkarlsson.jdbc.JdbcRunner.execute(JdbcRunner.java:66)
                at com.github.kagkarlsson.jdbc.JdbcRunner.execute(JdbcRunner.java:54)
                at com.github.kagkarlsson.scheduler.jdbc.JdbcTaskRepository.rescheduleInternal(JdbcTaskRepository.java:251)
                at com.github.kagkarlsson.scheduler.jdbc.JdbcTaskRepository.reschedule(JdbcTaskRepository.java:247)
                at com.github.kagkarlsson.scheduler.task.ExecutionOperations.reschedule(ExecutionOperations.java:62)
                at XXX.PollingTaskConfiguration.lambda$null$1(PollingTaskConfiguration.java:42)
                at com.github.kagkarlsson.scheduler.ExecutePicked.complete(ExecutePicked.java:104)
                at com.github.kagkarlsson.scheduler.ExecutePicked.executePickedExecution(ExecutePicked.java:88)
                at com.github.kagkarlsson.scheduler.ExecutePicked.run(ExecutePicked.java:68)
                at com.github.kagkarlsson.scheduler.FetchCandidates.lambda$run$1(FetchCandidates.java:89)
                at java.util.Optional.ifPresent(Optional.java:159)
                at com.github.kagkarlsson.scheduler.FetchCandidates.lambda$run$2(FetchCandidates.java:87)
                at com.github.kagkarlsson.scheduler.Executor.lambda$addToQueue$0(Executor.java:52)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:750)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 71) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
                at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:265)
                at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1676)
                at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:615)
                at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:537)
                at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7730)
                at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:3786)
                at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:268)
                at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:242)
                at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.execute(SQLServerPreparedStatement.java:515)
                at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
                at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
                at com.github.kagkarlsson.jdbc.JdbcRunner.lambda$execute$2(JdbcRunner.java:85)
                ... 17 common frames omitted

opened by rafaelhofmann 8

Comprehensive Spring-boot example showing most features
Create a simple web-application with Spring Boot showing off db-scheduler's features. This will be a better starting point for Spring boot users.

Ideas:

Page rendering the executions in the database. Also render data if possible (convert to json if java serialization used).

Allow triggering execution to run now

Also show what is currently executed (have some task sleep)

Allow for ad-hoc scheduling of some task via web

Audit table showing what executions have run. In-memory store. (auto-update if want to be fancy)

For understandability. Sometimes things happen too fast to catch. Ala. Kubernetes events

Show what is scheduler and what is scheduler-client

Create new RecurringTaskWithPersistentSchedule via web-request. Variable schedule, typically cron

Immediate execution

JobChaining

Long-running tasks, show currently executing

.. ?

enhancement
opened by kagkarlsson 0
Regularly log scheduler-statistics
To be helpful, and to help pin down issues, the scheduler should regularly log important statistics such as:

Average fetch-time (if too high -> missing index?)

Average dead-execution check (if too high -> missing index?)

Throughput such as

executions/s (and last 5min)

fetches/s (and last 5min)

schedules/s (and last 5min)

Detected anomalies

multiple schedulers

unresolved tasks

..?

The logger could possibly reset statistics every run. (though perhaps keep global averages) Configurable interval. Perhaps 1h as default to avoid log spamming
enhancement
opened by kagkarlsson 0
Guard for creating multiple Scheduler-instances against same table but different task-sets
Generate scheduler-id on startup.

Unique scheduler-id for each instance in the same JVM

Warn if multiple instances using same table_name

Serious warn if multiple instances using same table_name but different task-set (task names)

enhancement help wanted
opened by kagkarlsson 0
Is there a plan to support MSSQL for lock_and_fetch. Using fetch polling strategy has performance issue.

We are using 11.5 version and facing performance issues. We are using MSSQL DB, we have 10 pod running with 8 threads configured, 10*8 = 80 threads total All other configuration are left to default. Scheduler is able to process only 340 to 360 transaction per hour. For our business each transaction take ~40 to 50 secs. Any suggestion to improve the throughput with 'fetch' polling would really help.

I read your comment[1], if I apply same logic for my case

Trigger/task task taks = 50s Total threads across pods = 80 80/50 = 1.6/sec = 1.6 * 60 * 60 = 5760 tasks/hour Bu it is @360/hour.

[1] - https://github.com/kagkarlsson/db-scheduler/issues/209#issuecomment-856512430
question

opened by AnanthaKrishnaV 5

Releases(11.6)

11.6(Nov 18, 2022)
PR #331 adds support for Spring Boot 3.0. Contributed by evenh

PR #329 adds support for setting a custom FailureHandler for RecurringTaskWithPersistentSchedule. Contributed by olayinkasf

PR #338 adds serialization-support for CronSchedule. Contributed by Sprytin

PR #340 upgrades hsqldb to v2.7.1

Additionally, the flaky test SchedulerTest was hopefully fixed.

Source code(tar.gz)
Source code(zip)
11.5(Sep 14, 2022)
PR #321 updates a number of dependency versions

Source code(tar.gz)
Source code(zip)
11.4(Sep 14, 2022)
PR #318 fix bug where immediate execution was not triggered for scheduling happening in the ComplationHandler

PR #319 introduces new ExecutionOperations.replace which will replace the current execution with a new one. For use in for example job-chaining scenarios, where the next step is scheduled when the previous completes.

PR #320 improves the interface for DeadExecutionHandler

Source code(tar.gz)
Source code(zip)
11.3(Sep 14, 2022)
PR #288 upgrades to Spring Boot 2.7.0. Contributed by evenh

PR #289 Github Actions maintenance. Contributed by evenh

PR #290 adds indexes to the default suggested schemas. Contributed by gurban-h

PR #310, #311 upgrades tp postgresql 42.4.1

PR #314 contains fixes to make build run on M1 Macs

Source code(tar.gz)
Source code(zip)
11.2(May 18, 2022)
PR #286 adds support for disabled schedules. For example, a Cron-schedule can be disabled by setting the pattern to -. This will remove any existing executions (and any state they might have).

Source code(tar.gz)
Source code(zip)
11.1(May 4, 2022)
PR #278 adds serializers GsonSerializer, JacksonSerializer and KotlinSerializer and documentation on how to migrate existing state to another serializer. See documentation for #Serializers

Source code(tar.gz)
Source code(zip)
11.0(Feb 23, 2022)
PR #257 adds a better api for creating recurring tasks where the Schedule need to live in the task_data. These types of tasks support runtime modifications of the schedule and multiple instances of the same type of recurring task. See example RecurringTaskWithPersistentScheduleMain.java and StatefulRecurringTaskWithPersistentScheduleMain.java

PR #256 adds the license so that it is detected by github. Contributed by axelfontaine.

PR #252 updates readme versions. Contributed by attilapapai.

Various dependency updates

Source code(tar.gz)
Source code(zip)
10.5(Dec 7, 2021)
PR #237 updates cron-utils dependency to 9.1.6

Source code(tar.gz)
Source code(zip)
10.4(Nov 16, 2021)
PR #230 removes transitive dependencies added via shaded dependency cronutils.

PR #231 makes the Scheduler shutdown faster without unnecessary delays by using a ScheduledExecutorService for housekeeping tasks.

PR #234 removes requirement for Spring Actuator when using the Spring Boot autostarter. Contributed by evenh

PR #238 fixes a bug where shutdownMaxWait was not set for Spring Boot autostarter.

Source code(tar.gz)
Source code(zip)
10.3(May 19, 2021)
PR #198 adds a new FailureHandler using exponential backoff. Contributed by zendern.

PR #199 replaces generic RuntimeExceptions with specific subclasses, allowing users distinguish between exceptions. Contributed by zendern.

PR #204 fixes a bug where recurring tasks were not getting scheduled (initial) when using a DataSource supplying connections with autocommit=false.

Source code(tar.gz)
Source code(zip)
10.2(Apr 12, 2021)
PR #195 Ability to decorate task execution changes Task from an abstract class into an interface. Contributed by reda-alaoui.

PR #196 Example persistent schedule that can be update runtime. See PersistentDynamicScheduleMain.java

PR #197 Better support for stateful recurring tasks, i.e. recurring tasks that updates the task-data after each run. See TrackingProgressRecurringTaskMain.java.

Source code(tar.gz)
Source code(zip)
10.1(Mar 24, 2021)
PR #191 fixes bug where new polling strategy lock-and-fetch got an exception if there were executions with unresolvable tasks in the database.

Source code(tar.gz)
Source code(zip)
10.0(Mar 15, 2021)
Rather big rewrite of Scheduler-internals to support multiple polling-strategies.

PR #175 New polling-strategy for Postgres, select-for-update based. Adds a new, low overhead, polling-strategy utilizing postgres' powerful select for update.. skip locked. See SchedulerBuilder.pollUsingLockAndFetch(lower, higher). Proven to handle 10k executions/s.

PR #182 Spring Boot Devtools-friendly default task data serializer. Contributed by k-jamroz

PR #179 Add some toStrings. Adds better toString methods to tasks and schedules. Contributed by runeflobakk

PR #186 Bump junit from 4.11 to 4.13.1 in /test/benchmark

PR #187 Add SchedulerBuilder helper for registering a shutdown hook. Adds a simpler way of registering the shutdown hook. See SchedulerBuilder.registerShutdownHook()

Source code(tar.gz)
Source code(zip)
9.4(Feb 15, 2021)
PR #174 removes requirement for beans validator for Spring Boot starter. Contributed by dmoidl.

PR #171 adds the ability to change the default log-level for failures, which occur when an ExecutionHandler throws a RuntimeException. Default has previosly been ERROR. Default is now changed to WARN, but can be set with the new builder-method .failureLogging(LogLevel.INFO, true). Contributed by dmoidl

Source code(tar.gz)
Source code(zip)
9.3(Jan 19, 2021)
PR #165 adds support for adding explicit LIMIT to queries rather than relying on setMaxRows (for performance reasons). Must be overridden per-database basis and is currently the new default for PostgreSQL. Contributed by rafal-kowalski

Source code(tar.gz)
Source code(zip)
9.2(Nov 30, 2020)
PR #161 upgrades shades jmrozanec/cron-utils to v9.1.3

PR #155 adds a number of improvements to the SchedulerClient interface.

getScheduledExecution(...) methods returning a List after popular demand

getScheduledExecution(...) takes an optional ScheduledExecutionFilter with options to include or exclude picked executions

getScheduledExecution(...) return type ScheduledExecution now exposes more of the underlying fields on Execution

Source code(tar.gz)
Source code(zip)
9.1(Nov 21, 2020)

PR #159 let SchedulerClient customize serializer. Contributions bybm46370.
Source code(tar.gz)
Source code(zip)
9.0(Nov 14, 2020)
PR #154 fixes bug described in #146. The Scheduler was interfering with externally managed transactions by calling explicit commit when autocommit was set to false. This bug was introduced in v7.1. For anyone using Spring to manage transactions, it is highly recommended to upgrade.

PR #153 avoids running default AutodetectJdbcCustomization when JdbcCustomization is overridden

PR #149 adds tests validating db-scheduler is compatible with Java 15. Contributions by evenh.

Source code(tar.gz)
Source code(zip)
8.2(Nov 11, 2020)
PR #139 adds FixedDelay.ofMillis(millis). Contributions by evenh.

PR #142 adds javadoc for SchedulerClient. Contributions by cunhazera

PR #143 fixes Spring deprecation (Health[Indicator -> Contributor]AutoConfiguration). Contributions by evenh.

PR #144 migrates db-scheduler CI from Travis to Github Actions. Contributions by evenh.

PR #138 makes Scheduler graceful shutdown wait configurable from the default 30m.

Source code(tar.gz)
Source code(zip)
8.1(Sep 23, 2020)
Version 8.1

Provide sources for shaded dependency com.cronutils

Source code(tar.gz)
Source code(zip)
8.0(Sep 23, 2020)
Version 8.0

PR #136 introduces a new feature where changes to the Schedule for a RecurringTask is detected by the Scheduler on startup. Any existing execution is rescheduled to the new next execution-time according to the new Schedule. Note: A Schedule must now indicate whether it is deterministic or not (i.e. if it will evaluate to the same instants).

PR #134 fixes some edge-cases (that were fairly likely for high-throughput cases) that caused the Scheduler to stop fetching further batches of due executions until next pollInterval.

PR #132 fixes race-condition for immediate-execution that may cause the execution to be "missed" until next pollingInterval.

PR #131 upgrades guava to 29.0.

PR #129 adjusts defaults for Spring Boot to match SchedulerBuilder. Contributions by evenh.

Source code(tar.gz)
Source code(zip)
7.2(Sep 23, 2020)
Version 7.2

PR #110 adds micrometer metrics support. Activated by setting .statsRegistry(new MicrometerStatsRegistry(...)) on the builder. If you are using the Spring boot starter, the micrometer metrics will be added if you have micrometer on the classpath. Contributions by evenh.

Source code(tar.gz)
Source code(zip)
7.1(Sep 23, 2020)
Version 7.1

PR #109 fixes db-scheduler for data sources returning connections where autoCommit=false. db-scheduler will now issue an explicit commit for these cases.

Source code(tar.gz)
Source code(zip)
7.0(Sep 23, 2020)
Version 7.0

PR #105 fixes bug for Microsoft Sql Server where incorrect timezone handling caused persisted instant != read instant. This bug was discovered when adding testcontainers-based compatibility tests and has strangely enough never been reported by users. So this release will cause a change in behavior for users where the database is discovered to be Microsoft SQL Server.

Source code(tar.gz)
Source code(zip)
6.8(Sep 23, 2020)
Version 6.8

PR #96 allow for overriding DbSchedulerStarter in Spring Boot starter. Contributed by evenh.

Upgraded to JUnit 5

Full indentation reformatting of the codebase due to mix of tabs and spaces.

Source code(tar.gz)
Source code(zip)
6.7(Sep 23, 2020)
Version 6.7

PR #87 allow for specifying the TimeZone for the Daily schedule and the Schedule string-parser (contributed by alex859)

PR #90 adds task-name to logging of failures (contributed by alex859)

Source code(tar.gz)
Source code(zip)
6.6(Sep 23, 2020)
Version 6.6

PR #86 changes Spring Boot HealthIndicator to opt-in rather than default on. (contributed by ystarikovich)

Source code(tar.gz)
Source code(zip)
6.5(Sep 23, 2020)
Version 6.5

PR #83 added additional exclusions of executions with unresolved task names to getScheduledExecutions() and getExecutionsFailingLongerThan(..).

PR #82 sets junit to test-scope in db-scheduler-boot-starter pom.xml. (contributed by ystarikovich)

Source code(tar.gz)
Source code(zip)
6.4(Sep 23, 2020)
Added configuration option from version 6.3 (deleteUnresolvedAfter(Duration)) to Spring Boot starter.

Source code(tar.gz)
Source code(zip)
6.3(Sep 23, 2020)
Version 6.3

PR #80 adds more graceful handling of unresolved tasks. Executions with unknown tasks will not (in extreme cases) be able to block other executions. They will also automatically be removed from the database after a duration controlled by builder-method deleteUnresolvedAfter(Duration), which currently defaults to 14d.

Source code(tar.gz)
Source code(zip)

Persistent cluster-friendly scheduler for Java

Related tags

Overview

db-scheduler

Features

Getting started

Who uses db-scheduler?

Examples

Recurring task

One-time tasks

More examples

Configuration

Scheduler configuration

Task configuration

Schedules

Spring Boot usage

Prerequisites

Getting started

Configuration options

Interacting with scheduled executions using the SchedulerClient

How it works

Recurring tasks

One-time tasks

Custom tasks

Dead executions

Performance

Polling strategy fetch-and-lock-on-execute

Polling strategy lock-and-fetch

Benchmark test

Things to note / gotchas

Versions / upgrading

FAQ

Why db-scheduler when there is Quartz?

Why use a RDBMS for persistence and coordination?

I am missing feature X?

Is anybody using it?

Comments

Prerequisites

For bug reports

Workaround

Steps to Reproduce the bug

Context

Logs

Releases(11.6)

11.6(Nov 18, 2022)

11.5(Sep 14, 2022)

11.4(Sep 14, 2022)

11.3(Sep 14, 2022)

11.2(May 18, 2022)

11.1(May 4, 2022)

11.0(Feb 23, 2022)

10.5(Dec 7, 2021)

10.4(Nov 16, 2021)

10.3(May 19, 2021)

10.2(Apr 12, 2021)

10.1(Mar 24, 2021)

10.0(Mar 15, 2021)

9.4(Feb 15, 2021)

9.3(Jan 19, 2021)

9.2(Nov 30, 2020)

9.1(Nov 21, 2020)

9.0(Nov 14, 2020)

8.2(Nov 11, 2020)

8.1(Sep 23, 2020)

Version 8.1

8.0(Sep 23, 2020)

Version 8.0

7.2(Sep 23, 2020)

Version 7.2

7.1(Sep 23, 2020)

Version 7.1

7.0(Sep 23, 2020)

Version 7.0

6.8(Sep 23, 2020)

Version 6.8

6.7(Sep 23, 2020)

Version 6.7

6.6(Sep 23, 2020)

Version 6.6

6.5(Sep 23, 2020)

Why `db-scheduler` when there is `Quartz`?