Ptubes is a database disaster recovery product based on the PITR (Point In Time Recovery) method

Environment

• Ptubes version: 1.0.0
• Operating System version: Linux
• Java version: 1.8

Steps to reproduce this issue

In some cases in our business, there will be some special time fields, for example: the time field value is "-838:59:59.000000", the datetime field value is "9999-12-31 23:59:59.999999", these special After the time field is delivered to the downstream, the precision and value are different from the upstream

Expected Behavior

The value of the downstream data is the same as the value of the upstream database, regardless of whether it is a special time field value

Actual Behavior

The field value of datetime is "9999-12-31 23:59:59.999999", and the precision of synchronization to the target table becomes 3 digits The value of the time field is "-838:59:59.000000", the value of the upstream table is: -35 days, 1:00:01, after delivery to the downstream, it becomes: 17:04:05

Environment

• Ptubes version: 1.0.0
• Operating System version: Linux
• Java version: 1.8

Steps to reproduce this issue

If the upstream database goes down, the readerTask will always try to reconnect. The Disruptor created during this process cannot be recycled normally, which will cause the Disruptor thread to leak, resulting in the creation of a large number of useless threads.

Expected Behavior

The thread is recycled normally during the retry process

Actual Behavior

The thread is not recycled properly

Environment

• Ptubes version: 1.0.0
• Operating System version: Linux
• Java version: 1.8

Steps to reproduce this issue

I have a problem. When my upstream database changes greatly, I will encounter errors and lose data. The server log is as follows

b2bssc-3 parse row data error, skip row data at txn = TxnContext[], row = RowContext[table=xxx,type=UPDATE] com.meituan.ptubes.reader.common.exception.BuffaloException: java.lang.IndexOutOfBoundsException: Index: 0

Expected Behavior

Data can be parsed and stored normally

Actual Behavior

Data parsing error, it seems that only the update type will have problems

What is the purpose of the change

fix bug https://github.com/meituan/ptubes/issues/3

Brief Changelog

There will be problems with the parsing of special time types. This time, we have made special parsing for special times.

Verify this change

• [x] Make sure there is a GitHub_issue field for changes (usually before you start working on it). Trivial changes like spelling mistakes do not require GitHub issues. Your pull request should only address this issue and no other changes - one PR addresses one issue.
• [x] Every commit in a pull request should have a meaningful subject line and body.
• [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• [x] Write the necessary unit tests to verify your logic corrections, mocks work better when there are cross-module dependencies.
• [x] GitHub Actions works fine on your own fork.

What is the purpose of the change

bug fix https://github.com/meituan/ptubes/issues/2

Brief Changelog

In BinlogContextParser, there are two context variables that should be thread-private, but they are shared during processing, causing concurrency problems. This modification gives them a clone

Verify this change

• [x] Make sure there is a GitHub_issue field for changes (usually before you start working on it). Trivial changes like spelling mistakes do not require GitHub issues. Your pull request should only address this issue and no other changes - one PR addresses one issue.
• [x] Every commit in a pull request should have a meaningful subject line and body.
• [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• [x] Write the necessary unit tests to verify your logic corrections, mocks work better when there are cross-module dependencies.
• [x] GitHub Actions works fine on your own fork.

What is the purpose of the change

fix issuse https://github.com/meituan/ptubes/issues/1

Brief Changelog

The startup of BinlogPipeline is divided into two steps. The first step is to initialize some resources. Only after the database is successfully linked, will BinlogPipeline be started and the thread initialized to avoid thread leakage.

Verify this change

• [x] Make sure there is a GitHub_issue field for changes (usually before you start working on it). Trivial changes like spelling mistakes do not require GitHub issues. Your pull request should only address this issue and no other changes - one PR addresses one issue.
• [x] Every commit in a pull request should have a meaningful subject line and body.
• [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• [x] Write the necessary unit tests to verify your logic corrections, mocks work better when there are cross-module dependencies.
• [x] GitHub Actions works fine on your own fork.

Environment

• Ptubes version: 1.0.0
• Operating System version: Linux
• Java version: 1.8

Steps to reproduce this issue

I encountered a problem. When my database crashes, it is normal to retry and reconnect to the database, but I observed that the task shutdown resource will retry to connect to the database, and the Producer thread is waiting on a condition.

Expected Behavior

It is normal to expect that the database can automatically reconnect and continue the data flow after the database is disconnected

Actual Behavior

The database could not be reconnected and the data flow was interrupted

Environment

• Ptubes version: 1.0.0

• Operating System version: Linux

• Java version: 1.8

Steps to reproduce this issue

Subscribe a table with a column type of geometry.

Expected Behavior

Reader can subscribe successfully and sdk can receive the data.

Actual Behavior

Reader parse binlog event error and skip the event.

Describe the feature

At present, ptubes does not support transaction sending data. All binlog events strip the transaction concept and are all delivered in the form of Events, resulting in the loss of ACID attributes.

When the industry observes the downstream library, half of the upstream operations in the same transaction may be observed: for example, the order placement process, the intra-transaction operation to create an order, and the deduction of the balance. It was originally an intra-transaction operation, with atomicity and synchronization through ptubes When going downstream, the business may observe that the balance has not changed, an order has been added / the balance has decreased, and there is no corresponding order

So we expect that ptubes can assemble the binlog into a transaction and restore the ACID attribute. Some of our functions require this ability

Describe the feature

At present, the concept of sharding is a logical concept, which only exists in the Client for parallel consumption. As a result, each Reader needs to pull the full binlog, and there is a visible performance upper limit for pulling and storing (cannot scale linearly)

1. From logical sharding to physical sharding, the concept of partition is introduced, each Reader does not need to pull all binlogs, reducing storage and processing pressure, and at the same time supporting parallel processing of data by Reader (each Reader processes a part of the partition), improving ptubes Scalability
2. The data of each partition can be subscribed separately to reduce the data backtracking when the service is duplicated
3. Each partition has a separate writing thread, effectively breaking through the performance bottleneck of single-threaded sequential writing

Describe the feature

I recently encountered a problem. Our business has some tasks with relatively large traffic, which will write a large amount of data. At the same time, because of business characteristics, we expect the data to be stored for a long time, which will cause our disk usage to be relatively high. In extreme cases will cause disk exhaustion So, I hope to have a new ability

1. Added a new StorageMode inside Reader, which supports S3 remote storage + Memory mode, and supports tiered storage
2. Support Binlog persistent storage
3. The real-time consumption stream maintains the same speed as before
4. The local disk will be used as the real-time data layer, S3 will be used as the persistent storage layer, and the non-latest data of the local disk will be asynchronously written to S3 through certain rules