Skip to content

Fix TRUNCATE operation not replicated to ClickHouse #165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 29, 2025

Conversation

bakwc
Copy link
Owner

@bakwc bakwc commented Jun 29, 2025

Description

This PR fixes the bug where TRUNCATE operations in MySQL were not being replicated to clear data on the ClickHouse side.

Fixes #155

Problem

When executing TRUNCATE TABLE in MySQL, the operation was detected in the binlog but not processed by the replication system. This resulted in:

  • MySQL table being cleared ✅
  • ClickHouse table retaining all data ❌
  • Data inconsistency between MySQL and ClickHouse

Solution

Implementation

  • Added TRUNCATE detection in DbReplicatorRealtime.handle_query_event()
  • Implemented handle_truncate_query() method that:
    • Parses the table name from TRUNCATE queries
    • Clears any pending insert/delete operations for the affected table
    • Executes TRUNCATE TABLE on the corresponding ClickHouse table
  • Added comprehensive test test_truncate_operation_bug_issue_155() to verify the fix

Changes Made

  1. mysql_ch_replicator/db_replicator_realtime.py:

    • Added TRUNCATE query detection in handle_query_event()
    • Implemented handle_truncate_query() method with proper error handling
    • Added logging for TRUNCATE operations
  2. test_mysql_ch_replicator.py:

    • Added reproduction test that initially fails (demonstrating the bug)
    • Test verifies both MySQL and ClickHouse are cleared after TRUNCATE
    • Test confirms replication continues to work after TRUNCATE operation

Testing

The test confirms:

  • Before fix: Test fails with "ClickHouse table should be empty after TRUNCATE, but contains 3 records"
  • After fix: Test passes - both MySQL and ClickHouse tables are properly cleared
  • Replication continues: New records are properly replicated after TRUNCATE
# Test command
python3 -m pytest -v -s test_mysql_ch_replicator.py -k test_truncate_operation_bug_issue_155

Impact

  • Resolves data consistency issues between MySQL and ClickHouse during TRUNCATE operations
  • Maintains existing functionality - no breaking changes to other operations
  • Follows established patterns in the codebase for query handling
  • Comprehensive logging for troubleshooting TRUNCATE operations

- Add TRUNCATE TABLE handling in DbReplicatorRealtime.handle_query_event()
- Implement handle_truncate_query() method to parse and execute TRUNCATE on ClickHouse
- Clear pending insert/delete operations when TRUNCATE is processed
- Add test_truncate_operation_bug_issue_155() to reproduce and verify the fix
- Test confirms TRUNCATE now properly clears data in both MySQL and ClickHouse

Fixes #155
@bakwc bakwc merged commit 6b81394 into master Jun 29, 2025
1 check passed
@bakwc bakwc deleted the fix-truncate-operation-issue-155 branch June 29, 2025 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Truncate operation not replicated
1 participant