etcd-snapper

🚀 Overview

etcd-snapper is a lightweight, Python-based tool designed to simplify the backup and restoration of etcd databases. It allows users to export etcd key-value pairs to a snapshot file and import them to another etcd instance, with support for TLS authentication, dry run mode, and detailed audit logging. The tool provides real-time progress bars for user feedback and is optimized for reliability and ease of use across various environments, including Linux, macOS, and other POSIX-compliant systems.

Key Features

• Snapshot Export/Import: Export etcd key-value pairs to a JSON-based snapshot file and import them to another etcd instance.
• Dry Run Mode: Simulate export/import operations without modifying the filesystem or etcd database, ideal for testing.
• Progress Tracking: Real-time progress bars using tqdm for export and import operations.
• Audit Logging: Detailed logs written to a file for debugging, with minimal console output for major events and errors.
• TLS Support: Connect to etcd instances with TLS certificates for secure communication.
• Robust Error Handling: Gracefully handles connection issues, invalid snapshot files, and other errors.
• Verification: Checks for completeness of export/import operations to ensure data integrity.

---

📦 Installation

Prerequisites

• Python: 3.8 or higher
• etcd: An accessible etcd server (version 3.x) running on the specified host and port
• Dependencies: Listed in requirements.txt

Steps

1. Clone the Repository

   git clone https://github.com/siyamsarker/etcd-snapper.git
   cd etcd-snapper

2. Install Dependencies

   pip install -r requirements.txt

   The requirements.txt includes:

   etcd3>=0.12.0
   tqdm>=4.66.1
   protobuf==3.20.3
   

   Note: The protobuf==3.20.3 version is pinned to avoid compatibility issues with etcd3. If you encounter issues, ensure this version is installed:

   pip install protobuf==3.20.3

3. Verify etcd Server Ensure your etcd server is running and accessible at the specified host/port (default: localhost:2379). For testing, you can use the provided load_sample_data.sh script to populate an etcd instance with sample data (see Testing).

---

🛠 Usage

The etcd-snapper tool is run via the migrator.py script, which supports exporting and importing etcd snapshots with various options.

Command-Line Interface

python3 migrator.py [options]

Arguments

Argument	Description	Default	Required
--export	Export snapshot from the source etcd instance	-	No
--import	Import snapshot to the target etcd instance	-	No
--snapshot-file	Path to a specific snapshot file for import	Latest in snap/	No
--host	etcd host address	localhost	No
--port	etcd port number	2379	No
--ca-cert	Path to CA certificate for TLS	None	No
--cert-key	Path to certificate key for TLS	None	No
--cert-cert	Path to certificate for TLS	None	No
--snap-dir	Directory to store snapshot files	snap	No
--dry-run	Simulate operations without changes	False	No

Notes:

• Either --export or --import must be specified, but not both.
• If --snapshot-file is not provided for --import, the latest snapshot in snap-dir is used.
• TLS options (--ca-cert, --cert-key, --cert-cert) are required only for secure etcd instances.

Examples

1. Export Snapshot

   python3 migrator.py --export --host localhost --port 2379

   • Exports all key-value pairs from the etcd instance at localhost:2379 to a snapshot file in the snap/ directory (e.g., snap/snapshot_20250716_143800.db).
   • Console output:

     Connecting to etcd at localhost:2379
     Successfully connected to etcd server
     Starting snapshot export to snap/snapshot_20250716_143800.db
     Exporting snapshot: 100%|██████████| 50000/50000 [00:50<00:00, 1000.00keys/s]
     Snapshot successfully exported to snap/snapshot_20250716_143800.db
     

2. Dry Run Export

   python3 migrator.py --export --host localhost --port 2379 --dry-run

   • Simulates exporting without writing to the filesystem.
   • Console output:

     Connecting to etcd at localhost:2379
     Successfully connected to etcd server
     [DRY RUN] Starting snapshot export to snap/snapshot_20250716_143800.db
     Exporting snapshot: 100%|██████████| 50000/50000 [00:50<00:00, 1000.00keys/s]
     [DRY RUN] Completed snap/snapshot_20250716_143800.db
     

3. Import Snapshot

   python3 migrator.py --import --host localhost --port 2380

   • Imports the latest snapshot from the snap/ directory to the etcd instance at localhost:2380.
   • Console output:

     Connecting to etcd at localhost:2380
     Successfully connected to etcd server
     Starting snapshot import from snap/snapshot_20250716_143800.db
     Clearing existing etcd data before import
     Importing snapshot: 100%|██████████| 50000/50000 [00:50<00:00, 1000.00keys/s]
     Snapshot successfully imported from snap/snapshot_20250716_143800.db
     

4. Import Specific Snapshot with TLS

   python3 migrator.py --import --snapshot-file snap/snapshot_20250716_143800.db \
     --host secure-etcd.example.com --port 2379 --ca-cert ca.pem --cert-key key.pem --cert-cert cert.pem

   • Imports a specific snapshot file to a secure etcd instance.

Logging

• Console: Displays only major events (e.g., connection status, start/complete of operations) and errors, along with a progress bar.
• Log File: Detailed logs, including every key-value operation (in dry run or normal mode), are written to logs/etcd-snapper_<timestamp>.log.
  • Example log entry:

    2025-07-16 14:38:00,123 - DEBUG - Logging initialized successfully
    2025-07-16 14:38:00,124 - DEBUG - Initialized EtcdSnapper with host=localhost, port=2379, snap_dir=snap, dry_run=False
    2025-07-16 14:38:00,125 - INFO - Connecting to etcd at localhost:2379
    2025-07-16 14:38:00,126 - DEBUG - Exporting key: key1 with value: value1
    ...
    

---

🧪 Testing

Setting Up a Test etcd Instance

1. Run etcd in Docker:

   docker run -d --name etcd-server -p 2379:2379 bitnami/etcd:latest

2. Load Sample Data: Use the provided load_sample_data.sh script to populate the etcd instance with 50,000 key-value pairs:

   chmod +x load_sample_data.sh
   ./load_sample_data.sh

   • This script inserts keys key1 to key50000 with corresponding values value1 to value50000.

3. Run Export Test:

   python3 migrator.py --export --host localhost --port 2379

   • Verify that a snapshot file is created in the snap/ directory.

4. Run Import Test: Start another etcd instance on a different port:

   docker run -d --name etcd-target -p 2380:2379 bitnami/etcd:latest

   Import the snapshot:

   python3 migrator.py --import --host localhost --port 2380

   • Verify the data in the target etcd instance using etcdctl:

     docker exec etcd-target /opt/bitnami/etcd/bin/etcdctl --endpoints=http://127.0.0.1:2379 get --prefix ""

Troubleshooting

• Protobuf Error: If you see TypeError: Descriptors cannot be created directly, ensure protobuf==3.20.3:

  pip install protobuf==3.20.3

• Connection Issues: Verify the etcd server is running and accessible at the specified host/port. Check TLS certificates if used.
• Log File Issues: Ensure the logs/ directory has write permissions (chmod -R u+rw logs).
• Snapshot File Issues: If no snapshot files are found for import, ensure snap-dir contains valid .db files.

---

🤝 Contributing

We welcome contributions to enhance etcd-snapper! Follow these steps to contribute:

Getting Started

1. Fork the Repository:

   git clone https://github.com/siyamsarker/etcd-snapper.git
   cd etcd-snapper

2. Create a Branch:

   git checkout -b feature/your-feature

3. Install Dependencies:

   pip install -r requirements.txt

4. Make Changes:

   • Add new features, fix bugs, or improve documentation.
   • Ensure code follows PEP 8 style guidelines.

5. Test Changes:

   • Use the load_sample_data.sh script to populate an etcd instance.
   • Run export/import tests to verify functionality.

6. Commit Changes:

   git commit -m "Add your feature or fix"

7. Push to GitHub:

   git push origin feature/your-feature

8. Open a Pull Request:

   • Submit a PR with a clear description of your changes.
   • Link to any relevant issues.

Contribution Guidelines

• Code Style: Adhere to PEP 8 standards (use flake8 or similar tools).
• Testing: Add unit tests for new features using a framework like pytest.
• Documentation: Update README.md and inline comments for any new functionality.
• Commit Messages: Use clear, descriptive messages (e.g., "Add dry run mode for export/import").
• Issues: Check existing issues or create a new one to discuss your feature/bug fix.

Suggested Improvements

• Implement SSH/SFTP for remote snapshot transfers.
• Add compression for snapshot files to reduce storage and transfer size.
• Support incremental snapshots for large datasets.
• Add unit tests.

---

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🌟 Acknowledgments

• Author: Siyam Sarker
• Dependencies:
  • etcd3-py for etcd client functionality
  • tqdm for progress bars
  • protobuf for protocol buffer support
• Community: Thanks to all contributors and users for feedback and support!

---

Made with ❤️ by Siyam Sarker
Report an Issue | Contribute