🟩 SyncPipe

A file synchronization tool over TCP using Node.js and TypeScript. SyncPipe provides real-time file synchronization between a server and multiple clients using raw TCP sockets.

Features

• Real-time file synchronization - Detects and syncs file changes instantly
• TCP-based communication - Uses raw TCP sockets for efficient data transfer
• Chunked file transmission - Large files are sent in manageable chunks
• Automatic reconnection - Clients automatically reconnect with exponential backoff
• File integrity verification - SHA-256 checksums ensure data integrity
• Graceful error handling - Robust error handling and recovery
• Heartbeat monitoring - Connection health monitoring
• Cross-platform - Works on Windows, macOS, and Linux

Quick Start

1. Installation

npm install

2. Build the project

npm run build

3. Start the server

# Development mode
npm run dev:server

# Production mode
npm run start:server

4. Start the client (in another terminal)

# Development mode
npm run dev:client

# Production mode
npm run start:client

Configuration

Copy .env.example to .env and modify as needed:

cp .env.example .env

Environment Variables

Variable	Default	Description
PORT	3000	Server port
SYNC_DIR	./data/server	Server sync directory
SERVER_HOST	localhost	Server hostname for client
SERVER_PORT	3000	Server port for client
WATCH_DIR	./data/client	Client watch directory

Architecture

Core Components

1. SyncServer - TCP server that manages file synchronization
2. SyncClient - TCP client that connects to server and watches local files
3. FileWatcher - Monitors file system changes using Node.js fs.watch
4. Utils - Common utilities for file operations and message handling

Message Protocol

SyncPipe uses a custom message protocol over TCP:

interface SyncMessage {
  type: 'FILE_LIST' | 'FILE_REQUEST' | 'FILE_CHUNK' | 'FILE_COMPLETE' | 'ERROR' | 'HEARTBEAT';
  payload: any;
  timestamp: number;
  id: string;
}

File Transfer Process

1. Discovery: Server sends file list to connected clients
2. Comparison: Client compares local files with server files
3. Request: Client requests missing/outdated files
4. Transfer: Server sends files in chunks with checksums
5. Verification: Client verifies each chunk and reconstructs file
6. Completion: Server confirms transfer completion

Development

Available Scripts

# Development
npm run dev:server    # Start server in development mode
npm run dev:client    # Start client in development mode

# Production
npm run build         # Build TypeScript to JavaScript
npm run start:server  # Start compiled server
npm run start:client  # Start compiled client

# Utilities
npm run clean         # Remove build artifacts

Testing

Manual Testing

1. Start the server: npm run dev:server
2. Start the client: npm run dev:client
3. Add/modify files in data/server/ - they should sync to data/client/
4. Monitor the console output for synchronization events

Testing Scenarios

• File Creation: Add new files to server directory
• File Modification: Edit existing files
• File Deletion: Delete files (currently one-way sync)
• Large Files: Test with files larger than chunk size (64KB)
• Directory Structure: Test nested directories
• Network Interruption: Test reconnection by restarting server
• Multiple Clients: Start multiple client instances

Production Deployment

Server Deployment

# Build the project
npm run build

# Set environment variables
export PORT=3000
export SYNC_DIR=/path/to/sync/directory

# Start server
npm run start:server

Client Deployment

# Build the project
npm run build

# Set environment variables
export SERVER_HOST=your-server-hostname
export SERVER_PORT=3000
export WATCH_DIR=/path/to/watch/directory

# Start client
npm run start:client

Process Management

Consider using a process manager like PM2 for production:

# Install PM2
npm install -g pm2

# Start server
pm2 start dist/server.js --name syncpipe-server

# Start client
pm2 start dist/client.js --name syncpipe-client

Limitations

• One-way sync: Currently only syncs from server to client
• No conflict resolution: Last write wins
• No compression: Files are transferred without compression
• Memory usage: Large files are loaded into memory for checksum calculation

License

MIT License - see LICENSE file for details