High-Volume Log Sorting System

🧠 High-Volume Log Sorting & Extraction System (WPF + Web API + Docker)

A high-performance hybrid system (desktop + cloud) developed for a private data analytics client.
It seamlessly processed, filtered, and archived 120+ TB of raw logs, extracting structured intelligence from unstructured data using advanced multithreaded parsing and rule-based pattern detection.

---

🏗️ Project Scope

• Cross-platform solution for parsing, deduplication, filtering, and secure encryption of massive log datasets
• Extracts structured artifacts such as: Emails, IP Addresses, Usernames, Android IDs, IMAP, Phone Numbers, FTP Credentials
• Supports custom rules and regex filters per log type for flexible adaptation
• Built with a WPF front-end and a horizontally scalable, Dockerized backend

---

⚙️ Architecture Overview

• Frontend: WPF with multithreaded file I/O and task queues bound to UI
• Parser Engine: Stream-based processing with parallel GB-scale file reading
• Rule Engine: Combines autogenerated regex with manual condition logic
• Deduplication: In-memory cache backed by lightweight SQLite
• Export Layer: Extracted results stored in encrypted SQLite format
• Archiving & Delivery: Custom-built archival → AES encryption → secure REST upload
• Backend: ASP.NET Web API (containerized, horizontally scalable)
• Database: PostgreSQL with sharding and replication for performance and fault-tolerance

---

🔧 Tech Stack

WPF, Multithreading, Regex, SQLite, ASP.NET Web API, Docker, PostgreSQL, Encryption, Custom Obfuscation

---

🔐 Security & Precision

• End-to-end AES-256 encryption for all archived outputs
• Custom archival format to maximize space efficiency and security
• Backend scaling to handle high-concurrency upload demands
• PostgreSQL replication ensures high availability and consistency

✅ Precision-First Regex Testing

• Over 100 unit tests targeting edge-case regex scenarios
• Prevented leakage of sensitive data and ensured pattern completeness
• Accuracy improvements led to measurable revenue gains for the client

---

🖥️ Pattern-Aware Parsing Interface

Built for commercial usage in a production “operator mode” environment.
Simple for users — complex logic hidden under the hood.

Key features:

• Directory selector with batch processing controls
• Real-time stats: total records, matches, deduplicated entries
• Automatic Pattern Generator: creates regex from live log samples on demand

---

🔒 Code Obfuscation & Protection

To ensure intellectual property security, all code was obfuscated using a custom-forked version of DotFuscator.

• Encrypted strings, distorted control flow, and hidden symbols
• Injected anti-decompilation signatures
• Obfuscation applied across both the WPF frontend and parser core logic

---

📊 Results & Impact

---

🧠 User Experience

Despite processing scale and architectural complexity,
users reported the tool as fast, lightweight, and reliable.
Most workflows executed in 1–2 clicks — no training required.

Even first-time users sorted terabytes of logs
without reading any documentation.

Thanks to real-time feedback and pattern automation,
false positives dropped drastically and task speed increased by 5x.

---

✅ Summary

• Tailor-made system for high-throughput data cleaning and transformation
• Combines intelligent regex generation, multithreaded parsing, and secure archival
• User-friendly interface with a hardened, production-grade backend
• Personally led all design, development, testing, deployment, and protection efforts