Sumit Kumar

• Developed Airflow DAGs to automate SQL DDL operations for multiple data load phases, improving efficiency by reducing manual updates and saving 4 developer hours per week.

• Spearheaded the design of dimension tables, enabling real-time analytics and reporting on claims data, thereby reducing claims processing time by two days. Optimized redshift storage by column encoding.

• Prepared a code for data ingestion from Flat Files, Oracle, and Redshift into the Bronze layer, followed by loading into the Silver layer for historical processing in the data warehouse using the medallion architecture.

• Established linting tools across multiple python repo's, resulting in a 27% reduction in developer debugging time by enforcing consistent code structure and enabling automated pre-commit code correction.

• Reengineered ETL data pipeline for Expedia recommendation email campaigns, improving data quality. Resolved diamond user tickets related to shopper campaigns data quality issues.

• Slashed expenses by 47% ($51,300/Month) with data cleanup, lifecycle rules, job restructuring, trimming cluster sizes, dropping from $1,35,000/month to $83,700/month. Dropped the s3 storage to 2.3 PB from 3.3 PB.

• Improved code, identified bottlenecks, and refined tasks. Achieved a 45% runtime reduction, bringing pipeline duration down from 11 hours to 6 hours, through code optimization, task parallelism in REST calls using multithreading.

• Implemented the 62 source tables migration, shaping new columns for extensive data coverage, while ensuring integration, quality and availability for downstream teams and models by using spark framework.

• Led the design, implementation, and management of a high-performance Elasticsearch infrastructure (700M+ records) to serve data via REST APIs (Scalatra, Django) from Elasticsearch and S3 data lakes.

• Collaborated with data scientists to integrate ML prediction models into the data engineering pipeline, eliminating the need for separate pipelines and streamlining end-to-end processing.

• Improved data system performance and efficiency by optimizing PostgreSQL queries (reducing response times from minutes to seconds) and migrating Spark 2 to 3 on Kubernetes with Helm charts.