Transunion
|Data Engineer
Bengaluru, Karnataka, India
Summary
Led cross-functional data ingestion and engineering initiatives, orchestrating the design, development, and deployment of scalable ETL pipelines for multi-terabyte data flows.
Highlights
Led a cross-functional Data Ingestion team, designing and deploying scalable, production-grade ETL pipelines using Python, SQL, and AWS Glue to manage multi-terabyte data flows from diverse sources.
Spearheaded end-to-end data lifecycle management across AWS, Azure, and GCP, implementing data sampling and partitioning techniques to optimize analytical workloads and model training pipelines.
Developed custom analytics models with Python and SQL for data profiling and anomaly detection, significantly improving client data reliability metrics at scale.
Migrated legacy SQL workloads to AWS using Redshift and S3-integrated pipelines, and refactored complex SQL queries to significantly improve compute efficiency and reduce cost.
Designed and implemented custom onboarding ETL workflows, reducing go-live time by 40% through reusable pipeline templates and metadata-driven processing.
Built and maintained robust data validation frameworks and automated auditing checkpoints, ensuring data quality, governance standards, and regulatory compliance for sensitive financial datasets.
Consistently exceeded SLA thresholds and led a team of junior data engineers, providing technical guidance and driving best practice adoption for scalable pipeline architecture and data quality optimization.
Created executive-level dashboards and business metric reports in Power BI/Tableau, delivering real-time visibility and actionable insights to directors and APMs.