Softensity transforms Data Management for a leading subsidiary in the Automotive industry
A prominent subsidiary in the automotive industry stands as a global leader in the used vehicle marketplace. As a trailblazer in automotive services, they sought to redefine the wholesale vehicle buying and selling experience through comprehensive data management solutions. To achieve this, they partnered with Softensity’s Data Services Team to design and implement state-of-the-art ETL (Extract, Transform, and Load) solutions, leveraging AWS resources for optimal performance and efficiency.
Project overview
Softensity’s Data Services Team embarked on a mission to design end-to-end ETL solutions for various data streams. The team not only designed new data pipelines but also supported and fine-tuned existing services, creating innovative ways of data management, storage, and analysis. The team, divided into multiple subteams, specialized in capturing mixed and unstructured data and using advanced transformation methods to produce real-time reports utilized by teams across the subsidiary, customers, and auto dealers. A critical component of the project was the establishment of dynamic CI/CD pipelines to streamline the client’s data processing workflow.
Company description
This automotive subsidiary is a cutting-edge services provider headquartered in Atlanta, Georgia. Since its kick-off in November 2018, the company has delivered multiple end-to-end solutions aimed at redefining the wholesale vehicle buying and selling experience. The company adheres to the Scaled Agile Framework (SAFe), ensuring that all teams are well-coordinated and aligned with the overall strategic vision.
The challenge
The subsidiary required Python developers with basic knowledge of Terraform and hands-on experience in Big Data management and analytics. The Softensity team was tasked with:
- Performing preliminary source-to-target data mapping
- Building a data warehouse and a data lake using the current technological stack or upgrading it with new tools
The solution
Softensity’s approach was thorough and strategic, focusing on leveraging the best tools and practices to meet the client’s needs:
- CI/CD Pipelines: Leveraged GitHub Actions and AWS products to set up a dynamic CI/CD pipeline, migrating from Serverless to Terraform for seamless service configuration on AWS cloud. Jenkinsfile was edited to eliminate messy syntax and code duplicates, introducing unit testing and code linting for high code consistency.
- Data Transformation: Transferred EMR clusters from Sqoop to Hadoop, used AWS Lambda for file transfer from SFTP to the in-house data lake, and automated schema creation in Snowflake. Replaced a custom Python solution with PySpark, increasing processing speed fourfold.
- ETL Optimization: Readjusted AWS Glue job parameters, reducing ETL building and processing time by 15%. Designed a multi-regional custom solution to save data during AWS outages and set up cost optimization solutions that reduced AWS costs by 2%.
Technology Stack
- Programming Languages and Frameworks: Python, PyTest (+Moto), Spark, Java, Groovy, Bash, JavaScript, SQL, Node.js, Go language 2.
- Platforms
- AWS: Lambda, Glue, EC2, EMR, Step Function, Kinesis stream, Kinesis firehose, SNS, SQS, SES, ActiveMQ, DMS, CloudWatch, S3, DynamoDB, RedShift, ElasticSearch (OpenSearch) + Kibana, Secrets Manager, Parameter Store, AIM, VPC, Cognito
- Snowflake
- Internal API
- Additional Stack: Terraform, Poetry, Jenkins, LucidChart, Flake8/Pylint, SonarQube, PagerDuty, Rally, GitHub
Results
Through collaboration with the automotive subsidiary, Softensity’s team successfully identified and mitigated existing bottlenecks, performed source-to-target data mapping, and created dynamic pipelines for collecting, transforming, and storing various types of big data. The team built a data warehouse and a data lake, managing the client’s massive personal and transactional data. By developing powerful algorithms, the team transformed raw data into insightful and actionable reports used within the subsidiary.
Key Achievements:
- Automated merchandising: Enhanced operational efficiencies, reducing manual processes.
- Improved product availability: Ensured better product availability and variety.
- Cost reduction: Significant cost reductions, contributing to improved margins.
- Enhanced customer experience: Consistent, accurate, and engaging product information across multiple touchpoints.
- Rapid response: Quickly adapted to market changes and industry challenges.
Conclusion
Softensity’s collaboration with this leading automotive subsidiary exemplifies the power of strategic partnership and innovation in data management. By leveraging advanced technologies and methodologies, Softensity has not only met but exceeded the client’s expectations, ensuring seamless data processing and robust infrastructure. This case study stands as a testament to the transformative impact of well-executed data management solutions in the automotive industry.