Design, build, and optimize scalable data pipelines using Databricks and Apache Spark to enable data-driven decision-making. Collaborate with data scientists to transform data and ensure quality.
Description and Requirements
Role Summary
We are seeking a skilled Databricks Data Engineer to design, build, and optimize scalable data pipelines and data platforms using the Databricks Lakehouse architecture. The role involves working closely with data scientists, analysts, and business stakeholders to enable data-driven decision-making through robust, high-quality data solutions.
A Databricks Data Engineer primarily focuses on building and maintaining pipelines that transform raw data into usable insights using platforms like Apache Spark and Databricks.
Key Responsibilities
1. Data Pipeline Development
2. Data Processing & Transformation
3. Performance Optimization
4. Data Quality & Governance
5. Platform & Workflow Management
6. Collaboration & Stakeholder Engagement
Required Skills & Competencies
Technical Skills
Engineering Practices
Soft Skills
Qualifications
Nice-to-Have
Tools & Technologies
About MetLife
Recognized on Fortune magazine's list of the "World's Most Admired Companies" and Fortune World's 25 Best Workplaces™, MetLife, through its subsidiaries and affiliates, is one of the world's leading financial services companies; providing insurance, annuities, employee benefits and asset management to individual and institutional customers. With operations in more than 40 markets, we hold leading positions in the United States, Latin America, Asia, Europe, and the Middle East.
Our purpose is simple - to help our colleagues, customers, communities, and the world at large create a more confident future. United by purpose and guided by our core values - Win Together, Do the Right Thing, Deliver Impact Over Activity, and Think Ahead - we're inspired to transform the next century in financial services. At MetLife, it's #AllTogetherPossible . Join us!
#BI-Hybrid
Role Summary
We are seeking a skilled Databricks Data Engineer to design, build, and optimize scalable data pipelines and data platforms using the Databricks Lakehouse architecture. The role involves working closely with data scientists, analysts, and business stakeholders to enable data-driven decision-making through robust, high-quality data solutions.
A Databricks Data Engineer primarily focuses on building and maintaining pipelines that transform raw data into usable insights using platforms like Apache Spark and Databricks.
Key Responsibilities
1. Data Pipeline Development
- Design, develop, and maintain scalable batch and streaming data pipelines on Databricks
- Implement end-to-end ingestion, transformation, and modeling (Bronze, Silver, Gold layers)
- Build ETL/ELT workflows for structured and unstructured data
2. Data Processing & Transformation
- Process large-scale data using Apache Spark (PySpark/Scala)
- Perform data cleansing, transformation, and enrichment
- Enable efficient data modeling for analytics and reporting
3. Performance Optimization
- Optimize Spark jobs for performance (partitioning, joins, caching)
- Troubleshoot and resolve pipeline performance bottlenecks
- Tune clusters, workloads, and resource utilization
4. Data Quality & Governance
- Ensure data quality, consistency, and reliability across pipelines
- Implement validation checks, monitoring, and alerting mechanisms
- Apply governance practices such as schema enforcement and versioning
5. Platform & Workflow Management
- Develop and manage Databricks notebooks, jobs, and workflows
- Orchestrate pipelines using tools like Databricks Workflows / Airflow
- Monitor pipeline execution and ensure SLAs are met
6. Collaboration & Stakeholder Engagement
- Work closely with data scientists, analysts, and architects to gather requirements
- Translate business requirements into scalable data solutions
- Support downstream reporting, analytics, and ML use cases
Required Skills & Competencies
Technical Skills
- Strong experience with Databricks & Apache Spark
- Proficiency in Python / PySpark / Scala / SQL
- Experience with ETL/ELT and Data Warehousing concepts
- Familiarity with Delta Lake, Lakehouse architecture
- Experience with Cloud platforms (Azure / AWS / GCP)
- Knowledge of big data technologies (Kafka, Hadoop, etc.)
Engineering Practices
- Experience with CI/CD, Git, and SDLC practices
- Exposure to pipeline orchestration tools (Airflow, ADF, etc.)
- Understanding of data governance and security
Soft Skills
- Strong problem-solving and analytical capabilities
- Effective communication and stakeholder management
- Ability to work in cross-functional teams
Qualifications
- Bachelor's / Master's degree in Computer Science, Engineering, or related field
- Typically 3+ years of experience in data engineering or big data environments
- Certifications in Databricks or cloud platforms are a plus
Nice-to-Have
- Experience with ML pipelines or feature engineering
- Exposure to real-time streaming frameworks
- Knowledge of Power BI / analytics tools (for enterprise setups)
Tools & Technologies
- Databricks (Lakehouse Platform)
- Apache Spark (PySpark / Scala)
- SQL
- Delta Lake
- Azure Data Factory / Airflow
- Cloud Storage (ADLS, S3, GCS)
About MetLife
Recognized on Fortune magazine's list of the "World's Most Admired Companies" and Fortune World's 25 Best Workplaces™, MetLife, through its subsidiaries and affiliates, is one of the world's leading financial services companies; providing insurance, annuities, employee benefits and asset management to individual and institutional customers. With operations in more than 40 markets, we hold leading positions in the United States, Latin America, Asia, Europe, and the Middle East.
Our purpose is simple - to help our colleagues, customers, communities, and the world at large create a more confident future. United by purpose and guided by our core values - Win Together, Do the Right Thing, Deliver Impact Over Activity, and Think Ahead - we're inspired to transform the next century in financial services. At MetLife, it's #AllTogetherPossible . Join us!
#BI-Hybrid
Similar Jobs at MetLife
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
The Senior Data Modeller II translates business requirements into scalable data models, ensuring data integrity and best practices while providing leadership in data analytics talent.
Top Skills:
Azure SynapseData Modelling Tools Such As ErwinDatabricksETLJSONNoSQLOracleRestSoapSQLXML
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
The Senior Data Modeler designs and maintains data models to support analytics, mentors junior modelers, and collaborates across business functions.
Top Skills:
Azure SynapseDatabricks
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
The role involves developing full-stack applications using React and Spring Boot, integrating services, and ensuring code quality while collaborating across teams.
Top Skills:
AzureCSSDockerGitHTMLJavaJavaScriptKubernetesNoSQLReactSpringSpring BootSQLTypescript
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
