Hands-on Lead Data Engineer to design, build, and scale Databricks-based ETL/ELT pipelines and lakehouse architectures using PySpark and Delta Lake. Responsibilities include ingestion (batch & real-time), Delta features, pipeline orchestration, Spark optimization, data quality/governance, production support, and migrating legacy platforms to cloud.
Lead Data Engineer
Primary Skills
- ETL Fundamentals, SQL, SQL (Basic + Advanced), Python, Data Warehousing, Modern Data Platform Fundamentals, Data Modelling Fundamentals, PLSQL, T-SQL, Stored Procedures
Specialization
- Databricks Engineering: Data Engineer
Job requirements
- Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark
- Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers)
- Develop data ingestion frameworks from multiple sources (batch & real-time)
- Implement Delta Lake features including schema evolution, ACID transactions, and optimization
- Create and manage Databricks notebooks, workflows, and jobs for production pipelines
- Ensure data quality, governance, and lineage across pipelines
- Optimize Spark jobs for performance (partitioning, caching, tuning)
- Collaborate with cross-functional teams (Data Scientists, Analysts, Architects)
- Support production deployments, monitoring, and troubleshooting of data pipelines
- Contribute to migration of legacy data platforms to modern cloud-based architecture
- Strong experience in:
- Databricks (Workflows, Notebooks, Jobs)
- PySpark / Spark SQL
- SQL & Data Warehousing concepts
- Hands-on experience with:
- Delta Lake / Lakehouse architecture
- Data pipeline development & orchestration
- Good understanding of:
- Data modeling, ETL/ELT processes, SCD concepts
- Experience in at least one:
- AWS (S3, Glue, EMR) / Azure (ADF, ADLS, Synapse)
- Workflow tools:
- Airflow / Databricks Workflows
- Version control:
- Git / CI-CD pipelines
- Experience with:
- Streaming frameworks (Kafka / Spark Streaming)
- DBT / Snowflake / Redshift / Fabric
- Exposure to:
- Unity Catalog, Autoloader, Delta Live Tables
- 5–10 years of experience in Data Engineering
- Minimum 2–4 years of hands-on Databricks experience
- Proven experience building enterprise-scale data platforms and pipelines
- Strong problem-solving and analytical skills
- Ability to work in a fast-paced, agile environment
- Good communication and stakeholder management
- Work on cutting-edge data & AI programs
- Opportunity to build large-scale data platforms
- Collaborative and innovation-driven culture
Job Description – Data Engineer (Databricks)
Company: Brillio Technologies
Role: Data Engineer – Databricks
Experience: 5–10 Years
Location: India (Hybrid/3 days)
Employment Type: Full-time
About Brillio
Brillio is a leading digital transformation partner delivering innovative solutions across Data, Cloud, AI, and Digital Engineering. We help global enterprises unlock value through cutting-edge technology and data-driven insights.
Role Overview
We are looking for a hands-on Data Engineer with strong Databricks expertise to design, build, and scale modern data platforms. The ideal candidate will have experience in PySpark, Delta Lake, and cloud-native data engineering and will work closely with analytics, product, and business teams to deliver high-performance data pipelines.
Key Responsibilities
Technical Skills (Must-Have)
Cloud & Tools
Good to Have
Experience Required
Soft Skills
Why Join Brillio
Similar Jobs
.jpeg)
Artificial Intelligence • Hardware • Information Technology • Machine Learning
The role involves leading SSD product development at Micron, defining strategies, managing customer relationships, and overseeing product roadmaps in the Data Center segment.
Top Skills:
CrystaldiskmarkNand StoragePcm10Ssd
Artificial Intelligence • Hardware • Information Technology • Machine Learning
The role involves layout design for analog and digital blocks, performing layout verification, developing layouts using tools, and supporting team members. Responsibilities include project management and effective communication with global teams.
Top Skills:
Ai ToolsAnalog DesignAutomationDigital DesignDrcLayout ToolsLvsMixed-Signal Design
Artificial Intelligence • Hardware • Information Technology • Machine Learning
Provide verification support for DRAM and emerging memory architectures by simulating, analyzing, and debugging pre-silicon IP designs. Develop test cases and maintain test benches while collaborating internationally on verification methodologies.
Top Skills:
AmsMixed SignalPerlPythonRvmSpiceUvmVerilogVerilog A/Ms
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
