Design, build, and optimize scalable Spark/PySpark data pipelines on Databricks. Develop ETL/ELT workflows using AWS EMR, S3, and Hadoop/Hive. Build and maintain data lake and warehouse solutions, integrate APIs, orchestrate workflows with Airflow/Autosys, ensure data quality and governance, and tune performance. Collaborate with analytics, product, and engineering teams.
Lead Data Engineer
Primary Skills
- Athena, Step Functions, Spark - Pyspark, ETL Fundamentals, SQL (Basic + Advanced), Glue, Python, Lambda, Data Warehousing, EBS /EFS, AWS EC2, Lake Formation, Aurora, S3, Modern Data Platform Fundamentals, PLSQL, Data Modelling Fundamentals, Cloud front
Specialization
- AWS Data EngineerIng Basic: Senior Data Engineer
Job requirements
- Design and build scalable data pipelines using Spark / PySpark on Databricks
- Develop and optimize ETL/ELT workflows for large-scale data processing
- Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing
- Build and maintain data lake and data warehouse solutions
- Develop and integrate APIs for data ingestion and consumption
- Implement data processing workflows using Airflow / Autosys
- Optimize data performance using partitioning, caching, and query tuning
- Handle structured and semi-structured data from multiple sources
- Ensure data quality, governance, and reliability of pipelines
- Collaborate with cross-functional teams (Analytics, Product, Engineering)
- Strong expertise in:
- Apache Spark / PySpark
- Databricks (mandatory)
- SQL (PostgreSQL or similar)
- Hands-on experience with:
- AWS EMR, S3
- Hadoop, Hive ecosystem
- Programming skills:
- Python (must-have)
- Scala (good exposure)
- Strong knowledge of:
- UNIX / Shell scripting
- ETL pipelines and data engineering fundamentals
- Experience with:
- Elasticsearch (data storage & retrieval)
- Workflow orchestration tools (Airflow, Autosys)
- Exposure to:
- API development & integration
- Version control tools (Git / SVN)
- Basic knowledge of HTML (for web/data tasks)
- Minimum 5+ years of Data Engineering experience
- Strong hands-on experience in Databricks + Big Data stack
- Proven experience working on large-scale distributed systems
- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
- Work on cloud-native, large-scale data platforms
- Exposure to cutting-edge Databricks & AWS ecosystems
- Collaborative and high-performance engineering culture
Job Description – AWS Senior Data Engineer
Company: Brillio Technologies
Role: Senior Data Engineer
Experience: 5–10 Years
Location: India (Hybrid/Remote)
Employment Type: Full-time
Role Overview
We are looking for a Senior Data Engineer with strong Databricks and Big Data expertise to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands-on experience with Spark, PySpark, AWS ecosystem, and modern data architectures, along with exposure to APIs and real-time data processing.
Key Responsibilities
Technical Skills (Must-Have)
Additional Skills
Experience Required
Education
Why Join Brillio
Similar Jobs
.jpeg)
Artificial Intelligence • Hardware • Information Technology • Machine Learning
The role involves leading SSD product development at Micron, defining strategies, managing customer relationships, and overseeing product roadmaps in the Data Center segment.
Top Skills:
CrystaldiskmarkNand StoragePcm10Ssd
Artificial Intelligence • Hardware • Information Technology • Machine Learning
The role involves layout design for analog and digital blocks, performing layout verification, developing layouts using tools, and supporting team members. Responsibilities include project management and effective communication with global teams.
Top Skills:
Ai ToolsAnalog DesignAutomationDigital DesignDrcLayout ToolsLvsMixed-Signal Design
Artificial Intelligence • Hardware • Information Technology • Machine Learning
Provide verification support for DRAM and emerging memory architectures by simulating, analyzing, and debugging pre-silicon IP designs. Develop test cases and maintain test benches while collaborating internationally on verification methodologies.
Top Skills:
AmsMixed SignalPerlPythonRvmSpiceUvmVerilogVerilog A/Ms
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
