Similar Jobs
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
As a Data Engineer, you'll build data models, design ETL processes, improve data quality, and mentor junior engineers, leveraging AWS and various data technologies.
Top Skills:
AirflowAmazon Web ServicesData Build ToolDatabricksEmrHiveJavaKinesisPythonRdsS3ScalaSparkSQLSqs
Software
Lead the design and implementation of scalable data engineering solutions on Azure, optimizing performance and mentoring teams. Ensure data quality and security while delivering analytics-ready datasets through collaboration.
Top Skills:
Azure Data FactoryAzure Data LakeMicrosoft FabricPower BIPythonSQLSynapse
Healthtech • Pharmaceutical
The Sr. Data Engineer designs and operates large-scale data solutions on GCP, optimizing data pipelines, mentoring team members, and providing insights for business decisions.
Top Skills:
AirflowCi/CdCloud BigqueryCloud DataflowCloud FunctionsCloud PubsubConcourseGitGoogle Cloud PlatformJenkinsLookerPythonShell ScriptingSQLTableau
At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Lead, Data Engineering, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Key Responsibilities
- Understanding the client requirement and architect robust data platforms on multiple cloud technologies.
- Take the lead on creating, defining and driving the data engineering solutions for complex client engagements
- Develop ETL/ELT processes to help extract and manipulate data from multiple sources.
- Work with the leadership to set the standards for software engineering practices within the data engineering team
- Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources.
- Monitor data systems performance and implement optimization strategies.
- Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
- Partner with data scientists and functional leaders in sales, marketing, and product to deploy machine learning models in production.
- Choose and use the right analytical libraries, programming languages, and frameworks for each task.
- Lead data engineering team.
Minimum Qualifications
- Demonstrated expertise with a minimum of 7+ years of experience in data engineering domain
- Advanced SQL skills and experience with relational databases and database design.
- Experience with data modeling
- Experience working with cloud Data Warehouse solutions (e.g., Snowflake, Redshift, BigQuery, Azure Synapse, etc.).
- Strong Python skills with hands-on experience on Pandas, NumPy and other data related libraries
- Experience with working on large data sets and distributed computing (e.g. Hive/Hadoop/Spark/Presto/MapReduce).
- In depth knowledge and understanding of data streaming tools like Apache Kafka, AWS Kinesis.
- Proficient in data pipeline and workflow management tools e.g. Airflow, AWS Step functions etc
- Experience with data visualization tools like PowerBI, Tableau, AWs Quicksight etc
- Experience with the AWS/Azure/GCP data engineering services
- Knowledge of AWS services viz., EC2, EMR, S3, Lambda, EBS, IAM, Redshift, RDS, GLUE ETL
- OR
- Knowledge of Azure services viz., ADF, Databricks, Azure Synapse Analytics, ADLS Gen2, Azure Devops
- Good understanding of NoSQL databases like Redis, Cassandra, MongoDB, or Neo4j.
- Excellent problem-solving, communication, and organizational skills.
- Proven ability to manage a highly skilled data engineering team.
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
