Design, build, and maintain scalable AWS-based ETL/ELT data pipelines and data platform solutions. Develop SQL and Python code, create tests and documentation, monitor and optimize pipeline performance, and collaborate with BI, AI/ML, and infrastructure teams while following best practices and data governance.
Key Responsibilities
- Design, develop, and optimize scalable ETL/ELT pipelines for data ingestion and transformation.
- Build and maintain robust data pipelines using AWS cloud services.
- Partner with Business Intelligence (BI), AI/ML, and Infrastructure teams to deliver reliable and scalable data platform solutions.
- Develop efficient SQL queries and Python scripts for data processing, automation, and analytics.
- Create and maintain technical documentation, including solution designs, data flow diagrams, and operational guides.
- Develop comprehensive unit tests and perform code reviews to ensure high-quality, reliable, and maintainable code.
- Monitor, troubleshoot, and optimize data pipelines to ensure performance, scalability, and reliability.
- Work on multiple projects simultaneously while managing priorities in a fast-paced environment.
- Follow coding standards, best practices, and data governance guidelines.
Required Skills & Qualifications
- 3–5 years of hands-on experience in Data Engineering.
- Strong experience in designing and developing ETL/ELT pipelines.
- Proficiency in SQL and Python programming.
- Hands-on experience with AWS services, including:
- Amazon Redshift
- AWS Lambda
- Amazon S3
- AWS Glue (preferred)
- AWS IAM
- Amazon CloudWatch
- Experience with workflow orchestration tools such as Apache Airflow (preferred).
- Understanding of data warehousing concepts and dimensional data modeling.
- Knowledge of version control systems such as Git.
- Experience writing unit tests and following software development best practices.
- Strong analytical, problem-solving, and debugging skills.
- Excellent communication and documentation skills.
- Ability to manage multiple tasks and work effectively in a collaborative environment.
Preferred Skills
- Experience with CI/CD pipelines and DevOps practices.
- Knowledge of Spark, PySpark, or Databricks is an advantage.
- Exposure to AI/ML data pipelines and feature engineering.
- Familiarity with Agile/Scrum development methodologies.
- AWS Certification (Developer Associate or Data Engineer Associate) is a plus.
Key Competencies
- AWS Data Engineering
- ETL/ELT Development
- SQL & Python
- Amazon Redshift
- AWS Lambda
- Amazon S3
- Data Warehousing
- Unit Testing
- Technical Documentation
- Problem Solving
- Team Collaboration
- Multitasking
- Agile Development
Educational Qualification
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- Relevant AWS certifications will be an added advantage.
Similar Jobs
Healthtech • Pet • Biotech
Design, build, and operate production-grade near-real-time data pipelines and APIs on AWS (Kinesis, Glue/Spark, Lambda, EMR). Implement IaC and CI/CD, optimize data lakes and operational models, improve observability and reliability, and participate in on-call support.
Top Skills:
Api GatewayAws GlueAws KinesisAws LambdaCloudFormationDynamoDBEmrIamJenkinsJSONPythonRest ApiS3Shell ScriptingSnsSparkSqsTerraformXML
Agency • Information Technology
Design, build and maintain ETL mappings and data models; write SQL and Snowflake queries; manage ETL/infrastructure on AWS (EC2); use Informatica, Control-M and ServiceNow; script in Python/Shell/PowerShell; collaborate with onshore/offshore teams and define BI delivery strategy.
Top Skills:
AngularAws Ec2Control-MDaticalInformaticaJenkinsPower BIPowershellPythonServicenowShellSnowflakeSQLStashTableauUdeployUnix
Agency • Information Technology
Design, build, and maintain ETL pipelines and frameworks on AWS. Deploy and troubleshoot containerized applications (Docker, EKS), support CI/CD and build tooling, work with SQL/Oracle and ORM tools, and use job schedulers like Autosys/Airflow. Collaborate in Agile/SCRUM environments.
Top Skills:
AirflowArtifactoryAutosysAWSDockerEksEtl FrameworkGitHibernateIbatisJenkinsMavenOraclePythonSQLStashUdeployUnix Shell
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
