Lead Data Engineer

Posted 3 Days Ago
Be an Early Applicant
Bengaluru, Karnataka
Senior level
Artificial Intelligence • Machine Learning
The Role
The Lead Data Engineer will guide a team of data engineers in creating and optimizing data processes for a large consumer data lake. Responsibilities include designing data pipelines, ensuring data quality, collaborating with cross-functional teams, and implementing data engineering best practices.
Summary Generated by Built In

ABOUT US

We are Ai Palette. We think the world would be a nicer place to be if Food Companies could create products that the consumers really want. So we are making it happen. We want to be the most preferred Food AI company in the world. We’re making it possible by building an AI-powered SaaS platform based on our founders’ experience in the Food Industry & AI.

We are a Series A round funded company with experienced investors like pi Ventures, Exfinity ventures and Anthill ventures. We are building a global company and are already working with customers across the globe. Our customers include Fortune 500 companies and FMCG brands.

We have won several accolades & awards in our short period of existence such as:

  • Top 10 Global Food & Retail Tech startup at Kickstart Innovation'20 at Switzerland
  • Top 15 Global Food tech Startup at Slingshot'19 at Singapore
  • Top 200 Global startups at HKSTP Epic'20 at HongKong
  • Top 500 Global Food Tech startups by Forward Fooding


WHAT’S IT LIKE TO WORK AT AI PALETTE? 


We're a growing technology startup headquartered in Singapore with our Engineering base in Bangalore and a team in US, We PaletteeRs are a highly passionate and motivated bunch of people that help each other do remarkable things and achieve extraordinary results every single day. We are active learners, have a positive impact on consumers’ lives and settle for nothing short of excellence. We face challenges together and we win together. In our vision to build the World's First AI Platform for New Product Innovation, there isn’t a day that goes by where we don’t have “Aha” moments.  We strive together to deliver world-class solutions that transform the way consumer products of today and tomorrow will be created. Join us!

Responsibilities:

  1. Lead a team of data engineers specialising in data engineering, providing technical guidance, mentoring, and performance feedback.
  2. Optimise and automate processes for regularly refreshing the world's largest consumer data lake, purpose-built for CPG.
  3. Drive higher levels of automation and faster process (TAT) for weekly, monthly, and quarterly refresh processes.
  4. Drive initiatives for ongoing improvements in the quality and completeness of data on the platform
  5. Design, develop, and maintain spark data pipelines, ensuring efficient and timely data acquisition from various sources.
  6. Develop and enforce data engineering best practices, standards, and processes.
  7. Identify and resolve issues related to data engineering, such as handling complex data structures, mitigating bottlenecks, and addressing scale issues.
  8. Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and develop scalable data engineering solutions.
  9. Collaborate with stakeholders to define data engineering project requirements, timelines, and deliverables.
  10. Perform data extraction, transformation, and loading (ETL) tasks to convert raw data into usable formats for downstream analysis and processing.
  11. Monitor data pipeline performance and implement mechanisms to ensure the reliability and scalability of data pipelines.
  12. Stay up to date with the latest trends and advancements in data engineering techniques and related technologies.

Requirements:

  • Bachelor's or master's degree in computer science, data engineering, or a related field.
  • Proven experience (8+ years) working as a data engineer, with a specialisation in data engineering.
  • Strong programming skills in languages such as Python, Java, and Spark Pipelines.
  • Proficiency in SQL and database technologies (e.g., PostgreSQL, MySQL, or Oracle) for data storage and retrieval.
  • Familiarity with cloud platforms (e.g., AWS, Azure, or Google Cloud) and related data services for scalable and reliable data crawling.
  • Knowledge of data modeling, data warehousing, and ETL processes.
  • Experience with data warehouses such as Snowflake, Redshift, Elastic Search
  • Strong analytical and problem-solving skills, with a focus on data quality and accuracy.
  • Excellent leadership and team management abilities, with a proven track record of leading data engineering teams.
  • Effective communication and collaboration skills, with the ability to explain complex technical concepts to non-technical stakeholders

BENEFITS

  • Great progression opportunities - we want you to grow with us.
  • Look after yourself with health insurance including Hospital/Surgical.
  • Learn new skills with sponsored training on MOOCs such as Coursera, Udemy.  

EQUAL OPPORTUNITY: 

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

Top Skills

Spark
The Company
93 Employees
On-site Workplace
Year Founded: 2018

What We Do

Ai Palette enables CPG companies to create consumer-winning products using Artificial Intelligence and Machine Learning. Its patented technology identifies emerging trends across 61B data points collated from 150+ data sources in real time, uncovering consumer drivers and motivations, and helping in creating product concepts that meet unmet consumer needs. Its Natural Language Processing algorithm can also understand 18 different languages including Asian languages, making it truly one of a kind. With Ai Palette, you can: - Quickly access consumer trends and insights without spending time and money on traditional market research. FoodGPT leverages a vast pool of over 61 billion data points, ensuring you stay in sync with the dynamic consumer landscape. - Combine 1st party, 2nd party and 3rd party data and data sources in a single place to get faster research without investing a lot of time on digging into data sources. - Generate compelling product concepts, fine tuned to resonate with your target audience, with elaborate product claims and messaging crafted in consumer language. - Validate winning product hypothesis by marrying sales, consumer panel data, and secondary research data. - Leverage powerful predictions for sales forecasting with each new product launch. - Make new markets penetration or establish novel categories successfully backed by consumer data. Headquartered in Singapore, it is currently working closely with some of the world's largest food companies, including Fortune 500 giants like Kelloggs', Nestle, Olam, Diageo etc.

Similar Jobs

Thanisandra Main Rd, Bangalore, Karnataka, IND
29154 Employees

Regeneron Logo Regeneron

Lead Data Engineer

Biotech • Pharmaceutical
Bengaluru, Karnataka, IND
15000 Employees

Cermati.com Logo Cermati.com

Lead Data Engineer (Java expertise required)

Fintech • Payments • Financial Services
Bengaluru, Karnataka, IND
468 Employees

Encora Logo Encora

Lead Data Engineer

Software • Consulting
Bangalore, Bengaluru, Karnataka, IND
7456 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account