Lead Data / ML Engineer

About the Role:

We are looking for a Lead Data / ML Engineer to take end-to-end technical ownership of a large-scale data and ML platform on AWS. You will design and build PySpark pipelines, bring ML models and LLM and agent-based capabilities into production, and own what you build from first design through to running it reliably in production.

This is a hands-on role. Architecture and leadership experience is welcome, but you will still write code every day, and you will be the person others turn to when a pipeline breaks, a job slows down, or a design needs a second opinion.

‍

About the Work:

You will work with high-volume consumer transaction data, turning it into insights and predictive models that help understand and anticipate consumer behaviour. The work spans large-scale data processing, feature engineering, machine learning, and applying LLMs and agents to enrich and make sense of the data.

‍

‍Experience: 7+ Years

Location: Kolkata (Rajarhat–Newtown)

Mode of Working: WFO / Hybrid / Remote

Timing: Ability to work in the US Eastern time zone. This may be relaxed to half day IST and half day US EST based on project needs.

What We Are Looking For:

• End-to-end ownership: you take a problem from design to production and stay accountable for it once it is live
• Hands-on depth: you write, review and debug production code yourself, not only design it
• Troubleshooting leadership: you help teammates diagnose failing jobs, performance problems and data issues, and leave them better at solving the next one
• Practical judgement on LLMs and agents: you know where they add real value in a data pipeline and how to measure whether they are working

‍

Key Responsibilities:

• Design, build and optimize large-scale ETL/ELT pipelines in PySpark on AWS (EMR, Glue), with strong attention to partitioning, skew, shuffle and cost.
• Productionize ML models and feature pipelines, including training data pipelines, batch and real-time inference, and model serving on SageMaker.
• Build LLM and agent-powered capabilities into data workflows, such as document and entity extraction, classification and enrichment at scale, RAG pipelines, and tool-calling agents, along with evaluation and monitoring of their output quality.
• Orchestrate complex workflows with Apache Airflow (MWAA or self-managed), ensuring reliability, observability and SLA adherence.
• Architect storage across S3 (data lake), Redshift (warehouse) and RDS, applying best practices for data modeling, partitioning, compression and cost optimization.
• Build containerized data services with Docker and ECS/Fargate, with CI/CD through GitHub Actions.
• Develop event-driven and serverless processing with Lambda, SQS, SNS and EventBridge.
• Use AI coding assistants (Claude Code, Cursor, Kiro) to speed up delivery, and set the team's standards for reviewing and validating AI-generated code.
• Be the go-to person for production issues: lead root-cause analysis on failing or slow jobs, data quality incidents and model problems, and help teammates debug their own work.
• Review code, guide design decisions and help engineers on the team grow technically.
• Implement data quality, monitoring and alerting frameworks, and enforce data governance, security and access-control policies.
• Contribute to infrastructure as code using Terraform, CloudFormation or CDK.

‍

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science or a related field.
• 7+ years in data engineering or ML engineering, with at least 4 years of hands-on PySpark on production workloads at scale.
• Strong AWS production experience across EMR, S3, Redshift, RDS, Lambda, SageMaker and ECS.
• Practical, hands-on experience building with LLMs: prompt design, embeddings and vector search, RAG, agent or tool-calling frameworks, and evaluating LLM output quality.
• Solid understanding of the ML lifecycle: feature engineering, training and validation, model deployment and monitoring.
• Experience working with large transactional or behavioural datasets is a strong plus.
• Strong Python and SQL. TypeScript is a plus.
• Experience with Apache Airflow and Docker in production.
• A track record of owning systems end to end in production, including on-call or production support, and of helping other engineers troubleshoot.
• Strong grasp of data modeling, governance and security best practices.

‍

Good To Have:

• AWS certifications (Data Engineer Associate, Machine Learning Specialty, Solutions Architect)
• Experience with feature stores, MLOps tooling or model serving at scale
• Infrastructure as code (Terraform, CloudFormation, CDK)
• Experience optimizing costs for large AWS data workloads

‍

Tech Stack at a Glance:

• Compute & Processing: PySpark on EMR, Glue, Lambda, ECS/Fargate, SageMaker
• LLM & Agents: LLM APIs, embeddings and vector stores, RAG, agent frameworks
• Orchestration: Apache Airflow (MWAA), Step Functions
• Storage & Warehousing: S3, Redshift, RDS (PostgreSQL / MySQL)
• Containers & DevOps: Docker, ECS, GitHub Actions, Terraform
• AI-Assisted Dev: Claude Code, Cursor, Kiro
• Languages: Python, SQL, Bash