LeadDataEngineer

## Description 3Pillar is an AI transformation partner on a mission to help enterprises build the AI-native products and intelligent agents that will define the next era of business. With teams across North America, Europe, Latin America, and Asia, we work with the most ambitious companies in financial services, healthcare, media, and technology — helping them move faster, modernize boldly, and compete on their own terms. Our HelixAI platform and Helix Pods delivery model put our engineers at the center of real agentic transformation — doing work that is open, portable, and built to last. We are building the future of enterprise AI   We are looking Lead Data Engineer to build, operate, and continuously improve the data pipelines, retrieval infrastructure, and ML/LLMOps foundations that power our AI initiatives. The resource will work on turning reference architectures and data contracts into robust, production-grade implementations that serve conversational AI assistants, dashboard copilots, autonomous agents, RAG applications, and predictive ML models. ## Key Responsibilities Data Pipeline Engineering : Build, test, and maintain production pipelines (batch & real-time) on Snowflake, PySpark, Delta Lake, and Kafka. Implement data quality checks, schema validation, and alerting at every pipeline stage. Migrate legacy ETL/DWH to cloud-native AWS/Azure architectures with measurable latency and cost improvements. Maintain CI/CD pipelines: automated testing, deployment, rollback, and IaC (Terraform, GitHub Actions). RAG, Vector & Retrieval Infrastructure: Build end-to-end retrieval infrastructure: document ingestion, embedding pipelines, vector store management (Pinecone, FAISS, ChromaDB, OpenSearch), and hybrid retrieval layers. Implement chunking, metadata filtering, and re ranking — tuning for precision, recall, and latency. Maintain data freshness and index consistency; instrument with context relevance and faithfulness metrics. Semantic Layer & Knowledge Infrastruct