Infotel UK
Technology
PhytonwithSparkDeveloper
Neural analysis suggests this role is
optimal for Mid+ candidates.
“Phyton with Spark Developer at Infotel UK. Skills: Python, PySpark, SQL, ETL. Design ingestion pipelines. Develop ingestion pipelines”
Industry & Context.
Troubleshoot performance bottlenecks; Troubleshoot memory issues; Troubleshoot job failures; Resourceful to quickly understand complexities
What They're Looking For.
Must Have
Python experience, PySpark experience, SQL experience, ETL experience, Object oriented experience, API experience, Microservices experience, Design patterns knowledge, Development principles knowledge, Linux/Unix environment knowledge, Testing phases knowledge, Documentation knowledge, New framework knowledge, Agile knowledge, Scrum knowledge, DevOps knowledge, Data Engineering development experience, Python (NumPy, pandas, Python Frameworks, Restful APIs, MS-SQL or Oracle) hands-on, PySpark (DataFrames, Spark SQL, Structured Streaming, performance tuning) hands-on, Advanced SQL (complex queries, stored procedures, query optimization) hands-on, Python packages (Pandas, NumPy) for cleaning, wrangling, analysis, visualization knowledge, Code/scripts development and maintenance experience, Bug fixing experience, Production support experience, Build tools (Maven) experience, DevOps tools (Bitbucket, Jenkins) experience, Ability to learn diverse technologies, Willingness to work on diverse technologies, Self-motivated, Good interpersonal skills, Good communication skills, Good co-ordination skills
Nice to Have
Flask knowledge, Front-end technologies knowledge
What You'll Do.
Design ingestion pipelines
Develop ingestion pipelines
Write analytical UDFs
Write window functions
Translate requirements
Maintain CI/CD pipelines
Monitor production workloads
Troubleshoot performance bottlenecks
Troubleshoot memory issues
Troubleshoot job failures
Document data lineage
Document pipeline design
Document operational run-books
Provide recommendations
Contribute towards innovation
Suggest new technical practices
Participate in Agile ceremonies
Mentor junior engineers
Champion best practices
Evaluate emerging technologies
Deliver proof-of-concepts
How You'll Work.
Team & Collaboration
CEFS data architects; Data scientists; Business analysts; Agile ceremonies; Junior engineers
Communication Scope
Co-ordination skills
Process & Methodology
Agile, Scrum, Sprint planning, Daily standups, Retrospectives
Full Job Description
**Responsibilities** _Direct Responsibilities_ · Design & develop robust ingestion, transformation, and enrichment pipelines with Python, PySpark, and SQL. * Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting. * Collaborate with CEFS data architects, data scientists, and business analysts to translate functional requirements into technical specifications. * Unit‑test, integrate‑test, and review code. * Maintain CI/CD pipelines (Git, Jenkins, Docker) for automated build, test, and deployment of jobs. * Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures. * Document data lineage, pipeline design, and operational run‑books in Confluence/SharePoint. · Keep up to date with latest technologies and trends and provide input, expertise and recommendations. _Contributing Responsibilities_ · Contribute towards innovation (e.g. AI/ML); suggest new technical practices for efficiency improvement. * Participate in Agile ceremonies (sprint planning, daily standups, retrospectives) and help groom the backlogs. * Mentor junior engineers and champion best practices in Python coding, Spark optimization, and data‑engineering patterns. * Evaluate emerging technologies and deliver proof‑of‑concepts for CEFS. **Technical & Behavioral Competencies** · Resourceful to quickly understand complexities involved and provide the way forward. · Good experience in technical analysis of n-tier applications with multiple integrations using object oriented, APIs & Microservices approaches. · Strong knowledge about design patterns and development principles. · Inclination and prior experience of working across SQL, Python and ETL. · Strong Hands-on experience - **Python** (NumPy, pandas, Python Frameworks, Restful APIs, MS-SQL or Oracle. · **PySpark** \- DataFrames, Spark SQL, Structured Streaming, performance tuning (partitioning, caching, broadcast joins). · **Advanced SQL** – complex queri
Applying for this Phyton with Spark Developer role?
Most applicants get filtered before a human reads their resume. See if yours makes the cut.
ANONYMOUS · UNFILTERED
What do employees actually say about Infotel UK?
Real rants from real employees. Read before you apply.