Data Engineer - Big Data

Description

Summary: Seeking an experienced Data Engineer – Big Data to design, develop, maintain, and optimize scalable Data Lake pipelines and Big Data solutions within an international and collaborative environment. Highlights: 1. Work with large-scale Data Lake environments and Big Data technologies 2. Contribute to complex projects within an international and collaborative team 3. Opportunity for continuous learning and career growth Who are we? Quanteam, founding entity of RAINBOW PARTNERS, is a consulting firm specialized in Banking and Financial Services. With more than 1200 consultants across 10 countries, we combine business and IT expertise to support leading institutions in transformation projects. Our core areas: ➡ Quantitative Finance ➡ Risk \& Compliance ➡ Operations \& Finance ➡ Transformation \& Organization ➡ Information Systems **We are looking for a Data Engineer – Big Data** **CONTEXT** In this role, you will join a transversal and international team composed of Data Engineers, BI Developers, Data Analysts, Business Analysts, and Software Developers. Based in Porto, the team works closely with multiple stakeholders and technical teams in both Porto and Paris, developing data\-driven solutions for Technology and Operations within a Corporate Functions IT environment. You will contribute throughout the full lifecycle of data initiatives, from requirements assessment and use\-case identification to data preparation, implementation, industrialization, maintenance, and performance optimization. This is an excellent opportunity for an experienced Data Engineer who is looking to work with large\-scale Data Lake environments, strengthen their expertise in Big Data technologies, and contribute to complex projects within an international and collaborative environment. **MAIN MISSION** To design, develop, maintain, and optimize scalable Data Lake pipelines and Big Data solutions using Python, PySpark, Hadoop, and related technologies, ensuring data reliability, performance, automation, and alignment with business and technical requirements. **MAIN RESPONSIBILITIES** * Design, develop, and maintain Data Lake pipelines using Python and PySpark. * Analyse business and technical requirements and propose appropriate architecture solutions within the Data Lake environment. * Gather, document, examine, and manage application requirements in an Agile environment. * Develop and implement data use cases ranging from basic to complex scenarios. * Build scalable, efficient, and automated data\-processing solutions. * Work with large volumes of structured and unstructured data. * Develop and optimize Hive, Spark SQL, and SQL queries. * Apply Data Lake best practices, including partitioning, bucketing, file\-format selection, and metadata management. * Maintain and improve existing data pipelines and processes. * Investigate and resolve performance, processing, and data\-quality issues. * Analyse execution plans and identify opportunities for performance improvement. * Provide technical support to users and other internal teams. * Contribute to source\-code management, build, testing, and deployment activities. * Collaborate closely with Data Engineers, Business Analysts, BI Developers, Software Developers, and business stakeholders. * Clearly communicate technical findings, data insights, and project progress to internal and external stakeholders. * Produce and maintain technical and functional documentation. * Contribute to continuous improvement initiatives and suggest new approaches to existing processes. * Share technical knowledge and best practices across the Porto and Paris teams. **PROFILE \& SKILLS** * Degree in Computer Science, Information Systems, Data Engineering, Software Engineering, or a related field. * At least 3 years of professional experience in Data Engineering, Big Data, or a similar role. * Strong experience designing and developing end\-to\-end data pipelines. * Experience working with large\-scale Data Lake environments. * Solid understanding of data modelling principles. * Strong knowledge of SQL and data\-processing concepts. * Experience analysing and optimizing queries involving large data volumes. * Familiarity with Data Lake best practices, including partitioning, bucketing, Parquet or ORC formats, and metadata management. * Experience investigating and resolving data\-quality and performance issues. * Strong analytical and problem\-solving skills. * Ability to identify patterns, trends, seasonality, and other relevant data behaviours. * Ability to translate business and application requirements into technical solutions. * Experience working with Agile methodologies and tools, including Scrum or Kanban. * Ability to work independently and become quickly autonomous. * Proactive, organised, and able to effectively prioritise different tasks. * Ability to work effectively within an international and distributed team. * Strong communication and interpersonal skills. * Innovative mindset and willingness to suggest improvements. * Willingness to learn, collaborate, and share knowledge with other team members. * Availability to travel abroad for short periods when required. **TECHNICAL SKILLS** * Python * PySpark * Hadoop * HDFS * Hive * Kerberos * Starburst * SQL * Data Modelling * Data Lake Architecture * Big Data Technologies * NoSQL Technologies * Visual Studio Code or similar development environments * Git * Jenkins * XL Deploy * XL Release * CI/CD * DevOps Concepts * Agile Methodologies **WHAT WE OFFER** * Gross annual salary aligned with your experience and seniority. * Permanent contract (CTI). * Meal allowance. * Health Insurance. * Extra holiday: Christmas Eve or New Year's Eve. * Access to Quanteam Academy for continuous learning and certifications. * Elevate Program to support professional development and career growth. * Career progression plan. * Ongoing follow\-up and technical support provided by a Consultant Expert. Pagamento: 24 500,00€ \- 42 500,00€ por ano Localização do trabalho: Trabalho remoto híbrido em Porto