Data Engineer
About the Role
We're looking for a highly skilled Data Engineer with extensive experience in the Microsoft Azure ecosystem and deep expertise in Databricks. You'll be responsible for building scalable and reliable data solutions that power our business intelligence, reporting, and advanced analytics initiatives. You will work on a modern data platform using a lakehouse architecture.
Key Responsibilities
• Design & Development: Architect and implement robust and scalable ETL/ELT pipelines using Azure Data Factory and Databricks.
• Data Processing: Develop and optimize data transformation workflows using PySpark and SQL within the Databricks environment.
• Data Lake Management: Manage and maintain data in Azure Data Lake Storage (ADLS), applying a medallion architecture (Bronze, Silver, Gold layers) for data quality and access control.
• Performance Optimization: Tune and optimize Databricks clusters and Spark jobs to improve pipeline performance and reduce costs.
• Data Governance & Security: Implement data governance frameworks and security best practices using tools like Databricks Unity Catalog and Microsoft Purview.
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions that meet business needs.
• Automation & DevOps: Build and maintain CI/CD pipelines for data solutions using Azure DevOps.
Qualifications
• Education: Degree in Computer Science, Engineering, or a related field.
• Experience: At least 3 years of professional experience in data engineering.
• Databricks, Microsoft Certifications is a plus.
• Technical Skills:
o Extensive hands-on experience with Microsoft Azure data services, including Azure Data Factory, Azure Data Lake Storage Gen2, and Azure Synapse Analytics.
o Proven expertise in Databricks, including Spark, PySpark, and Databricks Workflows.
o Proficiency in Python and SQL.
o Familiarity with data warehousing, data modeling (e.g., star schema), and lakehouse architecture patterns.
o Experience with real-time data streaming technologies (e.g., Azure Event Hubs) is a plus.
- Departamento
- IT - Group
- Tipo de emprego
- Tempo inteiro
CHEGOU A TUA HORA
Lisboa
MAIS QUE UM EMPREGO
Na DECO PROteste, cada dia é uma oportunidade de fazer a diferença na vida dos consumidores. Saltamos da cama com este sentido de compromisso e quando chegamos ao trabalho é como se nunca tivessemos saído de casa. Em cada colega está um sorriso, em cada equipa um suporte e em cada porta um futuro por descobrir.
Aqui, valorizamos o talento, a proximidade e o trabalho em equipa.
Junta-te a nós! Na DECO PROteste, damos confiança às escolhas dos consumidores.
Sobre a DECO PROteste
A DECO PROteste resulta da parceria entre a Euroconsumers e a DECO.
A DECO – Associação Portuguesa para a Defesa do Consumidor é uma associação privada, sem fins lucrativos e de utilidade pública que detém 15% da DECO PROteste.
Os restantes 85% são detidos pela Euroconsumers "Uma organização que nasce da colaboração de diferentes organizações de Consumidores presente em – Portugal, Espanha, Bélgica, Itália."
Já trabalha na DECO PROteste?
Vamos recrutar juntos o(a) seu(sua) próximo(a) colega.