Pyspark with ADB
Clicking "Apply for this Position" redirects you to Capgemini's official careers portal
We are looking for a skilled PySpark Developer with experience in Azure Databricks (ADB) and Azure Data Factory (ADF) to design, develop, and deploy data solutions for large-scale data processing and analytics. You will implement ETL/ELT pipelines using Azure Data Factory and collaborate with data teams to transform structured and unstructured datasets into meaningful insights.
You will be responsible for optimizing PySpark jobs and data pipelines for performance and reliability, ensuring data quality, integrity, and adherence to regulatory and industry standards throughout all stages of data processing. The role also involves conducting financial risk assessments, identifying vulnerabilities in data workflows, and implementing strategies to mitigate financial risks associated with data transformation and aggregation.
Troubleshooting and debugging pipeline issues, applying best practices for data security, compliance, and privacy within the Azure environment, and documenting technical specifications, data flows, and solution architecture are key responsibilities. Experience in Financial, Risk, Compliance, or Banking domains is a plus.