Freelance DATA : Recevez facilement les CV que vous avez choisi

Résumé d'expériences de Mehdi,
freelance DATA habitant Paris (75)

Experience

❖ Société générale – DSI CFT - March 2021 - Today - Paris

Project « Finance District » :

Mission: Finance District supports the business areas of the CFT IT department in the design and
implementation of solutions related to data
Position: Technical Lead Big Data Engineer.

Lead tasks:
❖ Daily technical support for data engineers.
❖ Implementation of a monitoring module for the execution plan of Quartier Finance.
❖ Design of a data quality management module.
❖ Design of a production job monitoring module.
❖ Supervision of the technical and functional migration from Talend Big data to Scala/Spark.
❖ Ensuring the migration of Spark 2.1 projects to 2.4.
❖ Implementation of a data anonymization solution.
❖ Implementation of the Scala/Spark CI/CD pipeline.
❖ Ensuring the migration from Hortonworks to CDP.
Technical tasks:
• Analyzing and implementing "hot fixes" in production.
• Implementation of a Scala/Spark framework to facilitate and standardize Scala/Spark
developments.
• Developing a solution for sending files via WebHdfs.
• Tuning the performance of Scala/Spark applications for batch interval, parallelism, and memory.
• Optimization of existing algorithms in Hadoop using SparkSession, Spark-SQL, Data Frames,
and Pair RDDs.
• Manipulating large datasets using Partitions, Spark memory capabilities, Spark Broadcasts, and
efficient Joins.
• Developing audit logic to optimize the load in append mode.
• Developing solutions for pre-processing large sets of structured and semi-structured data with
different formats (text files, Avro, sequence files, JSON records).
• Use of Parquet, ORC, Avro files according to the technical need.
• Study of the choice of Spark partitions for "write HDFS" and the calculation of the coalesce to
avoid the shuffle
• Implementation of performance improvements based on job monitoring via SparkUI
• Adding "persist" via Spark and "serialization" of data as needed
• Use of RDD, DataFrame, DataSet according to the technical need.
• Development of some UDF on Spark 2.4
• Deployment and Orchestration of the project via ControlM
• Use of scalaTest to ensure unit tests and code coverage at the SonarQube level
• Development of test cases, test scenarios with TDD logic using Cucumber.
• Log recovery via Scala/Kafka & Spark Streaming from YARN for analysis and monitoring of
the application.

Project « PRORATA VAT »:
Mission: Equip accounting and fiscal business with a digital solution for calculating the "PRORATA
TVA" tax base
Position: Technical Lead Big Data Engineer.
Lead tasks:
❖ Facilitate or lead agility ceremonies.
❖ Implementation of the Data solution architecture using "Scala/Spark" and establishing a link
with visualization tools.
❖ Optimization of the performance and scalability of "Scala/Spark" data systems for jobs.
❖ Ensuring the quality and reliability of data.
❖ Synchronize with web restitution and PowerBI teams.
❖ Collaborate with security teams to ensure the security of data.
❖ Definition of standards and best practices for data projects.
Technical tasks:
• Writing technical specifications.
• Definition of big data architecture.
• Scala/Spark development of calculations for the "prorata tva" bases.
• Implementation of Scala/Spark jobs.
• Development of data extraction jobs from APIs.
• Analyze, design and build Modern data solutions using Azure PaaS service to support visualization
of data. Understand current Production state of application and determine the impact of new
implementation on existing business processes.
• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a
combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake
Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage,
Azure SQL, Azure DW) and processing the data in In Azure Databricks.
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and
load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, writeback tool and backwards.
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation
and aggregation from multiple file formats for analyzing & transforming the data to uncover
insights into the customer usage patterns.
• Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark
databricks cluster.
• Experienced in performance tuning of Spark Applications for setting right Batch Interval time,
correct level of Parallelism and memory tuning

Project « C3S TAX »:
Mission: Proposal for a technical data/big data and Azure Cloud architecture to meet the need for
industrialization of the C3S tax calculation.
Position: Data Architect
Architect tasks:
❖ Define an architectur...