Ming - Développeur APACHE HADOOP

février 2019 - Present
Paris Area, France - Société Générale
Big Data Engineer/Optimization Enthuiast (Spark SQL, Hadoop, Scala, Apache Oozie)

- using Apache oozie, optimize data process procedure and reduce waiting time of Tableau from 3 hours to 25 minutes
- using Spark DAG, analyse bottleneck of spark execution and reduce the time from more than 90 minutes to 25 minutes
- using Spark SQL 2.2, work on construction of data pipeline to feed transformed data to Tableau
- using Ansible, Jenkins and Git, work on process of CI/CD cooperate with BA to understand business logic and propose technical solution cooperate with Tableau to analyse calculate needs and "take the heavy lift" on spark side optimize application.properties to group parameters by usage(oozie, spark, application) and delete useless and duplicate parameters optimize Jenkins job so less humain error will be made which takes time to be seen communicate with cluster management team to ensure the execution of spark on different environment(disk space insufficient, queue saturated for submitting spark job)
- share knowledge with newly arrived developer, sometimes do pair programming to simplify the integration

janvier 2018 - janvier 2019 (1 an 1 mois)
Paris Area, France - Renault Digital
Big Data Engineer(Spark SQL, Hadoop, Scala, Apache Oozie)

- Based on Apache Kafka and Apache Spark Streaming, I build up a streaming system which responses to user demands. With the support of RESTful service, user without SQL knowledge could explore the datalake content easily and finally get the result saved to HDFS for later use.
- Based on Apache Oozie, I configure a DataLake Loader to fetch data frequently and feed to datalake.

mai 2017 - novembre 2017 (7 mois)
Paris Area, France - Le Groupe La Poste
Big Data Engineer(Apache Kafka, Apache Avro, Cassandra)

- At a big data project, I shared my knowledge about web, participated to the development of message treatment platform.
I also practiced some big data technologies and utilities.

janvier 2017 - avril 2017 (4 mois)
Big Data Certification
Big Data Engineer(Spark, Scala)
By taking MOOC on ********, I obtain my first big data experience about
scala, spark, k-means, mongodb and find my first job as a Big Data Engineer

avril 2014 - janvier 2017 (2 ans 10 mois)
Paris - Akka Technologies
Java EE Developer

- Within almost three years, I participated in multiple projects, from Java application for embedded system to android app for lighting system and intelligent transport vehicle and then web application.
- By cooperating with experienced and smart engineers, I gained a good understand about Java programming, the structure and component of web
application, some popular web technologies.

mars 2013 - août 2013 (6 mois)
Région de Toulouse, France -Pad-Ways
Embedded Software Engineer - Intern
- Develop/Test/Integrate Linux driver for Android tablet
- Configure source code for new image file of Android
- Configure input device of capacitive button in Android
- Analyze client's needs, optimize system function and realize it
- Cooperate with the project manager and manage project progress
- Write technique blogs to summarize key points and structure of project
Environnement: GNU/Linux(Ubuntu), source code Android, Android HAL/JNI/
NDK, Gcc, VIM, GIT, ARM Cortex-A8, MPR121, protocol SPI/I2C