Project: DevOps Big Data
Mission: Set up a DevOps tool for Big Data applications
The goal of the mission is to set up a Devops deployment system for Big Data treatments.
Deploy the solution within all EDF units with support and project training
The mission within the project consists of:
ï¼ Create, industrialize and automate bigdata infrastructure projects in Infra As code
ï¼ Set up the CI/CD pipeline on projects with tests, validations, securityï¼
ï¼ Propose and implement architectural developments in conjunction with the architectsï¼
ï¼ Guarantee the security of project data by implementing the right protection mechanismsï¼
ï¼ Participate in the study, costing, framing in connection with architects and project managersï¼
ï¼ Take charge of all BUILD and RUN aspects (You build it, you run it)ï¼
Project: Set up a Data-Lake
Mission: Architect, Hadoop Administrator, Data-Lake, DevOps
The objective of the mission is the definition of the architecture of the new datalake, then the implementation and
management of the Big Data HortonWorks infrastructure and also the implementation of a security policy and
data governance.
The mission within the project consists of:
ï¼ Definition of the technical architecture
ï¼ Setting up the technical infrastructureï¼
ï¼ Installation, configuration of software bricks: Hortonworks, Talend BigData, Vertica, Attunityï¼
ï¼ Connect all components and make them work togetherï¼
ï¼ Hortonworks Hadoop Administrationï¼
ï¼ Configuration and customization of Big Data securityï¼
ï¼ Set up the CI / CD installation pipeline with tests, validations, securityï¼
ï¼ Automation of the installation of infrastructure components with Ansibleï¼
Project: Compliance
Function: Big Data Engineer
Mission: Data Analyzer / Big Data Developer
The client has a global responsibility for all IT tools for market abuse control, compliance, and operational risk. In
addition to this direct responsibility, it is also possible that the steering projects executed in other related entities.
The Project Objective is the implementation of solutions based on Hadoop stack: Hortonworks distribution,
parquet storage, un-normalization in HIVE, SOLR indexing, the use of SPARK / Java for distributed process, the
use of oozie and spring also, data-visualization with Tableau Software.
The mission within the project consists of:
Analysis and design of the big data solution to put place
Loading data in the DataLake
SPARK / JAVA development of new detection patterns
Restitution via a WEB interface or the TABLE software tool
Development and calculation of KPI / KRI
Indexing data with SOLR
Project: Amazon open data project
Mission: Exploration of the data and analyzes feelings (Opinion of the Amazon customers)
ï¼ Development of several algorithms on Apache Spark by using the functional language Scala
ï¼ and Python
ï¼ Implémention of the algorithms by using the language R with RStudio
ï¼ Loading of the data by using Pig/Hive /Impala on a distribution Hadoop Cloudera
ï¼ Development of a MapReduce program in Java to make statistics on the basis of
data
ï¼ Construction of the whole of the characteristics to lead classification, by using them
methods of text mining
ï¼ Proposal and use for a set of learning techniques supervised (trees of
decision, random forests, svm, bayesien naive) to build a model of analysis of feeling.
ï¼ Proposal for a set of measurements to compare the performances of these methods (accuracy, precision,
recall, F-measurement, ROC, AUC).
ï¼ Calculations were carried out on a cluster composed of 10 waiters, of which storage capacity
live reached 40 GO of RAM by waiter and the total secondary storage capacity of the cluster reaches 10 TO.
Project: Projects under HADOOP
Mission: Architecture and development Big data
For projects of the department &ldquoEDF data warehouses&rdquo, that is brought to conduct many studies of architecture
then to implement these studies of architecture within the framework of operational projects.
For example: Installation of Kafka/Spark within the framework of a brick of acquisition at the current, put in work
of ELK for the acquisition of log and bricks of reporting.
The mission within the project consists with:
ï¼ To carry out the studies of architectures big dated in support with the architect referent from the department.
ï¼ To carry out in a specific way of the developments to implement these architectures.
Projects carried out:
ï¼ Accosting of new a CRM (MY) to the Big platform Dated, Vue customer 360°: real time with Kafka and
Spark Streaming, storage and indexing in HBase and ElasticSearch, alignment customer with Comscore and
WebAnalytics
ï¼ Creation of a reference frame for the eligible customers with the electronic invoice (AIFE Chorus): Spark
batch, Apache Nifi, HBase, Java, XML
ï¼ Development of several batchs, Hive, Spark, for the integration of data of several applications in the platform
Hadoop (Linky, GazPar, Comscore&hellip)
Project: SOA portal
Mission: Technical Lead and SOA Architect
ï¼ Participation in the installation of the technical architecture of the project in collaboration with
Basis teams, Architecture and Interfaces
ï¼ Participation in the drafting and the update of the document of technical architecture (DAT)
ï¼ Configuration of WebDispatcher (Load-balancing), to improve the performances
ï¼ The installation of the environments (Dev., test, preprod, prod), Unix, Oracle, RAC, SAP&hellip
ï¼ Administration of the waiters of application WAS
ï¼ Administration of the environment of development
ï¼ Drafting of the SFD and Standard and TUFF, YOU
ï¼ Formative expert/(2 developers) on technologies /J2EE Java
ï¼ Piloting of several developers onshore and offshore oil rig in a SOA context 4 layers
ï¼ Definition of the best practices project
ï¼ Design object and development of the microphone-services in Java
Context:
The infrastructure of the project is based on the structuring principles of the High Availability and the replication,
based on the fault tolerance of a principal site and a recovery plan of activity on a site of help. With this intention,
the infrastructure put in work rests on two rooms for the principal site and of a room in a site of distant help.
Project: Processing data of counting and publication flows (Data-processing Direction and telecommunications)
Mission: Technical Lead
ï¼ Framing of the development team
ï¼ Participation in the optimization of the technical architecture of the application
ï¼ Analyzes, design object, and definition of the technical specifications in UML
ï¼ Development java
Context:
The project rests on an architecture on 3 levels, of standard application Web/light customer based on the use
of:
ï¼ Apache (Web server)
ï¼ Weblogic Server (waiter of application/middleware)
ï¼ DBMS Oracle (persistence of the data)
ï¼ CFT, ftp and MQSeries-JMS for the exchanges with partners
The Apache waiter and the Weblogic field are lodged on the same physical machine, the Weblogic field
being composed of two authorities:
ï¼ WLS-01 for the presentation layer
ï¼ A cluster WLS-02 and WLS-03 for the layer services trades
The Oracle database is lodged on a separate machine. Base files them being stored on an external bay of discs.
Functional axes of:
ï¼ The management of the producing contract
ï¼ Data processing of counting (control, correction, validation and calculation of energies)
ï¼ Publication of flows
ï¼ Invoicing of the components of the access to the public network of distribution and covering
Project:Data-processing direction - development of a gate Web of monitoring quasi real time of a network IP VPN
(IP VPN Dashboard Monitoring) in Nimble mode
Mission: Refer technical
ï¼ Analyzes, design in UML
ï¼ The selection of spots/modules has to deliver, at
the beginning of each sprint
ï¼ Delivery of the selected modules has the end of
each Sprint
ï¼ Immediate notification of problems, cut-off
points to the scum-master degree
ï¼ Drafting of documentation
Context: The purpose of this project of VPN monitoring, definite jointly with Siemens is to provide an instrument
panel simplified to follow the good performance of the network. That would make it possible to go down in the
entrails from the infrastructure network, and to collect information concerning the utilization ratio, and to analyze
the elements of the topology of the network like following the gravity of the incidents
Project: Data-processing direction - European Approval of cars
Mission: Technical manager
ï¼ Participation in the installation of the technical architecture of the project in collaboration with
teams Structures of the customer
ï¼ Participation in the installation of the data model and in its optimization (Oracle 10)
ï¼ Formation and framing of the development team and assignment of the spots
ï¼ Analyzes, design object, and definition of the technical specifications in UML
ï¼ Development of the batchs in Java (Spring Batch)
ï¼ The support of production of the applications, follow-ups of the deliveries
Context:
The project aims at the application and the generation of the CoC document.
CoC is in application in Europe since 1996 for the particular vehicles.
It is about a document referring certain features of the vehicle (dimensions, mass, CO2, etc.) + clean information
(color, chassis number).
This document is obligatory for the registration in certain countries (Germany, Italy, etc&hellip). Optional in the other
countries (the manufacturer must be able to produce it with the request when a vehicle registered in a country A
must be transferred in a country B, in order to facilitate the administrative formalities in the country of destination).
Project: Direction of the information systems - the management of the industrial accidents
Mission: Responsible for the design and development
ï¼ The definition of the technical specifications in UML
ï¼ Design object and development of the IHM, unit tests
ï¼ Technical receipt and support functional receipt
ï¼ Provision of the packages of deliveries
ï¼ The support of production of the applications, follow-ups of the deliveries