Data Engineer

Extracting data. Information is all around us, but to make use of it, we have to extract it from its sources. In terms of corporate data, the source can be some database, an internal ERP/CRM system, etc. Data can come from IoT sensors scattered, say, across an aircraft or a manufacturing facility. Information may also be obtained from public sources available online.
Transformation. Raw data will not make much sense to end users, because it’s hard to analyze it in its initial form. The transformation stage is about cleaning, structuring, and formatting data sets to make them consumable for analysis and reporting.
Loading and storage. We need to keep extracted data somewhere. In data engineering, the concept of a data warehouse relates to the most common type of repository for data gathered for analytical purposes. But sometimes more complex data pipelines are developed that include such elements as data lakes, data lakehouses, data marts, and so on.

Skills for any specialist correlate with the responsibilities they’re in charge of. The required skill set would vary for every project or organization, as there is a wide range of things data engineers could do. But generally, their activities can be sorted into three main areas: engineering, database/warehouse management, and data science.

Typical technology stack includes:

Elastic Stack, MongoDB, Cassandra, Redis, MySQL, GraphDB, Java, Apache Karaf, Hibernate, Apache Camel, Spring Boot, Drupal

Storage Engines: Elastic Stack, Memcached, MySQL, GraphDB, Virtuoso
PLs/Frameworks: Java, PHP, Spring {Boot, Data}
CMS: Drupal