ETL term stands for the Extract, Transform and Load (ETL) and which is explained in different levels. ETL is a process of data warehousing and which is responsible for extract data out of the (RAW) source systems and placing it into a data warehouse QVD. ETL involves the following tasks:
Extracting the data:-
Extracting the data from raw source systems (ERP, SAP, FLAT FILES(TXT, EXCEL, CSV), other oprational systems), data from different raw source systems is converted into one Consolidated Data Model in warehouse format which is ready for transformation processing.
Transforming the data:-
Transforming the data may involve the following tasks:
- Applying business rules (so-called derivations, e.g., calculating new measures and dimensions),
- Cleaning (e.g., Date fields, Currency Conversion, finding Sum and Count metrics, Mapping NULL to 0 or Date Calculations etc.),
- Cleaning or Filtering (e.g., selecting only certain columns to load),
- Splitting a column into multiple columns and vice versa,
- Joining together data from multiple sources (e.g., Joins, Concatenation, Lookup, Merge),
- Transposing rows and columns,
- Applying any kind of simple or complex data validation (e.g., Loading manipulated or calculated row are empty then reject the row from processing.
Loading the data:-
Loading the data into a QVW application is same as loading data warehouse or data repository to other applications.