Size & Effort Estimation Model for ETL

This model will simplify and accelerate the process of size & effort estimation for ETL job development using a ETL tool. The purpose of this paper is to define a method for estimating ETL job development for DataStage / Informatica jobs by calculating job size and complexity based on the job pattern and specific challenges within the job. This Size & Effort Estimation model will help to predict how many Horizontal and vertical design blocks should be produced and how many developers are required and how long it will take. The approach is based on data typically produced during early stages of software development from various successfully executed projects.

Full paper can be viewed in this link: ETL jobs Effort Estimation

Instead of a Function Point estimation which is difficult and doesn’t work very well for tool based ETL development, a Horizontal and vertical design blocks or Level of detail (granularity) based Estimation approach is adapted in this model. An effective limit of granularity is defined as individual jobs which are no longer possible or require modularising

Advantages

This is an estimating process based upon historical results.

–> Estimates can be explained easily from the ETL perspective.
–> Can generate estimates which can be revised using the same process (same inputs).

The proposed model is confirmed by empirical evidence based on previous projects. Size & Effort estimation is based on vertical and horizontal design blocks and then final effort is calculated by multiplying with skill weight and data volume weight. Estimation also includes integration testing of all the horizontal and vertical design blocks with performance testing. These guidelines could then go into a modeling spreadsheet that estimates all ETL jobs in a project. However this does not take into account of the complexity of business rules: having to write a lot of transformation custom code or custom stages. Those should be treated as special cases and added to the calculation accordingly.

This method requires less effort and can be applied in the project sizing or proposals or earlier development cycle or later enhancements.

Comments

• Archives

  • June 2008 (2)
  • May 2008 (5)
  • April 2008 (10)

• Recent Posts

  • The Utopian “E” Convergence - EAI, EII, ETL, MOM, B2Bi, BPM, B2B Gateway, BPI, IDM, MDM,…
  • Extraction from HiPortfolio using DRO
  • Performance Improvement Techniques in DataStage
  • Information Server Overview
  • Protected: DataStage – Process Management

• Recent Comments

  • Mohan on Performance Improvement Techniques in DataStage
  • Shaun on Size & Effort Estimation Model for ETL
  • friend.kak on DataStage Overview | Tutorial | Essentials
  • Wah on Performance Improvement Techniques in DataStage
  • sanath on DataStage Overview | Tutorial | Essentials

• Pages

  • About
  • Contact
  • Main
  • Portfolio

• Categories

  Select Category DataStage Articles  (9) Hiportfolio  (1) So you want series  (4) SOA  (4) Tech Articles  (6)

• DataStage Articles

  • Datastage 7x / 8x Enterprise Edition with Teradata
  • DataStage Overview | Tutorial | Essentials
  • Datastage Parallel (C++) routine to create files dynamically
  • Information Server Overview
  • Performance Improvement Techniques in DataStage
  • Process Management in DataStage
  • Real Time Integration using Datastage
  • Surrogate Key Generation in DataStage

• ETL

  • Size & Effort Estimation Model for ETL

• Hiportfolio

  • Extraction from HiPortfolio using DRO

• Main Links

  • Home(Ubliss)
  • Office Photos

• So you want series

  • So you want to do IT in Australia !
  • So you want to do IT in Europe !
  • So you want to do IT in India !
  • So you want to do IT in New Zealand !
  • So you want to do IT in UK !

• SOA

  • Rock your Data Warehousing with SOA
  • The Utopian “E” Convergence - EAI, EII, ETL, MOM, B2Bi, BPM,…