ETL: More than Middleware?

[Ultramundane]

ETL: More than Middleware?

[Ultramundane]

http://www.middleware.org/whatis.html

[Ultramundane]

How important is a core competency?

I think you have answered by question below.

Thanks and you folks are awesome.

[chulett]

You seem to be having this conversation with yourself...

[Ultramundane]

You seem to be having this conversation with yourself...

Yeah, I do like to argue with myself sometimes. But seriously, I am looking for opinions other than my own. Sorry about that.

[kduke]

I agree. You do like to argue with yourself. Seriously, this never works. When admins monitor DataStage without knowledge of the product then it never runs smooth. They call about things that are not a problem and ignore serious problems.

[eostic]

Ok. I'll bite.

It is vastly more than just "another piece of middleware." It's often the core of the entire data integration infrastructure for thousands of companies. And as that "core," represents the lion's share of intellectual capital about "the" data and metrics that drive the enterprise. Often the folks who build decision support systems (still a majority of the apps that use ETL) know more about the data, it's meaning, its relationships, and its ideosyncracies than anyone else in the organization. Their understanding is reflected in ETL (whatever tool, or 3gl code they might be using) and in every other part of the lifecycle that surrounds it......profiling, data quality, modeling, cubing services, reporting, portals, legacy data stores, and more..........lots more.....

It can't be ignored by anyone, let alone be a delivery vehicle who results are the responsibility of end users.

Sure there are configurations where ETL is just a "movement process," but even there, treating it as just "plumbing" is a continuation of the habits that got us here in the first place (with hundreds of definitions of customer, no awareness of metadata, yada, yada.).

Ernie