Reading Partitioned data from File Connector stage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
samyamkrishna
Premium Member
Premium Member
Posts: 258
Joined: Tue Jul 04, 2006 10:35 pm
Location: Toronto

Reading Partitioned data from File Connector stage

Post by samyamkrishna »

Hi,

I am able to read the data from HDFS files.
the folder structure as below.

/data/projectname/zonename/dbname/tablename/partfilexxxxx*

Question:

How do I read the read if the data is stored in partition on business_effective_date like below

/data/projectname/zonename/dbname/tablename/effective_date=20170915/partfilexxxxx*
/data/projectname/zonename/dbname/tablename/effective_date=20170916/partfilexxxxx*

Should I read them separately or is there a way to read from all the effective_date sub folders at once using file connector stage?
Cheers,
Samyam
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

That should all be handled automatically for you. You (the user) should remain unaware of how Hadoop partitions its data.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply