people,
When using a sequential stage with the read method as 'File Pattern', i cannot use "Number of Readers Per Node" option as they are mutually exclusive..
can someone explain why they are mutually exculusive ??
thanks
Sequential File Stage in PX
Moderators: chulett, rschirm, roy
-
- Charter Member
- Posts: 38
- Joined: Wed Mar 17, 2004 1:16 am
- Location: USA
-
- Participant
- Posts: 75
- Joined: Tue May 13, 2003 4:14 am
- Location: California
- Contact:
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
The number of Readers Per Node has a complex start up involving counting the size of the files, dividing the file across the specified number of readers to partition your data. I'm guessing that they don't allow this option for File Patterns due to complexity it adds to the stage code. It may be an option that is much easier to apply programmatically when you have a fully defined set of files rather then a fuzzy file pattern.
You can get around the problem by retrieving your file list at the sequence job level and running the job multiple times with different file names.
You can get around the problem by retrieving your file list at the sequence job level and running the job multiple times with different file names.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
As well as everything Vincent said, note that Readers Per Node is intended only for fixed-width format files. It's fully described in the Parallel Job Developer's Guide
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
One final note - the goal of multiple readers is to take advantage of multiple nodes - instead of one single stream of data, you have as many as there are readers. On the other hand, if you are using either filesets or the file pattern, you will already be taking advantage of multiple nodes - one per file
Now, if you could specify multiple readers to be use for each file (fixed length, of course), that would be great - but I believe that would be a discussion for another forum ( PX enhancements?)
Now, if you could specify multiple readers to be use for each file (fixed length, of course), that would be great - but I believe that would be a discussion for another forum ( PX enhancements?)