We use the osh scripts to run the DS jobs, and we have configure the config.apt file to run it in 3 servers 6 nodes.
But we found that the import progress is only run in 1 server, the other operaters is running in 3 servers.
We have huge data files, and the import progress take too much time, i think it is the bottleneck in performance.
How to import data files in parallel?
Thanks!
How to import data files in parallel
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 5
- Joined: Tue May 26, 2015 8:27 am
Unless your input sequential file has a fixed line length, your reader process must run sequentially and can only run on one server and on one node.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Have you thought of taking a sample file for the import table definition?
something like:
IHTH (I Hope This Helps),
something like:
Code: Select all
head -1000 source_file.txt > source_sample.txt
Roy R.
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Re: How to import data files in parallel
Specify the multiple readers per node property.Johnny0638 wrote:How to import data files in parallel?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
This may be of interest: 11.3 Sequential File stage file pattern parallel processing.
Choose a job you love, and you will never have to work a day in your life. - Confucius