How to import data files in parallel

Johnny0638 · Post by **Johnny0638** » Mon Dec 28, 2015 8:09 pm

We use the osh scripts to run the DS jobs, and we have configure the config.apt file to run it in 3 servers 6 nodes.
But we found that the import progress is only run in 1 server, the other operaters is running in 3 servers.
We have huge data files, and the import progress take too much time, i think it is the bottleneck in performance.
How to import data files in parallel?
Thanks!

ArndW · Post by **ArndW** » Tue Dec 29, 2015 7:26 am

Unless your input sequential file has a fixed line length, your reader process must run sequentially and can only run on one server and on one node.

roy · Post by **roy** » Tue Dec 29, 2015 11:49 am

Have you thought of taking a sample file for the import table definition?

something like:

Code: Select all

head -1000 source_file.txt > source_sample.txt

IHTH (I Hope This Helps),

chulett · Post by **chulett** » Tue Dec 29, 2015 5:16 pm

Import as in the operator, Roy... i.e. read the file.

ray.wurlod · Post by **ray.wurlod** » Fri Jan 01, 2016 1:23 am

Johnny0638 wrote:How to import data files in parallel?

Specify the multiple readers per node property.

qt_ky · Post by **qt_ky** » Fri Jan 01, 2016 10:25 am

This may be of interest: 11.3 Sequential File stage file pattern parallel processing.

DSXchange

How to import data files in parallel

How to import data files in parallel

Re: How to import data files in parallel