Hello ,
We are implementing Schema file approach for processing nearly 50 CSV Files with using 50 different schema files.
If data volume in a CSV file is 1 million records, but schema file is propogating only 100 Records. we tried with multiple scenarions where If:
1. More than 100 records in CSV file : Processes only first 100 records.
2. CSV has got exact 100 records: Processes succesfully
3. Less than 100 records : Processes succesfully
What could be the issue around it? Any patches available for this?
Working with 8.1.x has become horrible now days, reggularly we are surprised with Issues. We installed FixPack-1 & Fix Pack-2.
Is Fixpack-3 also in market?
Can you post the actual schema file? Can you please also post the score from one run of the job with more than 100 records? There may be a limiter there. What stage type are you using to read the CSV files?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
CSV files are Tab delimited. we are using Sequential file stage to read the records which is reading all the Records but column Import stage where schema file is assinged is propogating only 100 records out of it.
Occurred: 11:50:04 On date: 10/02/2011 Type: Info
Event: ReadFile,0: Import complete; 105366 records imported successfully, 0 rejected.
Occurred: 11:50:06 On date: 10/02/2011 Type: Info
Event: Column_Import,0: Field import complete; 100 records converted successfully, 0 rejected.
Occurred: 11:50:06 On date: 10/02/2011 Type: Info
Event: Load_Table,0: Records inserted: 100 (...)