loading list csv files with Stage Folder

Brunodatastage · Post by **Brunodatastage** » Fri Dec 23, 2005 11:19 am

Hi everyone,
I have a job that reads the contents of all the csv files (all having the same format)
in a given directory and post in a Sequential File.

Stage Folder ----> TRasformer ---> Sequential file

When I run the Job, I get only the Column Names and field in the Resulting File is nullable=no.
Why in the sequential file for 2 rows I've blank rows, without errors/warnings in the log ?

The list files in the directory (names,dimensions):
------------------------
file_1.csv (10 MB)
file_2.csv (19 MB)
file_3.csv (18 MB)
file_4.csv (12 MB)
file_5.csv (9 MB)
file_6.csv (12 MB)
---------------------------

The result in the Sequential file
-----------------------------------------
file_1.csv

file_4.csv
file_5.csv
file_6.csv
------------------------------------------

How can I resolved it?
Thanks and regards
Ciao
B.

chulett · Post by **chulett** » Fri Dec 23, 2005 11:39 am

What is the end result you are shooting for? The Folder stage may not be the most... reliable... way to get a list of filenames, if that is all you are after. I would *guess* those two files may be 'too big' to process correctly through that stage.

Have you considered using ExecDOS to list them using an operating system command? Depending on exactly how you intend to use this list, if may be simpler / faster / more reliable that way.

ray.wurlod · Post by **ray.wurlod** » Fri Dec 23, 2005 2:08 pm

You don't seem to understand what the Folder stage reads. It has two "columns" - the first gets the file name (which must be non null, for obvious reasons), the second gets the entire file contents as a string.

This is not the correct stage type for what you want to do.

As Craig suggests, an operating system command executed via a before-stage subroutine (ExecDOS) in a job, or an Execute Command activity in a job sequence, is probably the most efficient solution. Something like:

Code: Select all

type origdir\*.csv > otherdir\new.csv

Brunodatastage · Post by **Brunodatastage** » Tue Dec 27, 2005 9:18 am

Thank's a lot for your suggestions but I Know that.
Unfortunately my job is imposed by the society's customer where i'm developing.
The really problem is that this problem compares ONLY on the test server, not in the development or production server.
For this reason I can't understand what happens.
The job's end results are the filenames in a particular folder.
Excuse me for my bad english but I don't use it frequently...

Thanks in Advance.
Bruno