Folder Stage seems to read file content, only need filenames

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
mario_j
Participant
Posts: 6
Joined: Thu Jun 26, 2014 1:12 pm
Location: Hamburg

Folder Stage seems to read file content, only need filenames

Post by mario_j »

Hi,
we have a server job that shall read all filenames from a directory and put it into a database table.
Within the folder stage only one column for the filename ist defined.
This job aborts, when in the directory is a file larger then 500 MB or when all files together are bigger then 500 MB.
It does not matter, what I am doing within the job, without sorting, transform and only writing to an hashed file the folder can be a little bit bigger than when writing to db stage, but it aborts with uvmalloc() memory exceeded error.
If there the files within the directory are smaller than 500 MB jobs are working fine.
If I delete some entries in the 500 MB file, so that it is smaller, the job works.

I can rebiuld it in parallel with external source using ls command, then it works. But using parallel job is not likely, because of sevrer routines that are used.

First question: does the folder stage really read all file content first, even I only use one column for filename?

Second: is there a parameter that limits memory for reading files or for the folder stage?

Thanks,
Mario
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Re: Folder Stage seems to read file content while only want

Post by chulett »

mario_j wrote:does the folder stage really read all file content first, even I only use one column for filename?
From what I recall and what you've documented - yes. If literally all you need are the filenames, I'd suggest an ExecSH command Before Job to list the filenames to a flat file and then use that file as your source in the job.
-craig

"You can never have too many knives" -- Logan Nine Fingers
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Did you ever get this resolved? Stumbled back on this today and just wanted to add that you could also use the Filter option of the Sequential File stage to feed in just the filenames using something like an "ls -1 <pattern>" O/S command.
-craig

"You can never have too many knives" -- Logan Nine Fingers
mario_j
Participant
Posts: 6
Joined: Thu Jun 26, 2014 1:12 pm
Location: Hamburg

Post by mario_j »

we did a workaround and use command stage instead of folder.
We also made a pmr, but they only stated out, what we already know and could not help.
Maybe problems with blocksize on our windows server. The process cannot allocate a big enough block.
Post Reply