File Pattern

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
kogads
Premium Member
Premium Member
Posts: 74
Joined: Fri Jun 05, 2009 5:36 pm

File Pattern

Post by kogads »

Hi All,
We have a requirement to capture all the files in a folder which have same metadata. I am using the file pattern to read all the files using sequential file stage. How can we track the file that is causing problem or the file that cant be extracted from the set of files.

Thanks in advanse.
keshav0307
Premium Member
Premium Member
Posts: 783
Joined: Mon Jan 16, 2006 10:17 pm
Location: Sydney, Australia

Post by keshav0307 »

check this post and try setting those parameters,.
viewtopic.php?t=121263
kogads
Premium Member
Premium Member
Posts: 74
Joined: Fri Jun 05, 2009 5:36 pm

Post by kogads »

I still cant understand how we can track the error file if we have the environment variable $APT_IMPORT_PATTERN_USES_FILESET set to true and File Name column is provides in sequential file?
anbu
Premium Member
Premium Member
Posts: 596
Joined: Sat Feb 18, 2006 2:25 am
Location: india

Post by anbu »

You can run an awk command to find list of files which dont match metadata

Code: Select all

for file in *
do
  awk -F"," -v no_of_fields=2 ' NF != no_of_fields { print FILENAME;  exit } ' $file
done
Replace comma by field delimiter and assign no_of_fields as per your input
You are the creator of your destiny - Swami Vivekananda
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

kogads wrote:I still cant understand how we can track the error file if we have the environment variable $APT_IMPORT_PATTERN_USES_FILESET set to true and File Name column is provides in sequential file?
It lets you know which one of the (possibly many) files you are reading that any particular record is from.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply