DSXchange

CLEAR.FILE is absolute. ALL records are removed from the file.

Every log event has a TIMESTAMP field so you could use a DELETE statement constraining on that.

To add to what Craig said, if you need the intermediate value, park it in a stage variable.

Can you tell us why you have to avoid stages in the job? Not just because "they" told you to, but their actual technical reason behind that directive.

Never be afraid to ask "why?"

I don't believe so.

I think the easiest solution is to use a loop in a Transformer stage to extract each of the delimited components from Business_Rule. This output can then stream into a Lookup stage to effect the conversions.

rkashyap wrote:What is the source of the ComponentValue and BusinessRule file?
How big is the ComponentValue file?
Which flavor of Unix are you using?

Why do any of those matter?

Begin by sorting the data by app_id then by id.
Then use stage variables to detect change of app_id (new group) and to process adjacent pairs of rows.

Note that there is no "look ahead" in DataStage - but you can always sort in reverse order!

Makes no difference to what Craig suggested.

You'll need to convert the string to a date if it's valid, otherwise set the value to null. Do these things in a Transformer stage.

Something like:

And have you checked that one or both of these environment variables is set for the executing user? (Tip: the second event in the job log will display environment variable values.)

So, ignoring the header for the moment, all you really need to do is to process label/value pairs. This is very similar to processing name/value pairs (for example in Hadoop) so should be very straightforward. Use stage variables to capture the values from the header rows to build your record struct...

Yes. The Operations Console is your management console for DataStage workload management. You can be as clever as you like with the queues.

Replace Delimiter property with Delimiter String property.

What error messages do you see? Have you properly provisioned all the required components?

At least three solutions exist. What have YOU tried?

Augmenting what Andy said, DataStage requires that the Null Field Value have the same length as a Char data type field. You can get around this by declaring it to be VarChar, and sorting it out afterwards.

It would probably be far easier to create the equivalent osh script (for sufficiently simple transformation rules that could be processed by a modify operator).