Need to handle new line termination for a field

Rahul Bharadwaj · Post by **Rahul Bharadwaj** » Fri Dec 30, 2011 2:49 am

Please let me know the Equivalent property for 'contains terminators' (Column property in server) in parallel job.

Thanks & Regards
Rahul

nagarjuna · Post by **nagarjuna** » Fri Dec 30, 2011 9:26 am

Save the server metadata as table definition in datastage repository..Go to that metadata & you have an option of sync to parallel ...You would get the parallel job equivalent...Hope this helps..

chulett · Post by **chulett** » Fri Dec 30, 2011 10:30 am

Could you explain for us the issue you are seeing and what things you've tried? Thanks.

ray.wurlod · Post by **ray.wurlod** » Fri Dec 30, 2011 4:27 pm

Rahul Bharadwaj wrote:Please let me know the Equivalent property for 'contains terminators' (Column property in server) in parallel job.

There isn't one.

chulett · Post by **chulett** » Fri Dec 30, 2011 5:17 pm

Ray is being pedantic again.

Then how about just "the equivalent" rather than "the equivalent property"... as in how would one handle the situation in PX.

ray.wurlod · Post by **ray.wurlod** » Fri Dec 30, 2011 6:43 pm

The parallel Sequential File stage is notorious for not being able to handle embedded newlines. This is not pedantry - search back through the myriad posts complaining about it.

Note that simplistic "solutions" such as using a shell script to delete the new lines don't cut it for me - I believe the customer's data is sacrosanct - if these newline characters are part of the data, then (lacking advice to the contrary) they're part of the data and must be handled accordingly.

My workaround, as often as not, has been to put a server Sequential File stage into a server Shared Container and to embed that in the parallel job.

chulett · Post by **chulett** » Fri Dec 30, 2011 6:50 pm

I recall issues with embedded quotes but thought there was some sort of equivalent for this in the PX Sequential File stage, guess I thunk wrong.

For those of you who have faced and overcome this issue, what solution did you architect? What was your 'work around'?

Ah... I see Ray added his while I was typing. Thanks!

pandeesh · Post by **pandeesh** » Fri Dec 30, 2011 8:40 pm

i have never come across this scenario.
Thanks Ray and Craig! Good learning!

Rahul Bharadwaj · Post by **Rahul Bharadwaj** » Mon Jan 02, 2012 12:28 am

Thanks Ray, I overcomed this situation by using server shared containers in parallel job. Thank you.

zulfi123786 · Post by **zulfi123786** » Mon Jan 02, 2012 12:35 am

We have address processing and its a big junkyard..... had single and double quotes as part of data and we had to use extended ascii characters to quote all fields.
Never had newline characters, the front end guys need to understand its no sense in capturing carriage return and new line charactes as part of any textual descriptions.

cparru · Post by **cparru** » Wed Apr 04, 2012 6:18 pm

Hi Guys,

I tried this solution tp load the address data which is broken into 2 lines. I was able to view the data in shared container/sequential file stage, but while loading it into the table (from a parallel job) it is still loading 2 records for one address. How do i remove that new line character(a square box

) from the output? ereplace didn't work for me, its not compiling!

Appreciate your suggestions!

ray.wurlod · Post by **ray.wurlod** » Wed Apr 04, 2012 6:32 pm

Can you please post what the "resolution" is?

My point made earlier is that you should NOT remove the newline, at least not without the customer's approval - it's part of the data. So you have to find a mechanism for loading the data in which any character is legitimate. I suspect you've used a bulk loader, which has interpreted the end-of-line in its own way.

DSXchange

Need to handle new line termination for a field

Need to handle new line termination for a field

Re: Need to handle new line termination for a field

Thank you ray

broken address