File Connector, Big Data Stages - BigInsights compatibility

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Re: File Connector, Big Data Stages - BigInsights compatibil

Post by chulett »

cdp wrote:In the Big Data File stage documentation there is no such mention and I understand that the Big Data File Stage CAN access non BigInsights HDFS:
To use the Big Data File stage, you must ensure that the stage can access the HDFS libhdfs.so shared library, its associated JAR libraries, and configuration files.
If you are using the Big Data File stage to access an InfoSphere BigInsights HDFS by way of the REST connection method or from an AIX operating system, the InfoSphere BigInsights libhdfs.so library is not required.
If your understanding is based on the If you highlighted, I assume you also need to bold the "by way of the REST connection method or from an AIX operating system" statement as that is what it applies to. Not just the "if BigInsights" bit. And that just to say you don't need to ensure access to the shared library. At least that's how I'm reading it.

Or does your understanding come from elsewhere?
-craig

"You can never have too many knives" -- Logan Nine Fingers
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

The Big Data File stage is available in DataStage Designer 11.3 on our system, and we do not have InfoSphere BigInsights installed. Its properties look the same as a regular Sequential File stage.
Choose a job you love, and you will never have to work a day in your life. - Confucius
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Available or available and usable?
-craig

"You can never have too many knives" -- Logan Nine Fingers
cdp
Premium Member
Premium Member
Posts: 113
Joined: Tue Dec 15, 2009 9:28 pm
Location: New Zealand

Re: File Connector, Big Data Stages - BigInsights compatibil

Post by cdp »

chulett wrote:And that just to say you don't need to ensure access to the shared library. At least that's how I'm reading it.?
Yes, you're right.
But anyway, the mystery remains for now:
To access a Hadoop Distributed File System (HDFS) with the Big Data File stage , you must make the libhdfs.so [...]
I understand 'a' as in 'any'
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Question for support, then. Or an English major. Or both! :wink:
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply