BigIntegrate Questions
Posted: Tue Sep 12, 2017 10:47 pm
Hi
We have BI 11.5 on Linux, Hortonworks Data platform. We are building a huge data lake in hadoop. I have few questions about how BI 11.5 works. I couldnt find answers to these in google. So could you please share some light.
1. Hive Queries - If i want to run any HIVE query on interactive mode using PUTTY, by default the execution engine is on Tez, but we can change this to use map reduce engine. But how does BigIntegrate works? For Example when i use a HIVE stage does it runs on Tez or MR engine? What about if i use File Connector stage to build a HIVE table?
2. Dynamic configuration File - Can someone provide the IBM link for some details on what is Dynamic config file and how it differs from traditional DS config file?
3. is there a way to start a tez session and keep it open, then run all my HIVE queries and then once i complete all HIVE queries, i would like to close the tez session. I would like to do this in a BI JOB. for example... I want to open the tez session using an execute command stage (shell script), then run all my jobs that uses HIVE load and extractions, once complete then finally close tez session. I dont know if BI uses tez in the background, but when i run H-SQL's it runs on tez... so i'm assuming that BI would also use tez in the background. but i may be completely wrong..
So any advice on these would be very helpful.
We have BI 11.5 on Linux, Hortonworks Data platform. We are building a huge data lake in hadoop. I have few questions about how BI 11.5 works. I couldnt find answers to these in google. So could you please share some light.
1. Hive Queries - If i want to run any HIVE query on interactive mode using PUTTY, by default the execution engine is on Tez, but we can change this to use map reduce engine. But how does BigIntegrate works? For Example when i use a HIVE stage does it runs on Tez or MR engine? What about if i use File Connector stage to build a HIVE table?
2. Dynamic configuration File - Can someone provide the IBM link for some details on what is Dynamic config file and how it differs from traditional DS config file?
3. is there a way to start a tez session and keep it open, then run all my HIVE queries and then once i complete all HIVE queries, i would like to close the tez session. I would like to do this in a BI JOB. for example... I want to open the tez session using an execute command stage (shell script), then run all my jobs that uses HIVE load and extractions, once complete then finally close tez session. I dont know if BI uses tez in the background, but when i run H-SQL's it runs on tez... so i'm assuming that BI would also use tez in the background. but i may be completely wrong..
So any advice on these would be very helpful.