Setting global variable from job?

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Setting global variable from job?

Post by jackson.eyton »

Hi everyone,
We have run into an issue where we need to pull in a value from a source, could be flat file or database table, doesn't matter to us. However I would like to have that value saved as a global variable that is accessible to any other job. Is something like this possible? Could execute something at the command line to set an evironment variable value or other variable value that I can then use in a parallel job?

To be fair I COULD do this via a file read in a sequence job, and pass that value down to all other jobs as a job variable. That would entail me having to add that job variable to every job. *NOTE: I've had issues passing values to specific parameters in a parameter set from a sequence job, hence my comment to add the job variable to each job as opposed to using a parameter in a set.

I read on here some time back regarding setting a job to read-only by setting a value for a job in a database table that stores job information. I could have misunderstood but is this a thing? Is there a database that stores various datastage project information? If so could someone point me in the right direction? My googlefoo must not be that good as I have yet to determine this on my own. :oops:

UPDATE: It looks like the datastage project and job information is stored in our XMETA database. Trying to determine what tables contain the projects environment variables.
-Me
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

I'm not clear on how you want to implement it. Do you want the variable value updated for every cycle, or as a one-time task?

Either way, invoking this dsadmin command for environment variables should be doable from a script or command line:

Code: Select all

${DSHOME}/bin/dsadmin ${CLI_ARGS} -envset ${VARNAME} -value "${VARVALUE}" ${PROJ_NAME}
We store the variables in a ssi file, sourced by a script that builds and runs the sample above, invoked by our scheduler app Control-M. We run this as a request job when we need to change a value or add a new variable (-envadd). It handles encrypted variables as well.

Edit: our script runs the command in a do-while loop for multiple variables in one job session.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Thanks FranklinE,
That is exactly what we were looking for! I will need to go through and add the Environment variable to every job, unless someone knows a way to make an env var available to all jobs without adding it to the jobs parameters? Not a huge deal to add one thing like this though. Our sequence for staging will go ahead and set the env var value.
-Me
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

Unfortunately, you will need to add the variable to every job that needs it. However, if you already have a variable or parameter in place (in a User Variable stage, perhaps) you can value it with the environment variable.

An alternative is exporting your Design code to a dsx file and editing it, but I really don't recommend that approach except as a last resort.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

So this kind of works, it seems the value does not update until the project is reloaded. For scheduled jobs this may indeed work just fine. For curiosity sake does anyone know if its possible to update a parameter value for a parameter set via CLI in a given project?
-Me
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

it seems the value does not update until the project is reloaded.
This is a bit mysterious to me. When we run env updates -- always in batch mode, I should point out -- we don't go through a "reload" process. The value is reflected in the repository, and any jobs that run after the update reflect the update.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Yea something is not right... using the datastage administrator I can see the correct value of the variable, yet when I go to run the job that uses said variable it shows the previous value. Doesn't matter if I close datastate and relaunch, recompile the job, etc. It doesnt matter if I place the environment variable in a parameter set, or add it outright to the job parameters. What appears to be happening is that when the variable is added to the jobs parameters it loads the variables current value at that time to the parameters default value. If the variable value changes it doesn't reflect that change in the job. Even if I add the variable to the job parameters and remove the default value from that, it then treats the variable as empty. As if the job does not read the values from the projects environment variables outside of the first time the variable is added to the job parameters.
-Me
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

Ah. Looks like a timing issue. The difference between us appears to be that updating the env variable(s) happens before any jobs that use them are invoked. The gap in timing for us is often hours, depending on how we are constrained from scheduling configuration changes.

Experiment: make sure no jobs are running or in a state about to run. Use Administrator to update the env variable. Click OK and close, then go to Director and run a job that uses it.

My expectation is that the Info entry that displays the parameters of the job will show the correct value.

If it doesn't, I'll jump out of my ground-level window. :wink:
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Ok, I just ran the following steps:
1. Created new Environment variable: APT_LASTDATEPROCESSED and set value to 2018-03-25

2. Added the environment variable the the parameters in a test job properties, this shows the default value of 2018-03-25

3. Compiled and ran the job with $APT_LASTDATEPROCESSED in a transform stage output link derivation. Verified the column populated in the database table with the value 2018-03-25

4. Opened administrator again and edited the value of APT_LASTDATEPROCESSED to 2018-03-28

5. Re-ran the job, the parameters tab if the Job Run Options window shows the old value for $APT_LASTDATEPROCESSED, confirmed no change in the database upon successful completion of the job, value 2018-03-25 is still being passed through.

6. Re-compiled the job and repeated step 5, no change.

7. Removed the environment variable from the parameters of the job properties and re-added. This now shows the updated value of 2018-03-28. Ran the job and confirmed the new value is populated in the table.


What REALLY gets me.... is I just ran a similar test using a parameter set and a value file.
http://www-01.ibm.com/support/docview.w ... wg21610390

What gets me here is that the value from a parameter set supposedly is SET once the job is compiled with whatever value is in that parameter set at that time. (my tests do not confirm this but instead confirm its the value as is set at the time the parameter set is added to the parameters of the job properties).
http://www-01.ibm.com/support/docview.w ... wg21448099

EDIT: previously unbeknownst to me there is a dropdown for the parameter set in the Job Run Options to select values (As Pre-Defined) or %ParamsetFileName%. Obviously, using the file name works as expected.
Last edited by jackson.eyton on Thu Mar 29, 2018 1:52 pm, edited 1 time in total.
-Me
FranklinE
Premium Member
Premium Member
Posts: 739
Joined: Tue Nov 25, 2008 2:19 pm
Location: Malvern, PA

Post by FranklinE »

Prepare for possible embarrassment I've felt many times. :wink:

What is the Default Value of $APT_LASTDATEPROCESSED in the job you run?

If it's not $PROJDEF, there's your problem.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

All the points to you sir! I see it now in the second link I sent, apparently needed to read a little closer. That indeed did the trick. Thanks!
-Me
Post Reply