datastage development process

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
abyss
Premium Member
Premium Member
Posts: 172
Joined: Thu May 22, 2014 12:43 am

datastage development process

Post by abyss »

Hi all
we have been using datastage for a few month now, we are still looking for a proper development process (check-in, check-out process, deployment process, version control process... etc, etc).

currently I have created different sandboxes for each developer, they can import the job into the sandbox and update the version number/comment in general tab them import the job back to shared project.

I just want to know what's the development processes you guys use at work, hopefully I can learn some from you guys

thanks
abyss
jhmckeever
Premium Member
Premium Member
Posts: 301
Joined: Thu Jul 14, 2005 10:27 am
Location: Melbourne, Australia
Contact:

Post by jhmckeever »

Hi abyss,

We've created a custom menu item in the DataStage Designer client which invokes a checkin-process.

The process is execute by Atlassian Bamboo (a build automation tool similar to Jenkins or Hudson) which extracts the currently open DataStage job from XMETA, invokes a set of unit tests, then checks the job into Stash (Atlassian's wrapped version of Git). The unit tests involve loading the DataStage job's OSH into a graph database and testing for anti-patterns, such as adjacent transformers, or anything else we want to identify. Some lead to warnings, some lead to the check-in being rejected. Bamboo can additionally update the job's long description with version/deployment information.

Jobs are then automatically deployed to an integration test environment (again using Bamboo) where they are tested against restricted data volumes with functional test cases. When all tests are passed the job becomes a candidate for promotion to production, or other test environments as necessary.

This process means all developers work in the same project, and we have good control over, and transparency of, our code releases.

HTH,
J.
<b>John McKeever</b>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
patpicos
Participant
Posts: 2
Joined: Wed Oct 30, 2013 6:39 am

Post by patpicos »

jhmckeever - would you mind doing a write up on the code validation steps that you have built ?

From a unit test perspective, are you testing unit tests against data or did you mean the code validation steps were your unit tests.


We are currently investigating how we can build jenkins pipelines, deploy continuous integration databases, load sample data, run automated tests, and if all successful, release the code to QA environment.
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

My current location puts assigned developers into a common dev project. When they're satisfied with unit testing, the lead exports objects into a dsx file, and we promote that to a separate test project. When user signs off, we promote to production. We don't bother with all the fancy, "proper" versioning numbers, check in, and check out processes. It's been working quite well that way for 10+ years. I have also seen much more formal, tedious, time-consuming processes in previous jobs. I guess it depends on the work culture (and possibly on which industry you're in) and how uptight or relaxed the organization is and how much effort they can afford throw into formalities... :wink:
Choose a job you love, and you will never have to work a day in your life. - Confucius
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

My previous project used the check-in/check-out capabilities exposed in Information Server Manager to interact with SubVersion. Developers had to be trained to use this mechanism universally, and there was some management overhead monitoring compliance.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
jhmckeever
Premium Member
Premium Member
Posts: 301
Joined: Thu Jul 14, 2005 10:27 am
Location: Melbourne, Australia
Contact:

Post by jhmckeever »

patpicos wrote:jhmckeever - would you mind doing a write up on the code validation steps that you have built? From a unit test perspective, are you testing unit tests against data or did you mean the code validation steps were your unit tests.
I'll send you a private message and we can discuss it.
<b>John McKeever</b>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
jhmckeever
Premium Member
Premium Member
Posts: 301
Joined: Thu Jul 14, 2005 10:27 am
Location: Melbourne, Australia
Contact:

DataStage Continuous Integration for Atlassian

Post by jhmckeever »

Those following this thread might be interested to know that my organisation's solution for implementing DevOps approaches with DataStage development (called 'MettleCI') has now been announced on the Atlassian Marketplace:

https://marketplace.atlassian.com/plugi ... .mettleci/

You can find out more here: https://www.mettleci.com

Image
<b>John McKeever</b>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
Post Reply