Has anybody tried using DataStage,(with or without DataStage Version Control, in conjunction with ClearCase for source control (checkin/checkout) and DS program version control? If so, could you explain how you did it and how it is working for you?
Thanks in advance.
Ed
DataStage and ClearCase usage for change control?
Moderators: chulett, rschirm, roy
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Admitting up front that I know nothing about ClearCase, I would imagine it would have difficulties, because DataStage is not just about code; in fact (apart from Routines) the DataStage developer does not work with code at all. Further, some code is not stored in an accessible fashion (for example Routines and Transforms are kept each in a single column in a hashed file). Job designs are stored as collections of job design objects, again in a hashed file. I would suspect that ClearCase can not access hashed files (which also argues against other source code control software).
With a little bit of management, Version Control can provide a basic level of check-out/check-in capability, if all DataStage objects are kept as read-only even in development, and only opened for update by a management process/person.
With a little bit of management, Version Control can provide a basic level of check-out/check-in capability, if all DataStage objects are kept as read-only even in development, and only opened for update by a management process/person.
VC and ClearCase
I'm at a client now that is using ClearCase for their Java application development, but basically only using Version Control for its DataStage source control. ClearCase is used to store entire-project .dsx backups since VC's intent is to be 'version control' and not as a backup process, but there hasn't been any thoughts to try to use CC for source code control.
I would think the best idea is to use only VC for code control, and consider CC as a place to store backups.
I would think the best idea is to use only VC for code control, and consider CC as a place to store backups.
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
thanks for DS/ClearCase response...
Thanks for all of your responses. All of your feedback
provided great discussion points for developing the
DS Source Control/Version Control process we will use.
It's good to know that the DS Version Control tool has proved
very useful. It looks like we will need to develop some custom
code to interface with ClearCase for Source Control - Checkin/Checkout.
I'll post our final solution, once it is completed and running.
Appreciate further postings on this topic.
Ed
provided great discussion points for developing the
DS Source Control/Version Control process we will use.
It's good to know that the DS Version Control tool has proved
very useful. It looks like we will need to develop some custom
code to interface with ClearCase for Source Control - Checkin/Checkout.
I'll post our final solution, once it is completed and running.
Appreciate further postings on this topic.
Ed
I have been using DataStage with Clearcase to handle promotions from development to integration testing, user acceptance ( and eventually production). We have been checking in DataStage components by functional categories into Clearcase using XML. XML gives better visibility as to what has been checked into Clearcase than a DSX export file would. Each export to XML includes all jobs, transforms, routines etc within a catgeory. We have been automating the process wherever possible. We plan on experimenting with checking individual components also.
Version Control Difficulties
Hi,
I currently work with a global MNC using DS for ETL efforts.There are almost 400 jobs in the projects for various data flows.
The key issue is multiple developers handling (read modifying) a single job design(as required by the end user group he/she supports).
Will a configuration management/version control tools like VSS, Clear Case or tru change work at all? Am told that version control tools allow only one check in , check out ( does it mean multiple developers cannot work on the job design?)
Can we build a library of DS objects( i.e jobs, transforms, routines etc so that re usability is enhanced)?
Do Version Control tools recognize changes made i.e I have job design A which is JDA.DSX and I change the job design A to Job design AC.dsx Can the configuration tool figure out that it is changed inspite of the same name?
PLEASE ANSWER ME WiZARDS!!
Thanks in advance for your attention!!
Regards
Srinivas.B
I currently work with a global MNC using DS for ETL efforts.There are almost 400 jobs in the projects for various data flows.
The key issue is multiple developers handling (read modifying) a single job design(as required by the end user group he/she supports).
Will a configuration management/version control tools like VSS, Clear Case or tru change work at all? Am told that version control tools allow only one check in , check out ( does it mean multiple developers cannot work on the job design?)
Can we build a library of DS objects( i.e jobs, transforms, routines etc so that re usability is enhanced)?
Do Version Control tools recognize changes made i.e I have job design A which is JDA.DSX and I change the job design A to Job design AC.dsx Can the configuration tool figure out that it is changed inspite of the same name?
PLEASE ANSWER ME WiZARDS!!
Thanks in advance for your attention!!
Regards
Srinivas.B
Ed
I posted this on tools4datastage.com but what I would do is export the jobs to one dsx. You then need to split that file into one dsx per job. Next you need to check those in. We used CC at a customer site and all we did was check in a whole dsx. This does nothing except archive the file. You still need to read that file into DS to find the one job you want to restore. You have no idea if the job was changed or when. I think all of these are serious problems. Some I have already addressed in my product DwNav.
Kim.
http://www.Duke-Consulting.com
I posted this on tools4datastage.com but what I would do is export the jobs to one dsx. You then need to split that file into one dsx per job. Next you need to check those in. We used CC at a customer site and all we did was check in a whole dsx. This does nothing except archive the file. You still need to read that file into DS to find the one job you want to restore. You have no idea if the job was changed or when. I think all of these are serious problems. Some I have already addressed in my product DwNav.
Kim.
http://www.Duke-Consulting.com
Version Control
I am told that DataStage Twister (7.0) will contain a much more robust solution for Version Control, including better integration within Designer and Manager.
We reviewed Version Control for 6.0, and found it to be woefully inadequate for our needs. We are currently using a file-based system with extracted .DSX files (one for each program), and we are hoping that Twister would provide a much more solid (and intuitive!) solution.
My advice: Wait until Twister comes out later this month (or next month? -- who knows) and review the latest possible solution for your needs.
-T.J.
We reviewed Version Control for 6.0, and found it to be woefully inadequate for our needs. We are currently using a file-based system with extracted .DSX files (one for each program), and we are hoping that Twister would provide a much more solid (and intuitive!) solution.
My advice: Wait until Twister comes out later this month (or next month? -- who knows) and review the latest possible solution for your needs.
-T.J.
Teej
VC only archives versions of a job. There is no diff to compare versions of any of these products because ClearCase and all these tools basically store a copy of an export file. There is locking on a job because ClearCase cannot be integrated with DataStage at the level of control. VC can do some of these things by marking a job as read only. Not a lot of real help. The key is a diff program to show you what changed. All these solutions are really snapshots of a job at a given time. No real change control management. You have to teach your developers to work through these tools because the tools cannot control the jobs.
Kim.
VC only archives versions of a job. There is no diff to compare versions of any of these products because ClearCase and all these tools basically store a copy of an export file. There is locking on a job because ClearCase cannot be integrated with DataStage at the level of control. VC can do some of these things by marking a job as read only. Not a lot of real help. The key is a diff program to show you what changed. All these solutions are really snapshots of a job at a given time. No real change control management. You have to teach your developers to work through these tools because the tools cannot control the jobs.
Kim.
Mamu Kim