DSXchange :: Alpine Consulting

Introduction

Migrating to any new version of a product can be intimidating so we tend to postpone it as much as we can. However, migration is a necessary step to keep up with technology and take advantage of enhancements.

IBM Information Server 8 (IISv8) is a major upgrade from the Ascential suite of products. The new version has a unified service-oriented architecture with a common Metadata Repository, common logging and common reporting. By eliminating the duplication of functions, the architecture efficiently uses hardware resources and reduces the amount of development and administrative effort required to deploy an integration solution.

In addition to the above, each of the modules has enhanced functionality which fills in most of the gaps that existed in the Ascential suite. Thus, upgrading to IISv8 is well worth the activity.

This article will concentrate on the various options that are available when migrating to the DataStage and QualityStage components of IBM Information Server.¹ The migration options take into account the risks, availability of resources, budget and time for the undertaking.

Before we get started, it will prove helpful to clarify the meaning of the verbs “upgrade” and “migrate”. “Upgrade” refers to replacing older binary files and programs with a newer version, while “migrate” refers to copying resources (text, data file, source code, etc.) to a new system.

Now we will explore the requirements and options available to you as an IISv8 adopter.

Pre-Migration Considerations

IBM Information Server requires a database² to store the common Metadata Repository and WebSphere Application Server (WAS), both of which are bundled with Information Server. Personnel with basic WAS and database administration skills will be required to maintain the above.

If you are planning to use an existing database for the Metadata Repository, make sure you have the correct version of database software (IISv8 is NOT compatible with prior versions of database software). As of this writing IISv8 supports³ the following databases for the Metadata Repository:

a.	IBM DB2 9.1 (default)
b.	Oracle 10g
c.	Microsoft SQL Server 2005

Note that DB2 9.1 and Microsoft SQL Server 2005 can not coexist on the same machine.

If you wish to install IISv8 on a machine that already has SQL Server, you must use Microsoft SQL Server 2005 for the Metadata Repository or uninstall Microsoft SQL Server 2005

If you want to use an existing installation of IBM WebSphere Application Server, then you require WAS release 6.0.2.11 or later.

Both the WAS and IISv8 have to be configured and synchronized. Organizations planning on leveraging LDAP integration should plan on spending significant efforts on this activity.

The DataStage Version Control component is not supported by IISv8. Organizations should plan on implementing another version control system, such as CVS (open source) or any other 3rd party version control system.

DataStage clients have to match the server versions, i.e., the DataStage 7.5.x client will not work with version 8.x server and vice versa.

Datasets that are created from version 7.x and below can not be read or modified by IISv8 jobs.

DataStage jobs that use the QualityStage plug-in are migrated by the installation process. Running these jobs after migration requires configuring the QualityStage plug-in to work with the DataStage/QualityStage engine in IISv8.

QualityStage .imf files cannot be used for migration to version 8. IISv8 provides a separate utility for migrating QualityStage 7.x objects to the DataStage client. Note that in IISv8, QualityStage is now a component of DataStage.

DataStage Migration

3.1.

Upgrading from versions earlier than 7.0

Versions prior to 7.0 cannot upgrade directly to IISv8. There are two available upgrade⁶ options:

1.	Upgrade DataStage server to DataStage version 7.5.2 and then run the IBM Information Server installation program to upgrade to version 8.
2.	Backup (export) DataStage design objects or source code (.dsx files) and associated data and uninstall DataStage before installing IBM Information Server. Then restore (import) your objects and data. This option can also be used to move DataStage to a different computer.

If you are using a UNIX or LINUX server, you can maintain your existing version of DataStage server and install IBM Information Server alongside it. You cannot do this in the case of Windows installations.

3.2.

Options for Migrating 7.x to 8.x

There are three options to migrate an existing DataStage 7.x environment to DataStage 8.x:

	1.	New Installation (Best Option)
	2.	Upgrade existing version using the Version 8 Installation Program
	3.	Co-Existing Versions (Multiple Instances)

New Installation (Best Option):

This involves installing IISv8 on a computer that has no DataStage versions. This can be on a new machine or an existing one where all previous DataStage installations have been removed. Export the jobs from the old version and import the jobs into the new computer containing version 8.x.

Advantages

	1.	New installation, if implemented on a different machine other then the existing DataStage server, has minimum impact on existing processes. There is time and room for testing the new installation independent of the existing processes before switching to IISv8.
	2.	If implemented on a new machine, the transition from the old version to IISv8 can be gradual, thus minimizing potential issues that could arise from unsupported objects.

Disadvantages

	1.	DataStage job dependencies need to be addressed separately as the location of dependent objects may change during migration. The jobs affected have to be modified to reflect the change.
	2.	Significant effort is needed for moving files such as hashed files, parameter files, configuration files and shell scripts to the new location. Datasets have to be rebuilt if they are used persistently across jobs.
	3.	If there is no available machine for the new installation, acquisition of a new machine adds to the migration cost.

Upgrade an existing 7.x version using the IISv8 installation program:

This option will replace the existing version with the new one. It will use the IISv8 installation program to migrate existing projects to 8.

During the installation process, your projects are prepared for migration to the new metadata repository. You then use the WebSphere DataStage and QualityStage Administrator client to migrate each of your projects.

Advantages

	1.	Use of this method does not require moving project dependencies. The dependencies remain in the existing location during the migration.
	2.	Reuse of hardware reduces the cost of migration.

Disadvantages

	1.	There is more impact on existing processes because the server will be unavailable during the entire upgrade and testing process.
	2.	The risk of incompatible and/or unsupported objects increases the potential for system down time and disrupted services.

Co-Existing Versions (Multiple Instances):

Different versions of DataStage can co-exist on UNIX computers. For example, IISv8 can be installed on a Linux computer already running DataStage 7.5. Note that different versions of IISv8 cannot be installed on the same computer (e.g. just 8.0 or 8.0.1, not both).

Advantages

	1.	Existing processes can run with minimum interruption, if at all, the downtime is only during installation.
	2.	Except datasets, the jobs in the new version can point to existing files, e.g. DataStage jobs can point to the same hashed files created in the old version.
	3.	Co-existing versions can reduce the cost of migration if the existing computer has enough resources for both versions.

Disadvantages

	1.	Co-existing versions can compete for resources, thus degrading performance.
	2.	Referencing files from the old installation can create more confusion about dependencies, adding to the complexity of the upgrade.
	3.	Datasets from version 7.5 and older are not compatible and can not be read or modified in version 8.x. If used persistently across jobs, they have to be recreated in IISv8.
	4.	Co-existing versions can increase the cost of migration if the existing computer does not meet the resource requirements of both versions.

QualityStage Migration

4.1.

The Migration Utility

The migration utility uses information in the QualityStage 7.x server project directory to build the .dsx file format that WebSphere DataStage requires to import jobs.

The migration utility migrates the following QualityStage 7.x objects:

Standardization rule set
Jobs in full legacy operational mode
Jobs in expanded form, in which some legacy operations are replaced by QualityStage 8.0 stages
Match specification

The migration utility creates a .dsx file that contains migrated jobs, rule sets, and match specifications in the Temp directory in the QualityStage 7.0 to 7.5 project directory.

4.2.

Two Options for QualityStage Migration

There are two options to migrate your existing QualityStage 7.x environment to QualityStage 8:

	1.	Full Legacy Operation Mode
	2.	Expanded Form

Full Legacy Operation Mode

In this mode the original job is replaced with a single instance of a QualityStage Legacy stage and Sequential File stages linked as source and target stages.

Use this option only for stable jobs that never need to be modified or jobs that are due to be replaced.

Advantages

All stages are QualityStage Legacy stage and hence consistent.

Disadvantages

You can make only minor changes to the resulting QualityStage Legacy stage

The following stages cannot be migrated using this mode because their functionality is not supported by the QualityStage Legacy stage

a.	Postal stages such as CASS and SERP
b.	Program stage
c.	Multinational Standardize stage
d.	WAVES stage
e.	Format Convert stage

Expanded Form

When you migrate a QualityStage 7.x job in expanded form, the utility replaces the original job with one or more parallel job stages for each stage in the 7.x job, and with Sequential File stages that are linked together to represent the 7.x job flow.

Advantages

	1.	Most of the stages are migrated to their equivalent version 8 stages and hence less reprogramming required.
	2.	Almost all stages are supported in this mode.

Disadvantages

	1.	There will be a mixture of data quality, processing, and Legacy stages when you open the job in the Designer client and as a result lacks consistency.
	2.	Some additional development may be required to realize logic that doesn’t migrate automatically.

Conclusion

There are three different options for migrating DataStage jobs and two options for migrating QualityStage jobs to the latest version of IBM Information Server. The options take into consideration all possible situations based on risks, available resources, time and budget. Once these options have been understood, migrating to IBM Information Server can be smooth and manageable.

The options available are summarized below. Our choice for the best DataStage migration option reflects our inclination towards minimum risk to existing processes, and hence, to use a new or separate machine for IISv8.

The choice for the best QualityStage migration option is based on our preference to take advantage of the new features and parallelism offered in the new release.

Module	Migration Options	Comments
DataStage
	New Installation (no previous version exists)	Best Option
	Upgrade Existing Version
	Multiple - Server Method (UNIX or Linux only)

QualityStage
	Full Legacy Operation Mode
	Expanded Form	Best Option

References

IBM Information Server Version 8.0.1 product documentation

http://www-1.ibm.com/support/docview.wss?uid=swg27009462#infos

IBM Information Server Version 8 system requirements

http://www-1.ibm.com/support/docview.wss?uid=swg27008923&rs=2321

Alpine Consulting Inc. whitepaper, What is New in DataStage version 8.x

About the Authors

Ranjana Vadakkepatti

Ranjana is a consultant with Alpine Consulting, Inc. She has been working with data warehousing and data cleansing technologies for the past 4 years and has developed solutions using DataStage and QualityStage. She has also successfully performed migrations to IBM Information Server v8.

Luisa Garcia

Luisa is a consultant at Alpine Consulting Inc. who specializes in data warehousing and business intelligence solutions. She has worked in several development and migration projects using DataStage.

Contact Information

We welcome your comments, questions and suggestions about this article. We would like to hear about your migration stories as well.

For additional information, please contact infoserver@alpineinc.com or call 800-905-5830.