Search found 733 matches

by FranklinE
Tue Oct 23, 2018 10:27 am
Forum: General
Topic: Best way to call a Shell Script
Replies: 5
Views: 3710

I avoid using the correct name "Execute Command" because it sounds like I'm ordering someone's demise.

But yes, that's the one, officer. I saw it do the deed.
:lol:
by FranklinE
Tue Oct 23, 2018 6:43 am
Forum: General
Topic: Best way to call a Shell Script
Replies: 5
Views: 3710

As Craig mentions, it depends on some specifics, but I use a specific standard for the first decision about "where": what level of error handling does it require? The Exec stage is best for this. It handles return codes, it provides explicit tracing of parameters, and (most important) writ...
by FranklinE
Mon Sep 10, 2018 7:17 am
Forum: General
Topic: Envisioning an ideal ETL environment
Replies: 2
Views: 2892

I'm a look at the abstract first sort of guy, but I quickly follow it with take a very close look at that is happening right now in the trenches. Best practices is a nice discussion to have, but unless you can count on your shop to actually follow them -- to be self-disciplined and self-enforcing --...
by FranklinE
Tue Sep 04, 2018 9:58 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Find previous business day
Replies: 7
Views: 4248

If you don't have a calendar function, like in Control-M, a simple design suggestion: Identify all jobs that depend on having an accurate previous business day date. Have the last job update a previous business date file with the current date. This implies that all of those jobs must complete before...
by FranklinE
Wed Aug 29, 2018 7:26 am
Forum: General
Topic: Latency between engine/database and clients
Replies: 5
Views: 2659

Yeah, late bordering on never. :wink: The trace made it rather clear. The physical location of the main gateway, where our guardians of security reside, is the culprit. The distance itself will impose an unavoidable increase in latency, but it should be tolerable. What we have now is basically a 10 ...
by FranklinE
Tue Aug 28, 2018 1:41 pm
Forum: General
Topic: Latency between engine/database and clients
Replies: 5
Views: 2659

We are implementing Cisco Any Connect, replacing Citrix. The issue is not on that level, I'll clarify. The big change is geographic distance. During the long transition to relocate the operational side, the firewall/gateway remains "local" to the developer workstation, but the engine and d...
by FranklinE
Tue Aug 28, 2018 7:49 am
Forum: General
Topic: Latency between engine/database and clients
Replies: 5
Views: 2659

Latency between engine/database and clients

We are migrating our "local" infrastructures to a remote site. We moved our DEV and INT instances recently, and the first fallout was an egregious increase in delay between the engine/database and both Designer and Director clients. We are discussing the issue internally, and have an IBM r...
by FranklinE
Wed Aug 15, 2018 8:52 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Is there a way to restart a job from point of failure?
Replies: 7
Views: 4826

High-level error handling design is where restartability is identified. Error handling is a part of the definition of the unit of work. Example: 1. Download file. If that fails, fix problem and rerun. 2. Process file. If there are no intermediate points of failure -- like commits -- if the process f...
by FranklinE
Mon Aug 06, 2018 6:53 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Create HIGH HEX value
Replies: 7
Views: 4340

You have to juggle the settings. Sometimes the two character sets, ASCII and EBCDIC, have implied conversions between them that you can't control. If it were my job -- and that means you might not find this valuable -- I'd customize the entire job to use EBCDIC throughout. At some point, you need to...
by FranklinE
Mon Aug 06, 2018 6:17 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Create HIGH HEX value
Replies: 7
Views: 4340

The only thing I can think of is that you are setting the char column under the control of an ASCII character set. That's why it's getting converted to xA7 or x40.

Check your project defaults. You may need to put peeks at every stage to uncover what's happening.

Good luck.
by FranklinE
Wed Aug 01, 2018 10:50 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: FTP Enterprise stage to read Mainframe variable blk dataset
Replies: 4
Views: 3342

We use a generic FTP Enterprise stage and job for all mainframe datasets. It has a single column definition of binary with unspecified length. Since the datasets have no delimiters, this creates a file on the server that is identical to the dataset source, a single stream of data full file to full f...
by FranklinE
Wed Aug 01, 2018 10:42 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: FTP Enterprise stage to read Mainframe variable blk dataset
Replies: 4
Views: 3342

CFF is very good with all of the COBOL formats, with some exceptions outlined in the FAQs. Binary (COMP) fields can have endian issues, for example. The prefix field showing the record length is the first obstacle, so far as I can tell. Removing it by reformatting to a fixed-length record is the sim...
by FranklinE
Tue Jul 31, 2018 1:17 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: FTP Enterprise stage to read Mainframe variable blk dataset
Replies: 4
Views: 3342

The use of variable length fields is the first problem. In the COBOL formatting, the binary record length "prefix" is explicitly defined as its own field, but I don't see a similar "prefix" for SEGMENT-DATA. That it's variable based on SEG-TYPE is not enough. You need the field l...
by FranklinE
Tue Jul 31, 2018 12:11 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Passing SQL through a string parameter
Replies: 15
Views: 8039

Late to this thread, not sure if this will help, but we successfully use the following in an ODBC Connector stage to select from MSSql: CALLS.local_start_time >= {ts '#CALL_START_DATE_1# 00:00:00.000'} AND CALLS.local_end_time<= {ts '#CALL_END_DATE_1# 23:59:59.000'} Single quotes around the paramete...
by FranklinE
Fri Jun 08, 2018 12:24 pm
Forum: General
Topic: Run and complete a DataStage Job based on a value in a table
Replies: 13
Views: 8237

I have a further thought around the manual update part. I suggest imposing a technical requirement on the mainframe developers that they accomplish the update with an ESP job. This would eliminate the need for a cyclic approach. You make the DataStage job(s) dependent on the sysdate update job. It/T...