Search found 4605 matches

by kduke
Fri May 20, 2016 4:03 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: How do I add business days to a timestamp?
Replies: 8
Views: 8435

14 days is not the same as 10 business days. A business day depends on if the business is open. So a holiday does not count as a business day. You have the check every day where it is open. Let us say you have a date dimension with BIZ_DAY = 'Y' then the SQL is a little tricky. You need count record...
by kduke
Fri May 20, 2016 3:40 pm
Forum: General
Topic: How to determine if a job is hung?
Replies: 4
Views: 6310

If a job is hung then several things might help figure it out. If you are getting blocked at the database then the DBA can help figure that out. When a UNIX process gets hung then the RAM size stays the same sometimes. If you have a memory leak then it grows until it fails or job runs out of rows. I...
by kduke
Fri May 20, 2016 3:25 pm
Forum: General
Topic: How to pass parameters to a generic job script?
Replies: 12
Views: 10114

dsjob -lparams $DSProject $JobName Will give you a list of parameters. If you send in a parameter to a job that does not exist in the job then it will fail. We put the parameter values in files. We would look for a parameter in jobname.prm. If it existed in this then we would run the job. If not th...
by kduke
Thu May 05, 2016 6:46 pm
Forum: General
Topic: Why ETL is hard
Replies: 13
Views: 9713

Awesome answer Craig. I too wish Ernie would start a new topic or use this one to expand on what these new tools are trying to address and solve. What questions are coming that we had no idea were even issues. Please educate us Ernie to what is out there. If there are cool concepts or tools being di...
by kduke
Thu May 05, 2016 6:31 pm
Forum: General
Topic: Happy Star Wars Day !
Replies: 3
Views: 3957

Cinco de Mayo. May the fifth be with you.
by kduke
Tue May 03, 2016 9:56 pm
Forum: General
Topic: Happy Star Wars Day !
Replies: 3
Views: 3957

Thanks Obi Wan.
by kduke
Tue May 03, 2016 9:54 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: SQL query character limit
Replies: 7
Views: 5970

The perfect example of doing too much in SQL. Break your SQL up into chunks. You are probably doing ranking. Do that SQL into lookups. Be creative. Learn the ETL tool. Quit relying on your SQL skills to do ETL.
by kduke
Tue May 03, 2016 9:50 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Parameter Set Value File Question
Replies: 4
Views: 4675

Why use a parameter set if you are not going to load the values into it? Use a plain parameter.
by kduke
Tue May 03, 2016 9:44 pm
Forum: General
Topic: Why ETL is hard
Replies: 13
Views: 9713

Great topic to bring back. I miss Ken Bland. He always added a level of deep thought. I still see developers using DataStage poorly. Their SQL takes forever to come back which tells me they did very little testing. It also means they are probably doing too much in SQL. Doing too many joins in their ...
by kduke
Fri Apr 08, 2016 8:25 am
Forum: General
Topic: How to generate custom status report for job
Replies: 2
Views: 3217

THe dsjob command can generate this in XML format so you can load the data.
by kduke
Sun Apr 03, 2016 8:56 pm
Forum: General
Topic: Way to generate automated Excel templates?
Replies: 8
Views: 7224

Explain 5 different layouts you want to merge. If you have 5 different tables then have 5 different jobs. If you have 5 tables that you want to merge into 1 report then sure you can do that.
by kduke
Thu Mar 31, 2016 2:56 pm
Forum: General
Topic: Way to generate automated Excel templates?
Replies: 8
Views: 7224

It can be done. You need to build your header or read it in from sequential file. Then build table tag. Then row tag. Field tag. Field value. Close field tag. Close row tag. Next row. Loop. Close table tag and all header tags. Each row needs a value to sort by. Start header rows with 10,000. Start r...
by kduke
Wed Mar 30, 2016 2:41 pm
Forum: General
Topic: Sanity Check Poll
Replies: 12
Views: 9974

I totally agree. Plus simplifies all integrations not just the EDW. The QA manager will love you. Along with CIO or CTO.
by kduke
Tue Mar 29, 2016 1:33 pm
Forum: General
Topic: Sanity Check Poll
Replies: 12
Views: 9974

EtlStats has a similar value. It is more important to know trends than the actual time a job takes to run. We found issues with DBAs not running stats and network issues or backups running during loads. All these can extend your load times. Being proactive always has its advantages.
by kduke
Tue Mar 29, 2016 1:25 pm
Forum: General
Topic: Sanity Check Poll
Replies: 12
Views: 9974

Thanks. Someone said why would I pay for that. It is just a few SQL statements. If your data warehouse has 200 tables and 5 sources then you need to 400 simple SQL statements to do a simple sanity check. You can't really do any trending based on those numbers because you do not store results. Is you...