Crazy debug idea

[PhilHibbs]

I've been editing my jobs adding debug code and debug outputs, and then deleting them afterwards because I don't want the overhead in the QA or production jobs. Then I have to add it back in again to test some change, and then delete it again.

I've had an idea how I can do this better.

All debug code is created in Shared Containers. Links go into the container, and nothing comes out. In the container are all the Sort/Aggregate/Lookup/Join/Sequential File stages. When the time comes to remove the debug code, the Shared Container stage in the job is deleted, and replaced with a similarly-named Shared Container with the same number of input links, but inside the Shared Container design, all the links are just terminated with Copy stages. When the job is compiled, I'm hoping that the links to these Copy stages should be optimized away into nothing. If the debug code needs to be reinstated, the Shared Container stage is deleted and replaced with the debug version.

Does this idea have legs?

[eostic]

I like that idea.... couldn't you also do this without having to edit the Job? What happens if you (first export it to .dsx) delete the "debug" container, then import the "dummy copy container" of the same name and re-compile?

Ernie

[PhilHibbs]

I like that idea.... couldn't you also do this without having to edit the Job? What happens if you (first export it to .dsx) delete the "debug" container, then import the "dummy copy container" of the same name and re-compile?

Ernie

Yes, that should work. Thanks, that's a great improvement! Now I just need to decide, should all debug code for a job be in a single SC, or should I have multiple SCs each with a different bit of code in it?

*Update*: Works like a dream!

[kumar_s]

Wouldn't you need to recompile the job when you rename using the dsx file edit opiton?

[PhilHibbs]

Wouldn't you need to recompile the job when you rename using the dsx file edit opiton?

Yes, you have to recompile the job after importing the Shared Container.

[trenicar]

Hi

I have done similar thing but instead of changing the SC I include a parameter/env variable that is turned off in prod and in the sc I check if the debug statements are needed, if not dont output!

Just my 10 cents

Tim

[PhilHibbs]

I have done similar thing but instead of changing the SC I include a parameter/env variable that is turned off in prod and in the sc I check if the debug statements are needed, if not dont output!

That will still create the debug output files, although with no data. Is the env var check performed in a Transformer or a Filter? In either case, the check has to be performed once per row of data, so you're incurring a performance overhead that my method does not.

[asorrell]

Another debug idea (only works at 8 though)...

I've been doing work at sites where NO changes are allowed in production. Everything must take the Dev->QA->Prod route. To help with debugging issues in production I've setup a parameter set called Debug that contains all the environment variables you'd ever want to use, with the debug settings turned off. Its inserted into every job so you can easily override the settings at runtime for problem diagnosis runs without having to modify and recompile.

It's also been a good educational tool because the developers are asking what specific variables in the set do and playing around with them. Its a good way for them to learn more about the architecture and execution of jobs.

[ray.wurlod]

Care to share the list of environment variables you use?

[trenicar]

The Env variable is checked as a constraint in the transformer, as you say it just doesnt generate any data.

But does the job!