Sanity Check Poll

kduke · Post by **kduke** » Thu Mar 24, 2016 8:42 am

Sanity Check Poll

1. Do not know what it is.
2. Know what it is but never seen it implemented anywhere.
3. Know what it is do not like it and never implement one. Kimball and Inmon are idiots.
4. Know what it is. Seen it implemented but think it is not useful.
5. Know what it is. Think it is valid QA technique.
6. Know what it is. Implement it every chance I get.

chulett · Post by **chulett** » Thu Mar 24, 2016 8:54 am

Kim, perhaps an explanation of what that is in the context of DataStage would help? Or is that meant to be part of poll, a.k.a. the first answer? We always do something I would consider to be a "sanity check" but no clue how closely that ties in to any sort of official... thing.

kduke · Post by **kduke** » Thu Mar 24, 2016 9:00 am

Working on it now Craig. Part of the issue is do they know what it is already. I will post what it is soon.

kduke · Post by **kduke** » Thu Mar 24, 2016 10:11 am

The CFO approves the budget for all departments. You want more money then make him/her happy. You want more developers same answer. Want more money as a consultant then know what a Sanity Check is and how to implement it. Same goes for being a DataStage admin. The more roles you know the more valuable you are on a team. Mentor somebody and make the team look better. You will all get paid more.

Take the poll. Help us understand what you know so we can help mentor you.

chulett · Post by **chulett** » Thu Mar 24, 2016 10:57 am

Hopefully that really wasn't just for me.

Thanks!

kduke · Post by **kduke** » Thu Mar 24, 2016 3:29 pm

Craig you are the expert. You don't need my help.

kduke · Post by **kduke** » Fri Mar 25, 2016 10:58 am

Here is the video on our tool that does a Sanity Check.

Auditing Tool - Overview

https://youtu.be/HvLv8OVnlzY

Description:
1. Quality Assurance tool for Sanity Checks
2. Completeness Reporting
3. Quick setup
4. Fast retrieval of measures
5. Flexible - any SQL statement that returns 1 numeric value can be used.
6. Complete audit trail
7. Textbook perfect solution

30:52

kduke · Post by **kduke** » Fri Mar 25, 2016 11:06 am

Any feedback would be great. Good or bad.

priyadarshikunal · Post by **priyadarshikunal** » Tue Mar 29, 2016 2:06 am

We "almost" always do some sanity check or reconciliation at the end of load for some key values. counts amounts for critical information. I prefer to know if there is a problem before business lads finding and start screaming. However I keep such sanity checks precise and minimum, limited to critical information.

But I like how the third option is phrased. I will go with that.

kduke · Post by **kduke** » Tue Mar 29, 2016 1:25 pm

Thanks. Someone said why would I pay for that. It is just a few SQL statements. If your data warehouse has 200 tables and 5 sources then you need to 400 simple SQL statements to do a simple sanity check. You can't really do any trending based on those numbers because you do not store results. Is your revenue up or down? Is this a peak revenue time of year? Simple trending is easily accomplished by running reports against audit trail history. We even narrowed our trending to day of the week. We knew exactly what our revenue was per customer per day of the week. We could easily trend from year to year as well.

If CFOs find out this tool exists then it will be required on every data integration project. There are so many benefits from these simple tools that automate tasks nobody wants to do anyway. Spend a day setting up your source and target tables then see how cool this is. They people that don't want these numbers never ran a sanity check before. Like he said. If I can get the bad news before someone else does then we as developers are way ahead of the game. I always like to stay ahead of the other lads.

kduke · Post by **kduke** » Tue Mar 29, 2016 1:33 pm

EtlStats has a similar value. It is more important to know trends than the actual time a job takes to run. We found issues with DBAs not running stats and network issues or backups running during loads. All these can extend your load times. Being proactive always has its advantages.

priyadarshikunal · Post by **priyadarshikunal** » Wed Mar 30, 2016 1:22 am

Won't it save on developers time, increasing productivity, and lower turn around time, if you get a ready made plug and play thingy. Time is money my friend. :lol

kduke · Post by **kduke** » Wed Mar 30, 2016 2:41 pm

I totally agree. Plus simplifies all integrations not just the EDW. The QA manager will love you. Along with CIO or CTO.