Match_Frequency Error: Input not sorted at record #

Infosphere's Quality Product

Moderators: chulett, rschirm

Post Reply
emeri1md
Participant
Posts: 33
Joined: Tue Jun 17, 2008 10:42 am

Match_Frequency Error: Input not sorted at record #

Post by emeri1md »

I have had several successfully run jobs before this, but now I'm getting an odd error. I am working with someone who has a few years of experience with QualityStage, and he hasn't seen anything like this either.

I have the usual setup of data running through a standardization stage (in some jobs, MNS in others) and then being split so I can get the match frequency to use in the match reference stage.

When I run the job, I get the following:

Code: Select all

Match_Frequency_25,1: Input not sorted at record 1606
This occurs for each node. The numbers are not always the same.

It seems to run fine if I create a new job and a new match spec to use in the frequency stage. If I change the columns or change the match spec, I get this error.

I don't want to keep creating the same jobs over and over again. I didn't have to before this. Has anyone seen an error like this before?

Further Information (6/20/08):

I just had it occur again. I had a working job. Then I changed the match spec so that the only pass in the spec matched on an additional column. I provisioned the spec/pass. Then I compiled and ran the job with the freq set to no match spec (to clear the data; had problems earlier and this fixed it). Then I ran the job and got the above error. I have the job for the reference data running right now, and now expect the same error (Got the error).

I hope this extra bit of information helps. Am I missing a step? Did I do the wrong step? Do I need a priest to sacrifice a live chicken to get it to work?

Further Information (Still 6/20/08):

I tried creating a copy of the match spec and using that, but it made no difference. I hope I don't have to create a brand new job (again) just to use the new match spec. I've created new jobs in the past and had them immediately fail with this problem. I believe I had to create both a new job and a new match spec.

I'd rather find a better resolution. In fact, I'd rather sacrifice that chicken.

Further Information (Still late in the workday, Friday, with no hope in sight):

I fixed the problem in the first update. The second update refers to a different (but similar) job with the same problem. The first update problem (FUP) was fixed by reverting the match spec back to the way it was, running the job without refering to the match spec, and then running the job with refering to the match spec.

If I figure out a real resolution to this problem, I'll post it here first thing.

Further Information (6/23/08):

I tried recreating the job from scratch, but using an existing match spec. Still get the error. It may work if I create a new job and match spec at the same time, but I somewhat recall trying that once already and still getting the error.

My coworker tried modifying a working match spec and got the error. Then he recompiled and ran the job again. It worked. So how can this happen, yet we can go long stretches with it working until the match spec if changed?

Current Theories (6/25/08):

We've tried many different things in the attempt to fix the problem. Everything seems to hinge on the match spec itself. If we modify it, it may or may not cause the problem. If the problem already exists, then it will likely occur again. Sometimes if you add a pass to a match spec, it may fail. But not always.

A case has been opened with IBM, but they have not yet been able to recreate the problem.
emeri1md
Participant
Posts: 33
Joined: Tue Jun 17, 2008 10:42 am

Further Information

Post by emeri1md »

I have been working on this some more and found some interesting information. I added a pass to a match spec that was already running. Then the error popped up.

I placed the pass into the holding area and ran it again; it worked. I added the pass again, but then deleted a matching column. I still got the error, but it occured at an earlier record. After removing a second column, the error listed 0 records. A third removal also removed the error.

There is something about using the larger number of columns that is causing the job to abort. For the job I'm talking about, the matching columns being removed are the MatchPrimaryWord1-5 that comes from standardizing by name. I started by removing 5 and worked my way back up. All of these columns match by CHAR. There is one additional matching column, but the match spec was working with it.
emeri1md
Participant
Posts: 33
Joined: Tue Jun 17, 2008 10:42 am

Post by emeri1md »

If the problem is already occuring, it seems as though it always aborts if I tell the Match_Freq stage not to use a match spec. If the job is working, it could still fail if I use a match spec, but not always.

Yet when I first ran these jobs, I wasn't using a match spec and it ran fine.
emeri1md
Participant
Posts: 33
Joined: Tue Jun 17, 2008 10:42 am

Quasi-Useful Workaround

Post by emeri1md »

I've been in contact with IBM support and was given a workaround until a patch could be released. Unfortunately, the workaround doesn't always work.

To get around the problem, you must run the job sequentially. It got one set of jobs running again, but not another.
emeri1md
Participant
Posts: 33
Joined: Tue Jun 17, 2008 10:42 am

Working Workaround

Post by emeri1md »

I now have a valid workaround for those having this problem. If you run the affected job on a single node configuration, it should run.

Though we still don't know the cause of the problem.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

I assume that your partitioning setting is (Auto) on all stages? That should mean you are getting Round Robin partitioning on sequential-to-parallel links and Same on parallel-to-parallel links, which ought not to affect sorting, but may locate any one key value on multiple nodes. Have you tried explicitly partitioning and sorting on the blocking field(s)?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
emeri1md
Participant
Posts: 33
Joined: Tue Jun 17, 2008 10:42 am

Post by emeri1md »

Have you tried explicitly partitioning and sorting on the blocking field(s)?
Yes, I have. I still get the same error. And keep in mind that I get the error even if I don't specify a match spec.
mayur25
Participant
Posts: 13
Joined: Fri Jul 04, 2008 11:13 pm

RE: Input data not sorted

Post by mayur25 »

hii

i am working on Quality stage i am also getting same error.
till standardization evrything was working properly then i am using that stan o/p file to get match freq report
but it gives me error that Input data not sorted #
can anyone give valuable suggetion.

thanks in advance
emeri1md wrote:
Have you tried explicitly partitioning and sorting on the blocking field(s)?
Yes, I have. I still get the same error. And keep in mind that I get the error even if I don't specify a match spec.
sri_vin
Premium Member
Premium Member
Posts: 20
Joined: Wed Aug 25, 2010 10:58 pm

Post by sri_vin »

Please check your fork joins.
Please check your network bandwidth.
Post Reply