CRC32 with Parallel Server-Job

rennratte · Post by **rennratte** » Thu Mar 10, 2005 10:41 am

Hi,

I always used CRC32 in Server-Jobs.
Now I need the same feature in a Parallel Job, but it is not in the function list. So does anyone know a workaround therefor or made some experiences?

Cheers,
Renate

ArndW · Post by **ArndW** » Thu Mar 10, 2005 11:05 am

Hello Renate (oder Rennratte),

since both products were born of different parents and use different technology there really is no "workaround" available - it is not a builtin function for Px and you would either need to build your own routine for it or use a command-line level UNIX program (freely available all over the internet).

You might wish to approach the problem without using a CRC check - it depends upon what you need to do and how much data you have.

mhester · Post by **mhester** » Thu Mar 10, 2005 12:21 pm

Here goes! If you truly want to use CRC32 in PX (not a bad idea) then you can use the following code to write a c program that will generate a CRC just like the server edition does. This code should return the same value as the server edition for a given string, but then there are no warranties and since it is public domain you would use at your own risk.

This version (as are most) is table driven and this code will build its own table so no need to build the table separately or hand type the 256 values into a header file and include said header.

If you're comfortable with c then this should be no problem as you will have to modify to work with DS data types and actually return a value instead of printing to stdout.

*** Disclaimer ***

Also, I want to add that since this is public domain and if it does not work correctly or as you would expect that you would not really be entitled to a refund of the purchase price since the refund would be very small and if it breaks you are more than welcome to keep both pieces

Code: Select all

/*----------------------------------------------------------------------------*\
*   Original code by: Craig Bruce
 *  This program is public-domain software.
 *  Generates a 32 bit CRC.
 *  based on the works of many different authors including Dr. Ross Williams and
 *  Mark R. Nelson.
 *  This program DOES generate the same CRC
 *  values as GZIP, PKZIP, and ZMODEM.
 *
\*----------------------------------------------------------------------------*/

#include <stdio.h>

int     main( int argc, char *argv[] );
unsigned long getcrc( FILE *fp );
void    crcgen( void );

unsigned long crcTable[256];

/******************************************************************************/
int main( int argc, char *argv[] )
{
    int     i;
    FILE   *fp;
    unsigned long crc;

    crcgen();
    if (argc < 2) {
        crc = getcrc( stdin );
        printf("crc32 = %08lx for <stdin>\n", crc);
    } else {
        for (i=1; i<argc; i++) {
            if ( (fp=fopen(argv[i],"rb")) == NULL ) {
                printf("error opening file \"%s\"!\n",argv[i]);
            } else {
                crc = getcrc( fp );
                printf("crc32 = %08lx for \"%s\"\n", crc, argv[i]);
                fclose( fp );
            }
        }
    }
    return( 0 );
}

/******************************************************************************/
unsigned long getcrc( FILE *fp )
{
    register unsigned long crc;
    int     c;

    crc = 0xFFFFFFFF;
    while( (c=getc(fp)) != EOF ) {
        crc = ((crc>>8) & 0x00FFFFFF) ^ crcTable[ (crc^c) & 0xFF ];
    }
    return( crc^0xFFFFFFFF );
}

/******************************************************************************/
void crcgen( void )
{
    unsigned long crc, poly;
    int     i, j;

    poly = 0xEDB88320L;
    for (i=0; i<256; i++) {
        crc = i;
        for (j=8; j>0; j--) {
            if (crc&1) {
                crc = (crc >> 1) ^ poly;
            } else {
                crc >>= 1;
            }
        }
        crcTable[i] = crc;
    }
}

kduke · Post by **kduke** » Thu Jan 25, 2007 8:43 pm

Michael

I got this to work but it is inconsistent. I am not sure I am compiling correctly. I am on an AIX box.

Code: Select all

/usr/vacpp/bin/xlC_r -O -G -qspill=32704 crc32.C
mv shr.o crc32.o

To compile the command line version:

Code: Select all

/usr/vacpp/bin/xlC_r -O -qspill=32704 crc32.C
mv a.out CRC32

To test command line version:

Code: Select all

CRC32 KIM
crc32 = aa587252 hex
crc32 = -1437044142 long

I think I need to leave off the -G option. I will try that tomorrow.

Final Version:

Code: Select all

#include <stdio.h> 
#include <ctype.h>
#include <string.h>

int     main( int argc, char *argv[] ); 
unsigned long getcrc( char* s ); 
void    crcgen( void ); 
unsigned long crc32( char* s ); 

unsigned long crcTable[256]; 

/******************************************************************************/ 
int main( int argc, char *argv[] ) 
{ 
    FILE   *fp; 
    unsigned long crc; 

    if (argc > 0) { 
        crc = crc32( argv[1] ); 
        printf("crc32 = %08lx hex \n", crc); 
        printf("crc32 = %d long \n", crc); 
    } 
    return( 0 ); 
} 

/******************************************************************************/ 
unsigned long crc32( char* s ) 
{ 
    int     i; 
    unsigned long crc; 

    crcgen(); 
	crc = getcrc( s ); 
    return( crc ); 
} 

/******************************************************************************/ 
unsigned long getcrc( char* s ) 
{ 
    register unsigned long crc; 
    int     c; 
	char* s1;
	
	s1=s;
	
	crc = 0xFFFFFFFF; 
	while (*s1)
	{
		c = *s1;
	    crc = ((crc >> 8) & 0x00FFFFFF) ^ crcTable[ (crc ^ c) & 0xFF ]; 
		s1++;
	}
    return( crc ^ 0xFFFFFFFF ); 
} 

/******************************************************************************/ 
void crcgen( void ) 
{ 
    unsigned long crc, poly; 
    int     i, j; 

    poly = 0xEDB88320L; 
    for (i=0; i<256; i++) { 
        crc = i; 
        for (j=8; j>0; j--) { 
            if (crc&1) { 
                crc = (crc >> 1) ^ poly; 
            } else { 
                crc >>= 1; 
            } 
        } 
        crcTable[i] = crc; 
    } 
}

I did not like the file pointer part so I switched it to command line arguments. I think it is easier to test. I am not very good with c. One trick though it needs to be named with ".C" and not ".c" to be a C++ program.

kduke · Post by **kduke** » Thu Jan 25, 2007 8:48 pm

The biggest issue was the values did not match between the DEV box and the TEST box. I also could not get it to match the Server values.

Code: Select all

PX
KIM DUKE
Peek_6,0: Exampl:-6172057588758100840
Peek_6,0: Exampl:5367974083337241752

Server
KIM DUKE
Peek_6,0: Exampl:-1725671448
Peek_6,0: Exampl:-1515929771

Command line
KIM
crc32 = aa587252 hex
crc32 = -1437044142 long
DUKE
crc32 = 4a7edffc hex
crc32 = 1249828860 long

Command line TEST
crc32 = aa587252 hex
crc32 = -1437044142 long

crc32 = 4a7edffc hex
crc32 = 1249828860 long

PX on TEST
Peek_6,0: Exampl:-6172057588725197672
Peek_6,0: Exampl:5367974083370144920

PX on TEST after binary copy of object code
Peek_6,0: Exampl:-6172057588725177192
Peek_6,0: Exampl:5367974083370165400

Server on TEST
Peek_6,0: Exampl:-1725671448
Peek_6,0: Exampl:-1515929771

I thought is I copied the object code with FTP instead of recompiling it then it would match but it did not match.

ray.wurlod · Post by **ray.wurlod** » Thu Jan 25, 2007 9:27 pm

If you are planning to use CRC32 to detect change, don't.

Parallel jobs have three different change detection stages - the one you choose depends on what format you require their output to be. The stages are Difference, Compare and Change Capture.

None of these risks the false positive that CRC32 can generate, particularly with large volumes of data.

DSguru2B · Post by **DSguru2B** » Thu Jan 25, 2007 9:28 pm

Hmm. I didnt know that a px version of CRC32 was built. I have to take a look at this. Will give it a shot sometime.

umamahes · Post by **umamahes** » Thu Apr 19, 2007 4:52 pm

what are the arguments for this crc32 function.It needs to arguments

Thanks
Uma

kduke · Post by **kduke** » Thu Apr 19, 2007 6:53 pm

It has one argument the string to calculate the CRC32 on.

ray.wurlod · Post by **ray.wurlod** » Thu Apr 19, 2007 10:01 pm

And, as you can see from the code, it returns an int32 (no surprise there!).

umamahes · Post by **umamahes** » Fri Apr 20, 2007 10:45 am

When i compiled the routine it is working fine

/usr/vacpp/bin/xlC_r -+ -O -g -c Testcrc.C -o Testcrc.o
/usr/vacpp/bin/xlC_r -G -qmkshrobj=1000 Testcrc.o -o Testcrc.so
Target "all" is up to date.

I called the .o file in the routine and i compiled the job it is compiled successfully.

when i run the job i am getting the following error.

Transformer_10: Failed to load the library "V0S10_Testrt_Transformer_10.o". Either the directory containing the library file is not on the library search path, or the library was compiled on a system that is incompatible with this system. Could not load "V0S10_Testrt_Transformer_10": rtld: 0712-001 Symbol Testcrc__FPc was referenced
from module /etldata/ifs/IFS_DEV/RT_BP6240.O/V0S10_Testrt_Transformer_10.o(), but a runtime definition
of the symbol was not found..

Thanks
Uma

DSguru2B · Post by **DSguru2B** » Fri Apr 20, 2007 12:27 pm

Try compiling with +Z option.

kduke · Post by **kduke** » Fri Apr 20, 2007 3:21 pm

This routine needs to be consistent. That was my problem it was not consistent across DEV, TEST and PROD. It needs to generate the same number across machines for the same string. It relies on overflow which was not consistent. Beware.