RESIZE 32 to 64

[narasimha]

Hi all,

I am trying to resize a hash file from 32 to 64 BIT. The RESIZE command is not taking the path name of the hash file. When i just give the name of the hash file without the path, the original hash file disappears. Has anybody come across such this situation !
-I am creating a empty Hash file
-Using the SETFILE
-Problems with RESIZE as above

[narasimha]

In detail----
Problem : I have to write more than 2 Gb of data into a hash file
Solution: Use a 64 BIT Hash File
1.Creating a blank Hash File (32 BIT)
A folder gets created in the specified hash file pathname

2.Making a pointer entry into tht VOC by using the SETFILE Command
So that i can use the RESIZE command

3.RESIZE into 64 BIT. ( without specifying the path name, running the command in Administrator)
The folder which was created, cannot be found anymore!

Summary :
a) I am getting problems when i am trying to RESIZE the hash file given its pathname, but it works fine if you dont specify its pathname.
b) The original hash file folder which was created initially( which we have to resize) is not found anymore!!

Is it mandatory to specify the file pathname while doing the RESIZE though it has a reference in the VOC ?

[ArndW]

Narasimha,

how are you creating the file? If you do a CREATE.FILE and subsequent RESIZE 64BIT on that file there is no problem. The RESIZE likes file names and not paths. When you resize from a DYNAMIC file type to a static one, the directory that used to contain the DATA.30 and OVER.30 files is no longer needed and deleted, this is probably what you are seeing. Non-dynamic files are stored as just one filesystem object.

[ray.wurlod]

The resize command can be run from the operating system prompt, in which case it will accept a pathname.

Otherwise, for example when running from Administrator, you will need a VOC pointer. You can create one with SETFILE.

[narasimha]

Thanks to Craig for helping me offline!
I got the same solution from him.
Ray and ArndW I really appreciate your answer
You guys are all 2 good.
- I am creating a hash file by just validating the job.
My first problem was that the Resize command was not accepting the absolute pathname of the hash file(I got the the answer that)
My other problem was, after Resizing from a type 30 (Dynamic) to a type 18 (Static). I was looking for the folder which had the hash file name
I did not have a clue that it would get converted to a single file!!
I think my problem is solved now, now i have to try to load more than 2 GB into it.

[ray.wurlod]

Even better is to ask "do I REALLY need to load more than 2GB into it?".

Do you really need all those columns? If they're not used in a job (there is no line drawn from the column in a Transformer stage) you don't need that column and therefore don't need to load it into the hashed file.

Do you really need all those rows? If you have records that are no longer current, they will never be looked up, so aren't needed in the hashed file.

[narasimha]

I was loading 5 fields, later removed 1 which was not needed.
I need to load all the records as I am looking for the missing records in same tables in different environments.
The size was exceeding 2 GB as the number of records is around 42 Million. So i dont have any other option, but to use a 64 BIT Hash File.

Thanks

[amsh76]

Can you please share the complete solution with all of us ? It can help others..

Thanks.

[narasimha]

I had to write a large amount of data into a hash file (more than 2 GB)
But a Dynamic hash file (Type 30) has a limitation of 2 GB space.
So i had to go for a 64 bit hash file.
There can be more than one solution for doing this, but the one i followed was-
- Create a empty Hash file
Created a simple job for creating a Hash File (Dynamic Type 30). Just Validate the job, you dont need to run the job. An empty hash file gets created.
- Use SETFILE command to create a pointer entry into the VOC

Code: Select all

SETFILE /hash_file_path/hash_filename hash_filename OVERWRITING

- Use RESIZE comand to change the hash file to the required type and size
(You can use the HFC.exe to determine the File Type, Modulo, Separation)
Ex. to resize to a Type 18 Static hash file

Code: Select all

RESIZE hash_filename 18 3937501 1 64BIT

There is a check to check if the hash file is really resized to 64 BIT

Code: Select all

UVFIXFILE hash_filename VLEVEL 0 TRACE 1

It should give an error, else it is still 32 BIT

Another method could be to create a 64 bit directly from the O/S using

Code: Select all

mkdbfile /hash_file_path/hash_filename 18 3843773 1 64BIT

There are many experts who can tell you other ways

[ray.wurlod]

Correction

Code: Select all

mkdbfile /hash_file_path/hash_filename 18 3843773 1 -64bit

[somu_june]

Hi

I want to Resize dynamic 30 hash file to 64 BIT. The code is given for static 18 and Iam trying with the above example for Dynamic 30 hash file Resize . Please correct me if my code is wrong

Code: Select all

RESIZE hash_filename 30 1 1  -64BIT or RESIZE hash_filename  -64 . 

Which one is correct . Please correct me if they are wrong.


Thanks,
somaRaju.

[ray.wurlod]

Neither is correct. When RESIZE is run within the DataStage environment the 64BIT keyword does not take a leading "-" character. Also, if you're not changing any of the parameters, it's better to use "*", since - for example - the "modulo" argument is interpreted as a command to change the MINIMUM.MODULUS setting.

Code: Select all

RESIZE hashed_file * * * 64BIT USING /directory/with/enough/free_space

The USING phrase means that a second file system can be used by RESIZE, so that you have two I/O channels functioning instead of one (provided the other directory is on a different file system), so that the whole operation completes more quickly.