Load all Hashed file into memory

mavrick21 · Post by **mavrick21** » Thu Aug 25, 2011 5:50 pm

Hello,

We have enough memory to load our entire project.

1) Is there a way to load all Hashed files, present in the project directory, in to memory before we start our ETL run?

2) If yes then would it, by any chance, alter the way our jobs are behaving? We have few jobs that just read the hashed files, few which just write to the hashed files and few which read and write to the same hashed file in the same job.

3) Once our ETL run completes is there a way to write all the hashed file contents present in memory back to the disk?

Thanks for your help
-Mav

SURA · Post by **SURA** » Thu Aug 25, 2011 6:02 pm

I doubt .....No

DS User

mavrick21 · Post by **mavrick21** » Thu Aug 25, 2011 6:09 pm

Gurus,

I stumbled up "Using System Caching" in "Hash Stage Disk Caching" PDF document. When I tried "DAEMON.FILE.CACHE START 60000" I get "Disk Cache not enabled." error.

How do I enable Disk Cache?

Thanks,
Mav

mavrick21 · Post by **mavrick21** » Thu Aug 25, 2011 6:13 pm

I wrote:How do I enable Disk Cache?

Think I got it. It's the same document.

Let me give it a try and will update this topic.

mavrick21 · Post by **mavrick21** » Tue Aug 30, 2011 4:09 pm

Is there a limit on the number of Hashed files that can be loaded into memory?

I'm trying to load 258 Hashed files into memory and it looks like it's taking for ever.

Thanks

ray.wurlod · Post by **ray.wurlod** » Tue Aug 30, 2011 5:45 pm

There's no limit till you start running out of memory. However, every byte of memory you use for cached hashed files is marked memory that is not available for any other purpose. So the more you use, the less you have for other tasks. The whole thing's a trade-off - supply and demand.

ray.wurlod · Post by **ray.wurlod** » Wed Sep 07, 2011 5:19 pm

I've learned something today. There is a limit.

mavrick21 · Post by **mavrick21** » Wed Sep 07, 2011 5:28 pm

Ray,

Most of our hashed files size is between 256 MB and 1.5 GB. Is there a work-around to catalog a few/most of them?

Thanks
Mav

ray.wurlod · Post by **ray.wurlod** » Wed Sep 07, 2011 5:46 pm

Not many.

The shared cache limit is 512MB according to that documentation.

Caching via the Hashed File stage gives you 999MB as the limit.

chulett · Post by **chulett** » Wed Sep 07, 2011 10:55 pm

I'm really curious what the goal is here and I'm wondering if all this curious energy isn't being misdirected. I get the impression you may not understand the use of "shared cache"... from what I recall from my play time with it many years ago, it is a way to cache hashed files into memory that many jobs would typically have open simultaneously. That way, rather than having five jobs caching five separate copies of hashed file X into memory, they could all share one copy. It's not really meant to be a place where every single hashed file you might want to use over the course of a run could be "preloaded".

At least that's what I recall.

mavrick21 · Post by **mavrick21** » Thu Sep 08, 2011 11:13 am

Craig,

When I initially posted my question I wanted to load every single hashed file into memory but after I stumbled upon "System caching is not intended to be used if only a single stage is creating or reading the file." in the PDF I realized my approach is wrong.

Craig/Ray,

Few more questions:

Caching via the Hashed File stage gives you 999MB as the limit.

1) Is this private link caching? If so can the Hashed File be preloaded into memory?

2) Is 999MB limit for a single Hashed File stage or the entire project?

Thank you.

ray.wurlod · Post by **ray.wurlod** » Thu Sep 08, 2011 4:44 pm

1) Yes.
2) Entire project. Or maybe entire server. Not individual hashed file.

qt_ky · Post by **qt_ky** » Sun Sep 11, 2011 12:51 am

This is an interesting question. I'm waiting for my premium membership to kick in to read all the responses so far, so I don't really know what answer you have got in detail just yet.

You asked about cataloging hashed files in bulk. What do you mean by catalog in this case?

mavrick21 · Post by **mavrick21** » Tue Sep 20, 2011 10:28 pm

@Eric,

Sorry for my late response. By Catalog I presume loading a Hashed file into memory. Please read through "Disk Caching Guide" manual for more information. I'm still an amateur in this topic and hence I'm trying to get clarifications from DS gurus by asking a lot of (possibly dumb) questions

@Ray/Craig,

I've a job which looks up on a huge Hashed file (< 999 MB). This Hashed file is used only by the job hence I'm using Private Link Caching and also have increased the Read Cache Size to 999 MB in Administrator. Can this Hashed file be cataloged (preloaded into memory) even before the job starts? If so can you please tell me the steps?

Thank you

ray.wurlod · Post by **ray.wurlod** » Wed Sep 21, 2011 3:18 am

With private link caching there is no way available to pre-load the contents of the hashed file into memory.

DSXchange

Load all Hashed file into memory

Load all Hashed file into memory

Re: Load all Hashed file into memory