Maximum Record Limit For lookup stage
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 10
- Joined: Mon Feb 22, 2010 1:58 am
- Location: Chennai
Maximum Record Limit For lookup stage
Hi all,
I have been told not to use Lookup stage if the Record counts Exceed by 1 Million by my senior datastage developers.
Can any body give an idea about the approximate count that could be handled in LookUp stage with out getting crashed.
Please provide your Input on this.
Currently i am using JOIN stage for 5 million records.But i should handle reject and Lookup failure conditions as well .Which is very easy in Lookup Stage.
Thanks In Advance
I have been told not to use Lookup stage if the Record counts Exceed by 1 Million by my senior datastage developers.
Can any body give an idea about the approximate count that could be handled in LookUp stage with out getting crashed.
Please provide your Input on this.
Currently i am using JOIN stage for 5 million records.But i should handle reject and Lookup failure conditions as well .Which is very easy in Lookup Stage.
Thanks In Advance
Re: Maximum Record Limit For lookup stage
Lookup stage can handle any number of records. There is no limit to that.
A general threshold of 1 million records is set just to be on the safer side for performance reasons.
It mainly depends on the resource allocated to the server machine, memory in particular.
Talk to the admin and find out what is the memory availability for the server machine.
A general threshold of 1 million records is set just to be on the safer side for performance reasons.
It mainly depends on the resource allocated to the server machine, memory in particular.
Talk to the admin and find out what is the memory availability for the server machine.
-
- Premium Member
- Posts: 892
- Joined: Thu Oct 16, 2003 5:18 am
The upper bound of the memory usable by the lookup stage is also dependant upon the platform implementation, see the threads on AIX (i.e. this thread). Using a number of "1 million records" is a bit misleading -> are the records 5 bytes wide or 500?
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The "reject" link in JOIN Stage means, the records which don't have the common key will go to reject link. Its the mistake of IBM Software Labs' Tester, for join not having the reject link. From 8.2 onwards DataStage will have the reject link in Join Stage.ray.wurlod wrote:I disagree.
What does a "reject" mean in the context of a join?
When I join tables using SQL there are no rejects! ...
vdlrao
Actually, no - it wasn't a mistake. It was designed to provide the same exact functionality as SQL joins - none of which support a "reject" capability.
And at this point there is no such thing as 8.2, my understanding is that the next major release (currently in beta) may be called 8.5 when it ships
And at this point there is no such thing as 8.2, my understanding is that the next major release (currently in beta) may be called 8.5 when it ships
I am not sure which is the next version of 8.1. But it is going to have the reject link for JOIN Stage from the next version onwards. Well what about the Merge Stage in DataStage? Did it designed to provide the same exact functionality as SQL Merge?. Does it not support reject link?asorrell wrote:Actually, no - it wasn't a mistake. It was designed to provide the same exact functionality as SQL joins - none of which support a "reject" capability.
And at this point there is no such thing as 8.2, my understanding is that the next major release (currently in beta) may be called 8.5 when it ships
vdlrao
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
Merge statement does exists in Oracle and to capture rejects it uses when not matched then clause. I think it was introduced in Oracle 9i version. However i don't know whether they took the idea from ETL tools ot ETL tools tried to mimic the Merge statement of oracle.
Well, unless you know the exact timeline of first use of it in SQL or ETL tool, it seems like the question of Egg and Chicken.
Well, unless you know the exact timeline of first use of it in SQL or ETL tool, it seems like the question of Egg and Chicken.
Priyadarshi Kunal
Genius may have its limitations, but stupidity is not thus handicapped.
Genius may have its limitations, but stupidity is not thus handicapped.