I am experiencing the error "The record is too big to fit in a block" in one of my job which uses MQ connector to read messages from a queue.
Scenario
- The MQ queue has around 15000 messages with the size of each message is some kb. Definelty less than a 0.5MB
- The Datastage connects to the MQ in client mode and reads the messages
- The message quantity is set to -1 so as to read all messages from the queue
I understand that the mq connector would read message by messgae and not line by line, in this case, does the connector waits until it reads all the messages in the queue before propagating it to the next stage ? or does it propagate the messages to next stage as it reads ?
Because, our transport block by default was set to 128kb and Datastage was saying it required around 750kb initially. Since it was below the recommened level of 1MB, I increased it to 900kb, but during the next run, the job again failed and said it needs 1.2MB, Though its a little above the recommended level, I could increase it to 1.2MB, but would like to understand if 1.2MB would be ok, or would it need more later ..
During functional and unit testing, we were testing with less than 10 messages in the queue, and we did not face any issues at those time.
Any pointers would be of great help.
Thanks
Sen