DSXchange

I have an ISD/Rest service that does a simple db2 lookup via a datastage job (job has ISD input/output stages and lookup with sparse db2 query). The job allows caller to perform lookup on customer name, it allows user to perform wildcard searches.

When the query result is a large I only receive a partial result set back at the web browser, when I make a subsequent call (even with different query parameter) I receive the second page of results from the first query. I've been searching the web/documentation and can't find answers to my questions:
1) Is the limit configurable, if so where, is it done by #rows or total bytes returned??
2) Since it appears server is holding the subsequent results (i.e. second page of results) what's the best way to handle this... is there something sent back in client's request that let's them know more data pending to be sent?

There is no formal buffering logic built into ISD. You would have to build that intelligence/paging yourself, so that it would understand subsequent calls.

ISD is effectively "stateless", and doesn't have the kind of "login, provide a handle, establish a session, etc." logic that a more formal application might.

It's do-able, but you would have to craft that sort of logic within DataStage, perhaps saving content somewhere and coming up with a ficticious sessionID for each incoming request, etc. etc.

It doesn't keep track of who/which each request came from, which is required for deep "session" tracking logic.

Ernie

The pagination described sounds like it is unwanted behavior. Perhaps it is a defect. I have not run into that myself.

I did run into large requests and results getting truncated back on version 8.7. We opened a PMR about it with Support. The workaround we used was to edit the properties of the ISD Input and Output stages. There you will find a Buffer Size setting under Options. The default size is 512 KB. We found in testing that our largest requests and responses required a size of 4096 KB. Support had described the buffer as not working as intended, and when it got full it was just truncating.

You might try disabling the ISD app, increasing those Buffer Size values, recompiling, and enabling the ISD app again. If you still see pagination then I would say to contact Support.

Also, if you want to help protect your ISD app and the size of the load on the network, then you can add your own logic into the job design. We did this on a few jobs that were similar in design to what you have described.

You can count the number of records in the result set and limit the output to the first N records, where N is either hard-coded or passed in as an ISD Input stage column. And if you want, you can include an indicator column in the ISD Output stage to tell the consumer whether or not they reached the limit.

Generally, if your consumers are searching on something and get a large result set, you want to ask what the likelihood is that they're going to pour through all the records. Beyond some number, it may be better to suggest narrowing the search.

Sorry for the delay in responding, been nutty lately and didn't get back to this till yesterday evening.

I changed buffer size on both ISD input/output stages but it didn't seem to make a difference with the number of records returned. I'm going to limit the records returned in the DS job (like you suggested) and will probably open a ticket (also like you suggested).

Thank you for the responses, I really appreciate it!!