DSXchange

We have a job writing to a dataset.
In the next job while reading this dataset the read is happening partially with pre-defined limit of 100 records.
Job stops reading any further.
Stats shows that 100 records were read whereas dataset contains 522 records.

log shows a message stating
"Open file limit (default=100) is being overridden with a value of 1024"

Any pointer for what could be the problem?

Thanks

The log message doesn't affect the number of rows being read, just the number of concurrent files that the process has open.
Your error must be coming from elsewhere. What do you mean by "with pre-defined limit of 100 records"? Do you set a limit yourself somewhere? The number 100 is too round to be a function of the data size, it must be either a job limit or stage constraint from a transform stage. If you write a small job to read this dataset and dump it straight into a peek stage, are 100 rows or more processed?