DSXchange

What does a timestamp look like after you extract it from SQL Server? Meaning, what format is it in? That will drive where you'll substring the year from.

Welcome to your own topic after splitting this from the post you jumped on the end of. Please edit your post and add the particulars for your environment.

Curious how this is an issue as there is no 'order' to how data is stored in a relational table.

What is your source, a database? If so, which one? Regardless, it should be as simple as getting the year from the timestamp, which could be just a substring of the first four characters.

There's always a reason... but there really isn't any way for us to know what it is. In my experience, the biggest culprit for project corruption is running out of disk space where the DataStage engine and/or Projects are installed. Any chance that happened? Are you actively monitoring that space? B...

If "one by one" means a separate run of the processing job for each file, that's what a Sequence job with the Start/End Loop stages will do. Any particular problem with doing them all at once?

I've been deleting these wayward posts up till now but took pity on this one for some reason, so split it out to here.

Same answer but it could be specific to a portion of the file, as in the file is corrupted. Or perhaps whatever you are doing to create the smaller file adds the appropriate delimiter. There's no way for us to know. We can't see your file and you haven't shared any of the settings you are using, so ...

h4harry1 wrote:Any other suggestion plz. I m not allowed to change source file at all.
Thanks

Whomever is producing the file needs to "change" it. What you are receiving right now is invalid.

That by itself won't work as the list needs to be delimited. Typically you would use a User Variables Activity stage to generate the list and your O/S should have an "m" option (off the top of my head) to return your list of filenames comma delimited.

We have the natural key from the fact data. In many cases that's all we have, making the stub dimension records extremely... stubby.

What have you tried? That's exactly what the Start/End Loop stages are for, all you need to do is build them a delimited list of the filenames first.

All we can tell from that is the record delimiter you used is wrong, as in doesn't match what's in the source file.

Yes, the dataset will become unusable. Deleting individual segments makes no sense. Your requirement should read: "if you are sure that the dataset PX_BEA.ds will never be referenced you can go ahead and delete the dataset and the corresponding PX_BEA.ds file" That latter would be automati...

My current (non-DataStage) project is taking a "pre-load" approach as well. After all of the dimensions have been updated from their sources, the incoming daily fact data is thrown up against each dimension and we create "stub" records for anything late arriving. They get a real ...