Talend DI

tOracleSCD defaults start date to system date and time

Details

  • Type: New Feature New Feature
  • Status: Accepted Accepted
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: 3.1.1
  • Fix Version/s: None
  • Component/s: None
  • Labels:
  • Environment:

    Operating System: XP
    Platform: Windows

Description

This (and my previous issues) are related to information that I got from this forum entry: http://www.talendforge.org/forum/viewtopic.php?pid=26792#p26792

I am very concerned to find out that this component only allows you to default your start dates to the system date and time. This is a major issue from a data warehousing point of view, and I am actually quite surprised that you are doing this.

Using system dates as the start date causes data and analysis problems in the data warehouse, because data is created one day in the source system, but because of load times, batch windows, load frequency's etc, might only be loaded 1 or more days later into the warehouse.

This means that according to the date that you are using, a record might only appear to be active in the warehouse days, or even months later. This will lead to users not being able to find their data, data will be grouped in incorrect financial periods, the list of problems are endless.

Take this as an example:

You have a daily load, but because of your batch window, your load only starts at 22:00. Data captured on Monday during the day will only get loaded that night. Due to run times, etc, your loads might often extend beyond 23:59. The jobs that run past this point will therefore have a create date of the following day, making them fall into a different effective period. A user then looking for a particular record for that day, will not find what they are looking for.

What happens if this is on the end of the month, or quarter or year? Data could then fall into the next reporting month, etc which is a much more serious issue. What happens when the loads are delayed for a couple of days due to system outages, etc? Your data could come in days later.

All in all, I think TALEND is a great tool, but I must say that I feel like this is a major short fall in the product.

I really do hope that you can improve this, it would be such a pity to be forced to not use this component because you don't allow for such basic, required functionality...

Activity

There are no comments yet on this issue.

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated: