Ask HN: Creation of training dataset with FeatureStore time travel

Hi, We are building a recommender system. I see 2 options for creating training dataset(joining with ground truth labels). I would like to get feedback from developers regarding usage of Feature Store in this regard.

Option 1:

1) We can log all the features from prediction API. UniqueResponseId, feature1, feature 2 We can pass UniqueResponseId in response and then it would be logged when customers take action to this response. UniqueResponseId, response1.

Then we can join these 2 data to form training dataset, UniqueResponseId, feature1, feature2, ..., action1 .

This approach seems like perfect as we are able to get exact features and their corresponding actions.

Option 2: We can have time travel in Feature Store. Time travel allows to get features at certain point of time. The approach would be: We log, responseId, time1 from prediction API. When customer takes action, we log responseId, action1

Then we do: getTime(responseId) returns time1 getFeatures(time1) ==> return feature 0, feature 1 join on responseId responseId, feature0, feature1, action1

This approach is complex and there are other disadvantages if features are changing rapidly as it is impossible to get exact features used during serving. The features can only be approximated in this option.

I am unsure if I should go with Option 2. Option1 seems like intuitive choice. I would like to learn from other people experience if there are other approaches or if I am missing anything here.