Is it possible to have staggered datapoints when training on streaming data?

[RaphaelChevallier]

Hello,

I'm pretty new to all of this ML and streaming ML techniques. I was wondering if it was possible with this library to support staggered datapoints. For example, my model takes in 3 different data points; price, quality, and product. However this data does not always come in together/ comes in at different times live streaming. Would it be possible for one iteration of data of the learning only has values for 2 of the 3 usual datapoints? and the third point is in the next iteration alone? Would I need to set the missing datapoints of each iteration as nulls? Would that still result in an accurate and functional model? Or should every datapoint be filled with a value for every iteration.

Thanks for the advice in advance.

[MaxHalford]

Hello there.

That's an interesting set of questions!

Would it be possible for one iteration of data of the learning only has values for 2 of the 3 usual datapoints? and the third point is in the next iteration alone?

That's not very clear to me. If I understand what you're saying, you're asking if a model can learn_one with a partial x. Models definitely can do that, but they will always assume each partial x is a separate observation. In other words, as an example, you can't do "partial gradient descent".

Would I need to set the missing datapoints of each iteration as nulls?

You can leave them missing, thus creating sparse vectors. Or you can use a method from the impute module.

I would say that if you have staggered data, it makes sense to want to merge these partial observations together before updating the model. Does that make sense?

I would say we can add some utilities to help you merge staggered data streams. But to do that I would need some more details on what your streams look like, ideally with an example. Indeed, it's not clear to me if this is something you should handle on your side, or if River should own this merging mechanism.

[RaphaelChevallier]

Thanks for the quick reply,

Here is my specific usecase that I am concerned about. I am monitoring live GPU and other computer parts availability and sales. I have APIs grabbing data values of quality, user reviews, price, amount available, etc going into the model for a specific model GPU/computer part by store. This is always changing and the API is not always sending data together in a uniform time. sometimes it updates all the columns at once. Sometimes for example a GPU model update arrives but only price and quality updates of it are sent through the api at the moment. With a few minutes or even hours later for that same gpu model at the same store the amount available and user reviews and price again are updated. By the time I wait for the next iteration of data the old iteration could be deprecated.

Would it still be smart to attach together these staggered training data points of the model through the gpu identifier. Once I have a new complete observation I then train the model with it? Even if a data point is a few minutes or even an hour old for that store?

[MaxHalford]

Thanks for the details.

I understand only a subset of the columns are updated. But I would fill in the other columns with the old values. So if only the price is updated, then build a new observation with all the old values and the new price. Makes sense?

[RaphaelChevallier]

That makes sense thanks