Limitations/Weaknesses of the Matrix Profile

[statistactics]

Hey everyone, given that this is a new method in time series mining, there has been nothing but praise for this method online.

Don't get me wrong, I love the matrix profile's versatility and efficiency. However, I think it's helpful to play the devil's advocate and discuss some limitations to perhaps find possible directions for improvement. After playing around with STUMPY and doing some reading, I concluded that the main limitations are:

1. Window sizes may not be robust for shorter time series - that is, different window sizes can lead to entirely different conclusions
2. Nature of Euclidean distance biases simple patterns (Addressed by the original authors with Annotation Vectors)
3. Distance metric is limited to Z-normalized Euclidean distance (I understand that this is for efficiency purposes)

Disclaimer: I have not read all 20+ research papers, but I have skimmed enough to have a decent grasp of the concept. Look forward to hearing everybody's thoughts!

[seanlaw]

@statistactics Thanks for initiating the discussion and welcome to the STUMPY community! Note that I am not affiliated with the work of the original authors but here are some comments/thoughts for everyone to consider:

Window sizes may not be robust for shorter time series - that is, different window sizes can lead to entirely different conclusions

At the end of the day, there is no silver bullet and every analysis approach will/must have limitations and tradeoffs. In general, I've found that as long as your window size is greater than or equal to the duration of your repeating event but no more than 2x the duration then you should be okay. Of course, it is certainly possible that your time series may contain motifs of varying lengths and this is an active area of ongoing research. It is certainly valuable to try multiple different window sizes and, prior to matrix profiles, this was simply infeasible as the computational cost was too high beyond toy examples.

Nature of Euclidean distance biases simple patterns (Addressed by the original authors with Annotation Vectors)

I'm not sure I fully understand your point so please feel free to elaborate. From my vantage point, "simple" is good and, if you don't know what you are looking for within your time series then "simple patterns" can be quite useful and can help orient where you should be looking first. Are you able to share a more concrete example?

Distance metric is limited to Z-normalized Euclidean distance (I understand that this is for efficiency purposes)

Indeed, beyond the simplified O(n) space complexity, computing all of the pairwise z-norm Euclidean distances efficiently is the fundamental breakthrough. The significant speed increase(s) would not be possible/tenable without this research. Note that the work has been extended to non-normalized Euclidean distance as well as generalized to p-norm as well, which is generally sufficient for a large percentage of use cases. Alas, matrix profiles is another tool in the tool belt. The difference is that it is much more usable now for larger time series beyond 1,000 datapoints.

[statistactics]

Hi @seanlaw, thanks a lot for your great work with the package and thorough response.

I'm not sure I fully understand your point so please feel free to elaborate. From my vantage point, "simple" is good and, if you don't know what you are looking for within your time series then "simple patterns" can be quite useful and can help orient where you should be looking first. Are you able to share a more concrete example?

The original researcher Prof Keogh spoke about it in this presentation (timestamped for convenience). I think the limitation described was that a vanilla matrix profile would bias uninformative patterns as opposed to useful ones (like in the Finger Flexion example in the video and STUMPY docs).

If you could share, I would love to hear what you believe the limitations of this tool are.

[JaKasb]

2. Nature of Euclidean distance biases simple patterns (Addressed by the original authors with Annotation Vectors)

I also observed this effect.
I smooth the data to reduce noise before motif-search. (Low Pass Filter or Total Variation Denoising)

Euclidean Distance is sensetive to pattern-complexity aka jiggly-ness.
This was adressed in the paper CID:
https://link.springer.com/article/10.1007/s10618-013-0312-3

Furthermore there is a paper that compensates the effect of random noise by estimating the random-noise:
https://biblio.ugent.be/publication/8605188/file/8605190
https://biblio.ugent.be/publication/8644995/file/8644997.pdf

A repeating cause of errors are constant subsequences due to the division by standard-deviation.
The trick is to add a small noise term to the data to eliminate constant subsequences.