Get original values after apply stump.match

[Dalla01]

Hello everyone,
Currently i'm work with Synthetic Aperture Radar (SAR) time series and these type of data generally returns the backscatter value (that's in dB and negative). I have lot of samples by the same target (crop filds) and tried to use stumpy.match to find common patterns to this targets.
So the question is, when I use the stumpy.match algorithm the data are normalized to z_norm, therefore when i plot in a chart the values are kind different by original values, cause the radar images pixel values that i study normally are between -25 dB and -4dB. So my question is is it possible to have original values again after match values by the query time series? I put an attachment to illustrate my issue.

Original time series (each line represents a sample in which I get pixel values):

Time series after use stumpy.match (to identify noisy samples and remove them)

So to identify noisy samples I need get back original values or know the pool of time series that not match with the query, is it possible to do it?
Ps. In this exemple the original pool of time series has 24 samples and after use stumpy.match I identified that has 18 matches with query. And it's just some initial tests i will improve the parameters.

Best regards.
André Garcia

[seanlaw]

@Dalla01 Apologies but I don't think I understand what you are trying to do. Are you able to provide a minimum-reproducible-code-example?

It looks like you have a query subsequence, Q_df, and you are comparing Q_df with each of the 24 other time series samples? Again, it's not clear to me what your inputs are and what your expected output(s) are. Can you please provide more information and I will try my best to help?

[Dalla01]

Hello Sean, thank you for your answer, and sorry for my mistakes in communication.
Yes, you catch the idea: "It looks like you have a query subsequence, Q_df, and you are comparing Q_df with each of the 24 other time series samples".
I put the code and a sample .csv to exemplify my issues. I hope that helps you to understand better.

csv file (only 6 samples "OBJECTIDs" to test)
TestSamples.csv

Follow the code: *Forgive me about the mistakes and redundancy in code, i'm new in coding and programing

import os
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import stumpy

df = pd.read_csv("myWayToFile/TestSamples.csv")

#Convert the VHt column to integers
df['VHt'] = pd.to_numeric(df['VHt'], errors='coerce')

#Drop rows with any NaN values in the 'VHt' column
df = df.dropna(subset=['VHt'])

#Unique object ID list (each OBJECTID is a time series sample)
unique_objectids = list(df['OBJECTID'].unique())

#list to catch the Dataframes by OBJECTID
output_list = []

#Get mean value grouped by date, to get a unique time series.
df_mean_grouped = df.groupby('Data').mean()
df_mean_grouped.reset_index(inplace=True)
df_mean_grouped['Data'] = df_mean_grouped['Data'].str.replace('-', '/')
df_mean_grouped['Data'] = pd.to_datetime(
    df_mean_grouped['Data'], format='%Y/%m/%d').dt.strftime('%Y/%m/%d')

for ids in unique_objectids:
    #Create a new DataFrame by filtering rows
    new_df = df.loc[df['OBJECTID'] == ids]
    new_df_grouped = new_df.groupby('Data').mean()
    new_df_grouped.reset_index(inplace=True)
    new_df_grouped['Data'] = new_df_grouped['Data'].str.replace('-', '/')
    new_df_grouped['Data'] = pd.to_datetime(
        new_df_grouped['Data'], format='%Y/%m/%d').dt.strftime('%Y/%m/%d')
    output_list.append(new_df_grouped)

#To build a time series with all samples like a sequence.
merged_df = pd.concat(output_list)

#Set query and timeseries to apply stumpy.match
query = df_mean_grouped['VHt']
ts = merged_df['VHt']

matches = stumpy.match(query, ts)

#Since MASS computes z-normalized Euclidean distances, we should z-normalize our subsequences before plotting
Q_z_norm = stumpy.core.z_norm(query.values)

#list to save matched ts z-nomalized
matched_ts = []

plt.suptitle(
    'Comparing The Query To All Matches (Default max_distance)', fontsize='18')
plt.xlabel('Time', fontsize='12')
plt.ylabel('polarization VH', fontsize='12')
for match_distance, match_idx in matches:
    match_z_norm = stumpy.core.z_norm(
        ts.values[match_idx:match_idx+len(query)])
    plt.plot(match_z_norm, lw=2)
    matched_ts.append(match_z_norm)
plt.plot(Q_z_norm, lw=4, color="black", label="Query Subsequence, Q_df")
plt.legend()
plt.show()

[seanlaw]

I think I understand better now. You have concatenated all of your time series together into one long time series and you are trying to figure out which of the samples have no match to the query. Is that right? If so, I think you should avoid concatenating and simply compare your query to each sample independently.

[Dalla01]

Thank you very much Sean, this works!

[seanlaw]

By the way, if you care about the amplitude when computing the distance, you can turn off z-normalization by setting normalize=False (i.e., this is unnormalized Euclidean distance)