Data processing and analysis

[mayacakmak]

We can move discussion about data to this issue.

One initial TODO:

• Write a script that converts (probably a subset of) the data into a CSV (spreadsheet compatible) format so we can more easily browse, take notes on, code, etc. open ended answers

[kavidey]

This script converts the json data into two csv files. One contains cycle data, and the other contains questionnaire responses. The original json file, along with the two csv files are uploaded to this google drive folder: https://drive.google.com/drive/u/0/folders/1R89qRGApJjWt0EQz45dtSLZaQNc5mKGz
The two output csv files are also in the data-processing folder

The cycle file only contains general data about each cycle (cycle length, target position/rotation/threshXY/threshXY, user id, interface type, etc.) It does not have data about any individual actions that the user took.

The only included cycles were cycles completed on the testing page and by users whos, user ID we logged as having completed this round of the study.

[mayacakmak]

Excellent! @csemecu can you give it a try?

[csemecu]

Yes! I will take a look at the data and start running some preliminary analyses with SPSS. I'll update you both early next week. One thing I was asking Kavi is if these files are de-identified, I'm thinking so.

[kavidey]

The results of the second batch are done processing. I added new files to the google drive folder. They are a combination of both batches (let me know if we want separate ones).

[mayacakmak]

Started combining/summarizing data in this spreadsheet for quick checks and visualization:
https://docs.google.com/spreadsheets/d/1snU2Rs7UUd3au_3eSIWlyoQcLhFxicjWH2Oe_wAJIBo/edit?usp=sharing
(in the same folder as the raw data)

[mayacakmak]

I've cleaned up the data in the spreadsheet, only including participants with a unique ID that has exactly 36 cycles associated--that is 156 out of 192 entries. @kaviMD we might want to remove those who do not have 36 cycles in the data processing pipeline already, but the spreadsheet can now handle mixed data to be pasted in. There were a few other small corruptions in the data, like the questionnaire answers being mis-ordered (probably due to some parts missing) but those were all eliminated anyways due to the number of cycles corruption.

Based on the distribution of the valid data I updated the 'todo' numbers in the database to run another batch (trying to get to 30 per condition but will likely have a drop with similar issues). I hope I got the ordering of the interfaces correct :) Working on getting the previous workers qualified so we can exclude them, will then start the next data collection.

[csemecu]

Yesterday I sent out an email with the results I got from SPSS. I used 166 participants. The ones that had more than 36 cycles I only used the first 36 and discarded the rest. But I can run everything again with those 156 that you mention. I added the exported results as an htm file in the drive folder

There are four significant groupings of the control types: target and targetdrag show no statistical difference between them.
I also plotted each control types with multiple bars to show click and press/release. The interface that benefits the most is the panel one so maybe that is something to discuss if we want to keep that interface as one of the accessible ones to be implemented with that transition.

[mayacakmak]

@csemecu Adding stats analysis discussion to this thread also--from Maru:

There are four significant groupings of the control types: target and targetdrag show no statistical difference between them.
I also plotted each control types with multiple bars to show click and press/release. The interface that benefits the most is the panel one so maybe that is something to discuss if we want to keep that interface as one of the accessible ones to be implemented with that transition.

(Note: the dependent variable/measurement shown here is cycleLength)

@csemecu Let's make sure you have the full/clean data before running the analysis again.

[mayacakmak]

@csemecu Messages crossed :)

Yes, it is unclear which 36 is the right one to include so let's just remove those participants. Also let's wait for the next batch.

[mayacakmak]

One question about analysis: We have 36 measurements from each participant but the data is pooled together. This analysis:

.. looks the same as it would look if each data point came from a different participant, instead of 36 data points from 166 participants. Shouldn't participant ID be a factor in the analysis?

[mayacakmak]

Initial look at questionnaire data is in the 'Interface comparisons' tab of the spreadsheet.

Replicated the scatterplots for Time -vs- dist_xy, dist_theta, thresh_xy, thresh_theta, with trendlines, showing some interesting trends, in the 'Target comparisons' tab of the spreadsheet.

@kaviMD I'm still thinking about your question about which interfaces to move on with for the SE3 study. I'm not sure a clear answer is in the results. I'm trying to think about the narrative and what reasoning would make most sense.

[mayacakmak]

@kaviMD I'm starting to look into the next batch of data. How long does get_firebase_data.py script usually take? Like you mentioned before, the web interface didn't work for exporting. One todo for future (perhaps already for SE3 study) is to save detailed session data (for recreating all actions) in a different branch of the database so it can be downloaded separately (at its own pace) if needed, but we get quicker access to measurement data.

The other question I have is, in what order should I then run the different scripts under data_processing/ once I have the data downloaded?

[mayacakmak]

Oh, looks like I'm done downloading, so the answer to the first question (for the latest data) is about 5 minutes.

[mayacakmak]

So far process_data.py and calculate_session_lengths.py seem outdated, because they require manually entering uids and have fewer uids than we had in the data already.

Tried json_to_csv.py but that requires a different json than the one get_firebase_data.py downloads (one that also has state branch of the database. I'll try this again after changing get_firebase_data.py to download the whole thing.

[mayacakmak]

Actually I was not able to download root of the database, so just went ahead and modified json_to_csv.py This part of the repo needs a bit more documentation ;)

@kaviMD the question labels in the questionnaire data seem to be mixed up in the latest data--I was already a bit suspicious the earlier data, feeling unsure which question id (e.g. section-0-question-1) corresponds to which actual question. Can you take a look if that's an issue in the collected data or in the data processing? We cannot tell apart the NASA TLX rating answers and statement agreement answers from one another so we really need to know which is which. I tagged you in the spreadsheets; will commit latest jsons and code.