Investigate deployment-level models for mode inference based on trajectory characteristics

[shankari]

In e-mission, we currently use two methods for determining the mode automatically:

• user level model to infer the (rich) mode from the user's prior travel history
• ad-hoc set of rules to infer a simplified set of base modes (walk, bike, car, bus, train). We do this by using the motion activity to distinguish between non-motorized (walk, bike) and motorized (car, bus, train) modes and then a GIS integration to split the bus, train modes based on OSM bus routes.

However, as we have deployments with more complex modes, neither of these are sufficient. For example, in Laos, we want to be able to distinguish between cars and motorcycles, both of which are motorized modes without fixed routes. To this effect, we want to have the ability to build a deployment-specific model that uses sensor-level data to predict the rich modes that are relevant to this deployment.

The steps to this are fairly simple:

• Pick a featurization that depends on the sensor characteristics and not the user
• Create training and test sets.
  • Note that we should have at least one scenario in which the test set does not contain any user overlap with the training set
• Build an RF model
• Cross-validate

The initial featurization we should use is from

Zheng, Y., Li, Q., Chen, Y., Xie, X., Ma, W.-Y., 2008. Understanding mobility based on GPS data. In: Proceedings of the 10th international conference on Ubiquitous computing, Association for Computing Machinery, Association for Computing Machinery, Seoul, South Korea. pp. 312–321.
https://dl.acm.org/doi/10.1145/1409635.1409677

I implemented this featurization in https://github.com/e-mission/e-mission-server/tree/random-forest-mode-detection
and in https://github.com/e-mission/e-mission-server/blob/random-forest-mode-detection/emission/analysis/classification/inference/mode/pipeline.py in particular

Please put the initial results here and then we can decide how to proceed

[shankari]

@iantei @Abby-Wheelis for visibility

[iantei]

I went through the paper which talks about the features, and we have incorporated all of these features with addition of a few more consideration like in the https://github.com/e-mission/e-mission-server/blob/random-forest-mode-detection/emission/analysis/classification/inference/mode/pipeline.py:

1. both start and end close to bus stop
2. both start and end close to train station
3. both start and end close to airport
   alongside others.

I followed the instructions mentioned in https://github.com/e-mission/e-mission-docs/blob/master/docs/install/manual_install.md to run the server.

source setup/activate.sh
export DB_HOST=mongodb://db/openpath_prod_usaid_laos_ev
./e-mission-py.bash bin/<script_name.py>

while loading the docker-compose for the public-dashboard with exposed ports as 27017.

[iantei]

My first initial idea was to run the script with a different dataset i.e. Laos EV dataset, since it would have both cars and motorcycles data.

However, I came across the below error code while executing the script intake_multiprocess.py:

Elaborated call stack -

(emission) ashrest2-41625s:e-mission-server ashrest2$ export DB_HOST=mongodb://db/openpath_prod_usaid_laos_ev
(emission) ashrest2-41625s:e-mission-server ashrest2$ ./e-mission-py.bash bin/intake_multiprocess.py 4
storage not configured, falling back to sample, default configuration
URL not formatted, defaulting to "Stage_database"
Connecting to database URL localhost
analysis.debug.conf.json not configured, falling back to sample, default configuration
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
analysis.trip_model.conf.json not configured, falling back to sample, default configuration
expectations.conf.json not configured, falling back to sample, default configuration
ERROR:root:habitica not configured, game functions not supported
Traceback (most recent call last):
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/net/ext_service/habitica/proxy.py", line 22, in <module>
    key_file = open('conf/net/ext_service/habitica.json')
FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
storage not configured, falling back to sample, default configuration
URL not formatted, defaulting to "Stage_database"
Connecting to database URL localhost
storage not configured, falling back to sample, default configuration
URL not formatted, defaulting to "Stage_database"
Connecting to database URL localhost
storage not configured, falling back to sample, default configuration
URL not formatted, defaulting to "Stage_database"
Connecting to database URL localhost
storage not configured, falling back to sample, default configuration
URL not formatted, defaulting to "Stage_database"
Connecting to database URL localhost
analysis.debug.conf.json not configured, falling back to sample, default configuration
analysis.debug.conf.json not configured, falling back to sample, default configuration
analysis.debug.conf.json not configured, falling back to sample, default configuration
analysis.debug.conf.json not configured, falling back to sample, default configuration
google maps key not configured, falling back to nominatim
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
nominatim not configured either, place decoding must happen on the client
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

analysis.trip_model.conf.json not configured, falling back to sample, default configuration
analysis.trip_model.conf.json not configured, falling back to sample, default configuration
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
analysis.trip_model.conf.json not configured, falling back to sample, default configuration
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
analysis.trip_model.conf.json not configured, falling back to sample, default configuration
expectations.conf.json not configured, falling back to sample, default configuration
expectations.conf.json not configured, falling back to sample, default configuration
expectations.conf.json not configured, falling back to sample, default configuration
expectations.conf.json not configured, falling back to sample, default configuration
ERROR:root:habitica not configured, game functions not supported
Traceback (most recent call last):
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/net/ext_service/habitica/proxy.py", line 22, in <module>
    key_file = open('conf/net/ext_service/habitica.json')
FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
ERROR:root:habitica not configured, game functions not supported
Traceback (most recent call last):
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/net/ext_service/habitica/proxy.py", line 22, in <module>
    key_file = open('conf/net/ext_service/habitica.json')
FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
ERROR:root:habitica not configured, game functions not supported
Traceback (most recent call last):
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/net/ext_service/habitica/proxy.py", line 22, in <module>
    key_file = open('conf/net/ext_service/habitica.json')
FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
ERROR:root:habitica not configured, game functions not supported
Traceback (most recent call last):
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/net/ext_service/habitica/proxy.py", line 22, in <module>
    key_file = open('conf/net/ext_service/habitica.json')
FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
(emission) ashrest2-41625s:e-mission-server ashrest2$ 

It complains about habitica.json not being available.

FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
ERROR:root:habitica not configured, game functions not supported

Also, am not sure if it's taking up the right Laos database instead of Stage_database.
We have 'habitica.json.sample' in the directory inside the repo.

Now, in the following file -
https://github.com/e-mission/e-mission-server/blob/random-forest-mode-detection/emission/analysis/classification/inference/mode/pipeline.py
Inside main function,

  modeInferPipeline = ModeInferencePipeline()
  modeInferPipeline.runPipeline()

There is no runPipeline() function in the class ModeInferencePipeline().

@shankari Could you please help me understand the following:

1. Am I loading the right MongoDB database correctly?
2. Am I using the right script to run the pipeline for inference for a new dataset?

It seems the above pipeline uses the mode being trained in
https://github.com/e-mission/e-mission-server/blob/52adee205f686d87e167bd4b1d166098938870c6/emission/analysis/classification/inference/mode/seed/pipeline.py

which needs to incorporate the split of training/test data and cross validation aspects.

Can I build up the above requirements based in the above pipeline code, once I am able to run the pipeline with Laos EV dataset?

[iantei]

There needs to be a slight modification in the approach while loading MongoDB database correctly -
After loading the mongo dataset into the public dashboard, and running the container for public dashboard.

Since the mongoDB is exposed in the port 27017, we need to assign it to localhost:27017 instead of db.

source setup/activate.sh
export DB_HOST=mongodb://localhost:27017/openpath_prod_usaid_laos_ev
./e-mission-py.bash bin/<script_name.py>

Running with this change, it is loading the right database i.e. openpath_prod_usaid_laos_ev. [Tested this changes with the latest - Server code; there needs to be additional changes in the existing RF branch, likely introducing db.conf file which is missing]

[iantei]

Can I build up the above requirements based in the above pipeline code, once I am able to run the pipeline with Laos EV dataset?

This is an old branch, which doesn't have the updates in the latest master branch.

I am inclining towards the idea of - understanding the workflow in which the pipeline of the RF model is being trained, model is saved and run with the new dataset & porting these changes to the master and working in a new PR altogether.
Reason: There are likely changes which needs to be made, which are apparently fixed in the Master branch, and the whole process of porting this changes are everything is finalized seems redundant.

[iantei]

We are not using habitica.json in the project anymore, so the error/ warning can be ignored. Or any code which specifies the usage of it can be analysed and removed.

[shankari]

I am inclining towards the idea of - understanding the workflow in which the pipeline of the RF model is being trained, model is saved and run with the new dataset & porting these changes to the master and working in a new PR altogether.

I am not sure where model training and evaluation is planned in here. I do not anticipate significant changes for porting - the data model has been unchanged for the past 5 years.

I would encourage you to build and evaluate the model before porting so we can the impactful results quickly and don't spin our wheels on porting something we won't use.

[iantei]

I am not sure where model training and evaluation is planned in here.

Yes, the model training is not done in https://github.com/e-mission/e-mission-server/blob/random-forest-mode-detection/emission/analysis/classification/inference/mode/pipeline.py

Rather it just loads the model being saved here -
https://github.com/e-mission/e-mission-server/blob/random-forest-mode-detection/emission/analysis/classification/inference/mode/seed/pipeline.py

where training of the model is being done:

  def buildModelStep(self):
    from sklearn import ensemble
    forestClf = ensemble.RandomForestClassifier()
    model = forestClf.fit(self.selFeatureMatrix, self.cleanedResultVector)
    return model

I couldn't find a place where the model evaluation is being done. But it does refer to 70% accuracy in one of the comment in mode/pipeline.py

# we documented to have ~ 70% accuracy in the 2014 e-mission paper.

[shankari]

Yes, the model training is not done in https://github.com/e-mission/e-mission-server/blob/random-forest-mode-detection/emission/analysis/classification/inference/mode/pipeline.py

I know that we don't do model training in the code. My comment was more around the model training step in your plan. Can you write out your proposed plan of action in bullet points (e..g 1. I will do this first, 2. I will do this next, ...)

I couldn't find a place where the model evaluation is being done. But it does refer to 70% accuracy in one of the comment in mode/pipeline.py

We generally don't do model evaluation in a production system (!!) that is done offline. The 70% accuracy is on the ad-hoc dataset that I collected as part of a class project - in 2014, I had just started the PhD program. The model accuracy will depend on the input dataset, which will depend on the deployment.

You will need to evaluate this on a few deployments, starting with Laos.

[iantei]

Here's my proposed plan of actions (on a high level):

Work on the random-forest-mode-detection branch.

1. Understand the way in which I can run the model/pipeline.py from the pre-existing scripts. Likely, intake_multiprocess.py.
2. Write an offline model evaluation script and run the existing model with the Loas dataset.
3. Document the evaluation results.
4. Discuss about the evaluation results, and proceed if it would be necessary to re-train the model.
5. After the model is deemed seems right, port these changes to the Master branch.

Please let me know about your thoughts on this.

Note: I am trying to figure out the issue with db.conf file (config file issues), referring to the initial PR we had some discussion about in op-admin repository.

[iantei]

Probably not an elegant solution to the issue, but I updated the db.conf.sample to the following:

{
    "timeseries": {
        "url": "mongodb://localhost:27017/openpath_prod_usaid_laos_ev",
        "result_limit": 250000
    }
}

This change helped me use the right mongo db. I am not looking for any clean fix here, since there is no issue with updated Master branch, where we can `export DB_HOST= and works as expected.

[iantei]

What happens when we run the intake_multiprocess.py script, after configuring the changes to db.conf.sample as mentioned above, as the following:
(emission) $ ./e-mission-py.bash bin/intake_multiprocess.py 4

Documenting the steps such that it helps me recall things better and easily later too.

import emission.pipeline.scheduler as eps
import emission.pipeline.intake_stage as epi
import emission.pipeline.classification.inference.mode.pipeline as eacimp

1.
eps.dispatch(split_lists, args.skip_if_no_new_data)
    dispatch():
          p = ctx.Process(target=epi.run_intake_pipeline, args=(pid, uuid_list, skip_if_no_new_data))

2.
epi.run_intake_pipeline():
        with ect.Timer() as crt:
logging.info("*" * 10 + "UUID %s: inferring transportation mode" % uuid + "*" * 10)
print(str(arrow.now()) + "*" * 10 + "UUID %s: inferring transportation mode" % uuid + "*" * 10)
      eacimp.predict_mode(uuid)

3.
 eacimp.predict_mode():
        mip = ModeInferencePipeline()
        mip.runPredictionPipeline(user_id, time_query)
              runPredictionPipeline(self, user_id, timerange):
                    self.loadModelStage()

4.
loadModelStage():
    import emission.analysis.classification.inference.mode.seed.pipeline as seedp
    self.model = seedp.ModeInferencePipelineMovesFormat.loadModel()

5.
SAVED_MODEL_FILENAME = 'seed_model.json'

loadModel():
   @staticmethod
  def loadModel():
    fd = open(SAVED_MODEL_FILENAME, "r")
    model_rep = fd.read()
    fd.close()
    return jpickle.loads(model_rep)

Apparently, we do not have seed_model.json.

This led to some interesting error logs in the terminal, but it well continued, and completed the execution of script without crash:

/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
2024-12-19T15:21:45.188388-07:00**********UUID <UUID_ID>: inferring transportation mode**********
Error while inferring modes, timestamp is unchanged
Traceback (most recent call last):
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/analysis/classification/inference/mode/pipeline.py", line 41, in predict_mode
    mip.runPredictionPipeline(user_id, time_query)
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/analysis/classification/inference/mode/pipeline.py", line 139, in runPredictionPipeline
    self.loadModelStage()
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/analysis/classification/inference/mode/pipeline.py", line 156, in loadModelStage
    self.model = seedp.ModeInferencePipelineMovesFormat.loadModel()
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/analysis/classification/inference/mode/seed/pipeline.py", line 93, in loadModel
    fd = open(SAVED_MODEL_FILENAME, "r")
FileNotFoundError: [Errno 2] No such file or directory: 'seed_model.json'
...
While getting section summary, section length = 0. This should never happen, but let's not crash if it does
While getting section summary, section length = 0. This should never happen, but let's not crash if it does

[iantei]

Well, I tried to run another script bin/analysis/create_static_model.py.

(emission) ashrest2-41625s:e-mission-server ashrest2$ ./e-mission-py.bash bin/analysis/create_static_model.py
storage not configured, falling back to sample, default configuration
Connecting to database URL mongodb://localhost:27017/openpath_prod_usaid_laos_ev
analysis.debug.conf.json not configured, falling back to sample, default configuration
DEBUG:root:START TRAINING DATA STEP
DEBUG:root:Section data set size = 0
DEBUG:root:Getting dataset size took 0.0019478797912597656
DEBUG:root:Querying confirmedSections 2024-12-19 16:53:19.226176
DEBUG:root:Querying confirmedSection took 0.0004892349243164062
DEBUG:root:Querying stage modes 2024-12-19 16:53:19.226729
DEBUG:root:Querying stage modes took 0.0005838871002197266
DEBUG:root:Section query with ground truth 2024-12-19 16:53:19.227361
DEBUG:root:Training set total size = 0
DEBUG:root:Getting section query with ground truth took 0.00034809112548828125
DEBUG:root:confirmedSectionCount = 0
INFO:root:initial loadTrainingDataStep DONE
DEBUG:root:finished loading current training set, now loading from backup!
DEBUG:root:START TRAINING DATA STEP
DEBUG:root:Section data set size = 0
DEBUG:root:Getting dataset size took 0.003607034683227539
DEBUG:root:Querying confirmedSections 2024-12-19 16:53:19.231980
DEBUG:root:Querying confirmedSection took 0.0003788471221923828
DEBUG:root:Querying stage modes 2024-12-19 16:53:19.232415
DEBUG:root:Querying stage modes took 0.0003960132598876953
DEBUG:root:Section query with ground truth 2024-12-19 16:53:19.232852
DEBUG:root:Training set total size = 0
DEBUG:root:Getting section query with ground truth took 0.0006999969482421875
INFO:root:loadTrainingDataStep DONE
DEBUG:root:Trying to find cluster locations for 0 trips
DEBUG:root:No points found in cluster input, nothing to fit..
DEBUG:root:Trying to find cluster locations for 0 trips
DEBUG:root:No points found in cluster input, nothing to fit..
DEBUG:root:Trying to find cluster locations for 0 trips
DEBUG:root:No points found in cluster input, nothing to fit..
INFO:root:generateBusAndTrainStopStep DONE
DEBUG:root:created data structures of size 0
INFO:root:generateFeatureMatrixAndResultVectorStep DONE
DEBUG:root:Stripped trips with mode: run 0, transport 0, mixed 0, unknown 0 unstripped 0
DEBUG:root:Stripping out distanceOutliers (array([], dtype=int64),), speedOutliers (array([], dtype=int64),), speedMeanOutliers (array([], dtype=int64),), speedVarianceOutliers (array([], dtype=int64),), maxSpeedOutliers (array([], dtype=int64),)
DEBUG:root:nonOutlierIndices.shape = 0
INFO:root:cleanDataStep DONE
DEBUG:root:generic features = [0, 1, 4, 5, 6, 7, 8]
DEBUG:root:advanced features = [10, 11, 12]
DEBUG:root:location features = [13, 14, 15, 16]
DEBUG:root:time features = [17, 18]
DEBUG:root:bus train features = [19, 20, 21]
INFO:root:selectFeatureIndicesStep DONE
Traceback (most recent call last):
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/bin/analysis/create_static_model.py", line 8, in <module>
    seed_pipeline.runPipeline()
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/analysis/classification/inference/mode/seed/pipeline.py", line 67, in runPipeline
    self.model = self.buildModelStep()
  File "/Users/ashrest2/NREL/2024_RF_Server/e-mission-server/emission/analysis/classification/inference/mode/seed/pipeline.py", line 328, in buildModelStep
    model = forestClf.fit(self.selFeatureMatrix, self.cleanedResultVector)
  File "/Users/ashrest2/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/sklearn/ensemble/_forest.py", line 345, in fit
    X, y = self._validate_data(
  File "/Users/ashrest2/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/sklearn/base.py", line 565, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/Users/ashrest2/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/sklearn/utils/validation.py", line 1106, in check_X_y
    X = check_array(
  File "/Users/ashrest2/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/sklearn/utils/validation.py", line 931, in check_array
    raise ValueError(
ValueError: Found array with 0 sample(s) (shape=(0, 13)) while a minimum of 1 is required by RandomForestClassifier.
(emission) ashrest2-41625s:e-mission-server ashrest2$

Well, the commit message states it is used to create and save model based on old-style data.
Accounting that we have 0 Training set total size, I see two possibility. One, the script is not loading the proper database to load the dataset.
Second, data structure has changed, and this script is outdated.