In this project, we investigate the performance of text classification methods on reddit posts from over 20 subreddits. We preprocess the data using natural language processing techniques such as stopword removal, TF-IDF weighting, χ2 test for feature selection. We used various classification algorithms and found that an Ensemble Classifier performed the best on this dataset. We also implemented Bernoulli Naive Bayes from scratch. Performances of all the classifiers and our implementation of Bernoulli NB are compared on the dataset. We achieved an accuracy of 61.02% on held-out validation set and 58.633% on Kaggle test set.
The repository contains 5 files:
- 1 Jupyter notebook file - PROJECT 2- FINAL.ipynb
- 2 Dataset files - reddit_train.csv and reddit_test.csv
- 1 ReadMe file - ReadMe.md
- 1 Project writeup - writeup.pdf
- Install the following libraries for Python (All the packages mentioned below can be installed using pip.
In Jupyter notebook, use: !pip install <package_name>):
- sklearn
- numpy
- pandas
- time
- re
- scipy
- itertools
- seaborn
- matplotlib
- nltk
- tqdm
- gensim
- pyLDAvis
- logging
- pprint
- wordcloud
- spacy
- Download all Jupyter notebook and Dataset files into one directory.
- Open Jupyter notebook into that directory.
- Select the required notebook (.ipynb file) and select "Run All" inside the jupyter notebook file.