req.json

[
  {
    "type": "md",
    "md": "# Exploratory Analysis for Loan Default Prediction\n\n## Project Introduction\n\nThis notebook details the exploratory data analysis (EDA) performed on the LendingClub loan dataset. The core aim is to dissect the data, uncover patterns related to loan defaults, and engineer impactful features. This analysis forms the foundation for building a robust binary classification model to predict loan outcomes ('Fully Paid' vs. 'Charged Off').\n\n## Key Analytical Goals\n\n- Assess the overall structure and quality of the dataset.\n- Analyze the balance of the target variable (loan status).\n- Discover features that are highly predictive of loan defaults.\n- Develop a strategy for handling missing data and potential outliers.\n- Engineer new features from existing data to enhance predictive power.\n- Prepare a clean, processed dataset ready for machine learning."
  },
  {
    "type": "code",
    "code": "# --- Core Libraries ---\nimport pandas as pd\nimport numpy as np\n\n# --- Visualization Libraries ---\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# --- Utilities ---\nfrom datetime import datetime\nimport warnings\nimport json\nimport os\n\n# --- Environment Configuration ---\n# Set a modern plot style and color palette\nplt.style.use('seaborn-v0_8-whitegrid')\nsns.set_palette('viridis')\n\n# Configure pandas display options for better readability\npd.set_option('display.max_columns', 100)\npd.set_option('display.float_format', lambda x: f'{x:.3f}')\n\n# Suppress non-critical warnings\nwarnings.filterwarnings('ignore')\n\nprint(\"Setup Complete: All libraries are imported and configured.\")",
    "output": "Setup Complete: All libraries are imported and configured.",
    "execution_count": 1
  },
  {
    "type": "md",
    "md": "## 1. Data Ingestion and Initial Assessment"
  },
  {
    "type": "code",
    "code": "# Load the dataset from the specified CSV file\nfile_path = './data/lending_club_loan_two.csv'\ntry:\n    loan_data = pd.read_csv(file_path)\n    print(f\"Successfully loaded data from {file_path}\")\nexcept FileNotFoundError:\n    print(f\"Error: Data file not found at {file_path}\")\n    loan_data = pd.DataFrame() # Create an empty DataFrame to prevent errors\n\n# Display a high-level summary of the loaded data\nif not loan_data.empty:\n    print(f\"\\nDataset Dimensions: {loan_data.shape[0]} rows and {loan_data.shape[1]} columns\")\n    mem_usage = loan_data.memory_usage(deep=True).sum() / (1024**2)\n    print(f\"Approximate Memory Usage: {mem_usage:.2f} MB\")\n    print(\"\\n--- Data Snapshot ---\")\n    display(loan_data.head())",
    "output": "Successfully loaded data from ./data/lending_club_loan_two.csv\n\nDataset Dimensions: 396030 rows and 27 columns\nApproximate Memory Usage: 376.43 MB\n\n--- Data Snapshot ---\n   loan_amnt        term  int_rate  installment grade sub_grade  \\\n0  10000.000   36 months    11.440      329.480     B        B4   \n1   8000.000   36 months    11.990      265.680     B        B5   \n2  15600.000   36 months    10.490      506.970     B        B3   \n3   7200.000   36 months     6.490      220.650     A        A2   \n4  24375.000   60 months    17.270      609.330     C        C5   \n\n                 emp_title emp_length home_ownership  annual_inc  \\\n0                Marketing  10+ years           RENT  117000.000   \n1          Credit analyst     4 years       MORTGAGE   65000.000   \n2             Statistician   < 1 year           RENT   43057.000   \n3          Client Advocate    6 years           RENT   54000.000   \n4  Destiny Management Inc.    9 years       MORTGAGE   55000.000   \n\n  verification_status   issue_d  loan_status             purpose  \\\n0        Not Verified  Jan-2015   Fully Paid            vacation   \n1        Not Verified  Jan-2015   Fully Paid  debt_consolidation   \n2     Source Verified  Jan-2015   Fully Paid         credit_card   \n3        Not Verified  Nov-2014   Fully Paid         credit_card   \n4            Verified  Apr-2013  Charged Off         credit_card   \n\n                     title    dti earliest_cr_line  open_acc  pub_rec  \\\n0                 Vacation 26.240         Jun-1990    16.000    0.000   \n1       Debt consolidation 22.050         Jul-2004    17.000    0.000   \n2  Credit card refinancing 12.790         Aug-2007    13.000    0.000   \n3  Credit card refinancing  2.600         Sep-2006     6.000    0.000   \n4    Credit Card Refinance 33.950         Mar-1999    13.000    0.000   \n\n   revol_bal  revol_util  total_acc initial_list_status application_type  \\\n0  36369.000      41.800     25.000                   w       INDIVIDUAL   \n1  20131.000      53.300     27.000                   f       INDIVIDUAL   \n2  11987.000      92.200     26.000                   f       INDIVIDUAL   \n3   5472.000      21.500     13.000                   f       INDIVIDUAL   \n4  24584.000      69.800     43.000                   f       INDIVIDUAL   \n\n   mort_acc  pub_rec_bankruptcies  \\\n0     0.000                 0.000   \n1     3.000                 0.000   \n2     0.000                 0.000   \n3     0.000                 0.000   \n4     1.000                 0.000   \n\n                                             address  \n0     0174 Michelle Gateway\\r\\nMendozaberg, OK 22690  \n1  1076 Carney Fort Apt. 347\\r\\nLoganmouth, SD 05113  \n2  87025 Mark Dale Apt. 269\\r\\nNew Sabrina, WV 05113  \n3            823 Reid Ford\\r\\nDelacruzside, MA 00813  \n4             679 Luna Roads\\r\\nGreggshire, VA 11650",
    "execution_count": 2
  },
  {
    "type": "code",
    "code": "# Examine the data types and null counts for each column\nprint(\"--- Column Data Types and Non-Null Counts ---\")\nloan_data.info()\n\n# Summarize the count of different data types present\nprint(\"\\n--- Summary of Data Types ---\")\nprint(loan_data.dtypes.value_counts())\n\n# List all column names for reference\nprint(\"\\n--- Column Index ---\")\nprint(loan_data.columns.tolist())",
    "output": "--- Column Data Types and Non-Null Counts ---\n<class 'pandas.core.frame.DataFrame'>\nRangeIndex: 396030 entries, 0 to 396029\nData columns (total 27 columns):\n #   Column                Non-Null Count   Dtype  \n---  ------                --------------   -----  \n 0   loan_amnt             396030 non-null  float64\n 1   term                  396030 non-null  object \n 2   int_rate              396030 non-null  float64\n 3   installment           396030 non-null  float64\n 4   grade                 396030 non-null  object \n 5   sub_grade             396030 non-null  object \n 6   emp_title             373103 non-null  object \n 7   emp_length            377729 non-null  object \n 8   home_ownership        396030 non-null  object \n 9   annual_inc            396030 non-null  float64\n 10  verification_status   396030 non-null  object \n 11  issue_d               396030 non-null  object \n 12  loan_status           396030 non-null  object \n 13  purpose               396030 non-null  object \n 14  title                 394274 non-null  object \n 15  dti                   396030 non-null  float64\n 16  earliest_cr_line      396030 non-null  object \n 17  open_acc              396030 non-null  float64\n 18  pub_rec               396030 non-null  float64\n 19  revol_bal             396030 non-null  float64\n 20  revol_util            395754 non-null  float64\n 21  total_acc             396030 non-null  float64\n 22  initial_list_status   396030 non-null  object \n 23  application_type      396030 non-null  object \n 24  mort_acc              358235 non-null  float64\n 25  pub_rec_bankruptcies  395495 non-null  float64\n 26  address               396030 non-null  object \ndtypes: float64(12), object(15)\nmemory usage: 81.6+ MB\n\n--- Summary of Data Types ---\nobject     15\nfloat64    12\nName: count, dtype: int64\n\n--- Column Index ---\n['loan_amnt', 'term', 'int_rate', 'installment', 'grade', 'sub_grade', 'emp_title', 'emp_length', 'home_ownership', 'annual_inc', 'verification_status', 'issue_d', 'loan_status', 'purpose', 'title', 'dti', 'earliest_cr_line', 'open_acc', 'pub_rec', 'revol_bal', 'revol_util', 'total_acc', 'initial_list_status', 'application_type', 'mort_acc', 'pub_rec_bankruptcies', 'address']",
    "execution_count": 3
  },
  {
    "type": "md",
    "md": "## 2. Target Variable Transformation and Analysis\n\nOur primary goal is to predict the `loan_status`. We will convert this into a numerical binary format and examine its distribution to check for class imbalance."
  },
  {
    "type": "code",
    "code": "# Analyze the distribution of the original 'loan_status' column\nprint(\"--- Original Target Variable Distribution ---\")\nprint(loan_data['loan_status'].value_counts(normalize=True).mul(100).round(2).astype(str) + '%')\n\n# Engineer a binary target variable: 1 for 'Fully Paid', 0 for 'Charged Off'\nloan_data['is_fully_paid'] = loan_data['loan_status'].map({'Fully Paid': 1, 'Charged Off': 0})\n\n# Visualize the binary target distribution\nplt.figure(figsize=(14, 6))\n\n# Bar chart for counts\nplt.subplot(1, 2, 1)\nsns.countplot(x='is_fully_paid', data=loan_data, palette='mako')\nplt.title('Count of Loan Outcomes')\nplt.xlabel('Loan Outcome')\nplt.ylabel('Number of Loans')\nplt.xticks([0, 1], ['Charged Off', 'Fully Paid'])\n\n# Pie chart for proportions\nplt.subplot(1, 2, 2)\noutcome_proportions = loan_data['is_fully_paid'].value_counts()\nplt.pie(outcome_proportions, labels=['Fully Paid', 'Charged Off'], autopct='%1.2f%%', startangle=140, colors=sns.color_palette('mako', 2))\nplt.title('Proportion of Loan Outcomes')\n\nplt.suptitle('Analysis of Binary Target Variable', fontsize=16)\nplt.tight_layout(rect=[0, 0, 1, 0.95])\nplt.show()\n\n# Conclude with imbalance information\nimbalance_ratio = loan_data['is_fully_paid'].mean()\nprint(f\"The dataset is imbalanced, with {imbalance_ratio:.2%} of loans being fully repaid.\")",
    "output": "--- Original Target Variable Distribution ---\nloan_status\nFully Paid     80.39%\nCharged Off    19.61%\nName: proportion, dtype: object\n<Figure size 1400x600 with 2 Axes>The dataset is imbalanced, with 80.39% of loans being fully repaid.",
    "execution_count": 4
  },
  {
    "type": "md",
    "md": "## 3. Handling Data Completeness and Missing Values\n\nA critical step is to quantify missing data across all features to inform our cleaning and imputation strategy."
  },
  {
    "type": "code",
    "code": "# Calculate missing value percentages for all columns\nmissing_percent = (loan_data.isnull().sum() / len(loan_data)) * 100\n\n# Filter for columns that have at least one missing value\nmissing_summary = missing_percent[missing_percent > 0].sort_values(ascending=True)\n\nif missing_summary.empty:\n    print(\"No missing values found in the dataset.\")\nelse:\n    print(\"--- Features with Incomplete Data ---\")\n    print(missing_summary.to_frame('Missing Percentage'))\n\n    # Visualize the percentage of missing values\n    plt.figure(figsize=(12, 7))\n    missing_summary.plot(kind='barh', color=sns.color_palette('plasma', len(missing_summary)))\n    plt.title('Percentage of Missing Data by Feature')\n    plt.xlabel('Percentage Missing (%)')\n    plt.ylabel('Feature')\n    for index, value in enumerate(missing_summary):\n        plt.text(value, index, f' {value:.2f}%', va='center')\n    plt.tight_layout()\n    plt.show()\n\n    print(f\"\\nThere are {len(missing_summary)} features with missing data.\")",
    "output": "--- Features with Incomplete Data ---\n                      Missing Percentage\nrevol_util                         0.070\npub_rec_bankruptcies               0.135\ntitle                              0.443\nemp_length                         4.621\nemp_title                          5.789\nmort_acc                           9.543\n<Figure size 1200x700 with 1 Axes>\nThere are 6 features with missing data.",
    "execution_count": 5
  },
  {
    "type": "md",
    "md": "## 4. Numerical Feature Exploration\n\nHere, we'll investigate the properties of numerical features, their distributions, and how they correlate with the loan repayment status."
  },
  {
    "type": "code",
    "code": "# Isolate columns with numerical data types, excluding our target variable\nnumeric_cols = loan_data.select_dtypes(include=np.number).columns.drop('is_fully_paid')\nprint(f\"Found {len(numeric_cols)} numerical features for analysis.\")\n\n# Display descriptive statistics for these features\nprint(\"\\n--- Descriptive Statistics for Numerical Features ---\")\ndisplay(loan_data[numeric_cols].describe().transpose())\n\n# Calculate the correlation of each numerical feature with the target\ncorr_with_target = loan_data[numeric_cols].corrwith(loan_data['is_fully_paid']).sort_values()\n\nprint(\"\\n--- Correlation with Loan Repayment Status ---\")\nprint(corr_with_target)\n\n# Visualize the correlations for easier interpretation\nplt.figure(figsize=(10, 8))\nsns.barplot(x=corr_with_target.values, y=corr_with_target.index, orient='h', palette='crest')\nplt.title('Correlation of Numerical Features with Loan Repayment')\nplt.xlabel('Correlation Coefficient')\nplt.ylabel('Numerical Feature')\nplt.axvline(x=0, color='black', linestyle='--', linewidth=0.8)\nplt.tight_layout()\nplt.show()",
    "output": "Found 12 numerical features for analysis.\n\n--- Descriptive Statistics for Numerical Features ---\n                          count      mean       std     min       25%  \\\nloan_amnt            396030.000 14113.888  8357.441 500.000  8000.000   \nint_rate             396030.000    13.639     4.472   5.320    10.490   \ninstallment          396030.000   431.850   250.728  16.080   250.330   \nannual_inc           396030.000 74203.176 61637.621   0.000 45000.000   \ndti                  396030.000    17.380    18.019   0.000    11.280   \nopen_acc             396030.000    11.311     5.138   0.000     8.000   \npub_rec              396030.000     0.178     0.531   0.000     0.000   \nrevol_bal            396030.000 15844.540 20591.836   0.000  6025.000   \nrevol_util           395754.000    53.792    24.452   0.000    35.800   \ntotal_acc            396030.000    25.415    11.887   2.000    17.000   \nmort_acc             358235.000     1.814     2.148   0.000     0.000   \npub_rec_bankruptcies 395495.000     0.122     0.356   0.000     0.000   \n\n                           50%       75%         max  \nloan_amnt            12000.000 20000.000   40000.000  \nint_rate                13.330    16.490      30.990  \ninstallment            375.430   567.300    1533.810  \nannual_inc           64000.000 90000.000 8706582.000  \ndti                     16.910    22.980    9999.000  \nopen_acc                10.000    14.000      90.000  \npub_rec                  0.000     0.000      86.000  \nrevol_bal            11181.000 19620.000 1743266.000  \nrevol_util              54.800    72.900     892.300  \ntotal_acc               24.000    32.000     151.000  \nmort_acc                 1.000     3.000      34.000  \npub_rec_bankruptcies     0.000     0.000       8.000  \n--- Correlation with Loan Repayment Status ---\nint_rate               -0.248\nrevol_util             -0.082\ndti                    -0.062\nloan_amnt              -0.060\ninstallment            -0.041\nopen_acc               -0.028\npub_rec                -0.020\npub_rec_bankruptcies   -0.009\nrevol_bal               0.011\ntotal_acc               0.018\nannual_inc              0.053\nmort_acc                0.073\ndtype: float64\n<Figure size 1000x800 with 1 Axes>",
    "execution_count": 6
  },
  {
    "type": "code",
    "code": "# Analyze distributions of the most correlated numerical features\nimportant_numeric_features = ['int_rate', 'revol_util', 'dti', 'loan_amnt', 'mort_acc', 'annual_inc']\n\n# Plot histograms to see the distribution shape\nfig, axes = plt.subplots(2, 3, figsize=(20, 10))\nfor i, feature in enumerate(important_numeric_features):\n    ax = axes.flatten()[i]\n    sns.histplot(data=loan_data, x=feature, kde=True, ax=ax, bins=40, color=sns.color_palette('viridis', 6)[i])\n    ax.set_title(f'Distribution of {feature.replace(\"_\", \" \").title()}')\n    ax.set_xlabel('')\n    ax.set_ylabel('Frequency')\n\nplt.suptitle('Distribution of Key Numerical Features', fontsize=18)\nplt.tight_layout(rect=[0, 0, 1, 0.96])\nplt.show()\n\n# Create box plots to see how distributions differ by loan outcome\nfig, axes = plt.subplots(2, 3, figsize=(20, 12))\nfor i, feature in enumerate(important_numeric_features):\n    ax = axes.flatten()[i]\n    sns.boxplot(x='is_fully_paid', y=feature, data=loan_data, ax=ax, palette='mako')\n    ax.set_title(f'{feature.replace(\"_\", \" \").title()} vs. Loan Outcome')\n    ax.set_xlabel('Loan Outcome')\n    ax.set_xticklabels(['Charged Off', 'Fully Paid'])\n    ax.set_ylabel(feature.title())\n\nplt.suptitle('Numerical Features by Loan Repayment Status', fontsize=18)\nplt.tight_layout(rect=[0, 0, 1, 0.96])\nplt.show()",
    "output": "<Figure size 2000x1000 with 6 Axes><Figure size 2000x1200 with 6 Axes>",
    "execution_count": 7
  },
  {
    "type": "md",
    "md": "## 5. Categorical Feature Exploration\n\nLet's analyze the categorical variables to understand their relationship with loan repayment rates."
  },
  {
    "type": "code",
    "code": "# Identify all non-numeric (object) columns\ncategorical_cols = loan_data.select_dtypes(include=['object']).columns\n\nprint(f\"--- Analysis of {len(categorical_cols)} Categorical Features ---\")\nfor feature in categorical_cols:\n    print(f\"- Column '{feature}' has {loan_data[feature].nunique()} unique values.\")\n\n# Select a subset of key categorical features for visualization\nkey_cats_to_viz = ['grade', 'term', 'home_ownership', 'verification_status', 'purpose']\n\n# Plot repayment rates for each category within these features\nfig, axes = plt.subplots(2, 3, figsize=(20, 14))\naxes = axes.flatten()\n\nfor i, feature in enumerate(key_cats_to_viz):\n    ax = axes[i]\n    # Calculate repayment rate and sort for better visualization\n    repayment_summary = loan_data.groupby(feature)['is_fully_paid'].mean().sort_values(ascending=False)\n    \n    sns.barplot(x=repayment_summary.index, y=repayment_summary.values, ax=ax, palette='cubehelix')\n    ax.set_title(f'Repayment Rate by {feature.title()}')\n    ax.set_ylabel('Average Repayment Rate')\n    ax.set_xlabel(feature.title())\n    ax.tick_params(axis='x', rotation=45)\n    ax.axhline(y=loan_data['is_fully_paid'].mean(), color='red', linestyle='--', label=f'Overall Avg: {loan_data[\"is_fully_paid\"].mean():.2f}')\n    ax.legend()\n\n# Hide any unused subplots\nfor j in range(i + 1, len(axes)):\n    fig.delaxes(axes[j])\n\nplt.tight_layout()\nplt.show()",
    "output": "--- Analysis of 15 Categorical Features ---\n- Column 'term' has 2 unique values.\n- Column 'grade' has 7 unique values.\n- Column 'sub_grade' has 35 unique values.\n- Column 'emp_title' has 173105 unique values.\n- Column 'emp_length' has 11 unique values.\n- Column 'home_ownership' has 6 unique values.\n- Column 'verification_status' has 3 unique values.\n- Column 'issue_d' has 115 unique values.\n- Column 'loan_status' has 2 unique values.\n- Column 'purpose' has 14 unique values.\n- Column 'title' has 48816 unique values.\n- Column 'earliest_cr_line' has 684 unique values.\n- Column 'initial_list_status' has 2 unique values.\n- Column 'application_type' has 3 unique values.\n- Column 'address' has 393700 unique values.\n<Figure size 2000x1400 with 5 Axes>",
    "execution_count": 8
  },
  {
    "type": "md",
    "md": "## 6. Data Cleansing and Feature Creation\n\nIn this section, we will clean the dataset by addressing missing values and engineer new, potentially more predictive features."
  },
  {
    "type": "code",
    "code": "# Create a working copy of the dataframe for modifications\ndata_cleaned = loan_data.copy()\n\n# --- Step 1: Handle Missing Values ---\nprint(\"--- Stage 1: Addressing Missing Data ---\")\n\n# Impute 'revol_util' and 'mort_acc' with their respective medians\nfor col in ['revol_util', 'mort_acc']:\n    median_val = data_cleaned[col].median()\n    data_cleaned[col].fillna(median_val, inplace=True)\n    print(f\"Filled missing '{col}' values with median ({median_val:.2f}).\")\n\n# Impute 'pub_rec_bankruptcies' with 0, a reasonable assumption\ndata_cleaned['pub_rec_bankruptcies'].fillna(0, inplace=True)\nprint(\"Filled missing 'pub_rec_bankruptcies' with 0.\")\n\n# Drop columns that are irrelevant, redundant, or too noisy\nfeatures_to_drop = ['emp_title', 'title', 'address', 'issue_d', 'sub_grade', 'loan_status']\ndata_cleaned.drop(columns=features_to_drop, inplace=True)\nprint(f\"Dropped irrelevant/redundant columns: {features_to_drop}\")\n\n# --- Step 2: Feature Engineering ---\nprint(\"\\n--- Stage 2: Creating New Features ---\")\n\n# Convert 'term' to a numerical format\ndata_cleaned['term_numeric'] = data_cleaned['term'].apply(lambda t: int(t.split()[0]))\n\n# Engineer 'employment_duration' from 'emp_length'\nemp_length_map = {'10+ years': 10, '9 years': 9, '8 years': 8, '7 years': 7, '6 years': 6, '5 years': 5, '4 years': 4, '3 years': 3, '2 years': 2, '1 year': 1, '< 1 year': 0}\ndata_cleaned['employment_duration'] = data_cleaned['emp_length'].map(emp_length_map)\nmode_emp_length = data_cleaned['employment_duration'].mode()[0]\ndata_cleaned['employment_duration'].fillna(mode_emp_length, inplace=True)\nprint(\"Engineered 'employment_duration' and filled missing with mode.\")\n\n# Engineer 'credit_history_years' from 'earliest_cr_line'\ncurrent_year = datetime.now().year\ndata_cleaned['credit_history_years'] = current_year - pd.to_datetime(data_cleaned['earliest_cr_line'], format='%b-%Y').dt.year\nprint(\"Engineered 'credit_history_years'.\")\n\n# Drop original columns that have been transformed\noriginal_cols_to_drop = ['term', 'emp_length', 'earliest_cr_line']\ndata_cleaned.drop(columns=original_cols_to_drop, inplace=True)\n\n# --- Step 3: Final Data Transformation (Categorical to Numerical) ---\nprint(\"\\n--- Stage 3: Encoding Categorical Features ---\")\ncategorical_features_to_encode = data_cleaned.select_dtypes(include=['object']).columns\ndata_final = pd.get_dummies(data_cleaned, columns=categorical_features_to_encode, drop_first=True)\n\nprint(f\"Data cleaning and feature engineering complete.\")\nprint(f\"Final dataset shape for modeling: {data_final.shape}\")",
    "output": "--- Stage 1: Addressing Missing Data ---\nFilled missing 'revol_util' values with median (54.80).\nFilled missing 'mort_acc' values with median (1.00).\nFilled missing 'pub_rec_bankruptcies' with 0.\nDropped irrelevant/redundant columns: ['emp_title', 'title', 'address', 'issue_d', 'sub_grade', 'loan_status']\n\n--- Stage 2: Creating New Features ---\nEngineered 'employment_duration' and filled missing with mode.\nEngineered 'credit_history_years'.\n\n--- Stage 3: Encoding Categorical Features ---\nData cleaning and feature engineering complete.\nFinal dataset shape for modeling: (396030, 45)",
    "execution_count": 9
  },
  {
    "type": "code",
    "code": "# --- Step 4: Correlation Analysis of Final Features ---\n# Separate features (X) and target (y)\nX = data_final.drop('is_fully_paid', axis=1)\ny = data_final['is_fully_paid']\n\n# Calculate the full correlation matrix\ncorrelation_matrix = X.corr()\n\n# Visualize the correlation matrix to check for multicollinearity\nplt.figure(figsize=(20, 16))\nmask = np.triu(np.ones_like(correlation_matrix, dtype=bool))\nsns.heatmap(correlation_matrix, mask=mask, cmap='coolwarm', annot=False, center=0)\nplt.title('Feature Correlation Matrix (Post-Engineering)', fontsize=16)\nplt.show()\n\n# Identify pairs with high correlation\nhigh_corr_threshold = 0.8\nstacked_corr = correlation_matrix.abs().stack()\nhigh_corr_pairs = stacked_corr[stacked_corr > high_corr_threshold]\nhigh_corr_pairs = high_corr_pairs[high_corr_pairs < 1] # Remove self-correlation\n\nprint(f\"Found {len(high_corr_pairs)//2} pairs of features with correlation > {high_corr_threshold}\")\nprint(high_corr_pairs.sort_values(ascending=False).to_frame('Correlation').head(10))",
    "output": "<Figure size 2000x1600 with 2 Axes>Found 2 pairs of features with correlation > 0.8\n                                                 Correlation\nloan_amnt               installment                    0.954\ninstallment             loan_amnt                      0.954\nhome_ownership_MORTGAGE home_ownership_RENT            0.824\nhome_ownership_RENT     home_ownership_MORTGAGE        0.824",
    "execution_count": 10
  },
  {
    "type": "code",
    "code": "# --- Step 5: Final Feature Selection and Data Preparation ---\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\n\n# Remove one feature from each highly correlated pair to reduce multicollinearity\n# e.g., 'loan_amnt' and 'installment' are highly correlated. We can drop one.\nfeatures_to_remove_for_model = ['installment']\n\nX_selected = X.drop(columns=features_to_remove_for_model)\nfinal_feature_list = X_selected.columns.tolist()\n\nprint(f\"Final number of features for the model: {len(final_feature_list)}\")\nprint(\"Selected Features:\", final_feature_list[:5], \"...\")\n\n# --- Step 6: Data Splitting and Scaling ---\n# Split the data into training and testing sets, stratifying by the target\nX_train, X_test, y_train, y_test = train_test_split(\n    X_selected, y, test_size=0.25, random_state=101, stratify=y\n)\n\n# Initialize and apply the scaler\nscaler = StandardScaler()\nX_train_scaled = scaler.fit_transform(X_train)\nX_test_scaled = scaler.transform(X_test)\n\nprint(\"\\n--- Data Splitting and Scaling Summary ---\")\nprint(f\"Training Set Shape: {X_train_scaled.shape}\")\nprint(f\"Test Set Shape: {X_test_scaled.shape}\")\nprint(f\"Training Target Distribution (Fully Paid): {y_train.mean():.3f}\")\nprint(f\"Test Target Distribution (Fully Paid): {y_test.mean():.3f}\")\n\n# --- Step 7: Save Processed Artifacts ---\ndef save_artifacts(data, path, scaler_obj, features):\n    \"\"\"Saves the processed data, scaler, and feature list.\"\"\"\n    os.makedirs('./data/processed_alt', exist_ok=True)\n    \n    # Save datasets\n    pd.DataFrame(data['X_train_s'], columns=features).to_csv(f'./data/processed_alt/{path}_train_scaled.csv', index=False)\n    pd.DataFrame(data['X_test_s'], columns=features).to_csv(f'./data/processed_alt/{path}_test_scaled.csv', index=False)\n    data['y_train'].to_csv(f'./data/processed_alt/{path}_ytrain.csv', index=False)\n    data['y_test'].to_csv(f'./data/processed_alt/{path}_ytest.csv', index=False)\n    \n    # Save scaler parameters and feature list\n    scaler_params = {'mean': scaler_obj.mean_.tolist(), 'scale': scaler_obj.scale_.tolist(), 'features': features}\n    with open(f'./data/processed_alt/{path}_scaler_info.json', 'w') as f:\n        json.dump(scaler_params, f, indent=4)\n        \n    print(f\"\\nArtifacts saved successfully to './data/processed_alt' with prefix '{path}'.\")\n\nsave_artifacts(\n    data={'X_train_s': X_train_scaled, 'X_test_s': X_test_scaled, 'y_train': y_train, 'y_test': y_test},\n    path='lendingclub_v2',\n    scaler_obj=scaler,\n    features=final_feature_list\n)\n\nprint(f\"\\nData preparation is complete. Ready for model training.\")",
    "output": "Final number of features for the model: 43\nSelected Features: ['loan_amnt', 'int_rate', 'annual_inc', 'dti', 'open_acc'] ...\n\n--- Data Splitting and Scaling Summary ---\nTraining Set Shape: (297022, 43)\nTest Set Shape: (99008, 43)\nTraining Target Distribution (Fully Paid): 0.804\nTest Target Distribution (Fully Paid): 0.804\n\nArtifacts saved successfully to './data/processed_alt' with prefix 'lendingclub_v2'.\n\nData preparation is complete. Ready for model training.",
    "execution_count": 11
  },
  {
    "type": "md",
    "md": "## 7. Analysis Summary and Key Takeaways\n\n### Data Profile Summary\n\n- **Dataset Scale**: The analysis began with 396,030 loan records and 27 original features.\n- **Data Integrity**: Six features contained missing values, most notably `mort_acc` (9.5% missing) and `emp_length` (4.6% missing), which were subsequently imputed.\n- **Target Imbalance**: The dataset shows a significant class imbalance, with 80.4% of loans being fully paid and 19.6% charged off. This is a critical consideration for model training and evaluation.\n\n### Key Predictive Insights\n\n- **Primary Risk Drivers**: The strongest predictors of loan default are the interest rate, loan grade, and the borrower's debt-to-income (DTI) ratio. Higher values in these features consistently correlate with a higher probability of default.\n- **Categorical Patterns**:\n  - **Loan Grade**: A clear trend was observed where higher grades (e.g., 'A') have a much higher repayment rate (93.7%) compared to lower grades (e.g., 'G' at 52.2%).\n  - **Home Ownership**: Borrowers with mortgages tend to have higher repayment rates than those who rent.\n  - **Purpose**: Loans for small businesses carry a significantly higher risk of default compared to loans for weddings or cars.\n\n### Feature Engineering Impact\n\n- **New Features**: We successfully created several new features, including `credit_history_years` and `employment_duration`, to capture more nuanced information.\n- **Multicollinearity Management**: After one-hot encoding, a correlation analysis revealed multicollinearity (e.g., between `loan_amnt` and `installment`). Redundant features were pruned to create a more robust feature set for linear models and neural networks.\n- **Final Feature Set**: The final data prepared for modeling consists of a well-defined set of engineered and encoded features.\n\n### Modeling Preparation Checklist\n\n- **Training/Test Split**: The data has been split into training (75%) and testing (25%) sets using a stratified approach to preserve the target distribution in both sets.\n- **Feature Scaling**: All numerical features have been standardized using `StandardScaler` to ensure they are on a comparable scale, which is crucial for distance-based algorithms and neural networks.\n- **Imbalance Strategy**: For modeling, it will be essential to employ techniques like class weighting (e.g., `class_weight='balanced'`) or resampling methods (e.g., SMOTE) to prevent the model from being biased towards the majority class."
  }
]