feat: AI moderation feature by jcapphelix · Pull Request #1 · edx/forum

jcapphelix · 2025-10-30T10:44:08Z

Description

Spammers have been spamming the edX forums in partner courses. This PR adds a feature of AI moderation. So manual efforts are reduced to remove the spam.

As of now it is just a MVP.

There are various changes supposed to be done on this.

Test Instructions

Install this forum on your devstack / environment (Open edX is on forum==0.3.8, edx/forum is on forum == 0.3.9)
Enable Ai Moderation waffle discussions.enable_ai_moderation
Set following in your environment files

AI_MODERATION_API_URL: 'URL_FOR_AI'
AI_MODERATION_CLIENT_ID: 'AI_CLIENT_ID'
AI_MODERATION_USER_ID: USER_ID_OF_GLOBAL_MODERATOR

Now try and create a spam / scam post, you should see it as reported (if you are a staff)
ModerationAuditLog in Django admin will also show you the spammed post

Related PRs and order of merging:-

Needs Migration command to be run

This PR adds new migration files so migration commands need to run if not run automatically

./manage.py lms migrate forum

Verification

To check if it's installed or not, you can go to your edxapp env in your desired environment and fire following :-

pip list | grep forum

This should list 0.3.9 as forum version

ToDo Check list:

Known issues :-

https://2u-internal.atlassian.net/browse/COSMO2-770

Deadline :- ASAP

Jira Link

Apt GIF

Copilot

Pull Request Overview

This PR adds AI-powered spam moderation to the forum platform. The implementation introduces a waffle flag-controlled feature that automatically classifies new posts and comments as spam using an external AI service, and flags detected spam content for moderation.

Adds is_spam field to Comment and CommentThread models with database migration
Implements AI moderation service that integrates with an external API for spam classification
Creates audit logging infrastructure via ModerationAuditLog model to track all AI moderation decisions

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
forum/toggles.py	Adds new waffle flag namespace and toggle for AI moderation feature
forum/serializers/contents.py	Adds `is_spam` field to content serializer
forum/migrations/0005_comment_is_spam_commentthread_is_spam_and_more.py	Creates database migration for spam flag and audit log table
forum/backends/mysql/models.py	Adds `is_spam` field to Content model and creates ModerationAuditLog model
forum/backends/mysql/api.py	Implements MySQL backend methods for flagging/unflagging spam
forum/backends/mongodb/threads.py	Adds `is_spam` parameter to thread insert/update operations
forum/backends/mongodb/comments.py	Adds `is_spam` parameter to comment insert/update operations
forum/backends/mongodb/api.py	Implements MongoDB backend methods for flagging/unflagging spam
forum/api/threads.py	Integrates AI moderation into thread creation flow
forum/api/comments.py	Integrates AI moderation into comment creation flow
forum/ai_moderation.py	Core AI moderation service implementation with API integration
forum/admin.py	Adds admin interface for spam flag and moderation audit logs
forum/init.py	Version bump to 0.3.9

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

forum/backends/mongodb/api.py

forum/backends/mysql/api.py

forum/backends/mongodb/threads.py

forum/backends/mongodb/comments.py

forum/backends/mongodb/api.py

forum/ai_moderation.py

mraman-2U

Test cases to be added for the new changes.

mraman-2U · 2025-11-04T12:50:36Z

forum/ai_moderation.py

+        self.system_message = getattr(
+            settings, "AI_MODERATION_SYSTEM_MESSAGE", self.DEFAULT_SYSTEM_MESSAGE
+        )
+        self.timeout = getattr(settings, "AI_MODERATION_TIMEOUT", 30)  # seconds


Get both read timeout and connectiion timout from config use the same in the post request

mraman-2U · 2025-11-04T12:52:33Z

forum/ai_moderation.py

+
+    def __init__(self):  # type: ignore[no-untyped-def]
+        """Initialize the AI moderation service."""
+        self.api_url = getattr(settings, "AI_MODERATION_API_URL", self.DEFAULT_API_URL)


Can we add None instead of taking test data as default values? Also keep None even for system message

mraman-2U · 2025-11-04T12:58:58Z

forum/ai_moderation.py

+        }
+
+        payload = {
+            "messages": [{"role": "user", "content": content}],


For content volume or size validation, is the a max char validation while getting the input in forums? If not, we have check for the max. token validation before calling XPert API. The max. token is the configurable one

Will take it post MVP, have added the point to https://2u-internal.atlassian.net/browse/COSMO2-770

mraman-2U · 2025-11-05T00:45:49Z

forum/ai_moderation.py

+
+    def _make_api_request(self, content: str) -> Optional[Dict[str, Any]]:
+        """
+        Make API request to AI moderation service.


Update to API request to XPert Service

mraman-2U · 2025-11-05T01:04:19Z

forum/ai_moderation.py

+
+        classification = moderation_result.get("classification", "not_spam")
+        reasoning = moderation_result.get("reasoning", "No reasoning provided")
+        is_spam = classification in ["spam", "spam_or_scam"]


Can these be made as enum?

Ideally yeah, but as of now we only have spam_or_scam and I kept spam as another option.

So when we do add more classification types we can make this as Enum. I'll add this as a note here :- https://2u-internal.atlassian.net/browse/COSMO2-770

mraman-2U · 2025-11-05T06:23:00Z

forum/ai_moderation.py

+            if not self.ai_moderation_user_id:
+                raise ValueError("AI_MODERATION_USER_ID setting is not configured.")
+            backend.flag_content_as_spam(content_type, content_id)
+            backend.flag_as_abuse(


For missing AI MODERATION USER ID config, None is being used as user id, will the model support to take "None"?

No, it won't support, hence the value error.

Posting of content will continue without any issue, just moderation won't happen if user is not defined.

Are we not storing with "None" as user id in DB ?

We can allow, but as flag abuse also needs a user, we raise a value error.

And allow the post flow to continue but not the AI moderation flow.

mraman-2U · 2025-11-05T06:26:29Z

forum/ai_moderation.py

+                return None
+        except (
+            requests.RequestException,
+            requests.Timeout,


For timeout, may need to build retry logic. It will be slowing down the synchronous request. Moving to async with retry will serve better.

Yes, that's good point.

I've added connection and read timeouts for now.

Adding retry logic and adding the calls to async can be done post MVP, work is logged https://2u-internal.atlassian.net/browse/COSMO2-770

mraman-2U · 2025-11-05T06:29:01Z

forum/ai_moderation.py

+    TODO:-
+     - Add content check for images
+    """
+    return ai_moderation_service.moderate_and_flag_content(


Cache all the flagged messages such that there is no need to hit the XPert api for duplicate messages.

That's a good point, it'll reduce our calls and overhead to Xpert API,

Have added this work item, https://2u-internal.atlassian.net/browse/COSMO2-770

mraman-2U · 2025-11-05T06:31:56Z

forum/ai_moderation.py

+        }
+        # Check if AI moderation is enabled
+        # pylint: disable=import-outside-toplevel
+        from forum.toggles import (


why don't this imported at the top?

Importing forum.toggles at top will bring openedx imports written in toggles file in the picture.
Which will fail our test cases. Hence it's written over here. It is fairly common to do it for such imports. However not that common that it does make us relax about it being present here.

This is already known and we do plan to improve the toggle call in other place instead of here.

https://2u-internal.atlassian.net/browse/COSMO2-770

mraman-2U · 2025-11-05T06:34:21Z

forum/backends/mysql/models.py

    updated_at: models.DateTimeField[datetime, datetime] = models.DateTimeField(
        auto_now=True
    )
+    is_spam: models.BooleanField[bool, bool] = models.BooleanField(


is db indexing required for this attribute? Assess this with existing indexes

added index

Copilot

Pull Request Overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 9 comments.

Comments suppressed due to low confidence (1)

forum/ai_moderation.py:1

[nitpick] Comment in toggles.py mentions 'discussions' but the implementation refers to 'forum' content. The terminology should be consistent with the codebase.

"""

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

forum/ai_moderation.py

forum/backends/mongodb/api.py

forum/ai_moderation.py

jcapphelix force-pushed the chintanjoshi-apphelix-2u/feat-forum branch 10 times, most recently from 820212e to a1cfe7d Compare November 3, 2025 20:18

Copilot AI mentioned this pull request Nov 4, 2025

Initial setup for AI moderation feature implementation #3

Closed

edx deleted a comment from Copilot AI Nov 4, 2025

jcapphelix force-pushed the chintanjoshi-apphelix-2u/feat-forum branch from a1cfe7d to 5a5d333 Compare November 4, 2025 07:37

jcapphelix marked this pull request as ready for review November 4, 2025 09:38

Copilot AI review requested due to automatic review settings November 4, 2025 09:38

jcapphelix changed the title ~~[DO NOT MERGE]: feat: AI moderation feature~~ feat: AI moderation feature Nov 4, 2025

Copilot AI reviewed Nov 4, 2025

View reviewed changes

jcapphelix mentioned this pull request Nov 4, 2025

fix: AI moderation fields in accessible_fields edx/edx-platform#26

Merged

jcapphelix force-pushed the chintanjoshi-apphelix-2u/feat-forum branch from 5a5d333 to ad5d296 Compare November 4, 2025 10:02

mraman-2U reviewed Nov 5, 2025

View reviewed changes

feat: AI moderation system

065b1a8

jcapphelix force-pushed the chintanjoshi-apphelix-2u/feat-forum branch from ad5d296 to 065b1a8 Compare November 5, 2025 09:05

Copilot AI review requested due to automatic review settings November 5, 2025 09:05

Copilot AI reviewed Nov 5, 2025

View reviewed changes

jcapphelix requested a review from mraman-2U November 5, 2025 09:09

mraman-2U approved these changes Nov 5, 2025

View reviewed changes

jcapphelix merged commit e19479b into master Nov 5, 2025
19 checks passed

jcapphelix deleted the chintanjoshi-apphelix-2u/feat-forum branch November 5, 2025 13:46

Conversation

jcapphelix commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Instructions

Related PRs and order of merging:-

Needs Migration command to be run

Verification

Known issues :-

Deadline :- ASAP

Jira Link

Apt GIF

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mraman-2U left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcapphelix Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

jcapphelix commented Oct 30, 2025 •

edited

Loading

jcapphelix Nov 5, 2025 •

edited

Loading