[proofpoint_essentials] Initial Release of Proofpoint Essentials

.github/CODEOWNERS

@@ -375,6 +375,7 @@
 /packages/prometheus/data_stream/query @elastic/obs-infraobs-integrations
 /packages/prometheus/data_stream/remote_write @elastic/obs-ds-hosted-services
 /packages/prometheus_input @elastic/obs-infraobs-integrations
+/packages/proofpoint_essentials @elastic/security-service-integrations
 /packages/proofpoint_itm @elastic/security-service-integrations
 /packages/proofpoint_on_demand @elastic/security-service-integrations
 /packages/proofpoint_tap @elastic/security-service-integrations

packages/proofpoint_essentials/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]

packages/proofpoint_essentials/_dev/build/docs/README.md

@@ -0,0 +1,130 @@
+# Proofpoint Essentials Integration for Elastic
+
+## Overview
+The Proofpoint Essentials integration with Elastic enables the collection of threats for monitoring and analysis. This valuable data can be leveraged within Elastic to analyze potential threat signals, including spam, phishing, business email compromise (BEC), imposter emails, ransomware, and malware.
+
+This integration utilizes the [Proofpoint Essentials Threat API](https://help.proofpoint.com/Essentials/Additional_Resources/API_Documentation/Essentials_Threat_API) to collect threat events.
+
+### Compatibility
+
+The Proofpoint Essentials integration uses the REST API. It uses the `/v2/siem/all` to collect threat events.
+
+### How it works
+
+The **threat** data stream uses the `/v2/siem/all` endpoint to gather all threats starting from the configured initial interval. Subsequently, it fetches the recent threats available at each specified interval.
+
+The gathered threat data is subsequently routed into individual data streams, each corresponding to a specific threat type.
+
+## What data does this integration collect?
+
+The Proofpoint Essentials integration collects threat events of the following types:
+
+- `clicks_blocked`: events for clicks on malicious URLs blocked by URL Defense.
+- `clicks_permitted`: events for clicks on malicious URLs permitted by URL Defense.
+- `message_blocked`: events for blocked messages that contain threats recognized by URL Defense or Attachment Defense.
+- `message_delivered`: events for delivered messages that contain threats recognized by URL Defense or Attachment Defense.
+
+### Supported use cases
+Integrating Proofpoint Essentials with Elastic SIEM enriches your security operations with targeted email threat intelligence. It enables the detection, investigation, and analysis of phishing, malware, and other email-based threats by leveraging detailed data on clicks and message events.
+
+## What do I need to use this integration?
+
+### From Proofpoint Essentials
+
+#### Collecting data from Essentials Threat API
+
+1. Navigate to 
+  - Go to **Account Management > Integrations**, then select the **Integration Keys** tab.
+2. Add a New Key
+  - Click **Add Integration Key** in the upper right-hand corner.
+3. Enter Key Details
+  - Provide a **description** to help identify the purpose of the key.
+  - In the **Access Type** dropdown, select **SIEM Threat Events**
+4. Set Scope
+  - If you are part of an **organisation**, the **Scope** field will be locked to **My Organisation Only**.
+  - If you are a **partner**, you can choose between:
+    - **My Organisation Only**
+    - **My Organisation and All Child Organisations**
+5. Create and Save Credentials
+  - After clicking **Create**, you’ll receive **API Key** and **API Key Secret**.
+6. Activation Time
+  - The key may take up to **30 minutes** to become active.
+
+For more details, check [Documentation](https://help.proofpoint.com/Essentials/Product_Documentation/Account_Management/Integrations/Integration_Keys).
+
+## How do I deploy this integration?
+
+### Agent-based deployment
+
+Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host.
+
+Elastic Agent is required to stream data from the syslog or log file receiver and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.
+
+### Agentless deployment
+
+Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. Agentless deployments provide a means to ingest data while avoiding the orchestration, management, and maintenance needs associated with standard ingest infrastructure. Using an agentless deployment makes manual agent deployment unnecessary, allowing you to focus on your data instead of the agent that collects it.
+
+For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html) 
+
+### Onboard / configure
+
+1. In the top search bar in Kibana, search for **Integrations**.
+2. In the search bar, type **Proofpoint Essentials**.
+3. Select the **Proofpoint Essentials** integration from the search results.
+4. Select **Add Proofpoint Essentials** to add the integration.
+5. Enable and configure only the collection methods which you will use.
+
+    * To **Collect Proofpoint Essentials logs via API**, you'll need to:
+
+        - Configure **URL**, **API Key**, and **API Key Secret**.
+        - Adjust the integration configuration parameters if required, including the Interval, Collect Customer Data, Collect Own Data, Preserve original event etc. to enable data collection.
+
+6. Select **Save and continue** to save the integration.
+
+### Validation
+
+#### Dashboards populated
+
+1. In the top search bar in Kibana, search for **Dashboards**.
+2. In the search bar, type **Proofpoint Essentials**.
+3. Select a dashboard for the dataset you are collecting, and verify the dashboard information is populated.
+
+## Troubleshooting
+
+For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems).
+
+## Scaling
+
+For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation.
+
+## Reference
+
+### ECS field reference
+
+#### Clicks Blocked
+
+{{fields "clicks_blocked"}}
+
+#### Clicks Permitted
+
+{{fields "clicks_permitted"}}
+
+#### Messages Blocked
+
+{{fields "message_blocked"}}
+
+#### Messages Delivered
+
+{{fields "message_delivered"}}
+
+### Inputs used
+
+These inputs are used in this integration:
+
+- [cel](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-cel)
+
+### API usage
+
+This integration uses the following APIs:
+
+- [Proofpoint Essentials Threat API](https://help.proofpoint.com/Essentials/Additional_Resources/API_Documentation/Essentials_Threat_API).

packages/proofpoint_essentials/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.1.0"
+  changes:
+    - description: Initial release.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/16073

packages/proofpoint_essentials/data_stream/clicks_blocked/fields/base-fields.yml

@@ -0,0 +1,16 @@
+- name: data_stream.type
+  external: ecs
+- name: data_stream.dataset
+  external: ecs
+- name: data_stream.namespace
+  external: ecs
+- name: event.module
+  external: ecs
+  type: constant_keyword
+  value: proofpoint_essentials
+- name: event.dataset
+  external: ecs
+  type: constant_keyword
+  value: proofpoint_essentials.clicks_blocked
+- name: '@timestamp'
+  external: ecs

packages/proofpoint_essentials/data_stream/clicks_blocked/fields/beats.yml

@@ -0,0 +1,6 @@
+- name: input.type
+  type: keyword
+  description: Type of filebeat input.
+- name: log.offset
+  type: long
+  description: Log offset.

packages/proofpoint_essentials/data_stream/clicks_blocked/fields/ecs.yml

@@ -0,0 +1,4 @@
+# Define ECS constant fields as constant_keyword
+- name: observer.vendor
+  type: constant_keyword
+  external: ecs

packages/proofpoint_essentials/data_stream/clicks_blocked/fields/fields.yml

@@ -0,0 +1,166 @@
+- name: proofpoint_essentials
+  type: group
+  fields:
+    - name: threat
+      type: group
+      fields:
+        - name: cc_addresses
+          type: keyword
+          description: 'A list of email addresses contained within the CC: header, excluding friendly names.'
+        - name: classification
+          type: keyword
+          description: The threat category of the malicious URL.
+        - name: click_ip
+          type: ip
+          description: The external IP address of the user who clicked on the link. If the user is behind a firewall performing network address translation, the IP address of the firewall will be shown.
+        - name: click_time
+          type: date
+          description: The time the user clicked on the URL.
+        - name: completely_rewritten
+          type: keyword
+          description: The rewrite status of the message.
+        - name: customer_eid
+          type: keyword
+          description: The customer's entity ID.
+        - name: customer_name
+          type: keyword
+          description: The customer's name, as configured in Essentials.
+        - name: event_type
+          type: keyword
+        - name: from_address
+          type: keyword
+          description: 'The email address contained in the From: header, excluding friendly name.'
+        - name: guid
+          type: keyword
+          description: The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.
+        - name: header_from
+          type: keyword
+          description: 'The full content of the From: header, including any friendly name.'
+        - name: header_reply_to
+          type: keyword
+          description: 'If present, the full content of the Reply-To: header, including any friendly names.'
+        - name: id
+          type: keyword
+          description: The unique id of the click.
+        - name: impostor_score
+          type: long
+          description: The impostor score of the message. Higher scores indicate higher certainty.
+        - name: malware_score
+          type: long
+          description: The malware score of the message. Higher scores indicate higher certainty.
+        - name: message_details_url
+          type: keyword
+          description: A permalink to the messages' details page.
+        - name: message_id
+          type: keyword
+          description: Message-ID extracted from the headers of the email message. It can be used to look up the associated message in PPS and is not unique.
+        - name: message_parts
+          type: group
+          fields:
+            - name: content_type
+              type: keyword
+              description: The true, detected Content-Type of the message_part. This may differ from the o_content_type value.
+            - name: disposition
+              type: keyword
+              description: If the value is "inline", the message_part is a message body. If the value is "attached", the message_part is an attachment.
+            - name: filename
+              type: keyword
+              description: The filename of the message_part.
+            - name: md5
+              type: keyword
+              description: The MD5 hash of the message_part contents.
+            - name: o_content_type
+              type: keyword
+              description: The declared Content-Type of the message_part.
+            - name: sandbox_status
+              type: keyword
+              description: The verdict returned by the sandbox during the scanning process.
+            - name: sha256
+              type: keyword
+              description: The SHA256 hash of the message_part contents.
+        - name: message_size
+          type: long
+          description: The size in bytes of the message, including headers and attachments.
+        - name: message_time
+          type: date
+          description: When the message was delivered to the user or quarantined by PPS.
+        - name: parent_eid
+          type: keyword
+          description: The parent's EID.
+        - name: parent_name
+          type: keyword
+          description: The parent's name, as configured in Essentials.
+        - name: phish_score
+          type: long
+          description: The phish score of the message. Higher scores indicate higher certainty.
+        - name: quarantine_rule
+          type: keyword
+          description: The name of the rule which quarantined the message. This appears only for messages_blocked events.
+        - name: recipient
+          type: keyword
+          description: An array containing the email addresses of the SMTP (envelope) recipients.
+        - name: reply_to_address
+          type: keyword
+          description: 'The email address contained in the Reply-To: header, excluding friendly name.'
+        - name: sender
+          type: keyword
+          description: The email address of the SMTP (envelope) sender. The user-part is hashed. The domain-part is cleartext.
+        - name: sender_ip
+          type: ip
+          description: The IP address of the sender.
+        - name: spam_score
+          type: long
+          description: The spam score of the message. Higher scores indicate higher certainty.
+        - name: stack_name
+          type: keyword
+          description: The name of the Essentials stack which processed the message.
+        - name: subject
+          type: keyword
+          description: The subject line of the message, if available.
+        - name: threat_id
+          type: keyword
+          description: The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints.
+        - name: threat_time
+          type: date
+          description: Proofpoint identified the URL as a threat at this time.
+        - name: threat_status
+          type: keyword
+          description: The current state of the threat.
+        - name: threats_info_map
+          type: group
+          fields:
+            - name: actors
+              type: nested
+              description: An array of structures which contain details about the actors associated with a threat.
+            - name: classification
+              type: keyword
+              description: The category of threat found in the message.
+            - name: detection_type
+              type: keyword
+            - name: threat
+              type: keyword
+              description: The artifact which was condemned by Proofpoint. The malicious URL, hash of the attachment threat, or email address of the impostor sender.
+            - name: threat_id
+              type: keyword
+              description: The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints.
+            - name: threat_status
+              type: keyword
+              description: The current state of the threat.
+            - name: threat_time
+              type: date
+              description: Proofpoint assigned the threat_status at this time.
+            - name: threat_type
+              type: keyword
+              description: Whether the threat was an attachment, URL, or message type.
+        - name: to_addresses
+          type: keyword
+          description: 'A list of email addresses contained within the To: header, excluding friendly names.'
+        - name: url
+          type: keyword
+          description: The malicious URL which was clicked.
+        - name: user_agent
+          type: keyword
+          description: The User-Agent header from the clicker's HTTP request.
+        - name: xmailer
+          type: keyword
+          description: 'The content of the X-Mailer: header, if present.'

packages/proofpoint_essentials/data_stream/clicks_blocked/manifest.yml

@@ -0,0 +1,9 @@
+title: Proofpoint Essentials Clicks Blocked Events
+dataset: proofpoint_essentials.clicks_blocked
+type: logs
+elasticsearch:
+  dynamic_dataset: true
+  dynamic_namespace: true
+  index_template:
+    mappings:
+      dynamic: true

packages/proofpoint_essentials/data_stream/clicks_permitted/fields/base-fields.yml

@@ -0,0 +1,16 @@
+- name: data_stream.type
+  external: ecs
+- name: data_stream.dataset
+  external: ecs
+- name: data_stream.namespace
+  external: ecs
+- name: event.module
+  external: ecs
+  type: constant_keyword
+  value: proofpoint_essentials
+- name: event.dataset
+  external: ecs
+  type: constant_keyword
+  value: proofpoint_essentials.clicks_permitted
+- name: '@timestamp'
+  external: ecs

packages/proofpoint_essentials/data_stream/clicks_permitted/fields/beats.yml

@@ -0,0 +1,6 @@
+- name: input.type
+  type: keyword
+  description: Type of filebeat input.
+- name: log.offset
+  type: long
+  description: Log offset.

packages/proofpoint_essentials/data_stream/clicks_permitted/fields/ecs.yml

@@ -0,0 +1,4 @@
+# Define ECS constant fields as constant_keyword
+- name: observer.vendor
+  type: constant_keyword
+  external: ecs