Add anomaly detector for AS hegemony cone (count of downstream networks)

[romain-fontugne]

Is your feature request related to a problem? Please describe.
Currently we cannot detect anomalies in the number of dependent networks because these value is computed just before storing it in the database.

Describe the solution you'd like

• We should compute the number of dependent networks from AS hegemony values in Kafka and push the results to Kafka.
• So we can also run an anomaly detector for it.
• And we'll need a new consumer to store these values to the postures database.

[rishabhknowss]

Hi Romain,

While reviewing the enhancement and going through the repository, I think that we need to add logic to compute the number of dependent networks in the psql/consumers/ASHegemony.py script.
The logic can be implemented as -

def compute_dependent_networks(as_hegemony_values):
    # Dictionary to store the count of dependent networks for each AS
    dependent_networks_count = {}

    # Iterate through AS hegemony values
    for as_value in as_hegemony_values:
        asn = as_value['asn']
        dependent_networks = as_value.get('dependent_networks', [])

        # Update the count for each dependent network
        for network in dependent_networks:
            if network not in dependent_networks_count:
                dependent_networks_count[network] = 1
            else:
                dependent_networks_count[network] += 1

    return dependent_networks_count

Additionally, creating a new consumer script to consume messages from the topic where the computed results are pushed, and adding a PostgreSQL schema for storing the count requests.
Well , I am new to Kafka as well as IHR and since I don't have hands-on experience with it, can you please guide me more towards implementing the solution

[romain-fontugne]

Hi,

A similar code is already in psql/consumers/ASHegemony.py:

kafka-toolbox/psql/consumers/ASHegemony.py

Line 193 in 89649fb

self.hegemonyCone[asn] += inc

We need to decouple the two functions that this script has, pushing ASHegemony values to Postgres and computing the number of dependent networks. So I think we should start with a new script that reads ASHegemony values from Kafka and store the number of dependent networks back into kafka.

[rishabhknowss]

Okay! On it

[ojninja16]

HI Romain, is this issue still open?..I would wanna work on this.

[romain-fontugne]

hey, sorry for the late reply. Yes this is still open. The first step is to move the computation of AS hegemony cone to kafka. That means read the AS hegemony kafka topic, compute AS hegemony cone, and push the results to a new kafka topic

[ojninja16]

hi hope you are doing well
I have been exploring the repository and reading up on Kafka topics to understand the basic requirements for using the scripts. However, I encountered an issue while trying to run the script with the following command:

docker run --rm --name ihr_hegemony_sink_4 \ -e KAFKA_HOST="kafka1:9092" -e DB_CONNECTION_STRING="host=momoka0.iijlab.net dbname=ihr" \ internethealthreport/psql_consumers ASHegemony.py ihr_hegemony 4

Unfortunately, I received the following error related to the connection:

psycopg2.OperationalError: could not translate host name "momoka0.iijlab.net" to address: Name or service not known

If you have any insights or suggestions on how to resolve this, I would greatly appreciate your help.

[ojninja16]

Hi @romain-fontugne upon further investigating issue with resolving the hostname momoka0.iijlab.net , I found that nslookup also returns:

server can't find momoka0.iijlab.net: NXDOMAIN

Could you please confirm the following:

• Is the hostname momoka0.iijlab.net correct?
• Is it publicly resolvable, or is it part of a private/internal network?
• If private, could you provide the corresponding IP address or access instructions?

Let me know if there’s anything else I need to configure on my side to resolve this issue.

Thank you for your assistance!

[romain-fontugne]

Hi, yes this name is not available anymore. I wonder you found that?
Anyways our Kafka cluster is not publicly available so you have to try that locally with a test Kafka cluster

[ojninja16]

Actually, I was using the readme to set up the repo and try things out using Psql consumers Readme , I was using docker to run AsHegemony.py as per the Readme

docker run --rm --name ihr_hegemony_sink_4
-e KAFKA_HOST="kafka1:9092" -e DB_CONNECTION_STRING="host=momoka0.iijlab.net dbname=ihr"
internethealthreport/psql_consumers ASHegemony.py ihr_hegemony 4

so that's where I was encountering this issue . If there are any other files/configs I need please let me know cause I have been stuck on this.