diff --git a/README.md b/README.md index 31d8b4da..c3af4355 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,22 @@ -# PECOS - Predictions for Enormous and Correlated Output Spaces +# PECOS4ANNIF +`pecos4annif` is a fork of Amazon's unmaintained ![pecos](https://github.com/amzn/pecos) library "Predictions for Enormous and Correlated Output Spaces". +This fork was created for the purpose of integrating pecos' Extreme Multi Label Classification Algorithm `X-Transformer` into the library cataloguing tool ![annif](https://github.com/NatLibFi/annif). + + PECOS is a versatile and modular machine learning (ML) framework for fast learning and inference on problems with large output spaces, such as extreme multi-label ranking (XMR) and large-scale retrieval. PECOS' design is intentionally agnostic to the specific nature of the inputs and outputs as it is envisioned to be a general-purpose framework for multiple distinct applications. -Given an input, PECOS identifies a small set (10-100) of relevant outputs from amongst an extremely large (~100MM) candidate set and ranks these outputs in terms of relevance. - +Given an input, PECOS identifies a small set (10-100) of relevant outputs from amongst an extremely large (~100MM) candidate set and ranks these outputs in terms of relevance. ### Features #### Extreme Multi-label Ranking and Classification + * X-Linear ([`pecos.xmc.xlinear`](pecos/xmc/xlinear/README.md)): recursive linear models learning to traverse an input from the root of a hierarchical label tree to a few leaf node clusters, and return top-k relevant labels within the clusters as predictions. See more details in the [PECOS paper (Yu et al., 2020)](https://arxiv.org/pdf/2010.05878.pdf). + fast real-time inference in C++ + can handle 100MM output space @@ -29,7 +34,7 @@ Given an input, PECOS identifies a small set (10-100) of relevant outputs from a ## Requirements and Installation -* Python (3.9, 3.10, 3.11, 3.12) +* Python (3.11 or higher) * Pip (>=19.3) See other dependencies in [`setup.py`](https://github.com/amzn/pecos/blob/mainline/setup.py#L135) @@ -37,31 +42,31 @@ You should install PECOS in a [virtual environment](https://docs.python.org/3/li If you're unfamiliar with Python virtual environments, check out the [user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). ### Supporting Platforms -* Ubuntu 20.04 and 22.04 -* Amazon Linux 2 + +* Ubuntu 22.04 + ### Installation from Wheel -PECOS can be installed using pip as follows: +`pecos4annif` can be installed using pip as follows: + ```bash -python3 -m pip install libpecos +python3 -m pip install pecos4annif ``` ### Installation from Source #### Prerequisite builder tools -* For Ubuntu (20.04, 22.04): + +* For Ubuntu (22.04): ``` bash sudo apt-get update && sudo apt-get install -y build-essential git python3 python3-distutils python3-venv ``` -* For Amazon Linux 2: -``` bash -sudo yum -y install python3 python3-devel python3-distutils python3-venv && sudo yum -y groupinstall 'Development Tools' -``` + #### Install and develop locally ```bash -git clone https://github.com/amzn/pecos +git clone https://github.com/NatLibFi/pecos/ cd pecos python3 -m pip install --editable ./ ``` diff --git a/aws_infra/multinode_batch_cdk/README.md b/aws_infra/multinode_batch_cdk/README.md deleted file mode 100644 index 07da091b..00000000 --- a/aws_infra/multinode_batch_cdk/README.md +++ /dev/null @@ -1,144 +0,0 @@ -# PECOS Distributed Training Infra AWS Multi-node Batch CDK - -This sub-folder contains AWS Multi-node Batch CDK code for fast generation of the infra for training distributed PECOS models. - -The PECOS code used for training is what you have in **local folder** when synth and deploy CDK, which facilitates the testing of local changes. - -## Prerequisite - -1. AWS account with access to VPC, Batch, EC2, S3, ECR. -2. Setup AWS credentials following [[guide](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html)]. -3. Configure versions: - ``` - export NPM_VERSION=v0.38.0 - export NODE_VERSION=14.6.0 - export CDK_VERSION=2.40.0 - ``` -4. Install `npm` and `node` if you haven't: - * Download nvm script: - ``` - curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/${NPM_VERSION}/install.sh | bash - ``` - * Append below to shell config (e.g. `.bashrc` or `.zshrc`), and source it: - ``` - # Append below - export NVM_DIR="$HOME/.nvm" - [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm - [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion - - # Source shell config - source - ``` - * Install latest CDK-supported `Node.js` and `npm`: - ``` - # NOTE: latest node version is not always supported by CDK, double-check before installation - nvm install ${NODE_VERSION} - ``` - * Check installation: - ``` - node -v - npm -v - ``` -5. Install AWS CDK: - ``` - npm install -g aws-cdk@${CDK_VERSION} - - # Check installation - cdk --version - ``` - -## Usage -### Workspace Preparation -Clone Code: -``` -git clone https://github.com/amzn/pecos.git -cd pecos -``` - -**All your command below will run in root folder of pecos.** -Export relative directory for CDK code: -``` -export CDKDIR=aws_infra/multinode_batch_cdk -export CDKAPP="python3 ./${CDKDIR}/app.py" -``` - -Create Python virtual environment and install dependencies: -``` -python3 -m venv $CDKDIR/.venv -source $CDKDIR/.venv/bin/activate -python3 -m pip install -r $CDKDIR/requirements.txt -``` - -### CDK Constructs Generation -Configure parameters: -``` -# Make sure AWS account/region are the same as environment’s AWS credential -./$CDKDIR/config_generator.py -``` - -This will generate a `param_config.json` file in the same directory as `config_generator.py`, which will be used for the CDK synth. - -Every time **CDK synthesize/deploy** needs to be redone for the parameters change. - -Bootstrap: -``` -# Replace contents in <> with your AWS account number and region -cdk -a $CDKAPP bootstrap aws:/// -``` - -CDK synthesize: -``` -cdk -a $CDKAPP synth -# Display change -cdk -a $CDKAPP diff -``` - -CDK deploy: -* Every time, **rerun deploy all stacks** for reflecting new changes from local PECOS code. -* An image for distributed PECOS containers will be built and uploaded, so it may take a while. -``` - cdk -a $CDKAPP deploy --all -``` - -You could open AWS CloudFormation web console [[link](https://console.aws.amazon.com/cloudformation/)] to check the deployed resources. - -If you do not need the infra anymore, **delete the S3 bucket** `s3://pecos-distributed-bucket--` generated by CDK first, then destroy: -``` -cdk -a $CDKAPP destroy --all -``` - -## Example - -Download `eurlex-4k` data and upload to the S3 bucket created by CDK: -``` -wget https://archive.org/download/pecos-dataset/xmc-base/eurlex-4k.tar.gz -tar -zxvf eurlex-4k.tar.gz -aws s3 cp --recursive ./xmc-base/eurlex-4k \ -s3://pecos-distributed-bucket--/input-eurlex-4k/ -``` - -Submit job by executing the following commands in the `aws_infra/multinode_batch_cdk` directory: -``` -./$CDKDIR/submit_job.py \ ---job-name pecos-train-xlinear-eurlex-4k \ ---input-folder input-eurlex-4k \ ---output-folder output-eurlex-4k \ ---num-nodes 2 \ ---cpu 1 \ ---memory 60000 \ ---commands 'mpiexec -n $AWS_BATCH_JOB_NUM_NODES -f /job/hostfile python3 -m pecos.distributed.xmc.xlinear.train \ --x $PECOS_INPUT/tfidf-attnxml/X.trn.npz \ --y $PECOS_INPUT/Y.trn.npz \ --m $PECOS_OUTPUT/eurlex_model \ ---nr-splits 2 -b 50 -k 100 -nst 16 -t 0.1 -python3 -m pecos.xmc.xlinear.predict \ --x $PECOS_INPUT/tfidf-attnxml/X.tst.npz \ --y $PECOS_INPUT/Y.tst.npz \ --m $PECOS_OUTPUT/eurlex_model > $PECOS_OUTPUT/eurlex_score.txt' -``` -in which `$PECOS_INPUT` contains everything downloaded from the S3 input folder given by your parameters, and you should put all outputs into `$PECOS_OUTPUT` for uploading to S3 bucket. -For full parameter list, please check `submit_job.py` - -After submitting the job, navigate to AWS Batch web console [[link](https://console.aws.amazon.com/batch/)] to check job running status. - -After job done, check output model `eurlex_model/` and score `eurlex_score.txt` in the folder `output-eurlex-4k` of S3 bucket generated by CDK. diff --git a/aws_infra/multinode_batch_cdk/app.py b/aws_infra/multinode_batch_cdk/app.py deleted file mode 100644 index 5f869646..00000000 --- a/aws_infra/multinode_batch_cdk/app.py +++ /dev/null @@ -1,74 +0,0 @@ -#!/usr/bin/env python3 -import aws_cdk -import sys -import os -from cdk_constructs.batch import PecosDistributedBatchStack -from cdk_constructs.vpc import PecosDistributedVPCStack -from cdk_constructs.iam import PecosDistributedIAMStack -from cdk_constructs.storage import PecosDistributedStorageStack -from cdk_constructs.ecr import PecosDistributedEcrStack -from cdk_constructs.param_config import PecosDistributedParamConfig - - -try: - param_config = PecosDistributedParamConfig.from_json( - os.path.join(os.path.dirname(os.path.realpath(__file__)), "param_config.json")) -except FileNotFoundError: - raise FileNotFoundError( - f"Configuration json: 'param_config.json' not found. " - f"Please run './config_generator.py' to generate." - ) - -cdk_env = aws_cdk.Environment(account=param_config.account, region=param_config.region) - -app = aws_cdk.App() - -vpc_stack = PecosDistributedVPCStack( - app, - "PecosDistributedVPCStack", - param_config, - stack_name=f"PecosDistributedVPCStack-{param_config.user_name}", - env=cdk_env -) - -storage_stack = PecosDistributedStorageStack( - app, - "PecosDistributedStorageStack", - param_config, - vpc_stack, - stack_name=f"PecosDistributedStorageStack-{param_config.user_name}", - env=cdk_env -) - -ecr_stack = PecosDistributedEcrStack( - app, - "PecosDistributedEcrStack", - param_config, - stack_name=f"PecosDistributedEcrStack-{param_config.user_name}", - env=cdk_env -) - -iam_stack = PecosDistributedIAMStack( - app, - "PecosDistributedIAMStack", - param_config, - storage_stack, - ecr_stack, - stack_name=f"PecosDistributedIAMStack-{param_config.user_name}", - env=cdk_env -) - -batch_stack = PecosDistributedBatchStack( - app, - "PecosDistributedBatchStack", - param_config, - vpc_stack, - storage_stack, - iam_stack, - ecr_stack, - stack_name=f"PecosDistributedBatchStack-{param_config.user_name}", - env=cdk_env -) - -aws_cdk.Tags.of(app).add("CREATEDBY", param_config.user_name) -app.synth() diff --git a/aws_infra/multinode_batch_cdk/cdk.json b/aws_infra/multinode_batch_cdk/cdk.json deleted file mode 100644 index 7a177e25..00000000 --- a/aws_infra/multinode_batch_cdk/cdk.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "app": "python3 app.py", - "watch": { - "include": [ - "**" - ], - "exclude": [ - "README.md", - "cdk*.json", - "requirements*.txt", - "source.bat", - "**/__init__.py", - "python/__pycache__", - "tests" - ] - }, - "context": { - "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true, - "@aws-cdk/core:stackRelativeExports": true, - "@aws-cdk/aws-rds:lowercaseDbIdentifier": true, - "@aws-cdk/aws-lambda:recognizeVersionProps": true, - "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true, - "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true, - "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true, - "@aws-cdk/core:checkSecretUsage": true, - "@aws-cdk/aws-iam:minimizePolicies": true, - "@aws-cdk/core:target-partitions": [ - "aws", - "aws-cn" - ] - } - } diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/__init__.py b/aws_infra/multinode_batch_cdk/cdk_constructs/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/batch.py b/aws_infra/multinode_batch_cdk/cdk_constructs/batch.py deleted file mode 100644 index f58b7997..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/batch.py +++ /dev/null @@ -1,174 +0,0 @@ -from aws_cdk import Stack -from aws_cdk import aws_ec2 -from aws_cdk import aws_batch - - -class PecosDistributedBatchStack(Stack): - _SHARED_DISK_MOUNT_NAME = "shared_data" - - def __init__( - self, - scope, - construct_id, - param_config, - vpc_stack, - storage_stack, - iam_stack, - ecr_stack, - **kwargs - ): - super().__init__(scope, construct_id, **kwargs) - - self.launch_template = self.create_launch_template( - vpc_stack.security_group, - storage_stack.shared_disk, - param_config.user_disk_gb_req, - param_config.user_name - ) - - self.compute_environment = self.create_compute_environment( - vpc_stack.vpc, - vpc_stack.security_group, - iam_stack.ecs_instance_profile, - self.launch_template, - param_config.user_name - ) - - self.job_queue = self.create_job_queue(self.compute_environment, param_config.user_name) - - self.job_definition = self.create_job_definition( - ecr_stack.ecr_assets.image_uri, - iam_stack.batch_job_role, - param_config.user_num_node, - param_config.user_mem_gb_req, - param_config.user_cpu_req, - param_config.user_name) - - def create_launch_template(self, security_group, shared_disk, user_disk_gb_req, user_name): - update_cmd = aws_ec2.UserData.for_linux() - update_cmd.add_commands( - "sudo yum update -y", - "sudo yum install -y amazon-efs-utils", - f"sudo mkdir -p /{self._SHARED_DISK_MOUNT_NAME}", - f"sudo mount -t efs -o tls {shared_disk.file_system_id}:/ /{self._SHARED_DISK_MOUNT_NAME}", - f"sudo chmod 777 /{self._SHARED_DISK_MOUNT_NAME}" - ) - - multipart_user_data = aws_ec2.MultipartUserData() - multipart_user_data.add_user_data_part( - user_data=update_cmd, - content_type=aws_ec2.MultipartBody.SHELL_SCRIPT - ) - - return aws_ec2.LaunchTemplate( - self, - id="PECOS-Distributed-Launch-Template", - launch_template_name=f"PECOS-Distributed-Launch-Template-{user_name}", - block_devices=[ - aws_ec2.BlockDevice( - device_name="/dev/xvda", - volume=aws_ec2.BlockDeviceVolume.ebs( - volume_size=user_disk_gb_req, - delete_on_termination=True, - encrypted=False, - volume_type=aws_ec2.EbsDeviceVolumeType.GP2 - ) - ) - ], - user_data=multipart_user_data - ) - - def create_compute_environment(self, vpc, security_group, instance_profile, launch_template, user_name): - return aws_batch.CfnComputeEnvironment( - self, - id="PECOS-Distributed-Compute-Environment", - compute_environment_name=f"PECOS-Distributed-Compute-Environment-{user_name}", - type="MANAGED", - state="ENABLED", - compute_resources=aws_batch.CfnComputeEnvironment.ComputeResourcesProperty( - type="EC2", - maxv_cpus=999, - minv_cpus=0, - desiredv_cpus=0, - instance_role=instance_profile.attr_arn, - instance_types=["optimal", "c5", "m5", "r5", "x1"], - allocation_strategy="BEST_FIT", - subnets=vpc.select_subnets( - subnet_type=aws_ec2.SubnetType.PRIVATE_WITH_NAT - ).subnet_ids, - security_group_ids=[security_group.security_group_id], - launch_template=aws_batch.CfnComputeEnvironment.LaunchTemplateSpecificationProperty( - launch_template_id=launch_template.launch_template_id, - version="$Latest" - ), - update_to_latest_image_version=True - ) - ) - - def create_job_queue(self, compute_environment, user_name): - return aws_batch.CfnJobQueue( - self, - id="PECOS-Distributed-Batch-Job-Queue", - job_queue_name=f"PECOS-Distributed-Batch-Job-Queue-{user_name}", - state="ENABLED", - compute_environment_order=[ - aws_batch.CfnJobQueue.ComputeEnvironmentOrderProperty( - compute_environment=compute_environment.attr_compute_environment_arn, - order=1 - ) - ], - priority=1 - ) - - def create_job_definition(self, ecr_image_uri, batch_job_role, user_num_node, user_mem_gb_req, user_cpu_req, user_name): - return aws_batch.CfnJobDefinition( - self, - id="PECOS-Distributed-Batch-Job-Definition", - job_definition_name=f"PECOS-Distributed-Batch-Job-Definition-{user_name}", - type="multinode", - node_properties=aws_batch.CfnJobDefinition.NodePropertiesProperty( - main_node=0, - num_nodes=user_num_node, - node_range_properties=[ - aws_batch.CfnJobDefinition.NodeRangePropertyProperty( - target_nodes="0:", - container=aws_batch.CfnJobDefinition.ContainerPropertiesProperty( - image=ecr_image_uri, - job_role_arn=batch_job_role.role_arn, - user="ecs-user", - privileged=True, - mount_points=[ - aws_batch.CfnJobDefinition.MountPointsProperty( - container_path=f"/{self._SHARED_DISK_MOUNT_NAME}", - read_only=False, - source_volume=f"{self._SHARED_DISK_MOUNT_NAME}" - ) - ], - volumes=[ - aws_batch.CfnJobDefinition.VolumesProperty( - host=aws_batch.CfnJobDefinition.VolumesHostProperty( - source_path=f"/{self._SHARED_DISK_MOUNT_NAME}" - ), - name=f"{self._SHARED_DISK_MOUNT_NAME}" - ) - ], - resource_requirements=[ - aws_batch.CfnJobDefinition.ResourceRequirementProperty( - type="VCPU", value=str(user_cpu_req) - ), - aws_batch.CfnJobDefinition.ResourceRequirementProperty( - type="MEMORY", value=str(user_mem_gb_req * 1024) - ) - ], - ulimits=[ - aws_batch.CfnJobDefinition.UlimitProperty( - hard_limit=-1, - name="memlock", - soft_limit=-1 - ) - ] - ) - ) - ] - ) - ) diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/Dockerfile b/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/Dockerfile deleted file mode 100644 index f6d733a8..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/Dockerfile +++ /dev/null @@ -1,70 +0,0 @@ -FROM amazonlinux:2 - - -ENV PYTHONUNBUFFERED=TRUE -ENV PYTHONDONTWRITEBYTECODE=TRUE - -# Fix vulnerability -RUN yum update -y gnupg2 -RUN yum -y updateinfo && yum -y --security update - -# Install dependencies -RUN yum install -y awscli sudo git -RUN yum install -y openssl openssh-clients openssh-server iproute -RUN yum install -y python3 python3-devel python3-distutils python3-venv -RUN yum groupinstall -y 'Development Tools' -RUN amazon-linux-extras install -y epel -RUN yum install -y openblas-devel - -# Install MPI -RUN yum install -y http://mirror.centos.org/centos/7/os/x86_64/Packages/mpich-3.2-3.2-2.el7.x86_64.rpm -RUN yum install -y http://mirror.centos.org/centos/7/os/x86_64/Packages/mpich-3.2-devel-3.2-2.el7.x86_64.rpm -RUN touch /etc/profile.d/mpich.sh -RUN echo 'export PATH=/usr/lib64/mpich-3.2/bin/:$PATH' | tee /etc/profile.d/mpich.sh -ENV PATH="/usr/lib64/mpich-3.2/bin/:${PATH}" - -RUN python3 -m pip install supervisor -RUN python3 -m pip install mpi4py - -# Install pecos from local source -ADD README.md /pecos-source/README.md -ADD Makefile /pecos-source/Makefile -ADD setup.py /pecos-source/setup.py -ADD pecos /pecos-source/pecos/ -RUN cd /pecos-source && make clean && make libpecos - -# Setup user -RUN groupadd -r amazon && useradd --no-log-init -r -g amazon ecs-user -RUN echo "ecs-user ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers -ENV USER ecs-user -ENV HOME /home/$USER -RUN echo $HOME -RUN mkdir -p $HOME && chown -R ecs-user:amazon $HOME - -# Enable password-less SSH -ENV SSHDIR $HOME/.ssh -RUN mkdir -p ${SSHDIR} \ -&& touch ${SSHDIR}/sshd_config \ -&& ssh-keygen -t rsa -f ${SSHDIR}/ssh_host_rsa_key -N '' \ -&& cp ${SSHDIR}/ssh_host_rsa_key.pub ${SSHDIR}/authorized_keys \ -&& cp ${SSHDIR}/ssh_host_rsa_key ${SSHDIR}/id_rsa \ -&& echo " IdentityFile ${SSHDIR}/id_rsa" >> ${SSHDIR}/config \ -&& echo " StrictHostKeyChecking no" >> ${SSHDIR}/config \ -&& echo " UserKnownHostsFile /dev/null" >> ${SSHDIR}/config \ -&& echo " Port 2022" >> ${SSHDIR}/config \ -&& echo 'Port 2022' >> ${SSHDIR}/sshd_config \ -&& echo 'UsePrivilegeSeparation no' >> ${SSHDIR}/sshd_config \ -&& echo "HostKey ${SSHDIR}/ssh_host_rsa_key" >> ${SSHDIR}/sshd_config \ && echo "PidFile ${SSHDIR}/sshd.pid" >> ${SSHDIR}/sshd_config \ -&& chmod -R 600 ${SSHDIR}/* \ -&& chown -R ${USER}:amazon ${SSHDIR}/ -RUN eval `ssh-agent -s` && ssh-add ${SSHDIR}/id_rsa -EXPOSE 22 - -# Add script -USER ecs-user -ARG SCRIPTS_DIR=aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts -ADD ${SCRIPTS_DIR}/supervisord.conf /etc/supervisor/supervisord.conf -ADD ${SCRIPTS_DIR}/dist-run.sh /supervised-scripts/dist-run.sh -RUN sudo chmod 755 /supervised-scripts/dist-run.sh -ADD ${SCRIPTS_DIR}/entry-point.sh /batch-runtime-scripts/entry-point.sh -RUN sudo chmod 755 /batch-runtime-scripts/entry-point.sh diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts/dist-run.sh b/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts/dist-run.sh deleted file mode 100644 index c0a89dfb..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts/dist-run.sh +++ /dev/null @@ -1,191 +0,0 @@ -#!/bin/bash -set -x - - -BASENAME="${0##*/}" -log () { - echo "${BASENAME} - ${1}" -} -HOST_FILE_PATH="/tmp/hostfile" -NODE_IP_FILE_PREFIX="/tmp/node_ip_" -NODE_IP_FILE_PATH="${NODE_IP_FILE_PREFIX}${AWS_BATCH_JOB_ID}" -AWS_BATCH_EXIT_CODE_FILE="/tmp/batch-exit-code" - - - -usage () { - if [ "${#@}" -ne 0 ]; then - log "* ${*}" - log - fi - cat <&2 - log "${2:-1}" > $AWS_BATCH_EXIT_CODE_FILE - kill $(cat /tmp/supervisord.pid) -} - -# Set child by default switch to main if on main node container -NODE_TYPE="child" -if [ "${AWS_BATCH_JOB_MAIN_NODE_INDEX}" == "${AWS_BATCH_JOB_NODE_INDEX}" ]; then - log "Running synchronize as the main node" - NODE_TYPE="main" -fi - -install_deps() { - if [[ ! -v BATCH_BOOTSTRAP ]]; then - echo "BATCH_BOOTSTRAP is not set. continuing" - else - $BATCH_BOOTSTRAP - fi -} - -get_ip() { - local ip=$(/sbin/ip -o -4 addr list eth0 | awk '{print $4}' | cut -d/ -f1) - if [[ -z $ip ]]; then - echo "Failed to get IP." - error_exit - else - echo "$ip" - fi -} - -# wait for all nodes to report -wait_for_nodes () { - log "Running as master node" - - touch $NODE_IP_FILE_PATH - ip=$(get_ip) - - if [ -x "$(command -v nvidia-smi)" ] ; then - NUM_GPUS=$(ls -l /dev/nvidia[0-9] | wc -l) - availablecores=$NUM_GPUS - else - availablecores=$(nproc) - fi - - log "master details -> $ip:$availablecores" - echo "$ip" >> $NODE_IP_FILE_PATH - - startTime=$(date +"%s") - # BATCH_BOOTSTRAP_TIMEOUT is time in minutes which will be used to wait. Default is 15 mins - if [[ ! -v BATCH_BOOTSTRAP_TIMEOUT ]]; then - echo " BATCH_BOOTSTRAP_TIMEOUT is not set. Defaulting to 15 minutes." - waitTime=$(expr 15 \* 60) - else - waitTime=$(expr $BATCH_BOOTSTRAP_TIMEOUT \* 60) - fi - - - - lines=$(ls ${NODE_IP_FILE_PREFIX}* | wc -l) - while [ "$AWS_BATCH_JOB_NUM_NODES" -gt "$lines" ] - do - currentTime=$(date +"%s") - duration=$(expr $currentTime - $startTime) - if [ "$duration" -gt "$waitTime" ] ; then - echo "Waited for $duration seconds . Exiting" - echo "1" > $AWS_BATCH_EXIT_CODE_FILE - kill $(cat /tmp/supervisord.pid) - exit 1 - fi - log "$lines out of $AWS_BATCH_JOB_NUM_NODES nodes joined, waited for $duration. check again in 30 second" - sleep 30 - - lines=$(ls ${NODE_IP_FILE_PREFIX}* | wc -l) - done - # Make the temporary file executable and run it with any given arguments - log "All nodes successfully joined" - - # remove duplicates if there are any. - cat $(ls ${NODE_IP_FILE_PREFIX}*) > $HOST_FILE_PATH - awk '!a[$0]++' $HOST_FILE_PATH > ${HOST_FILE_PATH}-deduped - cat $HOST_FILE_PATH-deduped - sudo mkdir -p /job/ - sudo chmod 777 /job - cp ${HOST_FILE_PATH}-deduped /job/hostfile - cat /job/hostfile - - if [[ ! -v BATCH_ENTRY_SCRIPT ]]; then - echo "BATCH_ENTRY_SCRIPT is not set. continuing" - else - $BATCH_ENTRY_SCRIPT - if [ $? -eq 0 ] - then - log "Writing exit code 0 to $AWS_BATCH_EXIT_CODE_FILE and shutting down supervisord" - echo "0" > $AWS_BATCH_EXIT_CODE_FILE - else - log "Writing exit code 1 to $AWS_BATCH_EXIT_CODE_FILE and shutting down supervisord" - echo "1" > $AWS_BATCH_EXIT_CODE_FILE - fi - fi - - kill $(cat /tmp/supervisord.pid) - exit 0 - -} - - -# Fetch and run a script -report_to_master () { - # get own ip and num cpus - # - ip=$(get_ip) - - if [ -x "$(command -v nvidia-smi)" ] ; then - NUM_GPUS=$(ls -l /dev/nvidia[0-9] | wc -l) - availablecores=$NUM_GPUS - else - availablecores=$(nproc) - fi - - log "I am a child node -> $ip:$availablecores, reporting to the master node -> -${AWS_BATCH_JOB_MAIN_NODE_PRIVATE_IPV4_ADDRESS}" - until echo "$ip" | ssh ${AWS_BATCH_JOB_MAIN_NODE_PRIVATE_IPV4_ADDRESS} "cat > $NODE_IP_FILE_PATH" - do - echo "Sleeping 5 seconds and trying again" - done - - while : - do - echo "Sleeping" - sleep 30 - done - - - - - log "done! goodbye" - exit 0 - } - - -# Main - dispatch user request to appropriate function -log $NODE_TYPE -install_deps -case $NODE_TYPE in - main) - wait_for_nodes "${@}" - ;; - - child) - report_to_master "${@}" - ;; - - *) - log $NODE_TYPE - usage "Could not determine node type. Expected (main/child)" - ;; -esac diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts/entry-point.sh b/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts/entry-point.sh deleted file mode 100644 index 25a9ad0f..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts/entry-point.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/bin/bash -# Launch supervisor -BASENAME="${0##*/}" -log () { - echo "${BASENAME} - ${1}" -} -AWS_BATCH_EXIT_CODE_FILE="/tmp/batch-exit-code" -python3 -m supervisor.supervisord -n -c "/etc/supervisor/supervisord.conf" -# if supervisor dies then read exit code from file we don't want to return the supervisors exit code -log "Reading exit code from batch script stored at $AWS_BATCH_EXIT_CODE_FILE" -if [ ! -f $AWS_BATCH_EXIT_CODE_FILE ]; then - echo "Exit code file not found , returning with exit code 1!" >&2 - exit 1 -fi -exit $(cat $AWS_BATCH_EXIT_CODE_FILE) diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts/supervisord.conf b/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts/supervisord.conf deleted file mode 100644 index f4ccc332..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/dockerfile/scripts/supervisord.conf +++ /dev/null @@ -1,34 +0,0 @@ -[supervisord] -logfile = /tmp/supervisord.log -logfile_maxbytes = 50MB -logfile_backups=10 -loglevel = info -pidfile = /tmp/supervisord.pid -nodaemon = false -minfds = 1024 -minprocs = 200 -umask = 022 -identifier = supervisor -directory = /tmp -nocleanup = true -childlogdir = /tmp -strip_ansi = false - -[program:sshd] -command=/usr/sbin/sshd -D -f /home/ecs-user/.ssh/sshd_config -h /home/ecs-user/.ssh/ssh_host_rsa_key -stdout_logfile=/dev/fd/1 -stdout_logfile_maxbytes=0 -redirect_stderr=true -autorestart=true -stopsignal=INT - - -[program:synchronize] -command=/supervised-scripts/dist-run.sh -stdout_logfile=/dev/fd/1 -stdout_logfile_maxbytes=0 -redirect_stderr=true -autorestart=false -startsecs=0 -stopsignal=INT -exitcodes=0,2 diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/ecr.py b/aws_infra/multinode_batch_cdk/cdk_constructs/ecr.py deleted file mode 100644 index cd9c426d..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/ecr.py +++ /dev/null @@ -1,22 +0,0 @@ -import os -from aws_cdk import Stack -from aws_cdk import aws_ecr_assets - - -class PecosDistributedEcrStack(Stack): - def __init__(self, scope, construct_id, param_config, **kwargs): - super().__init__(scope, construct_id, **kwargs) - - self.ecr_assets = aws_ecr_assets.DockerImageAsset( - self, - id="PECOS-Distributed-Ecr-Image", - directory=os.path.join( - os.path.dirname(os.path.realpath(__file__)), - "../../../" - ), - file=os.path.join( - "aws_infra/multinode_batch_cdk/cdk_constructs", - "dockerfile", - "Dockerfile" - ) - ) diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/iam.py b/aws_infra/multinode_batch_cdk/cdk_constructs/iam.py deleted file mode 100644 index c17e9e1a..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/iam.py +++ /dev/null @@ -1,98 +0,0 @@ -from aws_cdk import aws_iam -from aws_cdk import Stack - - -class PecosDistributedIAMStack(Stack): - def __init__(self, scope, construct_id, param_config, storage_stack, ecr_stack, **kwargs): - super().__init__(scope, construct_id, **kwargs) - - self.ecs_instance_role, self.ecs_instance_profile = self.create_ecs_instance_role(param_config.user_name) - self.batch_job_role = self.create_batch_job_role( - param_config.user_name, - storage_stack.s3_bucket.bucket_name, - ecr_stack.ecr_assets.repository.repository_arn - ) - - def create_ecs_instance_role(self, user_name): - ecs_instance_role = aws_iam.Role( - self, - id="PECOS-Distributed-Ecs-Instance-Role", - role_name=f"PECOS-Distributed-Ecs-Instance-Role-{user_name}", - assumed_by=aws_iam.CompositePrincipal(aws_iam.ServicePrincipal("ec2.amazonaws.com")), - managed_policies=[ - aws_iam.ManagedPolicy.from_aws_managed_policy_name( - "service-role/AmazonEC2ContainerServiceforEC2Role" - ), - ] - ) - ecs_instance_profile = aws_iam.CfnInstanceProfile( - self, - id="PECOS-Distributed-Ecs-Instance-Profile", - instance_profile_name=f"PECOS-Distributed-Ecs-Instance-Profile-{user_name}", - roles=[ecs_instance_role.role_name] - ) - return ecs_instance_role, ecs_instance_profile - - def create_batch_job_role(self, user_name, s3_bucket_name, erc_repo_arn): - job_role = aws_iam.Role( - self, - id="PECOS-Distributed-Batch-Job-Role", - assumed_by=aws_iam.ServicePrincipal("ecs-tasks.amazonaws.com"), - role_name=f"PECOS-Distributed-Batch-Role-{user_name}", - description="Job role for Batch PECOS Inference.", - inline_policies={ - "ECR-Read-Policy": self.create_ecr_read_policy(erc_repo_arn), - "User-S3-Read-Write-Policy": self.create_s3_rw_policy(s3_bucket_name) - }, - managed_policies=[ - aws_iam.ManagedPolicy.from_aws_managed_policy_name("CloudWatchLogsFullAccess") - ] - ) - return job_role - - @classmethod - def create_ecr_read_policy(cls, erc_repo_arn): - policy_json = { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "EcrRead", - "Effect": "Allow", - "Action": [ - "ecr:GetDownloadUrlForLayer", - "ecr:BatchGetImage" - ], - "Resource": erc_repo_arn - }, - { - "Sid": "EcrAuthorize", - "Effect": "Allow", - "Action": "ecr:GetAuthorizationToken", - "Resource": "*" - } - ] - } - return aws_iam.PolicyDocument.from_json(policy_json) - - @classmethod - def create_s3_rw_policy(cls, s3_bucket_name): - s3_bucket_arn = f"arn:aws:s3:::{s3_bucket_name}" - policy_json = { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "S3BucketReadWrite", - "Action": [ - "s3:Get*", - "s3:Put*", - "s3:List*" - ], - "Effect": "Allow", - "Resource": [ - s3_bucket_arn, - s3_bucket_arn + "/*" - ] - } - ] - } - return aws_iam.PolicyDocument.from_json(policy_json) diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/param_config.py b/aws_infra/multinode_batch_cdk/cdk_constructs/param_config.py deleted file mode 100644 index 03f8eb72..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/param_config.py +++ /dev/null @@ -1,30 +0,0 @@ -import json - - -class PecosDistributedParamConfig(object): - """ - Parameters for PECOS distributed jobs - """ - - def __init__( - self, - account, - region, - user_name, - user_disk_gb_req - ): - self.account = account - self.region = region - self.user_name = user_name - self.user_disk_gb_req = user_disk_gb_req - # Default value for generating multi-node batch constructs - # Overridable at submitting job - self.user_num_node = 2 - self.user_mem_gb_req = 350 - self.user_cpu_req = 1 - - @classmethod - def from_json(cls, json_path): - with open(json_path) as f: - param_dict = json.load(f) - return cls(**param_dict) diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/storage.py b/aws_infra/multinode_batch_cdk/cdk_constructs/storage.py deleted file mode 100644 index 8de88482..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/storage.py +++ /dev/null @@ -1,61 +0,0 @@ -from aws_cdk import aws_s3 -from aws_cdk import aws_efs -from aws_cdk import aws_ec2 -from aws_cdk import aws_iam -from aws_cdk import Stack -from aws_cdk import RemovalPolicy - - -class PecosDistributedStorageStack(Stack): - def __init__(self, scope, construct_id, param_config, vpc_stack, **kwargs): - super().__init__(scope, construct_id, **kwargs) - - self.s3_bucket = self.create_s3_bucket( - param_config.account, - param_config.user_name - ) - self.shared_disk = self.create_efs( - vpc_stack.vpc, - vpc_stack.security_group, - param_config.user_name - ) - - def create_s3_bucket(self, account, user_name): - return aws_s3.Bucket( - self, - id="PECOS-Distributed-Bucket", - bucket_name=f"pecos-distributed-bucket-{account}-{user_name}", - block_public_access=aws_s3.BlockPublicAccess.BLOCK_ALL, - removal_policy=RemovalPolicy.DESTROY - ) - - def create_efs(self, vpc, security_group, user_name): - shared_efs = aws_efs.FileSystem( - self, - id="PECOS-Distributed-EFS", - file_system_name=f"PECOS-Distributed-EFS-{user_name}", - vpc=vpc, - vpc_subnets=aws_ec2.SubnetSelection( - subnet_type=aws_ec2.SubnetType.PRIVATE_WITH_NAT - ), - security_group=security_group, - enable_automatic_backups=False, - encrypted=False, - performance_mode=aws_efs.PerformanceMode.GENERAL_PURPOSE, - removal_policy=RemovalPolicy.DESTROY - ) - shared_efs.node.default_child.file_system_policy = aws_iam.PolicyDocument( - statements=[ - aws_iam.PolicyStatement( - effect=aws_iam.Effect.ALLOW, - principals=[aws_iam.AnyPrincipal()], - actions=[ - "elasticfilesystem:ClientMount", - "elasticfilesystem:ClientWrite", - "elasticfilesystem:ClientRootAccess", - ], - conditions={"Bool": {"elasticfilesystem:AccessedViaMountTarget": "true"}}, - ) - ] - ) - return shared_efs diff --git a/aws_infra/multinode_batch_cdk/cdk_constructs/vpc.py b/aws_infra/multinode_batch_cdk/cdk_constructs/vpc.py deleted file mode 100644 index 195037a4..00000000 --- a/aws_infra/multinode_batch_cdk/cdk_constructs/vpc.py +++ /dev/null @@ -1,47 +0,0 @@ -from aws_cdk import Stack -from aws_cdk import aws_ec2 - - -class PecosDistributedVPCStack(Stack): - def __init__(self, scope, construct_id, param_config, **kwargs): - super().__init__(scope, construct_id, **kwargs) - - self.vpc = self.create_vpc(param_config.user_name) - self.security_group = self.create_security_group(self.vpc, param_config.user_name) - - def create_vpc(self, user_name): - vpc = aws_ec2.Vpc( - self, - id="PECOS-Distributed-VPC", - vpc_name=f"PECOS-Distributed-VPC-{user_name}", - cidr="10.0.0.0/16", - max_azs=2, - nat_gateways=1, - subnet_configuration=[ - aws_ec2.SubnetConfiguration( - name="public", cidr_mask=24, - reserved=False, subnet_type=aws_ec2.SubnetType.PUBLIC - ), - aws_ec2.SubnetConfiguration( - name="private", cidr_mask=24, - reserved=False, subnet_type=aws_ec2.SubnetType.PRIVATE_WITH_NAT - ), - ], - enable_dns_hostnames=True, - enable_dns_support=True - ) - return vpc - - def create_security_group(self, vpc, user_name): - security_group = aws_ec2.SecurityGroup( - self, - id="PECOS-Distributed-Security-Group", - security_group_name=f"PECOS-Distributed-SG-{user_name}", - vpc=vpc, - allow_all_outbound=True - ) - security_group.connections.allow_internally( - port_range=aws_ec2.Port.all_traffic(), - description="Allow hosts inside the security group to connect to each other." - ) - return security_group diff --git a/aws_infra/multinode_batch_cdk/config_generator.py b/aws_infra/multinode_batch_cdk/config_generator.py deleted file mode 100755 index eeab52dc..00000000 --- a/aws_infra/multinode_batch_cdk/config_generator.py +++ /dev/null @@ -1,55 +0,0 @@ -#!/usr/bin/env python3 -import os -import json - -def input_with_default(user_input, default_val): - if not user_input: - return default_val - return user_input - -def int_input_value_check(user_input, default_val, max_val, min_val): - int_input = int(input_with_default(user_input, default_val)) - if int_input > max_val or int_input < min_val: - raise ValueError(f"Input should be in [{min_val}, {max_val}], got: {int_input}") - return int_input - -def get_parameters(): - param_dict = {} - param_dict["account"] = None - while not param_dict["account"]: - param_dict["account"] = input("Please enter AWS 12-digit account ID (cannot be empty): ") - if not (param_dict["account"].isdigit() and len(param_dict["account"]) == 12): - print(f"AWS account ID should be integer and have 12 digits, got: {param_dict['account']}") - param_dict["account"] = None - - param_dict["region"] = input_with_default( - input("Please enter AWS region. The default is us-east-1: "), - "us-east-1" - ) - - param_dict["user_name"] = input_with_default( - input( - f"Please enter your name for tagging AWS stacks. " - f"The default is current OS user: " - ), - os.getlogin() - ) - - param_dict["user_disk_gb_req"] = int_input_value_check( - input( - f"Please enter disk size requirement(GB) for each node. Range 1GB ~ 15TB.\n" - f"PECOS training recommendations: >=1000GB(1TB).\n" - f"The default is 1000: " - ), - default_val=1000, - max_val=15000, - min_val=1 - ) - - # dump json - with open(os.path.join(os.path.dirname(os.path.realpath(__file__)), "param_config.json"), "w") as fp: - json.dump(param_dict, fp) - - -if __name__ == "__main__": - get_parameters() diff --git a/aws_infra/multinode_batch_cdk/requirements.txt b/aws_infra/multinode_batch_cdk/requirements.txt deleted file mode 100644 index 0549a89d..00000000 --- a/aws_infra/multinode_batch_cdk/requirements.txt +++ /dev/null @@ -1,3 +0,0 @@ -boto3 -aws-cdk-lib==2.40.0 -constructs>=10.0.0,<11.0.0 diff --git a/aws_infra/multinode_batch_cdk/submit_job.py b/aws_infra/multinode_batch_cdk/submit_job.py deleted file mode 100755 index 1bae3ada..00000000 --- a/aws_infra/multinode_batch_cdk/submit_job.py +++ /dev/null @@ -1,243 +0,0 @@ -#!/usr/bin/env python3 - -import argparse -import json -import logging -import os -import boto3 - - -class PecosBatchJobArgs(object): - """Class to parse and store PECOS multi-node Batch job arguments.""" - - def __init__(self): - self._configure_attr(self._get_args()) - - @classmethod - def _configure_attr(self, args): - """Configure self attributes from parsed args""" - # Copied From args - args_dict = vars(args) - for key, val in args_dict.items(): - setattr(self, key, val) - - # Created from args - with open(args.cdk_config) as f: - cdk_conf_dict = json.load(f) - self.account = cdk_conf_dict["account"] - self.region = cdk_conf_dict["region"] - self.user_name = cdk_conf_dict["user_name"] - self.job_definition = f"PECOS-Distributed-Batch-Job-Definition-{self.user_name}" - self.job_queue = f"PECOS-Distributed-Batch-Job-Queue-{self.user_name}" - self.input_s3_arn = f"s3://pecos-distributed-bucket-{self.account}-{self.user_name}/{args.input_folder}/" - self.output_s3_arn = f"s3://pecos-distributed-bucket-{self.account}-{self.user_name}/{args.output_folder}/" - - @classmethod - def _get_args(self): - """Get Batch job args""" - parser = argparse.ArgumentParser() - parser.add_argument( - "--cdk-config", - metavar="CDK_CONF_PATH", - type=str, - default=f"{os.path.join(os.path.dirname(os.path.realpath(__file__)), 'param_config.json')}", - help="CDK parameter configuration file path" - ) - parser.add_argument( - "--job-name", - metavar="JOB_NAME", - type=str, - required=True, - help="AWS Batch Job Name" - ) - parser.add_argument( - "--input-folder", - metavar="INPUT", - type=str, - required=True, - help="S3 input folder name" - ) - parser.add_argument( - "--output-folder", - metavar="OUTPUT", - type=str, - required=True, - help="S3 output folder name" - ) - parser.add_argument( - "--num-nodes", - metavar="NUM_NODES", - type=int, - required=True, - help="Number of nodes for multi-node processing Batch jobs" - ) - parser.add_argument( - "--cpu", - metavar="CPU_COUNTS", - type=int, - required=True, - help=f"Number of vCPU for each node" - ) - parser.add_argument( - "--memory", - metavar="MEM_SIZE", - type=int, - required=True, - help=f"Memory size in megabytes for each node" - ) - parser.add_argument( - "--commands", - metavar="COMMANDS", - required=True, - type=str, - help=f"Commands to be executed on main node" - ) - parser.add_argument( - "--batch-bootstrap-timeout-min", - metavar="BATCH_BOOTSTRAP_TIMEOUT_MIN", - type=int, - default=30, - help="Timeout(min) for Batch nodes to finish joining main node and bootstrapping" - ) - return parser.parse_args() - - -class PecosBatchJobSubmitter(object): - """Class to submit PECOS Multi-node Processing Batch Jobs. - """ - - _WORKSPACE = "/pecos_workspace" - _JOB_SCRIPT = "run_mnp_job.sh" - - def __init__(self, pecos_batch_job_args): - if not isinstance(pecos_batch_job_args, PecosBatchJobArgs): - raise ValueError(type(pecos_batch_job_args)) - self._args = pecos_batch_job_args - self._batch_client = boto3.client("batch", region_name=self._args.region) - - def _prepare_workspace_cmd(self): - """Prepare workspace folder command""" - return [ - f"sudo mkdir -p {self._WORKSPACE}", - f"sudo chown -R $USER:amazon {self._WORKSPACE}", - f"cd {self._WORKSPACE}", - f"mkdir -p {self._args.input_folder}", - f"mkdir -p {self._args.output_folder}", - f"export PECOS_WORKSPACE={self._WORKSPACE}", - f"export PECOS_INPUT={os.path.join(self._WORKSPACE, self._args.input_folder)}", - f"export PECOS_OUTPUT={os.path.join(self._WORKSPACE, self._args.output_folder)}" - ] - - def _download_input_s3_cmd(self): - """Download input from s3 command""" - return [ - f"aws s3 cp --quiet --recursive {self._args.input_s3_arn} $PECOS_INPUT" - ] - - def _upload_output_s3_cmd(self): - """Upload output to S3 command""" - return [ - f"aws s3 cp --quiet --recursive $PECOS_OUTPUT {self._args.output_s3_arn}" - ] - - def _cleanup_cmd(self): - """Clean up command""" - return ["sudo rm -rf $PECOS_BUILDS_WORKSPACE"] - - def _main_node_job_cmd(self, job_cmd): - """Job execution command only run on main node. - - For multi-node procesing jobs, need to write commands into a bash script on disk, - and set as env BATCH_ENTRY_SCRIPT for scheduler to execute. - The job command consists of doing work and uploading output, - and only main node needs to execute this bash script. - """ - job_script_path = os.path.join(self._WORKSPACE, self._JOB_SCRIPT) - job_cmd = "\n".join(["#!/bin/bash", "set -e"] + job_cmd) - commands = [ - f"echo '{job_cmd}' > {job_script_path}", - f"cat {job_script_path}", - f"chmod 755 {job_script_path}", - f"export BATCH_ENTRY_SCRIPT={job_script_path}", - f"export BATCH_BOOTSTRAP_TIMEOUT={self._args.batch_bootstrap_timeout_min}", - f"/batch-runtime-scripts/entry-point.sh", - ] - return commands - - def _get_batch_job_spec(self, commands): - """Create a Multi-node processing Batch job spec""" - return { - "jobName": self._args.job_name, - "jobQueue": self._args.job_queue, - "jobDefinition": self._args.job_definition, - "nodeOverrides": { - "numNodes": self._args.num_nodes, - "nodePropertyOverrides": [ - { - "targetNodes": "0:", - "containerOverrides": { - "command": ["bash", "-c", " && ".join(commands)], - "resourceRequirements": [ - {"type": "VCPU", "value": str(self._args.cpu)}, - {"type": "MEMORY", "value": str(self._args.memory)}, - ], - }, - } - ], - }, - } - - def submit(self): - """Submit Batch job""" - # Assemble batch job commands - commands = [] - commands += self._prepare_workspace_cmd() - commands += self._download_input_s3_cmd() - commands += self._main_node_job_cmd( - ["pwd", "cd $PECOS_BUILDS_WORKSPACE && ls", self._args.commands] + self._upload_output_s3_cmd() - ) - commands += self._cleanup_cmd() - - # Batch job specs - batch_job_spec = self._get_batch_job_spec(commands) - - batch_job_spec_json = json.dumps(batch_job_spec, indent=2, sort_keys=True) - logging.info(batch_job_spec_json) - - job_id = self._batch_client.submit_job(**batch_job_spec)["jobId"] - logging.info(f"Submitted {self._args.job_name} to job queue {self._args.job_queue} - {job_id}") - - -if __name__ == "__main__": - """ - Prerequisite - ------------ - 1. Setup AWS credentials. - 2. Upload input data to the folder created in PECOS distributed S3 bucket - - - Sample Commands - --------------- - ./submit_job.py \ - --job-name pecos-train-xlinear-eurlex-4k \ - --input-folder input-eurlex-4k \ - --output-folder output-eurlex-4k \ - --num-nodes 2 \ - --cpu 1 \ - --memory 60000 \ - --commands 'mpiexec -n $AWS_BATCH_JOB_NUM_NODES -f /job/hostfile python3 -m pecos.distributed.xmc.xlinear.train \ - -x $PECOS_INPUT/X.trn.npz \ - -y $PECOS_INPUT/Y.trn.npz \ - -m $PECOS_OUTPUT/eurlex_model \ - --nr-splits 2 -b 50 -k 100 -nst 16 -t 0.1 - - python3 -m pecos.xmc.xlinear.predict \ - -x $PECOS_INPUT/X.tst.npz \ - -y $PECOS_INPUT/Y.tst.npz \ - -m $PECOS_OUTPUT/eurlex_model > $PECOS_OUTPUT/eurlex_score.txt' - """ - logging.basicConfig(level=logging.INFO) - - pecos_batch_job_args = PecosBatchJobArgs() - job_submitter = PecosBatchJobSubmitter(pecos_batch_job_args) - job_submitter.submit() diff --git a/examples/MACLR/README.md b/examples/MACLR/README.md deleted file mode 100644 index 9497e458..00000000 --- a/examples/MACLR/README.md +++ /dev/null @@ -1,102 +0,0 @@ -## Preliminaries: Install all the required packages. - -To install all dependencies, run the following command: -``` -pip install -r requirements.txt -``` - - -## Prepare the dataset. - - -* Download datasets following instructions in ``dataset/README.md``. -* The following files should be available in /dataset/: - - trn.json - - tst.json - - lbl.json - - all_pairs.txt - - -## Configure the distributed training of Accelerate - -Accelerate provides a CLI tool that unifies all launcher. To use it, just run -```bash -accelerate config -``` -on your machine and reply to the questions asked. - -Or you can directly use the config file ``accelerate_config.yaml`` when you run the script. - -## Stage I: Multi-scale Adaptive Clustering and Label Regularization - -To run the pre-training Stage I, here is an example on LF-Amazon-131K: -```bash -export dataset=LF-Amazon-131K -export mode=ict -export log=test -export model=bert-base-uncased - -bash run.sh $dataset $mode $log $model -``` - -## Generate pseudo positive pairs using the encoder -You can run the following script to generate pseudo positive pairs from the encoder for self training -```bash -export dataset=LF-Amazon-131K -export mode=construct-pseudo -export log=test -export model= -bash evaluate.sh $dataset $mode $log $model -``` - -Then you will have a file containing top 5 potential labels for each training instance at ``/dataset//pseudo_pos.json``. - -## Stage II: Self-training with pseudo positive pairs -For Stage II, you only need to change ``mode`` to ``self-train`` and provide the path to the current model -```bash -export dataset=LF-Amazon-131K -export mode=self-train -export log=test -export model= - -bash run.sh $dataset $mode $log $model -``` - -## Fine-tune the encoder on few-shot data sampled by label -First, prepare ``label_index.json`` for sampling the subset by label -```bash -python label_index.py --dataset LF-Amazon-131K -``` - -Then change ``mode`` to ``finetune-label`` to start the fine-tuning procedure -```bash -export dataset=LF-Amazon-131K -export mode=finetune-label -export log=test -export model= - -bash run.sh $dataset $mode $log $model -``` - -## Fine-tune the encoder on few-shot data sampled by pair - -Change ``mode`` to ``finetune-pair`` to start the fine-tuning procedure -```bash -export dataset=LF-Amazon-131K -export mode=finetune-pair -export log=test -export model= - -bash run.sh $dataset $mode $log $model -``` - -## Evaluate the pre-trained model -We also provide pre-trained models for all four datasets at [this link](https://archive.org/download/maclr-www22/pretrained-models/). You can download them with ``wget``, decompress the model, and run the following command to evaluate the model -```bash -export dataset=LF-Amazon-131K -export mode=evaluate -export log=test -export model= - -bash evaluate.sh $dataset $mode $log $model -``` \ No newline at end of file diff --git a/examples/MACLR/accelerate_config.yaml b/examples/MACLR/accelerate_config.yaml deleted file mode 100644 index 6063c7fc..00000000 --- a/examples/MACLR/accelerate_config.yaml +++ /dev/null @@ -1,10 +0,0 @@ -compute_environment: LOCAL_MACHINE -deepspeed_config: {} -distributed_type: MULTI_GPU -fp16: false -machine_rank: 0 -main_process_ip: null -main_process_port: null -main_training_function: main -num_machines: 1 -num_processes: 8 diff --git a/examples/MACLR/dataset.py b/examples/MACLR/dataset.py deleted file mode 100644 index 0d6a2319..00000000 --- a/examples/MACLR/dataset.py +++ /dev/null @@ -1,221 +0,0 @@ -import os -import pickle -import json -import random -import torch -import numpy as np -from tqdm import tqdm -from nltk.tokenize import sent_tokenize as sent_tok -from torch.utils.data import Dataset -import logging, sys - -loggers = {} -def get_logger(name='default'): - try: - import loguru - return loguru.logger - except ImportError: - pass - - global loggers - if loggers.get(name): - return loggers.get(name) - else: - logger = logging.getLogger(__name__) - logger.setLevel(logging.DEBUG) - - handler = logging.StreamHandler(sys.stdout) - handler.setLevel(logging.DEBUG) - - # formatter = logging.Formatter( - # fmt='%(levelname)s:%(name)s:%(asctime)s %(message)s', datefmt='%H:%M:%S') - # formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') - formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s') - handler.setFormatter(formatter) - logger.addHandler(handler) - loggers[name] = logger - return logger - -logger = get_logger() -class SimpleDataset(Dataset): - def __init__(self, data, transform=None): - '''Simple dataset - ''' - self.instances = data - self.transform = transform - - def __len__(self): - return len(self.instances) - - def __getitem__(self, index): - if self.transform: - return self.transform(self.instances[index]) - else: - return self.instances[index] - - -class ICTXMCDataset(Dataset): - """ - ICT data generator - """ - def __init__(self, tokenizer, dataset): - self.path = os.path.abspath(os.getcwd()) - self.data_path = os.path.join(self.path, 'dataset', dataset) - self.tokenizer = tokenizer - self.passage_sent_dict, self.titles = self._sent_tokenize_passages(dataset) - self.valid_passage_ids = [pid for pid in self.passage_sent_dict if len(self.passage_sent_dict[pid]) > 1] - self.passageids_set = set(self.passage_sent_dict.keys()) - - # This could be done on the fly to save memory - # self.passage_dict = self._get_pid_to_tok(passage_path) - - def __len__(self): - return len(self.valid_passage_ids) - - def __getitem__(self, idx): - inst_id = self.valid_passage_ids[idx] - inst = self.passage_sent_dict[inst_id] - assert(len(inst) > 1) - label = self.titles[inst_id] - - label_tokens = self.tokenizer.encode(label) - full_inst = " ".join(sent.replace(label, "") if random.random() > 0.1 else sent for sent in inst) - inst_tokens = self.tokenizer.encode(full_inst, max_length=288, truncation=True) - - return label_tokens, inst_tokens, idx - - - def _sent_tokenize_passages(self, dataset): - """Tokenize the passage text into a list of sentences""" - try: - data_path = os.path.join(self.path, 'dataset', dataset) - passages_path = os.path.join(data_path, 'passages.pkl') - titles_path = os.path.join(data_path, 'titles.pkl') - passages = pickle.load(open(passages_path, 'rb')) - titles = pickle.load(open(titles_path, 'rb')) - except: - data_path = os.path.join(self.path, 'dataset', dataset) - json_path = os.path.join(data_path, 'trn.json') - passage_lines = open(json_path, 'r').readlines() - passages = {} - titles = {} - for line in tqdm(passage_lines): - json_line = json.loads(line) - passages[json_line['uid']] = sent_tok(json_line['content']) - titles[json_line['uid']] = json_line['title'] - # save as pickle files - passages_path = os.path.join(data_path, 'passages.pkl') - titles_path = os.path.join(data_path, 'titles.pkl') - pickle.dump(passages, open(passages_path, 'wb')) - pickle.dump(titles, open(titles_path, 'wb')) - return passages, titles - - def _get_pid_to_tok(self, passage_path): - passage_path_cache = passage_path + '.cache.pt' - try: - pid_tok_dict = torch.load(passage_path_cache) - logger.info(f'Loading from {passage_path_cache}!') - except: - passage_dict = self._load_passages(passage_path) - pid_tok_dict = {} - for k, v in passage_dict.items(): - pid_tok_dict[k] = self.tokenizer.encode(v) # roberta tokenizer API might be different - torch.save(pid_tok_dict, passage_path_cache) - logger.info(f'{passage_path_cache} saved!') - return pid_tok_dict - - def _load_passages(self, passage_path): - passage_lines = open(passage_path, 'r').readlines() - passages = {} - for line in tqdm(passage_lines): - psgs = json.loads(line) - if isinstance(psgs, dict): psgs = [psgs] - for psg in psgs: - passages[psg['uid']] = psg['title'] + '\t' + psg['content'] - return passages - -class PosDataset(ICTXMCDataset): - def __init__(self, tokenizer, dataset, labels, mode, sample_pairs=None): - super(PosDataset, self).__init__(tokenizer, dataset) - - self.pos_pair = [] - self.labels = labels - self.mode = mode - if self.mode == 'self-train': - with open(os.path.join(self.data_path, 'pseudo_pos.json')) as fp: - with open(os.path.join(self.data_path, 'pseudo_pos_tfidf.json')) as fp2: - i = 0 - for line, line2 in zip(fp, fp2): - pseudo_pair = json.loads(line.strip()) - pseudo_pair2 = json.loads(line2.strip()) - pid = pseudo_pair['uid'] - if len(self.passage_sent_dict[pid]) < 1: - continue - tfidf_pred = [] - if len(pseudo_pair2['predict_ind'])==0: - tfidf_pred = [] - else: - for tfidf_ind in pseudo_pair2['predict_ind'][:1]: - self.pos_pair.append((pid, tfidf_ind, i)) - tfidf_pred.append(tfidf_ind) - for ind, score in zip(pseudo_pair['predict_ind'][:1], pseudo_pair['score'][:1]): - if score > 36 and ind != tfidf_pred: - # if score > 30 and ind not in tfidf_pred: - self.pos_pair.append((pid, ind, i)) - i = i + 1 - else: - self.pos_pair = sample_pairs - - - def __len__(self): - return len(self.pos_pair) - - def __getitem__(self, idx): - - inst_id = self.pos_pair[idx][0] - label_id = self.pos_pair[idx][1] - idx = self.pos_pair[idx][2] - inst = self.passage_sent_dict[inst_id] - full_inst = self.titles[inst_id] + '\t' + " ".join(sent for sent in inst) - - label = self.labels[label_id] - label_tokens = self.tokenizer.encode(label) - inst_tokens = self.tokenizer.encode(full_inst, max_length=288, truncation=True) - return label_tokens, inst_tokens, idx - -def padding_util(examples, padding_id, seq_len): - length = max([len(example) for example in examples]) - length = min(length, seq_len) - batch = np.ones((len(examples), length)) * padding_id - for i, example in enumerate(examples): - idx = min(len(example), length) - batch[i, :idx] = example[:idx - 1] + [example[-1]] - return torch.tensor(batch, dtype=torch.long) - - -def ICT_batchify(examples, padding_id=0, max_label_len=64, max_instance_len=288): - """ - batch_size x query_length, num_passages x passage_length, batch_size x1 (labels) - """ - batch_size = len(examples) - label_len = max([len(example[0]) for example in examples]) - label_len = min(label_len, max_label_len) - label_tokens = np.ones((batch_size, label_len)) * padding_id - indices = np.zeros(batch_size) - instances = [] - count = 0 - for i, example in enumerate(examples): - idx = min(len(example[0]), label_len) - label_tokens[i, :idx] = example[0][:idx - 1] + [example[0][-1]] - inst = example[1] - instances.append(inst) - count += 1 - indices[i] = example[2] - - instance_len = max([len(inst) for inst in instances]) - instance_len = min(instance_len, max_instance_len) - inst_tokens = np.ones((count, instance_len)) * padding_id - for i, inst in enumerate(instances): - idx = min(len(inst), instance_len) - inst_tokens[i, :idx] = inst[: idx - 1] + [inst[-1]] - return torch.tensor(label_tokens, dtype=torch.long), torch.tensor(inst_tokens, dtype=torch.long), torch.tensor(indices, dtype=torch.long) \ No newline at end of file diff --git a/examples/MACLR/dataset/README.md b/examples/MACLR/dataset/README.md deleted file mode 100644 index c00bb274..00000000 --- a/examples/MACLR/dataset/README.md +++ /dev/null @@ -1,5 +0,0 @@ -This is where all the datasets are saved. Four datasets used in the paper can be downloaded via -``` -dataset=${your_dataset} -bash download_data.sh ${dataset} -``` \ No newline at end of file diff --git a/examples/MACLR/dataset/download_data.sh b/examples/MACLR/dataset/download_data.sh deleted file mode 100644 index 4c54bbb2..00000000 --- a/examples/MACLR/dataset/download_data.sh +++ /dev/null @@ -1,8 +0,0 @@ -dataset=$1 -if [ ${dataset} != "LF-Amazon-131K" ] && [ ${dataset} != "LF-WikiSeeAlso-320K" ] && [ ${dataset} != "LF-Wikipedia-500K" ] && [ ${dataset} != "LF-Amazon-1M" ]; then - echo "dataset=${dataset} is not yet supported!" - exit -fi - -wget https://archive.org/download/maclr-www22/datasets/${dataset}.tar.gz -tar -zxvf ${dataset}.tar.gz \ No newline at end of file diff --git a/examples/MACLR/evaluate.py b/examples/MACLR/evaluate.py deleted file mode 100644 index 451eb4b1..00000000 --- a/examples/MACLR/evaluate.py +++ /dev/null @@ -1,145 +0,0 @@ -import argparse -import os, sys -import logging -import json -import csv -from tqdm.auto import tqdm -import torch -from torch.utils.data import DataLoader, SequentialSampler -import sentence_transformers as sent_trans -import transformers -from transformers import set_seed -import accelerate -from accelerate import Accelerator -from dataset import SimpleDataset, padding_util -from model import build_encoder, DualEncoderModel -from utils import perform_eval, eval_and_cluster -import pandas as pd -import warnings -from main import parse_args -warnings.filterwarnings("ignore") - -os.environ["TOKENIZERS_PARALLELISM"] = "false" -logger = logging.getLogger(__name__) - -# Initialize the accelerator. We will let the accelerator handle device placement for us in this example. -args = parse_args() -distributed_args = accelerate.DistributedDataParallelKwargs(find_unused_parameters=True) -accelerator = Accelerator(kwargs_handlers=[distributed_args]) -device = accelerator.device -# Make one log on every process with the configuration for debugging. -logging.basicConfig( - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", - filename=f'xmc_{args.dataset}_{args.log}_evaluate.log', - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -logger.info(accelerator.state) - -# Setup logging, we only want one process per machine to log things on the screen. -# accelerator.is_local_main_process is only True for one process per machine. -logger.setLevel(logging.INFO if accelerator.is_local_main_process else logging.ERROR) -ch = logging.StreamHandler(sys.stdout) -logger.addHandler(ch) -if accelerator.is_local_main_process: - transformers.utils.logging.set_verbosity_info() -else: - transformers.utils.logging.set_verbosity_error() - -logger.info(sent_trans.__file__) - -# If passed along, set the training seed now. -if args.seed is not None: - set_seed(args.seed) - -# Load pretrained model and tokenizer -if args.model_name_or_path == 'bert-base-uncased' or args.model_name_or_path == 'sentence-transformers/paraphrase-mpnet-base-v2': - label_encoder = build_encoder( - args.model_name_or_path, - args.max_label_length, - args.pooling_mode, - args.proj_emb_dim, - ) -else: - label_encoder = sent_trans.SentenceTransformer(args.model_name_or_path) - -tokenizer = label_encoder._first_module().tokenizer - -instance_encoder = label_encoder - -model = DualEncoderModel( - label_encoder, - instance_encoder, -) -model = model.to(device) - -# the whole label set -data_path = os.path.join(os.path.abspath(os.getcwd()), 'dataset', args.dataset) -all_labels = pd.read_json(os.path.join(data_path, 'lbl.json'),lines=True) -label_list = list(all_labels.title) -label_ids = list(all_labels.uid) -label_data = SimpleDataset(label_list, transform=tokenizer.encode) - -# label dataloader for searching -sampler = SequentialSampler(label_data) -label_padding_func = lambda x: padding_util(x, tokenizer.pad_token_id, 64) -label_dataloader = DataLoader(label_data, sampler=sampler, batch_size=16, collate_fn=label_padding_func) - -# test data -data_path = os.path.join(os.path.abspath(os.getcwd()), 'dataset', args.dataset) -try: - accelerator.print("load cache") - all_instances = torch.load(os.path.join(data_path, 'all_passages_with_titles.json.cache.pt')) - test_data = SimpleDataset(all_instances.values()) -except: - if args.mode == 'construct-pseudo': - test_path = os.path.join(data_path, 'trn.json') - else: - test_path = os.path.join(data_path, 'tst.json') - all_instances = {} - test_ids = [] - with open(test_path) as fp: - for line in fp: - inst = json.loads(line.strip()) - all_instances[inst['uid']] = inst['title'] + '\t' + inst['content'] - test_ids.append(inst['uid']) - simple_transform = lambda x: tokenizer.encode(x, max_length=288, truncation=True) - test_data = SimpleDataset(list(all_instances.values()), transform=simple_transform) - inst_num = len(test_data) - -sampler = SequentialSampler(test_data) -sent_padding_func = lambda x: padding_util(x, tokenizer.pad_token_id, 288) -instance_dataloader = DataLoader(test_data, sampler=sampler, batch_size=128, collate_fn=sent_padding_func) - -# Prepare everything with our `accelerator`. -model, label_dataloader, instance_dataloader = accelerator.prepare(model, label_dataloader, instance_dataloader) - -if args.mode == 'construct-pseudo': - D, I, _ = perform_eval(accelerator.unwrap_model(model), label_dataloader, label_ids, instance_dataloader, inst_num, test_ids, accelerator) - pseudo_pair_path = os.path.join(data_path, 'pseudo_pos.json') - if accelerator.is_local_main_process: - with open(pseudo_pair_path, 'w') as f: - for row_id in tqdm(range(inst_num)): - inst_id = test_ids[row_id] - item = {'uid': inst_id} - predict_target = [] - predict_score = [] - for col_id, score in zip(I[row_id][:5], D[row_id][:5]): - predict_target.append(int(col_id)) - predict_score.append(float(score)) - item['predict_ind'] = predict_target - item['score'] = predict_score - f.write(json.dumps(item) + '\n') - -else: - # prepare pairs - reader = csv.reader(open(os.path.join(data_path, 'all_pairs.txt'), encoding="utf-8"), delimiter=" ") - qrels = {} - for id, row in enumerate(reader): - query_id, corpus_id, score = row[0], row[1], int(row[2]) - if query_id not in qrels: - qrels[query_id] = {corpus_id: score} - else: - qrels[query_id][corpus_id] = score - eval_and_cluster(args, logger, 0, accelerator.unwrap_model(model), label_dataloader, label_ids, - instance_dataloader, inst_num, test_ids, qrels, accelerator) \ No newline at end of file diff --git a/examples/MACLR/evaluate.sh b/examples/MACLR/evaluate.sh deleted file mode 100644 index 0ea3e5bc..00000000 --- a/examples/MACLR/evaluate.sh +++ /dev/null @@ -1,9 +0,0 @@ -dataset=$1 -log=$2 -mode=$3 -model=$4 -accelerate launch --config_file accelerate_config.yaml evaluate.py \ - --dataset $dataset \ - --log $log \ - --mode $mode \ - --model-name-or-path $model \ diff --git a/examples/MACLR/label_index.py b/examples/MACLR/label_index.py deleted file mode 100644 index eadfd59c..00000000 --- a/examples/MACLR/label_index.py +++ /dev/null @@ -1,32 +0,0 @@ -import pandas as pd -import json -import os -import argparse - -parser = argparse.ArgumentParser() -parser.add_argument( - "--dataset", - type=str, default="LF-Amazon-131K", - help="the dataset to run the experiments", -) -args = parser.parse_args() - -data_path = os.path.join(os.path.abspath(os.getcwd()), 'dataset', args.dataset) -label_index = {} -all_labels = pd.read_json(os.path.join(data_path, 'lbl.json'),lines=True) -label_ids = list(all_labels.uid) -with open(os.path.join(data_path, 'trn.json')) as fp: - for line in fp: - item = json.loads(line.strip()) - pid = item['uid'].strip() - for ind in item['target_ind']: - label_id = label_ids[ind].strip() - if label_id in label_index: - label_index[label_id].append(pid) - else: - label_index[label_id] = [ind, pid] - -with open(os.path.join(data_path, 'label_index.json'), 'w') as f: - for k, v in label_index.items(): - item = {'uid': k, 'ind': v[0], 'instance': v[1:]} - f.write(json.dumps(item) + '\n') \ No newline at end of file diff --git a/examples/MACLR/loss.py b/examples/MACLR/loss.py deleted file mode 100644 index 86a7da30..00000000 --- a/examples/MACLR/loss.py +++ /dev/null @@ -1,69 +0,0 @@ -import torch -import torch.distributed as dist -import torch.nn.functional as F -import mpu_utils - -def compute_loss(mask, logits_mask, logits): - exp_logits = torch.exp(logits) * logits_mask - softmax_scores = logits - torch.log(exp_logits.sum(1, keepdim=True)) - mean_log_prob_pos = (mask * softmax_scores).sum(1) / mask.sum(1) - return -mean_log_prob_pos.mean() - -def loss_function(label_emb, inst_emb, labels, accelerator): - assert label_emb.shape[0] == inst_emb.shape[0], "{} is not equal to {}".format(label_emb.shape[0], inst_emb.shape[0]) - assert label_emb.shape[1] == inst_emb.shape[1] - local_batch_size = label_emb.shape[0] - - # [global_batch_size, hidden_dim] - global_batch_size = dist.get_world_size() * local_batch_size - all_label_emb = mpu_utils.AllgatherFromDataParallelRegion.apply(label_emb) - all_inst_emb = mpu_utils.AllgatherFromDataParallelRegion.apply(inst_emb) - a_norm = all_label_emb - b_norm = all_inst_emb - retrieval_scores = torch.mm(b_norm, a_norm.transpose(0, 1)) - qd_max, _ = torch.max(retrieval_scores, dim=1, keepdim=True) - qd_stable_scores = retrieval_scores - qd_max.detach() - softmax_scores = F.log_softmax(qd_stable_scores, dim=1) - - labels = torch.arange(global_batch_size).long().to(accelerator.device) - loss = F.nll_loss(softmax_scores, labels, reduction='mean') - reduced_losses = mpu_utils.average_losses_across_data_parallel_group([loss]) - stats_dict = dict(loss=reduced_losses[0]) - return loss, stats_dict - -def loss_function_reg(label_emb, inst_emb, inst_emb_aug, reg_emb, labels, accelerator): - assert label_emb.shape[0] == inst_emb.shape[0], "{} is not equal to {}".format(label_emb.shape[0], inst_emb.shape[0]) - assert label_emb.shape[1] == inst_emb.shape[1] - - all_label_emb = mpu_utils.AllgatherFromDataParallelRegion.apply(label_emb) - all_inst_emb = mpu_utils.AllgatherFromDataParallelRegion.apply(inst_emb) - all_inst_emb_aug = mpu_utils.AllgatherFromDataParallelRegion.apply(inst_emb_aug) - all_reg_emb = mpu_utils.AllgatherFromDataParallelRegion.apply(reg_emb) - - labels = labels.contiguous().view(-1, 1) - all_labels = accelerator.gather(labels) - num_inst = all_label_emb.shape[0] - num_reg = all_reg_emb.shape[0] - - mask = torch.eq(all_labels, all_labels.transpose(0, 1)).float() - zero_mask = torch.zeros(num_inst, num_reg).to(accelerator.device) - - a_norm = all_label_emb - b_norm = all_inst_emb - inst_lbl_scores = torch.mm(b_norm, a_norm.transpose(0, 1)) - inst_lbl_max, _ = torch.max(inst_lbl_scores, dim=1, keepdim=True) - inst_lbl_stable_scores = inst_lbl_scores - inst_lbl_max.detach() - - c_norm = torch.cat([all_inst_emb_aug, all_reg_emb]) - real_scores = torch.mm(b_norm, c_norm.transpose(0, 1)) - real_max, _ = torch.max(real_scores, dim=1, keepdim=True) - real_stable_scores = real_scores - real_max.detach() - real_mask = torch.cat([mask, zero_mask], dim=1) - - contrast_loss = compute_loss(mask, torch.ones_like(mask), inst_lbl_stable_scores) - reg_loss = compute_loss(real_mask, torch.ones_like(real_mask), real_stable_scores) - loss = contrast_loss + 1*reg_loss - reduced_losses = mpu_utils.average_losses_across_data_parallel_group([loss, contrast_loss, reg_loss]) - - stats_dict = dict(loss=reduced_losses[0], contrast_loss=reduced_losses[1], reg_loss=reduced_losses[2]) - return loss, stats_dict \ No newline at end of file diff --git a/examples/MACLR/main.py b/examples/MACLR/main.py deleted file mode 100644 index c97c360b..00000000 --- a/examples/MACLR/main.py +++ /dev/null @@ -1,439 +0,0 @@ -import argparse -import logging -import math -import os, sys -import pickle -import random -import json -from tqdm.auto import tqdm -import numpy as np -from torch.utils.data import DataLoader, RandomSampler, SequentialSampler -import torch -import torch.nn as nn -import torch.distributed as dist -import torch.nn.functional as F -import transformers -import sentence_transformers as sent_trans -import accelerate -from accelerate import Accelerator -from transformers import ( - AdamW, - SchedulerType, - get_scheduler, - set_seed, -) -from utils import eval_and_cluster -from dataset import SimpleDataset, ICTXMCDataset, PosDataset, ICT_batchify, padding_util -from loss import loss_function_reg, loss_function -from model import build_encoder, DualEncoderModel -import csv -import pandas as pd -import warnings -warnings.filterwarnings("ignore") - -os.environ["TOKENIZERS_PARALLELISM"] = "false" -logger = logging.getLogger(__name__) -retriever_report_topk_accuracies = [1, 5, 10, 20] - - -def parse_args(): - parser = argparse.ArgumentParser(description="Pretrain two-tower Transformer models with ICT") - # Data - parser.add_argument( - "--corpus-pkl-path", - type=str, - help="a processed pickle file that contains title_list and block_list", - ) - parser.add_argument( - "--mode", - type=str, default="ict", - help="the mode of the training procedure", - ) - parser.add_argument( - "--dataset", - type=str, default="LF-Amazon-131K", - help="the dataset to run the experiments", - ) - parser.add_argument( - "--log", - type=str, default="test", - help="log file", - ) - parser.add_argument( - "--ratio", - type=float, default=0.01, - help="Sampling ratio", - ) - # Model - parser.add_argument( - "--model-name-or-path", - type=str, default="bert-base-uncased", - help="Path to pretrained model or model identifier from huggingface.co/models.", - ) - parser.add_argument( - "--max-label-length", - type=int, default=64, - help="maximum label length for pre-training (default: 64)", - ) - parser.add_argument( - "--max-inst-length", - type=int, default=288, - help="maximum block (i.e., title + text) length for pre-training (default: 288)", - ) - parser.add_argument( - "--pooling-mode", - type=str, default="cls", - help="Can be a string: mean/max/cls.", - ) - parser.add_argument( - "--proj-emb-dim", - type=int, default=512, - help="embedding size of the projection layer in two-tower models", - ) - # Optimizer - parser.add_argument( - "--per-device-train-batch-size", - type=int, default=16, - help="training batch size per GPU device (default: 8)", - ) - parser.add_argument( - "--learning-rate", - type=float, default=3e-5, - help="Initial learning rate (after the potential warmup period) to use.", - ) - parser.add_argument( - "--weight-decay", - type=float, default=0, - help="Weight decay to use.", - ) - parser.add_argument( - "--max-train-steps", - type=int, default=50000, - help="Total number of training steps to perform (default: 10,000)", - ) - parser.add_argument( - "--gradient-accumulation-steps", - type=int, default=1, - help="Number of updates steps to accumulate before performing a backward/update pass.", - ) - parser.add_argument( - "--lr-scheduler-type", - type=SchedulerType, - default="linear", - help="The scheduler type to use.", - choices=["linear", "cosine", "cosine_with_restarts", "polynomial", "constant", "constant_with_warmup"], - ) - parser.add_argument( - "--num-warmup-steps", - type=int, default=2000, - help="Number of steps for the warmup in the lr scheduler (default: 1,000)", - ) - parser.add_argument( - "--logging-steps", - type=int, default=100, - help="Number of steps for the logging information (default 100)", - ) - parser.add_argument( - "--eval-steps", - type=int, default=1000, - help="Number of steps for evaluation (default 100)", - ) - parser.add_argument( - "--saving-steps", - type=int, default=2000, - help="Number of steps for the saving checkpoint (default 1000)", - ) - # Output - parser.add_argument( - "--output-dir", - type=str, default='ckpt', - help="Where to store the final model.", - ) - parser.add_argument( - "--seed", - type=int, default=None, - help="A seed for reproducible training.", - ) - args = parser.parse_args() - # sanity check - args.output_dir = os.path.join(os.path.abspath(os.getcwd()), args.output_dir) - if args.output_dir is not None: - os.makedirs(args.output_dir, exist_ok=True) - return args - -def main(): - # Initialize the accelerator. We will let the accelerator handle device placement for us in this example. - args = parse_args() - distributed_args = accelerate.DistributedDataParallelKwargs(find_unused_parameters=True) - accelerator = Accelerator(kwargs_handlers=[distributed_args]) - device = accelerator.device - # Make one log on every process with the configuration for debugging. - logging.basicConfig( - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", - filename=f'xmc_{args.dataset}_{args.mode}_{args.log}.log', - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, - ) - logger.info(accelerator.state) - - # Setup logging, we only want one process per machine to log things on the screen. - # accelerator.is_local_main_process is only True for one process per machine. - logger.setLevel(logging.INFO if accelerator.is_local_main_process else logging.ERROR) - ch = logging.StreamHandler(sys.stdout) - logger.addHandler(ch) - if accelerator.is_local_main_process: - transformers.utils.logging.set_verbosity_info() - else: - transformers.utils.logging.set_verbosity_error() - - logger.info(sent_trans.__file__) - - # If passed along, set the training seed now. - if args.seed is not None: - set_seed(args.seed) - - # Load pretrained model and tokenizer - if args.model_name_or_path == 'bert-base-uncased' or args.model_name_or_path == 'sentence-transformers/paraphrase-mpnet-base-v2': - query_encoder = build_encoder( - args.model_name_or_path, - args.max_label_length, - args.pooling_mode, - args.proj_emb_dim, - ) - else: - query_encoder = sent_trans.SentenceTransformer(args.model_name_or_path) - - tokenizer = query_encoder._first_module().tokenizer - - block_encoder = query_encoder - - model = DualEncoderModel( - query_encoder, - block_encoder, - args.mode - ) - model = model.to(device) - - # the whole label set - data_path = os.path.join(os.path.abspath(os.getcwd()), 'dataset', args.dataset) - all_labels = pd.read_json(os.path.join(data_path, 'lbl.json'),lines=True) - label_list = list(all_labels.title) - label_ids = list(all_labels.uid) - label_data = SimpleDataset(label_list, transform=tokenizer.encode) - - # label dataloader for searching - sampler = SequentialSampler(label_data) - label_padding_func = lambda x: padding_util(x, tokenizer.pad_token_id, 64) - label_dataloader = DataLoader(label_data, sampler=sampler, batch_size=16, collate_fn=label_padding_func) - - # label dataloader for regularization - reg_sampler = RandomSampler(label_data) - reg_dataloader = DataLoader(label_data, sampler=reg_sampler, batch_size=4, collate_fn=label_padding_func) - - if args.mode == 'ict': - train_data = ICTXMCDataset(tokenizer=tokenizer, dataset=args.dataset) - elif args.mode == 'self-train': - train_data = PosDataset(tokenizer=tokenizer, dataset=args.dataset, labels=label_list, mode=args.mode) - elif args.mode == 'finetune-pair': - train_path = os.path.join(data_path, 'trn.json') - pos_pair = [] - with open(train_path) as fp: - for i, line in enumerate(fp): - inst = json.loads(line.strip()) - inst_id = inst['uid'] - for ind in inst['target_ind']: - pos_pair.append((inst_id, ind, i)) - dataset_size = len(pos_pair) - indices = list(range(dataset_size)) - split = int(np.floor(args.ratio * dataset_size)) - np.random.shuffle(indices) - train_indices = indices[:split] - torch.distributed.broadcast_object_list(train_indices, src=0, group=None) - sample_pairs = [pos_pair[i] for i in train_indices] - train_data = PosDataset(tokenizer=tokenizer, dataset=args.dataset, labels=label_list, mode=args.mode, sample_pairs=sample_pairs) - elif args.mode == 'finetune-label': - label_index = [] - label_path = os.path.join(data_path, 'label_index.json') - with open(label_path) as fp: - for line in fp: - label_index.append(json.loads(line.strip())) - np.random.shuffle(label_index) - sample_size = int(np.floor(args.ratio*len(label_index))) - sample_label = label_index[:sample_size] - torch.distributed.broadcast_object_list(sample_label, src=0, group=None) - sample_pairs = [] - for i, label in enumerate(sample_label): - ind = label['ind'] - for inst_id in label['instance']: - sample_pairs.append((inst_id, ind, i)) - train_data = PosDataset(tokenizer=tokenizer, dataset=args.dataset, labels=label_list, mode=args.mode, sample_pairs=sample_pairs) - - train_sampler = RandomSampler(train_data) - padding_func = lambda x: ICT_batchify(x, tokenizer.pad_token_id, 64, 288) - train_dataloader = torch.utils.data.DataLoader(train_data, sampler=train_sampler, batch_size=args.per_device_train_batch_size, - num_workers=4, pin_memory=False, - collate_fn=padding_func) - - - try: - accelerator.print("load cache") - all_instances = torch.load(os.path.join(data_path, 'all_passages_with_titles.json.cache.pt')) - test_data = SimpleDataset(all_instances.values()) - except: - all_instances = {} - test_path = os.path.join(data_path, 'tst.json') - if args.mode == 'ict': - train_path = os.path.join(data_path, 'trn.json') - train_instances = {} - valid_passage_ids = train_data.valid_passage_ids - with open(train_path) as fp: - for line in fp: - inst = json.loads(line.strip()) - train_instances[inst['uid']] = inst['title'] + '\t' + inst['content'] - for inst_id in valid_passage_ids: - all_instances[inst_id] = train_instances[inst_id] - test_ids = [] - with open(test_path) as fp: - for line in fp: - inst = json.loads(line.strip()) - all_instances[inst['uid']] = inst['title'] + '\t' + inst['content'] - test_ids.append(inst['uid']) - simple_transform = lambda x: tokenizer.encode(x, max_length=288, truncation=True) - test_data = SimpleDataset(list(all_instances.values()), transform=simple_transform) - inst_num = len(test_data) - - sampler = SequentialSampler(test_data) - sent_padding_func = lambda x: padding_util(x, tokenizer.pad_token_id, 288) - instance_dataloader = DataLoader(test_data, sampler=sampler, batch_size=128, collate_fn=sent_padding_func) - - - # prepare pairs - reader = csv.reader(open(os.path.join(data_path, 'all_pairs.txt'), encoding="utf-8"), delimiter=" ") - qrels = {} - for id, row in enumerate(reader): - query_id, corpus_id, score = row[0], row[1], int(row[2]) - if query_id not in qrels: - qrels[query_id] = {corpus_id: score} - else: - qrels[query_id][corpus_id] = score - - logging.info("| |ICT_dataset|={} pairs.".format(len(train_data))) - - - # Optimizer - # Split weights in two groups, one with weight decay and the other not. - no_decay = ["bias", "LayerNorm.weight"] - optimizer_grouped_parameters = [ - { - "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], - "weight_decay": args.weight_decay, - }, - { - "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], - "weight_decay": 0.0, - }, - ] - optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=1e-8) - - # Prepare everything with our `accelerator`. - model, optimizer, train_dataloader, label_dataloader, reg_dataloader, instance_dataloader = accelerator.prepare( - model, optimizer, train_dataloader, label_dataloader, reg_dataloader, instance_dataloader) - - # Scheduler and math around the number of training steps. - num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps) - # args.max_train_steps = 100000 - args.num_train_epochs = math.ceil(args.max_train_steps / num_update_steps_per_epoch) - args.num_warmup_steps = int(0.1*args.max_train_steps) - lr_scheduler = get_scheduler( - name=args.lr_scheduler_type, - optimizer=optimizer, - num_warmup_steps=args.num_warmup_steps, - num_training_steps=args.max_train_steps, - ) - - # Train! - total_batch_size = args.per_device_train_batch_size * accelerator.num_processes * args.gradient_accumulation_steps - logger.info("***** Running training *****") - logger.info(f" Num examples = {len(train_data)}") - logger.info(f" Num Epochs = {args.num_train_epochs}") - logger.info(f" Instantaneous batch size per device = {args.per_device_train_batch_size}") - logger.info(f" Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}") - logger.info(f" Gradient Accumulation steps = {args.gradient_accumulation_steps}") - logger.info(f" Learning Rate = {args.learning_rate}") - logger.info(f" Total optimization steps = {args.max_train_steps}") - # Only show the progress bar once on each machine. - progress_bar = tqdm(range(args.max_train_steps), disable=not accelerator.is_local_main_process) - completed_steps = 0 - from torch.cuda.amp import autocast - scaler = torch.cuda.amp.GradScaler() - cluster_result = eval_and_cluster(args, logger, completed_steps, accelerator.unwrap_model(model), - label_dataloader, label_ids, instance_dataloader, inst_num, test_ids, qrels, accelerator) - reg_iter = iter(reg_dataloader) - trial_name = f"dim-{args.proj_emb_dim}-bs-{args.per_device_train_batch_size}-{args.dataset}-{args.log}-{args.mode}" - for epoch in range(args.num_train_epochs): - model.train() - for step, batch in enumerate(train_dataloader): - batch = tuple(t for t in batch) - label_tokens, inst_tokens, indices = batch - if args.mode == 'ict': - try: - reg_data = next(reg_iter) - except StopIteration: - reg_iter = iter(reg_dataloader) - reg_data = next(reg_iter) - - if cluster_result is not None: - pseudo_labels = cluster_result[indices] - else: - pseudo_labels = indices - with autocast(): - if args.mode == 'ict': - label_emb, inst_emb, inst_emb_aug, reg_emb = model(label_tokens, inst_tokens, reg_data) - loss, stats_dict = loss_function_reg(label_emb, inst_emb, inst_emb_aug, reg_emb, pseudo_labels, accelerator) - else: - label_emb, inst_emb = model(label_tokens, inst_tokens, reg_data=None) - loss, stats_dict = loss_function(label_emb, inst_emb, pseudo_labels, accelerator) - loss = loss / args.gradient_accumulation_steps - - scaler.scale(loss).backward() - scaler.unscale_(optimizer) - torch.nn.utils.clip_grad_norm_(model.parameters(), 1) - if step % args.gradient_accumulation_steps == 0 or step == len(train_dataloader) - 1: - scaler.step(optimizer) - scaler.update() - lr_scheduler.step() - optimizer.zero_grad() - progress_bar.update(1) - completed_steps += 1 - - if completed_steps % args.logging_steps == 0: - if args.mode == 'ict': - logger.info("| Epoch [{:4d}/{:4d}] Step [{:8d}/{:8d}] Total Loss {:.6e} Contrast Loss {:.6e} Reg Loss {:.6e}".format( - epoch, args.num_train_epochs, - completed_steps, args.max_train_steps, - stats_dict["loss"].item(), - stats_dict["contrast_loss"].item(), - stats_dict["reg_loss"].item(), - ) - ) - else: - logger.info("| Epoch [{:4d}/{:4d}] Step [{:8d}/{:8d}] Total Loss {:.6e}".format( - epoch, args.num_train_epochs, - completed_steps, args.max_train_steps, - stats_dict["loss"].item(), - ) - ) - if completed_steps % args.eval_steps == 0: - cluster_result = eval_and_cluster(args, logger, completed_steps, accelerator.unwrap_model(model), - label_dataloader, label_ids, instance_dataloader, inst_num, test_ids, qrels, accelerator) - unwrapped_model = accelerator.unwrap_model(model) - - unwrapped_model.label_encoder.save(f"{args.output_dir}/{trial_name}/label_encoder") - unwrapped_model.instance_encoder.save(f"{args.output_dir}/{trial_name}/instance_encoder") - - if completed_steps >= args.max_train_steps: - break - -if __name__ == "__main__": - main() - diff --git a/examples/MACLR/model.py b/examples/MACLR/model.py deleted file mode 100644 index cc9ceec0..00000000 --- a/examples/MACLR/model.py +++ /dev/null @@ -1,69 +0,0 @@ -import torch -import torch.nn as nn -import sentence_transformers as sent_trans -class DualEncoderModel(torch.nn.Module): - def __init__(self, label_encoder, instance_encoder, mode='ict'): - super(DualEncoderModel, self).__init__() - self.label_encoder = label_encoder - self.instance_encoder = instance_encoder - self.mode = mode - - def forward(self, label_tokens, inst_tokens, reg_data): - # [local_batch_size, query_seq_len] - label_att_mask = ~(label_tokens.eq(0)) - inst_att_mask = ~(inst_tokens.eq(0)) - - - label_f = {'input_ids': label_tokens, 'attention_mask': label_att_mask} - label_emb = self.label_encoder(label_f) - label_emb = label_emb["sentence_embedding"] - - inst_f = {'input_ids': inst_tokens, 'attention_mask': inst_att_mask} - inst_emb = self.instance_encoder(inst_f) - inst_emb = inst_emb["sentence_embedding"] - - if self.mode == 'ict': - reg_att_mask = ~(reg_data.eq(0)) - reg_f = {'input_ids': reg_data, 'attention_mask': reg_att_mask} - reg_emb = self.label_encoder(reg_f) - reg_emb = reg_emb["sentence_embedding"] - - inst_f_aug = {'input_ids': inst_tokens.detach().clone(), 'attention_mask': inst_att_mask.detach().clone()} - inst_emb_aug = self.instance_encoder(inst_f_aug) - inst_emb_aug = inst_emb_aug["sentence_embedding"] - return label_emb, inst_emb, inst_emb_aug, reg_emb - else: - return label_emb, inst_emb - -def build_encoder( - model_name_or_path, - max_seq_length, - pooling_mode, - proj_emb_dim, - drop_prob=0.1, -): - base_layer = sent_trans.models.Transformer(model_name_or_path, max_seq_length=None) - pooling_layer = sent_trans.models.Pooling( - base_layer.get_word_embedding_dimension(), - pooling_mode=pooling_mode, - ) - dense_layer = sent_trans.models.Dense( - in_features=pooling_layer.get_sentence_embedding_dimension(), - out_features=proj_emb_dim, - activation_function=nn.Tanh(), - ) - # normalize_layer = sent_trans.models.LayerNorm(proj_emb_dim) - normalize_layer = sent_trans.models.Normalize() - dropout_layer = sent_trans.models.Dropout(dropout=drop_prob) - proj_layer = sent_trans.models.Dense( - in_features=512, - out_features=128, - activation_function=nn.Tanh(), - ) - # encoder = sent_trans.SentenceTransformer( - # modules=[base_layer, pooling_layer, dense_layer, normalize_layer, dropout_layer], - # ) - encoder = sent_trans.SentenceTransformer( - modules=[base_layer, pooling_layer, dense_layer], -) - return encoder \ No newline at end of file diff --git a/examples/MACLR/mpu_utils.py b/examples/MACLR/mpu_utils.py deleted file mode 100644 index c75b79ca..00000000 --- a/examples/MACLR/mpu_utils.py +++ /dev/null @@ -1,84 +0,0 @@ - -import torch -import torch.nn as nn -import torch.distributed as dist - - - -# Intra-layer model parallel group that the current rank belongs to. -_TENSOR_MODEL_PARALLEL_GROUP = None -# Inter-layer model parallel group that the current rank belongs to. -_PIPELINE_MODEL_PARALLEL_GROUP = None -# Model parallel group (both intra- and pipeline) that the current rank belongs to. -_MODEL_PARALLEL_GROUP = None -# Embedding group. -_EMBEDDING_GROUP = None -# Data parallel group that the current rank belongs to. -_DATA_PARALLEL_GROUP = None - -_VIRTUAL_PIPELINE_MODEL_PARALLEL_RANK = None -_VIRTUAL_PIPELINE_MODEL_PARALLEL_WORLD_SIZE = None - -# These values enable us to change the mpu sizes on the fly. -_MPU_TENSOR_MODEL_PARALLEL_WORLD_SIZE = None -_MPU_PIPELINE_MODEL_PARALLEL_WORLD_SIZE = None -_MPU_TENSOR_MODEL_PARALLEL_RANK = None -_MPU_PIPELINE_MODEL_PARALLEL_RANK = None - -# A list of global ranks for each pipeline group to ease calculation of the source -# rank when broadcasting from the first or last pipeline stage -_PIPELINE_GLOBAL_RANKS = None - - -def get_data_parallel_group(): - """Get the data parallel group the caller rank belongs to.""" - #assert _DATA_PARALLEL_GROUP is not None, f"data parallel group={_DATA_PARALLEL_GROUP}, which is not initialized" - return _DATA_PARALLEL_GROUP - - -def get_group_world_size_rank(): - group = get_data_parallel_group() - rank = torch.distributed.get_rank(group=group) - world_size = torch.distributed.get_world_size(group=group) - return group, rank, world_size - - -def get_data_parallel_world_size(): - """Return world size for the data parallel group.""" - return torch.distributed.get_world_size(group=get_data_parallel_group()) - - -def get_data_parallel_rank(): - """Return my rank for the data parallel group.""" - return torch.distributed.get_rank(group=get_data_parallel_group()) - - -class AllgatherFromDataParallelRegion(torch.autograd.Function): - """ https://github.com/NVIDIA/Megatron-LM/blob/main/pretrain_ict.py#L57 """ - @staticmethod - def forward(ctx, input_): - assert input_.dim() == 2 - group, rank, world_size = get_group_world_size_rank() - tensor_list = [torch.empty_like(input_) for _ in range(world_size)] - torch.distributed.all_gather(tensor_list, input_, group=group) - tensor_list[rank] = input_ - output = torch.cat(tensor_list, dim=0).contiguous() - return output - - @staticmethod - def backward(ctx, grad_output): - group, rank, world_size = get_group_world_size_rank() - assert grad_output.shape[0] % world_size == 0 - dim_size = grad_output.shape[0] // world_size - output_list = torch.split(grad_output, dim_size, dim=0) - # get chunk from this rank - output = output_list[rank].contiguous() - return output - - -def average_losses_across_data_parallel_group(losses): - """Reduce a tensor of losses across all GPUs.""" - averaged_losses = torch.cat([loss.clone().detach().view(1) for loss in losses]) - torch.distributed.all_reduce(averaged_losses, group=get_data_parallel_group()) - averaged_losses = averaged_losses / torch.distributed.get_world_size(group=get_data_parallel_group()) - return averaged_losses diff --git a/examples/MACLR/requirements.txt b/examples/MACLR/requirements.txt deleted file mode 100644 index 807044fa..00000000 --- a/examples/MACLR/requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -torch==1.8.0 -beir -accelerate -nltk -pandas -tqdm -faiss-gpu \ No newline at end of file diff --git a/examples/MACLR/run.sh b/examples/MACLR/run.sh deleted file mode 100644 index 80b113e9..00000000 --- a/examples/MACLR/run.sh +++ /dev/null @@ -1,15 +0,0 @@ -dataset=$1 -mode=$2 -log=$3 -model=$4 -accelerate launch --config_file accelerate_config.yaml main.py \ - --dataset $dataset \ - --mode $mode \ - --log $log \ - --model-name-or-path $model \ - --pooling-mode cls \ - --proj-emb-dim 512 \ - --per-device-train-batch-size 16 \ - --learning-rate 1e-5 \ - --max-train-steps 100000 \ - --eval-steps 5000 \ \ No newline at end of file diff --git a/examples/MACLR/utils.py b/examples/MACLR/utils.py deleted file mode 100644 index e6533890..00000000 --- a/examples/MACLR/utils.py +++ /dev/null @@ -1,168 +0,0 @@ -import numpy as np -import math -import faiss -import torch -from tqdm.auto import tqdm -from beir.retrieval.evaluation import EvaluateRetrieval - - -def perform_eval(model, label_dataloader, label_ids, instance_dataloader, inst_num, test_ids, accelerator): - label_bert = model.label_encoder - inst_bert = model.instance_encoder - torch.cuda.empty_cache() - label_bert, inst_bert = accelerator.prepare(label_bert, inst_bert) - # label embeddings - label_embeds = np.zeros((len(label_dataloader)*16*8, 512)).astype('float32') - count = 0 - label_bert.eval() - with torch.no_grad(): - for i, batch in enumerate(tqdm(label_dataloader, desc='Embedding Labels', disable=not accelerator.is_local_main_process)): - batch_att_mask = ~(batch.eq(0)) - feature = {'input_ids': batch, 'attention_mask': batch_att_mask} - embed = label_bert(feature)["sentence_embedding"] - output = accelerator.gather(embed) - output = output.data.cpu().numpy().astype('float32') - num_label = output.shape[0] - label_embeds[count:count + num_label, :] = output - count += num_label - label_num = len(label_ids) - label_embeds = label_embeds[:label_num] - label_embeds = label_embeds.astype('float32') - - - # instance embeddings - inst_embeds = np.zeros((len(instance_dataloader)*128*8, 512)).astype('float32') - count = 0 - inst_bert.eval() - with torch.no_grad(): - count = 0 - for i, batch in enumerate(tqdm(instance_dataloader, desc='Embedding Instances', disable=not accelerator.is_local_main_process)): - batch_att_mask = ~(batch.eq(0)) - feature = {'input_ids': batch, 'attention_mask': batch_att_mask} - embed = inst_bert(feature)["sentence_embedding"] - output = accelerator.gather(embed) - output = output.data.cpu().numpy().astype('float32') - num_inst = output.shape[0] - inst_embeds[count:count+num_inst, :] = output - count += num_inst - accelerator.print("embedding") - inst_embeds = inst_embeds.astype('float32') - inst_embeds = inst_embeds[:inst_num] - test_inst_embeds = inst_embeds[-len(test_ids):] - - - accelerator.print("Finish embedding") - D, I = get_knn(test_inst_embeds, label_embeds, accelerator, bsz=64) - label_bert.train() - inst_bert.train() - del label_embeds - - return D, I, inst_embeds - - -def perform_clustering(step, features, accelerator): - # num_cluster = [10000, 20000, 40000, 80000, 100000, 100000] - num_cluster = [2500, 5000, 10000, 20000, 40000, 80000] - if step>=0 and step<50000: - i = step // 10000 - d = features.shape[1] - k = num_cluster[i] - clus = faiss.Clustering(d, k) - clus.verbose = False - clus.niter = 20 - clus.nredo = 5 - clus.seed = 0 - clus.max_points_per_centroid = 1000 - clus.min_points_per_centroid = 1 - if accelerator.is_local_main_process: - clus.verbose = True - res = faiss.StandardGpuResources() - flat_config = faiss.GpuIndexFlatConfig() - flat_config.useFloat16 = False - flat_config.device = accelerator.local_process_index - index = faiss.GpuIndexFlatL2(res, d, flat_config) - features = features.astype('float32') - clus.train(features, index) - num_inst = features.shape[0] - bsz = 16 - nr_batch = int(math.ceil(num_inst / bsz)) - D_list, I_list = [], [] - for bidx in range(nr_batch): - sidx = bidx * bsz - eidx = min((bidx + 1) * bsz, num_inst) - D, I = index.search(features[sidx:eidx], 1) - D_list.append(D) - I_list.append(I) - idxs = np.concatenate(I_list) - cluster_result = [int(n[0]) for n in idxs] - else: - cluster_result = [None for _ in range(features.shape[0])] - torch.distributed.broadcast_object_list(cluster_result, src=0, group=None) - cluster_result = torch.LongTensor(cluster_result).to(accelerator.device) - return cluster_result - else: - return None - -def get_knn(inst_embeddings, label_embeddings, accelerator, top_k=100, bsz=65536): - accelerator.print("FAISS") - # logging.info("FAISS indexer building") - res = faiss.StandardGpuResources() - flat_config = faiss.GpuIndexFlatConfig() - flat_config.useFloat16 = False - flat_config.device = accelerator.local_process_index - indexer = faiss.GpuIndexFlatIP(res, inst_embeddings.shape[1], flat_config) - indexer.add(label_embeddings) - # logging.info("FAISS indexer searching") - num_inst = inst_embeddings.shape[0] - nr_batch = int(math.ceil(num_inst / bsz)) - D_list, I_list = [], [] - accelerator.print("index") - for bidx in tqdm(range(nr_batch)): - sidx = bidx * bsz - eidx = min((bidx + 1) * bsz, num_inst) - D, I = indexer.search(inst_embeddings[sidx:eidx], top_k) - D_list.append(D) - I_list.append(I) - D = np.concatenate(D_list) - I = np.concatenate(I_list) - return D, I - - -def eval_and_cluster(args, logger, step, model, label_dataloader, label_ids, - instance_dataloader, inst_num, test_ids, qrels, accelerator): - D, I, inst_embeds = perform_eval(model, label_dataloader, label_ids, instance_dataloader, inst_num, test_ids, accelerator) - num_tst = len(test_ids) - results = {pid: {} for pid in test_ids} - accelerator.print("Results") - for row_id in range(num_tst): - inst_id = test_ids[row_id] - for col_id, score in zip(I[row_id], D[row_id]): - lid = label_ids[col_id] - results[inst_id][lid] = float(score) - - #### evaluate - k_values = [1,3,5,10,20,100] - if accelerator.local_process_index == 0: - for row_id in range(num_tst): - pid = test_ids[row_id] - for col_id, score in zip(I[row_id], D[row_id]): - tid = label_ids[col_id] - results[pid][tid] = float(score) - accelerator.print("end") - ndcg, _map, recall, precision = EvaluateRetrieval.evaluate(qrels, results, k_values) - logger.info(ndcg) - logger.info(_map) - logger.info(precision) - logger.info(recall) - del results - - del D, I - - # clustering - if args.mode != 'ict': - return None - else: - cluster_num = inst_num - len(test_ids) - cluster_features = inst_embeds[:cluster_num] - cluster_result = perform_clustering(step, cluster_features, accelerator) - return cluster_result \ No newline at end of file diff --git a/examples/README.md b/examples/README.md deleted file mode 100644 index 3ce0d999..00000000 --- a/examples/README.md +++ /dev/null @@ -1,10 +0,0 @@ - -# PECOS Python Examples - -This directory contains examples of commonly-used PECOS Python API -and experimental codes to reproduce results from papers using PECOS. - -## About these examples - -* Each folder should be self-contained and clearly specify the package dependencies. -* This directory may not be actively maintained. diff --git a/examples/ann-hnsw-pq4bits/Makefile b/examples/ann-hnsw-pq4bits/Makefile deleted file mode 100644 index 9693b2b0..00000000 --- a/examples/ann-hnsw-pq4bits/Makefile +++ /dev/null @@ -1,11 +0,0 @@ -CXX=g++ -CXXFLAGS=-fopenmp -O3 -std=c++14 -fPIC -DNDEBUG -Wall -g -lblas -EXTRA_INCLUDE_FLAGS=-I../../pecos/core/ -ARCHFLAG=-march=native - -all: go - -go: example.cpp - ${CXX} -o go ${CXXFLAGS} example.cpp -I. ${EXTRA_INCLUDE_FLAGS} ${ARCHFLAG} -clean: - rm -rf *.so *.o go diff --git a/examples/ann-hnsw-pq4bits/README.md b/examples/ann-hnsw-pq4bits/README.md deleted file mode 100644 index e4fc9f37..00000000 --- a/examples/ann-hnsw-pq4bits/README.md +++ /dev/null @@ -1,75 +0,0 @@ - - -## Notice -- Currently we only support L2 distance with 4 Bits Product Quantization. -- We are working on extending to angular and ip distance measures. - -## Install Prerequisite - -To run this project, prerequisite is the same as building PECOS. - -* For Ubuntu (18.04, 20.04): -``` bash -sudo apt-get update && sudo apt-get install -y build-essential git python3 python3-distutils python3-venv -``` -* For Amazon Linux 2 Image: -``` bash -sudo yum -y install python3 python3-devel python3-distutils python3-venv && sudo yum -y groupinstall 'Development Tools' -``` -One needs to install at least one BLAS library to compile PECOS, e.g. `OpenBLAS`: -* For Ubuntu (18.04, 20.04): -``` bash -sudo apt-get install -y libopenblas-dev -``` -* For Amazon Linux 2 Image and AMI: -``` bash -sudo amazon-linux-extras install epel -y -sudo yum install openblas-devel -y -``` - -## Prepare Data - -Get the exemplar sift-128-eucldiean dataset - -```bash -wget https://archive.org/download/pecos-dataset/ann-benchmarks/sift-euclidean-128.tar.gz -``` - -Extract the dataset - -```bash -tar -xf sift-euclidean-128.tar.gz -``` - -The prepared dataset consists of 3 .npy files : X.trn.npy (training data), X.tst.npy (testing data) and Y.tst.npy (10 Nearest neighbors in training data of test data). - -## Compile the source code - -```bash -Make clean go -``` - -a runnable named "go" will be generated. - -## Running the compiled runnable - -the runnable take arguments in the following form : -```bash -./go data_folder model_folder space M efC #threads efs num_rerank sub_dimension -``` - -data_folder is the place where 3 npy files stored. model_folder is the place to store the trained model. If a saved model is found, we will load the model instead of training a new one. space denotes the distance measure to use. Currently, we only support L2. M is the maximal edge connection used in HNSW. efC is the Maximal connecting edges during construction used in HNSW. #threads is the number of threads to build the graph. Up to now, these hypaer-parameters relate to the construction, and they will be used to name the trained model directory. efs is the search queue size in the inference step. num_rerank is the number of points in the queue that we will further rerank again using original features instead of quantized distance. sub_dimension is the dimension of each subspace in Product Quantization. If sub_dimension is set to 0, it will use default scheme. That is, if original data dimension <= 400, we use sub_dimension == 1, otherwise we use sub_dimension == 2. - -Here, we provide an example of command executing the runnable : - -```bash -./go sift-euclidean-128 sift-euclidean-128 l2 8 500 24 10 10 0 -``` - -## Experiment - -The compiled source code in example.cpp already repeats the inference 10 times. So to evaluate under ann-benchmark protocol, we could simply use python to iterate hyper-parameters and record done results. - -```bash -python run.py -``` diff --git a/examples/ann-hnsw-pq4bits/example.cpp b/examples/ann-hnsw-pq4bits/example.cpp deleted file mode 100644 index fe0028e5..00000000 --- a/examples/ann-hnsw-pq4bits/example.cpp +++ /dev/null @@ -1,127 +0,0 @@ -#include -#include -#include -#include "utils/matrix.hpp" -#include "utils/scipy_loader.hpp" -#include "ann/hnsw.hpp" - - - -class StopW { - std::chrono::steady_clock::time_point time_begin; -public: - StopW() { - time_begin = std::chrono::steady_clock::now(); - } - - float getElapsedTimeMicro() { - std::chrono::steady_clock::time_point time_end = std::chrono::steady_clock::now(); - return (std::chrono::duration_cast(time_end - time_begin).count()); - } - - void reset() { - time_begin = std::chrono::steady_clock::now(); - } -}; - - -int num_rerank; -int sub_dimension; -using pecos::ann::index_type; - -typedef float32_t value_type; -typedef uint64_t mem_index_type; -typedef pecos::NpyArray scipy_npy_t; - - -auto npy_to_drm = [](scipy_npy_t& X_npy) -> pecos::drm_t { - pecos::drm_t X; - X.rows = X_npy.shape[0]; - X.cols = X_npy.shape[1]; - X.val = X_npy.array.data(); - return X; -}; - - -template -void run_dense(std::string data_dir , char* model_path, index_type M, index_type efC, index_type max_level, int threads, int efs) { - // data prepare - scipy_npy_t X_trn_npy(data_dir + "/X.trn.npy"); - scipy_npy_t X_tst_npy(data_dir + "/X.tst.npy"); - scipy_npy_t Y_tst_npy(data_dir + "/Y.tst.npy"); - auto X_trn = npy_to_drm(X_trn_npy); - auto X_tst = npy_to_drm(X_tst_npy); - auto Y_tst = npy_to_drm(Y_tst_npy); - // model prepare - index_type topk = 10; - pecos::ann::HNSWProductQuantizer4Bits indexer; - FILE* fp = fopen(model_path, "rb"); - if (!fp) { - // if subspace_dimension is set to 0, it will use default scheme. That is, - // if dimension <= 400, we use subspace_dimension 1, otherwise we use 2. - indexer.train(X_trn, M, efC, 0, 200, threads, max_level); - std::cout<< "After train" <::max(); - // REPEAT 10 times and report the best result - for (int repeat = 0; repeat < 10; repeat++) { - double inner_latency = 0.0; - for (index_type idx = 0; idx < num_data; idx++) { - StopW stopw = StopW(); - auto ret_pairs = indexer.predict_single(X_tst.get_row(idx), efs, topk, searcher, num_rerank); - double ss = stopw.getElapsedTimeMicro(); - inner_latency += ss; - } - latency = std::min(latency, inner_latency); - } - // inference and calculate recalls - double recall = 0.0; - for (index_type idx = 0; idx < num_data; idx++) { - auto ret_pairs = indexer.predict_single(X_tst.get_row(idx), efs, topk, searcher, num_rerank); - std::unordered_set true_indices; - - for (auto k = 0u; k < topk; k++) { - true_indices.insert(Y_tst.get_row(idx).val[k]); // assume Y_tst is ascendingly sorted by distance - } - for (auto dist_idx_pair : ret_pairs) { - if (true_indices.find(dist_idx_pair.node_id) != true_indices.end()) { - recall += 1.0; - } - } - } - recall = recall / num_data / topk; - latency = latency / num_data / 1000.; - std::cout<< recall << " : " << 1.0 / latency * 1e3 << "," <>(data_dir, model_path, M, efC, max_level, threads, efs); - } -} diff --git a/examples/ann-hnsw-pq4bits/run.py b/examples/ann-hnsw-pq4bits/run.py deleted file mode 100644 index 3a887371..00000000 --- a/examples/ann-hnsw-pq4bits/run.py +++ /dev/null @@ -1,9 +0,0 @@ -import os -cmd = "./go sift-euclidean-128 sift-euclidean-128 l2 %d 500 24 %d %d 0" -for args in [8, 16, 24, 36, 48, 64, 96]: - for efs in [10, 20, 40, 80, 120, 200, 400]: - os.system(cmd % (args, efs, efs)) - if efs - 20 > 0: - os.system(cmd % (args, efs, 20)) - if efs - 50 > 0: - os.system(cmd % (args, efs, 50)) diff --git a/examples/fm-for-xmc/FM_O_dk.md b/examples/fm-for-xmc/FM_O_dk.md deleted file mode 100644 index 7d6d20fd..00000000 --- a/examples/fm-for-xmc/FM_O_dk.md +++ /dev/null @@ -1,44 +0,0 @@ -# FM $O(dk)$ weight gradient calculation - -> Written by Andrew Bai (July 2022) - -Let $X \in \mathbb{R}^d$ and $P \in \mathbb{R}^{k \times d}$. We will ignore the linear terms for now. - -A factorization machine $\phi$ is defined as follows -$$ -\begin{align} - \phi(X) :&= \frac{1}{2} \bigg( \| PX \|^2 - \sum_{i=1}^k \| P_{i, :} \circ X \|^2 \bigg) \\ - &= \frac{1}{2} \bigg( \sum_{i=1}^k \big( \sum_{j=1}^d P_{i, j} \cdot {X_j} \big)^2 - \sum_{j=1}^d {X_j}^2 \cdot \big( \sum_{i=1}^k {P_{(i, j)}}^2 \big) \bigg) \\ - &= \frac{1}{2} \bigg( \sum_{i=1}^k \sum_{j=1}^d P_{i, j} {X_j} \sum_{j'=1}^d P_{i, j'} {X_{j'}} - \sum_{j=1}^d {X_j}^2 \cdot \big( \sum_{i=1}^k {P_{(i, j)}}^2 \big) \bigg) \label{eq:fm_scalar_form} -\end{align} -$$ -Given a binary classification setting where $y \in \{+1, -1\}$ and the model is trained with logistic regression, the loss function is as follows -$$ - l(X, y) = \log(1 + \exp(-y \cdot \phi(X))) -$$ -We now derive the derivative of the loss function with respect to one single weight parameter $P_{i,j}$ -$$ -\begin{equation} - \frac{\text{d}l(X, y)}{\text{d}P_{i, j}} - = \frac{\text{d}l(X, y)}{\text{d}\phi(X)} \cdot \frac{\text{d}\phi(X)}{\text{d}P_{i, j}} - = \frac{-y}{1 + \exp(y \cdot \phi(X))} \cdot \frac{\text{d}\phi(X)}{\text{d}P_{i, j}} \label{eq:loss_grad} -\end{equation} -$$ -According to Eq (3) the only terms in $\phi(X)$ involving $P_{i, j}$ is -$$ -\begin{equation} - \phi_{i,j}(X) = \frac{1}{2} \big( 2 \cdot P_{i,j} X_j \cdot \sum_{j'=1}^d P_{i, j'}X_{j'} - {P_{i, j}}^2{X_j}^2 - {X_j}^2 \cdot {P_{i, j}}^2\big) -\end{equation} -$$ -Let us pre-compute the embedding $Z = PX$ and memoize the results. We then proceed to compute the second term of Eq (5) -$$ -\begin{align} - \frac{\text{d}\phi(X)}{\text{d}P_{i, j}} - = \frac{\text{d}\phi_{i, j}(X)}{\text{d}P_{i, j}} - &= X_j \cdot \sum_{j'=1}^d P_{i, j'} X_{j'} - 2P_{i, j}{X_j}^2 \\ - &= X_j \cdot Z_i - 2P_{i, j}{X_j}^2 \label{eq:phi_grad} -\end{align} -$$ -Memoizing $Z$ takes $O(dk)$ time. Eq (8) can be calculated in $O(1)$ by looking up the memoized $Z_i$. - -Thus, the total complexity of calculating the weight gradient of FM is $O(dk)$. \ No newline at end of file diff --git a/examples/fm-for-xmc/Makefile b/examples/fm-for-xmc/Makefile deleted file mode 100644 index 75730588..00000000 --- a/examples/fm-for-xmc/Makefile +++ /dev/null @@ -1,16 +0,0 @@ -CXX=g++ -CXXFLAGS=-fopenmp -O3 -std=c++14 -fPIC -DNDEBUG -Wall -DUSEOMP -DVECGRAD -fno-math-errno #-DDETERMINISTIC -DBENCHMARK -DVERBOSE -fopt-info-vec-optimized -INCLUDE_FLAGS=-I. -I../../pecos/core/ -LIB_FLAGS=-lopenblas -ARCHFLAG=-mavx #-march=native - -all: fm_train fm_embgen - -fm_train: fm_train_driver.cpp - ${CXX} -o fm_train ${CXXFLAGS} -g fm_train_driver.cpp ${INCLUDE_FLAGS} ${ARCHFLAG} ${LIB_FLAGS} - -fm_embgen: fm_generate_embs.cpp - ${CXX} -o fm_embgen ${CXXFLAGS} -g fm_generate_embs.cpp ${INCLUDE_FLAGS} ${ARCHFLAG} ${LIB_FLAGS} - -clean: - rm -rf *.so *.o fm_train fm_embgen diff --git a/examples/fm-for-xmc/README.md b/examples/fm-for-xmc/README.md deleted file mode 100644 index 42c7ded8..00000000 --- a/examples/fm-for-xmc/README.md +++ /dev/null @@ -1,56 +0,0 @@ -# Factorization Machine XMC -Factorization machines (FM) are models that factorized input features into smaller dimensions for retrieval systems. -The factorization enables modeling of cross-terms between input features, which isn't possible with purely inner-product based models. -FM are particularly useful when the input feature dimension is sparse (e.g. TD-IDF embedding) and cross-terms between feature dimensions contain relevant information to perform prediction. -This library implements efficient training of factorization machines for extreme multilabel classification. -The optmization algorithm is AdaGrad with L2 regularization, with the option of parallelization with HogWild!. - -## Install -``` -git clone -b v0.4.0 https://github.com/amzn/pecos.git -cd pecos/example/fm -make all -``` -You might need to specify path to OpenBlas with `-L[/openblas_dir/lib]` - -## Training -``` -./fm_train $PARAMS $Q_TRN_PATH $QP_PAIR_TRN_PATH $Q_TST_PATH $QP_PAIR_TST_PATH $P_PATH $MODEL_PATH -``` -Trained model will be saved in `MODEL_PATH`. - -### Parameters -* `-t`: Number of training epochs (int, default 10) -* `-k`: Number of factorized dimensions (int, default 4) -* `-l`: L2 regularization factor (float, default 2e-5) -* `-r`: Adagrad learning rate (float, defalt 2e-2) -* `--n_threads`: Number of threads for parallel training when OpenMP is enabled. Current implementation adopts [HogWild!](https://arxiv.org/abs/1106.5730) lock-free, parallelized gradient descent. (int, default 1) -* `--auto-stop`: Flag to early-stop training when testing loss starts to increase. Factorization machines are know to overfit and is not fixable by increasing L2 regularization. -* `--factorized`: Flag to perform gradient update per training sample, as opposed to per factor. Improves the training speed from $O(d^2k)$ to $O(dk)$ where $d$ is the number of input feature dimensions and $k$ is the number of factors (see more details [here](./FM_O_dk.pdf)). The improvement in training speed comes as no cost of trained model performance. *Highly recommend turning on.* -* `--dense`: Flag to specify whether input features in `X_TRN_PATH`, `X_TST_PATH`, and `Z_PATH` are dense (expect Numpy [`.npy`](https://numpy.org/doc/stable/reference/routines.io.html)) or sparse (expect Scipy spare [`.npz`](https://docs.scipy.org/doc/scipy/reference/sparse.html)). - -### Training and testing data paths -* `Q_TRN_PATH`: Path to training query features file ( `npy` or `npz`) storing a $n_q \times d_q$ matrix, where $n_q$ is the number of unique training queries and $d_q$ is the dimension of query features. -* `QP_PAIR_TRN_PATH`: Path to training (query, product) pairs (`npz`) storing a $n_q \times n_p$ sparse binary matrix $M$, where $n_q$ is the number of unique training queries and $n_p$ is the number of unique products. $M[i, j] == 1$ indicates that the $i$-th query and $j$-th product is a positive pair. -* `Q_TST_PATH`: Path to testing query features file ( `npy` or `npz`) storing a $n_q \times d_q$ matrix, where $n_q$ is the number of unique testing queries and $d_q$ is the dimension of query features. -* `QP_PAIR_TST_PATH`: Path to testing (query, product) pairs (`npz`) storing a $n_q \times n_p$ sparse binary matrix $M$, where $n_q$ is the number of unique testing queries and $n_p$ is the number of unique products. $M[i, j] == 1$ indicates that the $i$-th query and $j$-th product is a positive pair. -* `P_PATH`: Path to product features file ( `npy` or `npz`) storing a $n_p \times d_p$ matrix, where $n_p$ is the number of unique products and $d_p$ is the dimension of product features. - -## FM to shifted inner product (SIP) -For better integration with existing inner-product-search based retrieval systems, factorization machines can embed queries and products into vectors such that inner product with the vectors recovers the factorization machine prediction. -To do so, simply run -``` -./fm_embgen $MODEL_PATH $Q_PATH $P_PATH $SAVE_DIR -``` -### Arguments -* `MODEL_PATH`: Path to saved model. -* `Q_PATH`: Path to query features file ( `npy` or `npz`) storing a $n_q \times d_q$ matrix, where $n_q$ is the number of unique queries and $d_q$ is the dimension of query features. -* `P_PATH`: Path to product features file ( `npy` or `npz`) storing a $n_p \times d_p$ matrix, where $n_p$ is the number of unique products and $d_p$ is the dimension of product features. -* `SAVE_DIR`: Directory to save the embeddings. - -### Importing embeddings into Python -Users may want to consume the binary embeddings in Python. To convert the embeddings to Numpy arrays, run -``` -python binary_emb_to_npy.py [EMB_DIR] -``` - diff --git a/examples/fm-for-xmc/binary_emb_to_npy.py b/examples/fm-for-xmc/binary_emb_to_npy.py deleted file mode 100644 index 1d9a6324..00000000 --- a/examples/fm-for-xmc/binary_emb_to_npy.py +++ /dev/null @@ -1,66 +0,0 @@ -import argparse -import os -import numpy as np -import struct - -def parse_args(): - parser = argparse.ArgumentParser() - - parser.add_argument("emb_dir", help="Directory of FM embeddings", type=str) - return parser.parse_args() - -def test_eof(f): - remaining_bytes = 0 - while True: - if f.read(1) != b'': - remaining_bytes += 1 - else: - break - if remaining_bytes != 0: - raise Exception(f"Expected to reach EOF but got {remaining_bytes} bytes left.") - -def construct_emb_with_bias(embs_dir, save=True, verbose=False): - with open(os.path.join(embs_dir, "X.emb"), 'rb') as f: - r = struct.unpack('i', f.read(4))[0] - c = struct.unpack('i', f.read(4))[0] - X_embs = np.reshape(np.fromfile(f, dtype=np.float32), (r, c)) - - test_eof(f) - - with open(os.path.join(embs_dir, "X.bias"), 'rb') as f: - r = struct.unpack('i', f.read(4))[0] - X_bias = np.reshape(np.fromfile(f, dtype=np.float32), (r, 1)) - - test_eof(f) - - with open(os.path.join(embs_dir, "Z.emb"), 'rb') as f: - r = struct.unpack('i', f.read(4))[0] - c = struct.unpack('i', f.read(4))[0] - Z_embs = np.reshape(np.fromfile(f, dtype=np.float32), (r, c)) - - test_eof(f) - - with open(os.path.join(embs_dir, "Z.bias"), 'rb') as f: - r = struct.unpack('i', f.read(4))[0] - Z_bias = np.reshape(np.fromfile(f, dtype=np.float32), (r, 1)) - - test_eof(f) - - X_embs_bias = np.concatenate([X_embs, X_bias, np.ones_like(X_bias)], axis=1) - Z_embs_bias = np.concatenate([Z_embs, np.ones_like(Z_bias), Z_bias], axis=1) - - if verbose: - print(f"X_embs_bias.shape={X_embs_bias.shape}, Z_embs_bias.shape={Z_embs_bias.shape}") - - if save: - np.save(os.path.join(embs_dir, 'X_embs_bias.npy'), X_embs_bias) - np.save(os.path.join(embs_dir, 'Z_embs_bias.npy'), Z_embs_bias) - else: - return X_embs_bias, Z_embs_bias - -def main(): - args = parse_args() - construct_emb_with_bias(args.emb_dir) - -if __name__ == '__main__': - main() \ No newline at end of file diff --git a/examples/fm-for-xmc/fm_generate_embs.cpp b/examples/fm-for-xmc/fm_generate_embs.cpp deleted file mode 100644 index 00eac064..00000000 --- a/examples/fm-for-xmc/fm_generate_embs.cpp +++ /dev/null @@ -1,125 +0,0 @@ -#include -#include -#include -#include - -#include "utils/matrix.hpp" -#include "utils/scipy_loader.hpp" - -#include "xmc/fm_solver.hpp" -#include "xmc/fm_inference.hpp" - -typedef float32_t value_type; -typedef uint64_t mem_index_type; -typedef pecos::NpyArray scipy_npy_t; -typedef pecos::ScipySparseNpz scipy_npz_t; - -using namespace std; - -auto npz_to_csr = [](scipy_npz_t& X_npz) -> pecos::csr_t { - pecos::csr_t X; - X.rows = X_npz.shape[0]; - X.cols = X_npz.shape[1]; - X.indices = X_npz.indices.array.data(); - X.indptr = X_npz.indptr.array.data(); - X.val = X_npz.data.array.data(); - return X; -}; - -int main(int argc, char** argv) { - vector args; - for(int i = 0; i < argc; i++) - args.push_back(string(argv[i])); - - // arguments - std::string model_path = args[1]; - std::string X_data_path = args[2]; - std::string Z_data_path = args[3]; - std::string emb_dir = args[4]; - - std::string X_emb_path = emb_dir + "/X.emb"; - std::string Z_emb_path = emb_dir + "/Z.emb"; - std::string X_bias_path = emb_dir + "/X.bias"; - std::string Z_bias_path = emb_dir + "/Z.bias"; - - // data loading. - scipy_npz_t X_npz(X_data_path); - scipy_npz_t Z_npz(Z_data_path); - - auto X = npz_to_csr(X_npz); // pecos::csr_t, [n_trn, d] - auto Z = npz_to_csr(Z_npz); - - std::cout << X.rows << " " << X.cols << std::endl; - std::cout << Z.rows << " " << Z.cols << std::endl; - - // load inference model. - typedef typename pecos::FactorizationMachineModel fm_t; - fm_t fm; - - FILE *fp; - fp = fopen(&model_path[0], "rb"); - fm.load(fp); - fclose(fp); - - fm.build_index(Z); - - // build X embs and bias - std::vector X_embs; - std::vector X_bias; - - X_embs.resize(X.rows * fm.k_size, 0); - X_bias.resize(X.rows, 0); - - pecos::drm_t X_embs_; - X_embs_.rows = X.rows; - X_embs_.cols = fm.k_size; - X_embs_.val = X_embs.data(); - - pecos::drm_t Wx_; - Wx_.rows = fm.wx_size; - Wx_.cols = fm.k_size; - Wx_.val = fm.Wx.data(); - - fm.smat_x_dmat(X, Wx_, X_embs_); - - for (size_t i = 0; i < X.rows; ++i) { - const auto& xi = X.get_row(i); - X_bias[i] = fm.get_bias(xi, Wx_); - } - - // test for correctness - pecos::drm_t Z_embs_; - Z_embs_.rows = Z.rows; - Z_embs_.cols = fm.k_size; - Z_embs_.val = fm.Z_embs.data(); - - // cout << pecos::do_dot_product(X_embs_.get_row(0), Z_embs_.get_row(0)) + X_bias[0] + fm.Z_bias[0] << endl; - // cout << fm.inference(X.get_row(0), 0) << endl; - - // write. - fp = fopen(&X_emb_path[0], "wb"); - pecos::file_util::fput_multiple(&(X_embs_.rows), 1, fp); - pecos::file_util::fput_multiple(&(X_embs_.cols), 1, fp); - - pecos::file_util::fput_multiple(&(X_embs[0]), X_embs.size(), fp); - fclose(fp); - - fp = fopen(&X_bias_path[0], "wb"); - pecos::file_util::fput_multiple(&(X_embs_.rows), 1, fp); - - pecos::file_util::fput_multiple(&(X_bias[0]), X_bias.size(), fp); - fclose(fp); - - fp = fopen(&Z_emb_path[0], "wb"); - pecos::file_util::fput_multiple(&(fm.num_zs), 1, fp); - pecos::file_util::fput_multiple(&(fm.k_size), 1, fp); - - pecos::file_util::fput_multiple(&(fm.Z_embs[0]), fm.Z_embs.size(), fp); - fclose(fp); - - fp = fopen(&Z_bias_path[0], "wb"); - pecos::file_util::fput_multiple(&(fm.num_zs), 1, fp); - - pecos::file_util::fput_multiple(&(fm.Z_bias[0]), fm.Z_bias.size(), fp); - fclose(fp); -} diff --git a/examples/fm-for-xmc/fm_train_driver.cpp b/examples/fm-for-xmc/fm_train_driver.cpp deleted file mode 100644 index 0694e831..00000000 --- a/examples/fm-for-xmc/fm_train_driver.cpp +++ /dev/null @@ -1,210 +0,0 @@ -#include -#include -#include -#include - -#if defined USEOMP -#include -#endif - -#include "utils/matrix.hpp" -#include "utils/scipy_loader.hpp" -#include "xmc/fm_solver.hpp" - -typedef float32_t value_type; -typedef uint64_t mem_index_type; -typedef pecos::NpyArray scipy_npy_t; -typedef pecos::ScipySparseNpz scipy_npz_t; - -using namespace std; - -auto npz_to_csr = [](scipy_npz_t& X_npz) -> pecos::csr_t { - pecos::csr_t X; - X.rows = X_npz.shape[0]; - X.cols = X_npz.shape[1]; - X.indices = X_npz.indices.array.data(); - X.indptr = X_npz.indptr.array.data(); - X.val = X_npz.data.array.data(); - return X; -}; - -auto npy_to_drm = [](scipy_npy_t& X_npy) -> pecos::drm_t { - pecos::drm_t X; - X.rows = X_npy.shape[0]; - X.cols = X_npy.shape[1]; - X.val = X_npy.array.data(); - return X; -}; - -int main(int argc, char** argv) { - // parse CLI arguments. - vector args; - for(int i = 0; i < argc; i++) - args.push_back(string(argv[i])); - - pecos::fm_solver::FMParameter param; - uint64_t k_size = 4; - std::string X_trn_path, Y_trn_path, X_tst_path, Y_tst_path, Z_path; - std::string model_path = ""; - std::string Y_prefix = ""; - bool is_emb_dense = false; - -#ifdef USEOMP - int n_threads = 1; -#endif - - int i = 1; - for(; i < argc; i++) { - if(args[i].compare("-t") == 0) - { - if(i == argc-1) - throw invalid_argument("need to specify number of iterations after -t"); - i++; - param.max_iter = atoi(args[i].c_str()); - if(param.max_iter <= 0) - throw invalid_argument("number of iterations should be greater than zero"); - } else if(args[i].compare("-k") == 0) { - if(i == argc-1) - throw invalid_argument("need to specify number of factors after -k"); - i++; - k_size = atoi(args[i].c_str()); - if(k_size <= 0) - throw invalid_argument("number of factors should be greater than zero"); - } else if(args[i].compare("-r") == 0) { - if(i == argc-1) - throw invalid_argument("need to specify eta after -r"); - i++; - param.eta = atof(args[i].c_str()); - if(param.eta <= 0) - throw invalid_argument("learning rate should be greater than zero"); - } else if(args[i].compare("--prefix") == 0) { - if(i == argc-1) - throw invalid_argument("need to specify prefix of Y after --prefix"); - i++; - Y_prefix = "." + args[i]; - } else if(args[i].compare("-l") == 0) { - if(i == argc-1) - throw invalid_argument("need to specify lambda after -l"); - i++; - param.reg = atof(args[i].c_str()); - if(param.reg < 0) - throw invalid_argument("regularization cost should not be smaller than zero"); - } else if(args[i].compare("--auto-stop") == 0) { - param.auto_stop = true; - } else if(args[i].compare("--factorized") == 0) { - param.factorized = true; - } else if(args[i].compare("--identity_biased_init") == 0) { - param.identity_biased_init = true; - } else if(args[i].compare("--dense") == 0) { - is_emb_dense = true; - } -#ifdef USEOMP - else if(args[i].compare("--n_threads") == 0) { - if(i == argc-1) - throw invalid_argument("need to specify number of threads after --n_threads"); - i++; - n_threads = atoi(args[i].c_str()); - if(n_threads <= 0) - throw invalid_argument("number of threads should be greater than zero"); - } -#endif - else { - break; - } - } - - X_trn_path = string(args[i]); - i++; - Y_trn_path = string(args[i]); - i++; - X_tst_path = string(args[i]); - i++; - Y_tst_path = string(args[i]); - i++; - Z_path = string(args[i]); - i++; - - model_path = string(args[i]); - -#ifdef USEOMP - // set threads - omp_set_num_threads(n_threads); -#endif - - typedef typename pecos::fm_solver::FMWorker fm_worker_t; - fm_worker_t fmw; - - const int seed = 1126; - - if (is_emb_dense) { - // load training data. - scipy_npy_t X_trn_npy(X_trn_path); - scipy_npz_t Y_trn_npz(Y_trn_path); - - auto X_trn = npy_to_drm(X_trn_npy); // pecos::drm_t, [n_trn, d] - // auto Z_trn = npy_to_drm(Z_trn_npy); - auto Y_trn = npz_to_csr(Y_trn_npz); - - std::cout << "X_trn.shape = (" << X_trn.rows << "," << X_trn.cols << ")" << std::endl; - std::cout << "Y_trn.shape = (" << Y_trn.rows << "," << Y_trn.cols << ")" << std::endl; - - // load testing data. - scipy_npy_t X_val_npy(X_tst_path); - scipy_npz_t Y_val_npz(Y_tst_path); - - auto X_val = npy_to_drm(X_val_npy); // pecos::csr_t, [n_trn, d] - auto Y_val = npz_to_csr(Y_val_npz); - - std::cout << "X_val.shape = (" << X_val.rows << "," << X_val.cols << ")" << std::endl; - std::cout << "Y_val.shape = (" << Y_val.rows << "," << Y_val.cols << ")" << std::endl; - - scipy_npy_t Z_npy(Z_path); - auto Z = npy_to_drm(Z_npy); - std::cout << "Z.shape = (" << Z.rows << "," << Z.cols << ")" << std::endl; - - // train - const uint64_t wx_size = X_trn.cols; - const uint64_t wz_size = Z.cols; - fmw.init(wx_size, wz_size, k_size, ¶m); - fmw.solve(X_trn, Z, Y_trn, X_val, Z, Y_val, seed); - } - else { - // load training data. - scipy_npz_t X_trn_npz(X_trn_path); - // scipy_npz_t Z_trn_npz(data_dir + "/Z.trn.npz"); - scipy_npz_t Y_trn_npz(Y_trn_path); - - auto X_trn = npz_to_csr(X_trn_npz); // pecos::csr_t, [n_trn, d] - auto Y_trn = npz_to_csr(Y_trn_npz); - - std::cout << "X_trn.shape = (" << X_trn.rows << "," << X_trn.cols << ")" << std::endl; - std::cout << "Y_trn.shape = (" << Y_trn.rows << "," << Y_trn.cols << ")" << std::endl; - - // load testing data. - scipy_npz_t X_val_npz(X_tst_path); - scipy_npz_t Y_val_npz(Y_tst_path); - - auto X_val = npz_to_csr(X_val_npz); // pecos::csr_t, [n_trn, d] - auto Y_val = npz_to_csr(Y_val_npz); - - std::cout << "X_val.shape = (" << X_val.rows << "," << X_val.cols << ")" << std::endl; - std::cout << "Y_val.shape = (" << Y_val.rows << "," << Y_val.cols << ")" << std::endl; - - scipy_npz_t Z_npz(Z_path); - auto Z = npz_to_csr(Z_npz); - std::cout << "Z.shape = (" << Z.rows << "," << Z.cols << ")" << std::endl; - - // train - const uint64_t wx_size = X_trn.cols; - const uint64_t wz_size = Z.cols; - fmw.init(wx_size, wz_size, k_size, ¶m); - fmw.solve(X_trn, Z, Y_trn, X_val, Z, Y_val, seed); - } - - // save model - FILE *fp; - fp = fopen(&model_path[0], "w"); - fmw.save(fp); - - return 0; -} diff --git a/examples/fm-for-xmc/xmc/fm_inference.hpp b/examples/fm-for-xmc/xmc/fm_inference.hpp deleted file mode 100644 index 2d9e2ecc..00000000 --- a/examples/fm-for-xmc/xmc/fm_inference.hpp +++ /dev/null @@ -1,225 +0,0 @@ -/* - * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance - * with the License. A copy of the License is located at - * - * http://aws.amazon.com/apache2.0/ - * - * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES - * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions - * and limitations under the License. - */ - -/* -* File: fm_inference.hpp -* -* Description: Provides functionality for performing PECOS FM prediction. -* -* Note about memory management: Any function that returns a matrix type has allocated memory -* and it is incumbent upon the user to deallocate that memory by calling the free_underlying_memory -* method of the matrix in question. -*/ - -#ifndef __INFERENCE_H__ -#define __INFERENCE_H__ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include "utils/matrix.hpp" -#include "ann/hnsw.hpp" -#include "third_party/nlohmann_json/json.hpp" -#include "third_party/robin_hood_hashing/robin_hood.h" - -namespace pecos { - -template -struct FactorizationMachineModel { - - typedef std::vector dvec_t; - typedef dense_vec_t dvec_wrapper_t; - typedef sparse_vec_t svec_wrapper_t; - typedef pecos::ann::Pair pair_t; - typedef pecos::ann::heap_t> max_heap_t; - - index_type wx_size, wz_size, k_size, num_zs = 0; - dvec_t Wx; // query feature matrix of size wx_size * k_size; - dvec_t Wz; // item feature matrix of size wz_size * k_size; - - dvec_t Z_embs; // [# of items, k_size] - dvec_t Z_bias; // [#indexed_items, 1] - - void init(index_type wx_size, index_type wz_size, index_type k_size) { - this->wx_size = wx_size; - this->wz_size = wz_size; - this->k_size = k_size; - - this->Wx.resize(wx_size * k_size, 0); - this->Wz.resize(wz_size * k_size, 0); - } - - void load(FILE *fp) { - pecos::file_util::fget_multiple(&(this->wx_size), 1, fp); - pecos::file_util::fget_multiple(&(this->wz_size), 1, fp); - pecos::file_util::fget_multiple(&(this->k_size), 1, fp); - - this->Wx.resize(this->wx_size * this->k_size, 0); - pecos::file_util::fget_multiple(&(this->Wx[0]), this->wx_size * this->k_size, fp); - - this->Wz.resize(this->wz_size * this->k_size, 0); - pecos::file_util::fget_multiple(&(this->Wz[0]), this->wz_size * this->k_size, fp); - } - - void svec_x_dmat(svec_wrapper_t x, const pecos::drm_t& Y, dvec_wrapper_t xY) const { - size_t nnz = x.get_nnz(); - // TODO: parallelize. - for (size_t i = 0; i < nnz; ++i) { - const auto& yi = Y.get_row(x.idx[i]); - pecos::do_axpy(x.val[i], yi, xY); - } - } - - void smat_x_dmat(const pecos::csr_t& X, const pecos::drm_t& Y, pecos::drm_t& XY) const { - // TODO: parallelize. - for (size_t i = 0; i < X.rows; ++i) { - svec_wrapper_t xi = X.get_row(i); // must add const. - dvec_wrapper_t xyi = XY.get_row(i); // cannot use auto& here. must specify type. - - svec_x_dmat(xi, Y, xyi); - } - } - - float get_bias(const svec_wrapper_t& x, const pecos::drm_t& W) const { - float bias = 0; - - // add ||xW||^2 term. - dvec_t xW; - xW.resize(this->k_size, 0); - dvec_wrapper_t xW_(xW); - svec_x_dmat(x, W, xW_); - bias += pecos::do_dot_product(xW_, xW_); - - // subtract diagonal term. - size_t x_nnz = x.get_nnz(); - for (size_t i = 0; i < x_nnz; ++i) { - const auto& wi = W.get_row(x.idx[i]); - bias -= x.val[i] * x.val[i] * pecos::do_dot_product(wi, wi); - } - - bias /= 2.; - return bias; - } - - template - void build_index(const MAT& Z) { - // Z.shape = [# of items, wz_size] -> sparse - // Wz.shape = [wz_size, k_size] -> dense - this->num_zs = Z.rows; - this->Z_embs.resize(this->num_zs * this->k_size, 0); - this->Z_bias.resize(this->num_zs, 0); - - // wrap shit in matrix - pecos::drm_t Z_embs_; - Z_embs_.rows = this->num_zs; - Z_embs_.cols = this->k_size; - Z_embs_.val = this->Z_embs.data(); - - pecos::drm_t Wz_; - Wz_.rows = this->wz_size; - Wz_.cols = this->k_size; - Wz_.val = this->Wz.data(); - - // build embeddings. - smat_x_dmat(Z, Wz_, Z_embs_); - - // build bias. - for (size_t i = 0; i < this->num_zs; ++i) { - const auto& zi = Z.get_row(i); - this->Z_bias[i] = get_bias(zi, Wz_); - } - } - - template - value_type inference(const VEC& x, const index_type z_idx) { - pecos::drm_t Wx_; - Wx_.rows = this->wx_size; - Wx_.cols = this->k_size; - Wx_.val = this->Wx.data(); - - // get x embedding. - dvec_t x_emb; - x_emb.resize(this->k_size, 0); - dvec_wrapper_t x_emb_(x_emb); - svec_x_dmat(x, Wx_, x_emb_); - - // get x bias. - float x_bias = get_bias(x, Wx_); - - // wrap Z_embs. - pecos::drm_t Z_embs_; - Z_embs_.rows = this->num_zs; - Z_embs_.cols = this->k_size; - Z_embs_.val = this->Z_embs.data(); - - dvec_wrapper_t z_emb_(Z_embs_.get_row(z_idx)); - const float z_bias = this->Z_bias[z_idx]; - - value_type score = pecos::do_dot_product(x_emb_, z_emb_) + x_bias + z_bias; - return score; - } - - template - max_heap_t& ranking(const VEC& x, const std::vector item_ids, const index_type topk) { - pecos::drm_t Wx_; - Wx_.rows = this->wx_size; - Wx_.cols = this->k_size; - Wx_.val = this->Wx.data(); - - // get x embedding. - dvec_t x_emb; - x_emb.resize(this->k_size, 0); - dvec_wrapper_t x_emb_(x_emb); - svec_x_dmat(x, Wx_, x_emb_); - - // get x bias. - float x_bias = get_bias(x, Wx_); - - // wrap Z_embs. - pecos::drm_t Z_embs_; - Z_embs_.rows = this->num_zs; - Z_embs_.cols = this->k_size; - Z_embs_.val = this->Z_embs.data(); - - max_heap_t topk_queue; - - for (auto i : item_ids) { - if (i >= this->num_zs) { - throw std::runtime_error("Item ids should be less then total number of items."); - } - dvec_wrapper_t z_emb_(Z_embs_.get_row(i)); - const value_type z_bias = this->Z_bias[i]; - - value_type score = pecos::do_dot_product(x_emb_, z_emb_) + x_bias + z_bias; - topk_queue.emplace(score, i); - } - - while (topk_queue.size() > topk) { - topk_queue.pop(); - } - - return topk_queue; - } - -}; -} // end namespace pecos - -#endif // end of __INFERENCE_H__ diff --git a/examples/fm-for-xmc/xmc/fm_solver.hpp b/examples/fm-for-xmc/xmc/fm_solver.hpp deleted file mode 100644 index ff45efb7..00000000 --- a/examples/fm-for-xmc/xmc/fm_solver.hpp +++ /dev/null @@ -1,628 +0,0 @@ -#ifndef __FM_SOLVER_H__ -#define __FM_SOLVER_H__ - -#include -#include -#include -#include "utils/matrix.hpp" -#include "utils/parallel.hpp" -#include "utils/random.hpp" -#include "utils/newton.hpp" - -#include "xmc/fm_utils.hpp" - -#ifdef USEOMP -#include -#endif - - -namespace pecos { - -namespace fm_solver { - -enum SolverType { - L2R_LOGLOSS_ADAGRAD=1 -}; - -struct FMParameter { - FMParameter( - int solver_type=L2R_LOGLOSS_ADAGRAD, - int max_iter=10, - float eta=0.02, - float reg=0.00002, - bool auto_stop=false, - bool identity_biased_init=false, - bool factorized=false - ): solver_type(solver_type), max_iter(max_iter), eta(eta), reg(reg), auto_stop(auto_stop), factorized(factorized) {} - - int solver_type; - size_t max_iter; - float eta, reg; - bool auto_stop, factorized, identity_biased_init; -}; - -#define INF HUGE_VAL -template -struct FMWorker { - - typedef std::vector dvec_t; - typedef sparse_vec_t svec_wrapper_t; - typedef dense_vec_t dvec_wrapper_t; - typedef random_number_generator<> rng_t; - - FMParameter param; - // u64_dvec_t index; // used to determine the subset of rows of X are used in the training. - dvec_t Wx, Wz; - index_type wx_size, wz_size, k_size; - - FMWorker(): wx_size(0), wz_size(0), k_size(0) {} - - void init(index_type wx_size, index_type wz_size, index_type k_size, const FMParameter *param_ptr=NULL) { - if(param_ptr != NULL) { - param = *param_ptr; - } - this->wx_size = wx_size; - this->wz_size = wz_size; - this->k_size = k_size; - // this->y_nnz = y_nnz; - - Wx.resize(wx_size * k_size, 0); - Wz.resize(wz_size * k_size, 0); - - // this->index.reserve(this->y_nnz); - } - - void lazy_init(size_t wx_size, size_t wz_size, size_t k_size, const FMParameter *param_ptr=NULL) { - if((wx_size != this->wx_size) - || (wz_size != this->wz_size) - || (k_size != this->k_size) - || ((param_ptr != NULL) && (param_ptr->solver_type != param.solver_type))) { - init(wx_size, wz_size, k_size, param_ptr); - } else { - param = *param_ptr; - } - } - - template - void solve(const XZ_MAT& X, const XZ_MAT& Z, const Y_MAT& Y, - const XZ_MAT& val_X, const XZ_MAT& val_Z, const Y_MAT& val_Y, - int seed=0) { - if(param.solver_type == L2R_LOGLOSS_ADAGRAD) { - solve_l2r_logloss_adagrad(X, Z, Y, val_X, val_Z, val_Y, seed); - } - } - - template - float forward(const VEC& xi, const VEC& zi) { - - pecos::drm_t curr_Wx; - curr_Wx.rows = this->wx_size; - curr_Wx.cols = this->k_size; - curr_Wx.val = this->Wx.data(); - - pecos::drm_t curr_Wz; - curr_Wz.rows = this->wz_size; - curr_Wz.cols = this->k_size; - curr_Wz.val = this->Wz.data(); - - const size_t x_nnz = xi.get_nnz(); - const size_t z_nnz = zi.get_nnz(); - - float v1, v2; - dvec_wrapper_t w1, w2; - - float t = 0; - // for loop over Xs and Zs - for (size_t j1 = 0; j1 < x_nnz + z_nnz; ++j1) { - - if (j1 < x_nnz) { - v1 = xi.val[j1]; - w1 = curr_Wx.get_row(get_ind(xi, j1)); - } - else { - v1 = zi.val[j1 - x_nnz]; - w1 = curr_Wz.get_row(get_ind(zi, j1 - x_nnz)); - } - - for (size_t j2 = j1 + 1; j2 < x_nnz + z_nnz; ++j2) { - - if (j2 < x_nnz) { - v2 = xi.val[j2]; - w2 = curr_Wx.get_row(get_ind(xi, j2)); - } - else { - v2 = zi.val[j2 - x_nnz]; - w2 = curr_Wz.get_row(get_ind(zi, j2 - x_nnz)); - } - t += pecos::do_dot_product(w1.val, w2.val, this->k_size) * v1 * v2; - } - } - return t; - } - - template - void backward(const VEC& xi, const VEC& zi, float kappa, drm_t& Gx, drm_t& Gz) { - - const size_t x_nnz = xi.get_nnz(); - const size_t z_nnz = zi.get_nnz(); - - #ifdef VECGRAD - // allocate space to place gradients. - dvec_t g1, g2; - g1.resize(this->k_size, 0); - g2.resize(this->k_size, 0); - - dvec_wrapper_t curr_g1(g1); - dvec_wrapper_t curr_g2(g2); - #endif - - // wrap vectors in matrices - pecos::drm_t curr_Wx; - curr_Wx.rows = this->wx_size; - curr_Wx.cols = this->k_size; - curr_Wx.val = this->Wx.data(); - - pecos::drm_t curr_Wz; - curr_Wz.rows = this->wz_size; - curr_Wz.cols = this->k_size; - curr_Wz.val = this->Wz.data(); - - for (size_t j1 = 0; j1 < x_nnz + z_nnz; ++j1) { - - float v1, v2; - dvec_wrapper_t w1, w2, G1, G2; - - if (j1 < x_nnz) { - v1 = xi.val[j1]; - w1 = curr_Wx.get_row(get_ind(xi, j1)); - G1 = Gx.get_row(get_ind(xi, j1)); - } - else { - v1 = zi.val[j1 - x_nnz]; - w1 = curr_Wz.get_row(get_ind(zi, j1 - x_nnz)); - G1 = Gz.get_row(get_ind(zi, j1 - x_nnz)); - } - - for (size_t j2 = j1 + 1; j2 < x_nnz + z_nnz; ++j2) { - - if (j2 < x_nnz) { - v2 = xi.val[j2]; - w2 = curr_Wx.get_row(get_ind(xi, j2)); - G2 = Gx.get_row(get_ind(xi, j2)); - } - else { - v2 = zi.val[j2 - x_nnz]; - w2 = curr_Wz.get_row(get_ind(zi, j2 - x_nnz)); - G2 = Gz.get_row(get_ind(zi, j2 - x_nnz)); - } - - float multiplier = kappa * v1 * v2; - - #ifdef VECGRAD - // zero_grad - std::fill(g1.begin(), g1.end(), 0); - std::fill(g2.begin(), g2.end(), 0); - - // g1 = lambda * w1 + kappa * w2 * v1 * v2. - // g2 = lambda * w2 + kappa * w1 * v1 * v2. - - pecos::do_axpy(this->param.reg, w1.val, curr_g1.val, this->k_size); - pecos::do_axpy(multiplier, w2.val, curr_g1.val, this->k_size); - - pecos::do_axpy(this->param.reg, w2.val, curr_g2.val, this->k_size); - pecos::do_axpy(multiplier, w1.val, curr_g2.val, this->k_size); - #endif - - // accumulate gradient square. - // pecos::do_ax2py(1.0, curr_g1, G1); - // pecos::do_ax2py(1.0, curr_g2, G2); - #pragma GCC ivdep - for (size_t d = 0; d < this->k_size; ++d) { - - #ifdef VECGRAD - float g1_ = g1[d]; - float g2_ = g2[d]; - #else - float g1_ = this->param.reg * w1.val[d] + multiplier * w2.val[d]; - float g2_ = this->param.reg * w2.val[d] + multiplier * w1.val[d]; - #endif - - G1.val[d] += g1_ * g1_; - G2.val[d] += g2_ * g2_; - - // descent. - w1.val[d] -= this->param.eta / std::sqrt(G1.val[d]) * g1_; - w2.val[d] -= this->param.eta / std::sqrt(G2.val[d]) * g2_; - } - } - } - } - - template - float forward_factorized(const VEC& x, const VEC& z, dvec_wrapper_t& emb_sum) { - - pecos::drm_t curr_Wx; - curr_Wx.rows = this->wx_size; - curr_Wx.cols = this->k_size; - curr_Wx.val = this->Wx.data(); - - pecos::drm_t curr_Wz; - curr_Wz.rows = this->wz_size; - curr_Wz.cols = this->k_size; - curr_Wz.val = this->Wz.data(); - - dvec_t ex, ez; - ex.resize(this->k_size, 0); - ez.resize(this->k_size, 0); - - dvec_wrapper_t curr_ex(ex); - dvec_wrapper_t curr_ez(ez); - - // generate embeddings O(dk). - mat_x_vec(curr_Wx, x, curr_ex); - mat_x_vec(curr_Wz, z, curr_ez); - - // for (size_t i = 0; i < this->k_size; ++i) std::cout << curr_ex.val[i] << " "; - // std::cout << std::endl; - // for (size_t i = 0; i < this->k_size; ++i) std::cout << curr_ez.val[i] << " "; - // std::cout << std::endl; - - float x_bias = get_bias(x, curr_Wx), z_bias = get_bias(z, curr_Wz); - - // std::cout << x_bias << " " << z_bias << std::endl; - - float t = pecos::do_dot_product(curr_ex.val, curr_ez.val, this->k_size) + x_bias + z_bias; - - pecos::do_axpy(1, curr_ex.val, emb_sum.val, this->k_size); - pecos::do_axpy(1, curr_ez.val, emb_sum.val, this->k_size); - - return t; - } - - template - void backward_factorized(const VEC& x, const VEC& z, dvec_wrapper_t& emb_sum, float kappa, - drm_t& Gx, drm_t& Gz) { - - pecos::drm_t curr_Wx; - curr_Wx.rows = this->wx_size; - curr_Wx.cols = this->k_size; - curr_Wx.val = this->Wx.data(); - - pecos::drm_t curr_Wz; - curr_Wz.rows = this->wz_size; - curr_Wz.cols = this->k_size; - curr_Wz.val = this->Wz.data(); - - const size_t x_nnz = x.get_nnz(); - const size_t z_nnz = z.get_nnz(); - - /* - dvec_t dummy; - dummy.resize(this->k_size, 0); - dvec_wrapper_t dummy_(dummy); - - dvec_t dummy2; - dummy2.resize(this->k_size, 0); - dvec_wrapper_t dummy2_(dummy2); - */ - - for (size_t d = 0; d < x_nnz + z_nnz; ++d) { - float v; - dvec_wrapper_t w, G; - - if (d < x_nnz) { - w = curr_Wx.get_row(get_ind(x, d)); - G = Gx.get_row(get_ind(x, d)); - v = x.val[d]; - } - else { - w = curr_Wz.get_row(get_ind(z, d - x_nnz)); - G = Gz.get_row(get_ind(z, d - x_nnz)); - v = z.val[d - x_nnz]; - } - #pragma GCC ivdep - for (size_t j = 0; j < this->k_size; ++j) { - float g = kappa * (emb_sum.val[j] - w.val[j] * v) * v + this->param.reg * w.val[j]; - G.val[j] += g * g; - w.val[j] -= this->param.eta / std::sqrt(G.val[j]) * g; - } - } - } - - template - long double eval_loss(const XZ_MAT &X, const XZ_MAT& Z, const Y_MAT& Y) { - - long double loss = 0; - -#ifdef USEOMP -#pragma omp parallel for schedule(static) reduction(+: loss) -#endif - for(size_t i = 0; i < Y.rows; ++i) { - - dvec_t dummy; - dummy.resize(this->k_size, 0); - dvec_wrapper_t dummy_(dummy); - - const auto& xi = X.get_row(i); - const auto& yi = Y.get_row(i); - - for (size_t j = 0; j < yi.get_nnz(); ++j) { - - const auto& zi = Z.get_row(yi.idx[j]); - - const float y = yi.val[j]; - - const double t = this->param.factorized ? forward_factorized(xi, zi, dummy_) : forward(xi, zi); - - const double expnyt = std::exp(-y * t); - loss += std::log1p(expnyt); - } - } - - loss /= Y.get_nnz(); - - return loss; - } - - void save(FILE *fp) const { - pecos::file_util::fput_multiple(&(this->wx_size), 1, fp); - pecos::file_util::fput_multiple(&(this->wz_size), 1, fp); - pecos::file_util::fput_multiple(&(this->k_size), 1, fp); - - pecos::file_util::fput_multiple(&(this->Wx[0]), this->wx_size * this->k_size, fp); - pecos::file_util::fput_multiple(&(this->Wz[0]), this->wz_size * this->k_size, fp); - } - - void load(FILE *fp) { - pecos::file_util::fget_multiple(&(this->wx_size), 1, fp); - pecos::file_util::fget_multiple(&(this->wz_size), 1, fp); - pecos::file_util::fget_multiple(&(this->k_size), 1, fp); - - this->Wx.resize(this->wx_size * this->k_size, 0); - pecos::file_util::fget_multiple(&(this->Wx[0]), this->wx_size * this->k_size, fp); - - this->Wz.resize(this->wz_size * this->k_size, 0); - pecos::file_util::fget_multiple(&(this->Wz[0]), this->wz_size * this->k_size, fp); - } - - template - void solve_l2r_logloss_adagrad(const XZ_MAT &X, const XZ_MAT& Z, const Y_MAT& Y, - const XZ_MAT &val_X, const XZ_MAT& val_Z, const Y_MAT& val_Y, int seed) { - rng_t rng(seed); - // initialize index by copy pointers to Y - u64_dvec_t y_rows, y_cols, index; - dvec_t y_vals; - - int cnt = 0; - for(size_t i = 0; i < Y.rows; ++i) { - const auto& yi = Y.get_row(i); - for (size_t j = 0; j < yi.get_nnz(); ++j) { - y_rows.push_back(i); - y_cols.push_back(yi.idx[j]); - y_vals.push_back(yi.val[j]); - index.push_back(cnt); - ++cnt; - } - } - - // initialize weights according to paper. - // Juan et al. (2016) Section 3.1 - const float sqrt_k = 1/std::sqrt(this->k_size); - #ifdef DETERMINISTIC - for(size_t i = 0; i < this->wx_size * this->k_size; i++) { - this->Wx[i] = sqrt_k / 2; - } - for(size_t i = 0; i < this->wz_size * this->k_size; i++) { - this->Wz[i] = sqrt_k / 2; - } - #else - for(size_t i = 0; i < this->wx_size * this->k_size; i++) { - this->Wx[i] = rng.uniform(0.0, sqrt_k); - } - if (param.identity_biased_init && (this->wx_size == this->wz_size)) { - std::cout << "Initialize weight with identity bias." << std::endl; - for(size_t i = 0; i < this->wz_size * this->k_size; i++) { - this->Wz[i] = this->Wx[i]; - } - } - else { - for(size_t i = 0; i < this->wz_size * this->k_size; i++) { - this->Wz[i] = rng.uniform(0.0, sqrt_k); - } - } - #endif - - // for implementing auto-stop - dvec_t prev_Wx; - dvec_t prev_Wz; - - // initialize G to 1 (gradient square sum) - // Juan et al. (2016) Section 3.1 - dvec_t Gx; - dvec_t Gz; - Gx.resize(wx_size * k_size, 1); - Gz.resize(wz_size * k_size, 1); - - // start training loop. - pecos::StopW stopw = pecos::StopW(); - - long double va_loss = std::numeric_limits::max(), - best_va_loss = std::numeric_limits::max(); - float momentum = 0.9; - int log_freq = 10000; - - for (size_t iter = 0; iter < param.max_iter; ++iter) { - - stopw.reset(); - - #ifndef DETERMINISTIC - // shuffle - rng.shuffle(index.begin(), index.end()); - #endif - - long double loss = 0; - int progress = 0; - double elapsed_time = 0, ema_ipt = 0; - -#ifdef USEOMP -#pragma omp parallel for schedule(static) reduction(+: loss) -#endif - for(auto i = index.begin(); i < index.end(); i++) { // cannot use C++11 syntactic sugar for OpenMP - // retreive data for given index. - const size_t ri = y_rows[*i]; - const size_t ci = y_cols[*i]; - const float yi = y_vals[*i]; - - const auto& xi = X.get_row(ri); - const auto& zi = Z.get_row(ci); - - pecos::drm_t curr_Gx; - curr_Gx.rows = this->wx_size; - curr_Gx.cols = this->k_size; - curr_Gx.val = Gx.data(); - - pecos::drm_t curr_Gz; - curr_Gz.rows = this->wz_size; - curr_Gz.cols = this->k_size; - curr_Gz.val = Gz.data(); - - // skip instance if feature is empty - if (xi.get_nnz() + zi.get_nnz() <= 1) continue; - - if (this->param.factorized) { - // allocate space to place embeddings. - dvec_t emb_sum; - emb_sum.resize(this->k_size, 0); - dvec_wrapper_t curr_emb_sum(emb_sum); - - // forward get embeddings ex, ez, and t = phi(x, z) - double t = forward_factorized(xi, zi, curr_emb_sum); - - double expnyt = std::exp(-yi * t); - loss += std::log1p(expnyt); - float kappa = -yi * expnyt / (1 + expnyt); - // std::cout << loss << std::endl; - - // backward - backward_factorized(xi, zi, curr_emb_sum, kappa, curr_Gx, curr_Gz); - } - else { - double t = forward(xi, zi); - - double expnyt = std::exp(-yi * t); - loss += std::log1p(expnyt); - float kappa = -yi * expnyt / (1 + expnyt); - - backward(xi, zi, kappa, curr_Gx, curr_Gz); - } - - if (std::isnan(loss)) { - throw std::overflow_error("Overflow in loss result in NaN. Considering reducing learning rate or increasing weight regularization."); - } - - #ifndef USEOMP - progress++; - - // show progress bar. - if (progress % log_freq == 0) { - double curr_elapsed_time = stopw.getElapsedTimeMicro() / 1000000.; - ema_dt = momentum * ema_dt + (1 - momentum) * (curr_elapsed_time - elapsed_time); - elapsed_time = curr_elapsed_time; - log_progress(progress, index.size(), elapsed_time, emd_dt); - } - #else - if(omp_get_thread_num() == 0) { - progress++; - - // show progress bar. - if (progress % log_freq == 0) { - double curr_elapsed_time = stopw.getElapsedTimeMicro() / 1000000.; - double ipt = omp_get_num_threads() * log_freq / (curr_elapsed_time - elapsed_time); - ema_ipt = momentum * ema_ipt + (1 - momentum) * (ipt); - elapsed_time = curr_elapsed_time; - log_progress(progress * omp_get_num_threads(), index.size(), elapsed_time, ema_ipt); - } - } - #endif - } - - float trn_time = stopw.getElapsedTimeMicro(); - - loss /= index.size(); - - - va_loss = eval_loss(val_X, val_Z, val_Y); - if (va_loss > best_va_loss) { - if (this->param.auto_stop) { - std::cout << std::endl << "Auto-stop. Use model at " << iter << "th iteration." << std::endl; - break; - } - } - else { - prev_Wx = Wx; - prev_Wz = Wz; - best_va_loss = va_loss; - } - - double sum_G = 0, cnt_G = 0; - for(auto& i : Gx) { - if (i == 1) continue; - sum_G += i; - cnt_G += 1; - } - for(auto& i : Gz) { - if (i != 1) continue; - sum_G += i; - cnt_G += 1; - } - - // flush out log_progress. - std::cerr << "\t\t\t\t\t\t\t\t\t\t\r"; - std::cerr.flush(); - - std::cout << std::fixed << std::setprecision(10); - std::cout << "iter: " << iter + 1 - << " logloss: " << loss - << " va_logloss: " << va_loss - << " trn_time: " << trn_time / 1000000. - << " avg_G: " << sum_G / cnt_G - << std::endl; - } - // restore best model - Wx = prev_Wx; - Wz = prev_Wz; - } - - void log_progress(int step, int total_steps, double elapsed_time, double ema_ipt) { - float progress_ratio = (float)step / total_steps; - - std::cout << "["; - int barWidth = 70; - int pos = barWidth * progress_ratio; - for (int i = 0; i < barWidth; ++i) { - if (i < pos) std::cout << "="; - else if (i == pos) std::cout << ">"; - else std::cout << " "; - } - // double elapsed_time = stopw.getElapsedTimeMicro() / 1000000.; - int elapsed_mins = (int)elapsed_time / 60, elapsed_secs = (int)elapsed_time % 60; - - double remaining_time = (total_steps - step) / ema_ipt; - int remaining_mins = (int)remaining_time / 60, remaining_secs = (int)remaining_time % 60; - - std::cerr << std::fixed << std::setprecision(3) - << "] " << (progress_ratio * 100.0) << "% " - << elapsed_mins << "m" << elapsed_secs << "s" - << " < " - << remaining_mins << "m" << remaining_secs << "s" - << " (" << ema_ipt << "instance/s)" - << "\t\t\t\t\r"; - std::cerr.flush(); - } - -}; - -} // end of namespace fm_solver -} // end of namespace pecos - -#endif // end of __FM_SOLVER_H__ diff --git a/examples/fm-for-xmc/xmc/fm_utils.hpp b/examples/fm-for-xmc/xmc/fm_utils.hpp deleted file mode 100644 index 65f4a278..00000000 --- a/examples/fm-for-xmc/xmc/fm_utils.hpp +++ /dev/null @@ -1,187 +0,0 @@ -/* - * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance - * with the License. A copy of the License is located at - * - * http://aws.amazon.com/apache2.0/ - * - * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES - * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions - * and limitations under the License. - */ - -#ifndef __FM_UTILS_H__ -#define __FM_UTILS_H__ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "utils/parallel.hpp" -#include "utils/scipy_loader.hpp" -#include "utils/matrix.hpp" - -namespace pecos { - -class StopW { - std::chrono::steady_clock::time_point time_begin; -public: - StopW() { - time_begin = std::chrono::steady_clock::now(); - } - - float getElapsedTimeMicro() { - std::chrono::steady_clock::time_point time_end = std::chrono::steady_clock::now(); - return (std::chrono::duration_cast(time_end - time_begin).count()); - } - - void reset() { - time_begin = std::chrono::steady_clock::now(); - } - -}; - - /* - template - void dmat_x_dvec(const M_MAT& M, const dense_vec_t& x, dense_vec_t& Mx) { - // TODO: parallelize. - for (size_t i = 0; i < M.cols; ++i) { - const auto& mi = M.get_col(i); - Mx.val[i] = pecos::do_dot_product(mi.val, x.val, M.rows); - } - } - - template - void svec_x_dmat(const sparse_vec_t& x, const Y_MAT& Y, dense_vec_t& xY) { - // TODO: parallelize. - for (size_t i = 0; i < x.nnz; ++i) { - const auto& yi = Y.get_row(x.idx[i]); - pecos::do_axpy(x.val[i], yi.val, xY.val, Y.cols); - } - } - - template - void smat_x_dmat(const X_MAT& X, const Y_MAT& Y, const Y_MAT& XY) { - // TODO: parallelize. - for (size_t i = 0; i < X.rows; ++i) { - auto& xi = X.get_row(i); - auto& xyi = XY.get_row(i); - - svec_x_dmat(xi, Y, xyi); - } - } - */ - - template - inline IX get_ind(const dense_vec_t& x, const IX i) { - return i; - } - - template - inline IX get_ind(const sparse_vec_t& x, const II i) { - return x.idx[i]; - } - - template - void mat_x_vec(const drm_t& M, const X_VEC& x, MX_VEC& Mx) { - // TODO: parallelize. - for (size_t i = 0; i < x.get_nnz(); ++i) { - const auto& mi = M.get_row(get_ind(x, i)); - pecos::do_axpy(x.val[i], mi.val, Mx.val, M.cols); - } - } - - template - void mat_x_mat(const drm_t& M, const MAT& X, const drm_t& MX) { - // TODO: parallelize. - for (size_t i = 0; i < X.rows; ++i) { - auto& xi = X.get_row(i); - auto& mxi = MX.get_row(i); - - mat_x_vec(M, xi, mxi); - } - } - - template - float get_bias(const VEC& x, const drm_t& M) { - float bias = 0; - - // add ||xW||^2 term. - std::vector Mx; - Mx.resize(M.cols, 0); - - dense_vec_t Mx_(Mx); - mat_x_vec(M, x, Mx_); - bias += pecos::do_dot_product(Mx_.val, Mx_.val, M.cols); - - // subtract diagonal term. - for (size_t i = 0; i < x.get_nnz(); ++i) { - const auto& mi = M.get_row(get_ind(x, i)); - bias -= x.val[i] * x.val[i] * pecos::do_dot_product(mi.val, mi.val, M.cols); - } - - bias /= 2.; - return bias; - } - - /* - template - VX get_bias(const sparse_vec_t& x, const M_MAT& M) { - VX bias = 0; - - // add ||xW||^2 term. - std::vector xM; - xM.resize(M.cols, 0); - - dense_vec_t xM_(xM); - svec_x_dmat(x, M, xM_); - bias += pecos::do_dot_product(xM_.val, xM_.val, M.cols); - - // subtract diagonal term. - for (size_t i = 0; i < x.nnz; ++i) { - const auto& mi = M.get_row(x.idx[i]); - bias -= x.val[i] * x.val[i] * pecos::do_dot_product(mi.val, mi.val, M.cols); - } - - bias /= 2.; - return bias; - } - - - template - VX get_bias(const dense_vec_t& x, const M_MAT& M) { - VX bias = 0; - - // add ||xW||^2 term. - std::vector Mx; - Mx.resize(M.cols, 0); - - dense_vec_t Mx_(Mx); - dmat_x_dvec(M, x, Mx_); - bias += pecos::do_dot_product(Mx_.val, Mx_.val, M.cols); - - // subtract diagonal term. - for (size_t i = 0; i < M.cols; ++i) { // k - const auto& mi = M.get_col(i); - for (size_t j = 0; j < M.rows; ++j) { // d - bias -= x.val[j] * x.val[j] * mi.val[j] * mi.val[j]; - } - } - - bias /= 2.; - return bias; - } - */ - -} // end namespace pecos - -#endif // end of __FM_UTILS_H__ diff --git a/examples/giant-xrt/OGB_baselines/ogbn-arxiv/gnn.py b/examples/giant-xrt/OGB_baselines/ogbn-arxiv/gnn.py deleted file mode 100644 index b513a7c7..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-arxiv/gnn.py +++ /dev/null @@ -1,185 +0,0 @@ -import argparse - -import torch -import torch.nn.functional as F - -import torch_geometric.transforms as T -from torch_geometric.nn import GCNConv, SAGEConv - -from ogb.nodeproppred import PygNodePropPredDataset, Evaluator - -from logger import Logger -import numpy as np -from pecos.utils import smat_util - - -class GCN(torch.nn.Module): - def __init__(self, in_channels, hidden_channels, out_channels, num_layers, - dropout): - super(GCN, self).__init__() - - self.convs = torch.nn.ModuleList() - self.convs.append(GCNConv(in_channels, hidden_channels, cached=True)) - self.bns = torch.nn.ModuleList() - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - for _ in range(num_layers - 2): - self.convs.append( - GCNConv(hidden_channels, hidden_channels, cached=True)) - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - self.convs.append(GCNConv(hidden_channels, out_channels, cached=True)) - - self.dropout = dropout - - def reset_parameters(self): - for conv in self.convs: - conv.reset_parameters() - for bn in self.bns: - bn.reset_parameters() - - def forward(self, x, adj_t): - for i, conv in enumerate(self.convs[:-1]): - x = conv(x, adj_t) - x = self.bns[i](x) - x = F.relu(x) - x = F.dropout(x, p=self.dropout, training=self.training) - x = self.convs[-1](x, adj_t) - return x.log_softmax(dim=-1) - - -class SAGE(torch.nn.Module): - def __init__(self, in_channels, hidden_channels, out_channels, num_layers, - dropout): - super(SAGE, self).__init__() - - self.convs = torch.nn.ModuleList() - self.convs.append(SAGEConv(in_channels, hidden_channels)) - self.bns = torch.nn.ModuleList() - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - for _ in range(num_layers - 2): - self.convs.append(SAGEConv(hidden_channels, hidden_channels)) - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - self.convs.append(SAGEConv(hidden_channels, out_channels)) - - self.dropout = dropout - - def reset_parameters(self): - for conv in self.convs: - conv.reset_parameters() - for bn in self.bns: - bn.reset_parameters() - - def forward(self, x, adj_t): - for i, conv in enumerate(self.convs[:-1]): - x = conv(x, adj_t) - x = self.bns[i](x) - x = F.relu(x) - x = F.dropout(x, p=self.dropout, training=self.training) - x = self.convs[-1](x, adj_t) - return x.log_softmax(dim=-1) - - -def train(model, data, train_idx, optimizer): - model.train() - - optimizer.zero_grad() - out = model(data.x, data.adj_t)[train_idx] - loss = F.nll_loss(out, data.y.squeeze(1)[train_idx]) - loss.backward() - optimizer.step() - - return loss.item() - - -@torch.no_grad() -def test(model, data, split_idx, evaluator): - model.eval() - - out = model(data.x, data.adj_t) - y_pred = out.argmax(dim=-1, keepdim=True) - - train_acc = evaluator.eval({ - 'y_true': data.y[split_idx['train']], - 'y_pred': y_pred[split_idx['train']], - })['acc'] - valid_acc = evaluator.eval({ - 'y_true': data.y[split_idx['valid']], - 'y_pred': y_pred[split_idx['valid']], - })['acc'] - test_acc = evaluator.eval({ - 'y_true': data.y[split_idx['test']], - 'y_pred': y_pred[split_idx['test']], - })['acc'] - - return train_acc, valid_acc, test_acc - - -def main(): - parser = argparse.ArgumentParser(description='OGBN-Arxiv (GNN)') - parser.add_argument('--device', type=int, default=0) - parser.add_argument('--log_steps', type=int, default=1) - parser.add_argument('--use_sage', action='store_true') - parser.add_argument('--num_layers', type=int, default=3) - parser.add_argument('--hidden_channels', type=int, default=256) - parser.add_argument('--dropout', type=float, default=0.5) - parser.add_argument('--lr', type=float, default=0.001) - parser.add_argument('--epochs', type=int, default=500) - parser.add_argument('--runs', type=int, default=10) - parser.add_argument('--data_root_dir', type=str, default='../../dataset') - parser.add_argument('--node_emb_path', type=str, default=None) - args = parser.parse_args() - - print(args) - - device = f'cuda:{args.device}' if torch.cuda.is_available() else 'cpu' - device = torch.device(device) - - dataset = PygNodePropPredDataset(name='ogbn-arxiv',root=args.data_root_dir, - transform=T.ToSparseTensor()) - - data = dataset[0] - data.adj_t = data.adj_t.to_symmetric() - - if args.node_emb_path: - data.x = torch.from_numpy(smat_util.load_matrix(args.node_emb_path).astype(np.float32)) - print("Loaded pre-trained node embeddings of shape={} from {}".format(data.x.shape, args.node_emb_path)) - - data = data.to(device) - - split_idx = dataset.get_idx_split() - train_idx = split_idx['train'].to(device) - - if args.use_sage: - model = SAGE(data.num_features, args.hidden_channels, - dataset.num_classes, args.num_layers, - args.dropout).to(device) - else: - model = GCN(data.num_features, args.hidden_channels, - dataset.num_classes, args.num_layers, - args.dropout).to(device) - - evaluator = Evaluator(name='ogbn-arxiv') - logger = Logger(args.runs, args) - - for run in range(args.runs): - model.reset_parameters() - optimizer = torch.optim.Adam(model.parameters(), lr=args.lr) - for epoch in range(1, 1 + args.epochs): - loss = train(model, data, train_idx, optimizer) - result = test(model, data, split_idx, evaluator) - logger.add_result(run, result) - - if epoch % args.log_steps == 0: - train_acc, valid_acc, test_acc = result - print(f'Run: {run + 1:02d}, ' - f'Epoch: {epoch:02d}, ' - f'Loss: {loss:.4f}, ' - f'Train: {100 * train_acc:.2f}%, ' - f'Valid: {100 * valid_acc:.2f}% ' - f'Test: {100 * test_acc:.2f}%') - - logger.print_statistics(run) - logger.print_statistics() - - -if __name__ == "__main__": - main() diff --git a/examples/giant-xrt/OGB_baselines/ogbn-arxiv/logger.py b/examples/giant-xrt/OGB_baselines/ogbn-arxiv/logger.py deleted file mode 100644 index b6e617ba..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-arxiv/logger.py +++ /dev/null @@ -1,44 +0,0 @@ -import torch - - -class Logger(object): - def __init__(self, runs, info=None): - self.info = info - self.results = [[] for _ in range(runs)] - - def add_result(self, run, result): - assert len(result) == 3 - assert run >= 0 and run < len(self.results) - self.results[run].append(result) - - def print_statistics(self, run=None): - if run is not None: - result = 100 * torch.tensor(self.results[run]) - argmax = result[:, 1].argmax().item() - print(f'Run {run + 1:02d}:') - print(f'Highest Train: {result[:, 0].max():.2f}') - print(f'Highest Valid: {result[:, 1].max():.2f}') - print(f' Final Train: {result[argmax, 0]:.2f}') - print(f' Final Test: {result[argmax, 2]:.2f}') - else: - result = 100 * torch.tensor(self.results) - - best_results = [] - for r in result: - train1 = r[:, 0].max().item() - valid = r[:, 1].max().item() - train2 = r[r[:, 1].argmax(), 0].item() - test = r[r[:, 1].argmax(), 2].item() - best_results.append((train1, valid, train2, test)) - - best_result = torch.tensor(best_results) - - print(f'All runs:') - r = best_result[:, 0] - print(f'Highest Train: {r.mean():.2f} ± {r.std():.2f}') - r = best_result[:, 1] - print(f'Highest Valid: {r.mean():.2f} ± {r.std():.2f}') - r = best_result[:, 2] - print(f' Final Train: {r.mean():.2f} ± {r.std():.2f}') - r = best_result[:, 3] - print(f' Final Test: {r.mean():.2f} ± {r.std():.2f}') diff --git a/examples/giant-xrt/OGB_baselines/ogbn-arxiv/mlp.py b/examples/giant-xrt/OGB_baselines/ogbn-arxiv/mlp.py deleted file mode 100644 index dbafaa2e..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-arxiv/mlp.py +++ /dev/null @@ -1,144 +0,0 @@ -import argparse - -import torch -import torch.nn.functional as F - -from ogb.nodeproppred import PygNodePropPredDataset, Evaluator - -from logger import Logger -import numpy as np -from pecos.utils import smat_util - - -class MLP(torch.nn.Module): - def __init__(self, in_channels, hidden_channels, out_channels, num_layers, - dropout): - super(MLP, self).__init__() - - self.lins = torch.nn.ModuleList() - self.lins.append(torch.nn.Linear(in_channels, hidden_channels)) - self.bns = torch.nn.ModuleList() - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - for _ in range(num_layers - 2): - self.lins.append(torch.nn.Linear(hidden_channels, hidden_channels)) - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - self.lins.append(torch.nn.Linear(hidden_channels, out_channels)) - - self.dropout = dropout - - def reset_parameters(self): - for lin in self.lins: - lin.reset_parameters() - for bn in self.bns: - bn.reset_parameters() - - def forward(self, x): - for i, lin in enumerate(self.lins[:-1]): - x = lin(x) - x = self.bns[i](x) - x = F.relu(x) - x = F.dropout(x, p=self.dropout, training=self.training) - x = self.lins[-1](x) - return torch.log_softmax(x, dim=-1) - - -def train(model, x, y_true, train_idx, optimizer): - model.train() - - optimizer.zero_grad() - out = model(x[train_idx]) - loss = F.nll_loss(out, y_true.squeeze(1)[train_idx]) - loss.backward() - optimizer.step() - - return loss.item() - - -@torch.no_grad() -def test(model, x, y_true, split_idx, evaluator): - model.eval() - - out = model(x) - y_pred = out.argmax(dim=-1, keepdim=True) - - train_acc = evaluator.eval({ - 'y_true': y_true[split_idx['train']], - 'y_pred': y_pred[split_idx['train']], - })['acc'] - valid_acc = evaluator.eval({ - 'y_true': y_true[split_idx['valid']], - 'y_pred': y_pred[split_idx['valid']], - })['acc'] - test_acc = evaluator.eval({ - 'y_true': y_true[split_idx['test']], - 'y_pred': y_pred[split_idx['test']], - })['acc'] - - return train_acc, valid_acc, test_acc - - -def main(): - parser = argparse.ArgumentParser(description='OGBN-Arxiv (MLP)') - parser.add_argument('--device', type=int, default=0) - parser.add_argument('--log_steps', type=int, default=1) - parser.add_argument('--use_node_embedding', action='store_true') - parser.add_argument('--num_layers', type=int, default=3) - parser.add_argument('--hidden_channels', type=int, default=256) - parser.add_argument('--dropout', type=float, default=0.5) - parser.add_argument('--lr', type=float, default=0.01) - parser.add_argument('--epochs', type=int, default=500) - parser.add_argument('--runs', type=int, default=10) - parser.add_argument('--data_root_dir', type=str, default='../../dataset') - parser.add_argument('--node_emb_path', type=str, default=None) - args = parser.parse_args() - print(args) - - device = f'cuda:{args.device}' if torch.cuda.is_available() else 'cpu' - device = torch.device(device) - - dataset = PygNodePropPredDataset(name='ogbn-arxiv',root=args.data_root_dir) - split_idx = dataset.get_idx_split() - data = dataset[0] - - if args.node_emb_path: - data.x = torch.from_numpy(smat_util.load_matrix(args.node_emb_path).astype(np.float32)) - print("Loaded pre-trained node embeddings of shape={} from {}".format(data.x.shape, args.node_emb_path)) - - x = data.x - if args.use_node_embedding: - embedding = torch.load('embedding.pt', map_location='cpu') - x = torch.cat([x, embedding], dim=-1) - x = x.to(device) - - y_true = data.y.to(device) - train_idx = split_idx['train'].to(device) - - model = MLP(x.size(-1), args.hidden_channels, dataset.num_classes, - args.num_layers, args.dropout).to(device) - - evaluator = Evaluator(name='ogbn-arxiv') - logger = Logger(args.runs, args) - - for run in range(args.runs): - model.reset_parameters() - optimizer = torch.optim.Adam(model.parameters(), lr=args.lr) - for epoch in range(1, 1 + args.epochs): - loss = train(model, x, y_true, train_idx, optimizer) - result = test(model, x, y_true, split_idx, evaluator) - logger.add_result(run, result) - - if epoch % args.log_steps == 0: - train_acc, valid_acc, test_acc = result - print(f'Run: {run + 1:02d}, ' - f'Epoch: {epoch:02d}, ' - f'Loss: {loss:.4f}, ' - f'Train: {100 * train_acc:.2f}%, ' - f'Valid: {100 * valid_acc:.2f}%, ' - f'Test: {100 * test_acc:.2f}%') - - logger.print_statistics(run) - logger.print_statistics() - - -if __name__ == "__main__": - main() diff --git a/examples/giant-xrt/OGB_baselines/ogbn-papers100M/logger.py b/examples/giant-xrt/OGB_baselines/ogbn-papers100M/logger.py deleted file mode 100644 index b6e617ba..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-papers100M/logger.py +++ /dev/null @@ -1,44 +0,0 @@ -import torch - - -class Logger(object): - def __init__(self, runs, info=None): - self.info = info - self.results = [[] for _ in range(runs)] - - def add_result(self, run, result): - assert len(result) == 3 - assert run >= 0 and run < len(self.results) - self.results[run].append(result) - - def print_statistics(self, run=None): - if run is not None: - result = 100 * torch.tensor(self.results[run]) - argmax = result[:, 1].argmax().item() - print(f'Run {run + 1:02d}:') - print(f'Highest Train: {result[:, 0].max():.2f}') - print(f'Highest Valid: {result[:, 1].max():.2f}') - print(f' Final Train: {result[argmax, 0]:.2f}') - print(f' Final Test: {result[argmax, 2]:.2f}') - else: - result = 100 * torch.tensor(self.results) - - best_results = [] - for r in result: - train1 = r[:, 0].max().item() - valid = r[:, 1].max().item() - train2 = r[r[:, 1].argmax(), 0].item() - test = r[r[:, 1].argmax(), 2].item() - best_results.append((train1, valid, train2, test)) - - best_result = torch.tensor(best_results) - - print(f'All runs:') - r = best_result[:, 0] - print(f'Highest Train: {r.mean():.2f} ± {r.std():.2f}') - r = best_result[:, 1] - print(f'Highest Valid: {r.mean():.2f} ± {r.std():.2f}') - r = best_result[:, 2] - print(f' Final Train: {r.mean():.2f} ± {r.std():.2f}') - r = best_result[:, 3] - print(f' Final Test: {r.mean():.2f} ± {r.std():.2f}') diff --git a/examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_sgc.py b/examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_sgc.py deleted file mode 100644 index fa4d6098..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_sgc.py +++ /dev/null @@ -1,151 +0,0 @@ -import argparse -from tqdm import tqdm - -import torch -import torch.nn.functional as F -from torch.utils.data import Dataset, DataLoader - -from ogb.nodeproppred import Evaluator - -from logger import Logger - - -class SimpleDataset(Dataset): - def __init__(self, x, y): - self.x = x - self.y = y - assert self.x.size(0) == self.y.size(0) - - def __len__(self): - return self.x.size(0) - - def __getitem__(self, idx): - return self.x[idx], self.y[idx] - - -class MLP(torch.nn.Module): - def __init__(self, in_channels, hidden_channels, out_channels, num_layers, - dropout): - super(MLP, self).__init__() - - self.lins = torch.nn.ModuleList() - self.lins.append(torch.nn.Linear(in_channels, hidden_channels)) - self.bns = torch.nn.ModuleList() - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - for _ in range(num_layers - 2): - self.lins.append(torch.nn.Linear(hidden_channels, hidden_channels)) - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - self.lins.append(torch.nn.Linear(hidden_channels, out_channels)) - - self.dropout = dropout - - def reset_parameters(self): - for lin in self.lins: - lin.reset_parameters() - for bn in self.bns: - bn.reset_parameters() - - def forward(self, x): - for i, lin in enumerate(self.lins[:-1]): - x = lin(x) - x = self.bns[i](x) - x = F.relu(x) - x = F.dropout(x, p=self.dropout, training=self.training) - x = self.lins[-1](x) - return torch.log_softmax(x, dim=-1) - - -def train(model, device, train_loader, optimizer): - model.train() - - total_loss = 0 - for x, y in tqdm(train_loader): - x, y = x.to(device), y.to(device) - optimizer.zero_grad() - out = model(x) - loss = F.nll_loss(out, y.squeeze(1)) - loss.backward() - optimizer.step() - total_loss += loss.item() * x.size(0) - return total_loss / len(train_loader.dataset) - - -@torch.no_grad() -def test(model, device, loader, evaluator): - model.eval() - - y_pred, y_true = [], [] - for x, y in tqdm(loader): - x = x.to(device) - out = model(x) - - y_pred.append(torch.argmax(out, dim=1, keepdim=True).cpu()) - y_true.append(y) - - return evaluator.eval({ - "y_true": torch.cat(y_true, dim=0), - "y_pred": torch.cat(y_pred, dim=0), - })['acc'] - - -def main(): - parser = argparse.ArgumentParser(description='OGBN-papers100M (MLP)') - parser.add_argument('--device', type=int, default=0) - parser.add_argument('--log_steps', type=int, default=1) - parser.add_argument('--sgc_dict_pt', type=str, required=True) - parser.add_argument('--num_layers', type=int, default=3) - parser.add_argument('--hidden_channels', type=int, default=256) - parser.add_argument('--dropout', type=float, default=0) - parser.add_argument('--lr', type=float, default=0.01) - parser.add_argument('--batch_size', type=int, default=256) - parser.add_argument('--epochs', type=int, default=30) - parser.add_argument('--runs', type=int, default=10) - args = parser.parse_args() - print(args) - - device = f'cuda:{args.device}' if torch.cuda.is_available() else 'cpu' - device = torch.device(device) - - sgc_dict = torch.load(args.sgc_dict_pt) - x = sgc_dict['sgc_embedding'] - split_idx = sgc_dict['split_idx'] - y = sgc_dict['label'].to(torch.long) - - train_dataset = SimpleDataset(x[split_idx['train']], y[split_idx['train']]) - valid_dataset = SimpleDataset(x[split_idx['valid']], y[split_idx['valid']]) - test_dataset = SimpleDataset(x[split_idx['test']], y[split_idx['test']]) - - train_loader = DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True) - valid_loader = DataLoader(valid_dataset, batch_size=128, shuffle=False) - test_loader = DataLoader(test_dataset, batch_size=128, shuffle=False) - - model = MLP(x.size(-1), args.hidden_channels, 172, args.num_layers, - args.dropout).to(device) - - evaluator = Evaluator(name='ogbn-papers100M') - logger = Logger(args.runs, args) - - for run in range(args.runs): - model.reset_parameters() - optimizer = torch.optim.Adam(model.parameters(), lr=args.lr) - for epoch in range(1, 1 + args.epochs): - train(model, device, train_loader, optimizer) - train_acc = test(model, device, train_loader, evaluator) - valid_acc = test(model, device, valid_loader, evaluator) - test_acc = test(model, device, test_loader, evaluator) - - logger.add_result(run, (train_acc, valid_acc, test_acc)) - - if epoch % args.log_steps == 0: - print(f'Run: {run + 1:02d}, ' - f'Epoch: {epoch:02d}, ' - f'Train: {100 * train_acc:.2f}%, ' - f'Valid: {100 * valid_acc:.2f}%, ' - f'Test: {100 * test_acc:.2f}%') - - logger.print_statistics(run) - logger.print_statistics() - - -if __name__ == "__main__": - main() diff --git a/examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_xrt.py b/examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_xrt.py deleted file mode 100644 index 441c7601..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_xrt.py +++ /dev/null @@ -1,160 +0,0 @@ -import argparse -from tqdm import tqdm - -import torch -import torch.nn.functional as F -from torch.utils.data import Dataset, DataLoader - -from ogb.nodeproppred import PygNodePropPredDataset, Evaluator - -from logger import Logger - -import numpy as np -from pecos.utils import smat_util - - -class SimpleDataset(Dataset): - def __init__(self, x, y): - self.x = x - self.y = y - assert self.x.size(0) == self.y.size(0) - - def __len__(self): - return self.x.size(0) - - def __getitem__(self, idx): - return self.x[idx], self.y[idx] - - -class MLP(torch.nn.Module): - def __init__(self, in_channels, hidden_channels, out_channels, num_layers, - dropout): - super(MLP, self).__init__() - - self.lins = torch.nn.ModuleList() - self.lins.append(torch.nn.Linear(in_channels, hidden_channels)) - self.bns = torch.nn.ModuleList() - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - for _ in range(num_layers - 2): - self.lins.append(torch.nn.Linear(hidden_channels, hidden_channels)) - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - self.lins.append(torch.nn.Linear(hidden_channels, out_channels)) - - self.dropout = dropout - - def reset_parameters(self): - for lin in self.lins: - lin.reset_parameters() - for bn in self.bns: - bn.reset_parameters() - - def forward(self, x): - for i, lin in enumerate(self.lins[:-1]): - x = lin(x) - x = self.bns[i](x) - x = F.relu(x) - x = F.dropout(x, p=self.dropout, training=self.training) - x = self.lins[-1](x) - return torch.log_softmax(x, dim=-1) - - -def train(model, device, train_loader, optimizer): - model.train() - - total_loss = 0 - for x, y in tqdm(train_loader): - x, y = x.to(device), y.to(device) - optimizer.zero_grad() - out = model(x) - loss = F.nll_loss(out, y.squeeze(1)) - loss.backward() - optimizer.step() - total_loss += loss.item() * x.size(0) - return total_loss / len(train_loader.dataset) - - -@torch.no_grad() -def test(model, device, loader, evaluator): - model.eval() - - y_pred, y_true = [], [] - for x, y in tqdm(loader): - x = x.to(device) - out = model(x) - - y_pred.append(torch.argmax(out, dim=1, keepdim=True).cpu()) - y_true.append(y) - - return evaluator.eval({ - "y_true": torch.cat(y_true, dim=0), - "y_pred": torch.cat(y_pred, dim=0), - })['acc'] - - -def main(): - parser = argparse.ArgumentParser(description='OGBN-papers100M (MLP)') - parser.add_argument('--device', type=int, default=0) - parser.add_argument('--log_steps', type=int, default=1) - parser.add_argument('--num_layers', type=int, default=3) - parser.add_argument('--hidden_channels', type=int, default=256) - parser.add_argument('--dropout', type=float, default=0) - parser.add_argument('--lr', type=float, default=0.01) - parser.add_argument('--batch_size', type=int, default=256) - parser.add_argument('--epochs', type=int, default=30) - parser.add_argument('--runs', type=int, default=5) - parser.add_argument('--data_root_dir', type=str, default='../../dataset') - parser.add_argument('--node_emb_path', type=str, default=None) - args = parser.parse_args() - print(args) - - device = f'cuda:{args.device}' if torch.cuda.is_available() else 'cpu' - device = torch.device(device) - - dataset = PygNodePropPredDataset(name='ogbn-papers100M',root=args.data_root_dir) - split_idx = dataset.get_idx_split() - data = dataset[0] - - if args.node_emb_path: - data.x = torch.from_numpy(smat_util.load_matrix(args.node_emb_path).astype(np.float32)) - print("Loaded pre-trained node embeddings of shape={} from {}".format(data.x.shape, args.node_emb_path)) - - x = data.x - y = data.y.to(torch.long) - train_dataset = SimpleDataset(x[split_idx['train']], y[split_idx['train']]) - valid_dataset = SimpleDataset(x[split_idx['valid']], y[split_idx['valid']]) - test_dataset = SimpleDataset(x[split_idx['test']], y[split_idx['test']]) - - train_loader = DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True) - valid_loader = DataLoader(valid_dataset, batch_size=args.batch_size * 4, shuffle=False) - test_loader = DataLoader(test_dataset, batch_size=args.batch_size * 4, shuffle=False) - - model = MLP(x.size(-1), args.hidden_channels, dataset.num_classes, - args.num_layers, args.dropout).to(device) - - evaluator = Evaluator(name='ogbn-papers100M') - logger = Logger(args.runs, args) - - for run in range(args.runs): - model.reset_parameters() - optimizer = torch.optim.Adam(model.parameters(), lr=args.lr) - for epoch in range(1, 1 + args.epochs): - train(model, device, train_loader, optimizer) - train_acc = test(model, device, train_loader, evaluator) - valid_acc = test(model, device, valid_loader, evaluator) - test_acc = test(model, device, test_loader, evaluator) - - logger.add_result(run, (train_acc, valid_acc, test_acc)) - - if epoch % args.log_steps == 0: - print(f'Run: {run + 1:02d}, ' - f'Epoch: {epoch:02d}, ' - f'Train: {100 * train_acc:.2f}%, ' - f'Valid: {100 * valid_acc:.2f}%, ' - f'Test: {100 * test_acc:.2f}%') - - logger.print_statistics(run) - logger.print_statistics() - - -if __name__ == "__main__": - main() diff --git a/examples/giant-xrt/OGB_baselines/ogbn-papers100M/sgc.py b/examples/giant-xrt/OGB_baselines/ogbn-papers100M/sgc.py deleted file mode 100644 index a65a9807..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-papers100M/sgc.py +++ /dev/null @@ -1,74 +0,0 @@ -import argparse -from tqdm import tqdm - -import torch -import torch.nn.functional as F -from torch_sparse import SparseTensor -from torch_geometric.utils import to_undirected, dropout_adj - -from ogb.nodeproppred import PygNodePropPredDataset -import numpy as np - - -def main(): - parser = argparse.ArgumentParser(description='OGBN-papers100M (MLP)') - parser.add_argument('--data_root_dir', type=str, default='../../dataset') - parser.add_argument('--num_propagations', type=int, default=3) - parser.add_argument('--dropedge_rate', type=float, default=0.4) - parser.add_argument('--node_emb_path', type=str, default=None) - parser.add_argument('--output_path', type=str, required=True) - args = parser.parse_args() - - # SGC pre-processing ###################################################### - - dataset = PygNodePropPredDataset(name='ogbn-papers100M', root=args.data_root_dir) - split_idx = dataset.get_idx_split() - data = dataset[0] - - x = None - if args.node_emb_path: - x = np.load(args.node_emb_path) - else: - x = data.x.numpy() - N = data.num_nodes - - print('Making the graph undirected.') - ### Randomly drop some edges to save computation - data.edge_index, _ = dropout_adj(data.edge_index, p = args.dropedge_rate, num_nodes= data.num_nodes) - data.edge_index = to_undirected(data.edge_index, data.num_nodes) - - print(data) - - row, col = data.edge_index - - print('Computing adj...') - - adj = SparseTensor(row=row, col=col, sparse_sizes=(N, N)) - adj = adj.set_diag() - deg = adj.sum(dim=1).to(torch.float) - deg_inv_sqrt = deg.pow(-0.5) - deg_inv_sqrt[deg_inv_sqrt == float('inf')] = 0 - adj = deg_inv_sqrt.view(-1, 1) * adj * deg_inv_sqrt.view(1, -1) - - adj = adj.to_scipy(layout='csr') - - - train_idx, valid_idx, test_idx = split_idx['train'], split_idx['valid'], split_idx['test'] - all_idx = torch.cat([train_idx, valid_idx, test_idx]) - mapped_train_idx = torch.arange(len(train_idx)) - mapped_valid_idx = torch.arange(len(train_idx), len(train_idx) + len(valid_idx)) - mapped_test_idx = torch.arange(len(train_idx) + len(valid_idx), len(train_idx) + len(valid_idx) + len(test_idx)) - - sgc_dict = {} - sgc_dict['label'] = data.y.data[all_idx].to(torch.long) - sgc_dict['split_idx'] = {'train': mapped_train_idx, 'valid': mapped_valid_idx, 'test': mapped_test_idx} - - print('Start SGC processing') - for _ in tqdm(range(args.num_propagations)): - x = adj @ x - sgc_dict['sgc_embedding'] = torch.from_numpy(x[all_idx]).to(torch.float) - torch.save(sgc_dict, args.output_path) - - -if __name__ == "__main__": - main() diff --git a/examples/giant-xrt/OGB_baselines/ogbn-products/graph_saint.py b/examples/giant-xrt/OGB_baselines/ogbn-products/graph_saint.py deleted file mode 100644 index c755de4a..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-products/graph_saint.py +++ /dev/null @@ -1,208 +0,0 @@ -import argparse - -import torch -from tqdm import tqdm -import torch.nn.functional as F - -from torch_geometric.data import GraphSAINTRandomWalkSampler, NeighborSampler -from torch_geometric.nn import SAGEConv -from torch_geometric.utils import subgraph - -from ogb.nodeproppred import PygNodePropPredDataset, Evaluator - -from logger import Logger -import numpy as np -from pecos.utils import smat_util - - -class SAGE(torch.nn.Module): - def __init__(self, in_channels, hidden_channels, out_channels, num_layers, - dropout): - super(SAGE, self).__init__() - - self.convs = torch.nn.ModuleList() - self.convs.append(SAGEConv(in_channels, hidden_channels)) - for _ in range(num_layers - 2): - self.convs.append(SAGEConv(hidden_channels, hidden_channels)) - self.convs.append(SAGEConv(hidden_channels, out_channels)) - - self.dropout = dropout - - def reset_parameters(self): - for conv in self.convs: - conv.reset_parameters() - - def forward(self, x, edge_index, edge_weight=None): - for conv in self.convs[:-1]: - x = conv(x, edge_index, edge_weight) - x = F.relu(x) - x = F.dropout(x, p=self.dropout, training=self.training) - x = self.convs[-1](x, edge_index, edge_weight) - return torch.log_softmax(x, dim=-1) - - def inference(self, x_all, subgraph_loader, device): - pbar = tqdm(total=x_all.size(0) * len(self.convs)) - pbar.set_description('Evaluating') - - for i, conv in enumerate(self.convs): - xs = [] - for batch_size, n_id, adj in subgraph_loader: - edge_index, _, size = adj.to(device) - x = x_all[n_id].to(device) - x_target = x[:size[1]] - x = conv((x, x_target), edge_index) - if i != len(self.convs) - 1: - x = F.relu(x) - xs.append(x.cpu()) - - pbar.update(batch_size) - - x_all = torch.cat(xs, dim=0) - - pbar.close() - - return x_all - - -def train(model, loader, optimizer, device): - model.train() - - total_loss = 0 - for data in loader: - data = data.to(device) - optimizer.zero_grad() - out = model(data.x, data.edge_index) - y = data.y.squeeze(1) - loss = F.nll_loss(out[data.train_mask], y[data.train_mask]) - loss.backward() - optimizer.step() - total_loss += loss.item() - - return total_loss / len(loader) - - -@torch.no_grad() -def test(model, data, evaluator, subgraph_loader, device): - model.eval() - - out = model.inference(data.x, subgraph_loader, device) - - y_true = data.y - y_pred = out.argmax(dim=-1, keepdim=True) - - train_acc = evaluator.eval({ - 'y_true': y_true[data.train_mask], - 'y_pred': y_pred[data.train_mask] - })['acc'] - valid_acc = evaluator.eval({ - 'y_true': y_true[data.valid_mask], - 'y_pred': y_pred[data.valid_mask] - })['acc'] - test_acc = evaluator.eval({ - 'y_true': y_true[data.test_mask], - 'y_pred': y_pred[data.test_mask] - })['acc'] - - return train_acc, valid_acc, test_acc - - -def to_inductive(data): - mask = data.train_mask - data.x = data.x[mask] - data.y = data.y[mask] - data.train_mask = data.train_mask[mask] - data.test_mask = None - data.edge_index, _ = subgraph(mask, data.edge_index, None, - relabel_nodes=True, num_nodes=data.num_nodes) - data.num_nodes = mask.sum().item() - return data - - -def main(): - parser = argparse.ArgumentParser(description='OGBN-Products (GraphSAINT)') - parser.add_argument('--device', type=int, default=0) - parser.add_argument('--log_steps', type=int, default=1) - parser.add_argument('--inductive', action='store_true') - parser.add_argument('--num_layers', type=int, default=3) - parser.add_argument('--hidden_channels', type=int, default=256) - parser.add_argument('--dropout', type=float, default=0.5) - parser.add_argument('--batch_size', type=int, default=20000) - parser.add_argument('--walk_length', type=int, default=3) - parser.add_argument('--lr', type=float, default=0.001) - parser.add_argument('--num_steps', type=int, default=30) - parser.add_argument('--epochs', type=int, default=50) - parser.add_argument('--eval_steps', type=int, default=2) - parser.add_argument('--runs', type=int, default=10) - parser.add_argument('--data_root_dir', type=str, default='../../dataset') - parser.add_argument('--node_emb_path', type=str, default=None) - args = parser.parse_args() - print(args) - - device = f'cuda:{args.device}' if torch.cuda.is_available() else 'cpu' - device = torch.device(device) - - dataset = PygNodePropPredDataset(name='ogbn-products', root=args.data_root_dir) - split_idx = dataset.get_idx_split() - data = dataset[0] - - # Load Pretrained node features from PECOS - if args.node_emb_path: - data.x = torch.from_numpy(smat_util.load_matrix(args.node_emb_path).astype(np.float32)) - print("Loaded pre-trained node embeddings of shape={} from {}".format(data.x.shape, args.node_emb_path)) - - # Convert split indices to boolean masks and add them to `data`. - for key, idx in split_idx.items(): - mask = torch.zeros(data.num_nodes, dtype=torch.bool) - mask[idx] = True - data[f'{key}_mask'] = mask - - # We omit normalization factors here since those are only defined for the - # inductive learning setup. - sampler_data = data - if args.inductive: - sampler_data = to_inductive(data) - - loader = GraphSAINTRandomWalkSampler(sampler_data, - batch_size=args.batch_size, - walk_length=args.walk_length, - num_steps=args.num_steps, - sample_coverage=0, - save_dir=dataset.processed_dir) - - model = SAGE(data.x.size(-1), args.hidden_channels, dataset.num_classes, - args.num_layers, args.dropout).to(device) - - subgraph_loader = NeighborSampler(data.edge_index, sizes=[-1], - batch_size=4096, shuffle=False, - num_workers=12) - - evaluator = Evaluator(name='ogbn-products') - logger = Logger(args.runs, args) - - for run in range(args.runs): - model.reset_parameters() - optimizer = torch.optim.Adam(model.parameters(), lr=args.lr) - for epoch in range(1, 1 + args.epochs): - loss = train(model, loader, optimizer, device) - if epoch % args.log_steps == 0: - print(f'Run: {run + 1:02d}, ' - f'Epoch: {epoch:02d}, ' - f'Loss: {loss:.4f}') - - if epoch > 9 and epoch % args.eval_steps == 0: - result = test(model, data, evaluator, subgraph_loader, device) - logger.add_result(run, result) - train_acc, valid_acc, test_acc = result - print(f'Run: {run + 1:02d}, ' - f'Epoch: {epoch:02d}, ' - f'Train: {100 * train_acc:.2f}%, ' - f'Valid: {100 * valid_acc:.2f}% ' - f'Test: {100 * test_acc:.2f}%') - - logger.add_result(run, result) - logger.print_statistics(run) - logger.print_statistics() - - -if __name__ == "__main__": - main() diff --git a/examples/giant-xrt/OGB_baselines/ogbn-products/logger.py b/examples/giant-xrt/OGB_baselines/ogbn-products/logger.py deleted file mode 100644 index b6e617ba..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-products/logger.py +++ /dev/null @@ -1,44 +0,0 @@ -import torch - - -class Logger(object): - def __init__(self, runs, info=None): - self.info = info - self.results = [[] for _ in range(runs)] - - def add_result(self, run, result): - assert len(result) == 3 - assert run >= 0 and run < len(self.results) - self.results[run].append(result) - - def print_statistics(self, run=None): - if run is not None: - result = 100 * torch.tensor(self.results[run]) - argmax = result[:, 1].argmax().item() - print(f'Run {run + 1:02d}:') - print(f'Highest Train: {result[:, 0].max():.2f}') - print(f'Highest Valid: {result[:, 1].max():.2f}') - print(f' Final Train: {result[argmax, 0]:.2f}') - print(f' Final Test: {result[argmax, 2]:.2f}') - else: - result = 100 * torch.tensor(self.results) - - best_results = [] - for r in result: - train1 = r[:, 0].max().item() - valid = r[:, 1].max().item() - train2 = r[r[:, 1].argmax(), 0].item() - test = r[r[:, 1].argmax(), 2].item() - best_results.append((train1, valid, train2, test)) - - best_result = torch.tensor(best_results) - - print(f'All runs:') - r = best_result[:, 0] - print(f'Highest Train: {r.mean():.2f} ± {r.std():.2f}') - r = best_result[:, 1] - print(f'Highest Valid: {r.mean():.2f} ± {r.std():.2f}') - r = best_result[:, 2] - print(f' Final Train: {r.mean():.2f} ± {r.std():.2f}') - r = best_result[:, 3] - print(f' Final Test: {r.mean():.2f} ± {r.std():.2f}') diff --git a/examples/giant-xrt/OGB_baselines/ogbn-products/mlp.py b/examples/giant-xrt/OGB_baselines/ogbn-products/mlp.py deleted file mode 100644 index 3fdc26d4..00000000 --- a/examples/giant-xrt/OGB_baselines/ogbn-products/mlp.py +++ /dev/null @@ -1,144 +0,0 @@ -import argparse - -import torch -import torch.nn.functional as F - -from ogb.nodeproppred import PygNodePropPredDataset, Evaluator - -from logger import Logger - -import numpy as np -from pecos.utils import smat_util - - -class MLP(torch.nn.Module): - def __init__(self, in_channels, hidden_channels, out_channels, num_layers, - dropout,BN=False): - super(MLP, self).__init__() - - self.lins = torch.nn.ModuleList() - self.lins.append(torch.nn.Linear(in_channels, hidden_channels)) - self.bns = torch.nn.ModuleList() - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - for _ in range(num_layers - 2): - self.lins.append(torch.nn.Linear(hidden_channels, hidden_channels)) - self.bns.append(torch.nn.BatchNorm1d(hidden_channels)) - self.lins.append(torch.nn.Linear(hidden_channels, out_channels)) - - self.dropout = dropout - self.BN = BN - def reset_parameters(self): - for lin in self.lins: - lin.reset_parameters() - for bn in self.bns: - bn.reset_parameters() - - def forward(self, x): - for i,lin in enumerate(self.lins[:-1]): - x = lin(x) - x = F.relu(x) - if self.BN: - x = self.bns[i](x) - x = F.dropout(x, p=self.dropout, training=self.training) - x = self.lins[-1](x) - return torch.log_softmax(x, dim=-1) - - -def train(model, x, y_true, train_idx, optimizer): - model.train() - - optimizer.zero_grad() - out = model(x[train_idx]) - loss = F.nll_loss(out, y_true.squeeze(1)[train_idx]) - loss.backward() - optimizer.step() - - return loss.item() - - -@torch.no_grad() -def test(model, x, y_true, split_idx, evaluator): - model.eval() - - out = model(x) - y_pred = out.argmax(dim=-1, keepdim=True) - - train_acc = evaluator.eval({ - 'y_true': y_true[split_idx['train']], - 'y_pred': y_pred[split_idx['train']], - })['acc'] - valid_acc = evaluator.eval({ - 'y_true': y_true[split_idx['valid']], - 'y_pred': y_pred[split_idx['valid']], - })['acc'] - test_acc = evaluator.eval({ - 'y_true': y_true[split_idx['test']], - 'y_pred': y_pred[split_idx['test']], - })['acc'] - - return train_acc, valid_acc, test_acc - - -def main(): - parser = argparse.ArgumentParser(description='OGBN-Products (MLP)') - parser.add_argument('--device', type=int, default=0) - parser.add_argument('--log_steps', type=int, default=1) - parser.add_argument('--use_node_embedding', action='store_true') - parser.add_argument('--num_layers', type=int, default=3) - parser.add_argument('--hidden_channels', type=int, default=256) - parser.add_argument('--dropout', type=float, default=0.0) - parser.add_argument('--lr', type=float, default=0.01) - parser.add_argument('--epochs', type=int, default=300) - parser.add_argument('--runs', type=int, default=10) - parser.add_argument('--bn', action='store_true') - parser.add_argument('--data_root_dir', type=str, default='../../dataset') - parser.add_argument('--node_emb_path', type=str, default=None) - args = parser.parse_args() - print(args) - - device = f'cuda:{args.device}' if torch.cuda.is_available() else 'cpu' - device = torch.device(device) - - dataset = PygNodePropPredDataset(name='ogbn-products', - root=args.data_root_dir) - split_idx = dataset.get_idx_split() - data = dataset[0] - - if args.node_emb_path: - data.x = torch.from_numpy(smat_util.load_matrix(args.node_emb_path).astype(np.float32)) - print("Loaded pre-trained node embeddings of shape={} from {}".format(data.x.shape, args.node_emb_path)) - - x = data.x - x = x.to(device) - - y_true = data.y.to(device) - train_idx = split_idx['train'].to(device) - - model = MLP(x.size(-1), args.hidden_channels, dataset.num_classes, args.num_layers,args.dropout,args.bn).to(device) - - evaluator = Evaluator(name='ogbn-products') - logger = Logger(args.runs, args) - - for run in range(args.runs): - model.reset_parameters() - optimizer = torch.optim.Adam(model.parameters(), lr=args.lr) - for epoch in range(1, 1 + args.epochs): - loss = train(model, x, y_true, train_idx, optimizer) - result = test(model, x, y_true, split_idx, evaluator) - logger.add_result(run, result) - - if epoch % args.log_steps == 0: - train_acc, valid_acc, test_acc = result - print(f'Run: {run + 1:02d}, ' - f'Epoch: {epoch:02d}, ' - f'Loss: {loss:.4f}, ' - f'Train: {100 * train_acc:.2f}%, ' - f'Valid: {100 * valid_acc:.2f}%, ' - f'Test: {100 * test_acc:.2f}%') - - logger.print_statistics(run) - logger.print_statistics() - - -if __name__ == "__main__": - main() diff --git a/examples/giant-xrt/README.md b/examples/giant-xrt/README.md deleted file mode 100644 index a80e721f..00000000 --- a/examples/giant-xrt/README.md +++ /dev/null @@ -1,162 +0,0 @@ -# GIANT-XRT: Node Feature Extraction by Self-supervised Multi-scale Neighborhood Prediction - -## Requirement and Install -First let's setup a conda enviroment -```bash -conda create -n "giant-xrt" python=3.8 -conda activate giant-xrt -``` -Next, we install pytorch and libpecos: -```bash -conda install pytorch==1.9.0 cudatoolkit=10.2 -c pytorch -pip install libpecos==0.2.2 -# check the pytorch version and cuda availability -python -c "import torch; print('torch={}, cuda={}'.format(torch.__version__, torch.cuda.is_available()))" -``` -Finall, we install GNN related packages -```bash -ptcu_version="1.9.0+cu102" -pip install torch-scatter -f "https://pytorch-geometric.com/whl/torch-${ptcu_version}.html" -pip install torch-sparse -f "https://pytorch-geometric.com/whl/torch-${ptcu_version}.html" -pip install torch-cluster -f "https://pytorch-geometric.com/whl/torch-${ptcu_version}.html" -pip install torch-spline-conv -f "https://pytorch-geometric.com/whl/torch-${ptcu_version}.html" -pip install torch-geometric -pip install ogb==1.3.2 -# our ogb version is 1.3.2 -python -c "import ogb; from ogb.graphproppred import PygGraphPropPredDataset; print(ogb.__version__)" -``` - -## Directory Layout -```bash -./giant-xrt -|---- dataset/ # OGB benchmark datasets -|---- OGB_baselines/ # OGB benchmark GNN models (e.g., mlp, graph-sage, graph-saint) -| |---- ogbn-arxiv/ -| | |---- mlp.py -| | |---- gnn.py -| | |---- logger.py -| | -| |---- ogbn-products/ -| |---- mlp.py -| |---- graph_saint.py -| |---- logger.py -| -|---- proc_data_xrt/ -| |---- ogbn-arxiv/ # default is empty, artifacts will be downloaded by download_data.sh -| |---- ogbn-products/ # default is empty, artifacts will be downloaded by download_data.sh -| |---- download_data.sh -| |---- vect_config.json # PECOS TFIDF vectorizer config file -| -|---- proc_data_xrt.py # create giant-xrt pre-training data -|---- proc_data_xrt.sh -|---- xrt_train.sh # pre-training with XR-Transformer in PECOS -|---- xrt_get_emb.sh # get node embeddings with the fine-tuned XR-Transformer -|---- run_ogb_baselines.sh # run GNN baselines on OGB benchmark datasets -``` - - -## Download GIANT-XRT Preporcessed Data -This step is required for all remaining sections! -We support three OGB datasets: `ogbn-arxiv`, `ogbn-products`, and `ogbn-papers100M`. -here, consider `ogbn-arxiv` as an example, which can be downloaded via - -```bash -cd ./proc_data_xrt -dataset=ogbn-arxiv -bash download_data.sh ${dataset} -cd ../ -``` - -After downloading the pre-processed data, you should see files under the `./proc_data_xrt/ogbn-arxiv/` folders -```bash -./gaint-xrt -|---- proc_data_xrt/ - |---- download_data.sh - |---- vect_config.json - |---- ogbn-arxiv/ - |---- params.json # hyper-paramters for GIANT-XRT pre-training - |---- X.all.txt # node raw text - |---- X.all.xrt-emb.npy # node embeddings from XR-Transformer - |---- xrt_models/ # XR-Transformer fine-tined models -``` - - -## Run GNN Baselines on OGB Datasets -For users who only want to take GIANT-XRT node embeddings for running GNN models: -```bash -dataset=ogbn-arxiv # can be either ogbn-arxiv, ogbn-products, ogbn-papers100M -# for ogbn-arxiv: mlp/graph-sage -# for ogbn-products: mlp/graph-saint; -# for ogbn-papers100M: mlp/sgc; -gnn_algo=mlp -bash ./run_ogb_baselines.sh ${dataset} ${gnn_algo} -``` - -### Results -For `ogbn-arxiv` and `ogbn-products`, we report the mean/std of 10 runs. -For `ogbn-papers100M`, we report the mean/std of 5 runs. - -| ogbn-arxiv | MLP | GraphSAGE | -|---|---|---| -| Test accuracy (%) | 73.06 ± 0.11 | 74.35 ± 0.14 | - -| ogbn-products | MLP | GraphSAINT | -|---|---|---| -| Test accuracy (%) | 80.49 ± 0.28 | 84.15 ± 0.22 | - -| ogbn-papers100M | MLP | SGC | -|---|---|---| -| Test accuracy (%) | 61.06 ± 0.13 | 66.19 ± 0.24 | - - -**Remark**: Note that we do not fix random seed as in the original OGB implementation. So the results can be slightly different (usually within 1 std). - - -## Run SOTA GNNs with GIANT-XRT on OGB Datasets - -For **ogbn-arxiv**, please check this [Repo](https://github.com/elichienxD/deep_gcns_torch). -

- -

- -For **ogbn-products**, please check this [Repo](https://github.com/elichienxD/SAGN_with_SLE). -

- -

- -For **ogbn-papers100M**, please check this [Repo](https://github.com/OctoberChang/GAMLP). -

- -

- - -## Pre-training with GIANT-XRT -This subsection is for advanced users who want to run the pre-training procedure. - -### Create Pre-training Data -```bash -dataset=ogbn-arxiv -bash proc_data_xrt.sh ${dataset} -``` - -### Pre-training GIANT-XRT -```bash -data_dir=./proc_data_xrt/ogbn-arxiv -bash xrt_train.sh ${data_dir} -``` - -### Get Node Embeddings by GIANT-XRT -```bash -bash xrt_get_emb.sh ${data_dir} -``` - -## Citation -If you find this useful, please consider citing our paper. -``` -@article{chien2021node, - title={Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction}, - author={Eli Chien and Wei-Cheng Chang and Cho-Jui Hsieh and Hsiang-Fu Yu and Jiong Zhang and Olgica Milenkovic and Inderjit S Dhillon}, - journal={arXiv preprint arXiv:2111.00064}, - year={2021} -} -``` diff --git a/examples/giant-xrt/bar-plot_ogbn-arxiv.png b/examples/giant-xrt/bar-plot_ogbn-arxiv.png deleted file mode 100644 index dc381c43..00000000 Binary files a/examples/giant-xrt/bar-plot_ogbn-arxiv.png and /dev/null differ diff --git a/examples/giant-xrt/bar-plot_ogbn-papers100M.png b/examples/giant-xrt/bar-plot_ogbn-papers100M.png deleted file mode 100644 index 7d6fd095..00000000 Binary files a/examples/giant-xrt/bar-plot_ogbn-papers100M.png and /dev/null differ diff --git a/examples/giant-xrt/bar-plot_ogbn-products.png b/examples/giant-xrt/bar-plot_ogbn-products.png deleted file mode 100644 index 50ba172a..00000000 Binary files a/examples/giant-xrt/bar-plot_ogbn-products.png and /dev/null differ diff --git a/examples/giant-xrt/proc_data_xrt.py b/examples/giant-xrt/proc_data_xrt.py deleted file mode 100644 index 22041121..00000000 --- a/examples/giant-xrt/proc_data_xrt.py +++ /dev/null @@ -1,75 +0,0 @@ - -import argparse -import os -import numpy as np -import scipy.sparse as smat -from pecos.utils import smat_util -from pecos.utils.featurization.text.vectorizers import Vectorizer -from pecos.utils.featurization.text.preprocess import Preprocessor - -import torch -from ogb.nodeproppred import PygNodePropPredDataset -from torch_geometric.data import DataLoader -from torch_geometric.utils import degree, is_undirected, to_undirected -from torch_geometric.utils.convert import to_scipy_sparse_matrix - - -def main(): - parser = argparse.ArgumentParser(description='Prepare data for Giant-XRT') - parser.add_argument('--raw-text-path', type=str, required=True, help="Path of raw text (.txt file, each raw correspond to a node)") - parser.add_argument('--vectorizer-config-path', type=str, required=True, help="a path to a json file that specify the tfidf hyper-paramters") - parser.add_argument('--data-root-dir', type=str, default="./dataset") - parser.add_argument('--xrt-data-dir', type=str, default="./proc_data_xrt") - parser.add_argument('--dataset', type=str, default="ogbn-arxiv") - parser.add_argument('--max-deg', type=int, default=1000) - args = parser.parse_args() - print(args) - - # Change args.save_data_dir to args.save_data_dir/args.dataset - save_data_dir = os.path.join(args.xrt_data_dir, args.dataset) - dataset = PygNodePropPredDataset(name=args.dataset, root=args.data_root_dir) - data = dataset[0] - edge_index = data.edge_index - - # Make sure edge_index is undirected!!! - if not is_undirected(edge_index): - edge_index = to_undirected(edge_index) - # Filtering nodes whose number of edges >= max_degree - Degree = degree(edge_index[0]) - Filtered_idx = torch.where(Degree < args.max_deg)[0] - print('Number of original nodes:{}'.format(data.x.shape[0])) - print('Number of filtered nodes:{}'.format(len(Filtered_idx))) - - # # Construct and save label matrix (adjacencey matrix) Y. - Y_csr_all = smat.csr_matrix(to_scipy_sparse_matrix(edge_index)) - Y_csr_trn = Y_csr_all[Filtered_idx] - smat_util.save_matrix(f"{save_data_dir}/Y.trn.npz", Y_csr_trn) - smat_util.save_matrix(f"{save_data_dir}/Y.all.npz", Y_csr_all) - print("Saved Y.trn.npz and Y.all.npz") - - # Apply the same filtering for raw text - with open(args.raw_text_path, "r") as fin: - node_text_list = fin.readlines() - print("|node_text_list={}".format(len(node_text_list))) - count = 0 - with open(f"{save_data_dir}/X.trn.txt", "w") as fout: - for cur_idx, line in enumerate(node_text_list): - if Filtered_idx[count].item() == cur_idx: - fout.writelines(line) - count += 1 - assert count == len(Filtered_idx), "count={}, len(Filtered_idx)={}".format(count, len(Filtered_idx)) - print("Saved X.trn.txt") - - # Apply the same filtering for tfidf features - vectorizer_config = Vectorizer.load_config_from_args(args) # using args.vectorizer_config_path - preprocessor = Preprocessor.train(node_text_list, vectorizer_config, dtype=np.float32) - preprocessor.save(f"{save_data_dir}/tfidf-model") - X_tfidf_all = preprocessor.predict(node_text_list) - X_tfidf_trn = X_tfidf_all[Filtered_idx] - smat_util.save_matrix(f"{save_data_dir}/X.all.tfidf.npz", X_tfidf_all) - smat_util.save_matrix(f"{save_data_dir}/X.trn.tfidf.npz", X_tfidf_trn) - print("Saved X.trn.npz and X.all.npz") - - -if __name__ == "__main__": - main() diff --git a/examples/giant-xrt/proc_data_xrt.sh b/examples/giant-xrt/proc_data_xrt.sh deleted file mode 100644 index 7726513d..00000000 --- a/examples/giant-xrt/proc_data_xrt.sh +++ /dev/null @@ -1,19 +0,0 @@ - -dataset=$1 -if [ ${dataset} != "ogbn-arxiv" ] && [ ${dataset} != "ogbn-products" ]; then - echo "dataset=${dataset} is not yet supported!" - exit -fi - -data_root_dir=./dataset -xrt_data_dir=./proc_data_xrt -max_degree=1000 - -python -u proc_data_xrt.py \ - --raw-text-path ${xrt_data_dir}/${dataset}/X.all.txt \ - --vectorizer-config-path ${xrt_data_dir}/vect_config.json \ - --data-root-dir ${data_root_dir} \ - --xrt-data-dir ${xrt_data_dir} \ - --dataset ${dataset} \ - --max-deg ${max_degree} - diff --git a/examples/giant-xrt/proc_data_xrt/download_data.sh b/examples/giant-xrt/proc_data_xrt/download_data.sh deleted file mode 100644 index a627217e..00000000 --- a/examples/giant-xrt/proc_data_xrt/download_data.sh +++ /dev/null @@ -1,9 +0,0 @@ - -dataset=$1 -if [ ${dataset} != "ogbn-arxiv" ] && [ ${dataset} != "ogbn-products" ] && [ ${dataset} != "ogbn-papers100M" ]; then - echo "dataset=${dataset} is not yet supported!" - exit -fi - -wget https://archive.org/download/pecos-dataset/giant-xrt/${dataset}.tar.gz -tar -zxvf ${dataset}.tar.gz diff --git a/examples/giant-xrt/proc_data_xrt/vect_config.json b/examples/giant-xrt/proc_data_xrt/vect_config.json deleted file mode 100644 index fb8ff15f..00000000 --- a/examples/giant-xrt/proc_data_xrt/vect_config.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "type": "tfidf", - "kwargs": { - "base_vect_configs": [ - { - "ngram_range": [1, 1], - "max_feature": 1000000, - "max_df_ratio": 0.98, - "analyzer": "word" - }, - { - "ngram_range": [2, 2], - "max_feature": 3000000, - "max_df_ratio": 0.98, - "analyzer": "word" - }, - { - "ngram_range": [3, 3], - "max_feature": 200000, - "max_df_ratio": 0.98, - "analyzer": "char_wb" - } - ] - } -} diff --git a/examples/giant-xrt/run_ogb_baselines.sh b/examples/giant-xrt/run_ogb_baselines.sh deleted file mode 100644 index bd3aaeea..00000000 --- a/examples/giant-xrt/run_ogb_baselines.sh +++ /dev/null @@ -1,73 +0,0 @@ - -dataset=$1 -gnn_algo=$2 - -if [ ${dataset} == "ogbn-arxiv" ]; then - RUNS=10 - if [ ${gnn_algo} == "mlp" ]; then - python -u OGB_baselines/${dataset}/mlp.py \ - --runs ${RUNS} \ - --data_root_dir ./dataset \ - --node_emb_path ./proc_data_xrt/${dataset}/X.all.xrt-emb.npy \ - |& tee OGB_baselines/${dataset}/mlp.giant-xrt.log - elif [ ${gnn_algo} == "graph-sage" ]; then - python -u OGB_baselines/${dataset}/gnn.py \ - --runs ${RUNS} \ - --data_root_dir ./dataset \ - --use_sage \ - --lr 8e-4 \ - --node_emb_path ./proc_data_xrt/${dataset}/X.all.xrt-emb.npy \ - |& tee OGB_baselines/${dataset}/graph-sage.giant-xrt.log - else - echo "gnn_algo=${gnn_algo} is not yet supported for ogbn-arxiv!" - fi -elif [ ${dataset} == "ogbn-products" ]; then - RUNS=10 - if [ ${gnn_algo} == "mlp" ]; then - python -u OGB_baselines/${dataset}/mlp.py \ - --runs ${RUNS} \ - --data_root_dir ./dataset \ - --node_emb_path ./proc_data_xrt/${dataset}/X.all.xrt-emb.npy \ - |& tee OGB_baselines/${dataset}/mlp.giant-xrt.log - elif [ ${gnn_algo} == "graph-saint" ]; then - CUDA_VISIBLE_DEVICES=1 python -u OGB_baselines/${dataset}/graph_saint.py \ - --runs ${RUNS} \ - --data_root_dir ./dataset \ - --eval_steps 10 \ - --epochs 50 \ - --num_layers 1 \ - --walk_length 1 \ - --hidden_channels 192 \ - --lr 1e-3 \ - --node_emb_path ./proc_data_xrt/${dataset}/X.all.xrt-emb.v2.npy \ - |& tee OGB_baselines/${dataset}/graph-saint.giant-xrt.v2.h-192_lr-1e-3..log - else - echo "gnn_algo=${gnn_algo} is not supported for ogbn-arxiv!" - fi -elif [ ${dataset} == "ogbn-papers100M" ]; then - RUNS=5 - if [ ${gnn_algo} == "mlp" ]; then - python -u OGB_baselines/${dataset}/mlp_xrt.py \ - --runs ${RUNS} \ - --data_root_dir ./dataset \ - --epochs 50 \ - --lr 1e-3 \ - --node_emb_path ./proc_data_xrt/${dataset}/X.all.xrt-emb.npy \ - |& tee OGB_baselines/${dataset}/mlp.giant-xrt.log - elif [ ${gnn_algo} == "sgc" ]; then - #python -u OGB_baselines/${dataset}/sgc.py \ - # --data_root_dir ./dataset \ - # --node_emb_path ./proc_data_xrt/${dataset}/X.all.xrt-emb.npy \ - # --output_path ./proc_data_xrt/${dataset}/X.all.xrt-emb.sgc.pt - python -u OGB_baselines/${dataset}/mlp_sgc.py \ - --runs ${RUNS} \ - --epochs 50 \ - --lr 1e-3 \ - --sgc_dict_pt ./proc_data_xrt/${dataset}/X.all.xrt-emb.sgc.pt \ - |& tee OGB_baselines/${dataset}/sgc.giant-xrt.log - else - echo "gnn_algo=${gnn_algo} is not supported for ogbn-papers100M" - fi -else - echo "dataset=${dataset} is not yet supported!" -fi diff --git a/examples/giant-xrt/xrt_get_emb.sh b/examples/giant-xrt/xrt_get_emb.sh deleted file mode 100644 index f18a8f74..00000000 --- a/examples/giant-xrt/xrt_get_emb.sh +++ /dev/null @@ -1,21 +0,0 @@ -#================= inputs ===================== -data_dir=$1 # e.g., ./proc_data_xrt/ogbn-arxiv -if [ -z ${data_dir} ] || [ ! -d ${data_dir} ]; then - echo "DATA_DIR does not exist: ${data_dir}" - exit -fi -X_txt_path=${data_dir}/X.all.txt # input node raw text for the entire graph -model_dir=${data_dir}/xrt_models # input pre-trained Giant-XRT models - -#================= output ===================== -X_emb_path=${data_dir}/X.all.xrt-emb.npy # output fine-tuned node embeddings from Giant-XRT - -#==================== train =================== -python -m pecos.xmc.xtransformer.encode \ - -t ${X_txt_path} \ - -m ${model_dir} \ - -o ${X_emb_path} \ - --batch-size 64 \ - --verbose-level 3 \ - |& tee ${model_dir}/predict.log - diff --git a/examples/giant-xrt/xrt_train.sh b/examples/giant-xrt/xrt_train.sh deleted file mode 100644 index 38895245..00000000 --- a/examples/giant-xrt/xrt_train.sh +++ /dev/null @@ -1,29 +0,0 @@ -#================= inputs ===================== -data_dir=$1 # e.g., ./proc_data_xrt/ogbn-arxiv -if [ -z ${data_dir} ] || [ ! -d ${data_dir} ]; then - echo "DATA_DIR does not exist: ${data_dir}" - exit -fi -Y_npz_path=${data_dir}/Y.trn.npz # training label matrix -X_txt_path=${data_dir}/X.trn.txt # training text -X_npz_path=${data_dir}/X.trn.tfidf.npz # training tfidf feature -X_pt_path=${data_dir}/X.trn.pt # save trn tensors here -params_path=${data_dir}/params.json # train/predict hyper-parameters in json file - -#================== outputs =================== -model_dir=${data_dir}/xrt_models -mkdir -p ${model_dir} -TMPDIR=${model_dir}/tmp -mkdir -p ${TMPDIR} -export TMPDIR=${model_dir}/tmp - -#==================== train =================== -python -m pecos.xmc.xtransformer.train \ - -t ${X_txt_path} \ - -x ${X_npz_path} \ - -y ${Y_npz_path} \ - -m ${model_dir} \ - --params-path ${params_path} \ - --verbose-level 3 \ - |& tee ${model_dir}/train.log - diff --git a/examples/msmarco-rankllama/README.md b/examples/msmarco-rankllama/README.md deleted file mode 100644 index 9014366f..00000000 --- a/examples/msmarco-rankllama/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# PECOS XMR Reranker on MS-Marco Dataset - -This is an example of PECOS-based RankingModel that reproduced the [RankLlaMA paper](https://arxiv.org/abs/2310.08319). - -## How to run - -### Training -```bash -torchrun --nnodes 1 --nproc-per-node 8 \ - -m pecos.xmr.reranker.train \ - --config_json_path ./msmarco_qwen2-7B.train.json -``` - -### Predictions -```bash -python -m pecos.xmr.reranker.predict \ - --config_json_path ./msmarco_qwen2-7B.pred.json -``` - -## Evaluation -We first convert the predictions from parquet to TREC format: -```python -python -u parquet_to_trec_eval.py -i inference_outputs/ms_marco/qwen2-7B -o inference_outputs/ms_marco/qwen2-7B.pred.trec -``` - -We then follow [Pyserini]() evaluation protocol to eval the NDCG@10, -and you should see the results like: -```python -python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-passage inference_outputs/ms_marco/qwen2-7B.pred.trec - -Results: -ndcg_cut_10 all 0.7619 -``` diff --git a/examples/msmarco-rankllama/msmarco_qwen2-7B.pred.json b/examples/msmarco-rankllama/msmarco_qwen2-7B.pred.json deleted file mode 100644 index b7b8baed..00000000 --- a/examples/msmarco-rankllama/msmarco_qwen2-7B.pred.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "target_data_folder": "./datasets/ms_marco/eval_aux/target", - "input_data_folder": "./datasets/ms_marco/eval_aux/input", - "label_data_folder": "./datasets/ms_marco/eval_aux/label", - "model_path": "./models/ms_marco/qwen2-7B/", - "output_dir": "./inference_outputs/ms_marco/qwen2-7B/", - "per_device_eval_batch_size": 1024, - "dataloader_num_workers": 1, - "dataloader_prefetch_factor": 10, - "rerank_max_len": 196, - "query_prefix": "query: ", - "passage_prefix": "document: ", - "inp_id_col": "inp_id", - "lbl_id_col": "lbl_id", - "inp_id_orig_col": "inp_id_orig", - "lbl_id_orig_col": "lbl_id_orig", - "keyword_col_name": "keywords", - "content_col_names": ["title", "contents"], - "append_eos_token": false, - "pad_to_multiple_of": 8 -} diff --git a/examples/msmarco-rankllama/msmarco_qwen2-7B.train.json b/examples/msmarco-rankllama/msmarco_qwen2-7B.train.json deleted file mode 100644 index 9f2063cf..00000000 --- a/examples/msmarco-rankllama/msmarco_qwen2-7B.train.json +++ /dev/null @@ -1,140 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmr.reranker.model###RankingModel.TrainParams" - }, - "target_data_folder": "./datasets/ms_marco/train/target", - "input_data_folder": "./datasets/ms_marco/train/input", - "label_data_folder": "./datasets/ms_marco/train/label", - "hf_trainer_args": { - "__meta__": { - "class_fullname": "pecos.xmr.reranker.trainer###RankingTrainer.TrainingArgs" - }, - "output_dir": "./models/ms_marco/qwen2-7B", - "ddp_find_unused_parameters": false, - "loss_fn": "listwise", - "loss_alpha": 1.0, - "group_size": 16, - "per_device_train_batch_size": 6, - "gradient_accumulation_steps": 8, - "disable_tqdm": false, - "logging_strategy": "steps", - "logging_first_step": false, - "learning_rate": 1e-4, - "max_steps": 1500, - "save_steps": 50, - "logging_steps": 10, - "save_strategy": "steps", - "save_total_limit": 5, - "seed": 42, - "data_seed": 42, - "bf16": true, - "dataloader_num_workers": 2, - "dataloader_prefetch_factor": 10, - "gradient_checkpointing": true, - "deepseed": { - "zero_optimization": { - "stage": 3, - "offload_optimizer": { - "device": "none", - "pin_memory": true - }, - "offload_param": { - "device": "none", - "pin_memory": true - }, - "overlap_comm": true, - "contiguous_gradients": true, - "sub_group_size": 1e9, - "reduce_bucket_size": 1e6, - "stage3_prefetch_bucket_size": "auto", - "stage3_param_persistence_threshold": "auto", - "stage3_max_live_parameters": 1e9, - "stage3_max_reuse_distance": 1e9, - "stage3_gather_16bit_weights_on_model_save": true - }, - "fp16": { - "enabled": "auto", - "loss_scale": 0, - "initial_scale_power": 10, - "loss_scale_window": 1000, - "hysteresis": 2, - "min_loss_scale": 1 - }, - "bf16": { - "enabled": "auto", - "loss_scale": 0, - "initial_scale_power": 10, - "loss_scale_window": 1000, - "hysteresis": 2, - "min_loss_scale": 1 - }, - "optimizer": { - "type": "AdamW", - "params": { - "lr": "auto", - "betas": "auto", - "eps": "auto", - "weight_decay": "auto", - "torch_adam": true - } - }, - "scheduler": { - "type": "WarmupDecayLR", - "params": { - "warmup_min_lr": "auto", - "warmup_max_lr": "auto", - "warmup_num_steps": "auto", - "total_num_steps": "auto" - } - }, - "gradient_accumulation_steps": "auto", - "gradient_clipping": "auto", - "steps_per_print": 1000, - "train_batch_size": "auto", - "train_micro_batch_size_per_gpu": "auto", - "wall_clock_breakdown": false - } - } - }, - "model_params": { - "__meta__": { - "class_fullname": "pecos.xmr.reranker.model###RankingModel.ModelParams" - }, - "encoder_config": { - "text_config": { - "model_type": "qwen2", - "name_or_path": "Qwen/Qwen2-7B", - "attn_implementation": "sdpa", - "trust_remote_code": true, - "token": null - }, - "numr_config": null, - "text_pooling_type": "last", - "head_size_list": [128] - }, - "model_modifier": { - "modifier_type": "peft", - "config_type": "LoraConfig" , - "config": { - "r": 16, - "lora_alpha": 32, - "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], - "modules_to_save": ["head_layers", "scorer"], - "lora_dropout": 0.1 - } - }, - "positive_passage_no_shuffle": false, - "negative_passage_no_shuffle": false, - "rerank_max_len": 196, - "query_prefix": "query: ", - "passage_prefix": "document: ", - "inp_id_col": "inp_id", - "lbl_idxs_col": "ret_idxs", - "score_col": "rel", - "keyword_col_name": "keywords", - "content_col_names": ["title", "contents"], - "append_eos_token": false, - "pad_to_multiple_of": 16 - } -} diff --git a/examples/msmarco-rankllama/parquet_to_trec_eval.py b/examples/msmarco-rankllama/parquet_to_trec_eval.py deleted file mode 100644 index 82c9a884..00000000 --- a/examples/msmarco-rankllama/parquet_to_trec_eval.py +++ /dev/null @@ -1,36 +0,0 @@ - -import argparse -import os -import pandas as pd - - -def main(args): - """ - Combine all results from the results folder and write them to the output file. - """ - result_files = [ - os.path.join(args.input_parquet_path, x) - for x in os.listdir(args.input_parquet_path) - ] - all_results = pd.read_parquet(result_files[0]) - for f in result_files[1:]: - all_results = pd.concat([all_results, pd.read_parquet(f)]) - # sort all results by 'inp_id' and then 'score' in descending order - all_results = all_results.sort_values(by=['inp_id', 'score'], ascending=[True, False]) - - cur_inp_id = None - with open(args.output_trec_path, "w") as fout: - for row in all_results.itertuples(): - if cur_inp_id != row.inp_id: - cur_inp_id = row.inp_id - rank = 0 - rank += 1 - fout.write(f"{row.inp_id} Q0 {row.lbl_id} {rank} {row.score} dense\n") - - -if __name__ == "__main__": - parser = argparse.ArgumentParser() - parser.add_argument("-i", "--input-parquet-path", type=str, required=True) - parser.add_argument("-o", "--output-trec-path", type=str, required=True) - args = parser.parse_args() - main(args) diff --git a/examples/overlap-xmc/README.md b/examples/overlap-xmc/README.md deleted file mode 100644 index 27c499e1..00000000 --- a/examples/overlap-xmc/README.md +++ /dev/null @@ -1,71 +0,0 @@ -# Label Disentanglement in Partition-based Extreme Multilabel Classification, NeurIPS 2021 - -This folder contains code to reproduce the key experiments in "[Label Disentanglement in Partition-based Extreme Multilabel Classification](https://arxiv.org/pdf/2106.12751.pdf)" - -## Get Started - -+ Clone the repository and enter `examples/overlap-xmc` directory. -+ First create a [virtual environment](https://docs.python.org/3/library/venv.html) and then install dependencies by running the following command: - -``` -pip install numba==0.52.0 -pip install scipy==1.4.1 -``` - -If you're unfamiliar with Python virtual environments, check out the [user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). - -After installing, create following folders: - -``` -mkdir dataset/ dataset-binned/ model/ -``` - -## Downloading Data - -The XMC datasets can be download at - -``` -# eurlex-4k, wiki10-31k, amazoncat-13k, amazon-670k, wiki-500k, amazon-3m -DATASET="wiki10-31k" -wget https://archive.org/download/pecos-dataset/xmc-base/${DATASET}.tar.gz -tar -zxvf ./${DATASET}.tar.gz -``` - -Then, move the data folders to `./dataset` folder: - -``` -cp -rf ./xmc-base ./dataset && rm -rf ./xmc-base -``` - -## Training and Evaluation - -The training and evaluation of our label disentablement model, run - -``` -# reproduce ours + XR-Linear: -bash run_base.sh ${DATASET} -# reproduce Figure2: -base run_binned.sh ${DATASET} -# reproduce VI (should be launched after completion of `run_binned.sh`): -base run_metric.sh -``` - -Recommended platform for training: [AWS p3.16xlarge](https://aws.amazon.com/cn/ec2/instance-types/p3/) instance or equivalent. - - -## Known Issues - -+ Be careful about the scipy and numba versions (needs to be 1.4.1 and 0.52.0, respectively). -+ For reproducing `Our method` + `X-Transformer` results, we need to upload the extracted features first. Currently they are missing. - -## Citation - -Please consider to cite this paper if you find our work helpful: -``` -@inproceedings{liu2021label, - title={Label disentanglement in partition-based extreme multilabel classification}, - author={Liu, Xuanqing and Chang, Wei-Cheng and Yu, Hsiang-Fu and Hsieh, Cho-Jui and Dhillon, Inderjit S}, - booktitle={Thirty-Fifth Conference on Neural Information Processing Systems}, - year={2021} -} -``` \ No newline at end of file diff --git a/examples/overlap-xmc/disentangle_metric.py b/examples/overlap-xmc/disentangle_metric.py deleted file mode 100644 index fd9fe0c2..00000000 --- a/examples/overlap-xmc/disentangle_metric.py +++ /dev/null @@ -1,298 +0,0 @@ -import argparse -import os -import pickle as pkl - -import numpy as np -import scipy.sparse as smat -from pecos.core.base import clib -from pecos.utils import smat_util -from pecos.utils.cluster_util import ClusterChain -from pecos.xmc import MLModel -from pecos.xmc.xlinear import XLinearModel - - -def parse_arguments(): - parser = argparse.ArgumentParser( - prog="Evaluate how well our model is good at semantic disentablement." - ) - parser.add_argument( - "-x", - "--inst-path", - type=str, - required=True, - metavar="PATH", - help="path to npz file of feature matrix", - ) - parser.add_argument( - "--y-origin", - type=str, - required=True, - metavar="PATH", - help="path to the npz file of the original label matrix", - ) - parser.add_argument( - "--y-binned", - type=str, - required=True, - metavar="PATH", - help="path to the binned label matrix", - ) - parser.add_argument( - "-m", - "--model-folder", - type=lambda p: os.path.abspath(p), - required=True, - metavar="DIR", - help="path to the model folder", - ) - parser.add_argument( - "--binned-mapper", type=str, required=True, help="path to the mapper file", - ) - parser.add_argument( - "--pseudo-label-mapper", - type=str, - default=None, - help="path to pseudo label mapper. If None, this variable is ignored.", - ) - parser.add_argument( - "--unused-labels", - type=str, - default=None, - help="path to unused label set. If None, this variable is ignored.", - ) - parser.add_argument( - "-b", - "--beam-size", - type=int, - required=True, - help="Beam size to calculate the matching matrix.", - ) - args = parser.parse_args() - return args - - -def get_matching_matrix(xlinear_model, Xt, beam_size=10): - """Compute the matching matrix. - - The matching matrix indicates which cluster(s) are selected for data point in X. The - final results is a sparse matrix of shape N x C, where N is the number of data, and C - is the number of clusters. - - Args: - xlinear_model: the pretrained model. - Xt: the feature matrix. - beam_size: beam size for inference. - - Returns: - The matching matrix in CSR format. - """ - matching_result = [] - batch_size = 8192 * 16 - kwargs = { - "beam_size": beam_size, - "only_topk": 30, - "post_processor": "l3-hinge", - } - model_chain = xlinear_model.model.model_chain - for i in range((Xt.shape[0] - 1) // batch_size + 1): - beg, end = i * batch_size, (i + 1) * batch_size - end = min(end, Xt.shape[0]) - X_selected = Xt[beg:end] - csr_codes = None - for level in range(len(model_chain) - 1): - cur_model = model_chain[level] - level_pred = cur_model.predict( - X_selected, - csr_codes=csr_codes, - only_topk=beam_size, - post_processor=kwargs["post_processor"], - ) - csr_codes = level_pred - matching_result.append(csr_codes) - matching_result = smat.vstack(matching_result, format="csr") - return matching_result - - -def positive_instances(Xt, Yt, underlying_label_ids): - """Find the instances having some particular label ids. - - For all labels in `underlying_label_ids`, return the list of instances containing - that label as ground-truth. - - Args: - Xt: The feature matrix of shape N x d, where N is number of instances, d is - feature dimension. - Yt: The label matrix of shape N x L, L is the size of label space. - underlying_label_ids: The set of target labels. - - Returns: - A list of positive instance ids and their feature vectors. - """ - row_ids_list = [] - Xt_subsets = [] - for label_id in underlying_label_ids: - row_ids = Yt.indices[Yt.indptr[label_id] : Yt.indptr[label_id + 1]] - Xt_subsets.append(Xt[row_ids]) - row_ids_list.append(row_ids) - return row_ids_list, Xt_subsets - - -def label_id_to_cluster_id(label_id, C, unused_labels): - """Map the label id to the cluster id according to clustering matrix. - - Args: - label_id: the label id. - C: the cluster matrix of shape L x C. - unused_labels: used to adjust the label id. - - Returns: - the cluster id. - """ - # count how many unused labels that are smaller than label_id - offset = sum([l < label_id for l in unused_labels]) - row_id = label_id - offset - assert C.indptr[row_id] + 1 == C.indptr[row_id + 1] - cluster_id = C.indices[C.indptr[row_id]] - return cluster_id - - -def match( - xlinear_model, beam_size, instance_ids_list, X_subsets, cid1, cid2, -): - """Given two clusters, distribute all instances to two groups. - - Separate all input features `X_subsets` into two subsets `x_cid1` and `x_cid2`, - according to the prediction results from `xlinear_model`. If the scores of an - instance in `cid1` is higher than `cid2`, than this instance is assigned to group1. - - Args: - xlinear_model: the model. - beam_size: beam size for inference. - instance_ids_list: the instance id of `X_subsets`. - X_subsets: the feature matrix. - cid1, cid2: the cluster ids of two clusters. - - Returns: - the instance ids of two subsets. - """ - x_cid1 = [] - x_cid2 = [] - for instance_ids, X_subset in zip(instance_ids_list, X_subsets): - matching_matrix = get_matching_matrix( - xlinear_model, X_subset, beam_size, - ).toarray() - mask = matching_matrix[:, cid1] > matching_matrix[:, cid2] - x_cid1.extend(instance_ids[mask]) - x_cid2.extend(instance_ids[~mask]) - return x_cid1, x_cid2 - - -def random_baseline(S1, S2): - """A random baseline that assigns all instances randomly to two groups. - - Args: - S1, S2: the ground truth assignment according to their semantic meanings. - - Returns: - VI scores of this random baseline. - """ - S = np.concatenate((S1, S2), axis=0) - experiment = [] - for _ in range(100): - np.random.shuffle(S) - selector = np.random.randn(len(S)) > 0 - K1 = S[selector] - K2 = S[~selector] - vi_sample = VI(S1, S2, K1, K2) - experiment.append(vi_sample) - return np.mean(experiment) - - -def VI(S1, S2, K1, K2): - """Computes the Variation of Information(VI) between two clusters. - - See: https://en.wikipedia.org/wiki/Variation_of_information for more information. - - Args: - S1, S2: the set of ground truth clusters. - K1, K2: the predicted clusters. - - Returns: - the VI score. - """ - assert len(S1) + len(S2) == len(K1) + len(K2) - n = len(S1) + len(S2) - eps = 1.0e-8 - p1 = len(S1) / n + eps - p2 = len(S2) / n + eps - q1 = len(K1) / n + eps - q2 = len(K2) / n + eps - r11 = len(np.intersect1d(S1, K1)) / n + eps - r12 = len(np.intersect1d(S1, K2)) / n + eps - r21 = len(np.intersect1d(S2, K1)) / n + eps - r22 = len(np.intersect1d(S2, K2)) / n + eps - vi = ( - r11 * (np.log(r11 / p1) + np.log(r11 / q1)) - + r12 * (np.log(r12 / p1) + np.log(r12 / q2)) - + r21 * (np.log(r21 / p2) + np.log(r21 / q1)) - + r22 * (np.log(r22 / p2) + np.log(r22 / q2)) - ) - return -vi - - -def main(args): - # Load Data - Xt = XLinearModel.load_feature_matrix(args.inst_path) - Yt_bin = XLinearModel.load_label_matrix(args.y_binned) - Yt = XLinearModel.load_label_matrix(args.y_origin).tocsc() - - # Optionally load mapper - mapper = {} - if args.pseudo_label_mapper is not None: - with open(args.pseudo_label_mapper, "rb") as reader: - mapper = pkl.load(reader) - inv_mapper = {v: k for k, v in mapper.items()} - - unused_label_set = {} - if args.unused_labels is not None: - with open(args.unused_labels, "rb") as reader: - unused_label_set = pkl.load(reader) - - # Mapper that maps from binned label to its components - with open(args.binned_mapper, "rb") as reader: - binned_label_mapper = pkl.load(reader) - - # Model prediction - xlinear_model = XLinearModel.load(args.model_folder) - - # label clustering matrix of size L x K - leaf_model = xlinear_model.model.model_chain[-1] - C = leaf_model.pC.buf.tocsr() - for fake_label_id, underlying_label_ids in binned_label_mapper.items(): - if len(underlying_label_ids) < 2 or fake_label_id not in inv_mapper: - continue - # given the label ids, gather all positive instances - instance_ids_list, X_subsets = positive_instances(Xt, Yt, underlying_label_ids) - if min(len(s) for s in instance_ids_list) <= 10: - continue - - pseudo_label_id = inv_mapper[fake_label_id] - cid_fake_label = label_id_to_cluster_id(fake_label_id, C, unused_label_set) - cid_pseudo_label = label_id_to_cluster_id(pseudo_label_id, C, unused_label_set) - assert cid_fake_label != cid_pseudo_label - x_cid1, x_cid2 = match( - xlinear_model, - args.beam_size, - instance_ids_list, - X_subsets, - cid_fake_label, - cid_pseudo_label, - ) - vi = VI(x_cid1, x_cid2, *instance_ids_list) - baseline_vi = random_baseline(x_cid1, x_cid2) - print(vi, baseline_vi, baseline_vi - vi) - - -if __name__ == "__main__": - args = parse_arguments() - main(args) diff --git a/examples/overlap-xmc/error_analyze.py b/examples/overlap-xmc/error_analyze.py deleted file mode 100644 index 35a721ba..00000000 --- a/examples/overlap-xmc/error_analyze.py +++ /dev/null @@ -1,255 +0,0 @@ -import argparse -import pickle as pkl - -import matplotlib.pyplot as plt -import numpy as np -import scipy.sparse as smat -from pecos.core import clib -from pecos.utils import smat_util -from pecos.xmc.xlinear.model import XLinearModel - - -def parse_arguments(): - parser = argparse.ArgumentParser() - parser.add_argument( - "-x", - "--inst-path", - type=str, - required=True, - metavar="PATH", - help="path to npz file of feature matrix", - ) - parser.add_argument( - "-y", - "--label-path", - type=str, - required=True, - metavar="PATH", - help="path to the npz file of the label matrix", - ) - parser.add_argument( - "-m", - "--model-folder", - type=str, - required=True, - metavar="DIR", - help="path to the model folder", - ) - parser.add_argument( - "--mapper", - type=str, - default=None, - help="path to pseudo label mapper. If None, this variable is ignored.", - ) - parser.add_argument( - "--unused-labels", - type=str, - default=None, - help="path to unused label set. If None, this variable is ignored.", - ) - parser.add_argument( - "-b", "--beam-size", type=int, default=10, help="Beam size at inference time.", - ) - args = parser.parse_args() - return args - - -def build_label_mapping_matrix(mapper, n_labels, n_new_labels): - """Build a matrix for label score aggregration. - - The label mapping matrix is of shape (n_labels + n_dup_labels) x (n_labels). - The content is - [ - I, - S, - ] - where I is identity matrix of shape n_labels x n_labels, and S is of shape - n_dup_labels x n_labels. S(i, j) = 1 iff (i + n_labels) is a duplication of - label j. - - Args: - mapper: A mapper from duplicated label id to its original id. - n_labels: Number of original labels. - n_new_labels: Equals to n_labels + n_dup_labels. - - Returns: - The sparse matrix for label score aggregation. - """ - eye = smat.diags(np.ones(n_labels), format="csc", dtype=np.float32) - rows, cols, data = [], [], [] - for row in range(n_labels, n_new_labels): - rows.append(row - n_labels) - cols.append(mapper[row]) - data.append(1) - more = smat.coo_matrix( - (data, (rows, cols)), - shape=(n_new_labels - n_labels, n_labels), - dtype=np.float32, - ) - mapper_mat = smat.vstack((eye, more), format="csc") - # normalize columns - # D = 1.0 / np.asarray(mapper_mat.sum(axis=0)).squeeze(axis=0) - # normalized_mapper = mapper_mat.dot(smat.diags(D, format="csc")) - # return normalized_mapper - return mapper_mat - - -def forward_matcher(xlinear_model, X, **kwargs): - """Forward through the matcher to get the leaf id. - - Args: - xlinear_model: The input xlinear model. - X: Input feature matrix. - kwargs: The config. - - Returns: - The matching matrix. - """ - csr_codes = None - model_chain = xlinear_model.model.model_chain - for layer_id in range(len(model_chain) - 1): - cur_model = model_chain[layer_id] - level_pred = cur_model.predict( - X, - csr_codes=csr_codes, - only_topk=kwargs["beam_size"], - post_processor=kwargs["post_processor"], - ) - csr_codes = level_pred - return csr_codes - - -def forward_ranker(xlinear_model, X, csr_codes, **kwargs): - """Forward the ranker using the matching results. - - Args: - xlinear_model: The input xlinear model. - X: Feature matrix. - csr_codes: The matching matrix. - kwargs: The config. - - Returns: - Y_hat: The score matrix of shape n_data x n_labels. - """ - model_chain = xlinear_model.model.model_chain[-1] - Y_hat = model_chain.predict( - X, - csr_codes=csr_codes, - only_topk=kwargs["beam_size"], - post_processor=kwargs["post_processor"], - ) - return Y_hat - - -def merge_pseudo_labels(mapper, truth, pred, merge_by="max"): - """Merge the prediction results of truth and prediction, combining the real labels - and pseudo labels. The combination algorithm can be max or mean. - - Args: - mapper: The label mapping matrix. Find the actual label id from the new label id. - truth: The groundtruth matrix. - pred: The prediction results. - merge_by: If "max" is used, aggregate the scores by max operator. If "mean" is - used, aggregated by average. - - Returns: - The aggregated scores. - """ - # pred: N x Lnew, Lnew x Lold ==> N x Lold - n_labels = pred.shape[1] - len(mapper) - n_new_labels = pred.shape[1] - if len(mapper) == 0: - return truth, pred - if merge_by == "mean": - normalized_mapper = build_label_mapping_matrix(mapper, n_labels, n_new_labels) - pred1 = clib.sparse_matmul(pred, normalized_mapper) - # debiasing - pred_sign = pred.sign() - bias = clib.sparse_matmul(pred_sign, normalized_mapper) - bias.data = 1.0 / np.clip(bias.data, a_min=0.1, a_max=None) - pred = pred1.multiply(bias) - else: - pred = pred.tolil() - truth = truth.tolil() - for pseudo_id, real_id in mapper.items(): - pred[:, real_id] = pred[:, real_id].maximum(pred[:, pseudo_id]) - return truth[:, :n_labels], pred[:, :n_labels] - - -def build_score_transformer(unused_label_set, total_labels): - """Build a matrix that mask-out the unused labels. - - This matrix is essentially all zeros in the row i if label i - is never being used. Otherwise D(i, i) = 1. - """ - D = np.ones(total_labels) - for unused_label in unused_label_set: - D[unused_label] = 0 - transformer = smat.diags( - D, shape=(total_labels, total_labels), format="csc", dtype=np.float32 - ) - return transformer - - -def do_analyze(args): - # Load Data - Xt = XLinearModel.load_feature_matrix(args.inst_path) - Yt = XLinearModel.load_label_matrix(args.label_path) - - # Optionally load mapper - mapper = {} - if args.mapper is not None: - with open(args.mapper, "rb") as reader: - mapper = pkl.load(reader) - unused_label_set = {} - if args.unused_labels is not None: - with open(args.unused_labels, "rb") as reader: - unused_label_set = pkl.load(reader) - - # Model prediction - xlinear_model = XLinearModel.load(args.model_folder) - kwargs = { - "beam_size": args.beam_size, - "only_topk": 160, - "post_processor": "l3-hinge", - } - - pred = None - batch_size = 8192 * 16 - pred_batches = [] - M_batches = [] - for i in range((Xt.shape[0] - 1) // batch_size + 1): - beg, end = i * batch_size, (i + 1) * batch_size - end = min(end, Xt.shape[0]) - X_batch = Xt[beg:end, :] - M_batch = forward_matcher(xlinear_model, X_batch, **kwargs) - # pred_batch = forward_ranker(xlinear_model, X_batch, M_batch, **kwargs) - pred_batch = xlinear_model.predict(Xt[beg:end, :], **kwargs) - M_batches.append(M_batch) - pred_batches.append(pred_batch) - - Mb = smat_util.binarized(smat.vstack(M_batches)) - C = xlinear_model.model.model_chain[-1].pC.buf - MC = clib.sparse_matmul(Mb, C.transpose()) - avg_inner_prod = MC.sum(axis=1).mean(axis=0)[0, 0] - - pred = smat.vstack(pred_batches) - unused_label_transformer = build_score_transformer(unused_label_set, pred.shape[1]) - # we set j-th column of pred to zero, iff j is an unused label, this is prevent label j - # from being accidentally ranked to the front. - pred = clib.sparse_matmul(pred, unused_label_transformer) - truth = Yt - - print("Merging pseudo labels") - truth, pred = merge_pseudo_labels(mapper, truth, pred, merge_by="mean") - truth = truth.tocsr() - pred = pred.tocsr() - print("Calculating metrics") - metric = smat_util.Metrics.generate(truth, pred, topk=10) - print(metric) - print("Average #inner prod: ", avg_inner_prod) - - -if __name__ == "__main__": - args = parse_arguments() - do_analyze(args) diff --git a/examples/overlap-xmc/make_combined_label.py b/examples/overlap-xmc/make_combined_label.py deleted file mode 100644 index 67cbb002..00000000 --- a/examples/overlap-xmc/make_combined_label.py +++ /dev/null @@ -1,150 +0,0 @@ -#!/usr/bin/env python -import argparse -import glob -import os -import pickle as pkl -import sys -from collections import defaultdict - -import numpy as np -import scipy as sp -import scipy.sparse as sps - - -def combine_Y(mapper, new_nr_labels, y): - """Create the new ground-truth label matrix given the label mapper. - - Args: - mapper: the mapper returned from `combine_from_cluster` function. - new_nr_labels: the number of new labels returned from `combined_from_cluster` function. - y: the old label matrix of shape nr_instance x nr_labels. - - Returns: - the new label matrix of shape nr_instance x new_nr_labels. - """ - new_y_cols = [] - new_y_rows = [] - y = y.tocsr() - for inst_id in range(y.shape[0]): - orig_labels = y.indices[y.indptr[inst_id] : y.indptr[inst_id + 1]] - for orig_label in orig_labels: - new_label = mapper[orig_label] - new_y_cols.append(new_label) - new_y_rows.append(inst_id) - new_y = sps.coo_matrix( - (np.ones_like(new_y_cols), (new_y_rows, new_y_cols)), - shape=(y.shape[0], new_nr_labels), - ) - return new_y.tocsr() - - -def cluster_chain(path, level_from_bottom=0): - """Obtain the clustering matrix given the model checkpoint. - - Args: - path: path to the model checkpoint. - level_from_bottom: the level of label tree starting from the bottom. - - Returns: - the clustering matrix `C` to be used for `combine_from_cluster` function. - """ - Cs = [] - for cf in sorted(glob.glob(path)): - C = sps.load_npz(cf) - Cs.append(C) - C_from_bottom = [Cs[-1]] - for C in Cs[::-1][1:]: - C_from_bottom.append(C_from_bottom[-1].dot(C)) - return C_from_bottom[level_from_bottom] - - -def combine_from_cluster(C, bin_size): - """Combine the labels in the same cluster. - - Given the clustering matrix `C`, we randomly group `bin_size` number of labels as a synthetic composite label, - which is further saved to our synthetic dataset. - - Args: - C: A scipy sparse matrix. - bin_size: An integer specifying how many labels to group into a new fake label. - - Returns: - A dictionary `mapper` that maps the id of original label to the id of new label. - As well as `new_label_count` meaning the number of labels in the new synthetic dataset. - """ - C = C.tocsc() - new_label_count = 0 - mapper = {} - for group_id in range(C.shape[1]): - # which labels are in the same cluster? - row_ids = C.indices[C.indptr[group_id] : C.indptr[group_id + 1]] - row_ids = row_ids.copy() - np.random.shuffle(row_ids) - # randomly group labels n-by-n - nr_new_labels_in_cluster = len(row_ids) // bin_size + len(row_ids) % bin_size - for id_of_bin in range(nr_new_labels_in_cluster): - for id_in_bin in range(bin_size): - idx = id_of_bin * bin_size + id_in_bin - if idx < len(row_ids): - mapper[row_ids[idx]] = id_of_bin + new_label_count - new_label_count += nr_new_labels_in_cluster - return mapper, new_label_count - - -def inversion_mapper(mapper): - """Find the list of label pairs that are mapped to the same bins. - - This amounts to finding the keys with the same labels in mapper. The results are sorted in increasing order. - - Args: - mapper: the result from `combined_from_cluster` function. - - Returns: - the inversion of mapper. - """ - invert_mapper = defaultdict(list) - for k, v in mapper.items(): - invert_mapper[v].append(k) - invert_mapper = {k: sorted(v) for k, v in invert_mapper.items()} - return invert_mapper - - -if __name__ == "__main__": - np.random.seed(0) - parser = argparse.ArgumentParser("Create dataset for Section 5.1 in our paper.") - parser.add_argument( - "--data", type=str, default="eurlex-4k", help="Name of the dataset." - ) - parser.add_argument( - "--level", - type=int, - default=0, - help="Select the level at which labels in the same clusters are combined.", - ) - parser.add_argument( - "--bin-size", - type=int, - default=2, - help="How many labels to group for each synthetic composite label.", - ) - args = parser.parse_args() - - data = args.data - bin_size = args.bin_size - combine_level = args.level - folder = f"./dataset/xmc-base/{data}/" - out_folder = f"./dataset-binned/{data}/" - C = cluster_chain(f"./model/{data}/ranker/**/C.npz", combine_level) - os.makedirs(out_folder, exist_ok=True) - - ytr = sps.load_npz(folder + "Y.trn.npz") - yte = sps.load_npz(folder + "Y.tst.npz") - - mapper, new_nr_labels = combine_from_cluster(C, bin_size) - new_ytr = combine_Y(mapper, new_nr_labels, ytr) - new_yte = combine_Y(mapper, new_nr_labels, yte) - invert_mapper = inversion_mapper(mapper) - with open(out_folder + "mapper.pkl", "wb") as writer: - pkl.dump(invert_mapper, writer) - sps.save_npz(out_folder + "Y.trn.npz", new_ytr) - sps.save_npz(out_folder + "Y.tst.npz", new_yte) diff --git a/examples/overlap-xmc/reorganize_clusters.py b/examples/overlap-xmc/reorganize_clusters.py deleted file mode 100644 index 7434fc63..00000000 --- a/examples/overlap-xmc/reorganize_clusters.py +++ /dev/null @@ -1,364 +0,0 @@ -import argparse -import os -import pickle as pkl -from collections import defaultdict - -import numpy as np -import scipy.sparse as smat -from numba import njit -from numba.core import types -from numba.typed import Dict -from pecos.core.base import clib -from pecos.utils import smat_util -from pecos.utils.cluster_util import ClusterChain -from pecos.xmc import MLModel -from pecos.xmc.xlinear import XLinearModel - - -def parse_arguments(): - parser = argparse.ArgumentParser( - prog="Reorganize the clusters, move some of the labels around to " - "improve recall." - ) - parser.add_argument( - "-x", - "--inst-path", - type=str, - required=True, - metavar="PATH", - help="path to npz file of feature matrix", - ) - parser.add_argument( - "-y", - "--label-path", - type=str, - required=True, - metavar="PATH", - help="path to the npz file of the label matrix", - ) - parser.add_argument( - "-m", - "--model-folder", - type=lambda p: os.path.abspath(p), - required=True, - metavar="DIR", - help="path to the model folder", - ) - parser.add_argument( - "-o", - "--model-folder-output", - type=lambda p: os.path.abspath(p), - required=True, - metavar="DIR", - help="path to the model output folder", - ) - parser.add_argument( - "-b", - "--beam-size", - type=int, - required=True, - help="Beam size to calculate the matching matrix.", - ) - parser.add_argument( - "--n_copies", type=int, default=2, help="number of copies for each label.", - ) - args = parser.parse_args() - return args - - -def get_matching_matrix(xlinear_model, Xt, beam_size=10): - """Get the matching matrix. - - The matching matrix indicates which cluster(s) are selected for data point in X. The - final results is a sparse matrix of shape N x C, where N is the number of data, and C - is the number of clusters. - - Args: - xlinear_model: the pretrained model. - Xt: the feature matrix. - beam_size: beam size for inference. - - Returns: - The binarized matching matrix in CSR format. - """ - matching_result = [] - batch_size = 8192 * 16 - kwargs = { - "beam_size": beam_size, - "only_topk": 30, - "post_processor": "l3-hinge", - } - model_chain = xlinear_model.model.model_chain - for i in range((Xt.shape[0] - 1) // batch_size + 1): - beg, end = i * batch_size, (i + 1) * batch_size - end = min(end, Xt.shape[0]) - X_selected = Xt[beg:end] - csr_codes = None - for level in range(len(model_chain) - 1): - model_l = model_chain[level] - level_pred = model_l.predict( - X_selected, - csr_codes=csr_codes, - only_topk=beam_size, - post_processor=kwargs["post_processor"], - ) - csr_codes = level_pred - matching_result.append(csr_codes) - matching_result = smat.vstack(matching_result, format="csr").sign() - return matching_result - - -@njit -def construct_new_C_and_Y( - counts_rows, - counts_cols, - counts, - row_ids, - row_ranges, - C_rows, - sort_idx, - nr_labels, - max_cluster_size, - n_copies, -): - """Determine the new clustering matrix and the new label matrix given the couting matrix. - - This function implements Eq.(10) in our paper. I.e. given the couting matrix C = Y^T * M, - we select the correct cluster id for each label one by one, in descending order of C entries, - possibly assign a label multiple times (`n_copies`) to different clusters. Finally, the new - cluster and new label matrix is returned. Notice that Numba is used here, this prevents us - from passing scipy sparse matrix directly. - - Args: - counts_rows, counts_cols, counts: The counting matrix in COO format. - row_ids, row_ranges: The indices and indptr of original Y matrix in CSC format. - C_rows: Clustering matrix C in LIL format, converted to list of numpy arrays. - sort_idx: Index of counts_{rows,cols} to sort them in decending order. - nr_labels: Number of original labels. - max_cluster_size: (Unused for now) Hard constraints to limit the number of labels - in each cluster (to balance cluster size). - n_copies: Max number of copies for each label (\lambda in our paper). - - Returns: - New cluster matrix (`new_C_*`), new label matrix (`new_Y_*`), the replicated label - assignment (`C_overlap_*`), number of duplicated labels (`nr_copied_labels`), a map - from new label id to the underlying label id (`mapper`), unused labels that never - show up in training (`unused_labels`), number of lightly used labels (`nr_tail_labels`). - """ - # construct empty cluster matrix and label matrix - nr_copied_labels = 0 - new_C_cols = [] - new_C_data = [] - new_Y_rows = [] - labels_included = set() - mapper = Dict.empty(key_type=types.int64, value_type=types.int64,) - cluster_size = Dict.empty(key_type=types.int64, value_type=types.int64,) - pseudo_label_count = Dict.empty(key_type=types.int64, value_type=types.int64,) - # results - C_overlap_rows, C_overlap_cols = [], [] - max_count = n_copies - # adding labels to clusters one by one in descending frequency - for idx in sort_idx: - label_id = counts_rows[idx] - leaf_id = counts_cols[idx] - if label_id in pseudo_label_count and pseudo_label_count[label_id] >= max_count: - continue - # If you need to contrain the max cluster size, then - # uncomment following two lines - # if label_count[leaf_id] >= max_cluster_size: - # continue - if leaf_id not in cluster_size: - cluster_size[leaf_id] = 1 - else: - cluster_size[leaf_id] += 1 - - if label_id not in pseudo_label_count: - pseudo_label_count[label_id] = 1 - else: - pseudo_label_count[label_id] += 1 - - if label_id in labels_included: - # add a pseudo label that duplicates label_id - pseudo_label_id = nr_copied_labels + nr_labels - mapper[pseudo_label_id] = label_id - # add one more row to C (in lil format) - new_C_cols.append([leaf_id]) - new_C_data.append([1]) - # add one more column to Yt - examples = row_ids[row_ranges[label_id] : row_ranges[label_id + 1]] - new_Y_rows.append(examples) - nr_copied_labels += 1 - else: - # add a new label - labels_included.add(label_id) - C_overlap_rows.append(label_id) - C_overlap_cols.append(leaf_id) - - # exit early if we have too many effective labels - if len(mapper) >= max_count * nr_labels: - break - # add missing labels back to clusters - nr_tail_labels = 0 - for label_id in range(nr_labels): - if label_id not in labels_included: - original_leaf_id = C_rows[label_id][0] - C_overlap_rows.append(label_id) - C_overlap_cols.append(original_leaf_id) - labels_included.add(label_id) - nr_tail_labels += 1 - - unused_labels = set() - for label_id in range(nr_labels): - if label_id not in labels_included: - unused_labels.add(label_id) - - # new_Y elements - new_Y_indptr = [0] - new_Y_indices = [] - for rows in new_Y_rows: - new_Y_indptr.append(new_Y_indptr[-1] + len(rows)) - new_Y_indices.extend(rows) - new_Y_data = np.ones(len(new_Y_indices), dtype=np.int32) - return ( - new_C_cols, - new_C_data, - new_Y_data, - new_Y_indices, - new_Y_indptr, - C_overlap_cols, - C_overlap_rows, - nr_copied_labels, - mapper, - unused_labels, - nr_tail_labels, - ) - - -def get_topk_clusters(xlinear_model, Xt, Yt, beam_size): - """Iterate over all labels, for each label, gather the training samples - from which the label is activated. Then for each training sample, gather - the top-1 (beam_size=1) leaf node. The topk leaves will be returned for - the reorganization step. - - Args: - xlinear_model: the pre-trained model. - Xt: traning features in csr format. - Yt: training labels in csr format. - - Returns: - None, xlinear_model will be modified in-place. - """ - # Get the clustering matrix in the last level - model_chain = xlinear_model.model.model_chain - leaf_model = model_chain[-1] - - # Get the matching matrix - M = get_matching_matrix(xlinear_model, Xt, beam_size=beam_size) - C = leaf_model.pC.buf - - # Get the counting matrix by YtM - counts = Yt.transpose().dot(M).tocoo() - counts.eliminate_zeros() - - counts_rows, counts_cols, counts = counts.row, counts.col, counts.data - sort_idx = np.argsort(counts)[::-1] - - Yt_csc = Yt.tocsc() - row_ranges = Yt_csc.indptr - row_ids = Yt_csc.indices - - # Cast C to lil format - C = C.tolil() - C_rows = C.rows - max_cluster_size = int(1.0 * C.shape[0] / C.shape[1]) - ( - new_C_cols, - new_C_data, - new_Y_data, - new_Y_indices, - new_Y_indptr, - C_overlap_cols, - C_overlap_rows, - out_labels, - mapper, - unused_labels, - nr_tail_labels, - ) = construct_new_C_and_Y( - np.asarray(counts_rows, dtype=np.int32), - np.asarray(counts_cols, dtype=np.int32), - np.asarray(counts, dtype=np.int32), - np.asarray(row_ids, dtype=np.int32), - np.asarray(row_ranges, dtype=np.int32), - [np.asarray(row, dtype=np.int32) for row in C_rows], - sort_idx, - Yt.shape[1], - max_cluster_size, - args.n_copies, - ) - C_overlap = smat.coo_matrix( - (np.ones_like(C_overlap_cols), (C_overlap_rows, C_overlap_cols)), - shape=C.shape, - dtype=C.dtype, - ).tocsr() - print(f"#copied labels: {out_labels}, #tail labels: {nr_tail_labels}") - - new_C = smat.lil_matrix((out_labels, C.shape[1]), dtype=C.dtype) - new_C.data = new_C_data - new_C.rows = new_C_cols - C = smat.vstack((C_overlap, new_C.tocsc()), format="csc") - - new_Y = smat.csc_matrix( - (new_Y_data, new_Y_indices, new_Y_indptr), - shape=(Yt.shape[0], len(new_Y_indptr) - 1), - dtype=Yt.dtype, - ) - Yt = smat.hstack((Yt, new_Y), format="csr") - - assert C.shape[1] == leaf_model.pC.buf.shape[1] - assert C.shape[0] == Yt.shape[1] - return C, Yt, dict(mapper), unused_labels - - -def main(args): - # Load Data - Xt = XLinearModel.load_feature_matrix(args.inst_path) - Yt = XLinearModel.load_label_matrix(args.label_path) - - # Model prediction - xlinear_model = XLinearModel.load(args.model_folder) - C, Yt, mapper, unused_labels = get_topk_clusters( - xlinear_model, Xt, Yt, args.beam_size - ) - - # Extract the cluster chain from model_chain - clusters = [m.pC.buf for m in xlinear_model.model.model_chain[:-1]] + [C] - chain = ClusterChain(clusters) - - # Save to folder - chain.save(args.model_folder_output) - - xt_out_name = os.path.join( - args.model_folder_output, os.path.basename(args.inst_path) - ) - smat.save_npz(xt_out_name, Xt) - yt_out_name = os.path.join( - args.model_folder_output, os.path.basename(args.label_path) - ) - smat.save_npz(yt_out_name, Yt) - mapper_file = os.path.join(args.model_folder_output, "pseudo_label_mapping.pkl") - with open(mapper_file, "wb") as writer: - pkl.dump(mapper, writer) - unused_labels_file = os.path.join(args.model_folder_output, "unused_labels.pkl") - with open(unused_labels_file, "wb") as writer: - pkl.dump(unused_labels, writer) - - -if __name__ == "__main__": - args = parse_arguments() - # save to new directory, avoid overwritting - assert ( - args.model_folder != args.model_folder_output - ), "You can't set the model desitination path to be the same as source path." - if not os.path.exists(args.model_folder_output): - os.makedirs(args.model_folder_output) - main(args) diff --git a/examples/overlap-xmc/run_base.sh b/examples/overlap-xmc/run_base.sh deleted file mode 100644 index 7da2e734..00000000 --- a/examples/overlap-xmc/run_base.sh +++ /dev/null @@ -1,45 +0,0 @@ -#!/bin/bash - -# Complete the training of our method -# By default, amazon-670k will be used -data=${1-amazon-670k} - -# Step 1. train a XR-Linear model -echo "Training XR-Linear model" -python -m pecos.xmc.xlinear.train \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.trn.npz \ - -y ./dataset/xmc-base/${data}/Y.trn.npz \ - -m ./model/${data} \ - --nr-splits 32 \ - -b 10 - - -# Step 4. create new dataset with overlapping label space -echo "Creating new dataset" -python reorganize_clusters.py \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.trn.npz \ - -y ./dataset/xmc-base/${data}/Y.trn.npz \ - -m model/${data} \ - -b 10 \ - --n_copies 2 \ - -o model/${data}-overlap - -# Step 5. train the model again on the new dataset -echo "Train our model" -python -m pecos.xmc.xlinear.train \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.trn.npz \ - -y ./model/${data}-overlap/Y.trn.npz \ - -c ./model/${data}-overlap \ - -m ./model/${data}-overlap \ - -b 10 - - -# Step 6. perform error analysis on our model -echo "Computing Prec@k and Recall@k" -python error_analyze.py \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.tst.npz \ - -y ./dataset/xmc-base/${data}/Y.tst.npz \ - -m ./model/${data}-overlap \ - -b 10 \ - --mapper ./model/${data}-overlap/pseudo_label_mapping.pkl \ - --unused-labels ./model/${data}-overlap/unused_labels.pkl diff --git a/examples/overlap-xmc/run_binned.sh b/examples/overlap-xmc/run_binned.sh deleted file mode 100644 index 22e282e6..00000000 --- a/examples/overlap-xmc/run_binned.sh +++ /dev/null @@ -1,63 +0,0 @@ -#!/bin/bash - -# Complete the training of our method -# By default, amazon-670k will be used -data=${1-amazon-670k} - -# Step 1. train a XR-Linear model -echo "Training XR-Linear model" -python -m pecos.xmc.xlinear.train \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.trn.npz \ - -y ./dataset/xmc-base/${data}/Y.trn.npz \ - -m ./model/${data} \ - --nr-splits 32 \ - -b 10 - - -# Step 2: create binned dataset -echo "Creating the binned dataset" -python ./make_combined_label.py \ - --data $data \ - --level 2 \ - --bin-size 2 - - -# Step 3. train the XR-Linear model again on the binned dataset -echo "Train the XR-Linear model on the binned dataset" -python -m pecos.xmc.xlinear.train \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.trn.npz \ - -y ./dataset-binned/${data}/Y.trn.npz \ - -m ./model/${data} \ - --nr-splits 32 \ - -b 10 - - -# Step 4. create new dataset with overlapping label space -echo "Creating new dataset" -python reorganize_clusters.py \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.trn.npz \ - -y ./dataset-binned/${data}/Y.trn.npz \ - -m model/${data} \ - -b 10 \ - --n_copies 2 \ - -o model/${data}-overlap - -# Step 5. train the model again on the new dataset -echo "Train our model" -python -m pecos.xmc.xlinear.train \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.trn.npz \ - -y ./model/${data}-overlap/Y.trn.npz \ - -c ./model/${data}-overlap \ - -m ./model/${data}-overlap \ - -b 10 - - -# Step 6. perform error analysis on our model -echo "Computing Prec@k and Recall@k" -python error_analyze.py \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.tst.npz \ - -y ./dataset-binned/${data}/Y.tst.npz \ - -m ./model/${data}-overlap \ - -b 10 \ - --mapper ./model/${data}-overlap/pseudo_label_mapping.pkl \ - --unused-labels ./model/${data}-overlap/unused_labels.pkl diff --git a/examples/overlap-xmc/run_metric.sh b/examples/overlap-xmc/run_metric.sh deleted file mode 100644 index f3fe761a..00000000 --- a/examples/overlap-xmc/run_metric.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/bash - - -data=amazon-670k -split=trn -python ./disentangle_metric.py \ - -x ./dataset/xmc-base/${data}/tfidf-attnxml/X.${split}.npz \ - --y-binned ./dataset-binned/${data}/Y.${split}.npz \ - --y-origin ./dataset/xmc-base/${data}/Y.${split}.npz \ - -m ./model/${data}-overlap \ - --binned-mapper ./dataset-binned/${data}/mapper.pkl \ - --pseudo-label-mapper ./model/${data}-overlap/pseudo_label_mapping.pkl \ - --unused-labels ./model/${data}-overlap/unused_labels.pkl \ - -b 10 diff --git a/examples/pecos-xrlinear-jmlr22/README.md b/examples/pecos-xrlinear-jmlr22/README.md deleted file mode 100644 index 7d103838..00000000 --- a/examples/pecos-xrlinear-jmlr22/README.md +++ /dev/null @@ -1,72 +0,0 @@ -# Experiment Code for PECOS Technical Report, JMLR 2022 - -This folder contains code to train XR-Linear models and reproduce experiments -in ["PECOS: Prediction for Enormous and Correlated Output Spaces"](https://arxiv.org/abs/2010.05878). - - -## Getting Started -* Clone the repository and enter `examples/pecos-xrlinear-jmlr22` directory. -* First create a [virtual environment](https://docs.python.org/3/library/venv.html) and then install dependencies -by running the following command: -```bash -pip install -r requirements.txt -``` -If you're unfamiliar with Python virtual environments, check out the -[user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). - - -## Downloading Data -The XMC datasets can be download at -``` bash -cd ./datasets -# eurlex-4k, wiki10-31k, amazoncat-13k, amazon-670k, wiki-500k, amazon-3m -DATASET="eurlex-4k" -wget https://archive.org/download/pecos-dataset/xmc-base/${DATASET}.tar.gz -tar -zxvf ./${DATASET}.tar.gz -``` - -## XR-Linear Models with Various Hierarchical Label Trees -For the results in Table 1 and Table 3, -we train and evaluate XR-Linear models with different branching factor of hierarchical label trees (HLTs), -which implicitly controls the tree depth of HLTs. - -For each braching factor `B={2, 8, 32}`, we first learn three HLTs under three different random seeds. -We then create an ensemble model by aggregating predictions from three XR-Linear models. -``` bash -bash exp_v1.sh ${DATASET} -``` - -The experiment results of Table 1 are available at -``` bash -tail ./exp_v1/saved_models/${DATASET}/nrs-32_*.log -n 3 -``` -Similarly, the experiment results of Table 3 are available at -``` bash -tail ./exp_v1/saved_models/${DATASET}/nrs-*_ensemble-average.log -n 3 -``` - - -## XR-Linear Models with Various Negative Sampling Scheme -For the results in Table 2, -we train and evaluate XR-Linear models with different negative sampling scheme. - -``` bash -NS_SCHEME="tfn+man" -bash exp_v2.sh ${DATASET} ${NS_SCHEME} -``` -The experiment results of Table 3 are available at -``` bash -tail ./exp_v2/saved_models/${DATASET}k/ns-${NS_SCHEME}/beam-50_ensemble-average.log -n 3 -``` - - -## XR-Transformer Models -To reproduce experiment results of XR-Transformer models, see -[link](https://github.com/amzn/pecos/tree/mainline/examples/xr-transformer-neurips21) - - -## Citation - -If you find this useful, please consider citing our paper. - -* ["PECOS: Prediction for Enormous and Correlated Output Spaces"](https://arxiv.org/abs/2010.05878) [[bib]](./bibtex/yu2020pecos.bib) diff --git a/examples/pecos-xrlinear-jmlr22/ensemble_evaluate.py b/examples/pecos-xrlinear-jmlr22/ensemble_evaluate.py deleted file mode 100644 index 4c00a211..00000000 --- a/examples/pecos-xrlinear-jmlr22/ensemble_evaluate.py +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env python3 -u - -import argparse - -from pecos.utils.smat_util import sorted_csr, CsrEnsembler, load_matrix - -def parse_arguments(): - parser = argparse.ArgumentParser() - - parser.add_argument( - "-y", - "--truth-path", - type=str, - required=True, - metavar="PATH", - help="path to the file of with ground truth output (CSR: nr_insts * nr_items)", - ) - parser.add_argument( - "-p", - "--pred-path", - type=str, - required=True, - nargs="*", - metavar="PATH", - help="path to the file of predicted output (CSR: nr_insts * nr_items)", - ) - parser.add_argument( - "--tags", - type=str, - required=True, - nargs="*", - metavar="PATH", - help="tags attached to each prediction", - ) - parser.add_argument( - "--ens-method", - type=str, - metavar="STR", - default="rank_average", - help="prediction ensemble method", - ) - - return parser - - -def do_evaluation(args): - """ Evaluate xlinear predictions """ - assert len(args.tags) == len(args.pred_path) - Y_true = sorted_csr(load_matrix(args.truth_path).tocsr()) - Y_pred = [sorted_csr(load_matrix(pp).tocsr()) for pp in args.pred_path] - print("==== evaluation results ====") - CsrEnsembler.print_ens(Y_true, Y_pred, args.tags, ens_method=args.ens_method) - - -if __name__ == "__main__": - parser = parse_arguments() - args = parser.parse_args() - do_evaluation(args) diff --git a/examples/pecos-xrlinear-jmlr22/exp_v1.sh b/examples/pecos-xrlinear-jmlr22/exp_v1.sh deleted file mode 100644 index 6fd8b7e8..00000000 --- a/examples/pecos-xrlinear-jmlr22/exp_v1.sh +++ /dev/null @@ -1,57 +0,0 @@ - -if [ -z ${1} ]; then - echo "bash exp_v1.sh [dataset]" - exit -fi -if [ ${1} != "eurlex-4k" ] && [ ${1} != "wiki10-31k" ] && [ ${1} != "amazoncat-13k" ] \ - [ ${1} != "amazon-670k" ] && [ ${1} != "wiki-500k" ] && [ ${1} != "amazon-3m" ]; then - echo "dataset=${1} is not support!" - exit -fi - -dataset=$1 -data_dir=./datasets/xmc-base/${dataset} -output_dir=./exp_v1 -beam_size=10 - -# EXP-1 -nrs_arr=( 2 8 32 ) -seed_arr=( 0 1 2 ) -for nr_splits in "${nrs_arr[@]}"; do - pred_npz_string="" - pred_tag_string="" - - for seed in "${seed_arr[@]}"; do - saved_model_dir=${output_dir}/saved_models/${dataset}/nrs-${nr_splits}_seed-${seed} - mkdir -p ${saved_model_dir} - pred_npz=${saved_model_dir}/Yp.tst.b-${beam_size}.npz - pred_tag=${nr_splits}_seed-${seed} - python -u xrl_train.py \ - -x ${data_dir}/tfidf-attnxml/X.trn.npz \ - -y ${data_dir}/Y.trn.npz \ - -m ${saved_model_dir} \ - --nr-splits ${nr_splits} \ - --seed ${seed} \ - --beam-size ${beam_size} \ - |& tee ${saved_model_dir}/train.log - python -u xrl_predict.py \ - -m ${saved_model_dir} \ - -x ${data_dir}/tfidf-attnxml/X.tst.npz \ - -y ${data_dir}/Y.tst.npz \ - -o ${pred_npz} \ - |& tee ${saved_model_dir}/eval.log - pred_npz_string="${pred_npz_string} ${pred_npz}" - pred_tag_string="${pred_tag_string} ${pred_tag}" - done - ## - ens_method_arr=( average rank_average softmax_average sigmoid_average ) - for ens_method in "${ens_method_arr[@]}"; do - python -u ensemble_evaluate.py \ - -y ${data_dir}/Y.tst.npz \ - -p ${pred_npz_string} \ - --tags ${pred_tag_string} \ - --ens-method ${ens_method} \ - |& tee ${output_dir}/saved_models/${dataset}/nrs-${nr_splits}_ensemble-${ens_method}.log - done - #exit -done diff --git a/examples/pecos-xrlinear-jmlr22/exp_v2.sh b/examples/pecos-xrlinear-jmlr22/exp_v2.sh deleted file mode 100644 index a67f9cfb..00000000 --- a/examples/pecos-xrlinear-jmlr22/exp_v2.sh +++ /dev/null @@ -1,62 +0,0 @@ - -if [ -z ${1} ] && [ -z ${2} ]; then - echo "bash exp_v1.sh [dataset] [ns_scheme]" - exit -fi -if [ ${1} != "eurlex-4k" ] && [ ${1} != "wiki10-31k" ] && [ ${1} != "amazoncat-13k" ] \ - [ ${1} != "amazon-670k" ] && [ ${1} != "wiki-500k" ] && [ ${1} != "amazon-3m" ]; then - echo "dataset=${1} is not support!" - exit -fi -if [ ${2} != "man" ] && [ ${2} != "tfn+man" ]; then - echo "ns_scheme=${2} is not support!" -fi - -dataset=$1 -ns_scheme=$2 # man, tfn+man -data_dir=./datasets/xmc-base/${dataset} -output_dir=./exp_v2 - -# EXP-2 -nr_splits=32 -seed_arr=( 0 1 2 ) -beam_arr=( 10 20 50 ) -for beam_size in "${beam_arr[@]}"; do - pred_npz_string="" - pred_tag_string="" - - for seed in "${seed_arr[@]}"; do - saved_model_dir=${output_dir}/saved_models/${dataset}/ns-${ns_scheme}/beam-${beam_size}_seed-${seed} - mkdir -p ${saved_model_dir} - pred_npz=${saved_model_dir}/Yp.tst.b-${beam_size}.npz - pred_tag=${ns_scheme}_seed-${seed} - python -u xrl_train.py \ - -x ${data_dir}/tfidf-attnxml/X.trn.npz \ - -y ${data_dir}/Y.trn.npz \ - -m ${saved_model_dir} \ - --seed ${seed} \ - --nr-splits ${nr_splits} \ - --negative-sampling ${ns_scheme} \ - --beam-size ${beam_size} \ - |& tee ${saved_model_dir}/train.log - python -u xrl_predict.py \ - -m ${saved_model_dir} \ - -x ${data_dir}/tfidf-attnxml/X.tst.npz \ - -y ${data_dir}/Y.tst.npz \ - -o ${pred_npz} \ - |& tee ${saved_model_dir}/eval.log - pred_npz_string="${pred_npz_string} ${pred_npz}" - pred_tag_string="${pred_tag_string} ${pred_tag}" - done - ## - ens_method_arr=( average rank_average softmax_average sigmoid_average ) - for ens_method in "${ens_method_arr[@]}"; do - python -u ensemble_evaluate.py \ - -y ${data_dir}/Y.tst.npz \ - -p ${pred_npz_string} \ - --tags ${pred_tag_string} \ - --ens-method ${ens_method} \ - |& tee ${output_dir}/saved_models/${dataset}/ns-${ns_scheme}/beam-${beam_size}_ensemble-${ens_method}.log - done - #exit -done diff --git a/examples/pecos-xrlinear-jmlr22/requirements.txt b/examples/pecos-xrlinear-jmlr22/requirements.txt deleted file mode 100644 index f53a8d6d..00000000 --- a/examples/pecos-xrlinear-jmlr22/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -libpecos==0.2.3 diff --git a/examples/pecos-xrlinear-jmlr22/xrl_predict.py b/examples/pecos-xrlinear-jmlr22/xrl_predict.py deleted file mode 100644 index afb06b34..00000000 --- a/examples/pecos-xrlinear-jmlr22/xrl_predict.py +++ /dev/null @@ -1,197 +0,0 @@ -# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance -# with the License. A copy of the License is located at -# -# http://aws.amazon.com/apache2.0/ -# -# or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES -# OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions -# and limitations under the License. -import argparse -import os -import sys -import logging -import time -from pecos.utils import smat_util -from pecos.xmc import PostProcessor -from pecos.xmc.xlinear.model import XLinearModel -from sklearn.preprocessing import normalize - - -LOGGER = logging.getLogger() -LOGGER.setLevel(logging.DEBUG) - -handler = logging.StreamHandler(sys.stdout) -handler.setLevel(logging.DEBUG) -formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") -handler.setFormatter(formatter) -LOGGER.addHandler(handler) - - -def parse_arguments(): - """Parse prediction arguments""" - - parser = argparse.ArgumentParser() - - # Required parameters - parser.add_argument( - "-x", - "--inst-path", - type=str, - required=True, - metavar="PATH", - help="path to the npz file of the feature matrix (CSR, nr_insts * nr_feats)", - ) - - parser.add_argument( - "-m", - "--model-folder", - type=str, - required=True, - metavar="DIR", - help="path to the model folder.", - ) - - # Optional - parser.add_argument( - "-k", - "--only-topk", - type=int, - default=None, - metavar="INT", - help="override the only topk specified in the model (default None to disable overriding)", - ) - - parser.add_argument( - "-b", - "--beam-size", - type=int, - default=None, - metavar="INT", - help="override the beam size specified in the model (default None to disable overriding)", - ) - - parser.add_argument( - "-pp", - "--post-processor", - type=str, - choices=PostProcessor.valid_list(), - default=None, - metavar="STR", - help="override the post processor specified in the model (default None to disable overriding)", - ) - - parser.add_argument( - "-y", - "--label-path", - type=str, - default=None, - metavar="PATH", - help="path to the npz file of the label matrix (CSR, nr_insts * nr_labels)", - ) - - parser.add_argument( - "-o", - "--save-pred-path", - type=str, - default=None, - metavar="PATH", - help="path to save the predictions (sorted CSR, nr_insts * nr_labels)", - ) - - parser.add_argument( - "-B", - "--max-pred-chunk", - default=10 ** 7, - metavar="INT", - type=int, - help="Max number of instances to predict on at once, set to avoid OOM. Set to None to predict on all instances at once. Default 10^7", - ) - - parser.add_argument( - "-n", - "--threads", - type=int, - default=-1, - metavar="THREADS", - help="number of threads to use (default -1 to denote all the CPUs)", - ) - - parser.add_argument( - "-so", - "--selected-output", - type=str, - default=None, - metavar="PATH", - help="path to the npz file of the selected output matrix (CSR, nr_insts * nr_labels), only-topk and beam-size are ignored if given", - ) - return parser - - -def do_predict(args): - """Predict and Evaluate for xlinear model - - Args: - args (argparse.Namespace): Command line arguments parsed by `parser.parse_args()` - """ - - # Load data - LOGGER.info("| loading data begin...") - start_time = time.time() - Xt = XLinearModel.load_feature_matrix(args.inst_path) - Xt = normalize(Xt, axis=1, norm="l2") - run_time_data = time.time() - start_time - LOGGER.info("| loading data finsihed | time(s) {:9.4f}".format(run_time_data)) - - LOGGER.info("| loading model begin...") - start_time = time.time() - if args.selected_output is not None: - # Selected Output - selected_outputs_csr = XLinearModel.load_feature_matrix(args.selected_output) - xlinear_model = XLinearModel.load( - args.model_folder, is_predict_only=True, weight_matrix_type="CSC" - ) - else: - # TopK - selected_outputs_csr = None - xlinear_model = XLinearModel.load(args.model_folder, is_predict_only=True) - run_time_io = time.time() - start_time - LOGGER.info("| loading model finsihed | time(s) {:9.4f}".format(run_time_io)) - - - # Model Predicting - LOGGER.info("| inference model begin...") - start_time = time.time() - Yt_pred = xlinear_model.predict( - Xt, - selected_outputs_csr=selected_outputs_csr, - only_topk=args.only_topk, - beam_size=args.beam_size, - post_processor=args.post_processor, - threads=args.threads, - max_pred_chunk=args.max_pred_chunk, - ) - run_time_pred = time.time() - start_time - LOGGER.info("| inference model finsihed | time(s) {:9.4f} latency(ms/q) {:9.4f}".format( - run_time_pred, - run_time_pred / Xt.shape[0] * 1000, - ) - ) - - # Save prediction - if args.save_pred_path: - smat_util.save_matrix(args.save_pred_path, Yt_pred) - - # Evaluate - if args.label_path: - Yt = XLinearModel.load_label_matrix(args.label_path) - metric = smat_util.Metrics.generate(Yt, Yt_pred, topk=10) - print("==== evaluation results ====") - print(metric) - - -if __name__ == "__main__": - parser = parse_arguments() - args = parser.parse_args() - do_predict(args) diff --git a/examples/pecos-xrlinear-jmlr22/xrl_train.py b/examples/pecos-xrlinear-jmlr22/xrl_train.py deleted file mode 100644 index f73e993a..00000000 --- a/examples/pecos-xrlinear-jmlr22/xrl_train.py +++ /dev/null @@ -1,423 +0,0 @@ -# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance -# with the License. A copy of the License is located at -# -# http://aws.amazon.com/apache2.0/ -# -# or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES -# OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions -# and limitations under the License. -import argparse -import os -import sys -import json -import logging -import time -from pecos.core import XLINEAR_SOLVERS -from pecos.utils import cli -from pecos.utils import smat_util, logging_util -from pecos.utils.cluster_util import ClusterChain -from pecos.xmc import Indexer, LabelEmbeddingFactory, PostProcessor -from pecos.xmc.base import HierarchicalKMeans -from pecos.xmc.xlinear.model import XLinearModel -from sklearn.preprocessing import normalize - - -LOGGER = logging.getLogger() -LOGGER.setLevel(logging.DEBUG) - -handler = logging.StreamHandler(sys.stdout) -handler.setLevel(logging.DEBUG) -formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") -handler.setFormatter(formatter) -LOGGER.addHandler(handler) - - -def parse_arguments(): - """Parse training arguments""" - - parser = argparse.ArgumentParser() - - parser.add_argument( - "--generate-params-skeleton", - action="store_true", - help="generate template params-json to stdout", - ) - - skip_training = "--generate-params-skeleton" in sys.argv - # ========= parameter jsons ============ - parser.add_argument( - "--params-path", - type=str, - default=None, - metavar="PARAMS_PATH", - help="Json file for params (default None)", - ) - # ======= actual arguments ======== - - # Required parameters - parser.add_argument( - "-x", - "--inst-path", - type=str, - required=not skip_training, - metavar="PATH", - help="path to the CSR npz or Row-majored npy file of the feature matrix (nr_insts * nr_feats)", - ) - - parser.add_argument( - "-y", - "--label-path", - type=str, - required=not skip_training, - metavar="PATH", - help="path to the CSR npz file of the label matrix (nr_insts * nr_labels)", - ) - - parser.add_argument( - "-m", - "--model-folder", - type=str, - required=not skip_training, - metavar="DIR", - help="path to the model folder.", - ) - - # Optional - - # Indexing parameters - parser.add_argument( - "-f", - "--label-feat-path", - type=str, - default=None, - metavar="PATH", - help="path to the CSR npz or Row-majored npy file of the label feature matrix (nr_labels * nr_label_feats)", - ) - - parser.add_argument( - "--nr-splits", - type=int, - default=8, - metavar="INT", - help="number of splits used to construct hierarchy (a power of 2 is recommended)", - ) - - parser.add_argument( - "--max-leaf-size", - type=int, - default=100, - metavar="INT", - help="The max size of the leaf nodes of hierarchical 2-means clustering. If larger than total number of labels, One-Versus-All model will be trained. Default 100.", - ) - - parser.add_argument( - "--spherical", - type=cli.str2bool, - metavar="[true/false]", - default=True, - help="If true, do l2-normalize cluster centers while clustering. Default true.", - ) - - parser.add_argument( - "--seed", type=int, default=0, metavar="INT", help="random seed (default 0)" - ) - - parser.add_argument( - "--kmeans-max-iter", - type=int, - default=20, - metavar="INT", - help="max number of k-means iterations for indexer (default 20)", - ) - - parser.add_argument( - "-n", - "--threads", - type=int, - default=-1, - metavar="INT", - help="number of threads to use (default -1 to denote all the CPUs)", - ) - - parser.add_argument( - "-c", - "--code-path", - type=str, - default=None, - metavar="PATH", - help="path to the npz file of the code matrix (CSC, nr_labels * nr_codes)", - ) - - parser.add_argument( - "-r", - "--rel-path", - type=str, - default=None, - metavar="PATH", - help="path to the npz file of the relevance matrix (CSR, nr_insts * nr_labels). Should have same sparsity pattern as label matrix.", - ) - - parser.add_argument( - "--rel-norm", - type=str, - choices=["l1", "l2", "max", "no-norm"], - default="no-norm", - metavar="STR", - help="norm type to row-wise normalzie relevance matrix for cost-sensitive learning", - ) - - parser.add_argument( - "--rel-mode", - type=str, - metavar="STR", - default="disable", - help="mode to use relevance score for cost sensitive learning ['disable'(default), 'induce', 'ranker-only']", - ) - - parser.add_argument( - "-um", - "--usn-match-path", - type=str, - default=None, - metavar="PATH", - help="path to the npz file of the user supplied matching matrix (CSR, nr_insts * nr_codes), will be add to negative sampling if given", - ) - - parser.add_argument( - "-uy", - "--usn-label-path", - type=str, - default=None, - metavar="PATH", - help="path to the npz file of the user supplied label importance matrix (CSR, nr_insts * nr_labels), will be add to negative sampling if given", - ) - - # Linear matching/ranking parameters - parser.add_argument( - "-s", - "--solver-type", - type=str, - default="L2R_L2LOSS_SVC_DUAL", - metavar="STR", - help="{} (default L2R_L2LOSS_SVC_DUAL)".format(" | ".join(XLINEAR_SOLVERS.keys())), - ) - - parser.add_argument( - "--Cp", - type=float, - default=1.0, - metavar="VAL", - help="coefficient for positive class in the loss function (default 1.0)", - ) - - parser.add_argument( - "--Cn", - type=float, - default=1.0, - metavar="VAL", - help="coefficient for negative class in the loss function (default 1.0)", - ) - - parser.add_argument( - "--bias", type=float, default=1.0, metavar="VAL", help="bias term (default 1.0)" - ) - - parser.add_argument( - "-ns", - "--negative-sampling", - type=str, - choices=["tfn", "man", "tfn+man", "usn", "usn+tfn", "usn+man", "usn+tfn+man"], - default="tfn", - metavar="STR", - dest="neg_mining_chain", - help="Negative Sampling Schemes", - ) - - parser.add_argument( - "-t", - "--threshold", - type=float, - default=0.1, - metavar="VAL", - help="threshold to sparsify the model weights (default 0.1)", - ) - - parser.add_argument( - "-z", - "--max-nonzeros-per-label", - type=int, - default=0, - metavar="NONZEROS", - help="keep at most NONZEROS weight parameters per label in model(default 0 to denote nr_features + 1)", - ) - - # Prediction kwargs - parser.add_argument( - "-k", - "--only-topk", - type=int, - default=None, - metavar="INT", - help="the default number of top labels used in the prediction", - ) - - parser.add_argument( - "-b", - "--beam-size", - type=int, - default=None, - metavar="INT", - help="the default size of beam search used in the prediction", - ) - - parser.add_argument( - "-pp", - "--post-processor", - type=str, - choices=PostProcessor.valid_list(), - default=None, - metavar="STR", - help="the default post processor used in the prediction", - ) - - parser.add_argument( - "--verbose-level", - type=int, - choices=logging_util.log_levels.keys(), - default=1, - metavar="INT", - help=f"the verbose level, {', '.join([str(k) + ' for ' + logging.getLevelName(v) for k, v in logging_util.log_levels.items()])}. Default 1", - ) - - return parser - - -def do_train(args): - """Train and Save xr-linear model - - Args: - args (argparse.Namespace): Command line arguments parsed by `parser.parse_args()` - """ - params = dict() - if args.generate_params_skeleton: - params["train_params"] = XLinearModel.TrainParams.from_dict({}, recursive=True).to_dict() - params["pred_params"] = XLinearModel.PredParams.from_dict({}, recursive=True).to_dict() - params["indexer_params"] = HierarchicalKMeans.TrainParams.from_dict( - {}, recursive=True - ).to_dict() - print(f"{json.dumps(params, indent=True)}") - return - - if args.params_path: - with open(args.params_path, "r") as fin: - params = json.load(fin) - - train_params = params.get("train_params", None) - pred_params = params.get("pred_params", None) - indexer_params = params.get("indexer_params", None) - - if train_params is not None: - train_params = XLinearModel.TrainParams.from_dict(train_params) - else: - train_params = XLinearModel.TrainParams.from_dict( - {k: v for k, v in vars(args).items() if v is not None}, - recursive=True, - ) - - if pred_params is not None: - pred_params = XLinearModel.PredParams.from_dict(pred_params) - else: - pred_params = XLinearModel.PredParams.from_dict( - {k: v for k, v in vars(args).items() if v is not None}, - recursive=True, - ) - - if indexer_params is not None: - indexer_params = HierarchicalKMeans.TrainParams.from_dict(indexer_params) - else: - indexer_params = HierarchicalKMeans.TrainParams.from_dict( - {k: v for k, v in vars(args).items() if v is not None}, - recursive=True, - ) - if args.seed: - indexer_params.seed = args.seed - - if not os.path.exists(args.model_folder): - os.makedirs(args.model_folder) - - LOGGER.info("| loading data begin...") - start_time = time.time() - X = XLinearModel.load_feature_matrix(args.inst_path) - X = normalize(X, axis=1, norm="l2") - Y = XLinearModel.load_label_matrix(args.label_path, for_training=True) - run_time_io = time.time() - start_time - LOGGER.info("| loading data finsihed | time(s) {:9.4f}".format(run_time_io)) - - LOGGER.info("| building HLT...") - start_time = time.time() - if args.code_path: - cluster_chain = ClusterChain.load(args.code_path) - else: - if args.label_feat_path: - label_feat = XLinearModel.load_feature_matrix(args.label_feat_path) - else: - label_feat = LabelEmbeddingFactory.create(Y, X, method="pifa") - - cluster_chain = Indexer.gen(label_feat, train_params=indexer_params) - run_time_hlt = time.time() - start_time - LOGGER.info("| building HLT finsihed | time(s) {:9.4f}".format(run_time_hlt)) - - # load label importance matrix if given - if args.usn_label_path: - usn_label_mat = smat_util.load_matrix(args.usn_label_path) - else: - usn_label_mat = None - # load user supplied matching matrix if given - if args.usn_match_path: - usn_match_mat = smat_util.load_matrix(args.usn_match_path) - else: - usn_match_mat = None - usn_match_dict = {0: usn_label_mat, 1: usn_match_mat} - - # load relevance matrix for cost-sensitive learning - if args.rel_path: - R = smat_util.load_matrix(args.rel_path) - else: - R = None - - pred_kwargs = {} - for kw in ["beam_size", "only_topk", "post_processor"]: - if getattr(args, kw, None) is not None: - pred_kwargs[kw] = getattr(args, kw) - - LOGGER.info("| training XR-Linear...") - start_time = time.time() - xlm = XLinearModel.train( - X, - Y, - C=cluster_chain, - R=R, - user_supplied_negatives=usn_match_dict, - train_params=train_params, - pred_params=pred_params, - pred_kwargs=pred_kwargs, - ) - run_time_xrl = time.time() - start_time - LOGGER.info("| training XR_Linear finsihed | time(s) {:9.4f}".format(run_time_xrl)) - - xlm.save(args.model_folder) - LOGGER.info("| Finished with run_time(s) | total {:9.4f} hlt {:9.4f} xrl {:9.4f}".format( - run_time_hlt + run_time_xrl, - run_time_hlt, - run_time_xrl, - ) - ) - - -if __name__ == "__main__": - parser = parse_arguments() - args = parser.parse_args() - logging_util.setup_logging_config(level=args.verbose_level) - do_train(args) diff --git a/examples/pefa-wsdm24/README.md b/examples/pefa-wsdm24/README.md deleted file mode 100644 index 73050544..00000000 --- a/examples/pefa-wsdm24/README.md +++ /dev/null @@ -1,96 +0,0 @@ -# Experiment Code for PEFA, WSDM 2024 - -This folder contains code to reproduce experiments in -["PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models"](https://arxiv.org/abs/2312.02429) - -## 1. Summary -In this repository, we demonstrated how to reproduce Table 2 (NQ-320K) and Table 3 (Trivia-QA) of our PEFA paper. -After following Steps 2-6 in the subsequent sections, you should be able to obtain - -| NQ-320K | Recall@10 | Recall@100 | -|---|---|---| -| DistilBERT + PEFA-XS | 80.52% | 92.23% | -| DistilBERT + PEFA-XL | 85.26% | 92.53% | -| MPNet + PEFA-XS | 86.67% | 94.53% | -| MPNet + PEFA-XL | 88.72% | 95.13% | -| Sentence-T5-base + PEFA-XS | 82.52% | 92.18% | -| Sentence-T5-base + PEFA-XL | 83.69% | 92.55% | -| GTR-T5-base + PEFA-XS | 84.90% | 93.28% | -| GTR-T5-base + PEFA-XL | 88.71% | 94.36% | - -| Trivia-QA | Recall@20 | Recall@100 | -|---|---|---| -| DistilBERT + PEFA-XS | 86.28% | 93.33% | -| DistilBERT + PEFA-XL | 84.18% | 91.24% | -| MPNet + PEFA-XS | 86.05% | 92.97% | -| MPNet + PEFA-XL | 86.13% | 92.42% | -| Sentence-T5-base + PEFA-XS | 78.39% | 88.57% | -| Sentence-T5-base + PEFA-XL | 75.13% | 87.24% | -| GTR-T5-base + PEFA-XS | 83.81% | 91.02% | -| GTR-T5-base + PEFA-XL | 85.30% | 92.38% | - - -## 2. Getting Started -* Clone the repository and enter `examples/pefa-wsdm24` directory. -* First create a [virtual environment](https://docs.python.org/3/library/venv.html) and then install dependencies by running the following command: -```bash -python3 -m pip install libpecos==1.2.1 -python3 -m pip install sentence-transformers==2.2.1 -``` - -## 3. Download Pre-processed Data for NQ320K and Trivia-QA -Our pre-processed datasets of NQ320K and Trivia-QA can be download at -```bash -mkdir -p ./data/xmc; cd ./data/xmc; -DATASET="nq320k" # nq320k or trivia -wget https://archive.org/download/pefa-wsdm24/data/xmc/${DATASET}.tar.gz -tar -zxvf ./${DATASET}.tar.gz -cd ../../ # get back to the pecos/examples/pefa-wsdm24 directory -``` - -Additional Notes on data-preprocessing -* We first obtained original NQ320K/Trivia-QA datasets from the [NCI Paper, Wang et al., NeurIPS22](https://github.com/solidsea98/Neural-Corpus-Indexer-NCI) -- We then pre-processed it into our format. -- Details about our data pre-processing scripts can be found in `./data/README.md`. - - -## 4. Generate Embeddings for PEFA Inference -Before running PEFA inference, we select an encoder to generating query/passage embeddings -```bash -DATASET="nq320k" # nq320k or trivia -ENCODER="gtr-t5-base" -bash run_encoder ${DATASET} ${ENCODER} -``` -The embeddings will be saved to `./data/embeddings/${DATSET}/` -Regarding the `ENCODER` used in our paper, -* For `nq320k`, we consider `{nq-distilbert-base-v1, multi-qa-mpnet-base-dot-v1, sentence-t5-base, gtr-t5-base}` -* For `trivia`, we consider `{multi-qa-distilbert-dot-v1, multi-qa-mpnet-base-dot-v1, sentence-t5-base, gtr-t5-base}` - -## 5. Run PEFA-XS -```bash -DATASET="nq320k" -ENCODER="gtr-t5-base" -bash run_pefa_xs.sh ${DATASET} ${ENCODER} -``` -The script `run_pefa_xs.sh` calls the `pefa_xs.py` with hard-coded hyper-parameters. -For example, it uses `threads=64`. If your machine has less CPU cores, please adjust it accordingly. - -## 6. Run PEFA-XL -```bash -DATASET="nq320k" -ENCODER="gtr-t5-base" -bash run_pefa_xl.sh ${DATASET} ${ENCODER} -``` -The script `run_pefa_xl.sh` calls the `pefa_xl.py` with hard-coded hyper-parameters. -For example, it uses `threads=64`. If your machine has less CPU cores, please adjust it accordingly. - -## 7. Citation -If you find this work useful for your research, please cite: -``` -@inproceedings{chang2024pefa, - title={PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models}, - author={Wei-Cheng Chang and Jyun-Yu Jiang and Jiong Zhang and Mutasem Al-Darabsah and Choon Hui Teo and Cho-Jui Hsieh and Hsiang-Fu Yu and S. V. N. Vishwanathan}, - booktitle={Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM '24)}, - year={2024} -} -``` diff --git a/examples/pefa-wsdm24/data/README.md b/examples/pefa-wsdm24/data/README.md deleted file mode 100644 index 30fc29fd..00000000 --- a/examples/pefa-wsdm24/data/README.md +++ /dev/null @@ -1,34 +0,0 @@ - -# Download Raw Data from NCI Paper -- NQ320K: https://drive.google.com/drive/folders/1epfUw4yQjAtqnZTQDLAUOwTJg-YMCGdD -- Trivia: https://drive.google.com/drive/folders/1SY28Idba1X8DNi4PYaDDH9CbUpdKiTXQ -``` - unzip the NQ320K folder to ./raw/NQ320K_data - unzip the Trivia folder to ./raw/trivia_newdata -``` - -# Process the Raw Data to XMC Format -- NQ320K: -``` - python proc_nq320k.py -``` - -- Trivia: -``` - python proc_trivia.py -``` - -You should see the following data artifacts -``` -./xmc/{nq320k|trivia} -|- X.trn.abs.txt -|- X.trn.d2q.txt -|- X.trn.doc.txt -|- X.trn.txt -|- X.tst.txt -|- Y.trn.abs.npz -|- Y.trn.d2q.npz -|- Y.trn.doc.npz -|- Y.trn.npz -|- Y.tst.npz -``` diff --git a/examples/pefa-wsdm24/data/proc_nq320k.py b/examples/pefa-wsdm24/data/proc_nq320k.py deleted file mode 100644 index 54af6ed1..00000000 --- a/examples/pefa-wsdm24/data/proc_nq320k.py +++ /dev/null @@ -1,102 +0,0 @@ - -import os -from collections import defaultdict -import numpy as np -import pandas as pd -import scipy.sparse as smat -from pecos.utils import smat_util - - -COL_NAME_LIST = [ - "query", "qid", "doc_id", - "bert_k30_c30_1", "bert_k30_c30_2", "bert_k30_c30_3", "bert_k30_c30_4", "bert_k30_c30_5", -] - -def load_df(input_tsv_path): - return pd.read_csv( - input_tsv_path, - encoding='utf-8', header=None, sep='\t', - names=COL_NAME_LIST, - dtype={"query": str, "qid": str, 'doc_id': str} - ).loc[:, ["query", "qid", "doc_id"]] - -def build_did2lid_map(df_inp, did_to_lid): - for i in range(len(df_inp)): - did_str = df_inp["doc_id"][i] - if did_str not in did_to_lid: - did_to_lid[did_str] = len(did_to_lid) - -def build_corpus_and_label_mat(df, did_to_lid, skip_same_lid=False): - qry_to_qid = defaultdict(str) - rows, cols = [], [] - inc_lid_set = set() - for i in range(len(df)): - query = df["query"][i] - did_str = df["doc_id"][i] - - lid = did_to_lid[did_str] - if skip_same_lid and lid in inc_lid_set: - continue - inc_lid_set.add(lid) - - if query not in qry_to_qid: - qry_to_qid[query] = len(qry_to_qid) - qid = qry_to_qid[query] - rows.append(qid) - cols.append(lid) - - vals = [1.0 for _ in range(len(rows))] - - num_inp, num_out = len(qry_to_qid), len(did_to_lid) - Y = smat.csr_matrix( - (vals, (rows, cols)), - shape=(num_inp, num_out), - dtype=np.float32, - ) - print("#Q {:7d} #L {:7d} NNZ {:9d}".format(num_inp, num_out, Y.nnz)) - id2query = [str(query).lower() for query, qid in sorted(qry_to_qid.items(), key=lambda x: x[1])] - return id2query, Y - - -def write_qtxt(id2qtxt, output_path): - with open(output_path, "w") as fout: - for query_txt in id2qtxt: - fout.write(f"{query_txt}\n") - -def main(): - df_trn = load_df("./raw/NQ_dataset/nq_train_doc_newid.tsv") - df_tst = load_df("./raw/NQ_dataset/nq_dev_doc_newid.tsv") - df_abs = load_df("./raw/NQ_dataset/nq_title_abs.tsv") - df_doc = load_df("./raw/NQ_dataset/NQ_doc_aug.tsv") - df_d2q = load_df("./raw/NQ_dataset/NQ_512_qg.tsv") - - did_to_lid = defaultdict(str) - build_did2lid_map(df_abs, did_to_lid) - print("After df_abs, #uniq_label {:9d}".format(len(did_to_lid))) - build_did2lid_map(df_doc, did_to_lid) - print("After df_doc, #uniq_label {:9d}".format(len(did_to_lid))) - build_did2lid_map(df_d2q, did_to_lid) - print("After df_d2q, #uniq_label {:9d}".format(len(did_to_lid))) - - id2qtxt_trn, Y_trn_all = build_corpus_and_label_mat(df_trn, did_to_lid, skip_same_lid=False) - id2qtxt_tst, Y_tst_all = build_corpus_and_label_mat(df_tst, did_to_lid, skip_same_lid=False) - id2qtxt_abs, Y_trn_abs = build_corpus_and_label_mat(df_abs, did_to_lid, skip_same_lid=True) - id2qtxt_doc, Y_trn_doc = build_corpus_and_label_mat(df_doc, did_to_lid, skip_same_lid=False) - id2qtxt_d2q, Y_trn_d2q = build_corpus_and_label_mat(df_d2q, did_to_lid, skip_same_lid=False) - - output_dir = "./xmc/nq320k" - os.makedirs(output_dir, exist_ok=True) - write_qtxt(id2qtxt_trn, f"{output_dir}/X.trn.txt") - write_qtxt(id2qtxt_tst, f"{output_dir}/X.tst.txt") - write_qtxt(id2qtxt_abs, f"{output_dir}/X.trn.abs.txt") - write_qtxt(id2qtxt_doc, f"{output_dir}/X.trn.doc.txt") - write_qtxt(id2qtxt_d2q, f"{output_dir}/X.trn.d2q.txt") - smat_util.save_matrix(f"{output_dir}/Y.trn.npz", Y_trn_all) - smat_util.save_matrix(f"{output_dir}/Y.tst.npz", Y_tst_all) - smat_util.save_matrix(f"{output_dir}/Y.trn.abs.npz", Y_trn_abs) - smat_util.save_matrix(f"{output_dir}/Y.trn.doc.npz", Y_trn_doc) - smat_util.save_matrix(f"{output_dir}/Y.trn.d2q.npz", Y_trn_d2q) - - -if __name__ == "__main__": - main() diff --git a/examples/pefa-wsdm24/data/proc_trivia.py b/examples/pefa-wsdm24/data/proc_trivia.py deleted file mode 100644 index ad80f6d0..00000000 --- a/examples/pefa-wsdm24/data/proc_trivia.py +++ /dev/null @@ -1,134 +0,0 @@ - -import os -from collections import defaultdict -import numpy as np -import pandas as pd -import scipy.sparse as smat -from pecos.utils import smat_util - - -COL_NAME_LIST = [ - "query", "qid", "doc_id", - "bert_k30_c30_1", "bert_k30_c30_2", "bert_k30_c30_3", "bert_k30_c30_4", "bert_k30_c30_5", -] - -QG_COL_NAME_LIST = [ - "query", "doc_id", - "bert_k30_c30_1", "bert_k30_c30_2", "bert_k30_c30_3", "bert_k30_c30_4", "bert_k30_c30_5", -] - - -def load_df(input_tsv_path): - return pd.read_csv( - input_tsv_path, - encoding='utf-8', header=None, sep='\t', - names=COL_NAME_LIST, - dtype={"query": str, "qid": str, 'doc_id': str}, - on_bad_lines="skip", - skip_blank_lines=True, - ).loc[:, ["query", "qid", "doc_id"]] - - -def load_qg_df(input_tsv_path): - return pd.read_csv( - input_tsv_path, - encoding='utf-8', header=None, sep='\t', - names=QG_COL_NAME_LIST, - dtype={"query": str, 'doc_id': str}, - on_bad_lines="skip", - skip_blank_lines=True, - ).loc[:, ["query", "doc_id"]] - - -def build_did2lid_map(df_inp, did_to_lid): - for i in range(len(df_inp)): - #did_str = df_inp["doc_id"][i] - try: - did_list = df_inp["doc_id"][i].split(",") - except: - print(i, type(df_inp["doc_id"][i]), df_inp["doc_id"][i]) - exit(0) - for did_str in did_list: - if did_str not in did_to_lid: - did_to_lid[did_str] = len(did_to_lid) - -def build_corpus_and_label_mat(df, did_to_lid, skip_same_lid=False): - qry_to_qid = defaultdict(str) - rows, cols = [], [] - inc_lid_set = set() - for i in range(len(df)): - query = df["query"][i] - try: - did_list = df["doc_id"][i].split(",") - except: - print(i, type(df["doc_id"][i]), df["doc_id"][i]) - exit(0) - for did_str in did_list: - lid = did_to_lid[did_str] - if skip_same_lid and lid in inc_lid_set: - continue - inc_lid_set.add(lid) - - if query not in qry_to_qid: - qry_to_qid[query] = len(qry_to_qid) - qid = qry_to_qid[query] - rows.append(qid) - cols.append(lid) - - vals = [1.0 for _ in range(len(rows))] - - num_inp, num_out = len(qry_to_qid), len(did_to_lid) - Y = smat.csr_matrix( - (vals, (rows, cols)), - shape=(num_inp, num_out), - dtype=np.float32, - ) - print("#Q {:7d} #L {:7d} NNZ {:9d}".format(num_inp, num_out, Y.nnz)) - id2query = [str(query).lower() for query, qid in sorted(qry_to_qid.items(), key=lambda x: x[1])] - return id2query, Y - - -def write_qtxt(id2qtxt, output_path): - with open(output_path, "w") as fout: - for query_txt in id2qtxt: - query_txt = query_txt.replace('\r\n', ' ') - query_txt = query_txt.replace('\n', ' ') - fout.write(f"{query_txt}\n") - -def main(): - df_trn = load_df("./raw/trivia_newdata/train.tsv") - df_tst = load_df("./raw/trivia_newdata/dev.tsv") - df_abs = load_df("./raw/trivia_newdata/trivia_title_cont.tsv") - df_doc = load_df("./raw/trivia_newdata/trivia_doc_aug.tsv") - df_d2q = load_qg_df("./raw/trivia_newdata/trivia_512_qg.tsv") - - did_to_lid = defaultdict(str) - build_did2lid_map(df_abs, did_to_lid) - print("After df_abs, #uniq_label {:9d}".format(len(did_to_lid))) - build_did2lid_map(df_doc, did_to_lid) - print("After df_doc, #uniq_label {:9d}".format(len(did_to_lid))) - build_did2lid_map(df_d2q, did_to_lid) - print("After df_d2q, #uniq_label {:9d}".format(len(did_to_lid))) - - id2qtxt_trn, Y_trn_all = build_corpus_and_label_mat(df_trn, did_to_lid, skip_same_lid=False) - id2qtxt_tst, Y_tst_all = build_corpus_and_label_mat(df_tst, did_to_lid, skip_same_lid=False) - id2qtxt_abs, Y_trn_abs = build_corpus_and_label_mat(df_abs, did_to_lid, skip_same_lid=True) - id2qtxt_doc, Y_trn_doc = build_corpus_and_label_mat(df_doc, did_to_lid, skip_same_lid=False) - id2qtxt_d2q, Y_trn_d2q = build_corpus_and_label_mat(df_d2q, did_to_lid, skip_same_lid=False) - - output_dir = "./xmc/trivia" - os.makedirs(output_dir, exist_ok=True) - write_qtxt(id2qtxt_trn, f"{output_dir}/X.trn.txt") - write_qtxt(id2qtxt_tst, f"{output_dir}/X.tst.txt") - write_qtxt(id2qtxt_abs, f"{output_dir}/X.trn.abs.txt") - write_qtxt(id2qtxt_doc, f"{output_dir}/X.trn.doc.txt") - write_qtxt(id2qtxt_d2q, f"{output_dir}/X.trn.d2q.txt") - smat_util.save_matrix(f"{output_dir}/Y.trn.npz", Y_trn_all) - smat_util.save_matrix(f"{output_dir}/Y.tst.npz", Y_tst_all) - smat_util.save_matrix(f"{output_dir}/Y.trn.abs.npz", Y_trn_abs) - smat_util.save_matrix(f"{output_dir}/Y.trn.doc.npz", Y_trn_doc) - smat_util.save_matrix(f"{output_dir}/Y.trn.d2q.npz", Y_trn_d2q) - - -if __name__ == "__main__": - main() diff --git a/examples/pefa-wsdm24/encoder.py b/examples/pefa-wsdm24/encoder.py deleted file mode 100644 index a5d9f50a..00000000 --- a/examples/pefa-wsdm24/encoder.py +++ /dev/null @@ -1,46 +0,0 @@ - -import sys -from pecos.utils import smat_util -from sentence_transformers import SentenceTransformer, LoggingHandler -import logging - -logging.basicConfig(format='%(asctime)s - %(message)s', - datefmt='%Y-%m-%d %H:%M:%S', - level=logging.INFO, - handlers=[LoggingHandler()]) - - -def main(model_name_or_path, input_txt_path, output_emb_path): - # load data - corpus = [line.strip() for line in open(input_txt_path, "r")] - logging.info("|corpus| = {}".format(len(corpus))) - - # load encoder - # e.g., model_name_or_path = "sentence-transformers/nq-distilbert-base-v1" - model = SentenceTransformer(model_name_or_path) - logging.info("model_name_or_path {}".format(model_name_or_path)) - - # Start the multi-process pool on all available CUDA devices - # https://github.com/UKPLab/sentence-transformers/blob/master/examples/applications/computing-embeddings/computing_embeddings_mutli_gpu.py - mp_pool = model.start_multi_process_pool() - - # encoding - #emb_arr = model.encode(corpus, show_progress_bar=True) - emb_arr = model.encode_multi_process(corpus, mp_pool, batch_size=256) - logging.info("emb_arr {}".format(emb_arr.shape)) - - # saving output as npy - smat_util.save_matrix(output_emb_path, emb_arr) - - #Optional: Stop the proccesses in the pool - model.stop_multi_process_pool(mp_pool) - - -if __name__ == "__main__": - if len(sys.argv) != 4: - print("python encoder.py [model_name_or_path] [input_txt_path] [output_emb_path]") - exit(0) - model_name_or_path = sys.argv[1] - input_txt_path = sys.argv[2] - output_emb_path = sys.argv[3] - main(model_name_or_path, input_txt_path, output_emb_path) diff --git a/examples/pefa-wsdm24/pefa_xl.py b/examples/pefa-wsdm24/pefa_xl.py deleted file mode 100644 index d71063da..00000000 --- a/examples/pefa-wsdm24/pefa_xl.py +++ /dev/null @@ -1,110 +0,0 @@ -import logging -import sys -import random -import numpy as np -from pecos.ann.hnsw import HNSW -from pecos.utils import smat_util -from pecos.xmc import LabelEmbeddingFactory -from sentence_transformers import LoggingHandler -from sklearn.preprocessing import normalize - -from utils import get_data_aug, get_eval_metric - - -logging.basicConfig(format='%(asctime)s - %(message)s', - datefmt='%Y-%m-%d %H:%M:%S', - level=logging.INFO, - handlers=[LoggingHandler()]) - - -# FIXED HYPER-PARAMETERS -threads = 64 -M, efC = 32, 500 -efS, topk_p = 300, 500 -metric_type = "ip" - - -def run_pefa_xl( - P_emb, Q_emb_trn, Y_trn_npz, Q_emb_tst, Y_tst_npz, - topk_q=32, lambda_erm=0.5, knn_type="v0", -): - logging.info("Building HNSW Index for f_erm") - train_params = HNSW.TrainParams(M=M, efC=efC, metric_type=metric_type, threads=threads) - index_P = HNSW.train(P_emb, train_params=train_params, pred_params=None) - pred_params = HNSW.PredParams(efS=efS, topk=topk_p) - searchers = index_P.searchers_create(num_searcher=threads) - Yp_erm = index_P.predict(Q_emb_tst, pred_params=pred_params, searchers=searchers, ret_csr=True) - Yp_erm.data = 1.0 - Yp_erm.data - - logging.info("Building HSNW Index for f_knn") - Yp_tst = None - if lambda_erm >= 0.0 and lambda_erm < 1.0: - train_params = HNSW.TrainParams(M=M, efC=efC, metric_type=metric_type, threads=threads) - index_Q = HNSW.train(Q_emb_trn, train_params=train_params, pred_params=None) - pred_params_q = HNSW.PredParams(efS=efS, topk=topk_q) - searchers = index_Q.searchers_create(num_searcher=threads) - qQT = index_Q.predict(Q_emb_tst, pred_params=pred_params_q, searchers=searchers, ret_csr=True) - if knn_type == "v0": - qQT.data = (1.0 - qQT.data) / float(topk_q) # normalizing - elif knn_type == "v1": - qQT.data = (1.0 - qQT.data) # no normalizing - else: - raise ValueError(f"knn_type={knn_type} is not valid!") - Yp_knn = qQT.dot(Y_trn_npz) - Yp_tst = lambda_erm * Yp_erm + (1.0 - lambda_erm) * Yp_knn - elif lambda_erm == 1.0: - Yp_tst = Yp_erm - else: - raise ValueError(f"lambda_erm={lambda_erm} should be in [0.0, 1.0]!") - Yp_tst = smat_util.sorted_csr(Yp_tst) - - # eval recall - eval_recall_str = get_eval_metric(Y_tst_npz, Yp_tst, topk=100) - logging.info("topk_q {:3d} lambda_erm {:.2f} knn_type {} | {}".format(topk_q, lambda_erm, knn_type, eval_recall_str)) - - -def main(input_xmc_dir, input_emb_dir, lambda_erm, topk_q, knn_type): - seed = 1234 - random.seed(seed) - np.random.seed(seed) - - logging.info("Loading input-to-label matrix..") - Y_trn = smat_util.load_matrix(f"{input_xmc_dir}/Y.trn.npz") - Y_tst = smat_util.load_matrix(f"{input_xmc_dir}/Y.tst.npz") - Y_abs = smat_util.load_matrix(f"{input_xmc_dir}/Y.trn.abs.npz") - Y_doc = smat_util.load_matrix(f"{input_xmc_dir}/Y.trn.doc.npz") - Y_d2q = smat_util.load_matrix(f"{input_xmc_dir}/Y.trn.d2q.npz") - - logging.info("Loading input embedding matrix..") - X_trn = smat_util.load_matrix(f"{input_emb_dir}/X.trn.npy") # trn set emb from real query text - X_tst = smat_util.load_matrix(f"{input_emb_dir}/X.tst.npy") # tst set emb from real query text - X_abs = smat_util.load_matrix(f"{input_emb_dir}/X.trn.abs.npy") # trn set emb from doc's abstract+title text - X_doc = smat_util.load_matrix(f"{input_emb_dir}/X.trn.doc.npy") # trn set emb from doc's content (first 512 tokens) - X_d2q = smat_util.load_matrix(f"{input_emb_dir}/X.trn.d2q.npy") # trn set emb from docT5query using doc's content - X_trn = normalize(X_trn, axis=1, norm="l2") - X_tst = normalize(X_tst, axis=1, norm="l2") - X_abs = normalize(X_abs, axis=1, norm="l2") - X_doc = normalize(X_doc, axis=1, norm="l2") - X_d2q = normalize(X_d2q, axis=1, norm="l2") - - logging.info("Gathering data augmentation..") - P_emb = LabelEmbeddingFactory.create(Y_abs, X_abs, method="pifa", normalized_Y=False) - X_aug, Y_aug = get_data_aug(X_trn, X_doc, X_d2q, Y_trn, Y_doc, Y_d2q, aug_type="v5") - - logging.info("Running PEFA-XL..") - run_pefa_xl( - P_emb, X_aug, Y_aug, X_tst, Y_tst, - topk_q=topk_q, lambda_erm=lambda_erm, knn_type=knn_type, - ) - - -if __name__ == "__main__": - if len(sys.argv) != 6: - print("python pefa_xl.py [input_xmc_dir] [input_emb_dir] [lambda_erm] [topk_q] [knn_type]") - exit(0) - input_xmc_dir = sys.argv[1] - input_emb_dir = sys.argv[2] - lambda_erm = float(sys.argv[3]) - topk_q = int(sys.argv[4]) - knn_type = str(sys.argv[5]) - main(input_xmc_dir, input_emb_dir, lambda_erm, topk_q, knn_type) diff --git a/examples/pefa-wsdm24/pefa_xs.py b/examples/pefa-wsdm24/pefa_xs.py deleted file mode 100644 index 0c7b2ef7..00000000 --- a/examples/pefa-wsdm24/pefa_xs.py +++ /dev/null @@ -1,101 +0,0 @@ - -import logging -import sys -import random -import numpy as np -from pecos.ann.hnsw import HNSW -from pecos.utils import smat_util -from pecos.xmc import LabelEmbeddingFactory -from sentence_transformers import LoggingHandler -from sklearn.preprocessing import normalize - -from utils import get_data_aug, get_eval_metric - - -logging.basicConfig(format='%(asctime)s - %(message)s', - datefmt='%Y-%m-%d %H:%M:%S', - level=logging.INFO, - handlers=[LoggingHandler()]) - - -# FIXED HYPER-PARAMETERS -threads = 64 -M, efC = 32, 500 -efS, topk = 300, 100 -metric_type = "ip" - - -def run_pefa_xs( - P_emb_npy, - Q_emb_trn, Y_trn_npz, - Q_emb_tst, Y_tst_npz, - lambda_erm=0.5, -): - if lambda_erm >= 0.0 and lambda_erm < 1.0: - pifa_emb = LabelEmbeddingFactory.create( - Y_trn_npz, - Q_emb_trn, - method="pifa", - normalized_Y=False, - ) - label_emb = lambda_erm * P_emb_npy + (1.0 - lambda_erm) * pifa_emb - elif lambda_erm == 1.0: - label_emb = P_emb_npy - else: - raise ValueError(f"lambda_erm={lambda_erm} should be in [0.0, 1.0]!") - - # build ANN index - train_params = HNSW.TrainParams(M=M, efC=efC, metric_type=metric_type, threads=threads) - index_P = HNSW.train(label_emb, train_params=train_params, pred_params=None) - - # inference - pred_params = HNSW.PredParams(efS=efS, topk=topk) - searchers = index_P.searchers_create(num_searcher=threads) - Yp_tst = index_P.predict(Q_emb_tst, pred_params=pred_params, searchers=searchers, ret_csr=True) - Yp_tst.data = 1.0 - Yp_tst.data - - # eval recall - eval_recall_str = get_eval_metric(Y_tst_npz, Yp_tst, topk=100) - logging.info("lambda_erm {:.2f} | {}".format(lambda_erm, eval_recall_str)) - - -def main(input_xmc_dir, input_emb_dir, lambda_erm): - seed = 1234 - random.seed(seed) - np.random.seed(seed) - - logging.info("Loading input-to-label matrix..") - Y_trn = smat_util.load_matrix(f"{input_xmc_dir}/Y.trn.npz") - Y_tst = smat_util.load_matrix(f"{input_xmc_dir}/Y.tst.npz") - Y_abs = smat_util.load_matrix(f"{input_xmc_dir}/Y.trn.abs.npz") - Y_doc = smat_util.load_matrix(f"{input_xmc_dir}/Y.trn.doc.npz") - Y_d2q = smat_util.load_matrix(f"{input_xmc_dir}/Y.trn.d2q.npz") - - logging.info("Loading input embedding matrix..") - X_trn = smat_util.load_matrix(f"{input_emb_dir}/X.trn.npy") # trn set emb from real query text - X_tst = smat_util.load_matrix(f"{input_emb_dir}/X.tst.npy") # tst set emb from real query text - X_abs = smat_util.load_matrix(f"{input_emb_dir}/X.trn.abs.npy") # trn set emb from doc's abstract+title text - X_doc = smat_util.load_matrix(f"{input_emb_dir}/X.trn.doc.npy") # trn set emb from doc's content (first 512 tokens) - X_d2q = smat_util.load_matrix(f"{input_emb_dir}/X.trn.d2q.npy") # trn set emb from docT5query using doc's content - X_trn = normalize(X_trn, axis=1, norm="l2") - X_tst = normalize(X_tst, axis=1, norm="l2") - X_abs = normalize(X_abs, axis=1, norm="l2") - X_doc = normalize(X_doc, axis=1, norm="l2") - X_d2q = normalize(X_d2q, axis=1, norm="l2") - - logging.info("Gathering data augmentation..") - P_emb = LabelEmbeddingFactory.create(Y_abs, X_abs, method="pifa", normalized_Y=False) - X_aug, Y_aug = get_data_aug(X_trn, X_doc, X_d2q, Y_trn, Y_doc, Y_d2q, aug_type="v5") - - logging.info("Running PEFA-XS..") - run_pefa_xs(P_emb, X_aug, Y_aug, X_tst, Y_tst, lambda_erm=lambda_erm) - - -if __name__ == "__main__": - if len(sys.argv) != 4: - print("python pefa_xs.py [input_xmc_dir] [input_emb_dir] [lambda_erm]") - exit(0) - input_xmc_dir = sys.argv[1] - input_emb_dir = sys.argv[2] - lambda_erm = float(sys.argv[3]) - main(input_xmc_dir, input_emb_dir, lambda_erm) diff --git a/examples/pefa-wsdm24/results/nq320k.pefa_xl.gtr-t5-base.log b/examples/pefa-wsdm24/results/nq320k.pefa_xl.gtr-t5-base.log deleted file mode 100644 index 8c31c72a..00000000 --- a/examples/pefa-wsdm24/results/nq320k.pefa_xl.gtr-t5-base.log +++ /dev/null @@ -1,7 +0,0 @@ -2023-12-04 07:15:53 - Loading input-to-label matrix.. -2023-12-04 07:15:53 - Loading input embedding matrix.. -2023-12-04 07:16:04 - Gathering data augmentation.. -2023-12-04 07:16:08 - Running PEFA-XL.. -2023-12-04 07:16:08 - Building HNSW Index for f_erm -2023-12-04 07:16:43 - Building HSNW Index for f_knn -2023-12-04 07:42:34 - topk_q 16 lambda_erm 0.10 knn_type v0 | R@10,20,100 0.8871 0.9128 0.9436 diff --git a/examples/pefa-wsdm24/results/nq320k.pefa_xl.multi-qa-mpnet-base-dot-v1.log b/examples/pefa-wsdm24/results/nq320k.pefa_xl.multi-qa-mpnet-base-dot-v1.log deleted file mode 100644 index dbef56dc..00000000 --- a/examples/pefa-wsdm24/results/nq320k.pefa_xl.multi-qa-mpnet-base-dot-v1.log +++ /dev/null @@ -1,7 +0,0 @@ -2023-12-04 18:50:20 - Loading input-to-label matrix.. -2023-12-04 18:50:20 - Loading input embedding matrix.. -2023-12-04 18:50:30 - Gathering data augmentation.. -2023-12-04 18:50:34 - Running PEFA-XL.. -2023-12-04 18:50:34 - Building HNSW Index for f_erm -2023-12-04 18:51:09 - Building HSNW Index for f_knn -2023-12-04 19:16:03 - topk_q 16 lambda_erm 0.10 knn_type v0 | R@10,20,100 0.8872 0.9184 0.9513 diff --git a/examples/pefa-wsdm24/results/nq320k.pefa_xl.nq-distilbert-base-v1.log b/examples/pefa-wsdm24/results/nq320k.pefa_xl.nq-distilbert-base-v1.log deleted file mode 100644 index b732208f..00000000 --- a/examples/pefa-wsdm24/results/nq320k.pefa_xl.nq-distilbert-base-v1.log +++ /dev/null @@ -1,7 +0,0 @@ -2023-12-04 06:50:27 - Loading input-to-label matrix.. -2023-12-04 06:50:27 - Loading input embedding matrix.. -2023-12-04 06:50:38 - Gathering data augmentation.. -2023-12-04 06:50:42 - Running PEFA-XL.. -2023-12-04 06:50:42 - Building HNSW Index for f_erm -2023-12-04 06:51:14 - Building HSNW Index for f_knn -2023-12-04 07:15:49 - topk_q 16 lambda_erm 0.10 knn_type v0 | R@10,20,100 0.8526 0.8817 0.9253 diff --git a/examples/pefa-wsdm24/results/nq320k.pefa_xl.sentence-t5-base.log b/examples/pefa-wsdm24/results/nq320k.pefa_xl.sentence-t5-base.log deleted file mode 100644 index 81922343..00000000 --- a/examples/pefa-wsdm24/results/nq320k.pefa_xl.sentence-t5-base.log +++ /dev/null @@ -1,7 +0,0 @@ -2023-12-04 19:16:06 - Loading input-to-label matrix.. -2023-12-04 19:16:06 - Loading input embedding matrix.. -2023-12-04 19:16:16 - Gathering data augmentation.. -2023-12-04 19:16:20 - Running PEFA-XL.. -2023-12-04 19:16:20 - Building HNSW Index for f_erm -2023-12-04 19:16:46 - Building HSNW Index for f_knn -2023-12-04 19:36:54 - topk_q 16 lambda_erm 0.10 knn_type v0 | R@10,20,100 0.8369 0.8820 0.9255 diff --git a/examples/pefa-wsdm24/results/nq320k.pefa_xs.gtr-t5-base.log b/examples/pefa-wsdm24/results/nq320k.pefa_xs.gtr-t5-base.log deleted file mode 100644 index 85c5d508..00000000 --- a/examples/pefa-wsdm24/results/nq320k.pefa_xs.gtr-t5-base.log +++ /dev/null @@ -1,5 +0,0 @@ -2023-12-04 05:33:52 - Loading input-to-label matrix.. -2023-12-04 05:33:52 - Loading input embedding matrix.. -2023-12-04 05:34:02 - Gathering data augmentation.. -2023-12-04 05:34:06 - Running PEFA-XS.. -2023-12-04 05:34:40 - lambda_erm 0.50 | R@10,20,100 0.8490 0.8851 0.9328 diff --git a/examples/pefa-wsdm24/results/nq320k.pefa_xs.multi-qa-mpnet-base-dot-v1.log b/examples/pefa-wsdm24/results/nq320k.pefa_xs.multi-qa-mpnet-base-dot-v1.log deleted file mode 100644 index ece362b8..00000000 --- a/examples/pefa-wsdm24/results/nq320k.pefa_xs.multi-qa-mpnet-base-dot-v1.log +++ /dev/null @@ -1,5 +0,0 @@ -2023-12-04 18:38:53 - Loading input-to-label matrix.. -2023-12-04 18:38:53 - Loading input embedding matrix.. -2023-12-04 18:39:35 - Gathering data augmentation.. -2023-12-04 18:39:41 - Running PEFA-XS.. -2023-12-04 18:40:15 - lambda_erm 0.50 | R@10,20,100 0.8667 0.8995 0.9453 diff --git a/examples/pefa-wsdm24/results/nq320k.pefa_xs.nq-distilbert-base-v1.log b/examples/pefa-wsdm24/results/nq320k.pefa_xs.nq-distilbert-base-v1.log deleted file mode 100644 index 7679a57d..00000000 --- a/examples/pefa-wsdm24/results/nq320k.pefa_xs.nq-distilbert-base-v1.log +++ /dev/null @@ -1,5 +0,0 @@ -2023-12-04 05:38:14 - Loading input-to-label matrix.. -2023-12-04 05:38:14 - Loading input embedding matrix.. -2023-12-04 05:38:24 - Gathering data augmentation.. -2023-12-04 05:38:28 - Running PEFA-XS.. -2023-12-04 05:39:05 - lambda_erm 0.50 | R@10,20,100 0.8052 0.8516 0.9223 diff --git a/examples/pefa-wsdm24/results/nq320k.pefa_xs.sentence-t5-base.log b/examples/pefa-wsdm24/results/nq320k.pefa_xs.sentence-t5-base.log deleted file mode 100644 index 3ac8cb5e..00000000 --- a/examples/pefa-wsdm24/results/nq320k.pefa_xs.sentence-t5-base.log +++ /dev/null @@ -1,5 +0,0 @@ -2023-12-04 18:41:37 - Loading input-to-label matrix.. -2023-12-04 18:41:37 - Loading input embedding matrix.. -2023-12-04 18:42:19 - Gathering data augmentation.. -2023-12-04 18:42:26 - Running PEFA-XS.. -2023-12-04 18:42:52 - lambda_erm 0.50 | R@10,20,100 0.8252 0.8673 0.9218 diff --git a/examples/pefa-wsdm24/results/trivia.pefa_xl.gtr-t5-base.log b/examples/pefa-wsdm24/results/trivia.pefa_xl.gtr-t5-base.log deleted file mode 100644 index a38e0680..00000000 --- a/examples/pefa-wsdm24/results/trivia.pefa_xl.gtr-t5-base.log +++ /dev/null @@ -1,7 +0,0 @@ -2023-12-05 01:04:04 - Loading input-to-label matrix.. -2023-12-05 01:04:04 - Loading input embedding matrix.. -2023-12-05 01:04:26 - Gathering data augmentation.. -2023-12-05 01:04:29 - Running PEFA-XL.. -2023-12-05 01:04:29 - Building HNSW Index for f_erm -2023-12-05 01:04:49 - Building HSNW Index for f_knn -2023-12-05 01:18:44 - topk_q 64 lambda_erm 0.50 knn_type v1 | R@10,20,100 0.7940 0.8530 0.9238 diff --git a/examples/pefa-wsdm24/results/trivia.pefa_xl.multi-qa-distilbert-dot-v1.log b/examples/pefa-wsdm24/results/trivia.pefa_xl.multi-qa-distilbert-dot-v1.log deleted file mode 100644 index 01d54788..00000000 --- a/examples/pefa-wsdm24/results/trivia.pefa_xl.multi-qa-distilbert-dot-v1.log +++ /dev/null @@ -1,7 +0,0 @@ -2023-12-04 23:40:47 - Loading input-to-label matrix.. -2023-12-04 23:40:47 - Loading input embedding matrix.. -2023-12-04 23:41:09 - Gathering data augmentation.. -2023-12-04 23:41:12 - Running PEFA-XL.. -2023-12-04 23:41:12 - Building HNSW Index for f_erm -2023-12-04 23:41:31 - Building HSNW Index for f_knn -2023-12-04 23:54:44 - topk_q 64 lambda_erm 0.50 knn_type v1 | R@10,20,100 0.7778 0.8418 0.9124 diff --git a/examples/pefa-wsdm24/results/trivia.pefa_xl.multi-qa-mpnet-base-dot-v1.log b/examples/pefa-wsdm24/results/trivia.pefa_xl.multi-qa-mpnet-base-dot-v1.log deleted file mode 100644 index 91549944..00000000 --- a/examples/pefa-wsdm24/results/trivia.pefa_xl.multi-qa-mpnet-base-dot-v1.log +++ /dev/null @@ -1,7 +0,0 @@ -2023-12-05 00:39:43 - Loading input-to-label matrix.. -2023-12-05 00:39:43 - Loading input embedding matrix.. -2023-12-05 00:39:48 - Gathering data augmentation.. -2023-12-05 00:39:50 - Running PEFA-XL.. -2023-12-05 00:39:50 - Building HNSW Index for f_erm -2023-12-05 00:40:10 - Building HSNW Index for f_knn -2023-12-05 00:51:57 - topk_q 64 lambda_erm 0.90 knn_type v1 | R@10,20,100 0.8055 0.8613 0.9242 diff --git a/examples/pefa-wsdm24/results/trivia.pefa_xl.sentence-t5-base.log b/examples/pefa-wsdm24/results/trivia.pefa_xl.sentence-t5-base.log deleted file mode 100644 index f27d9de7..00000000 --- a/examples/pefa-wsdm24/results/trivia.pefa_xl.sentence-t5-base.log +++ /dev/null @@ -1,7 +0,0 @@ -2023-12-05 00:47:39 - Loading input-to-label matrix.. -2023-12-05 00:47:39 - Loading input embedding matrix.. -2023-12-05 00:47:44 - Gathering data augmentation.. -2023-12-05 00:47:46 - Running PEFA-XL.. -2023-12-05 00:47:46 - Building HNSW Index for f_erm -2023-12-05 00:48:00 - Building HSNW Index for f_knn -2023-12-05 00:58:36 - topk_q 64 lambda_erm 0.70 knn_type v1 | R@10,20,100 0.6763 0.7513 0.8724 diff --git a/examples/pefa-wsdm24/results/trivia.pefa_xs.gtr-t5-base.log b/examples/pefa-wsdm24/results/trivia.pefa_xs.gtr-t5-base.log deleted file mode 100644 index 962aeeda..00000000 --- a/examples/pefa-wsdm24/results/trivia.pefa_xs.gtr-t5-base.log +++ /dev/null @@ -1,5 +0,0 @@ -2023-12-04 06:00:03 - Loading input-to-label matrix.. -2023-12-04 06:00:03 - Loading input embedding matrix.. -2023-12-04 06:00:17 - Gathering data augmentation.. -2023-12-04 06:00:19 - Running PEFA-XS.. -2023-12-04 06:00:38 - lambda_erm 0.30 | R@10,20,100 0.7888 0.8381 0.9102 diff --git a/examples/pefa-wsdm24/results/trivia.pefa_xs.multi-qa-distilbert-dot-v1.log b/examples/pefa-wsdm24/results/trivia.pefa_xs.multi-qa-distilbert-dot-v1.log deleted file mode 100644 index 48db23f9..00000000 --- a/examples/pefa-wsdm24/results/trivia.pefa_xs.multi-qa-distilbert-dot-v1.log +++ /dev/null @@ -1,5 +0,0 @@ -2023-12-04 05:58:31 - Loading input-to-label matrix.. -2023-12-04 05:58:31 - Loading input embedding matrix.. -2023-12-04 05:58:51 - Gathering data augmentation.. -2023-12-04 05:58:53 - Running PEFA-XS.. -2023-12-04 05:59:13 - lambda_erm 0.30 | R@10,20,100 0.8122 0.8628 0.9333 diff --git a/examples/pefa-wsdm24/results/trivia.pefa_xs.multi-qa-mpnet-base-dot-v1.log b/examples/pefa-wsdm24/results/trivia.pefa_xs.multi-qa-mpnet-base-dot-v1.log deleted file mode 100644 index 6bf7be47..00000000 --- a/examples/pefa-wsdm24/results/trivia.pefa_xs.multi-qa-mpnet-base-dot-v1.log +++ /dev/null @@ -1,5 +0,0 @@ -2023-12-04 18:44:59 - Loading input-to-label matrix.. -2023-12-04 18:44:59 - Loading input embedding matrix.. -2023-12-04 18:45:19 - Gathering data augmentation.. -2023-12-04 18:45:21 - Running PEFA-XS.. -2023-12-04 18:45:41 - lambda_erm 0.30 | R@10,20,100 0.8115 0.8605 0.9297 diff --git a/examples/pefa-wsdm24/results/trivia.pefa_xs.sentence-t5-base.log b/examples/pefa-wsdm24/results/trivia.pefa_xs.sentence-t5-base.log deleted file mode 100644 index 0bb60780..00000000 --- a/examples/pefa-wsdm24/results/trivia.pefa_xs.sentence-t5-base.log +++ /dev/null @@ -1,5 +0,0 @@ -2023-12-04 18:47:42 - Loading input-to-label matrix.. -2023-12-04 18:47:43 - Loading input embedding matrix.. -2023-12-04 18:48:04 - Gathering data augmentation.. -2023-12-04 18:48:08 - Running PEFA-XS.. -2023-12-04 18:48:23 - lambda_erm 0.30 | R@10,20,100 0.7243 0.7839 0.8857 diff --git a/examples/pefa-wsdm24/run_encoder.sh b/examples/pefa-wsdm24/run_encoder.sh deleted file mode 100644 index ad9e3f2f..00000000 --- a/examples/pefa-wsdm24/run_encoder.sh +++ /dev/null @@ -1,29 +0,0 @@ - -data_set=$1 -model_name=$2 -if [ -z ${data_set} ] || [ -z ${model_name} ]; then - echo "run_encoder.sh [data_set] [model_name]" - exit -fi - -if [ ${data_set} != "nq320k" ] && [ ${data_set} != "trivia" ]; then - echo "only support data_set={ nq320k | trivia }!" - exit -fi - -model_name_or_path="sentence-transformers/${model_name}" -input_data_dir="./data/xmc/${data_set}" -output_emb_dir="./data/embeddings/${data_set}/${model_name}" -mkdir -p ${output_emb_dir} - -txt_name_arr=( "tst" "trn" "trn.abs" "trn.doc" "trn.d2q" ) -for txt_name in "${txt_name_arr[@]}"; do - input_data_path="${input_data_dir}/X.${txt_name}.txt" - output_emb_path="${output_emb_dir}/X.${txt_name}.npy" - output_log_path="${output_emb_dir}/X.${txt_name}.log" - python -u encoder.py \ - ${model_name_or_path} \ - ${input_data_path} \ - ${output_emb_path} \ - |& tee ${output_log_path} -done diff --git a/examples/pefa-wsdm24/run_pefa_xl.sh b/examples/pefa-wsdm24/run_pefa_xl.sh deleted file mode 100644 index 9e425d41..00000000 --- a/examples/pefa-wsdm24/run_pefa_xl.sh +++ /dev/null @@ -1,36 +0,0 @@ - -# For Table 2 (data_set=nq320k), -# model_name = [nq-distilbert-base-v1, multi-qa-mpnet-base-dot-v1, sentence-t5-base, gtr-t5-base] -# For Table 3 (data_set=trivia), -# model_name = [multi-qa-distilbert-dot-v1, multi-qa-mpnet-base-dot-v1, sentence-t5-base, gtr-t5-base] -data_set=$1 # "nq320k" -model_name=$2 # "gtr-t5-base" -if [ -z ${data_set} ] || [ -z ${model_name} ]; then - echo "bash run_pefa_xl.sh [data_set] [model_name]" - exit -fi - -if [ ${data_set} == "nq320k" ]; then - knn_type="v0" - topk_q=16 - lambda_erm=0.1 -elif [ ${data_set} == "trivia" ]; then - knn_type="v1" - topk_q=64 - if [ ${model_name} == "multi-qa-distilbert-dot-v1" ]; then - lambda_erm=0.5 - elif [ ${model_name} == "multi-qa-mpnet-base-dot-v1" ]; then - lambda_erm=0.9 - elif [ ${model_name} == "sentence-t5-base" ]; then - lambda_erm=0.7 - elif [ ${model_name} == "gtr-t5-base" ]; then - lambda_erm=0.5 - else - echo "For trivia, model_name=${model_name} is not support!" - fi -else - echo "can not set knn_type due to unknown data_set!" -fi -input_xmc_dir=./data/xmc/${data_set} -input_emb_dir=./data/embeddings/${data_set}/${model_name} -python -u pefa_xl.py ${input_xmc_dir} ${input_emb_dir} ${lambda_erm} ${topk_q} ${knn_type} diff --git a/examples/pefa-wsdm24/run_pefa_xs.sh b/examples/pefa-wsdm24/run_pefa_xs.sh deleted file mode 100644 index 4c6dd4f4..00000000 --- a/examples/pefa-wsdm24/run_pefa_xs.sh +++ /dev/null @@ -1,23 +0,0 @@ - -# For Table 2 (data_set=nq320k), -# model_name = [nq-distilbert-base-v1, multi-qa-mpnet-base-dot-v1, sentence-t5-base, gtr-t5-base] -# For Table 3 (data_set=trivia), -# model_name = [multi-qa-distilbert-dot-v1, multi-qa-mpnet-base-dot-v1, sentence-t5-base, gtr-t5-base] -data_set=$1 # "nq320k" -model_name=$2 # "gtr-t5-base" -if [ -z ${data_set} ] || [ -z ${model_name} ]; then - echo "bash run_pefa_xs.sh [data_set] [model_name]" - exit -fi - -if [ ${data_set} == "nq320k" ]; then - lambda_erm=0.5 -elif [ ${data_set} == "trivia" ]; then - lambda_erm=0.3 -else - echo "can not set lamda_erm due to unknown data_set!" -fi - -input_xmc_dir=./data/xmc/${data_set} -input_emb_dir=./data/embeddings/${data_set}/${model_name} -python -u pefa_xs.py ${input_xmc_dir} ${input_emb_dir} ${lambda_erm} diff --git a/examples/pefa-wsdm24/utils.py b/examples/pefa-wsdm24/utils.py deleted file mode 100644 index ef33c7f6..00000000 --- a/examples/pefa-wsdm24/utils.py +++ /dev/null @@ -1,43 +0,0 @@ - -import numpy as np -from pecos.utils import smat_util - - -def get_eval_metric(Y_true, Y_pred, topk=100): - Y_pred = smat_util.sorted_csr(Y_pred) - metric = smat_util.Metrics.generate(Y_true, Y_pred, topk=topk) - eval_str_fnc = lambda evals, topks: " ".join("{:.4f}".format(v) for i, v in enumerate(evals) if i in topks) - eval_str_obj = "R@10,20,100 {}".format(eval_str_fnc(metric.recall, [9, 19, 99])) - return eval_str_obj - - -def get_data_aug( - X_trn, X_doc, X_d2q, - Y_trn, Y_doc, Y_d2q, - aug_type="v0", -): - if aug_type == "v0": - X_aug = [X_trn] - Y_aug = [Y_trn] - elif aug_type == "v1": - X_aug = [X_doc] - Y_aug = [Y_doc] - elif aug_type == "v2": - X_aug = [X_d2q] - Y_aug = [Y_d2q] - elif aug_type == "v3": - X_aug = [X_trn, X_doc] - Y_aug = [Y_trn, Y_doc] - elif aug_type == "v4": - X_aug = [X_trn, X_d2q] - Y_aug = [Y_trn, Y_d2q] - elif aug_type == "v5": - X_aug = [X_trn, X_doc, X_d2q] - Y_aug = [Y_trn, Y_doc, Y_d2q] - else: - raise ValueError(f"aug_type={aug_type} is not support") - X_aug = np.vstack(X_aug) - Y_aug = smat_util.vstack_csr(Y_aug) - return X_aug, Y_aug - - diff --git a/examples/pina/DataPrep_forXCrepo.py b/examples/pina/DataPrep_forXCrepo.py deleted file mode 100644 index d04bf3e7..00000000 --- a/examples/pina/DataPrep_forXCrepo.py +++ /dev/null @@ -1,65 +0,0 @@ -import argparse -import scipy.sparse as smat -import numpy as np -from pecos.utils import smat_util -import sklearn -import os -import sys -from xclib.data import data_utils - -def main(): - parser = argparse.ArgumentParser(description='DataPrep_forXCrepo') - parser.add_argument('--work_dir', type=str, default='.') - parser.add_argument('--dataset', type=str, default='LF-Amazon-131K') - args = parser.parse_args() - print(args) - - cur_dir = f'{args.work_dir}/dataset/{args.dataset}' - - if args.dataset in ['LF-Amazon-131K','LF-WikiSeeAlso-320K','LF-Amazon-1.3M']: - # Read files with features and labels (old format from XMLRepo) - features, tabels, num_samples, num_features, num_labels = data_utils.read_data(f'{cur_dir}/train.txt') - features = features.astype(np.float32) - sklearn.preprocessing.normalize(features,copy=False) - smat.save_npz(f'{cur_dir}/X_bow.trn.npz',features) - smat.save_npz(f'{cur_dir}/normalized/Y.trn.npz',tabels) - smat.save_npz(f'{cur_dir}/raw/Y.trn.npz',tabels) - - features, tabels, num_samples, num_features, num_labels = data_utils.read_data(f'{cur_dir}/test.txt') - features = features.astype(np.float32) - sklearn.preprocessing.normalize(features,copy=False) - smat.save_npz(f'{cur_dir}/X_bow.tst.npz',features) - smat.save_npz(f'{cur_dir}/normalized/Y.tst.npz',tabels) - smat.save_npz(f'{cur_dir}/raw/Y.tst.npz',tabels) - - TEST = data_utils.read_sparse_file(f'{cur_dir}/Yf.txt',header=True) - sklearn.preprocessing.normalize(TEST,copy=False) - smat.save_npz(f"{cur_dir}/Y_bow.npz",TEST) - elif args.dataset in ['LF-Wikipedia-500K']: - # Read files with labels (old format from XMLRepo) - _, tabels, num_samples, num_features, num_labels = data_utils.read_data(f'{cur_dir}/train.txt') - smat.save_npz(f'{cur_dir}/normalized/Y.trn.npz',tabels) - smat.save_npz(f'{cur_dir}/raw/Y.trn.npz',tabels) - _, tabels, num_samples, num_features, num_labels = data_utils.read_data(f'{cur_dir}/test.txt') - smat.save_npz(f'{cur_dir}/normalized/Y.tst.npz',tabels) - smat.save_npz(f'{cur_dir}/raw/Y.tst.npz',tabels) - - # Read files with features (BoW, dim = 500000. The feature in the old format has dim = 2381304.) - X_trn = data_utils.read_sparse_file(f"{cur_dir}/trn_X_Xf.txt", header=True) - X_trn = sklearn.preprocessing.normalize(X_trn,norm='l2') - - X_tst = data_utils.read_sparse_file(f"{cur_dir}/tst_X_Xf.txt", header=True) - X_tst = sklearn.preprocessing.normalize(X_tst,norm='l2') - - L_bow = data_utils.read_sparse_file(f"{cur_dir}/lbl_X_Xf.txt", header=True) - L_bow = sklearn.preprocessing.normalize(L_bow,norm='l2') - - smat.save_npz(f"{cur_dir}/Y_bow.npz",L_bow) - smat.save_npz(f"{cur_dir}/X_bow.trn.npz",X_trn) - smat.save_npz(f"{cur_dir}/X_bow.tst.npz",X_tst) - else: - raise ValueError(f'Dataset {args.dataset} is not supported yet!') - - -if __name__ == "__main__": - main() diff --git a/examples/pina/Ensemble-PINA.py b/examples/pina/Ensemble-PINA.py deleted file mode 100644 index ef0dda0c..00000000 --- a/examples/pina/Ensemble-PINA.py +++ /dev/null @@ -1,42 +0,0 @@ -from pecos.utils.smat_util import sorted_csr, CsrEnsembler, load_matrix, Metrics -import scipy.sparse as smat -import argparse -import os - -def main(): - parser = argparse.ArgumentParser(description='PrepareXYstack') - parser.add_argument('--work_dir', type=str, default='.') - parser.add_argument('--dataset', type=str, default='LF-Amazon-131K') - parser.add_argument('--model_name', type=str, default='v0') - parser.add_argument('--DS_model_names', type=str, default='v0,v0-s1,v0-s2', help="The DS_model_name should be seperated by ','. For example: 'v0,v0-s1,v0-s2'.") - parser.add_argument('--feature_name', type=str, default='BoW') - parser.add_argument('--ens_name', type=str, default='softmax', choices = ['rank', 'softmax', 'sigmoid']) - parser.add_argument('--L_option', type=str, default='Lft_xrt') - parser.add_argument('--Pk', type=str, default='5') - parser.add_argument('--Use_A', type=str, default='false') - args = parser.parse_args() - print(args) - - - feature_dir=f"{args.work_dir}/dataset/{args.dataset}" - TAGS = args.DS_model_names.split(',') - assert len(TAGS)>1 # Assume to ensemble at least 2 models! - - P_paths = [] - for tag in TAGS: - P_paths.append(f"{args.work_dir}/models_LF/xtransformer/{args.dataset}/{args.model_name}/{args.feature_name}/XYstack/downstream/{tag}/{args.Pk}/{args.L_option}/P.20.npz") - - Y_true = sorted_csr(load_matrix(f"{args.work_dir}/dataset/{args.dataset}/raw/Y.tst.npz").tocsr()) - Y_pred = [sorted_csr(load_matrix(pp).tocsr()) for pp in P_paths] - print("==== evaluation results ====") - ens = getattr(CsrEnsembler, f"{args.ens_name}_average") - cur_pred = ens(*Y_pred) - print(Metrics.generate(Y_true, cur_pred, topk=10)) - PATH = f"{args.work_dir}/models_LF/xtransformer/{args.dataset}/{args.model_name}/{args.feature_name}/XYstack/downstream/{args.DS_model_names}/{args.Pk}/{args.L_option}" - os.makedirs(PATH,exist_ok=True) - smat.save_npz(f"{PATH}/P.20.{args.ens_name}.npz",cur_pred) - print("Ensembled P matrix saved!") - print(f"Saved model path: {PATH}") - print(f"To evaluate, please use this path with ./scripts/Ensemble_evaluations.sh") -if __name__ == "__main__": - main() diff --git a/examples/pina/PINA_augmentation.py b/examples/pina/PINA_augmentation.py deleted file mode 100644 index d97a8c73..00000000 --- a/examples/pina/PINA_augmentation.py +++ /dev/null @@ -1,244 +0,0 @@ -from pecos.xmc.xtransformer.model import XTransformer -import scipy.sparse as smat -import numpy as np -from pecos.utils import smat_util -from pecos.utils.featurization.text.preprocess import Preprocessor -import sklearn -import os -from pecos.xmc import Indexer, LabelEmbeddingFactory -import sys -from pecos.core import clib as pecos_clib -from tqdm import tqdm -import argparse - - - -def CSR_rowwise_softmax(P): - P.data = np.exp(P.data).astype(np.float32) - P = sklearn.preprocessing.normalize(P, norm='l1') - return P - -def main(): - parser = argparse.ArgumentParser(description='PrepareXYstack') - parser.add_argument('--model_name', type=str, required=True) - parser.add_argument('--feature_name', type=str, default='BoW') - parser.add_argument('--work_dir', type=str, default='.') - parser.add_argument('--dataset', type=str, default='LF-AmazonTitles-131K') - parser.add_argument('--L_option', type=str, default='Lft_xrt', choices=['Lf', 'Lft', 'Lf_xrt','Lft_xrt','Lxrt']) - parser.add_argument('--Pk', type=int, default=5, help='Should be =< 20!!!') - parser.add_argument('--Use_A', type=int, default=0, help='Use true neighbor for training data') - parser.add_argument('--batch_size', type=int, default=256, help='batch size when applying XR-Transformer') - parser.add_argument('--num_workers', type=int, default=48, help='number of workers XR-Transformer') - parser.add_argument('--text_normalization', type=str, default="raw", help='Use raw or normalized text.') - args = parser.parse_args() - print(args) - - # !!! only_topk =<20 !!! - topk = 5 - - feature_dir=f"{args.work_dir}/dataset/{args.dataset}" - params_path=f"{args.work_dir}/scripts/params/xtransformer/{args.dataset}/{args.model_name}.json" - model_dir=f"{args.work_dir}/models_LF/xtransformer/{args.dataset}/{args.model_name}/{args.feature_name}/XYstack" - - # Remember to replace 20 with your own top k if you have modified it! - P_trn = smat.load_npz("{}/P.20.trn.npz".format(model_dir)) - P_tst = smat.load_npz("{}/P.20.tst.npz".format(model_dir)) - - if len(args.L_option)>2 and args.L_option[-3:]=="xrt": - xtf = XTransformer.load(model_dir) - - # use softmax row-wise to turn it into a probability. Since P contains negative values... - P_trn = smat_util.sorted_csr(P_trn,only_topk=topk)[:] - P_tst = smat_util.sorted_csr(P_tst,only_topk=topk)[:] - - if P_trn.min()<0 or P_tst.min()<0: - P_trn = CSR_rowwise_softmax(P_trn) - P_tst = CSR_rowwise_softmax(P_tst) - - N_trn = P_trn.shape[0] - print("{}/{}/Y_all.npz".format(feature_dir,args.text_normalization)) - Y_trn = smat.load_npz("{}/{}/Y_all.npz".format(feature_dir,args.text_normalization))[:N_trn, :] - - print(f"P_trn shape is {P_trn.shape}, max: {P_trn.max()}, min: {P_trn.min()}") - print(f"P_tst shape is {P_tst.shape}, max: {P_tst.max()}, min: {P_tst.min()}") - print(f"Y_trn shape is {Y_trn.shape}, max: {Y_trn.max()}, min: {Y_trn.min()}") - - - # Get features of pretraining XMC output space - if args.L_option == "Lft_xrt": - # Generate xrt dense embedding for label text - with open(f"{args.work_dir}/dataset/{args.dataset}/{args.text_normalization}/output-items.txt") as f: - text = f.readlines() - L_emb = xtf.encode(text, batch_size=args.batch_size, batch_gen_workers=args.num_workers) - - # Generate xrt dense embedding for instance text - with open("{}/dataset/{}/{}/X.trn.txt".format(args.work_dir,args.dataset,args.text_normalization)) as f: - text = f.readlines() - X_emb = xtf.encode(text, batch_size=args.batch_size, batch_gen_workers=args.num_workers) - - # Prepare [L|X] for dense embedding. - L_emb = smat.csr_matrix(L_emb,dtype=np.float32) - X_emb = smat.csr_matrix(X_emb,dtype=np.float32) - All_emb = smat_util.vstack_csr([L_emb,X_emb]) - # Row normalization - All_emb = sklearn.preprocessing.normalize(All_emb,norm='l2') - - # Load stacked instance feature - if args.feature_name in ['BoW']: - X_all = smat.load_npz('{}/X_bow.all.npz'.format(feature_dir)).astype(np.float32) - else: - X_all = smat.load_npz('{}/X.tfidf.all.npz'.format(feature_dir)).astype(np.float32) - - X_all = sklearn.preprocessing.normalize(X_all,norm='l2') - - # Concat sparse and dense embedding - Lf1 = smat_util.hstack_csr([X_all,All_emb]) - - elif args.L_option == "Lf_xrt": - # Use PIFA - # # Load stacked instance feature and multilabel matrix - if args.feature_name in ['BoW']: - X_all = smat.load_npz('{}/X_bow.all.npz'.format(feature_dir)).astype(np.float32) - else: - X_all = smat.load_npz('{}/X.tfidf.all.npz'.format(feature_dir)).astype(np.float32) - X_all = sklearn.preprocessing.normalize(X_all,norm='l2') - Y_all = smat.load_npz('{}/dataset/{}/{}/Y_all.npz'.format(args.work_dir,args.dataset,args.text_normalization)).astype(np.float32) - - # Produce PIFA embedding - Lf1 = LabelEmbeddingFactory.create(Y_all, X_all, method="pifa") - - # Generate xrt dense embedding for label text and instnace text - with open("{}/dataset/{}/{}/output-items.txt".format(args.work_dir,args.dataset,args.text_normalization)) as f: - text = f.readlines() - L_emb = xtf.encode(text, batch_size=args.batch_size, batch_gen_workers=args.num_workers) - with open("{}/dataset/{}/{}/X.trn.txt".format(args.work_dir,args.dataset,args.text_normalization)) as f: - text = f.readlines() - X_emb = xtf.encode(text, batch_size=args.batch_size, batch_gen_workers=args.num_workers) - - L_emb = smat.csr_matrix(L_emb,dtype=np.float32) - X_emb = smat.csr_matrix(X_emb,dtype=np.float32) - All_emb = smat_util.vstack_csr([L_emb,X_emb]) - # Row normalization - All_emb = sklearn.preprocessing.normalize(All_emb,norm='l2') - # Concat, X|L - Lf1 = smat_util.hstack_csr([All_emb,Lf1]) - - elif args.L_option == "Lft": - # Load stacked instance feature - if args.feature_name in ['BoW']: - X_all = smat.load_npz('{}/X_bow.all.npz'.format(feature_dir)).astype(np.float32) - else: - X_all = smat.load_npz('{}/X.tfidf.all.npz'.format(feature_dir)).astype(np.float32) - - Lf1 = sklearn.preprocessing.normalize(X_all,norm='l2') - - elif args.L_option == "Lf": - # Load stacked instance feature - if args.feature_name in ['BoW']: - X_all = smat.load_npz('{}/X_bow.all.npz'.format(feature_dir)).astype(np.float32) - else: - X_all = smat.load_npz('{}/X.tfidf.all.npz'.format(feature_dir)).astype(np.float32) - - X_all = sklearn.preprocessing.normalize(X_all,norm='l2') - Y_all = smat.load_npz('{}/dataset/{}/{}/Y_all.npz'.format(args.work_dir,args.dataset,args.text_normalization)).astype(np.float32) - - # Produce PIFA embedding - Lf1 = LabelEmbeddingFactory.create(Y_all, X_all, method="pifa") - - elif args.L_option == "Lxrt": - # Generate xrt dense embedding for label text and instnace text - with open("{}/dataset/{}/{}/output-items.txt".format(args.work_dir,args.dataset,args.text_normalization)) as f: - text = f.readlines() - L_emb = xtf.encode(text, batch_size=args.batch_size, batch_gen_workers=args.num_workers) - with open("{}/dataset/{}/{}/X.trn.txt".format(args.work_dir,args.dataset,args.text_normalization)) as f: - text = f.readlines() - X_emb = xtf.encode(text, batch_size=args.batch_size, batch_gen_workers=args.num_workers) - - L_emb = smat.csr_matrix(L_emb,dtype=np.float32) - X_emb = smat.csr_matrix(X_emb,dtype=np.float32) - Lf1 = smat_util.vstack_csr([L_emb,X_emb]) - - else: - print("Not implemented") - - # Apply row wise l2 normalization - Lf1 = sklearn.preprocessing.normalize(Lf1,norm='l2') - print(f"Feature shape for the pretraining XMC output space: {Lf1.shape}") - - # Prepare PINA augmentation - - # This allows for multi-hop generalization in the future... - Hops_trn = [] - Hops_tst = [] - Hops_true = [] - - # 0-Hop, also include xrt emb!!! - if args.feature_name in ['BoW']: - X_trn = smat.load_npz("{}/X_bow.trn.npz".format(feature_dir)) - X_tst = smat.load_npz("{}/X_bow.tst.npz".format(feature_dir)) - else: - X_trn = smat.load_npz('{}/X.tfidf.trn.npz'.format(feature_dir)).astype(np.float32) - X_tst = smat.load_npz('{}/X.tfidf.tst.npz'.format(feature_dir)).astype(np.float32) - - with open(f"{args.work_dir}/dataset/{args.dataset}/{args.text_normalization}/X.trn.txt") as f: - text = f.readlines() - X_emb_trn = xtf.encode(text, batch_size=args.batch_size, batch_gen_workers=args.num_workers) - - with open(f"{args.work_dir}/dataset/{args.dataset}/{args.text_normalization}/X.tst.txt") as f: - text = f.readlines() - X_emb_tst = xtf.encode(text, batch_size=args.batch_size, batch_gen_workers=args.num_workers) - - X_emb_trn = smat.csr_matrix(X_emb_trn,dtype=np.float32) - X_emb_tst = smat.csr_matrix(X_emb_tst,dtype=np.float32) - X_emb_trn = sklearn.preprocessing.normalize(X_emb_trn,norm='l2') - X_emb_tst = sklearn.preprocessing.normalize(X_emb_tst,norm='l2') - - X_trn = sklearn.preprocessing.normalize(X_trn,norm='l2') - X_tst = sklearn.preprocessing.normalize(X_tst,norm='l2') - - X_trn = smat_util.hstack_csr([X_emb_trn,X_trn]) - X_tst = smat_util.hstack_csr([X_emb_tst,X_tst]) - - X_trn = sklearn.preprocessing.normalize(X_trn,norm='l2') - X_tst = sklearn.preprocessing.normalize(X_tst,norm='l2') - - Hops_trn.append(X_trn) - Hops_tst.append(X_tst) - Hops_true.append(X_trn) - - # 1-Hop - X_trn = pecos_clib.sparse_matmul(P_trn,Lf1) - X_tst = pecos_clib.sparse_matmul(P_tst,Lf1) - X_true = pecos_clib.sparse_matmul(Y_trn,Lf1) - - # Apply row wise l2 normalization - X_trn = sklearn.preprocessing.normalize(X_trn,norm='l2') - X_tst = sklearn.preprocessing.normalize(X_tst,norm='l2') - X_true = sklearn.preprocessing.normalize(X_true,norm='l2') - - Hops_trn.append(X_trn) - Hops_tst.append(X_tst) - Hops_true.append(X_true) - - # Concat all hops. - X_cat_trn = smat_util.hstack_csr(Hops_trn) - X_cat_tst = smat_util.hstack_csr(Hops_tst) - X_cat_true = smat_util.hstack_csr(Hops_true) - - # Apply row wise l2 normalization - X_cat_trn = sklearn.preprocessing.normalize(X_cat_trn,norm='l2') - X_cat_tst = sklearn.preprocessing.normalize(X_cat_tst,norm='l2') - X_cat_true = sklearn.preprocessing.normalize(X_cat_true,norm='l2') - - print(f"X_trn shape is {X_cat_trn.shape}, max: {X_cat_trn.max()}, min: {X_cat_trn.min()}") - print(f"X_tst shape is {X_cat_tst.shape}, max: {X_cat_tst.max()}, min: {X_cat_tst.min()}") - print(f"X_true shape is {X_cat_true.shape}, max: {X_cat_true.max()}, min: {X_cat_true.min()}") - - smat.save_npz(f'{model_dir}/X_trn_P{args.Pk}{args.L_option}.npz',X_cat_trn.astype(np.float32)) - smat.save_npz(f'{model_dir}/X_tst_P{args.Pk}{args.L_option}.npz',X_cat_tst.astype(np.float32)) - smat.save_npz(f'{model_dir}/X_true_{args.L_option}.npz',X_cat_true.astype(np.float32)) - - print("All Set!!") - -if __name__ == "__main__": - main() diff --git a/examples/pina/PrepareXYstack-raw.py b/examples/pina/PrepareXYstack-raw.py deleted file mode 100644 index 23e119f2..00000000 --- a/examples/pina/PrepareXYstack-raw.py +++ /dev/null @@ -1,65 +0,0 @@ -import scipy.sparse as smat -import numpy as np -from pecos.utils import smat_util -import sklearn.preprocessing -import os -import sys -from tqdm import tqdm -import argparse - -def main(): - parser = argparse.ArgumentParser(description='PrepareXYstack-raw') - parser.add_argument('--work_dir', type=str, default='.') - parser.add_argument('--dataset', type=str, default='LF-Amazon-131K') - args = parser.parse_args() - print(args) - - data_dir= f"{args.work_dir}/dataset/{args.dataset}/raw" - feature_dir = f"{args.work_dir}/dataset/{args.dataset}" - - Y = smat.load_npz(f"{data_dir}/Y.trn.npz") - num_X, num_L = Y.shape - - # Preparing Y_all.npz - I_X = smat.identity(num_X,dtype=np.float32,format='csr') - I_L = smat.identity(num_L,dtype=np.float32,format='csr') - - X_LX = smat_util.hstack_csr([Y,I_X]) - L_XL = smat_util.hstack_csr([I_L,Y.transpose().tocsr()]) # This should be improved in the future... - Y_all = smat_util.vstack_csr([X_LX,L_XL]) - smat.save_npz("{}/Y_all.npz".format(data_dir),Y_all) - del Y_all, Y, I_X, I_L, X_LX, L_XL - - # Stack bag of word feature - X_bow_trn = smat.load_npz(f"{feature_dir}/X_bow.trn.npz") - Y_bow = smat.load_npz(f"{feature_dir}/Y_bow.npz") - - # Normalization! - sklearn.preprocessing.normalize(X_bow_trn,copy=False) - sklearn.preprocessing.normalize(Y_bow,copy=False) - - X_bow_all = smat_util.vstack_csr([X_bow_trn,Y_bow]) - smat.save_npz(f"{feature_dir}/X_bow.all.npz",X_bow_all) - del Y_bow, X_bow_trn, X_bow_all - - # Prepare text file (X_all.txt), note that the order should be X|L - with open("{}/trn.txt".format(data_dir),'r') as f: - X_lines = f.readlines() - assert num_X == len(X_lines) - print(num_X,len(X_lines)) - with open("{}/output-items.txt".format(data_dir),'r') as f: - L_lines = f.readlines() - assert num_L == len(L_lines) - print(num_L,len(L_lines)) - - with open("{}/X_all.txt".format(data_dir),"w") as f: - for line in tqdm(X_lines): - f.write(line) - for line in tqdm(L_lines): - f.write(line) - - print("All Done!") - -if __name__ == "__main__": - main() - diff --git a/examples/pina/README.md b/examples/pina/README.md deleted file mode 100644 index f33cf045..00000000 --- a/examples/pina/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# PINA: Leveraging Side Information in eXtreme Multi-label Classification via Predicted Instance Neighborhood Aggregation, ICML 2023 - -This folder contains code to train XR-Transformer+PINA models and reproduce experiments -in ["PINA: Leveraging Side Information in eXtreme Multi-label Classification via Predicted Instance Neighborhood Aggregation"](https://arxiv.org/pdf/2305.12349.pdf). - -## Getting Started -* Clone the repository and enter `examples/pina` directory. -* First create a [virtual environment](https://docs.python.org/3/library/venv.html) and then install dependencies -by running the following command: -```bash -pip install libpecos pandas gdown urllib3==1.26.6 -``` -If you're unfamiliar with Python virtual environments, check out the -[user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). - -* Install [pyxclib](https://github.com/kunaldahiya/pyxclib) -* Verify pytorch and CUDA: -```bash - python -c "import torch; print('torch={}, cuda={}'.format(torch.__version__, torch.cuda.is_available()))" -``` - -## Training and Evaluation -To train and evaluate PINA+XR-Transformer model, run -``` bash -chmod a+x ./scripts/* -DATASET="LF-Amazon-131K" -bash scripts/run_pina.sh ${DATASET} -``` -Recommended platform for training: [AWS p3.16xlarge instance](https://aws.amazon.com/ec2/instance-types/p3/) or equivalent. - -### References: - -[1] XC Repo: http://manikvarma.org/downloads/XC/XMLRepository.html diff --git a/examples/pina/convert_format.pl b/examples/pina/convert_format.pl deleted file mode 100644 index 74c721bb..00000000 --- a/examples/pina/convert_format.pl +++ /dev/null @@ -1,47 +0,0 @@ -my $inpfile; -open($inpfile,"<",$ARGV[0]); - -my $ftfile; -open($ftfile,">",$ARGV[1]); - -my $lblfile; -open($lblfile,">",$ARGV[2]); - -my $ctr = 0; -while(<$inpfile>) -{ - chomp($_); - - if($ctr==0) - { - my @items = split(" ",$_); - $num_inst = $items[0]; - $num_ft = $items[1]; - $num_lbl = $items[2]; - - print $ftfile "$num_inst $num_ft\n"; - print $lblfile "$num_inst $num_lbl\n"; - } - else - { - my @items = split(" ",$_,2); - - if($_ =~ /^ .*/) - { - print $lblfile "\n"; - print $ftfile $items[0]."\n"; - } - else - { - my @lbls = split(",",$items[0]); - print $lblfile join(" ",map {"$_:1"} @lbls)."\n"; - print $ftfile $items[1]."\n"; - } - } - - $ctr++; -} - -close($inpfile); -close($ftfile); -close($lblfile); diff --git a/examples/pina/evaluate.py b/examples/pina/evaluate.py deleted file mode 100644 index b74bdc83..00000000 --- a/examples/pina/evaluate.py +++ /dev/null @@ -1,63 +0,0 @@ -# Example to evaluate -import sys -import xclib.evaluation.xc_metrics as xc_metrics -import xclib.data.data_utils as data_utils -from scipy.sparse import load_npz -import scipy.sparse as sparse -import numpy as np -import json -import os - - -def load_overlap(data_dir, filter_label_file='filter_labels.txt'): - docs = np.asarray([]) - lbs = np.asarray([]) - if os.path.exists(os.path.join(data_dir, filter_label_file)): - filter_lbs = np.loadtxt(os.path.join( - data_dir, filter_label_file), dtype=np.int32) - if filter_lbs.size > 0: - docs = filter_lbs[:, 0] - lbs = filter_lbs[:, 1] - else: - print("Overlap not found") - print("Overlap is:", docs.size) - return docs, lbs - - -def _remove_overlap(score_mat, docs, lbs): - score_mat[docs, lbs] = 0 - score_mat = score_mat.tocsr() - score_mat.eliminate_zeros() - return score_mat - - -def main(targets_label_file, train_label_file, predictions_file, A, B, docs, lbls): - true_labels = _remove_overlap( - data_utils.read_sparse_file( - targets_label_file, force_header=True).tolil(), - docs, lbls) - trn_labels = data_utils.read_sparse_file( - train_label_file, force_header=True) - inv_propen = xc_metrics.compute_inv_propesity(trn_labels, A=A, B=B) - acc = xc_metrics.Metrics( - true_labels, inv_psp=inv_propen, remove_invalid=False) - predicted_labels = _remove_overlap( - load_npz(predictions_file+'.npz').tolil(), - docs, lbls) - rec = xc_metrics.recall(predicted_labels, true_labels, k=10)[-1]*100 - print("R@10=%0.2f" % (rec)) - args = acc.eval(predicted_labels, 5) - print(xc_metrics.format(*args)) - - -if __name__ == '__main__': - train_label_file = sys.argv[1] - targets_file = sys.argv[2] # Usually test data file - predictions_file = sys.argv[3] # In mat format - data_dir=sys.argv[4] - # configs = json.load(open(sys.argv[5]))["DEFAULT"] - A = 0.6 - B = 2.6 - filter_data = "filter_labels_test.txt" - docs, lbls = load_overlap(data_dir, filter_label_file=filter_data) - main(targets_file, train_label_file, predictions_file, A, B, docs, lbls) diff --git a/examples/pina/pecos_dataform_full.py b/examples/pina/pecos_dataform_full.py deleted file mode 100644 index ddb65293..00000000 --- a/examples/pina/pecos_dataform_full.py +++ /dev/null @@ -1,74 +0,0 @@ -import pandas as pd -import sys, os -import re -import string - -regex_punctuation = re.compile("[%s]" % re.escape(string.punctuation)) -regex_space = re.compile(r"\s+") - -def normalize(xx): - xx = regex_punctuation.sub(" ", xx) - xx = regex_space.sub(" ", xx).strip().lower() - return xx - - -def process_json_data(dataname, part): - raw_dir = f"{dataname}/raw/" - norm_dir = f"{dataname}/normalized/" - os.makedirs(norm_dir, exist_ok=True) - - assert part in ["trn", "tst", "lbl"] - df = pd.read_json(f"{raw_dir}/{part}.json", lines=True) - - X_title = list(df.title) - X_content = list(df.content) - - - Y = list(df.target_ind) - X_norm = [] - X = [] - - if "titles" in dataname.lower(): - for xxx in X_title: - X.append(xxx) - X_norm.append(normalize(xxx)) - else: - for i in range(len(X_title)): - xxx = X_title[i] + " " + X_content[i] - X.append(xxx) - X_norm.append(normalize(xxx)) - - # X_norm = [normalize(xxx) for xxx in X] - - - if part in ["trn", "tst"]: - # trn.txt tst.txt - # l_1,l_2,...,l_kxxxx xxxx xxxx - with open(f"{raw_dir}/{part}.txt", 'w') as fout: - for xx, yy in zip(X, Y): - fout.write(",".join([str(y) for y in yy]) + '\t' + xx + '\n') - - with open(f"{norm_dir}/{part}.txt", 'w') as fout: - for xx, yy in zip(X_norm, Y): - fout.write(",".join([str(y) for y in yy]) + '\t' + xx + '\n') - - with open(f"{raw_dir}/X.{part}.txt", 'w') as fout: - for xx in X: - fout.write(xx + '\n') - with open(f"{norm_dir}/X.{part}.txt", 'w') as fout: - for xx in X_norm: - fout.write(xx + '\n') - else: - with open(f"{raw_dir}/output-items.txt", 'w') as fout: - for xx in X: - fout.write(xx + '\n') - - with open(f"{norm_dir}/output-items.txt", 'w') as fout: - for xx in X_norm: - fout.write(xx + '\n') - - -dataname = f'./dataset/{sys.argv[1]}' -process_json_data(dataname, "tst") -process_json_data(dataname, "trn") -process_json_data(dataname, "lbl") diff --git a/examples/pina/scripts/download_data.sh b/examples/pina/scripts/download_data.sh deleted file mode 100755 index bdf27533..00000000 --- a/examples/pina/scripts/download_data.sh +++ /dev/null @@ -1,47 +0,0 @@ -#================= inputs ===================== -dataset=$1 # This is th dataset name (i.e LF-Amazon-131K). -work_dir="." -mkdir -p dataset -cd dataset -echo "$(pwd)" - -echo "Downloading ${dataset}" -if [[ $dataset == "LF-Amazon-131K" ]] -then - rawtext_id="1WuquxCAg8D4lKr-eZXPv4nNw2S2lm7_E" - BoW_id="1YNGEifTHu4qWBmCaLEBfjx07qRqw9DVW" - BoW_dname="LF-Amazon-131K" -elif [[ $dataset == "LF-WikiSeeAlso-320K" ]] -then - rawtext_id="1QZD4dFVxDpskCI2kGH9IbzgQR1JSZT-N" - BoW_id="1N8C_RL71ErX6X92ew9h8qRuTWJ9LywE8" - BoW_dname="LF-WikiSeeAlso-320K" -elif [[ $dataset == "LF-Amazon-1.3M" ]] -then - rawtext_id="12zH4mL2RX8iSvH0VCNnd3QxO4DzuHWnK" - BoW_id="1Davc6BIfoTIAS3mP1mUY5EGcGr2zN2pO" - BoW_dname="LF-AmazonTitles-1.3M" -fi - -echo $rawtext_id - -gdown $rawtext_id -gdown $BoW_id - -unzip $dataset.raw.zip -unzip -j $BoW_dname.bow.zip -d $dataset - -data_dir="${dataset}" -mkdir -p ${data_dir}/normalized -mkdir -p ${data_dir}/raw - -cd ${dataset} - -mv *.json.gz raw - -cd raw - -gunzip *.gz - -echo "${dataset} downlowded and unzipped!!!" - diff --git a/examples/pina/scripts/ensemble_evaluation.sh b/examples/pina/scripts/ensemble_evaluation.sh deleted file mode 100755 index 87ac2848..00000000 --- a/examples/pina/scripts/ensemble_evaluation.sh +++ /dev/null @@ -1,18 +0,0 @@ -#================= inputs ===================== -ens_name="softmax" -dname=$1 -model_name=$2 -DS_model_names=$3 -topk=20 -model_dir="./models_LF/xtransformer/${dname}/${model_name}/BoW/XYstack/downstream/${DS_model_names}/5/Lft_xrt" - -python3 Ensemble-PINA.py --dataset ${dname} \ - --model_name ${model_name} \ - --DS_model_names ${DS_model_names} - -echo "===After reciprocal pair removal===" -python3 -u ./evaluate.py \ - "./dataset/${dname}/trn_X_Y.txt" \ - "./dataset/${dname}/tst_X_Y.txt" \ - "${model_dir}/P.${topk}.${ens_name}" ./dataset/${dname} \ - |& tee ${model_dir}/Reciprocal_Removed_eval.log diff --git a/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-pre.json b/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-pre.json deleted file mode 100644 index f1a07c15..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-pre.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 110, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 110, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-s0.json b/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-s0.json deleted file mode 100644 index dee2a541..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-s0.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 40, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 40, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-s1.json b/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-s1.json deleted file mode 100644 index f00e7b9f..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-s1.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 40, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 40, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-s2.json b/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-s2.json deleted file mode 100644 index 008016a0..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Amazon-1.3M/v0-raw-s2.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 40, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 40, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-pre.json b/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-pre.json deleted file mode 100644 index 0670fe15..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-pre.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 12000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "disable", - "rel_norm": "no-norm", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "noop" - } - ] - } - } - } -} \ No newline at end of file diff --git a/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-s0.json b/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-s0.json deleted file mode 100644 index 0670fe15..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-s0.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 12000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "disable", - "rel_norm": "no-norm", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "noop" - } - ] - } - } - } -} \ No newline at end of file diff --git a/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-s1.json b/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-s1.json deleted file mode 100644 index 4a040940..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-s1.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 12000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "disable", - "rel_norm": "no-norm", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "noop" - } - ] - } - } - } -} \ No newline at end of file diff --git a/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-s2.json b/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-s2.json deleted file mode 100644 index e3982474..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Amazon-131K/v0-raw-s2.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 12000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "disable", - "rel_norm": "no-norm", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "noop" - } - ] - } - } - } -} \ No newline at end of file diff --git a/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-pre.json b/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-pre.json deleted file mode 100644 index 0670fe15..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-pre.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 12000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "disable", - "rel_norm": "no-norm", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "noop" - } - ] - } - } - } -} \ No newline at end of file diff --git a/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-s0.json b/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-s0.json deleted file mode 100644 index 44852088..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-s0.json +++ /dev/null @@ -1,342 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} \ No newline at end of file diff --git a/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-s1.json b/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-s1.json deleted file mode 100644 index 6d3c59b2..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-s1.json +++ /dev/null @@ -1,342 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} \ No newline at end of file diff --git a/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-s2.json b/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-s2.json deleted file mode 100644 index 9e9185b6..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-WikiSeeAlso-320K/v0-raw-s2.json +++ /dev/null @@ -1,342 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} \ No newline at end of file diff --git a/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-pre.json b/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-pre.json deleted file mode 100644 index 3370601c..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-pre.json +++ /dev/null @@ -1,350 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 16, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 60000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-s0.json b/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-s0.json deleted file mode 100644 index 5116061c..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-s0.json +++ /dev/null @@ -1,350 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 16, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 60000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-s1.json b/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-s1.json deleted file mode 100644 index cd92ea46..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-s1.json +++ /dev/null @@ -1,350 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 16, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 60000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-s2.json b/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-s2.json deleted file mode 100644 index 44b0b3ea..00000000 --- a/examples/pina/scripts/params/xtransformer/LF-Wikipedia-500K/v0-raw-s2.json +++ /dev/null @@ -1,350 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 16, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 60000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "pre_tokenize": false, - "pre_tensorize_labels": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/pina/scripts/run_pina.sh b/examples/pina/scripts/run_pina.sh deleted file mode 100755 index 596feae0..00000000 --- a/examples/pina/scripts/run_pina.sh +++ /dev/null @@ -1,28 +0,0 @@ -dataset=$1 - -echo *** data downloading and preprocessing *** -./scripts/download_data.sh ${dataset} - -python3 DataPrep_forXCrepo.py --dataset ${dataset} - -python3 pecos_dataform_full.py ${dataset} - -perl convert_format.pl ./dataset/${dataset}/train.txt ./dataset/${dataset}/trn_X_Xf.txt ./dataset/${dataset}/trn_X_Y.txt -perl convert_format.pl ./dataset/${dataset}/test.txt ./dataset/${dataset}/tst_X_Xf.txt ./dataset/${dataset}/tst_X_Y.txt -python3 PrepareXYstack-raw.py --dataset ${dataset} -echo "Data Preparation for ${dataset} is done!" - -echo *** PINA pre-training *** - -./scripts/xtransformer-XYstack-raw.sh v0-raw-pre BoW ${dataset} - -python3 PINA_augmentation.py --model_name v0-raw-pre --dataset ${dataset} - -echo *** Downstream Model Training *** - -./scripts/xtransformer-XYstack-DS-raw.sh v0-raw-pre v0-raw-s0 BoW ${dataset} 5 Lft_xrt -./scripts/xtransformer-XYstack-DS-raw.sh v0-raw-pre v0-raw-s1 BoW ${dataset} 5 Lft_xrt -./scripts/xtransformer-XYstack-DS-raw.sh v0-raw-pre v0-raw-s2 BoW ${dataset} 5 Lft_xrt - -echo *** Evaluation *** -./scripts/ensemble_evaluation.sh ${dataset} "v0-raw-pre" "v0-raw-s0,v0-raw-s1,v0-raw-s2" diff --git a/examples/pina/scripts/xtransformer-XYstack-DS-raw.sh b/examples/pina/scripts/xtransformer-XYstack-DS-raw.sh deleted file mode 100755 index 7998fb42..00000000 --- a/examples/pina/scripts/xtransformer-XYstack-DS-raw.sh +++ /dev/null @@ -1,68 +0,0 @@ -#================= inputs ===================== -model_name=$1 -DS_model_name=$2 -feature_name=$3 -work_dir="." -dname=$4 -topk=20 -P_topk=$5 -L_option=$6 -Use_A=$7 - -feature_dir="${work_dir}/dataset/${dname}/tfidf/default/XYstack/raw" -params_path=${work_dir}/scripts/params/xtransformer/${dname}/${DS_model_name}.json - -# model_dir="${work_dir}/models_LF/xtransformer/${dname}/${model_name}/${feature_name}/XYstack" -model_dir="${work_dir}/models_LF/xtransformer/${dname}/${model_name}/${feature_name}/XYstack/downstream/${DS_model_name}/${P_topk}/${L_option}" -pretrain_dir="${work_dir}/models_LF/xtransformer/${dname}/${model_name}/${feature_name}/XYstack" -mkdir -m777 -p ${model_dir} - -X_trn=${work_dir}/dataset/${dname}/raw/X.trn.txt - -if [[ "$Use_A" == "true" ]]; then - echo Using true A for training - Xf_trn=${pretrain_dir}/X_true_${L_option}.npz -else - echo Not using true A for training - Xf_trn=${pretrain_dir}/X_trn_P${P_topk}${L_option}.npz -fi - -Y_trn=${work_dir}/dataset/${dname}/raw/Y.trn.npz - -X_tst=${work_dir}/dataset/${dname}/raw/X.tst.txt -Xf_tst=${pretrain_dir}/X_tst_P${P_topk}${L_option}.npz -Y_tst=${work_dir}/dataset/${dname}/raw/Y.tst.npz - -# ================ training ==================== -if [ -f "${model_dir}/train.log" ]; then - echo ${model_dir}/train.log exists, skip... -else -python3 -m pecos.xmc.xtransformer.train -t ${X_trn} -x ${Xf_trn} -y ${Y_trn} -m ${model_dir} --only-topk $topk\ - --params-path ${params_path} \ - |& tee ${model_dir}/train.log -fi -# ================ eval ======================== -if [ -f "${model_dir}/eval_tst.log" ]; then - echo ${model_dir}/eval_tst.log exists, skip... - cat ${model_dir}/eval_tst.log -else -python3 -m pecos.xmc.xtransformer.predict -t ${X_tst} -x ${Xf_tst} -m ${model_dir} --only-topk $topk\ - -o ${model_dir}/P.${topk}.npz \ - |& tee ${model_dir}/eval_tst.log - -echo "===Before reciprocal pair removal===" -python3 -m pecos.xmc.xlinear.evaluate -y ${Y_tst} -p ${model_dir}/P.${topk}.npz -k 10 \ - |& tee ${model_dir}/eval_tst.log -fi - -if [ -f "${model_dir}/Reciprocal_Removed_eval.log" ]; then - echo Result from ${model_dir}/Reciprocal_Removed_eval.log - cat ${model_dir}/Reciprocal_Removed_eval.log -else -echo "===After reciprocal pair removal===" -python3 -u evaluate.py \ - "./dataset/${dname}/trn_X_Y.txt" \ - "./dataset/${dname}/tst_X_Y.txt" \ - "${model_dir}/P.${topk}" ./dataset/${dname} \ - |& tee ${model_dir}/Reciprocal_Removed_eval.log -fi diff --git a/examples/pina/scripts/xtransformer-XYstack-raw.sh b/examples/pina/scripts/xtransformer-XYstack-raw.sh deleted file mode 100755 index e9c81966..00000000 --- a/examples/pina/scripts/xtransformer-XYstack-raw.sh +++ /dev/null @@ -1,68 +0,0 @@ -#================= inputs ===================== -model_name=$1 -feature_name=$2 -work_dir="." -dname=$3 -topk=20 - - -params_path=${work_dir}/scripts/params/xtransformer/${dname}/${model_name}.json - -model_dir="${work_dir}/models_LF/xtransformer/${dname}/${model_name}/${feature_name}/XYstack" -mkdir -m777 -p ${model_dir} - -X_all=${work_dir}/dataset/${dname}/raw/X_all.txt -X_trn=${work_dir}/dataset/${dname}/raw/X.trn.txt -X_tst=${work_dir}/dataset/${dname}/raw/X.tst.txt - -if [[ $feature_name == "BoW" ]] -then - echo $feature_name - feature_dir="${work_dir}/dataset/${dname}" - Xf_all=${feature_dir}/X_bow.all.npz - Xf_trn=${feature_dir}/X_bow.trn.npz - Xf_tst=${feature_dir}/X_bow.tst.npz -else - echo $feature_name - feature_dir="${work_dir}/dataset/${dname}/tfidf/${feature_name}/XYstack/normalized" - Xf_all=${feature_dir}/X.tfidf.all.npz - Xf_trn=${feature_dir}/X.tfidf.trn.npz - Xf_tst=${feature_dir}/X.tfidf.tst.npz -fi -Y_all=${work_dir}/dataset/${dname}/raw/Y_all.npz - -# ================ training ==================== -if [ -f "${model_dir}/train.log" ]; then - echo ${model_dir}/train.log exists, skip... -else -python3 -m pecos.xmc.xtransformer.train -t ${X_all} -x ${Xf_all} -y ${Y_all} -m ${model_dir} \ - --params-path ${params_path} \ - |& tee ${model_dir}/train.log -fi -# ================ eval ======================== -if [ -f "${model_dir}/eval_tst.log" ]; then - echo ${model_dir}/eval_tst.log exists, skip... -else -python3 -m pecos.xmc.xtransformer.predict -t ${X_all} -x ${Xf_all} -m ${model_dir} --only-topk $topk\ - -o ${model_dir}/P.${topk}.npz \ - |& tee ${model_dir}/eval_tst.log - -python3 -m pecos.xmc.xlinear.evaluate -y ${Y_all} -p ${model_dir}/P.${topk}.npz -k 10 \ - |& tee ${model_dir}/eval_tst.log -fi -# =============== get prediction ============== -if [ -f "${model_dir}/P.${topk}.trn.npz" ]; then - echo ${model_dir}/P.${topk}.trn.npz exists, skip... -else -python3 -m pecos.xmc.xtransformer.predict \ - -t ${X_trn} -x ${Xf_trn} -m ${model_dir} --only-topk $topk \ - -o ${model_dir}/P.${topk}.trn.npz -fi - -if [ -f "${model_dir}/P.${topk}.tst.npz" ]; then - echo ${model_dir}/P.${topk}.tst.npz exists, skip... -else -python3 -m pecos.xmc.xtransformer.predict \ - -t ${X_tst} -x ${Xf_tst} -m ${model_dir} --only-topk $topk \ - -o ${model_dir}/P.${topk}.tst.npz -fi diff --git a/examples/qp2q/README.md b/examples/qp2q/README.md deleted file mode 100644 index f6991060..00000000 --- a/examples/qp2q/README.md +++ /dev/null @@ -1,102 +0,0 @@ -# Session-Aware Query-Autocompletion using eXtreme Multi-Label Ranking, KDD 2021 - -This folder contains code to train session-aware query-autocompletion models and reproduce experiments -in ["Session-Aware Query-Autocompletion using eXtreme Multi-Label Ranking, KDD 2021"](https://arxiv.org/abs/2012.07654). - -## Getting Started -* Clone the repository and enter `examples/qp2q` directory. -* First create a [virtual environment](https://docs.python.org/3/library/venv.html) and then install dependencies -by running the following command: -```bash -pip install -r requirements.txt - -# NOTE: The original nltk version used in the experiment -# CAUTION: nltk<=3.6.3 is known to contain an Inefficient Regular Expression and is vulnerable to regular expression denial of service attacks -# Details: https://github.com/advisories/GHSA-2ww3-fxvq-293j -pip install nltk==3.4.5 -``` -If you're unfamiliar with Python virtual environments, check out the -[user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). - -## Pre-procesing data -* Download AOL Search Logs data from [here](http://www.cim.mcgill.ca/~dudek/206/Logs/AOL-user-ct-collection/). -* Extract files into `RAW_DATA_DIR` folder. This folder should contain 10 files with name template: ` user-ct-test-collection-**.txt`. -* Run the following command to pre-process the data and to create train/test/dev splits (as used in this paper). - - ```bash - cd examples/qp2q - python utils/process_aol_dataset_from_orig_aol_files.py --data_dir --out_dir - ``` -* The processed data is present in `DATA_DIR` folder, which contains three folders: `train`, `test`, and `dev`. -Each folder contains a `.json` file. Each line in the file contains a data point stored in JSON format. - - -## Training Models - -* First run the setup script in `qp2q/bin` folder - - ```bash - cd examples/qp2q - source bin/setup.sh - ``` -* Train a model - - ```python models/train_model.py --config ``` - - Config files for various configurations of the models from [Table 2](https://arxiv.org/abs/2012.07654) in the paper - are present in `qp2q/config` folder. - - Please configure `fdir` parameter in config files to point to directory containing training data. By default, - training data is assumed to present in `data/aol/train` directory where `data` directory is assumed to be present - in the same folder as `qp2q` folder. - - Trained models will be stored in folders under `results` folder. - - The `data`, `results` and `qp2q` folders are assumed to be present in the same directory, with the data - directory organized as illustrated below. - ``` - ├── qp2q - ├── results - ├── data - | ├── aol - | ├── train - | ├── train.json - | ├── test - | ├── test.json - | ├── val - | ├── val.json - ``` - - -## Evaluation - -* To generate query suggestions and evaluate the predictions for proposed models: - - ```bash - python eval/run_eval.py --gt --out_dir --model_dir - ``` -* To generate query suggestions and evaluate the predictions for Most-Frequent-Query (MFQ) baseline: - - First generate a dictionary mapping each prefix to a list of top-k (k=10) query suggestions and then run eval. - ```bash - python utils/create_pref_to_top_k_suggestions_dict.py --k 10 --fdir --out_fname - python eval/run_mfq_eval.py --gt --out_dir --topk_file - ``` - - -## Citation - -If you find the code useful, please consider citing our paper. - -* [Session-Aware Query Auto-completion using Extreme Multi-label Ranking (Yadav et al., KDD 2021)](https://arxiv.org/pdf/2012.07654.pdf) [[bib]](../../bibtex/yadav2021session.bib) - -```bibtex -@inproceedings{Yadav2021session, -author = {Yadav, Nishant and Sen, Rajat and Hill, Daniel N. and Mazumdar, Arya and Dhillon, Inderjit S.}, -title = {Session-Aware Query Auto-Completion Using Extreme Multi-Label Ranking}, -year = {2021}, -isbn = {9781450383325}, -publisher = {Association for Computing Machinery}, -address = {New York, NY, USA}, -url = {https://doi.org/10.1145/3447548.3467087}, -doi = {10.1145/3447548.3467087}, -booktitle = {Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery \& Data Mining}, -pages = {3835–3844}, -numpages = {10}, -keywords = {session-aware, extreme multi-label ranking, multi-label, auto-complete}, -location = {Virtual Event, Singapore}, -series = {KDD '21} -} -``` - diff --git a/examples/qp2q/__init__.py b/examples/qp2q/__init__.py deleted file mode 100644 index d933a529..00000000 --- a/examples/qp2q/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# Init File for the package diff --git a/examples/qp2q/bin/setup.sh b/examples/qp2q/bin/setup.sh deleted file mode 100644 index 48c173cb..00000000 --- a/examples/qp2q/bin/setup.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/usr/bin/env bash - -export CURR_ROOT=`pwd` -export PYTHONPATH=$CURR_ROOT/..:$PYTHONPATH \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_0.json b/examples/qp2q/config/config_qp2q_0.json deleted file mode 100644 index 9b318291..00000000 --- a/examples/qp2q/config/config_qp2q_0.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_0", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "00", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "", - "prefix_vect_data": "", - - "use_label_feat": false, - "label_vectorizer": "", - "label_vect_data": "", - - "indexer_type": "hierarchicalkmeans", - "depth": -1 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_1.json b/examples/qp2q/config/config_qp2q_1.json deleted file mode 100644 index 511c569a..00000000 --- a/examples/qp2q/config/config_qp2q_1.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_1", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "01", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "", - "prefix_vect_data": "", - - "use_label_feat": true, - "label_vectorizer": "c-tfidf", - "label_vect_data": "train_data", - - "indexer_type": "hierarchicalkmeans", - "depth": -1 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_10.json b/examples/qp2q/config/config_qp2q_10.json deleted file mode 100644 index d476ae0e..00000000 --- a/examples/qp2q/config/config_qp2q_10.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_10", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "10", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": false, - "label_vectorizer": "", - "label_vect_data": "", - - "indexer_type": "trieindexer", - "depth": 16 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_11.json b/examples/qp2q/config/config_qp2q_11.json deleted file mode 100644 index 839ea967..00000000 --- a/examples/qp2q/config/config_qp2q_11.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_11", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "11", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "poswgtd_c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": true, - "label_vectorizer": "use_prefix_vectorizer", - "label_vect_data": "", - - "indexer_type": "hkmeans_w_mlc", - "depth": 5 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_2.json b/examples/qp2q/config/config_qp2q_2.json deleted file mode 100644 index 1c659703..00000000 --- a/examples/qp2q/config/config_qp2q_2.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_2", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "02", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": false, - "label_vectorizer": "", - "label_vect_data": "", - - "indexer_type": "hierarchicalkmeans", - "depth": -1 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_3.json b/examples/qp2q/config/config_qp2q_3.json deleted file mode 100644 index e68a6b29..00000000 --- a/examples/qp2q/config/config_qp2q_3.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_3", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "03", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": true, - "label_vectorizer": "use_prefix_vectorizer", - "label_vect_data": "", - - "indexer_type": "hierarchicalkmeans", - "depth": -1 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_4.json b/examples/qp2q/config/config_qp2q_4.json deleted file mode 100644 index ec3f1da2..00000000 --- a/examples/qp2q/config/config_qp2q_4.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_4", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "04", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "poswgtd_c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": false, - "label_vectorizer": "", - "label_vect_data": "", - - "indexer_type": "hierarchicalkmeans", - "depth": -1 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_5.json b/examples/qp2q/config/config_qp2q_5.json deleted file mode 100644 index c8ff8df2..00000000 --- a/examples/qp2q/config/config_qp2q_5.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_5", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "05", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "poswgtd_c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": true, - "label_vectorizer": "use_prefix_vectorizer", - "label_vect_data": "", - - "indexer_type": "hierarchicalkmeans", - "depth": -1 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_6.json b/examples/qp2q/config/config_qp2q_6.json deleted file mode 100644 index 9cbdd9b1..00000000 --- a/examples/qp2q/config/config_qp2q_6.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_6", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "06", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "poswgtd_c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": true, - "label_vectorizer": "use_prefix_vectorizer", - "label_vect_data": "", - - "indexer_type": "hybridindexer", - "depth": 1 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_7.json b/examples/qp2q/config/config_qp2q_7.json deleted file mode 100644 index 2f3ed18b..00000000 --- a/examples/qp2q/config/config_qp2q_7.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_7", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "07", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "poswgtd_c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": true, - "label_vectorizer": "use_prefix_vectorizer", - "label_vect_data": "", - - "indexer_type": "hybridindexer", - "depth": 2 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_8.json b/examples/qp2q/config/config_qp2q_8.json deleted file mode 100644 index 3318f768..00000000 --- a/examples/qp2q/config/config_qp2q_8.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_8", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "08", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "poswgtd_c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": true, - "label_vectorizer": "use_prefix_vectorizer", - "label_vect_data": "", - - "indexer_type": "hybridindexer", - "depth": 3 -} \ No newline at end of file diff --git a/examples/qp2q/config/config_qp2q_9.json b/examples/qp2q/config/config_qp2q_9.json deleted file mode 100644 index 1f49ff9c..00000000 --- a/examples/qp2q/config/config_qp2q_9.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "exp_id": "config_9", - "seed": 0, - - "data_type": "aol", - "f_compressed": false, - "misc": "09", - - "query_prefix_delimiter": "<@@>", - - "pref_vectorizer": "poswgtd_c-tfidf", - "prefix_vect_data": "train_data", - - "use_label_feat": false, - "label_vectorizer": "", - "label_vect_data": "", - - "indexer_type": "trieindexer", - "depth": 16 -} \ No newline at end of file diff --git a/examples/qp2q/eval/__init__.py b/examples/qp2q/eval/__init__.py deleted file mode 100644 index d933a529..00000000 --- a/examples/qp2q/eval/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# Init File for the package diff --git a/examples/qp2q/eval/eval_helper.py b/examples/qp2q/eval/eval_helper.py deleted file mode 100644 index 80c83e46..00000000 --- a/examples/qp2q/eval/eval_helper.py +++ /dev/null @@ -1,213 +0,0 @@ -import gc -import json -import logging -from abc import ABCMeta, abstractmethod -from typing import Any, ClassVar, Dict - -from qp2q.models import pecosq2q - -LOGGER = logging.getLogger(__name__) - - -class WorkerMeta(ABCMeta): - """Collects all eval_helpers in cls.subtypes automatically, - when AbstractWorker is inherited. - - Notes: - ----- - This is a meta-class: - https://docs.python.org/3/library/abc.html#abc.ABCMeta - """ - - subtypes: ClassVar[ - Dict[str, Any] - ] = {} # static variable for storing classnames->class for eval_helpers - - def __new__(cls, name, bases, attr): - cls = super().__new__(cls, name, bases, attr) - if cls.__name__ != "AbstractWorker": - WorkerMeta.subtypes[cls.__name__] = cls - return cls - - -class AbstractWorker(metaclass=WorkerMeta): - """ - An abstract class for different evaluators for different types of models. - - Any class that inherits AbstractWorker will be available in EVAL_HELPER_DICT, - with key as class name and value as the class itself. - """ - - def __init__(self, topk): - """ - Initialize AbstractWorker class. - - Parameters - ---------- - topk : int - number of suggestions to return - """ - if (not topk) or (not isinstance(topk, int)) or (topk < 0): - raise TypeError("`topk` should be a positive integer") - - self.topk = topk - - @abstractmethod - def get_suggestions(self, **kwargs): - """ - Abstract method to get suggestions. - - Not Implemented - - """ - raise NotImplementedError() - - @staticmethod - def _return_results(suggestions_with_score): - """ - Return topk results. - - Parameters - ---------- - suggestions_with_score : iterator - iterator of tuples where first item suggestion - and the second item is its score - Returns - ------- - list - A list of suggestions - - """ - return [suggestion_with_score[0] for suggestion_with_score in suggestions_with_score] - - -class PecosSuggester(AbstractWorker): - """Generate Suggestions using PECOS Auto-complete suggestion model""" - - def __init__( - self, - model_path, - beam_size, - topk, - ): - """ - Initialize suggestor. - - Parameters - ---------- - model_path: str - path containing saved model file(s) - beam_size: int - beam_size to use in generating suggestions with PECOS models - topk: int - number of suggestions to return - - """ - super(PecosSuggester, self).__init__(topk=topk) - self.beam_size = beam_size - self.pecos_model = pecosq2q.PecosQP2QModel.load(model_path, realtime=True) - gc.collect() - - def get_suggestions(self, request): - """ - Return next query suggestions - - Parameters - ---------- - request: dict with keys = prefix and prev_query - - Returns - ------- - list - list of next query suggestions - - """ - prefix = request["prefix"] - prev_query = request["prev_query"] - responses = self.pecos_model.get_suggestions( - prev_query=prev_query, - prefix=prefix, - topk=self.topk, - beam_size=self.beam_size, - ) - - responses = [(r[0], r[1]) for r in responses[: self.topk]] - return self._return_results(responses) - - -class PrefFreqSuggester(AbstractWorker): - """ - Generate Suggestions by returning most frequent queries matching the prefix - It uses a dictionary to retrieve pre-computed list of queries matching a prefix - If there are not enough queries matching the prefix then it fetches queries - matching a smaller prefix (removing one character at a time). - Queries retrieve using smaller prefix are appended to list of queries that exactly match the prefix. - """ - - def __init__(self, model_path, topk): - """ - Initialize suggestor. - - Parameters - ---------- - model_path: str - path containing saved model file - topk: int - number of suggestions to return - """ - super(PrefFreqSuggester, self).__init__(topk=topk) - LOGGER.info("Loading model from file") - with open(model_path, "r") as f: - self.pref_to_topk = json.load(f) - LOGGER.info("Finished loading model from file") - - def _get_suggestions(self, prefix): - """ - - Parameters - ---------- - prefix - - Returns a list of 2-tuple where first value is label and second is its frequency - ------- - - """ - if prefix in self.pref_to_topk: - if len(self.pref_to_topk[prefix]) >= self.topk: - return self.pref_to_topk[prefix][: self.topk] - elif len(prefix) > 0: - more_labels = self._get_suggestions(prefix=prefix[:-1]) - perfect_match_labels, _ = zip(*self.pref_to_topk[prefix]) - perfect_match_labels = set(perfect_match_labels) - all_labels = self.pref_to_topk[prefix] + [ - (l, f) for (l, f) in more_labels[: self.topk] if l not in perfect_match_labels - ] - return all_labels[: self.topk] - else: - return self.pref_to_topk[prefix][: self.topk] - elif len(prefix) > 0: - """ - In case prefix does not have top-k suggestions already computed, and prefix is greater than 1 char, - then suggest using prefix[:-1], i.e after removing 1 char from prefix - """ - return self._get_suggestions(prefix=prefix[:-1]) - else: - return [] - - def get_suggestions(self, request): - """ - Return next query suggestions - - Parameters - ---------- - request: dict with keys = prefix and prev_query - - Returns - ------- - list - list of next query suggestions - """ - prefix = request["prefix"] - responses = self._get_suggestions(prefix=prefix) - - return self._return_results(responses) diff --git a/examples/qp2q/eval/eval_pred_data.py b/examples/qp2q/eval/eval_pred_data.py deleted file mode 100644 index b52207a1..00000000 --- a/examples/qp2q/eval/eval_pred_data.py +++ /dev/null @@ -1,95 +0,0 @@ -import argparse -import logging -import numpy as np -from pathlib import Path -import json -import sys -import os -import csv -from nltk.translate import bleu_score -from nltk.translate.bleu_score import SmoothingFunction -from tqdm import tqdm - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -def calc_mrr_bleu(gt_data_iterator, pred_data_iterator): - - all_mrr = [] - all_bleu = [] - for j, (gt_line, preds) in tqdm(enumerate(zip(gt_data_iterator, pred_data_iterator))): - gt_dict = json.loads(gt_line) - gt_label = gt_dict["next_query"].lower() # ground truth next_query - preds = [p for p in json.loads(preds.lower())] - mrr = 0.0 - for i, curr_pred in enumerate(preds): - if gt_label == curr_pred: - mrr = 1.0 / (i + 1) - break - - all_mrr.append(mrr) - - wgtd_bleu_score = 0.0 - normalizer = 0.0 - for i, curr_pred in enumerate(preds): - wgtd_bleu_score += ( - bleu_score.sentence_bleu( - [gt_label.split()], - curr_pred.split(), - smoothing_function=SmoothingFunction().method1, - ) - / (i + 1) - ) - normalizer += 1.0 / (i + 1) - wgtd_bleu_score = wgtd_bleu_score / normalizer if normalizer > 0 else wgtd_bleu_score - all_bleu.append(wgtd_bleu_score) - - return {"mrr": np.mean(all_mrr), "bleu": np.mean(all_bleu), "num_samples": len(all_mrr)} - - -def calc_mrr_bleu_from_file(gt_file, pred_file): - with open(gt_file, "r") as orig_gt_reader, open(pred_file, "r") as pred_reader: - return calc_mrr_bleu(gt_data_iterator=orig_gt_reader, pred_data_iterator=pred_reader) - - -def main(): - parser = argparse.ArgumentParser(description="Eval predictions given gt data") - parser.add_argument("--gt_file", type=str, required=True, help="gt data file") - parser.add_argument("--pred_file", type=str, required=True, help="pred data file") - parser.add_argument("--out_file", type=str, required=True, help="File to save results in") - - args = parser.parse_args() - _gt_file = args.gt_file - _pred_file = args.pred_file - _out_file = args.out_file - - out_dir = os.path.dirname(_out_file) - Path(out_dir).mkdir(exist_ok=True, parents=True) - - res = calc_mrr_bleu_from_file(gt_file=_gt_file, pred_file=_pred_file) - LOGGER.info("Eval Result") - LOGGER.info(json.dumps(res)) - - res["gt_file"] = _gt_file - res["pred_file"] = _pred_file - - _out_file = _out_file + ".csv" - if os.path.exists(_out_file): - with open(_out_file, "a") as csvfile: - writer = csv.DictWriter(csvfile, fieldnames=res.keys()) - writer.writerow(res) - else: - with open(_out_file, "w") as csvfile: - writer = csv.DictWriter(csvfile, fieldnames=res.keys()) - writer.writeheader() - writer.writerow(res) - - -if __name__ == "__main__": - main() diff --git a/examples/qp2q/eval/eval_pred_latency.py b/examples/qp2q/eval/eval_pred_latency.py deleted file mode 100644 index 35c7560d..00000000 --- a/examples/qp2q/eval/eval_pred_latency.py +++ /dev/null @@ -1,162 +0,0 @@ -import os -import sys -import time -import json -import random -import logging -import argparse -import numpy as np -import os.path as path - -from qp2q.eval.gen_pred_data import get_model - -logger = logging.getLogger(__name__) - - -def _get_inference_time(model, sample): - """Computes inference time for one call to get_suggestions.""" - start = time.time() - model.get_suggestions(sample) - end = time.time() - return end - start - - -def _benchmark_model(model, gt_data_file, latency_data_file, num_samples, warmup_samples): - """Benchmark latencies for a given model. - - Parameters: - ---------- - model: obj - model file or suggester - gt_data_file: str - path to ground truth data file - latency_data_file: str - path where the results are saved - num_samples: int - number of samples on which latency is evaluated. - warmup_samples: int - number of samples to be used for warmup. - """ - logger.info("Reading groundtruth file.") - with open(gt_data_file, "r") as fp: - input_data = [json.loads(line) for line in fp] - samples = random.sample(input_data, num_samples + warmup_samples) - result = [_get_inference_time(model, sample) * 1000 for sample in samples] - result = result[warmup_samples:] - latency_dict = { - "min": np.min(result), - "max": np.max(result), - "mean": np.mean(result), - "p50": np.median(result), - "p75": np.percentile(result, q=75), - "p99": np.percentile(result, q=99), - } - logger.info("==== latency results in milli-seconds====") - logger.info(latency_dict) - with open(latency_data_file, "w") as fp: - json.dump(latency_dict, fp) - logger.info("Saved at {}".format(latency_data_file)) - - -def eval_pred_latency(config_dict, pred_data_path, gt_data_file, num_samples, warmup_samples): - """ - This method takes in model configurations to generate prediction results - for given sample data. - Latency results are then saved under save_dir - * /._latency_data.json - * contains lists of query suggestions - The model configs will be dumped into the following file: - * /_latency_config.json - - Parameters - ---------- - config_dict : dict - This dict contains the model and inference specific params - pred_data_path: str - Path to result file - gt_data_file : str - Ground-truth data file - num_samples: int - Number of samples to use for evaluating latency - warmup_samples: int - Number of samples to be used for warmup and not in latency calculations", - """ - save_dir = os.path.dirname(pred_data_path) - os.makedirs(path.abspath((save_dir)), exist_ok=True) - - # loading models - logger.info("Loading models") - model = get_model(config_dict) - if not model: - raise Exception("No models could be loaded") - - # save the eval configs (if not empty) for future reference - _out = path.join(path.abspath(save_dir), config_dict["name"] + "_latency_config.json") - json.dump(config_dict, open(_out, "w")) - - latency_data_file = pred_data_path.replace("pred_data", "latency_data") - _benchmark_model( - model=model, - gt_data_file=gt_data_file, - latency_data_file=latency_data_file, - num_samples=num_samples, - warmup_samples=warmup_samples, - ) - - logger.info("Latency Eval finished") - - -def main(argv): - - parser = argparse.ArgumentParser("Evaluate inference latency of models") - parser.add_argument( - "--config_file", - type=str, - required=True, - help="Path of the config json to compute predictions", - ) - parser.add_argument( - "--save_dir", type=str, required=True, help="Directory to save prediction data" - ) - parser.add_argument( - "--gt_file", - type=str, - required=True, - help="Path to datafile containing ground-truth data", - ) - parser.add_argument( - "--num_samples", - type=int, - required=False, - default=10000, - help="Number of samples to be used for evaluating latency", - ) - parser.add_argument( - "--warmup_samples", - type=int, - required=False, - default=100, - help="Number of samples to be used for warmup and not in latency calculations", - ) - - args = parser.parse_args(argv) - config_file = args.config_file - save_dir = args.save_dir - gt_file = args.gt_file - num_samples = args.num_samples - warmup_samples = args.warmup_samples - - with open(config_file, "r") as f: - config_dict = json.load(f) - eval_pred_latency( - config_dict=config_dict, - save_dir=save_dir, - gt_data_file=gt_file, - num_samples=num_samples, - warmup_samples=warmup_samples, - ) - - -if __name__ == "__main__": - logger.info(sys.argv) - main(sys.argv) diff --git a/examples/qp2q/eval/gen_pred_data.py b/examples/qp2q/eval/gen_pred_data.py deleted file mode 100644 index f12ea973..00000000 --- a/examples/qp2q/eval/gen_pred_data.py +++ /dev/null @@ -1,100 +0,0 @@ -import os -import sys -import json -import logging -import os.path as path -import argparse - -from tqdm import tqdm -from qp2q.eval.eval_helper import WorkerMeta - -logger = logging.getLogger(__name__) - - -def get_model(eval_config): - model_info = WorkerMeta.subtypes - logger.info("Loading {0} for {1}".format(eval_config["driver"], eval_config["name"])) - - model_klass = model_info[eval_config["driver"]] - model = model_klass(**eval_config["args"]) if "args" in eval_config else model_klass() - - return model - - -def generate_predictions(config_dict, pred_data_path, gt_data_file): - """ - This method takes in model configurations to generate prediction results - for given sample data. The prediction results are then saved under save_dir - * /.pred_data - * contains lists of query suggestions - The model configs will be dumped into the following file: - * /.json - - Parameters - ---------- - config_dict : dict - This dict contains the model and inference specific params - pred_data_path: Name of output file - gt_data_file : Ground-truth data - """ - save_dir = os.path.dirname(pred_data_path) - os.makedirs(path.abspath(save_dir), exist_ok=True) - - # loading samples - logger.info("Loading samples") - gt_data = [] - with open(gt_data_file) as reader: - gt_data = [json.loads(line) for line in reader] - - # loading models - logger.info("Loading models") - model = get_model(config_dict) - if not model: - raise Exception("No models could be loaded") - - # save the eval configs (if not empty) for future reference - _out = path.join(path.abspath(save_dir), config_dict["name"] + "_config.json") - json.dump(config_dict, open(_out, "w")) - - logger.info("Beginning eval") - with open(pred_data_path, "w") as pred_file: - for curr_gt_data_item in tqdm(gt_data): - # Getting query suggestions from models - next_query_suggestions = model.get_suggestions(curr_gt_data_item) - single_line = json.dumps(next_query_suggestions) - pred_file.write(single_line + "\n") - - logger.info("Eval finished") - - -def main(argv): - - parser = argparse.ArgumentParser("Generate next query predictions") - parser.add_argument( - "--config_file", - type=str, - required=True, - help="Path of the config json to compute predictions", - ) - parser.add_argument( - "--save_dir", type=str, required=True, help="Directory to save prediction data" - ) - parser.add_argument( - "--gt_file", - type=str, - required=True, - help="Path to datafile containing ground-truth data", - ) - args = parser.parse_args(argv) - config_file = args.config_file - save_dir = args.save_dir - gt_file = args.gt_file - - with open(config_file, "r") as f: - config_dict = json.load(f) - generate_predictions(config_dict=config_dict, save_dir=save_dir, gt_data_file=gt_file) - - -if __name__ == "__main__": - logger.info(sys.argv) - main(sys.argv) diff --git a/examples/qp2q/eval/run_eval.py b/examples/qp2q/eval/run_eval.py deleted file mode 100644 index 1bf17da1..00000000 --- a/examples/qp2q/eval/run_eval.py +++ /dev/null @@ -1,111 +0,0 @@ -import os -import sys -import logging -import argparse -from pathlib import Path - -from qp2q.eval.gen_pred_data import generate_predictions -from qp2q.eval.eval_pred_latency import eval_pred_latency - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -def eval_helper(model_path, gt_file, gen_pred_data, eval_latency, out_dir): - - LOGGER.info("Param: GT_FILE={}".format(gt_file)) - LOGGER.info("Param: MODEL_PATH{}".format(model_path)) - LOGGER.info("Param: GEN_PRED_DATA={}".format(gen_pred_data)) - LOGGER.info("Param: EVAL_LATENCY={}".format(eval_latency)) - - gt_filename = gt_file.split("/")[-1] - pred_data_path = f"{out_dir}/{gt_filename}.pred_data" - - LOGGER.info(f"Evaluating model at : {model_path} \n") - - # Create config file for evaluating using PECOS Model - config_dict = { - "name": gt_filename, - "driver": "PecosSuggester", - "args": {"model_path": model_path, "beam_size": 10, "topk": 10}, - } - - if gen_pred_data: - LOGGER.info( - "\n Generating data for for model : {m} in res dir :{o} using ground-truth at {g} \n".format( - m=model_path, o=out_dir, g=gt_file - ) - ) - generate_predictions( - config_dict=config_dict, pred_data_path=pred_data_path, gt_data_file=gt_file - ) - - LOGGER.info( - "\n Computing metrics for model : {m} in res dir :{o} using ground-truth at {g} \n".format( - m=model_path, o=out_dir, g=gt_file - ) - ) - command = f"python eval/eval_pred_data.py --out_file {out_dir}/eval_results.json --pred_file {pred_data_path} --gt_file {gt_file} " - LOGGER.info(command) - os.system(command) - - if eval_latency: - LOGGER.info("Evaluating latency ") - eval_pred_latency( - config_dict=config_dict, - gt_data_file=gt_file, - pred_data_path=pred_data_path, - num_samples=100000, - warmup_samples=1000, - ) - - -def main(): - parser = argparse.ArgumentParser(description="Run evaluation using a trained model \n ") - parser.add_argument( - "--model_path", - required=True, - type=str, - help="Path to model dir that stores actual parameters of the model.", - ) - parser.add_argument("--gt", required=True, type=str, help="Path to file with ground-truth data") - parser.add_argument("--out_dir", required=True, type=str, help="Directory to save eval result") - parser.add_argument( - "--gen_pred_data", default=1, type=int, help="Whether to generate pred data or not" - ) - parser.add_argument( - "--eval_latency", - default=0, - type=int, - help="Whether to evaluate latency for generation or not", - ) - - args = parser.parse_args() - model_path = args.model_path - out_dir = args.out_dir - gt_file = args.gt - gen_pred_data = bool(args.gen_pred_data) - eval_latency = bool(args.eval_latency) - - Path(out_dir).mkdir(exist_ok=True, parents=True) - LOGGER.info("Beginning eval") - - if os.path.isfile(gt_file): - eval_helper( - model_path=model_path, - gt_file=gt_file, - gen_pred_data=gen_pred_data, - eval_latency=eval_latency, - out_dir=out_dir, - ) - else: - LOGGER.info("Invalid gt file = {}".format(gt_file)) - - -if __name__ == "__main__": - main() diff --git a/examples/qp2q/eval/run_mfq_eval.py b/examples/qp2q/eval/run_mfq_eval.py deleted file mode 100644 index 5a81b3fe..00000000 --- a/examples/qp2q/eval/run_mfq_eval.py +++ /dev/null @@ -1,101 +0,0 @@ -import os -import sys -import logging -import argparse - -from qp2q.eval.gen_pred_data import generate_predictions -from qp2q.eval.eval_pred_latency import eval_pred_latency - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -def eval_helper(topk_file, gt_file, out_dir, gen_pred_data, eval_latency): - - LOGGER.info("Param: GT_FILE={}".format(gt_file)) - LOGGER.info("Param: TOPK_FILE={}".format(topk_file)) - LOGGER.info("Param: GEN_PRED_DATA={}".format(gen_pred_data)) - LOGGER.info("Param: EVAL_LATENCY={}".format(eval_latency)) - - gt_filename = gt_file.split("/")[-1] - pred_data_path = f"{out_dir}/{gt_filename}.pred_data" - - # create eval config for evaluating with Most-Frequent-Query Baseline - config_dict = { - "name": "MFQ", - "driver": "PrefFreqSuggester", - "args": { - "model_path": topk_file, - "topk": 10, - }, - } - - if gen_pred_data: - generate_predictions( - config_dict=config_dict, pred_data_path=pred_data_path, gt_data_file=gt_file - ) - - LOGGER.info("\n Computing eval metrics ") - command = f"python eval/eval_pred_data.py --out_file {out_dir}/eval_results.json --pred_file {pred_data_path} --gt_file {gt_file} " - LOGGER.info(command) - os.system(command) - - if eval_latency: - LOGGER.info("Evaluating latency ") - eval_pred_latency( - config_dict=config_dict, - gt_data_file=gt_file, - pred_data_path=pred_data_path, - num_samples=100000, - warmup_samples=1000, - ) - - -def main(): - parser = argparse.ArgumentParser(description="Run evaluation using a trained model \n ") - parser.add_argument( - "--topk_file", - required=True, - type=str, - help="Dictionary file mapping each prefix to top-k most frequency queries matching the prefix", - ) - parser.add_argument("--gt", required=True, type=str, help="Path to file with ground-truth data") - parser.add_argument("--out_dir", required=True, type=str, help="Directory to save eval results") - parser.add_argument( - "--gen_pred_data", default=1, type=int, help="Whether to generate pred data or not" - ) - parser.add_argument( - "--eval_latency", - default=0, - type=int, - help="Whether to evaluate latency for generation or not", - ) - - args = parser.parse_args() - topk_file = args.topk_file - out_dir = args.out_dir - gt_file = args.gt - gen_pred_data = bool(args.gen_pred_data) - eval_latency = bool(args.eval_latency) - - LOGGER.info("Beginning eval") - - if os.path.isfile(gt_file): - eval_helper( - topk_file=topk_file, - gt_file=gt_file, - gen_pred_data=gen_pred_data, - eval_latency=eval_latency, - out_dir=out_dir, - ) - else: - LOGGER.info("Invalid gt file = {}".format(gt_file)) - - -if __name__ == "__main__": - main() diff --git a/examples/qp2q/models/__init__.py b/examples/qp2q/models/__init__.py deleted file mode 100644 index d933a529..00000000 --- a/examples/qp2q/models/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# Init File for the package diff --git a/examples/qp2q/models/indices.py b/examples/qp2q/models/indices.py deleted file mode 100644 index 9b298bd4..00000000 --- a/examples/qp2q/models/indices.py +++ /dev/null @@ -1,901 +0,0 @@ -""" -This module contains contains indexing utils for training category -specific models, i.e user supplied top-level clustering of labels. -""" -import sys -import pygtrie -import logging -import numpy as np -import scipy as sp -import scipy.sparse as smat -import multiprocessing as mp -from itertools import chain, repeat -import sklearn.preprocessing as skprep -from sklearn.preprocessing import normalize - -from pecos.xmc import Indexer -from pecos.utils import cluster_util -from pecos.xmc.base import HierarchicalKMeans - - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -class TrieWrapper(pygtrie.CharTrie): - """ - Wrapper around pygtrie.CharTrie for purpose of creating cluster matrix chains - """ - - def get_children(self): - if isinstance(self._root, pygtrie._Node): - if isinstance(self._root.children, pygtrie._Children): - # child_char is char at root for corresponding trie denoted by child_root - for child_char, child_root in self._root.children.iteritems(): - child_trie = TrieWrapper() - child_trie._root = child_root - assert isinstance(child_trie._root, pygtrie._Node) - child_trie._sorted = self._sorted - yield child_char, child_trie - elif isinstance(self._root.children, pygtrie._OneChild): - child_trie = TrieWrapper() - child_trie._root = self._root.children.node - child_char = self._root.children.step - assert isinstance(child_trie._root, pygtrie._Node) - child_trie._sorted = self._sorted - yield child_char, child_trie - else: - raise Exception( - "Not handled child {} of type {}".format( - self._root.children, type(self._root.children) - ) - ) - else: - raise Exception( - "Not handled child {} of type {}".format( - self._root.children, type(self._root.children) - ) - ) - - @property - def n_children(self): - if isinstance(self._root, pygtrie._Node): - return len(self._root.children) - else: - raise Exception("Not handled root {} of type {}".format(self._root, type(self._root))) - - @property - def is_leaf(self): - return self.n_children == 0 - - @property - def n_keys(self): - return len(self.keys()) - - @property - def n_leaves(self): - - if self.is_leaf: - return 1 - else: - n_leaves = 0 - for child_char, child_trie in self.get_children(): - n_leaves += child_trie.n_leaves - return n_leaves - - def build_cluster_chain(self, depth): - - cluster_chain = self._build_sparse_cluster_chain_helper(depth=depth) - - assert len(cluster_chain) == depth + 1 - # Merge all child cluster chains level wise - new_chain = [] - for curr_level in range(depth + 1): - mats_to_merge = cluster_chain[curr_level] - - row_offsets = np.concatenate(([0], np.cumsum([mat.shape[0] for mat in mats_to_merge]))) - col_offsets = np.concatenate(([0], np.cumsum([mat.shape[1] for mat in mats_to_merge]))) - total_n_rows = row_offsets[-1] - total_n_cols = col_offsets[-1] - - all_rows = [mat.row + row_offset for mat, row_offset in zip(mats_to_merge, row_offsets)] - all_cols = [mat.col + col_offset for mat, col_offset in zip(mats_to_merge, col_offsets)] - - new_row_idxs = np.concatenate(all_rows) - new_col_idxs = np.concatenate(all_cols) - - assert len(new_row_idxs) == len(new_col_idxs) - new_data = np.ones((len(new_row_idxs))) - - new_mat = smat.csr_matrix( - (new_data, (new_row_idxs, new_col_idxs)), - shape=(total_n_rows, total_n_cols), - dtype=sp.float32, - ) - new_chain.append(new_mat) - - cluster_chain = cluster_util.ClusterChain(new_chain) - return cluster_chain - - def _build_sparse_cluster_chain_helper(self, depth): - """ - This version builds a chain of SPARSE cluster matrices in scipy.sparse.coo_matrix format. - Given a trie, builds cluster chain of length (depth + 1). - Basically, encode clustering induced by a trie as a chain of sparse matrices, - where sparse_matrix[i] is of shape [n_nodes_at_level_i_plus_1, n_nodes_at_level_i] - and leaf_nodes are assumed to be present at level = depth - - While creating cluster chain, the clustering induced by trie is collapsed after given depth is reached - i.e. after reaching level = depth, the subtree under nodes at level = depth are collapsed into a leaf node - for purpose of creating a cluster chain - - If given depth is such that a path in trie is shorted than given depth, then the cluster chain is created - as if the path is extended using a branching factor of 1 upto the required depth - - We assume level 0 contains just 1 node which is the root node so, if depth == 0, then cluster chain - has just 1 matrix of shape (n_labels, 1) - - - Parameters - ---------- - depth: int - Depth of trie to consider when building cluster chain i.e. depth at which we assume leaf nodes are present - This depth can be smaller or greater than depth of trie as long as it is not negative - - Returns - ------- - List of list of scipy sparse matrices in coo format - Each list of scipy sparse matrices correspond to sparse matrices for nodes at a particular level of cluster chain - """ - assert depth >= 0 - - if depth == 0: - # Creating chain for node because of depth constraint - new_chain = [[smat.coo_matrix(np.ones((self.n_keys, 1)))]] - elif self.is_leaf: - # Creating chain for a leaf node - new_chain = [[smat.coo_matrix(np.ones((self.n_keys, 1)))]] * (depth + 1) - else: - all_cluster_chains = [] - - if "" in self: # A substring ends at this node - par_child_smat = smat.coo_matrix( - np.ones((self.n_children + 1, 1)) - ) # 1 for dummy child - # A substring ends at this node so create a dummy node for that - dummy_cluster_chain = [[smat.coo_matrix([[1]])]] * (depth) - all_cluster_chains.append(dummy_cluster_chain) - else: - par_child_smat = smat.coo_matrix(np.ones((self.n_children, 1))) - - for child_char, child_trie in self.get_children(): - child_cluster_chain = child_trie._build_sparse_cluster_chain_helper(depth=depth - 1) - all_cluster_chains += [child_cluster_chain] - assert len(child_cluster_chain) == depth - - # Merge all child cluster chains level wise - new_chain = [] - for curr_level in range(depth): - mats_to_merge = [] - for chain in all_cluster_chains: - mats_to_merge += chain[curr_level] - new_chain += [mats_to_merge] - - new_chain = [[par_child_smat]] + new_chain - assert len(new_chain) == (depth + 1) - - return new_chain - - -class TrieIndexer(Indexer): - @classmethod - def gen(cls, feat_mat, label_strs=[], depth=1, **kwargs): - - is_sorted = all(label_strs[i] <= label_strs[i + 1] for i in range(len(label_strs) - 1)) - if not is_sorted: - raise Exception( - "label_strs should be sorted in order to build a cluster matrices correctly. " - "If not sorted then rows in last matrix in cluster chain will not correspond to " - "columns in training- data label matrix " - ) - - LOGGER.info("Starting Trie Indexing") - trie = TrieWrapper() - trie.update({lstr: 1 for lstr in label_strs}) - LOGGER.info("Added all labels to trie") - LOGGER.info("Number of keys = {}".format(trie.n_keys)) - - cluster_chain = trie.build_cluster_chain(depth=depth) - LOGGER.info("Finished building cluster chain") - return cluster_chain - - -class HybridIndexer(Indexer): - @classmethod - def gen( - cls, - feat_mat, - label_strs=[], - depth=1, - spherical=True, - max_iter=20, - max_leaf_size=100, - seed=0, - **kwargs, - ): - - # try: - is_sorted = all(label_strs[i] <= label_strs[i + 1] for i in range(len(label_strs) - 1)) - if not is_sorted: - raise Exception( - "label_strs should be sorted in order to build a cluster matrices correctly. " - "If not sorted then rows in last matrix in cluster chain will not correspond to " - "columns in training- data label matrix " - ) - - LOGGER.info("Starting Hybrid-Trie Indexing") - trie = TrieWrapper() - trie.update({lstr: 1 for lstr in label_strs}) - LOGGER.info("Added all labels to trie. Now building trie till depth = {}".format(depth)) - - trie_chain = trie.build_cluster_chain(depth=depth) - flat_clust = smat.csc_matrix( - trie_chain.chain[-1] - ) # Use last mat in chain to define flat clustering - - LOGGER.info("Flat clust shape :{}".format(flat_clust.shape)) - remaining_chain = PreClusteredHierarchicalKMeans.gen( - feat_mat=feat_mat, - init_mat=flat_clust, - hierarchical_codes=False, - spherical=spherical, - max_leaf_size=max_leaf_size, - max_iter=max_iter, - seed=seed, - ) - - LOGGER.info("Built remaining cluster chain using HC 2-means :".format(flat_clust.shape)) - final_chain = cluster_util.ClusterChain(trie_chain.chain[:-1] + remaining_chain.chain[1:]) - return final_chain - - -class HKMeans_w_MLC(Indexer): - KMEANS = 0 # KMEANS - SKMEANS = 5 # Spherical KMEANS - - @classmethod - def gen( - cls, - feat_mat, - nr_splits=2, - max_leaf_size=100, - imbalanced_ratio=0.0, - imbalanced_depth=100, - spherical=True, - seed=0, - max_iter=20, - threads=-1, - dtype=sp.float32, - mlc_mats=[], - use_freq=True, - **kwargs, - ): - if nr_splits != 2: - raise NotImplementedError - - cluster_chain = hierarchical_kmeans_w_mlc( - feat_mat=feat_mat, - mlc_mats=mlc_mats, - use_freq=use_freq, - max_leaf_size=max_leaf_size, - imbalanced_ratio=imbalanced_ratio, - imbalanced_depth=imbalanced_depth, - spherical=spherical, - seed=seed, - max_iter=max_iter, - threads=threads, - ) - cluster_chain = cluster_util.ClusterChain(cluster_chain) - return cluster_chain - - @staticmethod - def convert_codes_to_csc_matrix(codes, depth): - nr_codes = 1 << depth - nr_elements = len(codes) - - indptr = sp.cumsum(sp.bincount(codes + 1, minlength=(nr_codes + 1)), dtype=sp.uint64) - indices = sp.argsort(codes * sp.float64(nr_elements) + sp.arange(nr_elements)) - C = smat.csc_matrix( - (sp.ones_like(indices, dtype=sp.float32), indices, indptr), - shape=(nr_elements, nr_codes), - ) - return C - - -def hierarchical_kmeans_w_mlc( - feat_mat, - mlc_mats: list, - use_freq, - max_leaf_size=100, - imbalanced_ratio=0.0, - imbalanced_depth=100, - spherical=True, - seed=0, - max_iter=20, - threads=-1, -): - """ - - Parameters - ---------- - feat_mat - mlc_mats: list - list of must link constraint matrix - use_freq - max_leaf_size - imbalanced_ratio - imbalanced_depth - spherical - seed - max_iter - threads - - Returns - ------- - - """ - - global run_kmeans - - def run_kmeans(cluster, c1, c2, min_size, max_iter, spherical=True): - if point_freq_global is None: - indexer = kmeans(feat_mat_global[cluster], None, c1, c2, min_size, max_iter, spherical) - else: - indexer = kmeans( - feat_mat_global[cluster], - point_freq_global[cluster], - c1, - c2, - min_size, - max_iter, - spherical, - ) - return cluster[indexer], cluster[~indexer] - - global kmeans - - def kmeans(feat_mat, freqs, c1=-1, c2=-1, min_size=50, max_iter=20, spherical=True): - if c1 == -1: - c1, c2 = sp.random.randint(feat_mat.shape[0]), sp.random.randint(1, feat_mat.shape[0]) - c1, c2 = feat_mat[c1], feat_mat[(c1 + c2) % feat_mat.shape[0]] - old_indexer = sp.ones(feat_mat.shape[0]) * -1 - - for _ in range(max_iter): - scores = sp.squeeze(sp.asarray(feat_mat.multiply(c1 - c2).sum(1))) - - if freqs is None: - indexer = get_split_wo_freq(scores=scores, min_size=min_size) - else: - indexer = get_split_w_freq(scores=scores, min_size=min_size, freqs=freqs) - - if sp.array_equal(indexer, old_indexer): - break - old_indexer = indexer - c1 = feat_mat[indexer].sum(0) - c2 = feat_mat[~indexer].sum(0) - if spherical: - c1 = normalize(c1) - c2 = normalize(c2) - return indexer - - global feat_mat_global, point_freq_global - feat_mat_global = feat_mat - point_freq_global = None - - random = sp.random.RandomState(seed) - cluster_chain = [] - clusters_big, clusters_small = [], [] - if feat_mat.shape[0] > max_leaf_size: - clusters_big.append(sp.arange(feat_mat.shape[0])) - else: - clusters_small.append(sp.arange(feat_mat.shape[0])) - - while ( - len(clusters_big) > 0 - ): # Iterate until there is at least one cluster with > max_leaf_size nodes - - curr_level = len(cluster_chain) - # Do balanced clustering beyond imbalanced_depth to ensure reasonably timely termination - if curr_level >= imbalanced_depth: - imbalanced_ratio = 0 - - # Enact Must-link constraints by creating connected components based on must-link constraints - if curr_level >= len(mlc_mats): - """If there are no must-link constraints for this level onward, then append an identity matrix which - says that the trivial thing that every point must link to itself!""" - n = feat_mat.shape[0] - mlc_mats.append(smat.csr_matrix(smat.diags(np.ones((n)), shape=(n, n)))) - - clusters_big_cc = [] - feat_mat_cc = [] - cum_idx_cc = 0 - old_cc_to_new_cc = np.zeros((mlc_mats[curr_level].shape[1])) - 1 - new_cc_to_old_cc = np.zeros((mlc_mats[curr_level].shape[1])) - 1 - num_points_per_cc = [] - for cluster in clusters_big: - - # Get constraints mat and features mat rows for this cluster - local_feat_mat = feat_mat[cluster] - local_mlc_mat = mlc_mats[curr_level][cluster] - - # Find # non zero cols in local_mlc_mat. That'll be # conn components(= num_CC) over points in cluster - num_points = len(cluster) - non_zero_cols = np.diff(local_mlc_mat.tocsc().indptr).nonzero()[0] - num_CC = non_zero_cols.shape[0] - - # Retain only non-zero cols in local_mlc_mat. Now it should be of shape num_points x num_CC - local_mlc_mat = local_mlc_mat[:, non_zero_cols] - local_num_points_per_cc = np.array( - np.sum(local_mlc_mat.ceil(), axis=0, dtype=int) - ).reshape(-1) - - # Get feature vec for each conn component using points in that conn comp. - # (# conn comp x # points) x (# points x # features) --> ( # conn comp x # features ) - local_feat_mat_w_mlc = local_mlc_mat.transpose() * local_feat_mat - feat_mat_cc.append(local_feat_mat_w_mlc) - num_points_per_cc.append(local_num_points_per_cc) - - assert local_mlc_mat.shape == (num_points, num_CC) - assert local_feat_mat.shape == (num_points, feat_mat.shape[1]) - assert local_feat_mat_w_mlc.shape == (num_CC, feat_mat.shape[1]) - - """ Assert that each cols sums to one, and sum of total matrix is equal to num_CC. - This is important for correctness when getting conn comp vector using point vectors. """ - assert (np.round(np.sum(local_mlc_mat, axis=0)) == np.ones((1, num_CC))).all() - assert int(np.round(np.sum(local_mlc_mat))) == num_CC - - """ Give indices to each conn comp, offsetting it using cum_idx_cc which keeps track - of # conn comp so far, and add this list to cluster_big_cc """ - cc_idxs = np.arange(num_CC) + cum_idx_cc - clusters_big_cc.append(cc_idxs) - - old_cc_to_new_cc[non_zero_cols] = cc_idxs - new_cc_to_old_cc[cc_idxs] = non_zero_cols - - cum_idx_cc += num_CC - - feat_mat_global_cc = smat.csr_matrix(smat.vstack(feat_mat_cc)) - if use_freq: - point_freq_global = np.concatenate(num_points_per_cc).reshape(-1) - assert point_freq_global.shape == (feat_mat_global_cc.shape[0],) - - clusters_big = clusters_big_cc - feat_mat_global = feat_mat_global_cc - LOGGER.info("Shape of new global feat matrix = {}".format(feat_mat_global.shape)) - - num_parent_clusters = len(clusters_big) + len(clusters_small) - new_clusters_big = [] - new_clusters_small = [] - cols_big, cols_small = [], [x + len(clusters_big) for x in range(len(clusters_small))] - seeds = [(random.randint(s), random.randint(1, s)) for s in map(len, clusters_big)] - min_sizes = [int(s * (0.5 - imbalanced_ratio)) for s in map(len, clusters_big)] - - with mp.Pool(threads if threads > 0 else mp.cpu_count()) as p: - for col, child_clusters in enumerate( - p.starmap( - run_kmeans, - zip( - clusters_big, - *map(list, zip(*seeds)), - min_sizes, - repeat(max_iter), - repeat(spherical), - ), - ) - ): - for cluster_cc in child_clusters: - """cluster is a list of connected component indices. - Convert this list to list of indices of points in these connected components""" - # Map new conn_comp indices to old conn_comp indices - cluster_cc = new_cc_to_old_cc[cluster_cc] - - # Get mlc matrix with only cols restricted to current list of conn components - local_mlc_mat = mlc_mats[curr_level][:, cluster_cc] - assert local_mlc_mat.shape == (feat_mat.shape[0], len(cluster_cc)) - - # Get points in these conn components, which have non zero value in their corresponding row - cluster = np.diff(local_mlc_mat.indptr).nonzero()[0] - if len(cluster) > max_leaf_size and len(cluster_cc) > 1: - new_clusters_big.append(cluster) - cols_big.append(col) - elif len(cluster) > max_leaf_size and len(cluster_cc) == 1: - """Add to small clusters, even though this cluster has more than max_leaf_size points - because this cluster has just one connected component and thus can not split further due - to must-link constraints - """ - new_clusters_small.append(cluster) - cols_small.append(col) - elif len(cluster) > max_leaf_size and len(cluster_cc) == 0: - # This condition is not possible but still having this for a sanity check - raise NotImplementedError - elif len(cluster) > 0: - new_clusters_small.append(cluster) - cols_small.append(col) - # else: # Do not raise error when a cluster is empty. - # raise NotImplementedError - - cols = cols_big + cols_small - - clusters_small.extend(new_clusters_small) - - curr_clust_mat = smat.csc_matrix( - (sp.ones(len(cols)), (range(len(cols)), cols)), - shape=(len(new_clusters_big + clusters_small), num_parent_clusters), - dtype=sp.float32, - ) - cluster_chain.append(curr_clust_mat) - - clusters_big = new_clusters_big - - LOGGER.info( - "Cluster chain shape at level = {} is {}".format(curr_level, curr_clust_mat.shape) - ) - - C = [] - for col, cluster in enumerate(chain(clusters_big, clusters_small)): - for row in cluster: - C.append((row, col)) - - cluster_mat_cc = smat.csc_matrix( - (sp.ones(feat_mat.shape[0]), list(map(list, zip(*C)))), - shape=(feat_mat.shape[0], len(clusters_big) + len(clusters_small)), - dtype=sp.float32, - ) - - cluster_mat = smat.csc_matrix(mlc_mats[-1] * cluster_mat_cc, dtype=sp.float32) - cluster_chain.append(cluster_mat) - LOGGER.info("Cluster chain shape at final level is {}".format(cluster_mat.shape)) - return cluster_chain - - -def build_prefix_mlc_mat(label_strs, max_pref_len): - """ - Generates a list must-link constraint (mlc) matrix using a trie. At level d, the constraint is that - two strings with the exact same prefix upto d chars should be in the same cluster. - Taking transitive closure of must-link constraints induces connected components at each level. - Shape of list of mlc matrices is [(# points, # conn_components_at_level_d) for d in range(1, max_pref_len+1)] - - Parameters - ---------- - label_strs: iterable - iterable over label strings - max_pref_len: int - Max depth upto which I need to create must link constraint cluster matrix using the trie - - - Returns - ------- - List of matrices encoding must link constraints for each level of hierarchical clustering. - """ - - trie_cluster_mat = TrieIndexer.gen(feat_mat=None, label_strs=label_strs, depth=max_pref_len) - - assert len(trie_cluster_mat) == max_pref_len + 1 - for level, mat in enumerate(trie_cluster_mat): - LOGGER.info("Trie Cluster Matrix: {} {}".format(level, mat.shape)) - LOGGER.info("") - - prefix_mlc_mats = [ - smat.csr_matrix(normalize(trie_cluster_mat[-1], axis=0, norm="l1")) - ] # Normalize so that sum of each col is 1 - for pref_len in range(max_pref_len, 0, -1): - last_mat = trie_cluster_mat[pref_len] - sec_last_mat = trie_cluster_mat[pref_len - 1] - new_last_mat = last_mat.dot(sec_last_mat) - - # Remove last 2 matrices and add newly created one - trie_cluster_mat = trie_cluster_mat[: pref_len - 1] + [new_last_mat] - - # Normalize so that sum of each col is 1 - curr_mlc_mat = smat.csr_matrix(normalize(new_last_mat, axis=0, norm="l1")) - prefix_mlc_mats.append(curr_mlc_mat) - - assert len(trie_cluster_mat) == pref_len - LOGGER.info( - "Multiplying matrix {} with {} to get {}".format( - sec_last_mat.shape, last_mat.shape, new_last_mat.shape - ) - ) - - prefix_mlc_mats.reverse() # Reverse the order to go from level 0 to level max_pref_len - prefix_mlc_mats = prefix_mlc_mats[ - 1: - ] # Remove first constraint mat because that effectively puts a constraint that everylabel should be one cluster. - for level, mat in enumerate(prefix_mlc_mats): - LOGGER.info("MLC Mat {} {}".format(level, mat.shape)) - assert mat.shape[0] == len(label_strs) - assert int(np.round(np.sum(mat))) == mat.shape[1] - assert (np.round(np.sum(mat, axis=0)) == np.ones((1, mat.shape[1]))).all() - - assert len(prefix_mlc_mats) == max_pref_len - return prefix_mlc_mats - - -def get_split_wo_freq(scores, min_size): - - n = len(scores) - indexer = scores >= 0 # Default way of assigning points to c1 and c2 by hinging at zero - if indexer.sum() < min_size: - indexer = np.zeros(n, dtype=np.bool) - indexer[sp.argpartition(-scores, min_size)[:min_size]] = True - elif (~indexer).sum() < min_size: - indexer = np.zeros(n, dtype=np.bool) - indexer[sp.argpartition(scores, min_size)[min_size:]] = True - - return indexer - - -def get_split_w_freq(scores, freqs, min_size): - - total_freqs = freqs.sum() - n = len(scores) # Number of points - indexer = ( - scores >= 0 - ) # First assign points with scores greater than zero to c1 and others to c2 - - c1_size = freqs[indexer].sum() - c2_size = freqs[~indexer].sum() - assert c1_size + c2_size == total_freqs - assert c1_size >= min_size or c2_size >= min_size - - if c1_size < min_size: - indexer = np.zeros(n, dtype=np.bool) - ordering = np.argsort( - -1 * scores - ) # Sort in descending order. Elements towards beginning will be close to c1 - freqsums = np.cumsum( - freqs[ordering] - ) # freqs[ordering] gives freqs for points in sorted order of their scores - part_idx = n - 1 - for i in range(n - 1): - if freqsums[i] < min_size <= freqsums[i + 1]: - part_idx = i + 1 - break - c1_idxs = ordering[: part_idx + 1] - c2_idxs = ordering[part_idx + 1 :] - assert freqs[c1_idxs].sum() >= min_size - - if freqs[c2_idxs].sum() < min_size: - LOGGER.info( - "WARNING: min_size = {} condition is violaed. c1,c2 sizes = {} ({:.4f}) {} ({:.4f})".format( - min_size, - freqs[c1_idxs].sum(), - freqs[c1_idxs].sum() / total_freqs, - freqs[c2_idxs].sum(), - freqs[c2_idxs].sum() / total_freqs, - ) - ) - - indexer[c1_idxs] = True - elif c2_size < min_size: - indexer = np.zeros(n, dtype=np.bool) - ordering = np.argsort( - scores - ) # Sort in ascending order. Elements towards beginning will be close to c2 - freqsums = np.cumsum( - freqs[ordering] - ) # yfreqs[ordering] gives yfreqs for points in sorted order of their scores - part_idx = n - 1 - for i in range(n - 1): - if freqsums[i] < min_size <= freqsums[i + 1]: - part_idx = i + 1 - break - c2_idxs = ordering[: part_idx + 1] - c1_idxs = ordering[part_idx + 1 :] - assert freqs[c2_idxs].sum() >= min_size - - if freqs[c1_idxs].sum() < min_size: - LOGGER.info( - "WARNING: min_size = {} condition is violaed. c1,c2 sizes = {} ({:.4f}) {} ({:.4f})".format( - min_size, - freqs[c1_idxs].sum(), - freqs[c1_idxs].sum() / total_freqs, - freqs[c2_idxs].sum(), - freqs[c2_idxs].sum() / total_freqs, - ) - ) - - indexer[c1_idxs] = True - - return indexer - - -def reduce_chain_len(cluster_chain, max_depth): - """ - - Parameters - ---------- - max_depth: int - Max depth of final cluster chain - cluster_chain: list - list of cluster chain - Returns - ------- - cluster chain with given max depth - """ - - if isinstance(cluster_chain, cluster_util.ClusterChain): - cluster_chain = cluster_chain.chain - - assert isinstance(cluster_chain, list) - n_levels = len(cluster_chain) - for level in range(n_levels, max_depth, -1): - last_mat = cluster_chain[level - 1] - sec_lat_mat = cluster_chain[level - 2] - new_mat = last_mat.dot(sec_lat_mat) - - cluster_chain = [mat for mat in cluster_chain[: level - 2]] + [new_mat] - - assert len(cluster_chain) == max_depth - cluster_chain = cluster_util.ClusterChain(cluster_chain) - return cluster_chain - - -def print_to_dense(cluster_chain): - return "\n".join(["{}".format(c.todense()) for c in cluster_chain]) - - -class PreClusteredHierarchicalKMeans(Indexer): - """Does hierarchical balanced kmeans starting from pre-defined top-level clusters.""" - - @classmethod - def gen( - cls, - feat_mat, - init_mat, - kdim=2, - max_leaf_size=100, - spherical=True, - seed=0, - max_iter=20, - threads=-1, - hierarchical_codes=True, - **kwargs, - ): - """Main clustering function. - - Parameters: - ---------- - feat_mat: smat.csr_matrix - label features to be used for clustering. - init_mat: smat.csc_matrix - initial pre-clustering as sparse matrix, rows are labels and columns are codes. - kdim: int - number of children for each parent node - spherical: bool - true: use spherical kmeans - false: use regular kmeans - seed: int - random seed - max_iter: int - max. iterations for clustering - threads: int - number of cores to be used - hierarchical_codes: bool - true: make a hierarchical tree from roots to codes - false: root branches into codes directly - - Returns: - ------- - cluster_chain representing the final clustering. - - """ - if not isinstance(feat_mat, smat.csr_matrix): - raise ValueError("feat_mat does not follow correct input format") - if feat_mat.dtype != np.float32: - raise ValueError("feat_mat does not follow correct data type") - if not isinstance(init_mat, smat.csc_matrix): - raise ValueError("init_mat does not follow correct input format") - if init_mat.dtype != np.float32: - raise ValueError("init_mat does not follow correct data type") - label_order = [] - all_cluster_chains = [] - for code in range(init_mat.shape[1]): - LOGGER.info("Training hierarchical clustering for code: {}".format(code)) - rel_labels = init_mat.indices[init_mat.indptr[code] : init_mat.indptr[code + 1]] - rel_feat = feat_mat[rel_labels, :] - all_cluster_chains.append( - cls.indexer_dict["hierarchicalkmeans"].gen( - feat_mat=rel_feat, - kdim=kdim, - max_leaf_size=max_leaf_size, - imbalanced_ratio=0.0000000000000000001, # Passing non-zero but very very small imbalanced ratio to avoid error thrown by PECOS package when a branch has just a single label. - spherical=spherical, - seed=seed, - threads=threads, - max_iter=max_iter, - ) - ) - label_order += list(rel_labels) - - if hierarchical_codes: - final_cluster_chain = _index_clusters(feat_mat, init_mat) - else: - final_cluster_chain = [ - smat.csc_matrix( - np.ones((init_mat.shape[1], 1)), - dtype=np.float32, - ) - ] - max_depth = max(len(c_chain) for c_chain in all_cluster_chains) - all_cluster_chains = [ - _extend_to_depth(c_chain, max_depth) for c_chain in all_cluster_chains - ] - for d in range(max_depth): - LOGGER.info("Joining matrices at depth {}".format(d)) - mat_list = [ - all_cluster_chains[cluster][d] for cluster in range(len(all_cluster_chains)) - ] - final_cluster_chain.append(_block_join(mat_list)) - inverse = [0] * len(label_order) - for i, p in enumerate(label_order): - inverse[p] = i - final_cluster_chain[-1] = final_cluster_chain[-1].tocsr()[inverse, :].tocsc() - return cluster_util.ClusterChain(final_cluster_chain) - - -def _extend_to_depth(chain, depth): - """Extends a cluster chain to a given depth.""" - if depth < len(chain): - return chain - num_req = depth - len(chain) - num_codes = chain[-1].shape[1] - new_chain = chain[0 : len(chain) - 1] - for i in range(num_req): - new_chain.append(smat.identity(num_codes, dtype=np.float32).tocsc()) - new_chain.append(chain[-1]) - return cluster_util.ClusterChain(new_chain) - - -def _block_join(mat_list): - """Joins a list of matrices as block diagonals of a larger matrix.""" - row_offset = 0 - col_offset = 0 - data = [] - indices = [] - indptr = [] - prev_len = 0 - for i, mat in enumerate(mat_list): - data.extend(mat.data) - indices.extend(mat.indices + row_offset) - if i < len(mat_list) - 1: - indptr.extend(mat.indptr[:-1] + prev_len) - else: - indptr.extend(mat.indptr + prev_len) - prev_len += mat.indptr[-1] - row_offset += mat.shape[0] - col_offset += mat.shape[1] - return smat.csc_matrix( - (data, indices, indptr), - shape=(row_offset, col_offset), - dtype=np.float32, - ) - - -def _index_clusters(feat_mat, init_mat): - """Creates a hierarchical binary tree till the top-level clusters supplied.""" - cluster_feat = init_mat.T.dot(feat_mat) - cluster_feat = cluster_feat.tocsr() - cluster_feat = skprep.normalize(cluster_feat, "l2", axis=1) - init_cluster = HierarchicalKMeans.gen( - feat_mat=cluster_feat, - kdim=2, - max_leaf_size=2, - imbalanced_ratio=0.0, - ) - return init_cluster.chain diff --git a/examples/qp2q/models/pecosq2q.py b/examples/qp2q/models/pecosq2q.py deleted file mode 100644 index 6d66588f..00000000 --- a/examples/qp2q/models/pecosq2q.py +++ /dev/null @@ -1,686 +0,0 @@ -"""The module contains a class to train pecos qp2q models.""" -import copy -import json -import logging -import os -import pathlib -import gc -import glob - -import numpy as np -import pandas as pd -import scipy as sp -import scipy.sparse as smat -from tqdm import tqdm -from abc import ABCMeta -from sklearn.base import BaseEstimator - - -from pecos.apps.text2text import Text2Text -from pecos.xmc import Indexer -from pecos.xmc import LabelEmbeddingFactory -from pecos.xmc.xlinear.model import XLinearModel -from pecos.utils.featurization.text.vectorizers import vectorizer_dict - -from qp2q.models.indices import ( - TrieIndexer, - HybridIndexer, - HKMeans_w_MLC, - build_prefix_mlc_mat, - reduce_chain_len, -) -from qp2q.models.vectorizers import * - -LOGGER = logging.getLogger(__name__) -MODEL_DICT = {} - - -class ModelMeta(ABCMeta): - """Collects all tacos models in model_dict""" - - def __new__(cls, name, bases, attr): - cls = super().__new__(cls, name, bases, attr) - if cls.__name__ != "BaseModel": - MODEL_DICT[cls.__name__.lower()] = cls - return cls - - -class BaseModel(BaseEstimator, metaclass=ModelMeta): - """Base class for TACOS models""" - - def __init__(self, **kwargs): - super(BaseEstimator, self).__init__() - pass - - def fit(self, X, y, **kwargs): - pass - - @classmethod - def load(cls, model_path): - pass - - def save(self, model_path): - pass - - def predict(self, X, **kwargs): - pass - - -class PecosQP2QModel(BaseModel): - """Class to train a pecos next query prediction model given input sparse-dataframe - with prev_query,prefix as rows and next query as cols""" - - def __init__( - self, - model=None, - vectorizer="SklearnTfidf", - cluster_matrix=None, - label_universe=None, - query_feature_cols=None, - vectorizer_class=None, - vectorizer_config={"token_pattern": r"(?u)\b\w+\b", "tokenizer": None}, - load_cluster_matrix=False, - load_trained_vectorizer=False, - weighted_pifa=False, - indexer_type="hierarchicalkmeans", - spherical_clustering=True, - fitted_=False, - query_prefix_delimiter="<@@>", - ): - """ - Parameters - ---------- - model: XLinearModel object, optional - core pecos model used for training, prediction - vectorizer: str - vectorizer type to be used in the model - cluster_matrix: list[csc matrix]/ cluster_chain from pecos.util/ None - clustering matrix, the output of the pecos clustering step - label_universe: list[str], optional - list of queries to restrict label universe for pecos prediction - query_feature_cols : list[str], optional - list of query features to be used in training, - should correspond to columns in query_features_df - vectorizer_class : str, optional - name of vectorizer class used in training, default is tfidf - vectorizer_config : dict - non-default vectorizer parameters loaded into vectorizer - load_cluster_matrix : bool - true if cluster matrix is loaded directly for training, default is false - load_trained_vectorizer : bool - true if trained vectorizer is loaded directly for training, default is false - weighted_pifa : bool - true if label embedding is weighted by score - indexer_type : str - indexer algorithm to be used during clustering, default is hierarchicalkmeans. - This value must be present in indexers.Indexer.indexer_dict - spherical_clustering : bool - True if cluster centers are to be l2 normalized - fitted_: bool - if the model is fitted or not. a model is fitted if - we have previously run the fit method. - query_prefix_delimiter: str - delimiter to separate query and prefix while - representing them as rows of sparse_data_frame - - Notes: - ----- - - When load_trained_vectorizer is True, - you must supply a trained vectorizer for the vectorizer argument. - Similarly, when load_cluster_matrix is True, you must provide a cluster_matrix. - - Otherwise these objects are generated during fit and transform. - """ - super().__init__() - self.model = model - self.vectorizer = vectorizer - self.cluster_matrix = cluster_matrix - self.label_universe = label_universe - self.query_feature_cols = query_feature_cols - self.vectorizer_class = vectorizer_class - self.vectorizer_config = vectorizer_config - self.load_cluster_matrix = load_cluster_matrix - self.load_trained_vectorizer = load_trained_vectorizer - self.weighted_pifa = weighted_pifa - self.indexer_type = indexer_type - self.spherical_clustering = spherical_clustering - self.fitted_ = fitted_ - self.query_prefix_delimiter = query_prefix_delimiter - - @classmethod - def load(cls, folder_path, realtime=False, query_prefix_delimiter="<@@>"): - """ - Class method to load a pre-trained model from given path - - Parameters - ---------- - folder_path : str - path to the folder to load the model - realtime: bool, default - False - load in realtime inference mode or not - query_prefix_delimiter: str - delimiter will be placed between previous query and prefix during inference - - Returns - ------- - PecosQP2QModel object - """ - model = Text2Text.load(folder_path, is_predict_only=realtime) - return cls(model=model, query_prefix_delimiter=query_prefix_delimiter) - - def save(self, folder_path): - """ - Method to save a trained model and its attributes - - Parameters - ------------ - folder_path : str - path to the folder to save the model - """ - os.makedirs(folder_path, exist_ok=True) - - self.model.save(pathlib.Path(folder_path, "xlinear_ensemble/0")) - self.vectorizer.save(pathlib.Path(folder_path, "preprocessor")) - self.cluster_matrix.save(pathlib.Path(folder_path, "cluster")) - - np.save( - pathlib.Path(folder_path, "query_feature_cols"), np.asarray(self.query_feature_cols) - ) - - with open(pathlib.Path(folder_path, "output_items.json"), "w") as f: - json.dump(self.label_universe, f) - - parameter_dict = self.get_params(deep=False) - vectorizer_cls = type(self.vectorizer).__name__.lower() - for key in list(parameter_dict): - if not isinstance(parameter_dict[key], (bool, str)): - parameter_dict.pop(key) - parameter_dict["vectorizer_class"] = vectorizer_cls - with open(pathlib.Path(folder_path, "model_config.json"), "w") as f: - json.dump(parameter_dict, f) - - with open(pathlib.Path(folder_path, "preprocessor/config.json"), "w") as f: - json.dump({"type": vectorizer_cls, "kwargs": {}}, f) - - with open(pathlib.Path(folder_path, "preprocessor/max_prefix_len.json"), "w") as f: - json.dump({"max_prefix_len": None}, f) - - other_params = { - "nr_ensembles": 1, - "kwargs": [ - { - "bias": 1.0, - "Cp": 1.0, - "Cn": 1.0, - "solver_type": 1, - "threshold": 0.1, - "negative_sampling_scheme": "tfn", - "pred_kwargs": {"beam_size": 10, "only_topk": 20, "post_processor": "l3-hinge"}, - "indexer_algo": "hierarchicalkmeans", - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": True, - "seed": 0, - "max_iter": 20, - "max_leaf_size": 100, - "label_embed_type": "pifa", - } - ], - } - with open(pathlib.Path(folder_path, "xlinear_ensemble/config.json"), "w") as f: - json.dump(other_params, f) - - LOGGER.info(f"Saved model and its attributes to {folder_path}") - - def get_output_items(self, indices): - """Get labels given indices. - Parameters: - ---------- - indices: list/ array - list of indices in [0,1,...,#labels] - - Returns: - ------- - labels corresponding to those indices - """ - return [self.model.get_output_item(i) for i in indices] - - def get_suggestions( - self, - prev_query, - prefix, - topk=10, - beam_size=10, - max_prefix_len=None, - max_query_tokens=100, - n_threads=1, - ): - """ - Return predicted queries given prefix - - Parameters: - ---------- - prev_query: str - last query - prefix: str - prefix to be matched - topk: int - num. entries to get - beam_size: int - pecos beam size - max_beam_size: int - maximum pecos beam size - max_prefix_len: int with default None - skip inference on longer prefixes - max_query_tokens: int - do not attempt to run inference on a larger number of query tokens - n_threads: int - how many threads to use during inference - - Returns: - ------- - suggested queries, list - - """ - split_prev_query = prev_query.split() - if ( - sum(len(x) for x in split_prev_query) == 0 - or len(split_prev_query) > max_query_tokens - or (max_prefix_len is not None and len(prefix) > max_prefix_len) - ): - return [] - - text = [self.query_prefix_delimiter.join([prev_query, prefix])] - params = { - "beam_size": beam_size, - "topk": beam_size * self.model.xlinear_models[0][1]["max_leaf_size"], - "threads": n_threads, - } - - try: - results = self.model.predict(text, **params) - except Exception as exc: - LOGGER.exception("Unexpected exception, returning empty response.", exc_info=exc) - return [] - - out = [] - for i, idx in enumerate(results.indices): - item = self.model.get_output_item(idx) - if ( - not item.startswith(prefix) or len(item) == 0 - ): # Filter-out suggestions that do not match prefix - continue - out.append((item, results.data[i])) - if len(out) == topk: - break - return out - - def _build_clusters( - self, - label_features, - max_leaf_size, - max_iterations, - seed, - depth=-1, - imbalanced_ratio=0.0, - imbalanced_depth=100, - nr_splits=2, - ): - """ - Method builds clusters using pifa embeddings of the labels. - - Parameters - ---------- - label_features : csr matrix - pifa embeddings of the labels - indexer_algorithm: str - option of clustering algorithm to use ('SKMEANS' or 'KMEANS') - seed : int - random seed - max_leaf_size : int - max size of leaf nodes in clustering - max_iterations : int - max iterations for the indexer - depth : int - Depth of trie: Useful only for TrieIndexer and HybridIndexer - imbalanced_depth : int - Parameter for Hierarchical k-means indexing - imbalanced_ratio : float - Parameter for Hierarchical k-means indexing - nr_splits : int - Parameter for Hierarchical k-means indexing - Notes - ----- - Computes and sets the cluster_matrix needed for training. - - """ - LOGGER.info("Creating index for training.") - if ( - self.indexer_type not in Indexer.indexer_dict - and self.indexer_type.lower() not in Indexer.indexer_dict - ): - raise ValueError(f"{self.indexer_type} is not supported in PECOS.") - - if self.indexer_type.lower() == "trieindexer": - - LOGGER.info("Depth of cluster matrix is = {}".format(depth)) - self.cluster_matrix = TrieIndexer.gen( - feat_mat=None, label_strs=self.label_universe, depth=depth - ) - - LOGGER.info("Created cluster matrix") - - elif self.indexer_type.lower() == "hybridindexer": - - LOGGER.info("Depth of trie in hybrid index is = {}".format(depth)) - self.cluster_matrix = HybridIndexer.gen( - feat_mat=smat.csr_matrix(label_features, dtype=sp.float32), - label_strs=self.label_universe, - depth=depth, - max_leaf_size=max_leaf_size, - seed=seed, - max_iter=max_iterations, - spherical=self.spherical_clustering, - ) - LOGGER.info("Created cluster matrix") - - elif self.indexer_type.lower() == "hkmeans_w_mlc": - LOGGER.info("Depth of constraint cluster matrix is = {}".format(depth)) - mlc_mats = build_prefix_mlc_mat(label_strs=self.label_universe, max_pref_len=depth) - LOGGER.info("Created constraint cluster matrix") - - self.cluster_matrix = HKMeans_w_MLC.gen( - feat_mat=smat.csr_matrix(label_features, dtype=sp.float32), - mlc_mats=mlc_mats, - use_freq=True, - max_leaf_size=max_leaf_size, - seed=seed, - max_iter=max_iterations, - spherical=self.spherical_clustering, - imbalanced_depth=imbalanced_depth, - imbalanced_ratio=imbalanced_ratio, - nr_splits=nr_splits, - ) - LOGGER.info("Length of cluster matrix = {}".format(len(self.cluster_matrix))) - - else: - self.cluster_matrix = Indexer.gen( - smat.csr_matrix(label_features, dtype=sp.float32), - indexer_type=self.indexer_type.lower(), - max_leaf_size=max_leaf_size, - seed=seed, - max_iter=max_iterations, - spherical=self.spherical_clustering, - imbalanced_depth=imbalanced_depth, - imbalanced_ratio=imbalanced_ratio, - nr_splits=nr_splits, - ) - if depth != -1: - LOGGER.info( - "Reducing length of cluster matrix from {} to {}".format( - len(self.cluster_matrix), depth - ) - ) - self.cluster_matrix = reduce_chain_len( - cluster_chain=self.cluster_matrix, max_depth=depth - ) - - try: - LOGGER.info("Length of cluster matrix = {}".format(len(self.cluster_matrix))) - except Exception as e: - LOGGER.info("Error raised : {} ".format(str(e))) - pass - - def fit( - self, - X, - y, - seed=121, - Cp=1.0, - Cn=1.0, - n_jobs=16, - threshold=0.1, - max_iterations=20, - max_leaf_size=100, - dim_for_PIFA=None, - label_text_features=None, - depth=0, - imbalanced_ratio=0.0, - imbalanced_depth=100, - nr_splits=2, - ): - """ - Class method to train the q2a model by breaking down the process into two steps: i. Clustering ii. Training. - - Parameters - ---------- - X : csr matrix, dtype=np.float32 - training feature matrix - y : csc matrix, dtype=np.float32 - training label matrix - n_jobs: int - number of threads to spawn, default = 16 - Cp : float - coefficient for the positive class in the loss function, default = 1.0 - Cn : float - coefficient for the negative class in the loss function, default = 1.0 - threshold : 0.1 - threshold to sparsify the model weights, default = 1.0 - seed : int - random seed, default = 121 - max_leaf_size : int - max size of leaf nodes in clustering, default = 100 - dim_for_PIFA : int or None - If label_features is not None, then label embedding created using PIFA embedding upto dim = dim_for_PIFA - concatenated with label_features - max_iterations : int - max iterations for the indexer, default = 20 - label_text_features: - dense or sparse matrix with #rows = Number of labels - depth : int - Depth of trie: Useful only for TrieIndexer, HybridIndexer, and hkmeans_w_mlc - imbalanced_depth : int - Parameter for Hierarchical k-means indexing - imbalanced_ratio : float - Parameter for Hierarchical k-means indexing - nr_splits : float - Parameter for Hierarchical k-means indexing - Returns - ------- - fitted PecosQP2QModel object - """ - - LOGGER.info("Starting model training") - LOGGER.info(f"Type(X) = {type(X)}, type(y) = {type(y)}") - if not X.shape[0] == y.shape[0]: - raise ValueError( - f"Number of samples in X ({X.shape[0]}) and y ({y.shape[0]}) don't match" - ) - - if self.load_cluster_matrix: - LOGGER.info("Using cluster matrix loaded during init") - else: - if self.indexer_type.lower() == "trieindexer": - LOGGER.info("Generating index structure using Trie") - self._build_clusters( - label_features=None, - max_leaf_size=max_leaf_size, - max_iterations=0, - seed=0, - depth=depth, - ) - else: - if self.weighted_pifa: - label_features = LabelEmbeddingFactory.pifa(X=X, Y=y) - y[y > 0] = 1 - else: - y[y > 0] = 1 - label_features = LabelEmbeddingFactory.pifa(X=X, Y=y) - - if label_text_features is not None: - # Concat PIFA features with some features extracted from label text - if label_features.shape[0] != label_text_features.shape[0]: - raise ValueError( - "Some labels in the label_text_features do not have features in label_features or the other way" - ) - - LOGGER.info("Beginning to append label text feature to PIFA embedding") - label_features = smat.csr_matrix( - smat.hstack([label_features[:, :dim_for_PIFA], label_text_features]) - ) - LOGGER.info( - "Appending features from label text to PIFA features. Final feature mat dim is {}".format( - label_features.shape - ) - ) - - self._build_clusters( - label_features=label_features, - max_leaf_size=max_leaf_size, - max_iterations=max_iterations, - seed=seed, - depth=depth, - imbalanced_depth=imbalanced_depth, - imbalanced_ratio=imbalanced_ratio, - nr_splits=nr_splits, - ) - del label_features - gc.collect() - - LOGGER.info("Training XLinear model") - self.model = XLinearModel.train( - X, - y, - self.cluster_matrix, - threads=n_jobs, - Cp=Cp, - Cn=Cn, - threshold=threshold, - ) - LOGGER.info("Training complete") - self.fitted_ = True - return self - - def predict( - self, - samples, - query_features_df=None, - topk=10, - beam_size=10, - batch_size=10000, - labels_to_keep=[], - ): - """ - Method to generate candidates for a list of input samples (queries). - - Parameters - ---------- - samples : list[str] - list of samples (queries) - query_features_df : pandas dataframe object - dataframe indexed by query with columns as query features, default = None - topk: int - number of outputs to be considered for each sample, default = 10 - beam_size: int - beam width for prediction, default = 10 - batch_size: int - size of batches to be predicted sequentially, default = 10000 - labels_to_keep : list[str] - list of asins to restrict the label universe for prediction, default = None - - Returns - ------- - Y : csr matrix - sparse Q-A matrix with pecos scores - """ - - if not self.fitted_: - raise ValueError("Error: model has not been trained") - if labels_to_keep: - self.model.set_output_constraint(labels_to_keep) - LOGGER.info(f"Restricted labels universe to {len(labels_to_keep)} asins") - - X = self.vectorizer.predict(samples) - if isinstance(query_features_df, pd.DataFrame): - X = self._combine_query_features(X, query_features_df) - LOGGER.info( - f"Added {len(self.query_feature_cols)} query features to the prediction feature matrix" - ) - num_queries = X.shape[0] - LOGGER.info(f"Inferring pecos scores for {num_queries} samples.") - batch_indices = np.array_split(np.arange(num_queries), max(num_queries // batch_size, 1)) - X = smat.csr_matrix(X, dtype=np.float32) - gc.collect() - Y_list = [] - for i in tqdm(range(len(batch_indices))): - X_current = X[batch_indices[i], :] - X_current.sort_indices() - Y_list.append( - self.model.predict( - X_current, - beam_size=beam_size, - only_topk=topk, - ) - ) - Y = smat.csr_matrix(smat.vstack(Y_list)) - LOGGER.info("Model prediction complete") - return Y - - def transform(self, sparse_data_frame, query_features_df=None, inplace=False): - """ - Method to featurize input sparse data and create X, y matrices for training. - - Parameters - ---------- - sparse_data_frame : SparseDataFrame object - query-asin sparse matrix used for model training - query_features_df: pandas dataframe object - dataframe must be indexed by query, default = None - inplace: bool - if false: a copy of sparse_data_frame data matrix is made for constructing y - if true: the changes are done in place - - Returns - ------- - X : csr matrix - feature matrix - y : csc matrix - label matrix - - Notes - ----- - There is an option to load a pretrained vectorizer to featurize the Q-A matrix and use non-text query features if available. - """ - self.label_universe = list(sparse_data_frame.i2c.values()) - LOGGER.info(f"Number of asins in label universe: {len(self.label_universe)}") - - if self.load_trained_vectorizer: - trained_vectorizer = self.vectorizer - LOGGER.info("Loaded pre-trained vectorizer") - else: - if self.vectorizer.lower() not in vectorizer_dict: - raise ValueError( - f"Invalid vectorizer type {self.vectorizer}, ensure vectorizer class inherits PECOS vectorizer" - ) - self.vectorizer = vectorizer_dict[self.vectorizer.lower()].train( - sparse_data_frame.i2r.values(), self.vectorizer_config - ) - trained_vectorizer = self.vectorizer - LOGGER.info(f"Finished training {self.vectorizer}...") - - X = trained_vectorizer.predict(list(sparse_data_frame.i2r.values())) - if isinstance(query_features_df, pd.DataFrame): - X = self._combine_query_features(X, query_features_df) - LOGGER.info( - f"Added {len(self.query_feature_cols)} query features to the training feature matrix" - ) - LOGGER.info(f"Shape of feature matrix (X) : {X.shape}") - if inplace: - y = sparse_data_frame.data_matrix - else: - y = copy.deepcopy(sparse_data_frame.data_matrix) - X = smat.csr_matrix(X, dtype=np.float32) - y = smat.csc_matrix(y, dtype=np.float32) - gc.collect() - return X, y diff --git a/examples/qp2q/models/train_model.py b/examples/qp2q/models/train_model.py deleted file mode 100644 index 28e298f7..00000000 --- a/examples/qp2q/models/train_model.py +++ /dev/null @@ -1,303 +0,0 @@ -import gc -import sys -import logging -import argparse -import numpy as np -from pathlib import Path -from sklearn.feature_extraction.text import TfidfVectorizer - -from qp2q.preprocessing.sparse_data_processing import SparseDataFrame -from qp2q.preprocessing.session_data_processing import parallel_get_qp2q_sparse_data -from qp2q.models.pecosq2q import PecosQP2QModel -from qp2q.models.vectorizers import TfidfQueryOnly, TfidfQueryPrefix, PositionProductTfidf -from qp2q.utils.Config import Config - - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -class Trainer(object): - def __init__(self, config): - assert isinstance(config, Config) - self.config = config - - def train(self): - - self.config.save_config(self.config.result_dir) - - # Create sparse data frame with tranining data - i2r, i2c, smat = parallel_get_qp2q_sparse_data( - fdir=self.config.fdir, compressed=self.config.f_compressed, n_jobs=self.config.n_jobs - ) - sdf = SparseDataFrame(data_matrix=smat, columns=i2c, rows=i2r) - - # Need to create sdf where col are sorted by their strings. - # This is need only when using TrieIndexer or HybridIndexer - sorted_labels = sorted(sdf.c2i.keys()) - sdf = sdf[:, sorted_labels] - - # Create vectorizer to vectorizer the data that is fed into PECOS models - input_vectorizer = self.get_input_vectorizer(sdf=sdf) - - LOGGER.info("Starting to train model") - model = PecosQP2QModel( - vectorizer=input_vectorizer, - query_prefix_delimiter=self.config.query_prefix_delimiter, - load_trained_vectorizer=True, - indexer_type=self.config.indexer_type, - ) - - LOGGER.info("Create X and Y matrices") - X, y = model.transform(sparse_data_frame=sdf) - - label_features = None - dim_for_PIFA = None - if self.config.use_label_feat: - - query_vectorizer = input_vectorizer.model_query - prefix_vectorizer = ( - input_vectorizer.model_prefix if hasattr(input_vectorizer, "model_prefix") else None - ) - label_vectorizer = self.get_label_vectorizer( - sdf=sdf, prefix_vectorizer=prefix_vectorizer - ) - label_features = label_vectorizer.transform( - [sdf.i2c[i] for i in sorted(sdf.i2c.keys())] - ) - - LOGGER.info("Label features shape:{}".format(label_features.shape)) - LOGGER.info("X query feats:{}".format(len(query_vectorizer.get_feature_names()))) - LOGGER.info("X all shape:{}".format(X.shape)) - - LOGGER.info("Using only label text embedding for indexing labels (no query pifa)") - dim_for_PIFA = 0 - - # Fit a model on training data - model = model.fit( - X=X, - y=y, - label_text_features=label_features, - dim_for_PIFA=dim_for_PIFA, - depth=self.config.depth, - imbalanced_depth=self.config.imb_depth, - imbalanced_ratio=self.config.imb_ratio, - nr_splits=self.config.nr_splits, - ) - - LOGGER.info("Finished training...") - model.save("{}/model".format(self.config.result_dir)) - LOGGER.info(f"Successfully saved model at {self.config.result_dir}") - - def get_input_vectorizer(self, sdf): - """ - Train input vectorizer. - Parameters: - ---------- - sdf: SparseDataFrame - - Returns: - ------- - A vectorizer for vectorizing input for PECOS model - """ - - LOGGER.info("Training input vectorizer") - ######################## Train Query Vectorizer #################################### - query_vectorizer_config = { - "strip_accents": "unicode", - "ngram_range": (1, 1), - "analyzer": "word", - "dtype": np.float32, - } - - query_vectorizer = TfidfVectorizer(**query_vectorizer_config) - query_vectorizer.fit( - list(set([q.split(self.config.query_prefix_delimiter)[0] for q in sdf.i2r.values()])) - ) - gc.collect() - LOGGER.info("Finished Training query vectorizer") - - if self.config.pref_vectorizer.lower() == "": - return TfidfQueryOnly( - model_query=query_vectorizer, delim=self.config.query_prefix_delimiter - ) - elif ( - self.config.pref_vectorizer.lower() == "c-tfidf" - or self.config.pref_vectorizer.lower() == "poswgtd_c-tfidf" - ): - - ######################## Train Prefix} Vectorizer #################################### - LOGGER.info("Training prefix vectorizer") - prefix_vectorizer_config = { - "strip_accents": "unicode", - "ngram_range": (1, 3), - "analyzer": "char", - "dtype": np.float32, - "use_idf": True, - } - - if self.config.pref_vectorizer.lower() == "c-tfidf": - prefix_vectorizer = TfidfVectorizer(**prefix_vectorizer_config) - elif self.config.pref_vectorizer.lower() == "poswgtd_c-tfidf": - prefix_vectorizer = PositionProductTfidf(**prefix_vectorizer_config) - else: - raise Exception("Invalid vectorizer {}".format(self.config.pref_vectorizer)) - - if self.config.prefix_vect_data == "label_text": - LOGGER.info("Using label text for training prefix vectorizer") - pref_vect_train_data = sdf.i2c.values() - elif self.config.prefix_vect_data == "label_text_all_pref": - LOGGER.info( - "Using all possible prefixes of label lext for training prefix vectorizer" - ) - pref_vect_train_data = ( - label[:i] for label in sdf.i2c.values() for i in range(1, len(label) + 1) - ) - elif self.config.prefix_vect_data == "train_data": - LOGGER.info("Using prefixes in train data for training prefix vectorizer") - pref_vect_train_data = ( - q.split(self.config.query_prefix_delimiter)[-1] for q in sdf.i2r.values() - ) - else: - raise Exception( - f"Invalid option for vectorizing prefix = {self.config.prefix_vect_data}.\n " - f"Choose from label_text, label_text_all_pref, train_data" - ) - - prefix_vectorizer.fit(pref_vect_train_data) - gc.collect() - - ############ Combine Query and Prefix Vectorizer #################################### - joint_vectorizer = TfidfQueryPrefix( - model_query=query_vectorizer, - model_prefix=prefix_vectorizer, - delim=self.config.query_prefix_delimiter, - ) - return joint_vectorizer - else: - raise Exception( - "Invalid arg for config.pref_vectorizer = {}".format(self.config.pref_vectorizer) - ) - - def get_label_vectorizer(self, sdf, prefix_vectorizer): - """ - Get labels vectorizer. - Parameters: - ---------- - sdf: SparseDataFrame - prefix_vectorizer : A trained prefix vectorizer. - Only used when self.config.label_vectorizer == "use_prefix_vectorizer" - Returns: - ------- - A vectorizer for vectorizing labels in labels space of PECOS model - """ - - if self.config.label_vectorizer == "use_prefix_vectorizer": - LOGGER.info("Reusing prefix vectorizer to get label embeddings") - label_vectorizer = prefix_vectorizer - else: - LOGGER.info( - "Training new {} vectorizer on {} for vectorizing labels".format( - self.config.label_vectorizer, self.config.label_vect_data - ) - ) - label_vectorizer_config = { - "strip_accents": "unicode", - "ngram_range": (1, 3), - "analyzer": "char", - "dtype": np.float32, - "use_idf": True, - } - - ############################### Choose a vectorizer class ################################################## - if self.config.label_vectorizer.lower() == "c-tfidf": - label_vectorizer = TfidfVectorizer(**label_vectorizer_config) - elif self.config.label_vectorizer.lower() == "poswgtd_c-tfidf": - label_vectorizer = PositionProductTfidf(**label_vectorizer_config) - else: - raise Exception( - "Label vectorizer = {} not supported".format(self.config.label_vectorizer) - ) - - ############################### Choose data to train the vectorizer on ############################# - if self.config.label_vect_data == "train_data": - LOGGER.info("Using prefixes in train data for training label vectorizer") - vect_train_data = ( - q.split(self.config.query_prefix_delimiter)[-1] for q in sdf.i2r.values() - ) - elif self.config.label_vect_data == "label_text": - LOGGER.info("Using label text for training label vectorizer") - vect_train_data = sdf.i2c.values() - elif self.config.label_vect_data == "label_text_all_pref": - LOGGER.info( - "Using all possible prefixes of label lext for training label vectorizer" - ) - vect_train_data = ( - label[:i] for label in sdf.i2c.values() for i in range(1, len(label) + 1) - ) - else: - raise Exception( - "Label vectorizer opt = {} not implemented ".format(self.config.label_vect_data) - ) - - label_vectorizer.fit(vect_train_data) - gc.collect() - - return label_vectorizer - - -def main(): - parser = argparse.ArgumentParser( - description="Train XMC models for next query predictions tasks" - ) - parser.add_argument("--config", type=str, help="Train Config file") - - temp_config = Config() - ################################## OPTIONAL ARGUMENTS TO OVERWRITE CONFIG FILE ARGS ################################ - for config_arg in temp_config.__dict__: - def_val = temp_config.__getattribute__(config_arg) - arg_type = type(def_val) if def_val is not None else str - parser.add_argument( - "--{}".format(config_arg), - type=arg_type, - default=None, - help="If not specified then value from config file will be used", - ) - #################################################################################################################### - - args = parser.parse_args() - - assert args.config is not None - config = Config(args.config) - for config_arg in temp_config.__dict__: - def_val = getattr(args, config_arg) - if def_val is not None: - old_val = config.__dict__[config_arg] - config.__dict__.update({config_arg: def_val}) - new_val = config.__dict__[config_arg] - LOGGER.info( - "Updating Config.{} from {} to {} using arg_val={}".format( - config_arg, old_val, new_val, def_val - ) - ) - - Path(config.result_dir).mkdir( - parents=True, exist_ok=True - ) # Create resultDir directory if not already present - config.update_random_seeds(config.seed) - config.save_config(config.result_dir) - - trainer = Trainer(config) - if config.mode == "train": - trainer.train() - else: - raise Exception("Invalid mode = {}.".format(config.mode)) - - -if __name__ == "__main__": - main() diff --git a/examples/qp2q/models/vectorizers.py b/examples/qp2q/models/vectorizers.py deleted file mode 100644 index 8d2f90bb..00000000 --- a/examples/qp2q/models/vectorizers.py +++ /dev/null @@ -1,419 +0,0 @@ -import os -import sys -import json -import array -import pickle -import logging -import pathlib -import numpy as np -from collections import defaultdict - -import scipy.sparse as smat -from pure_sklearn.map import convert_estimator -from sklearn.feature_extraction.text import TfidfVectorizer -from sklearn.preprocessing import normalize -from sklearn.utils import _IS_32BIT - -import pecos.utils.featurization.text.vectorizers as pecos_vects - - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -def _make_float_array(): - """Construct an array.array of a type suitable for scipy.sparse indices.""" - return array.array("f") - - -class TfidfQueryPrefix(pecos_vects.Vectorizer): - """Vectorizer that processes Query and Prefix joined by a delim""" - - def __init__( - self, - model_query=None, - model_prefix=None, - delim="<@@>", - max_prefix_len=None, - ): - """ - Parameters: - ---------- - model_query: TfidfVectorizer object - query encoder - model_prefix: CountVectorizer object - prefix encoder - delim: str - delim between query and prefix - max_prefix_len: int - if supplied then part of the prefix till this length - is used for training and prediction. Default is None, in which case the full - prefix is used. - """ - self.model_query = model_query - self.model_prefix = model_prefix - self.delim = delim - self.query_vocab_len = len(self.model_query.vocabulary_) - self.prefix_vocab_len = len(self.model_prefix.vocabulary_) - self.max_prefix_len = max_prefix_len - - def save(self, vectorizer_folder): - """ - Save objects in pkl format. - - Parameters - ---------- - vectorizer_folder: str - Folder inside which to store the serialized objects as `vectorizer_.pkl` - - """ - os.makedirs(vectorizer_folder, exist_ok=True) - with open(pathlib.Path(vectorizer_folder, "vectorizer_query.pkl"), "wb") as pfile: - pickle.dump(self.model_query, pfile, protocol=pickle.HIGHEST_PROTOCOL) - with open(pathlib.Path(vectorizer_folder, "vectorizer_prefix.pkl"), "wb") as pfile: - pickle.dump(self.model_prefix, pfile, protocol=pickle.HIGHEST_PROTOCOL) - with open(pathlib.Path(vectorizer_folder, "delim.json"), "w") as jfile: - json.dump({"delim": self.delim}, jfile) - with open(pathlib.Path(vectorizer_folder, "max_prefix_len.json"), "w") as jfile: - json.dump({"max_prefix_len": self.max_prefix_len}, jfile) - - @classmethod - def load(cls, vectorizer_folder, realtime=False): - """ - Load a saved object. - - Parameters - ---------- - vectorizer_folder: str - Folder to load the model from - realtime: bool - if true then it is loaded for realtime inference - - """ - with open(pathlib.Path(vectorizer_folder, "vectorizer_query.pkl"), "rb") as pfile: - model_query = pickle.load(pfile) - with open(pathlib.Path(vectorizer_folder, "vectorizer_prefix.pkl"), "rb") as pfile: - model_prefix = pickle.load(pfile) - with open(pathlib.Path(vectorizer_folder, "delim.json"), "r") as jfile: - delim = json.load(jfile)["delim"] - try: - with open(pathlib.Path(vectorizer_folder, "max_prefix_len.json"), "r") as jfile: - max_prefix_len = json.load(jfile)["max_prefix_len"] - except Exception: - LOGGER.warning("max_prefix_len.json file not found. Max Prefix Len set to null") - max_prefix_len = None - if realtime: - model_query = convert_estimator(model_query) - model_prefix = convert_estimator(model_prefix) # convert to predict only faster version - return cls( - model_query=model_query, - model_prefix=model_prefix, - delim=delim, - max_prefix_len=max_prefix_len, - ) - - def predict(self, corpus): - """Predict on corpus - - Parameters: - ---------- - corpus: list/iterator - corpus where each eliment is of the form - - Returns: - ------- - encoded corpus - """ - query_features = self.model_query.transform( - [sample.split(self.delim)[0] for sample in corpus] - ) - prefix_features = self.model_prefix.transform( - [sample.split(self.delim)[1][: self.max_prefix_len] for sample in corpus] - ) - query_features = self._convert_to_csr(query_features, self.query_vocab_len) - prefix_features = self._convert_to_csr(prefix_features, self.prefix_vocab_len) - return normalize(smat.hstack([query_features, prefix_features]), "l2", axis=1) - - def _convert_to_csr(self, features, dimension): - """ - Helper function to convert dictionary of features to sparse csr_matrix - - Parameters: - ---------- - features: list(dictionary) (or sparse csr matrix) - a sparse matrix represented as list of dictionary, each element of the list is a row. - Each row's dictionary has indices mapped to values - dimension: int - dimension 1 of the sparse csr matrix that is required - - Returns: - ------- - sparse csr matrix. - """ - if isinstance(features, smat.csr_matrix): - return features - data = [] - indices = [] - ptr = 0 - indptr = [ptr] - for f in features: - data += list(f.values()) - indices += list(f.keys()) - ptr += len(f) - indptr.append(ptr) - - return smat.csr_matrix( - (data, indices, indptr), - shape=(len(features), dimension), - dtype=np.float32, - ) - - -class PositionProductTfidf(TfidfVectorizer): - """ - Tfidf vectorizer first creates a - position discounted CountVectorizer where an ngram - at position i counts for 1/(i+1) instead of 1 unit - and then multiplies the discounted count vector with tfidf vector - """ - - def __init__(self, **kwargs): - super(PositionProductTfidf, self).__init__(**kwargs) - - def fit_transform(self, raw_documents, y=None): - """Fit and transform method for vectorizer. - - Parameters: - ----------- - raw_documents: lst(str) - list of raw documents for that need to be vectorized - y: None - this parameter is not needed here but kept for the sake of consistency - with scikit learn. - Similar to https://github.com/scikit-learn/ - scikit-learn/blob/0fb307bf3/sklearn/feature_extraction/text.py#L1808 - - Returns: - ------- - X : sparse matrix of (n_samples, n_features) - Tf-idf-weighted document-term matrix. - """ - self.fit(raw_documents=raw_documents, y=y) - return self.transform(raw_documents) - - def _count_vocab_w_pos(self, raw_documents, fixed_vocab): - """Create sparse feature matrix, and vocabulary where fixed_vocab=False""" - if fixed_vocab: - vocabulary = self.vocabulary_ - else: - # Add a new value when a new vocabulary item is seen - vocabulary = defaultdict() - vocabulary.default_factory = vocabulary.__len__ - - analyze = self.build_analyzer() - j_indices = [] - indptr = [] - - values = _make_float_array() - indptr.append(0) - for doc in raw_documents: - feature_counter = {} - curr_ngram_len = 0 - ngram_offset = 0 - # the analyze function breaks the doc into list of n-grams - # starting with 1-gram, 2-gram and so on till n. - # so if we want till 2-grams, then - # "iphone" will become [i, p, h, o, n, e, ip, ph, ho, on, ne] - # note that the position of ip is 0 and not its position in the above - # list, so the extra logic is needed for position weighting - for feat_pos, feature in enumerate(analyze(doc)): - if len(feature) > curr_ngram_len: - curr_ngram_len = len(feature) - ngram_offset = feat_pos - try: - feature_idx = vocabulary[feature] - if feature_idx not in feature_counter: - feature_counter[feature_idx] = 1 / (feat_pos - ngram_offset + 1) - else: - feature_counter[feature_idx] += 1 / (feat_pos - ngram_offset + 1) - except KeyError: - # keyerror can occur only if fixed_vocab is true - # but it does not mean that it will necessarily occur - continue - - j_indices.extend(feature_counter.keys()) - values.extend(feature_counter.values()) - indptr.append(len(j_indices)) - - if not fixed_vocab: - # disable defaultdict behavior - vocabulary = dict(vocabulary) - if not vocabulary: - raise ValueError( - "empty vocabulary; perhaps the documents only" " contain stop words" - ) - - if indptr[-1] > np.iinfo(np.int32).max: # = 2**31 - 1 - if _IS_32BIT: - raise ValueError( - ( - "sparse CSR array has {} non-zero " - "elements and requires 64 bit indexing, " - "which is unsupported with 32 bit Python." - ).format(indptr[-1]) - ) - indices_dtype = np.int64 - - else: - indices_dtype = np.int32 - j_indices = np.asarray(j_indices, dtype=indices_dtype) - indptr = np.asarray(indptr, dtype=indices_dtype) - values = np.frombuffer(values, dtype=np.single) - - X = smat.csr_matrix( - (values, j_indices, indptr), - shape=(len(indptr) - 1, len(vocabulary)), - dtype=self.dtype, - ) - X.sort_indices() - return vocabulary, X - - def transform(self, corpus, **kwargs): - """Transform documents to document-term matrix. - Uses the vocabulary and document frequencies (df) learned by fit (or - fit_transform). - - Parameters: - ---------- - corpus : iterable - An iterable which yields either str, unicode or file objects. - - Returns: - ------- - X : sparse matrix of (n_samples, n_features) - Tf-idf-weighted document-term matrix. - """ - - self._check_vocabulary() - # use the same matrix-building strategy as fit_transform - _, X = self._count_vocab_w_pos(corpus, fixed_vocab=True) - - all_tfidf_vecs = self._tfidf.transform(X, copy=False) - - return all_tfidf_vecs - - -class TfidfQueryOnly(pecos_vects.Vectorizer): - """Vectorizer that processes takes Query and Prefix joined by a delim - as input but generates a vector using query ONLY""" - - def __init__(self, model_query=None, delim="<@@>"): - """ - Parameters: - ---------- - model_query: TfidfVectorizer object - query encoder - delim: str - delim between query and prefix - """ - self.model_query = model_query - self.delim = delim - - def save(self, vectorizer_folder): - """ - Save objects in pkl format. - - Parameters - ---------- - vectorizer_folder: str - Folder inside which to store the serialized objects as `vectorizer_.pkl` - - """ - os.makedirs(vectorizer_folder, exist_ok=True) - with open(pathlib.Path(vectorizer_folder, "vectorizer_query.pkl"), "wb") as pfile: - pickle.dump(self.model_query, pfile, protocol=pickle.HIGHEST_PROTOCOL) - with open(pathlib.Path(vectorizer_folder, "delim.json"), "w") as jfile: - json.dump({"delim": self.delim}, jfile) - - @classmethod - def load(cls, vectorizer_folder): - """ - Load a saved object. - - Parameters - ---------- - vectorizer_folder: str - Folder to load the model from - - """ - with open(pathlib.Path(vectorizer_folder, "vectorizer_query.pkl"), "rb") as pfile: - model_query = pickle.load(pfile) - with open(pathlib.Path(vectorizer_folder, "delim.json"), "r") as jfile: - delim = json.load(jfile)["delim"] - return cls(model_query=model_query, delim=delim) - - @classmethod - def train(cls, trn_corpus, config={}, dtype=np.float32): - """Train vectorizer from corpus. - - Paraneters: - ---------- - trn_corpus: list/iterator - training corpus where each eliment is of the form - config: dict - config file for training with keys: - config_query: configuration to set params for query encoder - delim: delimiter to be used (default = <@@>) - dtype: dtype object - datatype for encoding returned - - Returns: - ------- - trained object of cls - """ - defaults_query = { - "encoding": "utf-8", - "strip_accents": "unicode", - "stop_words": None, - "ngram_range": (1, 1), - "min_df": 1, - "lowercase": True, - "norm": "l2", - "dtype": dtype, - } - delim = "<@@>" - config_query = config.get("config_query", {}) - - try: - model_query = TfidfVectorizer(**{**defaults_query, **config_query}) - except TypeError: - raise Exception( - f"vectorizer config {config} contains unexpected keyword arguments for TfidfVectorizer" - ) - if "delim" in config: - delim = config["delim"] - model_query.fit(list(set([sample.split(delim)[0] for sample in trn_corpus]))) - return cls(model_query, delim) - - def predict(self, corpus, **kwargs): - """Predict on corpus - - Parameters: - ---------- - corpus: list/iterator - corpus where each eliment is of the form - - Returns: - ------- - encoded corpus - """ - query_features = self.model_query.transform( - [sample.split(self.delim)[0] for sample in corpus] - ) - query_features.sort_indices() - return query_features diff --git a/examples/qp2q/preprocessing/__init__.py b/examples/qp2q/preprocessing/__init__.py deleted file mode 100644 index d933a529..00000000 --- a/examples/qp2q/preprocessing/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# Init File for the package diff --git a/examples/qp2q/preprocessing/session_data_processing.py b/examples/qp2q/preprocessing/session_data_processing.py deleted file mode 100644 index 0d10a590..00000000 --- a/examples/qp2q/preprocessing/session_data_processing.py +++ /dev/null @@ -1,206 +0,0 @@ -"""The module contains functions to preprocess input -datasets into usable format.""" -import gc -import gzip -import json -import logging -import multiprocessing as mp -import pathlib -import sys -from itertools import repeat -import numpy as np -import scipy.sparse as smat - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) - -logger = logging.getLogger(__name__) - -_FUNC = None # place holder to Pool functions. - - -def _worker_init(func): - "init method to invoke Pool." - global _FUNC - _FUNC = func - - -def _worker(x): - "init function to invoke pool" - return _FUNC(x) - - -def open_file_helper(filename, compressed, mode="rt"): - """ - Supports reading of gzip compressed or uncompressed file. - - Parameters: - ---------- - filename : str - Name of the file to open. - compressed : bool - If true, treat filename as gzip compressed. - mode : str - Reading mode. - - Returns: - -------- - file handle to the opened file. - """ - return gzip.open(filename, mode=mode) if compressed else open(filename, mode) - - -def _get_unique_rows_cols(filename, compressed, delim="<@@>"): - """Function to load a json file in the format of processed session-data - for qp2q. Then it returns dictionary of queryprefix as r2i and next_query - as c2i. - """ - r2i = {} - c2i = {} - logger.info("Processing file for rows and columns: {}".format(filename)) - with open_file_helper(filename, compressed) as fp: - for line in fp: - try: - pline = json.loads(line) - except json.decoder.JSONDecodeError: - logger.warn(f"Failed to parse: {line}") - continue - query_prefix = delim.join([pline["prev_query"], pline["prefix"]]) - kw = pline["next_query"] - if query_prefix not in r2i: - r2i[query_prefix] = 1 - if kw not in c2i: - c2i[kw] = 1 - return r2i, c2i - - -def _transform_file_to_matrix_qp2q(filename, compressed, delim, g_r2i, g_c2i): - """ - Helper Function to extract qp2q matrix from input_file which was generated - as a output of the function parallel_process_session_data_qp2p. - Parameters: - ---------- - input_file: filename - full filepath of input dataframe - compressed: bool - compressed or not - delim: str - delim separating query and prefix - g_r2i: dictionary - mapping for input items - g_c2i: dictionary - mapping of output item - - Returns: - ------- - qp2q count matrix - """ - rows = [] - cols = [] - data = [] - logger.info("Processing file for matrix: {}".format(filename)) - with open_file_helper(filename, compressed) as fp: - for line in fp: - try: - pline = json.loads(line) - except json.decoder.JSONDecodeError: - logger.warn(f"Failed to parse: {line}") - continue - query_prefix = delim.join([pline["prev_query"], pline["prefix"]]) - kw = pline["next_query"] - freq = 1 - data.append(freq) - rows.append(g_r2i[query_prefix]) - cols.append(g_c2i[kw]) - matrix = smat.coo_matrix((data, (rows, cols)), shape=(len(g_r2i), len(g_c2i)), dtype=np.float32) - return matrix - - -def parallel_get_qp2q_sparse_data(fdir, compressed, delim="<@@>", n_jobs=4): - """Process session data to sparse matrix and dictionaries mapping rows and columns. - - Parameters: - ---------- - fdir: str - path to directory having all the files in json format - compressed: bool - files being compressed or not - delim: str - delimiter between query and prefix - n_jobs: int - number of threads to be used - - Returns: - ------- - dictionary mapping row index to row names - dictionary mapping col index to col names - qp2q sparse csr matrix containing freq. of occurences. - - """ - if compressed: - extension = "*.gz" - else: - extension = "*.json" - - if pathlib.Path(fdir).is_dir(): - files = pathlib.Path(fdir).glob(extension) - else: - raise ValueError(f"{fdir} is not a valid directory") - - files = [str(f) for f in files] - - logger.info("Getting qp2q unique rows and columns from files in {}".format(fdir)) - if n_jobs > 1: - with mp.Pool(processes=n_jobs) as pool: - dicts = pool.starmap( - _get_unique_rows_cols, - zip(files, repeat(compressed), repeat(delim)), - ) - else: - dicts = [_get_unique_rows_cols(file, compressed, delim) for file in files] - - g_r2i = {} - g_c2i = {} - for dic in dicts: - g_r2i.update(dic[0]) - g_c2i.update(dic[1]) - - g_i2r = {} - g_i2c = {} - for i, k in enumerate(g_r2i.keys()): - g_r2i[k] = i - g_i2r[i] = k - for i, k in enumerate(g_c2i.keys()): - g_c2i[k] = i - g_i2c[i] = k - - del dicts - gc.collect() - logger.info("Number of unique rows: {}".format(len(g_r2i))) - logger.info("Number of unique cols: {}".format(len(g_c2i))) - if n_jobs > 1: - with mp.Pool( - processes=n_jobs, - initializer=_worker_init, - initargs=( - lambda x: _transform_file_to_matrix_qp2q(x, compressed, delim, g_r2i, g_c2i), - ), - ) as pool: - matrices = pool.map(_worker, files) - else: - matrices = [ - _transform_file_to_matrix_qp2q(x, compressed, delim, g_r2i, g_c2i) for x in files - ] - matrices = [m.tocsr() for m in matrices] - qp2q_matrix = matrices[0] - for i in range(1, len(matrices)): - qp2q_matrix += matrices[i] - - del matrices - gc.collect() - - return g_i2r, g_i2c, qp2q_matrix diff --git a/examples/qp2q/preprocessing/sparse_data_processing.py b/examples/qp2q/preprocessing/sparse_data_processing.py deleted file mode 100644 index aaf85dba..00000000 --- a/examples/qp2q/preprocessing/sparse_data_processing.py +++ /dev/null @@ -1,563 +0,0 @@ -""" -This module contains the SparseDataFrame object -""" -import gc -import logging -import os -import pathlib -import pickle -import random -import copy -from collections import Counter -from multiprocessing import Pool - -import numpy as np -import scipy.sparse as smat - - -SEED = 111 -np.random.seed(SEED) -random.seed(SEED) - -LOGGER = logging.getLogger(__name__) -logging.basicConfig( - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) - -_FUNC = None # place holder to Pool functions. - - -def worker_init(func): - "init method to invoke Pool." - global _FUNC - _FUNC = func - - -def worker(x): - "init function to invoke pool" - return _FUNC(x) - - -def parallel_array_apply(arr, func, threads=8, len_threshold=None): - """ - Apply function in parallel to array. - - Parameters: - ---------- - arr: np.array - input array - func: function vectorized - function to be applied - threads: int - number of processes to use - len_threshold: int - no parallelism for arr len less than len_threshold - - Returns: - ------- - Function applied to the array. - """ - if len_threshold is None: - len_threshold = threads - if len(arr) < len_threshold or threads == 1: - return func(arr) - chunks = np.array_split(arr, threads) - with Pool(processes=threads, initializer=worker_init, initargs=(func,)) as pool: - results = pool.map(worker, chunks) - return np.concatenate(results) - - -class SparseDataFrame(object): - """ - The goal for this class object is to mimic a - sparse dataframe (limited functionality as of now). It is basically a sparse - matrix that can be indexed by other indenitifiers, - and not just row and column numbers. - For instance if the rows are asins and columns are queries, then we can get a - sparse sub-matrix as follows: - - R = ['B07K569FH8', 'B075PK7V2T'] - C = ['revvl 2 plus case', 'radio for 2011 gmc sierra'] - - SD[R,C] will give the corresponding sub-matrix, where SD is the sparse data frame. - """ - - def __init__(self, data_matrix, columns, rows): - """ - Parameters - ------------ - data: sparse csr matrix - input data matrix - columns: list/iterable - iterable with column names numbered from index 0 to end - rows: list/iterable - iterable with row names numbered from index 0 to end - Returns - ------------ - Nothing - """ - assert isinstance( - data_matrix, smat.csr_matrix - ), "data_matrix has to be scipy sparse csr matrix" - self.data_matrix = data_matrix # the actual sparse matrix - self.i2r = {} # indextorow - self.r2i = {} # rowtoindex - self.i2c = {} # indextocolumn - self.c2i = {} # columntoindex - self.shape = self.data_matrix.shape - assert data_matrix.shape[0] == len( - rows - ), "Number of rows are not equal to data_matrix dim 0 shape" - assert data_matrix.shape[1] == len( - columns - ), "Number of columns are not equal to data_matrix dim 1 shape" - for i in range(len(rows)): - self.i2r[i] = rows[i] - self.r2i[rows[i]] = i - for i in range(len(columns)): - self.i2c[i] = columns[i] - self.c2i[columns[i]] = i - - def save(self, folder_path): - """ - Function to save sparse data frame in the goven folder path. - It creates the folder if it does not exists - Parameters - ------------- - folder_path: str - folder path to save the object in - """ - os.makedirs(folder_path, exist_ok=True) - i2rpath = str(pathlib.Path(folder_path, "i2r.pkl")) - i2cpath = str(pathlib.Path(folder_path, "i2c.pkl")) - with open(i2rpath, "wb") as f: - pickle.dump(self.i2r, f, protocol=pickle.HIGHEST_PROTOCOL) - with open(i2cpath, "wb") as f: - pickle.dump(self.i2c, f, protocol=pickle.HIGHEST_PROTOCOL) - - matrix_path = str(pathlib.Path(folder_path, "matrix.npz")) - smat.save_npz(matrix_path, self.data_matrix) - - @classmethod - def load(cls, folder_path): - """ - Method to load an object of the class given a folder_path - - Parameters - ------------ - folder_path: str - path containing a saved sdf - - Returns - ------------- - An object of the class - """ - data_matrix = smat.load_npz(str(pathlib.Path(folder_path, "matrix.npz"))) - i2r = pickle.load(open(str(pathlib.Path(folder_path, "i2r.pkl")), "rb")) - i2c = pickle.load(open(str(pathlib.Path(folder_path, "i2c.pkl")), "rb")) - return cls(data_matrix=data_matrix, columns=i2c, rows=i2r) - - @classmethod - def load_from_dataframe(cls, dataframe, col_row, col_col, col_val): - """ - Class method to convert a pandas - dataframe into a sparse dataframe - Parameters - ------------ - dataframe: pandas dataframe - input dataframe - col_row: str - the column mapped to the rows of the matrix - col_col: str - the column in the dataframe mapped to the columns of the matrix - col_val: str - the column in the dataframe which has the values - Returns - ------------ - parseDataFrame object, with the rows - as col_row and columns as col_col - and the corresponding values as col_val - """ - pdata = dataframe.loc[dataframe[col_val] != 0] - pdata = pdata.set_index([col_row, col_col]) - mat = smat.csr_matrix((pdata[col_val], (pdata.index.codes[0], pdata.index.codes[1]))) - return cls( - data_matrix=mat, - columns=pdata.index.levels[1], - rows=pdata.index.levels[0], - ) - - def __getitem__(self, key): - """ - Custom getitem method: - self['hello','goodbye'] returns the value stored in the matrix - corresponding to row 'hello' - and column 'goodbye' - self[['hello','world'],['good','indexing']] returns the sparse 2 by-2 - SparseDataFrame corresponding to rows - ['hello','world'] and columns ['good','indexing'] - self[['hello'],:] will fetch all the columns corresponding to the row - ['hello'] - Parameters - ------------ - key: tuple - tuple of row and column iterables - sub-dataframe - Returns - ------------ - a sparse dataframe or a single scalar value - """ - k0 = np.array(key[0]).reshape(-1) - k1 = np.array(key[1]).reshape(-1) - out_data, rows, columns = self.get_submatrix_data(k0, k1) - if isinstance(out_data, smat.csr_matrix): - return SparseDataFrame(data_matrix=out_data, rows=rows, columns=columns) - return out_data - - def get_submatrix_data(self, rows, columns): - """ - This method can be useful, - if we only want to get the corresponding csr matrix. - Parameters - ------------- - rows: list(str) - rows for which we need to fetch - sub-matrix - columns: list(str) - rows for which we need to fetch - sub-matrix - Returns - ------------- - output csr matrix/individual scalar value - row iterator/dict - col iterator/dict - """ - if rows is None or isinstance(rows, slice): - rs = np.arange(self.data_matrix.shape[0]) - rows = self.i2r - elif rows[0] is None or isinstance(rows[0], slice): - rs = np.arange(self.data_matrix.shape[0]) - rows = self.i2r - else: - rs = [self.r2i[i] for i in rows] - rs = np.array(rs) - if columns is None or isinstance(columns, slice): - cs = np.arange(self.data_matrix.shape[1]) - columns = self.i2c - elif columns[0] is None or isinstance(columns[0], slice): - cs = np.arange(self.data_matrix.shape[1]) - columns = self.i2c - else: - cs = [self.c2i[i] for i in columns] - cs = np.array(cs) - if len(rs) == 1 and len(cs) == 1: - out_data = self.data_matrix[rs[0], cs[0]] - else: - out_rows = self.data_matrix[rs, :] - out_data = out_rows[:, cs] - return out_data, rows, columns - - def get_index2rows(self, indices=None): - """ - Parameters - ------------ - indices: list/iterable - indices that corresponds to rows - Returns - ------------ - row names corresponding to the indices - """ - if indices is None: - indices = np.arange(self.data_matrix.shape[0]) - return [self.i2r[i] for i in indices] - - def get_index2columns(self, indices=None): - """ - Parameters - ------------ - indices: list/iterable - indices that corresponds to columns - Returns - ------------ - column names corresponding to the indices - """ - if indices is None: - indices = np.arange(self.data_matrix.shape[1]) - return [self.i2c[i] for i in indices] - - def get_rows2index(self, rows=None): - """ - Parameters - ------------ - indices: list/iterable - list of row names - Returns - ------------ - indices corresponding to those rows - """ - if rows is None: - return np.arange(self.data_matrix.shape[0]) - return [self.r2i[r] for r in rows] - - def get_columns2index(self, columns=None): - """ - Parameters - ------------ - indices: list/iterable - list of column names - Returns - ------------ - indices corresponding to those columns - """ - if columns is None: - return np.arange(self.data_matrix.shape[1]) - return [self.c2i[c] for c in columns] - - def transpose(self): - """ - Out of place transpose of the whole object. - Returns - ----------- - sparse data-frame object transposed - """ - data_matrix = smat.csc_matrix(self.data_matrix).transpose() - return SparseDataFrame(data_matrix=data_matrix, rows=self.i2c, columns=self.i2r) - - def transpose_(self): - """ - Inplace transpose operation - Returns - ----------- - void - """ - self.data_matrix = smat.csc_matrix(self.data_matrix).transpose() - self.i2r, self.i2c = self.i2c, self.i2r - self.r2i, self.c2i = self.c2i, self.r2i - - def set_values(self, rows, columns, values, lil_matrix=False, merge_type="replace"): - """ - Set the given indices with the given 'values' - It might be more efficient to convert the matrix m in lil_matrix format - before doing a lot of these operations - for instance rows = [1,2] and columns = [4,5] means entries (1,4) - and (2,5) will be edited - values: the actual values to be written into the supplied indices - Parameters - ------------- - rows: list/iterable - indices denoting rows - columns: list/iterables - indices denoting column - lil_matrix: bool - boolean value denoting whether to covert back and forth to - lil_matrix while changing sparsity pattern - merge_type: str - 'replace': original values are replaced - 'min' : minimum of the values are kept in place of collision - 'max' : maximum of the values are kept in place of collision - 'add' : add the two values - """ - if lil_matrix: - self.data_matrix = smat.lil_matrix( - self.data_matrix - ) # this may help to speed things up if we are modifying a lot - # of values all at once - row_indices = self.get_rows2index(rows) - column_indices = self.get_columns2index(columns) - prev_values = np.array( - [self.data_matrix[row_indices[i], column_indices[i]] for i in range(len(row_indices))] - ) - if merge_type == "replace": - write_values = values - elif merge_type == "max": - write_values = np.maximum(values, prev_values) - elif merge_type == "min": - write_values = np.minimum(values, prev_values) - elif merge_type == "add": - write_values = values + prev_values - else: - raise NotImplementedError - self.data_matrix[row_indices, column_indices] = write_values - if lil_matrix: - self.data_matrix = smat.csr_matrix(self.data_matrix) - - def shape(self): - """ - Returns the shape of the data_matrix - """ - return self.data_matrix.shape - - def join(self, sdf, merge_type="replace", threads=1): - """ - Parameters: - ---------- - sdf: SparseDataFrame - another sparse dataframe - merge_type: str - 'replace': original values are replaced - 'min' : minimum of the values are kept in place of collision - 'max' : maximum of the values are kept in place of collision - 'add' : add the two values - threads: int - number of processes to use. - - Returns: - ------- - The values in sdf are used to replace/add to the corresponding - values in the original dataframe - """ - new_i2r, new_r2i = self._merge_index_dictionaries(self.r2i, sdf.r2i) - new_i2c, new_c2i = self._merge_index_dictionaries(self.c2i, sdf.c2i) - self.data_matrix = smat.coo_matrix(self.data_matrix) - sdf.data_matrix = smat.coo_matrix( - sdf.data_matrix - ) # this will be changed back to original format - gc.collect() - data_dic_self = self._map_rows_cols_join( - self.data_matrix, - self.i2r, - self.i2c, - new_r2i, - new_c2i, - threads=threads, - ) - LOGGER.info("Finished mapping rows and columns from self.") - del self.data_matrix - gc.collect() - - data_dic_sdf = self._map_rows_cols_join( - sdf.data_matrix, - sdf.i2r, - sdf.i2c, - new_r2i, - new_c2i, - threads=threads, - ) - LOGGER.info("Finished mapping rows and columns from sdf.") - gc.collect() - - if merge_type == "replace": - data_dic_self.update(data_dic_sdf) - else: - data_dic_self = Counter(data_dic_self) - data_dic_sdf = Counter(data_dic_sdf) - if merge_type == "add": - data_dic_self += data_dic_sdf - elif merge_type == "min": - data_dic_self &= data_dic_sdf - elif merge_type == "max": - data_dic_self |= data_dic_sdf - else: - raise NotImplementedError("Merge type not implemented.") - del data_dic_sdf - gc.collect() - row, col = zip(*list(data_dic_self.keys())) - values = list(data_dic_self.values()) - del data_dic_self - gc.collect() - self.data_matrix = smat.coo_matrix( - (values, (row, col)), shape=(len(new_i2r), len(new_i2c)) - ).tocsr() - sdf.data_matrix = sdf.data_matrix.tocsr() # changed back to original format - self.i2r = new_i2r - self.r2i = new_r2i - self.c2i = new_c2i - self.i2c = new_i2c - self.shape = self.data_matrix.shape - gc.collect() - - def vstack(self, sdf): - """ - In place version of vstack. The columns of sdf are assumed - to be a subset of columns of self. - If there is a row in sdf which is already present in self, - it is ignored and not added to self. - A usecase of this is where self contains the phrasedocs - for original + expanded + pecos inferred and - sdf points to predictions on unseen queries. In this use case, - the columns (asins) are assumed to be the - same whereas unseen queries (by definition) dont have phrase doc scores. - If there are unseen queries which - are already present in the set of (original + expanded + pecos inferred), - we ignore them and keep the original - data unchanged. - - Parameters - ------------ - sdf (sparseDataFrame): The sparse dataframe to vstack. - Sife Effect - ------------ - Inplace update of self such that the resulting sparse matrix is a - vertical stack of self and sdf. - Returns - ----------- - Nothing - """ - num_rows_original = len(self.i2r) - - # Get the rows which are not overlapping in the same order as in original. - non_overlapping_rows = [] - non_overlapping_indices = [] - for i in range(len(sdf.i2r)): - row = sdf.i2r[i] - if row not in self.r2i: - non_overlapping_rows.append(row) - non_overlapping_indices.append(i) - - # Append the rows from sdf to the end of the rows of self.data_matrix - # by updating the mapping. - for i, r in enumerate(non_overlapping_rows): - self.i2r[num_rows_original + i] = r - self.r2i[r] = num_rows_original + i - - non_overlapping_m = sdf.data_matrix[non_overlapping_indices].tocoo() - # Map the columns to the indices of self. - col_indices = [self.c2i[sdf.i2c[i]] for i in non_overlapping_m.col] - updated_m = smat.csr_matrix( - (non_overlapping_m.data, (non_overlapping_m.row, col_indices)), - shape=(len(non_overlapping_rows), len(self.c2i)), - ) - self.data_matrix = smat.vstack([self.data_matrix, updated_m]) - - @staticmethod - def _merge_index_dictionaries(dic_one, dic_two): - """ - Merge dic_one and dic_two and create new index dictionaries. - """ - new_o2i = copy.deepcopy(dic_one) - new_o2i.update(dic_two) - new_i2o = dict() - rows = new_o2i.keys() - for i, r in enumerate(rows): - new_o2i[r] = i - new_i2o[i] = r - del rows - gc.collect() - return new_i2o, new_o2i - - @staticmethod - def _map_rows_cols_join(data_matrix, i2r, i2c, new_r2i, new_c2i, threads=1): - """ - Helper function to map rows and columns to new indices, - during join. - """ - global _FUNC - row_mapper = np.vectorize(lambda x: new_r2i[i2r[x]]) - column_mapper = np.vectorize(lambda x: new_c2i[i2c[x]]) - LOGGER.info("Created vectorized mappings of rows and columns") - rows, cols = ( - parallel_array_apply(data_matrix.row, row_mapper, threads=threads), - parallel_array_apply(data_matrix.col, column_mapper, threads=threads), - ) - _FUNC = None - gc.collect() - keys = tuple(zip(rows, cols)) - del rows, cols - gc.collect() - data_dic = dict() - data_dic.update(zip(keys, data_matrix.data)) - del keys - gc.collect() - return data_dic diff --git a/examples/qp2q/requirements.txt b/examples/qp2q/requirements.txt deleted file mode 100644 index df2b8bb8..00000000 --- a/examples/qp2q/requirements.txt +++ /dev/null @@ -1,9 +0,0 @@ -scikit-learn==0.24.2 -scipy==1.10.0 -numpy==1.22.0 -pandas==1.0.1 -tqdm==4.66.3 -pygtrie==2.4.2 -pure-predict==0.0.4 -libpecos==0.1.0 - diff --git a/examples/qp2q/utils/Config.py b/examples/qp2q/utils/Config.py deleted file mode 100644 index d8f3fd90..00000000 --- a/examples/qp2q/utils/Config.py +++ /dev/null @@ -1,113 +0,0 @@ -import json -import random -import os -import numpy as np -from pathlib import Path -import logging - -LOGGER = logging.getLogger(__name__) - - -class Config(object): - def __init__(self, filename=None): - - self.config_name = filename - - self.base_res_dir = "../results" # Directory where result folder is created - self.exp_id = "0_Debug" - self.seed = 0 - - # Data specific params - self.data_type = "aol" - self.fdir = "../data/aol/train" # Directory where training data files are stored - self.f_compressed = False # Are training data files compressed or not? - - self.misc = "" # Suffix to append to result dir - - self.mode = "train" - self.n_jobs = 8 - - # Model specific params - self.query_prefix_delimiter = ( - "<@@>" # Delimiter used when concatenating query and prefix in the input - ) - self.pref_vectorizer = "poswgtd_c-tfidf" # Type of prefix vectorizer - self.prefix_vect_data = "train_data" # What data to use to train vectorizer for prefix - - self.use_label_feat = False # Append label text features instead of PIFA from prefix when creating label embedding for indexing - self.label_vectorizer = "use_prefix_vectorizer" # Type of label vectorizer - self.label_vect_data = "train_data" # data used to train a label vectorizer - - self.indexer_type = "hierarchicalkmeans" # Type of label indexing method to use. - self.depth = ( - -1 - ) # This is used for depth of Trie Index, or together with must-link-constraints etc - - self.imb_depth = 100 # imbalance_depth param used for Hierarchical kmeans - self.imb_ratio = 0.0 # imbalance_ratio param used for Hierarchical kmeans - self.nr_splits = 2 # Branching factor param for Hierarchical kmeans - - if filename is not None: - self.__dict__.update(json.load(open(filename))) - - self.np_seed = None - self.update_random_seeds(self.seed) - - def to_json(self): - return json.dumps(filter_json(self.__dict__), indent=4, sort_keys=True) - - def save_config(self, exp_dir, filename="train_config.json"): - Path(exp_dir).mkdir(exist_ok=True, parents=True) - with open(os.path.join(exp_dir, filename), "w") as fout: - fout.write(self.to_json()) - fout.write("\n") - - def __getstate__(self): - state = dict(self.__dict__) - - return state - - @property - def result_dir(self): - - # Update model name using selected config params - model_name = "d={d}_v={v}_i={i}_s={s}{m}".format( - d=self.data_type, - v=self.pref_vectorizer, - i=self.indexer_type.lower(), - s=self.seed, - m="_{}".format(self.misc) if self.misc != "" else "", - ) - - LOGGER.info("Model name is = {}".format(model_name)) - result_dir = "{base}/{exp_id}/{model}".format( - base=self.base_res_dir, exp_id=self.exp_id, model=model_name - ) - - LOGGER.info("Updated res dir is = {}".format(result_dir)) - return result_dir - - def update_random_seeds(self, random_seed): - - self.seed = random_seed - random.seed(random_seed) - - self.np_seed = random.randint(0, 1000) - np.random.seed(self.np_seed) - - -def filter_json(the_dict): - res = {} - for k in the_dict.keys(): - if ( - type(the_dict[k]) is str - or type(the_dict[k]) is float - or type(the_dict[k]) is int - or type(the_dict[k]) is list - or type(the_dict[k]) is bool - or the_dict[k] is None - ): - res[k] = the_dict[k] - elif type(the_dict[k]) is dict: - res[k] = filter_json(the_dict[k]) - return res diff --git a/examples/qp2q/utils/__init__.py b/examples/qp2q/utils/__init__.py deleted file mode 100644 index d933a529..00000000 --- a/examples/qp2q/utils/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# Init File for the package diff --git a/examples/qp2q/utils/analyse_eval_results.py b/examples/qp2q/utils/analyse_eval_results.py deleted file mode 100644 index 953a8fe9..00000000 --- a/examples/qp2q/utils/analyse_eval_results.py +++ /dev/null @@ -1,313 +0,0 @@ -import os -import csv -import sys -import json -import pickle -import logging -import argparse -import numpy as np -from tqdm import tqdm -from pygtrie import CharTrie -from collections import defaultdict -from nltk.translate import bleu_score -from nltk.translate.bleu_score import SmoothingFunction - - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -def calc_mrr(gt_data_iterator, pred_data_iterator): - all_mrr = [] - all_bleu = [] - for j, (gt_label, preds) in tqdm( - enumerate(zip(gt_data_iterator, pred_data_iterator)), total=len(gt_data_iterator) - ): - preds = [p for p in json.loads(preds.lower())] - mrr = 0.0 - for i, curr_pred in enumerate(preds): - if gt_label == curr_pred: - mrr = 1.0 / (i + 1) - break - - all_mrr.append(mrr) - wgtd_bleu_score = 0.0 - normalizer = 0.0 - for i, curr_pred in enumerate(preds): - wgtd_bleu_score += ( - bleu_score.sentence_bleu( - [gt_label.split()], - curr_pred.split(), - smoothing_function=SmoothingFunction().method1, - ) - / (i + 1) - ) - normalizer += 1.0 / (i + 1) - wgtd_bleu_score = wgtd_bleu_score / normalizer if normalizer > 0 else wgtd_bleu_score - all_bleu.append(wgtd_bleu_score) - - return { - "mrr": np.mean(all_mrr) if len(all_mrr) > 0 else 0.0, - "mrr_err": np.std(all_mrr) if len(all_mrr) > 0 else 0.0, - "bleu": np.mean(all_bleu) if len(all_bleu) > 0 else 0.0, - "bleu_err": np.std(all_bleu) if len(all_bleu) > 0 else 0.0, - "num_points": len(all_mrr), - } - - -def get_data_by_freq_and_pref_len(gt_filename, pred_filename, label_trie, only_seen_labels): - - freq_to_data = defaultdict(list) - pref_len_to_data = defaultdict(list) - with open(gt_filename, "r") as all_gt_file, open(pred_filename, "r") as main_pred_file: - for line_ctr, (line_gt, line_pred) in enumerate(zip(all_gt_file, main_pred_file)): - gt_dict = json.loads(line_gt) - gt_label = gt_dict["next_query"].lower() # ground truth next_query - prefix = gt_dict["prefix"].lower() # prefix - - curr_label_freq = label_trie[gt_label] if gt_label in label_trie else 0 - - freq_to_data[curr_label_freq] += [(gt_label, line_pred)] - if only_seen_labels: - if curr_label_freq > 0: - pref_len_to_data[len(prefix)] += [(gt_label, line_pred)] - else: - pass - else: - pref_len_to_data[len(prefix)] += [(gt_label, line_pred)] - - new_freq_to_data = defaultdict(lambda: ([], [])) - for freq in freq_to_data: - gt_labels, preds = zip(*freq_to_data[freq]) - new_freq_to_data[freq] = list(gt_labels), list(preds) - - new_pref_len_to_data = defaultdict(lambda: ([], [])) - for pref_len in pref_len_to_data: - gt_labels, preds = zip(*pref_len_to_data[pref_len]) - new_pref_len_to_data[pref_len] = list(gt_labels), list(preds) - - return new_freq_to_data, new_pref_len_to_data - - -def write_results(pred_fname, gt_fname, feat_name, feat_to_scores, res_file): - eval_metrics = list(calc_mrr([], []).keys()) - fieldnames = [feat_name] + eval_metrics + ["pred_fname", "gt_fname"] - - with open(res_file, "w") as f: - csv_writer = csv.DictWriter(f, fieldnames=fieldnames) - csv_writer.writeheader() - for feat in sorted(feat_to_scores): - temp_dict = {feat_name: feat, "pred_fname": pred_fname, "gt_fname": gt_fname} - temp_dict.update(feat_to_scores[feat]) - csv_writer.writerow(temp_dict) - - -def get_bucketed_scores(feat_to_data, feat_buckets): - feat_bkt_to_scores = {} - for min_feat_val, max_feat_val in zip(feat_buckets[:-1], feat_buckets[1:]): - gt_data = [] - pred_data = [] - for curr_feat_val in range(min_feat_val, max_feat_val): - gt_data += feat_to_data[curr_feat_val][0] - pred_data += feat_to_data[curr_feat_val][1] - - feat_bkt_to_scores[min_feat_val, max_feat_val] = calc_mrr( - gt_data_iterator=gt_data, pred_data_iterator=pred_data - ) - - return feat_bkt_to_scores - - -def analyse(gt_filename, pred_filename, label_trie, only_seen_labels): - - freq_to_data, pref_len_to_data = get_data_by_freq_and_pref_len( - gt_filename=gt_filename, - pred_filename=pred_filename, - label_trie=label_trie, - only_seen_labels=only_seen_labels, - ) - - freq_to_scores = { - freq: calc_mrr(gt_data_iterator=gt_data, pred_data_iterator=pred_data) - for freq, (gt_data, pred_data) in freq_to_data.items() - } - - pref_len_to_scores = { - pref_len: calc_mrr(gt_data_iterator=gt_data, pred_data_iterator=pred_data) - for pref_len, (gt_data, pred_data) in pref_len_to_data.items() - } - - out_dir = os.path.dirname(pred_filename) - - test_fname = gt_filename.split("/")[-1] - # Write results for each label freq to csv file - res_file = f"{out_dir}/{test_fname}_eval_res_by_seen_in_train_bkt_{only_seen_labels}.csv" - bktd_freq_to_scores = get_bucketed_scores(feat_to_data=freq_to_data, feat_buckets=[0, 1, 80000]) - write_results( - pred_fname=pred_filename, - feat_name="freq_bkt", - feat_to_scores=bktd_freq_to_scores, - res_file=res_file, - gt_fname=gt_filename, - ) - - res_file = f"{out_dir}/{test_fname}_eval_res_by_freq_{only_seen_labels}.csv" - write_results( - pred_fname=pred_filename, - feat_name="freq", - feat_to_scores=freq_to_scores, - res_file=res_file, - gt_fname=gt_filename, - ) - - res_file = f"{out_dir}/{test_fname}_eval_res_by_freq_bkt_{only_seen_labels}.csv" - bktd_freq_to_scores = get_bucketed_scores( - feat_to_data=freq_to_data, feat_buckets=[0, 1, 32, 128, 512, 2048, 8192, 32000, 80000] - ) - write_results( - pred_fname=pred_filename, - feat_name="freq_bkt", - feat_to_scores=bktd_freq_to_scores, - res_file=res_file, - gt_fname=gt_filename, - ) - - # Write results for each label prefix-length to csv file - res_file = f"{out_dir}/{test_fname}_eval_res_by_pref_len_{only_seen_labels}.csv" - write_results( - pred_fname=pred_filename, - feat_name="pref_len", - feat_to_scores=pref_len_to_scores, - res_file=res_file, - gt_fname=gt_filename, - ) - - # Write results for each label prefix-length to csv file bucketed - max_pref_bkt = 26 - max_pref = max(pref_len_to_scores.keys()) - pref_feat_bkts = list(range(max_pref_bkt)) - if max_pref > max_pref_bkt: - pref_feat_bkts = pref_feat_bkts + [max_pref + 1] - res_file = ( - f"{out_dir}/{test_fname}_eval_res_by_pref_len_bkt_{max_pref_bkt}_{only_seen_labels}.csv" - ) - bktd_pref_len_to_scores = get_bucketed_scores( - feat_to_data=pref_len_to_data, feat_buckets=pref_feat_bkts - ) - write_results( - pred_fname=pred_filename, - feat_name="pref_len_bkt", - feat_to_scores=bktd_pref_len_to_scores, - res_file=res_file, - gt_fname=gt_filename, - ) - - -def main(): - parser = argparse.ArgumentParser( - description="Breakdown performance of model given prediction file" - ) - parser.add_argument( - "--gt_file", type=str, required=True, help="file containing ground-truth data" - ) - parser.add_argument( - "--pred_file", type=str, required=True, help="file containing prediction data" - ) - parser.add_argument( - "--label_file", type=str, default=None, help="file containing label freq trie" - ) - - np.random.seed(0) - - args = parser.parse_args() - _gt_file = args.gt_file - _pred_file = args.pred_file - _label_file = args.label_file if args.label_file != "" else None - - LOGGER.info("Loading label freq trie from {}".format(_label_file)) - if _label_file is None: - label_trie = defaultdict(int) - else: - with open(_label_file, "rb") as f: - label_trie = pickle.load(f) - assert isinstance(label_trie, CharTrie) - LOGGER.info("Loaded label freq trie") - - analyse(gt_filename=_gt_file, pred_filename=_pred_file, label_trie=label_trie) - - -def final_paper_analysis(): - parser = argparse.ArgumentParser( - description="Breakdown performance of model given prediction file" - ) - parser.add_argument("--opt", type=int, default=-1, help="option") - parser.add_argument( - "--base_dir", type=str, required=True, help="dir with all relevant result folder" - ) - parser.add_argument( - "--label_file", - type=str, - default="", - help="pickle file containing dict/trie that maps " "label to its frequency", - ) - parser.add_argument("--gt_file", type=str, required=True, help="json file containing gt data") - args = parser.parse_args() - opt = args.opt - base_dir = args.base_dir - label_file = args.label_file - gt_file = args.gt_file - - pecos_08 = f"{base_dir}/7_ReproduceRes/d=aol_v=poswgtd_c-tfidf_i=hybridindexer_s=0_08" - freq = f"{base_dir}/8_FinalResults/PrefFreqSuggester" - seq2seqGRU = f"{base_dir}/8_Baseline/d=aol_m=HRED_s=0" - seq2seqGRU_freq = f"{base_dir}/8_Baseline/d=aol_m=HRED_s=0_rerank" - - LOGGER.info("Loading label freq trie from {}".format(label_file)) - if label_file is None: - label_trie = defaultdict(int) - only_seen_labels = False - else: - label_trie = pickle.load(open(label_file, "rb")) - only_seen_labels = True - assert isinstance(label_trie, CharTrie) - LOGGER.info("Loaded label freq trie") - - if opt == 1: - all_dirs = [pecos_08] - elif opt == 2: - all_dirs = [freq] - elif opt == 3: - all_dirs = [seq2seqGRU] - elif opt == 4: - all_dirs = [seq2seqGRU_freq] - else: - all_dirs = [pecos_08, freq, seq2seqGRU, seq2seqGRU_freq] - - for curr_dir in tqdm(all_dirs): - print("Running for dir = {}".format(curr_dir)) - try: - pred_file = f"{curr_dir}/eval/{curr_dir.split('/')[-1]}_test.json.pred_data" - analyse( - gt_filename=gt_file, - pred_filename=pred_file, - label_trie=label_trie, - only_seen_labels=only_seen_labels, - ) - analyse( - gt_filename=gt_file, - pred_filename=pred_file, - label_trie=defaultdict(int), - only_seen_labels=False, - ) - except Exception as e: - print(f"1) Error processing {curr_dir} {e}") - - -if __name__ == "__main__": - # main() - final_paper_analysis() diff --git a/examples/qp2q/utils/create_label_freq_trie.py b/examples/qp2q/utils/create_label_freq_trie.py deleted file mode 100644 index 458b26ad..00000000 --- a/examples/qp2q/utils/create_label_freq_trie.py +++ /dev/null @@ -1,70 +0,0 @@ -import os -import sys -import pickle -import logging -import argparse -import numpy as np -from pathlib import Path -from pygtrie import CharTrie - -from qp2q.preprocessing.session_data_processing import parallel_get_qp2q_sparse_data -from qp2q.preprocessing.sparse_data_processing import SparseDataFrame - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -def create_query_freq_based_trie(fdir, f_compressed, fname): - - i2r, i2c, smat = parallel_get_qp2q_sparse_data(fdir=fdir, compressed=f_compressed, n_jobs=16) - sdf = SparseDataFrame(data_matrix=smat, columns=i2c, rows=i2r) - - LOGGER.info("Created sparsedataframe") - - trie = CharTrie() - - LOGGER.info("Computed sum of each col") - flat_mat = np.sum(sdf.data_matrix, axis=0) # Compute sum for each col - assert flat_mat.shape == (1, sdf.data_matrix.shape[1]) - - LOGGER.info("Creating label_freq dictionary") - label_freq = {c: flat_mat[0, sdf.c2i[c]] for c in sdf.c2i} - LOGGER.info("Created label_freq dictionary") - - trie.update(label_freq) - LOGGER.info("Created trie") - - res_dir = os.path.dirname(fname) - Path(res_dir).mkdir(exist_ok=True, parents=True) - with open("{}".format(fname), "wb") as trie_file: - pickle.dump(trie, trie_file, protocol=pickle.HIGHEST_PROTOCOL) - - LOGGER.info("Saved trie in {}/{}".format(res_dir, fname)) - - -def main(): - parser = argparse.ArgumentParser(description="Create label frequency trie") - parser.add_argument("--fdir", type=str, required=True, help="path to data files") - parser.add_argument("--f_compressed", type=int, default=0, help="are data files compressed?") - parser.add_argument( - "--out_fname", - type=str, - default="data", - help="name of trie model file used when saving trie", - ) - - args = parser.parse_args() - fdir = args.fdir - f_compressed = bool(args.f_compressed) - fname = args.out_fname - - create_query_freq_based_trie(fdir=fdir, f_compressed=f_compressed, fname=fname) - - -if __name__ == "__main__": - main() diff --git a/examples/qp2q/utils/create_label_space_dict.py b/examples/qp2q/utils/create_label_space_dict.py deleted file mode 100644 index 8f228bb0..00000000 --- a/examples/qp2q/utils/create_label_space_dict.py +++ /dev/null @@ -1,82 +0,0 @@ -import os -import sys -import json -import logging -import argparse -from pathlib import Path - -from qp2q.preprocessing.session_data_processing import parallel_get_qp2q_sparse_data -from qp2q.preprocessing.sparse_data_processing import SparseDataFrame - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -def get_suffixes(query_text): - - q_token_list = query_text.split() - suffixes = [" ".join(q_token_list[ctr:]) for ctr in range(len(q_token_list))] - - return suffixes - - -def create_label_dict(fdir, f_compressed, out_file, add_suffix): - - i2r, i2c, smat = parallel_get_qp2q_sparse_data(fdir=fdir, compressed=f_compressed, n_jobs=8) - sdf = SparseDataFrame(data_matrix=smat, columns=i2c, rows=i2r) - train_next_queries = {q: 1 for q in sdf.c2i} - - out_dir = os.path.dirname(out_file) - Path(out_dir).mkdir(exist_ok=True, parents=True) - - if add_suffix: - LOGGER.info("num_train_labels {}".format(len(train_next_queries))) - train_next_queries = { - q_suffix: 1 for q in train_next_queries for q_suffix in get_suffixes(q) - } - LOGGER.info("num_train_labels with suffixes {}".format(len(train_next_queries))) - json.dump(train_next_queries, open(out_file, "w")) - else: - LOGGER.info("num_train_labels {}".format(len(train_next_queries))) - json.dump(train_next_queries, open(out_file, "w")) - - -def main(): - parser = argparse.ArgumentParser(description="Create a dictionary of labels in training data") - parser.add_argument("--fdir", type=str, required=True, help="training data file directory") - parser.add_argument( - "--out_file", type=str, required=True, help="name of output label dictionary file" - ) - parser.add_argument( - "--f_compressed", - type=int, - choices=[1, 0], - default=0, - help="are files in train data directory compressed. 1 for yes and 0 for no", - ) - parser.add_argument( - "--add_suffix", - type=int, - choices=[1, 0], - default=0, - help="Add suffixes of all labels in the label dictionary. 1 for yes and 0 for no", - ) - - args = parser.parse_args() - _fdir = args.fdir - _out_file = args.out_file - _f_compressed = bool(args.f_compressed) - _add_suffix = bool(args.add_suffix) - - create_label_dict( - fdir=_fdir, out_file=_out_file, f_compressed=_f_compressed, add_suffix=_add_suffix - ) - - -if __name__ == "__main__": - main() diff --git a/examples/qp2q/utils/create_pref_to_top_k_suggestions_dict.py b/examples/qp2q/utils/create_pref_to_top_k_suggestions_dict.py deleted file mode 100644 index 88bb5bda..00000000 --- a/examples/qp2q/utils/create_pref_to_top_k_suggestions_dict.py +++ /dev/null @@ -1,102 +0,0 @@ -import os -import sys -import json -import argparse -import logging -import numpy as np -from tqdm import tqdm -from pathlib import Path -from pygtrie import CharTrie - -from qp2q.preprocessing.session_data_processing import parallel_get_qp2q_sparse_data -from qp2q.preprocessing.sparse_data_processing import SparseDataFrame - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -def get_top_k_w_label_trie(prefix, label_trie, k): - if label_trie.has_subtrie(prefix): # if prefix exists in trie - sorted_labels = [ - (label, freq) - for label, freq in sorted( - label_trie.iteritems(prefix=prefix), key=lambda x: x[1], reverse=True - ) - ] - - return sorted_labels[:k] - else: # if prefix does not exist in trie then get suggestions using smaller prefix - if len(prefix) > 0: - return get_top_k_w_label_trie(prefix=prefix[:-1], label_trie=label_trie, k=k) - else: - return [] - - -def create_pref_to_topk_dict(fdir, f_compressed, fname, k): - - i2r, i2c, smat = parallel_get_qp2q_sparse_data(fdir=fdir, compressed=f_compressed, n_jobs=16) - sdf = SparseDataFrame(data_matrix=smat, columns=i2c, rows=i2r) - - LOGGER.info("Created sparsedataframe") - - flat_mat = np.sum(sdf.data_matrix, axis=0) # Compute sum for each col - LOGGER.info("Computed sum of each col") - assert flat_mat.shape == (1, sdf.data_matrix.shape[1]) - - LOGGER.info("Creating label_freq dictionary") - label_freq = {c: int(flat_mat[0, sdf.c2i[c]]) for c in sdf.c2i} - LOGGER.info("Created label_freq dictionary") - - all_prefs = {label[:i]: [] for label in label_freq for i in range(len(label) + 1)} - LOGGER.info("Number of prefixes = {}".format(len(all_prefs))) - - label_trie = CharTrie() - label_trie.update(label_freq) - LOGGER.info("Created label trie") - - all_prefs_to_topk = { - pref: get_top_k_w_label_trie(prefix=pref, label_trie=label_trie, k=k) - for pref in tqdm(all_prefs) - } - - res_dir = os.path.dirname(fname) - Path(res_dir).mkdir(exist_ok=True, parents=True) - - with open(fname, "w") as f: - json.dump(all_prefs_to_topk, f) - - LOGGER.info("Saved top-k dict in {}".format(fname)) - - -def main(): - parser = argparse.ArgumentParser( - description="Create a dict where key = prefix of a label and value = top-k suggestions matching the given prefix" - ) - parser.add_argument("--fdir", type=str, required=True, help="path to data files") - parser.add_argument("--f_compressed", type=int, default=0, help="are data files compressed?") - parser.add_argument( - "--out_fname", - type=str, - default="data", - help="name of trie model file used when saving trie", - ) - parser.add_argument( - "--k", type=int, required=True, help="max number of suggestions to store for each prefix" - ) - - args = parser.parse_args() - fdir = args.fdir - f_compressed = bool(args.f_compressed) - fname = args.out_fname - k = args.k - - create_pref_to_topk_dict(fdir=fdir, f_compressed=f_compressed, fname=fname, k=k) - - -if __name__ == "__main__": - main() diff --git a/examples/qp2q/utils/process_aol_dataset_from_orig_aol_files.py b/examples/qp2q/utils/process_aol_dataset_from_orig_aol_files.py deleted file mode 100644 index 13959152..00000000 --- a/examples/qp2q/utils/process_aol_dataset_from_orig_aol_files.py +++ /dev/null @@ -1,402 +0,0 @@ -import os -import re -import csv -import sys -import json -import glob -import argparse -import logging -import numpy as np -from pathlib import Path -from datetime import datetime -from collections import defaultdict - -logging.basicConfig( - stream=sys.stdout, - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s ", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, -) -LOGGER = logging.getLogger(__name__) - - -def get_all_data(fname): - """ - - Parameters - ---------- - fname: file containing raw aol data - - Returns dict with key = user_id - value = list of (query, query_time) tuples - ------- - - """ - LOGGER.info("Starting reading all data from {}".format(fname)) - - date_format = "%Y-%m-%d %H:%M:%S" - with open(fname, "r") as f: - reader = csv.reader(f, delimiter="\t") - all_data = defaultdict(list) - header = next(reader, None) - for ctr, row in enumerate(reader): - - user_id = row[0] - q_time = row[2] - query_tokens = [q_token for q_w_dot in row[1].split() for q_token in q_w_dot.split(".")] - query_tokens = [ - re.sub(r"\W+", "", token) for token in query_tokens - ] # Remove non-alpha numberic characters - - query = " ".join(query_tokens) - query = query.lower().strip() - if len(query) == 0: # Skip empty queries - continue - all_data[user_id] += [(query, datetime.strptime(q_time, date_format))] - - LOGGER.info("Finished reading all data from {}".format(fname)) - return all_data - - -def create_sessions(all_data, sess_len): - """ - - Parameters - ---------- - all_data: dict with key = user_id - value = list of (query, query_time) tuples - sess_len : int - Minimum time difference between two queries required to put them in different sessions - Returns list of (sessions, min_session_time, max_session_time) tuples where each session is a list of queries - ------- - - """ - LOGGER.info( - "Splitting data into session using min query time diff = {} seconds".format(sess_len) - ) - all_session_data = [] - for user_id in all_data: - - if len(all_data[user_id]) > 0: - prev_query, prev_query_time = all_data[user_id][0] - curr_session = [prev_query] - curr_session_times = [prev_query_time] - for curr_query, curr_query_time in all_data[user_id][1:]: - diff = curr_query_time - prev_query_time - if diff.total_seconds() > sess_len: # Start new session - if len(curr_session) > 0: - all_session_data += [ - (curr_session, min(curr_session_times), max(curr_session_times)) - ] - - # Start a new session - curr_session = [curr_query] - curr_session_times = [curr_query_time] - else: - curr_session += [curr_query] - curr_session_times += [curr_query_time] - prev_query_time = curr_query_time - - if len(curr_session) > 0: - all_session_data += [ - (curr_session, min(curr_session_times), max(curr_session_times)) - ] - - LOGGER.info( - "Created data w/ {} sessions using min query time diff = {} seconds".format( - len(all_session_data), sess_len - ) - ) - return all_session_data - - -def deduplicate_session_data(all_session_data): - """ - Removes duplicate copies of queries when they appear consecutively in a session - Parameters - ---------- - all_session_data: list of (sessions, min_session_time, max_session_time) tuples where each session is a list of queries - - Returns list of (session, min_session_time, max_session_time) tuples where session is deduplicated - ------- - - """ - # For each session, create prev query and next query pairs - LOGGER.info( - "Removing duplicate consecutive queries from {} sessions".format(len(all_session_data)) - ) - # Deduplicate session data - dedup_sess_data = [] - for session, min_sess_time, max_sess_time in all_session_data: - - prev_query = session[0] if len(session) > 0 else None - dedup_sess = [prev_query] if len(session) > 0 else [] - for curr_query in session: - if curr_query == prev_query: # Ignore duplicate consecutive queries - continue - else: - dedup_sess += [curr_query] - prev_query = curr_query - - if len(dedup_sess) > 1: # Only consider sessions that have at least 2 queries - dedup_sess_data += [(dedup_sess, min_sess_time, max_sess_time)] - - LOGGER.info( - "Finished removing duplicate consecutive queries and we have {} sessions now ".format( - len(dedup_sess_data) - ) - ) - return dedup_sess_data - - -def split_sessions_into_train_test_val(all_sessions): - """ - Splits session data into train/test/val lists based on session start time. - 1 March - 15 May -> Train - 16 May - 23 May -> Test - 24 May - 31 May -> Val - Parameters - ---------- - all_sessions: list of (session, min_session_time, max_session_time) tuples where session is deduplicated - - Returns train/test/val session lists - ------- - - """ - date_format = "%Y-%m-%d %H:%M:%S" - train_time_range = datetime.strptime("2006-03-01 00:00:00", date_format), datetime.strptime( - "2006-05-15 23:59:59", date_format - ) - val_time_range = datetime.strptime("2006-05-16 00:00:00", date_format), datetime.strptime( - "2006-05-23 23:59:59", date_format - ) - test_time_range = datetime.strptime("2006-05-24 00:00:00", date_format), datetime.strptime( - "2006-05-31 23:59:59", date_format - ) - train_sessions, test_sessions, val_sessions = [], [], [] - for curr_session, min_sess_time, max_sess_time in all_sessions: - - assert ( - min_sess_time <= max_sess_time - ), "Min session = {} time should be less than max session time = {}.".format( - min_sess_time, max_sess_time - ) - - if train_time_range[0] <= min_sess_time <= train_time_range[1]: - train_sessions.append(curr_session) - elif val_time_range[0] <= min_sess_time <= val_time_range[1]: - val_sessions.append(curr_session) - elif test_time_range[0] <= min_sess_time <= test_time_range[1]: - test_sessions.append(curr_session) - else: - raise Exception( - "This session range is not handled = ({}, {})".format(min_sess_time, max_sess_time) - ) - - return train_sessions, test_sessions, val_sessions - - -def create_query_pairs(all_session_data): - query_pairs = [ - (prev_query, next_query) - for s_queries in all_session_data - for prev_query, next_query in zip(s_queries[:-1], s_queries[1:]) - ] - return query_pairs - - -def convert_to_train_data_format(query_pairs, out_fname): - """ - Create training data by sampling prefix from next query in the query pairs and write it to a file - Parameters - ---------- - query_pairs: list of (prev_query, next_query) pairs - out_fname : name of file where data is where the data is written - - Returns : None - """ - np.random.seed(0) - Path(os.path.dirname(out_fname)).mkdir(exist_ok=True, parents=True) - with open(out_fname, "w") as writer: - for prev_query, next_query in query_pairs: - # Sample prefix len uniformly at random - pref_len = int(np.random.choice(range(1, len(next_query) + 1), size=1)[0]) - new_datapoint = { - "prev_query": prev_query, - "prefix": next_query[:pref_len], - "next_query": next_query, - } - writer.write(json.dumps(new_datapoint) + "\n") - - LOGGER.info("Created train dataset") - - -def convert_to_gt_data_format(query_pairs, out_fname): - """ - Create test data by sampling prefix from next query in the query pairs and write it to a file - Parameters - ---------- - query_pairs: list of (prev_query, next_query) pairs - out_fname : name of file where data is where the data is written - - Returns : None - """ - np.random.seed(0) - Path(os.path.dirname(out_fname)).mkdir(exist_ok=True, parents=True) - with open(out_fname, "w") as writer: - for prev_query, next_query in query_pairs: - pref_len = int( - np.random.choice(range(1, len(next_query) + 1), size=1)[0] - ) # Uniformly sample a prefix - prefix = next_query[:pref_len] - output = {"prev_query": prev_query, "prefix": prefix, "next_query": next_query} - writer.write(json.dumps(output) + "\n") - - LOGGER.info("Created test dataset") - - -def main(): - parser = argparse.ArgumentParser(description="Process AOL Dataset (from original AOL files)") - parser.add_argument( - "--data_dir", - type=str, - required=True, - help="Directory where all raw AOL log files are stored", - ) - parser.add_argument("--out_dir", type=str, required=True, help="Dir to store out file") - parser.add_argument( - "--sess_len", - type=int, - default=1800, - help="Min time gap (in seconds) b/w two consecutive queries to put them in different sessions ", - ) - - np.random.seed(0) - - args = parser.parse_args() - data_dir = args.data_dir - out_dir = args.out_dir - sess_len = args.sess_len - - Path(out_dir).mkdir(exist_ok=True, parents=True) - - os.system("cp {} {}/".format(sys.argv[0], out_dir)) - os.system( - "echo {} >> {}/command_to_run.txt".format(" ".join([str(x) for x in sys.argv]), out_dir) - ) - - data_files = sorted([str(f) for f in glob.glob("{}/*.txt".format(data_dir))]) - LOGGER.info("Data files = {}".format(data_files)) - train_sessions = [] - test_sessions = [] - val_sessions = [] - all_sessions = [] - for curr_file in data_files: - LOGGER.info("\nProcessing file = {}\n".format(curr_file)) - all_data = get_all_data(fname=curr_file) - dedup_sessions = deduplicate_session_data( - all_session_data=create_sessions(all_data=all_data, sess_len=sess_len) - ) - curr_train, curr_test, curr_val = split_sessions_into_train_test_val(dedup_sessions) - - all_sessions += dedup_sessions - train_sessions += curr_train - val_sessions += curr_val - test_sessions += curr_test - - assert len(train_sessions) + len(val_sessions) + len(test_sessions) == len( - all_sessions - ), "{} + {} + {} != {}".format( - len(train_sessions), len(val_sessions), len(test_sessions), len(all_sessions) - ) - - LOGGER.info("Total number of sessions = {}".format(len(all_sessions))) - LOGGER.info( - "Total number of query pairs = {}".format( - len(create_query_pairs([sess for sess, _, _ in all_sessions])) - ) - ) - LOGGER.info( - "Total number of unique query pairs = {}".format( - len(set(create_query_pairs([sess for sess, _, _ in all_sessions]))) - ) - ) - LOGGER.info( - "Total number of queries = {}".format(len([q for sess, _, _ in all_sessions for q in sess])) - ) - LOGGER.info( - "Total number of unique queries = {}".format( - len(set([q for sess, _, _ in all_sessions for q in sess])) - ) - ) - - LOGGER.info("Total number of train sessions = {}".format(len(train_sessions))) - LOGGER.info( - "Total number of train query pairs = {}".format(len(create_query_pairs(train_sessions))) - ) - LOGGER.info( - "Total number of train unique query pairs = {}".format( - len(set(create_query_pairs(train_sessions))) - ) - ) - LOGGER.info( - "Total number of train queries = {}".format( - len([q for sess in train_sessions for q in sess]) - ) - ) - LOGGER.info( - "Total number of unique train queries = {}".format( - len(set([q for sess in train_sessions for q in sess])) - ) - ) - - LOGGER.info("Total number of test sessions = {}".format(len(test_sessions))) - LOGGER.info( - "Total number of test query pairs = {}".format(len(create_query_pairs(test_sessions))) - ) - LOGGER.info( - "Total number of test unique query pairs = {}".format( - len(set(create_query_pairs(test_sessions))) - ) - ) - LOGGER.info( - "Total number of test queries = {}".format(len([q for sess in test_sessions for q in sess])) - ) - LOGGER.info( - "Total number of unique test queries = {}".format( - len(set([q for sess in test_sessions for q in sess])) - ) - ) - - LOGGER.info("Total number of val sessions = {}".format(len(val_sessions))) - LOGGER.info( - "Total number of val query pairs = {}".format(len(create_query_pairs(val_sessions))) - ) - LOGGER.info( - "Total number of val unique query pairs = {}".format( - len(set(create_query_pairs(val_sessions))) - ) - ) - LOGGER.info( - "Total number of val queries = {}".format(len([q for sess in val_sessions for q in sess])) - ) - LOGGER.info( - "Total number of unique val queries = {}".format( - len(set([q for sess in val_sessions for q in sess])) - ) - ) - - convert_to_train_data_format( - query_pairs=create_query_pairs(train_sessions), - out_fname="{}/train/train.json".format(out_dir), - ) - convert_to_train_data_format( - query_pairs=create_query_pairs(val_sessions), out_fname="{}/val/val.json".format(out_dir) - ) - convert_to_gt_data_format( - query_pairs=create_query_pairs(test_sessions), out_fname="{}/test/test.json".format(out_dir) - ) - - -if __name__ == "__main__": - main() diff --git a/examples/spmm/README.md b/examples/spmm/README.md deleted file mode 100644 index fddecc6d..00000000 --- a/examples/spmm/README.md +++ /dev/null @@ -1,109 +0,0 @@ - -# PECOS for Sparse-to-Sparse Matrix Multiplication (SpMM) -**Sp**arse-to-sparse **M**atrix **M**ultiplication (**SpMM**) is one of the key primitives in large-scale linear algebra operations, with a broad range of applications in machine learning and natural language processing. For example, a graph convolution step on sparse input features involves a SpMM operation. Another usage of SpMM is the computation of PIFA features in e**X**treme **M**ulti-label **C**lassification (**XMC**) community that aggregate sparse input TFIDF features associated with a label as its label embedding. - -However, to the best of our knowledge, very few linear algebra libraries support **SpMM** with fast parallelism on CPU machines. Therefore, we enable PECOS with a highly optimized multi-core CPU implementation for the **SpMM** operation. See the [Python API usage](#Python-API-Usage) and [Benchmarking Results](#Benchmarking-Results) for more details. - -

- -

- - -## Requirements and Installation -To use PECOS SpMM functionality without comparing to baselines, just pip install libpecos. -``` bash - pip install libpecos -``` - -## Python API Usage - -Perform a **CPU parallel** SpMM operation of the sparse matrix `X` and the sparse matrix `Y`. -``` python ->>> from pecos.core import clib as pecos_clib ->>> Z = pecos_clib.sparse_matmul(X, Y, eliminate_zeros=False, sorted_indices=True, threads=-1) -``` - -### Parameters -- `X` ([`scipy.sparse.csr_matrix`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) or [`scipy.sparse.csc_matrix`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html)): the first *sparse* matrix to be multiplied. -- `Y` ([`scipy.sparse.csr_matrix`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) or [`scipy.sparse.csc_matrix`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html)): the second *sparse* matrix to be multiplied. -- `eliminate_zeros` (bool, optional): if true, then eliminate (potential) zeros created by maxnnz in output matrix Z. Default is false. -- `sorted_indices` (bool, optional): if true, then sort the Z.indices for the output matrix Z. Default is true. -- `threads` (int, optional): The number of threads. Default -1 to use all CPU cores. - -### Toy Examples -``` python ->>> import numpy as np ->>> import scipy.sparse as smat ->>> from scipy.sparse import linalg ->>> from pecos.core import clib as pecos_clib ->>> X = smat.random(1000, 1000, density=0.01, format='csr', dtype=np.float32) ->>> Y = smat.random(1000, 1000, density=0.01, format='csr', dtype=np.float32) ->>> Z_true = X.dot(Y) ->>> Z_pred = pecos_clib.sparse_matmul(X, Y) ->>> print("||Z_true - Z_pred|| = ", linalg.norm(Z_true - Z_pred)) -``` - -## Benchmarking Results -We compare PECOS running time with a few popular linear algebra packages, including SciPy, Intel-MKL, Pytorch, and Tensorflow. -* Intel-MKL: `pip install sparse-dot-mkl==0.7.3` (Note that it requires MKL library, see [link](https://github.com/flatironinstitute/sparse_dot#requirements)) -* PECOS: `pip install libpecos==0.1.0` -* Pytorch: `pip install torch==1.9.0+cpu` -* Tensorflow: `pip install tensorflow==2.5.0` - -All the experiment results are conducted on a AWS [x1.32xlarge](https://aws.amazon.com/ec2/instance-types/x1/) instance with 128 CPU and 1.9T memory. -We note that Pytorch and Tensoflow results are from the same CPU machine without using any GPU. - -### Requirements -We also provide the conda environment with the those libraries installed for you to reproduce the results -``` bash - conda env create -f conda_env.yml - conda activate pecos-spmm -``` - -### Problem Setup -We consider benchmarking the SpMM operation `Z=(Y.T).dot(X)`, where -* `X` is the sparse instance-to-feature TFIDF matrix of shape `N by D` -* `Y` is the sparse instance-to-label relevant matrix of shape `N by L` - -The input matrices are from public XMC benchmark datasets as follows. - -### Data Statistics -The XMC datasets can be download at -``` bash -# eurlex-4k, wiki10-31k, amazoncat-13k, amazon-670k, wiki-500k, amazon-3m -DATASET="wiki10-31k" -wget https://archive.org/download/pecos-dataset/xmc-base/${DATASET}.tar.gz -tar -zxvf ./${DATASET}.tar.gz -``` - -| Data | N (#instance) | D (#feature) | L (#label) | nnz(X) | nnz(Y) | nnz(Z) -| :---- | ----: | ----: | ----: | ----: | ----: |----: | -| eurlex-4k | 15,449 | 186,104 | 3,956 | 4,194,123 | 82,265 | 6,126,348 -| wiki10-31k | 14,146 | 101,938 | 30,938 | 9,526,572 | 263,705 | 72,574,211 -| amazoncat-13k | 1,186,239 | 203,882 | 13,330 | 84,415,397 | 5,979,439 | 33,409,040 -| amazon-670k | 490,449 | 135,909 | 670,091 | 37,119,040 | 2,674,356 | 146,741,011 -| wiki-500k | 1,779,881 | 2,381,304 | 501,070 | 689,526,754 | 8,446,236 | 1,255,206,075 -| amazon-3m | 1,717,899 | 337,067 | 2,812,281 | 84,600,285 | 61,916,857 | 1,375,859,565 - -From the following, we assume all ${DATASET} sub-folders are located at `./data/${DATASET}` - - -### Multi-thread Comparison -``` bash -bash run_exp.sh ${DATASET} multi-thread -``` - -

- -

- -### Single-thread Comparison -``` bash -bash run_exp.sh ${DATASET} single-thread -``` -

- -

- - - diff --git a/examples/spmm/conda_env.yml b/examples/spmm/conda_env.yml deleted file mode 100644 index 05d85868..00000000 --- a/examples/spmm/conda_env.yml +++ /dev/null @@ -1,99 +0,0 @@ -name: pecos-spmm -channels: - - intel - - defaults -dependencies: - - _libgcc_mutex=0.1=main - - _openmp_mutex=4.5=1_gnu - - ca-certificates=2020.12.5=0 - - certifi=2020.12.5=py38he139614_0 - - intel-openmp=2021.3.0=intel_3350 - - intelpython=2021.3.0=7 - - ld_impl_linux-64=2.35.1=h7274673_9 - - libffi=3.3=he6710b0_2 - - libgcc-ng=9.3.0=h5101ec6_17 - - libgomp=9.3.0=h5101ec6_17 - - libstdcxx-ng=9.3.0=hd4cf53a_17 - - mkl=2021.3.0=intel_520 - - ncurses=6.2=he6710b0_1 - - openssl=1.1.1k=h14c3975_1 - - pip=21.1.3=py38h06a4308_0 - - python=3.8.10=h12debd9_8 - - readline=8.1=h27cfd23_0 - - setuptools=52.0.0=py38h06a4308_0 - - sqlite=3.36.0=hc218d9a_0 - - tbb=2021.3.0=intel_511 - - tk=8.6.10=hbc83047_0 - - wheel=0.36.2=pyhd3eb1b0_0 - - xz=5.2.5=h7b6447c_0 - - zlib=1.2.11=h7b6447c_3 - - pip: - - absl-py==0.13.0 - - astunparse==1.6.3 - - backcall==0.2.0 - - cachetools==4.2.2 - - charset-normalizer==2.0.3 - - click==8.0.1 - - decorator==5.0.9 - - filelock==3.0.12 - - flatbuffers==1.12 - - gast==0.4.0 - - google-auth==1.33.0 - - google-auth-oauthlib==0.4.4 - - google-pasta==0.2.0 - - grpcio==1.34.1 - - h5py==3.1.0 - - huggingface-hub==0.0.12 - - idna==3.2 - - ipython==7.25.0 - - ipython-genutils==0.2.0 - - jedi==0.18.0 - - joblib==1.0.1 - - keras-nightly==2.5.0.dev2021032900 - - keras-preprocessing==1.1.2 - - libpecos==0.1.0 - - markdown==3.3.4 - - matplotlib-inline==0.1.2 - - numpy==1.19.5 - - oauthlib==3.1.1 - - opt-einsum==3.3.0 - - packaging==21.0 - - parso==0.8.2 - - pexpect==4.8.0 - - pickleshare==0.7.5 - - prompt-toolkit==3.0.19 - - protobuf==3.17.3 - - ptyprocess==0.7.0 - - pyasn1==0.4.8 - - pyasn1-modules==0.2.8 - - pygments==2.9.0 - - pyparsing==2.4.7 - - pyyaml==5.4.1 - - regex==2021.7.6 - - requests==2.26.0 - - requests-oauthlib==1.3.0 - - rsa==4.7.2 - - sacremoses==0.0.45 - - scikit-learn==0.24.2 - - scipy==1.7.0 - - sentencepiece==0.1.96 - - six==1.15.0 - - sparse-dot-mkl==0.7.3 - - tensorboard==2.5.0 - - tensorboard-data-server==0.6.1 - - tensorboard-plugin-wit==1.8.0 - - tensorflow==2.5.0 - - tensorflow-estimator==2.5.0 - - termcolor==1.1.0 - - threadpoolctl==2.2.0 - - tokenizers==0.10.3 - - torch==1.9.0 - - tqdm==4.61.2 - - traitlets==5.0.5 - - transformers==4.8.2 - - typing-extensions==3.7.4.3 - - urllib3==1.26.6 - - wcwidth==0.2.5 - - werkzeug==2.0.1 - - wrapt==1.12.1 -prefix: /home/ec2-user/miniconda3/envs/pecos-spmm diff --git a/examples/spmm/figs/pecos-spmm_example-v1.png b/examples/spmm/figs/pecos-spmm_example-v1.png deleted file mode 100644 index 6f7278b4..00000000 Binary files a/examples/spmm/figs/pecos-spmm_example-v1.png and /dev/null differ diff --git a/examples/spmm/figs/pecos-spmm_example-v2.png b/examples/spmm/figs/pecos-spmm_example-v2.png deleted file mode 100644 index 4460116b..00000000 Binary files a/examples/spmm/figs/pecos-spmm_example-v2.png and /dev/null differ diff --git a/examples/spmm/figs/pecos-spmm_example-v3.png b/examples/spmm/figs/pecos-spmm_example-v3.png deleted file mode 100644 index 2f0a6fe5..00000000 Binary files a/examples/spmm/figs/pecos-spmm_example-v3.png and /dev/null differ diff --git a/examples/spmm/run_exp.py b/examples/spmm/run_exp.py deleted file mode 100644 index a1fa771b..00000000 --- a/examples/spmm/run_exp.py +++ /dev/null @@ -1,147 +0,0 @@ - -import argparse -import os -import time -import numpy as np -import scipy.sparse as smat -from pecos.utils import smat_util -from pecos.core import clib as pecos_clib - -SPMM_ALGO_LIST = ["pecos", "tensorflow", "pytorch", "intel-mkl", "scipy"] - - -def csr_to_coo(A): - A_coo = smat.coo_matrix(A) - # (nnz, 2) - indices = np.vstack([A_coo.row, A_coo.col]).T - # (nnz, ) - values = A_coo.data - return indices, values - - -def do_spmm_exp(args): - # load data - Y = smat_util.load_matrix(args.y_npz_path).astype(np.float32) - X = smat_util.load_matrix(args.x_npz_path).astype(np.float32) - YT_csr = Y.T.tocsr() - X_csr = X.tocsr() - - # The #threads is control by env variables (except for pecos) - # e.g., export OMP_NUM_THREADS=16, export MKL_NUM_THREADS=16. - run_time = 0.0 - if args.spmm_algo == "pecos": - start = time.time() - Z = pecos_clib.sparse_matmul( - YT_csr, X_csr, - eliminate_zeros=False, - sorted_indices=True, - threads=args.threads, - ) - run_time += time.time() - start - Z_data = Z.data - elif args.spmm_algo == "intel-mkl": - from sparse_dot_mkl import dot_product_mkl - # make sure set the index to int64 for large matrices - # export MKL_INTERFACE_LAYER=ILP64 - start = time.time() - Z = dot_product_mkl(YT_csr, X_csr, reorder_output=True) - run_time += time.time() - start - Z_data = Z.data - elif args.spmm_algo == "scipy": - # scipy will not sorted the indices for each row, - # so we do it explicitly - start = time.time() - Z = YT_csr.dot(X_csr) - Z.sort_indices() - run_time += time.time() - start - Z_data = Z.data - elif args.spmm_algo == "pytorch": - import torch - def get_pt_data(A_csr): - A_indices, A_values = csr_to_coo(A_csr) - A_pt = torch.sparse_coo_tensor( - A_indices.T.astype(np.int64), - A_values.astype(np.float32), - A_csr.shape, - ) - return A_pt - YT_pt = get_pt_data(YT_csr) - X_pt = get_pt_data(X_csr) - start = time.time() - Z_pt = torch.sparse.mm(YT_pt, X_pt) - run_time += time.time() - start - Z_data = Z_pt.coalesce().values().numpy() - elif args.spmm_algo == "tensorflow": - import tensorflow.compat.v1 as tf - from tensorflow.python.ops.linalg.sparse import sparse_csr_matrix_ops - def get_tf_data(A_csr): - # Define (COO format) Sparse Tensors over Numpy arrays - A_indices, A_values = csr_to_coo(A_csr) - A_st = tf.sparse.SparseTensor( - A_indices.astype(np.int64), - A_values.astype(np.float32), - A_csr.shape, - ) - return A_st - # Tensorflow (v2.5.0) usage, as of 07/20/2021: - # https://www.tensorflow.org/api_docs/python/tf/raw_ops/SparseMatrixSparseMatMul - with tf.Session() as sess: - YT_st = get_tf_data(YT_csr) - X_st = get_tf_data(X_csr) - sess.run(YT_st) - sess.run(X_st) - YT_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(YT_st.indices, YT_st.values, YT_st.dense_shape) - X_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(X_st.indices, X_st.values, X_st.dense_shape) - start = time.time() - Z_sm = sparse_csr_matrix_ops.sparse_matrix_sparse_mat_mul(a=YT_sm, b=X_sm, type=tf.float32) - Z_st = sparse_csr_matrix_ops.csr_sparse_matrix_to_sparse_tensor(Z_sm, tf.float32) - Z_data = sess.run(Z_st.values) - run_time += time.time() - start - else: - raise ValueError(f"spmm_algo={args.spmm_algo} is not valid") - run_time = time.time() - start - print("algo {:16s} time(s) {:9.5f} nnz(Z) {:12d} mu(Z.data) {:8.4f}".format( - args.spmm_algo, - run_time, - len(Z_data), - np.mean(Z_data), - ) - ) - - -def parse_arguments(): - """Parse arguments""" - - parser = argparse.ArgumentParser() - - # Required parameters - parser.add_argument( - "-x", "--x-npz-path", - type=str, required=True, - help="path to the CSR npz of the sparse instance-to-feature matrix", - ) - parser.add_argument( - "-y", "--y-npz-path", - type=str, required=True, - help="path to the CSR npz of the sparse instance-to-label matrix", - ) - # Optional - parser.add_argument( - "-algo", "--spmm-algo", - choices=SPMM_ALGO_LIST, - type=str, default="pecos", - help=f"SpMM algorithm (default pecos). Available choices are {', '.join(SPMM_ALGO_LIST)}", - ) - parser.add_argument( - "-t", "--threads", - type=int, default=-1, - help=f"number of threads for spmm_algo=pecos (default -1 to use all)", - ) - - return parser - - -if __name__ == "__main__": - parser = parse_arguments() - args = parser.parse_args() - do_spmm_exp(args) diff --git a/examples/spmm/run_exp.sh b/examples/spmm/run_exp.sh deleted file mode 100644 index 2a7751a8..00000000 --- a/examples/spmm/run_exp.sh +++ /dev/null @@ -1,53 +0,0 @@ - -dataset=$1 -exp_mode=$2 -if [ -z "${dataset}" ] || [ -z "${exp_mode}" ]; then - echo "bash run_exp.sh [dataset] [exp_mode]" - exit -fi - -x_npz_path=./data/${dataset}/tfidf-attnxml/X.trn.npz -y_npz_path=./data/${dataset}/Y.trn.npz -seed_arr=( 1 2 3 4 5 ) - -if [[ "${exp_mode}" == "single-thread" ]]; then - n_thread=1 - algo_arr=( scipy pecos intel-mkl pytorch tensorflow ) - for algo in "${algo_arr[@]}"; do - log_dir=./results/${dataset}/${algo} - mkdir -p ${log_dir} - for seed in "${seed_arr[@]}"; do - export MKL_INTERFACE_LAYER=ILP64 - export OMP_NUM_THREADS=${n_thread} - export MKL_NUM_THREADS=${n_thread} - python3 -u run_exp.py \ - --x-npz-path ${x_npz_path} \ - --y-npz-path ${y_npz_path} \ - --spmm-algo ${algo} \ - --threads ${n_thread} \ - |& tee ${log_dir}/t-${n_thread}.s-${seed}.log - done - done -elif [[ "${exp_mode}" == "multi-thread" ]]; then - thread_arr=( 2 4 8 16 32 ) - algo_arr=( pecos intel-mkl ) - for algo in "${algo_arr[@]}"; do - log_dir=./results/${dataset}/${algo} - mkdir -p ${log_dir} - for n_thread in "${thread_arr[@]}"; do - for seed in "${seed_arr[@]}"; do - export MKL_INTERFACE_LAYER=ILP64 - export OMP_NUM_THREADS=${n_thread} - export MKL_NUM_THREADS=${n_thread} - python3 -u run_exp.py \ - --x-npz-path ${x_npz_path} \ - --y-npz-path ${y_npz_path} \ - --spmm-algo ${algo} \ - --threads ${n_thread} \ - |& tee ${log_dir}/t-${n_thread}.s-${seed}.log - done - done - done -else - echo "exp_mode=${exp_mode} is not support! Consider {single-thread, multi-thread}" -fi diff --git a/examples/xr-transformer-neurips21/README.md b/examples/xr-transformer-neurips21/README.md deleted file mode 100644 index 8e4a85c9..00000000 --- a/examples/xr-transformer-neurips21/README.md +++ /dev/null @@ -1,72 +0,0 @@ -# Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification, NeurIPS 2021 - -This folder contains code to train XR-Transformer models and reproduce experiments -in ["Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification"](https://arxiv.org/abs/2110.00685). - -## Getting Started -* Clone the repository and enter `examples/xr-transformer-neurips21` directory. -* First create a [virtual environment](https://docs.python.org/3/library/venv.html) and then install dependencies -by running the following command: -```bash -pip install -r requirements.txt -``` -If you're unfamiliar with Python virtual environments, check out the -[user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). - -## Downloading Data -The XMC datasets can be download at -``` bash -# eurlex-4k, wiki10-31k, amazoncat-13k, amazon-670k, wiki-500k, amazon-3m -DATASET="wiki10-31k" -wget https://archive.org/download/pecos-dataset/xmc-base/${DATASET}.tar.gz -tar -zxvf ./${DATASET}.tar.gz -``` - -## Training and Evaluation -To train and evaluate XR-Transformer model, run -``` bash -bash run.sh ${DATASET} -``` -Recommended platform for training: [AWS p3.16xlarge instance](https://aws.amazon.com/ec2/instance-types/p3/) or equivalent. - -## Getting XR-Transformer Embeddings -We also release the fine-tuned XR-Transformer encoders on which users can -generate instance embeddings. -The XR-Transformer encoders can be download at -``` bash -# eurlex-4k, wiki10-31k, amazoncat-13k, amazon-670k, wiki-500k, amazon-3m -DATASET="wiki10-31k" -wget https://archive.org/download/xr-transformer-encoders/${DATASET}.tar.gz -mkdir -p ./encoders -tar -zxvf ./${DATASET}.tar.gz -C ./encoders -``` - -The XR-Transformer embeddings of training and testing instances can be generated by: -```bash -# for eurlex-4k, wiki10-31k, amazoncat-13k, MODEL_NAME can be bert|roberta|xlnet -# for amazon-670k, wiki-500k, amazon-3m, MODEL_NAME can be bert1|bert2|bert3 -MODEL_NAME="bert" -model_dir="./encoders/${DATASET}/${MODEL_NAME}" - -python3 -m pecos.xmc.xtransformer.encode \ - --text-path xmc-base/${DATASET}/X.trn.txt \ - --model-folder ${model_dir} \ - --batch-gen-workers 16 \ - --save-emb-path ${model_dir}/X.emb.trn.npy \ - --batch-size 128 - -python3 -m pecos.xmc.xtransformer.encode \ - --text-path xmc-base/${DATASET}/X.tst.txt \ - --model-folder ${model_dir} \ - --batch-gen-workers 16 \ - --save-emb-path ${model_dir}/X.emb.tst.npy \ - --batch-size 128 -``` -Embeddings will be saved at `${model_dir}/X.emb.trn.npy` and `${model_dir}/X.emb.tst.npy`. - -## Citation - -If you find this useful, please consider citing our paper. - -* [Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification (Zhang et al., NeurIPS 2021)](https://arxiv.org/pdf/2110.00685.pdf) [[bib]](./bibtex/zhang2021fast.bib) - diff --git a/examples/xr-transformer-neurips21/ensemble_evaluate.py b/examples/xr-transformer-neurips21/ensemble_evaluate.py deleted file mode 100644 index 4c00a211..00000000 --- a/examples/xr-transformer-neurips21/ensemble_evaluate.py +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env python3 -u - -import argparse - -from pecos.utils.smat_util import sorted_csr, CsrEnsembler, load_matrix - -def parse_arguments(): - parser = argparse.ArgumentParser() - - parser.add_argument( - "-y", - "--truth-path", - type=str, - required=True, - metavar="PATH", - help="path to the file of with ground truth output (CSR: nr_insts * nr_items)", - ) - parser.add_argument( - "-p", - "--pred-path", - type=str, - required=True, - nargs="*", - metavar="PATH", - help="path to the file of predicted output (CSR: nr_insts * nr_items)", - ) - parser.add_argument( - "--tags", - type=str, - required=True, - nargs="*", - metavar="PATH", - help="tags attached to each prediction", - ) - parser.add_argument( - "--ens-method", - type=str, - metavar="STR", - default="rank_average", - help="prediction ensemble method", - ) - - return parser - - -def do_evaluation(args): - """ Evaluate xlinear predictions """ - assert len(args.tags) == len(args.pred_path) - Y_true = sorted_csr(load_matrix(args.truth_path).tocsr()) - Y_pred = [sorted_csr(load_matrix(pp).tocsr()) for pp in args.pred_path] - print("==== evaluation results ====") - CsrEnsembler.print_ens(Y_true, Y_pred, args.tags, ens_method=args.ens_method) - - -if __name__ == "__main__": - parser = parse_arguments() - args = parser.parse_args() - do_evaluation(args) diff --git a/examples/xr-transformer-neurips21/params/amazon-3m/bert1/params.json b/examples/xr-transformer-neurips21/params/amazon-3m/bert1/params.json deleted file mode 100644 index 3afc4d75..00000000 --- a/examples/xr-transformer-neurips21/params/amazon-3m/bert1/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/amazon-3m/bert2/params.json b/examples/xr-transformer-neurips21/params/amazon-3m/bert2/params.json deleted file mode 100644 index 6c2e695f..00000000 --- a/examples/xr-transformer-neurips21/params/amazon-3m/bert2/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/amazon-3m/bert3/params.json b/examples/xr-transformer-neurips21/params/amazon-3m/bert3/params.json deleted file mode 100644 index 7478367d..00000000 --- a/examples/xr-transformer-neurips21/params/amazon-3m/bert3/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 100, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 8.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/amazon-670k/bert1/params.json b/examples/xr-transformer-neurips21/params/amazon-670k/bert1/params.json deleted file mode 100644 index e81ca9f6..00000000 --- a/examples/xr-transformer-neurips21/params/amazon-670k/bert1/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 12000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "disable", - "rel_norm": "no-norm", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/amazon-670k/bert2/params.json b/examples/xr-transformer-neurips21/params/amazon-670k/bert2/params.json deleted file mode 100644 index 8a1fb9f5..00000000 --- a/examples/xr-transformer-neurips21/params/amazon-670k/bert2/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 12000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "disable", - "rel_norm": "no-norm", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/amazon-670k/bert3/params.json b/examples/xr-transformer-neurips21/params/amazon-670k/bert3/params.json deleted file mode 100644 index 391de00b..00000000 --- a/examples/xr-transformer-neurips21/params/amazon-670k/bert3/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 32, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 4000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 12000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 200, - "logging_steps": 50, - "save_steps": 1000, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "disable", - "rel_norm": "no-norm", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/amazoncat-13k/bert/params.json b/examples/xr-transformer-neurips21/params/amazoncat-13k/bert/params.json deleted file mode 100644 index 7654a013..00000000 --- a/examples/xr-transformer-neurips21/params/amazoncat-13k/bert/params.json +++ /dev/null @@ -1,277 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 128, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 128, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.0001, - "hidden_dropout_prob": 0.05, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 10000, - "max_num_labels_in_gpu": 65536, - "max_steps": 5000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.0001, - "hidden_dropout_prob": 0.05, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 10000, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.0001, - "hidden_dropout_prob": 0.05, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 10000, - "max_num_labels_in_gpu": 65536, - "max_steps": 30000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 10000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.0001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 0.25, - "Cn": 0.5, - "max_iter": 100, - "eps": 0.01, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.0001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 0.25, - "Cn": 0.5, - "max_iter": 100, - "eps": 0.01, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.0001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 0.25, - "Cn": 0.5, - "max_iter": 100, - "eps": 0.01, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 200, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/amazoncat-13k/roberta/params.json b/examples/xr-transformer-neurips21/params/amazoncat-13k/roberta/params.json deleted file mode 100644 index 3f487f8d..00000000 --- a/examples/xr-transformer-neurips21/params/amazoncat-13k/roberta/params.json +++ /dev/null @@ -1,277 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 128, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 1000, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 128, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 1000, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "roberta-base", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.0001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 10000, - "max_num_labels_in_gpu": 65536, - "max_steps": 5000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "roberta-base", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.0001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 10000, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "roberta-base", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.0001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 10000, - "max_num_labels_in_gpu": 65536, - "max_steps": 30000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 10000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.0001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 0.5, - "Cn": 0.5, - "max_iter": 100, - "eps": 0.01, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.0001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 0.5, - "Cn": 0.5, - "max_iter": 100, - "eps": 0.01, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.0001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 0.5, - "Cn": 0.5, - "max_iter": 100, - "eps": 0.01, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 200, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/amazoncat-13k/xlnet/params.json b/examples/xr-transformer-neurips21/params/amazoncat-13k/xlnet/params.json deleted file mode 100644 index bb2aaf8c..00000000 --- a/examples/xr-transformer-neurips21/params/amazoncat-13k/xlnet/params.json +++ /dev/null @@ -1,277 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 128, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 100000, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 128, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 100000, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "xlnet-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.0001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 10000, - "max_num_labels_in_gpu": 65536, - "max_steps": 5000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "xlnet-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.0001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 10000, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "xlnet-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.0001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 10000, - "max_num_labels_in_gpu": 65536, - "max_steps": 30000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 1000, - "logging_steps": 500, - "save_steps": 10000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.0001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 0.5, - "Cn": 0.5, - "max_iter": 100, - "eps": 0.01, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.0001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 0.5, - "Cn": 0.5, - "max_iter": 100, - "eps": 0.01, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.0001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 0.5, - "Cn": 0.5, - "max_iter": 100, - "eps": 0.01, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 200, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/eurlex-4k/bert/params.json b/examples/xr-transformer-neurips21/params/eurlex-4k/bert/params.json deleted file mode 100644 index 37129658..00000000 --- a/examples/xr-transformer-neurips21/params/eurlex-4k/bert/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 16, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 0, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": null, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 0, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 600, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 800, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 1200, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 2.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 2.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 2.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 2.0, - "Cn": 4.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 25, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/eurlex-4k/roberta/params.json b/examples/xr-transformer-neurips21/params/eurlex-4k/roberta/params.json deleted file mode 100644 index 8e268f5c..00000000 --- a/examples/xr-transformer-neurips21/params/eurlex-4k/roberta/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 16, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 1, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": null, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 1, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "roberta-base", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 600, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "roberta-base", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 800, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "roberta-base", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 1200, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 2.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 2.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 2.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 2.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 25, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/eurlex-4k/xlnet/params.json b/examples/xr-transformer-neurips21/params/eurlex-4k/xlnet/params.json deleted file mode 100644 index 162532b3..00000000 --- a/examples/xr-transformer-neurips21/params/eurlex-4k/xlnet/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 16, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 2, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": null, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 2, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "xlnet-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 600, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "xlnet-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 800, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "xlnet-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 1200, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": false, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 2.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 2.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 1.0, - "Cn": 2.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 2.0, - "Cn": 4.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 75, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 25, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/wiki-500k/bert1/params.json b/examples/xr-transformer-neurips21/params/wiki-500k/bert1/params.json deleted file mode 100644 index e6760dca..00000000 --- a/examples/xr-transformer-neurips21/params/wiki-500k/bert1/params.json +++ /dev/null @@ -1,342 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10011, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/wiki-500k/bert2/params.json b/examples/xr-transformer-neurips21/params/wiki-500k/bert2/params.json deleted file mode 100644 index c86d7154..00000000 --- a/examples/xr-transformer-neurips21/params/wiki-500k/bert2/params.json +++ /dev/null @@ -1,342 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10012, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/wiki-500k/bert3/params.json b/examples/xr-transformer-neurips21/params/wiki-500k/bert3/params.json deleted file mode 100644 index c660bf33..00000000 --- a/examples/xr-transformer-neurips21/params/wiki-500k/bert3/params.json +++ /dev/null @@ -1,342 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 8, - "min_codes": 64, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 32, - "min_codes": null, - "max_leaf_size": 100, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10013, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 10000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 500, - "max_num_labels_in_gpu": 65536, - "max_steps": 20000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 0.0001, - "adam_epsilon": 1e-08, - "warmup_steps": 5000, - "logging_steps": 500, - "save_steps": 5000, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 5, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 128 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 50, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/wiki10-31k/bert/params.json b/examples/xr-transformer-neurips21/params/wiki10-31k/bert/params.json deleted file mode 100644 index 7a80f98d..00000000 --- a/examples/xr-transformer-neurips21/params/wiki10-31k/bert/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10001, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": null, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10001, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 1000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 1000, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "bert-base-uncased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 400, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/wiki10-31k/roberta/params.json b/examples/xr-transformer-neurips21/params/wiki10-31k/roberta/params.json deleted file mode 100644 index 34df4d7e..00000000 --- a/examples/xr-transformer-neurips21/params/wiki10-31k/roberta/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10002, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": null, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10002, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "roberta-base", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 600, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "roberta-base", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 400, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "roberta-base", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 200, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/params/wiki10-31k/xlnet/params.json b/examples/xr-transformer-neurips21/params/wiki10-31k/xlnet/params.json deleted file mode 100644 index 2c88bfee..00000000 --- a/examples/xr-transformer-neurips21/params/wiki10-31k/xlnet/params.json +++ /dev/null @@ -1,301 +0,0 @@ -{ - "train_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.TrainParams" - }, - "preliminary_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": 128, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10003, - "kmeans_max_iter": 20, - "threads": -1 - }, - "refined_indexer_params": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalKMeans.TrainParams" - }, - "nr_splits": 16, - "min_codes": null, - "max_leaf_size": 16, - "imbalanced_ratio": 0.0, - "imbalanced_depth": 100, - "spherical": true, - "seed": 10003, - "kmeans_max_iter": 20, - "threads": -1 - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "xlnet-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 400, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "xlnet-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 400, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams" - }, - "model_shortcut": "xlnet-base-cased", - "negative_sampling": "tfn+man", - "loss_function": "weighted-squared-hinge", - "bootstrap_method": "weighted-linear", - "lr_schedule": "linear", - "threshold": 0.001, - "hidden_dropout_prob": 0.1, - "batch_size": 32, - "batch_gen_workers": 16, - "max_active_matching_labels": 1000, - "max_num_labels_in_gpu": 65536, - "max_steps": 200, - "max_no_improve_cnt": -1, - "num_train_epochs": 10, - "gradient_accumulation_steps": 1, - "weight_decay": 0.0, - "max_grad_norm": 1.0, - "learning_rate": 5e-05, - "adam_epsilon": 1e-08, - "warmup_steps": 100, - "logging_steps": 50, - "save_steps": 200, - "cost_sensitive_ranker": true, - "use_gpu": true, - "checkpoint_dir": "", - "cache_dir": "", - "init_model_dir": "" - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.TrainParams" - }, - "mode": "full-model", - "ranker_level": 1, - "nr_splits": 16, - "min_codes": null, - "shallow": false, - "rel_mode": "induce", - "rel_norm": "l1", - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.TrainParams" - }, - "neg_mining_chain": [ - "tfn", - "tfn", - "tfn", - "tfn+man" - ], - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.TrainParams" - }, - "threshold": 0.001, - "max_nonzeros_per_label": null, - "solver_type": "L2R_L2LOSS_SVC_DUAL", - "Cp": 4.0, - "Cn": 1.0, - "max_iter": 100, - "eps": 0.1, - "bias": 1.0, - "threads": -1, - "verbose": 0, - "newton_eps": 0.01 - } - ] - } - }, - "do_fine_tune": true, - "only_encoder": false, - "fix_clustering": false, - "max_match_clusters": 32768, - "save_emb_dir": "" - }, - "pred_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.model###XTransformer.PredParams" - }, - "matcher_params_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.xtransformer.matcher###TransformerMatcher.PredParams" - }, - "only_topk": 20, - "post_processor": "l3-hinge", - "ensemble_method": "concat-only", - "truncate_length": 256 - } - ], - "ranker_params": { - "__meta__": { - "class_fullname": "pecos.xmc.xlinear.model###XLinearModel.PredParams" - }, - "hlm_args": { - "__meta__": { - "class_fullname": "pecos.xmc.base###HierarchicalMLModel.PredParams" - }, - "model_chain": [ - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "l3-hinge" - }, - { - "__meta__": { - "class_fullname": "pecos.xmc.base###MLModel.PredParams" - }, - "only_topk": 100, - "post_processor": "noop" - } - ] - } - } - } -} diff --git a/examples/xr-transformer-neurips21/requirements.txt b/examples/xr-transformer-neurips21/requirements.txt deleted file mode 100644 index 11849695..00000000 --- a/examples/xr-transformer-neurips21/requirements.txt +++ /dev/null @@ -1,30 +0,0 @@ -certifi==2023.7.22 -charset-normalizer==2.0.6 -click==8.0.1 -docopt==0.6.2 -filelock==3.1.0 -huggingface-hub==0.0.17 -idna==3.2 -importlib-metadata==4.8.1 -joblib==1.2.0 -numpy==1.22.0 -packaging==21.0 -pyparsing==2.4.7 -PyYAML==5.4.1 -regex==2021.9.24 -requests==2.32.0 -sacremoses==0.0.46 -scikit-learn==1.0 -scipy==1.10.0 -sentencepiece==0.1.96 -six==1.16.0 -threadpoolctl==2.2.0 -tokenizers==0.10.3 -torch==1.9.1 -tqdm==4.66.3 -transformers==4.11.0 -typing-extensions==3.10.0.2 -urllib3==1.26.7 -yarg==0.1.9 -zipp==3.5.1 -libpecos==0.2.2 diff --git a/examples/xr-transformer-neurips21/run.sh b/examples/xr-transformer-neurips21/run.sh deleted file mode 100644 index 92c3deb7..00000000 --- a/examples/xr-transformer-neurips21/run.sh +++ /dev/null @@ -1,43 +0,0 @@ -data=$1 -data_dir="./xmc-base/${data}/" - -if [ ${data} == "eurlex-4k" ]; then - models=(bert roberta xlnet) - ens_method=softmax_average -elif [ ${data} == "wiki10-31k" ]; then - models=(bert) - ens_method=rank_average -elif [ ${data} == "amazoncat-13k" ]; then - models=(bert roberta xlnet) - ens_method=softmax_average -elif [ ${data} == "wiki-500k" ]; then - models=(bert1 bert2 bert3) - ens_method=sigmoid_average -elif [ ${data} == "amazon-670k" ]; then - models=(bert1 bert2 bert3) - ens_method=softmax_average -elif [ ${data} == "amazon-3m" ]; then - models=(bert1 bert2 bert3) - ens_method=rank_average -else - echo Unknown dataset $1! - exit -fi - -Preds="" -Tags="" - -for mm in "${models[@]}"; do - bash train_and_predict.sh ${data} ${mm} ${data_dir} - Preds="${Preds} models/${data}/${mm}/Pt.npz" - Tags="${Tags} ${mm}" -done - -Y_tst=${data_dir}/Y.tst.npz # test label matrix - -python ensemble_evaluate.py \ - -y ${Y_tst} \ - -p ${Preds} \ - --tags ${Tags} \ - --ens-method ${ens_method} \ - |& tee models/${data}/ensemble.log diff --git a/examples/xr-transformer-neurips21/train_and_predict.sh b/examples/xr-transformer-neurips21/train_and_predict.sh deleted file mode 100644 index 3a595abd..00000000 --- a/examples/xr-transformer-neurips21/train_and_predict.sh +++ /dev/null @@ -1,40 +0,0 @@ -#================= inputs ===================== -data_name=$1 -model_name=$2 -data_dir=$3 - -X_trn=${data_dir}/X.trn.txt # training text -X_tst=${data_dir}/X.tst.txt # test text - -Y_trn=${data_dir}/Y.trn.npz # training label matrix -Y_tst=${data_dir}/Y.tst.npz # test label matrix -X_feat_trn=${data_dir}/tfidf-attnxml/X.trn.npz # training tfidf feature -X_feat_tst=${data_dir}/tfidf-attnxml/X.tst.npz # test tfidf feature - -model_dir=models/${data_name}/${model_name} -mkdir -p ${model_dir} - -params_dir=params/${data_name}/${model_name} - -python3 -m pecos.xmc.xtransformer.train \ - --trn-text-path ${X_trn} \ - --trn-feat-path ${X_feat_trn} \ - --trn-label-path ${Y_trn} \ - --model-dir ${model_dir} \ - --params-path ${params_dir}/params.json \ - |& tee ${model_dir}/train.log - -python3 -m pecos.xmc.xtransformer.predict \ - --feat-path ${X_feat_tst} \ - --text-path ${X_tst} \ - --model-folder ${model_dir} \ - --batch-gen-workers 16 \ - --save-pred-path ${model_dir}/Pt.npz \ - --batch-size 128 \ - |& tee ${model_dir}/predict.log - -python3 -m pecos.xmc.xlinear.evaluate \ - -y ${Y_tst} \ - -p ${model_dir}/Pt.npz \ - --topk 10 \ - |& tee ${model_dir}/result.log diff --git a/examples/xrlinear-mscm-www22/README.md b/examples/xrlinear-mscm-www22/README.md deleted file mode 100644 index feed453f..00000000 --- a/examples/xrlinear-mscm-www22/README.md +++ /dev/null @@ -1,14 +0,0 @@ -# Enterprise-Scale Search: Accelerating Inference for Sparse Extreme Multi-Label Ranking Trees, WWW 2022 - -This folder contains code to reproduce experiments in -["Enterprise-Scale Search: Accelerating Inference for Sparse Extreme Multi-Label Ranking Trees"](https://arxiv.org/pdf/2106.02697.pdf) - -## Experiment Code -Please visit this [repo](https://github.com/UniqueUpToPermutation/pecos/tree/benchmark) -for detailed instructions to reproduct the experiment results. - -## Citation -If you find this useful, please consider citing our paper. -* ["Enterprise-Scale Search: Accelerating Inference for Sparse Extreme Multi-Label Ranking Trees"](https://arxiv.org/pdf/2106.02697.pdf)[[bib]](./bibtex/etter2021accelerating.bib) - - diff --git a/setup.py b/setup.py index b79b7b2f..ff5a88bd 100644 --- a/setup.py +++ b/setup.py @@ -123,10 +123,10 @@ def get_version(cls): ) setuptools.setup( - name="libpecos", + name="pecos4annif", version=VersionHelper.get_version(), description="PECOS - Predictions for Enormous and Correlated Output Spaces", - url="https://github.com/amzn/pecos", + url="https://github.com/NatLibFi/pecos", author="Amazon.com, Inc.", license="Apache 2.0", packages=setuptools.find_packages(where="."), diff --git a/tutorials/kdd22/README.md b/tutorials/kdd22/README.md deleted file mode 100644 index 221dab2c..00000000 --- a/tutorials/kdd22/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# KDD 2022 Hands-on Tutorial - PECOS: Prediction for Enormous and Correlated Output Spaces - -In this tutorial, we will introduce several key functions and features of the PECOS library. -By way of real-world examples, the attendees will learn how to efficiently train large-scale machine learning models for enormous output spaces, and obtain predictions in less than 1 millisecond for a data input with million labels, in the context of product recommendation and natural language processing. -We will also show the flexibility of dealing with diverse machine learning problems and data formats with assorted built-in utilities in PECOS. -By the end of the tutorial, we believe that attendees will be easily capable of adopting certain concepts to their own projects and address different machine learning problems with enormous output spaces. - -* Presenters: Hsiang-Fu Yu (Amazon Search), Jiong Zhang (Amazon Search), Wei-Cheng Chang (Amazon Search), Jyun-Yu Jiang (Amazon Search), and Cho-Jui Hsieh (UCLA) - -* Contributer: Wei Li (Amazon Search) - -## Agenda - -| Time | Session | Presenter | Material | -|---|---|---|---| -| 9:00 AM - 9:20 AM | Session 1: Introduction to PECOS | Dr. Hsiang-Fu Yu (Amazon) | [Slides](https://www.cs.utexas.edu/~rofuyu/talks/pecos-tutorial-kdd22-opening.pdf) | -| 9:20 AM - 10:00 AM | Environment Setup | | | -| 9:30 AM - 10:00 AM | Coffee Break | | | -| 10:00 AM - 10:30 AM | Session 2: How to Apply PECOS to Extreme Multi-label Classification | Dr. Jyun-Yu Jiang (Amazon) | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%202%20Extreme%20Multi-label%20Classification%20with%20PECOS.ipynb) | -| 10:30 AM - 11:00 AM | Session 3: How to Perform Approximate Nearest Neighbor Search in PECOS | Dr. Wei-Cheng Chang (Amazon) | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%203%20Approximate%20Nearest%20Neighbor%20Search%20in%20PECOS.ipynb) | -| 11:00 AM - 11:20 AM | Session 4: Useful Utilities in PECOS | Dr. Jyun-Yu Jiang (Amazon) | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%204%20Utilities%20in%20PECOS.ipynb) | -| 11:20 AM - 11:40 AM | Session 5: How to Leverage Transformers in PECOS | Dr. Jiong Zhang (Amazon) | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%205%20eXtreme%20Multi-label%20Classification%20with%20XR-Transformer.ipynb) | -| 11:40 AM - 12:00 PM | Session 6: Research with PECOS | Prof. Cho-Jui Hsieh (UCLA & Amazon) | [Slides](http://web.cs.ucla.edu/~chohsieh/pecos_research.pdf) | - - -## Tutorial Instructions - -### Miniconda Installation -```bash -mkdir -p ~/miniconda3 -wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh -bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 -rm -rf ~/miniconda3/miniconda.sh -~/miniconda3/bin/conda init bash -~/miniconda3/bin/conda init zsh -``` - -### Tutorial Material Execution -```bash -conda create -n tutorial_env python=3.9 -y -conda activate tutorial_env -python -m pip install libpecos==0.4.0 matplotlib panda requests jupyterlab -mkdir -p ~/pecos_tutorial_playground -cd ~/pecos_tutorial_playground -git clone https://github.com/amzn/pecos -python -m jupyterlab.labapp --ip=0.0.0.0 --port 8888 --no-browser --allow-root --notebook-dir=pecos/tutorials/kdd22 -``` diff --git a/tutorials/kdd22/Session 2 Extreme Multi-label Classification with PECOS.ipynb b/tutorials/kdd22/Session 2 Extreme Multi-label Classification with PECOS.ipynb deleted file mode 100644 index 4ea4e8f5..00000000 --- a/tutorials/kdd22/Session 2 Extreme Multi-label Classification with PECOS.ipynb +++ /dev/null @@ -1,1414 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "67e70878", - "metadata": {}, - "source": [ - "# How to Apply PECOS to Extreme Multi-label Classification\n", - "\n", - "Prediction for Enormous and Correlated Output Spaces (PECOS) is a versatile and modular machine learning framework for solving prediction problems with very large outputs spaces. For a given input instance, we apply PECOS to the eXtreme Multilabel Classification (XMC) problem to find and rank the most relevant items from an enormous but fixed and finite output space. Generally, PECOS trains an XMC model that takes numerical features to rank labels from the enormous output space. PECOS also provides feature extraction functions for text data, such as TF-IDF (this session) and Transformers (Session 5).\n", - "\n", - "

\n", - "\n", - "
\n", - "\n", - "Using PECOS, we can tackle lots of real-world large-scale applications with only few commands or limited programming codes.\n", - "\n", - "

\n", - "\n", - "
\n", - "\n", - "\n", - "In this part of the tutorial, we will use XR-Linear as an example to demonstrate how to use PECOS to tackle real-world problems and understrand the model architecture in PECOS." - ] - }, - { - "cell_type": "markdown", - "id": "11c281dc", - "metadata": {}, - "source": [ - "## Outline in this Session\n", - "\n", - "1. Experimental dataset preparation\n", - "2. Hands-on PECOS in only few commands \n", - "3. Code with the PECOS library\n", - "4. Build your customized PECOS XR-Linear model" - ] - }, - { - "cell_type": "markdown", - "id": "41d87d24", - "metadata": {}, - "source": [ - "## 1. Experimental Dataset Preparation\n", - "\n", - "`eurlex-4k`, `wiki10-31k`, `amazoncat-13k`, `amazon-670k`, `wiki-500k`, and `amazon-3m` are available." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "1073ac9c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "xmc-base/wiki10-31k/output-items.txt\r\n", - "xmc-base/wiki10-31k/tfidf-attnxml\r\n", - "xmc-base/wiki10-31k/tfidf-attnxml/X.trn.npz\r\n", - "xmc-base/wiki10-31k/tfidf-attnxml/X.tst.npz\r\n", - "xmc-base/wiki10-31k/X.trn.txt\r\n", - "xmc-base/wiki10-31k/X.tst.txt\r\n", - "xmc-base/wiki10-31k/Y.trn.npz\r\n", - "xmc-base/wiki10-31k/Y.trn.txt\r\n", - "xmc-base/wiki10-31k/Y.tst.npz\r\n", - "xmc-base/wiki10-31k/Y.tst.txt\r\n" - ] - } - ], - "source": [ - "DATASET = \"wiki10-31k\"\n", - "! wget -nv -nc https://archive.org/download/pecos-dataset/xmc-base/{DATASET}.tar.gz\n", - "! tar --skip-old-files -zxf {DATASET}.tar.gz \n", - "! find xmc-base/{DATASET}/*" - ] - }, - { - "cell_type": "markdown", - "id": "057fb642", - "metadata": {}, - "source": [ - "### Numerical Feature and Label Format in PECOS\n", - "\n", - "In PECOS, numerical features of instances can be in either a [dense NumPy matrix](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html) or a [Compressed Sparse Row (CSR) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) of shape `(nr_inst, nr_feat)`, where `nr_inst` and `nr_feat` are numbers of instances and features. \n", - "\n", - "Similary, labels of instances can be also presented as a dense or a sparse matrix of shape `(nr_inst, nr_labels)`, where `nr_labels` is the number of labels in the XMC problem. The following figure shows an example of the sparse matrix for label representations of six instances.\n", - "\n", - "

\n", - "\n", - "
\n" - ] - }, - { - "cell_type": "markdown", - "id": "fcf8d41d", - "metadata": {}, - "source": [ - "## 2. Hands-on PECOS in Only Few Commands\n", - "\n", - "PECOS provides convenient command-line interfaces to establish a pipeline from feature extraction to training and inference. Specifically, a PECOS XR-Linear model for text data can be established and evaluated using the following command-line modules without writing any code.\n", - "\n", - "* Text Vectorizer: `pecos.utils.featurization.text.preprocess`\n", - "* XR-Linear Train/Predict/Evaluate: `pecos.xmc.xlinear.train`, `pecos.xmc.xlinear.predict`, `pecos.xmc.xlinear.evaluate`\n", - "\n", - "All of these commands can be supplied with JSON-format configuration files (See Section 4.2.2 and Appendix 2). In this section, we first use the default setting to have a quick hands-on demo.\n", - "\n", - "\n", - "### 2.1. Text Vectorizer\n", - "\n", - "PECOS text vectorizer `pecos.utils.featurization.text.preprocess` extracts TF-IDF features. The options `build` and `run` learn the vectorizer and extract features, respectively. The `--help` argument will list all available command-line options. In the default setting, the vectorizer learns a unigram TF-IDF without any filtering. If you have prepared a JSON-format vectorizer configuration file, the argument `--vectorizer-config-path` can help customize the vectorizer (See Section 4.2.2)." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "6b47e6d2", - "metadata": {}, - "outputs": [], - "source": [ - "! python3 -m pecos.utils.featurization.text.preprocess build \\\n", - " --text-pos 0 \\\n", - " --input-text-path xmc-base/{DATASET}/X.trn.txt \\\n", - " --output-model-folder simplest.{DATASET}.vectorizer \\\n", - " --from-file true" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "870f6b0d", - "metadata": {}, - "outputs": [], - "source": [ - "! python3 -m pecos.utils.featurization.text.preprocess run \\\n", - " --text-pos 0 \\\n", - " --input-preprocessor-folder simplest.{DATASET}.vectorizer \\\n", - " --input-text-path xmc-base/{DATASET}/X.trn.txt \\\n", - " --output-inst-path simplest.{DATASET}.X.trn.npz \\\n", - " --from-file true\n", - "! python3 -m pecos.utils.featurization.text.preprocess run \\\n", - " --text-pos 0 \\\n", - " --input-preprocessor-folder simplest.{DATASET}.vectorizer \\\n", - " --input-text-path xmc-base/{DATASET}/X.tst.txt \\\n", - " --output-inst-path simplest.{DATASET}.X.tst.npz \\\n", - " --from-file true" - ] - }, - { - "cell_type": "markdown", - "id": "a34bb4e2", - "metadata": {}, - "source": [ - "### 2.2. Train, Predict, Evaluate a PECOS Model\n", - "\n", - "With feature and label matrices, the pipeline of training, prediction, and evalution can be easily established with the modules `pecos.xmc.xlinear.train`, `pecos.xmc.xlinear.predict`, and `pecos.xmc.xlinear.evaluate`. The `--help` argument will list all available command-line options. If you have prepared a JSON-format configuration file, the argument `--params-path` can help customize training and prediction procedures (See Appendix 2)." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "b2316bff", - "metadata": {}, - "outputs": [], - "source": [ - "! python3 -m pecos.xmc.xlinear.train \\\n", - " -x simplest.{DATASET}.X.trn.npz \\\n", - " -y xmc-base/{DATASET}/Y.trn.npz \\\n", - " -m simplest.{DATASET}.model \\\n", - " --nr-splits 16 \\\n", - " -t 0.1" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "580ecfc7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "==== evaluation results ====\r\n", - "prec = 83.89 78.41 72.53 67.68 63.45 59.72 56.48 53.53 50.86 48.39\r\n", - "recall = 4.95 9.21 12.64 15.60 18.11 20.34 22.30 24.07 25.63 27.00\r\n" - ] - } - ], - "source": [ - "! python3 -m pecos.xmc.xlinear.predict \\\n", - " -x simplest.{DATASET}.X.tst.npz \\\n", - " -y xmc-base/{DATASET}/Y.tst.npz \\\n", - " -m simplest.{DATASET}.model \\\n", - " -o simplest.{DATASET}.Y.tst.pred.npz \\\n", - " -b 10 \\\n", - " -k 10" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "5e75b408", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "==== evaluation results ====\r\n", - "prec = 83.89 78.41 72.53 67.68 63.45 59.72 56.48 53.53 50.86 48.39\r\n", - "recall = 4.95 9.21 12.64 15.60 18.11 20.34 22.30 24.07 25.63 27.00\r\n" - ] - } - ], - "source": [ - "! python3 -m pecos.xmc.xlinear.evaluate \\\n", - " -y xmc-base/{DATASET}/Y.tst.npz \\\n", - " -p simplest.{DATASET}.Y.tst.pred.npz \\\n", - " -k 10" - ] - }, - { - "cell_type": "markdown", - "id": "b0c731f5", - "metadata": {}, - "source": [ - "## 3. Code with the PECOS library\n", - "\n", - "PECOS includes the comprehensieve Python library and interfaces so that we can easily utilize PECOS in the code-level with more flexibility.\n", - "\n", - "### 3.1. Loading Features and Labels\n", - "For convenience, PECOS also provides APIs `load_feature_matrix` and `load_label_matrix` for loading features and labels from binary files in arbitary formats.\n", - "Note that for the sparse format, training labels should be loaded as a [Compressed Sparse Column (CSC) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) while testing labels should be loaded as a CSR matrix for the purpose of computational efficiency. " - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "c518d892", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Training features X_trn is a csr matrix of shape (14146, 101938).\n", - "Training labels Y_trn is a csc matrix of shape (14146, 30938).\n", - "Testing features X_tst is a csr matrix of shape (6616, 101938).\n", - "Testing labels Y_tst is a csr matrix of shape (6616, 30938).\n" - ] - } - ], - "source": [ - "import numpy as np\n", - "from pecos.xmc.xlinear.model import XLinearModel\n", - "\n", - "DATASET = \"wiki10-31k\"\n", - "\n", - "X_trn = XLinearModel.load_feature_matrix(f\"xmc-base/{DATASET}/tfidf-attnxml/X.trn.npz\")\n", - "Y_trn = XLinearModel.load_label_matrix(f\"xmc-base/{DATASET}/Y.trn.npz\", for_training=True)\n", - "\n", - "X_tst = XLinearModel.load_feature_matrix(f\"xmc-base/{DATASET}/tfidf-attnxml/X.tst.npz\")\n", - "Y_tst = XLinearModel.load_label_matrix(f\"xmc-base/{DATASET}/Y.tst.npz\", for_training=False)\n", - "\n", - "print(f\"Training features X_trn is a {X_trn.getformat()} matrix of shape {X_trn.shape}.\")\n", - "print(f\"Training labels Y_trn is a {Y_trn.getformat()} matrix of shape {Y_trn.shape}.\")\n", - "print(f\"Testing features X_tst is a {X_tst.getformat()} matrix of shape {X_tst.shape}.\")\n", - "print(f\"Testing labels Y_tst is a {Y_tst.getformat()} matrix of shape {Y_tst.shape}.\")" - ] - }, - { - "cell_type": "markdown", - "id": "150fea14", - "metadata": {}, - "source": [ - "### 3.2. Semantic Label Indexing and Cluster Chain in XR-Linear\n", - "\n", - "The first step of training an XR-Linear model is to conduct semantic label indexing and establish the *hierarchial label tree* for resursive training the XR-Linear model and its inference. \n", - "\n", - "

\n", - "\n", - "
\n", - "\n", - "PECOS supports any method for semantic label indexing. In the PECOS library, as a build-in method, we provide Label Representation via Positive Instance Feature Aggregation (PIFA) for semantic label indexing with only the need of positive instances and their features in training data. PECOS can also consider additional label features `Z` of shape `(nr_labels, nr_label_feat)` in either dense or sparse matrix format, where `nr_label_feat` is the number of label features. These representations and features for each label are concatenated or combined as label embedding in `LabelEmbeddingFactory` in PECOS.\n", - "\n", - "To conduct semantic label indexing, PECOS learns an indexer based on label embedding. PECOS currently supports to use the **Hierarchical K-Means** for semantic label indexing with a hyper-parameter `nr_splits` (the number of clusters in each layer, or `B` in [our report](https://arxiv.org/pdf/2010.05878.pdf)), which decides the depth `D` of the hierarchical label tree. " - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "26794215", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "4 layers in the trained hierarchical label tree.\n" - ] - } - ], - "source": [ - "from pecos.xmc import Indexer, LabelEmbeddingFactory\n", - "\n", - "label_feat = LabelEmbeddingFactory.create(Y_trn, X_trn, method=\"pifa\")\n", - "# label_feat = LabelEmbeddingFactory.create(Y_trn, X_trn, Z, method=\"pifa_lf_concat\") # for using label features Z\n", - "\n", - "cluster_chain = Indexer.gen(label_feat, nr_splits=16, indexer_type=\"hierarchicalkmeans\")\n", - "\n", - "print(f\"{len(cluster_chain)} layers in the trained hierarchical label tree.\")" - ] - }, - { - "cell_type": "markdown", - "id": "02ffda21", - "metadata": {}, - "source": [ - "### 3.3. Training XR-Linear Negative Sampling and Sparsification\n", - "\n", - "Negative sampling plays an important role in solving the XMC problem. PECOS currently provides two negative sampling schemes, including Teacher Forcing Negatives (TFN) and Matcher Aware Negatives (MAN). Please refer to [our report](https://arxiv.org/pdf/2010.05878.pdf) for more details about negative sampling schemes.\n", - "\n", - "To reduce model sizes and improve efficiency, PECOS conduct model sparsification with a hyper-parameter `threshold`. The model weights with absolute values smaller than the threshold will be discarded." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "bd3d6527", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Training time: 39.7176 seconds.\n" - ] - } - ], - "source": [ - "import time\n", - "start_time = time.time()\n", - "\n", - "# For negative_sampling_scheme in model training, \"man\" and tfn+man\" are also available.\n", - "xlm = XLinearModel.train(X_trn, Y_trn, C=cluster_chain, threshold=0.1, negative_sampling_scheme=\"tfn\")\n", - "\n", - "training_time = time.time() - start_time\n", - "print(f\"Training time: {training_time:.4f} seconds.\")" - ] - }, - { - "cell_type": "markdown", - "id": "20f5cfa7", - "metadata": {}, - "source": [ - "PECOS supports serializing and loading the trained model into binary on disk with convenient interfaces. Note that model loading with `is_predict_only=True` could lead to faster prediction speed by disabling the flexibility of model modification." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "3d5d468d", - "metadata": {}, - "outputs": [], - "source": [ - "xlm.save(f\"{DATASET}.xlm.model\")\n", - "xlm = XLinearModel.load(f\"{DATASET}.xlm.model\", is_predict_only=False)" - ] - }, - { - "cell_type": "markdown", - "id": "4b6038ec", - "metadata": {}, - "source": [ - "### 3.4. Prediction and Evaluation\n", - "\n", - "As a tree model, the inference method significantly affects the prediction efficiency of XR-Linear in PECOS. As illustrated in the following figure, the prediction process in PECOS employs a beam search with a hyper-parameter `beam_size`. The other hyper-parameter `only_topk` also needs to be decided to limit the predicted most relevant labels for each instance. The `predict` function of the trained model will result in a CSR matrix of shape `(nr_inst, nr_labels)` and exactly `only_topk` non-zero columns for each row (or instance).\n", - "\n", - "
\n", - "
\n", - "
" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "7f851bc1", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Y_pred is a csr matrix of shape (6616, 30938) and 66160 non-zero elements.\n" - ] - } - ], - "source": [ - "Y_pred = xlm.predict(X_tst, beam_size=10, only_topk=10)\n", - "\n", - "print(f\"Y_pred is a {Y_pred.getformat()} matrix of shape {Y_pred.shape} and {Y_pred.nnz} non-zero elements.\")" - ] - }, - { - "cell_type": "markdown", - "id": "fb1ed22c", - "metadata": {}, - "source": [ - "For evaluation, we evaluate the trained model with conventional ranking metrics, including Precision@K and Recall@K. PECOS also provides the evaluation interface for predicted sparse matrices." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "4c57da1a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "prec = 84.36 78.20 72.67 68.03 63.85 60.23 56.86 53.85 51.07 48.63\n", - "recall = 4.99 9.17 12.65 15.67 18.28 20.56 22.50 24.21 25.74 27.15\n" - ] - } - ], - "source": [ - "from pecos.utils import smat_util\n", - "metrics = smat_util.Metrics.generate(Y_tst, Y_pred, topk=10)\n", - "print(metrics)" - ] - }, - { - "cell_type": "markdown", - "id": "ae9baf18", - "metadata": {}, - "source": [ - "### 3.5. PECOS and One-versus-All (OVA) Model\n", - "\n", - "PECOS also supports to train an OVA model without leveraing clustering hierarchy if needed. \n", - "\n", - "**Training OVA models is time-consuming, we suggest to try the following code offline after the tutorial. Note that training the above XR-Linear model is 26 times faster than training an OVA model using an AWS *i3.4xlarge* instance.**\n", - "\n", - "```python\n", - "import time\n", - "start_time = time.time()\n", - "\n", - "xlm_ova = XLinearModel.train(X_trn, Y_trn, C=None, negative_sampling_scheme=\"tfn\") \n", - "\n", - "training_time_ova = time.time() - start_time\n", - "print(f\"Training time for the OVA model: {training_time_ova:.4f} seconds.\")\n", - "pecos_faster_ratio = training_time_ova / training_time\n", - "print(f\"XR-Linear is {pecos_faster_ratio:.2f} times faster than the OVA model\")\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "1663277d", - "metadata": {}, - "source": [ - "# 4. Customized PECOS Model\n", - "\n", - "Besides pre-defined models in PECOS, such as XR-Linear, it is also convenient for users to customize PECOS for specific purposes and usage. Specifically, we suggest to establishing a model class to wrap fundamental PECOS functions and tailored operations. As a result, the customized model can be easily constructed and consumed for arbitrary data types and feature extractors. \n", - "\n", - "## 4.1. Structure of a Customized PECOS Model\n", - "\n", - "Even though a customized machine learning pipeline can be seperated into several independent scripts, we recommend declaring a customized PECOS model as a **model class** for better re-usability and code maintenance.\n", - "\n", - "A customized PECOS model should at least consist of the following components:\n", - "\n", - "* `preprocessor` or `encoder`: The procedure, which can be a method or a functionable object, pre-processes or encodes an arbitrary input with the designated data format into features. For example, text data and image data can be encoded by BERT and ResNet.\n", - "* `train()`: The training method takes a set of training data with a preprocessor, learns a primitive PECOS model, and returns a PECOS-based customized machine learning model. The training function could be a class method to construct the model object with the learned model and essential components after training.\n", - "* `model`: A primitive PECOS model taking pre-processed features is capable of deriving the predictions for arbitrary testing data. The model weights should be learned by `train()`. \n", - "* `predict()`: The prediction method takes arbitrary testing data and infers the prediction based on the pre-processor and the learned model.\n", - "* `save()`: The saving function serializes the trained model, including model weights and configuration, for further usage.\n", - "* `load()`: The loading function reads the serialized model so that the trained model can be loaded and re-used.\n", - "\n", - "
\n", - "
\n", - "
\n", - "\n", - "\n", - "In this part of the tutorial, we will use the task of *extreme multi-label text classification* as an example to demonstrate how to **customize a PECOS model that can handle text data with either a conventional bag-of-words (BoW) model or a deep learning model as the text encoder for feature extraction**.\n" - ] - }, - { - "cell_type": "markdown", - "id": "c3acc325", - "metadata": {}, - "source": [ - "## 4.2. Example: eXtreme Multi-label Text Classification (XMTC)\n", - "\n", - "The task of extreme multi-label text classification (XMTC) seeks to find relevant labels from an extreme large label collection for a given text input. Many real-world applications can be formulated as XMTC tasks, such as recommendation systems, document tagging, and semantic search. \n", - "\n", - "In this section, we guide through how to establish a customized PECOS model for XMTC tasks. We will walk through (1) PECOS' built-in BOW model for text preprocessing and vectorizing; (2) how to customize a PECOS model; and (3) \n", - "advanced usage of XR-Transformer based on deep learning." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "237164ca", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "text2text_demo/output-labels.txt\r\n", - "text2text_demo/testing-data.txt\r\n", - "text2text_demo/training-data.txt\r\n" - ] - } - ], - "source": [ - "! wget -nv -nc https://archive.org/download/text2text_demo.tar.gz/text2text_demo.tar.gz\n", - "! tar --skip-old-files -zxf text2text_demo.tar.gz\n", - "! find text2text_demo/*" - ] - }, - { - "cell_type": "markdown", - "id": "3b2ec0e6", - "metadata": {}, - "source": [ - "### 4.2.1. Preprocessor: Text Preprocessing and Vectorizing\n", - "\n", - "The preprocessor plays a role of encoding input data into machine readable vector representations. Any encoder that can transform text data into a vector representation can be considered as the preprocessor or encoder of a customized PECOS model for XMTC tasks.\n", - "\n", - "In the PECOS library, we provide [various text vectorizers](https://github.com/amzn/pecos/blob/mainline/pecos/utils/featurization/text/vectorizers.py), such as TF-IDF, hashing, and pretrained transformer, as **built-in preprocessors** to deal with text data. In this tutorial, we will utilize the [n-gram](https://en.wikipedia.org/wiki/N-gram) [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) model as our preprocessor.\n", - "\n", - "#### Label Space File Format for Built-in Text Preprocessors\n", - "\n", - "Label space is also essential for text preprocessors, especially for understanding the label space size to create the appropriate label matrix. The label IDs start from zero and can be referred to the line numbers and corresponding text descriptions in the label space file." - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "3f48f4f7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Artificial intelligence researchers\r\n", - "Computability theorists\r\n", - "British computer scientists\r\n", - "Machine learning researchers\r\n", - "Turing Award laureates\r\n", - "Deep Learning\r\n" - ] - } - ], - "source": [ - "! cat \"./text2text_demo/output-labels.txt\"" - ] - }, - { - "cell_type": "markdown", - "id": "e0862645", - "metadata": {}, - "source": [ - "#### Data File Format for Built-in Text Preprocessors\n", - "\n", - "PECOS built-in text preprocessors majorly take the files of text data with labels in a tab-separated values (TSV) format. Each line in the TSV file consists of two elements that represent the comma-separated label IDs and the input text of a data instance. " - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "bd5ebfc6", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "0,1,2\tAlan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.\r\n", - "0,2,3\tHinton was co-author of a highly cited paper published in 1986 that popularized the backpropagation algorithm for training multi-layer neural networks.\r\n", - "3,4,5\tHinton received the 2018 Turing Award, together with Yoshua Bengio and Yann LeCun, for their work on artificial intelligence and deep learning.\r\n", - "0,3,5\tYoshua Bengio is a Canadian computer scientist, most noted for his work on artificial neural networks and deep learning.\r\n" - ] - } - ], - "source": [ - "! cat ./text2text_demo/training-data.txt" - ] - }, - { - "cell_type": "markdown", - "id": "566d1eb5", - "metadata": {}, - "source": [ - "The data file format also supports to represent the label relevance for cost-sensitive learning by using double colons to separate a label and its relevance.\n", - "\n", - "

\n", - "0::0.1,1::0.2,2::0.8 <TAB> Alan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.

\n" - ] - }, - { - "cell_type": "markdown", - "id": "4d4e4419", - "metadata": {}, - "source": [ - "#### Training a Text Preprocessor\n", - "\n", - "The preprocessor model `Preprocessor` is defined in `pecos.utils.featurization.text.preprocess`. Given a training text corpus and the configuration dictionary, the class method `Preprocessor.train` will train a corresponding text preprocesssor. Besides, the built-in preprocessors also support serialization with the function `save()` for the re-usability.\n", - "\n", - "With the previously mentioned data and label space file formats, the utility function `Preprocessor.load_data_from_file(input_text_path, output_text_path)` returns a dictionary with three keys:\n", - "\n", - "* `label_matrix`: a `(num_inst, num_labels)` CSR matrix for the labels of each instance.\n", - "* `label_relevance`: `None` or a `(num_inst, num_labels)` CSR matrix for the relevance of each label in cost-sensitive learning if available.\n", - "* `corpus`: a list of string as the text corpus in the input_text_path.\n", - "\n", - "The configuration settings of text preprocessor including the preprocessor type and hyper-parameters should be defined in a dictionary. Specifially, the key `type` defines the preprocessor choice while the key `kwargs` represents the hyper-parameters. In this tutorial, we adopt n-gram TFIDF features containing *word unigrams*, *word bigrams*, and *character trigrams*. Note that each of the n-gram feature can have different hyper-parameters, such as `max_feature` and `max_df`. Users need to properly set max_feature (e.g., hundred of thousands or millions) based on the corpus size and downstream tasks." - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "b7f70a8f", - "metadata": {}, - "outputs": [], - "source": [ - "from pecos.utils.featurization.text.preprocess import Preprocessor\n", - "\n", - "input_text_path = \"./text2text_demo/training-data.txt\"\n", - "output_text_path = \"./text2text_demo/output-labels.txt\"\n", - "model_folder = \"./text2text_demo/pecos-text2text-model\"\n", - "\n", - "parsed_result = Preprocessor.load_data_from_file(input_text_path, output_text_path) # Read files\n", - "corpus = parsed_result[\"corpus\"] # Corpus input text: List of strings\n", - "Y = parsed_result[\"label_matrix\"] # Label Matrix: Sparse Matrix\n", - "\n", - "vectorizer_config = {\n", - " \"type\": \"tfidf\",\n", - " \"kwargs\": {\n", - " \"base_vect_configs\": [\n", - " \n", - " {\n", - " \"ngram_range\": [1, 1],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"word\",\n", - " },\n", - " {\n", - " \"ngram_range\": [2, 2],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"word\",\n", - " },\n", - " {\n", - " \"ngram_range\": [3, 3],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"char_wb\",\n", - " },\n", - " ],\n", - " },\n", - " }\n", - "\n", - "preprocessor = Preprocessor.train(corpus, vectorizer_config)\n", - "preprocessor.save(model_folder) " - ] - }, - { - "cell_type": "markdown", - "id": "a0300f8c", - "metadata": {}, - "source": [ - "#### Preprocessing with a Trained Text Preprocessor\n", - "\n", - "The function `predict` of a trained text preprocessor encodes texts in a **text data file** into a CSR matrix of shape `(num_inst, dim)` as numerical vector representations, where `num_inst` is the number of instances in the file; `dim` is the number of feature dimensions." - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "3b182171", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The file consists of 4 instances with {X.shape[1]}-dimensional features in a {X.getformat()} matrix.\n", - "\n", - "Text 0: Alan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.\n", - "Text 1: Hinton was co-author of a highly cited paper published in 1986 that popularized the backpropagation algorithm for training multi-layer neural networks.\n", - "Text 2: Hinton received the 2018 Turing Award, together with Yoshua Bengio and Yann LeCun, for their work on artificial intelligence and deep learning.\n", - "Text 3: Yoshua Bengio is a Canadian computer scientist, most noted for his work on artificial neural networks and deep learning.\n", - "\n", - "The cosine similarity is 0.0076 between text 0 and text 1.\n", - "The cosine similarity is 0.0325 between text 0 and text 2.\n", - "The cosine similarity is 0.0082 between text 1 and text 2.\n", - "The cosine similarity is 0.0366 between text 0 and text 3.\n", - "The cosine similarity is 0.0267 between text 1 and text 3.\n", - "The cosine similarity is 0.0943 between text 2 and text 3.\n" - ] - } - ], - "source": [ - "# Obtaining numerical vectors from text\n", - "X = preprocessor.predict(corpus)\n", - "\n", - "print(f\"The file consists of {X.shape[0]} instances \"\n", - " \"with {X.shape[1]}-dimensional features \"\n", - " \"in a {X.getformat()} matrix.\\n\")\n", - "\n", - "from sklearn.metrics.pairwise import cosine_similarity\n", - "\n", - "sim = cosine_similarity(X)\n", - "\n", - "for i, ti in enumerate(corpus):\n", - " print(f\"Text {i}: {ti}\")\n", - "\n", - "print(\"\")\n", - "for i in range(X.shape[0]):\n", - " for j in range(i):\n", - " print(f\"The cosine similarity is {sim[i][j]:.4f} between text {j} and text {i}.\")" - ] - }, - { - "cell_type": "markdown", - "id": "18fcd09b", - "metadata": {}, - "source": [ - "#### Command-line Interface\n", - "\n", - "The above vectorizer operations can also be achieved by the following commands with a JSON-format configuration file:\n", - "\n", - "```bash\n", - "python3 -m pecos.utils.featurization.text.preprocess build \\\n", - " --text-pos 1 \\\n", - " --input-text-path ./text2text_demo/training-data.txt \\\n", - " --vectorizer-config-path /path/to/vectorizer-config.json \\\n", - " --output-model-folder ./text2text_demo/pecos-text2text-model\n", - "\n", - "python3 -m pecos.utils.featurization.text.preprocess run \\\n", - " --input-preprocessor-folder ./text2text_demo/pecos-text2text-model \\\n", - " --text-pos 1 \\\n", - " --input-text-path ./text2text_demo/training-data.txt \\\n", - " --output-inst-path /path/to/X.npz\n", - " --label-pos 0 \\\n", - " --output-label-path /path/to/Y.npz \\\n", - " --label-text-path ./text2text_demo/output-labels.txt\n", - "```\n", - "\n", - "#### Efficiency of PECOS Built-in TF-IDF Vectorizer\n", - "\n", - "Moreover, the TF-IDF vectorizer in PECOS is implemented in C++ and efficient." - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "a3d6f675", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "PECOS TFIDF time: 27.13237s, result shape=(14146, 10858825), nnz=37194670\n" - ] - } - ], - "source": [ - "vectorizer_config = {\n", - " \"type\": \"tfidf\",\n", - " \"kwargs\": {\n", - " \"base_vect_configs\": [ \n", - " {\n", - " \"ngram_range\": [1, 2],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"word\",\n", - " },\n", - " ],\n", - " },\n", - " }\n", - "\n", - "input_text_path = \"xmc-base/wiki10-31k/X.trn.txt\"\n", - "corpus = Preprocessor.load_data_from_file(input_text_path, text_pos=0)[\"corpus\"]\n", - "\n", - "import time\n", - "start_time = time.time()\n", - "preprocessor = Preprocessor.train(corpus, vectorizer_config)\n", - "X = preprocessor.predict(input_text_path)\n", - "print(f\"PECOS TFIDF time: {time.time() - start_time:.5f}s, result shape={X.shape}, nnz={X.nnz}\")" - ] - }, - { - "cell_type": "markdown", - "id": "e63c62ce", - "metadata": {}, - "source": [ - "As a baseline method, we compare with the [Sklearn TFIDF vectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html):\n", - "```python\n", - "start_time = time.time()\n", - "preprocessor = Preprocessor.train(\n", - " corpus,\n", - " {\"type\": \"sklearntfidf\", \"kwargs\":{\"ngram_range\": [1, 2], \"max_df\": 0.98}},\n", - ")\n", - "X = preprocessor.predict(corpus)\n", - "print(f\"Sklearn TFIDF time: {time.time() - start_time:.5f}s, result shaepe={X.shape}, nnz={X.nnz}\")\n", - "```\n", - "\n", - "**Training Sklearn TFIDF models is time-consuming, we suggest to try the following code offline after the tutorial. The execution results using an AWS *i3.4xlarge* instance are as follows:**\n", - "```\n", - "Sklearn TFIDF time: 221.40709s, result shaepe=(14146, 7269690), nnz=33505461\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "75f5aaf8", - "metadata": {}, - "source": [ - "### 4.2.2 Customized PECOS Model with TF-IDF Preprocessor\n", - "\n", - "\n", - "After being powered with text preprocessors, following the [aforementioned illustration](#Structure-of-a-Customized-PECOS-Model), we demonstrate an example of declaring a **customized PECOS model class** based on a TF-IDF preprocessor and a XR-Linear model." - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "3893c23b", - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "from os import path\n", - "import pathlib\n", - "from pecos.utils.featurization.text.preprocess import Preprocessor\n", - "from pecos.xmc.xlinear.model import XLinearModel\n", - "from pecos.xmc import Indexer, LabelEmbeddingFactory\n", - "from pecos.utils import smat_util\n", - "\n", - "class CustomPECOS:\n", - " def __init__(self, preprocessor=None, xlinear_model=None, output_items=None):\n", - " self.preprocessor = preprocessor\n", - " self.xlinear_model = xlinear_model\n", - " self.output_items = output_items\n", - " \n", - " @classmethod\n", - " def train(cls, input_text_path, output_text_path):\n", - " \"\"\"Train a CustomPECOS model\n", - " \n", - " Args: \n", - " input_text_path (str): Text input file name. \n", - " output_text_path (str): The file path for output text items.\n", - " vectorizer_config (str): Json_format string for vectorizer config (default None). e.g. {\"type\": \"tfidf\", \"kwargs\": {}}\n", - " \n", - " Returns:\n", - " A CustomPECOS object\n", - " \"\"\"\n", - " # Obtain X_text, Y\n", - " parsed_result = Preprocessor.load_data_from_file(input_text_path, output_text_path)\n", - " Y = parsed_result[\"label_matrix\"]\n", - " corpus = parsed_result[\"corpus\"]\n", - "\n", - " # Train TF-IDF vectorizer\n", - " preprocessor = Preprocessor.train(corpus, {\"type\": \"tfidf\", \"kwargs\":{}}) \n", - " X = preprocessor.predict(corpus) \n", - " \n", - " # Train a XR-Linear model with TF-IDF features\n", - " label_feat = LabelEmbeddingFactory.create(Y, X, method=\"pifa\")\n", - " cluster_chain = Indexer.gen(label_feat)\n", - " xlinear_model = XLinearModel.train(X, Y, C=cluster_chain)\n", - " \n", - " # Load output items\n", - " with open(output_text_path, \"r\", encoding=\"utf-8\") as f:\n", - " output_items = [q.strip() for q in f]\n", - " \n", - " return cls(preprocessor, xlinear_model, output_items)\n", - " \n", - " def predict(self, corpus):\n", - " \"\"\"Predict labels for given inputs\n", - " \n", - " Args:\n", - " corpus (list of strings): input strings.\n", - " Returns:\n", - " csr_matrix: predicted label matrix (num_samples x num_labels)\n", - " \"\"\"\n", - " X = self.preprocessor.predict(corpus)\n", - " Y_pred = self.xlinear_model.predict(X)\n", - " return smat_util.sorted_csr(Y_pred)\n", - "\n", - " def save(self, model_folder):\n", - " \"\"\"Save the CustomPECOS model\n", - "\n", - " Args:\n", - " model_folder (str): folder name to save\n", - " \"\"\"\n", - " self.preprocessor.save(f\"{model_folder}/preprocessor\")\n", - " self.xlinear_model.save(f\"{model_folder}/xlinear_model\")\n", - " with open(f\"{model_folder}/output_items.json\", \"w\", encoding=\"utf-8\") as fp:\n", - " json.dump(self.output_items, fp)\n", - "\n", - " @classmethod\n", - " def load(cls, model_folder):\n", - " \"\"\"Load the CustomPECOS model\n", - "\n", - " Args:\n", - " model_folder (str): folder name to load\n", - " Returns:\n", - " CustomPECOS\n", - " \"\"\"\n", - " preprocessor = Preprocessor.load(f\"{model_folder}/preprocessor\")\n", - " xlinear_model = XLinearModel.load(f\"{model_folder}/xlinear_model\")\n", - " with open(f\"{model_folder}/output_items.json\", \"r\", encoding=\"utf-8\") as fin:\n", - " output_items = json.load(fin)\n", - " return cls(preprocessor, xlinear_model, output_items)" - ] - }, - { - "cell_type": "markdown", - "id": "fcdbb2c6", - "metadata": {}, - "source": [ - "### 4.2.3. Operating the Customized PECOS Model\n", - "\n", - "With a well-declared model class, the customized PECOS model can be modularized and very convenient to use." - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "id": "24134357", - "metadata": {}, - "outputs": [], - "source": [ - "# Declare the path for model serialization and preprocessor configuration.\n", - "model_folder = \"./text2text_demo/pecos-CustomPECOS-model\"\n", - "\n", - "# Train and save the trained model\n", - "input_text_path = \"./text2text_demo/training-data.txt\"\n", - "output_text_path = \"./text2text_demo/output-labels.txt\"\n", - "model = CustomPECOS.train(input_text_path, output_text_path)\n", - "model.save(model_folder)\n", - "\n", - "# Load the trained model and predict\n", - "model = model.load(model_folder)\n", - "testing_text_path = \"./text2text_demo/testing-data.txt\"\n", - "Y_pred = model.predict(testing_text_path)" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "id": "31efd9ac", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Text Input: In 1989, Yann LeCun et al. applied the standard backpropagation algorithm on neural networks for hand digit recognition.\n", - "Score 0.9515: Machine learning researchers\n", - "Score 0.8233: Artificial intelligence researchers\n", - "Score 0.4659: Deep Learning\n", - "Score 0.2779: British computer scientists\n", - "Score 0.0569: Turing Award laureates\n", - "Score 0.0129: Computability theorists\n" - ] - } - ], - "source": [ - "test_texts = Preprocessor.load_data_from_file(testing_text_path, output_text_path)[\"corpus\"]\n", - "\n", - "for i, text in enumerate(test_texts):\n", - " print(f\"Text Input: {text}\")\n", - " for j in range(Y_pred.indptr[i], Y_pred.indptr[i + 1]):\n", - " pred_label = model.output_items[Y_pred.indices[j]]\n", - " pred_score = Y_pred.data[j]\n", - " print(f\"Score {pred_score:.4f}: {pred_label}\")" - ] - }, - { - "cell_type": "markdown", - "id": "32908310", - "metadata": {}, - "source": [ - "## Appedix 1: Model Parameters in PECOS Implementation\n", - "\n", - "### A1.1. Cluster Chain in PECOS Implementation\n", - "\n", - "Specifically, PECOS trains a *cluster_chain* of `D` matching matrices `C[d]`, where `C[d]` is a CSC matrix of shape `(L[d], K[d])`; `L[d]` and `K[d]` are the numbers of labels and clusters in the layer `d`. Note that the clusters of a layer would be the labels of the next layer. The labels of the last layer `L[D - 1]` would be the labels of the overall XMC problem `nr_labels`." - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "id": "6b0cb55e", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "4 layers in the trained hierarchical label tree with C[d] as:\n", - "cluster_chain[0] is a csc matrix of shape (2, 1)\n", - "cluster_chain[1] is a csc matrix of shape (32, 2)\n", - "cluster_chain[2] is a csc matrix of shape (512, 32)\n", - "cluster_chain[3] is a csc matrix of shape (30938, 512)\n" - ] - } - ], - "source": [ - "print(f\"{len(cluster_chain)} layers in the trained hierarchical label tree with C[d] as:\")\n", - "for d, C in enumerate(cluster_chain):\n", - " print(f\"cluster_chain[{d}] is a {C.getformat()} matrix of shape {C.shape}\")" - ] - }, - { - "cell_type": "markdown", - "id": "790b21dc", - "metadata": {}, - "source": [ - "### A2.2. Model Weights in PECOS Implementation\n", - "\n", - "Model weights in an XR-Linear model are also accessible as `model_chain` for analysis and computations. For the i-th layer in the hierarchy, the model weights of matchers/rankers are available as a CSC matrix of shape `(nr_feat + 1, L[i])`, which concatenates weights for features and the bias term. " - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "id": "9e101f6b", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "model_chain[0].W is a csc matrix of shape (101939, 2)\n", - "model_chain[1].W is a csc matrix of shape (101939, 32)\n", - "model_chain[2].W is a csc matrix of shape (101939, 512)\n", - "model_chain[3].W is a csc matrix of shape (101939, 30938)\n" - ] - } - ], - "source": [ - "for d, m in enumerate(xlm.model.model_chain):\n", - " print(f\"model_chain[{d}].W is a {m.W.getformat()} matrix of shape {m.W.shape}\")" - ] - }, - { - "cell_type": "markdown", - "id": "73a599a0", - "metadata": {}, - "source": [ - "## Appendix 2: Customized Parameters and Advanced Training Options\n", - "\n", - "PECOS also supports using customized parameters and several advanced training options, such as different solvers and cost-sensitive learning.\n", - "\n", - "### A2.1. Customized Parameters\n", - "\n", - "The parameters for either of indexing, training, and inference can be easily customized by feeding a dictionary into the corresponding parameter class and its constructor:\n", - "\n", - "* Semantic Indexing (Hierarchical K-Means): `HierarchicalKMeans.TrainParams.from_dict(dict)`\n", - "* Training: `XLinearModel.TrainParams.from_dict(dict)`\n", - "* Inference: `XLinearModel.PredParams.from_dict(dict)`\n", - "\n", - "Although most of the parameters can be also passed by `kwargs` of Python methods, **we encourage to use the dictionary to designate the parameters because it is easier to manage, modularize, and store parameters in certain formats like JSON.**\n", - "\n", - "For XR-Linear models, the default values and skeleton of the parameters can be revealed and generated by the following command:" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "id": "1ddc9bfa", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\r\n", - " \"train_params\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.xlinear.model###XLinearModel.TrainParams\"\r\n", - " },\r\n", - " \"mode\": \"full-model\",\r\n", - " \"ranker_level\": 1,\r\n", - " \"nr_splits\": 16,\r\n", - " \"min_codes\": null,\r\n", - " \"shallow\": false,\r\n", - " \"rel_mode\": \"disable\",\r\n", - " \"rel_norm\": \"no-norm\",\r\n", - " \"hlm_args\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###HierarchicalMLModel.TrainParams\"\r\n", - " },\r\n", - " \"neg_mining_chain\": \"tfn\",\r\n", - " \"model_chain\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###MLModel.TrainParams\"\r\n", - " },\r\n", - " \"threshold\": 0.1,\r\n", - " \"max_nonzeros_per_label\": null,\r\n", - " \"solver_type\": \"L2R_L2LOSS_SVC_DUAL\",\r\n", - " \"Cp\": 1.0,\r\n", - " \"Cn\": 1.0,\r\n", - " \"max_iter\": 100,\r\n", - " \"eps\": 0.1,\r\n", - " \"bias\": 1.0,\r\n", - " \"threads\": -1,\r\n", - " \"verbose\": 0,\r\n", - " \"newton_eps\": 0.01\r\n", - " }\r\n", - " }\r\n", - " },\r\n", - " \"pred_params\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.xlinear.model###XLinearModel.PredParams\"\r\n", - " },\r\n", - " \"hlm_args\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###HierarchicalMLModel.PredParams\"\r\n", - " },\r\n", - " \"model_chain\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###MLModel.PredParams\"\r\n", - " },\r\n", - " \"only_topk\": 20,\r\n", - " \"post_processor\": \"l3-hinge\"\r\n", - " }\r\n", - " }\r\n", - " },\r\n", - " \"indexer_params\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###HierarchicalKMeans.TrainParams\"\r\n", - " },\r\n", - " \"nr_splits\": 16,\r\n", - " \"min_codes\": null,\r\n", - " \"max_leaf_size\": 100,\r\n", - " \"imbalanced_ratio\": 0.0,\r\n", - " \"imbalanced_depth\": 100,\r\n", - " \"spherical\": true,\r\n", - " \"seed\": 0,\r\n", - " \"kmeans_max_iter\": 20,\r\n", - " \"threads\": -1\r\n", - " }\r\n", - "}\r\n" - ] - } - ], - "source": [ - "! python3 -m pecos.xmc.xlinear.train --generate-params-skeleton" - ] - }, - { - "cell_type": "markdown", - "id": "35472517", - "metadata": {}, - "source": [ - "### A2.2. Training Parameters for Hierarchial Models in XR-Linear\n", - "\n", - "Hierarchical models could have different parameters over layers. To have customized parameters for the hierarchical model, `hlm_args` needs to be designated in the parameter dictionary. The values of `model_chain` and `neg_mining_chain` in `hlm_args` can be **a single dictionary** of general parameters for all layers or **a list of dictinoaries** for specific parameters of individual layers.\n", - "\n", - "#### General Parameters for All Layers\n", - "\n", - "```\n", - "train_params_l1 = XLinearModel.TrainParams.from_dict(\n", - " {\n", - " ...\n", - " \"hlm_args\": {\n", - " ...\n", - " \"neg_mining_chain\": \"tfn\", # Negative sampling scheme for all layers\n", - " \"model_chain\":{...}, # Parameters for all layers\n", - " }\n", - " ...\n", - " })\n", - "```\n", - "\n", - "#### Specific Parameters of Individual Layers\n", - "\n", - "```\n", - "train_params_l1 = XLinearModel.TrainParams.from_dict(\n", - " {\n", - " ...\n", - " \"hlm_args\": {\n", - " ...\n", - " \"neg_mining_chain\": [\n", - " \"tfn\", # Negative sampling scheme for layer-0\n", - " \"tfn\", # Negative sampling scheme for layer-1\n", - " \"tfn+man\", # Negative sampling scheme for layer-2\n", - " ...\n", - " ],\n", - " \"model_chain\": [\n", - " {...}, # Parameters for layer-0\n", - " {...}, # Parameters for layer-1\n", - " {...}, # Parameters for layer-2\n", - " ...\n", - " ],\n", - " }\n", - " ...\n", - " })\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "81f86b8a", - "metadata": {}, - "source": [ - "### A2.3. Variety of Solvers\n", - "\n", - "The solver for optimization can be adjusted by the argument `solver_type` in the `train` function. PECOS currently provides the following solvers for training each matcher/ranker:\n", - "\n", - "* \"L2R_L2LOSS_SVC_DUAL\" (default): L2-regularized L2-loss Dual SVM\n", - "* \"L2R_L1LOSS_SVC_DUAL\": : L2-regularized L1-loss Dual SVM\n", - "* \"L2R_LR_DUAL\": L2-reguarlized Logistic Regression" - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "id": "8f26ee42", - "metadata": {}, - "outputs": [], - "source": [ - "xlm_l1_kwargs = XLinearModel.train(\n", - " X_trn, Y_trn,\n", - " C=cluster_chain,\n", - " threshold=0.1,\n", - " negative_sampling_scheme=\"tfn\",\n", - " solver_type=\"L2R_L1LOSS_SVC_DUAL\")" - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "id": "197926a7", - "metadata": {}, - "outputs": [], - "source": [ - "train_params_l1 = XLinearModel.TrainParams.from_dict(\n", - " {\n", - " \"hlm_args\": {\n", - " \"threshold\": 0.1,\n", - " \"neg_mining_chain\": \"tfn\",\n", - " \"model_chain\":{\n", - " \"solver_type\": \"L2R_L1LOSS_SVC_DUAL\",\n", - " },\n", - " }\n", - " }\n", - ")\n", - "\n", - "xlm_l1_dict = XLinearModel.train(\n", - " X_trn, Y_trn,\n", - " C=cluster_chain,\n", - " train_params=train_params_l1)" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "id": "eddf91a6", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation Metrics with L2R_L1LOSS_SVC_DUAL (by method kwargs)\n", - "prec = 83.65 77.27 72.01 67.67 63.91 60.53 57.33 54.53 52.03 49.66\n", - "recall = 4.94 9.06 12.53 15.59 18.26 20.64 22.67 24.52 26.23 27.72\n", - "\n", - "Evaluation Metrics with L2R_L1LOSS_SVC_DUAL (by dictionary)\n", - "prec = 83.65 77.27 72.01 67.67 63.91 60.53 57.33 54.53 52.03 49.66\n", - "recall = 4.94 9.06 12.53 15.59 18.26 20.64 22.67 24.52 26.23 27.72\n" - ] - } - ], - "source": [ - "Y_pred_l1_kwargs = xlm_l1_kwargs.predict(X_tst, beam_size=10, only_topk=10)\n", - "Y_pred_l1_dict = xlm_l1_dict.predict(X_tst, beam_size=10, only_topk=10)\n", - "metrics_l1_kwargs = smat_util.Metrics.generate(Y_tst, Y_pred_l1_kwargs, topk=10)\n", - "metrics_l1_dict = smat_util.Metrics.generate(Y_tst, Y_pred_l1_dict, topk=10)\n", - "\n", - "print(\"Evaluation Metrics with L2R_L1LOSS_SVC_DUAL (by method kwargs)\")\n", - "print(metrics_l1_kwargs)\n", - "\n", - "print(\"\\nEvaluation Metrics with L2R_L1LOSS_SVC_DUAL (by dictionary)\")\n", - "print(metrics_l1_dict)" - ] - }, - { - "cell_type": "markdown", - "id": "9f481ed3", - "metadata": {}, - "source": [ - "## Appendix 3: Cost-sensitive Learning\n", - "\n", - "PECOS supports to adjust the cost of each training instance. To enable cost-sensitive learning, we need to provide a **relevance matrix** `R_trn` with the same shape to the label matrix `Y_trn` for the argument `R`. When `R` is `None` (default), cost-sensitive learning is disable. \n", - "\n", - "Since PECOS models are usually hierarhical, costs for upper layers also need to be decided as the cost-sensitive learning mode by the argument `rel_mode`. Currently, PECOS supports the following cost-sensitive learning modes:\n", - "\n", - "* `\"disable\"` (default): The cost-sensitive learning is disable.\n", - "* `\"induce\"`: Induce the costs into upper layers by the clustering chain.\n", - "* `\"ranker-only\"`: Only apply cost-sensitive learning to the model in the last ranker layer without induction.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "382277f3", - "metadata": {}, - "outputs": [], - "source": [ - "# An exmaple of using training label frequency scores as costs. \n", - "import copy\n", - "from sklearn.preprocessing import normalize\n", - "\n", - "R_trn = copy.deepcopy(Y_trn)\n", - "\n", - "# Training parameters for cost-sensitive learning.\n", - "train_params_cost = XLinearModel.TrainParams.from_dict(\n", - " {\n", - " \"rel_mode\": \"induce\",\n", - " \"rel_norm\": \"l1\",\n", - " \"hlm_args\": {\n", - " \"neg_mining_chain\": \"tfn\",\n", - " \"model_chain\":\n", - " [\n", - " {\n", - " \"threshold\": 0.1,\n", - " \"Cp\": 1.0,\n", - " \"Cn\": 1.0,\n", - " },\n", - " {\n", - " \"threshold\": 0.1,\n", - " \"Cp\": 8.0,\n", - " \"Cn\": 1.0,\n", - " },\n", - " {\n", - " \"threshold\": 0.1,\n", - " \"Cp\": 4.0,\n", - " \"Cn\": 1.0,\n", - " },\n", - " {\n", - " \"threshold\": 0.1,\n", - " \"Cp\": 4.0,\n", - " \"Cn\": 1.0,\n", - " },\n", - " ],\n", - " }\n", - " })\n", - " \n", - "# Cost-sensitive learning.\n", - "xlm_cost = XLinearModel.train(\n", - " X_trn, Y_trn,\n", - " C=cluster_chain,\n", - " R=R_trn,\n", - " train_params=train_params_cost)" - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "id": "559c15cb", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation Metrics with Cost-sensitive Learning\n", - "prec = 84.93 80.15 74.50 69.32 64.73 61.05 57.68 54.52 51.70 49.10\n", - "recall = 5.01 9.42 13.00 15.99 18.52 20.83 22.85 24.54 26.09 27.43\n", - "\n", - "Original Evaluation Metrics\n", - "prec = 84.36 78.20 72.67 68.03 63.85 60.23 56.86 53.85 51.07 48.63\n", - "recall = 4.99 9.17 12.65 15.67 18.28 20.56 22.50 24.21 25.74 27.15\n" - ] - } - ], - "source": [ - "Y_pred_cost = xlm_cost.predict(X_tst, beam_size=10, only_topk=10)\n", - "metrics_cost = smat_util.Metrics.generate(Y_tst, Y_pred_cost, topk=10)\n", - "print(\"Evaluation Metrics with Cost-sensitive Learning\")\n", - "print(metrics_cost)\n", - "print(\"\\nOriginal Evaluation Metrics\")\n", - "print(metrics)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/tutorials/kdd22/Session 3 Approximate Nearest Neighbor Search in PECOS.ipynb b/tutorials/kdd22/Session 3 Approximate Nearest Neighbor Search in PECOS.ipynb deleted file mode 100644 index 765b7148..00000000 --- a/tutorials/kdd22/Session 3 Approximate Nearest Neighbor Search in PECOS.ipynb +++ /dev/null @@ -1,864 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "30e3e801-3659-4455-b405-5a2b083ea952", - "metadata": { - "tags": [] - }, - "source": [ - "# How to Perform Approximate Nearest Neighbor Search in PECOS" - ] - }, - { - "cell_type": "markdown", - "id": "e5073aac", - "metadata": { - "tags": [] - }, - "source": [ - "## Introduction\n", - "Recall that PECOS is a scalable ML library for predictions on enormous and correlated output space. Therefore, we also support inference of embedding-based models such as dual-encoders, which is often formulated as a Maximum Inner Product Search (**MIPS**) or equivalently, an approximate nearest neighbor (**ANN**) search problem.\n", - "\n", - "In PECOS, we implemented a state-of-the-art graph-based ANN search algorithm, namely **H**ierarchical **N**avigable **S**mall **W**orld (**HNSW**) model. The life-cycle of HNSW can be divided into twp steps:\n", - "* *Training*: given user-provided database vectors, build the HNSW graph data structures for indexing;\n", - "* *Prediction*: given any query vector, return the top-K approximate nearest vectors indexed in the database.\n", - "\n", - "More specifically, the **search (i.e., inference)** procedure of HNSW can be summarized as:\n", - "* For each layer, conduct best first search traversal. The best candidate serves as an initial point to next layer;\n", - "* Traverse from top layer (course-grain graph, long-range link) to bottom layer (fine-grain graph, short-range link);\n", - "* The bottom layer graph contains all database items as the nodes.\n", - "

" - ] - }, - { - "cell_type": "markdown", - "id": "2eaba286-135c-4694-9756-638c8b3b1169", - "metadata": {}, - "source": [ - "## Highlight of PECOS-HNSW\n", - "In this part of tutorial, we introduce the usage of PECOS-HNSW to tackle ANN search problem, and highlight some key functionalities and features in PECOS-HNSW:\n", - "\n", - "* support inference on both sparse and dense input features;\n", - "* support SIMD instructions (AVX, AVX256, and AVX512) and select the best available one in runtime;\n", - "* achieve new SOTA results compared to other popular graph-based ANN libraries (e.g., NMSLIB and HNSWLIB).\n", - "

" - ] - }, - { - "cell_type": "markdown", - "id": "94b45789-b24f-4ed5-8d9c-0568c2d4d1cc", - "metadata": { - "tags": [] - }, - "source": [ - "## Benchmarking PECOS-HNSW with NMSLIB/HNSWLIB\n", - "\n", - "### Disclaimer \n", - "We follow [ANN-Benchmark](https://github.com/erikbern/ann-benchmarks) evaluation protocol, which conducts inference on each test query sequentially (**batch_size=1**) with the **single-thread** setup. \n", - "The benchmarking results are based on an r5dn-24xlarge (**w/ avx512 supports**) AWS instance with 96 Intel(R) Xeon(R) Platinum 8259CL CPUs @ 2.50GHz. With distinct environments, the magnitude of improvements could be also different.\n", - "\n", - "
\n", - "
\n", - " \n", - "
\n", - "
\n", - " \n", - "
\n", - "
\n", - "\n", - "### Results on *RCV1-47236-angular*\n", - "\n", - "* For RCV1, the instances in training/test set are 781,265 and 23,149, respectively. The feature dimension is 47,236.\n", - "* PECOS-HNSW achieves an average of **1.9x** speedup compared to the NMSLIB package.\n", - "\n", - "### Results on on *SIFT-128-euclidean*\n", - "\n", - "* For SIFT, the instances in training/test set are 1,000,000 and 10,000, respectively. The feature dimension is 128.\n", - "* PECOS-HNSW achieved an average of **1.3x** speedup compared to the HNSWLIB package.\n", - "* PECOS-HNSW++ (ongoing work) achieved an average of **3x** speedup compared to the HNSWLIB package." - ] - }, - { - "cell_type": "markdown", - "id": "d74d7466-e379-4378-9992-19f3bdb09ccc", - "metadata": { - "tags": [] - }, - "source": [ - "## Hands-on Tutorial" - ] - }, - { - "cell_type": "markdown", - "id": "abb5ff7e", - "metadata": {}, - "source": [ - "### Data Loading" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "140a0d24", - "metadata": {}, - "outputs": [], - "source": [ - "! wget https://archive.org/download/pecos-dataset/ann-benchmarks/rcv1-angular-47236.tar.gz\n", - "! tar -zxvf ./rcv1-angular-47236.tar.gz" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "46dc982b", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "n_trn 781265 n_tst 23149 data_dim 47236\n" - ] - } - ], - "source": [ - "import numpy as np\n", - "import os, time\n", - "from pecos.utils import smat_util\n", - "from pecos.ann.hnsw import HNSW\n", - "X_trn = smat_util.load_matrix(\"./rcv1-angular-47236/X.trn.npz\").astype(np.float32)\n", - "X_tst = smat_util.load_matrix(\"./rcv1-angular-47236/X.tst.npz\").astype(np.float32)\n", - "Y_tst = smat_util.load_matrix(\"./rcv1-angular-47236/Y.tst.npy\")\n", - "print(\"n_trn {:7d} n_tst {:7d} data_dim {:7d}\".format(\n", - " X_trn.shape[0], X_tst.shape[0], X_trn.shape[1])\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "bb73c619", - "metadata": {}, - "source": [ - "### Training Indexer\n", - "\n", - "To train a [PECOS-HNSW](https://github.com/amzn/pecos/tree/v0.4.0/pecos/ann/hnsw) model, training parameters need to be defined in an object of [HNSW.TrainParams](https://github.com/amzn/pecos/blob/v0.4.0/pecos/ann/hnsw/model.py#L33) as the argument `train_params`.\n", - "\n", - "The key parameters of training a [PECOS-HNSW](https://github.com/amzn/pecos/tree/v0.4.0/pecos/ann/hnsw) model include:\n", - "* `M` (default 32): The maximum number of edges per node for each layer. A larger M leads to a larger model size and greater memory consumption. Higher/lower M are more suitable for high/low dimensional data or the pursue of high/low recall.\n", - "* `efC` (default 100): The size of the priority queue for best first search in construction. `efC` can be considered as the trade-off between efficiency and accuracy for indexing. A higher `efC` results in longer construction time but better quality of indexing.\n", - "* `metric_type` (default ip): The distance metric type for ANN search. PECOS-HNSW currently supports Euclidean distance (`l2`); and inner product (`ip`)\n", - "* `threads` (default -1): The number of threads for training, or -1 to use all available cores.\n", - "\n", - "Detailed hyper-parameters can be found in the original HNSW paper ([Malkov et al, TPAMI 2018](https://arxiv.org/abs/1603.09320)).\n", - "\n", - "The parameters for inference can be also decided as the argument `pred_params` during model construction so that the model can be directly applied for inference without further parameter designation.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "553aaf55", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "HNSW Indexer | M 32 efC 100 metric ip | time(s) 12.000147581100464\n" - ] - } - ], - "source": [ - "M, efC = 32, 100\n", - "metric = \"ip\"\n", - "train_params = HNSW.TrainParams(\n", - " M=M,\n", - " efC=efC,\n", - " metric_type=metric,\n", - " threads=-1,\n", - ")\n", - "start_time = time.time()\n", - "model = HNSW.train(X_trn, train_params=train_params, pred_params=None)\n", - "print(\"HNSW Indexer | M {} efC {} metric {} | time(s) {}\".format(\n", - " M, efC, metric, time.time() - start_time),\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "6a150c44", - "metadata": {}, - "source": [ - "### Save and Load Indexer" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "bf7905f3", - "metadata": {}, - "outputs": [], - "source": [ - "model_folder = \"./rcv1.pecos-hnsw.index\"\n", - "model.save(model_folder)\n", - "del model\n", - "model = HNSW.load(model_folder)" - ] - }, - { - "cell_type": "markdown", - "id": "b1af6ee7", - "metadata": {}, - "source": [ - "### Inference and Evaluation\n", - "\n", - "To conduct inference, prediction parameters need to be defined in an object of [HNSW.PredParams](https://github.com/amzn/pecos/blob/v0.4.0/pecos/ann/hnsw/model.py#L51) as the argument `pred_params`.\n", - "\n", - "The key parameters of inference with a PECOS-HNSW model include:\n", - "* `efS` (default 100): The size of the priority queue for best first search during inference. Similar to efC, efS can be considered as the trade-off between search efficiency and accuracy. A higher efS results in more accurate results with slower speed. efS is required to be greater than topk.\n", - "* `topk` (default 10): The number of approximate nearest neighbor to be returned. \n", - "* `threads` (default -1): The number of searchers for parallel inference, -1 to use all available searchers.\n", - "\n", - "Users should also construct `searchers` to avoid memory overhead\n", - "```\n", - "searchers = model.searchers_create(num_searcher=1) # multiple searchers inference multiple queries in parallel\n", - "```\n", - "\n", - "The predict function derives search results based on a query matrix of shape (# of data points for inference, # of dimensions), `pred_params`, and `searchers`. " - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "e25e31d4", - "metadata": {}, - "outputs": [], - "source": [ - "pred_params = HNSW.PredParams(efS=100, topk=10)\n", - "searchers = model.searchers_create(num_searcher=1) # multiple searchers inference multiple queries in parallel\n", - "start_time = time.time()\n", - "indices, distances = model.predict(\n", - " X_tst,\n", - " pred_params=pred_params,\n", - " searchers=searchers,\n", - " ret_csr=False,\n", - ")\n", - "pred_time = time.time() - start_time" - ] - }, - { - "cell_type": "markdown", - "id": "dc06d282-ba6b-4a57-a963-1203fcc87c63", - "metadata": {}, - "source": [ - "The argument `ret_csr` (default `true`) decides the format of returned results as:\n", - "\n", - "* If `ret_csr` is false, the returned results would be two matrices of shape (# of data points, topk), which indicate the topk indices in the training corpus and the corresponding distances for each testing instance.\n", - "* If `ret_csr` is true, the returned results would be a [Compressed Sparse Row (CSR) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) of shape (# of data points, # of points in the training corpus). Each row contains sorted topk distance values at the corresponding columns (i.e., indices in training corpus). The data for each row (i.e., `data[indptr[i]:indptr[i + 1]]`) are also sorted by the distance values." - ] - }, - { - "cell_type": "markdown", - "id": "0ce9aefa", - "metadata": { - "tags": [] - }, - "source": [ - "### Evaluation" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "38401700", - "metadata": {}, - "outputs": [], - "source": [ - "def compute_recall(neighbors, true_neighbors):\n", - " total = 0\n", - " for gt_row, row in zip(true_neighbors, neighbors):\n", - " total += np.intersect1d(gt_row, row).shape[0]\n", - " return total / true_neighbors.size" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "b0b6d72a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "HNSW inference | R@10 0.9020 Throughput(q/s) 1478.575 latency(ms/q) 0.6763\n" - ] - } - ], - "source": [ - "recall = compute_recall(indices, Y_tst)\n", - "throughput = indices.shape[0] / pred_time\n", - "latency = 1.0 / throughput * 1000.\n", - "print(f\"HNSW inference | R@10 {recall:.4f} Throughput(q/s) {throughput:8.3f} latency(ms/q) {latency:8.4f}\")" - ] - }, - { - "cell_type": "markdown", - "id": "75880f0d", - "metadata": { - "jp-MarkdownHeadingCollapsed": true, - "tags": [] - }, - "source": [ - "## Appendix: Recall vs Throughput Trade-off" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "12bd7fb6", - "metadata": {}, - "outputs": [], - "source": [ - "def run_pecos(X_trn, X_tst, Y_tst):\n", - " metric = \"ip\"\n", - " M_list = [16]\n", - " efC = 500\n", - " topk = 10\n", - " efS_list = [10, 20, 40, 80, 120, 200, 400, 600]\n", - " for M in M_list:\n", - " train_params = HNSW.TrainParams(\n", - " M=M,\n", - " efC=efC,\n", - " metric_type=metric,\n", - " threads=-1,\n", - " )\n", - " start_time = time.time()\n", - " model = HNSW.train(X_trn, train_params=train_params, pred_params=None)\n", - " print(\"Indexer | M {} efC {} metric {} | train time(s) {}\".format(\n", - " M, efC, metric, time.time() - start_time)\n", - " )\n", - " \n", - " for efS in efS_list:\n", - " pred_params = HNSW.PredParams(efS=efS, topk=topk)\n", - " searchers = model.searchers_create(num_searcher=1)\n", - " \n", - " start_time = time.time()\n", - " indices, distances = model.predict(X_tst, pred_params=pred_params, searchers=searchers, ret_csr=False)\n", - " pred_time = time.time() - start_time\n", - " \n", - " recall = compute_recall(indices, Y_tst)\n", - " throughput = indices.shape[0] / pred_time\n", - " latency = 1.0 / throughput * 1000.\n", - " print(\"inference | efS {:3d} R@10 {:.4f} Throughput(q/s) {:8.3f} latency(ms/q) {:8.4f}\".format(\n", - " efS, recall, throughput, latency)\n", - " )" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "0b4af0fb", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Indexer | M 16 efC 500 metric ip | train time(s) 47.05828666687012\n", - "inference | efS 10 R@10 0.7737 Throughput(q/s) 5228.054 latency(ms/q) 0.1913\n", - "inference | efS 20 R@10 0.8552 Throughput(q/s) 3665.046 latency(ms/q) 0.2728\n", - "inference | efS 40 R@10 0.9043 Throughput(q/s) 2406.416 latency(ms/q) 0.4156\n", - "inference | efS 80 R@10 0.9320 Throughput(q/s) 1504.624 latency(ms/q) 0.6646\n", - "inference | efS 120 R@10 0.9432 Throughput(q/s) 1122.713 latency(ms/q) 0.8907\n", - "inference | efS 200 R@10 0.9536 Throughput(q/s) 763.034 latency(ms/q) 1.3106\n", - "inference | efS 400 R@10 0.9622 Throughput(q/s) 433.572 latency(ms/q) 2.3064\n", - "inference | efS 600 R@10 0.9656 Throughput(q/s) 305.857 latency(ms/q) 3.2695\n" - ] - } - ], - "source": [ - "run_pecos(X_trn, X_tst, Y_tst)" - ] - }, - { - "cell_type": "markdown", - "id": "ccea74a4-ca0d-4839-b1fa-b42ce86aea82", - "metadata": { - "jp-MarkdownHeadingCollapsed": true, - "tags": [] - }, - "source": [ - "## Appendix: Plot Recall vs Throughput Curve " - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "cc70d86f-a4fc-46b6-a079-860db59efc96", - "metadata": {}, - "outputs": [], - "source": [ - "def get_pareto_frontier(Xs, Ys, maxX=True, maxY=True):\n", - " '''Pareto frontier selection process'''\n", - " sorted_list = sorted([[Xs[i], Ys[i]] for i in range(len(Xs))], reverse=maxY)\n", - " pareto_front = [sorted_list[0]]\n", - " for pair in sorted_list[1:]:\n", - " if maxY:\n", - " if pair[1] >= pareto_front[-1][1]:\n", - " pareto_front.append(pair)\n", - " else:\n", - " if pair[1] <= pareto_front[-1][1]:\n", - " pareto_front.append(pair)\n", - " return pareto_front\n", - "\n", - "def plot_one(\n", - " results_dict,\n", - " xlim, ylim, title,\n", - " FONTSIZE=28):\n", - " import matplotlib.pyplot as plt\n", - " f, axs = plt.subplots(1, 1, figsize=(10,10))\n", - " for algo_name in results_dict.keys():\n", - " algo_dict = results_dict[algo_name]\n", - " pareto_front = get_pareto_frontier(algo_dict[\"recall\"], algo_dict[\"throughput\"])\n", - " Xs_list, Ys_list = zip(*pareto_front)\n", - " axs.plot(\n", - " Xs_list,\n", - " Ys_list,\n", - " label=algo_name,\n", - " ms=7, mew=3, lw=3,\n", - " color=algo_dict[\"color\"],\n", - " linestyle=algo_dict[\"linestyle\"],\n", - " marker=algo_dict[\"marker\"],\n", - " )\n", - " axs.set_xlim([xlim, 1.01])\n", - " axs.set_ylim([0.0, ylim])\n", - " axs.tick_params(axis='both', which='major', labelsize=FONTSIZE-8)\n", - " axs.tick_params(axis='both', which='minor', labelsize=FONTSIZE-8)\n", - " axs.set_ylabel(\"Throughoput (#queries/sec)\", fontsize=FONTSIZE-4)\n", - " axs.set_xlabel(\"Recall10@10\", fontsize=FONTSIZE-4)\n", - " axs.set_title(title, fontsize=FONTSIZE)\n", - " axs.legend(fontsize=18)\n", - " #axs[i, j].set_legend(loc='upper center', bbox_to_anchor=(0.76, 1.01), ncol=1, fancybox=True, shadow=True, fontsize=FONTSIZE-8)\n", - " axs.grid(visible=True, which='major', color='black', linestyle='-')\n", - " axs.grid(visible=True, which='minor', color='gray', linestyle='--')\n", - " axs.minorticks_on()" - ] - }, - { - "cell_type": "markdown", - "id": "4c7927f8-626a-4ac6-a889-47c8bd023954", - "metadata": { - "jp-MarkdownHeadingCollapsed": true, - "tags": [] - }, - "source": [ - "### plot RCV1" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "be44342f-f53a-4266-a865-42d1e4f0cc0e", - "metadata": {}, - "outputs": [], - "source": [ - "def get_results_rcv1():\n", - " results_dict = {\n", - " \"PECOS-HNSW\": {\n", - " \"color\": \"blue\",\n", - " \"marker\": \"D\",\n", - " \"linestyle\": \"-\",\n", - " \"recall\": [0.7733, 0.8545, 0.9043, 0.9325, 0.9434, 0.9533, 0.9621, 0.9657, 0.9678],\n", - " \"throughput\": [5250.297, 3677.292, 2409.959, 1508.349, 1125.047, 763.752, 433.872, 305.747, 237.651],\n", - " },\n", - " \"HNSW(NMSLIB)\": {\n", - " \"color\": \"black\",\n", - " \"marker\": \"o\",\n", - " \"linestyle\": \"--\",\n", - " \"recall\": [0.7790, 0.8581, 0.9055, 0.9326, 0.9426, 0.9523, 0.9608, 0.9644, 0.9663],\n", - " \"throughput\": [2710.256, 1924.505, 1271.085, 800.999, 597.873, 404.518, 229.553, 161.879, 124.806],\n", - " }\n", - " }\n", - " return results_dict" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "d8466781-720d-4eeb-8762-314f08f3c656", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAApMAAAJ/CAYAAAAkkFgEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAEAAElEQVR4nOyde3gU1fnHP28IMSYkQLgoiKgRUQTFGkBBCSgK2oCgYMECEhS19dLW2tpfveIFrfd6oVZEiQIKCgoKKigKqKhcKiiCIgWK3CThGogQQs7vjzO72YTdzW4yJ9nJns/zzJPszJnvvvO+u5M3Z855jyilsFgsFovFYrFYqkJCbRtgsVgsFovFYvEuNpm0WCwWi8VisVQZm0xaLBaLxWKxWKqMTSYtFovFYrFYLFXGJpMWi8VisVgslipjk0mLxWKxWCwWS5WxyaTFUsOIyHwRUSIyv7ZtsVgsFi/j3EuViIyubVviGZtMWowjIj0DvvAVt19EZJOIvC8iN4pIg9q21+JNRGRDiM9YiYjsEJHFIvKEiJwWpe4pInKniHwiIv8TkSJn2yQiH4jIHSJyYoVz/h3w/r+O8v1ODjj3owrHGohItoj8RUTeEJH1AW03RPM+1cH5rgb6ODdM21BxCbf1DKJztIhcLiJjReQrEdkpIodEZJeILBGRf1SMQwh7uorI7SLypoh8KyJbReSgiBSKyPcikhfs/SvRrCciQ0VkhnO9Bxz7VovIRBEZKSL1otG0WDyFUspudjO6AT0BFeG2AehY2zYb9sd851rnhzju88Xo2rbVS5vz2YnkM3YI+EsEeg2Bl4CSCDRLgdeB451zuwYcmxLlddwXcO7wCsc+CffdqSE/HwfsqfDeuS7ExbcdBo6roHEmUBjBuQeAmyqxf2WEdrwBJEfgj9OB/0Sg16i2vyN1cbP3y9jYErFYapbngX8FvG4KnAr8GWgLnAC8LyKnKqUKa8E+i/fZAvQJeH0UcDJwBTAYSAQeE5ENSqlpwQScHq73gHbOrh3Aa8ACR78EaAGc7+ieDAwBvgL+qZT6QkTWoD/Tl4lIulJqb4T2D3N+7gPeqmhawO87gaVAN6Ame/THAunAdqB5BO17A0mVtPkV8Krz+0dKqc0VjqdTdo1fArOBxUA+kAH8GrgJHevnROSAUuqlEO91AJjn6HwPbEP7sjnQEfgdcCJwJfqfhCGhjBaRDsDHQDPgIJAHfABsQj/5ywQuAgaEvXqLxevUdjZrt7q/Ub5ncnSINknoG7yv3W21bbdBf8zH9kya8OsGKumhA64N8O+3IdqkAN8GtMsDGobRrAfkopOrPwXsvzNA49oIr+H8gHMmBDl+PXAV0Caa63bRxwOd9/oZuDXA1txq6j4ToPXbIMe7oXsKO4TR6Ar84mjsAhqEaJdYiS1HA18E2HNmiHZHAaudNj8B7cJoJgJiOj7xuNn7ZWxsdsykJSZQShUDowN2XVxLpljqMEr3Vv3XedlBRI4N0mwM0MH5PU8plauU2hNG87BSKg/IAr4JODQR/UcO4OoITQxs90qQ9xqnlHpdKbU2Qj3XEJFGwLPOy9vQCZsbuvXRCTLAXuDtim2UUouUUr9RSq0MpaOU+oKypx6NCHEPUUqVhLNHKfUL8HTAru4hmt4GnIaO8W+UUqvDaJYoJ/OxWOoiNpm0xBL/Cfj9+HANReQMZyD+KhHZ60yIWCsiL4nIWZWce5SI3CwiH4vIdmcQ/24RWSMiH4nI30WkXZDzIpqFLSK5ARMJTgzXtsJ5G0Qk8A/OvUEmJuS5cS3RICIZIjJaRJY6kx0OiMhPzgSGsJNLKvpMRFqKyOOOfb84eh+LyJXVsTFKvg74vdznTESaAjc4L7cBt0QqqpT6SSn1ccDrjegxjgDdReSEcOeLyFHoR6sA/0M/Uo8lHkU/2p+nlJrkou6v0cNdAN5wkrmq8knA722qoRM4xCa54kFnMs3vnZcfOYmsUUSkhTPxaZqI/Cgi+52JQ5tFZKaIDBaRkH/TpfxEyJ7OvoEi8qFz7zggIv8VkWdC/JNVUa+JiDwqIj843+WfHa3LneNh74MBx0ZX8j55Uo0JZiKSKSK3ici7zj32F2f7n4hMFZFLKjm/3HWISJKI/EFEFolIvoiUisg/q2JbXcKOmbTEEocDfj8UrIGICPAw8FeO/GfoZGcbKSL3KqUeCHL+scCHlPU8+WjobKcAvdBjp0KOlYoFauJaROQCYDrQuMKhVsAgYJCITAeGKaUOVKLVDZiBHl/mIxm4ALhARJ5QSv2lKnZGSbjP2RD0Y06A8UqpfdV8r1eAC9FjHYcDD4Zpexm6Rw1gYiz1ZIlINjAKPS7wRpflRwT8fkRvbJQEjs08HLJV5QR+X74Pcrwb+jsA8I5vp4gkOfuLgW2V9YJGipO8+sZhVqQl+rNzGXCtiFwRwec2QUReRX8mA8lE/wM1SER6KKV+DGHPGeh7zzEBu5PR40MvEpFx6KECtYqInETZk4iKtHa234jIJGBkBPFqgr4fnu2elXUDm0xaYonTA37fEKLN05T1Fn2Fnm27Fv147HT0IPxzgPtFZKdSamyF85+lLPl6HX1j2IxOKo5B3yT6UvZ4sibxTVT41nldcbISlH+0aPRaRORM4H302LDDwAvoCSF7nPe9zfk5ED1R4Tdh5FoAM53f7wQWose3dQbucY7fJiLvK6XmVcXeKAj3OesR8PtsF95rOnrCSgMqTyYDH3G/GrJVDeP0mI5DJ8QPK6XWuKjdBMhxXv5XKfVZNSUD4xfysXMQOxLQE3DaAzdTNmHme2BOkFPODfj9GxFpg/4n9zLKEtp9IvIBcL9S6tuKAlHim3j1Mfo7+S168lEaOgG8Dj1m9GL0521EEI1AHkAnxLOBCcB6dKKUC/wW/X18mSCP+EUPd/iAskTydfSQju3o3uA/oMf2dozyGk1QD53Yz0Env6vQk60y0JPjbkLHfBiwDri3Er2XgTOAycAU9GS845z3iW9qe9Cm3er+RgQTcJx2rxF+EP5FAcdvCaFRL0BnLwHlOND/ORc7x56oxOYmQfbNJ8zEmYB2uQF2nhitToS+qta1RBi3LykrezMghA0LA+y9LMy1KmAjTumcCm3aomfYKuDtanzONlD5BJzeAfbMDXL8R8rK0xzl0uc/L+A9zwnRphn6nwAFfO72dVfT/gcd/e+BpBCf89wqat8coHFPNe08zvnOK3RiE0lZn4KA96+4rSXEpBp0AuZrdz165n0onYPAkGpemxAw6SpEG19JqVLglCDHe1awa3QInZcD2nQMcvypgON/C3K8HvoJROB7nRikXUQTZwK+P0E/3+F0gFSgRSV+9cVyH0Em2lX4nCvghurEsq5udsykpVZxxt2cLyLvUTYI/zP0zM2K/J/z8z2l1LNBjqOUOoz+b/Mg+r/2wLF4GUB95/ew49GUUjsiu4Jaw+i1iEhndA8v6EeuM4LoHkD3gPgeDVU2vvAWpdRPQXTWoP/4AGRHa2tlOONK24nI3QHvsw+4I0jzJs7PPUqpgy6ZEPjoNtREnKsoe1JU3Ue9riG69M3tzsvfKz1Rzk18/lBUozfWGf4yDv2dB90bGHbYRRgOoe81Z6nQk2oyAn5/Gp20jEVPyDkK/fj0bkcrCXhFRH5VRXtQmsomXd2PTo4F3UMajq/RyWcwHg34PbCn19dLneu8/KZCW5+th9Hjjqvqf9dQSu1XSm0Nc1yhn7AcRsfwokok5yulXnDRxDqDTSYtNU25SSXom9+nwKXonraXgEtVhbErIpKO/s8a4M1wb6CU2oUuTAz60Y+PHc57AAwXES8P8zB9LYEzYceHaqSUWg/4Vmnp7vyxCcYe4N0w77fU+ZnhPEarDidU+IwdQD/euh89HnIJcLFSammQc33JyP5q2hDIfPSEGoDBomcvV8Q3du0Awf+RqnGcR78vov9peVUp9YnL+u3QwxwAFiilNlRD7j70RB7Qn8eKw1tCcT76sWVH9NjWe9D3pPvR9SpD1e9MDfg9GRijlLpZKfWDUqpY6clYD6JLUYFOKMMNcYgKEUkQPZntVBHp4CT97dDjKqHyR8yTnUTqCJRS36P/2QL9CD2QTpSN6w2n8TPBhwfUKiJSX0RaOf9c+vzWEn0/hcr95ubEszqFl/+YWuoeP6ALPgcbPP4rysalTBCRCRFq+mclKqUOisjr6N60QUAXEXkT/cf+cycJ9QQ1cC2+sZil6OQrHF8Cl6B7ZNpSNuYzkDVKqdIwGjsDfk8DdkdmZtQcAF5SSn0Z4ngherJRaojjUaOUUiIyEbgL3fOZQ1kPqS+p6uS8nKmU2u3We1dERI7jyMlUPvY7/xz4uAk9NnAnYGJi1IiA36vcGysi16F7AUE/mh4aKsmpiJM4BfKJiDyHToRGAL8SkfOC3JMCe912oscgBtOfKCJ/Qo9fvkREGiqnzJSINCd00fdiVWFsqtP7OhSdoJ5D2USxYDQNcwwqH0+6Cz3ON63C/sDJfssq0VgK9K+kjXGcf96uR//D9ivCF9CvzG8r3LKrrmF7Ji01zfPonoAz0F/sHPTEjkPOvgUicmqQ8yJZaSMYKRVe30JZHbvW6Ecc7wI7RGS5iNwtIs3wBiavxfcYb28Ejwu3BTmvIkWVaAQmmtUdzL6Fss/YGege7VvRvYPJwL9F5K8hzvX1UDQM08taFQIf4VZ81B22tqTLjEEn+8E2/z9oInI88JDz8nalVL6bRji9nr6VfvYDQVciikBnCPBv5+Um4CKl1Pbq2Ob8I+ZLdM8k+HCIwNJBH1cyJOID52cCuhapjxsJHYu5gQIikoyeLDMR/XkOl0gSwfFIv48Vv4uB/4hU9plw9TNTFUQkAz2r/Dl0Al7ZSkyV+c0zHQ41je2ZtNQ021X5wsPLgfdE5F10iY0M4DUR6eKMvfEReFP7A+XryYWj3ONKpZdovEJEstCzj3uiew0S0Y84OgJ/EZGhSqlZEV9VLVBD1xJRD0+McUgdWdx6gYi8AixCj2t7SETmK6Uq9rouR89ITUD/sxOqBzMqlFI/isgi9AzaHBHJUErtDOhtAp2Uzw0pUrPcjO6Z2g4UOUlbRc4J/F1EfP90fKqOXA6xIr3QE2YApod4GhEWEbkMnaQnOHZepJT6X/izIkMptVpEfkSX1xrEkQnlxoDfjxgHHKZtVf+5uxM9FAj0GOmx6Lq824BffL3+IrIQPQNbgonEIU9TlsDPQE8u+gb9eTng68EWkY3omrOV+a065abqNDaZtMQESqnZIvJv9H/rZ6MHeQeurVsQ8PsvQZKFaN9vGc5jGhFJRd+Ah6LLYqQDU0XkZKVUYK+b77/1ynr0XXtEGglVvJbK8D12bigiyZX0TgYWON4ZslUto5TaJSIj0AliIvAER074WYBOHkCXVXIlmXR4BZ1MJqHXCH8eXWPTVzh9coV/oFxHKZVL2QSKcPh6ZZujqyNUxu+cDeBydImqcFTrEbeIXIweW1of3Vt0sVLqh2h1KiEfnUyeEOTYdwG/V9aTHnjcH1+l1GjKr/oVFOcfjlHOy0+BC8MMGQn1ZMAtAnvmKkuMKzuu0MmbkfupM85+sPNyslJqWJjmoYZ+WCLEPua2xBL3UdaTeK9TANjHCsp6yc53802dGX8fKKWGo8e1gX483rdCU9+jrcpuPKe5aV80RHEtleFL1hMoG88XCl8P1UHAtRqEJlBKLUbXfgQ9Yaji6hdT0PUvQReADjUBoyq8QdlYu6sr/IQYmsVtEhFJQyecoHvtoprYIyLd0b1MR6G/k5copb4Je1LV8PWcBus1XRjw+8mV6AQeryzJDkYGZf+wvRkqkXQ+q8GGCLlJYBKdFbKVprL7hun76SmUVbyYGqqRiJyG7oW3VAObTFpiBmesk6/swvEE9F44Y7YWOS+vdCYSmCCwYHbFwdjrnJ9tnf96j8AZZzewmjb4Eo7qjtkLdy2V8WHA79eEaiR6mTTfzO9PXSynY5IHKPvH5O7AA0qpAso+g8dSthZ1pTizRC8MddyZWOMr3H6uiHSk7LPytap+YWvXUEr9SSkl4TZgZMApIwOOzahE/krKxjJHtdKPiHRBjx1MQY/7y3H+QXAVpzSWr0fyiLg4M8991QAuEJGgCZHolWsGOC+LqHzSSjACnyCG66UbhfmnjUvRlRkAhjq9pkcgIscAfSrR8t1PO4dqIHrhhIorfEVKpH77XZhjlgixyaQl1niMsmTq7xVK3vhmTKYAb4leRzkoIlJPRIaJSKuAfZki0iPUOQ69A35fX+GYr55jEvCnIO8p6IHeLSp5j8rw1UUL2ePhwrWExRlL6PsjPUKCrMHtJM4TKLtpR5x41SZOL5ZvCbxuQRLAOynrmc0VkQmh/nkAf5mW4ehE4cxK3j6w93ESZT0icdEr6VCl3lgnsfgAPcP4ILqQ/qfRvLGIdHfKwYRrc1wFu0LVv3zY+ZkCjA2RWN1NWXmdl6tYpzOfsuoGVwWbGOYkv0FnlLuJM9zF548zKatBGmhLAvofsiPWNK+A7356jujlOivqNCJgUlgVWEvZP40jgsVHRPqhxwdbqokdM2mJKZRS20TkJXRZkpPQY/9ecY7NEZEn0LOWuwCrReQF9E0pH31TPwk9Lu0KdM/SGZTVXmuNLv3xPfox2RLnWCm61tgAysaT/QRUnLQyG52UnQSMFr0U3DR0j8OpwO/Rj+A/B86rhhsWOe9xmYjc4Oj5Euy9Tg9uda8lEq5DJ5RHATNF5F/Oe+1FL0H2F7R/QT9+eyeYSIzyIGVlS+5GL1MHgFKqyPkj8x66dl8u0E9EJqNLL21Fj307lrLPWtsI33cuetLEsZT1uBwisnGJiF62r+IwD19C2kBEcisc+yyCYtc1htOT7UscFqkQaz8HOe9kdG+5rwfwPmBrJYnhriATgbKAJ52JKu+hh8/4xmO3Qo9hHYkeawy6ZmVeMHGl1Fsi8g66QPhVwLFOWaH16PiOoGzM3v+IYHxkiPcpdT57N6ETuM9E5En0ak0N0fU1b0Q/jt9C5J/FqjIa3bt8LPAPp4f9VcqWU/wj+nuxGH2fhuAT+V5A210feEdEHkAPH6jnnHcresnG/1CFtbCVUjtEL4aRgy5dNldEnkfHojn6qUAuuoe0EVWfHGUBu5yi3cxvRLicYkD749E9DwpdezKhwvG/BxwPtx0kYAkyjlxOLNS2kSDLiDkaXdFjfUKd+zDVX07xLMqWF6y45bl1LRHG7kL0pJpw7zGNEEvXVXatAe3C+ixCWzcQZtm1IO3fD3jP84Mcb4ieBFYSgZ8Po/+gtozgfR+rcO7MKK4xNwJbArdcg9/r3GjfB10U3HfO9Qav2/9dqaDzpyjOnwCkVGJXCvoftXA63xNkecMofd0QvWpNqPfYgU7SQ37fKH/P6Bnhd+kIHzrHO6KTx3C+uybg9TEhdG4Jo7EfPbY2jzDf64D2o4McOx6dPIZ6j/8Bp4e7Xly4N8XDZh9zW2IOpZfc8z1makvZf/e+4w+jB1c/hP7vdwf6D/4+9ASQaehxMMep8r0yn6KXB3sI3RP1I7qXrQTds/kx+r/hdkqpoMVplVJfoP9LzkP3BB4Cfkb/QemjlPp71a/c/x7L0Unr6+hkMNg4xGpfS4S2fIz29f3ox7h70CvvbEZPZMlRSg1SVV+6rjYJfCx4d8WDSqk9Sqlr0b2Td6P/UP+EnqBzAO2DD9D/3JyolLpaKbUlgvet+Gi3yssIehDfI+7aWulnPHq2/lh0j/96dNJSjP7eLEIn+2cqpUYqpcLWY1RKFSml+qJ76mahewYPoe9J89GPUDuqCHtgw7zPHvTTjrvRYzgPoO93q4HHnfdYGFrBXZx7yunoigg/ou9RBejJVL9VSgX27kLZOMuKOs+ilzB8j7JVvTai/4nLUkq9XU07f0Lfrx9D/2046NiyAt27fZZSalV13sOiESfzjllEpBf6C9kV/YhjB/rL9LRS6r0KbbuhZ7Ceiy4++iO6rtSzKkTJDRHpi35c51th5TvgX0qpkGN5nPIiN6G/TIfR/zE+rmK8LqHFYrFYLDWBiIxHr9azWSnVqrL2Fm8T0z2TIvIoesxKJ/SA+SfQ49aaUbZOs69tf/R4i2z0qiDPoSdKPIUu9xFM/2b0iiEd0IPhX0SPN8sTkcdDnPM4uleqhdN+Enrc2LuOnsVisVgscYuIHE3ZmGQ3a7VaYpSY7ZkUvd7qOPQjoetVhVlwIlJfKXXI+T0dPXOrIXCeUmqpsz8Z/bivK3CVUmpKwPknosey7Ed3p29w9jdGT2Y4GejmPNb0ndMN/Wjkv0Bn5ax/7GgtQ5cfOM2nZbFYLBZLXcOZELVOBUkgnHJIL1FW2q2/8tbkPEsViMmeSaf0wRj02IkjEkkAXyLpMAjdWznFl0g6bQ5QVrj59xUkrkHPUn0uMPlzEkTfmrQV60/5Xo/xJZLOORvQY3COonztNYvFYrFY6hp/R1fTuFdELhGRs0Skq4j8Dj2O3ZdIfoJ++mep48RqaaCL0cnhP4FSEclBP4o+ACwO7C108NWJ+yCI1kJ06ZZuInKUKiuqHO6c9yu0ieR93kcPjr4QuDfIcYvFYrFY6gqnEr7c0ZfAb4L1XlrqHrGaTPoq4h9AT24pV0vMqRE2SOlVUaBsCakjlnJTSpWIyHp0XbxM9Oy3ys7ZKiL7gVYikqJ03blU9PJa+5RSWyueg57sA2FqfInI9cD1zsuslJSUUE2rRGlpKQkJ7nY2m9A0pWvK1gMHDpCcXFn93eiwfjWjayJWYP1qYxXfsYLy8UpOTubw4cMcPnw4sIQOIkJCQgL16tWjXr1654pIfmpq6MVnrF+99d0qKioqUEoFr8dZ27WJgm3A8+iaTiXAN+givQ3QE13mUKGOFjohVATUFKyg97lzvGvAvmJnX2KIczY7x1s4r1s6rzeFaF/fOX4wkmtMSUlRbvP22297QtOUrilbs7KyXNe0fjWjayJWSlm/2li5r2lK194HveVXL323gKUqRE4TkxNwnFVNrkfXhCo3oUVEUtCFrFvhTJARkTXoWninqCCrPYjI5+iK/P4JNSJSjE4A6yulSoKcsxmdQLZUuqeyJTrBDFrmQETqoxPUYqVUpWsqp6amqv3791fWzBIDdOrUiaVLl1be0FLr2Fh5Bxsrb2Hj5R1MxUpElimlOgU7FpMTcChbh/RrVWFmtNJFZOc4L31LNfkKojYMoefbvztgX6Tn7KnwM5r3qFHy8vI8oWlK15StJrB+tfHykl9trNzXNKUb77EypeslW2uDWE0mf3B+7g5x3DeT+ugK7Y8Yrygiieh1jkvQa3BWfI9g57RAl/nZ5CSvKKX2o3smGzjHK3KK8/OIMZg1RXHxEZPeY1LTlK4pW01g/Wrj5SW/2ljZWJnA+tVb8QpHrCaT89DjD08XkWA2+ibkrHd+fuz8vCRI22z0+qmLVNlM7srOubRCm+qcY7FYLBaLxVJnickxkwAiMhO4DPizUuqpgP290aV59qDXw93jFC3/L3ot0EiLlp+EntldK0XLTYyZjPcZbKZsNTH+xPrVjK6psULx7lcbq/iOFdj7oJdstWMmy3MT8BPwpIh8JCKPicg09ILwh4FRSqk9AEqpvcB16LW154vIeGcpxuXoRHIaMDVQXCm1HvgrkAEsFZGxIvIUevb4ycATqkI9S6XUIuBJ5/g3IvKUiIwFljo6f4kkkTTF3LlzPaFpSteUrSawfrXx8pJfbaxsrExg/eqteIUjVutMopTaJCJZwD3oHspsYC+6mv7DSqnFFdrPEJEewJ3AQCAZvcTin4FnVJAuWKXUsyKyAfgLcDU6uV4F3KWUeiWEXbeJyLfoZPd6oBT4D/CYUmpWtS+8GmzcuNETmqZ0TdlqAutXGy8v+dXGysbKBNav3opXOGI2mQRQuij5Lc4WSfvPgV9H+R7vEuVyT0qpPCAvmnMsFovFUvPs2bOHgoKCak10OOuss1i9enXlDWNA15Stjz76qGds9ZJfTehGGqt69eqRlpZGRkYGRx1VaUXDsMR0MmmJjpycHE9omtI1ZasJrF9tvLzkV6/G6sCBA/z888+0atWKo48+GhGpkmZxcTFJSUlumWhU15StSinatWvnqqb1qxndSGKllOLQoUPs3buXjRs30rp162ollLE8ZtISJfn5+ZU3igFNU7qmbDWB9auNl5f86tVY5efn06xZM1JSUqqcSAKUlByxroUrmNA1ZasJrF9rL14iQlJSEk2bNqVx48bs3LmzWno2maxDLF68uPJGMaBpSteUrSawfrXx8pJfvRqrAwcO0KBBg2prmlqtzISul1ZWs36NjXilp6dTWFhYLQ2bTFosFoulTlJSUkJioh3NZbGEo379+hw+fLhaGjaZrENkZWV5QtOUrilbTWD9auPlJb96OVbVebztIyUlpdoaNaVrylYTWL/GRrzc+I7YZLIO0apVK09omtI1ZasJrF9tvLzk13iPlYmJF6Z0TdlqAutXb8UrHDaZrEPMnDnTE5qmdE3ZagLrVxsvL/k13mO1e/du1zVN6Zqy1QTWr96KVzhsMmmxWCwWSxUoLYVXXtE/LZZ4xo5MrmFEpB/QLzU1lXHjxgHQpUsXmjVrxuzZswFo3bo1vXv3Zvz48YDuBs/NzWXGjBls374dgIEDB7J27VpWrFgBQLdu3UhLS/NrZmZmkp2dTV5eHqDHZQwbNoxp06b5SwAMHjyYlStX8t133wHQo0cPEhMTmTdvHgBt27alWbNmfs309HSGDBnClClT2Lt3LwBDhw5lyZIlrFmzBoBevXpRUlLCggULAGjfvj0dOnRg6lS9mmVGRgaDBg1i//79ft3c3FwWLlzIunXrAOjTpw+FhYUsWrQIgI4dO9KmTRumT58OQPPmzRkwYAB5eXn+QsSjRo3i8OHDfs2cnBzy8/P9MzuzsrJo1aqVv9eiZcuW9O3bl/Hjx/vXRh01ahSzZs1iy5YtAPTv359NmzbRvXt3xo0b52qc9uzZw7hx41yLU+fOnZk8eTJ79uxhypQprsVp0qRJflvditPcuXPZuHEje/bsYfPmza7FadmyZXTv3p3ly5e7Fqc5c+YAuiZbcXGxa3ECKCoqAnAtTj69Y445ho8++si1OAE0bNiQ5cuXuxYn0I+jN2/e7GqcMjMzOeaYY/z3gV/96lcA7Ny50z/BoHHjxhw4cIBffvkFgLS0NAD/bNbk5GRSUlL8ca1Xrx7169cvp5GRkcG+fUX8/veJvP760Xz8cQnPP1/C/v1a4+ijjyY5OZldu3b5NTIyMtixYwelTubZpEkToKyUUXp6OqWlpezbty+oRmJiIo0bN6agoADfgm5NmzZl7969FBcX8/nnn3P55ZcTSGpqKm3btmXgwIFce+21JCcnM2PGDEaOHEkoTj75ZFasWOH/PKWmpgLw3nvv8fvf/57vvvuOvXv30qBBA04//XRycnL44x//yMGDB/1lbho3bszSpUt56qmnWLRoEdu3byc1NZXTTz+d3/zmNwwfPpyGDRty+PBh8vPzSUhIoEmTJsyaNYvnnnuOVatW8fPPP5Oens5xxx1H586d+fOf/8xJJ50UNk4ZGRlcddVVTJkyhdWrV3PKKadQVFTEgQMH/LGeOHEi119/PS+99BK/+c1vSE5OZsWKFXTqpJecvuOOO/jzn/9cLk779u3j7LPPpkGDBixfvrxcnD755BPGjRvHqlWr2LFjB40bNyYzM5OsrCxuuukmmjRpwtKlS/n1r3/Nrbfeyt///ncaNmxISUkJ+/fvZ/HixfTt25ekpCTWrFlDSkoK9evXp1GjRuTn5zN48GDmz5/P9u3bSUxM5NChQwA0atSI4uLicnFKTExkz549/r8x6enpFBQUAHpcZNOmTdm1a1e5OPm+C4WFhaxcufKI71Pg36ewKKXsVgtbSkqKsniDrKys2jbBEiE2Vt6hJmK1atUqI7qHDys1cqRSULaNHKn31xaffPKJAtRVV12lJk6cqF599VX1j3/8Q5122mkKUNddd51SSqkJEyYoQP3hD39QEydOPGKbMWNGOd3t27erc889VwHqnHPOUQ8++KB66aWX1BNPPKH69++v6tWrp6688spy5/zrX/9S9erVU02bNlV/+ctf1Pjx49Vjjz3m17n44ovVvn37jjgHUJmZmeqee+5RL774onrooYfU8OHDVdOmTdWbb74ZkR9GjBihAJWfnx/0uO/6A/XWr1+vAAWolJQUtWXLliPOO+GEE1T79u3L7bv99tsVoM4880w1ZswYNX78eHX//ferK6+8UqWlpaklS5YopZQqLCxUiYmJ6rzzzjtC96GHHlKJiYkKUB9++GG5Y4cOHVKpqanqjDPOiOjafXz33XdRtVcqsu8KsFSFyGlqPamK181EMvniiy96QtOUrilbTfzRs341o2sqQYl3v3o1Vm4lk9u3b/f/HiyRrGpCGahbXXzJ5L333ltu/549e1TLli2ViKht27YFTaZCUVpaqrKzsxWg7rjjjqBt1qxZo8aMGeN/PXfuXCUi6swzzwya0N15550KUMOHD/df/6FDh1SjRo1U69at1Z49e4445+DBg2rHjh2V2qtU9ZLJTp06KUBdf/31R5x3/PHHl0smf/75Z5WQkKA6d+6siouLj2hfWFioCgsL/a+7du2qkpKS1P79+8u169mzp7r00kvVscceq+68885yxxYtWqQAdcstt0R07T5qI5m0YybrEKUGBu6Y0DSla8pWE1i/2nh5ya/xHqsybRg1CiZMCH58wgR9PJbclZ6eTteuXVFK+Yc+RMqsWbNYuHAhgwcPZujQoUHbnHLKKdxxxx3+1//3f/8HwGuvvUbTpk2PaP/AAw9wzjnnMHHiRFatWgVAQUEBu3fvpnPnzqSnpx9xTlJSEhkZGVHZXhXOOeccLr/8cl5++WV++OGHsG3XrVtHaWkp2dnZ1K9f/4jjDRo0KFcw/4ILLvAPR/BRUlLC4sWL6dGjBz169OCTTz4ppzF//nz/ubGOTSbrEAkJ7ofThKYpXVO2msD61cbLS36N91hB5Ymkj1hLKJVSrF27FqBccldYWEhBQcERW+CKLNOmTQPg+uuvj+i91q9fz3/+8x+6detG+/btg7YREa699loA/3jZY445hgYNGrBw4cJKk7hI2blzZ9Dr8411DMXDDz+MUoq///3vYdtlZmYClBsXHA5fQuhLEAGWLFlCUVGRP5lcsmRJOf/Pnz8fEaFHjx6V6tc6obos7ea9x9wWM9hxeN7Bxso71OaYyWCPqGNlqyq+x9z33Xefys/PV9u3b1crVqxQo0aNUoA699xzlVJlj3lDbTfddJNf8+yzz1aA2rFjR0SPTt95552IHssuW7ZMAWrgwIH+fY8//rgCVL169VTnzp3VH/7wBzVp0iS1devWqPzge8xd2RbsMbfv2q+77joFqC+++MLfJtiYyZtvvlkBKikpSXXv3l399a9/VW+++abauXPnEXYVFRWppKQk1bVrV/++hx56SDVo0EAdOnRIrVq1SgFqzpw5Sqmy8ZIdO3aM6vqVso+5LdVk1qxZntA0pWvKVhNYv9p4ecmv8R4rL3HvvffSrFkzmjdvTseOHXn55Ze57LLLmDFjRrl299xzDx9++OER2y233OJv46syEOzRczB87Rs2bBi2nU9vx44d/n233XYb77zzDr1792bVqlU888wzDBs2jFatWnHttdf6Zy1HyvTp04Ne31//+tdKzx09ejQpKSncfvvt/n3Bhk8888wzvPrqq3Tr1o3Fixfz2GOPceWVV9KiRQv+9re/lVui8Oijj+acc85h6dKl/t7H+fPn06VLFxITE2nXrh3Nmzf391z6eim98IgbbGmgOkUkXe2xoGlK15StJrB+tfHykl/jPVZeYvjw4Vx99dWIiL80ULDxhmeccQYXXXRRWC1f0ucrx1MZvvZ79uwJ286XdAaOKQTo168f/fr14/Dhw6xatYp58+bx9NNP8/LLL5OYmMgLL7wAwLZt28qdF2xMZXZ2dtAxm5s2bar0Olq2bMmf/vQnHnroId5991369esXtJ2IMHz4cIYPH05xcTHffPMNc+fO5Z///CePPvoojRo1Kve4/IILLuDTTz/ls88+o1evXnz++ef88Y9/LGezb9ykL6ns2bNnpfbGArZn0mKxWCxxRbQPn7dvz+fwYQhTnrEcI0fC4cOR6VbcV10yMzO56KKL6NWrF+eee261Jq506NABgK+//jqq9v/5z3/CtvMdb9euXdDj9erV44wzzuBPf/oTS5YsoWHDhrzyyiv+nr4WLVqU26644oqI7IuGv/3tbzRp0oQ77rgjokldSUlJdOrUiTvuuINPP/0UEeGll14q1yZw3KSv57Fr167+4z169GDp0qXs27eP+fPnk5CQQHZ2trsXZgibTNYh+vfv7wlNU7qmbDWB9auNl5f8Gu+xatSoEQkJMH585QnlyJG6XSTzgBo1auSKfYEcffTRrmkNHDgQwF9IvjJOOukkzjrrLBYtWsTq1auDtlFK+ZOswYMHV6rZtGlTTj75ZA4ePOgvwF3x0fUTTzwRkX3RkJ6ezl133cXKlSt55ZVXoprYdeqpp9K4cWM2b95cbn/Xrl1JTk7mk08+Yf78+Rx99NHlHmP36NGDkpIS5s+fz+eff85ZZ51F48aNXbsmk9hksg4RSfd9LGia0jVlqwmsX228vOTXeI+Vb2WgyhLKaBLJQF03CRynV1369etHdnY2r7/+Oq+//nrQNmvXruXhhx/2v/b9PnTo0HJjIn2MHj2aL7/8kuHDh3PaaacBejUo3ypPFfnxxx9ZtWoVTZs2pVmzZgBcdNFF5basrKxqXWcobrzxRk488UTuvfde/yo6PrZt28by5cuDnvfpp5+yc+dOTj/99HL7jzrqKLp27cqyZcuYNWtWuV5J0D27TZo04bHHHmP//v2eecQNdsxknWLZsmWuf6lMaJrSNWWrCaxfbby85Nd4j1VRUZF/WUFfQgnlywRFm0hW1HWLSBPUTz/99IgEycfQoUMREUSEadOm0a9fPx588EE++ugjLrvsMo499lh2797NZ599xjvvvOPvwQS45JJLeOaZZ/jjH/9Iu3btGDlyJKeeeiq7du3irbfeYtGiRVx88cU8//zz/usvKiqiZ8+edOjQgUsuuYRTTjkFpRTff/89r776KgcOHGDs2LE1XqIqKSmJBx54gOHDhwPlSytt2rSJzp07c84559CrVy8yMzM5ePAgK1asYPLkydSvX5+HHnroCM0LLriATz75hEWLFnHfffeV+wyICN27d/dPlvLK5BuwyaTFYrFYLFFRMaGsSiJZ2zzzzDMhjw0ZMoTERJ0eNGvWjE8//ZR//OMfLFiwgCeeeMK/7nPHjh15+umnj1jr++abb+a8887jySef5LXXXuPnn38mNTXVP7v86quvpl69ev4Z2o0aNeLll19m7ty5vPPOO2zdupUDBw7QrFkzevTowS233FJridXQoUN54oknjuiFPO200xg7diwffvghU6ZM4eeff+bQoUO0aNGCAQMGcNttt/nXhg+k4mPtivTo0YMZM2ZQr149unfv7vr1GCNUzSC7ea/O5Ndff+0JTVO6pmw1UQ/P+tWMrqnahfHuV6/Gyq3lFCsugefj8GGl8vKqviZ3KN3qYEJTqarVLqwMU7Z6ya8mdGujzqTtmaxD+MaTxLqmKV1TtprA+tXGy0t+jfdY+XrpKpKQACNGuK9bHUxomsKUrV7yq5fiFQ4PdcpbKsO3NFWsa5rSNWWrCaxfbby85Nd4j1VldRNjSdeUrSawfvVWvMJhk0mLxWKxWCwWS5WxyWQdonXr1p7QNKVrylYTWL/aeHnJr/Eeq6SkJNc1TemastUE1q/eilc46sbDeg8hIv2AfqmpqYwbNw6ALl260KxZM//jmdatW9O7d29/odikpCRyc3OZMWMG27dvB3Qx2bVr17JixQoAunXrxqmnnurXzMzMJDs7m7y8PABSUlIYNmwY06ZNY+fOnYAuGLty5Uq+++47QM8iS0xMZN68eQC0bduW8847z6+Znp7OkCFDmDJlin85rKFDh7JkyRLWrFkDQK9evSgpKfHXDGvfvj0dOnRg6tSpAGRkZDBo0CAKCgr8urm5uSxcuJB169YB0KdPHwoLC1m0aBEAHTt2pE2bNkyfPh2A5s2bM2DAAPLy8vxlMEaNGgXg18zJySE/P5/FixcDkJWVRatWrZg5cyagl8vq27cv48ePp7S0lISEBEaNGsWsWbP8y7H179+fTZs20b17d8aNG+danNLS0ti4cSPjxo1zLU6dO3dm8uTJAEyZMsW1OE2aNImioiLGjRvnWpzmzp3Lxo0bAdi8ebNrcVq2bBndu3dn+fLlrsVpzpw5gC7GXFxc7Gqc0tLS/PFyK04AV199NR999JGrcbr00ktZvny5a3ECaNWqFZs3b3Y1TpmZmfTs2dN/H/DNpt25c6e//mLjxo05cOAAv/zyS7k4+JYMTE5OJiUlxR/XevXq0bhx43IaGRkZFBUV+UvrVNQ4+uijSU5OZteuXX6NjIwMduzY4V9NpUmTJogI+fn5gL6/lpaWsm/fvqAaiYmJNG7cmIKCApSzVE7Tpk3Zu3evP24NGzakfv36fs2UlBSSkpLYvXs3APXr16dRo0b+46DHmO7evZtDhw4BemZ1cXGx//OUmppKYmIiaWlp5Ofnk5SURHp6ur+AuIjQtGlTdu3aRUlJSVAfN2jQgISEBP/n/KijjqJBgwYUFxeTn59PQkICTZo0qXacMjIy/Br5+fmuxWnfvn1+W92KU0lJiX+N7v3797sWpz179pCWluafcR9pnAoLC1m5cuUR36fAv09hCTUzx27em839wgsveELTlK4pW03MOrV+NaNraoZwvPvVq7Fyazb39u3bXdGpCV1TtpqYzW39aka3NmZz28fcFovFYrFYLJYqY5PJOkS8jxPx0tgT61cbLy/5Nd5jJSKua5rSNWWrCaxfvRWvcIhynulbapbU1FTlGy9hiW06derE0qVLa9sMSwTYWHmHmojV6tWradeundH3iBdWrVp1xFrTltikKrGK5LsiIsuUUp2CHbM9k3UI33qesa5pSteUrSawfrXx8pJf4z1WvskVXtA1ZasJrF+9Fa9w2GSyDuGb8RjrmqZ0TdlqAutXGy8v+TXeY+Wb/eoFXVO2msD61VvxCodNJi0Wi8VisVgsVcYmk3WIgQMHekLTlK4pW01g/Wrj5SW/xnusGjdu7LqmKV1TtpogWlu3b99Ow4YNefHFF13VjQSvfAZmzpxJx44d+fHHH13VrQybTNYh1q5d6wlNU7qmbDWB9auNl5f8Gu+x8hW9jmXd+fPnIyI88sgjIduICH379vW/7tmzJyJCZmamv6B2IKNHj0ZEjpgotW7dOq6//npOO+00UlJSaNy4Me3atWPEiBF88skn/nYtWrTg+OOPD2rLaaedRv369f1F6gOZMmUKIsITTzxRbv9dd91Fs2bNGDlypH/fhg0bEBFEhDvvvBM40q8nnngiHTp0KLcvNzfXf16oiWBPPfWUv01FO7dt28Zf/vIXOnToQFpaGunp6ZxyyikMGTKEt956q1zbnj170qBBg6DvEWhrsHa+GARu6enpnH766dx1113+ou0++vfvT9u2bfnb3/4W9P1MYZPJOoRvVYhY1zSla8pWE1i/2nh5ya/xHivfKiyB7Nixg/Hjx/PAAw8wfvx4duzY4YpudfGtkBIN69ev5/nnn4+o7dKlSznjjDN444036N27N0899RT33nsvF154IYsWLfKvrAQ6Odq0adMRCf7WrVv54YcfSExMZP78+Ue8hy8hveCCC/z7Nm3axMsvv8wtt9xCYmLwxfv++c9/snXr1qj8mpyczIQJE4IemzBhAsnJyUB5v/7vf/+jY8eOjB07lnPPPZd//OMfPPzww/Tt25fvv/8+pF4wIrX1/vvvZ+LEiUycOJGHHnqIdu3aMWbMGHr16uVfrcfHsGHDePvtt/2rcdUEdjlFi8VisVgiRCnFmDFjGDNmTLlepVtuuYU777yTO++801O1A48++mhOOukkHnzwQa655hr/koOhuO+++ygqKmL58uV07NjxiOPbtm3z/37BBRcwZcoU5s+fT5s2bfz7fQnklVdeGTSZnD9/Po0aNeKss87y73vhhRcQEa666qqgdvlKTY0ePZoHH3ww7DUEcvnll/P666/z5JNPctRRR/n3L1myhG+//Zbf/va3vPbaa+XOefzxx9m+fTszZsygf//+R2gG+sAtLr30Ujp1KqvKc/PNN3PFFVfw9ttvs2LFCv/SoQAXXXQRDz74IP/+97959tlnXbclGLZnsg7RrVs3T2ia0jVlqwmsX228vOTXeI9V4KPHMWPGcPfddx/xKPXAgQPcfffdjBkzpkq6bhGYEEVCQkICDz/8MAUFBTz66KOVtv/xxx9p0qRJ0EQS4Nhjj/X/7utZDHz0DTpZPO200xg8eDCbN28uN75v69atrFmzhuzsbBISylKUN998k06dOtG8efOg73vOOedw+eWX8/LLL/vXg4+EkSNHsmvXriNKSk2YMIFmzZr5hwUE+tVnb69evYJqBvqgMqrzGWjZsiVwZKH+1NRUunfvzrRp06qsHS02maxDVPYfZaxomtI1ZasJrF9tvLzk13iPlS+p2bFjR6XJ4pgxYyJ+5B2YLLnFL7/8QkFBQdAtFJdddhnnn38+Tz31VKW9aieffDI7duw4YlxgME455RSOO+64I3of58+fT48ePTjvvPOOeNTt+z3wEffPP//MDz/8QJcuXcK+38MPP4xSinvvvbdS23z86le/4qyzzuLll1/27ztw4ACvv/46w4YNo379+kD5lWpOPvlkAF588UWqu/BLpJ+BPXv2+OP43//+l5dffpkJEyZw/vnnBy1Q3rVrV7Zt28b3339fLfsixSaTdYg5c+Z4QtOUrilbTWD9auPlJb/WxVgFm9gQarvuuusAePvttyudNHPgwAGaNm1a7vzRo0cf0a5fv34kJyeHbVMVHnjgAZo1axZ0C8cjjzzC/v37K7Xjrrvuon79+gwcOJC2bdtyzTXX8Pzzz7N69eqg7S+44AK2bNnCmjVrgLKex549e6KU4uyzzy7Xc+lLJnv27Onft2rVKqAsiQvFqaeeyjXXXMPMmTP58ssvw7YN5JprruGjjz5i06ZNALz11lvs3r2ba665xt8mMO633XYb6enp/PnPf+aEE05g6NCh/POf/2TZsmURv6ePvXv3RtTuoosu8sexTZs2XHvttVx88cW8//77QYdV+HxVU+MmbTJpsVgsFksEbN26tbZNqJThw4fz4YcfBt3C0a1bNwYMGMBLL73kT/yC0bVrV5YtW8aIESPYs2cPEyZM4MYbb+T0008nOzubdevWlWvv62H0JYm+nz169PD/rNgzmZGRwZlnnunfl5+fD0BGRkal1z969GhSUlK4/fbbK23rY+jQodSvX59XXnkF0I+4O3fufMQMcB+ZmZmsWLGCm266CYDXXnuNW2+9lU6dOnHmmWdWKamsjLFjx/rjOG3aNG699Vbef/99Bg0aFHQmfpMmTYCaW3DAJpN1iMzMTE9omtI1ZasJrF9tvLzk13iPle9RZIsWLVzXdpu2bdty0UUXBd0qw/eY+O9//3vYdmeccQZ5eXn8/PPPbNiwgVdeeYXu3bvz6aef0r9//3LJTcVxk/Pnz+eUU06hRYsWHHXUUfTo0cM/uzvUeElfz1skj5RbtmzJ73//ez799FPefffdStuDTlIvu+wy8vLy+N///sfHH39crvwQcMQM8hNPPJHnnnuOjRs3smXLFt544w369evHt99+S9++fY8o2ROKSMe4dunSxR/HgQMH8uSTT3L//fczZ86cco/offh8VVOTwWwyWYfIzs72hKYpXVO2msD61cbLS36ti7EaPXo0SqmINt8f68svv9xfKiYUycnJ7Nixo9z5wR4dv/vuuxw+fDhsm6oQ7QScQE477TRGjhzJW2+9xVdffRXROSeccAJXX301CxYs4LzzzmPlypUsXrzYf/ykk07ihBNOYMGCBUDZeEnQk0/OP/98EhISmD9/ftDxkoD/EX2kCdpdd91FkyZNuOOOO44omxOKa665hrVr13LdddeRlJR0xKzxcH5t0aIFV155Je+88w6//e1v2bZtG++9915E71udCTh9+vQB4OOPPz7imM9XlQ1vcAubTNYh8vLyPKFpSteUrSawfrXx8pJf4z1Wvgk1TZo08RfGDsWdd94Z0ePYQF032b9/f7XOv++++0hJSYm66LWIcM455wCwefPmcscuuOACtm7dyieffMKaNWv8yeSOHTto2LAhZ511Fp988knIZLJ9+/YAEa/qcujQIe666y5Wrlzpf3RdGb1796ZVq1Z8+OGHXHHFFTRq1Kjc8Uj9eu655wJH+iAU1fkM+GpfFhYWHnHMV9sz1KN6t7F1JmsYEekH9EtNTWXcuHGA7r5u1qwZs2fPBqB169b07t3bX3E/KSmJ3NxcZsyY4R//MHDgQNauXesv0NutWzcOHjzo18zMzCQ7O9t/Y01JSWHYsGFMmzbN/x/L4MGDWblypX+Abo8ePUhMTGTevHmAflxy+PBhv2Z6ejpDhgxhypQp/kHDQ4cOZcmSJf4xNr169aKkpMT/X2j79u3p0KEDU6dOBfTjhEGDBrFz506/bm5uLgsXLvSPtenTpw+FhYUsWrQIgI4dO9KmTRt/MdzmzZszYMAA8vLy/I9TRo0axd69e/2aOTk55Ofn+/9DzsrKolWrVsycORPQj0L69u3L+PHjKS0tJSEhgVGjRjFr1ix/WYn+/fuzadMmunfvzrhx41yLU1paGgUFBYwbN861OHXu3JnJkydTUFDAlClTXIvTpEmT/La6Fae5c+eyceNGCgoK2Lx5s2txWrZsGd27d2f58uWuxck3mWPv3r0UFxe7Fico6zlwK05FRUUAlJaW8tFHH7kWJ4Di4mKWL1/uWpwAWrVqxebNm12NU2ZmJqWlpf77gK/23s6dOzl8+DCgl687cOCAv1i0b/a37w9ycnIyKSkp/vjUq1evnMb1119PaWkpDz/8cLlJGcnJyfzpT3/i+uuvZ9++fSQnJ7Nr1y6/RkZGBjt27PD3lDVp0oTDhw/7xwOmp6dTWlrKvn37AF3/MVAjMTGRxo0bU1BQ4H+E2bRpU/9nc/fu3YB+vOnTTElJISkpyX/Mh++4LxnZvXu3//dmzZpx44038vjjj/s/l4cOHSItLY38/Hw+//xzcnJy/JoiQtOmTdmyZQvvv/8+oD/r+/bt8/v4vPPOIy8vz5+In3322ZSWllJSUkJ+fj5dunRhxowZpKam0qRJE4455hhKSkrKxen000/niy++8Nvui5MvGfPFYufOnZSUlHDllVfy9NNPc88991BcXMzRRx/tb1NYWOj/3WcDwD/+8Q9+/PFHLrzwQv8+X7x8sUpPT2f+/Pm0a9eOo48+ulycSktL/bPcW7Vq5deo6PeGDRtSUlLC/v37/T+TkpI4dOgQSil2795No0aNyM/PL5fEBsapUaNGvPHGGwC0a9eOoqIiEhMT2bNnD2lpaXz22Wccc8wxZGRkkJ+f74/Trl27KCkpAcp/FwoLC1m5cuUR36fAv09hibSb327ubikpKcptJk6c6AlNU7qmbM3KynJd0/rVjK6JWCll/erVWK1atcoVzYKCgqD7XnzxRXX//ferF198MWibquhWlU8++UQBavTo0SHbAConJ8f/ukePHio1NfWIdrt371ZNmjRRgALUkiVL1HfffaeUUqp9+/bq2GOPVddff7169tln1UsvvaRGjx6t2rZtqwB19dVXH6G3ceNGv9ZJJ53k3++7/hkzZviPDxw4MKjto0ePVvXq1VNbtmwpt3/9+vUKUDfddNMRuhMnTvTrtm/fvtx5I0aMUIDKz88P6S+llHrzzTcVoJ599ln/vpycHNWoUSM1fPhw9dRTT6mXX35ZPfTQQyorK0sB6oILLlCHDx/2t+/Ro4dKSkpSDzzwwBHb3//+d79vg8Xj3nvvVYC6//771cSJE9XEiRPV2LFj1VVXXaVERLVq1Ur9/PPP5c5ZvHixSklJUTfffHPYawskku8KsFSFyGlqPamK181EMmkxg6k/ehb3sbHyDjURK7eSSS/gSyYfe+yxkG0iTSaVUurJJ58MmkzOmTNH3XjjjerMM89UTZo0UfXq1VMZGRmqZ8+e6qWXXiqXRAVy8sknK0Dl5uYecWzHjh1KRI5I2gLZvHmzSkxMVI8//ni5/cGSSR+lpaXqrLPOciWZnDBhgn/fF198of785z+rTp06qebNm6vExETVsGFDde6556onnnhCHThwoJxGjx49/L4Mtr3++uv+dqGSycAtMTFRnXDCCep3v/vdEcm1UkqNGTNGAerbb78Ne22B2GTSo5uJZPLNN9/0hKYpXVO2mvijZ/1qRtdUghLvfvVqrNxKJnfs2OGKTk3omrLVl0y6SbS23nDDDeqkk05SxcXFrupGgpc+A+3atVOXX355VOdUN5m0E3DqEJHOdKttTVO6pmw1gfWrjZeX/BrvsfKNt/SCrilbTRCtrffffz87duxgwoQJrupGglc+AzNmzODHH3/kkUcecVW3MuwEHIvFYrFYLDFP8+bN2bNnT22bEdMMGDCAFStWcMopp9To+4ruubTUNKmpqaq6JRwqsmfPHho2bFhjmqWlMHEiDB8O0S4xW9O2VodOnTqxdOlSVzVN2eolv5rQNRErsH71aqxWr15Nu3btqq1ZUlJyRNFqNzCha8rWVatWBV0DujpYv5rRrUqsIvmuiMgypVSnYMfsY+46xMqVK2tMs7QURo2C3Fz9M8K6sJXqVgcTmqYwZauX/Brv8fKSX+M9VpWtxx1LuqZsNYH1q7fiFQ6bTNYhTCzoHkzTl0j6hq1MmBB9QllTtsYqpmz1kl/jPV5e8mu8x8pX69ALuqZsNYH1q7fiFQ6bTFqiomIi6aMqCaXFYrFYLBbvY5PJOoRviSpTmqESSR/RJJSmbY11TNnqJb/Ge7y85Fcvx8qNeQG+VXLcxoSuKVtNYP0aG/Fy4ztik8k6hInBwT7NyhJJH5EmlCZt9QKmbPWSX+M9Xl7yq1djlZiY6F86zmKxBOfQoUP+ZUOrik0m6xC+NYDd1ow0kfQRSUJpylavYMpWL/k13uPlJb96NVbJycn+ta6rg2/dbrcxoWvKVhNYv8ZGvPbu3VvtHlKbTFoqZeLEyBNJHxMm6PMsFoultmjWrBn5+fkUFRW58ijPYqkrKKUoLi6moKCAXbt2kZGRUS097zy7sFRK27ZtjWhmZ8OCBdEllCNH6vqT4XTdxoSmKUzZ6iW/xnu8vORXr8YqOTmZY445hm3btnHw4MEqax44cICCggI3zDOua8rWbdu2ISKualq/mtGNNFb16tUjLS2N1q1bc9RRR1XrPW3R8lrCRNHy/fv3k5qaakQzmkfdQ4bA5MnhC5mbtNVtTBRXNmWrl/xqQtdUIex496uNVXzHCux90Eu2mvpu2aLlccLkyZONaSYkwPjxusexMj79FJYvj0zXTUxomsKUrV7ya7zHy0t+tbGysTKB9au34hUOm0xaIibShHLzZjj/fJg6tWbsslgsFovFUnvYx9w1jIj0A/qlpqZe9+STTwLQpUsXmjVrxuzZswFo3bo1vXv3Zvz48QAkJSWRm5vLjBkz2L59OwADBw5k7dq1rFixAoBu3bqxePFifxmMzMxMsrOzycvLAyAlJYVhw4Yxbdo0du7cCcDgwYNZuXKlf8WIHj16kJiY6J8N2bZtWzZt2kRRUREA6enpDBkyhNdem8LYsb9i0aJT/dfVrdsPnH32f3nllT4UFpaVGBg+/CeefjqdN9/UmWVGRgaDBg3iscce86+fm5uby8KFC1m3bh0Affr0obCwkEWLFgHQsWNH2rRpw/Tp0wFo3rw5AwYMIC8vj+LiYgBGjRrFc889R3JyMgA5OTnk5+ezePFiALKysmjVqhUzZ84EoGXLlvTt25fx48dTWlpKQkICo0aNYtasWWzZsgWA/v37s2nTJiZNmkS7du1ci1NaWhqvv/46GRkZrsWpc+fOTJ48mZ07d3LiiScyZMgQpkyZwt69ewEYOnQoS5YsYc2aNQD06tWLkpISFixYAED79u3p0KEDU6eWj9OkSZPYtGkTGRkZrsVp7ty5bNy4kZ07dzJ8+HDX4rRs2TJWr17NiBEjXIvTnDlzADh48CA33HCDa3ECPYvztttucy1Ovu9pgwYNaN68uWtxAqhfvz6/+tWvXIsTwHvvvcfYsWNdjVNmZibbt2/3z+B2I07p6en4cDNOubm5vPDCC/6xam7EKScnh1mzZvnHy7kRJ9997/HHH6ddu3auxSk7O5snn3ySjIwM1+Lku+9t2LCBjIwM1+K0cOFCli5dSkZGhmtx8t33du7cycUXX+xanGbPns3q1avp06ePa3Hy3fduuOGGkI+5bTJZS5gYM1mTBI6hHDlS91gmJMAPP0D//vqnj379YNIkCLgnewpT408s7mNj5R1srLyFjZd3sGMmLdViypQpNabpe+Sdl1eWSAKceip8+SVccklZ23ffha5dYe3a2rE1FjFlq5f8Gu/x8pJfbaxsrExg/eqteIXDJpN1CN8jmJrSTEiAESOOnLXdqBHMmgV//WvZvlWroEsX+Oij2rE11jBlq5f8Gu/x8pJfbaxsrExg/eqteIXDJpMWI9SrB48+qguX+8pX7dqleyyfeQbs6AqLxWKxWOoGdsxkLeG1OpPVYfFiuPxycMYNA3D11YcYN64+1ayTWg5bXy12PwM1pWtrF3rHVhsrb9lq74PesdWOmbRUiyVLlsSkZpcusGSJ/unj1Vfrc+GFsG1bteX9mLh+U5iyNVY/AzWpa4J496uNlY2VCaxfvRWvcNhksg7hK38Qi5otW+olGa++umzfokXQuTM4lQ2qjYnrN4UpW2P5M1BTuiaId7/aWNlYmcD61VvxCodNJi01RnKynv39xBMgUgrApk26wHkdmdBmsVgsFkvcYZPJOkSvXr1iXlME/vxneOmlbTg1yzlwAK66Cu64Q9evrComrt8Upmz1wmfAtK4J4t2vNlY2ViawfvVWvMJhk8k6hG/1m1jXBOjWrZDFi3VdSh8PP6wLnle1UoIpW01gylYvfQbiPV5e8quNlY2VCaxfvRWvcNhksg7hWyIq1jV9um3bwldfwaWXlu2fNQvOPRd+/LFqml7BpF+9oGlS1wTx7lcbKxsrE1i/eite4YjZZFJENoiICrEFnQMsIt1E5D0R2Skiv4jINyLyJxGpF6y9c05fEZkvIntEZJ+IfCUiIyqxbYSILHba73HO71vda45HGjbUK+TcfnvZvtWr9czvDz+sPbssFovFYrFERmJtG1AJe4B/Btm/r+IOEekPTAcOAFOBnUA/4CngPODKIOfcDDwL7AAmAcXAICBPRM5QSv0lyDmPA7cBm4AXgSRgCPCuiNyilHou6qt0ifbt23tCs6JuvXrwyCNwxhl6ve+DB2H3bl3g/Ikn4I9/1GMta8tWE9SEX2NZ06SuCeLdrzZWNlYmsH71VrzCopSKyQ3YAGyIsG06sB04CHQK2J8MLAIUMKTCOSeiE88dwIkB+xsDa51zulY4p5uzfy3QuILWDkfvxEhsTklJUW6ze/duT2iG0128WKmWLZXSa+ToLTdXqQMHqq5ZXbKyslzXrGm/xpqmKV0TsVLK+tXGKr5jpZS9D3rJVlPfLWCpCpHTxOxj7igZBDQDpiil/GXflVIHgLucl7+vcM41wFHAc0qpDQHn7AIecl7+rsI5vtdjnHa+czYAYx29kdW5kOowdepUT2iG0+3cGZYuhXPOKduXlwcXXFB5gXNTtpqgpv0aa5omdU0Q7361sbKxMoH1q7fiFY5YTyaPEpFhInKHiPxRRC4IMf7xQufnB0GOLQSKgG4iErh4X7hz3q/QpjrnWKKkRQuYPx9GBIxc/eIL6NRJJ5oWi8VisVhih1hPJo8FJgJj0GMnPwZ+FJEeFdr5CswcUUpeKVUCrEePD82M8JytwH6glYikAIhIKnAcsM85XhHf/OO2lV6VITIyMjyhGYlucjJMmABPPgkJzqd082bo3h1ee61qmrFEbfk1VjRN6pog3v1qY2VjZQLrV2/FKxyiH4PHHiJyL/Ap8B1QiE4EbwauR49N7KqUWuG0XQOcApyilFobROtz9HjHbkqpL5x9xUB9oL6TcFY8ZzPQEmiplNoqIi2BzcBmpVSrIO3royfwFCuljqp43GlzvWM/IpJ19tlnR+GR+GXPnnNZv/4hDh9O9+875pg8jjvuX/6VdEyyevVq2rVrZ/x9LNXHxso72Fh5Cxsv72AqVsuWLVumlOoU9GCowZSxugGPoyfBvB2wb42zr02Icz6nwoQadOKngMQQ52x2jrdwXrd0Xm8K0b6+c/xgJNdhYgLOxIkTPaFZFd0fflDq1FPLT8zJyVEqcOyyKVtNDGaOFb/WlqYpXVMDz+PdrzZW8R0rpex90Eu22gk4kfFv52d2wL49zs+GIc7x7d9dhXP2VPgZzXvUKEVFRZ7QrIqur8D5r39dtm/27PIFzk3ZaoJY8WttaZrUNUG8+9XGysbKBNav3opXOLyYTOY7P1MD9v3g/DxivKKIJAInASXAugjPaeHob1JKFQEopfajeysbOMcrcorz84gxmBZ3aNgQ3nkH/va3sn3ff68LnM+dW3t2WSwWi8US14TqsozVDeiDfpy8KmDfNc6+V4K0v9A5tqDC/vud/fcFOSeoHvCqs39kkHNC6gXbTDzmPnjwoCc03dCdPFmp5OSyR94JCUo9+ughVVrqkoEBmHhkEKt+rSlNU7qmHu/Eu19trOI7VkrZ+6CXbLWPuR1EpJ0ze7ri/hMB3wozkwIOTQMKgCEi0imgfTLwoPPy+QpyE9BFzm92dH3nNAbucF7+u8I5vtd3Ou0C7brJ0ZsQ/urMsXDhQk9ouqH729/Cp5/Cccfp16WlcPvtiVxzjV5BJ9aJVb/WlKZJXRPEu19trGysTGD96q14hSMmk0lgMLBNRGaLyL9E5BERmQasBtoA76En4gCglNoLXAfUA+aLyHgReRRYDnRFJ5vlKoMqpdYDfwUygKUiMlZEngK+AU4GnlDOzO+AcxYBTzrHvxGRp0RkLLDU0fmLCiiAXtOsW7eu8kYxoOmWbqdOsGSJHjfpIy8PevaErcGKN8UQsezXmtA0qWuCePerjZWNlQmsX70Vr3DEajL5CTALnbT9Fvgz0AP4DBgB9FVKFQeeoJSa4bRZCAwEbgEOOecOcbpoqXDOs8Bl6PJDV6PL9mwDclWQdbmdc25Dr3KzzWl/tXN+P1WL63LHK74C57m5Zfu+/FKvpLNkSW1ZZbFYLBZLHBHq+bfdzG4mxkxu2LDBE5omdEtLlbr77h0qIaFsHGVysh5bWV1MjD/xil9NaZrSNTVWKN79amMV37FSyt4HvWSrHTNpqRaFhYWe0DShKwK/+c0W3n8fGjXS+w4cgKFD9ezvw4ddfbtq4xW/mtI0qWuCePerjZWNlQmsX70Vr3DYZLIOsWjRIk9omtJdtGgRvXvD4sUQWPz/0Ufhsstgz57Q59Y0XvOrCUzpmiDe/WpjZWNlAutXb8UrHDaZtNQ5TjlFj5vMySnb9957eqLOGlsF1GKxWCwWV7HJZB2iY8eOntA0pRuomZ4OM2fC//1f2fHvv4dzzomNAude9asXdE0Q7361sbKxMoH1q7fiFQ6bTNYh2rRp4wlNU7oVNevVg4cfhtdeg+RkvW/3brj0UnjyST1Np7bwsl9jXdcE8e5XGysbKxNYv3orXuGwyWQdYvr06Z7QNKUbSvOqq44scH7bbTBypJ6kUxvUBb/Gqq4J4t2vNlY2ViawfvVWvMJhk0lLXNCpEyxdCl27lu175RVvFDi3WCwWiyWWiSqZFJGmInKliDwjIl+JyEYRKRKREhEpEJHvRWSGiPxFRLqYMtoSnObNm3tC05RuZZrHHguffKJ7JH189VXZSjo1SV3ya6zpmiDe/WpjZWNlAutXb8UrHKIqGTgmIgL8GrgWyAESfYfCnOYT3QC8DOQppTZXy9I6Rmpqqtq/f39tmxGXKAXPPAN//rN+5A1w1FHw0ku6LmVFOnXqxNKlS2vWSEuVsLHyDjZW3sLGyzuYipWILFNKdQp2LDHYzoATrwAeBE5FJ4+l6PWxlwIrgAJgF3AAaOxsJwKdgSzgJOB+4B4RGQ88oJTaVv1L8i4i0g/ol5qayrhx4wDo0qULzZo1Y/bs2QC0bt2a3r17M378eACSkpLIzc1lxowZbN++HYCBAweydu1aVqxYAUC3bt347LPPSEjQnc2ZmZlkZ2eTl5cHQEpKCsOGDWPatGns3LkTgMGDB7Ny5Uq+++47AHr06EFiYiLz5s0DoG3btvz3v//lsFPxOz09nSFDhjBlyhT27t0LwNChQ1myZAlrnJo7vXr1oqSkhAULFgDQvn17OnTowNSpemn0jIwMBg0axCOPPELjxo0ByM3NZeHChf41Svv06UNhYaG//lbHjh1p06aNf2xJ8+bNGTBgAHl5eRQX61U1R40axT//+U8aNGgAQE5ODvn5+SxevBiArKwsWrVqxcyZMwE4+eSWfPBBXwYMOEBRUTIHD8KwYfDWW//l4os/JiFB0b9/fzZt2kT37t0ZN26ca3FKS0vjtddeo0mTJq7FqXPnzkyePJkdO3Zw0kknuRanSZMm8dNPP9GkSRPX4jR37lw2btzIjh07uPrqq8PGqWXLlvTt25fx48dTWlpKQkICo0aNYtasWWzZsgXAH6dly5bRvXt3li9f7lqc5syZA8D+/fu56aabXIsTwO7du7n99ttdi1NRUREAiYmJtG7d2rU4gV4p7ZxzznEtTgCtWrVi8+bNrsYpMzOTjRs3UlJS4lqc0tPTOXDgAMnJya7GKTc3l7Fjx5KamupanHJycnj77bdJSkpyLU6++57vPuhWnLKzs3niiSdo0qSJa3Hy3ffWr19PkyZNXIvTwoULWbJkCU2aNHEtTr773o4dO+jdu7drcZo9ezbdu3fngw8+cC1OvvteWEItjQN8BRwGSoCPgeuAjFDtQ2icBzwHbEUnovuAgdFo1NXNxHKKL7zwgic0TelWRfPHH5Vq165sCUZQ6tJLldq1q6yNiaWp6rpfa0vX1DJi8e5XG6v4jpVS9j7oJVtjbTnFXwGvAKcppS5USr2olNpZeXpaLlH9XCl1M9AK/Zh8E9A+Gg2LxSRt2ugC5337lu17/31b4NxisVgslkgJOWZSRForpTa6+mZ6/GVLZcdPGhkz6esOj3VNU7rV0Tx8GO6+W9el9NGwIUyZAnfd5f74k3jxa03rmhorFO9+tbGK71iBmXhZv3rruxVuzGTIK3A7kXQ0lU0kzTHXwNIuJjRN6VZHs149eOih8gXO9+zRSzL+/PNQ1wucx4tfa0PXBPHuVxsrGysTWL96K17hsHUm6xC+wb2xrmlK1w3Nq66Czz6DVq3069JS2LTpVkaMcLfAebz5tSZ1TRDvfrWxsrEygfWrt+IVjmjrTB4tItki0jmCtp2dtslVN89iqXmysnTdyW7dyvZNnAg9eoAzkc5isVgsFouPUDNzgm3oGd2HgSciaDvOaZsbzXvEy2ZiNvemTZs8oWlK123NAweUuuaa8jO9W7RQ6quvqq8dz341qWtqFmO8+9XGKr5jpZSZeFm/euu7RRVncwdjkPNzYgRtx6FrU/4myvewVJH8/HxPaJrSdVvzqKNg/Hg4/vjHqFdP79u6FbKzdU9ldYhnv5rWNUG8+9XGysbKBNav3opXOKJNJk8FitEFyyvjP07b06I1ylI1fAWFY13TlK4JTRFo3nwqc+aAU2Odgwfh6qvhr3/Vs8CrQrz71aSuCeLdrzZWNlYmsH71VrzCEW0yeSywz+nuDItSqhQodM6xWDxNr156HOXpp5fte/xxXZ9y9+5aM8tisVgsllon2mRyL9BIRI6urKHTphHwSxXsslSBrKwsT2ia0jVlq4+TT4YvvoB+/cr2ffABnHMO/PBDdFrWr+bj5Sbx7lcbKxsrE1i/eite4Yg2mfzGOeeKCNoOBOoBK6M1ylI1Wvnq2cS4pildU7YGkp4OM2bAnXeW7VuzRieU778fuY71a83Eyy3i3a82VjZWJrB+9Va8whFtMvkGelLNkyIScllEEekAPAko5xxLDeBbJD7WNU3pmrK1IgkJ8OCDenWco50++j179CPvxx8nogLn1q81Fy83iHe/2ljZWJnA+tVb8QpHtMnky+jJN82AJSLynIjkiMgZztZXRP4FLAaaonslx7lrssUSGwwerAucH3+8fl1aqiflXH21uwXOLRaLxWKJZaJKJpVSJUAOsBxIBn4PvOO8Xg7MBG5wji0HcpRSh9wy1hKeli1bekLTlK4pW8Nx9tl6Ys5555XtmzSp8gLn1q+1E6+qEu9+tbGysTKB9au34hUOiWBi9pEniSQBo4DhQBaQ6BwqAZYBrwIvKaWKXbKzzpGamqr2799f22ZYIqBTp04sXbo0bJuDB+Gmm+Cll8r2tWgBb7+tx1NaaoZIYmWJDWysvIWNl3cwFSsRWaaU6hTsWJXW5lZKFSul/qWU6gqkoMv/HAukKKW6KqWet4lkzTN+/HhPaJrSNWVrJBx1FLz4IjzzDOUKnPfoAa++emR769fajVe0xLtfbaxsrExg/eqteIUjsfIm4XEefW93wZa4QET6Af1SU1MZN04PJ+3SpQvNmjVj9uzZALRu3ZrevXv7P2RJSUnk5uYyY8YMtm/Xrh44cCBr165lxQpdP75bt2788ssvfs3MzEyys7PJy8sDICUlhWHDhjFt2jR27twJwODBg1m5ciXfffcdAD169CAxMZF58+YB0LZtWw4dOuTXTE9PZ8iQIUyZMoW9e/cCMHToUJYsWcKaNWsA6NWrFyUlJSxYsACA9u3b06FDB6ZOnQpARkYGgwYNoqCgwK+bm5vLwoULWbduHQB9+vShsLCQRYsWAdCxY0fatGnD9OnTAWjevDkDBgwgLy+P4mL9P8uoUaPYvXu3XzMnJ4f8/Hx/QdisrCxatWrlH+zcsmVL+vbty/jx4yktLSUhIYFRo0Yxa9YstjjPp/v378+mTZvo3r0748aNiyhOt9ySy65dn/Hoo53Yvz+ZgwdhxAiYNGkFV1yxmPPP70paWhrbt29n3LhxrsWpc+fOTJ48mYKCAqZMmeJanCZNmuS31a04zZ07l40bN1JQUMDmzZtdi9OyZcvo3r07y5cvd+X7lJaWxpw5cwDYu3cvxcXFrsUJ8J/nVpyKiooAKC0t5aOPPnItTgDFxcUsX77ctTiBnsW6efNmV+OUmZlJSUmJ/z7gRpzS09MpLS11PU65ubnl7lluxCknJ4d9+/b5Nd2Ik+++57sPuhWn7Oxs/73FrTj57ns+XbfitHDhQr+mW3Hy3fcKCgpYtmyZa3GaPXs23bt354MPPnAtTr77XlhCrbMYyQYcA3QCsqujE4+bibW5X3zxRU9omtI1ZWtV1jldu1ap9u3Lr+t9ySVK7dqlj1u/mtE1tSZtvPvVxiq+Y6WUmXhZv3rru0WYtbmrOmZyMHAn4CsPpJRSiQHHGwFvossIXamU2hX1m9Rx7JhJ71DV8SeFhTBsGLzzTtm+tm3161NPddFAix87rss72Fh5Cxsv7+CJMZMi8jDwGtABvfa2QieNfpRSu4GfgQuAwdG+h6VqzJo1yxOapnRN2VpV0tL0BJy77irbt2YNdOkCo0d/ZeQ9veTXWItXOOLdrzZWNlYmsH71VrzCEVUyKSK9gb+h19weAjQA8kM0fwWdZPapjoGWyNkSrhZNDGma0jVla3VISIAHHoCpU8sKnO/dC/ff34XHHouswHk0eMmvsRivUMS7X22sbKxMYP3qrXiFI9qeyZvRPZF/U0q9oZQ6HKbtF07bjlU1zmKpK/zmN/D552UFzpUSbr9dFzj/xa5eb7FYLBYvE2owZbAN/ej6MJAasG8rcDhE+11AUTTvES+biQk427Zt84SmKV1Ttro5mHnbNqXOO6/8xJzOnZXatMktfe/41YSuqYHn8e5XG6v4jpVSZuJl/eqt7xZhJuBE2zPZCNirlIp05ohU3sTiFps2bfKEpildU7a6yTHHwMcfw4ABZaNDliyBzp3hKxeGUXrJr16Il49496uNlY2VCaxfvRWvcESbTO4E0kXk6MoaishxQDqwrSqGWaLHV28q1jVN6Zqy1W2SkuCSS97m2WfLFzjPzoZXXqmetpf86pV4gfWrjZWNlQmsX70Vr3BEm0wudn5eGkHbm5yfn0b5HhZLnUcEbr4Z5s6FjAy9r7gYcnPhttugpKRWzbNYLBaLJWKiTSbHox9dPyQiIVcnF5HrgNvRE3D+XXXzLNHQpUsXT2ia0jVlqwl8tl54oX7M3b592bEnn4ScHNhVheqsXvKrF+MV65qmdG2sbKxMYP3qrXiFI6pkUin1LrrGZFtgmYg8jV6bGxH5g4g8LiIr0QlkAvC8UuoLl222hKBZs2ae0DSla8pWEwTampkJX3wB/fuXHZ87F845B1avrrquW3jpM2CKePerjZWNlQmsX70Vr3BEXbQcyAWeBpoDtwBpzv6ngFuB053XTwB/qKZ9lijwrXEb65qmdE3ZaoKKtqalwVtvlS9w/uOPcO658N57Vdd1Ay99BkwR7361sbKxMoH1q7fiFY6ok0mlVIlS6lbgVGAM8DHwPbAGPT7yEaCDUuqvSqlSN421WOoyvgLnb7wBKSl639690LcvPPKI+wXOLRaLxWJxg8TKmwRHKbUWuNtFWyzVpHXr1p7QNKVrylYThLP1yivhlFP0Y++NG3US+X//B998A+PHl62kE62uCVtjUdcE8e5XGysbKxNYv3orXuEQZbs7aoXU1FS1f3+k5Tojo7S0lISEqoxcqFlNU7qmbO3UqRNLly51VTMSW7dvh0GD4NOAegidOun1vlu1qrputHjpM2AiVmD9amMV37GC2rsPxoqul2w19d0SkWVKqU7Bjrl6BSJyqYj8Q0SeEpFL3NS2VM748eM9oWlK15StJojE1ubN4aOP4Prry/YtXaoLnH/5ZdV1o8VLnwFTxLtfbaxsrExg/eqteIUjqmRSRH4jIltE5MUgx/4NzAL+ip54M1tE/uWOmRZLfJKUBP/+N4wdW1bgfNs26NED8vJq1TSLxWKxWIAoH3OLyGvAYGCQUurtgP3ZwHzn5ZfAL0BP53U/pVQU81HrNiLSD+iXmpp63ZNPPgnoOlPNmjXzz+pq3bo1vXv39v/HkpSURG5uLjNmzGD79u0ADBw4kLVr17JixQoAunXrxmeffebvLs/MzCQ7O5s8J+NISUlh2LBhTJs2jZ07dwIwePBgVq5cyXfffQdAjx49SExMZN68eQC0bduW//73vxw+fBiA9PR0hgwZwpQpU9i7dy8AQ4cOZcmSJaxZswaAXr16UVJSwoIFCwBo3749HTp0YOrUqQBkZGQwaNAgHnnkERo3bgxAbm4uCxcuZN26dQD06dOHwsJCFi1aBEDHjh1p06YN06dPB6B58+YMGDCAvLw8iouLARg1ahT//Oc/adCgAQA5OTnk5+ezeLGus5+VlUWrVq2YOXMmAC1btqRv376MHz/e/5hh1KhRzJo1iy1btgDQv39/Nm3axKRJk2jXrp1rcUpLS+O1116jSZMmUcXprbd28sILF7N/f7L/89Sr1zf8/e+7OPfcTkyePJkdO3Zw0kknuRanSZMm8dNPP9GkSRPX4jR37lw2btzIjh07uPrqq12L07Jly1i9ejUjRoxwLU5z5swBYP/+/dx0003V/j517tyZyZMnA7B7925uv/121+JUVFQEQGJiIq1bt3YtTgBKKc455xzX4gTw3nvvMXbsWFfjlJmZycaNGylxqv67Eaf09HQOHDhAcnKyq3HKzc1l7NixpKamuhannJwc3n77bZKSklyLk+++9/jjj9OuXTvX4pSdnc0TTzxBkyZNXIuT7763fv16mjRp4lqcFi5cyJIlS2jSpIlrcfLd93bs2EHv3r1di9Ps2bNZvXo1ffr0cS1OvvveDTfcEPIxd7TJ5PfAKUBzpdSOgP15wNXAOKXU75x9dwAPAjOUUldE/CZxgokxkxYzmBp/UhXWr4fLLoOVK8v2XXwxTJ0KTm4e18RSrCzhsbHyFjZe3sELYyabAQcCE0mH3ujVbv4ZsG+s87NulHf3ADNmzPCEpildU7aaoKq2nnQSLFoEAwaU7fvwQ+jSRRc495Jf4yFeNa1pStfGyn1NU7rxHitTul6ytTaINplMAw4F7hCRE4FjgS1Kqe99+5VSe4Dd6ATUUgP4uq5jXdOUrilbTVAdW9PSYPp0uOeesn1r1+oVc+bNO8oF68rjpc+AKbz0efWSrSbw0vV7yVYTWL96K17hiDaZ3AmkiUhGwL6LnZ+fBWlfH9hXFcMsFktoEhLgvvvgzTfLCpwXFsLYsZfYAucWi8ViqVmUUhFvwHvAYeAB5/XRwHJn37UV2h4LlAIro3mPeNlSUlKU2xQUFHhC05SuKVuzsrJc13TT1q+/Vqp1a6V0Cqm33/5WqaIid/S99BkwESulvPV59YqtNlbesjXW74Omdb1kq6nvFrBUhchpou2ZfAEQ4A4R+Q74ETgT2AW8UaHtBc7Pb6J8D0sVWbt2rSc0TemastUEbtp61lmwZAl0716277XX9OtNm6qv76XPgCm89Hn1kq0m8NL1e8lWE1i/eite4YgqmVRKzQQeRk+2aQe0RD/6Hq6UKqzQfITz86PqGmmJDN/0/ljXNKVrylYTuG2rr8B5dvYq/75ly/SKOV98UT1tL30GTOGlz6uXbDWBl67fS7aawPrVW/EKR9Qr4Cil7gRORtebvBRoo5R6P7CNiNRHPxK/FXjHBTstFkslJCXB0KGf8a9/QWKi3vfzz9CzJ0yYUKumWSwWi6UOkxjuoIhkKKV2VtyvlPof8L9Q5ymlDgHPVN88SzR069bNE5qmdE3ZagKTfu3QAdq10+t679gBxcVwzTXwzTfw2GNliWYs2OoVvPR59ZKtJvDS9XvJVhNYv3orXuGorGfyZxFZKCJ/E5H2NWKRpcqkpaV5QtOUrilbTWDarz176nGUZ5xRduyf/4Rf/xp2HvHvYWSabhPv8fKSX22sbKxMYP3qrXiFo7JkcjdwPvAQ8I2IrBORZ0Skj4gkGbfOEhW+JZBiXdOUrilbTVATfvUVOL/88rLjH36o61GuXl01TTeJ93h5ya82VjZWJrB+9Va8wlFZMtkcnUw+CqwCTgRuRo+HLBCRt0TkWhE5xqiVFoulSjRoANOmwb33lu3zFTifNav27LJYLBZL3SFsMumUFlqklPq7UuoMypLJuejxlgOAccBmEVksIveIyNmGbbaEIDMz0xOapnRN2WqCmvRrQgKMHq2TysAC55ddBv/4R+UFzr30GTCFlz6vXrLVBF66fi/ZagLrV2/FKxyiqrhUhoikABcB/YBfAy2cQwrYCsx2tg+VUr9U39S6RWpqqtq/f7+rmsXFxSQluTv6wISmKV1Ttnbq1ImlS5e6qllbfl2xAvr3h/8FTJ+76ioYP74s0YxWs6qY0DURK/DW59UrttpYecvWunQfjBVNU7qmvlsiskwp1SnYsahLA/lQShUppd5RSl2nlDoO6AzcD3yNTiyvA94GdojIbBH5vX0cbpa8vDxPaJrSNWWrCWrLrx076ok52dll+15/Xb8OVeDcS58BU3jp8+olW03gpev3kq0msH71VrzCUeVksiJKqWVKqdFO1nocOpl8F73U4qXAc8ANbr2fxWKpGs2a6Yk4v/td2T5fgfNFi2rPLovFYrF4E9eSyUCUUtuUUi8ppQYATdCPwf8FbDHxfhZNSqjnlDGmaUrXlK0mqG2/JiXB88/rrWKB85dfrppmtMR7vLzkVxsrGysTWL96K17hqPKYSUvVEJF+QL/U1NTrnnzySQC6dOlCs2bNmD17NgCtW7emd+/ejB8/HoCkpCRyc3OZMWMG27dvB2DgwIGsXbvWvxRTt27dSEtL85cZyMzMJDs729+FnpKSwrBhw5g2bRo7nUKDgwcPZuXKlXz33XcA9OjRg8TERObNmwdA27Zt6dy5M5MnTwYgPT2dIUOGMGXKFPbu3QvA0KFDWbJkCWvWrAGgV69elJSUsGDBAgDat29Phw4dmDp1KgAZGRkMGjSISZMmUVRUBEBubi4LFy5k3bp1APTp04fCwkIWOd1kHTt2pE2bNkyfPh2A5s2bM2DAAPLy8iguLgZg1KhRzJ07l40bNwKQk5NDfn4+ixcvBiArK4tWrVoxc+ZMAFq2bEnfvn0ZP348paWlJCQkMGrUKGbNmsWWLfp/nv79+7Np0yYmTZpEu3bt6myc1qxpwUsvXcLu3fX9n9NLL/2Rd945hSlTvBOnZcuWsXr1akaMGFEn4wR14/u0bNkyAN577z3Gjh1r4xTjcfLd9x5//HHatWtn4xTjcZo9ezarV6+mT58+rsfphhtuCDlmEqVUxBuQAfQGzglyrCUwFdgG7AJeB1pGox9PW0pKinKbN9980xOapnRN2ZqVleW6Zqz5df16pc48Uyk9t1tvF12k1I4dsWdrOEzESilvfV69YquNlbdsjYf7YE1rmtI19d0ClqoQOU20j7mvB94HfhO4U0SSgYXAIHRtyoZOm/kikhrle1iqyM5olzapJU1TuqZsNUGs+fXEE+Hzz2HgwLJ9H30EXbrAypVmnl7Ee7xi7TNQ05qm8NL1e8lWE1i/eite4Yg2mezj/JxcYX8ukAnsBH4HjAA2Ayej61JaLJYYp0EDeOMNXZPSx3//C//4R3/efbfWzLJYLBZLrBOqyzLYBmxAz85OrrB/nrN/VMC+XkAp8GU07xEvm4nH3Lt37/aEpildU7aaeGQQ636dPl2p1NSyR94iSo0Zo1RpqSvySikzPjD1eMdLn1ev2Gpj5S1b4/E+aFrTlK4XHnM3A3YrpQ74dohIItDVSRzfDGj7sZNgnhp9imupCitXrvSEpildU7aaINb9esUVukzQiSfq10rBnXfqAufOuPRqE+/xivXPgGlNU3jp+r1kqwmsX70Vr3BEm0wKUHEMZBaQDKxQSu3x7XSy2D3A0dWy0BIxvllvsa5pSteUrSbwgl/PPFMXOG/btqyi19Sp0L07/PRT9fXjPV5e+AyY1DSFl67fS7aawPrVW/EKR7TJ5E9AfRE5M2DfAOfnp4ENRSQBSAPyq2ydxWKpVZo2hT/9aTY33li27z//0QXOP/+89uyyWCwWS+wQbTL5Mbp38nkR6SwilwE3otfjrjhE/3SgPhBikTaL2/To0cMTmqZ0TdlqAi/59cILsxk7Fv7977IC59u3wwUXwEsvVV033uPlpc+AjZWNlQmsX70Vr3BEm0w+AhQC5wJfotfeTgMWKaU+rtD2MnSSaRdoqyESfX/pY1zTlK4pW03gRb/ecAPMm6d7KwEOHYJRo+APf4CSkqrregEvfV69ZKsJvHT9XrLVBNav3opXOKJKJpVSG4ALgAXAAWA7MAHoH9hOROqh1+YW4CM3DLVUjm9lgFjXNKVrylYTeNWv2dl6HOWZAQNdnn0WLrkEduyoum6s46XPq5dsNYGXrt9LtprA+tVb8QpH1GtzK6X+o5S6UCmVqpRqoZS6VilVsepmKXAW0Bj4wAU7LRZLjHDiiXqmd2CB83nzdIHzOjKW3GKxWCxREHUyGQlOSaI9zmYX/64h2rZt6wlNU7qmbDWB1/2amqoLnN93X9m+devg3HPhnXeqrhureOnz6iVbTeCl6/eSrSawfvVWvMJR5WRSRBJF5BwRGSQiV7tpVIj3GyYiytlGhWjTV0Tmi8geEdknIl+JyIhKdEeIyGKn/R7n/L5h2tcTkVtF5BsR+UVEdorIeyLSrbrXWF06d+7sCU1TuqZsNUFd8GtCAtxzD7z1lk4uAfbtgwEDYMwYXZuyKrqxiJc+r16y1QReun4v2WoC61dvxSscVUomReRvwDb05Jqp6HGTgccbicgqEVkrIi2ra6SIHA88B+wL0+Zm9IzyDsAk4EWgJZAnIo+HOOdxIA9o4bSfBJwBvOvoVWwvwBTgSSDJseltIBtYKCL9K55Tk0yeXHGVy9jUNKVrylYT1CW/Xn45fPFF+QLnd91VeYHzeI9XXfoMxBJeun4v2WoC61dvxSscUSeTIjIZeAg9HnI9cMQ8TqXUbvQknZOAIdUx0EngJgA7gH+HaHMi8Dh6bfBOSqmblFK3AmcC/wVuE5GuFc7pBtzmHD9TKXWrUuomdBH2ncDjjm4gQ4BB6CT6LKXUX5VS16InJR0GXhSRtOpcr8XiRc44Q0/M6dmzbN/UqXD++bBxY62ZZbFYLJYaIKpkUkSGAFeheyW7KaXaoBOvYExGz+a+qFoWwh+AC4GRwP4Qba4BjgKec2acA6CU2oVOfAF+V+Ec3+sxTjvfORuAsY7eyArn/N75eVfgkpJKqSXoHtpm6GSzVkhPT/eEpildU7aaoC76tWlTmDuXcgXOv/4aOneGzz6rum4s4KXPq5dsNYGXrt9LtprA+tVb8QpLqEW7g23Ah+geuCsD9m0FDgdpm4zutdwYzXtU0GgH/AI85bweja5dOapCu8+c/V2DaLRwjv1UYf8mZ3+LIOd0dY59GuR69gOJQc65yjlnYiTXlpKSEm49dUsMkZWVVdsmeI4XXlAqMVEp/dBbqfr1lXrxRfPva2PlHWysvIWNl3cwFStgqQqR00RbLfNXTsJU6XxNpdQBEdmD7q2LGhFJBCYCG4E7Kml+qvNzTRA7torIfqCViKQopYpEJBU4DtinlNoaRO9H52fgNKuTgXrAOqVUsBLNwc4ph4hcD1zv/E6nTp3CXVPU7Ny5k4yMjJjXNKVrytbVq1d7IlamdKuqmZn5K9ate5SSksYcOgTXXQd33DGF449/CpHDRmw1ESuILb/Whq6NVXzHCux90Eu2mvpuhSVUlhlsQxcq31VhX9CeSedYITphq0qv5P3oXtCuAftGE7xnstjZf0SPoXN8MwG9kOiJOQrYFKJ9fef4wYB93Zx9n4U45xTn+A+RXJ+JnskXXnjBE5qmdE3ZauK/vHjx64YNSnXsWNZDCUpdeKFSBQVmbDX1H3ms+bWmdW2s4jtWStn7oJdsrY2eyWgn4OQD6ZFMMhGRU4BUqrA2t4icg+6NfEIp9UW051ssltjghBPg889hUMBI4o8/1gXON29uXHuGWSwWi8U9QmWZwTZ0WZzDwDUB+0KNmRzntH0+yvdIBH4AVgFHVTg2muA9k/nO/iYhNPc5x1Oc16nO68IQ7Zs6x38O2Nfe2fdtiHM6Oce/iuQ6TfRM7tu3zxOapnRN2Wriv7x482tpqVL331++h7JBg1I1Y4YLBgZg6j/yWPVrTema0LSx8pat9j7oHVu90DP5LHqG9oMi0iFYAxE5SkTGAKOc5Oq5KN+jAXrcYTvgQEChcgXc67R50dn3T+f1D87PI8YrikgLnB5SpVQRgFJqP/rRdwPneEVOcX4GjsH8Lzo5znTGc0ZyTo2yZMkST2ia0jVlqwniza8icPfd8Pbb0KCB3rdvn0Rc4Ly2iVW/1pRuvH+3bKzMYP3qrXiFI6pkUin1OfAYcCzwlYjMBNIARORJEZmGTtL+zznlHqVUtKv1HgReCrF97bT5zHntewT+sfPzkiB6l1ZoQ4XXEZ2jdCmgRUAK0D2K96kx1qxxP481oWlK15StJohXvw4YoAucn3RS2b677oIhQ2B/qMJfMUCs+9W0brx/t2yszGD96q14hSPqouVKqb8Bt6J76fqhkysB/ghcAWQARcAflFIPhdIJo/+LUmpUsI2yWeSvOPumOq8noJPQmwMLjYtIY8pmglcseO57fafTznfOicBNjt6ECuc87/x8UESSA87pDAxGP26fHu01WyzxRIcOusD5qadu9u974w1b4NxisVi8SpWWU1RKPQ0cj36U/TLwPjAXeBVd2PsEpVS0j7erjFJqPfBXdCK7VETGishTwDfokj5HTORRSi1CL4t4MvCNiDwlImOBpY7OX1RAAXSHKcA09Mzur0XkURF5CfgEXTboOqXUXlPXWRm9evXyhKYpXVO2miDe/dqkCcyYcYCbAxYtXb4cOnWCTz91/e2qjVf8ako33r9bNlZmsH71VrzCEW2dST9KqT3oRPJl98ypOkqpZ0VkA/AX4Gp0orwKvVrNKyHOuU1EvkX3RF4PlAL/AR5TSs0K0l6JyFXox93XALegyyUtBB50EtRao6QkWPnL2NM0pWvKVhNYv4JICc8+C2eeCTfdBIcOQX4+9OoFY8fqupSxgpf86iVbTeCl6/eSrSawfvVWvMJRpZ7J2kIpNVopJUqp8SGOv6uU6qGUSlNKpSqlOodKJAPOyXPapTrn9QiWSAa0L1FKPaWUOkMpdbRSqrFS6te1nUgCLFiwwBOapnRN2WoC69cy3euu0+WCmjnLGxw6BNdfDzffrH+PBbzo11jXNIWXrt9LtprA+tVb8QqHp5JJi8VSNzn/fFi6FM46q2zf2LHQpw8UFNSaWRaLxWKJgJCPuUXE9/h6q1Lqzgr7okEppa6tinGW6Gjfvr0nNE3pmrLVBNavR+q2bg2ffQYjR8Kbb+p9n3yiC5zPnAlnnGHEjIjwsl9jVdMUXrp+L9lqAutXb8UrHOHGTOY6P78H7gzYp9CztyNFATaZrAE6dAha+jPmNE3pmrLVBNavwXVTU2HqVOjYUZcMAli/Hrp2hUmTdGmh2sDrfo1FTVN46fq9ZKsJrF+9Fa9whHvMfZ+zja2w7/6AY5Fs97tutSUoU6dOrbxRDGia0jVlqwmsX0PrisCdd8KMGWUFzvfvh8svhwceqJ0C53XBr7GmaQovXb+XbDWB9au34hWOkD2TSqn7ItlnsVgsJujfXxc4798f1q3T++65B775BvLydC+mxWKxWGqfqCbgiMiZztbAlEGWqpORkeEJTVO6pmw1gfVrZLodOsDixXDhhWX7pk2D886D//3PiFlBqWt+jQVNU3jp+r1kqwmsX70Vr3CIiuKZkYiUomsxHquUsnMsq0FqaqraH8vrx1n8dOrUiaVLl9a2GXHNoUNw223w7LNl+5o1g+nToXvA4qY2Vt7Bxspb2Hh5B1OxEpFlSqlOwY5FWxpoD7DHJpKxyaRJkzyhaUrXlK0msH6NTrd+fXjmGXjxRf076ALnF14I48YZMa8cddWvtalpCi9dv5dsNYH1q7fiFY5ok8k1QFrgutSW2KGoqMgTmqZ0TdlqAuvXqumOGqULnDdvrl+XlMANN5StoGOKuu7X2tA0hZeu30u2msD61VvxCke0yylOBDqjlyusgf6AuoeI9AP6paamMs7pUunSpQvNmjVj9uzZALRu3ZrevXszfrxe6CcpKYnc3FxmzJjB9u3bARg4cCBr165lxYoVAHTr1o2DBw/6NTMzM8nOziYvLw+AlJQUhg0bxrRp09i5cycAgwcPZuXKlXz33XcA9OjRg8TERObNmwdA27ZtOXz4sF8zPT2dIUOGMGXKFPbu1UuQDx06lCVLlrBmzRpArzNaUlLir+rfvn17OnTo4J+xlpGRwaBBg9i5c6dfNzc3l4ULF7LOmWXRp08fCgsLWbRILyrUsWNH2rRpw/Tp0wFo3rw5AwYMIC8vj+LiYgBGjRrF3r17/Zo5OTnk5+ezePFiALKysmjVqhUzZ84EoGXLlvTt25fx48dTWlpKQkICo0aNYtasWWzZsgWA/v37s2nTJrp37864ceNci1NaWhoFBQWMGzfOtTh17tyZyZMnU1BQwJQpU1yL06RJk/y2uhWnuXPnsnHjRgoKCti8eXOV4vTwwx8xevRZ/PRTUwD+9S/46KMtXHDBJSxfvty1OM2ZMweAvXv3Ulxc7FqcAP95bsXJ90eptLSUjz76yLU4ARQXF7N8+XJXvk/Lli0DoFWrVmzevNnVOGVmZlJaWuq/D7gRp/T0dCNxys3NLXfPciNOOTk5FBUV+TXdiJPvvue7D7oVp+zsbP+9xa04+e57Pl234rRw4UK/pltx8t33CgoKWLZsmWtxmj17Nt27d+eDDz5wLU6++144oh0zKcBbQB/0utQTlFKlEQtY/JgYM1lcXExSUlLMa5rSNWWrifEn1q/V1y0q0gXO33ijbF9S0maWLj3O9QLn8eTXmtI0Na7LK9dvStfeB73lVy99t9wcM/kSsBsoQfdMbhOR90TkFRF5OcT2UvXMt0TKwoULPaFpSteUrSawfq2+bkoKTJkCY8aU7SsuPo6uXeHtt6tpXAXiya81pWkKL12/l2w1gfWrt+IVjmgfc+dSfgWcpsAllZxjV8CpIXyPtWJd05SuKVtNYP3qjq4I3HGHLiE0dCjs26cLnF9xBdx3n15FJyHaf5kN2VoTmqZ04/27ZWNlButXb8UrHNEmk7ZoucViiTkuuwy+/BLOPnsTxcWtALj3Xl3g/JVXyhc4Ly2FiRNh+HB3Ek2LxWKJe5RSdquFLSUlRbnNhg0bPKFpSteUrVlZWa5rWr+a0e3Y8UJ14YVK6UUX9daxo1K+tzp8WKmRI/X+kSP169qy1Ut+NaFp4nullHeu35SuvQ96y69e+m4BS1WInMb+X16HKCws9ISmKV1TtprA+tWMbmLiHubMgT/8oWzfihXQqRPMn69LC02YoPdPmKBfl0YwhTDe/Rrv3y0bKzNYv3orXuGoVjIpmqYi0totgyxVx1eqINY1TemastUE1q/mdBMT4emnYfz4sgLnBQW6wLkvkfQRaUIZ736N9++WjZUZrF+9Fa9wVCmZFJGuIvIOsBf4GVhX4XgjEXlJRMaLSIoLdlosFktUXHstfPKJXnYR9EPvYETTQ2mxWCyWI4k6mRSRm4CFQF8gFT2zWwLbKKV2o2d6jwQGVttKS0R07NjRE5qmdE3ZagLr15qJV9eucMEFlberLKGMd7/G+3fLxsoM1q/eilc4okomRaQL8DRQCvwf0BrdMxmMCegk89fVMdASOW3atPGEpildU7aawPrVfLxKS3WCGFjUPBzhEsp492u8f7dsrMxg/eqteIUj2p7JP6MTxNFKqUeVUpvCtF3g/Dy7SpZZosa3nFOsa5rSNWWrCaxfzcdr4sQjx0hWxoQJ+ryKxLtf4/27ZWNlButXb8UrHNEmk92dn/+qrKFSahdQCLSK1iiLxWKpLsOH6+UWo2HkSH2exWKxWCIn2mSyKbBXKbUnwvaHq/AelirSvHlzT2ia0jVlqwmsX83HKyFBz+iONKHMzdXtgxUyj3e/xvt3y8bKDNav3opXOESFmuIYrLHIdiADSFFKFTv7tgLNlVL1KrRtih5PuVkpZUsHVSA1NVXt37+/ts2wRECnTp1YunRpbZthiYBgsfKNnazskffo0XrVHEvNYL9X3sLGyzuYipWILFNKdQp2LNpewxXoMZPdK2uIXsdbgK+ifA9LFcnLy/OEpildU7aawPq15uIVaQ/l6NHw3HPBj8W7X+P9u2VjZQbrV2/FKxzRJpOvohPEh0WkQahGItIbuB9QwMtVN88SDcXFxZ7QNKVrylYTWL/WbLxCJZRXXw0XX1z2+pZbYNKkI8+Pd7/G+3fLxsoM1q/eilc4EqNsPwm4GugFfCUi44GjAESkH3ACcCnQB52ovq2Uet89cy0Wi6Vq+BJK0I+8R47Ur3/5RSeUX3yhj+XmQsOG0K9frZlqsVgsniKqMZMATo/kRKA/uufxiCbOz7eAq5VSRdWysI7hJN39UlNTr3vyyScB6NKlC82aNWP27NkAtG7dmt69ezPe+cuXlJREbm4uM2bMYPv27QAMHDiQtWvXsmLFCgC6detGamoqH374IQCZmZlkZ2f7u9BTUlIYNmwY06ZNY+fOnQAMHjyYlStX8t133wHQo0cPEhMTmTdvHgBt27YlKyuL119/HYD09HSGDBnClClT2Lt3LwBDhw5lyZIlrFmzBoBevXpRUlLCggW6MlT79u3p0KEDU6dOBSAjI4NBgwYxadIkior0RyM3N5eFCxeybp1eSKlPnz4UFhb6l5nq2LEjbdq08ZdQaN68OQMGDCAvL8//X92oUaOYO3cuGzduBCAnJ4f8/HwWL14MQFZWFq1atWLmzJkAtGzZkr59+zJ+/HhKS0tJSEhg1KhRzJo1iy1btgDQv39/Nm3axKRJk2jXrp1rcUpLS2POnDmuxqlz585Mnjw5ruO0bNkyVq9ezYgRI8LGKTExiXr1cklLm0FBwXbHH4Po0yeZ//43xYllKa+8sp29e98B4KSTTqJHjx6uxiktLY2rrrrK9ThdffXVfPbZZ67G6dJLL2XHjh2uxQngvffeY+zYsa5/n84//3xeffVV1+KUnp7Ob37zG9544w3Xv08LFixg/fr1rsUpJyeH7du3s2TJEtfi5LvvPf7447Rr187e92LwvhcYp9mzZ7N69Wr69OnjepxuuOGGkGMmUUpVaUP3Tk4E1gL7gV+ADcDrQJ+q6sbLlpKSotzm/fff94SmKV1TtmZlZbmuaf1qRrc6sdqyRamTT1ZKL7yoVFqaUkuW6GPx7tdYi1U4vHL9pnTtfdBbfvXSdwtYqkLkNNE+5g5MQucB86p6vsV9fP/1xLqmKV1TtprA+jX24tWiBXz4IZx3HmzdCoWFcMkl8Omn1q+xFqtweOn6vWSrCaxfvRWvcNgakBaLxeJw0kk6oczI0K937IDevWHHjpDzDS0WiyXusclkHSInJ8cTmqZ0TdlqAuvX2I1X+/bw/vuQmqpfb9oEL774G37+2d338ZJfYzVWwfDS9XvJVhNYv3orXuGIKpkUkXuqspky3lKe/Px8T2ia0jVlqwmsX2M7Xl26wMyZkJSkX69fn8gll8Du3e69h5f8GsuxqoiXrt9LtprA+tVb8QpHtD2To4F7o9h87S01gG9mWKxrmtI1ZasJrF9jP169esGUKWXLKy5frssFFblUn8JLfo31WAXipev3kq0msH71VrzCEe0EnFcJXg7IR0MgCzge2Am8W0W7LBaLpda5/HJ46aWyYueffQaDBsGMGWW9lhaLxRLvRJVMKqVyI2knIsOAcUCJUuq6KthlqQJZWVme0DSla8pWE1i/eideubnw7bc/8eSTxwN6POWIEXqlnHr1qq7rJb96JVbgrev3kq0msH71VrzCYWQCjlJqEnArcI2IjDDxHpYjadWqlSc0TemastUE1q/eitfttydx991lr6dMgZtv1hUpq4qX/OqlWHnp+r1kqwmsX70Vr3CYnM39KnAY+L3B97AE4KueH+uapnRN2WoC61fvxeu++3QC6ePf/4a77qqepgm89BkwgZeu30u2msD61VvxCoexZFIp9QtQBJxu6j0sFoulphCBp5+GoUPL9j30EDz+eO3ZZLFYLLGAsWRSRE4E0oFSU+9hKU/Lli09oWlK15StJrB+9Wa8EhJgwgTo27fs2F//Cs7yt1XSdBsvfQZM4KXr95KtJrB+9Va8wiGqOoN+QomKHANMAbKBT5RSF7n+Jh4nNTVV7d+/v7bNsERAp06dWLp0aW2bYYmAmorVL7/ApZfCggX6dUICTJ2qZ3pbIsN+r7yFjZd3MBUrEVmmlOoU7Fi0RctfrmR7TUQWAhvQiWQpMKbaV2CJiPFV6R6pBU1TuqZsNYH1q7fjdfTR8M47cPbZ+nVpKfz2t3opxqpquoWXPgMm8NL1e8lWE1i/eite4Yi2zmQuus6kRNB2C3CzUuqTaI2yVI3SUvdHFJjQNKVrylYTWL96P17p6fDBB9C9O/zwAxw6BAMGwEcfQdeuVdN0Ay99Bkzgpev3kq0msH71VrzCEW0yeV8lx0uA3cC3wOdKqcNVMcpSNRIS3B8Ca0LTlK4pW01g/Vo34tWsme6NPO88+OknvTrOr3+tH3+feWbVNKuLlz4DJvDS9XvJVhNYv3orXuEwMmbSUjl2zKR3sGOFvENtxWrNGjj/fPAts3vssXq1nJNPrnFTPIP9XnkLGy/vUBtjJqPtmbRUExHpB/RLTU1l3LhxAHTp0oVmzZoxe/ZsAFq3bk3v3r39YymSkpLIzc1lxowZbN++HYCBAweydu1aVqxYAUC3bt349ttvKSwsBCAzM5Ps7Gzy8vIASElJYdiwYUybNo2dO3cCMHjwYFauXMl3330HQI8ePUhMTGTevHkAtG3bll27dvkXok9PT2fIkCFMmTKFvXv3AjB06FCWLFnCmjVrAOjVqxclJSUscGYmtG/fng4dOjB16lQAMjIyGDRoEE899RSpqakA5ObmsnDhQtatWwdAnz59KCwsZNGiRQB07NiRNm3aMH36dACaN2/OgAEDyMvLo7i4GIBRo0bxwgsvUM9ZkiQnJ4f8/Hz/uqdZWVm0atXKX9OrZcuW9O3bl/Hjx1NaWkpCQgKjRo1i1qxZbNmyBYD+/fuzadMmunfvzrhx41yLU1paGm+88QYNGzZ0LU6dO3dm8uTJ7Nmzh+OPP961OE2aNImtW7fSsGFD1+I0d+5cNm7cyJ49e/jtb3/rWpyWLVtG9+7dWb58uWtxmjNnDgBKKUaOHBk2TtOmXcWvf300+/cnsm0b9OxZzNSpW1i5cu4RcQIoKiriT3/6k2txKnIWDT/mmGNITU11LU4ADRs25NRTT3UtTqCLNW/evNnVOGVmZrJ//35+/vnnkHGK5vsE+r7XoEED9u3b52qccnNzmTBhAiLiWpxycnL46KOPOHjwoGtx8t33fPdBt+KUnZ3N008/TcOGDV2Lk+++99NPP9GwYUPX4rRw4UK+/vprGjZs6FqcfPe9PXv2cOGFF7oWp9mzZ9O9e3c++OAD1+Lku++FRSllt1rYUlJSlNu88MILntA0pWvK1qysLNc1rV/N6JqIlVKR27pwoVLJyUrptXGUOv10pQoKqqcZLV75DNR2rGpb05SuvQ96y69e+m4BS1WInCaqnkkRuTqa9pUksa+6pWWxWCyxQPfuMH069O8PJSWwapUeQzlvHjRoUNvWWSwWiyFCZZnBNnSpn8NubNG8b13cTPRMbtu2zROapnRN2WrivzzrVzO6pv4jj9bW115TSqSsh7JXL6V++aV6mpHilc9ArMSqtjRN6dr7oLf86qXvFmF6JqOdRrTQ2faiywMJsAn4ytl+Cti/J6B9sM3iMps2bfKEpildU7aawPq1bsfrqqtg7Niy1/Pm6X0lJVXXjBQvfQZM4KXr95KtJrB+9Va8whFVMqmU6gksAxoCLwEnK6VOUEp1c7YTgUzgRaAROou9INjm6lVYAPwDcWNd05SuKVtNYP1a9+P1+9/DmIAlG2bMgOuu0wXOq6oZCV76DJjAS9fvJVtNYP3qrXiFI9oVcIYBtwKPKKWuU0qtr9hGKbVBKXUD8A/gzyLyW3dMtVgsFm/x97/DX/5S9jovT79WtiKbxWKpQ0T7mPsm9LjJhyNo+w+n7U3RGmWpGl26dPGEpildU7aawPo1PuIlAo8+CtdeW7bvqad0j6WX/BoPsappTVO68R4rU7pesrU2iLbO5OnAXqXU3soaKqX2isheoH2VLLNETbNmzTyhaUrXlK0msH6Nn3iJwAsvwO7deqY3wN13g8iJnHWWK+aVw0ufARN46fq9ZKsJrF+9Fa9wRNszqYCGItK8soZOm0ZVMcpSNXzFf2Nd05SuKVtNYP0aX/GqVw8mT4aLLy7bd9ddjXDqLbuKlz4DJvDS9XvJVhNYv3orXuGINpn8D3qm9qMRtH3UaWvXX7JYLHHPUUfBW2/BueeW7RsxAmbNqj2bLBaLxQ2iTSZ9CeJwEflQRC4SkaN9B0Uk2dk3FxiO7smMJPG0uEDr1q09oWlK15StJrB+jc94NWgAs2dDhw769eHDcOWV4Kzu5gpe+gyYwEvX7yVbTWD96q14hUNUlNMKReSv6Mk1PkrRNSVBlwxKQCecCvibUupxF+ysc6Smpqr9+/e7qulbwzPWNU3pmrK1U6dOLF3qbge79asZXROxAvdt3boVzj9fsW6dXpc5LQ3mz4ezz66+tlc+A16JlSlNU7r2Pugtv3rpuyUiy5RSnYIdi/oKlFKPAT2A+c6uekCGs9Vz9s0Dsm0iWbP4FnSPdU1TuqZsNYH1a3zHq0ULuOaaKbRooV8XFkKfPvD999XX9tJnwAReun4v2WoC61dvxSsc0c7mBkAp9RnQS0QaA78CfNOR8oGvlVK7XLLPYrFY6iTNmhUydy5kZ8OuXVBQAL17w2efQR158mWxWOKEKiWTPpyk8WOXbLFUk6SkJE9omtI1ZasJrF9tvJKSkujQAd57Dy66CPbvh59+0jO+P/0UmldaMyO0rtvYWHnnOxDvsTKl6yVba4Oox0xa3MHEmEmLGUyNP7G4j1dj9dFHkJMDxcX69a9+BZ98Ag0b1q5dJvFqrOIVGy/v4IkxkyJyhYhcXX2zLG4zY8YMT2ia0jVlqwmsX228AjUvughefx184/C//hr69YOiourpuoWNlfuapnTjPVamdL1ka21QlcfcTwDHA6+6bEtcICL9gH6pqamMGzcO0MspNWvWzF+8tHXr1vTu3ds/MDcpKYnc3FxmzJjB9u3bARg4cCBr165lxYoVAHTr1o2ffvrJr5mZmUl2djZ5eXkApKSkMGzYMKZNm8bOnTsBGDx4MCtXruS7774DoEePHiQmJjJv3jwA2rZty9atW/2a6enpDBkyhClTprB3r14EaejQoSxZsoQ1a9YA0KtXL0pKSljg1Dpp3749HTp0YOrUqQBkZGQwaNAgvv/+e79ubm4uCxcuZN26dQD06dOHwsJCFi1aBEDHjh1p06YN053lQ5o3b86AAQPIy8uj2OnKGTVqFGvXrvVr5uTkkJ+fz+LFiwHIysqiVatWzJw5E4CWLVvSt29fxo8f759NN2rUKGbNmsWWLVsA6N+/P5s2baJ79+6MGzfOtTilpaWxatUqtm/f7lqcOnfuzOTJkykoKODAgQOuxWnSpEls3LiR7du3uxanuXPnsnHjRgoKCujcubNrcVq2bBndu3dn+fLlrsVpzpw5AOzdu5fi4mLX4gT4zwuM03PPjeDGG48C9KPunJz9PPvsJhYtqjxORU7mWVpaykcffeRanACKi4tZvny5a3ECaNWqFZs3b3Y1TpmZmWzbts1/H3AjTunp6ezdu9e175MvTrm5ueXuWW7EKScnhw0bNvg13YiT777nuw+6Fafs7Gz/fdCtOPnue+vWrWP79u2uxWnhwoV+W92Kk+++V1BQwPHHH+9anGbPnk337t354IMPXIuT774XFqVUVBuwHjgcZP9WoCRavXjdUlJSlNu88MILntA0pWvK1qysLNc1rV/N6JqIlVI169cnnlAKyrarrlLq8OHq61YHGyvvfAfsfdBbfvXSdwtYqkLlhqEO6PP4FHgIyAEaqcqTySP2263mksmCggJPaJrSNWWriS+m9asZXVM30Zr26113lU8of/97pUpLq69bVWysvPMdsPdBb/nVS9+tcMlkZWMmzwP+D3gHKBCRb4HmACLSsvJ+T0tNsnbtWk9omtI1ZasJrF9tvMJp3n8/3Hhj2evnn4e7766+blWxsfLOdyDeY2VK10u21gaVJZPHAUOAfwErgXbA0egVbn4SkXUi8oqIXAfUjfntHsY37iHWNU3pmrLVBNavNl7hNEXg2Wfht78t2zdmDDzxRPV0q4qNlXe+A/EeK1O6XrK1NgibTCqltiql3lBK3aKUOgtoin6crYAvgZboNbj/DTQGEJHpIvInEfmViIhR6y0Wi6WOkpAAeXm6ZJCPv/wFXn651kyyWCyWoERVGkgptRsodn4/D2gEXADcC/ziNLsceBJYCuwUkVku2WqphG7dunlC05SuKVtNYP1q4xWJZv368OabepUcH9ddB87E0SrrRouNlXe+A/EeK1O6XrK1NgibTIrIbSLSRUTqBTuulDqglFqglHoQ2Ovs7gTcBswADgGXumivJQxpaWme0DSla8pWE1i/2nhFqnn00fDOO7qQOUBpqX78/eGH1dONBhsr73wH4j1WpnS9ZGttUFnP5GPAF8AeEflQRO4FUsOdoJT6j1LqKaXUQKVUc6C9S7ZaKsFXGyrWNU3pmrLVBNavNl7RaDZsCB98AG3b6tfFxXD55fDll9XTjRQbK+98B+I9VqZ0vWRrbVBZMvknYDqwB+iFfpzdFEBElojIEyJymYg0CSWglPreJVstFoslbmneXPdGHn+8fr1/P/z61/Dtt7Vrl8VisVQ2AecZpdRvlFLHAW2AkejEEiALuBV4G9hOWcmg34rI8eZMtoQiMzPTE5qmdE3ZagLrVxuvqmi2bq0TyqZN9etdu6B3b3AWu6mybmXYWHnnOxDvsTKl6yVbawPRdSijOEFkPdAaaAH0AM4HsoEzKzTdiC56vkAp9VL1Ta1bpKamqv3797uqWVxcTFKSuxWaTGia0jVla6dOnVi6dKmrmtavZnRNxApiz6//+Q/07AmFhfr1SSfBZ59By5axZ2so4iVWNa1r74Pe8quXvlsiskwp1SnYsahmcweilNqulHpTKfVHpdSv0L2TAGOAz4FjgWHAuKroi8gjIjJPRH4SkV9EZKeIfC0i94Z6rC4i3UTkPaftLyLyjVOmKOgEIuecviIyX0T2iMg+EflKREZUYtsIEVnstN/jnN+3KtfpJhGtnxkDmqZ0TdlqAutXG6/qaJ59NsyaBcnJ+vX69dCnD+zcGXu21jReun4v2WoC61dvxSscVU4mQ6GUukcplU35skFV4Vb0ZJ8PgaeByUAJMBr4puKjdBHpDyxE95K+DTyHLqT+FDAl2BuIyM3Au0AHYBLwIrp2Zp6IPB7inMeBPHTP7IvOeWcA7zp6FovFYpzsbJg2DRIT9euVK/UYygMHEmvXMIvFEndU5a7zFbChskZKqYPAAmerCulKqQMVd4rIGOAO4O/Ajc6+dHRidxjoqZRa6uy/G/gYGCQiQ5RSUwJ0TgQeB3YCnZRSG5z99wNLgNtEZLpS6ouAc7qhyx79F+islNrl7H8MWAY8LiKzfFo1TUpKiic0TemastUE1q82Xm5o5uTA/7N37uFRVOcf/5wAISYSkTsYEVNEESpKABEMWEHQAgWFCgqVoEH689aqtbVcvHCx2qK2VapAFJBQoYBAJQoIIlFRgVCoIEgRNYR7iBgghJDk/P6Y2c2FZLObzMnuJO/neebZzMyZ77zzfncmZ2fOnDNvHowaZY3i/cUXcOrUbYwbB/XrOxCkjXjlnnOgtntlStdNsQaF8gbtDnQCDgMFTun52E8nrBF4Pii27F572bwyyt9sr9tQavlke/mzZWxTph7wlr18TBnblKtX1hQZGXneIOpCaBIXFxfsEAQ/qa1ezZihtVWdtKY77tD63LlgR+Wb2uqVWxG/3IMpr4Atupw6jZOPubsA1fFa0iD787/Flt1sf64qo3wqkAP0UEoV/63ua5v3S5WpyjbVxpIlS1yhaUrXVKwmkLyKX05qPvAATJ1aNP/OOzBunFW1dALxyj3nQG33ypSum2INBo41rtFaZzilVRyl1O+AC4GLsCqsN2JVJJ8vVuxK+3NPGXHl22+gd8Cq7O7yY5tDSqnTQIxSKlJrnaOUigIuAU5prQ+VEer/7M92gRyfk2RlZblC05SuqVhNIHkVv5zWHD/eegHnpZes+TffhIYNYfp0UKpq2uKVe86B2u6VKV03xRoMAu4aqLpRSh0GmhdbtApI0FofKVZmD3AFcIXWem8ZGp8CPYAe2m4DqZTKA+oB9bTW+WVscwDrZZxWduWyFXAAOKC1jimjfD2sccvztNZltlZSSt0P3G//Hde5c2d/UuA3x44do2nTpiGvaUrXVKy7du2iffv2jmpKXs3omvAK3JNXreHrr3/P6dN3epe1ajWDli3nVElXvHLPOSDXQXfl1U3nVlpaWrldA/lqmxhW3rqqTJXVxapQ3g58DRwEOhdbtwervWLbcrb91F5/Q7FlefayuuVsc8Be39Keb2XPZ5RTvp69/qw/x2OizeSJEydcoWlK11SsJtqfSF7N6JpqK+SmvGZmntB33FGyDeU//lE1TfHKPeeAXAfdlVc3nVtUss3k10qpMb76aAwEpdSVSqm3gCcrs73W+ojWehnQD2iM9TKMB8+oPBeVs7ln+YlKbPNjqc9A9lGt7NixwxWapnRNxWoCyav4Zer4d+/ewT//CX36FC178EH45z8rryleueccqO1emdJ1U6zBwFdlshmQBHxjdxR+VaDiSqlIpdRdSql/AzuBkUCVhn3RWn8PfAV0UErZg4rxtf15XntFpVRd4HKsPiqLDTrmc5uWWH1cZmitc+z9nsa6W3mhvb40V9if57XBrC527tzpCk1TuqZiNYHkVfwymdf69WH5crj+emuZ1jB6NKSkVF7TLbjNKzdomkLy6i6/fOGrMnk5VmfhzYGngJ32iDKvKaUSlVLdlFKxSqmLlVJRSqlLlFIdlVKDlFLPKqXewxoVJxkYCOwABmit/+ZA3K3szwL780P789YyyvYCIoGN2ur70oOvbW4rVaYq2wiCIFQ7F14I770HHTpY8/n5MGwYfPxxcOMSBKEGUt7zb13UFvBS4AXgEFBoTwV+TJ5ya4Ch2C/7+DNh3S28qIzlYVjDNWrg02LLo4FjwFmsDsg9yyOAjXb5EaW0LgdygeNAm2LLLwb2UqqNpb2uh718L3BxseVtbJ3c4lq+JhNtJnfv3u0KTVO6pmI10f5E8mpG11RbITfn9cABrS+/vKj9ZHS01mlpVdN0AvHKXbHKddA9sQajzWSFXQNprfcDf1BKjQd+jtVmsSfWEIJltafMtCtwnwCLtfVYOlB+DvxJKfUJ8K1dUWsO9Mbq3ucwMLZYjNlKqbHAEuAjpdRCrJFtfoHVBdASYFGp4/pWKfUE8Hdgi1JqEdZLOcOAGOBFXWz0G3ubjUqpl4DHsIZ0XII1ZONwoBHwsA7S6DcAdetWaGdIaJrSNRWrCSSv4ld15bVVK/jgA7jxRjh8GLKz4dZbrTuUV15ZjkgFmqGMm70KVU1TSF7d5Zcv/O60XGtdoLV+V2v9sNa6M1bfj5cBcVh37K4Cmmitm2mth2itp1eyIgmwFngDaArcATyBdXczC3gW6KC1/qpUfMuxKpupdtmHgXNYFb8Rdq269DG9glXh3Ancg9Vtz2Gsrod+V04eHgfG2OXut7fbCQzSWr9ayeN1hHXr1rlC05SuqVhNIHkVv6ozrz/5CaxZAxdfbM0fOwa33AL791deM1Rxu1ehqGkKyau7/PJFpavE2mp/uN+eHEVrvQN4qBLbfYp1VzOQbd4F3g1wm7nA3EC2EQRBCCY//anVhrJPH8jJsSqSt9wCqanQrFmwoxMEwc04OZyiEGTatXN+8B0TmqZ0TcVqAsmr+BWMvHbvbr3lXa+eNf/119Yj7x9/LHeTCjVDjZriVShpmkLy6i6/fBHyI+DUVKKiovTp01XqJek8Tp8+TVRUVMhrmtI1FWuXLl3YsmWLo5qSVzO6JryCmpfXJUtg+HAoLLTme/WCVavgggsqrxko4pW7YpXroHtiNXVuKaXKHQFH7kzWIBYsWOAKTVO6pmI1geRV/ApmXocNg1mziuZTU+HOO+Hcucprhgo1zatQ0DSF5NVdfvlCKpOCIAi1kPvug+nTi+ZXroSEhKK7lYIgCP4ilckaRHR0tCs0TemaitUEklfxKxTy+vjjMH580fw//wmPPGL1SFlZzWBTU70KpqYpJK/u8ssX0mYySJhoMymYwVT7E8F5xKvA0doau/u114qWTZoEkyeb3a945S7EL/cgbSaFKrFw4UJXaJrSNRWrCSSv4leo5FUpePVVuOuuomVTpsDLL1deM5jUZK+CpWkKyau7/PKFVCZrENnZ2a7QNKVrKlYTSF7Fr1DKa1gYzJsHPy/WS+9jj8GcOZXXDBY13atgaJpC8uouv3wRUGVSKfWUUuqxAMo/opR6KvCwBEEQhOqkXj1YvNgadtFDYiIsWxa8mARBcAcBtZlUShUCh7XWrfws/y3QWmtd1hjetRKl1CBgUFRU1NiXXnoJgG7dutG0aVNSUlIAaN26Nf369SMpKQmA8PBwEhISWL58OUePHgVg6NCh7N27l+3btwPQo0cP6tatS2pqKgCxsbH06tWLuXPnAhAZGcmoUaNYsmQJWVlZAAwfPpwdO3awc+dOAHr37k3dunW9wzu1a9eODh06sMz+bxIdHc2IESNYuHCh99fUyJEj2bx5M3v27AGgT58+5Ofns2HDBgA6dOhAx44dWbTIGhq9UaNGDBs2jHnz5nH27FkAEhISSE1NZd++fQD079+fkydPsnHjRgA6depE27ZtWbp0KQDNmjVjyJAhzJ07l7y8PAASExN57733OHjwIAADBgzg2LFjbNq0CYC4uDhiYmJYsWIFAK1atWLgwIEkJSVRWFhIWFgYiYmJrFy50qsxePBgMjIySE5Opn379o751KBBA95//33CwsIc86lr164sWLCAwsJCGjZs6JhPycnJnDp1irCwMMd8WrNmDenp6RQWFjJo0CDHfEpLS2PXrl2MHj3aMZ9Wr14NwKWXXkqfPn0c8wkgKiqKkSNHOuZTTk4OAL/85S9JS0urtE85OeHMmzeabdsUAHXrFpCUdJhOnZzzCeC9995jxowZjvoUGxtLXFwcixcvdsyn6OhoBg0axLvvvuuoTwkJCaxbt4799piWVT2fwLruZWRkePPjhE+e69706dNp3769Yz716tWLN998k7CwMMd88lz3Tpw4QVhYmGM+paamsnfvXsLCwhzzyXPdKywspGvXro75lJKSwq5du+jfv79jPnmue+PGjSu3zSRaa78noBA4GED5b4GCQPZRW6bIyEjtNOvXr3eFpildU7HGxcU5ril5NaNrwiuta19ejxzRul07ra3Xc7SOiMjXn39e9diKI165K1a5DjqvaUrX1LkFbNHl1GlMt5lsAuQY3odg4/n1FeqapnRNxWoCyav4Fcp5bdYM1qyBmBhrPje3DrfdBjt2VFnaOLXNq+rQNIXk1V1++aKuCVGl1EXAGCAK2G5iH4K7OH78OMuWLePQoUO0bNmSU6dOBTskQRB8cNll8MEHEB8PmZnwww/Qrx98+ilcfnmwoxMEIZTwWZlUSj0NlH6BprlSqsBPfQ3UjLGCXECfPn1CTlNrzbRp05g2bRq5ubne5fXr1ycnJ4cJEyaglKpqmICZ4zeFqVhD8TtQ3bomqK15veoqa8zu3r0LOX06jEOHoG9f+OQTaNnSsd04Sm31yqSmKSSv7vLLF/485lbFJl1q3td0CJgCvOh41EKZ5Ofnh5zmtGnTmDRpUomKJMDZs2eZNGkS06ZNq5J+cUwcvylMxRqK34Hq1jVBbc5rXBy89loG9etb8/v2WXco7fckQo7a7JUpTVNIXt3lly8qqkz+FbjcnmKxKonHii0ra7oMuFhrHaO1fsZutClUA5431EJF8/jx4xVWFqdNm8bx48crvY/imDh+U5iKNdS+A8HQNUFtz+uZM6tYvBjq2P1y7NgBAwZAKA7iVdu9qu3nlSldN8UaDHxWJrXWP2qtv7en74BU4ONiy8qa9mutf6yW6IWQZtmyZefdkSxNbm6ut+shQRBCl0GDwO4hBIDPP4fbbwe7hy9BEGoxAb3NrbW+SWs9zFQwQtXo0KFDSGkeOnTIr3LfffddpfdRHBPHbwpTsYbadyAYuiao7Xn1aI4aBa+8UrT8gw9g5Ego8LcVfTUgXtXu88qUrptiDQYynGINomPHjiGl2dLPFvrTp0/n4Ycf9rvyWR4mjt8UpmINte9AMHRNUNvzWlzzoYdg8uSidUuXwrhxVo+UoYB4VbvPK1O6boo1GFRmOMWAJ1PBCyXx9OIfKpq33347ERERFZY7e/YsM2fOrPJb3SaO3xSmYg2170AwdE1Q2/NaWnPiRPjtb4vm33gDfv/70KhQile1+7wypeumWINBoP1MPoP1Rre/eN4An1xRQaHm0bhxYyZMmMCkSZPKLdOkSRMyMzMZNmwYLVq0KLHu8OHDhIWF0axZM9OhCoIQAErBiy/CiRNF7SinT4dGjeCPfwxmZIIgBINAK5Nv4bsyeREQB1wKZAHvVjIuoRI0atQo5DQnTJgAcF4/k/Xq1eOpp55i/PjxrF+/nubNm5+37Z/+9Cdef/117rzzTh588EGuv/56n3cvTRy/KUzFGorfgerWNUFtz2tZmmFhMHu2VaFcvtxaNn48XHwx/PrXjofgN+JV7T6vTOm6KdZgoEz03KOUGgXMAhZorcc6voMaQFRUlD4div1qGKL0CDi33347jRs3Lrf8qVOnuOSSS8jOzvYu69y5Mw899BAjRozgggsuqI6wAejSpQtbtmyptv0JlUe8qn5yc61ugj780JpXCv75Txgxwvd24pW7EL/cgymvlFJpWusuZa0z8gKO1joZeBS4Vyk12sQ+hPNJTk4OWc3GjRuTmJjIpEmTSExM5P333/dZ/tChQ1x99dUllm3dupV7772XmJgYnnjiCfbt22ck1urAVKyh/B2oLl0T1Pa8+tKMiLDuTHbrZs1rDb/6Fbz3nuNh+IV4VbvPK1O6boo1GJh8m/stoAD4P4P7EIqRk5PjCk1/dK+44go+++wztmzZwpgxY0q8yJOVlcX06dNp27YtAwcO5P3330drbSxWEwQrr6GiaVLXBLU9rxVpNmhgVR49v//y82HoUPj4Y8dDqRDxqnafV6Z03RRrMDBWmdRanwFygKsrKisI5REXF8ebb75JRkYGf/nLX7j88su967TWpKSk+HzBRxCE6qFxY1izBtq0seZzc2HgQPjPf4IaliAI1YCRNpMASqk2wD4gW2vd0MhOXIyJNpN5eXmEh4eHvGZVdAsKCli1ahWvvvoqq1atAmDOnDkkJCSU0Pzxxx+56KKLHInVRPuTUMtrdWua0jXVVqi25zUQzb174cYb4cgRa75pU/jkE2jXrmQ58cpdscp10D2xBqPNZKBvc/u7w+bAHKw3v6XFbjGUUoOAQVFRUcyaNQuAbt260bRpU1JSUgBo3bo1/fr1IykpCYDw8HASEhJYvnw5R48eBWDo0KHs3buX7du3A9CjRw++/vpr7zjXsbGx9OrVi7l2vx2RkZGMGjWKJUuWkJWVBcDw4cPZsWMHO3fuBKB3797UrVuXdevWAdCuXTvOnDnD/v37AYiOjmbEiBEsXLjQ+2LMyJEj2bx5M3v27AGgT58+5Ofne8cb7dChAx07dvT2pdWoUSOGDRvGa6+95n2JJiEhgdTUVG8byP79+3Py5Ek2btwIQKdOnWjbti1Lly4FoFmzZrz//vu88MILfPTRR5w6dYrCwkLvsWqtmT59OlFRUXTp0oXOnTtz/fXXExMTw4oVKwBo1aoVAwcOJCkpicLCQsLCwkhMTGTlypUcPHgQgMGDB5ORkUF8fDyzZs1yzKcGDRqwePFioqOjHfOpa9euLFiwgOzsbGJiYhzzKTk5mcOHDxMdHV0pn4YMGcLcuXPJy8sDIDExkTVr1pCenk52djZ33XUXx44dY9OmTYB1J7qyPqWlpREfH8+2bdsc82n16tUAhIWFcc899zjmE1j9qz788MOO+eR5XHbZZZdRp04dx3wCaNq0KZdffrnfPv36103429/u4MQJOHYMbrjhJB98kIvW6aSlpQEQExPDgQMHHPUpNjaWgoICvv/+e8d8io6O9nZh5qRPCQkJvPXWWxQWFjrm04ABA9iwYQOnTp3yy6eKzico+v/kuQ465VOvXr34+9//TnR0tGM+ea57GRkZREdHO+ZTamoq27ZtIzo62jGfPNe97OxsfvaznznmU0pKCvHx8axatcoxn+YWH0e1PLTWfk/AmxVM/8Qav/sMVnvJc8DPAtlHbZkiIyO108ycOdMVmqZ0PZpr167VWD9kNKCbN2+uJ06cqPfv318p3bi4OCfD1Fq7M69u0DXhldaS18pofvqp1pGRWluv5Gh91VVaHz1atF68clesch10T6ymzi1giy6nThNom8kEYLT9WdY0ArgRqA8cAn6ptV4f4D4EoUrs2LGDevXqeeePHDnC1KlTadOmDcOGDWP9+vWeH0eCIBiiRw945x3wnIq7d8Ntt0Gx3r4EQagplFfLLGsCnq5gmgA8CPQC6gSiXdsmE3cmv/vuO1domtItrnno0CE9ZcoUfckll5S4S+mZrr76aj1jxgydnZ1doa6JX3luzWuo65r6RV7b81oVzcWLtQ4LK7pD2bu31jk54pUpXVOxynXQPbEG485kQG0mtdbPOlKDFYxw8uRJV2ia0i2u2aJFCyZOnMgf/vAH/v3vf/Pqq6/y0Ucfedd/9dVXPPjggxw6dIgpU6Y4HksgsYa6rptiNUVtz2tVNIcNg5kzYaw9fMWGDTB8OGhdx6HoShJqx1/durX9vDKl66ZYg4HJfiaFasbTIDjUNU3plqVZr149hg4dyvr169mxYwcPPPAAF154IWC9VDF27PkDNHkaxJvE7XkNZV0T1Pa8VlUzMRH+/Oei+Xffhe++exoTp1ooHn916tb288qUrptiDQZVrkwqpeoqpZrak5G3wwXBCTp06MCMGTM4cOAAr7zyCr/97W9p3bp1iTLfffcdsbGxPPfcc9433gRBqDpPPAFPPlk0n5X1c37zG+vhtyAI7qZSlUml1EVKqfFKqa1YHZMftqccpdRWpdSTSilnOvkT/KZTp06u0DSl669mdHQ0Dz30EC+++OJ561577TW+//57JkyYwKWXXsqoUaM4deqU4y/s1MS8hoquCWp7Xp3SfO45GDeuaP7VV+GZZxyR9hLKx18durX9vDKl66ZYg0HAlUml1I3ALmAKcC1WX5XKnuray6YBu5RSPZ0KVKiYtm3bukLTlG5VNbXWrFy50jufl5fHggUL+Prrr+nSpQtvvvkmZ86cqWqYQO3Ka3XrmqC259UpTaVgxgyrzaSHyZPhr391RB4I7eOvDt3afl6Z0nVTrMEgoMqkUuoKYBXQAsgCngP6Ax3tqZ+9LNMus8reRqgGPJ2mhrqmKd2qaiql2Lp1K8nJyXTv3r3Euq1bt3LfffcRExPDE0884e0QurLUprxWt64JantendSsUwfeeguioz/1Lnv0UZg3zxn9UD9+07q1/bwypeumWINBoHcmnwUigTTgKq31RK31B1rrr+xprdZ6ItDeLhOF1WWQILiC+vXrM3LkSD777DPS0tK49957UUp512dlZTF9+nTatm3Lf//73yBGKgjuJTwcfvKT39Oz2LOr++6D5ctLlisstCqZ1fBOnCAIVSDQymQfrH767tNaHy+vkNY6C7jPnu1bydiEAGnWrJkrNE3pOq3ZuXNn3njjDa655hr+8pe/cPnll3vXdejQgZ/+9KeV1q7NeTWta4LanlcTmmFhZ1m5EjxNxgoKrMff9mh5FBZab4EnJFif/lYo3XL8pnRr+3llStdNsQYDFciLBUqpHOCs1vpiP8ufAOppraMqF17NJSoqSp8+fTrYYQh+0KVLF7Zs2UJBQQGrVq1ixowZDBkyhPvvv79Eufnz57NhwwYefPBBrrvuuiBFW7vxeCWEPh6vjhyBG2+EvXut5VFRsHYtzJoFc+YUlR8zBpKSIEw6tAsKcm65B1NeKaXStNZdyloX6Gn5PRChlKqwt1m7TH0gPcB9CJXEr8HYQ0DTlK6pWD3UqVOHAQMG8N57753XP6XWmhdffJE33niDzp0707NnT95++23y8vKqNVY35dW0X05S2/Nq0qvmza3K4yWXWPOnT0Pv3iUrkmDN+3OH0k3H76ZYTSB5dZdfvgi0MvkvIBwYXlFBu0x9YGGgQQmVo7yKS6hpmtI1FWtZFG9HCZCWlsb27du98xs3buTuu++mdevWTJo0iYyMjBLlJa/V61dVqe15Ne3VZZfBBx9Ao0ae/ZVdzp8KpZuO302xmkDy6i6/fBFoZfI5YBMwUyk1orxCSqnhwEzgc+BPlQ9PENxBXFwcn3zyCSNGjKBu3aK++48cOcLUqVNp06YNw4YNY/369Y73WSkINYErr6TECznl4e8dSkEQqo9A20w+hXW38QEgGuux9wbggF2kFdAbaAP8CLwGnC1LS2s9ubJB1wRMtJksLCwkzOEGRSY0TemaijXQ9ieHDx9m9uzZzJw5kwMHDpy3vmfPnmzYsIE6dZwfm9hNeTWha6qtUG3Pq2mvPC/blH607Yvy2lC65fhN6YbKddAfJK/uug462WbyGeBJ4CKsTsrbAKOBP9pTAnC5va6hXfbpUtMzSHdBRlizZo0rNE3pmoo1UFq0aMGkSZP49ttvWbx4MTfddFOJ9ddffz0ffPCBkX27Ka+h4pc/1Pa8mvZq/vzAKpJglZ8///zlbjp+N8VqAsmru/zyRaBjab+F1TWQUEmUUoOAQVFRUcyaNQuAbt260bRpU1JSUgBo3bo1/fr1IykpCYDw8HASEhJYvny5d7zooUOHsnfvXm87vR49evC///2P9HTrfafY2Fh69erlbdwbGRnJqFGjWLJkCVlZWQAMHz6cHTt2sHPnTgB69+5N3bp1WWf3zdGuXTu+/fZbb5zR0dGMGDGChQsXkp2dDcDIkSPZvHkze/bsAaBPnz7k5+ezYcMGwOpCp2PHjixatAiARo0aMWzYMLZt2+aNNSEhgdTUVG9H4P379+fkyZNs3LgRsIabatu2rbdz12bNmjFkyBDmzp3rbW+SmJjIjh07vJoDBgzg2LFjbNq0CbAeQ8fExLBixQoAWrVqxcCBA0lKSvL+MkxMTGTlypUcPHgQgMGDB5ORkUF8fDyzZs2qtE+XXnop77//Ph999BFbtmzhF7/4BZ9++inp6elen+68805atmxJ9+7dGT16dMA+de3alQULFpCZmcmJEycc8yk5OZn09HTS09Md82nNmjWkp6eTmZnJT3/6U8d8SktLIz4+nm3btjlyPjVo0IDVq1cDkJ2dzc0331zl88njE1h9lt56662O+ZSTkwNYdzrWrl3rmE9gtevatm2bYz4BxMTEcODAAVJSUigshFtu+TkffBCDv/To8TXXXXeO778v8ik2NpbvvvvOe81ywqfo6Giys7Md9ykhIaHENcsJnwYMGMDu3bu980745Lnuea6DTpxPnuve1q1bSU9Pd8wnz3Vv3759pKenO+ZTamqqN1anfPJc9zIzM2natKljPqWkpBAfH8+qVasc88mvl4S01jIFYYqMjNROM3PmTFdomtI1FWtcXJxjWjk5OVrrkrFmZGToOnXqaEBfeumleurUqfrIkSOV0ndTXk3oOulVcWp7XqvDq4ICrceM0RoqnsaMscpXV6y13SutzZxbkld3XQeBLbqcOk3QK1W1dTJRmczIyHCFpildU7GaODGLx/rUU09prDv+3ik8PFyPHDlSb9y4URcWFlZK10Ssoa5r6iJa2/NaXV75U6H81a/Kr0iairW2e6W1+etgqOu6KdZgVCal+9caxLFjx1yhaUrXVKwmKB7rfffdx/jx42natKl3WV5eHgsWLKBHjx506dKFN998kzNnzgSkayJWN+iaoLbntbq8CguzXqoZM6b8MlFRvjsud9PxuylWE0he3eWXLypdmVRK3aSU+odS6nOl1Df29Lm97CbnQhT8xdOeKdQ1TemaitUExWNt3bo106ZNY//+/cyfP5/rr7++RNmtW7dy3333cckll1TYdsVNeXWrX6GsaUq3Or2qqEL5+uuweHH527vp+N0Uqwkkr+7yyxcBVyaVUk2UUquBdcA4oBvWG9yX23+PA9YppVYppZo4Gawg1GTq16/PqFGj+Pzzz9myZQtjxowhIiLCu/6HH36gZcuWQYxQEKqH0hXKhAQYOrRofWIi2O8XCYIQAgRUmVRKhQMfAH2xuv/5HJgG/J89TbOXKeAWYI29jVANxMXFuULTlK6pWE1QUaxxcXG8+eabZGRk8Oc//5nLL7+ctm3bcsstt5Qod/bsWf7+97/zww8/+KVrItZQ0zVBbc9rMLzyVCjnzoU33rD+vvxya112NowYUfZIOW46fjfFagLJq7v88kWgdyYfAjoBPwD9tdY9tdaTtNYz7WmS1roncCtwwi77oKMRC+USE+N/txrB1DSlaypWE/gba+PGjXniiSf43//+xwcffHBe57aLFy/mN7/5DZdccgljx47lxIkTQYs1VHRN4Kbvq5tirYiwMBg92vps2BAWLYJ69ax1mzfDH/94/jZuOn43xWoCyau7/PJFoJXJ4Vhvm96vtS6312Wt9Rrgfqw7lOUOuyg4i6ePqlDXNKVrKlYTBBprnTp1aNOmzXnLZ8yYAcCZM2dISkqib9++9OzZk7ffftuxMV/d9B0whZu+r26KNVC6doXnny+af+klWLmyZBk3Hb+bYjWB5NVdfvki0MrklUAusMyPssvsslcFGpQgCBWjtea+++6jU6dOJZZv3LiRu+++m9atWzNp0iQyMjKCFKEgOM+jj8LAgUXzo0eDfMUFIbgEWpmsB5yz+xvyida6EDhH4KPsCJWkVatWrtA0pWsqVhM4EatSisTERP7zn//wySefcNddd5UY7/vIkSNMnTqVNm3aMGzYMPbv3x+0WKtT1wRu+r66KdbKoJTVjtLzdDArC+66C/LzrXk3Hb+bYjWB5NVdfvlC+VEvLCqs1C6gHdBVa721grJxwGbga611+ypFWQOJiorSp0+fDnYYgh906dKFLVu2BDsMvzh8+DCzZ89m5syZHDhwwLv8wgsv5MCBA0RHRwcxOvO4yavaTlW9+vhjuOkmKCy05idMgKlTnYlNOB85t9yDKa+UUmla6y5lrQv0zuR7WO0g31BKNS2vkFKqOfAGVvvKlAD3IVQSzxicoa5pStdUrCYwFevKlSuZNGkS3377LYsXL+amm24C4J577jmvIrl37152795doaabvgOmcNP31U2xVoX4eJg8uWj+uedg7Vp3Hb+bYjWB5NVdfvki0MrkC0AWcA2wWyn1vFLqVqXUT5VSXZVSQ5VSrwLf2GV+AP7sbMhCeRR6fqKHuKYpXVOxmsB0XuvVq8ewYcNYv349X375Jb///e/PKzt58mTat29P3759WbZsGfme54TVHKsbcNP31U2xVpUnn4S+fa2/tYZRo+CHH+o7vh/xygySV3f55YuAKpNa66PAz4EjwMXAE1h3Hrdh9S/5L6z+JiOBQ8Bt9jZCNVC625hQ1TSlaypWE1RnXjt27Mhll11WYtmxY8dYtGgRAOvWreOOO+4gNjaW5557jqNHS56ybvoOmMJN31c3xVpV6tSB+fOhWTNr/sgRmDPnZpz+/yxemUHy6i6/fBFQm0nvRko1BB4GhgIdKaqUFgI7gCXAq1rrE45EWQORNpPuoSa2Fdq3bx+/+93vWLFixXm/jMPDw7nzzjt58MEHuf7661FKBSnKwKmJXtVUnPRq7Vro18+6OwkwbRqMH++ItGAj55Z7cEObSQC01ie01lO01tcCFwAt7ekCrfW1WuupUpGsflaW7nAtRDVN6ZqK1QTBzmtsbCzvvPMO3377LePHj6dp06Im0Hl5eSQnJ3PDDTfQpUsXfvvb31KZH51OxRoKuOn76qZYnaJv35KVx0mTrBd0nEK8MoPk1V1++aLK91e11ue01kfs6ZwTQQmV4+DBg67QNKVrKlYThEpeW7duzbRp09i/fz/z58+ne/fuJdZv3bqVNWvWGLk7Wdv9CpXvQLA0neaZZ+DGG62/Cwvh7rvh+HFntMUrM0he3eWXL6QPyGpGKTUIGBQVFcWsWbMA6NatG02bNiUlxXrxvXXr1vTr18/7lld4eDgJCQksX77c255t6NCh7N27l+3btwPQo0cPzp4969WMjY2lV69ezJ07F4DIyEhGjRrFkiVLyMrKAmD48OHs2LGDnTt3AtC7d2/q1q3LunXrAGjXrh0FBQVezejoaEaMGMHChQvJzs4GYOTIkWzevJk9e/YA0KdPH/Lz89mwYQMAHTp0oGPHjt72eY0aNWLYsGFkZWV5dRMSEkhNTWXfvn0A9O/fn5MnT7Jx40YAOnXqRNu2bVm6dCkAzZo1Y8iQIcydO9c7yktiYiLZ2dlezQEDBnDs2DE2bdoEWOOfxsTEeEcbaNWqFQMHDiQpKYnCwkLCwsJITExk5cqV3pN78ODBZGRkEB8fz6xZsxzzqUGDBmRmZjJr1izHfOratSsLFiwgMzOThQsXVsqnQYMGkZOTw80338xnn33GZ599Rm5uLp07d2bWrFklfMrMzOSuu+7i9OnTAfu0Zs0a0tPTyczM5MCBA475lJaWRnx8PNu2bXPMp9WrVwOQnZ1NXl6eYz4B3u2cOJ+Sk5PJyckBrAb9a9eurfL55PEJrLvV27Ztc8wnsIaRO3DggKM+/d//tec//+nK6dMRZGRA374ZbN0aw9KllffJ0wuC0z4lJCSUuGY54dOAAQPIycnxajrhk+e657kOOuGT57rnuQ46cT4V///k0XXKp9TUVK+mUz55rnuZmZmkpaU55lNKSgrx8fGsWrXKMZ881z1fVLbNZBjQA6u95MVYnZmXi9Z6sq/1tRETbSaPHDlC8+bNQ17TlK6pWE20P3FDXo8fP84///lPhg4dWqJj3XPnztGmTRsiIyP5v//7P8aMGcPFF18c1Fg9mGor5Kbvq1tiNeXV/Pk/cM89Rd/Hl16yRs2pCrXdK6i910GTmqZ0XdFmUil1O5AObABmAFOBp8uZnrE/hWrAxLB5pobic1OsJnBDXhs3bszDDz/MoUOHSixfsWIFBw8eZO/evTz++ONccskljB07lm3btgUtVtO46fvqplhNcPXV+3jssaL5P/wBNm+umqZ4ZQbJq7v88kVAlUmlVF9gMdAKa6jET7G6A3qrnGme/SlUA57b3aGuaUrXVKwmcHNejx8/TsOGDb3zZ86cISkpieuuu46ePXvy9ttvex/vBKIbyrjp++qmWE2QlpbGn/4EXbta8+fOwfDh8OOPVdM0gXgleXWTX74I9M7keHubDcDlWuteWuu7tNZjfE3Ohy0IQrAYN24cBw4cYPbs2Vx77bUl1m3cuJG7776b1q1bM2nSpBrzq1twF+HhsHAheAZ9+vZbSEws6jpIEARnCbQyGYc1RGKC1vpQRYWF6qVbt26u0DSlaypWE7g9r5GRkSQmJrJ161Y++eQT7rrrLurVK2o6feTIEaZOncrChQurNVZTuOn76qZYTeCJNTYWio9Ut2QJzJxZNU2nEa8kr27yyxeBViYVkK21/t5EMELVKN5XYChrmtI1FasJakpelVL07NmTf/7zn6SnpzN58mQuueQSACIiIrj33nvP28bzlmRt96umfAdCjeKx/vKX8OtfF6377W/hv/+tmqaTiFeSVzf55YtAK5O7gCilVISJYISq4eliI9Q1TemaitUENTGvLVq0YNKkSXz77bcsWbKEqVOn0qhRoxJltm/fTosWLXj44Ye9XVa4ATd9X90UqwlKx/rSS3DNNdbfZ8/CnXfCqVNV03QK8Ury6ia/fBFoZfIfWH1T/spALIIg1ADq1avH0KFDefzxx89bN2PGDE6ePMmrr77KM888Q9++fVm2bBn5+flBiFSoDVxwASxaBFFR1vzXX8ODDwY3JkGoaQRUmdRazwPeAP6qlBphJiShsrRu3doVmqZ0TcVqgtqY18LCQrZu3Vpi2bp167jjjjuIjY3lueee83amG2qEcl6rQ9ft59ZVV8E//lE0/9ZbMG9e1TSdQLySvLrJL1+U22m5UupNH9sNAS7C6m9yC3DSR1mttb6vsgHWVEx0Wu7pKT/UNU3pmorVRAewtTWvWms+/PBDZsyYwYoVKygsLCyxPjw8nDvvvJMHH3yQ66+/PuBhHE111hvqeTWta0IzGF4lJBRVIiMjIS3NqmhWRbMquMUrkOugm2INtU7LE4DR9mfpqSHWyziXAUPLKVN8EqoBE23QTLVrc1OsJqiteVVK0adPH9555x2mTZvG+PHjSzRAz8vLIzk5mRtuuIGvv/7asf1WlVDPq2ndmnJuvfpqUeUxJ8dqP3nmTNU0q4J4JXl1k1++8DU297PVFoUgCLWORo0a8eSTT/LUU0+xePFiZsyYweeffw5AfHw8V5W6ZaS1DvhOpSAU58ILrfaT3bpZL+N8+SU89hi89lqwIxMEd1NuZVJrHbTKpFKqMXA7MAD4KXAJkAd8CcwB5mitC8vYrgcwEegOXAD8D3gTeEVrXVDOvgYCvwOuA+oAO4F/2O1Dy4tvNPAgcDVQAPwHmK61XlmZ43WK8PBwV2ia0jUVqwkkr0W69evXZ9SoUYwaNYq0tDRmzJjBwIEDzyv/0ksvsX79eh588EH69+9v5JFTRbGGuqYp3Zp0bl1zDfztb0VdBr3+OvzsZ9ZdyspqVhbxSvLqJr98UW6byWCilPo18BpwCFiP1TazOXAHVlvNpcAvdbHglVKD7eW5wCIgCxgEXAks0Vr/soz9PAS8Ahy3t8kDhgExwIta69+Vsc104HEgA1gChAMjgEbAw1rrV/05RhNtJgUzmGp/IvhPQUEBV1xxBd9++y0AP/nJT3jggQcYM2YMF198sbeceOUegumV1jBiBPzrX9Z8dDT85z9WR+dC2ci55R5Crc1kMNkD/AKI0VqP1Fr/UWt9L3AVsB+rneYdnsJKqWhgNtZdwpu01vdprZ8ArgU+A4aVfvtcKdUGmI5V6eyitX5Qa/0ocA3wDfC4UuqGUtv0wKpIfgNco7V+VGv9INbIQFnAdFs3KCxfvtwVmqZ0TcVqAslrYLqbN2/2ViQBvvnmGx5//HEuueQSxo4dy7Zt25wPsBg1Na/B1DSFP7EqBbNmFVUes7Ot8bvLG1JevDKD5NVdfvkioMqkUqpXgNP1SqkrlVK+2maeh9b6Q631u6UfZWutDwOv27M3FVs1DGgKLNRabylWPhfrsTfA/5Xazb1AfeBVrfV3xbb5AXjOnv11qW0889Pscp5tvgNm2HpBG4vcRLcqprpqcVOsJpC8BqbbvXt3/ve///HYY4/RsGFD7/IzZ86QlJTEddddR8+ePcnKyiKvvBpBNcUaTE1TujXx3LroImv8bs8ooFu2wJNPVk0zUMQryaub/PJFoHcmP8J67OzvtBH4CjiplFqllLrFgZjP2Z/Fezm+2f5cVUb5VCAH6KGUqu/nNu+XKlOVbQRBcIC2bdvy4osvcuDAAWbPns21115bYv3GjRv59ttv6dmzZ3ACFFxH167w5z8Xzb/8Mrz7bvDiEQTXorX2ewK+A74FTgGF9nQWOGhPZ4stP2WXzyq2rAB4JpB9ltp/XayXcDTQv9jyzfayuHK222Gvb19s2TF7WeNytjllr4+056Ps+ZPllG9irz/iz7FERkZqp8nMzHSFpildU7HGxcU5ril5rbpuYWGh/uSTT/Rdd92l69Wrp+3zT0+ZMsWhCIuoTXmtLk0T55XWgcdaWKj1oEFaWy0ptW7USOv09Kpp+otbvNJaroNuitXUuQVs0eXUaQJ9/NxGKfUI8BfgPeAF4HOt9TkApVQ94HrgD8AtwJ+11v9QSsXay8YCk5RS67XWGwLZt83zQEfgPa316mLLL7I/fyxnO8/yhgFuE2WXy6nkPkqglLofuN/+my5dymzHWmlOnz5NlGfMsBDWNKVrKtZdu3a5witTum6ItX379mRmZnLo0CEWL158Xjuk9PR0IiIiaNy4MXXq1AlqrCY1Tema0DRxXkHlYs3Pv4h69RZw7lwLsrKgffttXHnlOJQqqLSmqViDoQlyHXRTrKbOLZ+UV8ssa8LqqqcAmOVH2VlYj6L7FVv2D6w7lIsC2a+97SNYdx52AY1Krdtjr2tbzraf2utvKLYsz15Wt5xtDtjrW9rzrez5jHLK17PXn/XneEzcmZw5c6YrNE3pmorVxK88yasZ3c6dO5+37KuvvvLetWzQoIF+6KGH9FdffVXm9pmZmXr27Nl68uTJevbs2d67BrU9ryY0Td09qWysH3+sdZ06RXcox4+vumZFuMUrreU66KZYg3FnMtA2k4/bF+XxfpQdjzVKzhPFlnlap/QIZKd2Fz5/w2p/+TOtdVapIp67ghdRNp7lJyqxzY+lPgPZhyAI1UhZnZrPmjXL+/fJkyd59dVXufrqq+nTpw/Lli0jPz8frTVTp04lJiaGsWPH8tRTTzF27FhiYmKYOnWq5wejUIO58UaYPLlo/k9/gg8+CF48guAmAq1MdgJ+1FpnVlTQLnMC6Fxs2XdY43g3LXur81FK/RarL8gdWBXJw2UU84y71q6M7esCl2PdJd3n5zYtsR5xZ2itc+zYT2PdrbzQXl+aK+zPPRUckjF69Aiojh40TVO6pmI1geS1+vx69tlneeWVV84bUefDDz/kjjvuIDY2ln79+jFp0iRyc3NLlMnNzWXSpEns2LHD8bjclNfacm49+STcYr8mqjWMGgWHD4tXppC8ussvXwRamawPRCulGlRU0C4TjdWpd+l95vizM6XUH4CXgW1YFcny3qH/0P68tYx1vYBIYKPW+qyf29xWqkxVtqk2GjSo0JaQ0DSlaypWE0heq8+v6OhoHnroIb766ivWrl3L7bffXmL0nP3797N27VqfGrNnz+b48eOOxuWmvNaWcyssDObPh+bNrfmjR60KZWSkeGUCOQfc5ZcvAq1M7rC3+b0fZZ+gaHhCAJRSjbDu+B2paGOl1CSsF27SgD4V3A1dAmQCI5RS3lanSqkIYKo9W3r01TlYb58/VLyjcaXUxRQ9xn+91Dae+Ql2Oc82bbCGVzxr6waF1atXV1woBDRN6ZqK1QSS1+r3SylFnz59eOedd/j2228ZP348TZv695Dk7NmzLFu2zNF43JTX2nRuNW8OCxZYHZsDrFsHjz1mpi9A8UrOATf55YtAK5OvYbWDHK+Ues1+S7sESqnLlVIzgAlY7SuLV+J+Zn/+x9dO7LGvJ2O97PMx8IhS6plSU4KnvNY6G+tN8TrAR0qpJKXUn7HuaN6AVdlcVHwfWutvsSq8jYAtSqkZSqmXgf8CP8EaTvGzUttsBF6y1/9XKfWyfaxbbJ3f6WIdoAuCEJq0bt2aadOmsX//foYOHerXNocOHTIclRAq9OkDEyYUzf/733F8/HHw4hGEUCfQroHmKaVuBO7D6uLmfqXUYawxtAFaAi3svxXwptZ6XjGJAVjtDldUsKvL7c86wG/LKbMBmFsstuVKqd5YldihQASwF3gM+LsuowW91voVpdR3wO+Ae7Aq118BE0vFXXybx5VSX2Ldibwf6+30rcBftNYrKzguo8QaGFjWhKYpXVOxmkDyGhp+1a9fn1tvvZWlS5dWWLZZs2aO7ttNeQ0Fr/zFqViffho2bICPPwatw7jrLti2DZo0cUQeEK/kHHCXX75QlXlLUSk1BpgEtCmnyPfAFK31m5UPrWYTFRWlT58+7ahmXl4e4eGlm6iGnqYpXVOxdunShS1btlRcMAAkr2Z0K+PV8ePHiYmJOe/lm9JceeWVvPLKK9xyixMDebkrr6HilT84GWtGBlx7LXiayw4YYI2QU0anAZXCLV6BXAfdFKupc0splaa1LrMDy0AfcwOgtZ6jtY4FumA9Xv6jPY21l8VKRbL6mTt3ris0TemaitUEktfQ8atx48ZMKP5Msxy+/vpr+vXrx5AhQ/jmm2+qvF835TVUvPIHJ2ONiYF5xZ5RpaRYQy46hXg11zW6boo1GFSqMulBa71Va/2G1voFe3rDXiadsgmC4BomTJjAlClTiIiIKLE8IiKCfv36lbhzsGLFCq6++mrGjx/PqVOnqjtUoZoZMABuuWW7d/4Pf4BNm4IYkCCEIFWqTAqhRWRkpCs0TemaitUEktfQ8kspxcSJE8nIyGD27NlMnjyZ2bNnk5GRwerVq3nxxRcZPXq0t3xeXh5/+tOfuPLKK0lOTq5Up+ZuymsoeVURJmK9++6ddOtm/Z2fD8OHw4kTVdcVr+QccJNfvqhUm0mh6phoMymYwVT7E8F5THv1xRdf8Mgjj7Cp1K2p6dOn8/jjjxvbb03EbefVt9/CddfBj/ZYaMOGwb/+5Vz7yVDHbX7VZkK+zaRSqqASU74zhyFUxJIlS1yhaUrXVKwmkLy606/rr7+ezz77jLlz59KihdVxRePGjbn33nsrrek0bvoOmMDU8V9+OSQlFV8Gr5fuibgSuk5T270ypeumWINBoI+5VSUmeZReTWRllR6yPDQ1TemaitUEklf3+hUWFsbo0aPZs2cPv//973nhhRe4+OKLS5T/8ccfycvL81vTVKyhrGkKk8c/bBg88EDR8kcftboLqqquk9R2r0zpuinWYBBoRe/yCqZrsd7o3glkAYMp6jNSEAShxtCgQQNeeOEF7rvvvvPWPfLII/z0pz/l/fffD0JkgklefBE6dbL+PnvWaj8p72EJtR6tteMT1njc64CjwKUm9uH2KTIyUjvNiRMnXKFpStdUrHFxcY5rSl7N6JrwSuvAYv3888811uhfGtADBgzQe/bsqZJmILjlOxAKXlVWc/duraOitAZrGjVK68LCqus6gVwH3ZVXN51bwBZdTp3GyCNorXUe8AjQBHjaxD6E89mxY4crNE3pmorVBJLXmuvXd999R3R0tHc+JSWFDh068Pvf/57s7OxKaQaCm74DJqiO47/yypLtJZOTS/ZHWVldJ6jtXpnSdVOswSCg4RQDQWu9UymVDdxqah9uRCk1CBgUFRXFrFmzAOjWrRtNmzYlJSUFsMYN7tevH0l2a+/w8HASEhJYvnw5R48eBWDo0KHs3buX7dut/s969OjB1q1b2blzJ2AN0dSrVy9vh6iRkZGMGjWKJUuWeNtoDB8+nB07dni36d27N3Xr1mXdunUAtGvXjl27dnnXR0dHM2LECBYuXOj9pzhy5Eg2b97Mnj17AOjTpw/5+fls2LABgA4dOtCxY0cWLbKGRm/UqBHDhg3j448/9uomJCSQmprKvn37AOjfvz8nT55k48aNAHTq1Im2bdt6h7xr1qwZQ4YMYe7cud52aYmJiXz22WdezQEDBnDs2DHvW7dxcXHExMSwYoU1kmerVq0YOHAgSUlJFBYWEhYWRmJiIitXruTgwYMADB48mIyMDOLj45k1a5ZjPjVo0IANGzawc+dOx3zq2rUrCxYsIDMzk/379zvmU3JyMunp6ezcudMxn9asWUN6ejqZmZm0adPGMZ/S0tKIj49n27Ztjvm0evVqALKzs+natatfPv34449MmjSJL774gqVLl6K15ty5c/zlL39h3rx53HbbbXTv3p0TJ07Qs2dPx3zKyckBoLCwkDNnzjjmE1hdIUVFRTnmE0BMTAwHDhxw1KfY2Fj27t3rPVeqej6Bdd3Lzs5m//79JXyKidnMDTe05LPPrgTggQcK+f77JbRsecIvnxISEkpcs5zwacCAAWzevNmr6YRPnuue5zrolE+9evXyXged8slz3du3bx87d+505HzyXPc2bdrEzp07HfPJc93LzMwkIiLCMZ9SUlKIj49n1apVjvnkV8fq5d2yrOqE9ag7Dzhjah9unkw85p45c6YrNE3pmorVxCMDyasZXVOPdyob65YtW3SPHj1KPPYGdLdu3fSTTz7pcJQWbvkOhJpXldE8dUrrq64qetzdsaPWOTlV160Kch10V17ddG5R3Y+5be7GuvN50OA+hGL07t3bFZqmdE3FagLJa+3wKy4ujk8++YTk5GRatWrlXb5p0yaef/55710FJ3HTd8AE1Xn8UVFWX5OegZN27IDf/rbqulWhtntlStdNsQaDQPuZbF3B1E4pdbNS6m/AP7B+hS82ErlwHnXrOt9qwYSmKV1TsZpA8lp7/FJKMXLkSL7++mvGjx9P/fr1AejZsyedO3d2KkQvbvoOmKC6j/+nP4W//a1oftYssJ+aVkm3stR2r0zpuinWYBDonclvK5h2AR8ADwERwCZgilPBCr7xtCUJdU1TuqZiNYHktfb5deGFFzJt2jS++uorhg0bRt++fVGlhk85duyYp5lQpXHTd8AEwTj+sWOtLoKKz3/zTdV1K0Nt98qUrptiDQZOd1peiNW/5AbgASBeay1jBgqCINjExsayePHiEo+9Ac6ePcsNN9zAbbfdxu7du4MUnVAZlLLuSP7kJ9b8yZNW5fLs2eDGJQjVRUCVSa11WAVTPa11U631zVrr17XWMpRiNdKuXTtXaJrSNRWrCSSv4ldpzZdffplvvvmG1atX89Of/pTHHnuMEydOVFnXCcSrijWjo2HhQqhXz5pPS4Mnn6y6bqDUdq9M6bop1mAgQx3WILp27eoKTVO6pmI1geRV/Cqtefz4ccLCrEtyfn4+L7/8Mu3atSMpKYmCgoJK6zqBeOWfZpcu8Je/FM3/9a/w739XXTcQartXpnTdFGswkMpkDcLT31aoa5rSNRWrCSSv4ldpzb/85S+kpaXRq1cv77Jjx44xduxYunXrxqefflopXScQr/zXfOQR+MUviuYTEsDuXrBKuv5S270ypeumWINBpSuTSqkLlVJ3KqWeV0q9YU/P28sudDJIQRCE2sC1117LRx99xKJFi7j00ku9y7du3cqNN97I3XffTUZGRhAjFCpCKZgzBzz2/fAD3HUXnDsX3LgEwSQBVyaVxXjgAPA28ASQYE9P2MsOKKWeVKVfVRSMUnwIt1DWNKVrKlYTSF7Fr/I0lVLceeed7N69m6eeeooITyeGwNtvv02XLl3Izc0NiVhDkVA4/kaN4O23oU4da37jRni6jIGFQyHWYOKma4ubYg0GKtBuKJRS84BRWG9v5wJpgOencgwQh9UtkAaStdajHYu2BhEVFaVPn5YX3d1Aly5d2LJlS7DDEPygJnr1/fff88QTT7B4sdVl75/+9CeerOjNDhdQE70qzfPPwx//WDS/ejX06xe8eKpCbfCrpmDKK6VUmta6S1nrAu20/A7gV/bsn4AWWut4rfVd9hQPtACet8uMUkrdXtnAhcBYuHChKzRN6ZqK1QSSV/HLX83LLruMf/3rX6xfv54hQ4bw6KOPnlfm2LFjAesGgnhVOc3f/75k5fFXv4JDh6qu64va7pUpXTfFGgwCfcx9P9Ydx4la6wla6+zSBbTW2Vrr8cAkrLuX91c9TMEfsrPPsyMkNU3pmorVBJJX8StQzZtuuolly5Z5R9Dx8NVXX9G6dWseeeQRfvjhh5CINZiE0vGHhcFbb0GLFtb80aMwahR4Xs4PpViDgZuuLW6KNRgEWpmMAwqAv1VU0C5TAJR5S1QQBEGoGlprfvOb35Cbm8srr7zCFVdcwYYNGwLqSkgwS/PmsGCB9WIOwIcfwnPPBTcmQXCagNpMKqVygdNa68Z+lj8ORGmtIyosXMsw0Wby9OnTREVFhbymKV1TsZpofyJ5NaNrqq1QqOb15MmTDBkyhA8//LDE8muuuYa///3v9O7du0r6HsSrqms+/TRMnmz9HRYG69dDXFxoxloWch10T6wh32YSOAo0VEq1qqigUuoSoCFwrIKigkNs3rzZFZqmdE3FagLJq/jlhGaDBg1Yu3YtS5Ys4bLLLvMu/+9//8tNN93EnXfeyffff1/l/YhXVdecNAk8XYgWFlrdBX3wwbYq65amtntlStdNsQaDQCuTqfbnS350+/OS/flRgPsQKsmePXtcoWlK11SsJpC8il9OaSqlGDp0KLt27WLKlCmEh4d71y1evJirrrqKZ555hpycnErvQ7yqumbduvDPf0Jj+7newYMwadKlFBZWWboEtd0rU7puijUYBPqY+1pgC9aLNZ9gvdGdqrXOsdc3Bn4G/AHoDBQCXbXW2xyN2sUopQYBg6Kiosa+9JJV3+7WrRtNmzYlJSUFgNatW9OvXz+SkpIACA8PJyEhgeXLl3P06FEAhg4dyt69e9m+fTsAPXr0YP369d7G+bGxsfTq1Yu5c+cCEBkZyahRo1iyZAlZWVkADB8+nB07drBz504AevfuTd26dVm3bh1gjRm6a9cu6tidpUVHRzNixAgWLlzobTQ8cuRINm/e7D0h+vTpQ35+Phs2bACgQ4cOdOzYkUWLFgHQqFEjhg0bxvPPP0+jRo0ASEhIIDU1lX379gHQv39/Tp48ycaNGwHo1KkTbdu2ZenSpQA0a9aMIUOGMHfuXPLy8gBITEzkpZde8vbZNWDAAI4dO8amTZsAiIuLIyYmhhUrVgDQqlUrBg4cSFJSEoWFhYSFhZGYmMjKlSs5ePAgAIMHDyYjI4Pk5GTat2/vmE8NGjRgwYIFNGnSxDGfunbtyoIFC8jMzCQ2NtYxn5KTk0lPT6dJkyaO+bRmzRrS09PJzMxk9OjRjvmUlpbGrl27GD16tGM+rV69GrAayT/yyCOO+QSQlZXFk08+6ZhPnspiZmYma9euZf369RTnuuuu49e//nXAPgHk5eVx4403OuYTwHvvvceMGTMc9Sk2Npa9e/d6h6V0wqfo6Giys7O9n1X16YsvGvPqq7d5fRk27DNuueXLKp9PYF33Fi9eTGRkpGM+ea5706dPp3379o751KtXL6ZPn06TJk0c88lz3du3bx9NmjRx5HzyXPc2bdpEkyZNHPPJc93LzMykf//+jvmUkpLCrl276N+/v2M+ea5748aNK/cxN1rrgCast7PzsV6uKbD/Pg6cKras0F5+f6D6tWWKjIzUTrN3715XaJrSNRVrXFyc45qSVzO6JrzS2p15/fjjj/V1112nsXrg0J9//nmVNZ2kNnv1u99pDdZUt67WHmsKCrSeO9f6rCxyHXTHd8CkrqlzC9iiy6nTBDwCjtZ6FtCLosfXYcDFQCTWHUuAD4F4u6xQTeTn57tC05SuqVhNIHkVv0zn9cYbb2Tz5s3MmjWLxx57jOuvv75EuYKCAo4fPx6Qphtwg1fTpoHHjvx8GDECsrIgMdEayzsxkUo//q7tXpnSdVOswaBSY3NrrTdqrfsATYC+wF321BdoorXuq7X+zLkwBX/w3LoPdU1TuqZiNYHkVfyqjrzWqVOHsWPH8uKLL55XLikpibZt2/LKK69U+A9NvHJWMzwcFi6ECy44C8B330HnztaY3mB9VrZCWdu9MqXrpliDQaAj4LyjlFqqlLocQGv9g9b6Q631Inv6UGv9g5lQBUEQBCf44YcfmDBhAidOnOCRRx7h2muv9bZFE6qHNm3gnnuKKhKlX7qvSoVSEKqbQO9MDgRu1Vp/ayIYoWp06NDBFZqmdE3FagLJq/gVzLymp6fTsGFD7/zOnTvp27cvd9xxB99+e/7lXbwyc/wjR0Zx1VXlr69MhbK2e2VK102xBoNAK5OHgXMmAhGqTseOHV2haUrXVKwmkLyKX8HMa6dOndi5cyfPP/88F154oXf5smXLaN++PRMnTqT4oArilfOahYUwc2ZXdu/2XS7QCmVt98qUrptiDQaBVibXAw2UUu1NBCNUDU/3BqGuaUrXVKwmkLyKX8HOa/369fnDH/7A119/zT333ONdfvbsWaZNm8aVV17J22+/jdZavHJYs7DQqiAuWBBecWECq1DWdq9M6bop1mAQaGXyeeAM8KpSqr6BeARBEIRqpFWrVsybN4/PPvuMrl27epcfOHCAu+++m3/84x9BjK5mMn9+0cs2/jJnjrWdIIQigVYmTwO/BroCO5RSv1FKdVNKXa6Ual3e5HzYQll4OgEPdU1TuqZiNYHkVfwKtbx2796dzz//nDlz5tC8eXMAWrRowT333CNeOaz5q1/BmDGBbTNmjLVdRdR2r0zpuinWYBDoCDgFldiH1lrXrcR2NZqoqChdvE2SELp06dKFLVu2BDsMwQ/EK2fIzs5m6tSpdOrUiZEjR5ZY98MPP3DhhRdSr169Ku2jtnvledTtzx3KMWMgKQnCKtWZnzPUdr/chCmvlFLljoAT6FdTVWIK4te/dpGcnOwKTVO6pmI1geRV/ArlvEZHR/PnP//ZW5Esrjlu3DiuueYa75BroYZbvAoLg5tuSq7wDuXNNwdWkazt55UpXTfFGgwCquhprcMqM5kKXiiJZyzRUNc0pWsqVhNIXsUvN+XVo/nRRx+xePFidu/eza233sovfvEL9u7d6/j+qoKbvMrNzSEpyfcj7//+F/wcqAiQ88qUrptiDQZS0RMEQRD84uDBgzRo0MA7/+6779KhQweefPJJTp48GcTI3EtYGGVWKC+4wPrMzIQHH6z+uAQhIMobtFsms1NkZKR2mrNnz7pC05SuqVjj4uIc15S8mtE14ZXWktfimocPH9b33nuvVkppwDu1bNlSz5s3TxcUFPilKV6V1C0o0HrMGK3B+kxJsf72TIsWBTdWuQ66J1ZT5xawRZdTp5E7kzWI1NRUV2ia0jUVqwkkr+KXm/JaXLN58+a88cYbbNq0iRtuuMG7/NChQ4wePZoePXqwadMmx2PwF7d65blDOXeu9fnzn1sv6Hh44AE4ciQwzVDHreeAG3Srm0pVJpVStyqlkpRSnyulvlZK7fMxfeN00ELZ7Nu3zxWapnRNxWoCyav45aa8lqXZpUsXPv30U+bPn0/Lli29y7/44guuv/56du7c6Xgc/uBmr8LCYPToopdtXnwRWtud6x0/DuPGWfcpA9EMZdx+DoSybnUTUGVSKVVPKfUOkAKMAboBVwBtKpgEQRCEGoZSilGjRrFnzx7++Mc/Eh5ujejyi1/8osaMORxMoqPhjTeK5lesgAULghePIJRHoP0//gEYgtVGJgVYDhwAch2NqgajlBoEDIqKimLWrFkAdOvWjaZNm5KSkgJA69at6devH0lJSQCEh4eTkJDA8uXLOXr0KABDhw5l7969bN++HYAePXrQqVMnr2ZsbCy9evVi7ty5AERGRjJq1CiWLFlCVlYWAMOHD2fHjh3eOwi9e/embt26rFu3DoB27drRu3dvr2Z0dDQjRoxg4cKFZGdnAzBy5Eg2b97Mnj17AOjTpw/5+fls2LABsAax79ixo3fIqEaNGjFs2DCUUl7dhIQEUlNTvb/Q+vfvz8mTJ9m4cSNgjSPctm1bli5dCkCzZs0YMmQIc+fOJS8vD4DExEQuvvhir+aAAQM4duyY93FbXFwcMTExrFixArBG/Rg4cCBJSUkUFhYSFhZGYmIiK1eu5ODBgwAMHjyYjIwM4uPjmTVrlmM+NWjQgLy8PGbNmuWYT127dmXBggXk5eWxcOFCx3xKTk72xuqUT2vWrCE9PZ28vDwOHDjgmE9paWnEx8ezbds2x3zydH/TpEkT8vLyHPMJoG5d6/LrlE+et0Jvvvlm1q5d65hPAJ07d2bbtm0+fXruuee48MILWbp0Kddffz1ACZ969OjB2bNnSUtLAyAmJoYDBw446lNsbCw333yz9zrghE/R0dH079/fcZ8SEhJo0qSJN9byfNq3bym9e/dkwwarcj5u3FkOHlxMw4Y55/k0YMAAfvKTn3g1q3o+QdH/J8910CmfevXq5b22OOWT57rn0XXKp9TUVK+mE+dT8f9PeXl5pKWlOeZTSkoK8fHxrFq1yjGfPNc9n5TXmLKsCdgFFAC/D2Q7marnBZwvv/zSFZqmdE3FaqIxs+TVjK6phue1Pa9V1Tx9+rS+9NJL9W233aZ3796ttRavAtE9eVLryy8vehlnwACtCwurphkoch10T6xueAGnDVAIvBLgdkI14PmlFOqapnRNxWoCyav45aa8VlXzhRdeYP/+/bz//vt07NiRxx9/nIKCygyoVjGhePxV1b3wwpIj5aSkWC/qVEUzFAh2XoOtaVK3ugm0MnkCOKm1PmMgFkEQBKEG8uOPP6KUAiA/P5+XXnqJHTt28MYbb1BYWBjk6NxB797wyCNF87/9LezfH7RwBKEEgVYmNwAXKaUuNRGMUDU6derkCk1TuqZiNYHkVfxyU16rqvnXv/6VtLQ0brzxRu+y/Px8EhMT6datm6N3Z0Lx+J3S/dOfoG1b6+/sbKvroNJvd9f288qUrptiDQaBVianYr1s84KBWIQq0tZzlQlxTVO6pmI1geRV/HJTXp3QvO6660hNTeXtt98mJibGuzwtLY2ePXsyatQoDhw4UOX9hOrxO6EbGWk93rZv8rJmDcyeXTXNYBIqeQ2Wpknd6ibQsbl3YL3NfatS6n2l1E1KqSgjkQkB43mbLNQ1TemaitUEklfxy015dUpTKcWIESPYvXs3LVu2JCIiwrtuwYIFdO/enXPnzlVpH6F8/E7o9uwJjz9eNP/44/Ddd1XTDBahlNdgaJrUrW7KrUwqpQrKmoBVwEVAP2AdkF1eWXvKr6ZjEQRBEFxAVFQUrVq1YteuXQwdOtS7/He/+x316tUrUfb48eMkJSUxZcoUkpKSOH78eHWHG3JMngxXXWX9feoU3HsvSNNTIZj4ujOpHJpkyMZqolmzZq7QNKVrKlYTSF7FLzfl1VSsbdq0YcmSJaxbt45hw4bxwAMPeNdprZk6dSoxMTGMHTuWp556irFjxxITE8PUqVM93dVVS6yh5tUFF8C8eUUj5axfD6+9VjXNYBBqea1uTZO61Y0q74RUSl3m1E601t87pVVTiIqK0qdPnw52GIIfdOnShS1btgQ7DMEPxCv3UJFXU6dOZdKkSeWunzJlChMnTjQRmmsYP956KQes9pTbtxe9oOM0cm65B1NeKaXStNZdylpX7l1DrfX3Tk2OH5FQJn71Uh8CmqZ0TcVqAsmr+OWmvFa3V8ePH2fatGk+y0ybNq3MR95uOv6q6j79NHTsaP2dkwNjxsCbb1ZNszoJ1bxWl6ZJ3erG5yNopdSHSqnF1RWMUDU8QzeFuqYpXVOxmkDyKn65Ka/V7dWyZcvIzfU9Sm9ubi7Lli07b7mbjr+quvXrW4+769Sx5j/5BFatutKByKqHUM1rdWma1K1uKmrPeBPQsxriEARBEAQADh065Gi5mkznzjBhQtH88uVd+frr4MUj1E7KbTMJoJQqBA5rrVtVX0i1AxNtJj0Dwoe6pildU7GaaH8ieTWja6qtUG3Pa3V7lZSUxNixYyvUuPvuu0lOTvaOrgPuOX4ndfPy4PrrYds2a757d+supeeOpRPIddA9sYZUm0nBfaxZs8YVmqZ0TcVqAsmr+OWmvFa3V7fffnuJPijL45///CdDhw4lKyvLu8xNx++Ubni49bjb06vS55/Diy86Im2UUM+raU2TutWNVCZrEOnp6a7QNKVrKlYTSF7FLzfltbq9aty4MROKP7v1wfLly/nyyy+98246fid1r7nGeiHHw6RJsHOnY/JGcENeTWqa1K1upDIpCIIghBwTJkxgypQp592hjIiI4Omnn+bBBx8EYOLEifTu3TsYIYYcf/gDtG59DLAefY8eDVUcUEgQ/MKfNpNngEVV2IfWWt9Xhe1rJCbaTB44cIBLLrkk5DVN6ZqK1UT7E8mrGV1TbYVqe16D6dXx48dZtmwZhw4domXLltx+++00btwYgPXr1xMfH0/dunVLxNqyZUtH26G5yasPPzzCbbc1x/OS8NSpJV/QqSxyHXRPrMFoM1m3rIWliABGV3bfgAakMmmjlBoEDIqKimLWrFkAdOvWjaZNm5KSkgJA69at6devH0lJSQCEh4eTkJDA8uXLOXr0KABDhw5l7969bN++HYAePXpw+PBhr0ZsbCy9evXy9mEVGRnJqFGjWLJkibd90fDhw9mxYwc77WchvXv3pm7duqxbtw6Adu3accEFF3g1o6OjGTFiBAsXLiQ7OxuAkSNHsnnzZvbs2QNAnz59yM/PZ8OGDQB06NCBjh07smiR9XukUaNGDBs2jH//+9/UsVuHJyQkkJqayr59+wDo378/J0+eZOPGjQB06tSJtm3bescwbdasGUOGDGHu3LnebhUSExP58MMPOXPmDAADBgzg2LFjbNq0CYC4uDhiYmJYsWIFAK1atWLgwIEkJSV5G0AnJiaycuVKDh48CMDgwYPJyMggPj6eWbNmOeZTgwYNWLZsGZGRkY751LVrVxYsWEBOTg4tWrRwzKfk5GQyMzOJjIx0zKc1a9aQnp5OTk4Ov/zlLx3zKS0tjfj4eLZt2+aYT6tXrwbgwgsvZNiwYY75BNYIL+PGjXPMp5ycHAA6d+7Mrl27HPMJ4IorrnDUJ4CYmBgOHDhQoU8ff/wxhYWFNG/enNtvv529e/d6Y+/RowcHDhzw+hQbG0t+fj5du3Zl6NChXH/99VX2KTo6mquuuoqPP/7YUZ8SEhLYsGEDp06dcsynAQMGcOpUGgMHNuedd64H4NlnNbm5S7n00ixatGjFDz8M5OzZJMA/nzzXPc910InzyXPdW7x4MZGRkY6cT8X/Px0+fJjIyEjHfEpNTWXHjh1ERkY65pPnfMrJySE+Pr7K51Px/0/x8fGsWrXKMZ/86gtTa13uBBQCucD6qky+9lFbp8jISO00M2fOdIWmKV1TscbFxTmuKXk1o2vCK60lr27xKj8/X1955ZUa6yaGfvTRR3Vubm6Vdd3mVX6+1t27aw3WdO21Wp85o/WYMdb8mDFaFxQEpivXQffEauo6CGzR5dRp/LkzmaW1/pkf5QRBEAQhaOzfv58jR454519++WU2bNjAwoULueKKK4IYWfVSpw7MnQvXXgu5uVaXQTfcUNR10Jw51mdSUtH43oJQFeRrVIOIi4tzhaYpXVOxmkDyKn65Ka9u8apNmzYsWbKEgQMHepdt3bqVzp07k5ycXGldN3p15ZXw3HNFyz0VSQ9z5kBiIhQWOh6C37gxr27RrW6kMlmDiImJcYWmKV1TsZpA8ip+uSmvbvKqY8eO/Pvf/+Zvf/sb4eHhAJw6dYpf/epXJCQkeNsoBoJbvXr4YWjevPyywa5QujWvbtCtbqQyWYPwNN4NdU1TuqZiNYHkVfxyU17d5pVSikceeYTPP/+8xOPtefPmERcXx3/+85+ANU1g0qvCQrj/fij21L9MglmhdGNe3aJb3UhlUhAEQaiRXHfddaSlpXHPPfd4l+3Zs4fu3bt73/CtiRQWWhVET9vIigj2HUrB/UhlsgbRqpXzQ6ib0DSlaypWE0hexS835dXNXjVo0IB58+bx1ltvERUVBcBdd91Fu3btKq3pFKa8mj/f/4qkhzlzYP58x8PxidvyagI3nVu+8NlpuWAOE52WC2Yw1QGs4DzilXsIhlf/+9//mDhxIm+88QYXXnhhte67Ogn0ziTAmDG+3+6Wc8s9BKPTcrkzWYPwdE4a6pqmdE3FagLJq/jlprzWFK+uuOIKFi1adF5F8tSpU/z1r38lPz8/YM2qYMqrsDCrYjhmjH/bVFSRNIXb8moCN51bvpDKZA2i0ECDFxOapnRNxWoCyav45aa81nSvHnroIR599FH69OlDRkaGI5r+YNIrfyuUCQnB62/SjXl1i251I5XJGoSTY9Ga1DSlaypWE0hexS835bUme/X+++8zb948AFJTU+nUqRP//ve/q6TpL6a98qdC2alT8Doud2te3aBb3UibySAhbSbdg7QVcg/ilXsIFa8KCgp47rnneOaZZ0rcJXr44Yf585//TERERBCjcwZfbSgjI2HHDrj8ct8aoeKXUDHSZtJGKTVMKfWKUupjpVS2UkorpXwOX6CU6qGUek8plaWUOqOU+q9S6rdKqTo+thmolPpIKfWjUuqUUuoLpdToCvYzWim1yS7/o739QF/bVBcrV650haYpXVOxmkDyKn65Ka812as6deowadIkNmzYwKWXXupd/sorr9C9e3e+/vpr13tV+g7l6NHQoYP1d06O1R9lMO4ruT2voaxb3YRkZRKYCDwEXAscqKiwUmowkAr0ApYBrwLhwMvAwnK2eQh4F+gIJAOzgVbAXKXU9HK2mQ7MBVra5ZOBnwLv2npB5eDBg67QNKVrKlYTSF7FLzfltTZ4deONN7Jt2zaGDBniXbZ9+3Y6d+7M4sWLMfEUrzq98lQo586FN9+0Js8T1rVrreXVjZwD7jq3fBGqlclHgXZANPB/vgoqpaKxKnYFwE1a6/u01k9gVUQ/A4YppUaU2qYNMB3IArporR/UWj8KXAN8AzyulLqh1DY9gMft9ddorR/VWj8IxNk6021dQRAEwYU0atSId955h1dffZX69esDkJOTw1tvvcXIkSPJyckJcoRVIyzMuisZFgbdusGjjxate+wxOHQoeLEJLkdrHdITcBOggeRy1t9rr59Xxrqb7XUbSi2fbC9/1l894C17+ZgytilXr7wpMjJSO83hw4ddoWlK11SscXFxjmtKXs3omvBKa8lrbfRq27Zt+sorr9T2tV336dNHFxQUOKLtIdhenT6tdWys1tZDbq3vuKP8snIddE+sps4tYIsup04TqncmA+Fm+3NVGetSgRygh1Kqvp/bvF+qTFW2qVbK6tIiFDVN6ZqK1QSSV/HLTXmtjV516tSJtLQ07r33Xi6++GLmz5/v+Ju3wfYqMtJ69O3hnXdgyRLHQyoXOQfcdW75om6wA3CAK+3P8wZa1VrnK6W+BToAscAuP7Y5pJQ6DcQopSK11jlKqSjgEuCU1rqsBwH/sz99js+llLofuN/+my5dynwpqtIcO3aMpk2bhrymKV1Tse7atcsVXpnSdVOsJrwCyWtt9yoyMpJBgwaVWKa1pqCggLp1K/9vNFS8atJkPJmZdwBw113HmTr1l9Stm12ijFwH3ROrqXPLJ+XdsgyViYofc++x17ctZ/2n9vobii3Ls5fVLWebA/b6lvZ8K3s+o5zy9ez1Z/09LhOPuWfOnOkKTVO6pmI18chA8mpG19TjndqeV/HqfM1JkybpFi1a6LVr1zqqW1Uqo/nDD1q3alX0uHv06PPLyHXQPbHKY26hSnTr1s0VmqZ0TcVqAsmr+OWmvIpXJTXXr1/P1KlTOXz4MLfccgsTJkwodyjGQHSdoDKaDRvCa68Vzc+bB6tXOxdTecg54K5zyxc1oTL5o/15UTnrPctPVGKbH0t9BrKPasfELXgTmqZ0TcVqAsmr+OWmvIpXJTULCgpo0qQJYD3de+655+jduzfff/99lXSdoLKav/gFjCjW78n998PJkw4FVQ5yDrjr3PJFTahMfm1/ntdeUSlVF7gcyAf2+blNSyAK65F2DoDW+jTWo+8L7fWlucL+PK8NZnWSkpLiCk1TuqZiNYHkVfxyU17Fq5Kaffv2Zfv27fTp08e7bOPGjVx77bUsXbq00rpOUBXNv/0NGje2/k5PhwkTHAqqHOQccNe55YuaUJn80P68tYx1vYBIYKPW+qyf29xWqkxVthEEQRBqIC1btmTNmjU899xz1KljDbR24sQJhg0bxv/93/9x5syZIEcYOM2aWRVKD6++Cp9+Grx4BPdQEyqTS4BMYIRSyvv6klIqAphqz75Waps5wFngoeIdjSulLgbG27Ovl9rGMz/BLufZpg3woK1Xxsin1Ufr1q1doWlK11SsJpC8il9uyqt4VbZmWFgYf/zjH0lNTeWyyy7zLn/99dfp1q0bO3furJRuVaiq5t13w89/bv2tNdx3H+TmOhBYGcg54K5zyxfKekEntFBKDQGG2LMtgP5Yj6k/tpdlaq1/V6r8EiAXa/jELOAXWF0ALQHu1KUOVCn1MPB34DiwCOsN72FADPBicf1i27wIPAZk2LrhwHCgMfCw1vpVf48xKipKnz592t/iflFYWOh4P2gmNE3pmoq1S5cubNmyxVFNyasZXRNegeRVvKpY88SJE4wdO5YlxTpq7NatG59//jlKqZCKtSL274err4ZTp6z58eNh9Wq5DrolVlPnllIqTWtdZp9DoXpn8lpgtD31t5fFFls2rHhhrfVyoDdWJ+VDgYeBc1gVvxGlK5L2Nq9gVTh3Avdg9f94GEgoqyJpb/M4MMYud7+93U5gUCAVSVMkFe99NoQ1TemaitUEklfxy015Fa8q1mzYsCH/+te/mDlzJhEREVxwwQW8+eab5VYk/dUNFCc0L70U/vznovkXXoCcHJ/dKFcKOQfcdW75IiQ7LddaPwM8E+A2nwI/D3Cbd4F3A9xmLjA3kG0EQRCEmo9Sivvvv58ePXqwe/duOnToEOyQKs24cbBwIaSmQkEBfPfdU+TnQxX6aBdqMKF6Z1KoBOHh4a7QNKVrKlYTSF7FLzflVbwKTLNjx44MGzbsvOVvvvkmL7zwAoWFhZXS9QenNMPCYPZsiIiw5s+cuYoXX3RE2oucA+46t3wRkm0mawMm2kwKZjDV/kRwHvHKPdQ2r7766iu6dOnCmTNnuOWWW5g/fz7NmzcPdlgV8uc/wx/+YP1dvz5s3w5XXul7GyG4SJtJoUosX77cFZqmdE3FagLJq/jlpryKV1XXnD59ure7oA8++IBOnTrxzDPPVFm3NE4f/2OPQefO1t9nz0JiItg3VquMnAPuOrd8IZXJGsTRo0ddoWlK11SsJpC8il9uyqt4VXXNmTNn8sc//tH7Qs6RI0d49tlnefLJJzl37hzHjx8nKSmJKVOmkJSUxPHjx4MWa3Hq1oU33wRr7A/45BN4vXTHeZVEzgF3nVu+kMqkIAiCIBimXr16PPfcc6xZs6bE4+0XXniB2NhYLrnkEsaOHctTTz3F2LFjiYmJYerUqYRCU7ROnaBFi3ne+T/8wRohRxC8aK1lCsIUGRmpnSYzM9MVmqZ0TcUaFxfnuKbk1YyuCa+0lryKV85qHj58WPfv318DFU5TpkwJaqwerruuu77qKq2trsy1vu02rQsLtS4o0HruXOszUGr7OWBK19S5BWzR5dRp5M5kDWLv3r2u0DSlaypWE0hexS835VW8clazefPmvPfee9x3330Vlp02bVpAj7xNeRUWdo433gBPt5nvvw9vvWW1oUxIqFxbSjkH3HVu+UIqkzWI7du3u0LTlK6pWE0geRW/3JRX8cp5zbCwMPLy8iosl5uby7Jly/zWNelVjx7w8MNF8/ffD3PsQYTnzAm8QinngLvOLV9I96PVjFJqEDAoKiqKWbNmAdaQW02bNiUlJQWwxurs16+ft2f88PBwEhISWL58ubex7tChQ9m7d6/3i9ijRw/Onj3r1YyNjaVXr17MnTsXgMjISEaNGsWSJUvIysoCYPjw4ezYscM7fmzv3r2pW7cu69atA6Bdu3YUFBR4NaOjoxkxYgQLFy4kOzsbgJEjR7J582b27NkDQJ8+fcjPz2fDhg0AdOjQgY4dO7Jo0SIAGjVqxLBhw8jKyvLqJiQkkJqayr59+wDo378/J0+eZOPGjQB06tSJtm3bsnTpUgCaNWvGkCFDmDt3rvdinJiYSHZ2tldzwIABHDt2jE2bNgEQFxdHTEwMK1asAKBVq1YMHDiQpKQk73BWiYmJrFy5koMHDwIwePBgMjIyiI+PZ9asWY751KBBAzIzM5k1a5ZjPnXt2pUFCxaQmZnJwoULHfMpOTnZG6tTPq1Zs4b09HQyMzM5cOCAYz6lpaURHx/Ptm3bHPNp9erVAGRnZ5OXl+eYT4B3O6d8ysnJAazh2dauXeuYTwB5eXls27bNMZ8AYmJiOHDggKM+xcbGUlhY6L0OOOFTdHS0EZ8SEhI4cuQI/vDuu+/SpEmTCn0aMGAAOTk53uN3wifPdc9zHbzppstZvrwv6emK0nXhOXPg66+/ZvnyJuzb59unXr16ea8tTvnkue55dJ3yKTU11avpxPlU/P9TZmYmaWlpjvmUkpJCfHw8q1atcuR8Kv7/ySflPf+WyX1tJr/88ktXaJrSNRWrifYnklczuqbaCtX2vIpXZrx65pln/Go3OXv27KDH6vGroEDrfv2K2k6WNY0Z418bytp+DpjSlTaTQpVo0KCBKzRN6ZqK1QSSV/HLTXkVr8wc/+DBg4nwDDFTDkopjh075remSa8KC61H2WvW+C7n7yNvOQfcdW75QiqTNQjPrelQ1zSlaypWE0hexS835VW8MnP8mzZtYsKECT7LaK0ZP348jzzyCAUFBRVqmopVa0ViYlEbyYrwp0Ip54C7zi1fSGVSEARBEILEhAkTmDJlynl3KCMiIoiJifHOv/LKKwwZMoRTp05Vd4gAZGX93O+KpIc5c2D+fDPxCKGFVCZrELGxsa7QNKVrKlYTSF7FLzflVbwy55VSiokTJ5KRkcHs2bOZPHkys2fPJiMjgz179vDLX/7SW37lypXEx8dz4MCBao+1UaP3GDMmsG3GjIFf/ar89XIOuOvc8oWy2lQK1U1UVJQ+ffq0o5p5eXmEh4eHvKYpXVOxdunShS1btjiqKXk1o2vCK5C8ilfB86qwsJAJEybw/PPPe5ddcsklrFy5kmuvvbZSmpWhS5cubNq0xe9H3WPGQFIShPm4ZVXbzwFTuqbOLaVUmta6S1nr5M5kDcKv1/dDQNOUrqlYTSB5Fb/clFfxynlNf3XDwsL405/+RFJSEnXrWr35HThwgBtvvJGVK1dWSrOyhIVZFcSK7lDecUfFFUmQc8CkbnUjlUlBEARBCHHuu+8+3n//fS666CIATp8+zeDBg3nllVeqNQ5/KpQBvHwu1BCkMlmDiIyMdIWmKV1TsZpA8ip+uSmv4lVoeNW3b182btxImzZtAOsR+MaNGyneXK06vCqvQukZavHjj/17FB4qeQ2Wpknd6kbaTAYJE20mBTOYan8iOI945R7Eq8pz5MgRBg8eTJ06dVi3bl2FfVU6QVl+efqdnDPHqlg2bQp//rO1rmFD2L0bmjc3HppQCmkzKVSJJUuWuELTlK6pWE0geRW/3JRX8Sq0vGrevDnr16/n3//+93kVyer0ynOHcu5c6/Ppp8HzcvKJE/Doo763D7W8VremSd3qRiqTNQjPmKahrmlK11SsJpC8il9uyqt4FXpeXXDBBTRu3LjEMq01c+bMYdu2bVWMzH/CwmD0aOszMhJef71o3dtvw/vvl79tKOa1OjVN6lY3UpkUBEEQhBrASy+9xHvvvceNN95ISkpKUGK45RYYNapo/v/+D6RFV81H2kwGCRNtJn/88Ufvm36hrGlK11SsJtqfSF7N6JpqK1Tb8ypehb5XWVlZtG3blh9++AGwuhT629/+xl133cWyZcs4dOgQLVu25Pbbbz/vjqY/BOLXsWNw1VXguen2+OMwffr55dyQV5OapnSlzaRQJXbs2OEKTVO6pmI1geRV/HJTXsWr0PeqUaNGfPzxx7Ro0QKw3vR++OGHad68OWPHjuWpp55i7NixxMTEMHXqVEzeSGraFF56qWj+5Zdh69bzy7khryY1TepWN1KZrEHs3LnTFZqmdE3FagLJq/jlpryKV+7wqkOHDjz++ON069bNu6ygoKBEmdzcXCZNmsS0adMc3Xdp7rkHbr7Z+ruwEO6/H/LzS5ZxS15NaZrUrW6kMikIgiAINYTo6GiWLFlCWAXDz0ybNo3jx48bi0Mp62Wc+vWt+bQ0qOb+1YVqRNpMVjNKqUHAoKioqLEv2c8BunXrRtOmTb0Nplu3bk2/fv1ISkoCIDw8nISEBJYvX87Ro0cBGDp0KHv3XHii4QAAKeRJREFU7mX79u0A9OjRgx9++IFdu3YB1uDxvXr18g7VFBkZyahRo1iyZIn37bHhw4ezY8cO7y+j3r17U7duXdatWwdAu3btuPjii/niiy8A6yI1YsQIFi5cSHZ2NgAjR45k8+bN7NmzB4A+ffqQn5/Phg0bAOuXcseOHVm0aBFgPYoZNmwYM2fORNk93CYkJJCamsq+ffsA6N+/PydPnmTjxo0AdOrUibZt27J06VIAmjVrxpAhQ5g7dy55eXkAJCYmsmjRIk6ePAnAgAEDOHbsGJs2bQIgLi6OmJgYVqxYAUCrVq0YOHAgSUlJFBYWEhYWRmJiIitXruTgwYMADB48mIyMDJKTk2nfvr1jPjVo0IAVK1YQERHhmE9du3ZlwYIF5Obm0qxZM8d8Sk5OJisri4iICMd8WrNmDenp6eTm5jJ06FDHfEpLS2PXrl2MHj3aMZ9Wr14NQMOGDb3H4oRPgPdYnPIpJyfHG/vhw4cd8wmgY8eOREZGOuYTwHvvvceMGTMc9Sk2NpYWLVp4j9UJn6Kjo7nuuuv4z3/+46hPnmM7ceKEYz4NGDCAL7/8kn/+85/Mnz+firjnnnuYN2+eT588173p06fTvn37gH16771rWbGim132HM88s5iuXZvRq1cvXn/9dSIiIhzzyXPdO3r0KBEREY75lJqayldffUVERIRjPnmue7m5ufTs2bPK51Px/0+7du2if//+jpxPxf8/jRs3rtw2k2itZQrCFBkZqZ1m7969rtA0pWsq1ri4OMc1Ja9mdE14pbXkVbxyl1eTJ0/WQIXT5MmT/datrF9nz2rdsaPWYE0DBmhdWFgUqwlq+3fA1LkFbNHl1GnkMXcNwvOLLdQ1TemaitUEklfxy015Fa/c5VXLli39KutvuaoQHg6zZhUNtZiSAosXW3+7La8mcNO55QupTAqCIAhCDeL222+vcIjFiIgI7rjjjmqJ54YbrP4mPTzyCNg9GAk1BKlM1iDatWvnCk1TuqZiNYHkVfxyU17FK3d51bhxYyZMmOCz3IQJE2jUqBGff/65tw2dSZ57Dlq1sv4+cgT+8Af35dUEbjq3fFLe82+Z3Ndm8tSpU67QNKVrKlYT7U8kr2Z0TbUVqu15Fa/c51VhYaGeMmWKjoiIKNFOMiIiQk+ZMkUXFhbq3bt364YNG+o2bdroHTt2+NR1wq933ilqOwlar16dU2XNsqjt3wFpMylUCc9bbaGuaUrXVKwmkLyKX27Kq3jlPq+UUkycOJGMjAxmz57N5MmTmT17NhkZGUycOJH8/HwGDx7MiRMn+O6777jhhhtYtWqV4/EU5/bbYciQovnRo3M5e9b5/dT270AwkMqkIAiCINRQGjduTGJiIpMmTSIxMdE7lGK9evWYPn06F154IQAnT55kwIAB/P3vf8e6CWWGV16BBg2svw8fvpjnnze2K6EakcpkDSI6OtoVmqZ0TcVqAsmr+OWmvIpXNdOrgQMH8umnn9K6dWvAGoLxN7/5DQ888ADnzp1zPDaAmBir/aSH556D3bux9w/z5lmfVSHYeQ0F3epGOi0PElFRUfr06dPBDkPwgy5durBly5ZghyH4gXjlHsSr0OHIkSMMGTKEzz//3Lusb9++LF68mIYNGwLO+lVQAD17gj0eBvHx8OGH1pCLc+bAmDGQlAQVDOIjlIOpc0spVW6n5WJVDWLhwoWu0DSlaypWE0hexS835VW8qtleNW/enPXr13PXXXd5l61du5YbbriBvXv3OhkeAHXqWH1P1qlj3YL8+GO46SarIgnWZ2Ji5e9Qhkpeg6lb3UhlsgbhGeor1DVN6ZqK1QSSV/HLTXkVr2q+VxERESxYsIBnn33Wu2z37t10797dSNdB11wDt9yy3Tv/6acl11elQhlKeQ2WbnUjlUlBEARBEFBK8dRTT7Fw4UJvp+f33XcfzZo1M7K/227b6n0ZpyyqeodSqD6kzWSQMNFm8vTp00RFRYW8pildU7GaaH8ieTWja6qtUG3Pq3hV+7z64osveOONN3j99dcJCwsr4dfx48dZtmwZhw4domXLltx+++3et8T9pbAQEhLOMX9+vQrLBtqGMpTzWh260mZSqBKbN292haYpXVOxmkDyKn65Ka/iVe3z6vrrr2fWrFmEFavBaa2ZNGkSMTExjB07lqeeeoqxY8cSExPD1KlT/e5SqLDQuuPoT0USAr9DGcp5rS7d6kYqkzWIPXv2uELTlK6pWE0geRW/3JRX8Uq8ApgyZQpTp04lNze3xPLc3FwmTZrEtGnT/NKZP7/oZRt/mTPH2s4f3JRXN51bvpDKpCAIgiAIPsnPz2fy5Mk+y0ybNo3jx49XqPWrX1mPrgNhzBhrOyE0kcpkDaJPnz6u0DSlaypWE0hexS835VW8Eq9OnDhBQUGBzzK5ubksW7asQq2wMKsN5LBh/r3JHGibSTfl1U3nli+kMlmDyM/Pd4WmKV1TsZpA8ip+uSmv4pV45e+IOIcOHfKrXFgYTJ58qMI7lJXpwNxNeXXTueWLusEOoLahlBoEDIqKimLWrFkAdOvWjaZNm5KSkgJA69at6devH0lJSQCEh4eTkJDA8uXLvf19DR06lL1797J9u9VPV48ePVi/fj0bNmwAIDY2ll69ejF37lwAIiMjGTVqFEuWLCErKwuA4cOHs2PHDnbu3AlA7969qVu3LuvWrQOgXbt27Nq1y6sZHR3NiBEjWLhwobdvrJEjR7J582Zvu48+ffqQn5/v3aZDhw507NiRRYsWAdCoUSOGDRvGsmXLaNSoEQAJCQmkpqayb98+APr378/JkyfZuHEjAJ06daJt27YsXboUgGbNmjFkyBDmzp1LXl4eAImJibz77rve/Q4YMIBjx46xadMmAOLi4oiJiWHFihUAtGrVioEDB5KUlERhYSFhYWEkJiaycuVKDh48CMDgwYPJyMggPj6eWbNmOeZTgwYNWLp0KU2aNHHMp65du7JgwQIyMzOJjY11zKfk5GTS09Np0qSJYz6tWbOG9PR0MjMzGT16tGM+paWlER8fz7Zt2xzzafXq1YDVF9zll1/umE8AWVlZPPnkk475lJOTA1hD4u3fv98xnwDy8vI4c+aMYz4BxMTEcODAAUd9io2NZe/evd58OeFTdHQ02dnZ/Oc//3HUp4SEhBLXLCd8GjBgAO+//75X0wmfPNe9q6++2lvGF//73/84fvx4hT716tWLZcuW0q1bE7755mZSU9uWqdew4bskJR3yyyfPdW/fvn00adLEMZ9SU1PZtGkTTZo0ccwnz3UvMzOT/v37O+ZTSkoK8fHxrFq1ypHzqfj/J59orWUKwhQZGamdZubMma7QNKVrKta4uDjHNSWvZnRNeKW15FW8qt1eaa11p06ddEREhAbKncLCwnR6enrAsRYUaD1mjNZw/nT55VqfOqW95ebOtT790XUSN30HTJ1bwBZdTp1GHnPXIDp06OAKTVO6pmI1geRV/HJTXsUr8apu3bpMmDDBZ5nCwkJuv/12jhw54pemJ1ZPG0rPI+/hw+Gii6y/v/0WnnyyqDuhhISKuwlyU17ddG75Qh5z1yA6duzoCk1TuqZiNYHkVfxyU17FK/EK8FYmp02bVqJ7oLp163rb/p06dYp69fzrP7J4rJ4KZe/e1lvbyckwerS17tVX4Ztv4P33rXlPt0LltaV0U17ddG75Qu5M1iA87T5CXdOUrqlYTSB5Fb/clFfxSrwCa7jFiRMnkpGRwezZs5k8eTKzZ8/m8OHDvPbaa7Rs2ZL33nvP2x4+0FjDwqwKZFiYVaEcOLBonaci6cFXR+Zuyqubzi1fyJ1JQRAEQRD8pnHjxiQmJpZY9utf/5pRo0Zx4YUXOrIPpeC112DNGrDfYzmPiu5QCtWHpL8G4e+vwWBrmtI1FasJJK/il5vyKl6JV/5QVkVy4cKFTJ48ucyhFn3FWlgITz1VfkXSQ1l3KN2UVzedW75QZRksmCcqKkqfPn062GEIftClSxe2bNkS7DAEPxCv3IN45S4q49cnn3xCnz59yMvLY8yYMcycOdOv9pSel20CGXKxMv1R1lRMnVtKqTStdZey1knaaxDJycmu0DSlaypWE0hexS835VW8Eq8qw6uvvurta3HOnDn8/Oc/58cff/SuLy/Wqo7d7aa8hpJfVUEqkzUITyeroa5pStdUrCaQvIpfbsqreCVeVYb58+czptgQN2vXrqVnz57ezrvLi7UyY3dfcQWMHIlP3argpu9AMJDKpCAIgiAIjlOvXj3eeOMNpkyZ4l22c+dOunfvztatWzl16hRJSUlMmTKFpKQkjh8/Dpzf76Q//O9/cP/9vvufFMwhbSaDhIk2k3l5eYSHh4e8pildU7GaaH8ieTWja6qtUG3Pq3hVu72CqvuVnJzMvffe6x3ju169eiilvI/BASIiIpgwYQITJkxAKVXptpP/+EceERHuyKubzi1pM1lLSE1NdYWmKV1TsZpA8ip+uSmv4pV4VVVGjRrFmjVraNiwIQDnzp0rUZEEyM3NZdKkSUybNg2o3B3KOXNgyJBMx+9Quuk7EAykMlmD2Ldvnys0TemaitUEklfxy015Fa/EKye46aabSElJQSnls9y0adNKPPKeNctqE+kvq1e3qnDIxUBx03cgGEhlUhAEQRCEauGrr74qs8/J4uTm5rJs2TLv/IIFVpvIQCj+drdgHqlM1iD69+/vCk1TuqZiNYHkVfxyU17FK/HKKQ4dOhRwucq83T1mjLWdU7jpOxAMZDjFGsTJkyddoWlK11SsJpC8il9uyqt4JV45RcuWLf0ud/z4cZYtW8ahQ4fo3r0lubm38/bbjSvcdsiQH0hKutjRDszd9B0IBnJnsgaxceNGV2ia0jUVqwkkr+KXm/IqXolXTnH77bcTERFRYbmkpCRiYmIYO3YsTz31FOPGjWXZshiuu24qUP5j8jFjoH//xY6PhOOm70AwkK6Bqhml1CBgUFRU1NiXXnoJgG7dutG0aVNSUlIAaN26Nf369SMpKQmA8PBwEhISWL58OUePHgVg6NCh7N27l+3btwPQo0cP1q9fT/369QGIjY2lV69ezJ07F4DIyEhGjRrFkiVLyMrKAmD48OHs2LGDnTt3AtC7d2/q1q3LunXrAGjXrh27du2iTp06AERHRzNixAgWLlxIdnY2ACNHjmTz5s3s2bMHgD59+pCfn8+GDRsA6NChAx07dmTRokWANQ7psGHDeP75571jkiYkJJCamuptiNy/f39OnjzpPck6depE27ZtWbp0KQDNmjVjyJAhzJ071/s2YGJiIi+99BLR0dEADBgwgGPHjrFp0yYA4uLiiImJYcWKFQC0atWKgQMHkpSURGFhIWFhYSQmJrJy5UoOHjwIwODBg8nIyCA5OZn27ds75lODBg1YsGABTZo0ccynrl27smDBAjIzM4mNjXXMp+TkZNLT02nSpIljPq1Zs4b09HQyMzMZPXq0Yz6lpaWxa9cuRo8e7ZhPq1evBiA7O5tHHnnEMZ8AsrKyePLJJx3zydP5cWFhIW3btnXMJ7C6L7nxxhsd8wngvffeY8aMGY76FBsby969ewmzaxJO+BQdHU12drb30ymfEhIS+Pvf/+69Zjnh04ABA1i8eDGRkZGO+eS57k2fPp327ds74tNnn33mPZcqw6WX/ob9+/963vIxY6Bv34V8990+mjRp4phPqampbNq0iSZNmjjmk+e6l5mZSf/+/R3zKSUlhV27dtG/f39Hzqfi/5/GjRtXbtdAaK1lCsIUGRmpnebzzz93haYpXVOxxsXFOa4peTWja8IrrSWv4lXt9kprZ/0qLCzUU6ZM0fXr19dYtxk1oCMiIvRPfvKTEsvKmurWrau7dXtZQ6YGrUHrMWO0Liiw9N2UVzedW8AWXU6dRh5z1yDatm3rCk1TuqZiNYHkVfxyU17FK/HKSZRSTJw4kS+//JLZs2czefJkZs+eTUZGBr///e8r3D4/P59Nmx6lTp0YYCoJCZqkJLyPtt2UVzf45Q9SmaxBeG6zh7qmKV1TsZpA8ip+uSmv4pV4ZYL169eTmJjIpEmTSExMpHHjxhw5csTv7QsKcoFJxMZOK9FG0k15dZNfvpDKpCAIgiAIIYG/b3sX57nnijo5F4KDVCZrEM2aNXOFpildU7GaQPIqfrkpr+KVeGWCsmL1923v4pTu5NxNeXWTX76Qt7mDRFRUlD59+nSwwxD8oEuXLmzZsiXYYQh+IF65B/HKXVSnX1OnTmXSpEkBbTN58uSAt6mpmPJKKVXu29xyZ7IGUZWuFqpT05SuqVhNIHkVv9yUV/HKeU1TujXBqwkTJjBlypSA7lAWfzzupry6yS9fSGWyBuHp0yrUNU3pmorVBJJX8ctNeRWvxCsTlBer523vjIwMXn75ZerW9T1YX0REBHfccUeFulXBTd+BYCCVSUEQBEEQQo7GjRvz29/+lqefftpnuQkTJngHwRCCg7SZDBIm2kx6esoPdU1TuqZiNdH+RPJqRtdUW6HanlfxqnZ7BcG9DmqtmTZtGtOmTSM3N9e7PCIiggkTJjBhwgSUUgHrmog1FHSlzaRQJdasWeMKTVO6pmI1geRV/HJTXsUr8coE/sZa/LF36U7OJ06cWKIiGYiuiVhDRbe68d0QQSgTpVQMMBm4FWgMHAKWA89qrX8IVlyecT9DXdOUrqlYTSB5Fb/clFfxSrwyQaCxNm7cmMTERMd1/cFN34FgIJXJAFFK/QTYCDQDVgC7gW7Ab4BblVI9tdbSe6ogCIIgCLUCecwdOP/Aqkg+orUeorV+Umt9M/AycCUwLViBDRgwwBWapnRNxWoCyav45aa8ilfilQkkr+7yyxdSmQwA+65kP+A7YEap1U8Dp4FfKaWiqjk0AI4dO+YKTVO6pmI1geRV/HJTXsUr8coEkld3+eULqUwGxs/szzVa68LiK7TWJ4FPgUige3UHBrBp0yZXaJrSNRWrCSSv4peb8ipeiVcmkLy6yy9fSGUyMK60P/eUs/5/9me7aohFEARBEAQh6Eg/kwGglJoFjAXGaq2Tylg/DRgPjNda/6mM9fcD99uzHYEdDod4EfCjCzRN6ZqKtQmQ6bCm5NWMrgmvQPIqXtVur0Cug26K1dS5dYXW+qIy12itZfJzAmYBGkgsZ/00e/0f/dDaYiI+N2i6MFZXeOXCvJqI1XGvJK/iVW33ypRfkteac27JY+7A8Px6KLtmXrT8hPlQyuRdl2ia0jUVqwkkr+KXm/IqXpnBTbGaQPJaQ/ySx9wBoJRKBGZj1c7HlbF+Ndbb3n211usq0NqiyxmWSAgtxCv3IF65B/HKXYhf7iEYXsmdycBYb3/2U0qVyJ1SqgHQE8gBPvdDa5bDsQnmEK/cg3jlHsQrdyF+uYdq90ruTAZIsbuPj2itXym2/CXgUWCm1vrXwYpPEARBEAShOpHKZICUMZziLuB6rD4o9wA9tAynKAiCIAhCLUEqk5VAKXUpMBm4FWgMHAKWAc9qrX8IZmyCIAiCIAjViVQmBUEQBEEQhEojL+AEgFIqRin1plLqoFLqrFLqO6XUX5VSF/u5/U1KKe3HdGmp7XyV9edln1pHVb0qpnOjUmqFvX2uUipdKfWeUupWH9tcrZT6l1LqqL3N10qpZ5VSF1T9yGoewfJKzqvK4aBfQ5VSHymlflRKnVFK7VRK/VEpFe5jmx62p1n2Nv9VSv1WKVWn6kdW8wiGV0qpNhWcWwudO8KagVJqmFLqFaXUx0qpbDtPyZXUCthzJ/5nyZ1JP1Hnt5XcDXTDaiv5NdCzoraSSqk2QEI5q38K3AHs0Fr/tNR2GvgemFvGdhm6jNF4ajNOeGXr/B/wD+A0VjOGDCAGy6dIYKLWelqpba4HPgTqAUuA/cDNQBessdv7aK3PVv0oawZB9krOqwBx0K/ngD8Cp4ClQBYQj3WerANu01qfK7XNYLtsLrDI3mYQ1jC3S7TWv3TgEGsMwfLK/j/3LbAdWF6G5A6t9ZLKHldNRCm1DeiEleMM4CpggdZ6VIA6AXvu2P8sE72k18QJWI01us3DpZa/ZC9/vYr6b9s6j5SxTgMfBTsHbpmc8Mo+sU4AZ4ArS61rj/UPLQeoX2x5HeArex+/KLY8zD5JNfBksPMTSlOwvLLXyXkVHL8622V/AGKLLVfAa/a6x0ptEw0cBc4CXYotj8D656mBEcHOTyhNQfSqjb18brBz4JYJq7J3hZ3Xm+z8JZv23Mn/WUFPohsm4Cd2Ur8Fwkqta4D1a+I0EFVJ/SbF/uE1LGO9/NOrZq+A5rbO9nLW/9de37jYspvtZRvKKB9rr/sO+4lAbZ+C6ZW9XM6r4Pg12db5SxnrLrbXfVNq+b328nllbFPueVdbpyB71QapTFbFu5uoRGWyMp47+T9L2kz6x8/szzVa68LiK7TWJ7FuBUcC3SupPxqoDyzWWp8op0xDpdS9SqnxSqkHlVKV3VdNxymvjgLHgHZKqSuKr1BKtcP6FblNl3xkcLP9uaq0mNZ6H1bXUZdhnaRCcL3yIOeV/zjlVwv7c1/pFdrqDeMHIFYpdXmxVeWeW0Aq1g/xHkqp+hXsu7YQTK88tFJKjbPPrXFKqWsCOgIhUCrjuWP/s6Qy6R9X2p97yln/P/uzXSX1x9qfM32U6QS8AUwDXgU+U0ptU0r91Mc2tRFHvNLWT7MHsc6RNKXUPKXUn5RSbwFpwE6gdBst09+TmkYwvfIg55X/OPX9zrQ/z6uAKKUaYt3xKr4/n/vWWudj3Y2pi/xQ8xBMrzzcAryOdW69DmxXSq1XSrWuYJ9C5aiM5479z5LKpH9cZH/+WM56z/KGgQorpXpjGbpDa72xnGIvYQ3V2BTrdnVXrPYMnYAPlVKXBLrfGoxjXmmtF2P9cjsB3AM8CfwK61HBHM7/tW7se1JDCaZXIOdVoDjlV4r9OdZ+WQMApZTCqnh4KP72qZxbgRFMr3KAKUCcvfxioDfWcMQ3AeuUUlEV7FcInMp47th5JZXJ4HO//VnuWJpa68e11hu11pla61Na6y3aenNxKVZ7y99VR6C1DaXUKGAt8DHWixyR9uc6rLtY0sVFiFAZr+S8Cg5a60+x7gY3BP6rlJqjlHoR+AK4D+sNVIDCshWE6qIyXmmtj2qtn9Jab9Van7CnVKxhiL8A2gKJ1XkcgnmkMukfntr5ReWs9yw/EYioUqoRMBTrLdT5lYjrdfuzVyW2rak44pXd1u5NrEekv9Ja79Zan9Fa78a645UG/FIpdZPT+65FBNMrX8h5VTZOfr/HAuOwuiu50/47G+vO1Td2maOG9l0bCKZXZWI3R/B0tyXnlvNUxnPHvidSmfSPr+3P8toNeBr9l9fuoDw8L978y8eLN744Zn/KI4MinPKqH1aXMxvKaMxciNXoH6xHOU7vu7YQTK98IedV2Tj2/dYWs7TWXbXWUVrrC7XWfbXWn2P1uVsIbPVn30qpulht+vIpuzlDbSSYXvlCzi1zVMZzx74nUpn0j/X2Zz+lVImcKaUaYLW7ygECHTXD8+JNuY+4K8DzVpZcQItwyivPW6FNy1nvWZ5XbNmH9ud5I64opWKxTtjvEb88BNMrX8h5VTamroPFdW4CWgMpWuvi7bjKPbew7nJFAhu1DAjgIZhe+ULOLXNUxnPn/mc51TdSTZ8IvDPQq4CrfOjF29t9WcF+rwHqlbM809a4O9j5CaXJCa+wRg3Q9sl3Tal112I1TSgEOhRb7qsD2MVIp+Wh5JWcV0Hyy14eXcayy4C9WB2Tl/YxGuuulnRaHvpedaZUP4f28j5Y/SlroEew8xOqExX0M4n1FOYq4CcOeO7Y/ywZTtFPyhimaBdwPVbfTnuwTo7jxcprAK21KkdvPjAKa8SbV3zsdy7WkGEfYw1zdBbri3Qr1hdhNjBOi5FenPJKKfUmMAbrjtYyrF9obYAhQDjwV631o6W2KT00VTrWRVSGUyyDYHkl51XlcNCvxVgVkq1Yw/NdDvwC67z5ldZ6URn7HoJ1TuVivVCVZW9zpb38TvGriGB5pZT6COvx6EasoQHB+pHm6dNwktZ6qlPHWROwv9tD7NkWQH+su4Ef28sytda/s8u2weoK63utdZtSOgF5bm/jzP+sYNfC3TQBl2J1M3II65/W98BfgYvLKKuxu8ArY93FWHdLyhzxplTZIcA7WL8Cs+39HgLepdgvCZmc9wpraKsE4COsznnzsS6m6/BxFwS4GutXXSZWJWUP8CxwQbDzEopTMLyS8yrofo3G+kd13NbYD8wD2lew757Ae7bHZ4AvgUeBOsHOSyhOwfAK6y3vlVgjp5yyr4HpWOOpxwc7J6E4Ac948l/O9F2xsm1KL6us58W2qfL/LLkzKQiCIAiCIFQaeQFHEARBEARBqDRSmRQEQRAEQRAqjVQmBUEQBEEQhEojlUlBEARBEASh0khlUhAEQRAEQag0UpkUBEEQBEEQKo1UJgVBEARBEIRKI5VJQRCEIKKUekYppe1ReUqv+85ed1O1ByYIguAnUpkUBCFkUUrNtStTpaeTSqmdSql/KKXaBzvOYKCUukYp9Wul1BtKqf8qpfLt3CwMQOMKe/t0pdRZpdRBpdRCpVRcJeLpoJR6Sin1iVJqv62XrZTarpSaoZTq6adOmFLqZ0qp3yul/qWU+raY778OIJ6+Sql3lVJHlVK5SqlvlFJ/U0o1D/TYBEHwTd1gByAIguAH57CGRwRr6MQmWEOAXQ3cp5QapbVeHKzggsRbQKfKbqyUugVYDkTai37EGhd4ODBUKTVGa53sh04s8Dzwy2KL87GGqbwIa1zma4AHlFJrgfu01uk+JKOxxgquNEqpCYBn/OdCrGH9YoFHgLuUUjdrrXdUZR+CIBQhdyYFQXADG7XWLeypORAB3IY1/m84MEcp1TSYAQaBc8A2IAkYB6z2d0OlVAtgCVZF8gOgjda6IVZlcgHWjYY3lFIdKtDpB2zFqkh+BfwGaA+Ea60bA/WxKnGPATuAvsAOpdSgCkI8DXwMvAzcDRwO4Nh+TlFF8kWgodb6IqAjVr6aAiuUUvX91RQEwTdSmRQEwXVorc9prVcBI+1FUcDQIIYUDLprra/TWo/VWs8igAoX8CTWHcD9wB1a6+8BtNZHgdFAGlYlfXJ5Akqp24AUrNw/Clyjtf671nq31lrbegVa62+11i8D1wK/x6pg/ksp1asc6R+BaK11L631Y1rrt4GzARzbc/bnMq3177TWJ+1YdgKDKLpLeX8AmoIg+EAqk4IguJnPsCoHYD3yLoFSKlwp9ZBS6mOlVJbdju97pdSbFbW1VEo1Vko9q5RKU0qdUErlKKX22G0Kh5RRvpfdJu8Lu+1hnt1eb5VSapgjR1sMrXVBZbZTSoUBI+zZ17TWp4qvt3VfsmcHKqWiy9CIxbqDeQb4udb6rxXFY1cs/4L1GL0usLysu8naojDQ47Lj6kDRo/+/lKGdAbxtz44svV4QhMohlUlBENyOsj/rlFioVEtgE/AKcCNW+72zQGtgDLBVKXVHmYJKxQNfA08BnYELsCpOsViVoWWlyl8IbMBqk9cNuNAu3xToDyxWSs2s4nE6xdWA5yWU8h6Nr7E/w7FyV5oXgYuBcVrrDyraoVIqUinVQikVrrVeDjxhb1/unc9K8jP780fgi3LKeI65m+2bIAhVRCqTgiC4mR5Yj1kB9nkWKqXqASuw7lKts8tFaK2jgVbAX7HaXc5XSv2kuKA9vxJojNXG7mYg0m4D2ADoB7xTKo5CrDaItwONtdbRdju9i4GHsO6e3q+U+iXBx3MHV2O1czwPrXUmcLRUeQCUUtcCQ4Cl9iPo4uv62neBT9t3gpcppa4GHgAOYfkA8HdgD/Crsu58VgFPrLt83N30HLMCrnJw34JQa5HKpCAIrkMpVU8p1R/wvG18DlhUrMhooCvWSxy3aa0/01qfA9BaH9JaPwrMxHoB5dFS8n/Cak+4B+iltV7veYSrtT6jtf5Aa12ifabWOkdr/Uut9XKtdVax5Se01jOwKlMU+wwmLe3PH7TWuT7KHSxV3oPn8fAzxRfaj/JXY93JjMRqGzkE6+7wdcXL2hW9eVg/BG4OKHrfeGI96KNM8XWlj00QhEoglUlBENxAD6XUYXs6AuQCq4A2WHcFx9nt4TyMtj//5qlElsEC+/MWzwL7seft9uxTnpc3HOBd+7O7UqqOz5Lm8dzJPVNBuRz7s/Sj4Nuw7vx5u9ax8zYL63/KS0BDe7trgW+w3sguzaf2Z8B9WvrAn2PLKfa3POYWBAeQfiYFQXAD9Shq51ecLKC/1nqLZ4FSqi5Wu0WAmUqpGeVoeip1lxZb1gXruqixKqt+Y+93NFY3OZ2ARlhtDosTgfXoOzMQ7VBBKaWAKymqiHsYgnVca7TWjxdbvl0pdTuwi/Nz4Xn7vLZ16SQINQ65MykIghvYoLVWWmuFVSG7FquNYiOs/hAvLla2eCWuMVYltKypiV3mgmLbeiqsP2qtf/Q3uGIv4CRhvXDTAigAjgFH7MlD1HkC1ctp+/MCn6WKOjMv/rZ3Y6zK9qFSZT2PsReVWo7Weh/Wo+7SeNpK5lUQRyD4c2yRxf4+VW4pQRD8RiqTgiC4Cq31Wa31duBOrDZ612C1f/RQ/Lp2nacS6mtyIKxJWC+XZGLdnWyutY7UWjfTWrcALilW1on9VQVPm8GLlVIRPsq1sj+LVxw9sZd+quWpGJbXVrGs5Zfbn/t9xBAonv208lGm+LrSlWJBECqBVCYFQXAldsfYj2DdAfylUqq3veq4vQysboACwXMH8SKl1EUBbOd5S/thrfVbduffxQml8aCLv818Xt+cAEqpJkCzUuWhKLctSm1ywv6MKWefZS33vMjzUfmhBown1vZ2f5plUfxt9l0O7lsQai1SmRQEwbVorfdQ9Gh1mr3sHOBpQ3lbgJJbsMaVVgFu66ks/aec9X0DjMMkuyiqNN9SThnP8jzgE89C+y3sfUD3UuU325+/Ki2klLoKuL7UshuxRqPZpLXeXHqbKrDe/rwI623+suhnf36htT5dThlBEAJAKpOCILid6fZnT6XUTfbfc+3PBKVUp9IbFKd4e0t7NBhPh+TPKqUa+BmDp33lT8vQvxCY4KeOcewK4UJ79gGlVIk2nPYdPU93Se9qrbNLSawB2pYat/tdrBdqeiml5iil2imlGiil+tjrPC87XayUehB4D+ut6kTHDgzQWn8FbLdnnyi9XinVCrjLni39EpEgCJVEKpOCILgarfV/gLX27ET78w3gc6yXdT5USo0t3jm2PRrLSKXUBuA3pSTHAyeBdkCqUupnnkemSqkLlFIDlFLvldrGMwrMS0qp3vZbzyilumJ1mt7YkYMthj2qTBPPhNWvI0B48eXljPLyPJCN1QzgHaVUa1uzKVZFvCvWXcmny9j2X/anJ9dorc8ACfY2CVijB2Vj+RINvG4XfQd4Favy3Vdr/WU5x3ZRqWPz/K+KKnVs9cvYfLz9OVQp9WfPDwK78/R3sTqe3wfMLmvfgiBUAq21TDLJJFNITlgVGw18VEG5W+xyGuhuL2uG9YjWs7wAq83fqWLLNPB0GXo/A34oViYX6+WafM+yUuVjsd7c9pQ/U2w/OViPVj3r2pTa9hl7+dwy4vjOXndTGeueoeRxlDedp1ssZ6eLlTuB1WenxuoEfpSPfH9kl72t1PI4rLfs92G1X3wN64WXcfb8cuB+rNGIfPn5kZ/HllDO9hOLlcnHqrx65o8BHYP93ZZJppo0yZ1JQRBcj7bGh/a0V5xkLzsK9MZ60eM9rEqE57H1buAtrDfCny9Dbz1Wf4ovADuwKiQRWB1wvw38olT5fVh9WyZjDUNYB6tytgDoqrVeQ4hh5+xaYA6QgdWdzhGsO4/dtdbJ5W/Nb7Aq2G8rpbydjmut07TWw7TWsVrrq7XW/6e1Pqi1nmnPD9Faz9K+R96pMlrrqViV5RSsHwX1sSq4f8eqSO7wsbkgCAGitNbBjkEQBEFwGUqpUcB8rDuvDwBv6Qr+oSil7gYOaq0/Mh+hIAjVhVQmBUEQhEphVw7fxLrz9x+s/j5XAen6/9u7YxuFYiAIoPMbOFEAKfkFFEALV8T1QBfUQEXEFEATCGkJ/CWIFxAIvRc6cjiyPLtVNf8dXWW02f8zXkIPSdY1ikDAFxAmAWibpuk3yS7J5u74klHA+cltwPk5Y0PQtp638xz4AMIkAA+bZ0f+ZZSXlkkWGYWnY8Y4oX1Vnd52QeBlhEkAANq0uQEAaBMmAQBoEyYBAGgTJgEAaBMmAQBoEyYBAGi7AgB0KE715rC3AAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "results_dict = get_results_rcv1()\n", - "plot_one(\n", - " results_dict,\n", - " xlim=0.75,\n", - " ylim=6000,\n", - " title=\"Results on RCV1-47236-angular\",\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "97528c4a-962c-44b6-b250-f03b60fe516f", - "metadata": { - "jp-MarkdownHeadingCollapsed": true, - "tags": [] - }, - "source": [ - "### Plot SIFT-128-euclidean" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "c55a3ee8-dfd6-4bb9-9cb1-66ce33469058", - "metadata": {}, - "outputs": [], - "source": [ - "def get_results_sift():\n", - " results_dict = {\n", - " \"PECOS-HNSW$^{++}$ (onging work)\": {\n", - " \"color\": \"red\",\n", - " \"marker\": \"X\",\n", - " \"linestyle\": \"-\",\n", - " \"recall\": [\n", - " 0.49571,\n", - " 0.68714,\n", - " 0.82385,\n", - " 0.91453,\n", - " 0.94811,\n", - " 0.97443,\n", - " 0.99149,\n", - " 0.59568,\n", - " 0.79981,\n", - " 0.91674,\n", - " 0.97189,\n", - " 0.98643,\n", - " 0.99524,\n", - " 0.99869,\n", - " 0.64926,\n", - " 0.85598,\n", - " 0.95698,\n", - " 0.99031,\n", - " 0.99637,\n", - " 0.99872,\n", - " 0.99925,\n", - " ],\n", - " \"throughput\": [\n", - " 65812.90,\n", - " 32797.30,\n", - " 24629.80,\n", - " 13685.60,\n", - " 9375.55,\n", - " 5805.03,\n", - " 2917.94,\n", - " 53613.60,\n", - " 34283.30,\n", - " 15798.90,\n", - " 11082.60,\n", - " 7682.11,\n", - " 4690.11,\n", - " 2398.27,\n", - " 46432.80,\n", - " 30219.90,\n", - " 17348.00,\n", - " 9557.86,\n", - " 6556.19,\n", - " 4058.41,\n", - " 2072.55,\n", - " ],\n", - " },\n", - " \"PECOS-HNSW\": {\n", - " \"color\": \"blue\",\n", - " \"marker\": \"D\",\n", - " \"linestyle\": \"-\",\n", - " \"recall\": [\n", - " 0.5640,\n", - " 0.7130,\n", - " 0.8350,\n", - " 0.9187,\n", - " 0.9522,\n", - " 0.9765,\n", - " 0.9922,\n", - " 0.7096,\n", - " 0.8404,\n", - " 0.9281,\n", - " 0.9759,\n", - " 0.9882,\n", - " 0.9958,\n", - " 0.9987,\n", - " 0.8014,\n", - " 0.9083,\n", - " 0.9685,\n", - " 0.9918,\n", - " 0.9964,\n", - " 0.9985,\n", - " 0.9990,\n", - " ],\n", - " \"throughput\": [\n", - " 38457.696,\n", - " 23871.502,\n", - " 13948.193,\n", - " 7802.484,\n", - " 5451.538,\n", - " 3436.597,\n", - " 1810.481,\n", - " 25668.715,\n", - " 16076.992,\n", - " 9331.247,\n", - " 5168.389,\n", - " 3611.553,\n", - " 2284.600,\n", - " 1218.401,\n", - " 17172.999,\n", - " 10774.329,\n", - " 6191.127,\n", - " 3414.756,\n", - " 2394.785,\n", - " 1527.418,\n", - " 831.661,\n", - " ],\n", - " },\n", - " \"HNSW(HNSWLIB-v0.4)\": {\n", - " \"color\": \"gray\",\n", - " \"marker\": \"o\",\n", - " \"linestyle\": \"--\",\n", - " \"recall\": [\n", - " 0.5610,\n", - " 0.7121,\n", - " 0.8340,\n", - " 0.9200,\n", - " 0.9525,\n", - " 0.9774,\n", - " 0.9926,\n", - " 0.7107,\n", - " 0.8397,\n", - " 0.9290,\n", - " 0.9763,\n", - " 0.9887,\n", - " 0.9961,\n", - " 0.9989,\n", - " 0.7997,\n", - " 0.9091,\n", - " 0.9693,\n", - " 0.9922,\n", - " 0.9967,\n", - " 0.9990,\n", - " 0.9992,\n", - " ],\n", - " \"throughput\": [\n", - " 29658.555,\n", - " 19267.245,\n", - " 11726.411,\n", - " 6800.556,\n", - " 4799.454,\n", - " 3042.605,\n", - " 1623.884,\n", - " 18731.696,\n", - " 12097.183,\n", - " 7304.240,\n", - " 4155.413,\n", - " 2949.101,\n", - " 1901.381,\n", - " 1040.917,\n", - " 11694.866,\n", - " 7513.240,\n", - " 4482.334,\n", - " 2546.814,\n", - " 1815.963,\n", - " 1182.672,\n", - " 655.921,\n", - " ],\n", - " }\n", - " }\n", - " return results_dict" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "7b1c7618-b89c-4173-849a-28fe7c27cff1", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqMAAAJ/CAYAAABMY9o8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAEAAElEQVR4nOydaXgUxdaA3wpJCAkECAQEIyKbyK6EXQgIgsiqoOAFJSjignrF5fKpXBW96nXFfUGEKKB4QQXBBVAJqyLgCigIiIqChD0hhDDkfD+qZzJJZk2mJ+mh3+fpZ2aqq06dPme650ytSkSwsbGxsbGxsbGxKQ+iylsBGxsbGxsbGxub0xc7GLWxsbGxsbGxsSk37GDUxsbGxsbGxsam3LCDURsbGxsbGxsbm3LDDkZtbGxsbGxsbGzKDTsYtbGxsbGxsbGxKTfsYNTGJgJRSmUqpUQplVneutjY2FQ8lFINjWeEKKXSPZxPdzvfsAz1ZBgydpVBXZsIxw5GbSICpVRPtwdn8eO4Umq3UuoTpdTNSqmq5a2vjbVRSnVXSk1XSm1SSh1RSjmUUtlKqV+UUh8rpR5SSvVSSsV4KOseBGR4kb/Lx/e5+LGrmMyyHA0DvP4opVQLI2B5WSm1Xil1wk1OzwDl1FFKXaeUmm3YMlspla+U2qeU+lwpdbtSKjFAWUopNVwp9b5S6nelVJ5x/G6kDVdKqUBk2djYhJfo8lbAxiYMxAFnGsclwL+UUkNE5PvyVav8UEo5d7uYIiIPlqcuVkIpFQ+8CQz3cLoq0MQ4+gP/Bm4CXg2bguHjaiCjLAKUUtcDrwCVPJxOBi4yjruVUleJyEofsmoAHwA9PZw+yzguAzKVUkNF5EhZdLexsQktdjBqE4m8Arzs9rk2cC5wB9AMOBv4RCl1rohkl4N+NtZlPjrQBNgJvA6sBw4ACUBDoAswGB0AlZW/gH5+8uQDfwKtfeRZAtQPQN6fAerl3sJ4EvgRiPGjQ3HqogPRk8AnwDJDzlGgATrgHWbo/bFSqquI/OBF1jsUBqK/A08YsgDaAv8CUow87wCXBqHnaYmIZFDGPxw2NoFiB6M2kcg+EdlULC1TKTUT/aN3EVAPGA88HW7lbKyJUupSCgPRZcBgEckrlm0NMEcpdStwMXCsjNWe9PBd9obXfEqpk6WQ54stwG3oQPw7EclTSj1IcMHoMXTQ+LSI7Ct27ltgoVLqdmAqOtB/BuhTXIhSKhXd4wHwK9BORI66ZVmplJoFfIf+I9pfKdVeRDYGoauNjY2J2GNGbU4bRCQfeNAt6eJyUsXGmgx1ez/RQyDqQjRLRWSN+WqFHxH5WkReEJGvfNnBj4ypIjLJQyDqnudZwBk09lJK1fKQravb+6nFAlGnnMPooNZTGRsbm3LGDkZtTje+cXvvsxtVKdVaKfWSUmqLUuqoUipXKbVdKfWGUqqdn7KVlVK3KKW+MCZjnFRKHVZKbVNKfaaUukcpdZ6HcgHNglelnOnqnBjjlvSAh0ksGaG4lmBQSiUppR5USm1QSh0yJp78oZSaZ7RI+ipbxGZKqfpKqacM/Y4b8r5QSl1RFh3RXcdOdpRRlk3gZBqvUUAjD+dj3d7v9CFnu5cypUYpdaFSaobxXDimlMpRSv2klHpBKdXYR7kHnfebH/k+Z7wXy9tHKfWm0pPospWeUPaHUmqR0hM3awR5bQE9Y5RS5yk9Y/4Pt/v2baVUhyDrSzbssk4pdUDpiWx/KaUWKKWG+imboJQaofSkwu+UnlR4UimVpZRaoZS6S/mZuOp2rQ8an9srpeYoPQHuhFJqj/E8ah/MddkEht1Nb3O6ccrt/UlPGZRSCngMuJuSf9gaG8dYpdQDIvKwh/JnoLtxWxU7Vd04mgK90WPZRpbiGsJGOK5FKdULeA+oWexUCnqi0HCl1HvAaH+tcEqprsAC9AQYJ3FAL3TL2tMicldp9AROuL0/D92VbGM+7oHjKQ/nt7q99xSsOnEPDrd6zRUASqnKwDTgGg+nmxvHDUqpG0VkRlnqCkCXGsAcPI+DTTGOgUAdivYMhaLuK4G3gMrF6rwKuEIpdWMQcqYD1YqdqgcMAYYopT4E/iEinoa+fASkeUivDfQwjpuVUpeKyM8B6HMz8BxFY6Qz0M+jIUqpESLygT85NoFjB6M2pxst3N7v8pLnOeBW4/064A10q8pRo/wEoBPwkFLqoIi8VKz8CxQGb++gA60/0cFvXeAC9I+Dz1YRk+iL/nF3Tu4oPtkL4JDbe1OvRSnVBj2OtzI60HgNeB84YtR7p/E6DCgArvQhrh6w0Hh/H7ASOA50AO43zt+plPpERD4vhbrfoCcmAbys9IoMXruYbUKGM8g4SdHWTSdL0GNFzwFuV0plFJ+YqPTyUBONj9uNMmXhfxR+F5YCb6NbZfPQ98Tt6IB0ulJqn4gsLmN9HlFKxQGfG3WCvq9fBb5Hf/fro4ck+LpvSlt3B3QQHI2eRPccsBhtg47AvejnyxY/coYBc9GT4v5AP3M2A3vRK6D8A/1HdzAw08u1RKOv/UNgA3qinkKPEb7MKHMOsEAp1c7Pn9p+hv4/Ac8CPxjy+6MnwsUCM5RSK0XkgK9rswkCEbEP+7D8gZ4lK8bxoI98b7vl+4eH833czt/qRUYlNzlHgRpu5+LQD2ZBT8zwpXMtD2mZRtlMP2XT3fRsGKycAG1VpmsJ0G9fGfILgKFedFjppu9gH9cq6JnUZ3nI0wz9IynAB6XUNQXIcavrODo4vw3oDMQFKKehm4wML3l2Gef/RAfjXo8A63TK21Waaw+wjgfdrqtniGQOcpP5oY98nYEsI99O4Gagu3FMcLv+v4EOZdTpOkOWw9N31shTxe17+SsQ7c1WQXxX0j2cf8Lt/OtAJS9yooAzg5Sd7na+oYfz693s0MfD+fro4NIpo8R3D6gFHDbOvw3EetH/Jjc5vT2cb+rHjn3Qf3YFuM5LHnE7PgUqe8hzjVuef4b6Hjqdj3JXwD7sIxQHPoJR44F3IfCxW55VxX8gjLyfGec/8lNfTQqDm+vd0uu71VEicArgOpw/YJl+8vn7ofApx5utiuUp07UEcK0d3OS/6SPfOehWMQGW+bhWAYb4kDPXyHOgDDr3R7faiocjH/gS3SJUz4eMhm5lMrzk2eWljhJHgHo75e0KtR/d6njQTa+eIZBXm8JgxoGeJe8r/5nAf9HDKYrb6QTwOMUCslLopIBfDJkv+cnbwq3+i73Zyo8M9+9KerFz1Sn8c/QjEBPktXiVbZxPdzvfsNg593v3dR91XOmWr8R3D5hsnNsHxPvR92sj75xS+u4Do/wiL+edeh4H6nrJE4VudRXg/bJ+x+2j8LAnMNlEIkUm5QD70cFnf3TA8AbQX0Qc7oWMrryexsd5vioQkUMULqXTxe3UAaMOgKuVUlYeCmP2tbivZjDdWyYR+RX9JwGguzFezxNHgEU+6ttgvCYFO5nDTZdP0N2vz6Bb2dyJQbfQPQJsV0rdUZo6bMD4rr2Lbo0GvTnDdz7yK2CEcXianBRrnLvKyFtaWqA3NQD/z4gt6HsIij4jQkUv9JJXAC+IiMcx8CbhvsTWTB/5PkC3fHpjiPH6sYjk+qnTuemBX1sak6GaKqVaOQ90yzno8e2++ExEit/bAIhIAYWTYH2NT7YJEiv/UNrYlIatwLMikuPh3PkU7gYzU+l1SQPhDOcbETmhlHoHGIMe7N5RKTUP3Xq3xghiLUEYrsU5FrUA3eXni6/Qa0lWRne5/+ghzzbjx8IbB93eV8P3j6RXRGQPeuzp3eh1NTujx+x1A1oa2eKBp5VSVUXkodLUY/CbiDQsQ/mAUEoloFugvbE1XMGOESzORK8HDHpC2iM+8kehxzM7xxJmAC+hxx2C9smt6C7WJ4EuSqkrReSUm4xzKAzsinNIRJybAaS6pS8PIq49w3+WoLnA7f0qE+T7wrmebAGFS2+VQEROKqW+RQfORVBKVQLaGR/HKKXGBFi3R1sqpbqhh8z0AZJ8lK/tR/5Pfs47nyPFJ1vZlAE7GLWJRNwn5USju5sHA9eiH6IrlN7NpfiM2jqlrC++2OdbgUT0wPkG6Ek4dwKilPoBPc7wVRHJouJj5rU4fzCOiv+1Kvd6KFccfy0r7oGqpy0og8IIfL83DgCUUs3RXcXOFp/JSqm3RGRXWeszmQ7Ach/nz8H7hL9Q8wIw2nj/OTDSz5+MmygMRB8WkfuLnd+ADnZ2o4dQXI4eU/qCW56ZeJ6NDXr713TjfaieEaHAfcWIPSbI94X7vXvCZ86SPQjuMkoTg1QpnmAsx/RAacsXI9DnSJmfITaF2MGoTSRSfAem79DbCS5Cz7ZMAt5WSnV0bx2h6MPlNnz/OLtTZKkR0TN5LzfWo7sS3fV/Afp+a2scdymlRolJs2xDRZiuRUKha0VARH5WSl2O7lLshu66H4qelWvjB6XU4+gJRwBr0WOA/QU744zXHOBRH/keRt/XVdF/TF/wkdcb7s+IYcC2AMtZpkckSMpy77rbchZ6MlbQKKV6UxiI7gSeAlajJzMecw7HUko9BPy71NramIodjNqcNojIR0qpV9GtIhegWzvecMuy3+39cSnjtomitxvcCK5u0O7AKPRSJYnAu0qpxiLi3urn/Nftbzy3ty5FUyjltfjD2d1VXSkV56d11L1r7qDXXBUAESlQSs1AB6Og12Kt0IhIJkX3mw87Sqn70UvngB6Xd6l4XlOyOM4NFzb7+g6J3rJ0C3rZnubFzvUMUE33Z8TRMjwjXC29SqkoHy2/vu5zd13qUcphJ6XEGVxXV0pV9vOHoa6X9APoYFYBUWWw5fVuOnX20Uvjq+veppyxJzDZnG5MobAl8wGllPtkh+8p/Kd/YSgrFZFjIvKpiFyNnkEKuutuYLGszvURiy8AX5zmfs6bRhDX4g/nj08URcfieaKT8XqCwFujypO/3N5HTMuvWSil7kTfm6DHevYVkSMBFndORIwJIK8zj8NnLu986/a+LM8I93VQfd3rvu5z97GaPcqgS2lwjtmOArzuSGRMRGvn6ZwxBtk5rrdbGSaWOcdpL/czXMjfM8amHLGDUZvTCtGLlL9mfDwLPTnHeS4L3TUIeveQM01Sw33B9eKD6Z3bGTYzZveXwJhNPqyMOjhbkLzNTA8UX9fij2Vu76/1lknprQidM+9XBdBtawpB/li6//D52qLytEcpdRO6axX0skl9JLjFxJ32baWU8hrYKb2vvTNwKa1PvkUvNwUwzt8Wkz5wr9/XtpmjfJxbTuEf61vCvHLHZ27vfU08ugzfwbZzk4qG6OEspcF53V5bkZVS51P4h9amAmIHozanI09SGIzdU+wh7tzeMx54XynlNcBSSlVSSo1WSqW4pTVSSnmbCOGkr9v7X4udW2G8xqJ3cSlepwJeRHfLlQXnhAdf+2eX9Vp8IiLr0WsHgp5gUmI7QyPwnknhD05pxvmFileUUvcZQY1XjB++u42Pp9DjlG08oJS6Bj3zHeA39ILmwQz1gMLlvGKBZ43Z9cXrqYS+b2KLlQkKozvdObP/TOAdpZTXyUlKqTil1C3GTknurKWwdfZOLzpfjQ7mvOlyBD1ZE/TKFK94kmPIilJK1fcmK1hE5GsKlzi6TuktfYvXeQaFfzK88Rx64xCAaUqpC3xlVkp19/BM+sV4vVAp1cRDmWT0mFSbCow9ZtTmtENE9iql3kBPlDgH3frwpnFuiVLqafSM8Y7AT0qp19BBYhY6SD0HvcXe5eixjK2B3Yb4BuglX35GL0mz3jhXgJ7VP5TCmbl/oLfPc+cjCrc2fNAIfOajZ3iei545fCGwhsIxiaVhrVHHYKXUDYY8Z4B+1GhBLuu1BML16IC0MrBQKfWyUddRdCvWXRQuIzNPRMozsKsN3ADcr5Ragv5O/IAe+xaFbt25BLiawqDnWRHxtIWl5VFKpRdLauf2/hKjRdvJXhH5tFj5ocAM9JjBXOCf6DGI1X1Uu1tEDhdLewbdsn4GevmmpsbYcOc2lK3Q48SdLZB7jTKlZRp6+aDh6KEpzmfEl+hxi1XR44S7o4PJGujlplyISJZSai561YA+wEdKqRfQfxLPRE8WHI3/+/x+dK9BW/REro7GtX+Hvp/roZceuwq9deeDZbju4tyMnigUDXyilHoW/fxy3w40GT38yePanoYdrkFvAVwb+FIpNRv9LPkdPcmpHtp3QylcpmuFm5i30Dt1JaBXSvkvhUMYugJ3oL8bX2LOeq82oaC8V923D/sIxUGA24G65T+Lwp1atqIH0LufvwfPO7l42tmliRc9fB2/A2296NYFPabMW9nHKPsOTO0o3EGq+JERqmsJ0HcXoScl+apjPl622/R3rW75fNosAD2fC9Aegt4x6r+A8iCnYXFbe8izCy+71pTSxiGVZ8gM1BYefYMO0IKRIXjYKciQ1Qa957y/8tuBNiG49mjj+3AqgDpzgCoeZCSj17T0Vu4ziu7i5O3akyjcOc7X8aCP72EJ2YHcL+gg19tz8iT6z6bTz16/e+g/cVle5BQ/rvFQfoaP/A70H50HnWl+vs8PetOz2Pc2ZPeSfdg7MNmcpojIHxitoehF1EcUO/8YunXjUXTL3QH0Qy0HPYFmPnAjentB95avVej1Ch8FvkB3IR01ymYZaROB80TkezwgIl+iZ/tnoFsiT6LX6lsM9BORe0p/5a46vkMHve+gg0lP4zDLfC0B6vIF2tYPoVs0jqB3fvoTvY7pABEZLv7XIjUVEfknurX4enRrzLfoINqBDuz3oAPjh9A2+T8xfr1szEVEfkAHpDcCn6AnkJ0wjr+MtBvRgegPIajPYXwfWqOX7foO3Sp6Cn2PbAZmo1tq64nIcQ8ysijcsetn9HfoMLoF70b0EBh/a14iIgdFpA+6dXAuerhDHvraf0cPExkPPF3Ky/VV9zvozUJmoe3svG//B1woIq8HKOdTdE/N7eix5HsMWXnoa1iCbmltLiJveSh/LbpHYhX6j/wJtB1mAV1F5LlSX6RNWFAV/VlprCF2C/qHsyY6KPgReE5EPi6Wtyt6dm9n9MK2v6D/Mb0gRdeTdC8zEN0V6Nx9ZzPwsoi86Sm/UWYMuou3Bfrh8y3wlHhZZ9EYr3QbMBb9o3scvaPMf0RkracyNjY2NjY2NjanAxU6GFVKPYGeCLAb/c92P7proz16/9h/ueUdgm5FyUPvaXwQ/U/xXGC+iFzhQf4t6AkRB4wy+ehxQCnA0yJyl4cyT6HHE+5Gt47FAiPRXSW3isiLxfIr9L/E4eju4EVG3hFAHDBMRBZiY2NjY2NjY3MaUmGDUaXU9eiB4m8C40Ukv9j5GDH2SjaWwNkOVAe6icgGIz0O3ZXYBbhKROa6lW+I7ho5BrQXY7s+Y2mQ9ehZxl2NLlNnma7oAeU7gA5i7M1tyNqIHkDdXNy2/lNKXQW8jZ4w0tvZ1aiU6oAe/H0EaCx6pxsbGxsbGxsbm9OKCjlm1FjO5RH0WJESgSiAMxA1GI5uMZ3rDESNPHkULsp9UzER16Jn8L7oHjwaAaZzS7kbi5Vxfn7EGYgaZXahlyepjO6Kd8dZ72T3MW+il7V519B7ePHrs7GxsbGxsbE5HaiQwSh6qYpk9HIPBUqpAUqpSUqpfyqlPC3NcJHx+qmHcyvRg8C7GkFuIGU+KZanVGWMltmuRv2rgqjHxsbGxsbGxua0oKKuM+pcDy4PPTmolftJpdRKYLgUbv11rvFaYptAEXEopX5Fr0/WCL2Uhr8ye5RSx4AUpVS8iOQa+3GfCeSIyJ7iZShceLeZW1pj9KSonSLiafs5T2Xcr3M8ehYklSpVal+5cnCb5RQUFBAVZd7/DbPl5+XlERdXfK3o0GG2/uGow+ryzfYxWN9GVpcP1r+XrS4/HHVY3cfhqMPq8oP1cW5u7n4RSQ4oc3mvLeXpQO8q4Vwf7Af0It9V0ctoLKHYunXogFJwW++xmLw1xvkubmn5Rlq0lzJ/GufrGZ/rG593e8kfY5w/4ZbW1Uhb7aVMU+P8Vn82ad++vQTLBx98EHSZiiS/NNccDGbrH446rC7fbB+LWN9GVpcvYv172eryw1GH1X0cjjqsLj9YHwMbJMC4r0JOYDJ2sxiPXius+ISgePSs9BSMCUZKqW3owK6peNjtRCm1Bh0YuiYkKaXy0QFkjHhotVRK/YkOQOuLbimtjw5Q/xSRFA/5Y9ABbr6IVDbSnBOe1ojIhR7KNEUH0ttE5Nzi591JTU2VDRs2+MoScaSmpnK6XfPphu3j0wPbz5GP7ePIJ1gfK6U2ikhqIHkr6pjRw8brt+6BKICI5KJbR0FvOQZ6Rjro2fSecKYfdksLtMyRYq9m1HHYy/kykZGRYYbYsMk3m3Dob3UfWN3HYH0bWV1+OLC6jSLhWWQ2kWAjq8s3k4oajG41Xg97Oe+cyV6lWP4SYy+VUtHonR0cwE4PdXgqUw+9TNNuI/hFRI6hW0arGueL09R4dR+DugO9KH4jQ49AyoSM/PwSixBYSr7ZhEN/q/vA6j4G69vI6vLDgdVtFAnPIrOJBBtZXb6ZVNRg9HP0WMoWSilPOjonNP1qvH5hvF7iIW8PIB5YKyLuWx76KtO/WJ5SlRG9lNNao/7uQdRTkn37/GaxsbGxsbGxsbEcgQ4uDfcBLEQHpBOLpfcFCtCto9WNtET0XtkngFS3vHHoYFCAkcXknIOerX8AaOiWXhO9gH6RCU9SdELSdqCmW3pDQ06euyzj3FVGmTVAnFt6B0PffUCiP3u0B5H77hMpKPA7aNjJqVOnAs5bGsyWb/aAeLP1D0cdVpcfjglMVreR1eWLWP9etrr8cNRhdR+How6ryzdzAlNFbRkFvff7H8AzSqnPlFJPKqXmAx+ju77HicgRABE5ClyPXkYpUyk13dhK9Dv07kvz0QvMuxCRX9FbjSYBG5RSLymlpqJn7zdGbwf6ZbEya4FnjPM/KKWmKqVeAjYYcu6SYmNcgblG/V2Bb5VSTyil3gCWG/peb+jvn0cegRtuAIenVaJKsnTp0oDylRaz5ZtNOPS3ug+s7mOwvo2sLj8cWN1GkfAsMptIsJHV5ZtJRV1nFBHZrZRqD9wPDEZ3tx9F7+3+mIh8XSz/AqVUGnAfMAzdKroduAN43ojSi9fxglJqF3AXcA162MIW9G5Jb3rR606l1I/oYHk8upX2G+BJEVnsIb8YW4KuRe/6dCu6BXUl8B8jwA2c11+HrCx45x3ws97X77//HpToYDFbvtmEQ3+r+8DqPgbr28jq8sOB1W0UCc8is4kEG1ldvplU2GAUQPSi9rcaRyD51wCXBlnHInSAG0yZDCAjiPwOYKpxlI5ateDAAf1+wQK45BJYuBCqe5uob2NzenPkyBH2799Pu3bt+Omnn/wXKCW2fP888cQTlr4Gq8sPRx1W93E46rC6/Mcff5wdO3ZQu3Ztqoc49qjQwaiNGw0bwtix8NRT+vOKFZCWBp98AvU8Te6HAQMGmKqS2fLNJhz6W90HVvVxXl4ef//9NykpKaSkpBDs7mXBkJ+fT2xsrC3fByLCeeedZ5p8q9soHD4wuw6r+zgcdVhdfkFBAfXq1WP37t1Urlw5pDtuVeQxozbFefJJeOKJws/ffw/dusH2Euv8A5CVleUxPVSYLd9swqG/1X1gVR9nZWWRnJxMfHw8p06dMrUuR4BjuE9X+eHA6jYKhw+s7udIsJHV5SuliI+Pp3bt2iH/bbCDUatx992QkQGVKunPv/6qA9JvvimR9euvvy6RFkrMlm824dDf6j6wqo/z8vKoWrUqAMeOHTO1Llt++WN1G4XDB1b3cyTYyOrynVSrVo28vLyQyrSDUSsyZoweN1rFWPN/3z7o2ROWLy9PrWxsKgwOh4PoaHsUko2NjU2oiY6ODnkrrB2MWpWBA2HZMqhRQ3/OztaTmt57z5Wlffv2pqpgtnyzCYf+VveBlX2slAIgPj7e1Hps+eWP1W0UDh9Y3c+RYCOry3fifLaGEjsYtTLdusGqVVC/vv6cnw9XXAGvvQZASkqKqdWbLd9swqG/1X1gdR8Dpk96sOWXP1a3UTh8YHU/R4KNrC7fTOxg1Oq0agVr10KzZvqzCNx4Izz0EAsXLDC16oULF5oq32zCob/ZdVhdfjg4fPiwLb8c5YcDq9soHD6wup8jwUZWl28mdjAaCZx9NqxeDR06FKY98ADd5s4Fk2cS29jY2NjY2NiUBXuEfwVHKTUIGJSSksK0adMA6NixI8nJyXz00UcANGjQgL59+5IxejR9c3JIMRa9bZmZyY5OnVg+diyXjRzJ9u3b+f777wHo2rUr1apVY8mSJQA0atSIHj16kJGRAeixJ6NHj2b+/PkcPHgQgBEjRrBp0yY2b97syrNjxw4+//xzAJo1a0aHDh2YM2cOAImJiYwcOZK5c+dy9Kje8XTUqFGsX7+ebdu2AdC7d28cDgcrVqzQOrdsSatWrXj33Xfp3r078+fPZ/jw4cyePZvc3FwA0tPTWblyJTt37gSgX79+ZGdns3at3syqbdu2NGnShPeM8bN16tRh6NChZGRkkJ+fD8C4ceM4deqUy6YDBgwgKyvLNXu8ffv2pKSkuFoG69evz8CBA5k+fToFBQVERUUxbtw4Fi9ezF9//QXAkCFD2L17Nxs3bnT5qXr16q46nH6aPn06oLtU0tPTWbBgAfv27QNg2LBhQfnJOXvSl5/S0tKIjo4ulZ+OHDnCtGnTfPoJICkpqVR+SkhIcNnHm5+WLl3q2lkkUD9lZ2e7lh6JiYnh8OHDnDx5EoAaNWqQn5/v0jMhIYHo6GiOHDni8ktiYiL79+8H9Pio2rVrc+jQIdeg/Zo1a5KXl8fx48c5deoUx48fJyoqymW/ypUrU7VqVQ4YG1VERUVRq1YtDh486Fpqyl0G6BmqANnZ2QDExcW5lqbKysqiUqVKJCUlFZGRlJREbm6ua2ZrcRlVqlQhLi6OQ4cOAbhkHDhwgIKCApd9jh49yokTJ1zfh4KCAnJycjzKiI6OpmbNmuzfvx/nxna1a9fm6NGjLr9Vr14dh8Ph+n7GxsZy8uRJV8tNTEwMNWrUKLI8THJycqn9dOrUKfbv3+/TTwBVq1YtlZ+gcJkzb35y3n+l8RPoSXe+/FSrVi1ycnJK7adTp04hIj79FB8fT2xsbKn8VK1aNbKyssp8P3nzU0xMjEuPstxPvvwUHR1NdnZ2me4nX346deoUOTk5Zb6fvPnJ+azw5aeyPPeioqLIycnh+PHjZGdns2nTJp+/T8GgPOySaVMBSU1NlQ0bNvjPmJ+vZ9vPnVuY1rs3fPABGDeWVUhNTSWga7axLGb5+KeffjJ1AW6b4NiyZQstWrQobzUiil27dtGzZ0927dplaplAsX0c+bj7OJBnrFJqo4ikBiLb7qaPNGJjYc4cuNVtB9XPP4devfQSUCHE2cJnVcKhv9l1WF1+OLD6xgBWlx8OrG4jq23AsWnTJqKjo1m2bFnIZPojUP0zMzNRSpWqdc7qfi4uPyMjA6UUmZmZPsstXLiQ2NhYfvnlFxO1840djEYiUVHw3HOsHzKkMG3jRrjwQgjhP2Jnt4RVCYf+Ztdhdfk2Zcf54+t+VK1alfbt2/Pcc8+5uiA95VNKUadOHZRSXtdlzc3N5dlnn6V79+4kJSURExND3bp1ufTSS8nIyCix3uDRo0d5+OGHueCCC6hWrRrx8fEMGjSIu+++m7///ttjHTt37mT8+PE0b96c+Ph4atasyXnnnceYMWNYHuD6yc7re8q5ZbIHlFIMHDjQq/1ef/11j+Xq1KlTolygOn/66acopZg8eXIJuV999RVKKVJSUlzdp+5ccsklREVFubpOKwp33HEH3bp14+KLLy5vVWxCwJAhQ2jdujWTJk0qNx3sMaORilJ8P3AgHQYM0LPrCwrgl1+ga1dYsgRaty5zFVFR1v4vEw79za7D6vIth/M+OvdcV5I6cgSSk8tRKc1VV13FpZdeiojw119/kZGRwe23387mzZtd43Ld8zk5evQoiYmJHn29fft2BgwYwLZt2+jTpw/33HMPtWvXZt++fXz22WeMHTuWLVu28ISxTfG2bdvo168fv/32G5dffjnXXXcdMTExfPrppzz33HPMnDmTRYsW0aVLF1cdGzZsIC0tjZiYGK655hpatmzJ8ePH+eWXX1i6dCnVqlWjV69eJlqukAcffJDRo0dTxbmhiBeC0fnCCy8kOjraY+vU8uXLiY6OJj8/n7Vr19KnTx/XOYfDwerVq2nVqhW1a9cO6XWWhS+//JJly5axwOTVWkpLjx49OH78ODExMeWtiqX45z//yZgxY9i8eTMtW7YMvwIiYh8WONq3by+l5r33RCpXFtELP4nUqCGyalXp5YWJMl2zjSUwy8dbtmwJvdBTp0TGjtX30OWXi3z4oX5VSr9+913o6wyA5cuXCyBPPvlkkfQjR45I/fr1RSkle/fu9ZrPG7m5uXLuuedKdHS0vPfeex7zfP311/LSSy+JiMixY8ekWbNmEhMTI4sXLy6Sb/PmzbJ+/XqpXr26JCcny969e13nBg4cKIB858V+e/bsCUjfQK4PkAEDBngsl5qaKoA8+uijfssFq3OXLl0kNjZWjh07ViS9b9++0r9/fznjjDPkvvvuK3Ju7dq1Asitt97qSrvpppukevXqUr16dalWrZoopVyfq1evLqs8PNdLU8YXo0ePltq1a0t+fn6R9M2bNwclx8YcHA6H63s2c+ZMAWT58uV+y2VnZ0t8fLzccsstXvO4+ziQZyywQQKMcexmjwhm8eLF+s3ll8Onn0Jiov58+DBcfDF8+GFo5FuUcOhvdh1Wlx8OQrL2XkEBjBsHM2fqz++/D4MH61cR/dquHQwbBsZKCKGitPonJibSpUsXRMS1okEw8qdPn87WrVu58847ufzyyz3m6dChAzfffDMAb7zxBtu2beP2229nwIABJfKmpqby6KOPkpWVxZNPPulK/+WXX6hVqxZt27b1WMcZZ5zhVXd/1xAMV155Je3bt+fxxx93za73RrA69+rVi/z8fNasWeNKczgcrFmzhrS0NLp06VJiOIKzJdW9Vfjll1/m8OHDHD58mB9++IEGDRq4Ph8+fJgLL7ywhC7OMrt27Qq4jDccDgcLFiygT58+Hlse9+/fz4QJEzjrrLOIjY3lrLPOYsKECSXs6RzL+MUXX/DUU0/RuHFjKleuTLNmzXjzzTdLyN21axeDBw8mMTGRxMREhgwZwq+//krDhg3p2bNnCbsVHzMaaH3O79GuXbsYNmxYQPUV57fffkMpxQMPPFAkvV+/fiilmDp1apH0Tp06lZgIFKwdP/vsMx5++GHOOecc4uLi+N///udTx0ceeQSlFLfeeqtrKFbVqlVdK9iUB3YwGsE4lxwC9N71K1ZA3br6c16eDlKdP65llW9BwqG/2XVYXX44cC5tUiZ++SWwe+X99+GFF3SAGiJKq7+IsH37doAi3by5ubns37/fdezdu5f9+/e7ltFx4vxRGj9+fED1BZI/PT2dmJgY17JrAI0bN+bAgQO8//77gV2YB9xtVPz63A9fKKX473//y5EjR3jkkUd85g1WZ2dA6d5Vv379eo4dO0ZaWhqdO3d2fXbiDKrS0tICqsMfobgPNm7cSE5ODh07dixxLjs7m65du/LKK6/Qr18/nn32WS655BJeeeUVLrzwwiLLWDm59957mTVrFjfccANPPPEEUVFRpKenFwnaDxw4QPfu3fn0009JT0/n8ccfJyEhgV69ehWxVyD4q+/kyZOu+hYtWlSq+s4++2waNWrEF1984UrLz89n9erVREVFFUk/evQoGzdu5KKLLnKlHTlyJGg73nXXXcydO5fRo0fz3HPPca7bMCJ3Tp06xU033cTkyZN57LHHeOGFF4oMz+nSpQt79+7l559/9m/MUBNoE6p9WK+b/rXXXiuZuH27SOPG4uqyB5H//lekoCA08kOI2d30ZusfjjqsLj8c3fT79u0rPOH+va9ohxeK6O8BZzfzlClTJCsrS/bt2yfff/+9jBs3TgDp3LlzkXzejuLd10lJSZKYmBiwzZOSkqRatWoez7l377Vu3VoAyc7OFhHdHR0TEyOANG3aVMaOHSsvv/xyUEMt9u3b5/f6vF1n8e79iy++WCpXriy7du1y5SleLlidc3NzJTY2Vrp06eJKe/TRR6Vq1apy8uRJWb16tQCyZMkSERE5efKkJCQkSNu2bb1e86+//ipnn312UDYKtkxxZsyYIYAsXLiwxLnx48cL4Bq24eTFF18UQCZPnuxKc3Yft2vXTk6cOOFK3717t8TGxsrIkSNdaXfffbcA8vLLLxeR60xPS0srku7058yZM4Oub9++fS65s2fPDqg+T4wbN05iYmJc3eUrVqwQQIYPHy7VqlWTkydPiojIhx9+KIDMnz/fVfbee+8N2o7NmjWTY8eOlXhWuHfT5+bmytChQyUmJkbefPNNj3rPmjWrhD7umNlNX+5Bln2YF4y6j8sqdkKkXTsp8kM4caIeExcK+SHC7GDUbP3DUYfV5YcjGC0ytq28A85SBKPFx+YVx1sQFhUVJYMHD3b50Jlv/PjxsmzZMtfxySefyLJly0qMf4yOjpYzzzwzYJtHR0dLSkqKx3PuP2LdunUTQHbv3u1K++GHH2TMmDFSp06dItfQvXt32bFjh9+68/PzvV6f+xFIMPrNN9+IUkquvvpqVx5P5YLVuXv37hITEyM5OTkioseL9u3b16V/nTp15J577hGRwvGit99+u99rDxR/36NAePzxxwXwOM60UaNGkpyc7Aq0nJw8eVKSk5OlZcuWrjRnkDRt2rQSclq1alXkudC8eXOpV6+e5OXlFcn3999/Bx2M+qsvPz/fVd+pYr+H3urzxNtvvy2AfPrppyIi8sADD0idOnXkyy+/FEDWrl0rIiITJ04UpZQcOHDAVfa8884L2o5Tp0516e+O8/x7770n3bp1k6pVq7p08sTHH3/sMfB3YmYwas+mj2B2795NXWe3vDt160JmJgwdql8Bpk6FrCyYMQMCnIXoVb5FCIf+ZtdhdfnhID8/39IzawPVf/z48VxxxRUopUhISKBZs2YkJSWVyNe0adMis7aPHTtGQkJCiXyJiYkeuwS9kZiYWKKr3xPOPNWrV3eltW7d2jXG77fffmPFihVMnz6dVatWMWTIEDZu3EhsbGyRHWacVKlSpciyVMWvL1jOP/98rrrqKubMmcNdd91FmzZtPOYLVGcnvXr1YtWqVaxevZrevXuzZs0a7r33XkD7uEePHq5xo87ufH/jE4MhFPeBUgrQjVjF+fPPP+nQoUOJJcKio6Np1qwZ33zzTYkyjRo1KpFWq1YtfvvtN9fnX3/9lY4dO+JwOKhcubIrvU6dOtSoUSMo/f3Vl5+f76qv+OoSwdTn7Hb/4osv6NevH1988QW9evWiZcuW1KxZky+++IIuXbrwxRdf0LZt2yL36a+//kpqampQdmzWrJlLf08+Tk9PJycnh5UrV/ocI+z0q9PP4cQeMxrBOLel9Ej16vDJJ3rcqJPZs2HIEAhwHI5P+RYgHPqbXYfV5YeDIus3lrbd8tAhCPQBnZFRujoC0d8HziCsd+/edO7c2WMgGoz8Vq1acfToUZ+Tnzzld45T9VbXzz//TMOGDalatarHPGeffTbXXHMNK1asoFu3bmzatMm1/esff/xBvXr1ihz//Oc/A7ZRoPznP/8hOjo64HUXfensxH3cqPt4UdB2SUtLY8OGDeTk5JCZmUlUVBQ9evQI2TWFwkbJxhJmzi01y0qlSpU8pnsKdkOhv7/6QvU9qlu3Li1atOCLL74gNzeXdevWcdFFF5GXl0daWhqff/45Bw4c4IcffigyXrS0xMfHA971HzFiBFFRUTz88MOu7VI94fRrcjksVWcHo6czcXHwv//BDTcUpn3yCfTpA35mk9rYnFbUqAGXXeY/39ixcPXVpqsTDoYNGwYEvguXc8a9r/xvvfUWJ0+e9Do73x2lFJ06dQJ0qxvoWerLli0rcvzrX/8KSL9gOOecc7jpppv49NNP/e5e409nJ126dCEuLo7ly5eTmZlJlSpV6NChg+t8WloaDoeDzMxM1qxZQ7t27ahZs2ZIridUtGrVCsDjTj0pKSls3bq1xCYIDoeDbdu2eWyVDISGDRuyffv2Ehtw7Nu3LzQrZZhU30UXXcQ333zDokWLyM/Pp3fv3gD07t2btWvX8sknnyAiJYLRRo0ahdyOo0aNYvbs2XzxxRcMHDjQa9Dq/CPp9HM4sYPRCMbTjMcSVKoEr7wC//53YdpXX0H37vDHH2WXX4EJh/5m12F1+eHAUxd0qbj//qKfGzcu+nnsWJg+Xe+AFkJCpn+Q8seNG8e5557LU089xcKFCz3m2bhxIy+//LIrf5MmTXjmmWf49NNPS+T95ptvuOeee0hOTubuu+92pS9btqzEDy/A8ePHWbp0KYBrP+y4uDj69OlT5GjRooUpNpo8eTKJiYkeg91gdHZSuXJlunTpwsaNG1m8eDFdunRxdeMnJCTQqlUratWqxZNPPsmxY8dC2kXvrKOsnH/++SQmJvLVV1+VONe7d2+ysrJK/Bl5/fXXycrK4rJA/sx5YNCgQezZs4ePPvqoSLqvnbZKS0JCgqu+d955p0z1XXTRRRQUFDBlyhQaNGhA48aNSUhI4KKLLuLEiRM89thjREdHl2j9Hjp0aKnt6MvHI0eO5J133mHVqlX079+fnJycEnm++uor6tat63U2vpnYY0YjmICb2pWChx6COnXgttt0l+FPP+ndmpYuhWJroAUtv4ISDv3NrsPq8sOBt20ug6ZtWz2s5f33deA5bRqMH6+XfDIpEIUQ6m/wzTffMHv2bNdnh8PhqmPo0KGu7vP4+HgWL17MgAEDGDp0KH379uXiiy+mVq1aZGVlsXz5cpYsWeIK1hISEvjwww+55JJLGDBgAMOGDaNnz55ER0ezdOlSFi1aRNWqVVmwYEGRdTgnTpzIgQMHGDx4MK1btyY+Pp4//viDt99+m23btnHNNdfQ2s+OcaG2EejlsO6++27+7f5HvYw69+rVi+XLl7N27VqmTJlSRH+lFN27d3ftbBTqXadCYaNKlSpx+eWXs2DBAk6cOFFkDOe1117L8uXLmTBhAt988w3nn38+3377LW+88QbnnntuqVuwJ02axNtvv82NN97I999/T/PmzVm1ahVr166ldu3aIR3f6Bya8fbbbzN27Fi+/vrrUtfXs2dPoqKi+Omnn0hPT3fJb9GiBWeccQZbtmyhc+fOVKtWrUi5f/3rX8ybN69UdvTn4+HDhxMTE8OVV15Jv379+OSTT0g01h/Pyclh1apVXHvttQFdX8gJdKaTfVhvNn2pluWZO1ckJqZwJFtSksiXX4ZOfhDYSztFvvywL+1UVr77TuS66wpXnjh1So48/3zQK1EEQ6BLO/nbWSmQpY9++eWXEuWOHTsmzzzzjHTr1k1q1Kgh0dHRUqdOHbn00kvlrbfeEofDUST/4cOHZcqUKdK2bVtJSEiQuLg4Oeecc+TOO+/0uJvSkiVL5Oabb5Y2bdpIrVq1pFKlSpKUlCQ9e/aUN954o8SsZm82KusOTJ7KHTt2TOrVq1eiXGl1XrVqlcvWmZmZRfQXEZk6daoAUqlSJTl8+LDf6w6GUN0H69atEzws/7N582bZt2+f3HTTTXLmmWe6VmO4+eabJSsrq0heXzsDpaWllVh+aufOnXLppZdK1apVpVq1ajJ48GDZuXOn1KpVS/r3718kr6/Z9P7qc9po586dctlllwVUny8uuOACAeStt94qIv8f//iHAHLvvfd6LFdaO/pa2smdxYsXS+XKlaVTp06u71lGRoYA8uOPP3q9HntpJ/sIXzAqIrJ0qUhCgrgC0vh4kY8/Dp38ALGD0ciXb7lgVKTEmrwhl18Mq8sXMX+rSKvbKBw+CGUd/fr1kwsvvLBIWrh9vH//fgHkhhtuMK2OUNdXkb9H559/vlx22WU+89jbgdqUigYNGpSu4MUX6yWfnLu25ObqrQ/nzAmN/ApCOPQ3uw6ryw8H7svrhIRi3XQhl18Mq8sPB1a3UTh8EMo6nn76ab788kvX+FizOX78eAn9//vf/wJw8cUXh6weZx2eZpyHor6K+j1asGABmzZt4vHHHw+xRoFjjxmt4CilBgGDUlJSmDZtGqAnlSQnJ7sGdDdo0IC+ffu6BjzHxsaSnp7O8ePHXWWGDRvG9u3b+d7YN7tr165Uq1aNJUuWAHoGX48ePVzr5sXHxzN6zRqOXXghCVlZ4HDA6NH8um4dy4yZdt27d2fHjh18/vnngF7rrEOHDswxgtbExERGjhzJ3LlzXWsLjho1ivXr17Nt2zZAD3p3OBysWLECgJYtW9KqVSveffdd1z65w4cPZ/bs2a4ZgOnp6axcudK15Ey/fv3Izs5m7dq1ALRt25YmTZq4thysU6cOQ4cOJSMjg/z8fEBPtgBc9hkwYABZWVmu5Vjat29PSkqKa+JG/fr1GThwINOnT6egoICoqCjGjRvH4sWLXVtmDhkyhN27d7uWQ+rYsSMtW7Z01eHNTwsWLGDfvn2l9hPorRidy3KMGDGCTZs2sXnzZkDP0o2Oji6Vn37//XemTZvm008ASUlJpfJTQkKCyz7e/LR06VJ+//33oPyUnZ3tWo+ydu3aHD582LUdYo0aNcjPz3fpmZCQQHR0NEeOHHH5JTEx0bV9pFKK2rVrc+jQIdeklZo1a5KXl+f60Tp+/DhRUVEu+1WuXJmqVau69pKOioqiVq1aHDx4kFOnTnmU4Rw75lzbMy4ujvj4ePLz88nKyqJSpUokJSUVkZGUlERubi55eXkeZVSpUoW4uDgOHToE4JJx4MAB12zhWrVqcfToUU6cOOH6PhQUFLgmOBSXER0dTc2aNdm/f7/uXjNsfPToUZffqlevjsPhcG2fGBsby8mTJ12zkWNiYqhRo0aRNUOTk5PL5Kf9+/f79VPVqlVL5aeoqCiXrt785Lz/SuOnuLg4HA6HXz/l5OSUyU8i4tNP8fHxxMbG+vXTmWeeyZ49ewC9hWZ+fj7VqlUjKysrJPdTcT8NGzaMc845h3PPPZeCggJWrVrF0qVL6dixI926dSMrKyvg+8mfn7Kzs+nfvz8pKSl06NCBgoICPvvsM5YuXUqnTp0YOHCgyx6l8VNOTk6Z7ydvfnI+K5x+CvR+ctowMTHRVd6Tn6KiosjJyeH48eNkZ2ezadMmn79PQRFoE6p9nEbd9O78+adIq1aFXfYg8n//J1JQYNkuXCd2N335y7dkN30xbPn+sbvpy1d+OOow08dPPfWUtGjRQhITEyUmJsY1Bvno0aMhrcdpo6eeekratGkT8vqs/j2yd2CyKT/q14eVK2HQIFizRqf997+wbx8qNbV8dbOxsbGxiXjuvPNOrrnmmrCt7nHnnXdy5513hqUuG409ZjSCCdn4lJo19RJPAwcWps2YQb9p08DHbg4VHauN04pE+eHA7K3tbPnlj9VtFA4fWN3PkWAjq8s3EzsYjWCca5uFhPh4+OADcJPZ4LvvoF8/MGEXjHAQUvuUUx1Wlx8Oajsn4tnyy0V+OLC6jcLhA6v7ORJsZHX5ZmIHoxGMc/HkkBEdDTNmgPuiu6tWQY8eYEzisRIht0851GF1+eHAOVHAll8+8sOB1W0UDh9Y3c+RYCOryzcTOxiNYJwztEOKUvD44/D004VpP/4I3bqBMUPeKphinzDXYXX54cDTto22/PDJDwdWt1E4fGB1P0eCjawu30zsYNSmdNxxB8vHjtWtpQC7dsGFF4KxrJGNjY2NjY2NTSDYwWgEM2zYMFPlt3nySfjwQ6hSRSdkZUHPnvDZZ6bWGyrMtk846rC6/HBQs2ZNW345yg8HVrdROHxgdT9Hgo2sLt9M7GA0gtm+fbv58vv3h88/1zPuAXJy4NJL4X//M7XuUGC2fcJRh9XlhwPnQuO2/PKRHw6sbqNw+MDqfo4EG1ldvpnYwWgE49zFx3T5XbrA6tWQkqI/nzwJI0fCyy+bWn9ZMds+4ajD6vLDgaet/Wz54ZMfDqxuo3D4wOp+jgQbWV2+mdjBqE1oaNFCL4rfvLn+LAITJsCDD+r3NjY2NjY2NjYesIPRCKZr167hld+ggV7qqVOnwrQpU+Dmm8HY+7ciYbZ9wlGH1eWHg6pVq9ryy1F+OLC6jcLhA6v7ORJsZHX5ZmIHoxFMtWrVwi+/dm09hrRfv8K0V1/V3fYnTpiqT7CYbZ9w1GF1+eEgKir0j7mCAnjzTf1qhnx3rC4/HFjdRuHwgdX9HAk2srp8M7Gu5jZ+WbJkSfnIT0jQs+z/8Y/CtPnz9WSno0dN1SkYzLZPOOqwuvxwcDTE37mCAhg3Tm9GNm4cHD5s7nc61PqHW344sLqNwuEDq/s5EmxkdflmYgejNuYQGwuzZsHttxemLV+ul376++/y0srGpkw4A9GZM/XnmTNh4sSqFBSUn06ZmZkopYocVatWpX379jz33HOcMobIZGRklMinlKJOnToopWjSpEkJ2Q6HgxkzZnDxxReTnJxMbGwstWrVolevXrzwwgvk5uaWKPPDDz8wZswYzj77bCpXrkxSUhLp6enMnDnTpUtxVq5cyeDBg2nYsCGVK1emTp06pKamctttt7Fz586A7JCeno5Siv3793s877z++fPnu9J27drlssN9993nsVzDhg3p0aNHifRFixZx8cUXk5KSQuXKlalXrx5du3blX//6l0uHTz/9FKUUkydPLlH+q6++QilF5cqVPdrxkksuISoqyuv12NhEEtHlrYCNeTRq1Kh85UdFwTPPQN26cM89Ou3bb/VuTUuXgsn6+cNs+4SjDqvLDweVK1cOiZzigaiTd96pQlwcTJ+uv/KhJlD9r7rqKi699FJEhL/++ouMjAxuv/12Nm/ezLRp01z5brvtNjp06OD6fPz4capUqVJiSEZWVhaDBw/mq6++olOnTtx+++3Uq1ePw4cPs3LlSiZOnMiqVav4n9sybq+88gq33norNWvWJD09nebNm3Po0CFmz57NtddeyzvvvMMHH3xAQkJCkTI333wzjRo1YsyYMZx11llkZWXx008/8c4779CjRw+/38NQ+PjZZ5/llltuoV69eiXOKaWKfJ40aRJPPPEEbdq04eabb6Zu3br89ddf/Pjjj7z66qtceeWV1K5dmwsvvJDo6GgyMzNLyFy+fDnR0dHk5+fz3XffcfbZZ7vOORwOVq9eTatWrUK233io7oPyIhz6m12H1eWbiojYhwWO9u3bS7CcOHEi6DKmyZ8+XSQqSkTPrRc54wyRb74R+fnnovkOHXK9Lc01B4PZ9glHHVaXb5aPt2zZ4np/6tSpMss7dUpk7NjCr6+nY+xYnS/U+NN/+fLlAsiTTz5ZJP3IkSNSv359UUrJ3r17ZebMmQLIvHnz/MovKCiQHj16CCDPP/+8x3q3bdsmjzzyiOvz0qVLRSklbdq0kaysrCJ5N2/eLPfdd58AcvXVV7vST548KTVq1JAGDRrIkSNHStRx4sQJOXDggM/rd17DmDFjBChRtxNP1//rr78KIKmpqQLI+PHjS5Q7++yzpWXLlq7Pf//9t0RFRUmHDh0kPz+/RP7s7GzJzs52fe7SpYvExsbKsWPHiuTr27ev9O/fX8444wy59957i5xbu3atAHLrrbf6vfZACcV94IvNmzebKt9s/cNRh9Xlu/vY/RnrDWCDBBjj2N30EUxGRkbFkX/ddfD+++D857Z3r55137w5DBsGixbp16Qk/RqG9S3Ntk846rC6/HBw4MCBMpX31iJanJkzdb5Qd9mXVv/ExES6dOmCiPjs6vYkf/HixaxcuZIRI0Zw6623eizXtGlT7r33Xtfn//u//wPg7bff9tia9/DDD9OpUydmzZrFjz/+CMD+/fs5fPgwHTp0IDExsUSZ2NhYkpKSfF+ol2sIhk6dOnHZZZcxY8YMtm7dWuK8+/CCnTt3UlBQQI8ePYiJiSmRt2rVqkVmNffq1Yv8/HzWrFnjSnM4HKxZs4a0tDTS0tJYtmxZERnOltRevXqV6brcKauNyptw6G92HVaXbyZ2N30FRyk1CBiUkpLi6mrr2LEjycnJfPTRRwA0aNCAvn37Mn36dEA/wNPT0zl8+LCrzLBhw9i+fbtrEfOuXbtSrVo11wSVRo0a0aNHD1fwER8fz+jRo5k/fz4HDx4EYMSIEWzatInNmzcDereHHTt28PnnnwPQrFkzOnTowJw5cwD9Yzhy5Ejmzp3rGlh99cKFxFx+OdG5uXpxfNBB6vvvF1608fmNJk1Y9tRTXHzXXcyePds1rio9PZ2VK1e6fmD79etHdnY2a9euBaBt27Y0adKE9957D4A6deowdOhQMjIyyM/PB2DcuHEcPXrUZZ8BAwaQlZXF119/DUD79u1JSUlh4cKFANSvX5+BAwcyffp0CgoKiIqKYty4cSxevJi//voLgCFDhrB79242btzo8lN+fr6rDm9+WrBgAfv27SuVn5y+8eWntLQ0oqOjg/LTqFGjWL9+Pfv372fatGn07t0bh8PBihUrAGjZsiWtWrXi3XffBSApKYnhw4cH7aeEhASXfbz5aenSpfz+++9B+Sk7O5usrCzXV+rw4cOcPHmSOnWSMZOZM/0Hrd44flzvnpKdnQ1AXFwc8fHxOBwOsrKyqFSpEklJSRw8eNAVHCUlJbnsnZOT49qBJTs7GxFh27ZtgJ5l65SbnZ3Ntm3bKDCiZhEhJycHpRQJCQkkJia6/HrllVeSk5NDXFwchw4dAiA6OpqaNWuyf/9+dOOHlvnNN9/QsWNH6tSpQ35+Pg6Hg2PHjgH6u+5wOLjyyitZt24db7/9No899hhRUVEkJCSwYsUKtm7dSt26dTlpPBdq1KhBfn6+6/oSEhKIjo7myJEjLpmJiYns378fh8PhuvZff/3VNc6yRo0a5OXlkZeXx969ewE4efKk67vhtMnx48e5++67+fDDD7nnnnuYPn16kQBURFxl6tevD8DChQsZM2YMDRs2JD4+3nX/FffT+eefD8DSpUtp164dAJs3b+bYsWO0adMGpRTvv/8+R44ccX3vly1bhlKK1q1bu+qtVasWOTk5nDBWJUlMTKSgoICcnBwAqlSp4tNPDocDEeHo0aOueqpXr17ET/Hx8cTGxnL48GEAYmJiqFGjRpF7KTk52XU/ufupWrVqZGVl+fQT6CEPtWvX5tChQzgcDkBvY5mXl+datL1q1apERUW5nkeVK1cu4oOoqChq1apV5F4oLsM57KT4/eTNT04/Z2dnu75LxWUUt7FTxoEDB1z3ky8/ORyOgO6n2rVrl8pPzmeFLz8Fcj9581NUVBQ5OTkcP36c7OxsNm3a5PP3KSgCbUK1D+t108+aNSvoMmGRv3Ch+OzvdD+uu06koCC0ihuYbZ9w1GF1+eHopt+/f7/rfaBfu/I4vOGuvyec3fRTpkyRrKws2bdvn3z//fcybtw4AaRz584iUthN7e2YMGGCS+YFF1wgQEBd5CIiH374oc9uZWf33saNGwWQYcOGuc499dRTAkilSpWkQ4cOctttt8ns2bNlz549AdUtom3k7Kb3d3jqpnde+/XXXy+AfPnll648Z599tjRv3rxIfbfccosAEhsbK927d5e7775b5s2bJwcPHiyhW25ursTGxkqXLl1caY8++qhUrVpVTp48KVu2bBFAlixZIiJ66EJCQoK0bds24OsPBH/fo7Jidje92fqHow6ryzezm95uGY1gRo8eXTHlDx6s1yH1s2zQh7VqMXjaNCg2eSBUmG2fcNRhdfnhoFatWuWtQpkIVP8HHniABx54wPU5KiqKwYMHF5m8BHD//ffTvXv3EuXPOuss13tni5SnrnNPOPNXr17dZz6nPGdrDMCdd95Js2bNeOWVV1i5ciXr168HdKvTmDFjeOGFF4iPj/cp191G7733nke9ly5dypNPPulTzoMPPsicOXP417/+xcqVK13plSpVKpLv+eefp2PHjsyYMYMvv/ySVatWAboF75///CePPvqoq0yVKlXo1KkTX331FceOHSMhIYHMzEy6du1KdHQ05513HnXq1CEzM5O+ffuyfv16jh07FtIuejh97oOKXIfV5ZuJPWY0gnFfwqTCyZ8wwff5xo155OyzzZmebGC2fcJRh9XlhwNntxyUrs3y1CkYOzawusaO1flLU08g+vti/PjxLFu2jM8++4wvv/ySrKwsFi5cSN26dYvka926NX369HEdF1xwAX369OHcc8915XEGc87uSX94CjI94S1oHTRoEB9//DFHjhzhhx9+YOrUqZx11lnMmDGDiRMnuvLt3bu3yOG0jbuNevToUeT6nEeLFi38Xkf9+vW5/fbbWbVqFYsWLXKlF1+SSinF1VdfzfLlyzl69Cjr16/nkUceITExkSeeeIInnniiSP5evXpx8uRJVq9eXWS8qJPOnTuzfPlyoHC8aM+ePf3qGwyBfo8qKuHQ3+w6rC7fTOxgNIKp0F98f2NKduzgvt9+C/1sEDfsh1v5yw8H3ta2DJSoKL1sk7+AdOxYc5Z3ClT/pk2b0qdPH3r37k3nzp0DmvjjTX6rVq0A+PbbbwOS4cz/zTff+MznPN+6dWuP5ytVqkTr1q25/fbbWb9+PdWrV+fNN9906VivXr0ix+WXX+71GkrLpEmTqFWrFvfee69rHKAvYmNjSU1N5d5772XVqlUopXjjjTeK5HG2cmZmZrpaPt2D0S5durBhwwZycnLIzMwkKirK49qmZSGUNioPwqG/2XVYXb6Z2MGoTfg5fBg++MBvtsEHDuiF821syhl/AalZgWh5MWzYMADXZDt/nHPOObRr1461a9fy008/ecwjIq4g7bLLLvMrs3bt2jRu3JgTJ064JlUsW7asyPH0008HpF8wJCYmMnnyZDZt2sSbb74ZVNlzzz2XmjVr8ueffxZJ79KlC3FxcSxfvpzMzEyqVKlSZK3XLl264HA4yMzMZM2aNbRr146aNWuG5HpsbKxAhDw6bTwxYsSIiim/Rg0I4Mfoh4QEuPrq0tURAGbbJxx1WF1+OAjVj7q3gHTMmAJTA1GzgxJP8gcNGkSPHj145513ePnllz2W2759O4899pjrs/P9qFGjPC4x8+CDD/LVV19x9dVX06ZNGwByc3NdKzQU55dffmHLli3Url2b5GS9CkLxrvf27dt7vYaycPPNN9OwYUMeeOABTpw4UWTM6N69e/nuu+88llu1ahUHDx4sMSSgcuXKdOnShY0bN7J48WK6dOlCbGys63zXrl2pVasWTz75JMeOHQt5Fz2Y/z0ym3DoXx73mpXkm4k9gSmC2bRpE926dauY8u+/v+hyTo0bw44dRbK0OXYMXnnF//jSUmK2fcJRh9Xlh4O8vLwi6z6WBWdACnoJp7Fj4dlnc4mKCo18T4RSf9ABk3PpGqf8uLg4QAeSzu0x58+fz6BBg5gwYQKzZs1i8ODBnHHGGRw+fJjVq1fz4YcfulpQQW9f+fzzz/PPf/6T8847j7Fjx3Luuee6dmD67rvvuPjii3nllVdcZXJzc+nZsyetWrXikksuoWnTpogIP//8M2+99RZ5eXm89NJLRPmJ9N2vJxTExsby8MMPc7XxZ9j9R3737t106NCBTp060bt3bxo1asSJEyf4/vvvmTNnDjExMTz66KMlZPbq1Yvly5ezdu1apkyZUuTciRMn6N69OwsWLHDlDTWh/h6Fm3Dob3YdVpdvJnYwGsFs3rzZ1ECiTPLbtoXLL9cB6dixMG0ajB+vf+GTk8G5rt0tt+hXEwJSs+0TjjqsLj8cHD9+PKQPaGdAmpamG+4PHDgOmPcDEGr9n3/+ea/nRo4cSXS0/llITk5m1apVvPXWW7zzzjs8/fTTHDlyhMTERNq2bctzzz3H2GLNxLfccgvdunXjmWee4e233+bvv/8mISGBpk2bMmPGDK655poirYw1atRgxowZLF26lA8//JA9e/aQl5dHcnIyaWlp3HrrrQEFZs61JUPJqFGjePrpp/nuu+8QtxlmzZs356WXXmLZsmXMnTuXv//+m5MnT1KvXj2GDh3KnXfe6Vpb1B3363AfL+rUPy0tjQULFlCpUiWPqx2UlVB/j8JNOPQ3uw6ryzeVQNeAsg/rrTP62muvBV0mrPK/+06vI+rcwuzUKZGMDL0laOfORScbv/himfUtjtn2CUcdVpcfjnVG9+3bZ0odtvzAMXsNSqvbKBw+MLsOq/s4HHVYXb69HahNqSj+77vCyW/bFl5/vXCwXVQUjBmjx5QuWaLHjDq55RZ46aWy1VcMs+0TjjqsLj8cOHdRseWXj/xwYHUbhcMHVvdzJNjI6vLNxA5GIxhnd1uFlu9tQfvERG5t2hQ6dy5MC3FAarZ9wlGH1eXb2NjY2NjYwWgE49yL3Kryj1WqpHdpMikgNVv/cNRhdfnhINCF22351sXqNgqHD6zu50iwkdXlm4kdjNpUbBIT4dNPTW0htbGxsbGxsSk/7GA0gmnWrJml5buoXt1zQOpl/cNACYf+VvdB2HxsIs5li2z55SM/HFjdRuHwgdX9HAk2srp8M7GD0QjGfYcPK8ovgqeAdMKEMgWk4dDf6j4Iq49NIj4+3pZfjvLDgdVtFA4fWN3PkWAjq8s3EzsYjWDmzJljafklcAaknToVppUhIA2H/lb3Qdh9bAIHDx605Zej/HBgdRuFwwdW93Mk2Mjq8s3EDkZtrEX16npSU4gCUhsbGxsbG5vyxV63pYKjlBoEDEpJSWHatGkAdOzYkeTkZD766CMAGjRoQN++fZlu7FMYGxtLeno6x44dc5UZNmwY27dv5/vvvwf0XsjVqlVjyZIlADRq1IgePXqQkZEB6Ob+0aNHM3/+fNe/rREjRrBp0yY2b94MQFRUFDt27HDNuG7WrBkdOnRwtaYlJiYycuRI5s6dy9GjRwG9q8n69evZtm0bAL1798bhcLj2p27ZsiWtWrXi3XffpXv37syfP5/hw4cze/ZscnNzAUhPT2fNPffQYuJE6v76qzbUhAmsWrOGn9LSaNu2LU2aNOG9994DoE6dOgwdOpSMjAzy8/MBGDduHHl5eS77DBgwgKysLL7++msA2rdvT0pKCgsXLgSgfv36DBw4kOnTp1NQUEBUVBTjxo1j8eLF/PXXXwAMGTKE3bt3s3HjRpefYmJiXHV489OCBQvYt29fqfx05MgRAJ9+SktLIzo6ulR+OnjwINOmTfPpJ4CkpCSPflq5ciU7d+4EoF+/fmRnZ7N27VoA2rZtS0JCgss+3vy0dOlSfv/996D8lJ2dTZaxi1elSpU4fPgwJ0+eBPSuP/n5+S49ExISiI6OdtkyNjaWxMRE9u/fD4BSitq1a3Po0CEcDgegt4fMy8vj+PHjOBwOjh8/TlRUlMt+lStXpmrVqq492qOioqhVqxYHDx7k1KlTJWRA4RqBzhmxcXFxxMfH43A4yMrKolKlSiQlJRWRkZSURG5urms7zOIyqlSpQlxcHIcOHXLZIikpiQMHDlBQUODS7ejRo5w4ccL1fSgoKCAnJ8ejjOjoaGrWrMn+/ftdOxPVrl2bo0ePuvxWvXp1HA4Hx44dc9n05MmTHD58GICYmBhq1Kjh8hHoHZ9K6yeHw8H+/ft9+gmgatWqpfKTiLh09eYn5/1XGj+JCA6Hw6efatWqRU5OTqn95HA4EBGffoqPjyc2NrZUfqpWrRpZWVllvp+8+SkqKsqlR1nuJ19+ioqKIjs7u0z3ky8/ORwOcnJyynw/efOT81nhy09lee5FRUWRk5PD8ePHyc7OZtOmTT5/n4JBOS/epmKTmpoqGzZsKG81wkpqaio+r/nIEejXD9atK0x7+WW46SbzlbMJCX59XEp++uknzjvvvJDLtSkdW7ZsoUWLFuWtho2J2D6OfNx9HMgzVim1UURSA5Ftd9NHMHPnzrW0fL946rK/+WZ45ZWAiodDf6v7oNx9HAKsPk7L6vLDgdVtFAnjIc0mEmxkdflmYgejEYyzi8Oq8gPCGZB27FiYFmBAGg79re6DCuHjMuLsgrPll4/8cGB1G4XDB1b3cyTYyOryzcQORm2sT/XqsHRpyYD01VfLTycbG5uQsW/fPqpXr87rr79e3qrYVCCee+45atWq5Rp/aWNd7GA0ghk1apSl5QeFp4D0ppt8BqTh0N/qPqhQPi4lSUlJIZWXm5vLN998w4oVK/jmm29MX2jan/6ZmZkopXjqqae85lFKMXDgQNfnnj17opSiUaNGVK1atUT+Bx98EKVUifG8O3fuZPz48TRv3pz4+Hhq1qzJeeedx5gxY1i+fLkrX7169TjrrLM86tK8eXOUUq6JfO7MnTsXpRRPP/10kfTJkyeTnJzM2LFjXWm7du1CKcUtt9zi1UYNGzakVatWRdLS09NRSlG9enXXxCV3MjIyUEoxf/58V1pSUhJ79+7lrrvuolWrVlSrVo3ExESaNm3KyJEjef/99115O3fuTOXKlV0TRdy55JJLUEoxefLkIulJSUl89dVXKKW49dZbXenF/eYNT/mcPnYederUoXbt2nTq1IlXX33VtFa0devW0adPH5eNLrnkEr777rtSy5s0aRJKKc4555wS52644QYqV67Mww8/XAaNCwn1syLS5JuJHYxGMOvXr7e0/KAJMiANh/5W90GF83Ep8BQUlAYRYeXKlUydOpVFixaRmZnJokWLeO6551i5ciVmTQYNlf6e+PXXX3n++ecDyrthwwZat27N//73P/r27cvUqVN54IEHuOiii1i7dq1r9QrQgdDu3bvZvn17ERl79uxh69atREdHk5mZWaIOZ0Dbq1cvV9ru3buZMWMGt956K9HRnheAKY2Njh49yn/+85+A8v7888+0bduWl156ic6dO/Pf//6Xxx57jIEDB/Lzzz8zc+ZMV95evXqRn5/vWjXCicPhYPXq1R6vPTc31+O1l5XKlSsza9YsZs2axWuvvcb//d//cerUKW666SbuuOOOkNXj5KuvviItLY1ff/2Vhx56iClTpvDLL7/QvXt3fvzxx6DlfffddzzzzDMe/zCBniF/44038vLLL3v8YxEsZt5rkSDfTOxgNIJxLp9kVfmlwtMYUi8BaTj0t7oPKqSPg8S5TEtZWbVqFcuXL3ctc+Lk1KlTLF++nFWrVoWknuKESv/iVKlShRYtWvDkk0+6lq7xxZQpU8jNzWXFihU8//zz3HDDDdx+++289NJL/PLLL0Va+5wBVfGgy/n56quv9hiMZmZmUqNGDdq1a+dKe+2111BKcdVVV3nVrTQ2Sk1N5ZVXXuG3337zm3fq1Kns27ePuXPnMn36dCZMmMCECROYOnUq3333XZHhA96uff369Rw7doyrr76a9evXFwkc8vLyXC3caWlpQV+LN6Kjoxk9ejSjR4/msssu46677mL16tXUq1evVMvv+OO2224jNjaWlStXMnHiRCZOnMjKlStRSnHnnXcGJevUqVNcf/319O/fn/bt23v9szd69GhOnDgRkusx616LFPlmYgejNpFHjRqeA9LXXis3lWysTW5urt9gc9WqVZZqmYiKiuKxxx7jwIEDPPHEE37z//LLL9SqVYu2bdt6PH/GGWe43jsDMveue9ABWvPmzbnyyiv5888/+eWXX1zn9uzZw7Zt2+jRowdRUYU/TfPmzSM1NZU6deoEdX3+eOyxx8jPzy/RZe4J51q5vXv39nje/dq7detGTEyMx2uvWrUqd9xxB/n5+axZs8Z1zuFwsGbNGlq3bk2tWrVKczkBExcXR1JSErGxsX7zOrvIf/jhhxLnjhw5QpUqVRg6dCgAv/32G+vXr+eKK67gzDPPdOU788wzueKKK/jss8/Yu3dvwHo+//zzbNmyhRdeeMFnvkaNGnHuuecyb968gGXbVDzsRe8jGG8PTqvILxPOgLRfPzAWSOfGG/XrDTcA4dHf6j6o0D4OEOfC1U4yMzNdi/f744ILLmDQoEH8/PPPJVpEi+NwOHjyySeLpKWlpdGzZ88iae+8806RFmdPeXzp743c3FzXgtWBMnjwYLp27crUqVOZMGFCkaCqOI0bN2br1q28//77XH755T7lNm3alDPPPNNjy2ivXr3o1q2bq7u6adOmrnNQtJv677//ZuvWrdx2221e68rLy+PEiRMer925ELkn2rVrxz/+8Q/mzJnDXXfd5TXIBr1RxPLly3n99de5/fbbUUp5zZuQkEDHjh35+uuvOXbsGAkJCa7r69atG61ataJu3bpkZmZy8cUXA7B161aOHTsW0i56J0675OXlsWXLFt588002b97Mfffd57fsmDFjeOKJJ3jrrbdKjEn+3//+R15eHmPGjAFg06ZNAHTp0qWEnM6dOzNjxgw2btzIgAED/Nb722+/8e9//5sHHniAs88+G8Cnzbt06cLs2bPJycnx2qUfCIHea6erfDOxW0YjGH8/nhVdfplxBqQdOhSm3Xijq4U0HPpb3QcV3sdhIpBu7PLmgQceIDk52ePhi//85z8cO3aMBx980Ge+yZMnExMTw7Bhw2jWrBnXXnstr7zyCj/99JPH/L169eKvv/5yBd7Ols+ePXtSrVo1LrjggiKth85g1D0w37JlC6ADYW+88cYbnHXWWR6v+48//vB77TExMUyaNMlnvn/+858kJiZyxx13cPbZZzNq1CieffZZ125rnq795MmTrtZPZ8un89rS0tKKXLuz1d3Xn5LScOzYMZctGjRoQMuWLXn66ad54IEHAhov26JFC1JTU3n77bdLTHh66623qFWrliu4dO78494q6sSZ9ueffwak90033USjRo0CHtfauHFjHA4HW7duDSi/TcXDDkYjmEBbfyqq/JBQo4ae1OQhIA2H/lb3gSV87IdQBJLl2eIQqP7jx49n2bJlHg9ftGrViqFDh/LGG2/4HCPcpUsXNm7cyJgxYzhy5AgzZ87k5ptvpkWLFvTo0cPVle2k+NhJ56tzTGRaWlqRltPMzEySkpJo06aNK80Z4PiaJTxkyBDmzZvn8brr1q3r89obNmzIzTffzJIlS/jiiy+85qtduzbff/89EyZMAODtt99m4sSJpKam0qZNmxJBafFrd44Xdb/2DRs2uLZ3/OKLL4iKigrpeFHQXfJOW8ybN4/Zs2czZMgQpkyZwkMPPRSQjDFjxrBnz54i36Nff/2VNWvWcNVVV7m6+53bcFauXNmjHhDYBJt33nmHTz/9lFdffbXIhDVfEwSdQxuc2yqXFrP/dFpdvplU2G56pdQu4Gwvp/8WkRL9SUqprsBkoDNQBfgFmAG8ICIe17FQSg0E7gLOByoBm4GXReRNH7qNASYALYBTwLfAUyKy2Ev+SsBtwFigKXAc+Ar4j4is9VTGJoQ4A9K+fcE5O/zGG2k+ahSMH1+uqtmEn549ewbdAtW8eXM++eQTny3F0dHR3HHHHVSpUsWnLF8TccpC06ZN6dOnT6nKPvbYYyxatIh77rmnyKz44rRu3do1UeS3335jxYoVTJ8+nVWrVjFkyBA2btzoCk7cx41eeOGFri75evXqAToge/LJJ9m6dSuJiYls27aNoUOHFhkv6uya9RWIpKSkkJaW5rEFOJBltyZPnsyMGTOYNGkSXzuH9HigYcOGvPjii7z44ovs2bOH1atXM2vWLBYtWsTAgQPZvHmzK2ju2rUrlStXdrV+ZmZmEh8fT2pqquvaT548yerVq+nduzdff/01bdq0oWbNmn71DYZKlSq5vhNZWVkkJyczatQo+vfvz4MPPsjw4cNp0aIFx48fd+1R7qR69epUqVKFq666ijvvvJO33nqLSy65BNCtoiLCNddc48rv/N4792R3xzmxJj4+3qe+Bw8e5Pbbb+e6666ja9euAV+n8/vhqyvfpmJT0VtGjwBTPBwlFtRTSg0BVgI9gA+AF4FYYCrgcU9DpdQtwCKgFTAbeB2oD2QopTwu2mekZwD1jPyzgdbAIkNe8fzKqP8ZQ58XDf16ACsNvU2hZcuWZokOi/yQ4qGFtMecOTBtmqnVWt0HlvKxF/wFh4EQHx9P9+7dfebp3r17SOoqjhkyi8tv3rw5Y8eO5f3332fdunUBlTv77LO55pprWLFiBd26dWPTpk1FgrlzzjmHs88+29W6npmZWaTl78ILLyQqKorMzEyP40UBV4Dpb5vDstioVq1a/Otf/2LDhg3873//C0h+vXr1uOKKK/jwww/5xz/+wd69e/n4449d5+Pi4ujcuTMbNmwgJyeHzMxMunbtSkxMDKC7v2vXrk1mZqZrZr0Z40W9XUO/fv0QEZfd3333XerVq1fkePfddwFtn0svvZQFCxa4Wt5mzZrFeeedRwe356nTV5664p1pnrrw3ZkyZQrHjh3j+uuvZ/v27a7D2eq6fft2j0MvnN8Pf0NS/BGOe83K8s2kogejh0XkQQ9HkUBRKZWIDgxPAT1F5DoRuRtoB3wJDFdKjSxWpiE6qD0IpIrIBBGZCLQBdgB3KqW6FCvTFbjTON9GRCaKyASgvSHnKUOuOyOB4cBaoJ2I3C0i1wG9DH1fV0qZ0gdYfLFnq8kPOZ667G+4wdSA1Oo+sJyPPRCqRem7d+9Or169Sqx1GR0dTa9evfwGq6XF7EX1nfKnTJlCfHy83/GTxVFK0alTJ6BkINKrVy/27NnDunXr2LZtW5FgtHr16rRr147ly5d7DUadf4bcZ937uobSMnHiROrVq8fkyZM5efJkUPI7d+4MeL52h8NBZmYma9asKXLtSil69Ojh89pDjfs1OK/RGVz269evxBCHfv36ufKPGTOG48ePM2/ePNasWcOOHTtcE5ecOJ8VX375ZYm6nQv6t2/f3qeOv/32G8eOHaNTp040bdrUdXz99dfk5ubStGlT+vfvX6Lc9u3biY6O5txzzw3QGp4J171mVflmUtGD0UAZDiQDc0XEtWWIiOShu+0BbipW5lqgMvCiiOxyK3MIeNT4eGOxMs7Pjxj5nGV2AS8Z8sYWK+Osd7Khj7PMeuBdQ+/hfq+wFDj/2ZqF2fJNwRmQGt1lgKkBqdV9YEkfFyNUWwU6A4iJEycyaNAgevbsyaBBg7jmmmvo0aOHaV2EZm916JRfv359/vnPf7JixYoirXxOli1b5nGYwvHjx1m6dCmgW/zccQZYL7/8MkCJMZFpaWmsWLGCzMxMateuXeLPT3JyMi1btuSrr74K6BpKS3x8PA8++CDbt2/3uOXo4sWLXa1z7hQUFLBo0SLA+7U/+eSTRcaLOklLS2Pjxo0sXryYqKgoevToUaZr8IfTRiLCwoULAVzBYb169ejTp0+RwzmcAmDAgAHUrl2bt956i7feeouoqChGjx5dRP7ZZ59Namoq8+bN46+//nKl//XXX8ybN4+LLrqoyGoN+/fv5+effy4yPGDSpEnMmzevxNGiRQvi4uKYN28eU6dOLXFtX331Fe3bty/TTHp3G5mF1eWbSYUdM2pQWSk1GmgAHAN+AFZ6GP95kfH6qQcZK4FcoKtSqrKInAigzCfF8gRSzyfAv408DwAopeKArkb9nhYp/AS42igz08N5GzOoUQOWLWNfu3bUcS54bSz3ZI8htfFHfHw8F1xwgeuzc5JNJDBp0iSmTZvmceetiRMncuDAAQYPHkzr1q2Jj4/njz/+4O2332bbtm1cc801tG7dukgZZ0C2YcMGzjnnnBJbhKalpTF16lT27t3LsGHDPAb0V1xxBQ8//DB79uwpEiCFmuuuu45nnnnG47W//PLLjB07lkGDBnHBBRdQvXp19u7dy3vvvcfGjRvp1atXiSWLOnfuTJUqVVi5ciVxcXF0dF/3GH3tDoeDtWvX0rZtW6pXr+5Rr+3bt3ud+T5x4kTX0lGecDgczJ49G9A7TuXl5fH++++zZs0a+vbtG/DSbTExMVx11VW8+OKLbNy4kT59+njscn/uuedcvQTObU1feOEFCgoKSmzx+uKLLzJlyhRmzpxJeno64HlZKGfeXbt2MXx4yXabHTt2sHXrVp/b4dpUfCp6MHoGMKtY2q9KqbEi4j7N19k2X2IqqIg4lFK/Ai2BRsBPAZTZo5Q6BqQopeJFJFcplQCcCeSIyB4Pujr7kZq5pTVGT4raKSKeZj54KhMy7H1wfVCjBl9OmcKQF18E5/7bN9wASsH114esGqv7wNI+NqhUqZItP0D51atX57777vO4pM4zzzzDwoULWb16Ne+99x6HDx+mevXqtGnThkmTJrkCCnfOOussGjduzI4dOzzOFO/evTtKKUTE66Sy66+/nv/85z+8/fbbXnfxCYWNKlWqxGOPPeZxDdU777yTpUuXsnLlSpYsWcLBgwdJSEjgvPPO4+mnn2bChAlFJl4BxMbG0rVrVz7//HPXfvXutG7dmpo1a3Lo0CEuvPBCr3pt3bqVf//73x7PjRs3zmcweuLECa6++mrX57i4OJo0acIjjzzCnXfeGVRr/pgxY3jhhRfIyckpMnHJna5du5KZmcnkyZOZPHkySim6du3KvHnzfK7jGgjedJ09ezaVK1f2+P0Llki6l62GMms/5bKilHoA3Zq4GchGB5K3AOOBPKCLiHxv5N2GnqXeVES2e5C1Bt1C2VVEvjTS8oEYIMZToKiU+hM9mam+EZzWB/4E/hSRFA/5Y4B8IF9EKhtpXYE1wBoRKfG0UUo1RQfD20SkxGAXpdR443qJi4trHwmTSYLhp59+4rzzzjO1jqoOBy/98gst3ZYc+U+DBiwo40B4m8Awy8dPPPGEzwXcbcJLXl5emcazTZkyhbVr17J48WLXJCCbikVZfVwaTpw4Qb9+/ejfv3/QY51tgsfdx3v37uVf//qXz/wbN27cKCKpPjM5ERFLHehJRwJ84Ja2zUhr4qXMGuN8F7e0fCMt2kuZP43z9YzP9Y3Pu73kjzHOn3BL62qkrfZSpqlxfqu/627fvr0Ey6xZs4IuU5Hkl+aag8Gl/8GDIqmpIlB4TJsW2jpMwuryzfLxli1bXO/3799vSh22/MDZvHlzmcr//fffkpiYKK+99prH81a3UTh8YHYdZfWxPzzpP3XqVKlZs6YcPHjQtDpCidXlu/vY/RnrDWCDBBjbWXEC06vGq/tob+cIaM+DbgrTD5eizJFir2bUcdjL+TJh9j7ZVtqH2xMu/WvWLDmpafx48DCRodR1mITV5YcDX1tC2vLNlx8K6tSpw5EjRxjvZUy31W0UDh9Ywc++8KT/7bffzsGDB0O2PqvV/WxlH1sxGHXOFnAfKOPcA6zE2EulVDRwDuAAdgZYpp4hf7eI5AKIyDF0a2lV43xxmhqv7mNQd6CXb2pk6BFIGZvywFtAOn16+elkY2NjY2NzGmDFYLSz8eoeWDr3cbvEQ/4eQDywVgpn0vsr079YnlKVEb2U01qjfk+LEHqrJySEYkB3eco3mxL6OwNS97Xwrr++TAGp1X1gdR9D4VaBtvzykR8OrG6jcPjA6n6OBBtZXb6ZVMhgVCl1njF7vXh6Q/QORqB3PnIyH9gPjFRKpbrljwOca2K8UkzcTOAEcIv7QvVKqZrAvcbHV4uVcX6+z8jnrtcEQ17xJZqc9f7H0MdZpgMwAt3S633/vTKwcuVKM8SGTb7ZeNS/Zk1YtixkAanVfWB1HwPk5OTY8stRfjiwuo3C4QOr+zkSbGR1+WZSUZd2GoHeAWkl8Bt6Nn1jYAAQB3yM25agInJUKXU9OijNVErNRe+INBi9hNN89ALzuJX5VSl1N/A8sEEp9S56UtNwIAV4WoyZ925l1iqlngHuAH5QSs1Hb/E5AkgCbhW3BfQN5gKXG3K/VUotAmoZZSoB14vI0dIayhc7d+70n6kCyzcbr/o7A9KLL4aNG3Wac7mnceNCU0eIsLp8MxERlFIe98oOJbb88sfqNgqHD6zu50iwkdXlOxETVmGqqMHocnQQeT7QDT1+8zCwGr3u6CwpZg0RWaCUSgPuA4ahg9bt6MDx+eL5jTIvKKV2AXcB16Bbiregd0t605NiInKnUupHdEvoeKAA+AZ4UkQWe8gvSqmr0N311wK3opemWgn8R0TWBm4Wm7DhLSBVCq67rnx1s/FLdHQ0DofDXgbIxsbGJsQ4HI4S2yKXlQoZjIpe0H6F34wly60BLg2yzCJgUZBlMoCMIPI7gKnGETbc9xa2onyz8au/p4DU2TIaYEBqdR9Y1cdxcXHk5ORQs2ZNEhMTTa3Lll/+WN1G4fCB1f0cCTayunwn2dnZIV9TtkKOGbUJDdnZ2ZaWbzYB6e8MSN22f2TcOHjjjdDVUQasLt8skpOTycrKIjc3l1Oniu8eHFqsvlyLlZeDcWJ1G9lLO/knEmxkdfkiQm5uLvv37yc5xBvDVMiWUZvQsHbtWlq1amVZ+WYTsP41a8Jnn0GfPvDNNzotwBZSq/vAqj6Oi4ujbt267N27l/3791OtWjXT6srOzrbl+2Hv3r1BbT0ZLFa3UTh8YHYdVvdxOOqwuvy9e/dSpUoV6tatG/rdtgJdHd8+yvcozU413nYrCRVmyzd7B6ag9T94UOSCCwp3aVJK5I03QltHkFhdvtk+FrG+jawuX6QC3sunmfxw1GF1H4ejDqvLD9bHRPgOTDYB0rZtW0vLN5ug9Xe2kDq77EV0C+mMGaGrI0isLj8cWN1GVpcfDqxuo3D4wOp+jgQbWV2+mdjBaATTpEkTS8s3m1LpX3wMqZ+A1Oo+sLqPwfo2srr8cGB1G4XDB1b3cyTYyOryzcQORiOY994zZS39sMk3m1Lrn5QUcEBqdR9Y3cdgfRtZXX44sLqNwuEDq/s5EmxkdflmYgejNjalIYiA1MbGxsbGxsY7Qc2mV0rVBnqh91nvBNQDaqN3ITqM3pLzZ/Ti9CtF5OtQKmsTHHXq1LG0fLMps/7OgLRPH/j228KAFODaa0NThx+sLj8cWN1GVpcfDqxuo3D4wOp+jgQbWV2+mSjxs62T0ms1XApch96O0xnA+lrDwSl0FzADyBCRP8uk6WlOamqqbNiwobzVCCupqalY4poPHiwMSEHv0jR9uisgtfGOZXxsUyZsP0c+to8jn2B9rJTaKCKpgeT12TKqlLoc+A96a06F3vryJ2AD8D26JfQQenvLmsbREOgAtAfOAR4C7ldKTQceFpG9AV+JDUqpQcCglJQUpk2bBkDHjh1JTk7mo48+AqBBgwb07duX6dOnAxAbG0t6ejpPPfWUa0eGYcOGsX37dr7//nsAunbtSrVq1ViyZAkAjRo1okePHmRkZAAQHx/P6NGjmT9/PgcPHgRgxIgRbNq0ic2bNwN6S7B+/frx+eefA9CsWTM6dOjAnDlzAL0bxMiRI5k7dy5Hjx4FYNSoUaxfv55t27YB0Lt3bxwOBytW6A23WrZsSatWrXj33Xfp3r078+fPZ/jw4cyePZvc3FwA0tPTWblypWvf9H79+pGdnc3atXpn1bZt29KkSRPX+Jk6deowdOhQMjIyyM/PB2DcuHE8++yzVK1aFYABAwaQlZXF11/rxvz27duTkpLCwoULAahfvz4DBw5k+vTpFBQUEBUVxbhx41i8eDF//fUXla++mn84HMT8+COIIOPG8ccff/DJGWe41t7z5qcFCxawb9++Uvnp0KFDTJo0yaef0tLSiI6OLpWfvvzyS2rVquXTTwBJSUml8lNCQoLre+3NT0uXLuX3338vtZ+io6OpXbs2f/31FwBDhgxh9+7dbDR21Qr0fvLmpwMHDjBo0KAy30/e/PT8889Tq1atMt9P3vwUHR1NgwYNynw/+fJTgwYN+PvvvwO+n4L104EDB6hXr16Z7ydvfsrIyKBKlSo+/VSW597x48dJT08v8/3ky08HDhxg0qRJZb6fvPmpe/fuTJs2rcz3kzc//f777zgcDp9+Kutzr1KlSjRu3LhM95MvPx04cICLLrqozPeTNz899thj1KpVq8z3kzc/JSQksG7duoDvp6DwtuYTsA44BTiAL4DrgaRA14wyZHQDXgT2oAPZHGBYMDLsw15n1CxCrv+BAyLnn19kHdLl11wT2jqKYfvYP1a3kdXli1jwXo4w+eGow+o+DkcdVpdfXuuMng+8CTQXkYtE5HURORhkoLtGRG4BUtDd/LuBlsHIsLGxDElJeh3S88/Xn0VImzULZs4sX71sbGxsbGwqMF7HjCqlGojI7yGtTPdX1hd7/GjQlGbMqLNbxSzMlm/2GCTT9Pc0hvSNN2Ds2JBXZfvYP1a3kdXlg4Xv5QiRH446rO7jcNRhdflmjhn1qnWoA1FDptiBaPhYunSppeWbjWn6e2gh5brroDTjaPxg+9g/VreR1eWHA6vbKBw+sLqfI8FGVpdvJvY6oxGMc/CzVeWbjan6GwHp/rPO0p9F9Oz6EAekto/9Y3UbWV1+OLC6jcLhA6v7ORJsZHX5ZhJUMKqUqqKU6qGU6hBA3g5G3rjSq2djY2GSkvho4sSiLaQmBKQ2NjY2NjZWJtiW0dHAcmBkAHmvDyKvjQkMGDDA0vLNJhz697nySt1l366dTghxQGr72D9Wt5HV5YcDq9soHD6wup8jwUZWl28mwQajw43XWQHknYZem/TKIOuwCRFZWVmWlm824dA/KyurcAxp8YD0zTdDI99ErO5jsL6NrC4/HFjdRmF7FlmYSLCR1eWbSbDB6LlAPnrBe398Y+RtHqxSNqHBuUCuVeWbTTj0d9VRq1bJgHTsWL3s09atRQsdPhy8fJOwuo/B+jayuvxwYHUbhfVZZFEiwUZWl28mwQajZwA54m09KDdEpADINsrY2Nh4CkivvRaaN4dhw2DRIv2alKRfvw/kP5+NjY2NjY21CTYYPQrUUEpV8ZfRyFMDOF4KvWxCQPv27S0t32zCoX+JOpwBadu2RdPffx8GD9avIvq1XTu/QantY/9Y3UZWlx8OrG6jcnkWWYxIsJHV5ZtJsMHoD0aZywPIOwyoBGwKVimb0JCSkmJp+WYTDv091lGrFrz6amAC3n8fXnhBB6iByg8hVvcxWN9GVpcfDqxuo3J7FlmISLCR1eWbSbDB6P/Qk5KeUUp53dZTKdUKeAYQo4xNObBw4UJLyzebcOjvtY7OnWHgQP8Cxo6FadP0Lk7ByA8RVvcxWN9GVpcfDqxuo3J9FlmESLCR1eWbSbDB6Az05KVkYL1S6kWl1AClVGvjGKiUehn4GqiNbhWdFlqVbWwihPHjfZ9v3FgHoiZvgWdjY2NjY1OeRAeTWUQcSqkBwGKgHXCTcRRHAd8Bg0XkZBl1tCkl9evXt7R8swmH/j7r8LfW6I4dMGSIntjkJSC1fewfq9vI6vLDgdVtVO7PIgsQCTayunwzUQFMjC9ZSKlYYBxwNdCewqDWAWwE3gLeEJH8EOl52pOamiobNmwobzXCSmpqKhF7zYcP61nzgdx/3brBe+9B3bqmqxVuItrHNi5sP0c+to8jn2B9rJTaKCKpgeQtVf+fiOSLyMsi0gWIRy/fdAYQLyJdROQVOxAtf6ZPn25p+WYTDv291lGjBlx2WWBC1qyBc8+FF18EhyMw+SHC6j4G69vI6vLDgdVtVK7PIosQCTayunwzCaqb3hMi4gD2hUAXGw8opQYBg1JSUpg2TQ+/7dixI8nJyXz00UcANGjQgL59+7q+iLGxsaSnp3Pw4EFXmWHDhrF9+3a+N5YJ6tq1K9WqVWPJkiUANGrUiB49epBhdB3Hx8czevRo5s+fz8GDBwEYMWIEmzZtYvPmzQDk5eWxY8cOPv/8cwCaNWtGhw4dmDNnDgCJiYmMHDmSuXPncvToUQBGjRrF+vXr2bZtGwC9e/fG4XCwYsUKAFq2bEmrVq1499136d69O/Pnz2f48OHMnj2b3NxcANLT01m5ciU7d+4EoF+/fmRnZ7N27VoA2rZtS5MmTXjvvfcAqFOnDkOHDiUjI4P8fP0fady4cRw+fNhlnwEDBpCVleVaNLh9+/akpKS4BoTXr1+fgQMHMn36dAoKCoiKimLcuHEsXryYv/76C4AhQ4awe/duNm7c6PJTXl6eq47ifqrbti1D3n/f5esjyclUd9tBI7tWLaodOGCcPAK33kr2s88S98YbzDQWynf6xpef0tLSiI6OLpWf9u3bx7Rp03z6CSApKalUfkpISHDZx5ufli5dyu+//15qPxUUFPj1UyD304IFC9i3Tz/q3O+n/fv3s2nTpjLfT9785PRBWe8nb34qKCjgs88+K/P95MtPDRo04O+//y7z/eTNT/v37ycjI8Onn6D0z72jR4+6vqdluZ+8+SknJ4cjR46U+X7y5af9+/dTUFBQ5vvJm5+6d+/OtGnTynw/efOTw+Fw+aAs95MvP506dYrMzMwy3U++/LR//37WrVtX5vvJm5+cz4qy3k/e/JSQkMC6desCvp+CQkRKfQB1gVSgR1nk2If/o3379hIsr7/+etBlKpL80lxzMJitf0B1XH65CIiMHSty8qR+dX4+dUpkyRKRpk11mvsxdqzIvn22jwPA6jayunwR69/LVpcfjjqs7uNw1GF1+cH6GNggAcY4pR0zOgK4D3Au7yQiEu12vgYwDz2R6QoRORR8mGzjjj1mNEL5/nu9jqhz1nxBAcyaBVdfXThp6cQJePpp+M9/4LjbHhI1asCjj+pZ+ZUqlYv6ZeW08LGN7efTANvHkU+FGjOqlHoMeBtohd57XtBBpwsROQz8DfQCRgRbh01oWLx4saXlm0049PdbR9u28PrrhYFnVBSMGVN09nzlynDvvfDTT0XHmR4+DDffDJ06gUl7Elvdx2D9+8Dq8sOB1W1UIZ5FFZxIsJHV5ZtJUMGoUqovMAm95/xIoCqQ5SX7m+ggtV9ZFLQpPc6xIlaVbzbh0D+gOrwsaF+Cs8/WOzJ9/LFeg9TJxo16Ef3x42H//tIp6gWr+xisfx9YXX44sLqNKsyzqAITCTayunwzCbZl9BZ0S+gkEfmfiJzykfdLI29bH3lsbGyCpX9/2LQJHnoIR0yMThPRLaznnqu7/AsKyldHGxsbGxubAAk2GO1kvM72l1FEcoCj6CWfbMqBIUOGWFq+2YRDf9PqiIuDf/+bw6tXw6BBhekHD8INN0CXLhCC8VtW9zFY/z6wuvxwYHUbWfpZFCYiwUZWl28mwQajNYCjInIswPwB9j/amMHu3bstLd9swqG/2XX8VqkSfPihPs45p/DE119Dx45w0006QC0lVvcxWP8+sLr8cGB1G0XCs8hsIsFGVpdvJsEGoweBRKVUFX8ZlVJnAonA3tIoZlN2nOuJWVW+2YRD/7D5YNAg2LwZ7r9fT3gC3XX/6qu6637GjFJ13Vvdx2D9+8Dq8sOB1W0UCc8is4kEG1ldvpkEG4w6p+z2DyDvBON1VZB12NjYlIYqVWDKFD2etL/bLbp/P1x3HVx4IXz7bfnpZ2NjY2Nj44Fgg9Hp6K73R5VS9b1lUkpdD/wLPYHp1dKrZ1MWOnbsaGn5ZhMO/cvFB02awEcfwQcfQIMGhelffgmpqXDrrXpZqNLKtxhWvw+sLj8cWN1GkfAsMptIsJHV5ZtJUMGoiCxCrzHaDNiolHoOvTc9SqnblFJPKaU2oQPQKOAVEfkyxDrbBEhycrKl5ZtNOPQvNx8oBUOH6rVJ770XnLPuCwr0Hvfnngtvvqm78ksj30JY/T6wuvxwYHUbRcKzyGwiwUZWl28mQS96D6QDzwF1gFuBakb6VGAi0ML4/DRwWxn1sykDzj1nrSrfbMKhf7n7ID4eHnlEd91ffHFh+r59kJ4OPXrADz+UXr4FKHcfnObyw4HVbRQJzyKziQQbWV2+mQQdjIqIQ0QmAucCjwBfAD8D29DjQx8HWonI3SJiL3ZoY1MRaNYMliyB+fMhJaUwffVquOACmDgRjhwpP/1sbGxsbE5bStMyCoCIbBeRf4tIHxFpKSLniUhPEblHRH4KpZI2paOB+3hBC8o3m3DoX6F8oBQMG6a77idNguhonX7qFDz7LDRvDnPmFOm6t7qPoYL54DSUHw6sbqNIeBaZTSTYyOryzUSJnzFjNhWD1NRU2RDkIuYFBQVERZX6/0a5y09NTSXYaw4Gs/UPRx1lkv/TT3DLLfDFF0XTe/SAl16CVq0s72Oo4D44DeSD9e9lq8sPRx1W93E46rC6/GB9rJTaKCKpgeQNqdZKqf5Kqf8qpaYqpS4JpWyb4Jk+fbql5ZtNOPSv0D447zz47DOYOxfquy2OsXIltGsHd93Fmy++WGYdy5sK7YPTQH44sLqNIuFZZDaRYCOryzeToIJRpdSVSqm/lFKvezj3KrAYuBs9cekjpdTLoVHTxsbGFJSCESPg55/hrruKdt0//TRXPvAAvPuu31n3NjY2NjY2pSU6yPxDgbrAx+6JSqkewHjj41fAcaAncINSarGIFMlvEzhKqUHAoJSUFKZNmwbotcSSk5NdM+caNGhA3759Xf+KYmNjSU9P5+jRo64yw4YNY/v27Xz//fcAdO3alWrVqrFkyRIAGjVqRI8ePcjIyAAgPj6e0aNHM3/+fA4a20mOGDGCTZs2sXnzZgAcDgc7duzg888/B6BZs2Z06NCBOXPmAJCYmMjIkSOZO3cuR48eBWDUqFGsX7+ebdu2AdC7d28cDgcrVqwAoGXLlrRq1Yp3332X7t27M3/+fIYPH87s2bPJzc0FID09nZUrV7Jz504A+vXrR3Z2NmvXrgWgbdu2NGnShPfeew+AOnXqMHToUDIyMsjPzwdg3Lhx5OTkuOwzYMAAsrKy+Pprva9D+/btSUlJYeHChQDUr1+fgQMHMn36dFdXyLhx41i8eDF//fUXoPcF3r17t2sXjI4dOyIirjq8+WnBggXs27evVH46dOgQgE8/paWlER0d7d9P7dtT6b776PbOO9Q3/JNw+DCMHEnuc8/x9+TJLDO2m3P3E0BSUlKp/JSQkOCyjzc/LV26lN9//73UfoqNjfXrp0DuJ29+OnDgAJs2bSrz/eTNTwcOHGDatGllvp+8+Sk6OprPPvuszPeTLz81aNCAv//+u8z3kzc/HThwgIyMjDLfT978dPz4cdf3NOD7KQg/HT9+nCNHjpT5fvLlpwMHDlBQUFDm+8mbn7p37860adPKfD9581N0dLTLB2W5n3z5qVKlSmRmZpbpfvLlpwMHDrBu3boy30/e/OR8VpT1fvLmp4SEBNatWxfw/RQMQY0ZVUr9DDQF6ojIAbf0DOAaYJqI3Gik3Qv8B1ggIpcHrZlNEUozZtTqhGM8oY0HRODtt3VL6V633Xyjo+GOO+Df/4aqVUNSle3j0wPbz5GP7ePIpyKNGU0G8twDUYO+6N2WnnVLe8l4te6WABZnwYIFlpZvNuHQ35I+UApGjYKff2b7oEFQqZJOdzjgiSf0WNP58y3TdW9JH0SQ/HBgdRtFwrPIbCLBRlaXbybBBqPVgJPuCUqphsAZwF8i8rMzXUSOAIfRAaxNOeBsWreqfLMJh/6W9kH16nwxcCB8843e197J7t1wxRVwySVgdGdVZCztgwiQHw6sbqNIeBaZTSTYyOryzSTYYPQgUE0pleSW5tzWZbWH/DFATmkUs7GxqSC0aaNn2L/5JtSpU5i+dCm0agX33QfHjpWffjY2NjY21kZEAj7QE5dOAQ8bn6sA3xlp1xXLewZQAGwKpg778Hy0b99egmX//v1Bl6lI8ktzzcFgtv7hqCPs8g8dErnlFpGoKBHdUa+PBg1EPvhApKAgKPlm+1gkAn1gMfki1r+XrS4/HHVY3cfhqMPq8oP1MbBBAoxxgm0ZfQ1QwL1Kqc3AL0Ab4BDwv2J5exmv3je+tjGV7du3W1q+2YRDf6v7oIT8GjXghRdgwwbo0qUw/fff4bLLYMAAqGDfi4jzgcXkhwOr2ygSnkVmEwk2srp8MwkqGBWRhcBj6MlK5wH10V33V4tIdrHsY4zXz8qqpE3pcC6/YFX5ZhMO/a3uA6/yzz9f72v/xhtQu3Zh+iefQMuWcP/9cPy4qboFSsT6wCLyw4HVbRQJzyKziQQbWV2+mQS9A5OI3Ac0BkYA/YEmIvKJex6lVAy6S38i8GEI9LSxsaloREXBtdfC1q1w0016Fj5Afj48/DC0aAGLFpWvjjY2NjY2FR6fwWixiUouROQ3EZknIktE5LCH8ydF5HkReU5E9odIV5sg6dq1q6Xlm0049Le6DwKSn5QEL78MX38NHd1Wctu1CwYPhkGD4NdfTdPRHxXCRqex/HBgdRtFwrPIbCLBRlaXbyb+Wkb/VkqtVEpNUkq1DItGNiGjWrVqlpZvNuHQ3+o+CEp+aip8+SVMm6YDVCeLF+tW0ocegry80Cvphwplo9NQfjiwuo0i4VlkNpFgI6vLNxN/wehh4ELgUeAHpdROpdTzSql+SqlY07WzKRPOLbqsKt9swqG/1X0QtPyoKLj+er3+6PXXF3bd5+XBAw/opaA++cS3jBBT4Wx0mskPB1a3USQ8i8wmEmxkdflm4i8YrYMORp8AtgANgVvQ40H3K6XeV0pdp5Sqa6qWNjY21qJWLd1C+uWX0L59YfqOHXDppXrm/W+/lZ9+NjY2NjYVBp/BqLFU1FoRuUdEWlMYjC4FooGhwDTgT6XU10qp+5VSF5iss02ANGrUyNLyzSYc+lvdB2WW36kTrFsHr7wCNWsWpi9YAOedx9g9e+DEibLV4YcKb6MIlx8OrG6jSHgWmU0k2Mjq8s1ESSn3l1ZKxQN9gEHApUA945QAe4CPjGOZiFSMNV4sTGpqqmzYsCGoMvn5+cTGmjeawmz5qampBHvNwWC2/uGow1Lys7Lg//4PZswomt60Kbz4IvTtG5p6imEpG0WgfLD+vWx1+eGow+o+DkcdVpcfrI+VUhtFJDWQvEEv7eRERHJF5EMRuV5EzgQ6AA8B36ID0+uBD4ADSqmPlFI32d354SUjI8PS8s0mHPpb3QchlZ+crNclXbsW2rUrTP/lF+jXD4YPhz/+CF19BpayUQTKDwdWt1EkPIvMJhJsZHX5ZlLqYLQ4IrJRRB40ouAz0cHoIvRWof2BF4EbQlWfjY2NRenSBdavhxdeILtSpcL0996D5s3h8cf1WqU2NjY2NqcFIQtG3RGRvSLyhogMBWqhu/FfBv4yoz4bz8THx1tavtmEQ3+r+8A0+dHRcMstDGvZEsaMKUzPzdVd+W3bwuefh6Qqy9ooQuSHA6vbKBKeRWYTCTayunwziTa7AhHJBz41DpsgUUoNAgalpKQwbdo0ADp27EhycjIfffQRAA0aNKBv375Mnz4dgNjYWNLT06lataqrzLBhw9i+fbtru7CuXbtSrVo111IQjRo1okePHq5m/vj4eEaPHs38+fM5ePAgACNGjGDTpk1s3rwZgLS0NHbs2MHnRtDQrFkzOnTowJw5cwBITExk5MiRzJ07l6NHjwIwatQo1q9fz7Zt2wDo3bs3DoeDFStWANCyZUtatWrFu+++S/fu3Zk/fz7Dhw9n9uzZ5ObmApCens7KlSvZuXMnAP369SM7O5u1a9cC0LZtW5o0acJ7770HQJ06dRg6dCgZGRnkGy1u48aNo3bt2i77DBgwgKysLL7++msA2rdvT0pKCgsXLgSgfv36DBw4kOnTp1NQUEBUVBTjxo1j8eLF/PWX/o81ZMgQdu/ezcaNG11+6tWrl6sOb35asGAB+/btK7WfAL9+io6OLpWfcnNzmTZtmk8/ASQlJZXKTydq1GBa166ckZxM2rx5VHfOsP/5Z+jTB7nySjIHDeIXQ6ZZfgrkfvLlp02bNoXkfvLkJ6cPyno/+fLTZ599Vub7aenSpfz+++8e/dSgQQP+/vtvU/2UkZERkvvJk5/OOecc131clvvJl5+OHDlS5vvJn58KCgp8+qks91P37t2ZNm1aSO4nT3668sorXT4oy/3kz0+ZmZllvp98+WndunVlvp+8+SkvL49p06aZ9txLSEhg3bp1Ad9PQSEiAR9AEtAX6OThXH3gXWAvcAh4B6gfjHz78H60b99egmXevHlBl6lI8ktzzcFgtv7hqMPq8ov4+ORJkalTRapVE4HCo2pVkaeeEsnPL1UdVreR1eWLWP9etrr8cNRhdR+How6ryw/Wx8AGCTDGCbabfjzwCXCle6JSKg5YCQxHr01a3ciTqZRKCD5EtgkFzn+MVpVvNuHQ3+o+CKuPo6Ph9tv1XvejRhWm5+TAXXfpSU+ZmUGLtbqNrC4/HFjdRpHwLDKbSLCR1eWbSbDBaD/jdU6x9HSgEXAQuBEYA/wJNEavS2pjY2MTGPXqwezZOvBs6bYL8ZYt0KsXjB4Ne/aUm3o2NjY2NiEm0CZU3eLKLvTs+Lhi6Z8b6ePc0noDBcBXwdRhH6Hrpj98+HDQZSqSfLO7fczWPxx1WF2+Xx/n5+su+qpVi3bdV6umu/RPnvRbh9VtZHX5Ita/l60uPxx1WN3H4ajD6vIrUjd9MnBYRPKcCUqpaKCLEXjOc8v7hRGgnht8iGwTCjZt2mRp+WYTDv2t7oNy93FMDNx5p57QNGJEYXp2NkycCBdcAKtX+xRhdRtZXX44sLqNIuFZZDaRYCOryzeTYINRBRQfA9oeiAO+F5EjzkQjKj4CVCmThjalxjmr0KryzSYc+lvdBxXGx2eeCXPnwmef6bVInfz4I3TvrpeH+vtvj0WtbiOryw8HVrdRJDyLzCYSbGR1+WYSbDD6BxCjlGrjljbUeF3lnlEpFQVUA7JKrZ2NjY2NO717w/ffw3//C+5r6r31Fpx7rt5W1OEoP/1sbGxsbIIm2GD0C3Tr6CtKqQ5KqcHAzej96BcVy9sCiAF2l1lLm1KRlpZmaflmEw79re6DCunj2FiYNEl33Q8fXph+5Ajceit06KC3HDWwuo2sLj8cWN1GkfAsMptIsJHV5ZtJsMHo40A20Bn4Cr33fDVgrYh8USzvYHSQuhabciE62tw9DcyWbzbh0N/qPqjQPj7rLJg3D5YsgaZNC9O/+w66dYNrr4WsLMvbyOryw4HVbRQJzyKziQQbWV2+mQQVjIrILqAXsALIA/YBM4Eh7vmUUpXQe9Mr4LNQKGoTPJ+HaDvF8pJvNuHQ3+o+sISP+/bVY0cfeQSquA1RnzkTmjXjr/vvh1OnTKve6j6whI/9YHUbRcKzyGwiwUZWl28mQe9NLyLfiMhFIpIgIvVE5DoRKb7SagHQDqiJvQ2ojY2N2VSuDPfeCz/9BEOHFqYfPkz3t9+GTp3A2ErPxsbGxqZiEXQwGgjGElNHjEPMqMPGP82aNbO0fLMJh/5W94HlfHz22fDBB/DRR9C4cWH6xo3QuTOMHw8HDoS0Sqv7wHI+9oDVbRQJzyKziQQbWV2+mZQ6GFVKRSulOimlhiulrgmlUl7qG62UEuMY5yXPQKVUplLqiFIqRym1Tik1xo/cMUqpr438R4zyA33kr6SUmqiU+kEpdVwpdVAp9bFSqquPMlWUUlOUUluVUnlKqX1Kqf8ppc4L3ALB06FDBzPFmy7fbMKhv9V9YFkfX3opbNoEDz2ExMXpNBF4/XVo1ky/FhSEpCqr+8CyPnbD6jaKhGeR2USCjawu30xKFYwqpSYBe9GTk95Fjxt1P19DKbVFKbVdKVW/rEoqpc4CXgRyfOS5BT2jvxUwG3gdqA9kKKWe8lLmKSADqGfknw20BhYZ8ornV8Bc4Bkg1tDpA6AHsFIpNcRDmcrAMuB+4CjwHHoc7WXABqVUJ78GKCVz5hTftdVa8s0mHPpb3QeW9nFcHPz738ydPBkGuv2/PHhQt5B26QIbNpS5Gqv7wNI+NrC6jSLhWWQ2kWAjq8s3k6CDUaXUHOBR9HjQX4ESi/qJyGH0JKdzgJFlUdAIAGcCB4BXveRpCDwFHARSRWSCiEwE2gA7gDuVUl2KlekK3GmcbyMiE0VkAnoR/4PAU4Zcd0YCw9FBeDsRuVtErkNP6joFvK6UqlaszB1AN2A+0ElEJonIPww58cAMY01WGxsbE8hOToZFi+DDD6Fhw8ITX38NHTvCTTfpANXGxsbGplwIKghSSo0ErkK3inYVkSbowM0Tc9Cz6fuUSUO4DbgIGAsc85LnWqAy8KIx4x8AETmEDpwBbixWxvn5ESOfs8wu4CVD3thiZW4yXie7b4kqIuvRLcTJ6CATcAXSznr+JSIFbmUWojcKaAGYsjhYYmKiGWLDJt9swqG/1X1gdR+D2zUMGgRbtsD99+sJT6C77l99VS+YP2NGqbrure6DiPKxLb9c6zCTSLCR1eWbSqCb2BvzkJahWwCvcEvbA5zykDcO3Wr6ezB1FJNxHnAcmGp8fhC9dum4YvlWG+ldPMioZ5z7o1j6biO9nocyXYxzqzxczzEg2kOZq4wys9zSmhhpW71c3z3G+Yf92aJ9+/ZyunE6XvPpRrn5+JdfRPr3F9HhaOHRpYvIN9+Uj04RjH0vRz62jyOfYH0MbJAA471gV0g93wiePvSXUUTylFJH0K2FQaOUigZmAb8D9/rJfq7xus2DHnuUUseAFKVUvIjkKqUSgDOBHBHZ40HeL8ar+9S0xkAlYKeIeNpv0FMZr3r5KONCKTUeGA8QFxdHamqqFzGeOXjwIElJSUGVqUjyf/rpp6CvORjM1j8cdVhdvtk+Bh/XIEJa48bc9ccf1MvP12lffsmpCy5gfnIyr9SvT04Ai0hb3QfhuA+sfi9bXX446rC6j8NRh9Xlm+rjQKNWHeSSBxwqluaxZdQ4l40O+ErTKvoQuhW2i1vag3huGc030ku0WBrn/8StFRQ9sUmA3V7yxxjnT7ildTXSVnsp05RiraDAP4y02V7KXGycX+LPHqX51/naa68FXaYiyTf7n7bZ+oejDqvLD0drit9ryMkRufdekZiYoq2kdeqIZGSIFBSUTX4Zsbp8Eevfy1aXH446rO7jcNRhdflmtowGO3EmC0j0MEmnBEqppkACpdib3phhfi/wtIh8GWx5Gxsbm4BJSNC7N23aBBdfXJi+bx+kp0OPHvDDD+Wmno2NjU3EE2jUqoNc5qJbK691S/M2ZnSakfeVIOuIBrYCW4DKxc49iOeW0SwjvZYXmTnG+Xjjc4LxOdtL/trG+b/d0loaaT96KZNqnF/nljbASFvkpcxw4/y7/uxSmn+dOTk5QZepSPLN/qdttv7hqMPq8sPRMhrUNRQUiMyfL5KSUrSVtFIlkdtvFzl8uGzyS4HV5YtY/162uvxw1GF1H4ejDqvLr0gtoy+gZ8j/RynVylMGpVRlpdQjwDgj0HoxyDqqosdQngfkuS10L8ADRp7XjbRnjc9bjdcSYy+VUvUwWmhFJBdARI6hu+6rGueL09R4dR/ruQMdXDcyxrMGUsarXj7KhIz169ebITZs8s0mHPpb3QdW9zEEeQ1KwbBhelvRSZPAOWb01Cl49llo3hzmzNEhamnklwKryw8HVrdRJDyLzCYSbGR1+WYSVDAqImuAJ4EzgHVKqYVANQCl1DNKqfnoIO//jCL3i8jmIHU6Abzh5fjWyLPa+Ozswv/CeL3Eg7z+xfJQ7HNAZUQv5bQWvTZo9wDr2YGegNVMKXVOELqFhG3bTIlxwybfbMKhv9V9YHUfQymvoWpV+O9/dff8RRcVpu/dC6NHQ69esHlz6eUHgdXlhwOr2ygSnkVmEwk2srp8Mwl6sXURmQRMRLcSDkIHZwr4J3A5kATkAreJyKPe5PiQf1xExnk6KJzF/6aR9q7xeSY6iL3FfaF6pVRNCmfiF18w3/n5PiOfs0xDYIIhb2axMq8Yr/9RSsW5lekAjEAPF3jP7VrErZ4n3Be3N3Zr6o4ejrDCl01sbGzKifPOg88+g7lzob7bZnIrVkC7dnDXXcTk5XktbmNjY2Pjn2CXdgJARJ5TSmUAw9CzzOuhA9u/0a2V80QkbFuaiMivSqm7gefRW2y+i55hPxxIwcNEKBFZq5R6Br1D0g9Gq24sOqhMAm4VtwX0DeaiA+7hwLdKqUVALaNMJeB6ETlarMwzwECjzDql1OdAA+AKdNB+rbgthh9Kevfu7XpfUACzZsHVV0NUiPZ7cpdvRcKhv9l1WF1+OCjzNSgFI0bo/e6nTNHd9adOgcMBTz/N1XXrQt26cOWVOm+IsX3sH6vbKBKeRWYTCTayunwzKXVYIiJHRGSG0UI5QET6i0i6iLwWzkDUTZ8XgMHAZuAa9Pqce4F0EbnLS5k70bss7TXyX2OUHyQiJca6Gi2dV6EDWAdwKzo4XQn0EL2rUvEyJ9BLOD0M1EC3Kl8MLAA6iMi60l6zPxwOvRxqQQGMG6cnBo8bV6pNZnzKtyrh0N/sOqwuPxyE7BqqVYOnnoLvvoO0wk3Tov/+G0aOhD599FjTEGP72D9Wt1EkPIvMJhJsZHX5ZmKpPdFF5EERUSIy3cv5RSKSJiLVRCRBRDqIyJt+ZGYY+RKMcmkisthHfoeITBWR1iJSRURqisilIrLWR5lcEblfRJqKSGURSRaRK0RkS+BXHzwrVqxwBaIzjQEHM2eGLiBdscLaowvCob/ZdVhdfjgI+TW0agXLl8Ps2XDGGYXpX3wBbdroiU85OSGrzvaxf6xuo0h4FplNJNjI6vLNxFLBqE1wFA9EnYQyILWxOS1RCkaNgp9/5sfevaFSJZ3ucMATT+ixpvPnF5l1b2NjY2PjGa9jRpVSM4y3e0TkvmJpwSAicl1plLMpPQUFsGjRED7+2PN5Z4A6fXrpx5C2bNmydAUrCOHQ3+w6rC4/HJh6DdWrc3TKFN2FP2ECrF6t03fvhiuugL594YUXoJm31d38Y/vYP1a3USQ8i8wmEmxkdflm4msCU7rx+jNwn1uaoGfPB4oAdjAaRpwtoh9/XNdnvrIGpK1aeVxq1jKEQ3+z67C6/HAQFhtVrw4rV+pZgnffrXdvAli6VHfr33033HcfxMeXTr6J2D6OfPnhqsNMIsFGVpdvJr5CkCnG8VKxtIfczgVyPBRyrW284q1r3htl6bJ/9913/WeqwIRDf7PrsLr8cBA2GykF11wDW7fCLbcU/sM7eRIefVR33S9YEHTXve1j/1jdRpHwLDKbSLCR1eWbideWURGZEkiaTcVi1qzAA1EnM2fqycFjxpijk43NaUWNGrpr/tpr4eab4auvdPrvv8Nll0H//vD889CkSbmqaWNjY1NRCKpzVinVxjiqmqWQTdm4+moYOza4MmPH6nLBkpSUFHyhCkQ49De7DqvLDwflZqPzz4c1a+CNN6B27cL0Tz6Bli3h/vvh+PHSyw8Rto8jX3646jCTSLCR1eWbiZIguoyUUgVAAXCGiOw3TSubEqSmpsqGDRsCyhtMV32lSvD443D77YUTgisKqampBHrNNtbktPHxwYMweTK8+mrRbvqGDXUr6aBB5aZaODht/HwaY/s48gnWx0qpjSKSGkjeYKetHAGO2IFoxSYqSk9K6tFju9+8p07BXXfBhRcGv1737NmzS6lhxSAc+ptdh9Xlh4MKYaOkJHj5Zfj6a+jYsTB91y4Y/P/snXl4VEXWh99KQoCEhDWoMSIiiwoSJAkKmCCgoMOqoKCgBo24oCKOy4zzucyoM+qojIoiMUpUUNCAIIuCIJugEMJmEFlFRAQJSwgECEnX90f1kkCW3u7trua+z3OfpKvvPXX6/FI31XWrTvVXxy+/eG/fByyNQ9++WXUYSSjESHf7RuJpZ3QLEFN+X3aL4CQsDG699dsqH9n36weXX+56/cMPaqvt//xHpUp0h+LiYp/9DCRm+G90HbrbN4OgilFyMnz/PWRmqg6qg1mz4LLL4F//gtP2ug8q/4MU3WMUCvciowmFGOlu30g83Zv+YyAFtW1mpv/dsTgdIUQ/oF9CQgKZmSrknTp1Ii4ujjlz5gDQrFkzevXqRVaW2pgqMjKS9PR0jhw5TKdOmWze3I0VK9o4bXbpspkXXjhF3box/PWv+5k79wrKysIpKYGnnoJ3393PyJEr+cc/+pKTk8PBg2p31yFDhpCfn8/GjRsBOHHiBNu3b2fhwoUAtG7dmpSUFCZPngxAbGwsQ4cOZcqUKRw5cgSAYcOGkZuby5YtWwC1l25paalz54i2bdvSrl07pk6dSmpqKjk5OQwePJhJkyY5G1p6ejpLly5lx44dAPTu3ZuioiJWrFCbYCUmJtKyZUumTZsGQNOmTRk4cCDZ2dmUlJQAkJGRwZEjR5wx7dOnD/v372fVqlUAJCUlkZCQwMyZaofX+Ph4+vbtS1ZWFjabjbCwMDIyMpg9ezZ79uwBYMCAAezevZu8vDynTiUlJc46qtJpxowZ/GlPBTRo0CC2bdvG+vXr7Vp1ISYmhnnz5gHQokUL0tLSyM7OBnBqU51O3bp1IyIiwiudCgoKyMzMrFYnUHOVvNEpOjraGZ+qdJo/fz67du3yWiegRp3caU9V6VRQUEB+fn61OkVFRTF8+PAKOg3NzaX4kUc4Z/ZshJSqE/rssxSOG8eKoUOpe9NNpKSkODXwtT1VpZPNZmPBggU+t6fqdGrWrBn79u3zuT1VpVNBQQHZ2dk+t6fKdBoyZAhHjx51/p360p6q0uno0aMUFhb63J6q06mgoACbzeZze6pKp9TUVDIzM31uT1XpZLPZnBpUpZOv972ysjIWL17sU3uqTqeCggJWrlzpc3uqSifHvcLX9lSVTtHR0axcudLt9uQRUkq3D1R+0S+AYlTu0DBPrrcO74+kpCTpKSdPnpRSSllWJuWIEVKC+llWVvG8DRukTE5W7zuOiAgpn3lGSruJau0bhTef2ROM9t+MOnS3b7TGUmoQox9+kDIpqWIDBCkHDpRy587g998NdG/Luts3ow7dNTajDt3te6oxsFq62cfx9DH9+8BhoBQ1MrpXCDFXCPGhEOKDKo73Pe8iW/iDpUuXAq45pNnZlSe4v/xy9eTw5Zehdm1VVlqqnhgmJUFubvX2dcUM/42uQ3f7ZhD0MbrySli5Us0pbdDAVT5jBlx6Kbvuuw9OnvStjmqwNA59+2bVYSShECPd7RuJp53RdNQj+nqoUdImwPXA7fb3qjosAoDjMQGoDuidd1a901JEBDzxBKxfD127usrz8+Gqq+DJJ8/MQlPevo6Y4b/Rdehu3wy0iFF4ONx/P2zZovKTOjh+nJYTJ6pvjPPn+15PJVgah759s+owklCIke72jcTTOaNW0vsQp00btavh22/D3/4GxcUqVdQrr6iBmg8+qNhZtbCw8CNxcSov6d13q73u161T5Vu3Qu/eMGgQjB0LF1wQUDctLCws/Iq7z/OtQ785ozt37vT4mvJs3y5ljx4Vp7EJIeXDD0tZVOS7/Zoweg6S0f6bUYfu9s2YM6ptjE6dkvKtt2RZTEzFRhgVJeVLL1U/odsDzGgHurdl3e2bUYfuGptRh+72g2nOqIVGFBUV+XR9ixawYIHKQhMTo8qkVDm6L78cvvnGiw3tgwhf4xMMdehu3wy0jVFEBDz4IJu//FLtee+guFg9tkhMBPtKYV+wNA59+2bVYSShECPd7RuJT51RoWgihGjmL4cs/Icj5YcvCAH33AMbN6ottR3s3An33HMR994LhYU+VxMQ/BGfQNehu30z0D1Gy7ZsgQ8/VPNnyicH/vlnuPZaGDoUfv/da/uWxqFv36w6jCQUYqS7fSPxqjMqhOgshPgSOALsA3ac9n4DIcT7QogsIUSUH/y0CDAXXABz5sBHH0HDhq7yzExo1w7mzg2cbxYWZwWpqbBmjZoz6nhUATB1KlxyCbz2Gpw6FTj/LCwsLLzE486oEGIUsBToC0SjVtWL8udIKQ+jVtqPAAb57KWFVyQmJvrVnhBw++3w009w442u8t27oU8ftVrfnn9YC/wdn0DUobt9M9A9RhXsR0TAI4/A5s0wbJir/OhRta9vhw6weLH39jUlpDTWuA4jCYUY6W7fSDzqjAohOgFvADbgb0Az1MhoZUxEdVL/4ouDFt7TsmVLQ+yeey5Mmwbvv19EXJyr/KOP1I6G06cbUq3fMSo+Ztahu30z0D1Gldo/7zyYNAkWLVKNzsFPP0H37jB8OPzxh/f2NSMkNdawDiMJhRjpbt9IPB0ZfRTVwXxOSvmKlHJ3Necusf/s6JVnFj7j2G7MCISA0tJP+eknuO02V/m+fSr7zC23gH0HsaDFyPiYVYfu9s1A9xhVa/+aa1T6p1dfhXr1XOWTJ6s8bf/7n9rBwlv7mhDSGmtUh5GEQox0t28knnZGU+0/36npRCnlIaAISPDUKQt9aNJE/d+bOVMN1jj4/HM1YPPJJ2oFvoWFhUHUqgV//ata0DRkiKu8qAjGjIGOHeG77wLnn4WFhUUNeNoZbQIckVK6u366zIs6LPxE06ZNTbPfv796Qnj33a73DxxQ09oGDPBpsa9hGB0fM+rQ3b4Z6B4jt+2ffz5MmaLysV1yiav8xx/V4qc771SPLry1H8QEjQZBat+sOowkFGKku30jEdKDYSshxJ9AIyBKSlliL/sDaCqlDD/t3Cao+aS/Symt1E8+kpycLFevXh1oN9xi/nyVDmrXLldZ/frw+uswYoR6xO8OycnJ6PKZLbzD0tggSkrUqvt//UvlJXVQvz688ALcd59aDGUSls6hj6Vx6OOpxkKIPCllsjvnejpquR41ZzS1phNRe9ILYKWHdVj4iezs7IDY79VL7Wn/wAOussJCNWrauzf8+quhbrmN0fExow7d7ZuB7jHyyn5kJDz5pHp0P3iwq7ywEB56CFJS4PvvvbcfZASlBkFk36w6jCQUYqS7fSPxtDP6EaqD+R8hRL2qThJC9AL+BUjgA+/ds/CFkpKSgNmPiVH72y9eDBdf7Cr/5huVl/Sdd9Se94HE6PiYUYfu9s1A9xj5ZP+CC9QE7nnzoFUrV/m6ddClC9x9N2EHDvjsY6AJag2CwL5ZdRhJKMRId/tG4mlndBKwEEgGVgohxgC1AYQQ/YQQDwoh5gBzgTrADCnlV/502EIvunWDDRvU+oow+1/b0aMwapTKQLNtW2D9s7A4K+jVS80dffFFqFvXVf7BBwx55hkYPx7KygLnn4WFxVmNR3NGAewjoh8DA1Ajn2ecYv85HbhDSllcyTkWbiKE6Af0S0hIuOfpp58GoFOnTsTFxTFnzhwAmjVrRq9evcjKygIgMjKS9PR0vvjiC/bv3w/AoEGD2LZtG+vXrwegS5cuxMTEMG/ePABatGhBWlqac5g/KiqK4cOHk5OTw0F7JvshQ4aQn5/Pxo0bAUhNTSUyMpKF9v2xW7duTUpKCpMnTwYgNjaWoUOHMmXKFI4cOcKOHU2ZMaM/mze7vgPVqWPj4YcLuOiimYSFSdq2bUu7du2YOnUqmzZtomvXrgwePJhJkyZRbJ/7lp6eztKlS9mxQ2381bt3b4qKipxboSUmJtKyZUtnmoumTZsycOBAsrOznd8cMzIymD9/PrvsE1v79OnD/v37WbVqFQBJSUkkJCQwc+ZMAOLj4+nbty9ZWVnYbDbCwsLIyMhg9uzZ7NmzB4ABAwawe/du8vLynDo1btyYr776qlqdZsyYwZ/2PFhG6NStWzciIiLc1glg2LBh5ObmsmXLFgB69uxJaWkpS5aojG3ldQJo1KiRVzo98cQTDLMnbzdKp7vuuou5c+dWq5M77SlkdEpOZtfgwTSzf34H+5s147vbbqPjfff5vT19/fXXjB8/3uf2FCidfvzxR3766SfDdLrsssu4/PLLfW5PgbzvTZo0iUsvvdQwna6++mo++uijanXytT3deuut5OXlGX7fM0qn9957DymlYe1p8uTJvPLKK263p3vvvdftOaNIKb06gJ6oTuk24BhwHNgJfAr09taudVR+JCUlSU/56quvPL7GaPsnTkj5j39IGR4upUr6pI4rr5Ry48aK53rzmT3B6PiYUYfu9o3WWEr9Y2SY/TlzpLz44ooNUQgp77lHyoICv1ale1vW3b4ZdeiusRl16G7fU42B1dLNPo7XaZeklAullLdLKVtKKaOllHWllM2llLdKKed5a9fCf+wqv5w9SOzXrq0W8+bmQvmdy1auhCuuUE8Rzdpe2+j4mFGH7vbNQPcYGWb/L3+B/HxW9+sHdeqoMinhvfegdWv1M9ATu91EWw1Msm9WHUYSCjHS3b6RWDlALQLCFVeoDunzz6uc3aCy0fzf/0GnTmp9hYWFhcHUqcOavn1h40bo29dVfvAgjBwJnTuDla7HwsLCaNwdQrUO/R7T79692+NrAmE/P1/KlJSKTwsjIqQ877xMeeKEX6qoFKPjY0Yduts34zG97jEy1f6XX0rZvPmZj+7vu0/KAwe8rsNonUNKA03r0F1jM+rQ3X7QPKYXQjzjzWFQP9qiBhyLl4Ldftu2sGIF/Pe/rqeFpaXwxx/30LGjeoRvBEbHx4w6dLdvBrrHyFT7/fqprdSeeUbNqQHVJX33XbXX/QcfBOWj+5DSQOM6jCQUYqS7fSPx9DH9c8CzHhyO8y0CgGPlnQ72IyLgscdg/Xq4+mpX+U8/qXSIjz8Ox4/7rTrA+PiYUYfu9s1A9xiZbr9uXfjnP9XOFddf7yovKFA7V1x9Naxda6hPnhJyGmhah5GEQox0t28k3iS9r+6YCexGpXc6BHxoL7ewcIvWrWHJEnjrLQgLU6kybDZ49VW14GnZsgA7aGFxttCyJcydC198Ac3K7ej8/feQnKx2cjp8OGDuWVhYhA4edUallOlSyhHVHDdJKS8E7gCigFIp5QhDPLeokaSkJC3th4XBgw/CZZcNpWdPV/nWrZCWpv4HHj3qez1Gx8eMOnS3bwa6xyig9oWAgQPVI4qnnnKtNrTZYNw49ej+o4/Uo/wAEtIaaFSHkYRCjHS3bySGrKaXUk4CxgB3CSHuNKIOi5pJSEjQ2n7t2nv45hvIyoLYWFf5uHFw+eWwYIFv9o3234w6dLdvBrrHKCjsR0ervGv5+XDdda7yP/+EO+9U3xI3bDDOyRoIihgFsX2z6jCSUIiR7vaNxMjUTh8BZcD9BtZhUQ2O3Rl0tQ9qYObuu1XmmT59XOU7d6r/iffcA4WF3tk2w3/dNTAjRkaje4yCyn7r1mqf+88/h/L/+L77Djp2hDFjVIO02WDz5orXGvhIP6hiFIT2zarDSEIhRrrbNxLDOqNSyuNAMXCZUXVYnD0kJMCsWTBpEjRq5CrPylKr8WfPDpxvFhZnFULA4MGwaRM8+aRafQhqb/v//U89uu/eHS65BAYNglmzeGX7dtVwBw1SqxQtLCwsymFYZ1QI0RyIBYIvD8hZQnx8vNb2T0cIGDZMTV8bPNhV/vvvKiPN7bfDgQPu2zPDf901MFtjI9A9RkFrv149eOkl9Xi+Rw9X+b59sHSp+n36dOjfnx6HD6t5pdOnQ4cOfu+UBm2MgsS+WXUYSSjESHf7RiKkARPPhRDnAFOANGCRlPJav1dylpGcnCxXn2U7oSQnJ1PdZ87JgVGj1LQ1B02bwjvvqP91FsFPTRpbaIKU8Nln8PDDFRtkddx9t9pyVAhjfbMwBasthz6eaiyEyJNSJrtzrqdJ7z+o4fhECLEU2InqiNqAFz2pw8J/ZGVlaW2/JgYPVqOkw4e7yv78U5UPHqwGaKrDDP911yDQGvsD3WOkhX0hYMgQ2LYNWrWq+fwRIyAz028dUS1iFED7ZtVhJKEQI93tG0mEh+enAxKVR7Qm9gAPSikXeeqUhX+wGbxTitH23aFxY/j4Y/V/8N57Yc8eVT5tGixaBG+8oR7tV/Y/zwz/ddcgGDT2Fd1jpJX9mBh47TXo37/qc845R+3oFOa/WWJaxSgA9s2qw0hCIUa62zcSTzuj/6zh/VLgMPAjsFxKWeaNUxb+IcyPN/tA2PeEvn3VivvHH1eLmgAOHlTzSKdMUf/7Ts96YYb/umsQTBp7i+4x0s5+dnb17+/bB/Hx8OmnFdNE+YB2MTLZvll1GEkoxEh3+0ZiyJxRC/9jzRl1nwULVMqnnTtdZbGxasDm7rutKWrBhDXPLMQ4fFitmnf3/0rv3moRVIcORnplYQJWWw59jJwz6unIqIXJCCH6Af0SEhLIzMwEoFOnTsTFxTFnzhwAmjVrRq9evZzzRSIjI0lPT2fcuHFERkYCMGjQILZt28Z6+wrWLl26EBMTw7x58wBo0aIFaWlpZNtHNaKiohg+fDg5OTkcPHgQgCFDhpCfn8/GjRud53Tu3JmFCxcC0Lp1a1JSUpg8eTIAsbGxDB06lClTpnDkyBEAhg0bRm5uLlu2bAGgZ8+elJaWsmTJEgDatm1Lu3btmDp1KqmpqeTk5DB48GAmTZpEcbHaHjQ9PZ2lS5eyY8cOAHr37k1RURErVqwAIDExkcWLW5Ke/geLF7cD4MgR1UF9/fXdDB++jL/9bQgTJkwgPDwcgD59+rB//37n3r5JSUkkJCQ487bFx8fTt29fsrKysNlshIWFkZGRwezZs9ljnxswYMAAdu/eTV5enlOnzZs3U2hPhFqVTjNmzOBP+6IPT3U6duwYY8aMqVanbt26ERER4ZVOubm51K9fv1qdABo1auSVTtHR0c6/66ZNmzJw4ECys7MpKSkBICMjg/nz57Nr1y6vdTr33HMBqtXJnfZUlU6FhYXccMMNPrenqnR65513qF+/vs/tqSqdzjnnHKKjo6vVqWXLlkybNs0tna7r0IGL3N27ft48mDePbVdeScspU5idn++VToWFhcTFxfncnqrS6fPPP3c+AvWlPVWlU1hYGDfffLPP7ak6nQoLC/nrX//qc3uq6r6XmppKZmamz+2pKp2OHTvGPvtCAF/aU3U6xcXF0bBhQ5/aU3U6FRYWkpaW5lF78kSn//73v9SvX9/t/0+e6hQdHc3KlSvdbk8eIaW0Dg2OpKQk6SkTJkzw+Jpgsu/NZz6dpUulbNlSSjVUo47oaCnfekvK8eON9V9K/TXQQeOa0D1G2tlft65ig7v44oqvb7lFyrvukjIsrGJ5ZKSUjzwi5f79gf8MIWbfjDqMbsuhECPd7XuqMbBautnH8WhkVAhxh+fd3So7wR/5y5aFRVWkpqp0hs8+C6+/rjaGOXZM7W/fsmU/evRQm8pYWFj4icREuOkmlVPUvmr+y3PPpf+BA+p1VpZavPToo2q/+y+/VNeVlKik+R98oJLpjx6ttiG1sLAIfdzttapOLjbUFp8+H57Uax3ejYzu3bvX42uCyb6/v2mvXCll27YVB2Pq1JHyv/+VsrTUr1U50V0D3TSuDN1jpKX9deukvPtuKcvKpJRSJnfsKGV2tvN1BZYtk7Jz54oNE6Q87zwpJ0yQ8tSpGqvTMkYm2jejDqPbcijESHf7Ro6Merr0aqn9OIJK7ySA3cBK+/FbufLCcudXdlgYzO7du7W27286dYK8PHj6adcOhidOqBX4Xbqo1fj+RncNdNO4MnSPkZb2ExNVQnv76l4pBNx5Z+XpnK6+GpYvhy++UFuIOvjjD5WvrV079Z6selGUljEy0b5ZdRhJKMRId/tG4lFnVEp5DZAH1AfeBy6WUl4opexiP5oDLYD3gAaoXnH3yg6/fgqLSnFMVNbVvhHUrg3/+hfk5sIFFxQ4y1etgiuugBdegFOn/Fef7hroqPHp6B4jbe17krZCCBg4EH78USXDP+8813ubN6vH/l27wnffVXq5tjEyyb5ZdRhJKMRId/tG4ukOTMOBMcDLUsp7pJS/nH6OlHKnlPJe4CXgUSHEbf5x1cLCf3ToAH//+xe8+CLYEw5w6pQaNU1JAXcXA1tYWPiRiAiV9mLbNvj3v1VONgfff68mgffvb8xjDAsLi4Dh6WP6Uah5o/9x49yX7OeO8tQpC//QqVMnre0bTefOKTz1lOp4Xnmlq3z9etUh/b//g5MnfatDdw101xj0j5Hu9r0iKgr+/nfYsQPGjHF9YwSYNQvat1dJg+2PJXWPkRkaBKXOHhAKMdLdvpF42hm9DDgipTxS04n2c44Abb1xzMJ34uLitLZvNA7/L7tMTVl77TWoU0e9V1YGL76oHt2vXOl7HUahu30z0D1Gutv3icaNVRqMzZvVdmqOR/82m1p136oVPPkkTWvVMtSNUNAgqHV2g1CIke72jcTTzqgE6gshmtZ0ov2cBt44ZeEfHMlsdbVvNOX9Dw9XmWY2bIC0NNc5mzapxU1//SvYcxp7XYcR6G7fDHSPke72/ULz5vDRR+oxxvXXu8pPnIBXXqFRp07w6qvqtQGEggZa6FwNoRAj3e0biaed0TWolfKvuHHuK/Zzrf3BLLShVStYtAjeftuV4tBmU4Mz7duDfSMOCwuLQJCYCF99BQsXQlKSs7hOcbFKi9G6NWRnq0cbFhYW2uBpZ9TRwbxdCPGNEOJaIURdx5tCiDr2svnA7aiRVHc6rhYG0KxZM63tG01V/oeFwQMPQH4+XHedq3z7drjmGhg1CoqKfKvDX+hu3wx0j5Hu9g2hRw+VAmPqVLj4Ylf5b7+pxPodOsCcOdWmg/KEUNBAS53LEQox0t2+kQjpYWMVQjyOWpzkwIbKKQoq5VMYqsMqgSellK/6wc+znuTkZLl6tWeDzI69hI3CaPvJycl4+pk9wR3/pYSJE9Uj/MJCV3mzZiqNYq9evtfhC7rbN1pj0D9GutsHg3UuKcGWmUnY88+DfQ9tJ926wcsvV1yh6AWhoIHubTkUYqS7fU81FkLkSSmT3TnXY6+llP8FugGL7UXhQCP7EW4vWwikWR3RwJKVlaW1faNxx38h4K674KefoF8/V/muXdC7t1rQe/iwb3X4gu72zUD3GOlu33AiI8mKjFTpoJ57ruIWokuWwFVXweDBahGUl4SCBrrrHAox0t2+kXjVhZZSfiel7Ak0Aa4FbrUf1wJNpJTXSSmX+89NC4vAEh8PM2fCJ5+oBb4OPvhArcZ3bK9tYWERIGJi4Nln1XyaUaNc26wBTJsGbdvC/fernZ0sLCyCCp/Gc6WUh6SU30opp9qPb6WUh/zlnIVvRJbPzaehfaPx1H8h4NZb1SjpzTe7yv/4AwYMgGHDoKCg4jW6a6C7xqB/jHS3bwYVPsM558C4cSoVxi23uMrLyuDdd6FlS7W7xZEaMxRWbt8AzNBAd51DIUa62zcSj+eMWgQGb+aM6o4Z8wl9Yfp0tdBp3z5XWVycWok/eLBnuyGerQS7xhb+IaA65+bCk0+qNBnladJEdUrvu69iUn0Lr7DacugTVHNGhRA3CSHu8PQ6C/OZMWOG1vaNxlf/b7pJjZLeUa417N+vBmMGD4a9e/XXQHeNQf8Y6W7fDKr9DCkpKhXU11+r1FAOCgpg9Gi45BL49FOVw80b+37ADA101zkUYqS7fSOJqPmUM3gNuAD4yM++WFSCEKIf0C8hIYHMzExAbfkVFxfnTHDbrFkzevXq5Zy8HBkZSXp6Olu2bHFeM2jQILZt28b69esB6NKlCzExMcybNw+AFi1akJaWRnZ2NgBRUVEMHz6cnJwcDh48CMCQIUPIz89no31f6BMnTrB9+3YWLlwIQOvWrUlJSWHy5MkAxMbGMnToUKZMmcIR+yOxYcOGkZuby5YtWwDo2bMnpaWlLLEn8Gzbti3t2rVj6tSppKamkpOTw+DBg5k0aRLF9qzz6enpLF26lB07dgDQu3dvioqKWLFiBQCJiYm0bNmSadOmAdC0aVMGDhxIdnY2JSUlAGRkZLBt2zZnfPr06cP+/ftZtWoVAElJSSQkJDBz5kwA4uPj6du3L1lZWc4VixkZGaxYMZuuXfcQG3sB06f3Ys8etYZv+nRYsKCU/v2j2bcvEyGq1mnGjBn8aV8F7KlODm2q06lbt25ERER4pdNPP/3En3/+Wa1OAI0aNfJKp+joaKcGVek0f/58du3a5bVONpuN2bNns2fPHgAGDBjA7t27ycvLA9xvT1XpVFBQQMuWLX1uT1Xp5NDA1/ZUlU42m40FCxb43J6q06lZs2bs27evxvbkrU4FBQUcPny45vb00Uec++231H7hBWIOHACAX36B226j4MknWXfrrVz78stn6LRjxw7n36kv7akqnY4ePUphYaHP7ak6nQoKCujfv7/P7akqnVJTU8nMzPS5PUHl9729e/c6NfClPVWnU1lZGYsXL/apPVWnU0FBAeedd57P7akqnRz3Cl/bU1U6RUdHs3LlSrf/P3mElNKjA/gFKKuk/A+g1FN71uHekZSUJD1lwoQJHl8TTPa9+cye4G//Dx+WcuRIKVVCKNdxww1S7trl16qcWBrXjO4x0t2+lEHYlk+ckPL116Vs1OjMBnvddVLm5flm30PM0ED3thwKMdLdvqcaA6ulu33Lat+EZcC/gT5AA1lzZ/SMcusIXGe0oKDA42uCyb7RNzej/F+wQMrmzSv+f4uJkXLCBCltNv/WZWlcM7rHSHf7UgZxWz58WMqnnpKybt0zO6W33irl9u2+2XcTMzTQvS2HQox0t29kZ7SmOaNdgb8BXwIFQogfgaYAQoh4z8dhLcxk27ZtWts3GqP879kTfvwRHn4YhFALBIuK4N574dprwf70xi9YGteM7jHS3b4ZeP0Z6teHF19UOUrvuUdtv+bg00/VfNLRo9mZm+sfR6vADA101zkUYqS7fSOpqTN6PjAUeAfIBy4F6qJ2WPpNCLFDCPGhEOIewFqOGGQ45nXoat9ojPS/Xj144w147LEvad3aVf7tt3D55fDmm9Wul3AbS+Oa0T1Guts3A58/Q3w8ZGaqPYAHDnSVnzoFb77J5TfeCM8/D8eO+VZPFZihge46h0KMdLdvJNV2RqWUf0gpP5NSPiSl7IBKcv8HaqvPH4B41B707wINAYQQ04QQjwghrhDCSm5jcXbTsuU+1q2DJ55wDboUF6tFvGlpPm0KY2Fh4W8uvRS++AKWL4euXZ3FkSdOwDPPqByl776rOqkWFhZ+w6PUTlLKw0CJ/feuQAOgO/AscNx+2o3A68Bq4KAQYraffLXwkC5dumht32jM8L9Lly7Urau2x/7hB7UJjIPly1WmmVdegdJS7+0bie4ag/4x0t2+Gfj9M3TpAsuWqa3VLrvMVb53r9rFqV07tauT9E+ebrPuRToTCjHS3b6RVNsZFUL8VQjRSQgRXtn7UsoTUsolUsoXAMd2FsnAX4EZwCngBj/6a+EBMTExWts3GjP8L19HSgrk5akBFsdOhSdPqnzcnTureaa+2DcC3TUG/WOku30zMOQzCAH9+sH69RS88gqcf77rvS1bVDLhq64Ce9ofXzD7XqQjoRAj3e0bSU0jo/8FvgcKhRDfCCGeBaKru0BKuUZKOVZKOUhK2RRoW935FsbhyP2lq32jMcP/0+uoXRv++U9YvRo6dnSVr14NSUnwr3+BPdWcV/b9je4ag/4x0t2+GRj6GSIimF6/Pmzdqh5x1K/vem/VKrjmGujb17tvk3YCcS/SjVCIke72jaSmzugjwDSgEOiJehzfBEAIkSuEeE0I0V8I0bgqA1LKn/3kq4VFyJCYCCtXwn/+49qJ8NQpePZZ1wiqhYVFEFG3rpr8vWMHPPaY+mbpYM4c1ajT08GeqNzCwsJ9alrA9KaU8hYp5flAS2AEqmMKkASMAb4A/sSV8uk2IcQFxrls4S4tWrTQ2r7RmOF/dXVERMDf/gbr1qmnfQ42bIArr4SnnoITJ7y37w901xj0j5Hu9s3A1Bg1agT//a9afXjnnepxPqj5ox9+CK1bw+OPg31nII/tG4TuOodCjHS3byRCejgBWwjxC9AMOA/oBlwNpAHtTzt1Fypp/hIp5fu+u3p2k5ycLFevXu3RNSUlJURGGpdxy2j7ycnJePqZPcFo/z2po6wM3npLdUCPH3eVX3IJfPCBmlPqi31v0V1j0D9GutsH/dtytfZ//FF9q5w7t2J5gwbw97/DQw+pUVWbTT3qb9PGdc7hw9CgQVDdi7xFd43NqEN3+55qLITIk1Imu3OuR6vpyyOl/FNK+bmUcrSU8grU6CjAi8By4FxgOJDpjX0hxMtCiIVCiN+EEMeFEAeFEGuFEM9WNS1ACNFFCDHXfu5xIcQGe5qpShdg2a/pK4RYLIQoFEIcFUKsFELcWYNvdwohVtnPL7Rf37ea88OFEGPs/jg+y1whhKFL37zaHzaI7BuNGf67W0d4ODzyiPq/ds01rvKff1YZZh59VKWE8ta+t+iuMegfI93tm0FAY3T55eox/aJF0KmTq/zwYbU6sVUryMqCu+9W3y4HDYJZs9TPRo1g0CC+fP55Q/0H/XUOpvv12WrfSLzujFaFlPIZKWUaFdM+ecMY1GKpb4A3gMlAKfAcsOH0qQBCiAHAUtQo7RfAOFQi/rHAlMoqEEI8CMwC2gGTgPdQuVOzhRCvVnHNq0A2amT4Pft1lwOz7PZOP1/Y63/d7s84u39pwFK73xYWAFx8MSxcCOPHq8T5oJ7+jR2r/uctWhRY/ywsLKrgmmtU/racHNUBdfD772p3J0dHYfp06N9f/ZQSpk9n8AsvqM6pxknLLSx8wZvO6EpUp69apJQny6V98oZYKeVVUsq7pJR/syfeTwH+jeow/t1xohAiFtUxLAOukVLeLaV8HOiAygYwWAgxtLxxIURz4FXgIJAspRwlpRyDmm6wHfirEKLzadd0QaWt2g60l1KOkVKOQs2fPQi8ardbnqHAYGAF0EFK+biU8m5UR70MeE8IYUg+hqioKCPMmmbfaMzw35s6wsLgvvtg40bo3dtVvmMH9Oih0hweOeK9fU/QXWPQP0a62zeDoImREKpTuXGj+kZ5zjnuVzJ9upqr46fcpaeju87Ber8+m+wbirub2Nd0AHuBMn/Zq6aeRNQOUN+UK7vLXvZhJef3sL+35LTyf9nL/1nJNZXaAz6yl4+o5JpK7aE67hLoXsk1Vdo7/UhKSpJnG2fjZz4dm03KiROlbNBASvVfSh0XXCDlV18F2jvfsTQ+OzhrdS4qkvJf/5IyPLxiA67sGDFCyrKyQHvsNWetxmcRnmoMrJZu9u38+Zg+GTBjKVc/+88N5cp62H9+Xcn5S4FioIsQolwujmqv+eq0c7y6RghRB+hir3+ZB/X4THFxMe+99x5LlixhzZo1FFc24dBHcnJy/G7TTMzw39c6hFDZYjZuhAHlJnT89hvccAN0776TQ4d887E6dNcYjP8Mlv3AE7QxqlcPnn4aJk6s9rSj55wDmZmuPYMNQHeddbhfh7p9I4nwlyEp5W5/2SqPEOIxoB5QH9XhvRrVEX2p3GmO5YlbKvGr1J4BoC2qs7zJjWv+EEIcAxKEEFFSymIhRDRwPnBUSvlHJa5utf9sXa7sYiAc2CGlrGzDx8qu8QkpJcuWLWPZsmWUlpayZ88eAL766itSU1NJTU1FOFKR+MhBD1KXBCNm+O+vOuLj1ZbZn30GDz4IBQWqfPHi5lx2mdouu3xn1V/orjEY/xks+4En6GM0Y0a1b9fbtw9GjlQLnQzqkOqus07361C1byQep3YyGyHEXqD8xJuvgXQp5b5y52wBWgGtpJTbKrGxHDVC2UVK+b29rASoBdSqrKMohPgdNTc13t45jQd+B36XUiZUcn4toAQokVLWtpd1QWUWWC6lvLqSa1qhOsNbpJRtKnl/JDASoE6dOklt29a8mVWrVq245JJLqnz/559/ZuvWrVW+7wn79+8nLi7OL7YqY9OmTVx66aWG2Tfaf6PqOHWqAb/99gSHDvWqUN6w4TwuuOC/1Kp12G916a4xGP8ZLPs1o3tb9sV+vdJSvl2/3q3HkM82b86cxlXuIeMTurdlXe/XoWTfU43z8vLcTu1U5cioECJMSmlzu1Y38dSulPJc+3XnoDqULwFrhRB9pZRr/O1fMCGlzMSeGsudPKPFxcWMHTuW0tLKBmEV7dq14/333/fLROfCwkLql98az88YnbfOaP+NruOLL+D++23s26f+zR061Jvw8N6MGwe33OLKxe0LumsMxn8Gy37N6N6WfbY/aJBaoFQdSUn8c9Uq/mnQyKjubVn3+3Uo2Pciz6jb51b3V79ZCDGiuhydniCEaCOE+Aj4mzfXSyn3SSm/AHoBjVGLfxw4doWqSgVH+WEvrik87acRdRyu4n2P+Pnnn6vtiAKUlpby88/+2aE1Pz/fL3YChRn+G1nHjTfCxImrSE93lRUUwNChcNNN8Edlk0k8RHeNwfjPYNkPPEEfo2eeqfj64ovPPCcvT+0HbNDTSt111v1+HQr2jaS6zmhTIAvYbk80X/Wz3yoQQkQJIW4VQnwJbASGAce8c1UhpfwV+AloK4RoYi/ebP95xtxLIUQEcBEqR+mOcm9Vd815qBynu6WUxfZ6j6Ee09ezv386jsRy5eegbkelb2ph98Oda7ymqKjIr+fVxMaNG/1iJ1CY4b/Rdfz22wYmToSvvoILymXenTEDLrtM7U7oy/823TUG4z+DZT/wBH2MEhPVN0SAESPUbhYjRqjX557rOu+FF9QOFwZ0SHXXORTu17rbN5LqOqMXoZLNnwM8A2y07yA0XgiRIYToJIRoIYRoKISIFkKcL4RoJ4ToJ4T4pxBiLmpXpklAXyAf6COlfMMPfsfbf5bZf35r/3l9JeemAVHACinlyXLl1V1zw2nneHWNlPIEKr9oFJDqQT1eERPjXrpSd8+z0Ifrr4f8fJWf1MHhw2ol/l/+Art2BcozCwsLQI2O3n23WqQUEaF+ZmfDtm3satfOdd7//qcWM5WVVWXJwiLkqLIzKqU8KKV8FDVy+CqqY9kOuBeYgEomvxUoAI6g9qJfD8wA/g/VYasLLARuBq6QUlaWEukMhBCthRBnPNoWQoQJIV5EjdqukFI6ktrk2P0YKoRILnd+HcCRdH/8aeYmAieBB8snqhdCNASesr9897RrHK//YT/PcU1zYJTd3uk5PBz1vmD3x3FNCjAE2A9MO/2zesMll1xCRET1CRIiIiKcE5DXrFnj0+q7bt26eX1tMGCG/0bXUd5+bKzKs/3tt9CiXJK1r7+Gtm3Vinubh7PAddcYzNXAsh8YtIhRYiK8955rtXxYGNx5J0RHc/yTT2DwYNe5WVlw++1w6pTv9drRXedQu1/raN9IapwpLaX8TUr5JJAADADeRnU6bYCo5DiA2mLzCaCFlLKXlHKa9GzZ/l+AvUKIb4QQmUKI/wghPkB1fp9CJdi/p5yPR+yvw4HFQogsIcQrwDqgM6qzOvW0z/UL8DjQCFgthHhbCDEWlTbqYuA1x8r7ctesQG3reTFqS9KxQoi3gdV2O49JKXee9lmm2Ovvglp49YoQ4n1gkd3fe+z++0xUVBSpqZUNwLpITU2lbt26HDhwgNmzZzNu3Di+/PJLDh8+7HF9NXV8gx0z/De6jsrsd+8OGzaove4d88ePHlU7N117LWzf7pt93QiEBpZ9c9EmRlUs6IiIioJPP1WdUweffgo33wwnTvilat11DtX7tU72jcTtZXtSyjIp5SyptuXsiMr9eSFqK8wuwCVAEyllUynlQCnlq/b5nd6wAHgfiANuQnUaB6G23Pwn0FZK+dNp/s0AuqGS3A8CHgJOAY8CQyvrDEsp3wL6o+az3oFKo7QXlTrqsSri8FdghP28kfbrNgL9pJTjKjlfArfa/Si1+3WT3c80KeVMN2PiFqmpqXTv3v2MP8qIiAi6d+/u7KwuWbLEufPB2rVreeutt5gzZ45H80kXLlzoT9dNxwz/ja6jKvvR0Wo/++++gzblkoYtWgTt28Mbb7j3FFB3jSFwGlj2zUP3GC1cuFA9uv/gA3jgAdcbM2eqfeyP+bTUwlWHxoTy/VoX+0bidTfaPv/yN/vhV6SU+cCDXly3HDWq6sk1s1AjuZ5ckw1ke3B+KTDWfhiKEIK0tDSSk5PJzMzkiiuuICYmhksuuaRCOqeUlBSOHj3KL7/8AoDNZmP16tWsW7eO5ORkrr76aqKjo41218JgunSBdevgn/+E//5XdUCLi9Wo6dSp6n9fNWlpLSwszCQsDMaNUzs3vfKKKvvmGzUpfPZsMDi1kYVFoDBu7zGLgBIVFUWHDh3o1q0bHTt2PCOv6AUXXMAdd9zBHXfcwQXllmGXlpbyww8/8MYbb7Bw4UKOHz9eZR2tW/tt46iAYIb/Rtfhjv06deA//4EffoDLL3eVf/89dOgAL70EVWUE011jCA4Nzmb7ZqB7jCrYF0I1yuefd5V99x307AkHDvinDg05W+7XwWzfSIJ+ByYLhTtJ70/n2LFjbo1uSinZvn07ixYtcm4f6qB27dqMHDmSRo0aeW3fW4xOomy0/2bU4an9khLVMX3hhYod0KQkNUravr1v9j3FjKT3wabB2WYf9G/LAbM/dqxK9eSgXTs1Ulo+HZSvdfgJ3TU2ow7d7XuR9N7tHZiskdEQZvLkyW6dJ4SgZcuWZGRkMHToUM45x7X7apMmTWjYsGGl17lrP1gxw3+j6/DUfmSkyqudl6c6oA4cr597TnVYvbUfjASbBmebfTPQPUZV2h8zBjIzXQuf8vMhNdWrXG2663w23q+Dzb6RWJ1RCydCCNq0acO9997L4MGDadKkCd27dz9jS6+CggJO+THliIX5tG+vHtu//DLUrq3KSkvV3NLkZDB4sNLCwsJd7rkHJk2CcPtmiNu2qQ7p1q2B9cvCwo/omwfAokZiY2O9uk4IQdu2bbn00kvP6IjabDamTp3KiRMniIqKorS0VNt0Et7GJ5jq8MV+RAQ88QQMGAB33QUrVqjyH3+EK6+Exx+Hyy6rfFRcJ4JZg7PBvhnoHqMa7d92G0RFwZAh6tHFrl2QlqYe2ZdPmO9LHUHO2X6/Dgb7RmLNGdUEb+aMGsH69euZMWOG83X9+vVJS0sjMTGRcMc3dz9hxnxCC0VZGbz9Nvz972q1vYM2beD996FrV2PqtTQ+O7B09hPz58PAgeBYWNqokSorP+cmQFgahz7WnFELr5gyZYrfbYaHh1OvXj3n68LCQmbNmsXbb7/N+vXrsXm6xU8AMSI+ZtfhL/vh4fDww2pUtHt3V/nmzeqJ4COP+CXVYUDQRYNQtW8GusfIbfu9esG8eeDY0vngQejRQ62291cdQYp1vw68fSOxOqMhzJEjftnYqQLt2rXj4YcfplevXoSFuf58Dh06xIwZMxg/fjz5+fnoMOJuRHzMrsPf9lu0gAULYMIE1/87KVWS/MsvV1uN6oZuGoSafTPQPUYe2U9NhYULwbGw9MgR6N1bNVx/1RGEWPfrwNs3Eo86o0KIZ4QQj9Z8pvP8h4UQz3julkUwU6tWLTp37kxcXBw9e/akTp06zvcKCgqYNm0aEyZM4I8//giglxbeEhYGI0fCxo3Qrp1r1e4vv6hUh/feC4WFAXTQwuJsJyUFFi+Gpk3V6+Ji6NMHZnm0f4uFRdDg0ZxRIYQN2CuljHfz/F+AZlJK/04mPIsQQvQD+iUkJNzz9NNPA9CpUyfi4uKYM2cOAM2aNaNXr15kZWUBEBkZSXp6Ojk5ORw8eBCAQYMGsW3bNtavXw9Aly5diImJYd68eQC0aNGCtLQ0srOzAZU0f/jw4RVsDBkyhPz8fDZu3AjAlVdeSb169fjmm284duwYx48fp6zcHpMXX3wxw4cPZ8qUKc5vbMOGDSM3N5ctW7YA0LNnT0pLS1myZAkAbdu2pV27dkydOpVNmzbRtWtXBg8ezKRJkyi2T2ZMT09n6dKl7NixA4DevXtTVFTECvsKnMTERFq2bMm0adMAaNq0KQMHDiQ7O5sSe96ijIwM5s6d68yr2qdPH/bv38+qVasASEpKIiEhgZkz1W6t8fHx9O3bl6ysLGw2G2FhYWRkZDB79mynjQEDBrB7927y8vKcOtWrV49v7cOJVek0Y8YM/vzzT690ql27NnfeeWe1OnXr1o2IiAjnVnGtW7cmJSXFmQYkNjaWoUOHVqrTpk0/s2pVG6ZPT+PIEVczbtr0JG+/XcrBg8pGo0aNvNLpiSeeYNiwYdXqNH/+fHbZU9l4o9Ott95aIYduZTq5056q0slms3H11Vf73J6q0unjjz8mLCysWp3caU9V6XTzzTeTl5fnc3uqTqevv/6a8ePH+9yeqtLJZrNRp04dn9tTVTqtWbOGrfbV6760p6p0atWqFR07dqxWp8ra08kffyT+jjuod+gQADIigoXp6exISTlDJ5vNxsiRI31uT1XpNGnSJC699FKf21NVOiUlJfH5559Xq5Ov970bb7yRjRs3+tSeqrvv2Ww2rrjiCp/bU1U6TZgwASGEz+2pKp0mT57MK6+84nZ7uvfee92eM+rcn9ydA7ABezw4/xegzJM6rKPyIykpSXrKokWLPL7GF/vFxcVywYIF8sUXX5Rff/31GeeXlpZ6ZN+bz+wJRsfHjDrMsr9nj5QDB0qpHtq7jjvukPLAAe/tG62xlKGjga72pdS/LQe1/V9+kfLii12NUggp33/fv3W4ge4am1GH7vY91RhYLd3s4xg9Z7QJUFzjWRaG4Ph2Z5b9unXr0rNnT0aPHk1aWtoZ58+fP5+PPvqI3377zVC/3MXo+JhRh1n2zzsPpk9X+9nHxbne/+gjuOwy+OILQ93wiVDRQFf7ZqB7jHyy37w5LF2qGiKoLundd8Obb/qvjiDAul8H3r6RGNIZFULUF0I8AkQD24yowyJ4iY6Opm7duhXKCgsLycvL45dffuGDDz5g8uTJZ2w9ahHcCAG33KLmkt56q6t83z646SaVAtH+NMfCwsJM4uNhyRK44gpX2ejR8O9/B84nCwsPqLYzKoR4VghR5jjsxeeUL6vsAA4CrwES0Hd/Ks3p2bNn0Nj/9ddfK6R92rZtG++99x5Tpkxh3759RrhXI0bHx4w6AmE/Lg4++QRmzFAjpg4++0wNznz6qRqcCRZCUQOd7JuB7jHyi/0mTVS6iy5dXGX/+Ac89RRIqb3O1v068PaNxJ2RUVHukKe9ru74A3ge1Sm1CAClpaVBY799+/Y8+OCDtG/fvkL55s2beffdd8nJyWH//v3+drFajI6PGXUE0v6AAWqUdMQIV9mBA2qzmIEDIVgGvkNZAx3sm4HuMfKb/QYNVB7SHj1cZf/5D4weTal9cYyuWPfrwNs3kpo6o/8DLrIfLVCdzP3lyio7LgQaSikTpJTP2SexWgQAxwrAYLHfqFEjbrzxRh544AHatm1b4b2NGzcyfvx4vvjiC+fqSKMxOj5m1BFo+w0bwgcfwNdfQ7NmrvIvv1SjpBMnBn6UNNAxOtvtm4HuMfKr/Xr1YM4c6NvXVfbWW5CRobZa0xTrfh14+0ZSbWdUSlkopfzVfuwElgLLypVVdvwmpbSyEFpUSVxcHIMHD+bee++lTZs2znIpJRs2bLDmkmpI796Qnw8PPOAqKyxUe95ffz38+mvgfLOwOOuoU0etOLzlFmdRmxUrYNgwOHUqgI5ZWFSORwuYpJTXSCkHG+WMhX85ffQx2Oyfe+65DB06lIyMDFq2bAmonGtG++3AjHqCXQN/2o+JUfvbL14MF1/sKp8/H9q1g/HjIRC7xQZTjM5G+2age4wMsV+rlprcXX4ezdSpMGgQnDjh//oMxrpfB96+kVjbgYYw7dq108L++eefz7BhwxgxYgR/+ctfEEJUeH/btm3MmzePY37eHN3o+JhRRzDa79YNNmyARx9VK/ABjh5Vo6Y9esA2k/NrBGOMzib7ZqB7jAyzHx4OWVnw0EOuslmz1CP8o0eNqdMgrPt14O0biTfbgXp8GOW8RfU4donQxX6zZs248MILK5RJKfn222/54YcfeOONN1i4cCHHjx/3S31Gx8eMOoLVflQUvPYaLF8Ol1ziKl+yBNq3h7FjzZu+FqwxOlvsm4HuMTLUflgYvPEGa6+/3lW2cKGaW3P4sHH1+hnrfh14+0YS4eH5z6FW1LuLYwX+vzysx8ICgB07djj3uD916hTfffcdubm5XHXVVXTu3JnatWsH2EOL6ujcGdauheefh5dfVh3Q48fVqOlnn8Hx480D7aKFRegjBLk33sgVqakq3RPAihXqUcX8+SotlIVFAPG0M/oR1XdG6wNJwAWoXKOzvPTLwg80atRIa/ug9rodMmQIixYtcu6Pe/LkSZYsWcKqVavo0qULnTp1IjIy0mPbZvivuwb+sF+nDrz4opqqNmKEeoQP8MMPIMQn/Oc/8NhjaoqbEegQo1C2bwa6x8i0e9HIkWq1/ejRqnDtWjWv5ptvVOL8IMa6XwfevpEIIzIvCSGGA5nAZCnlPX6v4CwkOTlZrl69OtBumEpycjKOzyylZOPGjSxevJgDBw5UOC86OpquXbuSnJxMLaN6NBZ+oaREjZA+/3zFRb1XXKFSRHXoEDDXLAykfFu2CAI++EClenL8/7/4YvXo/rRpUp5gaRz6eKqxECJPSpnszrmGLGCSUk4CxgB3CSHuNKIOi5qZNGmS1vbLI4SgXbt2PPDAAwwcOJCGDRs63zt27Bjz589nwYIFHtk0w3/dNfC3/chIePppWLMGUlJc5WvXqtfPPAMnT1Z+rc0GH37o+Yp83WIUavbNQPcYmX4vuusutdI+wv5wdPt2uPpqCOK9za37deDtG4mRq+k/AsqA+w2sw6IaiouLtbZfGWFhYSQmJjJq1Cj69u1LbGyss/yqq67yyJYZ/uuugVH227VTU9bOP/8N6tRRZaWlasQ0KQlWrap4vs2mBnLS09VPTzqkusYoVOybge4xCsi9aOhQmDZNfUME2L0b0tLgxx8N98UbrPt14O0biWGdUSnlcaAYuMyoOizOXsLDw0lKSuKhhx7ihhtuoGvXrhVGSwGOHz/Ohg0bsAUiuaVFjUREwLnnfsz69WpQxsHGjWrh0xNPqMVOjo7oxInq/YkTPe+QWlhYVEL//jB7tkp/AbBvn5pDmpsbWL8szjoMmTMKIIRoDuwAjkgpGxhSyVmEN3NGS0pKvFrYEyz2fZ2DtHDhQr777juaNGnCNddcw2WXXVYhh6nR/ptRh+72HRrbbPDOO/C3v0H5dLItW0Lr1jB37pnXjhihUiiG1fCVWvcY6W4fjJ9PqHuMAn4v+u476NMHjhxRr2Ni1Jaiqalu29ddYzPq0N2+kXNGPV1N764D5wATUSvvrRnNPiCE6Af0S0hIIDMzE4BOnToRFxfHnDlzAJWfs1evXmRlZQEQGRlJeno677//PuHh4QAMGjSIbdu2sX79egC6dOlCTEwM8+bNA9Sq9bS0NLKzswGIiopi+PDh5OTkOPeKHzJkCPn5+WzcuBGA+vXrk5yczMKFCwFo3bo1KSkpTJ48GYDY2FiGDh3KlClTOGK/yQ0bNozc3Fy22Ocm9ezZk9LSUueeum3btqVdu3ZMnTqV1NRUcnJyGDx4MJMmTXI+gkhPT2fp0qXs2LEDgN69e1NUVMSKFSsASExMJD4+nuXLlwNQUFBATk4OtWrVol69etSuXZt77rnH+VkB+vTpw/79+1llfz6clJREQkICM2fOBCA+Pp6+ffuSlZWFzWYjLCyMjIwMZs+e7dy+dMCAAezevZu8vDynTr/88gv79++vVqcZM2Y4MwV4qtPx48cZPXp0tTp169aNiIgIr3RavXo1sbGx1eoEahWnNzpFR0c7/64TEpry448D6dNnD5s2qZW927ZVnSR/4kSl7YQJZcyaVbVOzZs358SJE9Xq5E57qkqnI0eOcP311/vcnqrSafz48cTGxvrcnqrS6cILLyQ8PLxanVq2bMm0adMAtUvawIEDyc7OpqSkBICMjAzmz5/Prl27gDPbU7Nmzdi3b5/P7akqnY4cOUKTJk18bk9V6fTFF184P6sv7akqnSIjI7nxxht9bk/V6XTkyBEeffTRynX66SeaPPgg/d9+m4jCQigqorRnT1b/3/9x1TPPuKVTamoqmZmZPrenqnQqKyvjV/u+wr60p+p0uuCCC6hbt65P7ak6nY4cOUJqaqrP7amq/09vvfUWMTExPrenqnSKjo5m5cqVbrcnj5BSun0AH9RwfILav/44ar7oKaC7J3VYR+VHUlKS9JQJEyZ4fE0w2ffmMzs4ceKEXLRokfz3v/8tn3vuuQpHZmam3Lp1q3z33Xf96G3l6K5BIDS22aR8910pIyKkVMt9qz9GjJCyrKzqOnSPke72pfStLbuD7jEyQwO36vjxRynPPdfVuCIjpfziC7fs666xGXXobt9TjYHV0s0+jqcjo+mo0U5Rw3kAe4AHpZSLPKzDwsJnateuzTXXXEOnTp1YsWIFq1at4pQ9l9CePXuYPHkytWrVYufOnTRv3jywzlpUQEpYuVItaHIHx1xSdx7ZW1hYVEO7drB0KfTsCb/9pnKxDR4MH30Et90WaO8sQhhPO6P/rOH9UuAw8COwXEpp0oZ/FpXRu3dvre37g6ioKK699lquuuoqli9fTm5uLmX2fShPnTrFhx9+SGJiIgMHDjSkft01CITGH3/s6mC6y8SJat3FnZUkktM9RrrbNwPdY2SGBm7X0aqVmkPas6eaI1NWBsOHq8nc9wQubXhQxegstW8kHnVGpZQ1dUYtgoiioiKt7fuTevXq0bt3bzp37syyZctYs2aNc5V9s2bNDKtXdw0CofHtt6s97D3pkI4Yoa6rDN1jpLt9M9A9RmZo4FEdzZqpEdLrrlPpLaRUuzcdOwaPPGKYj9URdDE6C+0bifVQK4RxTGzX1b4RxMbG0qdPHx566CHq1q1LkyZNSExMPOO8Q4cO+aU+3TUIhMZhYeqR+4gR7p1f06p63WOku30z0D1GZmjgcR3nnae+FSYlucrGjIEXXnDt3GQiQRmjs8y+kfjcGRVCRAgh4uyHIavzLSz8TYMGDWjQoAH33nuvM+OAg507d/Lmm2+Sk5NDQUFBgDw8u3G3Q+pueicLCwsvaNxYbRNaPhHw00+rHGwB6JBahC5e3cKFEPWFEE8JIdagEtvvtR/FQog1Qoi/CSHq+9NRC8+pbMRPJ/tGk5iYSERExe9PUkoWLVJr7jZu3Mg777zDjBkzvB4p1V2DQGpcU4fU3Y6o7jHS3b4Z6B4jMzTwuo769eHrr9UjewevvAIPPmjqzhNBHaOzxL6ReNwZFUJcDWwCngc6oOadCvsRYS97EdgkhOjqL0ctPKdly5Za2zeayvw/deoUdevWdb6WUrJ+/XrGjRvHrFmzKCws9LkOf6K7/ZqoqkMaGQmZme6NiOoeI93tm4HuMTJDA5/qiI6GL79UOzY5eOcdtce9u2kvfCToY3QW2DcSjzqjQohWwNfAucBB4N9Ab6Cd/ehlLyuwn/O1/RqLAOBIqqurfaOpzP/IyEiGDh1KRkYGF198sbPcZrOxZs0a3nrrLebOnev2RHHdNQgGjR0d0vR0V1lJiVpX4Q66x0h3+2age4zM0MDnOurUgZwcuPVWV9mHH6rX9mTtRqJFjELcvpF4k9opCsgDrpdSHjjt/Z+ABUKI14F5QBLwLDDcV0ctLMzk/PPPZ/jw4ezatYtvv/3WufNHWVkZubm5rF27lpSUFK677roKW4xaGENYGLz/Pvz0E9g3IWHRItD4qZSFhX7UqqVyr0VHq2+IoDqoxcXUNvGRvUXo4elj+p6opPd3V9IRdSKlPAjcbX95rZe+WfhI06ZNtbZvNO7436xZM+68805uv/12EhISnOWlpaUUFhbW2BHVXYNg0jgsrOLj+kVubqehe4x0t28GusfIDA38Vkd4uJojM3q0q2zuXP63bRsYmFpIqxiFqH0jEdKDFXFCiGLgpJSyoZvnHwZqSSmjvXPPwkFycrJcvXp1oN0wleTkZILpM0sp2bZtG4sWLeKPP/7ggQceIC4u7oxzrJFS9/FU482b4ZJL1O8NGkBBgfrfaBHcBFtbtvADUsIzz6hUTw6uugrmzoWGbnURLDTD03YshMiTUia7c66nI6O/AnWEEDXe/u3n1AZ2eViHhZ/Izs7W2r7ReOq/EIJWrVpxzz33cM8995zRET158iTvvPMOy5cvp8Q+h0p3DYJN49atVfpDgMOH1f/Bmp4O6h4j3e2bge4xMkMDv9chBDz/PLz0kqvshx+gRw/Yv9+/daFpjELMvpF42hn9DIgEhrhx7hBUZ3SKp05Z+IcSgyeVG23faLz1XwhBfHz8GeWrVq2ioKCABQsW8Oabb/LDDz9w8uRJX92slrNNYyGge3fX6+eeg4yM6jukusdId/tmoHuMzNDAsDqefBLeesv1et06SEuD33/3azVaxyhE7BuJp53RfwOrgAlCiKFVnSSEGAJMAH4A/uO9exYWeiClZNOmTc7Xx44dY968efz555+sXr2asrKyAHoXWqSlVXw9cWLNHVILCwsDefBB/nnhha5caz//DKmp8MsvgfXLQhs8nTP6DGq08wEgFvXYfgng+AoUD3QDmgOFwHig0qEhKeW/vHX6bMSbOaM2m40wA7emMdq+0fPM/O1/WVkZ69atY+nSpRw5cqTCew0aNCAtLY3ExES/1nm2aWyzwS23QGUZTKpKgq97jHS3D/q15VCzb0YdycnJrH7iCRg2zJV79PzzYcEC10RvHwiFGOluP5jmjD4H/A2oj0py3xy4E/i7/UgHLrK/18B+7rOnHc/Zf1oYzPz587W2bzT+9j88PJykpCQeeughrr/+eurVq+d87/Dhw3z55Ze8/fbbbNiwAZufhvHOJo1tNjUCWlUqvapGSHWPke72zUD3GJmhgSk633ILfPEF1K6tXv/+u3qUsX69z6ZDIUa62zcST/OMfoRK7WRhEkKIfkC/hIQEMjMzAejUqRNxcXHMmTMHUOmHevXqRZY971tkZCTp6els2LCBXbvU+rFBgwaxbds21ttvCl26dCEmJoZ58+YB0KJFC9LS0pwToKOiohg+fDg5OTkcPHgQgCFDhpCfn89Ge7bxEydO0KpVKxYuXAhA69atSUlJYfLkyQDExsYydOhQpkyZ4hwpHDZsGLm5uWzZsgWAnj17UlpaypIlSwBo27Yt7dq1Y+rUqaSmppKTk8PgwYOZNGkSxcXFAKSnp7N06VJ27NgBQO/evSkqKmLFihWA2hKtZcuWzgTATZs2ZeDAgWRnZzvn1GRkZJCfn++MT58+fdi/fz+r7Eksk5KSSEhIYObMmQDEx8fTt29fsrKynN8+MzIymD17Nnv27AFgwIAB7N69m7y8PAC6devGokWLOHHihLPzefDgQWbOnMmKFSuoU6cO6enpzJgxgz///NMrnQ4ePMj1119frU7dunUjIiLCK53WrFnDrl27qtUJoFGjRl7pFB0d7fy7rkqn+fPns3PnLj7+uBsrVrShOiZOVD+vuioLUDrZbLZqdXK3PVWlU0FBAQkJCT63p6p0cmjga3uqSiebzcaCBQt8bk/z58+vsj01a9aMffv2+dSeqtOpoKCAvXv3+tyeqtLpp59+cn42X9pTVTodPXqUzp07+9yeqtOpoKCAXr16VauTL/e91NRUMjMz6dSpEwkffkj9O++k1smTsH8/8pprmHHffey/6KIa21NVOu3cudN5r/ClPVWnU1lZGYsXL/apPVWnU0FBAQ0bNvS5PVWlk+Ne4Wt7gsrve9HR0axcudLt9uQRUkrr0OBISkqSnjJhwgSPrwkm+958Zk8w2n9HHSdOnJBLly6VL730knzuuefk999/71f7RhIMGpeVSTlihJQql4x7x4gR6jozPoNlv2Z0b8u62zejjjM0Xr5cyvr1XY2yXj0pFy/22n4oxEh3+562Y2C1dLOPE/BOlnUY1xndvXu3x9cEk32j/4EZ7f/pdRw/flwuXbpUnjp16ozzFixYILdt2yZtNpvX9o0gGDTOznavA3r6kZ1tzmew7NeM7m1Zd/tm1FGpxnl5UjZu7GqUdepI+dVXXtkPhRjpbt/Izqixs4EtAsp+A3K9mWnfaMzwv3wdderUITU1lYiIirNj9uzZw3fffcekSZPIzs5m586dXtk3gmDQ+PbbK+685A4tW0LPnup33WOku30z0D1GZt+LTKNjR1i61JUc+MQJ6N8fpk/32FQoxEh3+0bidWdUCHGNEOIdIcQPQojt9uMHe9k1/nPRwlsc80t0tW80ZvjvTh1Lly51/r5r1y4+/PBDPv74Y3bv3u0X+74QDBqHhalV8p50SLdtgzZt4F//gu++yzPOOfTXIBg09hXdYxQs9yJDuOwyWLYMLrxQvT51Si10mjTJIzOhECPd7RuJpwuYEEI0ASbj2nO+/N6HFwEpwL1CiG+A4VLKAp+9tLAIYf7yl79Qr1491q5d61zotGPHDnbs2EGrVq3o3r075zlGFs5SHB1ScC1SqowLLoDfflO/FxfDs89Cw4ZDaNAAbrvtzLRPFhYWJnDxxapDeu21sGULlJXBHXfAsWNw772B9s4iCPDo1iyEiAS+QXVEBSqp/YvA/fbjRXuZAK4D5tuvsQgASUlJWts3GjP8d6eO2NhY+vbty4MPPkiHDh0q7G2/detWMjMz+eyzz5wrGj217wvBpHFNI6QjRsDOnSqtYfv2rvJDh+px++3QuTMsX+5/v3TXIJg09hbdYxQs9yJDueAC9cj+8svVaynhvvvgtdfcujwUYqS7fSPxdJzgQSAROAT0llJ2lVI+LaWcYD+ellJ2Ba4HDtvPHeVXjy3cJiEhQWv7RmOG/57U0bBhQwYMGMCoUaO43HHDtrNp0ybGjx/PgQMHvLbvDcGmcVUd0vIJ73v2hDVr4L334JxzXOesWgVXXw1DhqhOq7/QXYNg09gbdI9RsN2LDOOcc2DxYkhJcZU99hj885+qc1oNoRAj3e0biaed0SGoPKMjpZTfVHWSlHI+MBI1QlrltqEWxuLIQaarfaMxw39v6mjcuDE33XQT999/P5deeqmzvGXLljRu3Nhn+54QjBqf3iGtbOel8HCVAH/rVrj++rXOHNwAn32mNoT5+9/htI2yvEJ3DYJRY0/RPUbBei8yhEaN1OOL8vv6PvccPP54tR3SUIiR7vaNxNPOaBvgBPCFG+d+YT/X933ALCzOQpo2bcott9zCyJEjad26Nd27dz/jnJMnT1JYWBgA7wKLo0OanV35FqAOYmLgxhtz+flnNSLq4ORJeOklaNVKjaCWlZnitoWFBUBsLHz1FfTu7Sp77TV44IEzt1CzOCvwtDNaCzhlzx9VLVJKG3AKLxZJWfiH+Ph4re0bjRn++6OO8847j1tvvfUMW6WlpRQVFfHWW2/x1VdfUVRU5HNdpxPMGoeFwZ131rwoKT4+nubNYcoUNWe0UyfXe3/+CSNHwhVXqMEab9C9nQWzxu6ie4x0uRf5lagomDkTbrzRVfbuu5Ce7trbvhyhECPd7RuJcKNf6TpZiE1AayBFSrmmhnOTgFxgs5Ty0urOtaiZ5ORkuXr16kC7YSrJycmcbZ/ZE3744QfnNmwAERERpKSk0LVrV6KjowPomfsEQmObDT79FP72Nzg9e1bfvvDqqyotlIX/sNpy6OO1xqWlqgNq36YTgJtugk8+ocL8GouA46nGQog8KWWyO+d6OjI6FzUP9H0hRFw1DpwDvI+aXzrHwzos/IRjj1ld7RuNGf4bWce5555LrVq1nK9LS0v5/vvvefPNN/n22285fvy4z3XorjGc+RnCwmDYMNi8WeUhjYpyvTd7NrRrB6NHg33La4/t+xvd7ZuB7jHS/V7kExER8NFH6hGFg+nTYeBAlZ/NTijESHf7RuJpZ/Rl4CDQHvhZCPGSEOJ6IcTlQogUIcQgIcQ4YLv9nEPAK/512cJdbAbPvTHavtGY4b+RdTRv3pzGjRtz6623cu655zrLS0pKWLZsGW+88QZLlizh5MmTXtehu8ZQ9WeIioKnn1aLnEaMAEdGrdJSePNNtYvTG2+oHN3e2PcXuts3A91jpPu9yGfCwtQj+jFjXGVffw1/+QvYpx+FQox0t28kHnVGpZR/An8B9gENgcdRI5/rUPlFP0PlG40C/gBusF9jEQDCDM7wbbR9ozHDf6PrCA8Pp3Xr1owcOZJbbrmFuDjXA4uTJ0+yePFi3njjDXbt2uWVfd01hpo/Q3w8fPABrF4N3bq5yg8dgkceUSOlX35Z9UJf3dvZ2aDx2W7frDp8Qgi1iOmZZ1xlS5aoRPkHD4ZEjHS3byQezRl1XiREA+AhYBDQDlen1gbkAznAOCnlYb94aWHNGbVwC5vNxsaNG1m8eDEH7c+Z69Spw+jRo6lTp06AvTuTYNNYSpgxQ2WZ2b694ns9esDrr0NiYkBc05pg09nC//hV4//+F554wvW6fXuYP79i4mAL0wmmOaMASCkPSymfl1J2AOoC59mPulLKDlLKF6yOaOCZPXu21vaNxgz/zdYgLCyMyy+/nFGjRjFgwAAaNGhA586dz+iIHj9+nDI38hnprjF49hmEUIt7N25UC5nq13e99+23atX9PffA3r3e2fcG3e2bge4xCoV7kV95/HF4+23X6w0bOJqUdOaKQz+ju85aaXwaPo/pSilPSSn32Y8aZldZmMmePXu0tm80ZvgfKA3CwsLo0KEDDz74IJ07dz7j/a+++opx48axbt26aucZ6a4xePcZateGv/4Vtm2DUaNUEn1QI6dZWSo/6X/+A8eP69/OzlaNzyb7ZtXhVx54QCUStj96rvf775CaeuYjCz+iu87aaVwOKwdokCOE6Af0S0hIIDMzE4BOnToRFxfHnDkqUUGzZs3o1auXcyVdZGQk6enpHD582HnNoEGD2LZtG+vXrwegS5cuxMTEOFMDtWjRgrS0NLKzswGIiopi+PDh5OTkOB/3DhkyhPz8fDZu3AjAiRMn2L59OwsXLgSgdevWpKSkMNmeoiM2NpahQ4cyZcoUjti3uhk2bBi5ubls2bIFgJ49e1JaWsqSJUsAaNu2Le3atWPq1KmkpqaSk5PD4MGDmTRpEsX2lZXp6eksXbqUHTt2ANC7d2+KiopYsWIFAImJibRs2ZJp06YBKnn8wIEDyc7OpqSkBICMjAyOHDnijE+fPn3Yv38/q1atAtQevwkJCc4dLeLj4+nbty9ZWVnYbDbCwsLIyMhg9uzZzhvAgAED2L17N3l5eU6dSkpKnHVUpdOMGTOc+857qpNDm+p06tatGxEREU6dzj//fH788UdA7dgxd+5c+vXrR35+vjNXqUOngoICMjMzq9UJoFGjRl7pFB0d7YxPVTrNnz/fOefVG52AGnWqrj0lJ0eyYUM6t9++lzVr1EKxo0fhqafgtdeKuPbapnTunE9srG/t6XSdHO3JoYGv7akqnWw2GwsWLPC5PVWnU7Nmzdi3b5/P7akqnQoKCsjOzva5PVWl09GjR51/p1Xp5Mt97+jRoxQWFvrcnqrTqaCgAJvN5nN7qkqn1NRUMjMz3f7/5JZOJ09yUUYGPT/4gLDSUti5k2PJySz829/o/+STXrWn6nQqKytj8eLFPrWn6nQqKChg5cqVPrenqnRy3Ct8bU9V6RQdHc3KlSvdbk+e4O2c0TCgC2q+aENUMvwqkVL+y+NKLCrgzZzRffv2cY6Bc2yMtm/0PDOj/TejDm/sb968mZkzZ56R+ikuLo5rrrmGSy+9FGFfWq67xuDfzzBvHjz6KPz0U8Xyzp1h7Fi48kq/VFOBYPwb8hTd27Lu9s2ow1CN585FDhqEOHFCvW7SRM0hveIKv1aju87BprGhc0aFEDcCu4AlwNvAC8CzVRzP2X9aBIDdBs+vMdq+0ZjhfzBq0KZNG0aPHk337t2pXS6p9P79+/n888/JzMxky5YtSCm11xj8q0Hv3rB+Pbzzjvp/6OD77+Gqq1T+Ui8TF1RJMP4NBRu6xygU7kWG8pe/sOV//wPHZh4FBdC9u2p4fkR3nXXW2KPOqBDiWuBzIB611edyVDqnj6o4PrT/tAgAjuF4Xe0bjRn+B6sGtWvXJi0tjdGjR5OamkpkZKTzvb179/Lpp5/y/vvv872fb/aBwN8aRETA/fer/KSPPQbh4a6FYJ98onZvevpp9SjfHwTr31AwoXuMQuFeZDRLhFB79jZooAoKC+G662DRIr/VobvOOmvs6ZzRp1Ad2CXAbVLKP/zvkoWFhVnUrVuXHj16cNVVV7F8+XJWrVpFqX1f6N9//51GjRoF2MPgpUEDlYGmQYPPWLv2VuxTwDhxAl54Ad5/H158Ee6807kGw8LCwheuukp1Pnv1gv374dgxlRh/2jT100JbPL1FJqG2+Ey3OqLBT6dOnbS2bzRm+K+LBlFRUVx33XWMHj2aTp06ER4eTrNmzbj66qsrnOfNHPNAY7QGffpcSk6Oys/dsaOr/I8/4K67IDkZFi/23r4uf0OBRPcYhcK9yGic/nfoAEuXqt0qQH37GzgQcnL8V4dB6G7fSDztjArgiJTyVyOcsfAv5Xfj0dG+0Zjhv24a1KtXjxtuuIGHH36Yvn370rRp0wrv//jjj3z66af88Yc+30XN0iAtDXJzVTYax/9JgLVr1fS2G29UqaK8tW8Uurdj0D9GoXAvMpoK/l9yCSxbBhddpF6fOgVDhsCHH/qvDgPQ3b6ReNoZ3QRECyGCbysXizNwpGzQ1b7RmOG/rhrExsZWSPsBFdOeZGZm8tlnnzlTfgQzZmoQFqYey2/ZonY1rFvXdd6MGXDZZSp/6eHD3tk3At3bMegfo1C4FxnNGf63aKFGSNu0Ua9tNkhPV6sL/VWHn9HdvpF42hl9BzXP9HYDfLGwsAhidu3axaFDh5yvN23axPjx45k+fToHDhwIoGfBR3Q0/POfqlM6fLir/NQptaVoy5Zqgxn79FwLCwtvSEhQHdLye/SOGgWvvBI4nyy8wqPOqJTyQ+B94H9CiKHGuGThL5o1a6a1faMxw3/dNShv/6KLLuL+++/n0ksvrXDOjz/+yNtvv83MmTM57MmQn0kEUoOEBPj4Y1i5Erp2dZUfOAAPPqi23P7qK+/t+wPd2zHoH6NQuBcZTZX+N22qFjWVT/L75JPq0YSHc9x111lnjatMei+E+KCa6wYC9VH5RlcDRdWcK6WUd3vroIXCm6T35XegMQKj7RudKNto/82oI1D2//jjDxYtWsTWrVsrlIeFhXHFFVeQlpZGbGxsjfbNSHofLBpIqdZYPPEE7NxZ8b3eveG116BtW+/te4sZ7UD3tqy7fTPqCLjGRUXQv3/F1YJjxqiGZd/Ew+c6fER3+4FKep8O3Gn/efrRALWY6UJgUBXnlD8sAoBjWy9d7RuNGf7rrkFV9s877zxuu+027r77blq0aOEst9ls5OXl8cMPPxjqlycEiwZCwM03w6ZN8NJLEBPjem/ePDVKev/9KmONN/a9Rfd2DPrHKBTuRUZTo/8xMTB3Ltxwg6ts7Fi4914oK6v6Ok/q8BHd7RtJdXlG/2maFxYWFlqSkJDA7bffzs6dO1m0aBG7du2iVq1adC3/TNqiAnXqqKeI6enqSWJWllp7YbPBu++qxPn/93/w8MNQboMsCwuLmqhbV60UvO02nIl/33tP5SPNzoZa1e5cbhFAquyMSikD1hkVQjQGbgT6AJcD5wMlwI/ARGCilNJWyXVdgP8DrgLqAluBD4C3pJSVfjUSQvQFHgOuAMKBjcA79vmxVfl3JzAKuAwoA9YCr0opZ1dxfjjwMDACaAUcB34AXpBSrqguFr5QflcdHe0bjRn+666Bu/abN29Oeno6O3bs4PDhw0Q7tu2zU1hYyJo1a+jcuTN16pibjCNYYnQ655wDEyaouaOPPqo2lwE4ckQ9yn/3XbUOo1at4PQ/mAhWjYPFvll1GInb/kdGwpQpKsnvxx+rsk8+UR3SqVOr/Yanu846a1zlnNFAIoS4DxgP/AEsQs1NPQe4CTVXdRpwsyznvBBigL38BDAVOAj0A9oAOVLKmyup50HgLeCA/ZoSYDCQALwmpXyskmteBf4K7AZygEhgKNAIeEhKOe608wVqy9TBwGZglv3cIUAdYJCUcmZNMfFmzqjumDGf0MIcZs2axZo1a6hTpw6dO3fmyiuvpHbt2pbGdqRUTxj/+lfYvLnie2lpagV+UlJgfPMHls6hT9BpbLOplfXvvusq69ULvvgCoqIC55fGBGrOaCDZAvQHEqSUw6SUf5dS3gVcAvyGmqd6k+NkIUQs8B5qlPIaKeXdUsrHgQ7A98Dg01f/CyGaA6+iOq3JUspRUsoxQHtgO/BXIUTn067pguqIbgfaSynHSClHoXamOgi8ardbnqGojugKoIOU8nH7gq7udn/fE0LEYAAzZswwwqxp9o3GDP9118Af9g8fPsy6desAOHHiBIsWLeLNN99kxYoVhIeH+2y/JnSIkRDQpw/8+CO8+SaU34V16VK1i9Odd8Lvv/tc1Rno3o5BD40Dad+sOozEY//DwlTO0cfKjSnNnw/XX68eP/ijDg/R3b6ReNQZFUKkeXhcKYRoI4Sobm7qGUgpv5VSzjr9UbyUci/g+JpzTbm3BgNxwBQp5epy559APbYHuP+0au4CagPjpJQ7y11zCPi3/eV9p13jeP2i/TzHNTuBt+32Rpx2jaPe/7P747gmFzUaG2f33+8YnZBch4Tn1WGG/7pr4A/7sbGxDBw4sMI+98XFxXzzzTf06NGDlStXUmpgwk0dYuSgVi146CHYuhUeeQQiyt05P/oIWrdW+UuLi/1WpfbtGPTSOBD2zarDSLzyXwg11+Wf5WYdLlsGPXuq/Gr+qMMDdLdvJJ6OjC5GPTZ391gB/AQUCSG+FkJc5wefT9l/lv/v1cP+8+tKzl8KFANdhBDlJ4tUd81Xp53j1TX2naq62Otf5kE9FhYhQ1hYGJdffjmjRo2if//+NGjQwPlenTp1+Prrr3nrrbfIy8ujzM1Vr6FOo0ZqIfDGjZCYuNNZXlwMzz2nOqUff6yeRFpYWFSDEGql4KuvuspWr4ZrroG9ewPmlkVFPJozKoTYCUjUaJ5j0sUp1JxLgMaAY7laMVAAxKJSQWG/9nkp5XNeOatGWNcC7YDrpZTz7OW5QDLqcXteJdflA22By6SUm+xl+4EmQBMp5RlfkYQQR4FoIFpKWSyEiAaOAkellGc8VhdCNAH2A39KKc+xl7UF8oF8KeXllVyTDOQCq6SUV57+fnm8mTN64MABGjdu7NE1wWTf6DlIRvtvRh062i8rK2Pt2rUsXbqUoqKKKYpbt27Nrbfe6tf6dIzR6fbXr2/MmDGwYUPF95KTVaf16qt9s290O9C9Letu34w6tNB4wgSVP83R72nVSq0ctCeL113nYNPYkzmjnj4+by6EeBj4LzAXeBn4QUp5yl5xLeBK4EngOuAVKeU7QogW9rJ7gKeFEIuklEs8qdvOS6iO6FxHR9ROffvPwiquc5Q38PCaaPt5xQbWcfo1ToQQI4GRoEaQkpPd0tTJsWPHzljV7E+Mtr9p0yaPP7MnGO2/GXXobD8sLIw6depw9dVXU9u+wnXy5Mm89tprfq1H5xiVtx8REcaFF/bj99/vp7S0CaAGeFJToWHDbzj//LeoXXuP1/aNRPe2rLt9M+rQReMbLryQZ3fuVJ2frVv5o1Ur7m/Vit116mivs9YaSyndPlCplsqATDfOzUQ9Su9VruwdwAZM9aRe+7UPo0ZWNwGNTntvi/29llVcu9z+fudyZSX2sogqrvnd/v559tfx9te7qzi/lv39k+XKutjLvqvimlb29zfX9PmTkpKkp0yYMMHja4LJvjef2ROM9t+MOnS3n5SUJE+ePCm/++47+fHHH0ubzVbhfZvNJrdt21ah/NixYzIvL08uXrxY5uXlyWPHjlVbh+4xOt3+kSNSPvWUlLVrS6mGeNQRGSnlk09KWVjom30j0L0t627fjDq00nj6dClr1XI1nnPPlTI/X3udg01jYLV0s4/n6ZzRv9o7T0+5ce5TqF2aHi9X9or9ZxdPKrWnYHoDNf+0u5Ty4GmnOEYY61M5jvLDXlxTeNpPI+o4XMX7FhYhT2RkJF27dmX48OGI07bt+/nnn5k0aRLvvfceW7ZsYcmSJYwdO5ZZs2axePFiZs2axdixY1m6dKnjC17IExMDL76oUkANLZcjpKQEXn4ZWrZUTyMNXBNmYaE3N94IX36pdqAANXe0Wzea/PprYP06i/G0M5oIFEopC2o60X7OYaBjubKdqH3s49ytUAjxCCoXaD6qI1rZjGNHZr7WlVwfAVyEGqXd4eY156Ee0e+WUhbbfT+GGi2tZ3//dFrZf24pV7YdNZLcooqMApVd4ze6dPGozx909o3GDP911yCQGkspWWzfZ/qPP/7g008/ZfHixWesvC8tLWXRokUsW1bZGkH9Y1SV/QsvhE8/hRUr4MpyM87374f77oMrroBvvvHevk6Eqsa61WEkfvf/+uvh66+hXj31+sABBrz5Jixf7t96yhEKf0dG4WlntDYQ605eTPs5saik8KfX6VZiEiHEk8BYYB2qI1pV3oJv7T+vr+S9NNRiqxVSypNuXnPDaed4dY1UqZxW2OtP9aAevxATY0j6UtPsG40Z/uuuQSA1Li0t5eKLLyYiwr2p7cuWLaO4kpxHuseoJvudO6sO6eTJcMEFrvL8fJXju29f+Pln7+3rQKA1CHb7ZtVhJIb4360bLFwIDRsCEH70qGo0Cxf6vy5C4+/IKDztjObbr3nCjXMfx7W9JgBCiEaoEcd9NV0shHgatWApD+hZw2hsDmrl/lD7CnWHjTrAC/aX40+7ZiJwEniwfKJ6IURDXNMQ3j3tGsfrf9jPc1zTHLU96Em73fI46n3B7o/jmhTULkz7UTtH+Z158+bVfFIQ2zcaM/zXXYNAalyrVi169erFww8/TPPmzWs8v7S0lJ8r6XXpHiN37IeFqe24N2+GF16A8msY5syBdu1U/tJKUitq344hODQIZvtm1WEkhvnfqRMsXgxNm6rXxcVqB4pZs/xeVSj8HRmFp53R8ah5oE8JIcbbV8lXQAhxkRDibeAfqPml5TuB3e0/11ZXiX3v93+hHnEvAx4WQjx32pHuOF9KeQS1Uj8cWCyEyBJCvIIaUe2M6qxOLV+HlPIXVIe5EbBaCPG2EGIssAG4GLUd6PenXbMCeN3+/gYhxFj7Z11tt/OYLJdA384Ue/1dgLVCiFeEEO+j8rCGA/fY/bewsKiEmJgYtzqjwBmpos426taFf/xDJc2/6y6VYhGgrAzGjVPzSceOVfNLLSws7LRvD0uXctQ+QsrJk3DTTWovewtT8KgzKqX8EHgf1SEdCWwVQvwuhFhtP34HtqF2KhLARPs1Dvqg5l3WtBf7Rfaf4cAjwLOVHOmn+TYD6IZKcj8IeAiVA/VRYKisZHWDlPIt1LajG4E77J9pL5AuK9mX3n7NX1G7LO21n3+H/fp+8rR96e3nS+BWux+ldr9usvuZJt3Yl95bWrQ447uCVvaNxgz/ddcgWDR29/FTZefpHiNv7J93Hrz/PuTlqdzeDg4fhkcfVSOlM2eqpcTBorEvBKMGwWTfrDqMxHD/27Rh3ZtvgqOe0lL1uGHi6Q87vScU/o6MwqOk986LhBgBPA00r+KUX1HJ7T/w3jWL8niT9L6kpITIyNOn7PoPo+0bnUTZaP/NqEN3++5qXFxczNixY6vdNjQiIoJHH32UunXrArBgwQKio6Np27YtsbGxfvP5dIJdAylVx/Pxx2HbtorvmMQGUQAAcTVJREFUde8OL798ipSUWpVf7Cd0b8u62zejDt01dtZRUADXXgubNrneePNNNc/FH/Y1/jsyMum9p4/pAZBSTpRStkDtenQP8Hf7cY+9rIXVEQ082dnZWts3GjP8112DYNE4KiqK1NTK1gC6SE1NdXZEjx49yvfff8/8+fP53//+x6xZs9hr0NZ/wa6BEDBwoNpa9LXXoH65RHOLFsGVV0aQkaH3zojBrkGg7ZtVh5GYFqP4eFiyRKWjcPDww/Cf//jHvoHorLFXnVEHUso1Usr3pZQv24/37WVnR8I/CwsL00hNTaV79+5nrK6PiIige/fuFTqra9euxWbfuF1KyZo1a5gwYQITJ04kPz+fsrIyU30PBiIj1SP6bdvgwQchPFyVSyl4/321M+K//w3HjwfWTwuLgBMXB99+q1JVOHjqKTUh2+reGIJH24Fa6EVUVJTW9o3GDP911yCYNBZCkJaWRnJyMj///DNFRUXExMRwySWXnOHnlVdeSVRUFLm5uezb50resWvXLnbt2kW9evVISkoiKSnJ53QoumnQpAm89RY88AA89hjMnavKjx5V/2snTICXXlIJ9U/bgyBo0U0Ds+2bVYeRmB6jBg1g/nzo3189QgD1be3oUbUKMMzzsbxQ+DsyCq/mjFqYjzdzRnXH6DlIFoHHaI2llPz222+sWrWKTZs2OUdLHYSFhdG9e3euvvpqw3wIdubPVyOmGzdWLL/qKvU/96qrfK/DasuhT8hqfPw43HyzypHm4K67IDPT9XjhLCFo5owKIcq8OKxN6QJETk6O1vaNxgz/dddAd42FEKxatYrBgwfzyCOP0K1bN+o5dlwBbDYbTR35Bb1Edw2OHMlh3ToYP149nXTwww/qKeVtt8GuXYa64DO6axAK9yKjCViM6taF6dNVh9TBBx/AsGFw6pTv9v2Izhp7Os4svDh8mpdq4T0HDx7U2r7RmOG/7hrorjG4PkNMTAzXXHMNjzzyCIMHD+bCCy+kQYMGtGzZssL5ZWVlLFiwgAOVZYivxr5RmGE/IkJtIbp1q1p1X35B7qefQps28H//p55QBiOhoIHR6N6WAxqjyEj45BO4805X2dSpMGgQnDjhu30/obPGnnYUL6rh6IBaUb8ROAgMwJUz1MLCwiLghIeH07ZtW9LT07n33nsJO23u16ZNm1i+fDnjxo1j0qRJbN68+YzH+6FK/frwyisqq82gQa7yEyfgxRfVIqcPPlBJ9C0szioiItQf/6hRrrJZs9R+u8eOBc6vEMGQOaNCiEjgK+ByIElK+ZvfKznL8GbOaGFhIfXL53HxM0bbN3oOktH+m1GH7vbNmGfm6WeYOHEiu057Ll2/fn2Sk5Pp2LHjGYsEdNegOvvLlsGYMSp5fnk6dIDXX1d5St1B97asu30z6tBdY7frkBL+/nd4+WVXWdeuak5pDdfq/ncUNHNG3UVKWQI8DDRB7ZZkEQDy8/O1tm80Zvivuwa6awyef4a0tDRat25doaywsJCFCxfy+uuvM2PGDPbs2eO1fU8JpP3UVFi1Cj78UKVfdLBuHfToofKXbt1qqHtuEcoa6FSHkQRNjIRQOUdfeMFVtny5ahAFBb7b9wGdNTYstZOUcqMQ4ghwvVF1nA0IIfoB/RISEsjMzASgU6dOxMXFMce+uq9Zs2b06tWLrKwsACIjI0lPT2f58uVstC+RHTRoENu2bWP9+vUAdOnShZiYGObNmweobcTS0tKcSXOjoqIYPnw4OTk5znkoQ4YMIT8/32nzxIkTnHvuuSxcuBCA1q1bk5KSwuTJkwGIjY1l6NChTJkyhSNHjgAwbNgwcnNz2bJlCwA9e/aktLSUJUuWANC2bVvatWvH1KlTSU1NJScnh8GDBzNp0iSKi4sBSE9PZ+nSpezYsQOA3r17U1RUxIoVKwBITEykZcuWTJs2DYCmTZsycOBAsrOzKbFvyp2RkcH333/v/Cx9+vRh//79rFq1CoCkpCQSEhKYOVPt1hofH0/fvn3JysrCZrMRFhZGRkYGs2fPdnZMBgwYwO7du8mzDyN16tSJtWvXOuuoSqcZM2bw559/eqXTwYMH6dq1a7U6devWjYiICK90WrFiBRs3bqxWJ4BGjRp5pVN0dLTz77oqnebPn+8cqfRGJ5vNxqFDh6rV6fT2NGTIEN59912Ki4s5fvy48zF9WVkZ69evZ/369dSvX59atWo5RyN8bU9V6bRkyRI2btzoc3uqSiebzcbx48er1alPn5YUFs5k/vxE5s9PpKRE/euYORNmzy7joYfC6NJlIYcO7ahUpwsuuJA33ywkMnIqYWHet6eq7nsFBQVs3brV5/ZUlU4rV670S3uqSqejR4/6pT1Vd98rKCigc+fOPrenqnRKTU0lMzPT7f9Pnuq0bds2pwa+tKfqdCorK+PUqVPutae4ODqlp9PBkWh+zRoOtm9Pve+/Z+nWrZXqVFBQQEREhFv/n7zRyXGv8LU9VaVTdHQ0K1eudLs9eYSU0pADiARKgONG1XE2HUlJSdJTJkyY4PE1wWTfm8/sCUb7b0Yduts3WmMpff8Mp06dkmvXrpWZmZnyueeecx4vvviiPH78uPYaeGr/t9+kvOMOKdXzStfRuLGUb70lZUlJxfPLyqRs3HimBClHjFCv/U2wxSjY7JtRx1l7v37vPSmFcDWEFi2kXLiw4jmHDnlv3wOCTWNgtXSzj2PkSvfbUCOve2o60cIYunXrprV9ozHDf9010F1j8P0zRERE0KFDB+655x4yMjJITEwkPDycyy+/nDp16lSwX1BQwI4dOxxfyP1CsGmckKAe2+fmQvn0rAcOqO2727dX0+ekBJsNMjLgwIH+AEycqF77ez1YsMUo2OybVYeRBG2MMjJg8mRXztEdO6BnT+jdWy1wGjQIGjWCQYPodc45/nX4NHTW2KPH9EKIZjWcUgdIQK2ivweQwOfeuWbhK6dvm6ibfaMxw3/dNdBdY/DvZzj//PM5//zzue6665yP7svbX758OevWraNx48akpKSQmJhInTp1fKozWDVOToalS2HaNHjiCfjlF1X+889qgfG110K9ejBjRsXrJk5UP7OyvNrEplKCNUbBYt+sOowkqGN0661Qpw4MHuz6pjV/vjocTJ9O8+nT4aab4JlnIDHRd4dPQ2eNPb0V/FLDsQn4BngQ1TFdBTzvL2ctPMMxV0ZX+0Zjhv+6a6C7xmDMZ4iOjnZuI+qwX1xc7FxAcODAAb7++mtef/11Zs+e7Zxz5Q3BrLEQ6v/vTz+pxcXld1ZdsODMjqgDf4+QBnOMgsG+WXUYSdDH6LLL3PuDnj5d7cdrQCYjnTX2d9J7Gyq/6BLgASBVSmkl4LKwsAh5bDYbHTt2JLJcxvhTp06Rl5fH+PHjyc7OZuPGjZSFYJLOOnXU6Oi2bTBypHvXGPXI3sIiILRpo0Y9a2LECLWVqBDG+6QRHo3pSimt3ZQ04vT0NLrZNxoz/NddA901BvNiVK9ePW644QZ69OjBhg0byM3NZf/+/c7zfv31V3799VdiYmJISkoiLS0N4cY/JJ00btLEsx0SJ05U53/4oW+P7HWKUSDsm1WHkWgRo/R0NfJZBcfj46mbmem/+SmnobPGhiS9t/A/3iS9P3bsGNHR0QZ5ZLx9o5MoG+2/GXXobt+MpPeBipGUkl9//ZXc3Fw2bdpUYVFT8+bNubP81oJe2PcX/rT/4Yfq/7GnxMfDdddBp07qaN++4pakNaFTjAJh34w6rPs1arFSNZ1RQI2M+nPCdDmCTeOAJ723CA4c+dR0tW80Zvivuwa6awyBi5EQgubNm3PzzTfzyCOPkJaW5vxHkZKScsb5e/fu5VQlw4o6aXz77ep/rafs2aM6sqNGQUqKmnt65ZVqdf7HH8PmzdU/ztcpRoGwb1YdRhL0MTp8GL74oubzJk5Uf9QGoLPGXi+9EkLUA/4CdATi7MX7gTXAXCnlUd/ds7CwsNCf2NhYunfvTlpaGj///DOXXHJJhfdtNhtTpkzh5MmTdOjQgZSUFBo1ahQgb70nLEwN+oBr1Xx1NGwIhYVndjRLStTOT/Yc3wDExqqOqmP0NCUFzj/ff75bWPhEgwZw443ujYzefrspLumEx51RoSY5/R14EqhXxWlHhRD/AV6W1jyAgBEbG6u1faMxw3/dNdBdYwiuGIWHh9O2bdszyrds2UJhYSEAP/zwAz/88AMtW7YkJSXFuWrfKPwdH3c7pI6nlSdOwNq1rs7nqlUqVePpHDkCCxeqw0F8vOqY1q59FS1aqHRTDRr49eMAUK9eLB9+qPoQRkz3C4V7kdFoEaNnnqnYGQ0Ph3ILFnd060YLgx7Rg94aezxnVAjxITActXr+BJAH7La/nQAkodI6SWCSlNK9iVEW1eLNnFHdMWM+oUVgsTRWbNq0iW+++YZDhw6d8V6DBg1ITk7miiuuICoqKgDeeYcj4X1lHdKaps0VFMDq1RU7qOXWglVLmzYVR1ATE9Vqf398DgOn+2mP1ZbtOOaNjhgBdevCO++o8ksugY0btf7jCZo5o0KImwDH+PJ/gHOllKlSylvtRypwLvCS/ZzhQogbPanDwn9MmTJFa/tGY4b/umugu8agR4wuvfRSHnroIW677TZatWpV4b3Dhw+zYMECxo4dy8yZM9m7d6/P9ZXHqPg4RkhPn0PqToeuSRO4/no10DR7NuzbBzt3wmefwWOPQbduUNU6jc2bYdIkePhhuOoq9Xg/ORkeeACys1VOVHeza53eoTYqHVUo3IuMRpsYPfMM3H23+iMfPNhVfuSIFveiQOHpY/qRqBHP/5NS/qeyE6SUR4CnhBBHgRfs17gxq9fC3xw5ckRr+0Zjhv+6a6C7xqBPjIQQtGrVilatWnHo0CFyc3NZu3YtJ06cAKC0tNS5u9O5557rlzrB2Pg4OqRffvklBw7093pkUQi48EJ13HyzKisrg02b1Kjphx/+xNGjl7FhA5SWVrz21CnIy1PH+PGqrF491UF1zD3t1AkuuKBi6seqRnaN2EEqFO5FRqNNjBIT4b331B9Taio0bqz2yt2zh9o//ui7/WrQWWNPO6NJQBnwhhvnvgH8E3BriNbCwsLCQtGwYUN69epF9+7defvtt6lbty579+4lPDycK6644ozzzUh74y1hYXDhhc/z2mv9/TrnMjwc2rVTR2npd4wceRnHj8O6dZCb63q8v3XrmdcePQqLF6vDwTnnuDqnyclqdPWTTyqv24gOqUUI4fhWExEB/fs7/2AuWrMmgE4FNx7NGRVCnACOSSkbu3n+ASBaSunb5swWVp5RA9Aib12I2w/lPKP+tB8VFcXu3bvZt28fyckVv9//+eefvPvuu1xyySWkpKTQvHlzt5Lpl7dvdDsIZFs+dOjM+af+mungrzmkoXAvsu7XVTB7NvTrB4Dt4osJ27rVsN2Xgk1jI/OM/gk0EELEu+HE+UADVLoniwCQm5urtX2jMcN/3TXQXWPQP0a5ubkIIbjgggvO6Ig63pdSsmnTJj766CPeeecdVq1axcmTJ922rzvVfYaGDVVC/X/8A2bOVDlNf/sNpk2DJ5+E7t1VXlNv8Ncc0lC4FxmNtjG69lo1LwQI275dLWIyCJ019rQzutT+83VR81fv1+0/F3tYh4Wf2LJli9b2jcYM/3XXQHeNQf8YVWdfSklRUVGFsoKCAr766itef/115s6dW2FLUk/t64Inn0EISEhQ24i/9BJ8+63KV/7TT6pj6Sn+yGEeCvcio9E2RnXqwF/+4nrtTmJ8L9FZY0/njL4KDAVuBs6z5xJdKqUsBhBCNAa6o3KQdgRswGv+c/fsQwjRD+iXkJBAZmYmAJ06dSIuLo45c+YA0KxZM3r16kWWPblfZGQk6enpHD582HnNoEGD2LZtG+vXrwegS5cuxMTEMG/ePABatGhBWloa2dnZAERFRTF8+HBycnI4ePAgAEOGDCE/P5+N9m92J06cYPv27Sy0J/5r3bo1KSkpzl0gYmNjGTp0KFOmTHFOrB42bBi5ubnORtOzZ09KS0tZsmQJAG3btqVdu3ZMnTqV1NRUcnJyGDx4MJMmTaK4uBiA9PR0li5dyg57MsLevXtTVFTEihUrAEhMTKRly5ZMmzYNgKZNmzJw4ECys7MpKSkBICMjgyNHjjjj06dPH/bv388qe5btpKQkEhISmDlzJgDx8fH07duXrKwsbDYbYWFhZGRkMHv2bPbs2QPAgAED2L17N3l5eU6dSkpKnHVUpdOMGTP4888/vdLJoU11OnXr1o2IiAivdCooKCAzM7NanQAaNWrklU7R0dHO+FSl0/z589m1a5fXOgE16uROe6pKp4KCAvLz831uT1Xp5NCgKp3OPfdcDhw4QHFxMSUlJZTaV/CUlJSQm5tLbm4ukZGRREdHc95553HzzTdX0Mlms7FgwQKf21N1OjVr1ox9+/b53J6q0qmgoIDs7Gyf21NqahRlZcPdStjv4Prr/+Cmm2LJzPT+vnf06FEKCwt9bk/V6VRQUIDNZvO5PVWlU2pqKpmZmT63p6p0stlsznuFL+2puvteWVkZixcvduv/kyc6hSUnc8FnnwFwbNIkTjzwgE/tqSqdHPcKX9tTVTpFR0ezcuVKt9uTJ3iTZ3Qk8A4qzyio1fWFQG2gruM0VEf0ASllpsdeWZyBN3NGt2/fzsUXX2yQR8bbN3oOktH+m1GH7vbNmDOqe4w8sX/y5EnWr1/v/CJxOiNHjuS8887z2r636NSWq8uPWhm33QavvOLbblChcC/SSWPT6zhyBOLi1NZioHZ1uOgiv1cTbBobuje9vXOZhuvxexjQEIjC1UH9Fki1OqKBpfT0HCea2TcaM/zXXQPdNQb9Y+SJ/dq1a9OpUyceeOAB7rjjDi699FLnYqaEhIQzOqI2m41Tp0751d9A4E8NqsqPWhWffKIS7f/nP2o3KW8IhXuR0Wgdo9hYNXHZgUGP6nXW2Ks1gFLKFVLKnkAT4FrgVvtxLdBESnmtlPJ7/7lp4Q2ORwu62jcaM/zXXQPdNQb9Y+SNfSEEF110EbfccgujR48mNTWVLl26nHHeypUrmTZtGmvXrtW6U+pvDWrqkN58s5pz6uDYMXjqKWjbFr78EjzdBDsU7kVGo32Mbiy3/09N+9d7ic4aezRnVAgxHfVY/jEp5S9SykOoUVALCwsLiyCkfv369OjR44xyKSW5ubmUlpby5Zdf8s0339ChQwdSUlJo2LBhADwNLhwdUqj4yL58Oqdvv4XRoyE/X723YwcMGAC9esH//geXXmq62xbBSv/+2IQgTEpYsULlF/Pj5hW64+nIaF/geinlL0Y4Y+Ff2rZtq7V9ozHDf9010F1j0D9GRtnfv39/hZX4x48f5/vvv+fNN9/kk08+YevWrXi6piBQGBUjR4f0L3/ZB5yZV7RHD1i7Ft56S6WQcjB/PrRvD2PGqJX6NREK9yKj0T5GcXEUJSaq36VUecb8jM4ae9oZ3Qvo+yznLKNdu3Za2zcaM/zXXQPdNQb9Y2SU/aZNmzJmzBhSU1Np0KBBhfe2bt3KJ598wrhx4/j+++85fvy4IT74CyM1CAuDSZPqkJ1deYL7iAh48EHYsgXuv9/1fmmpGh1t3Rref7/6XKShcC8ymlCIUeTQoa4XBswb1VljTzuji4AYIYT18EEDHOkndLVvNGb4r7sGumsM+sfISPtRUVFs27aNhx56iFtvvZWWLVtWeP/gwYPMnz+f9957L6hHSY3W4PPPp3LnndXvtNSkCbzzDqxZA2lprvL9+9Xq/E6d1NPZygiFe5HRhEKMppdvQwsXujds7gE6a+xpZ/Ql4DgwTghR2wB/LCwsLCxMJiwsjNatWzNs2DAeeughrrrqKurUce3i3L59e4+2GD2bSUxUe95PnQoXXOAqz8uDrl1h+HD4/feAuWcRQI41agSOXdRKS8Ge49PC887oMeA+IAXIF0KMFkJ0EkJcJIRoVtXhf7ct3KFRo0Za2zcaM/zXXQPdNQb9Y2S2/UaNGtG7d2/GjBlDv379iI+PJykp6YzrZsyYwaJFi5wJwwNJsGkgBNxyC/z8MzzzjNqEx8HkyWemggqFe5HRhEKMGjVqVDENg59X1eussUdJ74UQZV7UIaWUnu70ZHEa3iS91x0zEqJbBBZLYz05cOAA48aNA1QaqUsvvZSUlBQuvPDCSkdQz3add+6Exx+HnJyK5S1awNix0K+f6sDqzNmusdv8/LMrzULdulBQAFFRgfXJTYIp6b3w4vAql6mF70yaNElr+0Zjhv+6a6C7xqB/jILRvmM7QFApon766Sc+/PBDxo8fz+rVq53bGppFMMaoPM2bw+efq1RQ5deYOFJBJSbuYdMm33ysCd3bcsjcry+5xNUZPX5cpV7wp31N8aijKKUM8+YwynmL6nHslaurfaMxw3/dNdBdY9A/RsFov1u3btx8881ceOGFFcr379/PnDlzeP311/nqq68q3ZLUCIIxRpXRvbtKBTVuXMVUUD/+GO9RKihv0L0th9T92qAE+DprbHUULSwsLCw8Ijw8nMsuu4z09HTuv/9+kpOTqVWrlvP9kydPsmrVKt5++22WLVsWQE+Dj4gIGDWq+lRQWVlQ5s2kOAs9KD9vdNYs0Hj3M3/h0ZxRi8DhzZzRkpISIiMjDfLIePtGz0Ey2n8z6tDdvhnzzHSPkS72T5w4wfr168nNzeXAgQPO8rvuuosbb7xR67ZspP316+Ghh2wsW1ZxbKhjR3jzTbUC3x/o3pZD6n4tpZq7sWuXemP+/Ip71/tq3yCCac6ohUYsXbpUa/tGY4b/umugu8agf4x0sV+nTh2uvPJKRo0axe23306bNm2Ij48nISGhwnklJSV8+eWX/O7H/Ea6xKgyEhPh6ae/PSMV1Jo1cPXV/ksFpXtbDqn7tRAVH9X7KQG+zhp71RkVQlwvhMgSQvwghNgshNhRzbHd305buMeOHTu0tm80Zvivuwa6awz6x0g3+0IIWrRowdChQ7nrrrvOWF2/fv161q5dS1ZWFu+99x7r1q2jtLTUpzp1i9Hp/PLLDo9SQXmD7m055O7X5R/Vf/FF9Vt0eWNfMzzqjAohagkhpgNzgBFAJ6AV0LyGw8LCwsLiLCM8PLzCayklubm5ztd79uxh5syZvP766yxYsIDDRq3e0YSoKPjnP2HTJhg82FV+7Bg89RS0bQtffqme8lpoTteuEBenft+7F374IbD+BBhP838+CQwEJKpDOgP4HfDh+5pFdQgh+gH9EhISyMzMBKBTp07ExcUxx757Q7NmzejVqxdZWVkAREZGkp6eTlRUlPOaQYMGsW3bNmdKli5duhATE8O8efMAaNGiBWlpaWRnZwNqm8Dhw4eTk5PDwYMHARgyZAj5+fls3LgRgEsvvZTt27ezcOFCAFq3bk1KSgqTJ08GIDY2lqFDhzJlyhRnYuxhw4aRm5vLli1bAOjZsyelpaUsWbIEgLZt29KuXTumTp1KamoqOTk5DB48mEmTJjlXCqanp7N06VLnt8DevXtTVFTECvtee4mJibRs2ZJp06YBag/ugQMHkp2d7Uw5k5GRQcOGDZ3x6dOnD/v372fVqlUAJCUlkZCQwMyZMwGIj4+nb9++ZGVlYbPZCAsLIyMjg9mzZ7Nnzx4ABgwYwO7du8nLy3Pq1LFjR2cdVek0Y8YM/vzzT690cow6VadTt27diIiI8EqnkpISMjMzq9UJVLJlb3SKjo52xqcqnebPn88u+9wqb3Tq3bt3jTq5056q0qmkpIT8/Hyf21NVOjk08LU9VaVTjx49WLBggc/tqTqdmjVrxr59+5g5cyY2m43GjRtz+PBhyuyrdI4fP87y5ctZvnw5tWvXJjo6mptvvpnff//dLZ1KSkrIzs72uT1VpVN8fLzz79SX9lSVTvHx8RQWFlbQ6fPPB/PUU9/w4YdJ7Nmjkpk7UkFddtlvvPlmGOecc8htnUpKSrDZbD63p6rue6mpqWRmZvrcnqrSqUePHk4NfGlP1enUrVs3Fi9e7FN7qu6+V1JSwsqVK506pbZpw6X79wOQ//zzrLA/uvf2vnfq1CkyMzPd/v/kqU7R0dGsXLnS7fbkCZ4mvd8EtAb+LqV8xePaLLzGmwVM+fn5tCuf1M7PGG3f6AnxRvtvRh262zdjAZPuMdLdPlSuc3FxMWvWrGH16tUUFhaecU2jRo0YOnQocY7Ro2rQPUbV2S8thQkT4Omn4dAhV3lEBDz4IDz7LDRo4Fsd/sC6X3thf+5c6NNH/d6iBWzb5tPuB8GmsZELmJoDNuAtD6+zCACOb8y62jcaM/zXXQPdNQb9Y6S7/aqIiori6quv5uGHH2bo0KFcfPHFFd4/fvw4Dcsn46wG3WNUnX1HKqitW31LBaV7Ww7J+3XPnhATo37fsQN+/NG/9jXC087oYaBISnncAF8sLCwsLM4ywsLCaNOmDcOHD+fBBx/kyiuvpHbt2nTs2JGIiIozyX7//Xc2btzofLx/NtG4Mbzzjlpl362bq3z/frjnHujUCZYvD5x/Fl5Quzb07et67ee96nXC087oEqC+EOKCGs+0CDiJiYla2zcaM/zXXQPdNQb9Y6S7fU9o3Lgx119/PY8++ihdK0mw+d1335GTk8Mbb7zB4sWLKSoqAvSPkSf2ExNh0SI8TgUVTDp7Q8jer/2Y4klnjT3tjL6AWqz0sgG+WPiZli1bam3faMzwX3cNdNcY9I+R7va9ITIykrp161YoKywsZPPmzQAUFRWxZMkS/ve//5GTk0NUVBRGbuASbBoIgTMV1LPPVp4K6t//rpgKKhh19oSQvV/fcIMaIQXYsEHNG/WnfU3wdG/6fNRq+uuFEF8JIa4RQkQb4pmFzzhWVepq32jM8F93DXTXGPSPke72/UWtWrVIS0ujXr16zjKbzcbGjRuZPn06EyZMIC8vz7nC358EqwZRUfDcc6pTenoqqH/8Q6WCmjlTpYLSReeqCNn7db160KuX67UPo6M6a1xlZ1QIUVbZAXwN1Ad6AQuBI1Wdaz98y2ZsYWFhYXHWExUVxTXXXMMjjzzCoEGDaNasWYX39+3bx+zZs3n99deZP3++oSOlwcaFF8Lnn8O330L5xdQ7dsDAgdC7N/zxR4NAuWdRE6cnwD8LqS7PqPf5BYyxY+EhTZs21dq+0Zjhv+4a6K4x6B8j3e37m/DwcNq1a0e7du3Yu3cvubm5rFu3Dpt9B5uTJ09y+PDhM3Z+8gVdNOjeHdauPTMV1DffwLffDubwYfdTQQUbIX2/7tcPwsNVSoTvv4c9eyA+3n/2NaDKPKNCiAv9VYmU8ld/2Tpb8SbPqO6YkYPSIrBYGp8dGK3z8ePHWbduHatXr+bgwYPccccdXHTRRRXO2blzJ02bNiUqKsowP4KJAwdUh3TChIo7TcbFqfmkI0ao/o+/sNqyj/TsqYa2QaVNuP/+wPpTCQHJMyql/NVfh9ueW/gVb3ZBCCb7RmOG/7proLvGoH+MdLdvBlOnTqVz5848+OCD3HHHHTRv3rzC+6dOneKzzz5j7NixzJw507krjbvoqEGopYIK+ft1+Uf1XqZ40rktV7uASQjxrRDic7OcsfAvRkzkN9O+0Zjhv+4a6K4x6B8j3e2bgeMzCCG46KKLznhEn5+fz/HjxyktLWXdunW89957ZGVlsWHDBkpLa17WoLMGjlRQ99yzwKNUUMFGyN+vBw50/b54Mdi3OfWb/SCnptX01wBnJnuzsLCwsLDQhOjoaM4777wKZb///jtffPEFY8eOZeHChZVuSerYsrSoqIg1a9Y49x/XDSEgOXmHR6mgLEzm/PPhyivV76WlMHt2YP0xmWr3phdC2IC9UkrPZ9Ja+BVv5ozabDbCwjxNJRs89o2eg2S0/2bUobt9M+aZ6R4j3e1DcLRlKSW///47ubm5le7iJISgTZs2dO3alfPPP59ly5axbNmyCiOnERERpKamkpqa6tcFUmbfi379FR57DHJyKp7TogW8/jr07+/5FunBoHGw11Gj/Zdfhr/9Tf0+YADMmOFf+z4STHvTW2jE/PnztbZvNGb4r7sGumsM+sdId/tm4M5nEEKQkJDAjTfeyJgxY+jRowexsbHO96WU/Pzzz+zdu5dly5axaNGiMx7hl5aWsmjRIpYtW2a6//6so3wqqMsvd51TPhXUpk2Gu+QRZ8X9uvxuTPPmqYSx/rQfxFid0RBm165dWts3GjP8110D3TUG/WOku30z8PQzREdHk5qayujRo7nlllucK+9r165Ny5Yta+xsLlu2zK+P7AN1L+reXc0dHTcOGjZ0lX/zDbRvD2PGwOHDhrvmFmfF/bp1a1ei2BMn4Ouv/Ws/iLE6oxYWFhYWZyVhYWFceuml3HHHHTzwwAP079+fHTt21LioqbS0lJ9//tkkL40lIgJGjYKtW+GBB8DxlLe0FP73P9U/eu89lQLTwgT8uFe9TlSX9N5BfSHEBz7UIaWUd/twvYWX9OnTR2v7RmOG/7proLvGoH+MdLdvBv74DHFxccTFxbFkyRK3zi8qKvK5TgfBcC9q3BjefhvuvRcefhgcYdi/H0aOhHffhTffhK4BWtIcDDEyxf5NN8Hzz6vfZ8+GkhKIjPSf/SDFnc5oHeBOL+0LQAJWZ9RLhBD9gH4JCQlkZmYC0KlTJ+Li4pgzZw4AzZo1o1evXmRlZQEQGRlJeno68+bNc37DHzRoENu2bWP9+vUAdOnShZiYGObNmwdAixYtSEtLc+Ypi4qKYvjw4eTk5HDQnmJiyJAh5Ofns3HjRgAuuOACTpw4wcKFCwFo3bo1KSkpTJ48GYDY2FiGDh3KlClTOHLkCADDhg0jNzeXLVu2ANCzZ09KS0ud/wDatm1Lu3btmDp1KqmpqeTk5DB48GAmTZrkfCyWnp7O0qVL2bFjBwC9e/emqKiIFStWAJCYmEjLli2d+/Q2bdqUgQMHkp2d7Ux9kZGRwbfffsvx48cB1Yj379/PqlWrAEhKSiIhIYGZM2cCEB8fT9++fcnKynJOEs/IyGD27NnOnIUDBgxg9+7d5OXlOXU6dOhQjTrNmDGDP//80yudysrKuP/++6vVqVu3bkRERHil07p164iKiqpWJ4BGjRp5pVN0dLTz77oqnebPn+98/OSNTsnJyaxdu7ZandxpT1XpVFxczLXXXutze6pKp88//5yoqCif21NVOnXs2JFNmzb53J6q06lZs2bs27fP5/ZUlU7FxcU0aNDA5/YUFRXFZZddhjt89913/Pjjj7Rr187n+17Dhg3p1auXz+2pOp2Ki4t5+OGH3WpPt946kzZtLmL69KspKKgLuFJB3XYb9Oq1gJMnd1TQKTU1lczMTJ/bU1U6NWjQwGnTl/ZUnU6JiYls3brVp/ZUnU7FxcV07ty5+vYkJfc0b47YuRMKC5n7+OMkPvGEW/e9nJwc6tat63N7qkqn6OhoVq5c6XZ78gR3VtOXAN97bLkcUsruvlxv4d1q+szMTEaOHGmQR8bbN3p1ptH+m1GH7vbNWE2ve4x0tw96teXi4mLGjh3rVv5RgFatWpGWlkZCQoLXdQbrvai4GF55RS3yLp/2KToannoKHn3UlSZKJ40DVYfb9v/6V5XWANSw9IQJ/rXvJYFeTX9QStndl8Ntzy0sLCwsLAJIVFQUqampbp+/detW3n//fSZNmsSpU6cM9Mx8oqLguefg559h8GBX+bFj8I9/QNu2MHMmVDOmZeEN5eeNzphxVkzYtRYwhTBJSUla2zcaM/zXXQPdNQb9Y6S7fTPw92dITU2le/fuRERUnMkWERFB9+7due+++2jnWPVcjlq1anlVX7Dfi9xJBXX8eHOffayOYI+RX+137gxNm6rf//wTvrc/nK4htYHObdnqjIYwvjw2Cgb7RmOG/7proLvGoH+MdLdvBv7+DEII0tLSGDNmDP369SMlJYV+/foxZswY0tLSOOeccxg0aBCjRo2iffv2CCHoVn4DeDtFRUVUNxXOKP+NqqO6VFA//TSlxlRQNht8+KH66Sm6xMgv9oWoGOD//hcGDYJGjdRP+5xNr+0HIVZnNIRxTG7W1b7RmOG/7hrorjHoHyPd7ZuBUZ8hKiqKjh07snv3bjp27EhUVFSF95s0acKNN97I6NGjuaD8xu+o9E/vvfce2dnZbN++vdpOqU73oqpSQUFEtamgbDbIyID0dPXT0w6pTjHyyb4jUJs3u8q+/BKmT1fzIaZPhw4dKu2U6tyWrc6ohYWFhYWFD9SvX/+MsrVr11JUVMSuXbuYNGkSH3zwAVu3bnVrpFQHHKmg1q6F8oPCjlRQnTrB8uWqzNG/mjhRvZ440bsO6VnB1q2uQFXH9Onw1lshM2HX6oyGMPHx8VrbNxoz/NddA901Bv1jpLt9MwjGGB07dqzCPuG7d+/mk08+ISsriy1btlTolOp8L2rfHhYtghYtnqRZM1d5+VRQt912Zv/K0w6pzjHyyH6bNirXaE2MGAGZmeqRvif2g5RqUztZBA/epHbSHTPS/lgEFkvjs4OzVefCwkK+++471q5dS9lpz63PO+880tLSaNOmDaJch0JXkpOTWbp0daWpoKpjxAjIyir/uN+CWbOgf/+q37/4YpXiIMKdVPH+I9CpnSw0xZG8Vlf7RmOG/7proLvGoH+MdLdvBsEao/r169OnTx8efvhhOnXqRHh4uPO9P/74g6lTpzJhwgTGjx/vL1erxAydq0oFVR3ujpCeVffrmpLGb98OQ4eeETSd27LVGQ1hbAZPyDHavtGY4b/uGuiuMegfI93tm0Gwxyg2NpYbbriB0aNHc9VVV1VIGXX48GFTRkbN1PmCCyAmxv3z3emQnjX368OH3duTfto01eM/edIz+0GK1RkNYcIMfu5htH2jMcN/3TXQXWPQP0a62zcDXWIUExND7969GT16NJ07d6ZWrVpceeWVZ+QzPXHihN87Fmbq/PHH7q3BKc/Eieq6qjhr7tcNGlRMel8dX3yhJu1++6379oMUa86oJlhzRi1CEUvjswNL58o5duwY4eHh1HHsqWln+vTp7Nmzh9TUVC6//HItOhnlNT599bw7DB0Kkydbc0cBlbKpQwfX64svVo/mHTRuDAcOVLxm+HB49VU45xzD3Drr5owKIQYLId4SQiwTQhwRQkghxKQarukihJgrhDgohDguhNgghHhECBFezTV9hRCLhRCFQoijQoiVQog7a6jnTiHEKvv5hfbr+1ZzfrgQYozdn+N2/+YKIbrUHAnfmD17ttb2jcYM/3XXQHeNQf8Y6W7fDHSNUXR0NHXq1Klg/8CBA+Tn53PgwAFmzJjBuHHjKl0A5Slm6hwWphYljRjh/jWzZsH/t3fu8VFV1+L/rhBegfAGhVIqyFMQFAK21CRELFTRiuIDC2iioLa2Pnvv9Xdrrb23t/f2akGu7bXGWKLAFRUURLSgFJIALU9fvF/1QXlIxIRACJDM/v2xzySTITOZycw5yR729/M5n5M5Z++111lr9sma/fzNb/RWo3VxXr2vhw2rmVGfk6MH4fqNmZMDhw7pnQfatavJM28eZy6+WO9jb2B3fZMMRoHHgZ8AlwH/qC+xiNwAFAIZwJvA74EWwCxgQYg8PwGWAkOAecALQA8gX0SeDpHnaSAf6O6knwdcCix15AWnF6f8mY4+v3f0ywAKHb1d4+DBg26Kd12+23ihv+k+MN3HYL6NTJfvBabbKFD+0aNHadGiRfXnr7/+mrfeeovf//73bNmypcFBqdd+jjYgPXkSfvEL6NdPr1hUWVn7/nn3vn7iCbj7bm3E5GR9zs/X5+bN9c4DO3fqJmWHFidPwn33wXe/G3KXpqZKUw1GHwb6A+2AH4VLKCLt0IFhFTBGKXW3Uuqf0IHsX4GbRWRyUJ6LgKeBY0CaUup+pdTDwFBgH/CoiHwnKM9o4FHn/lCl1MNKqfuBEY6cpx25gUwGbgbWAZcppf5JKXU3kOXo+4KIRDHM22KxWCyJzMCBA3nooYcYM2ZMre77kpISli5dyrPPPsumTZuoDI7WmiD1BaTZ2bBkCVxySc21Q4fg3nvh0kv1vfN2JOGwYXorK/+4haQkuPPO2uMYuneHV16BP/9Zd+X7+dvfYMQI+NnP4MQJb/VuKEqpJn0AYwAFzAtx/y7n/kt13LvKuVcQdP3fnOu/ilQe8LJzPaeOPHXKQ7fWKiCrjjwh5dV1jBgxQkXL4cOHo87TlOQ35JmjwW39vSjDdPlu+1gp821kunylzK/LjSW/oqJCFRYWqt/+9rfqySefrHXMnDlT7dixI+Yy4kU4H1dVKZWTo5QOLfWRk6OvK6XU2bNK5eUp1aNH7TSg1He/q9S6dfZ9XS/l5ars4YeVat68tgG/+U2lFi+OSxHR1mNgk4ow1muqLaPRcJVz/nMd9wqBcmC0iLSMMM+7QWkalEdEWgGjnfKLoignbhw4cMAt0Z7Idxsv9DfdB6b7GMy3kenyvcB0G4WS37JlS9LT03nwwQcZO3YsKSkp1feOHz9O8+bNYy7DC4JbSIMXuk9O1j3Se/bocaOBQyHXroXRo+H221uwe3f4cnw+eOmlhg+ZNPp71Lo1u6ZMgY8/hjFjaq5/8QVMnAg33ACffaav+Xywa1ft/CUl7ukWAU1+Nr2IjAFWAfOVUlPruL8RSEN3t2+u4/5WYDBwiVJqh3PtKNAF6KKU+qqOPCeANkAbpVS5iLQBTgAnlFLndKuLSBfgKPClUuoC59pgYCuwVSl1aR150oCNwAal1BUhnv0e4B6AVq1ajRg8eHBdyUJy9OhRunbtGlWepiR/x44dDBo0yDX5buvvRRmmy3fbx2C+jUyXD+bX5aYiv1mzZnzrW9+ib9++nDx5krX+zd8dRISkpKQ6x5U2hbqslHDs2LV06vQOIqFjj8rK9hw6dDdHj96CUoEBdyVdu75J9+4v0Lz5sXNkf/bZL/jqqx/QufNbfOtb/x62jLpoKn6OWb5SXHvsGA8dOECngOEcp5KSeP7CC7n49Gmu/+or/tKhA0s7d+b6r75iTEkJqzt04IXu3dkT8KMnkGjr8ebNmyOeTd/o3fD1HdTfTb/bud83xP21zv3vBFw741xLDpHnH8797s7nHs7nAyHSN3funw64Ntq5tiZEnn7O/V2R2KEh3VzPP/981Hmakny3u/bc1t+LMkyX70U3vek2Ml2+UubX5aYm/8yZM+rYsWPnXP/www/VU089pdasWaNOnz4dUxnR4oaP9+1TavLkc7vu27RR6pe/VKqsTKerbxhApDQ1P8cs/6uvlJox41wD1nfcdJNSH354jnzbTW9pEKNGjTJavtt4ob/pPjDdx2C+jUyX7wWm2yha+c2bN6djx461rvl8PoqKijh58iTvv/8+s2fPZs2aNZx2dugx0c99+uj5ORs3wsiRZdXXT56EX/0K+vaFP/wB7rrr3DVNI91mNJCm5ueY5XfqpJcmWLtWzwiLlDfegGef9XT2WCIEo6XOuX2I+/7rJQ3IUxp0dqOMkhD3Y8btrje35buNF/qb7gPTfQzm28h0+V5guo3iIb+kpISzZ89Wfy4vL2flypXMnj2bwsJC2rcP9a+o6ZOWBm+8cZx33qkdUx05Aj/5iR4nWhfRBqQm+LlB8kePhs2b4amnoFnIpddryMnRQawH29T6SYRg1D8Kt3/wDRFJBnoDlcD+CPN0R48XPaCUKgdQSp1Ed923de4H0885Bw6v3odevqmPo0ckeeLKsmXL3BLtiXy38UJ/031guo/BfBuZLt8LTLdRPOR36tSJBx54gOuuu65W4Hnq1ClWrVrF3LlzWb16NadOnYq5rMbgnXeWcc018MEHOsj8xjciyxdNQGqCnxssv3lzvdRTbm54IRdfrNN4vBVWIgSjf3HO36/jXgaQAqxTSp2OMM81QWkalEcpVYFeXzQFSI+iHIvFYrFYoqZZs2aMGDGCn/70p/zgBz+o1ZWvlKKgoIDZs2dTVFTXAi9m0KwZ3HEHXBXFOjQN6bJPWOoLiPftg3vu8dxYiRCMLgSKgcnODHWgemmlXzsfnwvKMwc4DfwkcKF6EekI/Kvz8Y9Befyff+6k8+e5CLjfkRe8E6+/3F87+vjzjARuQ8/AX1TvEzaQXr16uSXaE/lu44X+pvvAdB+D+TYyXb4XmG6jeMtv1qwZl19+Offffz833HADnTp1qr53+vRpI1tHA200d64+omHOnPrzmObnqOWXlMCbb9YvKBJjxZkmubSTiEwEJjofLwTGo7vZ/T/nipVSPwtKvxCoQG+/eQz4ATDAuX6rCnpQEfkp8D/AV8Cr6Bn2NwM9gd8Fyg/I8zvgEeCAI7cFOqjsDPxUKfX7oPQCvObI3YnefrSzk6cVMEkptSQSm6SlpalNmzZFkrQan89HkotN7W7LT0tLI9pnjga39feiDNPlu+1jMN9GpssH8+tyIsj/5JNPWLNmDSUlJTz44IO0bdu2VpqzZ89GtW5pMF762OfTLZ3Bk5bCkZ0NL74Yvvc5Efxcr/xJk/QEpXAELwTrEK2PRSTipZ2aasvoZcCdzjHeudYn4NrNgYmVUouBTPQi95OAnwJn0YHj5OBA1MnzLDpg3QbcgV7P8zCQXVcg6uR5FMhx0t3j5NsGXB8ciDrpFXC7o0elo9dNjp4ZkQaiDSUvL89N8a7Ldxsv9DfdB6b7GMy3kenyvcB0G7ktPykpifXr1/OjH/2Iu+6665xAtLS0lJkzZ7J8+XJONNHtIwNtFO2+9wB799a/Xbvpfo5I/hNPhL8fIhB1m7om1jQ6SqkngSejzLMWuDbKPEvRrZXR5MkH8qNIXwnMcg6LxWKxWBqFpKQkunc/dw7u2rVrqaio4G9/+xubNm1i+PDhXHnllaSmnrPHS5PBH5BCZC2ka9bo7dpnzIBf/xoSYBGJhjFsGNx0k24dveQS2L695l4jBaLQdFtGLXGgRYsWRst3Gy/0N90HpvsYzLeR6fK9wHQbNea7yOfzcejQoerPlZWVbNiwgdmzZ/POO+9QWlpaZz6vqUv/+lpIp0yBRx7R242CXjYzNxf69YNnnoGAlbBClhFPmoz8J57Q+6/+9rc113r2bLRAFJromFHLuTRkzKjpeDGe0NK4WB+fH1g/N22UUuzevZuCgoJagSnoyVCXXXYZV155JR06dAgpozF9XNcY0sBGvl27dFD6zju18w0cqIPS8eM5/1AKysuhY8eaqPzQIbjwwpBZzscxo5Y4sHjxYqPlu40X+pvuA9N9DObbyHT5XmC6jRr7XSQiDBgwgBkzZvDDH/6QbwQs4llVVcXmzZt59tlneeuttzhz5ozrutZFOP2DW0iDe5sHDNArGi1bBv0DVhffuRO+/324/nrYs8d8P0clXwTatNEL4vt5//246xQpNhhNYL788kuj5buNF/qb7gPTfQzm28h0+V5guo2ayrtIROjXrx933303U6ZMoWfPntX3fD4f//jHP2KacR8L9envD0jz80P3Nl97LXzyCTz9NAQOh337bRg8GJ57rjfHj8dX70Ca5Pdo3Liav997L37KRIkNRi0Wi8VisVQjIvTt25e77rqLadOmVa9fmZGRgQRtEVlVVdUYKtZJUhLceWf4YY8tWsCjj+qW0Lvuqtnx8uxZWLFiGP376+7+82aB/O99r+bv997zdD/6Wiil7GHAMWLECBUtxcXFUedpSvIb8szR4Lb+XpRhuny3fayU+TYyXb5S5tdl0+XHWobP51OfffaZ8vl851x//vnn1aJFi1RmZmaMGobHLRtt3KjU6NFK6Sis5hg5Uqm//jW+ZTXJ71FlpVIdO9Y8+CefhEwabT0GNqkIYxzbMprA7N2712j5buOF/qb7wHQfg/k2Ml2+F5huo6b+LhIRevXqdU6r6O7duzl06BCffPIJY8aMYeHCha51Rbtlo7Q0vezT/PnQtWvNeNiNG+E734Fp0+DgwfiU1SS/R82awdixNZ8bqaveBqMJzEf1rfDbxOW7jRf6m+4D030M5tvIdPleYLqNTH0Xffrpp9V/iwjbtm3jueee4/XXX+fIkSNxLctNG4nAD38Ijz8+j5//HFq2rLk3b56e9PSb30BFRWzlNNnvUXBXfSPQJBe9t9QgItcD1/fs2ZPc3FwARo0aRdeuXVm2bBmg96MdN25c9e4LLVq0IDs7m5KSkuo8kyZNYu/evdVf1tGjR5Oamsry5csB6NOnDxkZGeTn5wOQkpLC1KlTWbhwIceOHQPgtttuY+vWrWzbtg2AiooK9u3bx8qVKwHo378/I0eOZP78+QC0a9eOyZMns2DBAo47o8KnTJnCxo0b2b17NwBjx46lsrKSgoICAAYPHsyQIUN49dVXSU9PZ+HChdx8883MmzeP8vJyALKzsyksLGT//v0AjB8/nrKyMtatWwfAsGHD6Nu3L4sWLQKgW7duTJw4kfz8/OqZoNOnT+f48ePV9pkwYQJHjx5lw4YNAIwYMYKePXuyZIneJKtHjx5cd9115OXlVW+5Nn36dN5++20OOj+bb7jhBg4cOMDmzZur/XTmzJnqMkL5afHixdWtCdH6ye+bcH7KzMwkOTm5QX4qLi4mNzc3rJ8AOnXq1CA/tWnTpto+ofy0YsUKPv/88wb7CajXT5HUp1B+Ki4uZuvWrTHXp1B+8vsg1voUyk8+n4/3338/5voUzk+9evXiyJEjMdenUH4qLi4mPz8/5voUyk8nTpyo/p7GUp9C+enEiROUlpbGXJ/C+am4uBifzxdzfQr204033sjatWtrtYhu376d7du307JlS/r168eNN95YLePs2bMMHTqUvXv3cvbsWZKSkiLyk8/nq/ZBLPUpnJ+aN6/i6qtXk5JyiIULv80HH/QG4ORJ+PnP4Q9/OMV//7ePEyfmIxK9n4qLi1m/fn3M9SmUn/zvimjrU2pxMbc7vju7ciUv/eEPJLdpc059atOmDevXr4+4PkWDXWfUEBqyzujWrVsZMmSISxq5L9/tdevc1t+LMkyX78XahKbbyHT5YH5dNl2+F2WMHTuWGTNmsGvXrnPu9evXj06dOrF582YqKyurrycnJ5Oenk56evo5QwCCaQwb/eUv8OCDsHVr7XRjx8Ls2XoGfizy401M8vv103umgn7wrKxzkth1Ri0Nwu2t3JryVnGR4IX+pvvAdB+D+TYyXb4XmG6jRHgXlZaWMnnyZO655x4GDRpU697evXtZv359rUAU9G5Pq1atoqioqF75kehfXl7Oli1bKCgoYMuWLdWtlZESXMZVV8EHH8Dvf6/XhvezcqXeVfOnPwWnYbZB8uNNTPIbuaveBqMJjL/p3FT5buOF/qb7wHQfg/k2Ml2+F5huo0R4F/np3r07t956K/fddx+XXHJJRHkKCgrYtGkT+/bt49ChQ5SWlnI2aK/OcPorpSgsLGTWrFksXbqU1atXs3TpUmbNmkVhYSGR9gDXVUZyMtx/v14K6v77a5aNqqrSQWr//vDccxAUZ0csP57EJL+Rg1E7ZtRisVgsFktcueCCC7jllltYvXp19djYUPh8vuqxwIE0b96clJQUrrnmmnPubdu2DREhJSWFHTt2VI+nDMTf8gp6jdRY6NxZB5/33gsPPaR7sgG++gp+/GP44x911/2YMTEV03hkZelI2+eDzZv1g3Xu7FnxtmU0genTp4/R8t3GC/1N94HpPgbzbWS6fC8w3UaJ8C4KRX1jQcNx9uxZSktLEZFz9F++fDmvv/46L730Up2BaCBFRUURddlHYqNLL9W7Zi5aBBddVHP94491PHfLLRCwyEDU8mMhJvkdOsCoUfpvpWqibY+wwWgCE+svwcaW7zZe6G+6D0z3MZhvI9Ple4HpNkqEd1EoIh3H2KNHD3r37s0FF1xAampq9UoYoGfPB+qvlIpqPGhlZSU7d+6sN12kNhKBm26CHTvg17+GlJSaewsXwqBB8MQTehZ+Q+Q3lJjlB24NumJFbLKixAajCUxDlldoSvLdxgv9TfeB6T4G821kunwvMN1GifAuCsXAgQNJTg4/IjA5OZmpU6dyxx13cN999/HII4/w+OOP89hjj/HAAw9w4YUX1tLf5/MxbNgwBg4cSPv27SPSo6ysrN400dqoVSu95NPu3TBlSs31igr493+HgQPhlVdqdths8t+jRtwa1AajFovFYrFYXCElJYX09PSwadLT02ndunWtayJCy5Yt6dix4znBbLNmzbj++uu57bbbIm4NdHMm+ze+oRfHX7sWRoyouX7ggF5MPz0dtmxxrfj4ccUV0Lat/vuzz2qWeiopcb1oG4wmMCmBfQcGyncbL/Q33Qem+xjMt5Hp8r3AdBslwrsoHOnp6WRlZZ0TVCYnJ5OVlVVvsAqh9Y+05TV4ualoyoiU0aNhwwZ48UXo1q3m+tq1etvRV17JwqXdUoE4+LhZs9prWM2cCZMmQadOMGkS/aJcKisa7KL3htCQRe9Nx4sF0S2Ni/Xx+YH1c+ITiY/Ly8vZuXMnZWVlpKamMnDgwLgEyYWFhdWz5usiKyurugXVH/PEMrEqEkpL9XjS2bMhcJWqdu3gl7+En/wEWrQ4N5/PB3PnwrRpNctIeYLPB9Onw5w54dPddJMeEDtsWL0i7aL3FkBvEWmyfLfxQn/TfWC6j8F8G5ku3wtMt1EivIsiISUlheHDh5OZmcnw4cOjCkTD6R9Ny+vq1at57bXXOH36dFRlREv79vDUU3r3pgkTaq4fPw6PPgpDh8K779bO448Hs7P12eeLrsyY9N+zp/5AFOCNN+DZZ+M+ntSuM5rAHItma4gmKN9tvNDfdB+Y7mMw30amy/cC022UCO8itwmnv4iQkZFBWlpa2JbXnTt3UlhYCEBeXh633XYbXbp0iaiMhtK/P7z9tg48c3JKOHKkAwC7dsG11+pAdeZM6Nu3dsOk/5yXF3kLaUz6DxigWz3feCN8upwcyM3VSwrEEdsyarFYLBaLxXjqa3k9ePBg9d/FxcXk5eWxa9cuT3S75hp44omF/O53uqvez7Jleo/7YcPObZicM6dhLaQNJjs77O0vWrbUgagL4wfsmFFDaMiY0dLS0oiXvWgIbst3e5yZ2/p7UYbp8r0YS2i6jUyXD+bXZdPle1GGKT7++OOPWbp0KZUB+3dmZmaSmZnJ8ePHPfHzkSPw+ON6olMkIVhOTmQtpDHbaNKkyFpGI2yutWNGLQBs3brVaPlu44X+pvvAdB+D+TYyXb4XmG6jRHgXuU289B86dCh33XVXraCtoKCABQsW8OGHH8aljFD4n+GCC+CFF2D9+tqz7kMRaQtpTDYqKYE334xMmblzG15OCGwwmsBs27bNaPlu44X+pvvAdB+D+TYyXb4XmG6jRHgXuU089e/evTv33HMPvXv3rr62e/duioqKOHr0aNzKCSbwGXw+eO45Il7qKZKANCYbdegAN95Yf7qcHD3VP87YYNRisVgsFst5RUpKClOnTuU73/lO9bWqqiry8vLYs2eP6+XPnRvZ5PVAXGqUrOGJJ8Lefqtz5+hmVEWBnU3fxBGR64Hre/bsSW5uLgCjRo2ia9euLFu2DIBevXoxbtw48vLyAGjRogXZ2dm0bNmyOs+kSZPYu3cvH330EQCjR48mNTWV5cuXA9CnTx8yMjKqtxPzV9SFCxdWz9C77bbb2Lp1a/Wvr759+7Jv3z5WrlwJQP/+/Rk5ciTz588HoF27dkyePJkFCxZw/PhxAKZMmcLGjRvZvXs3AGPHjqWyspKCggIABg8ezJAhQ3j11VdJT09n4cKF3HzzzcybN696L+Ls7GwKCwvZv38/AOPHj6esrIx169YBMGzYMPr27cuiRYsA6NatGxMnTiQ/P58zZ84AMH36dFJTU6vtM2HCBI4ePcqGDRsAGDFiBD179mTJkiWA3jf5uuuuIy8vD5/PR1JSEtOnT+ftt9+uHhR/ww03cODAATZv3lztpyFDhlSXEcpPixcv5kvn53G0fvKP+Q7np8zMTJKTkxvkp4qKCnJzc8P6CaBTp04N8lObNm2q7RPKTytWrODzzz9vsJ8yMzPr9VMk9SmUnyoqKti6dWvM9SmUn/w+iLU+hfLT6NGjef/992OuT+H81KtXL44cORJzfQrlp4qKCvLz82OuT6H81K1bt+rvaSz1KZSfunXrRmlpacz1KZyfKioq8Pl8MdenUH5KT08nNzc35voUyk+jR4+u9kEs9SnYT8eOHaNDhw6UOLsMVVVVsWLFClatWhX3915FRQXr16+nb9++nDq1iNGjM1m3bgCRMnr0Lk6dKmDz5rr9dPr0aXJzc2OqT59dfjm9P/gAX1ISSQHNsLtGj2Zms2ZcsHFjxPUpKpRS9jDgGDFihIqWvXv3Rp2nKclvyDNHg9v6e1GG6fLd9rFS5tvIdPlKmV+XTZfvRRmm+/jw4cPq6aefVtu3b3etjOBnqKpSKidHKT2NKfxx++06vT9ffn7N51DyG8SHHyp1991KXXJJTeFPPqlUVVXUPgY2qQhjHNtNn8D4fxGaKt9tvNDfdB+Y7mMw30amy/cC022UCO8it3Fb/wsuuIC2bdvWuW3o2cAtlGIg+BmSknSvd05O/Xm3b9e7OoVbGD8uNho2TM+uChhPy6WXur4dlA1GLRaLxWKxnPfUtUXo4cOHeeaZZ1ybABZpQPrRRzBuHNx5Z+2F8V1Zh1QELrqo5vPf/x7nAs7FBqMJTP/+/Y2W7zZe6G+6D0z3MZhvI9Ple4HpNkqEd5HbNIaNTp06xauvvkp5eTkLFy7kvffewxdD5BfqGUIFpP4lPf1s2gTz5tVOExiQxtVGgcHop5/GT24I7ASmBGbkyJFGy3cbL/Q33Qem+xjMt5Hp8r3AdBslwrvIbRrDRuXl5SQFdE+vW7eOw4cPM2nSpHN2d2qI/ED8ASnoADNwbfnTp+H++0PL9beUzp4dRxsFdtN7EIzaltEExj9r0FT5buOF/qb7wHQfg/k2Ml2+F5huo0R4F7lNY9ioc+fOzJgxg379+lVf279/Py+88AKHDx8GdMC6ZcsWCgoK2LJlS/Ws+kjkB+MPSPPzawJRn0+3iNbHnDnw/e8fiF+XvW0ZtVgsFovFYml8WrVqxe23387q1aspLCwEoKSkhLy8PAYMGMDu3btrbS367rvvkp6eTnp6ep1jUOsjKUmPC4WayUqRrke6bt0Apk+P01KgwcGocnfreNsymsC0a9fOaPlu44X+pvvAdB+D+TYyXb4XmG6jRHgXuU1j2khEyMrKYvLkybRo0QLQ65Fu3769ViAKUFlZyapVqygqKopYfigadWH8Tp2gbVv994kT8NVXcRAaGlEuR7uW+JCWlqY2RdJWn0CkpaVxvj3z+Yb18fmB9XPic774uLi4mFdeeaV6sf1QJCcn8/DDD0c9trS8vJydO3dSVlZG27apPPfcQF58MXIZgWNNY+bSS8G/3/3GjaTdd19UPhaRzUqptEjS2pbRBGbBggVGy3cbL/Q33Qem+xjMt5Hp8r3AdBslwrvIbZqKjbp06cIVV1xRb7rKykpWrFjBZ599RllZGUqpsPKVUhQWFjJr1iyWLl3K6tWrefvtpfTuPYuHHioE6m84/Pa3P4vvbp0eTmKyY0YTGP9WdKbKdxsv9DfdB6b7GMy3kenyvcB0GyXCu8htmpKNTp06FVG6jz76qHrrzObNm6OU4rXXXqNjx4506tSp+mjfvj1FRUWsWrXqHBmVlZV06LCKhx6CZ57JCFvenj0d+fRT6NMnIvXqx8NJTDYYtVgsFovFYomQ1NTUqPP4d3HasWNHrett27blRz/6UZ1jTAPp0qWIu+9O48UXU2jdupxBg3bStm0ZJ06ksnPnQMrLU/jqq3ZceSW8/z5ccknUKp6Lh8GoHTNqCA0ZM3ry5EnatGnjkkbuy3d7DJLb+ntRhunyvRhnZrqNTJcP5tdl0+V7UYbpPo6mjPLycmbNmnXO5KVAkpKSGDBgAKWlpRw7doyKioo60/Xq1Ythw4axdOnSesudMOE65s8/SZs2RTRvXlO2SDIFBen85S/pgNClCyxfDsOH1ysyPAsXwi236L+vvZa0I0fY9P770KFDRNntmFELABs3bjRavtt4ob/pPjDdx2C+jUyX7wWm2ygR3kVu05RslJKSQnp6etg0mZmZ3HrrrcyYMYN/+Zd/4Z//+Z+5/PLLuemmmxgzZgxDhw6lZ8+e9OjRg7KysojKXbOmiA4dVtUKRAGUqiQjYxVXXVUAQHExZGXBmjURia0bn6/2tPxPPuG/9+3Ts+wnTdL7k8YRG4wmMLt37zZavtt4ob/pPjDdx2C+jUyX7wWm2ygR3kVu09RslJ6eTlZWFsnJtUc7Jicnk5WVdU6w2rp1aw4fPsyll15KZmYmN954I3fffTfjx4+PuNu/tLS0Hp0K6d5dL7p//Ljey37FitppfD546aV69rP3L3D61ls11774gqtKSvR6o2+8AZddFteg1I4ZtVgsFovFYokCESEjI4O0tLTqpZhSU1MZOHBg1Ms5DRw4kHfffTdst39kOin++Med3HPPcI4cgVOn4PrrYcECuPHG2ovoFxSEWQJqz57IFjh94w3o2BFeeAEasMB/ILZlNIEZO3as0fLdxgv9TfeB6T4G821kunwvMN1GifAucpumaqOUlBSGDx9OZmYmw4cPDxuIhpIfSbd/t27dItKnffsyioqgVy/9+cwZPezz5Zdr7+Y0Z47+XFlZR0vpgAFw0031F5aTA7m5MQeiYIPRhCbWX1mNLd9tvNDfdB+Y7mMw30amy/cC022UCO8it0kEG4WTX1+3/6hRoyIqIzU1lX79oKgI+vXT16qq9BajwY2dc+boWffZ2TowrRWQZmeHL+jii3UgGqdFTW03fRNHRK4Hru/Zsye5ubkAjBo1iq5du7Js2TJAz8YbN24ceXl5ALRo0YLs7GyWLFlCB2fW26RJk9i7d2/1mmejR48mNTWV5cuXA9CnTx8yMjLIz88H9C+1qVOnsnDhwuqdJm677Ta2bt3Ktm3bAKioqCA5OZmVK1cC0L9/f0aOHMn8+fMBvfXZ5MmTWbBgQfX6bVOmTGHjxo3VY3PGjh1LZWUlBQV64PXgwYMZMmQIr776Kunp6SxcuJCbb76ZefPmUV6ux8JkZ2dTWFjI/v37ARg/fjxlZWWsW7cOgGHDhtG3b18WLVoE6F+UEydOJD8/nzNnzgAwffp0li5dWl3uhAkTOHr0KBs2bABgxIgR9OzZkyVLlgDQo0cPrrvuOvLy8vD5fCQlJTF9+nTefvttDh48CMANN9zAgQMH2Lx5c7Wf1qxZU11GKD8tXryYL7/8skF+OnbsGI899lhYP2VmZjbYT+vWraNLly5h/QTQqVOnBvmpTZs21d/rUH5asWIFn3/+eYP95PP52LNnT1g/RVKfQvmpuLiYH/zgBzHXp1B+WrRoEV26dIm5PoXyk8/n44svvoi5PoXzU69evThy5EjM9SmUn4qLi+nRo0fM9SmUn5YtW1Zt01jqUyg/nThxggsvvDDm+hTOT8XFxTz22GMx16dQfkpPTyc3Nzfm+hTKT3v37q22Vyz1KZyfqqqqOHToUEz1KZyfiouLGTt2bJ1+eumllzhz5gydO3dm5MiRfPjhh5SVldGqVSsuvvhi/vGPfxAJGzZsYMuWLUyfPp1f/GIFjz02goMHO4dMv2ePPs+Zoxfev+KKF0hKgnG5uVwUrqB9+/j6lls4PnMmy997r9pPgfUpKpRS9jDgGDFihIqW559/Puo8TUl+Q545GtzW34syTJfvto+VMt9GpstXyvy6bLp8L8ow3cdelBGr/IKCAvXkk0+GPJ5++ula6auqlPrhD5XSs44iO3JylKr66mulRCLLkJ8fUl9gk4owxrEtownM4MGDjZbvNl7ob7oPTPcxmG8j0+V7gek2SoR3kdskgo1ile8fV1pUVFSryz85OZn09HSSArrM/ZOV/u//oitDd+V3IG/iTSS9uSh84pwcmDYtugJCYIPRBGbIkCFGy3cbL/Q33Qem+xjMt5Hp8r3AdBslwrvIbRLBRrHKD5zB//HHH/Ppp5+ilKJ3794MHTq0epcn0EuERjIhvi7mzAFuyCWPN0gixMZIOTlhpuNHj53AlMD4x7WYKt9tvNDfdB+Y7mMw30amy/cC022UCO8it0kEG8VDvlKKDRs28N5777Fr1y52797N8uXLmTVrFnl5eejecd1gmZNzbv7WrcsZPnwLGRkFDB++hdaty+ssZ86STkz/1nv4qJkpv82/UkCcA1GwLaMWi8VisVgsRlBUVFQ9oSqQyspKTpw4QVFRERkZGSQl6XgR/C2kioyMItLTa28les0171JUlE5hod5KNJA5n40ls9fj3Pn5vwOwtl07Bv/v/+pIN46BKNhgNKHp1KmT0fLdxgv9TfeB6T4G821kunwvMN1GifAucptEsFGs8svLyykqKgqbpqioiLS0NFJSUmoFpPv2FXHVVavOSd+8eWX19cLCjFr3cnJgWp9W8Av9uaVSeo0oFxB/k66laZOWlqY2bdrU2Gp4SlpaGufbM59vWB+fH1g/Jz7Wx+6zZcsWli5dWm+65s2b06ZNG2699Va6d+/OiRPlPPXULJKSQq9zevZsMjNnPsypU7orPjsbMjLgzpLZJD3yEAALunZlsrMcVySIyGalVFokae2Y0QRm3rx5Rst3Gy/0N90HpvsYzLeR6fK9wHQbJcK7yG0SwUaxyi8rK4so3dmzZykpKaFZs2YA7N69M2wgCrqFdPz45bRuXU5Wlr52110wfeH3q8eNtnSx8dIGowmMfxFeU+W7jRf6m+4D030M5tvIdPleYLqNEuFd5DaJYKNY5aempkaVvlWrVkDkQexll33MI4/MIimpkPx8HXjOWTeA6eThQ2hZa4um+GLHjFosFovFYrE0cQYOHMi7774bdlvR5ORk7r33XkSEtm3bAtEFsc2bV5KevoqzZ2vGkM7hLgB+WPWzGLQPjx0zaggNGTN65swZWrRo4ZJG7st3ewyS2/p7UYbp8r0YZ2a6jUyXD+bXZdPle1GG6T72oox4yC8sLGTVqnMnIvnJysoiI6P2RKTy8nJmzZoVNogNJngMKcC1zeeytGJaxBPp7ZhRC6C/tCbLdxsv9DfdB6b7GMy3kenyvcB0GyXCu8htEsFG8ZCfnp5OVlYWycm1O7aTk5Pp1atX9S5NgaSkpNR5PRzNm1cyaNDOWtfeOTuN6Xecxo3eettNn8Ds37/faPlu44X+pvvAdB+D+TYyXb4XmG6jRHgXuU0i2Cge8gN3Ydq5cydlZWWkpqYycOBA5s2bh4jUmS/UVqLhaNv23LGmc+a3hDUryXuzC0mXD2v4gwRhg1GLxWKxWCwWg0hJSWH48OERpw8MYpcvX87HH39cb55LL/2ETp2OUVCQwddfd66+PuezsWQOv5M7bzoBTzwBw2IPSm0wmsCMHz/eaPlu44X+pvvAdB+D+TYyXb4XmG6jRHgXuU0i2KgpyE9JSWH8+PFs37693hbSrl2/omvXrxg27GP+/vfevPzyVCCJHP7ENObCGwo6doQXXoAQLbKRYseMJjCRLufQVOW7jRf6m+4D030M5tvIdPleYLqNEuFd5DaJYKOmIj/aMaQi0KfP37njjnlkM4c8ppOE0ls05ebGHIiCDUYTmnXr1hkt32280N90H5juYzDfRqbL9wLTbZQI7yK3SQQbNSX5oSZChaN377+T3PGM/nDxxToQjdMe9babvokjItcD1/fs2ZPc3FwARo0aRdeuXVm2bBkAvXr1Yty4ceQ5m9C2aNGC7OxsSkpKqvNMmjSJvXv38tFHHwEwevRoUlNTWb58OQB9+vQhIyOD/Px8QP9ymjp1KgsXLuTYsWMA3HbbbWzdupVt27YBUFFRwb59+1i5ciUA/fv3Z+TIkcyfPx+Adu3aMXnyZBYsWMDx48cBmDJlChs3bmT37t0AjB07lsrKSgoKCgAYPHgwQ4YM4dVXXyU9PZ2FCxdy8803M2/evOoFg7OzsyksLKweDD5+/HjKysqqK+KwYcPo27cvixYtAqBbt25MnDiR/Px8zpzRFWn69OkcP3682j4TJkzg6NGjbNiwAYARI0bQs2dPlixZAkCPHj247rrryMvLw+fzkZSUxPTp03n77bc5ePAgADfccAMHDhxg8+bN1X46c+ZMdRmh/LR48WK+dLZYi9ZPft+E81NmZibJyckN8lNxcTG5ublh/QR6z+WG+KlNmzbV9gnlpxUrVvD555832E9AvX6KpD6F8lNxcTFbt26NuT6F8pPfB7HWp1B+8vl8vP/++zHXp3B+6tWrF0eOHIm5PoXyU3FxMfn5+THXp1B+OnHiRPX3NJb6FMpPJ06coLS0NOb6FM5PxcXF+Hy+mOtTKD+lp6eTm5sbc30K5Sefz1ftg1jqUzg/VVVVsXr16pjqUzg/FRcXs379+pjrUyg/+d8Vkdaniy++mO3bt1NRUcHZs2frXZRfBIozuzF9cR55+6ZTesstHJ85k+XvvVftp8D6FA12nVFDaMg6o+vXr+eKK65wSSP35bu9bp3b+ntRhunyvVhn1HQbmS4fzK/Lpsv3ogzTfexFGU1Z/ksvvcSnn35ab7r9+y/i5ZfvJIc/6e76/Dlw5511po1mnVHbMprA9O3b12j5buOF/qb7wHQfg/k2Ml2+F5huo0R4F7lNItioKctv165dROlOnmwDOLsy9etH3pTvxmW8px0zmsD4uwFMle82Xuhvug9M9zGYbyPT5XuB6TZKhHeR2ySCjZqy/OBdm0IxcOAOMjIKAcWcPelMvycpLovg22DUYrFYLBaL5Tymc+fOtGrVu950zZv7uOqqVWRkFAEwZw7MnRt7+TYYTWC6detmtHy38UJ/031guo/BfBuZLt8LTLdRIryL3CYRbNTU5T/66FROnepNJFOJ0tOLaN26nOxsmDYtpmIBO4HJGBoygcl0vJjcYmlcrI/PD6yfEx/rY/Px+eDuu2Hv3kKuvnpVvenfeut6hg4dzosv1r3CUzQTmGzLaALTkOUVmpJ8t/FCf9N9YLqPwXwbmS7fC0y3USK8i9wmEWzU1OXPnQv5+ZCUFFkjZdu2ZeTn2256Sz341ywzVb7beKG/6T4w3cdgvo1Ml+8FptsoEd5FbpMINmrq8qdNg+xsOHEiNaL0J06kxq2b3gajFovFYrFYLBYAduwYSFVV+PCwqiqJHTsGxa1MO2bUEBoyZjRwBxo3cFu+22OQ3NbfizJMl+/FODPTbWS6fDC/Lpsu34syTPexF2U0dfkvvaRbRlu3LudnP/sdzZqFXrOpqiqJp59+lFOnUsjPr3vdeztm1ALAihUrjJbvNl7ob7oPTPcxmG8j0+V7gek2SoR3kdskgo2auvxp0yAnBwYN2hk2EAVo1szHoEE7ycmx3fRGISI9ReRPInJQRE6LyKci8oyIdHSrTP++tqbKdxsv9DfdB6b7GMy3kenyvcB0GyXCu8htEsFGTV1+UhLk5UFGRllE6TMyysjLq3smfdRlxy7CUh8icjGwGcgBNgCzgP3Ag8BfRaRzI6pnsVgsFovFQlISTJ0a2QSmqVNT4xKIgg1GveJ/gW7AA0qpiUqpx5RSV6GD0gHAf7hR6IQJE9wQ65l8t/FCf9N9YLqPwXwbmS7fC0y3USK8i9wmEWxkivxBgwaSnJwcNk1ycjKXXBK/CUw2GHUZp1V0HPAp8Ieg278ETgLTRKRNvMs+evRovEV6Kt9tvNDfdB+Y7mMw30amy/cC022UCO8it0kEG5kiPyUlhfT09LBp0tPTad26dVzKAxuMekGWc16hlKo1IlgpVQasBVKAb8e74A0bNsRbpKfy3cYL/U33gek+BvNtZLp8LzDdRonwLnKbRLCRSfLT09PJyso6p4U0OTmZrKyseoPVaAnfDmuJBwOc8+4Q9/egW077Ays90chisVgsFoslBCJCRkYGaWlp7NixkzVryli9ejbz588jJSUl/uXZdUbdRURygRnADKVUXh33/wP4V+BflVL/GXTvHuAe5+MAYFeUxbcHSqNWuunI7wIUuyjfbf29KMN0+W77GMy3kenywfy6bLp8L8ow3cdelGG6/Gh9/C2lVNeIUiql7OHiAeQCCpge4v5/OPf/nxtlu/1sLsvfZLL+CeIDo32cIDYyWr4XfjbdRgnyLjLax4ngZ5N9bMeMuo//V0r7EPf910tcKHupCzK9lO82Xuhvug9M9zGYbyPT5XuB6TZKhHeR2ySCjUyX7xq2m95lRGQ68AL6F8u9ddxfjh4zerVSyo4ZDUBENqkItxKzmIn18fmB9XPiY32c+LjpY9sy6j6rnPM4EallbxFJBb4LlAN/81oxA8htbAUsrmN9fH5g/Zz4WB8nPq752LaMekBA6+cDSqlnA67PBB4GnldK3ddY+lksFovFYrE0FjYY9QBn4ft16F2YlgA7gCvQa5DuBkYrpb5qPA0tFovFYrFYGgcbjHqEiHwT+Dfg+0Bn4BDwJvArpdTXjambxWKxWCwWS2Nhg1GLxWKxWCwWS6NhJzBZPENEeorIn0TkoIicFpFPReQZEenYAFnDReT/ROSAI+uIiBSIyB1u6G6JjFh9LCJjRERFcHzT7Wex1E286rGIXCkiS5z8FSLyuYi8IyLfd0t3S+TE0c+TRGS1iJSKyCkR2SYi/09EWrilu6V+RORmEXlWRIpE5LjzXp3XQFkxf1dsy6jFE+oYN7sTGIUeN7sL+G6k42ZF5CfAbOBrYBnwD6ATMAQ4oJSaHPcHsNRLPHwsIhcB2SFuXwrcBGxVSl0aH60t0RCveiwiPwL+FziJHq50AOiJ9m8K8LhS6j/ceAZL/cTRz78B/h9wAlgEHAPSgTT09tfXKKXOuvEMlvCIyIfAMLRvDgADgflKqalRyonP/3Y3V+u3hz38B7AcvdPUT4Ouz3Su/zFCOeMAnyMvtY77zRv7Wc/XI14+DiP/FUfOA439rOfrEQ8fA83Rm3ycAgYE3RsEVKCXu2vZ2M97vh5x8vNwJ+3XQJ+A6wI859x7pLGf9Xw9nGCxn+OPMY4/5jXGd0UpZVtGLe7j/HLaC3wKXKyU8gXcS0VP5hKgm1LqZD2yPgL6Ar2UXYGgyRBPH4eQ3wX9690H9FBKlcRBbUsUxMvHInIBcBj4WCk1rI77H6NbwbvYOu49cfTzvwG/AJ5WSv1T0L2O6FbS/Uqpi+P+EJaoEJEx6DXRo2oZjed7344ZtXhBlnNeEfhlBVBKlQFr0V1z3w4nRESGAEOBFcAxEckSkZ+JyKMiMjZ4UwGLp8TFx2G4E2gJvG4D0UYjXj7+EjgK9BeRfoE3RKQ/urXmQxuINhrx8vOFznl/8A2lV5D5GugjIr1jU9fSiMTtvW//eVu8YIBz3h3i/h7n3L8eOSOd85fAauAvwFPA08D7wIci0rfhalpiIF4+DsUM5/x8A/NbYicuPla6O+5+9P+fzSLykoj8p4i8DGwGtgG3xEFfS8OIV10uds7nBJsi0gHwT24ZEHzfYgxxe+/bYNTiBe2dc2mI+/7rHeqR08053w1cBExwZPcH5qG79pbZWZqNQrx8fA4ikol+6W1VSq2LXjVLnIibj5VSrwNXoceO3gE8BkxDT2iaQx2taRbPiJeflznnGc7ERABERIDAyWlRr6ZiaTLE7Z1gg1GLSfi/r82AyUqpd5RSx5VSe9D/0DahA9NJjaWgxRXucc527+sEQUSmonszitCTllKc80rg98CCxtPOEg+UUmuBF9GByMciMkdEfgesRzco7HSS+uqWYDmfsMGoxQv8v47ah7jvv15Sjxz//cNKqb8G3nC6/pY4H0dFqZ8lduLl41qISCf0j4tTwNwGaWaJF3HxsTMu9E/o7vhpSqmdSqlTSqmd6NbRzcAtzqQKi/fEsy7PAO5FL/Fzq/P3cfTs7X1Omi8boqSlSRC374oNRi1esMs5hxo34p/EEGrcSbCckhD3/duqto5MLUsciZePg/FPXHrNTlxqdOLl43Ho5Z0K6pj04AMKnY8jGqKkJWbiVpeVJlcpNVIp1UYp1VYpdbVS6m/oYVU+YEvsKlsaibh9V2wwavGCVc55XPCMd2f5h++i1xX8Wz1y/oYeU3aRiLSp4/4Q5/z3GHS1NIx4+TgY/8Ql20Xf+MTLxy2dc9cQ9/3XzzRESUvMuFWXA+WMAXoBy5RSocYbWpo+cfuu2GDU4jpKqX3o5ZguQs+iDeRXQBtgbuA6ZCIyUEQGBskpR49BagX82hkI709/KXrnnkpgYfyfwhKOePk4EBFJR48jtBOXmgBx9HGRc75ZRIYG3hCRy4Cb0Ytl/yVuylsiJp51WUTa1XHtW0Ae+sfG4/HT3OIWItLc8XGtNWEb8l0JWYZd9N7iBXVsGbYDuAK9TtluYHTguoIiogCUUhIkpx1QAFyGHgi/FrgAvY1ga+AhpdRslx/HUgfx8nHA/bnAVPSOS8+6q70lEuJYj/8E5KADkjeBz9D/0CYCLYBnlFIPu/s0llDE0c+vA99Cd8UfQy/z9AP0MI1pSqlXXX8YS52IyER0fQO9Jux49CoW/h+LxUqpnzlpL0L3OH6mlLooSE5U35WQ+thg1OIVIvJN4N+A7wOd0bszvAn8ylkEOTBtyEBFRNqi9zu+Bf2iOwVsQO/0scLNZ7CEJ44+7ggcRLeQ2R2XmhDx8LHTq3EnujdjGJCKntjyAfCCUsrOpm9k4uTnO9GrYQxE+/gIusX7v5RSO9x+BktoRORJ4JdhklQHnuGCUed+xN+VkPrYYNRisVgsFovF0ljYMaMWi8VisVgslkbDBqMWi8VisVgslkbDBqMWi8VisVgslkbDBqMWi8VisVgslkbDBqMWi8VisVgslkbDBqMWi8VisVgslkbDBqMWi8VisVgslkbDBqMWi8ViGCLypIgoEcmv496nzr0xnitmsVgsDcAGoxaLJaEQkXwnGAs+ykRkm4j8r4gMamw9GwMRGSoi94nIiyLysYhUOraJeMcjEenn5P9cRE6LyEERWSAiIxqgz2AReUJE1ojIF4684yLykYj8QUS+G6GcJBHJEpF/FpHXROTvAX6/Lwp9rhaRpSLypYhUiMg+EZktIhdE+2wWiyVykhtbAYvFYnGJs+j9sAEE6AJc4hx3i8hUpdTrjaVcI/EyevvNBiEi3wMWAynOpVL0vta3AZNEJEcpNS8COX2A/0Jv6eunEr0laHtgqHP8WETeB+5WSn0eRmQ79DaTDUZEfg782vnoA04AfYAHgNtF5Cql1NZYyrBYLHVjW0YtFkuisk4pdaFzXAC0Aq4BPgVaAHNEpGtjKtgInAU+BPKAe4HlkWYUkQuBhehA9D3gIqVUB3QwOh/duPGiiAyuR844YAs6EN0OPAgMAloopToDLdFB4CPAVuBqYKuIXF+PiieBImAW8EPgcBTPdi01gejvgA5KqfbAELS9ugJLRKRlpDItFkvk2GDUYrGcFyilziql/gxMcS61ASY1okqNwbeVUpcrpWYopXKJImADHkO3QH4B3KSU+gxAKfUlcCewGR3k/1soASJyDbAMbfuHgaFKqf9RSu1USilHXpVS6u9KqVnAZcA/owPU10QkI4ToUqCdUipDKfWIUuoV4HQUz/Yb5/ymUupnSqkyR5dtwPXUtJLeE4VMi8USITYYtVgs5xt/RQcXoLvsayEiLUTkJyJSJCLHnHGMn4nIn+obayoinUXkVyKyWURKRKRcRHY7Yyon1pE+wxmTuN4Ze3nGGa/4ZxG5OS5PG4BSqqoh+UQkCZjsfHxOKXUi8L4jd6bz8ToRaVeHjD7oFtRTwLVKqWfq08cJTJ9CDwNIBhbX1ZqtNL5on8vRazA1QxeeqkP2AeAV5+OU4PsWiyV2bDBqsVjOR8Q5N6t1UaQ7sAF4FrgSPX7xNNALyAG2iMhNdQoUSQd2AU8Aw4HW6MCrDzqYejMofVugAD0mcRTQ1knfFRgPvC4iz8f4nPHiEsA/iSdU1/4K59wCbbtgfgd0BO5VSr1XX4EikiIiF4pIC6XUYuCfnPwhW14bSJZzLgXWh0jjf+ZRjt8sFkscscGoxWI53xiN7iYG2O+/KCLNgSXoVrKVTrpWSql2QA/gGfS407kicnGgQOfz20Bn9BjDq4AUZwxkKjAOeCNIDx96DOaNQGelVDtnnGJH4Cfo1tt7ROQWGh9/C7JCj/M8B6VUMfBlUHoAROQyYCKwyOlCD7x3tdMKfdJpiX5TRC4BfgwcQvsB4H+A3cC0ulpeY8Cv644wrav+ZxZgYBzLtlgs2GDUYrGcJ4hIcxEZD/hne58FXg1IcicwEj0J5hql1F+VUmcBlFKHlFIPA8+jJ/A8HCT+P9HjKXcDGUqpVf4uaKXUKaXUe0qpWuNTlVLlSqlblFKLlVLHAq6XKKX+gA7GCDg3Jt2d89dKqYow6Q4Gpffj795+MvCiMxRhObolNQU9NnQiunX68sC0TqD4EvqHxFVRaR8ev64Hw6QJvBf8bBaLJUZsMGqxWBKV0SJy2DmOABXAn4GL0K2S9zrjAf3c6Zxn+4PQOpjvnL/nv+B0297ofHzCP/klDix1zt8WkWZhU7qPvyX5VD3pyp1zcFf2NeiWx+qlkRy75aL/D80EOjj5LgP2oWfEB7PWOUe9pmkYInm28oC/bTe9xRJn7DqjFoslUWlOzTjHQI4B45VSm/wXRCQZPW4T4HkR+UMImf6g8JsB19LQ71KFDnYjxin3TvQyR8OATugxl4G0QnfdF0cju6kgIgIMoCaQ9zMR/VwrlFKPBlz/SERuBHZwri38s//PtyW5LJaExraMWiyWRKVAKSVKKUEHdJehx2h2Qq+H2TEgbWAQ2BkdxNZ1dHHStA7I6w94S5VSpZEqFzCBKQ89YelCoAo4ChxxDj9tzhHgLSedc+uwqWoWww+cbd8ZHawfCkrr74Z/Neg6Sqn96K76YPxjRc/Uo0c0RPJsKQF/nwiZymKxNAgbjFosloRHKXVaKfURcCt6jOJQ9PhPP4Hvwsv9QWy4Iw5q/QI9OacY3Tp6gVIqRSnVTSl1IfCNgLTxKC8W/GMmO4pIqzDpejjnwMDTr3twT5w/sAw1VrOu672d8xdhdIgWfzk9wqQJvBccVFsslhixwajFYjlvcBZWfwDdAnmLiGQ6t75yroFexika/C2Y7UWkfRT5/LPkf6qUetlZPD6QprQfeuBs8nPWZgUQkS5At6D0UGPbC4OylDjnniHKrOu6fyLU6tCqRo1f10HOeqp1EbiawI44lm2xWLDBqMViOc9QSu2mpmv4P5xrZwH/GNJrohS5Cb2vukSZ1x9sfRDi/tVR6uEmO6gJur8XIo3/+hlgjf+iMwt+P/DtoPQbnfO0YEEiMhC4IujalejdkDYopTYG54mBVc65PXo1hboY55zXK6VOhkhjsVgaiA1GLRbL+cjTzvm7IjLG+TvfOWeLyLDgDIEEjjd1diPyL2j/KxFJjVAH//jSS+uQ3xb4eYRyXMcJKBc4H38sIrXGsDotiv7lrpYqpY4HiVgB9A3at34pekJShojMEZH+IpIqImOde/7JYh1F5H7gHfSs9ulxezBAKbUd+Mj5+E/B90WkB3C78zF4EpbFYokDNhi1WCznHUqpD4D3nY+PO+cXgb+hJzv9RURmBC6u7uwGNEVECoAHg0T+K1AG9AcKRSTL3+UrIq1FZIKIvBOUx78L0UwRyXRmnSMiI9GL7neOy8MG4Oxq1MV/oNf1BGgReD3ELkP/BRxHD2N4Q0R6OTK7ogP5kehW0V/Wkfc15+y3NUqpU0C2kycbvXvVcbRf2gF/dJK+AfweHbxfrZT6JMSztQ96Nv//tzZBz9ayjuz/6pwnich/+39QOIvvL0VvXLAfeKGusi0WS4wopexhD3vYI2EOdGCkgNX1pPuek04B33audUN3MfuvV6HHPJ4IuKaAX9YhLwv4OiBNBXpyUqX/WlD6PuiZ8/70pwLKKUd3DfvvXRSU90nnen4denzq3BtTx70nqf0coY5z5AbY7GRAuhL0mq0KvYnA1DD2Xu2kvSbo+gj0Kgf70eM3n0NPGLrX+bwYuAe9G1Y4f66O8NmyQ+R/PCBNJTr49X8+Cgxp7O+2PeyRqIdtGbVYLOclSu+P7h+v+Qvn2pdAJnqizDvoIMTf7b4TeBk9I/+/6pC3Cr2e5m+BreiAphV6AfdXgB8Epd+PXtt0HnobzWbo4G4+MFIptYImhmOzy4A5wAH0ckhH0C2f31ZKzQudmwfRAforIlK9aL1SarNS6malVB+l1CVKqR8ppQ4qpZ53Pk9USuWq8Ds/xYxS6tfoYHsZ+kdFS3SA/D/oQHRrmOwWiyUGRCnV2DpYLBaL5TxARKYCc9Etvz8GXlb1/BMSkR8CB5VSq93X0GKxNAY2GLVYLBaLZzjB5Z/QLY8foNd7/TPwuVJKOWNn+6FXE5iBbon9AEhTeiKVxWJJMGwwarFYLBZPEZHLgFnAmIDLlegJTO2oWSD/DHqHqseUUmUeqmixWDzEBqMWi8ViaRSctUNvRE/+6gl0QE8Y24VeDuolpdQ/Gk1Bi8XiCTYYtVgsFovFYrE0GnY2vcVisVgsFoul0bDBqMVisVgsFoul0bDBqMVisVgsFoul0bDBqMVisVgsFoul0bDBqMVisVgsFoul0bDBqMVisVgsFoul0fj/RYh93t+FtcgAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "results_dict = get_results_sift()\n", - "plot_one(\n", - " results_dict,\n", - " xlim=0.55,\n", - " ylim=60000,\n", - " title=\"Results on SIFT-128-euclidean\",\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "149cb520-26dc-40cd-a6e5-328062ca36d6", - "metadata": { - "jp-MarkdownHeadingCollapsed": true, - "tags": [] - }, - "source": [ - "## Appendix: Install PECOS and NMSLIB" - ] - }, - { - "cell_type": "markdown", - "id": "b58276b2", - "metadata": { - "jp-MarkdownHeadingCollapsed": true, - "tags": [] - }, - "source": [ - "### Install via Conda \n", - "```bash\n", - "conda create -n pecos-hnsw-tutorial python=3.8\n", - "conda activate pecos-hnsw-tutorial\n", - "\n", - "pip install pyarrow pandas ipython jupyterlab\n", - "```\n", - "\n", - "### Install PECOS from Source\n", - "\n", - "We will install PECOS from source with the -march=native flag to optimize the best SIMD instruction available in your machine. More details available in https://github.com/amzn/pecos#installation-from-source\n", - "\n", - "```bash\n", - "# prerequisite, assuming amazon linux 2 \n", - "sudo yum -y install python3 python3-devel python3-distutils python3-venv && sudo yum -y groupinstall 'Development Tools' \n", - "sudo amazon-linux-extras install epel -y\n", - "sudo yum install openblas-devel -y\n", - "# pecos with -march=native flag\n", - "git clone https://github.com/amzn/pecos\n", - "cd pecos\n", - "PECOS_MANUAL_COMPILE_ARGS=\"-march=native\" python -m pip install --editable .\n", - "```\n", - "\n", - "### Install NMSLIB from Source\n", - "\n", - "We follow the install guide [install guide](https://github.com/erikbern/ann-benchmarks/blob/master/install/Dockerfile.nmslib) from ANN-Benchmark to install NMSLIB from source for the best performance.\n", - "\n", - "```bash\n", - "# pre-requisite, assuming amazon linux 2\n", - "sudo yum -y install cmake boost-devel eigen3-devel\n", - "git clone https://github.com/searchivarius/nmslib.git\n", - "cd nmslib/similarity_search\n", - "cmake . -DWITH_EXTRAS=1\n", - "make -j4\n", - "pip install pybind11\n", - "cd ../python_bindings/\n", - "python setup.py build\n", - "python setup.py install\n", - "python -c 'import nmslib'\n", - "```\n", - "\n", - "### Install via Docker (as in ANN-Benchmkark)\n", - "\n", - "```bash\n", - "# install some basic stuff\n", - "sudo yum -y update\n", - "sudo yum install -y git curl zip unzip vim gcc-c++ htop\n", - "\n", - "# https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html\n", - "# sudo yum update -y\n", - "# sudo amazon-linux-extras install docker\n", - "sudo service docker start\n", - "sudo systemctl enable docker\n", - "sudo usermod -a -G docker ec2-user\n", - "docker info\n", - "```\n", - "\n", - "### Install Docker Image\n", - "\n", - "```bash\n", - "# install miniconda fist!\n", - "conda create -n ann-benchmarks python=3.8\n", - "conda activate ann-benchmarks\n", - "\n", - "# install ANN package supported by ann-benchmarks\n", - "git clone https://github.com/erikbern/ann-benchmarks.git\n", - "cd ann-benchmarks\n", - "pip install -r requirements.txt\n", - "\n", - "# install docker containers\n", - "python -u install.py --algorithm faiss\n", - "python -u install.py --algorithm hnswlib\n", - "python -u install.py --algorithm n2\n", - "python -u install.py --algorithm pecos\n", - "python -u install.py --algorithm scann\n", - "python -u install.py --algorithm ngt\n", - "python -u install.py --algorithm nmslib\n", - "python -u install.py --algorithm diskann\n", - "python -u install.py --algorithm pynndescent\n", - "\n", - "# list all dockers\n", - "docker image ls\n", - "REPOSITORY TAG IMAGE ID CREATED SIZE\n", - "ann-benchmarks-hnswlib latest 2e1ea8d11df7 2 hours ago 1.04GB\n", - "ann-benchmarks-nmslib latest 1e094d3e96f7 3 hours ago 1.64GB\n", - "ann-benchmarks-faiss latest 44e5bd15bfcd 5 hours ago 4.9GB\n", - "ann-benchmarks-scann latest 5151abe3b09e 5 hours ago 2.76GB\n", - "ann-benchmarks latest c2c612131da4 5 hours ago 938MB\n", - "```\n", - "\n", - "### Enter Docker Env\n", - "\n", - "```bash\n", - "EFS_DIR=/PATH/TO/pecos-hnsw-kdd22\n", - "DOCKER_IMAGE=ann-benchmarks-nmslib\n", - "\n", - "docker run --rm -it -v ${EFS_DIR}:/home/app/ws \\\n", - " --entrypoint /bin/bash ${DOCKER_IMAGE}\n", - "```" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/tutorials/kdd22/Session 4 Utilities in PECOS.ipynb b/tutorials/kdd22/Session 4 Utilities in PECOS.ipynb deleted file mode 100644 index 3257e41e..00000000 --- a/tutorials/kdd22/Session 4 Utilities in PECOS.ipynb +++ /dev/null @@ -1,1103 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "b1ebc316", - "metadata": {}, - "source": [ - "# Useful Utilities in PECOS\n", - "\n", - "PECOS provides various useful interfaces and utility functions for XMC problems and related tasks. In this session, we will present some utilities in PECOS for efficient matrix operations and hierarchical clustering.\n", - "\n", - "## Outline\n", - "\n", - "1. Sparse Matrix Operations\n", - "2. Hierarchical Clustering" - ] - }, - { - "cell_type": "markdown", - "id": "c2b3f61e", - "metadata": {}, - "source": [ - "## 1. Sparse Matrix Operations\n", - "\n", - "Most of the computations in PECOS are based on sparse matrices, so PECOS also provides various useful and efficient operation utilities for sparse matrices." - ] - }, - { - "cell_type": "markdown", - "id": "5fba58d4", - "metadata": {}, - "source": [ - "### 1.1 Genric Matriox IO and Conversion\n", - "\n", - "`smat_util.load_matrix` and `smat_util.save_matrix` provide generic interfaces for loading and storing matrices in arbitrary common formats, including [dense matrix](https://numpy.org/doc/stable/reference/generated/numpy.array.html) in NumPy or different sparse matrix formats (i.e., [sparse Compressed Sparse Row (CSR) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html), [sparse Compressed Sparse Column (CSC) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html), and [sparse COOrdinate (COO) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html))." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "385ed0ba", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Dense Matrtix IO\n", - "mat is a matrix with a shape (2, 3).\n", - "[[0.32706124 0.94765886 0.16764024]\n", - " [0.29065096 0.23160388 0.3871939 ]]\n", - "mat_loaded is a matrix with a shape (2, 3).\n", - "[[0.32706124 0.94765886 0.16764024]\n", - " [0.29065096 0.23160388 0.3871939 ]]\n", - "\n", - "csr Sparse Matrix IO\n", - "mat is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (1, 1)\t0.3859982113301277\n", - " (2, 1)\t0.5399444869534915\n", - " (3, 1)\t0.008896715300809821\n", - " (4, 2)\t0.9634283904734527\n", - "mat_loaded is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (1, 1)\t0.3859982113301277\n", - " (2, 1)\t0.5399444869534915\n", - " (3, 1)\t0.008896715300809821\n", - " (4, 2)\t0.9634283904734527\n", - "\n", - "csc Sparse Matrix IO\n", - "mat is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (2, 0)\t0.8430107100693552\n", - " (0, 3)\t0.5602000410939516\n", - " (3, 3)\t0.4358575080842668\n", - " (4, 3)\t0.454532975053182\n", - "mat_loaded is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (2, 0)\t0.8430107100693552\n", - " (0, 3)\t0.5602000410939516\n", - " (3, 3)\t0.4358575080842668\n", - " (4, 3)\t0.454532975053182\n", - "\n", - "coo Sparse Matrix IO\n", - "mat is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (2, 0)\t0.6008844736009242\n", - " (3, 3)\t0.5296164005351621\n", - " (3, 2)\t0.6884529935778093\n", - " (0, 3)\t0.847894528567365\n", - "mat_loaded is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (2, 0)\t0.6008844736009242\n", - " (3, 3)\t0.5296164005351621\n", - " (3, 2)\t0.6884529935778093\n", - " (0, 3)\t0.847894528567365\n", - "\n" - ] - } - ], - "source": [ - "from pecos.utils import smat_util\n", - "import numpy as np\n", - "import scipy.sparse as smat\n", - "\n", - "print(\"Dense Matrtix IO\")\n", - "mat = np.random.rand(2, 3)\n", - "print(f\"mat is a {type(mat)} matrix with a shape {mat.shape}.\")\n", - "print(mat)\n", - "smat_util.save_matrix(\"mat.npz\", mat)\n", - "mat_loaded = smat_util.load_matrix(\"mat.npz\")\n", - "print(f\"mat_loaded is a {type(mat_loaded)} matrix with a shape {mat_loaded.shape}.\")\n", - "print(mat)\n", - "print(\"\") \n", - "\n", - "for matrix_format in [\"csr\", \"csc\", \"coo\"]:\n", - " print(f\"{matrix_format} Sparse Matrix IO\")\n", - " mat = smat.random(5, 4, density=0.2, format=matrix_format)\n", - " print(f\"mat is a {type(mat)} matrix\"\n", - " f\" with a shape {mat.shape} and {mat.nnz} non-zero values.\")\n", - " print(mat)\n", - " \n", - " smat_util.save_matrix(\"mat.npz\", mat)\n", - " mat_loaded = smat_util.load_matrix(\"mat.npz\")\n", - " print(f\"mat_loaded is a {type(mat_loaded)} matrix\"\n", - " f\" with a shape {mat_loaded.shape} and {mat_loaded.nnz} non-zero values.\")\n", - " print(mat_loaded)\n", - " print(\"\") " - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "2579b855", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Original Matrix mat\n", - " [[0.16936286 0.78425304 0.8562633 0.61722574 0.4486684 0.23233178]\n", - " [0.79099373 0.3961628 0.91564054 0.58414229 0.43155964 0.55876417]\n", - " [0.44718835 0.05151288 0.42833526 0.12533758 0.2968885 0.82826553]\n", - " [0.57886779 0.45415528 0.24104546 0.04155873 0.7281743 0.08374103]] \n", - "\n", - "csr_mat = dense_to_csr(mat)\n", - "csr_mat is a matrix with a shape (4, 6) and 24 non-zero values.\n", - "[[0.16936286 0.78425304 0.8562633 0.61722574 0.4486684 0.23233178]\n", - " [0.79099373 0.3961628 0.91564054 0.58414229 0.43155964 0.55876417]\n", - " [0.44718835 0.05151288 0.42833526 0.12533758 0.2968885 0.82826553]\n", - " [0.57886779 0.45415528 0.24104546 0.04155873 0.7281743 0.08374103]] \n", - "\n", - "csr_mat_topk = dense_to_csr(mat, topk=2)\n", - "csr_mat is a matrix with a shape (4, 6) and 8 non-zero values.\n", - "[[0. 0.78425304 0.8562633 0. 0. 0. ]\n", - " [0.79099373 0. 0.91564054 0. 0. 0. ]\n", - " [0. 0. 0.42833526 0. 0. 0.82826553]\n", - " [0. 0.45415528 0. 0. 0.7281743 0. ]] \n", - "\n" - ] - } - ], - "source": [ - "mat = np.random.rand(4, 6)\n", - "\n", - "print(f\"Original Matrix mat\\n\", mat, \"\\n\")\n", - "\n", - "print(\"csr_mat = dense_to_csr(mat)\")\n", - "csr_mat = smat_util.dense_to_csr(mat)\n", - "print(f\"csr_mat is a {type(csr_mat)} matrix\"\n", - " f\" with a shape {csr_mat.shape} and {csr_mat.nnz} non-zero values.\")\n", - "print(csr_mat.toarray(), \"\\n\")\n", - "\n", - "print(\"csr_mat_topk = dense_to_csr(mat, topk=2)\")\n", - "csr_mat_topk = smat_util.dense_to_csr(mat, topk=2)\n", - "print(f\"csr_mat is a {type(csr_mat_topk)} matrix\"\n", - " f\" with a shape {csr_mat_topk.shape} and {csr_mat_topk.nnz} non-zero values.\")\n", - "print(csr_mat_topk.toarray(), \"\\n\")" - ] - }, - { - "cell_type": "markdown", - "id": "b2746e3c", - "metadata": {}, - "source": [ - "### 1.2. Memory-efficient Sparse Matrix Operations\n", - "\n", - "To manipulate with sparse matrix, PECOS provides many useful memory-efficient functions. For example, for CSR matrices, we have following functions to combine multiple matrices.\n", - "\n", - "* `hstack_csr([mat, mat, mat]`\n", - "* `vstack_csr([mat, mat, mat]`\n", - "* `block_diag_csr([mat, mat, mat]`\n", - "\n", - "These funcations are also available for CSC matrices as `hstack_csc`, `vstack_csr`, and `block_diag_csr`.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "b9dac617", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Original Matrix mat\n", - " [[0.77445782 0.45465131]\n", - " [0. 0.43456434]\n", - " [0. 0. ]] \n", - "\n", - "hstack_csr([mat, mat, mat])\n", - "[[0.77445782 0.45465131 0.77445782 0.45465131 0.77445782 0.45465131]\n", - " [0. 0.43456434 0. 0.43456434 0. 0.43456434]\n", - " [0. 0. 0. 0. 0. 0. ]] \n", - "\n", - "vstack_csr([mat, mat, mat])\n", - "[[0.77445782 0.45465131]\n", - " [0. 0.43456434]\n", - " [0. 0. ]\n", - " [0.77445782 0.45465131]\n", - " [0. 0.43456434]\n", - " [0. 0. ]\n", - " [0.77445782 0.45465131]\n", - " [0. 0.43456434]\n", - " [0. 0. ]] \n", - "\n", - "block_diag_csr([mat, mat, mat])\n", - "[[0.77445782 0.45465131 0. 0. 0. 0. ]\n", - " [0. 0.43456434 0. 0. 0. 0. ]\n", - " [0. 0. 0. 0. 0. 0. ]\n", - " [0. 0. 0.77445782 0.45465131 0. 0. ]\n", - " [0. 0. 0. 0.43456434 0. 0. ]\n", - " [0. 0. 0. 0. 0. 0. ]\n", - " [0. 0. 0. 0. 0.77445782 0.45465131]\n", - " [0. 0. 0. 0. 0. 0.43456434]\n", - " [0. 0. 0. 0. 0. 0. ]] \n", - "\n" - ] - } - ], - "source": [ - "from pecos.utils import smat_util\n", - "import scipy.sparse as smat\n", - "\n", - "mat = smat.random(3, 2, density=0.5, format=\"csr\")\n", - "print(f\"Original Matrix {type(mat)} mat\\n\", mat.toarray(), \"\\n\")\n", - "\n", - "print(f\"hstack_csr([mat, mat, mat])\")\n", - "print(smat_util.hstack_csr([mat, mat, mat]).toarray(), \"\\n\")\n", - "\n", - "print(f\"vstack_csr([mat, mat, mat])\")\n", - "print(smat_util.vstack_csr([mat, mat, mat]).toarray(), \"\\n\")\n", - "\n", - "print(f\"block_diag_csr([mat, mat, mat])\")\n", - "print(smat_util.block_diag_csr([mat, mat, mat]).toarray(), \"\\n\")" - ] - }, - { - "cell_type": "markdown", - "id": "5f9cf7f1", - "metadata": {}, - "source": [ - "### 1.3. Sparse-to-sparse Matrix Multiplication (SpMM)\n", - "\n", - "Many operations in PECOS or XMC problems rely on Sparse-to-sparse Matrix Multiplication (SpMM), such as the computation of PIFA features. It is also one of the key primitives in large-scale linear algebra operations, with a broad range of applications in machine learning and natural language processing.\n", - "\n", - "For SpMM, PECOS provides a highly optimized multi-core CPU implementation with state-of-the-art performance, where the underlying operations are implemented and optimized in C/C++.\n", - "Specifically, the Python interface and parameters are as follows:\n", - "\n", - "```python\n", - "from pecos.core import clib as pecos_clib\n", - "Z = pecos_clib.sparse_matmul(X, Y, eliminate_zeros=False, sorted_indices=True, threads=-1)\n", - "```\n", - "* Parameters\n", - " * `X` (scipy.sparse.csr_matrix or scipy.sparse.csc_matrix): the first sparse matrix to be multiplied.\n", - " * `Y` (scipy.sparse.csr_matrix or scipy.sparse.csc_matrix): the second sparse matrix to be multiplied.\n", - " * `eliminate_zeros` (bool, optional): if true, then eliminate (potential) zeros created by maxnnz in output matrix Z. Default is false.\n", - " * `sorted_indices` (bool, optional): if true, then sort the Z.indices for the output matrix Z. Default is true.\n", - " * `threads` (int, optional): The number of threads. Default -1 to use all CPU cores." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "b2e6c54a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "||Z_true - Z_pred|| = 0.0\n" - ] - } - ], - "source": [ - "import numpy as np\n", - "import scipy.sparse as smat\n", - "from scipy.sparse import linalg\n", - "from pecos.core import clib as pecos_clib\n", - "X = smat.random(1000, 1000, density=0.01, format='csr', dtype=np.float32)\n", - "Y = smat.random(1000, 1000, density=0.01, format='csr', dtype=np.float32)\n", - "Z_true = X.dot(Y)\n", - "Z_pred = pecos_clib.sparse_matmul(X, Y)\n", - "print(\"||Z_true - Z_pred|| = \", linalg.norm(Z_true - Z_pred))" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "14e4bded", - "metadata": {}, - "outputs": [], - "source": [ - "import time\n", - "DATASET = \"wiki10-31k\"\n", - "X = smat_util.load_matrix(f\"xmc-base/{DATASET}/tfidf-attnxml/X.trn.npz\").astype(np.float32)\n", - "Y = smat_util.load_matrix(f\"xmc-base/{DATASET}/Y.trn.npz\").astype(np.float32)\n", - "YT_csr = Y.T.tocsr()\n", - "X_csr = X.tocsr()" - ] - }, - { - "cell_type": "markdown", - "id": "bb5af1e9", - "metadata": {}, - "source": [ - "#### Benchmarking Sparse Matrix Muplication\n", - "\n", - "The SpMM utility has state-of-the-art performance in efficiency as shown in the following figure.\n", - "\n", - "
\n", - "
\n", - "
\n", - "\n", - "In this part, we provide some hands-on instructions for benchmarking different methods for SpMM." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "c71cffdb", - "metadata": {}, - "outputs": [], - "source": [ - "# Benchmarking SciPy\n", - "\n", - "start = time.time()\n", - "Z = YT_csr.dot(X_csr)\n", - "Z.sort_indices()\n", - "run_time_scipy = time.time() - start" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "2f2b9795", - "metadata": {}, - "outputs": [], - "source": [ - "# Benchmarking PyTorch\n", - "\n", - "import torch\n", - "\n", - "def csr_to_coo(A):\n", - " A_coo = smat.coo_matrix(A)\n", - " indices = np.vstack([A_coo.row, A_coo.col]).T\n", - " values = A_coo.data\n", - " return indices, values\n", - "\n", - "def get_pt_data(A_csr):\n", - " A_indices, A_values = csr_to_coo(A_csr)\n", - " A_pt = torch.sparse_coo_tensor(\n", - " A_indices.T.astype(np.int64),\n", - " A_values.astype(np.float32),\n", - " A_csr.shape,\n", - " )\n", - " return A_pt\n", - " \n", - "YT_pt = get_pt_data(YT_csr)\n", - "X_pt = get_pt_data(X_csr)\n", - "start = time.time()\n", - "Z_pt = torch.sparse.mm(YT_pt, X_pt)\n", - "run_time_pytorch = time.time() - start" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "54b24694", - "metadata": {}, - "outputs": [], - "source": [ - "# Benchmarking PECOS\n", - "\n", - "start = time.time()\n", - "Z = pecos_clib.sparse_matmul(\n", - " YT_csr, X_csr,\n", - " eliminate_zeros=False,\n", - " sorted_indices=True\n", - ")\n", - "run_time_pecos = time.time() - start" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "e12f29e9", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD4CAYAAADmWv3KAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAW/0lEQVR4nO3deZSldX3n8fcHEFkEDNIqitDigqMMspQoB6Oho44oOh4Vlbjr0CbHBUYzc3BiEjGTBCfBNY5nWkARFAVUNhcUFKMJA3RDsx+HBkVRAo0aWWQR/M4f9yktiqpbT917n9vVl/frnHvqPvv3culv/er7/J7fL1WFJGnybLKhA5AkdcMEL0kTygQvSRPKBC9JE8oEL0kTarMNHcBMO+ywQy1fvnxDhyFJG401a9bcUlXL5tq2pBL88uXLWb169YYOQ5I2Gkmun2+bJRpJmlAmeEmaUCZ4SZpQJnhJmlAmeEmaUCZ4SZpQJnhJmlAmeEmaUCZ4SZpQnT3JmmQ34IszVu0K/FVVfaSL6y0/4qtdnFbAj4568YYOQdIAOkvwVfUDYE+AJJsCPwW+0tX1JEn3N64SzR8D11bVvGMmSJJGa1wJ/jXASXNtSLIyyeokq9evXz+mcCRp8nWe4JNsDrwUOGWu7VW1qqqmqmpq2bI5R7yUJA1gHC34A4GLq+qmMVxLktQYR4I/hHnKM5Kk7nSa4JNsDTwf+HKX15EkPVCnMzpV1R3AI7q8hiRpbj7JKkkTasEWfJJHAvsDjwHuBK4AVlfVbzuOTZI0hHkTfJIDgCOA7YFLgJuBLYCXAU9IcipwdFXdOoY4JUmL1K8F/yLg0Kr68ewNSTYDDqJ3A/VLHcUmSRrCvAm+qv5bn233Aqd1EZAkaTQWvMma5LAk26bn2CQXJ3nBOIKTJA2uTS+atzR19hcAfwC8Hjiq06gkSUNrk+DT/HwRcEJVXTljnSRpiWqT4Nck+Sa9BH92km0Au0hK0hLX5knWt9KbuOO6qvp1kkcAb+40KknS0Pr1g9971qpdEyszkrSx6NeCP7r5uQWwD3AZvdr7HsBqYL9uQ5MkDWPeGnxVHVBVBwA3Avs0k3LsA+xFb35VSdIS1uYm625Vdfn0QlVdAfyH7kKSJI1Cm5uslyU5BjixWX4tvXKNJGkJa5Pg3wz8GXBYs/zPwCc7i0iSNBILJviqugv4cPOSJG0k2owHvz/wfmCXmftX1a7dhSVJGlabEs2xwH8F1gD3dRuOJGlU2iT4X1XV1zuPRJI0Um0S/HeS/APwZeDu6ZVVdfFCByZ5OHAMsDtQ9EamPH+wUCVJi9EmwT+z+Tk1Y10BK1oc+1HgG1X1yiSbA1stMj5J0oDa9KI5YJATJ9kOeA7wpuY89wD3DHIuSdLitZnRabskH0qyunkd3STvhTweWA98OsklSY5JsvUc5185fe7169cP8BEkSXNpM1TBccBtwKua163Ap1sctxmwN/DJqtoLuAM4YvZOVbWqGedmatmyZa0DlyT116YG/4SqesWM5SOTrG1x3A3ADVV1QbN8KnMkeElSN9q04O9M8uzphebBpzsXOqiq/g34SZLdmlV/DFw1UJSSpEVr04L/M+D4GXX3X9LcOG3hncDnmh401+FMUJI0Nm160awFnp5k22b51rYnb46dWmg/SdLotelF83dJHl5Vt1bVrUn+IMn/HEdwkqTBtanBH1hV/z69UFW/BF7UWUSSpJFok+A3TfLQ6YUkWwIP7bO/JGkJaHOT9XPAuUmm+76/GTi+u5AkSaPQ5ibrB5NcCjyvWfU3VXV2t2FJkobVpgUPcDVwb1Wdk2SrJNtU1W1dBiZJGk6bXjSH0nsK9f80qx4LnNZhTJKkEWhzk/XtwP70xqChqq4BHtllUJKk4bVJ8Hc3Q/0CkGQzeuPBS5KWsDYJ/rtJ/gewZZLnA6cAZ3YbliRpWG0S/BH0xnW/HHgb8DXgfV0GJUkaXptukr8FPgV8Ksn2wE5VZYlGkpa4Nr1ozkuybZPc19BL9B/uPjRJ0jDalGi2a0aQfDnw2ap6Jr2x3SVJS1ibBL9Zkh3pTdd3VsfxSJJGpE2C/wBwNrCuqi5KsitwTbdhSZKG1eYm6yn0ukZOL18HvGL+IyRJS8G8Lfgk72turM63fUWSg7oJS5I0rH4t+MuBM5PcBVxMry/8FsCTgD2Bc4C/6zpASdJg5k3wVXU6cHqSJ9Ebi2ZHeuPRnAisrKo7xxOiJGkQbWrw1zDgTdUkPwJuA+6jN9ywE3BL0pi0HQ9+GAdU1S1juI4kaYY23SQlSRuhrhN8Ad9MsibJyrl2SLIyyeokq9evX99xOJL04NFmLJonJzk3yRXN8h5J2o4m+eyq2hs4EHh7kufM3qGqVlXVVFVNLVu2bFHBS5Lm16YF/yngvcBvAKrqMuA1bU5eVT9tft4MfAXYd7AwJUmL1SbBb1VVF85ad+9CByXZOsk20++BFwBXLD5ESdIg2vSiuSXJE2im6UvySuDGFsc9CvhKkunrfL6qvjFooJKkxWmT4N8OrAKekuSnwA+B1y10UDNmzdOHC0+SNKg2DzpdBzyvKbNsUlW3dR+WJGlYCyb4JA8H3gAspzc2PABV9a4uA5MkDadNieZrwP+lN/jYb7sNR5I0Km0S/BZV9e7OI5EkjVSbbpInJDk0yY5Jtp9+dR6ZJGkobVrw9wD/APwFTVfJ5ueuXQUlSRpemwT/HuCJjggpSRuXNiWadcCvuw5EkjRabVrwdwBrk3wHuHt6pd0kJWlpa5PgT2tekqSNSJsnWY8fRyCSpNGaN8EnObmqXpXkcn7fe+Z3qmqPTiOTJA2lXwv+w83Pg8YRiB5clh/x1Q0dwsT60VEv3tAhaInol+A/AexdVdePKxhJ0uj06yaZsUUhSRq5fi34xyb52Hwb7SYpSUtbvwR/J7BmXIFIkkarX4L/uV0kJWnj1a8Gf8/YopAkjdy8Cb6qnjXOQCRJo9VmsLGhJNk0ySVJzur6WpKk3+s8wQOHAVeP4TqSpBlaJfimFf6YJDtPv1oetxPwYuCYYYKUJC3egoONJXkn8NfATfx+0u0C2oxF8xHgvwPb9Dn/SmAlwM47t/q9IUlqoc1wwYcBu1XVzxdz4iQHATdX1ZokfzTfflW1ClgFMDU19YBBzSRJg2lTovkJ8KsBzr0/8NIkPwK+AKxIcuIA55EkDaBNC/464LwkX+X+Mzp9qN9BVfVe4L0ATQv+z6vqdQNHKklalDYJ/sfNa/PmJUnaCLSZ0elIgCQPa5ZvX+xFquo84LzFHidJGtyCNfgkuye5BLgSuDLJmiRP6z40SdIw2txkXQW8u6p2qapdgPcAn+o2LEnSsNok+K2r6jvTC025ZevOIpIkjUSrXjRJ/hI4oVl+Hb2eNZKkJaxNC/4twDLgy81rWbNOkrSEtelF80vA6fkkaSMzb4JP8pGqOjzJmfTGnrmfqnppp5FJkobSrwU/XXP/x3EEIkkarXkTfFVNT7i9Z1V9dOa2JIcB3+0yMEnScNrcZH3jHOveNOI4JEkj1q8GfwjwJ8Djk5wxY9M2wC+6DkySNJx+Nfh/BW4EdgCOnrH+NuCyLoOSJA2vXw3+euB6YL/xhSNJGpU2g409K8lFSW5Pck+S+5LcOo7gJEmDa3OT9Z+AQ4BrgC2B/wJ8osugJEnDa5Pgqap1wKZVdV9VfRp4YbdhSZKG1WawsV8n2RxYm+R/0bvx2uoXgyRpw2mTqF/f7PcO4A7gccArugxKkjS8Ni34W4B7quou4MgkmwIP7TYsSdKw2rTgzwW2mrG8JXBON+FIkkalTYLfYuZE2837rfrsD0CSLZJcmOTSJFcmOXKYQCVJi9Mmwd+RZO/phST7AHe2OO5uYEVVPR3YE3hhkmcNFKUkadHa1OAPB05J8jMgwKOBVy90UFUVMN3yf0jzesC48pKkbrSZ0emiJE8BdmtW/aCqftPm5M0N2TXAE4FPVNUFc+yzElgJsPPOO7eNW5K0gHlLNElWND9fDrwEeHLzekmzbkHNg1F7AjsB+ybZfY59VlXVVFVNLVu2bICPIEmaS78W/HOBb9NL7rMVvQm4W6mqf0/yHXpPwF6xqAglSQPpN5rkXzc/3zzIiZMsA37TJPctgecDHxwoSknSovWb8OPd/Q6sqg8tcO4dgeObOvwmwMlVddbiQ5QkDaJfiWabYU5cVZcBew1zDknS4PqVaHwwSZI2Ym0m/Ng1yZlJ1ie5OcnpSXYdR3CSpMG1eZL188DJ9GrqjwFOAU7qMihJ0vDaJPitquqEqrq3eZ0IbNF1YJKk4bQZquDrSY4AvkCv//urga8l2R6gqn7RYXySpAG1SfCvan6+bdb619BL+NbjJWkJajMWzePHEYgkabT6Pei0oqq+Pd+4M1XVeqgCSdL4jWUsGknS+C04Fg3wgar64cxtSSzbSNIS16ab5JfmWHfqqAORJI1Wvxr8U4CnAdvNqsNvi/3gJWnJ61eD3w04CHg496/D3wYc2mFMkqQR6FeDPx04Pcl+VXX+GGOSJI1AvxLNx2kmyU5yyOztVfWuDuOSJA2pX4lm9diikCSNXL8SzfHjDESSNFoLDlXQTJZds9dX1YpOIpIkjUSbwcb+fMb7LYBXAPd2E44kaVTaDDa2Ztaqf0lyYUfxSJJGpE2JZvsZi5sA+wDbtTjuccBngUfRK/GsqqqPDhinJGmR2pRo1tBL0KFXmvkh8NYWx90LvKeqLk6yDbAmybeq6qqBo5UktdbZePBVdSNwY/P+tiRXA48FTPCSNAb9HnSacxz4aYsZDz7JcmAv4II5tq0EVgLsvPPObU8pSVpAvxb8qcDa5gW9Es201uPBJ3kYvREpD6+qW2dvr6pVwCqAqampB3THlCQNpl+Cfzm9eVf3AE4HTqqqdYs5eZKH0Evun3MGKEkar3nHg6+q06rqNfRmdroWODrJ95M8t82JkwQ4Fri6qj40kmglSa21mfDjLuBXwK3Aw2g/Fvz+wOuBFUnWNq8XDRamJGmx+k66Ta9Esy9wDvDRqmo9AFlVfZ/71+0lSWPUrwZ/DnAZ8H3gocAbkrxheqPDBUvS0tYvwb95bFFIkkbO4YIlaUK1uckqSdoImeAlaUINlOCTbD7qQCRJo7Vggk9yXjOWzPTyvsBFXQYlSRpem+GC/x74RpKP0RsN8kDsYSNJS16b4YLPTvKnwLeAW4C9qurfOo9MkjSUNiWavwQ+DjwHeD9wXpIXdxyXJGlIbUo0jwD2rao7gfOTfAM4Bvhqp5FJkobSpkRz+Kzl64HndxWQJGk0+g029pGqOjzJmfQm+Lifqnppp5FJkobSrwV/QvPzH8cRiCRptPqNRbMmyabAyqp67RhjkiSNQN9eNFV1H7CLT65K0sanTS+a64B/SXIGcMf0Sqfhk6SlrU2Cv7Z5bQJs06x7wE1XSdLS0ibBX1VVp8xckeTgjuKRJI1Im9Ek39tynSRpCenXD/5A4EXAY5uBxqZtC9y70ImTHAccBNxcVbsPG6gkaXH6teB/BqwG7gLWzHidAfynFuf+DPDCIeOTJA2oXz/4S4FLk3y+qn6z2BNX1T/PHEdekjRebW6yLk/y98BTgS2mV1bVrp1FJUkaWpubrJ8GPkmv7n4A8FngxFEFkGRlktVJVq9fv35Up5WkB702CX7LqjoXSFVdX1XvB0Y2HnxVraqqqaqaWrZs2ahOK0kPem1KNHcn2QS4Jsk7gJ8CD+s2LEnSsNq04A8DtgLeBewDvB5440IHJTkJOB/YLckNSd46TKCSpMVpM+HHRc3b21nEZNtVdcigQUmShtfvQacz+h3ohB+StLT1a8HvB/wEOAm4AMhYIpIkjUS/BP9oenOvHgL8Cb1Jtk+qqivHEZikpWX5EV/d0CFMrB8dNbKOifcz703Wqrqvqr5RVW8EngWsA85retJIkpa4vjdZkzyUXp/3Q4DlwMeAr3QfliRpWP1usn4W2B34GnBkVV0xtqgkSUPr14J/Hb0p+g4D3pX87h5rgKqqbTuOTZI0hH6jSbZ5CEqStESZxCVpQpngJWlCmeAlaUKZ4CVpQpngJWlCmeAlaUKZ4CVpQpngJWlCmeAlaUKZ4CVpQpngJWlCmeAlaUKZ4CVpQnWa4JO8MMkPkqxLckSX15Ik3V9nCT7JpsAngAOBpwKHJHlqV9eTJN1fly34fYF1VXVdVd0DfAH4zx1eT5I0Q985WYf0WOAnM5ZvAJ45e6ckK4GVzeLtSX7QYUxLxQ7ALRs6iLbywQ0dwZKw0Xxnfl+/82D5znaZb0OXCb6VqloFrNrQcYxTktVVNbWh41B7fmcbH7+zbks0PwUeN2N5p2adJGkMukzwFwFPSvL4JJsDrwHO6PB6kqQZOivRVNW9Sd4BnA1sChxXVVd2db2NzIOqJDUh/M42Pg/67yxVtaFjkCR1wCdZJWlCmeAlaUKZ4EckyV8kuTLJZUnWJnlAn/9mv6kkH2vevynJ+mb/q5IcOt6oJ1+S+5r/vlckOSXJVvPs9x+b/dYm+UWSHzbvzxni2p9J8srBo39wm++7m7F++nVEs/4hSY5Kck2Si5Ocn+TAZtt2ST7bDJtybfN+u2bbJkk+1lzn8iQXJXn8hvvko7PB+8FPgiT7AQcBe1fV3Ul2ADafa9+qWg2snrHqi1X1jiSPBK5MckZV3dR91A8ad1bVngBJPgf8KfCh2TtV1eXA9H6fAc6qqlPbXCDJplV134ji1e/N9939bv0sfwPsCOze/Dt8FPDcZtuxwBVV9YbmfEcCxwAHA68GHgPsUVW/TbITcEdnn2qMbMGPxo7ALVV1N0BV3VJVP0vyjCT/muTSJBcm2SbJHyU5a/YJqupm4Fpgl6YFsgx+17pYN72soXwPeGKSDyQ5fHplkr9NcthcByQ5pGnVXZH8/nnDJLcnOTrJpcB+Sd7Q/PV2aZITZpziOc3/A9fZmh/K94Anzrexad0fCrxzxr/Dm6rq5CRPBPah9wtg2geAqSRPoPfv98aq+m1z3A1V9cuOPsdYmeBH45vA45L8vyT/O8lzm77/XwQOq6qnA88D7pzvBEl2BXYF1gEnAq9tNj0PuLSq1nf6CSZcks3oDXx3OXAcMN2S24TeMxonznHMY4APAivote6fkeRlzeatgQua7/aXwPuAFc3yzF8WOwLPpvcX3lGj/lwPBrO+O4AtZ5VoXk0v+f+4qm6d4xRPBdbO/Cureb8WeBpwMvCS5lxHJ9mry88zTpZoRqCqbk+yD/CHwAH0Evvf0msVXNTscytAktmHvzrJs4G7gbdV1S+SHAecDnwEeAvw6XF8jgm1ZZK1zfvvAcdW1T1Jft78Q34UcElV/XyOY58BnDf9y7UpEzwHOA24D/hSs98K4JSqugWgqn4x4xynNS3Dq5qSgdp7wHfXvH9AiSbJHoNepKpuSLIbve9xBXBukoOr6txBz7lUmOBHpGkRnAecl+Ry4O0tD/1iVb1j1rl+kuSmJCvojcr52rkPVQvz1WuPAd4EPJpei36x7mpZd797xvsH/HZXX/N9d3NZB+ycZNs5WvFXAXsm2WS6DNP85bZns42mrPN14OtJbgJeBmz0Cd4SzQgk2S3Jk2as2hO4GtgxyTOafbZp/tRs6xh6ZYNTvIHXia8AL6TXSj97nn0uBJ6bZIf05jc4BPjuHPt9Gzg4ySMAkmzfQbzqo6p+Ta+F/9GmPEqSZU1LfB1wCb0y2rT3ARdX1bokezfluOnEvwdw/Xg/QTdM8KPxMOD49Lo6Xkav5vdX9O7Of7y5EfctYItFnPOM5ryWZzrQzFHwHeDk+X6BVtWNwBHNfpcCa6rq9Dn2u5JeSe67zXf9gF46GqnZNfjpexvvA9bTK4ddAZwFTLfm3wo8uekieS3w5GYdwCOBM5tjLgPuBf5pXB+mSw5VsEQlmQI+XFV/uKFjmURNS+1i4OCqumZDxyN1wRb8EpTegxtfAt67oWOZROlNHbkOONfkrklmC16SJpQteEmaUCZ4SZpQJnhJmlAmeEmaUCZ4SZpQ/x/ET4IYm4/UTQAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "from matplotlib import pyplot as plt\n", - "plt.bar(\n", - " [1,2,3],\n", - " [run_time_scipy, run_time_pytorch, run_time_pecos],\n", - " tick_label = [\"SciPy\", \"PyTorch\", \"PECOS\"])\n", - "\n", - "plt.ylabel(\"Matrix Multiplication Time (seconds)\");" - ] - }, - { - "cell_type": "markdown", - "id": "3e9cc45c", - "metadata": {}, - "source": [ - "### 1.4. Sparse Matrix Operations for Working with Arbitrary Data Formats\n", - "\n", - "PECOS is a general machine learning framework and able to fit arbitary data format and interact with different data manipulation and analysis libraries like [Pandas](https://pandas.pydata.org/). In the following example, we will show how to learn a PECOS model with Pandas-loaded data of text, categorical, and numerical features based on sparse matrix operations." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "9f0f17a7", - "metadata": {}, - "outputs": [], - "source": [ - "import pecos\n", - "import pandas as pd\n", - "import numpy as np" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "526820f8", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "2022-08-13 06:42:02 URL:https://archive.ics.uci.edu/ml/machine-learning-databases/00461/drugLib_raw.zip [1133354/1133354] -> \"drugLib_raw.zip\" [1]\n", - "Archive: drugLib_raw.zip\n", - " inflating: drugLibTest_raw.tsv \n", - " inflating: drugLibTrain_raw.tsv \n" - ] - } - ], - "source": [ - "! wget -nv -nc https://archive.ics.uci.edu/ml/machine-learning-databases/00461/drugLib_raw.zip\n", - "! unzip -o drugLib_raw.zip" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "ac8ab46f", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Training DataFrame consists of 3107 instances.\n", - "Testing DataFrame consists of 1036 instances.\n", - "Index(['Unnamed: 0', 'urlDrugName', 'rating', 'effectiveness', 'sideEffects',\n", - " 'condition', 'benefitsReview', 'sideEffectsReview', 'commentsReview'],\n", - " dtype='object')\n" - ] - } - ], - "source": [ - "train_df = pd.read_csv(\"drugLibTrain_raw.tsv\", sep=\"\\t\")\n", - "test_df = pd.read_csv(\"drugLibTest_raw.tsv\", sep=\"\\t\")\n", - "print(f\"Training DataFrame consists of {len(train_df)} instances.\")\n", - "print(f\"Testing DataFrame consists of {len(test_df)} instances.\")\n", - "print(train_df.columns)" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "6d6ee989", - "metadata": {}, - "outputs": [], - "source": [ - "label_name = \"effectiveness\"\n", - "text_features = [\"condition\", \"benefitsReview\", \"sideEffectsReview\", \"commentsReview\"]\n", - "categorical_features = [\"sideEffects\"]\n", - "numerical_features = [\"rating\"]" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "c25b1cb2", - "metadata": {}, - "outputs": [], - "source": [ - "X_trn_list = []\n", - "X_tst_list = []" - ] - }, - { - "cell_type": "markdown", - "id": "4f72d047", - "metadata": {}, - "source": [ - "#### Label Encoding\n", - "\n", - "To encode labels into the sparse matrix format compatible to PECOS, [OneHotEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) and [MultiLabelBinarizer](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html#sklearn.preprocessing.MultiLabelBinarizer) are helpful for the scenarios of multi-class and multi-label classification." - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "4f0d9ec7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Y_trn is a csr matrix with a shape (3107, 5) and 3107 non-zero values.\n", - "Y_tst is a csr matrix with a shape (1036, 5) and 1036 non-zero values.\n" - ] - } - ], - "source": [ - "from sklearn.preprocessing import OneHotEncoder\n", - "\n", - "label_encoder = OneHotEncoder(dtype=np.float32)\n", - "Y_trn = label_encoder.fit_transform(train_df[[label_name]])\n", - "Y_tst = label_encoder.transform(test_df[[label_name]])\n", - "\n", - "print(f\"Y_trn is a {Y_trn.getformat()} matrix with a shape {Y_trn.shape} and {Y_trn.nnz} non-zero values.\")\n", - "print(f\"Y_tst is a {Y_tst.getformat()} matrix with a shape {Y_tst.shape} and {Y_tst.nnz} non-zero values.\")" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "62ec7371", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Y_trn_mlb is a csr matrix with a shape (3107, 5) and 3107 non-zero values.\n", - "Y_tst_mlb is a csr matrix with a shape (1036, 5) and 1036 non-zero values.\n" - ] - } - ], - "source": [ - "from sklearn.preprocessing import MultiLabelBinarizer\n", - "\n", - "label_encoder_multilabel = MultiLabelBinarizer(sparse_output=True)\n", - "Y_trn_mlb = label_encoder.fit_transform([[lbl] for lbl in train_df[label_name].tolist()])\n", - "Y_tst_mlb = label_encoder.fit_transform([[lbl] for lbl in test_df[label_name].tolist()])\n", - "print(f\"Y_trn_mlb is a {Y_trn_mlb.getformat()} matrix with a shape {Y_trn_mlb.shape} and {Y_trn_mlb.nnz} non-zero values.\")\n", - "print(f\"Y_tst_mlb is a {Y_tst_mlb.getformat()} matrix with a shape {Y_tst_mlb.shape} and {Y_tst_mlb.nnz} non-zero values.\")" - ] - }, - { - "cell_type": "markdown", - "id": "260a9a8a", - "metadata": {}, - "source": [ - "#### 1.4.2. Text Feature Encoding\n", - "\n", - "As introduced in Session 1, we can use PECOS vectorizer for featurize text data. In addition, the encoder of [XR-Transformer](https://github.com/amzn/pecos/tree/mainline/pecos/xmc/xtransformer) can be also utilized for deriving text features with proper fine-tuning." - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "96fef619", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "condition: (3107, 3759) and (1036, 3759) in training and testing.\n", - "benefitsReview: (3107, 72861) and (1036, 72861) in training and testing.\n", - "sideEffectsReview: (3107, 64321) and (1036, 64321) in training and testing.\n", - "commentsReview: (3107, 91731) and (1036, 91731) in training and testing.\n" - ] - } - ], - "source": [ - "from pecos.utils.featurization.text.vectorizers import Vectorizer\n", - "\n", - "for feature_name in text_features:\n", - " vectorizer_config = {\n", - " \"type\": \"tfidf\",\n", - " \"kwargs\": {\n", - " \"base_vect_configs\": [\n", - "\n", - " {\n", - " \"ngram_range\": [1, 2],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"word\",\n", - " },\n", - " ],\n", - " },\n", - " } \n", - " train_texts = [str(x) for x in train_df[feature_name].tolist()]\n", - " test_texts = test_df[feature_name].tolist()\n", - " vectorizer = Vectorizer.train(train_texts, config=vectorizer_config)\n", - " X_trn_local = vectorizer.predict(train_texts)\n", - " X_tst_local = vectorizer.predict(test_texts)\n", - " print(f\"{feature_name}: {X_trn_local.shape} and {X_tst_local.shape} in training and testing.\")\n", - " \n", - " X_trn_list.append(X_trn_local)\n", - " X_tst_list.append(X_tst_local)" - ] - }, - { - "cell_type": "markdown", - "id": "38e75fa2", - "metadata": {}, - "source": [ - "#### 1.4.3. Categorical Feature Encoding\n", - "\n", - "Similar to labels, categorical features can also be considered as one-hot or multi-hot embeddings." - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "386b96ad", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "sideEffects: (3107, 5) and (1036, 5) in training and testing.\n" - ] - } - ], - "source": [ - "from sklearn.preprocessing import OneHotEncoder\n", - "\n", - "for feature_name in categorical_features:\n", - " local_encoder = OneHotEncoder(dtype=np.float32)\n", - " X_trn_local = local_encoder.fit_transform(train_df[[feature_name]])\n", - " X_tst_local = local_encoder.transform(test_df[[feature_name]])\n", - " print(f\"{feature_name}: {X_trn_local.shape} and {X_tst_local.shape} in training and testing.\")\n", - " \n", - " X_trn_list.append(X_trn_local)\n", - " X_tst_list.append(X_tst_local)" - ] - }, - { - "cell_type": "markdown", - "id": "6deaf4e8", - "metadata": {}, - "source": [ - "#### 1.4.4. Numerical Features Encoding\n", - "\n", - "Numberical features can be directly incorporated as model inputs after some simple normalization." - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "90668ea4", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "rating: (3107, 1) and (1036, 1) in training and testing.\n" - ] - } - ], - "source": [ - "from scipy.sparse import csr_matrix\n", - "from sklearn.preprocessing import StandardScaler\n", - "\n", - "for feature_name in numerical_features:\n", - " X_trn_values = train_df[[\"rating\"]].values\n", - " X_tst_values = test_df[[\"rating\"]].values\n", - " scaler = StandardScaler()\n", - " X_trn_local = csr_matrix(scaler.fit_transform(X_trn_values), dtype=np.float32)\n", - " X_tst_local = csr_matrix(scaler.transform(X_tst_values), dtype=np.float32)\n", - " print(f\"{feature_name}: {X_trn_local.shape} and {X_tst_local.shape} in training and testing.\")\n", - " \n", - " X_trn_list.append(X_trn_local)\n", - " X_tst_list.append(X_tst_local)" - ] - }, - { - "cell_type": "markdown", - "id": "d2441580", - "metadata": {}, - "source": [ - "#### 1.4.5. Feature Concatenation\n", - "\n", - "PECOS provides easy-going utility functions for efficient matrix operations. The `hstack_csr` function can concatenate different features for each individual instance. More detils about other utilities will be introduced later in this session." - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "id": "d0d3e69c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "X_trn is a csr matrix with a shape (3107, 232678) and 653987 non-zero values.\n", - "X_tst is a csr matrix with a shape (1036, 232678) and 164272 non-zero values.\n" - ] - } - ], - "source": [ - "from pecos.utils import smat_util\n", - "\n", - "X_trn = smat_util.hstack_csr(X_trn_list)\n", - "X_tst = smat_util.hstack_csr(X_tst_list)\n", - "\n", - "print(f\"X_trn is a {X_trn.getformat()} matrix with a shape {X_trn.shape} and {X_trn.nnz} non-zero values.\")\n", - "print(f\"X_tst is a {X_tst.getformat()} matrix with a shape {X_tst.shape} and {X_tst.nnz} non-zero values.\")" - ] - }, - { - "cell_type": "markdown", - "id": "5e61775b", - "metadata": {}, - "source": [ - "#### 1.4.6. Model Training and Testing" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "id": "38189597", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "prec = 52.80 40.69 30.92 24.52 20.00\n", - "recall = 52.80 81.37 92.76 98.07 100.00\n" - ] - } - ], - "source": [ - "from pecos.xmc.xlinear.model import XLinearModel\n", - "xlm = XLinearModel.train(X_trn, Y_trn)\n", - "\n", - "Y_pred = xlm.predict(X_tst, beam_size=10, only_topk=5)\n", - "metrics = smat_util.Metrics.generate(Y_tst, Y_pred, topk=5)\n", - "print(metrics)" - ] - }, - { - "cell_type": "markdown", - "id": "1f833846", - "metadata": {}, - "source": [ - "## 2. Hierarchical Clustering\n", - "\n", - "Hierarchical clustering is an essential function for tree-based XMC models and plays a role of the indexer in PECOS. Accordingly, PECOS also implements hierarchical K-means algorithms in the manner of efficient C/C++, which can also be considered as useful functions for arbitrary tasks. The Python interface of PECOS hierarchical K-means algorithms is as follows:\n", - "\n", - "```python\n", - "from pecos.xmc import HierarchicalKMeans\n", - "HierarchicalKMeans.gen(feature_matrix, ... [training parameters])\n", - "```\n", - "* Training Parameters\n", - " * `nr_splits` (int, optional): The out-degree of each internal node of the tree. Ignored if `imbalanced_ratio != 0` because imbalanced clustering supports only 2-means. Default is `16`.\n", - " * `min_codes` (int): The number of direct child nodes that the top level of the hierarchy should have.\n", - " * `max_leaf_size` (int, optional): The maximum size of each leaf node of the tree. Default is `100`.\n", - " * `spherical` (bool, optional): True will l2-normalize the centroids of k-means after each iteration. Default is `True`.\n", - " * `seed` (int, optional): Random seed. Default is `0`.\n", - " * `kmeans_max_iter` (int, optional): Maximum number of iterations for each k-means problem. Default is `20`.\n", - " * `threads` (int, optional): Number of threads to use. `-1` denotes all CPUs. Default is `-1`.\n", - " \n", - "#### Clustering Chains\n", - "\n", - "Similar to the results of semantic label indexing in PECOS, the hierarchical clustering results will be returned as a list of `D` CSC matrices `C[d]` to denote hierarchical cluster assignments over layers, where `D` is the layers of resulting hierarchical clusters." - ] - }, - { - "cell_type": "markdown", - "id": "3b3b073c", - "metadata": {}, - "source": [ - "### 2.1. Naive Clustering as Degenerated Hierarchical Clustering\n", - "\n", - "When `min_codes` and `max_leaf_size` as the stopping criteria are large enough, the hierarchical clustering will be degenerated to conventional naive clustering." - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "id": "4a5da8c1", - "metadata": {}, - "outputs": [], - "source": [ - "from pecos.utils import smat_util\n", - "import time\n", - "DATASET = \"wiki10-31k\"\n", - "X = smat_util.load_matrix(f\"xmc-base/{DATASET}/tfidf-attnxml/X.trn.npz\").astype(np.float32)\n", - "Y = smat_util.load_matrix(f\"xmc-base/{DATASET}/Y.trn.npz\").astype(np.float32)\n", - "YT_csr = Y.T.tocsr()\n", - "X_csr = X.tocsr()" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "id": "088d87a3", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "2 layers in the trained hierarchical clusters with C[d] as:\n", - "cluster_chain[0] is a csc matrix of shape (4, 1).\n", - "cluster_chain[1] is a csc matrix of shape (14146, 4).\n" - ] - } - ], - "source": [ - "from pecos.xmc.base import HierarchicalKMeans\n", - "import scipy.sparse as smat\n", - "import numpy as np\n", - "\n", - "num_splits = 4\n", - "cluster_chain = HierarchicalKMeans.gen(\n", - " X_csr,\n", - " min_codes=num_splits,\n", - " nr_splits=num_splits,\n", - " max_leaf_size=np.ceil(X_csr.shape[0]/num_splits))\n", - "\n", - "print(f\"{len(cluster_chain)} layers in the trained hierarchical clusters with C[d] as:\")\n", - "for d, C in enumerate(cluster_chain):\n", - " print(f\"cluster_chain[{d}] is a {C.getformat()} matrix of shape {C.shape}.\")" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "id": "ef1c9ee1", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "(14146,) (14146,)\n" - ] - }, - { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAC5EUlEQVR4nOydd5hcVdnAf+feO322l/RGekISWkKo0ntTLIAFBcWGgCCKYEFABAuIiIWigtJFPpFeQodACJDee7JJtu/s9Ln3vt8fd3Z3Zmc22WQ3JIT7ex4esnfOPefcKec9561KRHBxcXFxcelA290TcHFxcXHZs3AFg4uLi4tLHq5gcHFxcXHJwxUMLi4uLi55uILBxcXFxSUPY3dPoCeqq6tl5MiRu3saLi4uLh8r5s6d2ygiNX3pY48VDCNHjuS9997b3dNwcXFx+VihlFrX1z5cVZKLi4uLSx6uYHBxcXFxycMVDC4uLi4uebiCwcXFxcUlD1cwuLi4uLjkscd6Jbm4uOxaRNJI7F5IPgP6QFT42yjPlN09LZc9AFcwuLh8QpHWSyD1FpAEcxGSegOq7neFg4urSnJx+SQi5gZIvQkkO64AKST65904K5c9BVcwuLh8ErGbQHm6XRSwtuyW6bjsWbiCwcXlk4hnUpGLfvCf/JFPxWXPo18Eg1LqJKXUMqXUSqXUVdtod7ZSSpRSB/XHuC4uLjuHUl5U+Z9AhZ3/8IHvEFTo/N09NZc9gD4bn5VSOnAHcDywEZijlHpCRBZ3a1cCXAq809cxXVxc+o7yzYTa2ZBZDFoVyhi2u6fksofQHyeGGcBKEVktImngIeDMIu2uB26my9rl4uLSS0QEST6P3fxN7NYrkMz8fulXKS/Ku58rFFzy6A/BMATYkPP3xuy1TpRSBwDDROSpbXWklLpIKfWeUuq9hoaGfpiai8vegURvR1qvhPTLkHwSafoSknpzd0/LZS9llxuflVIacAtwxfbaisidInKQiBxUU9OndOIuLnsNIimI3Q0kOq4ASaT9N7txVi57M/0hGDYBuefQodlrHZQA+wKvKKXWAjOBJ1wDtItLL7EjOMKgG65rqcsuoj8EwxxgrFJqlFLKC5wDPNHxooi0iUi1iIwUkZHAbOAMEXGr8Li49AatGrSqbhcN8B2+W6bTHUm9gt1wIvaWadjNX0PM9bt7Si59pM+CQURM4GLgOWAJ8IiILFJKXaeUOqOv/bu47A2ICJJZhphrdvhepRSq/DZQJVn30hDoQ1ElP94FM90xJLMQabkErDVAAtJvI83nIpLZ3VNz6QP9kitJRJ4Gnu527Wc9tD2qP8Z0cfm4IOY6pPlrIE0gghhjUZX3oLTyXvehvNOg9i1IzwEVBM9+OOa73YvEHwTSOVdskDik3wbfkbtrWi59ZPd/s1xc9nKk9VKw60ASOAnrliCR63e4H6V8KN/hKO8Be4RQAEDSgN3togL3xPCxZg/5drm47J2IHQVzGfmLpwmpl3fXlPoVFfgsEOh+FXyH7o7puPQTrmBwcdmVKB/QPVkdsANqpL4gknaE0y5C+Q6Gkqsc+wc66KNQlfeiVHdh4fJxwq3H4OKyC1HKgwS/AvF/0hWH4IfQpbt0XBELidwAiUcBC/FMQZXfhtIH9ftYWuhcJPgFkCRKC/Z7/y4fPa5gcHHZxaiSKxBjOMQfBuVDhS5E+Y/dpWNK7G+QeIxOw3BmPtJyEar6f7tkPKU0xyjuslfgCgYXl12MUgoV/DwEP//RDZp4lPy0ZDaYaxFrE0of0tNdLi6Aa2Nwcdkl2IkXsRtOwN66H3bzNxCr7qOdgPIVuSiA96Odh8vHElcwuLj0M5L+ANouB2tt1qf/daTpXESsj24SwQvJ9xbygnc6SndzkLlsH1cwuLjkICKItQWRnc8OL/F/AamcKzZIBNIfXRYYLXgWlF4N2mBQpRA4E1V++0c2vsvHG9fG4OKSRdIfIq2XgN3i/B36Oip8CUqpHewoRWHSO0V+hPCuRwt+AYJf+EjHdNk7cE8MLi6ASBJpuQDsLTi7/RTE/gapF3a4LxX8PIVBXzp4D+6Hme5axFyPHbkRu/VyJPkSIkWyurrs9biCwcUFIP1ukYsJJPH4DnelfEdCyeVOsjt00EejKu/DST685yKZJUjTGRD/l1MMqO1yt+bDJxRXleTiAlkf/CLqH1WyU91pofOR4JeyQV+hPk/vo0Daf5/N55R9HyQB8fuQ8DdRWtnunJrLR4x7YnBxAfAcAFoN+XslHyp0/k53qZT+sREKAFirKRCOyuMWBPoE4goGFxecyF1V+SD4T3WK4hhTUBV/RXkm77Y5iR1FzFWIFBqtxY4h8YexIzcjqVf7xxbgPYLCvE4KjFF979vlY4WrSnJxyaL0KlT5nqFTt6N/huifQOmAhpTehBY4AQCxI0jTmWA149hBHgDfCX2euyr5HpJ+E+x654JYqLLf7fG2ke0h5nok8jNIvw/6AFTJ1Sj/0bt7Wns0rmBwcdnDkNQ7EP0LkOrS7LRdgXhnofQaJH4/WI10xkpIApLPIZmLUJ6xOz2u0iqg+mlIvwN2K/gO3aFiQnsiIhmk+VywmwAbrHVOfYyqf6E8U3f39PZYXFWSy16PiCDJ57GbL8Ru+S6SnrO7p7RNJPk0XZlYsygNUq87/84sJD+ADudkYS7v89hK6SjfoajAKR97oQBAerYTfZ5XDyOFxB/YXTP6WOCeGFz2eiR2p6OWyS62knodKfsdWuD4HevH3IgkHgW7HRU4DeU9YBfMFtAqcHT9uVXQNNBKnX96p2eFRE50tpjgmbJr5vNxRkyc4MK8i26Fue3gnhhc9mpETIj9mfwdeBKiO6aPl8xCpOlUiN0FifuR5q9ix+7r17l2oIKfB+Wla0EznLQWviMdI7P/NNBHZF1svYAfgl9CGcN3yXw+1vgOoVAwBFCBz+2O2XxscAWDy96NJLMpKrphN+5YN5Gbsj7+Jo7iPwnR3yHF+u4jSh+MqnwIvJ8CbSgEzkBVPQrmCqTxJGg4AuyN4DsNVfIDVNWDaKU/6vd57A0o5UdV/hP00YDmxKWUXIXyzdzdU9ujcVVJLns1Sgsj+iiwVuZc1cG7gzWJrdWF10Qco6Y+uE9zLIbyjEdV3pkzVBJpPMVJxgcgMUj+DwJ/2q0utR8HlGcSquaZbGJEr1NUyGWbuO+Qy16PKv89qEogiLMX0kCSSGZJ7zvxHEjBz0X5QRuwQ3MRSSKZBYi1YycW0rPJN6CC46r66I71swchdhxJ/A+JP4xY9bt8PKX8rlDoJf3yLimlTlJKLVNKrVRKXVXk9W8ppRYopT5USr2hlJrUH+O6uPQG5RkHNa+BMRRH35zJ1kg4BzFXbu92p4/SqxyjsAqBCgB+VPmvUUrv9TzsxAtI/Uyk+Xyk4Sjstp/vQGCah+IpO4oV5HEQqwG75ZvYW6Zg1x+GHX+413Pd1Yi5Hmk4Gon8DInciDQch6Re2d3TcsnSZ1WScn4ZdwDHAxuBOUqpJ0RkcU6zB0TkL9n2ZwC3ACf1dWwXl96irJWItYEuTx8BUkjsblTZTdu/Xx8CNS9D6hWwY+D7FEqv7nxdMouR5ItOTqHA6SitMu9+sVuc4j25bqaJ/wPfTPCfvP0H8B7s6MclQdfJwYcKfrFocxFBms8Haw1ggd0AkRsRrQrlP2774+1iJHIDSBu5pyBp/RHUvrVDwtZl19AfJ4YZwEoRWS1O7P5DwJm5DUQ6FKMAhCjc+ri47FrsJqD7gmPvUB4gpfwo/0mo4Nl5QsGOPYA0nQOxPyHtv0UajkfMbjaJ9Luguu/DEkjimV6ObaCqHnTSVqgg6Pugym/rOUjLXA72JiC3alwCif2jV+PtcjLzKFCNSXyHnQJcdg39YXweAmzI+XsjUJB4Xin1XeByHP+6Y4p1pJS6CLgIYPhw1/XOZduI1YQk/g3WZpT/WPAe3nNRHc8B5C+SAAGU/7S+zUGS0H4zXTEFKZA00v5rVMVfuhpqlRTuhwzQe2+jUPoQVOVdvWxtUeimCY5X1R6AMRoy3SraKSMbw+Gyu/nILDEicoeIjAZ+BPykhzZ3ishBInJQTY1bm9alZ8Ta5LhuRm+HxANI68VI+409tldaCFX+h6yNIAz4wH8CBD7dt4lYm6FAGAmk38du/ip2yyVIep5jvNaH4eyLwLEPeFHBL/dt/J4wJjrJAHN/4iqACn5p14y3g6jSH2fjMDw4c/Q7bqQf87xMewv9cWLYBAzL+Xto9lpPPAT8uR/GdfkEI9G/gETpPAVIAuIPIqFvoPTawvbpd5HIr5y4Bm0wlP0YrT907foQiu7MpR3SbwEKSb0EwXOh9NeQehJSr4I+yikbagxHrAZH1WWMQRWom3YOpRRU3Iu0/QAyHziLcOjbTvbYPQDlmQLVTyPx/4DEUIFTUZ59d/e0XLL0x7dwDjBWKTUKRyCcA5yX20ApNVZEVmT/PBVYgYtLX8gspkA1pLxgrYNugkGsOqT5G3RGP9sboO0HiOfFPFvBzqCUFym5FiI/wVEV6dlxOuYmQAbi/4T4w1B6LVr1lc4rYmG3XgnJZ5y6B3ig4g6Ud3qf5tQ5N2MoquohJ/obfcdrV+9ilD4YVXLx7p6GSxH6rEoS51t3MfAcsAR4REQWKaWuy3ogAVyslFqklPoQx86w89VPXFwgm+qgW+0AyYAxvrBt8mkKdOtiQ/LZnRpaJIHd9lPsrQdgb50Jdh1U/Q9V8gMIXwoUcyF1vKCIXIfYcedK/GFIPg+knYA1aUVavlm0/kJfUMrY44SCy55Nv5xbReRp4Olu136W8+9L+2Mcl08uYm11XC6NcSjlRYW+gSSfd66JBQiU/BjVkWgu7+bugWE47QsCxno5l9YfOm6rHa6n0b8AHlT46yCCxP8Bdg/eTkp3XEi1yU7kcvcsquB47PTTqaG3iN0M5gZHnfVxqjrnsktwU2K47NGIWEjbjyD5nOO1IgoJfxdF3NGZayGU3Qzew1DGsOKd+E+B6B/Jy1aqNPCfhEgKid0PqZfB2AcV+nrP/eBUTiM1K78vnNrIhL+e1e3/EWm+IJvuuftJJQ36UOffWjWOfSLHW0nMj9QzR0SQ9t9C/F5HFScmUvoLtGAfjfIuH2tcweCyRyPxRyH5Ao4baMcO/WYE5UQgawOg6j/b3OUqYyhU/BmJ/BSsjaDVggohTZ93ThvS6vSfeQ9JPgnVTzgBbcVnRPEwnC4BoDxTofYtJ0Yhci2OEMk4KTRC33CC4AAVvghJvUqXq6sPPFNRxphevz99Jv0GJP6Fo87KqrAiP0N8h6D0gR/dPFz2KNzEIS67BMksQ2L3IcmXssbPnST5JEXVLYizI7c2I4lHttuN8h2GVjMLSn4FdrOjzrHrQLbSFY1sgbQjTV8sDFDr6EcLgz6y21UN/Pk7bKW8aMEzUTUvQvgSCJ6PqrgbLdxlbFWeKajKe8F7uJP9M3TBDsQp9A+SfCEbTZ07eQ1Sb3yk83DZs3BPDC79jh29HaJ3AbbjbaMPgcqHd053rddCRqNne0ASMh8WXJXky0jsASADgU+jPJOQ5DMQ+9M2+up4gDqk6QtQ87IjCHL7tZvBWt/tBgVa8ROG0qtQ4Yt6HEp590dV/m3b89mVaLU4sRW5Bm8tGwPh8knFPTG49CtibYbonTjqkay3jblup0spqtA32Pb+xZ+Nau7Cjt2LtF4CmVch8xZErkSaToPYH+m1wVkykHq+8Hr6g2wRnVwsSL/cu373MFTwc9lEfB1LgccRCr4jdue0kMxi7JbvYDeeid1+O9L9VOOyS3EFg0v/klmY9cnPJeUUmO8BsRqwm7+OvWUS9taDsWP3dr2oD6B4agcF+EEflleNS8SG6G0U1ETe4fRcdqGKBUAflPWCysUAYwSSmY/deDb2lmnYjWcjmQU7OOZHj9IHoKr+4wS+GeMg+EVU1aP9Fmi3M0hmOdJ0LqReAnMJxO5Emr++2+bzScQVDC79izGmSD1dL/QQ1SoiSMvXIP0mYIK0QPstSOJJ5/XE8/S8qGfdVHOEgFgbsxHRvUWHkmvpSlWRg+/YgkvKMwm8+wP+7BXDMYIHzkSavwLmAiAB5gKk+cs7XndhN6CMEWjlv0OrfhKt9GrUbs5XJLG7cD7Tjs89BZmFSGbZbpzVJwtXMLj0K8oYBYHTgUD2SgC0ClToK8VvsFY7/vPds4DG70OsJojeQr7+u4NsRLG1xnG37KD18h2cse7YKEqvd9Ja4wVV4WQu7cErR1XcCSVXOlXgAl9AVf0XlX6/8CQhVja4rsjsrU1I/BHHFtIX43z3fkWcU9PHGWsrBSo/pbuZVz9CXOOzS7+jSn/pxAik3syqes4qMOJ20dNpwEaidzg5h7aJBclXoAzEbgVz8XbadycNyScg/D2ofgGF7QiybdQEcALsvgyhrgR4kspQaL/ICq9u2PFHIHI9Tg1iDbQaqHoEpZXv4Nyzo0gaifwGEg/h7LQV4jsaVXbTTve5W/Gfmk3LnavKs8Gz/+6a0ScO98Tg0u8opVC+I9FKf4wW+tI2hAKOm6Y+mPxaCQZYzZB4jMJU2UWQRuzITYho7NxX2obGY6FhJtJ0FmJ29zrqBb4TKaz3oGWv50zVjmaFQgpIOMZ5qw6J3cnOIpHrsrEIHSo1gdQrSGv/JRwQSSLx/2C3/xZJvbJLTyUq+FnwHwf4nEy4Kowqvx2lBXfZmC75uCcGl48ckbSzECaecjxgwhdD4pGsgVoHBOyNO9KjE3mcmu14KGU+pCtobAex66Hp08iAd3coBbQyhkH5bUjkJ06chFaJKvulE1yXi7nMMc5LrnE8Dam3oGTHpytiQeJxCgWoDen3EDtSPE3Ijoxhx5Cmz2SLGiWQ+D/BezSq4vd96rcnlNJR5b9DrC2OWskz0U3H/RHjCgaXjxxpvTQbQJUCaxW0zUdV3ImU3wX1B7Fzi7oJ1mKwHE+lTgO0XU+vTh15JJyU2b6jdugu5T8afG+AxB37SPp5xFwH/lNQera+iD68K8K4Ew08E3dwjh1sL+dT35PndRRD6vxcJAGpWUhmEcozuc/994TSB4Ibfb1bcFVJLh8pYm2B1Ovku5MmkdifUKQpppPfMZJgbYCy34H/s+y4UAAQpO06JPU6YjU6c+4lSimnjGfTqU6Zz/YbkYYjsRNPOK/rNRD8ouPJBDjG7jAq9J2dmKeTORXfMRT+lDUnf5S2E8eQ7mQWUCCsleacflz2SlzB4NIjImkkPQfJLEGkn8p0221Fah8DVlM23cRY8ne5Bmgj2eGvasv5EL+9D/PciLRciDQcgTQcj934acSq77G5mGuR5MvYZgMSuQZH8Nk4O3oL2q5EzJUAqJKrUOV/hMC5EP4uqvrZbSbu2x6q7CbwHJl/0TMDVX7LTveZ39dBdHmZZREbjD23sI5YW5HUbMRuyb+emo3deCr2lqnYTed1fiYu+ah++8H3MwcddJC8995722/oskuQzAInQyiW85++D6ry3r7rq8VCGo7o5nroh/C3nbQSkZ/i7E4FpybyROff1sIivXXLTLpL0cHYD4whTh0H5YHAF536C5Gru65JmuLutUDgs2hlPZcf7Qt22zWQ+B95O3ttOKrs56g+RjGLpJCmz4G5gs4TmD7WeV5loIIXoAKn9GmM/sLJFvtrpzCS8jmfR8n30UIXIOYapPFMut4jBaoEVfPqXpVqXCk1V0QO6ksf7onBpQAn6Oy7IG2Orl4SYC5H2n/X576V0lEVdzkumioI+MB3JPhOgsg1OC6KHYu9Dt59wVpUvDNtMFS9Bvokp5Yzfqe/XfK1tsCcmy3uk031Eb8P2n6Yk/01So9CAcDagtgR7OjfnEI/iWexzU19Tvcgdjsk/kuBusdej7R8F0nP6VP/4AU7N+AMsFY4rsGZ+Ujbj7Fj9/dxjH4i/Q7EH8D5jNqBFLTfipirkMR/yE+Dnj3NpT6e6Ux2Ja7x2aUQuw7s1m4XM47B0fqOU49A+cB3wrZdUXtAeSZDzetgrgStDKUPQGL/clJp55GG5P/R46nAbkbJBqh+zImctjYixhRoPnuH59R7chf+pJO2oVfGci94j0YaTwE74tyTeBjQEDxI6Guo8Pd3rtKaROjZyJxEonejKvtQ+CczP5uFticjd8JJThj64s6P0U84acyLCNrUm9nTXLdnEGGbwvwTiisYXApRZRRdjFUAaTg++4cG6kaoehhljHb8883FThEaVYYkngBrNco7E3xHg92MtN/s5P/XB6PCV6J8M7v61sqc6Na8Yb3ZH25PpJDUG2je6c6pA5Do3/v27D2i02Uz2Am8hzvpPuxW8hciG0hB7C4kfj+ij0CVXIHyHdb7vrXBzgmsJxff7QYJbgdJst1TmN3HMfoLfRDOyTFHWCsD9AEo70FI/MFur0nWeO+SiysYXApQWhgJnuPEFnSqOXyOf37uj0oUErke8Z/l2AY6dOzKyC7oCSTxKHiPBHOpUyQHE+wmpOWbUPVAl7uj/zhou7bbTDKO3cHe0MNMbYjdg618KBVCtFqI9l3d1YXXKa4jERzdevdduR+MSWC+v51+PKjym5G2q+h5d+rUgsBciLR8Gyr/ifJO69Usnapxf0JaLnRKnea9GEAF+niC8u5PQX3tPAzwfapvY/QTKnAmEv2TUwkPE/CCVgm+o1HKi5Td6AQYSpuzQSn79cczOnwX4woGl+KEvu0s5Om5Ts7+8Peg7YpujQTS8502uRXWcv30JZ4thamRr99NIbH7UOU3Z9slKcyIam9DKHSQhuhtCB46DeVF6VjUO3b8huNtI5lsGo3u6gcFVf+B5s/nP29nXz4IngvaIIguKjL3bA0JFYDAuSitDPEe5qg0tqt6SiHxv6O8v99Ou1xsMKY52UjtNjrdfgOfh0BhmU6x6pD4v8CqQ/lOAP9JKFX8VKCUFyr/4cSfWJucZ1PB7GcmYIxFlf1iB+a661BaGVT/HxK900lo6D3YKdeaDZDTAqch/lNxPgP/zqnuPgG4gsGlALHboemMLrWHlYTYHRQWdAH0KsfDSLovjLkU8x6ywVyWdYMV54fc21oJhTMunFcBQRzBlML52hsQOA3lOxJp+izYPVWJK6ZSq0bVvoFSCrGbkdjtWWEoOAIhCIHTQCIo/6ngO865Lfg5JPl/jm1F4tt+HjuynefJaZ1Z7KSp7vTm8oFnGqrir0W9bcRc60QySxIwkeTLkH4DtQ2PKeWZCNXPgbRmDf0eJwEiHpQxvNdz/ShQ+kBU2c96fl0pCtxvXfJwBYNLAZL4T3Zh6lhsk07VssC5kLg/e113VEehC6H9pu302LEYd8NcgcTuAHM1JJ8n/0TR36Ry+s+qGSI/R9AobrjVQCvPqsW6Xffu17nTVFolVD2CRH7t7NY9U1ElPyy+WNqtoA0DWUHhCSaXHVP/SPSvdAmF7LNm5jtR39qoIu3/nBVMHYI4AYn/IeFLtlnnWSkFKicltzG65zmJQPo1JPUG6MOziRT7IdjO5SPBFQwuhZjrKFB3iAWZOY5qRJWC71Oo8MWgDUTiD4G5ikJ1Sl6nRa5lIHoHjmF3V3uGFDuNbEv1ZEPbVVDyE4j8jM64CuVFlfwor6UyRqMq/7rN0UXSSONnQBq22Q40CJwN/h2IC7C3UCBglJGNFSkUDJgrKUxr7XFUh/2UgkIi1zgpxyUO+JHYX6H6CUeQuuzx9IvDt1LqJKXUMqXUSqXUVUVev1wptVgpNV8p9ZJSakR/jOuyiyjqV58Cc1HWs6beCaayW51MqpX3Q+gi0AbsxGAWvf8aKgp3973VEe+omkog/R5YG1HVT6NKrkSV/gxVM2vnVCepV533rjek3+r9LDMrshHIvu6vIMa+2PFHsJu+iN1yCZKZ77zkO7KwvZhgTOj1uNuck7nW+X50qsuSYLciuZX5XPZo+nxiUE7i+juA44GNwByl1BMikpsY/wPgIBGJK6W+Dfwa+EJfx3bpf8RuhuT/iryikb+7TiPx+1Blv3L02MHPOrvCnaK3p4ViahfBSUu6K9wlU5B8BlXyPTDO73lWIkjsboj/3bE1+E9DlV4F5mqk7adgLgetgt7lbbKdE4C5BDyTeh7TjiItF0BmmZO3CBvwZOs3gyq/HWm/ARJP4hjWFZJ6BSr/hgpdiKRmgbUOUI5QKL1hp2JSimKuKp5B9mNQ6tTFoT9USTOAlSKyGkAp9RBwJtApGEQkN7RwNvClfhjXZVeQWeIUuy/IANodO993PbOwl/d1xwfGlKzL504an5UGsovSY+i1nf8UO+6kC0+9CPpQVPh7KM9kJP53iP2x66SVeAyxt0J6thMhDVl1T29RdBciYtUh7b93Cth4puFUnluME+GbbaRVQdkdKO++ThR2yzfpErqCE+z2B7TK+6Dq/yDzHlj14J2J0qtyxqoHuwWMMdssWNQjnslFvgc+8M4s2txlz6M/BMMQINencCNw8DbaXwg8U+wFpdRFwEUAw4fvWZ4OnxiMfSis2dzxNem2cFv1SPI58J3gGCIL7usNKTDn06dFXSJ9u39bqDJnCJHsDj3rmmouR1IvI9rArIDMVb+lIPUKhb7/vRFeyhnTcOI7JPU60nYd2Ou6mljrsv1068uOo/Rqx1/fas4azrst0NZmZxSlwJsfDS2SQVp/4ERzq2wt6/K/9DqeovMJ9IFI+FsQvdOZozJAH4IK7v7IaJfe8ZEan5VSXwIOAopGw4jIncCd4CTR+win5pJF6YOQ4Och8e/sDtjv+Kx798+mf8jB/BBp/T7oI1ClV4P/5GwKi2IdV4E0U3xh7KvheRd+VVLPYDe1QOjrzmkqz8AuYG/expy6z0uDwNeyKauXALqTurrzNCGOm2nZb1FKQzLLnZxVBXEPNsVtK5bjSQWgj8KJAM51i/WC7zjEXOXEOnj2zSuAI7H7snmD0o5AkTjSchHUvumk994BtPDFiO9E59SkDwXfETvch8vuoz8+qU1Abs7godlreSiljgOuAT4lsk2nd5fdiNjtzm41oDuCQd8HFfw00nhGD3eYYK1yFjD/6ThfqSIeSCXXQuR7/TjTHsbZFWTegcgqem8kN8Az1Vn882SDB/wzoeV75J8wFKhqKPsVmr8rfbYkHqFnoalwvLk63oMABM7szH6rlA4VdzgLO+J4lRljIPMB0nh/NvW5DhV3d50Ikt2ys4IzvrkEPFN6+ew5M/SMBc/YHb7PZffTH4JhDjBWKTUKRyCcA5yX20AptT/wV+AkEek5qb3LbkUyy5Dm85xFRDkrmqq4G6WVI1rJdvTkSUg+2sNrGmTepn/TZPe3UCgDYj30K2BH6Z3xWDnRtuW/g8wypO1HYG8FVY4quxGJ3k1hlLU4bqytFyO1r3S5dMq2ntELpddB6nnHHhD4NCrwWec2sZyo5sRjYIwF37FOIF/yCYjdS65dQlq/BTVvAFpx+5CYoMp78dwuexN9FgwiYiqlLgaew9nC/E1EFimlrgPeE5EngN8AYeDRbGDQehHpaQvqspuQtp/SmXCtY+Fo+xFUv4QKX4y0/oidK7tpQ+JRdqnKp8+0bef1BM7PRVHooZWLhgp/B6VVYmcWgN0EBBy1TOIRJ01Dj1hI7H7HCwpQgU9nU0V3f899UHIlyn9s1hbgceJKsiktJHItJJ6gUwBlFoMxMhtE2G3xlwRYa5H0fCeIsduz4J3RpyJCLh9P3EI9LgCIJJCt+1FML64GfIhSfiT5nJOgzFxJ30tw7ilUAL2MLwBAc+o260OdTLFFCYAxDsx5hddVCKSx6F0OHgh+Fa30SgDs+H+h/UaQNtCGQvhilP84x/jdcgFdtgYvquph0KqR+pkUCABjnGPUznSvzeBF1b6GNH85W4in27NWz0IzBm9jvi57Gv1RqMe1Bn1CEZFsrhsd9MFI83cpuqNXpXQEQ4lY2WjavUQo6OOy78GOYDs1pSsfgoZDKe5imygiFLLX9XFgbkswZLDif6dJP4Wa0GS04JlI4AwgjVIdn4MgbVd2y7cURyI/Q5X9podpt6LKf440X0jXCSQAgVNQWiXFzX4exwhurs8m3NvgxCZYW8Az2XHX3cPyJLn0D65g+AQi1hak+avODx0L52vQw2LvmQqJR7D1gU6KiJ1SJe2J+LJCbmdsFRbEH3XqIPRUA6En9OEQvgjafp710ioiWMTktoW/YUjFeXx73AmAjUT/6izOWOA9KZvCPO8mJ8ZBqwV9cI5LKzjeSCeivNOh8j4nENFuBv+ZqGA2zjTwuWx6ko7PVwNjFNjtSPPnssFqOeozazWSehmqn8uLgXDZO3AFwycQab2i2055G+6i6TnZ0pBFql99bCjJ2r2TdAnAFMi2agxsh9jvofwuaL1gx+5LPQ/eaWgDnLQXdsPx2UW8Cw2ImooH177J0QMmM057DGJ307lop/5N0dOdPqyrNkPzhY76SSzw7IcquRwA5d0P5f1zwa0qdCFibXbclLGcZIDltyGRG7Juy0Wy40oKSTyOCn99x94Dlz0eVzB8whCxnIjXXtO3esS7Hy+q4o/gPQBpOg/MhXQtctE+9GtB2w924r4UtP8OCX4OpQKgCjOO2kDMMjDF5r3m1YwL3k/+Sa2YXVBDlVwNgDLGQM3Lji1IBVHG0O3OSikDVXYtUno1SLozPYaYq3sYDyBdWBjIZa9gV1RNd9nDEGsrklmCiInzkfdhp/yxw4Pog5CW72Q9gootcspRC+mT2aGfhDTv3JSUAmur8+/AOTiBaHmz4Zrh71Fq2AwMlNMrN1mtFuU7NGcIDeUZ1yuhkD81b37OJN+R9Px98TuGcJe9DlcwfEwRO4Yd/Qt201ewIzc6+W26t5EMdutlSMOxSPN5SP1hkPnA0Sd/YkhD48mQfn0bbbIpqq0l9F1dprP9jK+aYwcAVPCz4Ds2T1xpCmq9ST5bs4kjaydmK7B1z56ai3fH0nTvACr8HcfVVYWccQBHkPkg/A3HbuGy1+Gqkj6GiFhI87lgrsEpyjIHiT+GlFyO8n2qc5co8fshOYvOFAfEnFrL2yiwsvfRGw+qntuIOBv8XuM9wwnmk9xgQOXkHRIbECj9ZWcqCqU08B2RLX/apbbzaxZfq/0A1XQyBL4I/lOzkck2ePZ3MrZiOwFonkmocH9GlefMXCuFqv9Beg5IM2JMRUmzY89wayXvtbiC4eNI+u1sMFKHi6EFtEP7jUj7TUj4MrTwhU5O/IKCO1Hn1ODSK3a4JHD68SIXBaiCwKcgdCGaMaTrFbGyp4VC7yhFwjFMR3/tVE7zHQ/hb6F5JiCSdqq0aeWOTWEX4givbF7MzGIkdg9YWxH/yajguSi1Z6omxY4675E+EGXss7un87HCFQwfR6zNFK1F3LHzjd6KLYkikazQu7QOewsenOfdA7ypZINTFjU9B6l6EKWFHdtP87lOSottRoWbTsqM1DOQehE7fCkq8BmU14lhshNPQ/QWxwXVeziq9Oe7xIVUMoscA35HGdHMIiQ9B1Vxe7+P1VfsxPOOc4AyQEzEdwiq/I97rBDrQNLzkMSDIBYq+PndpqpzbQwfR7wHs+3FzoLYnx13xU80GXojFD664H8Ba4Wz4wYkcn1WyPeUo6nI/aQheivScDSSehVJvQltP3Y2ARKF1ItIy5fZFRkNpDPOoaPvJKReQay6fh+rL4gdzXqMJZ33hCSk3kbiD+/uqW0TO/GCE4GeeByS/0WaL8SOP7Zb5uIKho8hyhgOJVfgGCSL7YAs9pro5I+A3qiLRPpLgNiQnIWkXsuW8NyZE5wJJJHWHyKxv5HvUmyCtSmb1rufseooONkoA6w9zGU182E2e2wuyWyuqD2Y6I0UCN72m3eJkN8ermD4mKKFvoqqeQXKbspmv9xRZfgnGxFIWqrHxT5XEKRsjahlsDJRit0fv1FrGdJ6WXY32wck0ZX0MI8eMqX2Ff8JdHetBQWeif0/Vl/QBhTJTKvBnp4MsIhnoXPq/+jVv65g+Bij9Cq0wOlQ+U9cwbBjWCjejAwkI8V/Akplww0EViTK+PbKTzHYl0ZTFUB5H0e3eyEUFIQugfD1QICiP1WlQfA8x+Mp9z4V2qn6CdtDhS4E74E4xZvCoEJZvb13u/d+lCjPWPAeRJcQ00D5UaE9PELbsx8Fv2Nj3G4pcOQan/dARARJ/Btif3XSOPjPQJVc1vkDFHMl0v4nyLzvVFfzHY/jY7635DHatYiAhnBU+WYUYIvzcyymUtIVjA20MbNkK883D+LTNRv5SNR0KoTyTnXqKARPR5JPQ+Q6HDWShRNH8AOU/0zE2gixu5x8RqoGlA9pPAUJnocKfrkzHXefp6R8qMq/O9HQdhN4pqBU9xPEnoGq+AsSf8BRHxnDUaGLUMao3T2tbaLKfuk4I3QmNNR7Toq4q+fipt3e87Djj0LkBrp0xz7wH4dWfqvjtdD8RfLzGykcwbB3F8bb4ZiCfuw3Yem0mV4G+j6qFCF+VPV/8lxRxdzgbBjsdlTgdJR3/67XxESidzqbic7vjR+C56OVXvERzdmlO3b8iS6PMd+hqNLrUHptj+1F0o47uthOe7WtwMbi9EfabVcw7IEUS6wGHlTtO0jzl8BcXOSu/qyO9vFgVwmKnvrf1eN1YYDvMLSKu3boLnvrzCJpOvyoAfNQfZh4azrGy1sXAXDUgElUeMPbucMFQFJvZGt2dwhq3amPXv1Mnz6P7eHWY9hbkeK7UrE29iAUYG8SCjuyAO+KxdoSR6O/0/1qw7NlULdnAPZk/4vnXzbGoMr/VNBarM2OgdIzofhOUuKF10jTlVp9x1nQup6L5/yt0zPm90uf5vaDLmBqhVuHYXtI7B/ke4xZzvfCXAyeybtpVr3DNT7vifjPJD83jgZaDbRetpsmtOfSn0JBxMlqaosq2u/2xuo4fGesLbRY5Yhs5wb/CRQKdC/4js8LxBIxsVu/jzScgLR8Damf6bi7FvR3DPnuywZ4D+mT8fKXC/9DwkqTtDMk7QwJK80NC3ePb/3HjqKeYQpkz3cldwXDHogquRT8x9O5o1RBx9i3w9XG9m66q3l6S7GYBBGoz/j5xbqDdkrY5J5cPCpNiVaPvU1PsQCU3gy+Q3K8inyglaJCX8zvO/7vbM6rlOPNJDGk5WLEzj8hqNJfZD1bvE5fxsQ+Gy/XRAtjFNbFGnaLb/3HDRU6F8ejrPOK4821CzzG+htXlbQHopQXSq9HCEDyqb77u3/M2FXq147Fu6fTQI0nyQ+Gfojqtovvjbqq++s6oFSxqGvHFtRmlfDsyj9REryQ4ypOxJt5G4zRThoErSL/luTTFNbFSCLxB/KK5CitDFV1fzbTro3SB2570r1gSKCCjYl8u8WgQMUu1ZHvNfhOgvAGiP3J8S40JqLKb0EpfXfPbLu4gmEPQsSC9Gywm5DoX8HqXpx970XEUapoO7nedF+nii3mxday3HaagjIj3ePr26J7u+73ZMRDwg5RorWiFHhpYqb3Pr65pI7b9TLuPeTqbO2FIuiDinvIxu5CQhcUuKNuy+tlR/nR5DO58v1/YYoTZGUonasmn9Vv/e/NKKVQ4YuQ0NeBzE55GO0uXMGwmxBJIO23QfJZ0CogeEHWra0p2+KTE5NgCbwTGUh9JsD0knqG+GI7dH9vhUCx+7rjVTaWsM14wWLjWXh5qnEQf9s6mW8PWsDxFRvQc9pYtoVXRTrvC+gWtST4VNl6nmsZyV0rnuKnU/NVSJ3PEroQSRbJ2ioxxw1Sr972g/aBg6vH8sDhl/Bs3YcAnDR4P4YG3RrPO4IjuD8+QgFcd9Xdht38NSfH/XY9V/Ze6tN+Wk0fP183g62ZILYobBTfHTSfz9es6nfDcm9cT1szHpYnKhjmjzLIGy/arlhA3Go5jy/PS2GhGBto5c4xLxPQnV122la0mj5K9TR+PV+9VJcKUutNAArDfyKq/CYcA6WJ0sK8XfcfqtO/ZoSvhYytMJSNp+OAoIKo2nf3uMhjl93LHhPHoJQ6CbgNR7V6t4jc1O31I4HfA1OBc0Tk39vrc28WDGLVIQ0nsrcHpPWELXDJqiOYH6vGErAc59DO173K4onJT1Fu9F5obk/lY9mgb8fV4vHGUdy6aT88yiIjOidVrON7g+YRNqzOvkXg3fYaDi7NN8qmbA9Xr53OmxGnMtvUUCPfGbSAgd44b0UG8lLLMH47+k0CWlfem0L1mccpMWo7yepiagzKXEFQz78HnBrN6IMd19bQhW4lNZdO9og4BuVYUu4Ajgc2AnOUUk+ISK7D/Xrgq8DOVE/f+7CjoPS9KfRgh3i2eTjzYtVkpLgRzlA270WqGeqPM9Ab36aAyNhw15Z9+V/zSE4qX8e3Bi/Cq2yUcgSQKQqPku2+1VvTAW7dtB9p0Uln5/VC6zBmlNQzIdCc3dXDykQZs1qHsn+4Ca/Wtfv3aRnOrFrTKRjmx6r5/qrDsVGkxPmZPdM8gpMr1qErG68mjiE8/2nA7gpsDNjLEC1/5kp1CAfTSbVtrUdSb0H5bSj/0dt5SheX3tEf7qozgJUislpE0sBDwJm5DURkrYjMZ4+omLIHYIwBQrt7FruN39dN61EoANR44oQMi1pPnBWJMh7YOrZHF1NDQVhL02L6eaxpNDesd9xNTRvWWEcwN+oEYumq8N5c3ovWonfzIkrYHt5uH8i6dA3XrpvBeUtP4IIVx7I2VVaQfE9EYUrXPkthU2Jk+N7g+fiUiUL4/aZpXLb6cNYnw/x24zTS9rZ/fpqSoj/QwpNREoness2+XFx2hP4wPg8BNuT8vRE4eGc6UkpdBFwEMHz43htZqZSG+E+CxD9391R2C2E9Q8TKNcYJfmUys2QrCVvn+pHvEtIz6ArKjQYmBVuYG63moJLGvH46Fsgv1K7kP02j2ZIJ8UrbEOKWjlezGG3MZnRp7+o5j/C1F9ibvcpisCfBYcOv4pHG/2NjugSA+bEqWk0vPs3CUJKdi4+kcTgBLUHK1hgXaOO6ke8w3Bdl31AbL6UupyT9L06oWMMlqz5FXSrISF87p1Wtw6/1Q1rlPagmgkgCUm+C8mYD7PbsqmkuhexRXkkicidwJzg2ht08nV2LuWh3z2C38fUBi7h+w3TEMeF2/rdfuIGJwRY8yu7c4WvKUS01mz1n8cyIxsRgC1vaQkj270AvihXl7rwnB5uZGGhhQbyStBjo2Pg0i7OqV4F3Oo32QmArAILiWyuP5sdD5zKztBFNHwAlV3NqWYYTQ1djShJfjpppQvkkJlV+Dts8mDnrr6Yx48dE59ZN+xG3Dc6oWotXWQQ9Q8HeSpdHmo9GGY2dWcsAb7F0Fx0Y4D9qm8/6USGZ+UjzV7suqCBUPoQyhu5QP1uTbUQycfYJD0Dvp+ywLr2nP97xTUBuBYyh2WsugNhx7NhD2JEbkOTz2OYGJPE4e5hM7ne25dNwcGl9dqfdtTInxeA/TaMpNTJoKv9mXdlUGMke+zSUsCZZio7NhEALAW3bZTJFwOqWrkIpuGX0G5w/YClTQ42cWbWGf45/gSqPiUq+wLED98WrdX1mDZkAv1g/HRsBu9mpsew7El1TeUIBvOCdiVhNaMZwIoEfQtaf3ULjT5unctLCM7hpw3SoehRKrgRtEKgKCJ5D7aCHGDT8Qyi9zbmODsZYUKVO3QUVAGMsquSqbT5zfyGSwI7cjF1/FHbjp5HkrJzXBGm9Ihudnf3PbkQi1/a6/7Rt8oP3/8nZr/2Ob8z+K6e+fBOL2zb2/4O4bJP+WJ3mAGOVUqNwBMI5wHn90O/HHrGjSNNZ2WN+wskPj53dKX/0VZk+CuKWzqONY3g3UoslinozSECzOK9mGSdXrkdTzo67mANRQybA620DOb1yHT6ty+BsisYgT4yMaHiVjSmwKFZJUDOJ2wbrkmE2psNMDDZz2eB56NswNYs4Rmm7SDZan2Zz4cClXDhwaf49qdf40rDvM69lLR+2rMMggaaEX496C0NZQAJpvwWt6l9Q8Q+k7YdgrcVx0hOI/RmJ/gEp/Qkzqk7HEo1cc5tfMzl+yHFoejmEvuz81w0teDIET855DhMy85wduTGhaCSymBuQ+D/B2ozynwj+U/pcm0FaLob0u0AK7DqnEl3F7Sjfp5y4Cqv7ntCGzNxe9//P1a/xTuMK0rZJGohbaS6fex9PHX2Ve3L4COmzYBARUyl1MfAczi/hbyKySCl1HfCeiDyhlJoOPA5UAKcrpX4hInt2esGdQETAWgN4UMYwJPFYtlxfh2qgNwXf9yy63CO338YGvrHiaDamwllPnK5TwW82HkBaND5dvZZqT5LxgRYWxyuzrqoAQsL2cM+WyTzRtA/3jJuFX7MQFFFT5w91jtrlogELsZXG+EArAd3CtBVjA21MDLayjz+Crm3bdVUpx1Op9xiQmoU39Qq3DVdsHPE1Wtv/w/jAFjy5HkOWY2ZT3mmomuew2/8GsVtx8htlVVqRGyitOZYb9zuXn817EJEkpig+W13PETXjdmBOWXdV74E9vi7mKqTps9miL2ZnjWlVduMOjZPXp7WlSyh0kkSif3EEgwqA8hemcNEG9XqM5zfPJ2Xn/04SVpo10XrGlPQ9xYdL7+gXESwiT4vIOBEZLSK/zF77mYg8kf33HBEZKiIhEanaK4WCtQlpPAlp/DTSeAp20xcg/QEf9whmS8AssgHvrtbJCLzVNpC6dKjTPbO7qugfW53awGlbcc3w9/jWoIWEtDS5AiQlBk2mn7cjA/FpNn7NosqT5pLB8whpGa7bMIMJwdbO4DFDE3zKIm4bnXEK281rhOLvm8djSfGEel1oOMI8BSRA4gy172HfcCxPKKxPlfFW/AiaU1FEMthtV0PsZgriVJQHMh9wWEWIpyb/l7vGzuLJyU9y8eC3oO2biLV12xPfASR6RzZ9e8cim4DEE87iXqy9uRFJPIlkFvScIE+iTjnR7thtAE4OoPAVdCWO0wA/qvTHvZ53la+w1oMlNmWeYK/7cOk7e7ei+yNEWi/LFtfJqggyHwAf7MYZ9Z6edtgbkiGeah5BSE9zbu2qTg+c3PvAuff+reP5V/14knbPX6m47cneoxjpjzLIu5J2y8N99RPy2iVtgw2pMLY4xmcbxwA9u30gk0PNBTt+XaPAriACyxLlbE0HmRpupCIbC5G0dFYkSjl/4HJStoauBK8S1ifDxG2DsYHWTsN3zC7FoB1fntdQCrzHQPoVbNH4xbp9eaV1MIZukFn+a74zPMjnyp8ors4SiwQDeGrFzzm9PMXoQE5iPLEh+RyEvtLj+7dDmKsp8A5XXrA2Q7fkenb0zxD9E52qL+90qPhLYbpufR/H9pFXL8QPgU93/qWFvogYo51Kc8qPCp6H8kzq9bQvHH0MC1o3kLKdU5ZPMzikehw1/tJe9+HSd1zB0A+IJCCzkD0lTCNlKV5uG8pBJfVUe3qOrjZthSDYaHiyWvcOAfF22wCuWnsIGdHQEP6xZRJ/GfsKY4ORzvtzhckbkcHE7G2lZhAmBZqwoTMwzKfZHBSu5/768Vg5pwufZnFAuJGEbWDaQkq8fGHpidgolsQrCxbdpK3zausQxgacXERpW+P7qw9jYawKQWGJ4pIhH/KFmtU0ZnyMC7bgy258Y5bBxSsPZ3miHF0JAc3kD6NfZ4Q/xpxImOkl7d2y3AikX4XKx3h202xebVtKSixSpiN47ljXwpEBg8G+7kF5PtAquPr9O/gwWs3tG8/g8NLNXDn0A8qMtGN3Mlcg9ceBtIDvSFTJTxyj9jZoSUd5vm4+7WaCowZM7lK3+I4CcyV5KVfEAmN8/tOY67NCIed7kp4DySchcFZeW6U0qLybRNO3aEu1UeNJoAVOQoW+lt/ONxPlm7nNeffEgVX7cNtB53P3ylm0pGOcOGgaXxx1eNd80x86xmxzNXjGoUqvRXn23amxXHrGFQz9gNh7VlpspYSHG8ZQaSSpMhp6VK1oSjhq/lnUeJI8NOFZLDRWxMup9SS4eeMBnSohGzBF55p1M3lk4vOd1x6rH81b7YO4cugHDPe1syReQYmWps3uWEpzB1YM9MbJiNYZSPZhtJqnmkdSYqRoN734NBtLFCeWr+eAcANx2+DvWyfxfOuITptFRhQ/WzeD60a+iyUKHWF+rIoHG8ZwYtUGhnmjPN44igWx6s4IZoBbN+3PuEAb+4ebyOWPdVNYmqhwAu4E4rbBj9YcyoPjn+Pgkk2kbY0kWl6OI5EEizZexs3LJ5PqFuimIbwfrWGwb11Oe4XShzCr2eLtSGXW+QBeaR3MpnSIv4+bhWChEg93dZR8Ckm/DzUv92gwXhut54LZfyFjm2Rsi3tXv8oVE0/jrGEzUKGvI6lXskZwJ/cSZTehtG4qmcz7RaLwE0jqdVQ3wQBw77qN3LPqMMAmZPi4Ydq5HNSHQkDFOKByH/40Y5+C62JtQVrO7zqxZOYjzV+GmpdQWmW/zuGTjisY+ojYUWg6iz3Jy8ij4LbRrxHWtz2nFtPHtwctYEZJPbPbB/KzdQej4ey4MxRGJm9Kdel/M6LzRPMoViVL+cqyY6nxJAjrGSqNFKm0QVLyv1oKm3mxriygTzUN5zcbDyCZXbw9yubUijV8pmYNo/ztgKM+erltWF4Mg4bN65EhnLrwNPYNNdGQCbAmWYbCRhdn8X4zMihPKHTwu437868JL+Zde6FlWLcobEV9JkCr7SNhGVy55hDuG/9SwbOM9y3ny7Ua99WPJ53zrBnRaMj4qU8HqPUmnPdSG0zIXs/99Ud0CgUAE53ViVLebBvIzLKWgvlib3V2777i8aJ/WPYsMTNFR8KPlG1y69KnOXnw/vj0MFQ97iz8diN4ZxTWeQDQRxTp2ZeNzs9nduMK/rbq5U41Tyod5/L37+Ppo35M2NNznEm/kXzKOfXkIpaToTjoOkL2J67/Vx8QO4pEfgH27o86zTWiKgUluoWmcnPrFLb3KpPPVq9ioDfBT9bOJGF7iNmerFAovGmwN4YINKT93LxhP1YlyxA0YraHtakyIpaPdakS0tL9ayXUeuKsTZXxjy0TSdkaf6iblhUeTnBbRnReiwxlhM8RCglL563IQLZmuna4XizG+FsBIW57eLd9IGuSZdkRFKVGCksUI/zt6AVqPcXmdP5ueXm8jJhdGJUrQEjL8LtN+1GXDmXdS/PRlXB2zSoCmoWR3RQowEbjb1sn8dklJ3HrxqkosQmwFTAL0mgApEXnqbbD0Itm2bVB2otcd1gWqaNYFqiGlKPuU0qhvAei/CcWFwrgVHwzpuZUkfOCVoIKnlPQ9Jm6D0ja+UGDGhrvNH00dUNEMhSqa8U5Dbn0K65g2EnEXI00HO3oYnfH+EUW+42prvxL2yoaA45e3qMEjwbzYtUFhmU6/fy7rm9Il3DovLP5yrJjOadmBU9O/h+leopclZGNVlDS0qcsQlqGaiPBP+oncsWqQ2mzCu0RjRk/r7cNYmGskru3TOSnaw/OGV+w0FierIAiURAeZXP9uhkkRee82hUENLNzwfYqk4CW4YBwA0mr6yv/TMuIIsuqMC3USFC3WByvJCMGf948mYSldyblk6xRvMJI89jEpzmvdjmTAk1oWZVPJpuI74nmfXg/VkvcEh6sH81RZZvwq65FTMNmsDfGDcMep/hPUTnBceZa7LbrsVsuQZLPdXoNTS4bWhARolC0pRPcsexZ7lk1iy2J1iL95rRXClV5N6rkJ9mKY99CVT9ZVJCUGAH0Iu99UP9oag0o/8kUKjlUtna2S3/iqpJ2EBFBUi9C5DcgbX3sy/l/f9QdUIqiu8fu43V4IHmU1el9U2Uk84y/2dYULsDCdwct4HM1K9GVYCjhU2Wb+F9zvj7Yp0xqPEk2psPUehJcPvgDZpZtRQGzIwP4xfrpjPG3sTJZ1jmGwmZisJlliQrWtZQwJtDK3WNf4va6acyNORXJrG6L536hBs6uXoWhbBbEKnmpZRgeZTPIG+dfE17goYaxrE+WML1kK6dVrUXD2el3YCgb3dHud13D5pQKJ9J2uK+dRfEqHm0cy4pEOSdXrsO04TPVazvbhw2L7wxexEMNaVbW1eYpFBO2zuz2AUwJNRHSTcYHWmmu8vFk0yjSojEp2Mz1I99FK1oCFAhfBdYGpPncbDyChaRfhcB7qNJruGTCybzfsoaMbWKKjYbixEFT+da7d5GxTXSlcd/q1/jT9AuZXD6s+BhkS8kGP4cKfq7HNgBnDz+YJza+h5U9NegoSjx+pleN3uZ9/YUyRkD575HIT50CRVo1quxX/VLC1CUfVzDsALa5FRpPAXo+3u8o/VmMJqxvJxUE8EG0igNLmzByVEzjAq2MC7SyNF5OWgxUZ3R2IWdWrc5L+fDDoR/wYsswEtKhkhFKdJOHJz7nVFno1s3BpVs5t2YF70drKNUzmOLseT3KZkMqzJpkKRnRqfQkGR+M8JUBy5i7ekDBPI4vX8fVw97Hpzkqs8NLN9NueXmkYQxnZ9Vj3xm0MJv6Qghkjcd2jnA8rXItjzaM6VQVKYSAbnFgeDO3bzkWhQe/MjFFMT9WxZJ4Bbfu83rRMqIDAsMwlEk6x5XWpywGehJ4NZuTKtbj1YT9wk1cNngeFlq31Bnd8YEykPbfZ42t2Q9LEhB/kK36F0lLgMeP+AGz6hcRzSQ5rGYcF87+S6cNwBaLjGVxy9KnuGfmt7YxVu8YFa7lj9Mv4A/LnmFjvJkDK/fhsgmnYGgfXQ1j5T8GfEfjxAf53drTuwhXMOwIbZewJwgFEUfTKjYY2U10wtIJ6YVJ43JjFBQwIdjSeV0A23bUIn8Y/Tr3bx3La5EhVBoJ3m4fXHTsn607mFtHv9n5t6GE4ys28ETzKBRQoqe5edRbBWmuO/BrNidUbOCerZMJqTQj/W2MD7TyTmQAW8xwpxrqiaZRzAhv5bnWYsZR4ftD5ncGuQF4NOG7gxdy0sLTeT9aw9Hlm2jI+Hm2eTgeTbhr7MuEdNOJixCncM9wX5TfjHqTWzbtR106zPhgC1cNfZ9yT5pSNrIwPoWh3gj7h5so0dOcWLGB8cHWos91eM0kKjesJZP2kBEdA4ugbnJy5Tp0pPNz0lQ2MeB2XZtTkH4r61WUfxKMWTbfn/NrViUrqPKG+dOMCxkZrqU1HSNpFX4H1kTrtzNW75laMYK7+0HI9AVHGAS2285l53EFw46QWbDLut5WGodiheaVQMTyUKpMMqJotzzUegujrLvfFzLsrj6AtGh8GK1gWriZrw1cytHlG3mpdShL4xW0WH66u5y+0z6Azyw6iRmlW/n6gMW0mD6OKt/Ej4a+T9J2jNYeTWjJGLzQMoz9S5oY44/kzWN+rAoQYuJhQrCViwcv5NKh80lYBr9YP4PZ7QOxULwRGURIy1DriVPfaYQWzq9dQoVRGJ9RpqfRsXm7fSBvt3elYfAqi0cbR/PVAcsAJ0o7aXsoMzJML23gwdIXAKdYz/+aRxIxvYwNtAKwMV1KU2uQ20e/zZjstcI3GTzpV/j72PU80DCUudFaJgRaOKdmRed77CtQFxk44n1bAsIPviMgvpHcTLECrEuVIAiN6XaumPtP/n3k5ZR6ApR4AjSnu9ynFTCpbMg2xnBxKcQVDEUQSUPyOSSzHMylkH6HLmNsf40B82JVbEyFmRJqZIg3hoazmxSBVsvLm22DCOgmh5Vuxt9tYdEUbMmEmBcLMCO8pahQ2F65S3DUJ7Nah1GhpxjmjzLUF+f8Acs4o2od5yw9gUQ3rx0bRV0mzP+aAjzbPByUQiFUGin+OOY1BnnjbE35KPdk+HT1Wl5uG8Klqw7nt6PeYlKolYSlcc+Wifxs2ByOKt+EP6sKAvBraW4a9TZfXXYMt45+k1I9jYZwufqQG9cfyAutQwlpJnaRXEcisCEV4tjyDbzSNjTPhTQtOu+11/LVAcsQgeXxCiaH8t1DVyZKuWjF0WREwxQNI+f9Vghz28vYx9+YV2az63XAWkiJAd8ctBiRxQiwJlnCF5cezyMTn6Mm7/PRIfAFlP84pPU7Tg9SJK22tRZVdq1Toc2uI2mZgMXP183Ic7HdnGyhIRWh1l/Gz6d8lh9+cD8AmlIYSucHE08v9tG7uPSIKxi6IZJEmj7nJEUr9mPtBzK24pJVR7I0UQ6AJRq37fMaU8PNgPBO+wCuWnMISjmafq+yuGfcywzxxTr7SFg6/9w6nrfbB7JfqJFb9nmzUPeNozbpSa3TMXbM9vCl5cfz+KRnGZAtYTnAm+DiQQv4Q93UnNxH0HGCsNCxRSFZ/fzmtM6166bz17GvUubJdOrPP1VWh6FsLlpxNPeNf5F208N1I95lQrAFr1YoaG2BHwz5kGojkZeP6Kph7/NamxOf8G50AOdbSwnl1kIG2i2DulSoIIbh81XL+dZgp/6FUjAm0MbSeDkTgq2dQumPdVOI2x3us+QtvCKOu233gjq2kBct3vkOZU9jowNJHt93Nu9lLuIIuRtD4URtG2NQJZejtBKoeRNJPgmRG6Cby2pdvI6nmp7hvKrxhDSd+dEAP1s9lFarK2bAp0xOqNhIWeqviDqMmdWH8viRV/Ba/RJ8uoejBkwmZHw0XkMuew+uYOiGxJ8Acz2Q2G7bHvvoYacuAk0ZL99beSTr0qXYWS+bSiPJpFALuhJsgV+sn+74+GfXxQQ6t26axm/3eQvTVmTQeKd9AK+0DcVG8WGsmsXZXXDH2B15hpKWQsvWF+5OxPTwattgZrUMwUbj6eYRfKGmq/j8GVVreKVtCEsSFYgtxMQLKIb52tmcDmLmLJ42GvNj1Yg4doQOfJrNEWWb0RS8317NIWVbqfYkiwoFcOIRhvvb8zOXAn7N4tf7vM01a2fyh9FvEDa6Fum0rbh5/f481Toqmy6j43SnOL5sHd8evCjPHrEwVsVP1x1MwvYwNtDCZUPmsTrHQyp3NpcMmsc5tSvzBECnNxnbOpH5IPglKkq+zwnKi8hF2TTZJWCM7zSaKi0MgS8gsbs6M7R2jNGWER5e9x6vbk5x3/hl7B/0cOWwBq5ZewgAYT3NveNeosqTxpN4D0neD/5TqS67kc8M36kiii4uwCdUMIjdisQfBnMlyndENk999q1IvUxvhEKHAThhG2hInoqh2GJhCaxKlHLh8mPIdHvbB3rjnYVjWkwfMStffSNozI9VIQIvtQ7lgYZxLEt0+Zkns3PoEAbg/D9jK7amAwzxJfLiFETg2eahXLdhRo73keLuLROZ1TqUe8e/iKYcg+7tY17Hzt66IFbFNWsP5o+jX+OC5cfQZOYbAIPbKJDj5M4TajyJHj2eLNsRaOuTYao8ybyTjlIwLdTIr0a+XVCb2asJJ1dt4KnWfXJcTx3h8LmaVXlCYV0yzJVrDuuMzF4Yr+KbK47GwlEZ5c4tqJmcWbWq8z3Nncv21XQaKnyR4wpK1iXUO915LySNJF8Buwm8h6OMYVBxN9L4GURizolDwT6BCN8evJA76qYwN1rL9JJ6Di/dzEBPjC2ZEJ+pWkWVJ9EliCUBicewPQehBT+zrcntFCkrg1czXE+gTwCfuAA3sVuQxlMh+kdI/hdp+ynS8m0nPsGOQPq17feRXRR05exk58eqWBjrOVfLc81D+Mayo/jK8uOLpppYES/Fq5zFq1RP5+m3OwjpGdotDxnRWJcqyZ8Pju6/+wKWEp376ifSZnmwsou7ZcO/G0bxiw0H45SaV+SqhzamQ7wdGYBpd+2MOzxppoSauH3064T1DN8YuDgvWMuvmXy2eiUZm7wi9ylb45XWIWRE43ebDuS4BWcSs4xOYQOO0ExaGigngd7UcLOzrGddS01b8c+t4zhmwad5LTIIn8pX6YjAPv5iMSWqYPF+snlktwhkp0FAmVQaSYJaBr8y8SmTnw6fQ8jYCbuSqkZV/KVokJjz/TsJafsREvkVdsMpfLj5bzy6qY6F0XxB69NsTq5YjwAtZldA4OiA86yTg815p7PsCBD5KZKet+Pz7oFlze8wb9VRsHUKWzfsz/t1f+y3vl32TD55giH+ANgRurJJJiD9DtJ8PlJ/CNurEwz5O0VDCQeGG/j5uhkF0cjtpofXWwfwqw3TWZysJncRzsVGoyHjLAoeTbh48Hz8ykRhY2BhKIszKlezNhGmLhXMCpHcwRSvtA7Ji+oF58N9sWUw16+fzmmLTuPkBafx1y2TmRetKjoPcE5Ad9Tty6mLTuOC5ccwq7XLo0VTMNLfjleZnFW9hl+MeJf9Qw1MDTZyXs0y7q0fzxuRQXg1m6StkbI1XmgZxi83HIizH1ekxeCC5cewJF7huI0KPFw/hozo6MrZ/Xs1uzOdx7pUmB+sOZQ7Nk/FFI1/N4wpmLlSjkAN64VpJf7dOIaE1SWMM3ZhZLamhJBh8sCE57hy6AccW76BrwxYwlFldUXfo5StMat1CC+35rr06hD8DlTPQtW+ifIdUnCfpN5GGs8CayNIDJEkP1pzAH9YMZtD+S7jg02kResmNBXDvFGOK3eC7rzK5pcj3+Ho8iYWxGsxpdihP4PE/lZ07jtKNB2lInYRk4J1eDSbWm+c8dzBmuan+6V/lz2TT54qyVxOQQEVkpCZvdNdtltetqSDJGydlOi83DqEVfFS4uLhiNJNqG26JAqHlm6m1pvoPIl8pnoN4wJtPNc8DKWEM6vWAHDZ6iOIWh7MIjl3Hm4cy4zSevYNNmHhZB29bt1BjPTH+CBa02lAfrB+LEHdpKfI5hPL1/Fq21CSYtCW8HPduukkLJ1Tq9Z3tupQ8XyqvI5PldeRsHSuXjsTUDRn1Ut+zSZjK4Z4o1wzbC4Z0Xi8cR8WxKvZkglz4YpjqdCT1BoxLKVzZk40cQeWwOON+3D+gGWcXb2aB+vH0mZ6acz4GOjL98LSVTZSeXMt1sIwtOuoygzP7juCWk+c8wcsJaiZfKpsE4837ZNnUBdReLBpN338ftM00qJhicZAT4ITKjbg0aQzMM4GZkcGcsOG6dgCpfobHFDS6tQdKLm0RzWLpF5BWi4ht3DTnGgt70VreGLy0znBidJ5uktYOv9tGslv93mzS0GmwK8srhv5IbGy/2K0nwt2kRLrdmvRefQWMdcisXvIxOdQoqfy1Ho+ZRFv/ztUntKnMVz2XD55gsGYBMyiUDj0nqStdR7hE5bOPVsmEtRNVibKuHT1EUwONPPrfd5CUPg1kwNLnuHC5ceyMV1YncpQNpcNmY+uulQngqMmmBxsds4FAucsO4HGTCBHB96Rx8j5Oy0631t1JKP8bVQbSRbGK6k2EtRngnmLYAaDNkunuFBYy8ttw/O8epJicM/WSZxatZ6UrTEvWs3WTIDTq9YBjk3g7i0Tebt9EDp2nhuoRxOm5aS5Prp8E79cfxAvtjrpGVosPy2Wr6j6TATqUkG+NcgxHIvAklg5DzSM5zNLTmVKqJHrRrzb6UUF8MPBH/ClR74AlnMyk80aVpOH+7wT+Gf9ODSEP4x+gyuHvs/tddNotzzUeBJELC8nV67n9ropRCxPVsUGN288kEpPkpklToCYhcISjXvrJ5CwnQjxZ5pHcGBJI8QfRdLvkSj9Gwvbm6nyljC6pCtiW9pvo3s1v0WxSg4MFyZg1JVjUH+4cQx3bZ7Mp6vXFKjEPESp8Plp5CFWbDiPkf5GBna+FwFUjo1B7GYk9i8wl4H3CFTwM522j2I4ZUHPBklRpqwCJ21NgeGaGfZqPjGCQSSJtHwnW7N226kjbHHy8uemmBCBlGho2GxMhUnaOpZoPNwwhtcjg/lq7WJu3nggCdvgmhFz84zRJXqGK4e+z6Wrj+w+K345YjaDvDFsgTXJUhKWzoRgS2ekrBPT4GNzOtTNaJsrILqur0mWsYYyNGw2pMM96AoLhYJfmXx/yDxeKhJpHLG8tJseXmodym110xCBWm+Cg0vqidsG61Ol6FgcXLKVicH8+IBcu0dAWVwyeH6nYOiYS8TysihWyf7hxk6PqpStc8P66RxVtomHGsfSlPFjoToX7QWxKi5ddQQPTni+01g7PtSKKsugTYtBiQVxhT0/TLgBEoPARuey1UdwYsV6fjDkfWo9cX689hBM0fAoi8WJys7+wQlMGx9o63Q/9SoBLH4w5AMuWHEsToanjgdM8FZzhGve/y2a5sOybfYtH8atB56PT/c4huZujPRH+CBaXXDdFpjbXstfNk8BYF2yhEnd4i7aLY0zXryFlG0R0A8jY6f4dNU6LhuyEBX6PPjPcD5Zu9mxqdntQBpSbyDJ/0Hlv3o+3UTvBEnSEXzX3diesHXCpV8teq/L3sEnxsYg0Tuc3Pak2V6ltYxofGP50Vy1ZiYZWzn+6srR71riZNn0K4sXW4YS0ExuHPk2jzWNYVWylJBmUm3kezVpCvYNNXefEVOC9RwQrmdTKuSkpdg0lfHB1k6h0HFvQDN7sAhAT7YCQwlTAw14lIUnz1jbff/nnDzuGjOLco/J6EAbWs77Y2ATtwyOX3gmN2UFX1IMHmt0Eqd5lc2ieAU2iqaMn6Pmn8XxC87g5vXTimaArfLk7pqdBidWbGBisLnLHRTYlAoxP17FbZunsTUTxETPW7RtNLakg3mG+DXRUrRDIqhSy1nMQ4I2sx3lsTtdg03ReKp5JDdsmM66ZAm3j3mDYb52yvQ0Y/2tqJz3J6BZlBqFdot9Ak5aa5+yOCOr5kvaGj9ZeyAJG2JmiqSd4f3mNVw+914akxHwHQ/k79KPKI+wODGImOXJq6utgANLGpkYbMGrLD6MVne+lyJOZtyfrj2IhG1iI8Qsk7To/Ld5HO8bD6GVXtO56Ev8IbCjdMVIJMBc5NRpAEQsxFyP2F0xMk4ajm4nuOzY7ZaPzfq3GVJ+asH74rL38IkRDCSfpbv6KLeGgYhT5jFh6fx24/6sSZXxVmQgD9SP69z1tlseHmvch1H+dsYEI1wxbB4/HTGXg0u28qXa5YBjvC1W97guFSR3UVbYbM2UcMLCs7hizWF8GK1iWaKcYgu9roQzqlbjU73JOy94MTmnZjn1mRApMTCUkwxbw86qbPJX7D/s8zpjQ04OqF+NfJshvhh+ZeJVFiP9kaL1i7veO+Fnw+cAsCxZQdI2aLe8PN48hmWJsjzhINnV5bDSfKPuOTXLO5PcpW2NunSIEiOVPSEVN9iTfQoNO7tYatyw9qCCpkpBtEoreOaU6PyrYTy1nhh/H/cSZ1at4XtDFhDIGv1BSNgaUauwXoNXWdwy6nWuHDqX/bKqsuWJioJZ2ghzmlfzhTduY7PxNfBMBfygQqBV46l6gIePuIq/NRzHqkRZ/iZEE/46bi4vTPkvp1St479No/hv00jmRmt4omkkq7J1KHJJ2iZvN2/Iv2iuoqja1NqApGYj9Ychjacj9TOx2291Unr7jnXmmYOm/Kja9ygdPJ8xAy4p7K8bIjYie0apW5cd5xOjSkKrBmtd3iVTnJ3nGy0DuLVuP8qNNBvTJZ0Le1oMnmsdzvkDl5G0Nb66/DhG+SIcXLKV1yKDyYjO8eUbGO5rZ0MqBNkl5Xcb9+NHw9/Hp6zOUKvf102jK8W0oFCd+X/Wp0r5/uojGOKN8lLrkGyqCOdH1Zrx8JuN+6NEOKykjjfaB+ele+hOSGX449jXGeNvZWyglZ+uOyQvrYXV7beqK5tliXJmlDq67sG+OA+Pf47VqVKCKoNqsbgwdhwtuh8NR8/uUxafrVmFUrA1HeTqtYfkLOIdKN5vr2F8oMuNtMPF98aRs7lg+bGsSpbhURbDfE5un2eah/GbjQdgo4qeNnJRYqNvTXNNdDprAhVolk3aY/RY1iA36K2DDelSbtm4Hz8e/j66BiN97UwJNTEnWtv5PL9cfyA3jpydF3CnKTi0bCtJS5G2FV5NqNCTRQvxAMTNFPeumcPV+z6AmGtBomBMRCmdGg9cPfkkpG0W+Q9t49XLWRw3uHjlYVQZCe4c9wp+ZbFfuIEyI8316w7CzHF/9mseBvnL8x/ddySSfJG82ByxEH0CtJyTH90f+wd4pqFC5yOpN7IBeTpOWdDfoOml2/pInK4lhUR+Don/AYL4T0aV3YBSbtK7jxP9IhiUUicBtwE6cLeI3NTtdR9wH3Ag0AR8QUTW9sfYuZi2xdN1H/B6/VJGhmr4/IhDqPE7X2YJfptlm69GVynG+NsQYFbrUG7esD/xbERvoxkq6DOgmdgCzzcPoyHtJ2oZfGPFMU7BFhz/esnTNcOzrSNYkyrhM1WrmRZuZIAnnhe0JnkKCwdbYEbJVm7ZMI1NdX7OGLmeJWYl16ybiUc5abAHemL8e+Iz3LVlckENBHB2zxOCLZToaTyacFjpFgq8j7IHho54Phudu7fsyxFlW3g3UstdWycTtTyMXb8V8/pmIo06mtrAOV9v4+Kr1pMUg1WJEqaGHZ33vVsnZJPnFe7qh/ljRYPADGVzauVa/lU/nmuGvUdAmdSlg9y04cBu3kI9BZEJkoG45mGFvyobaKGDdER4O++ugY2eDaobG2jh5bZ824YAL7UN5zvWQuriYW7dOJXlyYpOtRPA65Eh3Lxhf34y4v2CWfh1IW1rrE6EmR+rxlB20ZKiFjYvbVnAWcOmM7F0RIFu3/LMRNG95I4fwlfwq8VPEbc9XDpgHiV6utPoe0zZRu70TmZT2tmQeJRO0PBx8uD9u03yVEg+DanZoDSQDIS/h7I35qnmHBJI4v/Q/Megqu5DMkvArgfPAU76jl4gkRsg8RSdbt/J5xA0VPlvenW/y55BnwWDUkoH7gCOBzYCc5RST4jI4pxmFwItIjJGKXUOcDPwhb6O3Z0rP/gXc5tWk7QzvKV0Ht/4Lo/MPINg7CcY1jIGez38pW4yb7cP5Nyq5fx+y355i0AhQqvppTnj4466KZhotFu9yzuzLFHJbXUl/HjYXKo9yaLZQLujLUxS9qM1PNwS4mFtIslzKsl8Qe/M27MhFeaeLZO5Yug8XmwdlnMScBbCjvQYX1p6PDeOfLsr948ImI4gkwYPalC+3lwQ7q8fywutw53TUsZm8w9iaNEuYTbrb6UcMK2MI05tY0qOIXR1oqTH93BetIrI+jAPvj2VjKVz9oGL+Oz0RWhAtSfBoxOfJaA5qbBrPQmOKNvEiznGb5XzbCq7h7ezQXkhPU2yXMfqHh6dkxLDRMcU2JwOsY8/Qpmeos3KV5F4lEVdKszYQCuDfXGWJgsDFbsHFObi1WyqPUlu2nggAAYaZhEbVruZ5Buz/8qocC1/mvF1Sj3ODvqx9e9w+7Jn2cd3CL8Y8Q6DfVGUCkH4B2iBE1mdfB2w2TfYnOcJ5NGEO8a8ymWrTyHgHcIBlaP40qgjCmovK2WgKv6KZBaDuRa8+6H0wUhqNt3Va6CBVt51r2ciMLHHZy9K4r/kq67SkHwKkV9vM2JaMkuQ9l857uTGVFTp1Shj5I6N7dJv9MeJYQawUkRWAyilHgLOBHIFw5nAtdl//xv4o1JKiWxPYdB7lkc2M7d5dWdN2oxYWJkEm7Z8i4nBJhbHK/nl+gNZkyxFEjq/az0IFbR7Ul8DILZi/eoqTm0/A+WVbbYtcjeaEg4r3YzCyaTa9VJ28crpz4oJ//tOCBJd5wn/Q83Yo32YBzknGROdt9oH8h1zAcHlXtpXV4AuqDFxtOEplHKily2Bq9fOzAoUpy9pA2nWUSMyBYcIQwmL45WdKjR9YZLuwdfJuM5zD1ZyxKmOl44l8GG0hhXJCgpjIgQN4aG5+yILw6QyjoD5w4uHUNdWwqXHzWZ6uJ5QjteXoYQrh85jVuvwrtOX6upXw+bAcD1p0bly6IeM8LeTsTXuqx/P37dOLBg/928LjTnRWo4oreO5lhF5p5CM6AzztbOpsYzqiMlQI0Kpx6TBDNKQCeBXJjNK6juXUBF4deko3lw5jKEVEc46YAklgTQg+JSFKT27gWbEYk20nr+seJ4fTjqT+S3ruW3p0yTtDAvjZZy95AQqPPDI4VdT5nNcm4cGq1gXa2BBrIoR/va81CblRoqHjvhxr1Q8yjMJPJOcBTizDPFMzapXU3R56XlRwS9vt69tD6QVypvtmDLFqkeazwPJGsDTryJNH0DNy04uKZePnP4QDEOAXIvXRqB7Bq/ONiJiKqXagCqgMbeRUuoi4CKA4cOH79AkNiWa0bqlY7YRvrfqcO4eO4vvrTrC2WErUEG7c23uCRGwXi9DPyTiCIUeGulz41j7Bbs5dgtVeoJfjZqNjeKatQcTtz10LJh2wc5J0D9MYKv8n5BKCZ4XI52CASBiernxf5+iYWklZCOdZUEJtij0kV0eP8mOimoxi+Cvt2C87+iSMweHiF8+EAlo2Q22oCP5mUMDWZ/R3LkoIVTS1WZ5oowr1hyWl4W04/l+NvxdhvjifPfFM4lnuk4dyYyHR9+dwsXHvEuJURhhHtBMwnqKiOWncHHXmRsdwP8mP0m5kXZ86XWLr9QuY32qhJdahzHIaOekyvWUe9K80jqUD2I1nfeX6Wm+NmCpIxhsQTQnncnZFau47B+nsbqhkv2G1/G3s1/Eo1touvBWZACrkmV8beDSzpn87PFjeGXpKBIZLz4jw/1vT+OHX5qFT1nUeBJsTG/7J5URi6XNc7EjS/BE3mV6uJTXI10R1Cnby1tNKzpVQldNOpPL5v6De7ZO5IiyOgKaiUfZWGjcuWUqlw7qpYpH0kjLRSxcvYE5a4cwqKyNo2d8FR8LIfUm6MNRJVegPGN71V+PBD4H8YfpitnwQ+CMbZ8WEv91VFxdV5y/k8/DLsj55LJ99ijjs4jcCdwJcNBBB+3QaWJK+XBS0t1rR5ERjTu3TOr9HEyQNgNEoYzipwTVaqLaLDzPRfA920bq9HJS51U6wiHrANPSZHDz0kmsG1fTLQtpcYlkBgx83Z5YNJBQ/m4rmdF5adEYyBWClkJWBCFHMHTo5wO/r8f4IE6Hx6pnTpzgn+tJXJGtk6sU7baXRXHnRCNpRTpZjT/QiJ7uOjl4/cKnv+HI8YSlc8dtw0ker3f7BjmZV48q34xCsDKF+nbT1ljUXoGkbKbWtKDlPF4s69HUKUC7vU8WinfbazmpcmPntUbTT0CZjPa18vfxs9CUo2w6o3It926dwD/qJ+JXJl8fsIjF8UpH3izxM3PSRr48ahEvvTGeqUO38sOTX2fCoEYMvetDODS4hYN8Wzujftc1lTFryWhSpvPQKdODbWs8PWcCw8a2szJVUfC83RkbaOUvo2dB3GacF349Ct5rr+F7q4+kwwPLr3UJ0/19L/Pg+Oe4bv00zllyPEeWb6HSSDInOpT9a08sWHBtW3js9fn8540FeD06Xzn+II7dfywSf4A/PKV4ZM6JZEwNr8fizy+v5P6rL6e09lfbnXdvUSU/dGwXiUcAAf9nUKVXbfsmiVIYW2Rlr7vsDvpDMGwCcq16Q7PXirXZqJw0pmU4Ruh+o9pXwjEDJvPClvwqa6ZoJK0ij5l1t1Q5gWTSpmPPD2EsTuBbtBVVZZI8sAY8ucZbQZ+XIPjbLWA72g7/oy3YlTqZU8pAd5LB2RUeVltVsNKE0Tr2Ri/2ej+YCjUwjbZPotMADGBNDWCX6Ghps3MRx6NIn1aeP2+7h2O5qZCO+xw/TkDwzI6SG8agMoLn9SiJK4q8JQLWa+UQ14mcvi/Bt9fi29jK8BExvv2zjfgm6vxozUzej9SQ7CED7bpUCefNO5Z200uq1oRNRpcQUza+iiSX3DSVCfPquOWRNnRd0HSwDcWN6w/qNIj6yJDAQ3dPp1s27c+EYBsj/e282TaQq9fOxBLFDSPfQVd25yIe0C0uGLiElclSzqxaQ0Cz+OHKQ5EtXsyMh3fNGt5Z9SmGDorxqzFvMdgTx7I1jJzARJ9PWBUpY0AmQZUnxdrGclS3pHUZTfFGZgB6qrBQUncUih8Pfb8zO2zHmn5QSQPTw/W8Fx2ITzc4tGa883mk50L0Dwz2JbhjzCv8c+t4/t04BtGq+PSwmVww+uiCMW7/7xs8/MqHJNPOQvvTfzxLLJnmwJrXeejdSaSzQi2R1mmwNB6aNYuLzug/c59SHlTpj6H0x72/x38yEvs73SPD8R/Xb/Ny2TH6QzDMAcYqpUbhCIBzgPO6tXkCOB94G/gsMKs/7QsdfG/8ybxav4S03bX78CqLwyPrmG90izC1wV4SQg1KOTvuVUH0eQnKnp+HSmXvXw2p5nKk1tOlKkoLvv+2oqx8Vaq+NYN+VyOZQ8OOaklTSKmO57V2kplyZHkou0AK0uLBWh5EP7IVnxYnrXkQv0bslmH472zAmJfArvKQOGcA1kgv0mAgTV5UyERfHoOmdqzykpwc2wJDE5BW4JN81389axDIpYd8BtLogZQGopCAl9gx40gqiwkzFjHokPV8adnxJOxsoNmxHorq40xoui+F7/+2EvhKgtTx1U408wYfssVLenw7pb9tYW3az/kzJ/Kp01vxBoSXGwex9sIulUoC5+TQnYjl5R9bJ/Dz4XP41cYuL6Z9/JGCgkSaEvYPNfDr9QdQnw4iTR5kow/tgHYcc4piI0EuX3c44xdluP7Ts/K/IgJbzCD/WDeBa0fNYeSA5s7TAgA+C/2oFihQNRa+LwqYWTWW8aGnimowjy3fgsd/JFdOOt2JlAYk8Qwdi6Wu4KsDl/HVgetRpdeggvmLpiU2S9vqeHD2HNLprhGSaZO/PjmbK08bgUe3OgUDQNoy+GD19pNG7mqUZwJS+nNovwHEAuWD0htRevG64y67nj4LhqzN4GLgORx31b+JyCKl1HXAeyLyBHAP8E+l1EqgGUd49DsDA+VcO+Wz/HLR49giiJh8Z9AKBr2XRN1Uh3bpQOxRXiShY88LI7ZyFsOIBzIWpc8uQWVydoQC4Ss3kvh2DeZ+QbQmE//fmzCWOj/W3B+498k2MMH7QoTUp8tJfaUaNMdTqEso5NxlAY978T+2FL8o0keXk7i4iviVgyChQcBG25hG/08K83AP2pg4ygB7oAcVSaGe92A0JAjOXocWS2NO9ZO4uAYG5AZkKdInleJ9LoJKZY3QPkXqlFJot9CDgqXnuJpmCpcsS3SaowEeaxxN2ta6XBy9Gog4AVG56a0VSKlOZkaIzOllaD5nB66NjyMlFmqd0OF6395i8OR9jsC2hijHdy2PYkuoYnG8gpaMl6ZMlwfO+9EaBnljefEGaVvjr5unkNwQwF4QhoyOdnBb3klN0IiYXt5qquXD9YPYb/hm/B7LScth6tzbMJHlzaXc6pnGK5GhMCEOS0OgBG18DLzSeerMJ184CJCw0+jGGDDn5z+RgrNGnktJ+1RuXvxfwoafc0cexmS9DOcnmrN4Kw20/OC25ZE6Ln3vH8StNNaRabQmD/a7ZdmcUdAciXHPrCAZM1+AeQ2baWNGF5v8R44WPBsJnA52M2jVXfVRXHYLahds3PuFgw46SN57772dujdjm2xOtFLrL8WnKcRcy6++8jiv/+d9zIxFZmaQxBWDwAPS5sF6pwLPymbCL69AyxS6Gm7HTl3Y3gPpA0OkyoeQGV6VE2DVDcum6m4nq6sYGunhpejxNFKhMMca+J+N0HbhJDg5nb+YmaCeEkrvWY0ynfmKBna1QfSekfm1PC3B968mvM85aRzSp5aROqfSaZMR5/SQbS5phfV8VeeCAuD3ZLjx7Bd4wTeAF1qLOwSIDdJsoEpNlBdI2s4pJZRvYxALjN8nCb26gdwgbtEgfUQ5yR/VsL13WwS8ymTwplbWDq7uFEiVRpK/j3uJsJ5BIehKuH7NgbzynwqMOTFMVUpq1GD0o2MwIF+f7cPEnl2K3eTjy4d+wLGTVrO1Lcxd86exbIwPbVMa1WphTXZcTCWmIY1e1OAkqiAwOuf3FLPR2izsAR7QFeWeIM9+6ovQeDJ5KSdUGX9p/TkPrZ9D0sqgAJ/m4eb9TuFg+VY2CE0AHbQqVM1LOKFBICKc8eqv2ZrsCiQUC2R1AHtxGC17qrRtIeRLkTYNTEvD79UoD4d58OovURrKd3F1+XijlJorIgf1pY+9Uix7NIPhoS7VkfKM5eoHrmTRm0t55NYneeeZ90ndZmNNDKPa0lChEL+nmOaiE9vQUGZ3c2gPZCBj1JIZUtFThBYAWrTL31uZNt7VrU7/W8BYrhClkNrugU9OcJp/bgOYOQXrbdDaLfTFCawpQSdF9MoAss6POayS+E1xtGGpbtMRjNejGG9FSf5oIMoL2vQI9tySbCCcYuiEBvwD4xxmbeaFlmFFn0dpQImF/X4p+swI+LUCr6aOZwws3OLYd8ARzEpDPAaxIwejSSYnQC1/nM79SwbMiMG4sjbWSVXnfJoyfs5edBKH0kB5eYLZsVqiP24lOL8BlbHxemIEFmyl/fQRjiTqfA6h1EjT3qaTMA3ufm06d782HcMwsafEUWYK/b9x/FP8XHfAV3ho3Vus8G5h31HDKPeEeKbuAzKSY8TJxo347mrA91QENBC/RuKaQUw5egKaMQq75lWIXO/47HsPJRX4Fg/O+xOprApUgKSd4U8r5jBzxsNI+2/AXAHe6Y7nkOqKpdmUaKYtnV+bXOmghqTxrDAwdEUs6Zw4YinnPkPT+PrJh3DOMfsT8Bam/CjGskgd1y94jJXtWxjgL+PKSWdweO2EXt3bE5HmdnRdI1RWGFjqsnvZKwVDMZRS7Hv4RPY9fCK2bZOIJomLzYU3PcjGlnbMgSVIyIu0p1DZRU0AdEXskJHYYR8lzy3dpvDIJTW2BoxuXjm5pzPTJvTmmvw55v7bEuyggWdeFPPAbqU+TVCxIvtqBSrtjGEtCMH6QOfuX+aXYFsKfVSOgU9AX5lEeRQkBQIKbUAadVITRHVsC9ZWeLhkzZF8d8C8bT6vJHSk3uvMzQBMQTTVddKxBa3RRNua6Zy3FfAQnzGK9PAKVHWUUpVieCDKskS5kzTPpjOThTQb2CuC0OBFJsd4X6+FtCA+IKFjrwqQrPMyK1MLCjw1WyhZmEBlsgFyGYGYRfi76zEPCJD6cjVSpoPAb8e9QfqLAa58+ESiKS+2rTht2lKe3hjEXuQh/eVKAqaHgOHl9ukXdD7zskgd/9s0l64vhfNkxjsxjOUpEt+vxS438L7cTvCaTcz443FkpmTweAdARVcVtMZoIxk7vyodQH2qDVNGYfpuJ1BZfFcf0vykN+lYm30QstFGJlA+YUx1Lb/+6Ze5+m/PsHDtlrx7PB6dw/Yd1WuhEDWTfOvdu4iZzkZmc7KVH3/4IP845Dt5qcV7S0t9G9d99rcsfXclADNO3p+r/nUJAffkssfwiREMuWiaRqg0SAi44/LP8blf3EvatGg7cwrB2WvxbGhFZUxih+5Dekw1eJx0C4lpQwgsqEN1N+bm0LlE2FIoQwQ8a5rQYml8S7ditGy7trQd8mIsjmHOq0ammWRDf5FmHYkWCSuzNdJDSyACrA0UurQuD0KuYLDB+3oU0RThH2xA25LBHOcn+d1a7AFeWBJAwnGUDnc9OhKOyBkrKwDEzs5pUc6uL2XDljQqoCNVzklMazIJXleXN18tZpIeXQ2WzYwNy7nl7PfJiMZ77bX8bN0M0qIjlsJeGELWBzsnrSrSNGsB5HEf6swk5uwKaM9f5NQbHsRWeclHVAa0ehPfs+34nm133h5DMXxJlMDQCE9ffh91LaWUBZMEjTSvj5qKpWm0Tx9BpNrm4jl/p8obJmVnOLJ2EjEzSbHgPvPAIOYB2dgWTZEY78ec4OPeS+7nmd8+S+WgClbPW8fwiUP49u+/yvXpZ7NuzF3oNgy4J8ZZp52PZVqMmz6Gnz58OTVDq/La3fLg61jvlyKmgCZYqwIEj23neweeyNCacs4+Ygor6xo7vZQ0pagIBxg7pDDdd0+8Ub+U7irnjG3y5Ka5XDphx4v13Hje71kyewWW6QjDOc9+yF+vuJfL/vLNHe7LZdfwycmu2gPDasq590fnMqGyHEPAGliKEkEzhfToKkcoAChFcsogElMHY3t1bF3D9ubnHRXA9jnt/Yu2QCZnF5ix8C3dSumLywm/vRZPS2KbaqnYzBFEzpxC7MjRWOvLsf8dwnrfjz2nDP+NjXjrInQpQ8Aq9dN+3HiM+yz0u+zimcXTCsnW01QtJsGbtqBtNdE3Z9DXpFEJwZifIHT5BtT6JGW/XIzxTBxtQQLP/9pRG9KQtcHYW7zYm73IRh/WG+VIswc1MGsL8Wkw3AdVBipjo+ZA8OI69A35HjBWWQAyFt5VjZTMacaMwq0/GcZVy2Y6iQJ1hfKANiUGFRnQbdTgFFRYSNAgNG8tnndi0J6/v1EDUtgnFmak6r6EK3FOEq//r4x00qmZPbQygl9LM2dWKbalwABjgSPATdtka6qN1kycZ+o+4PWGpUUOkAo8mmOc7/AaC2hkji0l5bPYtGILC15bQqwtzpLZK7j0+79mU7zQczuwzKLpv+tIJzNYps2yd1fyk9Py4w02NLTy4gcrsDqMyrYCU2Nm2/4cUjMOgDMOmcx5R++P32ugaxrjhtbwp0vO3mbAWW/ZGetkKpFi/quLO4UCQCaV4ZWH3+rzfFz6j0/kiaE744fV8sCNjorgX9c/yoPvricNhF9eSfTYcZ3GXPHoiGWjpZ0vtaBhBwyUJaiMhVlbQvToMYReW4l/7gYQSE4eCAp8yxoIvrtuu3MRwKwMkpw8yImJ6LjuD0Cds7D7VzV1qrsA2o8Zg5T4KX1qsaPiBiIlfqwBYTqXQtvGu7yRwEMbwRC0xhRakV+2ctxnKLlsA8qC0F2bUZZz+vH9OkLs7FGIT0eNSCAtBrImALZCDUmiTc0JSFLK8T7SFUyziR0xktIXVuap01Kjqwi/tgrf6gaGfzXJtV8bxdzKQYUrjg76lHbnBFRhOuYB00YqNAK315P6zBDsEkcNoUbG0SbHwFAkv1iJ//5mRCknUE8Ec5yXxHcHYA/xoK9JE/hjPXdcM4SScouDjnZSjy+YHeK3lw3r/EDsWk/XM2XJiOX0qWx6tb8ywa4wUO1pcrJakKlQZDIWdNM6as0m6XhXTivbslm/ZCOfG3ghkVQa/bSptFcFyJjdVFACkaau+5RSXHzW4Xzz9ENIZyxC/p5TdvTE4TUT8lP8pWx8r8RYpM/lNxOauOjzZ1FW2rsIbN3Q0XUNu1uaX19gx+flsutwBUM3Pn3pqbz80Js0bGzCu7Ud37NLqT5zP5ojcdJLN2PM39zZVlk2WsImObGW+Izh+JY3Yge9pEdV46lrJzh3A8G5G7YxWj4CoCmix48jr1oPbNOIbbQk8L23sdNDSQElzy+j/ZSJWOUBEDC2RAi9sQZlOgEYmUGleDdHOkIrutk3cicl2IYiPbyS2FFjIOKsYNLsQY2LYZzalM1fLoVz7kADKgxaPz+NwDvrMLa2o8cyBOdsyD634rknB5CMKWQ6RdJxgLKS+P/VDKaQPrEMa7wPY0EClRRCr6+m/YTxoGtoE+Oddo305yvJHFmCPi+JuaYSY2Qj5omlne+lNcFP9OYhqK+t5dqvjSIYtlAaRCO6Y9rwKqyRXqwJvq5Q8ty3JmUTCpjEpHsgXta6ntNeZQRtU6bTZtL51nwYRajNf7/SNpk5rXRfKs2MRXNLlJYvHgReHVLpbuOCR9M4ZNLIgo/Ao+t49MJI9N4Q9vi5Y8aF3LDgMVa1bSF0WyOx71QxT0WZrxbxn+cWcd8xlzC2avtxB4bH4ISvHcUL971GOuEIMF/Qx2evOH2n5uaya3AFQzdCpUHunPc73n3mA+rXN7L/sVMYPmEIALZtc9kRP2XJ205Rno6F3PZ7QNNITXV+GGbJTu5+FEROnYRdHuy5jaZIjazAt7rZWWM0hVlbQuCD/GBzPZ6m7N/zsMJeJOAhOXUwbWdPxdgaxf/BBqLHjKXi/rnEDh1JaPb6vEINouja1dqgREgcPKJLrQadNgsZGkNfn0JbmcI8rsz5Rmnk1/Q0BX1NFHNaGL0phh7PFKjREvU24tVRCzNYS0qQLQHHmD4igTYuRviqTWibnQSAnjejpPcNEZ8yEjE0fEvrKX9sPvH9arHPzF+kZaCHTK0He7mBjCkrlK+awjwsjPe5CPGooxoUn8Ku0MkcGcaYn6T0jFVIUCP1xUrSZ2XTXtiCarW4ovktFk4Zyn+aRueMm3UB7jgdZYTAjZuxy4PoLQly9Xxak82Xqw7lny1vYcVNMBSeV9rRX4sgepeQVkohIiTH14KvSJpzEUd12Z7ipH1H0RtEhOXvrcKybMZPH42+DcExqWwoDxx+KW/9dw4/+MLDSNBRlQlg2cLVb/yLR8/8Ya/GvfgPF1I5oJxn7pmF4dE5+/LTOfO7J/XqXpePBlcwFEE3dA45vdANWNM0bnvjBt55+n1e//ds3trawLqhYdB1SJvO4qprWMMqSQ8vx7u+tfPenvb7nZtHQyN22CjMwWU9tOwidugovGuaUQLJCbVkBpciAQ8qURjFapX6iZ44wfGQ0hTpEj+Z4eWIUlghL6kJA1EZm+B7G8AWxKMTP3AY4dlru+YtYAeKeLBYCv/djXhfjIBStLaPQPlMtJOioEvn0wVuq8fzXhxvXQQtYZI+owzVaOJ5M5a7lDp1tScMQtb6O08fsjqA9lYMvS7n2VKCZ2GC6JerQddJj6gk+M46wq+sI3baUKzx/i71nw3SaKCNSlI0ZkqB5EaC+yD6+2GEL12P7/FWMLO2iJiN7+/NZDKlWKfo6JtMtDcz3Hz8oYyIxfBgk+muD+qsVSqYFSUkDhiBUR8l/MpKVDIDYY1pPzuM78w8ledPepbWpggSUCQvqCHzyGg8T7RS8b84RlLhC3hp3tKKVeEvOHVkB6H0uaWEI2kWnLqQQecfVeRhu2jc1MQVR19Ly5ZWUBAqC/LbWdcyZMygbd63fmUd5kRPvuDXFBs8bT3f1A3DY/CVa7/AV67t98z7Lv3EJ974vKMopZh56oFc+ffv8qu/fodQKEDAa1D21GJ8yxvQIkk8G1pQ0TSRI0dhh53TQ9cy2fVvAcRvEDtwCM1fPojUhG6uf7m1RzuwbEpeWdlpH0iPqgKvQeywUYiuOX0qEKVoP34c0VMmOTv9zvQZCvEa6JEEkbOngaZI7juIttMnIx4NZVn4l9cXPLdnY2t++TcRsCGtDcEq8ZOYOgQJerH1IOasKqwFYeyFIYLfr8P7ZhRl2lhlAdKnlZI8v9oJfusmLVXGwtjaTl5mPVsjU57viQOgUhadHXh04jOGg4LAb7eimk0kjZOwM6Fhzy2eulkE0DXUShPxKKx9vCS+XUvo5i1oaSDTdXLKDC6l9ZwDyGi12M9VIW8aWOcFSA7xsjxZQWZbPyVDI3HaYMTvITO8gpZz9yd541iCjx/AT77n2La0lUmMZUkS3x+IDPSAoZH5TCUtfx/BjWt/ycV//Dr+kA9dtYO/u2eBoEJJPHURdKUIlBSvlrY10crr9Uupi7dwyzf+ypY19SSiSRLtSZo2tfCrL/2h52fIMvWwiahooWdDhdrGKdflY4d7YugDE4bV8uQNF/Li+8v5y+2vEa7PzwbpfW0NKuRFDSqFphh21midux5Gpw8nPa6WgkQ/IngXb0FZNt5NbYihk5wyCNvQ8Gxs6/Sw0SNJTNsmPbqatvIAvsVb8K5tJjWqkszwijwDdicK7BI/4jOcXa2mY1WHic8cSfj11eithW604TdW03b6vtghb5ctQVNkhpZhnjmF0LOL4cChTn+WjmwKIrZNavhgPGtXYZX5ic8YgXFsA/g10qeU4Xm1HTpSdWSfx7u6yTm1HJGTqqHII4ihodIWEsi+6NExa8J4tkTx/aSV6Of3cQRii1OsSNb6YHTXqUGsbGzE8hDtx1Sjfb6V8uuXEPx9l1Ds+ERsr07kpIl5qjSzuhqtLYJWm+nWuhCPbqClPRg+DxnT4rBP7cMZR0/k4PJ9eOv/5nLn758gsq6NzBHhgpOAic1tT/6H0yLj+cIPz+T+x54lPa0N673yrkaG4A01omkKX9DHwafsXzCHO5Y9x4Pr3sSjdDJioY1swmt1BWyKCMvnrMLMmOiGzpLIJlZH65lcNpRR4S4byKSZ4zj45QHMnlgPXudu3Vb88MAze3x+l48frmDoI2UhP2cfMZUnxw9m47L8IvcDR9Vy8oXHIrbwwK/+Q5rCIKbQW2sBSI+v7VoAbSEwex3BhZtz6pGBd30L0SPzc9sE5tWRGlMNhsKqCqFHkmiJDOnxAwoD7DqwbMSj5xtT9aytJDu+5dXQMoLKnlhUyiTw9hoyg8tJTx7YJRyUQjRFemyNc6LIHVPTMGud3XpmaDkgiF93UnKP8xO/cgD+uxrRmi0nhYYNmiX4l9UTP3SUI9RE0BscgdsRcGiV+NBiacKzlhM9dhyia6BrJKYMxvPScpQIpfcuQ0uapEZXkThoGPayMJoHGJF01EebvdjvhdAiKdgvQ/iBtUXfKumYe/eTm62QDT6Mpc34/96I1mphTguQuHgAUt31s9JEMcBfyt8u+w51De0MrS6jNOTnjf97hy+c/w0S0aRj1waUmXuuzI5v2ix5bSmb7n8H3dD542vXcU3b/7GmsgGrweMUaipNUXlrnBlnzeBbvzsfbzfPo8VtG3l43Vuk/7+98w6Polob+O/MbN90Eggl9N67ggJ2sFBUihXsvdd77eXau9druV6viuUDVOwdLBQRaVKlkxAgQEhPts+c74/ZbHazmxAIUrzzex6e7M6emXn3hJx3zlv1EIFweWsxMgV1TjmWNTV5Lc5kwwR3629vs7hsC0IIdKlzZs5gbul2RmTci3+/hW+W/sbHeb+Rmp7MRf1OpHtqq4TzZ3JkYpqSDhC3vn4VDrcdu9OG3WnD4Xbwt3du4Ly7zkJRFbRgvFIAI7Ipee5mUt9bgvvbtdjW7ETZU4lS4YuJuzcWDh3n0ny0lJqSCGq5j7SPVmDbUAhBDev2MoQEtcwbX5ZCSghp2NbsjF1/dEnyV2tI/mFD5FCwbRMqRnUl2DQJLdWBt29Lqk7ugt7EGf9wrAjUcj/2NbsgqBnVaaUETcey06jRJDwBQ4nkOyLlwUNDk6l4ox1V/2gR2zFOlzWyC4GlxGM4xDEywtVyP6HmqYiqIGnvL8GxfDtqYSVaigNdAeu2Mqx7qlAr/TiX7yDlqz9AA31lMtoXmWifZ6IvSYUg2AsLcK7dhmVFfF6Jr4NhwhJ1/O4seeW4ntuFuiuE8Essiz2479xmOKZ3+LF+XYrzX7upOmspb93yDl1zskhxO/j4n1/xj0nP4a3wxUSEWRZVGTWkon9vIYnl61L8ngCeci8Pnv0014WGYfu1AiXZi8VfhfvZXaRtVbhn+s00a5MVJ+eiok21ynYYEVcMqgkxdbjtjL91DJdd/SDzdqzFpwfxagH8eoiPti7kmx2/xyS5jew3iOdOupTHh19oKoW/IOaO4QDR89huvL3hn8z9aCFCCIaNP5r0poYjOTUzGavdEpPUA0ZEUfUioHqDiMJK7PkliZPTwqilvsgOQnPbUHxBdLslHJYaDqmUEufS7QTaZADCsNmHNCNqxaoSrPZl6BIUgW1LEZadFQitJtzVvrkIX8/mlJ/ZO+b+oayk+KdnKbFvKUYp9+H+Nc8wVTmtVB3TFtcSo6mOLa8EhEBf60b1+rEv3o1SECDYPRX7ZzWN/KSAULPkGLONY0VBTOy/0CXWbaWUn9KVlFnrUCv8+JolQ1BDJtmNXUD1WMBSUI590Vb8/VvFRlYBtt/L8ZdnJHTo+rtnY99SjHVbKUpAQ7coNf6PkI5zYUGkaq0hFyglISyzynA9X1hjigK+f2cOvYZ157hzhvLWPdMSPigIv8R9az7e65uh9kpCy/PieHU36rYax3vBpl28d9t0HMsKqC4gEcxKYtOxzTnxtlcZ1K01t4wfQfOMmnaf2Y40bIoFr1aT3+C0WDlz0sns3L0GLagz+upT+PyV79jUsQicsQEQQanx0MoP+Tj/N14YeDEbF2zk8cn/ZM+2IpzJTq56dgojpxwf931Mjlz+ktVVDzeqyj1M6Xg9FSWV6JqOUASuZCeDn5rE7DumIUqNp1VpUQhmJxs1jEo9KN6gES6a4JpSgKdzFqomCXTMRKkK4FhZgFruBSEQmkRLc+Lp0wIt3YVtSxHegTmxph5dB0XBNW8zztWx9XRkuEaUv0etKBVd4v5hPZ5hHZCqcR/3z5uwb4nP3q2daVwysS9YVVI/+N0oSKhLpEVBt6oIrxHCqmW6KR/VDemuMYckfbUGW0F5JE+jWr6SCX1Jn74Mf+csqo5tDxaVlBm/Yy2JLSpXLUvV0W2M7yPDCnHjHtQyD95+OThW7MC9ODbnRLdbCDZxYdtRjnRZqRrSlmCLVJQyH66FuVh3Vcb9bqrDXYVfxn02cFRfup97NFOnvBonX/T5StsMxt0xlo+veTPhGCUqQUxLsVM63phXQhquxfk4NxXRtm1TLrxvAsPHD8GvBTl3/gvs9pUR0DVsQiXDnsyMYTfhUI159nv9jE2djOfkJHyXZYEz3phgVyxMzjqGz4a/g6+qRvnaXTae+fFBugzqSN4f25h6/3Ry12xjwMm9ueDe8aRkNCz5zeTAYFZXPUJwp7h4efHjvHXfdNYu3EDXozpx0UOTyMrJpFuLLN6+4U2qcvdgt1o4/6KTmPLARPILy1j+zTJevPiVmGtVq/GKER0IZbpRfCFSv/ojJhNaYiy4whPAvTAPxRcimJ2M16KiVPqxbSxE8QTxd2iC1iQJLcNljI9eeIVAq51PEY6Ssm8uwra5CMIhsqKeZ4to5eD6bSuWPZWIgFZjIgvpKBLKR3VFKgKtVVqs70NKKk/pClLinrMJx8Y9SAFaugslqFN5XEcjMius8MrP7EXaR8tRy+I7qlkKK/GHlSGqQqBTZjh7TqC7rEbR1ehq2CGNYJsMrEVVCG+QpNkbjOguXRJqmoRuUVBqVdwVAAmUggQWLt/MgsUbjD5JtZIKo+dJ31HGf+f+jvXoNrh/zYu7lq7pWCwqoZCGr2uzSMRZ8vfrsG4vA02SuyqfJy96CdWicsy4wbw95Fqm5/3C7yW59EprzUmWrqya/Qcd+rQhvVkaiqqgKAq2Hyvwj09HCotRJdenGz8Bvx7ih88WxJXTCPiCfDf1J9KbpXL90Xfhq/IhdcmOjTv57atlvLHmuXpzJAAqgl4+3Pora8q20z+jLeNaDcZpMbOhDxXmjuEwIeALoFrVuD+gC/rews5V4R7Hes2CE0xzUD6uNymfryKUlYRjw55wCGmUaSP8UwrwDMhBy0oyKsRGjQllOCkf05O0j1eiVAUQId3YuTRNouKMHjGx+MZFBdaNhViKPUZ4blWg7hwNETZ3eYIIKREyfhcBoFtVKk/sRLBNRsIM4xphddKmL0N3Wqg8oROuRflUHt8xdhckJbb1u0n+aVOsLGHZK0Z2Me5TG00n+UtjZxJ9jrdvS7y9mmPfXIRa7jN2GV7DfBdsnoIttzheCUQnCEbfP+q7134fMx+qgufoNvh7NiftnUXG/NUaoygCXZcEspOpGN0D4QuR/v6SuAKPnQe251+/PRE1PZKXb3qTL1+fhdVmJRgIMuWBiUy6YxzPXv4Ks9+fh58Q/pEpaJ3tyCQVbaBRIFFBMGh1JnkPLcNbURO5ploUzrzxdCxWCx8++xmhQI2ZzJns5L4PbmXgKX0SfFMDbyjAOfOfp8hfSUAPYVcs5LiaMHXodViU/cvW/l/mQOwYTOfzYYLNYUv4VPXC1/eS3SYLNc2FiEoys5b6sOUWI20WHOt2IzQ9ZtcQbTIXEmx5xbh/2ogIK5fqf5ZiL47l2yk9sxdVR7fF160ZlcM7UHFa9xilYNlchG3dLtAlwQ6ZeAe2pvS8AfjbZyBVgW5R4goKSpuFsrP6UHpOPwKt02MirKIRuk6oadjcUF9xNwXKzuiOv3MWSd+vM5zctSvdhnc6cffAyOB2haPA4lAVvINiGxEJwPX7dtI+WYlz9U7DVBdOItSS7Vi3lRJqnoK/fQaBZkn4O2YSSnPW0XcuwXtViZMTjPmorsdVdUy7hNpDD/+u3WV+3HklYQd5/EBPeezOadkPq/jmvz8Q9AXxlHsI+oJMffAD8tbkc/2/LuOsG08nIzWF1kstdAlmYR1k+BtUoeBQbdwweRJWmyVm12CxWTj10hMpLiiJUQoASEnp7vqT374tWE5JoCrSktevh9jhLWF+4bp6zzP58zBNSYc5TZqnM3XDP1m7cAN+b4Atq7by3sMfUVFiZNAG2qSjpTnDpRaMpUFXQQrDzFGNtbAq8SIEuJbtAJsFf8dM/F2bGqaJKKWgFHlInbWekvP61yTKhX9UndwVX4kHpSoA3gDJP2w08huaJVM1ogM4rUhNpfLkLqS/uxjhM8qHS1Uxnqx1SdXQdshEmdVRWHZWkDR7PYonYCTltc8g2CrdqBkUTUjDmleMt1c2zpU745ZKtSpAXWjpTsPfoesR7SV0iVph2NMjwaQWBU//lki3A81uwVpURbB1WsSZL8r9pH68AjVUTxQBRCKt4hACS0EZae/vItQ0iWBWEtZij7GbI/acoDdAxtpC8k/siO62opT7I5/bnDZOnjIi5tKLvlkW4x8ACHgDfPjsF9z6n6u59NHzuPRRo2V7SNf4ascyfti5mpaudM5reywtXRk8P+9hXrzmddb+tpHsds249oWLad21JcPHH82cDxfEXF8LaQw4OTaAoTb5nj34tNis/aCusd1bXO95Jn8epmI4AlAUhe5DugDQ74RenHXD6Ux74hOmPjAdkVcSP14DmSC0KaETGyOr17UoH9fCrUgBFSd1JtTWiGiy7Cgj5bu1SEWgJ9kTXMGw9wt/iJQ5m8MCCCx7qhDVT4+KwL5ud83CKqHi+E4IXRJsnoJ021DKfSiVfiPqqTpyqNp8FdJJ/mp1TdtVXeJYvwf7pmLKMlzoTdyR42qxB+fKAkJNXBF/QPR31W2qcUwQ58uwryvE160Zvj4tsG0oxL1oa/x3ddsQGIrWm5OOc+k2vP1axZizZIqdypM7k/L9+poue7VtSBDj04n5SJdYt5UhAKXCDxaFUIoDEdJRK/0x30lRFTxbi8l4exG6PfbPufeI7ky8bUzMsWZtsrA5bZECdtXMencOY64ZSaf+7SPHLIrKmFYDGdMq1iqR06UlT81+IG5uBp/Wn9FXn8InL32D1WpB13Vue+Ma0pulxY2NZmBGBz7cujAmakoRCv3T29V7nsmfh6kYjlBOu+xEZj7/BaW7y+OaqEB9ebjEmXSqhrVHBDRSP1uFokljQUtwHaXSHylvXZuknzbWLHRhh6x7zibKz+6D65dcHH/sQmg1T7yu3/IoO6s36JKk79Zi21qCVBSElDUOZSmxbi1BqfTH2c4FxiKtpzhiDuphx3SwTQb+7tkkzdkM4QxfgREWXFclWOfSfISm4+vbwlAqCRoyWcI7jmCrNLCoRnJdgkq4Iqjj65hpZGdbFRzbysAbjPtdxTmuMcKYq82CAiPJzVrsQVcEQoDFbiHkD4UjkQVSGgpQjaqVJRTBlAcmYrHG/omfdMFw3rpvWpxi0IIhvn5jdoxi2FeEEFzx5GQm3TGOXXmFtOneCrsz8cNENEdnduKE7B7MKliFKhRCUmNC66Ppmtpyv2UxaRymj+EIJaVJMq8tf4Zz/jaWnC4tUMIlNYSA1KwUbv3P1SjRC5YwFgswisZFL09KRcBYTFumGtFJgOZQkVYFXakZ656zyciF0PTYJCxdoiSKAir2QEDD8cfOmBwJAEuZj/R3FpPy5RpsW0sRmkQJaoiQTtKPGxBeI/w2efZ6kn7JjfGfVFM73wEhkJlJlJ/ZC1+fFgQ6N6V48kAqT+xk7BSqv2+CYoMioBmLuKKglHhxL4yNBqp9d7XYA5puXCuBySjYJh3Pse3xHtUGabOANxDn36lTedcuO14tty5RhaBFh2ySM5IQQqBYVKP5Ui1cyU46D6zJkg8EQ6zK3UmlrnPlk5Ox1MrnIFy9NZrd+Xu4d+wTjE2bwmW9bmHRN8vqkjiG1MwUOg/o0CClYNxacH+vCbw99Bru7nkm0469kRu6ntqgc03+HMwdwxFMetNULnnkfE6/4hQu730Lfo8fXZN4K7zMencOX1a9x7b1O7A5rTRvl82uvEJevulN5v62joqh7XFs2G1UZJUS2/pdhNJdKMUe1KoAFp+Gt0c2oawknEvyUT1BbNvKSJ3xO4H2TQi2TiPULAUw7Oq6y4bqiX0K1Z1WbJv2JHzyBoxQzz1VcQukDJuiQulOgtkpRjZ37TGAr3dNjoVlRxmuhXmoFX4CLVPxDGmLdNnAZiHQNgO1xItzqRHd5VyYZ/g/qhfHoIZz8dZwJVmZ2KkNeAbm4OtlNFCy5pdCUCeY5Qo7H8LRVCHd2G2Ea1TpKQ58A3JwrtlFvZmLMRMj4pRDNVpIJ3/t9ogyqP3kD0aU0OPf3YsSTsZbuHYrt7/2uWE2DGkc070tNredUGlNvofNYWXUJSdG3UfjpmPvoWh7MbouyVvt4cGzn+bZOQ/ReUCH2rc8ILRPakb7pH3vIb2vFGzZxbOXv8qqeWtJb5bG5U9ewPGTjvnT73skYSqGvwAfPfcFAV8QPbyYBXxB1i3ayJaVW2NMA9ltm/Lgx3dw0/B7WaAKPIPbGGYQKUl7Z2OMKQIg0DaDUKs0Ah0zseUWo+6pQkt34lq0FdeKHeh2C7rbRjDTja97U1yLt8Us4GpVgOQ5m+ptAVlt0ooxqUjQXTa0TDcVLVKxb9pD0s+b4s4TAQ10HbXES8rXf0RMWfZNe7DurqT0nH4opV6suyogpBNsnoyeZEfxBXH9sgV/12YITce+ugB7filSFVSO6ICl2BPfL6JPC3y9W0A4GU8t8WDNLcJzTHtDCYQzzlFEbElqDCdzoEUK9vzSemailolPDe8pND1eKSZQGooqaNmxOQNH9mX8LWfQtLVRGsMXCHHrq5/h8df8bhf8kce5L17Iqme+YfOKPJo0T+fq5y6iS9QOY/nPa6gq80Sin8D4f/XpS99w+5vX1vs9GsLurYX4PAFyurQ4IG1GG4oW0rhlxH0U7yhB1yWF+Xt45tKXadI8nd7Dux80OQ53GqUYhBAZwHSgLZALTJRSxnlDhRDfAEcD86SUZ9T+3KRx7Ni0M67EgqIqFOYXxdmMhRA8+d29TH/hS2YsXccelwrbShKaVxxrdlLZKs1IBuuQCR0yIaTjWmg4ZRV/CMUfQk2wkMbcM/wzUQ4DtT6TqiCU6UbLdBuLrVXF3zETx/IdqKWx9YySftxI+egeOJZvjzHnCAlKVYCUj5ZjKfEaNntNR0t1UHF6dxRviNT3lyBCOlXHtkd0bUawTROCrVKRDiuOFTviZPP1bA5WFaXUS+rHKwxfRKesWEUgwst6ODqp9gzUlbsghTDMdg4LarkfrCqVvZvjLPHStX1zcn/6g1AgVM8MG13QJj8wkeOinny1kMYn7/2MuqUIstyRHZIvGGLZziL+u+xppJQJF+agL34nIqXE7/HHHd8XPBVe7hv3BH8sWI+iKKRnp/Lo1/fQqlP9fSAOFKvmraWqzBuj8PyeAJ++9LWpGKJo7I7hb8BsKeXjQoi/hd/fmWDcU4ALuLKR9zNJwLFnHsXyH1fji/qjDQVC9Dy2a8LxdqedyX87i8lAaWEZ57e9hkRBnNYdZZH6SsZFjaJ4tU1GjX3eq35KDmYlEWibYZiIohcrXaKlu7CUemOUi1rpJ+3/lhqd32pfVNOxFBkKq9o/oVb4sf2xC/uWYoQQRpawKgjmpNecF9SwrzNKb0eWjiiHgGthXiRzW7pscbuDyPiI7DpKIETwhM7IZfnYVu+MmKmqh/kmD8LjMP4UrbnFuH/ZQtLifHK6teS2JyYjpeT6o++qUzlYbBaSM5IYOnZQ5NiuvEJuOvYeKss8WP1B0gWUn9EDLSsJIaBZulH1tq6n9b4n9Iz7zOG2M+rSExOObyiv3/EOa35ZR9BvfJedWwp58OyneH3Fs426bkPRND1hqoymNdDM9z9CY53PY4G3w6/fBsYlGiSlnA1UNPJeJnVw8uQRDBjZB7vThivFhc1h46bXriSlyd5r1KRlpfLgJ4lbMip+jaQFuSjlPoQviH1jIcnfNSzpqLaxozpUVaoisWlJVaga1h5fl6z4kxWBZU9lorMAw6SUMKGt9jFNYt1VScXIrpRMHoSWlUTSrPWG8tMltrW7SXt/SYwCEmHhHat3QlDDUlTjE7EUJpBJCIQviFpUhfCHsOaW4P5+Ha0ddkJpzjilANBuYwmOYg/uWeuwbSo0ggAcVrat2c6tx91Piw7ZjLrkhLhbuVNddBrQnrHXjeLlRU/ElNt+8drXKd5Ziq/ShwhqKAGN5FlGSXK71cLFIwdHxu7M3c17j3zE1AdnkL/OaBFrd9p54vv7aNGhGapVxZns4KKHz6k3g7khzPt4YUQpgLEL2ba+gNLC+pPgqslft52bht3LqfZzuaD9Ncz/5Ld9un/v4d2w2a0xysHusjP6qlP26Tp/dRpVEkMIUSqlTAu/FkBJ9fsEY48DbqvPlCSEuAK4AqB169YD8vLy9lu2/0Xy121nZ24hXQd3JDk9cdeyuvj81e946fo3IsXZFEVw63+v4fupP/P7j6uwWFS0kJ4wNLYuIiMVEWmso6fY0ZzWuAJ0EVOSRaHszF5oSXajppGUOFYX4A6br+ozR9W+Vm10VcE7MAdf33AYZFAj/b0lEAghHRYUb6jOa+sCvINbo+6pwr65yCjvoQiKLxocGxmlS+xrdpI0f4uhCK0qZWN7GrkWAY206UtRPbFmO6vdSiDJSsnpPZBqOLlQQPJ360gt9XHza1fx2cvfsGre2pjzHG47z/78ENvWF7BnezEDR/ahXU8je3tMyoV4K2tFiimC7k9N5Kqzh9GtteHkXf7zau45/TFCwRBSgsWqcve0myOtbaWUVJV5cCY5UOvq77EPXNLtRvJr9S2x2q3MLHoTh6v+KKaAL8B5ra+ivKgikuJid9p45ueHYvwjeyPvj208ceGLbFy2haQ0N5c8eh5nXPnXUQwHoiTGXhWDEGIWkJ3go7uBt6MVgRCiREqZnmBsgxRDNP9rtZIOB3blFTLv44VY7VZGXXx85An0m//+wL9u+G+MqaohSIzieHqak+Rv12EJVz2trluEjC82B8aCG2idjp7iwLqjDMueqphr7o9ikIpAd1opm9DX6FwHEAiR9PMm7JuL9nrdaP9A9NhAThoVp3QxEtykRCn3kfLxSlR/iEB2MpUndEJG5X6k/3chSoKS2+UjuxrZ09FtTb1B0pduI2dsf4r+2E7Vd38YCW5h7C4b7lQ33govwUAIRVWYfP8EJt0xjkt73MTWP7bH3MOV7GRm8ZsxpVcu63Uzeau3xYxr0iKd/8t/LcaUFNQ0/IEQSQ0MQa2LH6fN55nLXsYfNkfaXTZOmXIcN/zr8r2e++sXS3js/BfwRNVpEopg1CUncMu/r9pnWbSQhqIqB9X5fTA4KNVVpZQn1SPALiFEcyllgRCiORDfLNjkiKFZmyzOvileb/84bX68UhDQtkcOrhQn6U3TaN2tJR+/+FVsOYQmbkLhgnWBnDTUMm8klj+6p0NcVJIusecWRz7TrSpKUENLsiNtKmqJJ1KkLtGCHr14R2cTVx7TrkYphFH89Tt1o69ZGwlYt5dh3VJEMCcd18I8HGt3I4CQ24qvT8sYpQAQyk7Gml8a+32FMHIylFqWXbtKyVGtKdlTgtrUhT6hL6kfLTe69KkKVruViqIKglG+h6kPzGDUJSdw+RMX8PCk5wj6g0hdYnfZufjRc+PqcW3fEFtuHaC4oAQtpGGxWpBS8vpXC3nru0UEQzptmqXxxGWn06FFZoPmrTbHn3MMdqeN6U9+grfSx6hLTmDsdaMadK6uJahBL2Vkl7uvNHQHpGka30+dw0/T55PZMoMJt46mTfec/brnkUJjnc+fAVOAx8M/P220RCaHHU1apEeqeVbjcNm5c+r1tOvZmpkvfMn3U38mo3ka5UWVhAIhhowZRNbZA/jPD0vQd1Vg3VkBuqxzIa8LARDSKTutG6GcdOOJ/LNViEDICE+VdSuH2td2LdlGedsMY7ei6SieIJYdNbbtQOt0/O0zUDxBnL9vjxSySyRTKNONp19LdJcNy/YyMt5eFFXNViAQ4d4aMsaRXnVMO9Jm/B7JU3CnumjdrSW/lnoJ1a4XJUQkq1qTIGwqgaPa4Jq9np7HdqWytIrNy2PNrVa7lQ+e+ZwvXv3OKM9ttdB1cEcuuG8C/U/sFfdd2vVqzYYlm2OONWvbNJIx/f3S9bz13SJ8YeWzpaCYq174iK8fvRxLon7iDWDo2EExjvKG0v/k3pHcjGpsThunNtIhvjeeufQV5n70K74qP4oi+HnGLzw392E69m33p973UNJY5/PjwMlCiA3ASeH3CCEGCiH+Uz1ICDEX+AA4UQixTQgxspH3NTmITLhtDNYox6bVbqVdrzZ06NOWpy7+F2/fN50tK7eyY+Mugv4g9390O3e9dyPnnTaYLEUl5dOVWHdX7Hf0kpCSpKXbIKihpzgondAXb79WNZXAifdXJ8JSXGWU2CjzYl+7i9RPVkZKgXuOakPFSZ0JdGmGr3cLSs4fYDiLExBolkzZmJ4E2zZByzZqPUVHJwkpUfwh7HkloOvYNhTiXLAF2/rdWLeWGLum8F+elJJBp/an+fo9RoVUTY/0vagdPiOBdsd355vANC7+x7lsXRNrAgLwewN89PwXVJV5CAVCBP1GTktOlxYJv8tNr16BM9lhtKV12XC47Nz232sin8+cuzKiFKpl8AVCrMotaMCMR8kuJYu/W87b909n9ntzCfjjw6P3hsNl5+kfH6Bdr9ZGhn9mCje+cgU9hnbZ52s1lD07ivlp+i+RnbCuGyG77z70wZ92z8OBRu0YpJRFQJy6llIuBi6Lej+sMfcxObS069ma5+Y8xNQHZrBzy26GjBnIuX8/k/KiCuZ8uCAmysTvCTD1gRkMPKUPboeNc5o1562orX5t5aBaVRwuG1VlXuoj22anZaXOxu27sRaUYwt3jKtWCnXlCEQjJKR8YzhwtSQ7IqQZHeRsqpGnUF1CRFVAEXgGtyYlQRSWd0BOjMNZCWePxxDSw7ub1UbCXK3KqDI8JZ5yL+//40PunnYzlVLnwZk/43dZI5nTta9ZMm8D60/YxEMTniFUy1ehWhTa9GjFpmW5MccD/iCPnPs8rbu1ZPzNZ5DTtSW5q/PRghod+7Xj3S0vM2/mb2ghjWPPHBxT9M5ui18idCkRmmTaE58w96Nfcae6aN+rNV2P6sSg0/qTu3IrqZnJtOpco4weu+AFFny2GF+VH4fbzv89NpN/LnwMpztx7a266NCnLf9e/sxB8w8U7SjBarcQjFJkUsKOTbv+1PseaszMZ5MG0al/ex7+7G8xx4oKSlFUFYi100fX31eEQElQhwcBDpeD8beewYX3TWBKx+vZmZvYRWWxWRhwUm9mvzeHpAQ+AQGgCnSbWm9kUTTBnDS84XwJGc7+jr2oINQ8JaagnVQFoSZu9JRYB2ywZSosUYyn/YjQClIVRvJfKLZOVG3TVyio8fQlL/P8vH+ghnQUT6CmWKGURjgtArXMi++7Ndzwzd0Js59VqxqnFKpvuHr+WlbPX8vX/5lNRvM0POVehBCkZCbz1Kz7Oe2yxOaYC04cwKJ1+ZFdg6oqZKcn83/Xv8nKeWsjJTmWzV6JxW5BC2o43Hb0kE73oZ15+LO/kbdmG798ujiSHOer8rMzdzffvf0TY69pmH8h7rsegAiphtCuZ7wvweqw7pcp7EjCLKJnst+06NCMlCaxYbE2h5URE4dE3g87+yisdmvcmLfX/5PPyqcy5YFJKIrCLf+5KqGjwJnkIKdrC8666fR6nw6FLhlwxgAGjBmA1W7BmuBJNxr7H7uwr9mF7rYh3fbYkFMw+mH7ggRapiAFBLPcFF84kPLTuxuLdpQi0Zom4+3WzNi5CIFUBb7uzRCaTFhaO9G3qCrzcEXvW3F8vIK06ctI/uYP0HQcy7aR9PMmUr5aQ+rMFcbOo446SsEGOtKLC0rxVfnxVvrYvXUPj57/Qp1jB3XJ4eEpo2jdNI0kh43je3fg/tOHsipKKVQT8oeQusRb4cPvDbB6/jr+7/FP2Lxia1xSmd8TYO3CDQ2S91Bic9i4Z9rNONx2XClOHG47Hfu2Y9Kd4w61aH8q5o7BZL9RFIWHP/sbd532CN4KH1pIo/eI7lxw7/jImJwuLbnjzWt54ZrX8VX5cCY7ufm1K2nRITYCut8JvXh+7sO8fPNbFO8o4ajTB3D06AGkZaXQZVBHALJyMtmxsSDu4R7AarOQk5nKHws3IHWJO9VF2Z66cyoF4FxZgFpSReXIbrHtQXWJUuEn9ZNVKP4QElBLvCR/v97obFdd5E7XjaqxEqx7KiPRVv7mqQTaZ6LuqTTMU3tp2FONlDLciQ2s28pwrCxAhHRsW4rq7asN4ejfOhRGvffUJesXbSTgD2KzJ26WdGL/TpzYv1Pk/fKfVqNaVajf+kfAF2TuR79y13s3xu0Y7S4bPYYmzsw/3Bg0qh8zCl5n1by12F12kjOS4qvT/sUwez6bNBpd18ldlU9SmitSvK02mqZRUVxJckbSXhvD18XWtdu569RHKC0si8TBg7EoOtwOpJRx3cn2hgSwKHgH5eDtlg1C4Po1F8eanXGLsW5RqDq+I4H2RqimpaAM129bseyuTFgWvHR8b1y/bsWys9z4XBGE0p1YCuMryiYilOmm4oROpM9cUa9yadkxG5vLxpYV8Y2Fqr9jffdzuO18WjY1LuKnLnwePxOzL4tPoEvAgFP68Pg39/DMZa/w0/T5BP1BbA4bLTs15/l5Dze4NHc13771I+/940OqyrwMO/sornx6Ms6kxEECBxIpJW/c9T4zn/8Si01FURT+/u4NHHX6gD/93vvKQUlwO1SYisEkEVJKtq7djr/Kzycvfc3KuX/QrldrugzswLQnPtlnxVCNarfgb5GKUlCG4qvbJOPt2RzPMe1A17GvKyRpzqaE43SbSsnkQaAYNZnUoipCmUmEmiaROnM5aqlvrwl1oTbp+Ef34rIu7fnhqS/YvXUPopa/xu6y89Ts+3nhmtfZtGxL3HU0lxURCu9sdFmTWBi+hGKzMPne8Zx/99kNmaYIi79bzkMTniboD0VqOAlFRHYtQhihpE9+fx/dh3RBSsnq+WtZNX8drTo3Z8jogfvsJzCS416J+Cqsdgu9h3fn8W/v3afr7A8LPl/Mo+c9H/P/y+608X7+q6Rk7L30zMHkoCS4mZgcTgghaNOtFQB3vHVd5PhXr89qWMxqHWj+EF1Sk9m2qxIluaZDWrTdXrcoaE1cxgKrSdxr6g7ZNHIsJAiFYKs0o+MboJR6CRzbHucXa+r/nsAx5xzDZTeOZvPs1Tjcdpq0SKd5+2asnr8uohyGjz8aZ5KDbeu2J7xOMCcdz4BW2HNLEP4ggXZNUCr8OFfuAE2SOaIL59111r5NFjDwlD58uOsN1i/ZTOG2InJX5ZPVqgneSg9zP1pIRvN0zv3bmREzoBCCnsd2o+ex3fb5XtVMf/KTmOquQX+IFXP+oKighCbNExZciFBV7mH94k00bZ1Jy477Xsn1p+nz4x46VIvKslkrGTFx6D5f73DHVAwmfwmGjhvEK7e8lfAzRVVQVGWvpavbdG/Fc3MfZufmXTRtk8kDZz7Nijlr0MMtSbEohNKdWHOLSFm5k8feu4V7z3gsLmwUAFVgKSgn1CK1JvQ0qJG2soDhZw7m17lbCNYToqtaVfplZbDlxzU8d+VrkQWxaEdsVfs5Hyyg86AOdT59K1V+pN1iNBgKozVxE2ybgd2qctyQnrz/6Efkr93B4FP7MWLS0Aab+mwOGz2PCfsJJtUcn3Dr2Aadv6/4PfE1gIUiCCQoER7Nzx8s4KmLXkK1qoSCGoNH9eOe6Tfv044lrWkqqkVFC9X8riWS5AYUqjwSMaOSTP4SpGWl8tjXd9Oio+HUbtIinfPuPpvh449m+PghqBYVJbxAW6yWhBFOuavzefyCF9m+cSeuZBePf3sPVz93EVaHFVVVUHwh0j5ZRda8XG5/7ALmzPglZqGoRndYKZnUj1B2SsRRLSp8JH+/DntuMWeNGsSM3Ffod1IvrDYLVrsl0na1GqvVgivZyf89NrPeHgh+X4CVc9bUqfSs28pI+mkjzl+2oJT7ItFUDpuFDqnJ/HTTe7z/yExmvzeX5658jYcmPNOwCT8EjLrkeOyumkRLRRE0b9eU7LZN6zynsrSKJy96Cb83gKfcS8AbYPG3v/PtWz/t073HXDMyJrrOYlXJyE6nz3F/zR4O5o7B5C9Dz2O78fb6f6KFtJinwZdueINQIBSpqRMKJl5ENy/PY/PyPJbOWsFZN57Oxf84l3HXncpJFwzni9e+p6KkkkEj+9JrWDeevuRlfnh/bsIIKW/v5khHVJKaAOm0oRZ7sNmt9BjaJdww6T4ASnaXcVGn62OKwwX8QXRdjzmWEAkLPl1MRot0SnaWYrFZCAVCONx2Kks96JqObYtRd8q5ZhetLj2WMecfR+dWWfz2nx/4sMwTqbXkq/Kz5LvlbF6RR/vebRo05/uKFtJY8v0KSnaVMuDk3mS2bNLgc8ffOpqduYV8++YP6Lqkfe823P/hbfWGMa+evxaLVSUQNY0+j5+5M3+tM3cjES07Nue5uQ/x5j3T2LZuOwNO6cOUByftdyDF4Y6pGEz+ctQ2EezKLUz4ZF8Xvio/Hz77Bef8bRyeCh/3jnmc3NX5ICXrftvI3dNu4qcZv8TUjopGS7bHZS4LTUdLd+Iv9DDjqU/pelQnsts25YWr/s2KOWtIzkjC7rZTsrMUMArGvXD16w2SNxgIUbh1DyecP4yTLxxBVk4TsnIyOTvzYgLRrUE1naabijnrWKNm0svfLo8pwAfG3G3fUFCnYtidvwdFERTvLOXjF7+iqszDyIuOZ+jYQXvNQq4oqeSGIXdRVFCC1I3idze+egWnTD6u3vO2rMyjYPNuuh3diRtfvpyrnplM0B8iKc2917lp0jIDrVZEl2pRaN6u7l1GXXTs245Hvvj7Pp93JGIqBpO/PMPOPprff1wV4zwU4Z4HdSaLBYJMf+pTlv+4ms3LcyOLy/KfVnNh+2vrrejp/jUPj91CIKoznFQVLDsrCIZ0pj4ww/B5BDW0kIbUJf7txXv9HkIRtOjYHD0UomBzbJa4rktmvzuXYWcfTU6XlhRs3hUxnUWTv9bohbBu8SY2r8iN/97+IJ0Gxfc2KNldxr2jH2PLyq3ouozIDUbW84TbxjD5/on1yv/+ozPZmVsYY/Z68erXGXbWUQlDTrWQxoPjn2bprBWoFpVQIMSVT09mzDWjGhzm2rFvO7oO7sjahRvwewMoqoLNaWP8LaMbdP7/KqaPweQvz4kXDGPI6IHYHFZcKU7sLhvX/+tSOg/sgMWqIhQRZ+OXuuTDpz9n1by1cU+c1U5QtY7qompVgKTv1uHcWmKUyQhqOH/LQwlfJ+AL4qvyEwqE9ikpzZ3qovfwbhQXlCb8XErJ63e8C0DT1plx2d+KqtB7hGET/+i5z9GCNd9LCqPkR3m/Vpz15Ps89ND/sXtrYeTzp6a8xMZluQR8wTi5fVV+Zjz56V6dwEtnrYjzhahWlS2r8hOOn/3eXJbNXonfE/YP+IK8dttU9uzYuxKN5tGv7mLyg5PocUxXTr5wOC8veiIuwdIkFnPHYPKXR1VV7nr/Jgq27GL31j106t8eV7KT0VeNpKKkkqoyD38/9RF2bt4VE2Hk99a90FntFlp0yCZvTT66Fr+4i5BO2y1leFULnuXbUAsrE/bVjj+ROsNuQ4EQs96ZE1PQrTZFBUbUkmpRufOdG3h4wjPGzkEYiuXyJy9k7syF/DxjQeQc3aLg65GNv3sz9BQnaDqf5G9n/pC/89Dr1zBoVF+WzFpR7y5Jl5KKkiqaNLfVOaZdrzbkrsqPuU7QH6rTrPPr54vjQkQtVgur563dpxBRm8PGxNvGMPG2MQ0+538dc8dg8j9D83bN6DOiB67kGrNFcnoS2W2b8sbq5+g9okfcOdU7itooisLE28fyecW7PPTZnQnHCG+Qjz69lyffuYnJD0zE5khcciKajGZpdaYpZ+U0wWqv+1lOURV6jejG5wtWM+2n32l9dEfey3uFG16+nL+/eyNTN75ERnYaz1/1WsziLDQd5+qduH/eDJqOuruCpFnrCZZ7eeTc5wgFQ9iddS/4AGlZKWRkp9U75sL7xuNMckS+g8NtZ8zVp8RUc42mVZcWcbseXddp1jZxdr3JgcNUDCYmGAv9yIuPx+GOtV0LVeH0K06Oc6wqqsLQsYOwOWwcffoAmraO7WhmdVgZPmEIQgh6D+/OpDvGMWTMoMj11Tpq7XQ9qhNXPTMl4Wfdj+6csOe2xWbBmewgo0UGvzR38vj0H3lh5hzOeuBtFufv4qQLhjNk9ECsNiuVpVVUlXoi50qLgnRaIaRj3V2Bc+k2Uj9fjS2vBLUqgLfCx8MTnuXsW86I6cksBFhsKq4UJ+5UF/fOuHWvzueWHZvzn9XPMenOcZx66QncM/0WrqzjuwKMve5UHEkOLDZjruwuG10GdYwkzZn8eZglMUxMwmiaxqPnvcDCL5YYyVCBEJc/eQHjrjuN5T+t5qXr36Bg8y469mvHja9eQbuerSPnblm1lbtOfYTKMg96SKPX8O48MPP2mMVU0zTmfvgrC79cisWm8vOMBXH1hgac0oeHP7uTM1znx0U9ZbdtyrUvXsLDk55B13RC4Q5zLTpmc/Ej5/Lxrt3MX5OLHvU3neKyM+vJqyLd1nRdZ3TSBfh9QTxHtTH6UACKJ0Dyd2sR3iCKJxizabE5rPx37Qss/HIpn770NUIIxt1wKtltspj+1GesmrcW1aJy2uUncvkTF2C17X1n1FD2bC9i5gtfkbc6n8Gn9+fUS0+ss9ifiYFZK8nE5E8g749t7Ni4k66DO9Zp5qhNwZZdvHT9G6yct5bsNllc/9Jl9BpWd/kHLaRxTqsrKSssi+RCONxGu9QBJ/dmXMZF6LWc3q4UJ5+WTmXG05/y1r3TYsp12JxW/BP6UeyMNb04rBZmPnAR2VH1fG474QF+276bymHta8qNS4nwhUj5cBkWT6yD2Jns5KnZ99NlYGy00sMTn2HB50siPg+pKgT7tOSMO8Zw89nD97v1p0njOBCKwfzNmZjUok23VgwZPbDBSiHgC3DDkLtZ/M3veMu9bFm5lb+f+gj5ddQvAsM5/PQP99OmR2tUi4IzycH595zNsWcaoZvterSOMc1YrCqDT+0HEHZAxy7eAW8QZdpShDfWMa2ogiYprphj599zNr6uTWN7UIT7SBxz0XFxeSCKKujQJzavIeAPMv/TRTGOcKHpWFYX8PH8lbz+1a/1zJjJ4Y6pGExMGsmib34n4A3EmH5CgSBf/ntWvee16Z7D6yue4eOSt/m45C3OufPMyGf3zLiFzFYZ2J02rHYrrbq04Lp/XgoYVUsToQpI2lqCzaKiCIHDauHWs0dgrbXQ9zuhF2qCMFkhBKdeMJxew7vFOMBdSU6KC0rixteFLxBi5ryVDR5vcvhhKgYTk0bi9/iRtWJMdU1vcAlwp9sRV1qhaU4Tsts1RUqJalHYvqGARd/+DsDE28Ykdl7rkskj+nL16KFcPHIQb94+iTPDWc616aXYwi1Dw2g6lgo/g/t3xJ3qisnRKNpRHNflzWa3cszYQTFRUlJV8HVrBoAIa5aQpvP5gtXc9trnvPTJPApLKxs0JyaHFlMxmJg0kkGn9otLVLM5bZx04fD9vuZnL3/LukWbIslwQX+I5654jcrSKoaPH8LFD58TF9aqWlRGnjecKacM5Nqxx9Alp+6yD3c9OpnM+blYij0ITwDn5iLuOW0oVpuVxd/8HpPUp+uSNQvWEwzEmqlue/NaRkwYilAVpMVQCt6BOThsFiaM6A3A7f/+nMen/cAPv2/knVlLmPDwO+w2lcNhj6kYTEwaSXJ6Ev/44u9ktszAYjOqol7x1OSaktT7wYLPF8f1VLbaLKz9bSMAk+4Yx30zbiU5Iwmr3UpyRhK3v3UdrTrVlNf2Vnr5+o3ZvPPQB6z5dX3Mtdr2yOGDuY9y17D+3Na1M//30rWMnnI8AEnp8TWI7E5bnO/B6XZw59Tr+aTiHYa8djGh4zridjs47/h+XDJqMBu272Hh2q14w9nOQU3H6w/w7qwl+z0vJgcHM/PZxOQA0GdED97f+iple8pJSnNjsTbuTyunSwtWzVsbk4gWCoTIjkruGnb20QwdN4jyokpSmsS2TC3ZXcbV/W+nqsyD3xNg+pOfcvZNRsXYalIzUxh33alx977w/gm8cvPbkXLfdpediXeMrbP1p8th4+GLR/HwxaNiju/YU4Za65ygprN5576VtDgQ6LrO91N/ZvZ7c0lrmsrE28bQsV+7gy7HkUKj/vcKITKA6UBbIBeYKKUsqTWmL/AKkAJowCNSyumNua+JyeGIEIK0rNQDcq1Jd4zjx2nz8XsDaEENu8vOoJF9adW5Rcw4VVVJbxp/z+lPfkLZnopIbSK/x88Hz3zOmGtH7bXb2emXn0xKRjIfPvcFWlBj7LWj9sss1rNdNsFaVW0dVgvDeh6YBVlKyexlG5g5bxUuu5XzT+xPv44tE4598ZrXmfXuXPweP0IIfvl0EU9+fy/dh3Q5ILL81WhUHoMQ4kmgWEr5uBDib0C6lPLOWmM6A1JKuUEI0QJYAnSTUpbWd20zj8Hkf52dubuZ+fyXFGzexTHjBnPy5BEN7jp24zF3s2ZBrPnInerivg9vo/+JiR3SfwYzfl7Osx/8hFXT0a0WerTL5qXrzsTWyB0VwGtfLmDqd4sjpiqHzcLjl53O8F7tY8aVF1VwTqsr42pM9TuxF09+f1+j5TjcOBx6Po8Fjgu/fhv4CYhRDFLK9VGvdwghdgNZQGkj721i8pcmu21Trnn+4v06t+ex3diwZHNMv4WAL0i7njkHSrwG0aosQIvpy6kq9+BMdjD59asPiFIIBEO8/e1ifFFNl3yBEP/8eF6cYijZVYpqVeMUw668QkwS01jnczMpZXVH9J1As/oGCyEGAzZgUx2fXyGEWCyEWFxYaP7STEz2l4m3jyE9Ow1nkgPVomJ32jj372c2OGnvQFCweRePnvs8FUUV6EGNquIqHrvgRbZvLNj7yXvB4w+i6fHVXgvL4iOeWnVuEVdGw2q3MGRMox6q/9LsVXULIWYBiYqX3x39RkophRB12qWEEM2Bd4ApUsqE9XullP8G/g2GKWlvspmYmCQmNTOF//7xPHM/WsiebUUMOKUPnfq33/uJB5B5Mxei1SrVrYc05s38jUl3jG3UtVPdDpo3SWHr7tLIMVVVGNItvvOcalG5/6PbuHf04yCMHJM23XP22ljof5m9KgYp5Ul1fSaE2CWEaC6lLAgv/LvrGJcCfAncLaU0c+VNTA4Cdqedky7Y/1yKxmJz2uK6yCmq0qDy43tDCMGTl5/BVS98SDCkI6WkaVoSt088PuH43sO7M2Pn66z+ZT0pTZLo0KftXqvB/i/TWOfzU0BRlPM5Q0p5R60xNuBr4HMp5fMNvbbpfDYxObIpLSxjSsfr8VR4I8ecyQ7eXv/PA2bSCmoaKzYVYLdZ6NGmmbnYc3gU0XscOFkIsQE4KfweIcRAIcR/wmMmAsOBi4QQv4f/9W3kfU1MTA5z0rJSeXbOQ/Q8tiuuFCc9junCsz89dED9HFZVZUDnVvRsm20qhQOIWXbbxMTE5C/E4bBjMDExMTH5i2EqBhMTExOTGEzFYGJiYmISg6kYTExMTExiMBWDiYmJiUkMpmIwMTExMYnhsA1XFUIUAnl7GZYJ7DkI4vwZHKmyH6lyw5Eruyn3wedIlT0TcEsps/Y6sh4OW8XQEIQQixsbr3uoOFJlP1LlhiNXdlPug8+RKvuBkts0JZmYmJiYxGAqBhMTExOTGI50xfDvQy1AIzhSZT9S5YYjV3ZT7oPPkSr7AZH7iPYxmJiYmJgceI70HYOJiYmJyQHGVAwmJiYmJjEctopBCDFKCLFOCLEx3ASo9ud2IcT08OcLhRBtoz77e/j4OiHEyCNBbiFEWyGEN6pnxasHU+4Gyj5cCLFUCBESQoyv9dkUIcSG8L8pB0/qRsutRc35ZwdP6sj99yb7LUKINUKIFUKI2UKINlGfHc5zXp/ch2zOGyD3VUKIlWHZ5gkhukd9dsjWlfD990v2/VpbpJSH3T9ABTYB7QEbsBzoXmvMNcCr4dfnANPDr7uHx9uBduHrqEeA3G2BVYf5nLcFegNTgfFRxzOAzeGf6eHX6Ye73OHPKg/zOT8ecIVfXx31/+Vwn/OEch/KOW+g3ClRr8cA34RfH7J15QDIvs9ry+G6YxgMbJRSbpZSBoBpQO3u4WOBt8OvPwROFEYLp7HANCmlX0q5BdgYvt7hLvehZq+ySylzpZQrAL3WuSOB76WUxVLKEuB7YNTBEJrGyX2oaYjsP0opPeG3vwKtwq8P9zmvS+5DSUPkLo966waqo3MO5boCjZN9nzlcFUNLID/q/bbwsYRjpJQhoAxo0sBz/ywaIzdAOyHEMiHEz0KIYX+2sHXJFWZf5u1wn/P6cAghFgshfhVCjDugku2dfZX9Uoz+6ftz7oGkMXLDoZvzBskthLhWCLEJeBK4YV/O/RNpjOywj2uLpbHSmhwwCoDWUsoiIcQA4BMhRI9aTwEmB542UsrtQoj2wA9CiJVSyk2HWqjaCCEuAAYCIw61LPtCHXIf1nMupfwX8C8hxHnAPcBB9d80hjpk3+e15XDdMWwHcqLetwofSzhGCGEBUoGiBp77Z7Hfcoe3qEUAUsolGPbEzn+6xAnkCrMv83a4z3mdSCm3h39uBn4C+h1I4fZCg2QXQpwE3A2MkVL69+XcP4nGyH0o53xf52waMG4/zz3Q7Lfs+7W2HCznyT46WiwYzrR21DhaetQacy2xTtwZ4dc9iHUSbebgOZ8bI3dWtZwYDqbtQMbhNOdRY98i3vm8BcMJmh5+fVBkb6Tc6YA9/DoT2EAth96hlh1j0dwEdKp1/LCe83rkPmRz3kC5O0W9Hg0sDr8+ZOvKAZB9n9eWg/Kl9nMiTgPWh/9z3R0+9hDG0weAA/gAwwn0G9A+6ty7w+etA049EuQGzgZWA78DS4HRh+GcD8KwbVZh7M5WR517Sfg7bQQuPhLkBoYCK8N/ZCuBSw/DOZ8F7Ar/v/gd+OwImfOEch/qOW+A3C9E/R3+SNTieyjXlcbIvj9ri1kSw8TExMQkhsPVx2BiYmJicogwFYOJiYmJSQymYjAxMTExicFUDCYmJiYmMZiKwcTExMQkBlMxmJiYmJjEYCoGExMTE5MY/h9YOay7NvNeOAAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "from sklearn.decomposition import TruncatedSVD\n", - "svd = TruncatedSVD(n_components=2)\n", - "X_svd = svd.fit_transform(X_csr)\n", - "\n", - "from matplotlib import pyplot as plt\n", - "cluster_x, cluster_y = X_svd[:, 0], X_svd[:, 1]\n", - "cluster_c = list(cluster_chain[-1].tocsr().indices)\n", - "print(cluster_x.shape, cluster_y.shape)\n", - "plt.scatter(cluster_x, cluster_y, c=cluster_c, s=25)" - ] - }, - { - "cell_type": "markdown", - "id": "3acd550d", - "metadata": {}, - "source": [ - "### 2.2. Tracing Cluster in Hierarchical Clustering" - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "id": "b6c75c21", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "4 layers in the trained hierarchical clusters with C[d] as:\n", - "cluster_chain[0] is a csc matrix of shape (4, 1).\n", - "cluster_chain[1] is a csc matrix of shape (32, 4).\n", - "cluster_chain[2] is a csc matrix of shape (256, 32).\n", - "cluster_chain[3] is a csc matrix of shape (14146, 256).\n" - ] - } - ], - "source": [ - "from pecos.xmc.base import HierarchicalKMeans\n", - "import scipy.sparse as smat\n", - "import numpy as np\n", - "\n", - "cluster_chain = HierarchicalKMeans.gen(X_csr, nr_splits=8)\n", - "\n", - "print(f\"{len(cluster_chain)} layers in the trained hierarchical clusters with C[d] as:\")\n", - "for d, C in enumerate(cluster_chain):\n", - " print(f\"cluster_chain[{d}] is a {C.getformat()} matrix of shape {C.shape}.\")" - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "id": "dc644526", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "55 instances belong to the first cluster in the layer-3.\n", - "442 instances belong to the first cluster in the layer-2.\n", - "3536 instances belong to the first cluster in the layer-1.\n" - ] - } - ], - "source": [ - "from scipy.sparse import linalg\n", - "from pecos.core import clib as pecos_clib\n", - "\n", - "current_cluster = cluster_chain[-1]\n", - "for i in range(len(cluster_chain) - 2, -1, -1):\n", - " print(f\"{current_cluster.getnnz(0)[0]} instances belong to the first cluster in the layer-{i + 1}.\")\n", - " current_cluster = pecos_clib.sparse_matmul(current_cluster, cluster_chain[i])" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "id": "f2203831", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The 10-th instance belongs to the cluster-228 in the layer-3.\n", - "The 10-th instance belongs to the cluster-28 in the layer-2.\n", - "The 10-th instance belongs to the cluster-3 in the layer-1.\n" - ] - } - ], - "source": [ - "inst_idx = 10\n", - "\n", - "current_cluster = cluster_chain[-1]\n", - "for i in range(len(cluster_chain) - 2, -1, -1):\n", - " print(f\"The {inst_idx}-th instance belongs to the cluster-{current_cluster.tocsr().indices[inst_idx]} in the layer-{i + 1}.\")\n", - " current_cluster = pecos_clib.sparse_matmul(current_cluster, cluster_chain[i])" - ] - }, - { - "cell_type": "markdown", - "id": "080f044a", - "metadata": {}, - "source": [ - "### 2.3. Performance Benchmarking\n", - "\n", - "Here we benchmark the efficiency performance of PECOS hierarchicaly clustering and compare with a pure Python implementation based on [sklearn.cluster.KMeans](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html)." - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "id": "cbcacb51", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "PECOS takes 1.0039 seconds for hierarchical clustering with a depth 5.\n" - ] - } - ], - "source": [ - "import time\n", - "from pecos.xmc.base import HierarchicalKMeans\n", - "\n", - "nr_splits = 4\n", - "\n", - "start_time = time.time()\n", - "cluster_chain = HierarchicalKMeans.gen(X_csr, nr_splits=nr_splits)\n", - "pred_time = time.time() - start_time\n", - "\n", - "cluster_depth = len(cluster_chain)\n", - "print(f\"PECOS takes {pred_time:.4f} seconds for hierarchical clustering with a depth {cluster_depth}.\")" - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "id": "5c8179cd", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "scikit-learn takes 58.2397 seconds for hierarchical clustering with a depth 5.\n" - ] - } - ], - "source": [ - "from sklearn.cluster import KMeans\n", - "import numpy as np\n", - "\n", - "start_time = time.time()\n", - "current_clusters = [X_csr]\n", - "for d in range(cluster_depth):\n", - " next_clusters = []\n", - " for cur_X in current_clusters:\n", - " if cur_X.shape[0] >= nr_splits:\n", - " kmeans = KMeans(n_clusters=nr_splits).fit(cur_X)\n", - " next_clusters.append(cur_X[kmeans.labels_ == 0])\n", - " next_clusters.append(cur_X[kmeans.labels_ == 1])\n", - " else:\n", - " next_clusters.append(cur_X)\n", - " \n", - " current_clusters = next_clusters\n", - "pred_time = time.time() - start_time\n", - "\n", - "print(f\"scikit-learn takes {pred_time:.4f} seconds for hierarchical clustering with a depth {cluster_depth}.\")" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/tutorials/kdd22/Session 5 eXtreme Multi-label Classification with XR-Transformer.ipynb b/tutorials/kdd22/Session 5 eXtreme Multi-label Classification with XR-Transformer.ipynb deleted file mode 100644 index 28e24106..00000000 --- a/tutorials/kdd22/Session 5 eXtreme Multi-label Classification with XR-Transformer.ipynb +++ /dev/null @@ -1,1013 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "3ee4c624-46ff-4a69-b315-097a4a471737", - "metadata": {}, - "source": [ - "# How to Leverage Transformers in PECOS\n", - "\n", - "Extreme multi-label text classification (XMC) seeks to find relevant labels from an\n", - "extreme large label collection for a given text input.\n", - "The current state of the art result on XMC benchmarks are established by **XR-Transformer** [[NeurIPS21](https://arxiv.org/pdf/2110.00685.pdf)], which leverages recursively fine-tuned transformer encoders in text feature extaction.\n", - "\n", - "In this section, we will demostrate how you can use XR-Transformer to solve the XMC problems.\n", - "\n", - "### Download dataset and fine-tuned Transformer encoders" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "d380a9b4-dcf4-4fab-b07f-e9e0c38a15b8", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "2022-08-13 21:39:55 URL:https://ia802308.us.archive.org/21/items/pecos-dataset/xmc-base/wiki10-31k.tar.gz [162277861/162277861] -> \"wiki10-31k.tar.gz\" [1]\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "xmc-base/wiki10-31k/output-items.txt\n", - "xmc-base/wiki10-31k/tfidf-attnxml\n", - "xmc-base/wiki10-31k/tfidf-attnxml/X.trn.npz\n", - "xmc-base/wiki10-31k/tfidf-attnxml/X.tst.npz\n", - "xmc-base/wiki10-31k/X.trn.txt\n", - "xmc-base/wiki10-31k/X.tst.txt\n", - "xmc-base/wiki10-31k/Y.trn.npz\n", - "xmc-base/wiki10-31k/Y.trn.txt\n", - "xmc-base/wiki10-31k/Y.tst.npz\n", - "xmc-base/wiki10-31k/Y.tst.txt\n", - "./work_dir/xr-transformer-encoder/wiki10-31k\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/text_encoder\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/text_encoder/pytorch_model.bin\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/text_encoder/config.json\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/C.npz\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/param.json\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/text_model\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/text_tokenizer\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/text_tokenizer/tokenizer_config.json\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/text_tokenizer/special_tokens_map.json\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/text_tokenizer/vocab.txt\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder/text_tokenizer/tokenizer.json\n", - "./work_dir/xr-transformer-encoder/wiki10-31k/bert/param.json\n" - ] - } - ], - "source": [ - "%%bash\n", - "DATASET=\"wiki10-31k\"\n", - "wget -nv -nc https://archive.org/download/pecos-dataset/xmc-base/${DATASET}.tar.gz\n", - "tar --skip-old-files -zxf ${DATASET}.tar.gz \n", - "find xmc-base/${DATASET}/*\n", - "wget -q https://archive.org/download/xr-transformer-demos/${DATASET}-bert.tar.gz\n", - "mkdir -p ./work_dir/xr-transformer-encoder\n", - "tar -zxf ./${DATASET}-bert.tar.gz -C ./work_dir/xr-transformer-encoder\n", - "find ./work_dir/xr-transformer-encoder/*" - ] - }, - { - "cell_type": "markdown", - "id": "242c18b9-5356-4385-bffb-8b8c18d7ae06", - "metadata": {}, - "source": [ - "## Outline in this Session\n", - "\n", - " 1. XR-Transformer Overview\n", - " 2. Hands on training and evaluation\n", - " 3. How to customize the parameter settings\n", - " 4. Command line interface tools\n", - " 5. Example pf using XR-Transformer on your custom dataset" - ] - }, - { - "cell_type": "markdown", - "id": "235e6dce-80eb-48f7-8604-f1695f04c878", - "metadata": {}, - "source": [ - "## 1. XR-Transformer Overview\n", - "\n", - "## 1.1 Benchmarking XR-Transformer on public XMC datasets\n", - "\n", - "A comparison of Precision@1,3,5 and training time on 3 public XMC benchmarking datasets.\n", - "\n", - "PECOS XR-Transformer achieves the highgest accuracy while taking significantly less time to train (20-50X faster than X-Transformer).\n", - "\n", - "\n", - "\n", - "\n", - "
\n", - "\n", - "\n", - "## 1.2 Training Procedures\n", - "\n", - "One important thing to note is that XR-Transformer leverages multi-resolution fine-tuning to allow tuning from easy to hard tasks. The training can be separated into three steps:\n", - "\n", - "* **Step1**: Label features are computed and are used to build preliminary hierarchical label tree (HLT).\n", - "* **Step2**: Fine-tune the transformer encoder on the chosen levels of the preliminary HLT.\n", - "* **Step3**: Concatenate final instance embeddings and sparse features and train the linear rankers on the refined HLT.\n", - "\n", - "

\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "id": "734e63c1-1d08-49c0-b060-1aed92a76159", - "metadata": {}, - "source": [ - "## 2. Hands on training and evaluation\n", - "### 2.1 Data Loading\n", - "\n", - "XR-Transformer model takes both raw text as well as text numerical features (such as TFIDF) as input." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "b99f59a5-b1f9-4d73-b09e-cf129923eb7f", - "metadata": {}, - "outputs": [], - "source": [ - "import logging\n", - "import numpy as np\n", - "from pecos.utils import smat_util, logging_util\n", - "\n", - "# set logging level to WARNING(1)\n", - "# you can change this to INFO(2) or DEBUG(3) if you'd like to see more logging\n", - "LOGGER = logging.getLogger(__name__)\n", - "logging_util.setup_logging_config(level=1)\n", - "\n", - "# load training data\n", - "X_feat_trn = smat_util.load_matrix(\"xmc-base/wiki10-31k/tfidf-attnxml/X.trn.npz\", dtype=np.float32)\n", - "Y_trn = smat_util.load_matrix(\"xmc-base/wiki10-31k/Y.trn.npz\", dtype=np.float32)\n", - "\n", - "with open(\"xmc-base/wiki10-31k/X.trn.txt\", 'r') as fin:\n", - " X_txt_trn = [xx.strip() for xx in fin.readlines()]\n", - "\n", - "# load test data\n", - "X_feat_tst = smat_util.load_matrix(\"xmc-base/wiki10-31k/tfidf-attnxml/X.tst.npz\", dtype=np.float32)\n", - "Y_tst = smat_util.load_matrix(\"xmc-base/wiki10-31k/Y.tst.npz\", dtype=np.float32)\n", - "\n", - "with open(\"xmc-base/wiki10-31k/X.tst.txt\", 'r') as fin:\n", - " X_txt_tst = [xx.strip() for xx in fin.readlines()]" - ] - }, - { - "cell_type": "markdown", - "id": "b37a6873-2fcf-4a37-9d43-166dbf3b689d", - "metadata": {}, - "source": [ - "### 2.2 Model Training and Evaluation\n", - "\n", - "In this section, we will compare the performance of three models:\n", - "1. XR-Linear model with only sparse TF-IDF features\n", - "2. XR-Transformer model without fine-tuning\n", - "3. XR-Transformer model with fine-tuning\n", - "\n", - "XR-Transformer parameters for 6 public XMC benchmark datasets (i.e. `Eurlex-4K`, `Wiki10-31K`,\n", - "`AmazonCat-13K`, `Wiki-500K`, `Amazon-670K`, `Amazon-3M`) are released. For this turoiral we will be using `Wiki10-31K` with `bert-base-uncased` encoder as an example." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "7e4d3ab3-31ec-4e10-8c1d-6bb30bdfbcc7", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "import json\n", - "import requests\n", - "from pecos.xmc.xtransformer.model import XTransformer\n", - "\n", - "# get XR-Transformer training params\n", - "param_url = \"https://raw.githubusercontent.com/amzn/pecos/mainline/examples/xr-transformer-neurips21/params/wiki10-31k/bert/params.json\"\n", - "params = json.loads(requests.get(param_url).text)\n", - " \n", - "wiki31k_train_params = XTransformer.TrainParams.from_dict(params[\"train_params\"])\n", - "wiki31k_pred_params = XTransformer.PredParams.from_dict(params[\"pred_params\"])\n", - "\n", - "# you can view the detailed parameter setting via\n", - "#print(json.dumps(wiki31k_train_params.to_dict(), indent=True))\n", - "#print(json.dumps(wiki31k_pred_params.to_dict(), indent=True))" - ] - }, - { - "cell_type": "markdown", - "id": "21f98b0f-cf5b-4844-a029-48ea339019f5", - "metadata": {}, - "source": [ - "#### Baseline 1: XR-Linear\n", - "Let's train a XR-Linear model on the TF-IDF features using the same hyper-parameters." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "2394cb6e-51f0-4485-85c2-74158445dc81", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation metrics of XR-Linear model\n", - "prec = 84.96 81.82 76.30 70.70 65.67 61.46 57.92 54.63 51.77 49.16\n", - "recall = 5.02 9.66 13.40 16.41 18.91 21.12 23.09 24.76 26.30 27.64\n" - ] - } - ], - "source": [ - "# construct label hierarchy\n", - "from pecos.xmc import Indexer, LabelEmbeddingFactory\n", - "cluster_chain = Indexer.gen(\n", - " LabelEmbeddingFactory.create(Y_trn, X_feat_trn, method=\"pifa\"),\n", - " train_params=wiki31k_train_params.refined_indexer_params,\n", - ")\n", - "\n", - "# train XR-Linear model\n", - "from pecos.xmc.xlinear import XLinearModel\n", - "xlm = XLinearModel.train(\n", - " X_feat_trn,\n", - " Y_trn,\n", - " C=cluster_chain,\n", - " train_params=wiki31k_train_params.ranker_params,\n", - " pred_params=wiki31k_pred_params.ranker_params,\n", - ")\n", - "\n", - "# predict on test set with XR-Linear model\n", - "P_xlm = xlm.predict(X_feat_tst)\n", - "\n", - "# compute metrics using ground truth\n", - "metrics = smat_util.Metrics.generate(Y_tst, P_xlm)\n", - "print(\"Evaluation metrics of XR-Linear model\")\n", - "print(metrics)" - ] - }, - { - "cell_type": "markdown", - "id": "d3a7618e-7302-4486-9d82-00e95ec9a61c", - "metadata": {}, - "source": [ - "#### Baseline 2: XR-Transformer without fine-tuning" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "24344e15-e183-45ad-abda-7239b6d5e144", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForXMC: ['cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias']\n", - "- This IS expected if you are initializing BertForXMC from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", - "- This IS NOT expected if you are initializing BertForXMC from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation metrics of XR-Transformer (not fine-tuned)\n", - "prec = 85.22 82.55 77.26 72.15 67.42 63.13 59.33 56.08 53.02 50.24\n", - "recall = 5.05 9.76 13.58 16.74 19.41 21.68 23.64 25.41 26.92 28.22\n" - ] - } - ], - "source": [ - "# define the problem\n", - "from pecos.xmc.xtransformer.module import MLProblemWithText\n", - "prob = MLProblemWithText(X_txt_trn, Y_trn, X_feat=X_feat_trn)\n", - "\n", - "# disable fine-tuning, directly use pre-trained bert model from huggingface\n", - "wiki31k_train_params.do_fine_tune = False\n", - "\n", - "# train XR-Transformer (without fine-tuning)\n", - "# this will be slow on CPU only machine\n", - "xrt_pretrained = XTransformer.train(\n", - " prob,\n", - " train_params=wiki31k_train_params,\n", - " pred_params=wiki31k_pred_params,\n", - ")\n", - "\n", - "# predict and compute metrics\n", - "P_xrt_pretrained = xrt_pretrained.predict(X_txt_tst, X_feat=X_feat_tst)\n", - "metrics = smat_util.Metrics.generate(Y_tst, P_xrt_pretrained)\n", - "print(\"Evaluation metrics of XR-Transformer (not fine-tuned)\")\n", - "print(metrics)" - ] - }, - { - "cell_type": "markdown", - "id": "fea3745a-2dc7-47e8-b3b4-c8c4ce331e05", - "metadata": {}, - "source": [ - "#### Model: XR-Transformer\n", - "For demo purpose, let's disable fine-tuning and load an already fine-tuned encoder directly (i.e. skip step 1&2).\n", - "\n", - "End-to-end training of XR-Transformer on **Wiki10-31K** dataset will take around 30min on **p3.16xlarge** instance.\n", - "If you are running this on equivalent or more powerful machine, you can also turn on `DO_FINE_TUNE_NOW` and train XR-Transformer end-to-end." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "bbada274-cfe3-4ef4-a5fe-448c00c3e6cd", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation metrics of XR-Transformer\n", - "prec = 87.95 83.54 78.79 73.95 69.43 65.14 61.08 57.70 54.63 51.97\n", - "recall = 5.25 9.89 13.84 17.14 19.99 22.36 24.35 26.16 27.73 29.21\n" - ] - } - ], - "source": [ - "DO_FINE_TUNE_NOW = False\n", - "\n", - "if DO_FINE_TUNE_NOW:\n", - " wiki31k_train_params.do_fine_tune = True\n", - "else:\n", - " # skip fine-tuning and use existing fine-tuned encoder\n", - " wiki31k_train_params.do_fine_tune = False\n", - " wiki31k_train_params.matcher_params_chain[0].init_model_dir = \"./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder\"\n", - "\n", - "# this will be slow on CPU only machine\n", - "xrt_fine_tuned = XTransformer.train(\n", - " prob,\n", - " clustering=cluster_chain,\n", - " train_params=wiki31k_train_params,\n", - " pred_params=wiki31k_pred_params,\n", - ")\n", - "\n", - "P_xrt_fine_tuned = xrt_fine_tuned.predict(X_txt_tst, X_feat=X_feat_tst)\n", - "metrics = smat_util.Metrics.generate(Y_tst, P_xrt_fine_tuned, topk=10)\n", - "print(\"Evaluation metrics of XR-Transformer\")\n", - "print(metrics)" - ] - }, - { - "cell_type": "markdown", - "id": "a2e7f85b-96f7-4801-ab86-410c5f0115a1", - "metadata": {}, - "source": [ - "### 2.3 Save and load model, get transformer embeddings\n", - "Note you can pass keyword arguments of `XLinear.load` to `XTransformer.load` such as `is_predict_only`." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "233050a5-0398-48d0-8ef8-fc58232de669", - "metadata": {}, - "outputs": [], - "source": [ - "model_folder = \"./work_dir/my_xrt\"\n", - "xrt_fine_tuned.save(model_folder)\n", - "del xrt_fine_tuned\n", - "xrt_fine_tuned = XTransformer.load(model_folder, is_predict_only=True)" - ] - }, - { - "cell_type": "markdown", - "id": "96b8e6fd-7755-4068-acdb-bae212c26c4f", - "metadata": {}, - "source": [ - "For BERT model, ebmeddings are from the [CLS] token." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "9cd7bcd2-0798-4bdd-8460-9e7f114d71fb", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Generated test embedding type= with shape=(6616, 768)\n" - ] - } - ], - "source": [ - "X_emb_tst = xrt_fine_tuned.encode(\n", - " X_txt_tst,\n", - " batch_size=256,\n", - " batch_gen_workers=8,\n", - ")\n", - "print(f\"Generated test embedding type={type(X_emb_tst)} with shape={X_emb_tst.shape}\")" - ] - }, - { - "cell_type": "markdown", - "id": "dbcb0589-54a2-41fa-bb25-0f9343f1a9bc", - "metadata": {}, - "source": [ - "### 2.4 Training without TFIDF features\n", - "\n", - "The XR-Transformer module can also be used with only text features when numerical features like TFIDF are not available." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "57b1860c-3bc7-4c46-ba57-3bab8ebd012f", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation metrics of XR-Transformer (without TFIDF)\n", - "prec = 86.23 81.57 76.04 70.60 65.52 61.38 57.65 54.31 51.45 48.81\n", - "recall = 5.11 9.61 13.28 16.31 18.80 21.00 22.90 24.54 26.06 27.38\n" - ] - } - ], - "source": [ - "prob_only_text = MLProblemWithText(X_txt_trn, Y_trn)\n", - "wiki31k_train_params.do_fine_tune = False\n", - "wiki31k_train_params.matcher_params_chain[0].init_model_dir = \"./work_dir/xr-transformer-encoder/wiki10-31k/bert/text_encoder\"\n", - "\n", - "# this will be slow on CPU only machine\n", - "xrt_only_text = XTransformer.train(\n", - " prob_only_text,\n", - " clustering=cluster_chain,\n", - " train_params=wiki31k_train_params,\n", - " pred_params=wiki31k_pred_params,\n", - ")\n", - "\n", - "P_xrt_only_text = xrt_only_text.predict(X_txt_tst)\n", - "metrics = smat_util.Metrics.generate(Y_tst, P_xrt_only_text, topk=10)\n", - "print(\"Evaluation metrics of XR-Transformer (without TFIDF)\")\n", - "print(metrics)" - ] - }, - { - "cell_type": "markdown", - "id": "307ac8e2-555d-4bc2-a187-6a590bc7944c", - "metadata": {}, - "source": [ - "## 3 How to customize the parameter settings\n", - "For your custom dataset, it is recommended to start from the pre-defined parameters or the default value and make proper modifications based on the specific problem.\n", - "\n", - "### 3.1 Training Parameters of XTransformer.\n", - "\n", - "```\n", - "xrt_train_params = XTransformer.TrainParams.from_dict(\n", - "{\n", - " \"do_fine_tune\": [true/false], # if true, do encoder fine-tuning\n", - " \"only_encoder\": [true/false], # if true, skip linear ranker training\n", - " \"max_match_clusters\": INT # max label resolution to fine-tune encoder on\n", - " \"preliminary_indexer_params\": {...}, # (HierarchicalKMeans.TrainParams) parameters to construct preliminary HLT \n", - " \"refined_indexer_params\": {...}, # (HierarchicalKMeans.TrainParams) parameters to construct refined HLT \n", - " \"matcher_params_chain\": [ # fine-tuning parameters. Can be dict or list of dict. If dict, all layers will share the same setting\n", - " {...}, # (TransformerMatcher.TrainParams) fine-tuning parameters for layer-0\n", - " {...}, # (TransformerMatcher.TrainParams) fine-tuning parameters for layer-1\n", - " ...\n", - " ],\n", - " \"ranker_params\": {...}, # (XLinearModel.TrainParams) ranker training parameters\n", - "}\n", - ")\n", - "```\n", - "\n", - "You can get the training and prediction parameters filled with default values by:" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "489c8bab-f7d6-4445-b4b8-c44097cb8415", - "metadata": {}, - "outputs": [], - "source": [ - "train_params = XTransformer.TrainParams.from_dict({}, recursive=True)\n", - "pred_params = XTransformer.PredParams.from_dict({}, recursive=True)" - ] - }, - { - "cell_type": "markdown", - "id": "4f58825d-9e85-4556-bf7f-1a9f1ddde8c4", - "metadata": {}, - "source": [ - "Detailed control over each layer's fine-tuning task is done through `matcher_params_chain`:" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "17ab552c-1845-47fd-9eb8-1d3c13bb9bb0", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\n", - " \"__meta__\": {\n", - " \"class_fullname\": \"pecos.xmc.xtransformer.matcher###TransformerMatcher.TrainParams\"\n", - " },\n", - " \"model_shortcut\": \"bert-base-cased\",\n", - " \"negative_sampling\": \"tfn\",\n", - " \"loss_function\": \"squared-hinge\",\n", - " \"bootstrap_method\": \"linear\",\n", - " \"lr_schedule\": \"linear\",\n", - " \"threshold\": 0.1,\n", - " \"hidden_dropout_prob\": 0.1,\n", - " \"batch_size\": 8,\n", - " \"batch_gen_workers\": 4,\n", - " \"max_active_matching_labels\": null,\n", - " \"max_num_labels_in_gpu\": 65536,\n", - " \"max_steps\": 0,\n", - " \"max_no_improve_cnt\": -1,\n", - " \"num_train_epochs\": 5,\n", - " \"gradient_accumulation_steps\": 1,\n", - " \"weight_decay\": 0,\n", - " \"max_grad_norm\": 1.0,\n", - " \"learning_rate\": 0.0001,\n", - " \"adam_epsilon\": 1e-08,\n", - " \"warmup_steps\": 0,\n", - " \"logging_steps\": 50,\n", - " \"save_steps\": 100,\n", - " \"cost_sensitive_ranker\": false,\n", - " \"pre_tokenize\": true,\n", - " \"pre_tensorize_labels\": true,\n", - " \"use_gpu\": true,\n", - " \"eval_by_true_shorlist\": false,\n", - " \"checkpoint_dir\": \"\",\n", - " \"cache_dir\": \"\",\n", - " \"init_model_dir\": \"\"\n", - "}\n" - ] - } - ], - "source": [ - "print(json.dumps(train_params.matcher_params_chain.to_dict(), indent=True))" - ] - }, - { - "cell_type": "markdown", - "id": "318b7124-00b2-4275-af9a-dbdcbcab39d5", - "metadata": {}, - "source": [ - "### 3.2 Getting the pre-trained models\n", - "\n", - "There are two ways to provide pre-trained Transformer encoder:\n", - "* **Download from huggingface repo** (https://huggingface.co/models): pre-trained model name provided in `model_shortcut` (under `XTransformer.TrainParams.matcher_params_chain`) will be automatically downloaded. (e.x. `bert-base-uncased`)\n", - "* **Load your custom model from local disk**: model path provided by `init_model_dir`. Model should be loadable through `TransformerMatcher.load()`\n", - "\n", - "Note that both `model_shortcut` and `init_model_dir` will only be used in the first fine-tuning layer, as the later ones will just continue on the final state from parent encoder.\n", - "\n", - "A simple example if you want to construct your custom pre-trained model for XR-Transformer fine-tuning:" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "f54abca4-d07b-4cef-a68d-6b1b40d83280", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias']\n", - "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", - "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n", - "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\n", - "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n", - "Some weights of the model checkpoint at work_dir/my_pre_trained_model/text_encoder were not used when initializing BertForXMC: ['classifier.bias', 'classifier.weight']\n", - "- This IS expected if you are initializing BertForXMC from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", - "- This IS NOT expected if you are initializing BertForXMC from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n", - "08/13/2022 21:48:31 - WARNING - pecos.xmc.xtransformer.matcher - XMC text_model of BertForXMC not initialized from pre-trained model.\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - " model loaded with encoder_type=bert num_labels=2\n" - ] - } - ], - "source": [ - "from pecos.xmc.xtransformer.matcher import TransformerMatcher\n", - "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n", - "\n", - "init_model_dir = \"work_dir/my_pre_trained_model\"\n", - "\n", - "# example to use your own pre-trained model, here we use huggingface model as an example\n", - "my_tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n", - "my_encoder = AutoModelForSequenceClassification.from_pretrained(\"bert-base-uncased\")\n", - "\n", - "# ...\n", - "# do my own modification/tuning/etc\n", - "# ...\n", - "\n", - "# save my own model to disk\n", - "my_tokenizer.save_pretrained(f\"{init_model_dir}/text_tokenizer\")\n", - "my_encoder.save_pretrained(f\"{init_model_dir}/text_encoder\")\n", - "\n", - "# then the `work_dir` can be fed as `init_model_dir` as initial model.\n", - "# Sanity check: if this dir can be loaded via TransformerMatcher.load(*)\n", - "matcher = TransformerMatcher.load(init_model_dir)\n", - "print(f\"{matcher.__class__} model loaded with encoder_type={matcher.model_type} num_labels={matcher.nr_labels}\")" - ] - }, - { - "cell_type": "markdown", - "id": "31c8cd72-12e1-4aa3-940e-24b36095e440", - "metadata": { - "tags": [] - }, - "source": [ - "## 4. Command line interface tools\n", - "You can achieve the same functionalities with the provided command line tools.\n", - "\n", - "Although we provide basic functionalities to supply training and prediction parameters in the CLI tool `pecos.xmc.xtransformer.train`, `pecos.xmc.xtransformer.predict` and `pecos.xmc.xtransformer.encode`,\n", - "you should supply parameters via a JSON file if you want full control over the training/prediction process.\n", - "\n", - "Similar to the python interface, you can also generate a `.json` file with all of the parameters that you can edit and fill in via\n", - "```bash\n", - "python3 -m pecos.xmc.xtransformer.train --generate-params-skeleton &> params.json\n", - "```\n", - "\n", - "After filling in the desired parameters into `params.json`, the training can be done end2end via:\n", - "```bash\n", - "python3 -m pecos.xmc.xtransformer.train \\\n", - " -t ${T_path} \\\n", - " -x ${X_path} \\\n", - " -y ${Y_path} \\\n", - " -m ${model_dir} \\\n", - " --params-path params.json\n", - "\n", - "python3 -m pecos.xmc.xtransformer.predict \\\n", - " -t ${Tt_path} \\\n", - " -x ${Xt_path} \\\n", - " -m ${model_dir} \\\n", - " -o ${Pt_path}\n", - "```\n", - "where\n", - "* `T_path` and `Tt_path` are the paths to the input text file of the training/test instances. Text files with `N`/`Nt` lines where each line is the text feature of the corresponding training/test instance.\n", - "* `X_path` and `Xt_path` are the paths to the CSR npz or Row-majored npy files of the training/test feature matrices with shape `(N, d)` and `(Nt, d)`.\n", - " * Note that you can use the PECOS built in text preprocessing/vectorizing module [pecos.utils.featurization.text.preprocess](https://github.com/amzn/pecos/tree/mainline/pecos/utils/featurization/text) to generate numerical features if you do not already have them.\n", - " * Usually providing instance numerical features is recommended. However, if you choose not to provide numerical features, `code-path` or `label-feat-path` is required to generate the hierarchical label trees.\n", - "* `Y_path` and `Yt_path` are the paths to the CSR npz files of the training/test label matrices with shape `(N, L)` and `(Nt, L)`.\n", - "* `model_dir` is the path to the model folder where the trained model will be saved to, will be created if not exist.\n", - "* `Pt_path` is the path to save the prediction label matrix with shape `(Nt, L)`\n", - "\n", - "To get the evaluation metrics for top-10 predictions:\n", - "```bash\n", - "python3 -m pecos.xmc.xlinear.evaluate \\\n", - " -y ${Yt_path} \\\n", - " -p ${Pt_path} \\\n", - " -k 10\n", - "```\n", - "You can also get the fine-tuned text embeddings via:\n", - "```bash\n", - "python3 -m pecos.xmc.xtransformer.encode \\\n", - " -t ${Tt_path} \\\n", - " -m ${model_dir} \\\n", - " -o ${Emb_path}\n", - "```\n", - "\n", - "where\n", - "* `Emb_path` is the path to save the prediction label matrix with shape `(Nt, hidden_dim)`" - ] - }, - { - "cell_type": "markdown", - "id": "b1293642-87cb-4148-b206-04fc38340c9c", - "metadata": {}, - "source": [ - "## 5. Example: Use XR-Transformer for your custom dataset\n", - "This section demostrates how you can use XR-Transformer on your custom dataset.\n", - "\n", - "**Note**: The data used here is a dummy dataset only for demo purposes, therefore we don't expect sensical results." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "43657390-f634-4ec9-b940-92ed511a0f32", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "2022-08-13 21:48:32 URL:https://ia601500.us.archive.org/21/items/text2text_demo.tar.gz/text2text_demo.tar.gz [674/674] -> \"text2text_demo.tar.gz\" [1]\n", - "text2text_demo/output-labels.txt\n", - "text2text_demo/testing-data.txt\n", - "text2text_demo/training-data.txt\n" - ] - } - ], - "source": [ - "! wget -nv -nc https://archive.org/download/text2text_demo.tar.gz/text2text_demo.tar.gz\n", - "! tar --skip-old-files -zxf text2text_demo.tar.gz\n", - "! find text2text_demo/*" - ] - }, - { - "cell_type": "markdown", - "id": "8d9af6ab-3f0e-4d0f-a059-6e699b8792df", - "metadata": {}, - "source": [ - "First format your input data into two files `training-data.txt` and `output-labels.txt`.\n", - "\n", - "Each line of `output-labels.txt` corresponds to the text representation of a label:" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "7fe80fa9-e6a3-455a-8f1e-26d181c58e9c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Artificial intelligence researchers\n", - "Computability theorists\n", - "British computer scientists\n", - "Machine learning researchers\n", - "Turing Award laureates\n", - "Deep Learning\n" - ] - } - ], - "source": [ - "! cat ./text2text_demo/output-labels.txt" - ] - }, - { - "cell_type": "markdown", - "id": "ca3f5ac8-b738-432d-806c-22e1f1d369e0", - "metadata": {}, - "source": [ - "The `training-data.txt` stores input corpus and training signals. Each line in the file consists of two elements that represent the comma-separated label IDs and the input text of a data instance: \n", - "\n", - "

\n", - "label_idx1,label_idx2,... <TAB> instance_text

" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "1d785ef4-6b0f-4827-a9c4-8a5ffe9aea99", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "0,1,2\tAlan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.\n", - "0,2,3\tHinton was co-author of a highly cited paper published in 1986 that popularized the backpropagation algorithm for training multi-layer neural networks.\n", - "3,4,5\tHinton received the 2018 Turing Award, together with Yoshua Bengio and Yann LeCun, for their work on artificial intelligence and deep learning.\n", - "0,3,5\tYoshua Bengio is a Canadian computer scientist, most noted for his work on artificial neural networks and deep learning.\n" - ] - } - ], - "source": [ - "! cat ./text2text_demo/training-data.txt" - ] - }, - { - "cell_type": "markdown", - "id": "1d2465b7-5f5c-4c88-b53f-114bdef3ef2e", - "metadata": {}, - "source": [ - "First parse the `training-data.txt` into training corpus and label matrix:" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "a7006bb0-15d4-46b3-aaea-38ca484eea92", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Constructed training corpus len=4, training label matrix with shape=(4, 6) and nnz=12\n" - ] - } - ], - "source": [ - "from pecos.utils.featurization.text.preprocess import Preprocessor\n", - "\n", - "parsed_result = Preprocessor.load_data_from_file(\n", - " \"./text2text_demo/training-data.txt\",\n", - " \"./text2text_demo/output-labels.txt\",\n", - ")\n", - "Y = parsed_result[\"label_matrix\"]\n", - "X_txt = parsed_result[\"corpus\"]\n", - "\n", - "print(f\"Constructed training corpus len={len(X_txt)}, training label matrix with shape={Y.shape} and nnz={Y.nnz}\")" - ] - }, - { - "cell_type": "markdown", - "id": "7b130f88-f40f-4bca-8675-7c454e5fd774", - "metadata": {}, - "source": [ - "Build TF-IDF model with training corpus:" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "a54c8b14-8fc0-49b6-999d-25b28d747755", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Constructed training feature matrix with shape=(4, 125) and nnz=151\n" - ] - } - ], - "source": [ - "vectorizer_config = {\n", - " \"type\": \"tfidf\",\n", - " \"kwargs\": {\n", - " \"base_vect_configs\": [\n", - " {\n", - " \"ngram_range\": [1, 2],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"word\",\n", - " },\n", - " ],\n", - " },\n", - "}\n", - "\n", - "tfidf_model = Preprocessor.train(X_txt, vectorizer_config)\n", - "X_feat = tfidf_model.predict(X_txt)\n", - "\n", - "print(f\"Constructed training feature matrix with shape={X_feat.shape} and nnz={X_feat.nnz}\")" - ] - }, - { - "cell_type": "markdown", - "id": "fae807f1-f34c-4d11-9e5a-50e87ed42443", - "metadata": {}, - "source": [ - "Train XR-Transformer with all default settings:" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "058ffcdf-54c7-4eff-af97-2418d5ff2b5b", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForXMC: ['cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias']\n", - "- This IS expected if you are initializing BertForXMC from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", - "- This IS NOT expected if you are initializing BertForXMC from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n", - "/home/ec2-user/miniconda3/envs/tutorial_env/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n", - " warnings.warn(\n" - ] - } - ], - "source": [ - "from pecos.xmc.xtransformer.model import XTransformer\n", - "from pecos.xmc.xtransformer.module import MLProblemWithText\n", - "prob = MLProblemWithText(X_txt, Y, X_feat=X_feat)\n", - "custom_xtf = XTransformer.train(prob)" - ] - }, - { - "cell_type": "markdown", - "id": "f94f8eae-47c7-4361-bed7-cdc5894e03a3", - "metadata": {}, - "source": [ - "Save tfidf model and XR-Transformer model to disk:" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "7c270281-c729-44f1-a9b4-a8765048e81e", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "custom_model_dir = \"work_dir/custom_model\"\n", - "os.makedirs(custom_model_dir, exist_ok=True)\n", - "\n", - "tfidf_model.save(f\"{custom_model_dir}/tfidf_model\")\n", - "custom_xtf.save(f\"{custom_model_dir}/xrt_model\")" - ] - }, - { - "cell_type": "markdown", - "id": "fd24024b-7d19-44c7-9d12-c9e496f23526", - "metadata": {}, - "source": [ - "Load tfidf model and XR-Transformer model from disk:" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "id": "318e04f4-be60-4b38-9c37-3b04092bef42", - "metadata": {}, - "outputs": [], - "source": [ - "custom_xtf = XTransformer.load(f\"{custom_model_dir}/xrt_model\")\n", - "tfidf_model = Preprocessor.load(f\"{custom_model_dir}/tfidf_model\")" - ] - }, - { - "cell_type": "markdown", - "id": "3a521df7-d595-43ac-8412-1060eaf2d5a9", - "metadata": {}, - "source": [ - "Predict on a test input:" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "id": "39922231-b05e-4a1c-b5ba-f23360d66adf", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Input text: In 1989, Yann LeCun et al. applied the standard backpropagation algorithm on neural networks for hand digit recognition.\n", - "Predicted label: Machine learning researchers\n", - "Predicted score: 0.7240481376647949\n" - ] - } - ], - "source": [ - "test_input = [\"In 1989, Yann LeCun et al. applied the standard backpropagation algorithm on neural networks for hand digit recognition.\"]\n", - "\n", - "P = custom_xtf.predict(\n", - " test_input,\n", - " X_feat=tfidf_model.predict(test_input),\n", - " only_topk=1\n", - ")\n", - "\n", - "with open(\"./text2text_demo/output-labels.txt\", 'r') as fin:\n", - " output_items = [ll.strip() for ll in fin.readlines()]\n", - "\n", - "for i, t in enumerate(test_input):\n", - " print(f\"Input text: {t}\")\n", - " print(f\"Predicted label: {output_items[P[i, :].indices[0]]}\")\n", - " print(f\"Predicted score: {P[i, :].data[0]}\")" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/tutorials/kdd22/imgs/dist-xlinear.png b/tutorials/kdd22/imgs/dist-xlinear.png deleted file mode 100644 index c7f2e952..00000000 Binary files a/tutorials/kdd22/imgs/dist-xlinear.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/illus_customized_model.jpg b/tutorials/kdd22/imgs/illus_customized_model.jpg deleted file mode 100644 index a292f01e..00000000 Binary files a/tutorials/kdd22/imgs/illus_customized_model.jpg and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-compare.png b/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-compare.png deleted file mode 100644 index bcc64f2b..00000000 Binary files a/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-compare.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-example.png b/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-example.png deleted file mode 100644 index 1559a197..00000000 Binary files a/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-example.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-rcv1.png b/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-rcv1.png deleted file mode 100644 index 6acc8d89..00000000 Binary files a/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-rcv1.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-sift.png b/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-sift.png deleted file mode 100644 index ac1b9e90..00000000 Binary files a/tutorials/kdd22/imgs/pecos-kdd22-tutorial-hnsw-sift.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos_beam_search.png b/tutorials/kdd22/imgs/pecos_beam_search.png deleted file mode 100644 index f13ac0b4..00000000 Binary files a/tutorials/kdd22/imgs/pecos_beam_search.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos_label_matrix.png b/tutorials/kdd22/imgs/pecos_label_matrix.png deleted file mode 100644 index 837cc9fc..00000000 Binary files a/tutorials/kdd22/imgs/pecos_label_matrix.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos_matcher_ranker.png b/tutorials/kdd22/imgs/pecos_matcher_ranker.png deleted file mode 100644 index 8302b8b6..00000000 Binary files a/tutorials/kdd22/imgs/pecos_matcher_ranker.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos_pipeline.png b/tutorials/kdd22/imgs/pecos_pipeline.png deleted file mode 100644 index 634edf2b..00000000 Binary files a/tutorials/kdd22/imgs/pecos_pipeline.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos_spmm.png b/tutorials/kdd22/imgs/pecos_spmm.png deleted file mode 100644 index 6f7278b4..00000000 Binary files a/tutorials/kdd22/imgs/pecos_spmm.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos_xmc_examples.png b/tutorials/kdd22/imgs/pecos_xmc_examples.png deleted file mode 100644 index 85ecafac..00000000 Binary files a/tutorials/kdd22/imgs/pecos_xmc_examples.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos_xmr_framework.png b/tutorials/kdd22/imgs/pecos_xmr_framework.png deleted file mode 100644 index dcef9d1c..00000000 Binary files a/tutorials/kdd22/imgs/pecos_xmr_framework.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/pecos_xrtransformer.png b/tutorials/kdd22/imgs/pecos_xrtransformer.png deleted file mode 100644 index 10ff1d75..00000000 Binary files a/tutorials/kdd22/imgs/pecos_xrtransformer.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/xrtransformer_prec135.png b/tutorials/kdd22/imgs/xrtransformer_prec135.png deleted file mode 100644 index 33713e27..00000000 Binary files a/tutorials/kdd22/imgs/xrtransformer_prec135.png and /dev/null differ diff --git a/tutorials/kdd22/imgs/xrtransformer_trainingtime.png b/tutorials/kdd22/imgs/xrtransformer_trainingtime.png deleted file mode 100644 index 3e9fbdf3..00000000 Binary files a/tutorials/kdd22/imgs/xrtransformer_trainingtime.png and /dev/null differ