Feature/trust eval integration

[shanghongsim]

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

• I added new unit tests to cover this change
• I believe this change is already covered by existing unit tests

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[logan-markewich]

It might be better if this implemented the actual base classes we have for evaluation?

llama_index/llama-index-core/llama_index/core/evaluation/base.py

Line 44 in 659f9fa

class BaseEvaluator(PromptMixin):

Similar to the other evaluation integrations

llama_index/llama-index-integrations/evaluation/llama-index-evaluation-tonic-validate/llama_index/evaluation/tonic_validate/answer_consistency.py

Line 12 in 659f9fa

class AnswerConsistencyEvaluator(BaseEvaluator):

[shanghongsim]

Hi Logan, thank you for reviewing my PR. I did see how others have implemented, but all of them implemented the BaseEvaluator to process query by query. For my integration, there are some metrics that work more on a macro level (i.e. over a few queries) and are unable to work on a single query level. Am I able to change the aevaluate to take in multiple queries? What the best way to do this? Grateful for any advice you can offer!

@abstractmethod
    async def aevaluate(
        self,
        query: Optional[str] = None,
        response: Optional[str] = None,
        contexts: Optional[Sequence[str]] = None,
        **kwargs: Any,
    ) -> EvaluationResult:
        """Run evaluation with query string, retrieved contexts,
        and generated response string.

        Subclasses can override this method to provide custom evaluation logic and
        take in additional arguments.
        """
        raise NotImplementedError

[logan-markewich]

That does make sense. Hmmm. Is there a way that those metrics can be tracked under a different method?

The only reason I ask for this is, this integration does not even depend on anything from llama-index right now (not using our documents/nodes, not our using llms or embeddings, not using our query engines, etc.), so it doesn't quite make sense to merge 🤔

[logan-markewich]

llama-index isn't a dependency on the package either lol so it seems like users should just use the trust_eval package or this code could live inside trust_eval