Develop workaround for pubmed API

[tubamos]

Domain

data pipeline

Description

Develop a strategy and write a workaround for the PubMed API's limitation that caps free users' results at 10K. The solution might involve segmenting API calls by publication year to avoid hitting the cap.

User Story

• As a user,
• I want as much of the lastest medical research information available as possible,
• to populate the custom context that will imporve the responses I get from querrying the LLMs.

Acceptance Criteria

• The system can handle PubMed API queries without hitting the 10K limit by segmenting searches by year or other criteria.
• The implementation includes a method for dynamically adjusting query parameters to stay within the API limits.
• Testing confirms that the workaround allows for complete data retrieval from PubMed without errors related to the limits.
• The solution is integrated into the existing data acquisition pipeline and works correctly in tandem with the rest of the scrapers.

Definition of Done

• The feature has been fully implemented.
• The feature has been manually tested and works as expected without critical bugs.
• The feature code is documented with clear explanations of its functionality and usage.
• The feature code has been reviewed and approved by at least one team member.
• The feature branches have been merged into the main branch and closed.
• The feature utility, function and usage have been documented in the respective project wiki on github.