Skip to content

Add caching and rate limiting support? #376

@SnowCheetos

Description

@SnowCheetos

When working in a notebook or when fetching from static endpoints, there often come times where the same endpoint is called, which would benefit from having a cached URL session. Also sometimes when I need to make a large volume of requests in a short amount of time, it risks hitting the AV rate limit (which I believe is 25/day for free tier, 75/min for the next tier, not sure after that). It'd be nice if I can pass in a custom requests session that handles these things. I know that for yfinance, you can construct a custom session and use it with the Ticker class like this

from requests import Session
from requests_cache import CacheMixin, SQLiteCache
from requests_ratelimiter import LimiterMixin, MemoryQueueBucket
from pyrate_limiter import Duration, RequestRate, Limiter
class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
    pass

session = CachedLimiterSession(
    limiter=Limiter(RequestRate(2, Duration.SECOND*5)),  # max 2 requests per 5 seconds
    bucket_class=MemoryQueueBucket,
    backend=SQLiteCache("yfinance.cache"),
)

ticker = yf.Ticker('msft', session=session)
# The scraped response will be stored in the cache
ticker.actions

I think the AV client could also benefit from something like this. Maybe add an optional attribute session to class AlphaVantage(object): so that def _handle_api_call(self, url): can default to use the custom session if available?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions