High memory consumption with large resource counts

[catherinevee]

Description

When running drift detection on large AWS environments (>5000 resources), the application experiences significant memory consumption, sometimes exceeding 8GB RAM. This impacts usability for enterprise customers with large-scale deployments.

Current Behavior

• Memory usage grows linearly with resource count
• No streaming/pagination for large result sets
• All resources loaded into memory simultaneously

Expected Behavior

• Memory usage should remain bounded regardless of resource count
• Implement streaming for large datasets
• Use pagination for API calls and result processing

Steps to Reproduce

1. Configure AWS provider with access to account with 5000+ resources
2. Run driftmgr discover --provider aws --all-regions
3. Monitor memory usage with top or task manager
4. Observe memory consumption exceeding 8GB

Environment

• DriftMgr version: 1.0.0
• OS: Ubuntu 22.04
• Go version: 1.23
• AWS account with ~7500 resources across 6 regions

Proposed Solution

• Implement resource streaming with configurable batch sizes
• Add --max-memory flag to limit memory consumption
• Use disk-based caching for large datasets
• Implement progressive loading in dashboard mode

Impact

This issue affects enterprise customers and prevents adoption in large-scale environments.