Expanding our guide to flaky tests

[kalverra]

---

Below is a summarization created by an LLM (gpt-4-0125-preview). Be mindful of hallucinations and verify accuracy.

Why

The changes made to the e2e-flaky-test-guide.md document shift its focus from specifically addressing E2E (End-to-End) flakiness in tests to a broader approach applicable to flaky tests in Go programming. These modifications aim to provide developers with a comprehensive guide on diagnosing and resolving flakiness in tests, emphasizing the impact even a minimal flakiness rate can have on development velocity. The updated guide includes general tips for reproducing and fixing flaky tests, specialized strategies for handling flaky tests in the Chainlink E2E testing environment, and realistic considerations for when tests run in CI/CD pipelines.

What

• e2e-flaky-test-guide.md
  • Updated the title to "Finding the Root Cause of Test Flakes in Go" from "Finding the Root Cause of E2E Test Flakes", broadening the scope of the guide beyond just E2E tests.
  • Introduced a new section explaining why even a small percentage of flakiness is worth fixing, using a mathematical example to highlight the compounded impact on development velocity.
  • Removed specific references to GitHub Runners' hardware limitations and detailed instructions on dealing with low-resource environments, indicating a shift away from environment-specific troubleshooting.
  • Added general tips for reproducing flakiness and detailed steps for diagnosing and fixing flaky tests, including running tests in isolation, checking for race conditions, and emulating CI environments locally.
  • Included instructions on using Docker and the act tool to emulate CI environments, emphasizing the importance of testing in conditions similar to the target deployment environment.
  • Provided guidance on when and how to give up on fixing a flaky test, suggesting documenting the issue and temporarily disabling the test if necessary.
  • Focused on Chainlink E2E tests with steps on finding flakes using flakeguard, exploring logs and debugging, and considering real-world scenarios that might affect test outcomes.
  • Highlighted the potential need to adjust resource allocations for E2E tests in CI environments, suggesting telemetry as a tool for understanding test resource consumption.

[cl-sonarqube-production]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube