Add helper method to wait for log message to be observed (#17589)

* Add helper method to wait for log message to be observed

There appears to be a race condition in tests whereby log messages are not
observed with a single interogation of the container logs. This commit adds a
helper method to wait for log messages. This should make the tests more
resilient when there is a delay in log messages being captured.

* Use local scop container ref

Due to a copy paste error, the wrong reference to container (instance var)
was being used. When test file that did not define this uses the helper
it failed. The helper should use the ref explicitly passed to the method.

Author: donoghuc

qa/docker/shared_examples/container_options.rb

@@ -88,8 +88,7 @@
       expect(output_plugins[0].dig('name')).to eql('elasticsearch')
 
       # check if logs contain the ES request with the resolved ${USER}
-      container_logs = @container.logs(stdout: true)
-      expect(container_logs.include?('https://kimchy:xxxxxx@es:9200')).to be true
+      wait_for_log_message(@container, 'https://kimchy:xxxxxx@es:9200')
     end
   end
 end

qa/docker/shared_examples/xpack.rb

@@ -48,14 +48,11 @@
         expect(settings['xpack.management.pipeline.id']).to eq("${XPACK_MANAGEMENT_PIPELINE_ID}")
         expect(settings['xpack.management.elasticsearch.hosts']).to eq("${XPACK_MANAGEMENT_ELASTICSEARCH_HOSTS}")
 
-        # get container logs
-        container_logs = container.logs(stdout: true)
-
         # check if logs contain node3 & node4 values actually resolved and used
-        expect(container_logs.include?('pipeline_id=>["*"]')).to be true
+        wait_for_log_message(container, 'pipeline_id=>["*"]', :stdout)
         # note that, we are not spinning up ES nodes, so values can be in errors or in pool update logs
-        expect(container_logs.include?('http://node3:9200')).to be true
-        expect(container_logs.include?('http://node4:9200')).to be true
+        wait_for_log_message(container, 'http://node3:9200', :stdout)
+        wait_for_log_message(container, 'http://node4:9200', :stdout)
       end
     end
   end

qa/docker/spec/spec_helper.rb

@@ -49,6 +49,13 @@ def wait_for_pipeline(container, pipeline = 'main')
   end
 end
 
+def wait_for_log_message(container, search_string, stream = :stdout)
+  Stud.try(40.times, [NoMethodError, Docker::Error::ConflictError, RSpec::Expectations::ExpectationNotMetError, TypeError]) do
+    container_logs = container.logs(stream => true)
+    expect(container_logs.include?(search_string)).to be true
+  end
+end
+
 def cleanup_container(container)
   unless container.nil?
     begin