Add Mutation Testing by ahawkins · Pull Request #22 · ahawkins/harness

ahawkins · 2014-04-15T19:48:31Z

Current work in progress with @mbj

ahawkins · 2014-04-15T19:50:52Z

@mbj So I'm almost done testing ::Harness only. Can't seem to get to the 100% mark though.

Here's the current report:

adam at mba in ~/p/harness(mutant*) bundle exec rake mutant
bundle exec mutant -I lib -r harness --use minitest '::Harness'
Mutant configuration:
Matcher:         #<Mutant::Matcher::Scope cache=#<Mutant::Cache> scope=Harness>
Strategy:        #<Mutant::Minitest::Strategy>
Expect Coverage: 100.000000%
Harness.collector:/Users/adam/projects/harness/lib/harness.rb:49
......
(06/06) 100% - 0.65s
Harness.config:/Users/adam/projects/harness/lib/harness.rb:10
...FFFF.
(04/08)  50% - 1.25s
Harness.count:/Users/adam/projects/harness/lib/harness.rb:22
...................................
(35/35) 100% - 5.06s
Harness.decrement:/Users/adam/projects/harness/lib/harness.rb:18
...............................
(31/31) 100% - 4.53s
Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37
........................F....................F.........F.
(54/57)  94% - 5.93s
Harness.gauge:/Users/adam/projects/harness/lib/harness.rb:41
...................................
(35/35) 100% - 5.10s
Harness.increment:/Users/adam/projects/harness/lib/harness.rb:14
...............................
(31/31) 100% - 4.50s
Harness.queue:/Users/adam/projects/harness/lib/harness.rb:45
......
(06/06) 100% - 0.63s
Harness.time:/Users/adam/projects/harness/lib/harness.rb:30
............................................
(44/44) 100% - 6.33s
Harness.timing:/Users/adam/projects/harness/lib/harness.rb:26
...................................
(35/35) 100% - 4.98s
Harness.config:/Users/adam/projects/harness/lib/harness.rb:10
evil:Harness.config:/Users/adam/projects/harness/lib/harness.rb:10:2b302
@@ -1,4 +1,4 @@
 def self.config
-  @config ||= Config.new
+  @s53eff8ffac026cdd4412 ||= Config.new
 end
evil:Harness.config:/Users/adam/projects/harness/lib/harness.rb:10:e076e
@@ -1,4 +1,4 @@
 def self.config
-  @config ||= Config.new
+  @config ||= Config
 end
evil:Harness.config:/Users/adam/projects/harness/lib/harness.rb:10:b6fc6
@@ -1,4 +1,4 @@
 def self.config
-  @config ||= Config.new
+  @config ||= nil.new
 end
evil:Harness.config:/Users/adam/projects/harness/lib/harness.rb:10:b452c
@@ -1,4 +1,4 @@
 def self.config
-  @config ||= Config.new
+  @config ||= nil
 end
(04/08)  50% - 1.25s
Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37
evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37:48223

evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37:48223

evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37:48223

(54/57)  94% - 5.93s
Subjects:  10
Mutations: 288
Kills:     281
Alive:     7
Runtime:   39.07s
Killtime:  38.95s
Overhead:  0.29%
Coverage:  97.57%
Expected:  100.00%
rake aborted!
Command failed with status (1): [bundle exec mutant -I lib -r harness --use...]
/Users/adam/projects/harness/Rakefile:11:in `block in <top (required)>'
Tasks: TOP => mutant
(See full trace by running task with --trace)

As you can see it's doing mutations to the config method. I added a test to assert that Harness.config always returns an instance of Harness::Config. Am I doing something wrong? I tested that it does in fact run all the tests and that is the case.

Secondly, is there any possible way to fix or ignore this mutation?

 def self.config
-  @config ||= Config.new
+  @s53eff8ffac026cdd4412 ||= Config.new
 end

mbj · 2014-04-15T20:03:05Z

@ahawkins I have a ticked for these. This mutation will be removed. Or at least be more intelligent on them. I personally prefer to refactor the need for ||= away, that also reduces some nasty races.

ahawkins · 2014-04-15T20:09:45Z

@mbj how do you refactor them?

solnic · 2014-04-15T20:20:23Z

blah @mbj you still didn't remove those? ;)

@ahawkins you can either refactor the code to not use ||= (which IMHO is a good thing to do) or just ignore those mutations

I explained (more or less) why I don't like ||= in such cases here but this is probably a broader topic related to things like using objects not classes, injecting dependencies, avoiding mutable state and maybe something else ;)

mbj · 2014-04-15T20:44:49Z

@ahawkins There is no general advice for all ||=.

Specific advice is possible:

def self.config
  @config ||= Config.new
end

You lazy initialize a mutable object under a global accessible location.

If you do this:

class Config
  INSTANCE = new
end

def self.config
  Config::INSTANCE
end

The mutant is dead.

Has the same public interface semantics. With less code, and race free.

I generally would not use global mutable state at all. But that is another topic.

Remove unused method

ahawkins · 2014-04-15T20:57:08Z

@mbj thanks.

Next question. Almost to 100% across the project. Cannot get AsyncQueue correct though. Is there some problem with attr_reader?

(11/11) 100% - 1.59s
Cannot find definition of: Harness::AsyncQueue#queue in /Users/adam/projects/harness/lib/harness/async_queue.rb:5
Harness::AsyncQueue#collector:/Users/adam/projects/harness/lib/harness/async_queue.rb:25
evil:Harness::AsyncQueue#collector:/Users/adam/projects/harness/lib/harness/async_queue.rb:25:9b3ef
@@ -1,4 +1,4 @@
 def collector
-  Harness.collector
+  Harness
 end
(05/06)  83% - 0.76s
Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:f05b8
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  @s0c72b44866a20ed13aff = Queue.new
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:5f13a
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  @queue = Queue
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:88a5f
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  @queue = nil
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:52267
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  nil
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:f18f1
@@ -1,12 +1,11 @@
 def initialize
-  @queue = Queue.new
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:daeda
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  @s613e120fb7c7fde109d5 = Queue.new
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
(55/61)  90% - 8.05s
Subjects:  3
Mutations: 78
Kills:     71
Alive:     7
Runtime:   10.46s
Killtime:  10.40s
Overhead:  0.55%
Coverage:  91.03%
Expected:  100.00%
rake aborted!
Command failed with status (1): [bundle exec mutant -I lib -r harness --use...]
/Users/adam/projects/harness/Rakefile:11:in `block in <top (required)>'
Tasks: TOP => mutant
(See full trace by running task with --trace)

I see mutant is saying cannot find definition for queue method. Should I define methods myself instead of using attr_reader?

mbj · 2014-04-15T21:02:10Z

@ahawkins The warnings about "Cannot find definition of Foo#bar" will are not really warnings. I added these at the time the matcher was not as good as today. I'll only emit these warnings in future when you explicitly asked for Foo#bar. Mutant does not know where to find ASTs for methods generated via attr_reader or definie_method in its current execution model.

You can savely ignore these warnings, they'll go away soon.

ahawkins · 2014-04-16T21:50:46Z

@mbj c5cc8a3 was definitely the hardest one yet. This is an interesting experience to say the least :)

mbj · 2014-04-16T22:01:23Z

@ahawkins Writing code in a way thats easily mutation testable is hard. But it pays back. Does not matter what you see on twitter. People think I prefer local maxima.

mbj · 2014-04-16T22:03:34Z

@ahawkins I'm busy. Cannot expand the following: Options for testing Threaded code:

Inject a predicable Thread mock. (Allows most scenarios to be steered explicitly!)
Ignore the Thread subject in mutation testing. But move as much code that gets executed in the thread to a mutation testable unit.
(IDEA) never tried. Create a thread mock thats sigle steppable via debug hooks.

solnic · 2014-04-16T22:08:59Z

@mbj what do you mean by local maxima in this context?

mbj · 2014-04-16T22:10:43Z

@solnic Anima::Builder was a local maxima. Where I did not spotted a better local maxima the new code. Dunno if I can ever prove we reached a global maxima of beautifulness. See linear optimization speak.

ahawkins · 2014-04-18T10:31:26Z

@mbj Only 3 mutations remaining the entire code base. I'm not sure what to do about them. They are "evil" mutants. The mutant output does not include any code changes though:

adam at mba in ~/p/harness(mutant*) bundle exec rake mutant
bundle exec mutant -d -I lib -r harness --use minitest '::Harness'
Mutant configuration:
Matcher:         #<Mutant::Matcher::Scope cache=#<Mutant::Cache> scope=Harness>
Strategy:        #<Mutant::Minitest::Strategy>
Expect Coverage: 100.000000%
Harness.collector:/Users/adam/projects/harness/lib/harness.rb:51
......
(06/06) 100% - 1.01s
Harness.config:/Users/adam/projects/harness/lib/harness.rb:12
....
(04/04) 100% - 0.77s
Harness.count:/Users/adam/projects/harness/lib/harness.rb:24
...................................
(35/35) 100% - 10.11s
Harness.decrement:/Users/adam/projects/harness/lib/harness.rb:20
...............................
(31/31) 100% - 9.26s
Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39
........................F....................F.........F.
(54/57)  94% - 7.81s
Harness.gauge:/Users/adam/projects/harness/lib/harness.rb:43
...................................
(35/35) 100% - 10.04s
Harness.increment:/Users/adam/projects/harness/lib/harness.rb:16
...............................
(31/31) 100% - 9.28s
Harness.queue:/Users/adam/projects/harness/lib/harness.rb:47
......
(06/06) 100% - 0.97s
Harness.time:/Users/adam/projects/harness/lib/harness.rb:32
............................................
(44/44) 100% - 12.79s
Harness.timing:/Users/adam/projects/harness/lib/harness.rb:28
...................................
(35/35) 100% - 9.90s
Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39
evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39:48d86

evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39:48d86

evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39:48d86

(54/57)  94% - 7.81s
Subjects:  10
Mutations: 284
Kills:     281
Alive:     3
Runtime:   72.04s
Killtime:  71.93s
Overhead:  0.15%
Coverage:  98.94%
Expected:  100.00%
rake aborted!
Command failed with status (1): [bundle exec mutant -d -I lib -r harness --...]
/Users/adam/projects/harness/Rakefile:11:in `block in <top (required)>'
Tasks: TOP => mutant
(See full trace by running task with --trace)

mbj · 2018-11-19T16:54:35Z

@ahawkins Catching up on a 4 y old thread. mutant-minitest was just merged to master. Do you want to retry? I'd be happy to advice.

mbj and others added 4 commits April 13, 2014 21:11

Add mutant dependency

4f40244

Mutant test integration to minitest

13e8d01

Use explicit signalling to detect active mutant

e33dde8

97.5 mutation test coverage on Harness only

f16544f

ahawkins added 2 commits April 15, 2014 22:46

NullCollector 100% mutation coverage

b3f00bd

FakeCollector 100% mutation covered

bcb9a0b

Remove unused method

100% mutation cover AsyncQueue

c5cc8a3

Fix config related mutations

8c42625

Conversation

ahawkins commented Apr 15, 2014

Uh oh!

ahawkins commented Apr 15, 2014

Uh oh!

mbj commented Apr 15, 2014

Uh oh!

ahawkins commented Apr 15, 2014

Uh oh!

solnic commented Apr 15, 2014

Uh oh!

mbj commented Apr 15, 2014

Uh oh!

ahawkins commented Apr 15, 2014

Uh oh!

mbj commented Apr 15, 2014

Uh oh!

ahawkins commented Apr 16, 2014

Uh oh!

mbj commented Apr 16, 2014

Uh oh!

mbj commented Apr 16, 2014

Uh oh!

solnic commented Apr 16, 2014

Uh oh!

mbj commented Apr 16, 2014

Uh oh!

ahawkins commented Apr 18, 2014

Uh oh!

mbj commented Nov 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mbj commented Nov 19, 2018 •

edited

Loading