Running 'zef test -v .' in cro-zeromq gives different failures every run

[sjn]

When running "zef test -v. " in the cro-zeromq rep, the tests fail, but subtly different every time. Also, some imes one of the tests hang.

Here are some of the different outputs I get:

=================================8<---------------------------
$ zef test -v .
===> Testing: Cro::ZeroMQ:ver('0.8')
t/zeromq-client.t ......... ok
t/zeromq-collector.t ...... ok
t/zeromq-distributor.t .... All 1 subtests passed
t/zeromq-message.t ........ ok
t/zeromq-pub-sub.t ........ No subtests run
t/zeromq-push-pull.t ...... ok
t/zeromq-req-rep.t ........ ok
t/zeromq-router-dealer.t .. ok
t/zeromq-service.t ........ ok
t/zeromq-xpub-xsub.t ...... ok
All tests successful.

Test Summary Report

t/zeromq-distributor.t (Wstat: 11 Tests: 1 Failed: 0)
Non-zero wait status: 11
Parse errors: No plan found in TAP output
t/zeromq-pub-sub.t (Wstat: 11 Tests: 0 Failed: 0)
Non-zero wait status: 11
Parse errors: No plan found in TAP output
Files=10, Tests=20, 4 wallclock secs
Result: FAILED
===> Testing [FAIL]: Cro::ZeroMQ:ver('0.8')
Aborting due to test failure: Cro::ZeroMQ:ver('0.8') (use --force-test to override)
in code at /home/sjn/.rakudobrew/moar-nom/install/share/perl6/site/sources/A5C13EF833A22B1E85E78D1CD88BE1B0B0B24DA1 (Zef::Client) line 375
in method test at /home/sjn/.rakudobrew/moar-nom/install/share/perl6/site/sources/A5C13EF833A22B1E85E78D1CD88BE1B0B0B24DA1 (Zef::Client) line 353
in sub MAIN at /home/sjn/.rakudobrew/moar-nom/install/share/perl6/site/sources/31FF9E8498781BF95868DF98AD861FCF1211BCC1 (Zef::CLI) line 38
in block at /home/sjn/.rakudobrew/moar-nom/install/share/perl6/site/resources/0031ECC6124802A80F7B7B073C50D0A1694E3020 line 1
in sub MAIN at /home/sjn/.rakudobrew/bin/../moar-nom/install/share/perl6/site/bin/zef line 2
in block at /home/sjn/.rakudobrew/bin/../moar-nom/install/share/perl6/site/bin/zef line 2

====================================8<-------------------

...and here's another run:

========8<---------------------------------------------------------------
$ zef test -v .
===> Testing: Cro::ZeroMQ:ver('0.8')
t/zeromq-client.t ......... ok
t/zeromq-collector.t ...... ok
t/zeromq-distributor.t .... All 1 subtests passed
t/zeromq-message.t ........ ok
[HANGS]
-------------------------------------------------->8=====================

...and a third run:

=====================8<-------------------------------------
$ zef test -v .
===> Testing: Cro::ZeroMQ:ver('0.8')
t/zeromq-client.t ......... All 3 subtests passed
t/zeromq-collector.t ...... ok
t/zeromq-distributor.t .... All 1 subtests passed
t/zeromq-message.t ........ ok
t/zeromq-pub-sub.t ........ All 3 subtests passed
t/zeromq-push-pull.t ...... ok
t/zeromq-req-rep.t ........ ok
t/zeromq-router-dealer.t .. ok
t/zeromq-service.t ........ ok
t/zeromq-xpub-xsub.t ...... ok
All tests successful.

Test Summary Report

t/zeromq-client.t (Wstat: 11 Tests: 3 Failed: 0)
Non-zero wait status: 11
t/zeromq-distributor.t (Wstat: 11 Tests: 1 Failed: 0)
Non-zero wait status: 11
Parse errors: No plan found in TAP output
t/zeromq-pub-sub.t (Wstat: 11 Tests: 3 Failed: 0)
Non-zero wait status: 11
Parse errors: No plan found in TAP output
Files=10, Tests=23, 5 wallclock secs
Result: FAILED
===> Testing [FAIL]: Cro::ZeroMQ:ver('0.8')
Aborting due to test failure: Cro::ZeroMQ:ver('0.8') (use --force-test to override)
in code at /home/sjn/.rakudobrew/moar-nom/install/share/perl6/site/sources/A5C13EF833A22B1E85E78D1CD88BE1B0B0B24DA1 (Zef::Client) line 375
in method test at /home/sjn/.rakudobrew/moar-nom/install/share/perl6/site/sources/A5C13EF833A22B1E85E78D1CD88BE1B0B0B24DA1 (Zef::Client) line 353
in sub MAIN at /home/sjn/.rakudobrew/moar-nom/install/share/perl6/site/sources/31FF9E8498781BF95868DF98AD861FCF1211BCC1 (Zef::CLI) line 38
in block at /home/sjn/.rakudobrew/moar-nom/install/share/perl6/site/resources/0031ECC6124802A80F7B7B073C50D0A1694E3020 line 1
in sub MAIN at /home/sjn/.rakudobrew/bin/../moar-nom/install/share/perl6/site/bin/zef line 2
in block at /home/sjn/.rakudobrew/bin/../moar-nom/install/share/perl6/site/bin/zef line 2

==================8<--------------------------------------

This happens when I'm using:

$ git show | grep ^commit
commit 0db37a0

$ perl6 -v
This is Rakudo version 2017.08-74-g13b6a33c2 built on MoarVM version 2017.08.1-52-gf59ac65d9

[Altai-man]

Let me guess, your ZeroMQ version is 4.2.x? Unfortunately, it is a known bug. Net::ZMQ library that cro uses to operate on zmq sockets currently supports only 4.1.x, and there are segfaults on latest ZMQ version.

[sjn]

When I run Net::ZMQ's tests, they all pass. If I then force-install Net::ZMQ, and then go to a dir containing this repo, and run the tests as above, they fail.

Is this a sign of Net:ZMQ's tests being thorough enough, or a sign of Cro::ZeroMQ's tests not using Net::ZMQ correctly? (or something else?)

[Altai-man]

Net:ZMQ's tests not being thorough enough

To be honest, I'd bet on this.

The situation with this is quite complicated, to be honest.

The thing is, as far as I know, @jnthn couldn't reproduce any segfaults on 4.1 with this module's tests. And there are many of it on 4.2. I may be wrong, of course, but I'd say that Net::ZMQ is at fault.

It is a really black magic to debug and I am not capable to do it right now.

[Altai-man]

We can consider trying out https://github.com/gabrielash/perl6-zmq (yet I don't sure how ready it is). I wonder is it thread-safe.

[sjn]

Heya. I bumped into @arnsholt recently, and he's in the middle of his first "OMG I'm a dad!" phase of his life (and quite happy, it seems :-) ). I'd guess he'll have no tuits for fixing bugs in Net::ZMQ in the foreseeable future....

[Altai-man]

@sjn well, there are people who can merge PRs into it's repository, so the problem is more of someone skillful enough(to crack memory-related problems) and willing to help. I am not that person, unfortunately.