crash when runing coala on the libreoffice core repo #3701

sils · 2017-02-05T20:47:28Z

coafile:

[Default]
bears = CPDBear, CPPCheckBear
files = **.cxx, **.hxx
language = c++

log:

[DEBUG][23:43:25] Running bear CPDBear...
[WARNING][23:43:25] Bear CPDBear failed to run. Take a look at debug messages (`-V`) for further information.
[DEBUG][23:43:26] The bear CPDBear raised an exception. If you are the author of this bear, please make sure to catch all exceptions. If not and this error annoys you, you might want to get in contact with the author of this bear.

Traceback information is provided below:

Traceback (most recent call last):
  File "/home/lasse/venvs/def/lib/python3.5/site-packages/coalib/bears/Bear.py", line 240, in execute
    return [] if result is None else list(result)
  File "/home/lasse/venvs/def/lib/python3.5/site-packages/bears/general/CPDBear.py", line 103, in run
    stdout_output, _ = run_shell_command(arguments)
  File "/home/lasse/venvs/def/lib/python3.5/site-packages/coalib/misc/Shell.py", line 103, in run_shell_command
    with run_interactive_shell_command(command, **kwargs) as p:
  File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/lasse/venvs/def/lib/python3.5/site-packages/coalib/misc/Shell.py", line 70, in run_interactive_shell_command
    process = Popen(command, **args)
  File "/usr/lib64/python3.5/subprocess.py", line 950, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.5/subprocess.py", line 1544, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 7] Argument list too long

The text was updated successfully, but these errors were encountered:

sils · 2017-02-05T20:50:48Z

my(max) idea is that we might try to run cpd with 10k files and it doesn't like that

Makman2 · 2017-02-05T21:48:38Z

looks even like it's a limit inside python (or even OS) :O
That's bad actually...

If possible, maybe CPD supports configuration files where you can also put the files to analyze?

sims1253 · 2017-02-05T22:09:02Z

there is --files for files and directories. Maybe we can bundle the files somehow or add directories

sims1253 · 2017-02-05T22:10:01Z

actually there is "There should be no limit to the number of ‘–files’, you may add… But if you stumble one, please tell us !" in their docs

impmihai · 2017-02-05T22:25:50Z

[Errno 7] Argument list too long
This is an OS related issue as i can read. Maybe splitting into multiple runs of coala over smaller ammounts of files would help?

Makman2 · 2017-02-06T15:17:19Z

Problem is you may loose context when splitting up files^^

Makman2 · 2017-02-06T15:18:57Z

But it's definitely a limit in python / OS:

>>> call(['echo'] + list(str(x) for x in range(1000000)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.5/subprocess.py", line 557, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib64/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 7] Argument list too long

Makman2 · 2017-02-06T15:22:18Z

So we need a configuration file, where we can put an arbitrary amount of information.

Makman2 · 2017-02-06T15:23:18Z

Interesting, just tested it: It's a limit in Python, not the OS!
Bash supports as many arguments as needed, I generated a bash file with echo 1 2 3 ... until 1000000 (one million), and it echoes properly^^

sims1253 · 2017-02-06T16:46:16Z

so then subprocess has a limeted amount of parameters you can give it? strange
If we could somehow give it directories instead of files that would maybe obscure the problem by shrinking the list length

Makman2 · 2017-02-06T16:50:01Z

but still we want to support passing in specific files, and if the user wants 10000+ specific files, then we have this problem again^^

impmihai · 2017-02-06T17:44:46Z

maybe writing down the parameters to a temporary file and read them from there?

Makman2 · 2017-02-06T18:30:07Z

cpd has to support it, and I can't find a suitable command line argument yet...

Until nowm the files to run on were given to CPD with the --files option, which would raise 'OSError: [Errno 7] Argument list too long'. Now, the files are saved in a temporary file, and that file is given to CPD with the --filelist option, which takes a csv file containing a list of files to run on. Fixes coala/coala#3701

Until now, the files to run on were given to CPD with the --files option, which would raise 'OSError: [Errno 7] Argument list too long'. Now, the files are saved in a temporary file, and that file is given to CPD with the --filelist option, which takes a csv file containing a list of files to run on. Fixes coala/coala#3701

impmihai · 2017-02-14T00:33:02Z

Ok, looks like the old version of CPD that circleCI has does not have the --filelist option that takes a file containing the filelist to run on. We need to update PMD on CI, am i right?

Until now, the files to run on were given to CPD with the --files option, which would raise 'OSError: [Errno 7] Argument list too long'. Now, the files are saved in a temporary file, and that file is given to CPD with the --filelist option, which takes a csv file containing a list of files to run on. Fixes coala/coala#3701

impmihai · 2017-03-11T00:40:58Z

Ok, it is worse than i thought. Pmd has been updated with the -filelist option, but cpd does not have that. All i can do is to file an issue to pmd to add that feature to cpd too. Shall I do this and wait for an update?

jayvdb · 2017-03-11T01:50:49Z

Yes please raise the bug upstream

…

On Sat, 11 Mar 2017 07:40 Mihai Ionut ***@***.***> wrote: Ok, it is worse than i thought. Pmd has been updated with the -filelist option, but cpd does not have that. All i can do is to file an issue to pmd to add that feature to cpd too. Shall I do this and wait for an update? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3701 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA69LI4vrIIq5b5o_-KybpGAqWqaSI2ks5rke2agaJpZM4L3nus> .

impmihai · 2017-03-15T20:00:22Z

I made a PR to PMD in order to add that option. Here it is: pmd/pmd#297 . When it is merged, i will finish the PR on this issue. 😄

Until now, the files to run on were given to CPD with the --files option, which would raise 'OSError: [Errno 7] Argument list too long'. Now, the files are saved in a temporary file, and that file is given to CPD with the --filelist option, which takes a csv file containing a list of files to run on. Fixes coala/coala#3701

impmihai · 2017-03-19T19:13:23Z

PR merged, waiting for their version release and I am ready to finish this too 😄

jayvdb · 2017-04-09T04:46:30Z

pmd/pmd@029ed7c7a isnt in the latest release v5.5.5, so it looks like this will be in v5.6.

https://pmd.github.io/ says that 5.5.5 will be the last release in the 5.5 series, so 5.6 is probably on the way soon, and it has only two blockers not yet done.

Until now, the files to run on were given to CPD with the --files option, which would raise 'OSError: [Errno 7] Argument list too long'. Now, the files are saved in a temporary file, and that file is given to CPD with the --filelist option, which takes a csv file containing a list of files to run on. Fixes coala/coala#3701

jayvdb · 2017-05-08T03:04:51Z

5.6 has been released, and includes this feature. https://pmd.github.io/pmd-5.6.0/overview/changelog.html

Until now, the files to run on were given to CPD with the --files option, which would raise 'OSError: [Errno 7] Argument list too long'. Now, the files are saved in a temporary file, and that file is given to CPD with the --filelist option, which takes a csv file containing a list of files to run on. Fixes coala/coala#3701

sils added the type/bug label Feb 5, 2017

gitmate-bot added the area/bears label Feb 5, 2017

coala-bot assigned impmihai Feb 13, 2017

impmihai linked a pull request Feb 13, 2017 that will close this issue

CPDBear: Fix for crash when many files are given coala/coala-bears#1429

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crash when runing coala on the libreoffice core repo #3701

crash when runing coala on the libreoffice core repo #3701

sils commented Feb 5, 2017

sils commented Feb 5, 2017

Makman2 commented Feb 5, 2017

sims1253 commented Feb 5, 2017

sims1253 commented Feb 5, 2017

impmihai commented Feb 5, 2017

Makman2 commented Feb 6, 2017

Makman2 commented Feb 6, 2017

Makman2 commented Feb 6, 2017

Makman2 commented Feb 6, 2017

sims1253 commented Feb 6, 2017

Makman2 commented Feb 6, 2017

impmihai commented Feb 6, 2017

Makman2 commented Feb 6, 2017

impmihai commented Feb 14, 2017

impmihai commented Mar 11, 2017

jayvdb commented Mar 11, 2017 via email

impmihai commented Mar 15, 2017 •

edited

Loading

impmihai commented Mar 19, 2017

jayvdb commented Apr 9, 2017

jayvdb commented May 8, 2017

crash when runing coala on the libreoffice core repo #3701

crash when runing coala on the libreoffice core repo #3701

Comments

sils commented Feb 5, 2017

sils commented Feb 5, 2017

Makman2 commented Feb 5, 2017

sims1253 commented Feb 5, 2017

sims1253 commented Feb 5, 2017

impmihai commented Feb 5, 2017

Makman2 commented Feb 6, 2017

Makman2 commented Feb 6, 2017

Makman2 commented Feb 6, 2017

Makman2 commented Feb 6, 2017

sims1253 commented Feb 6, 2017

Makman2 commented Feb 6, 2017

impmihai commented Feb 6, 2017

Makman2 commented Feb 6, 2017

impmihai commented Feb 14, 2017

impmihai commented Mar 11, 2017

jayvdb commented Mar 11, 2017 via email

impmihai commented Mar 15, 2017 • edited Loading

impmihai commented Mar 19, 2017

jayvdb commented Apr 9, 2017

jayvdb commented May 8, 2017

impmihai commented Mar 15, 2017 •

edited

Loading