Test suite consumes an extraordinary amount of memory #104
Description
Hi,
there is a bug report for the Debian packaged version which says that the ruffus test suite consumes an extraordinary amount of memory on the build machine. The bug was filed in 2016 for version 2.6.3 of ruffus where a memory consumption of 18GB was measured. The bug reporter did a new measure for the recent version 2.8.1 he gets 38GB so it became worse. I tried to reproduce the issue and can confirm that if I do
while true; do grep Committed_AS /proc/meminfo; sleep 1; done
in a separate xterm while running the build time tests I get something like
Committed_AS: 9807008 kB
Committed_AS: 22324324 kB
Committed_AS: 33082052 kB
Committed_AS: 33082052 kB
Committed_AS: 17073508 kB
Committed_AS: 41512736 kB
Committed_AS: 45553976 kB
Committed_AS: 26653296 kB
Committed_AS: 5887392 kB
while the test suite outputs something like
test_pool_manager.py::TestExecutionEngines::test_pipeline_fails_with_unknown_manager PASSED [ 25%]
test_pool_manager.py::TestExecutionEngines::test_pipeline_runs_with_gevent_manager <- ../../../../../../../usr/lib/python2.7/unittest/case.py SKIPPED [ 50%]
test_pool_manager.py::TestExecutionEngines::test_pipeline_runs_with_multiprocessing PASSED [ 75%]
test_pool_manager.py::TestExecutionEngines::test_pipeline_runs_with_multithreading PASSED [100%]
===================== 3 passed, 1 skipped in 2.50 seconds ======================
pytest -v test_cmdline.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.15+, pytest-3.6.4, py-1.7.0, pluggy-0.8.0 -- /usr/bin/python2
cachedir: ../../../../../.pytest_cache
rootdir: /home/andreas/debian-maintain/salsa/med-team/build-area/python-ruffus-2.8.1, inifile:
collecting ... collected 4 items
test_cmdline.py::Test_cmdline::test_argparse PASSED [ 25%]
test_cmdline.py::Test_cmdline::test_optparse PASSED [ 50%]
test_cmdline.py::Test_cmdline::test_verbose PASSED [ 75%]
test_cmdline.py::Test_cmdline::test_verbose_abbreviated_path PASSED [100%]
Please note: I'm definitely not sure that this is the affected test neither that these are the only tests which push the memory limit way more far than other programs but I confirm that the reporter is right and that the issue can be found somewhere close here. I'd suggest some bisecting to find the actual test to find out the reason.
Kind regards, Andreas.