Dealing with grade "ERROR mogul was killed because it hit the CPU limit of 60 seconds" message
- Because of a Freeze problem with mogul used from NFSv3-mounted CSDS2014, Grade releases after 20140109 introduced a 60 seconds elapsed time limit on the "test mogul by asking for bond angle of CO2" that is invoked both by grade -checkdeps and at the start of every grade run.
- The problem has occurred if you see a grade ERROR message of the form:
grade -checkdeps edited out loads of lines Setting tool 'mogul' to '/public/xtal/ccdc2013/bin/mogul' from environment variable $BDG_TOOL_MOGUL test mogul by asking for bond angle of CO2 .. ERROR mogul was killed because it hit the CPU limit of 60 seconds. ERROR this should not happen for this simple test! ERROR We have found this problem if CSDS 2014 is used from a filesystem mounted ERROR using NFSv3 because of locking problems (lockd errors reported by dmesg). ERROR Please contact buster-develop@globalphasing.com if you need advice.
- Currently it is proving difficult to track down and solve the issues behind this bug because we cannot reproduce the conditions under which it occurs.
- So we are asking for your help in understanding what happens. Please could you perform the tests on this page and provide answers to the questions in mogul_co2test_questions.txt.
- Please download the file mogul_co2test.tar.gz and unpack it using the command
tar xzf mogul_co2test.tar.gz
- This will create a directory mogul_co2test so first cd into this
cd mogul_co2test
Does mogul work properly if run interactively? (Q1 to Q3)
- start mogul and load co2.pdb.
- Provided you have mogul on your path or set as an alias this is normally done by
mogul co2.pdb
- this should fire up a window:
- N.B., if you do not see a window like this then are big problems with your CSDS installation and you need to ask for help from the CCDC support@ccdc.cam.ac.uk . (Once you have got mogul working interactively then try again with grade)
- Please answer Q1 on the questionnaire.
- Use your mouse to left click on the OK button as shown.
- In mogul click on the three atoms (left, center and right) then on the Search button as shown.
- The window should rapidly (<1 sec) change to provide results of the mogul search:
- N.B., if mogul stalls at this stage then either (a) your CSDS installation has problems or (b) you have hit the NFSv3 issue.
- N.B., if the window takes a long time to appear (>10 seconds) it is very likely you have hit the NFSv3 issue.
- Please answer Q2 and Q3 on the questionnaire.
- Exit mogul.
How often to you see get the "ERROR mogul was killed ..." message?
- Please answer Q4 and Q5 on the questionnaire.
- Please setup Global Phasing Software and run
grade -checkdeps
- Please answer Q6 on the questionnaire by pasting the output produced.
Script to help diagnose any mogul stall.
- Please run the script run_test_once.bash that is supplied in the tar ball. This should be run by specifying the full path of your mogul executable. For instance at Global Phasing mogul is in /public/xtal/CCDC/CSD_System/bin/mogul so we run:
./run_test_once.bash /public/xtal/CCDC/Linux/CSD_System_2014/bin/mogul
- If there are periods where grade persistently reports "ERROR mogul was killed ..." please run the script then. If not then run it immediately.
- Please answer Q7 on the questionnaire by pasting the output produced.
- The script should run in less than a couple of seconds
- but if it takes longer leave it for at least an hour before interrupting it.
- If the ./run_test_once.bash script takes a long time then run
dmesg | tail -30
- Then please answer Q8 on the questionnaire by pasting the output produced.
- Finally please email the filled in questionnaire to buster-develop@globalphasing.com
Page by Oliver Smart. Original version 9th April 2014, last modified 10th April 2014. Address problems, corrections and clarifications to buster-develop@globalphasing.com