mogul wrapper using local disk & -csdpath -datapath to work around NFS (particularly v3) issue
- There are issues using the latest 2014 mogul from a CSDS installation accessed from a NFS-mounted file system. For details see SoftwareMogulRelease2014NFSissues and GradeErrorMessageMogulKilledCPULimit for details
- One way to avoid these issues is to make a full installation of CSDS on a local disk of the machine you want to run grade or buster-report on. But this is laborious requiring access to the CSDS installation media and licence.
- Fortunately there is an easier way that has been suggested by Simon Bowden at CCDC. This is to make local copies of the directories csd/ and data/ from the NFS-mounted CSDS installation. A wrapper script is then used to invoke mogul using its -csdpath and -datapath options. This avoids disk-intensive access over NFS and provides performance that matches that of having a full local installation.
- The prerequisite for setting up the wrapper is to be able to create a directory on a local disk to contain 6Gb of data. The data directories and wrapper can be set up by a general user (it does not use the CSDS media).
- for example here I will use the directory /local/osmart because the /local directory is a local disk with plenty of room and I have write permission for /local/osmart you will have to use different locations that are valid on your workstation
- The first task is to copy the data directories from the NFS-mounted CSDS installation to the local directory.
- Here the NFS-mounted CSDS installation is in /nfs/CSDS2014/ and so the data directories are /nfs/CSDS2014/csd and /nfs/CSDS2014/data.
- The copy can performed using rsync commands (the commands run are show in bold).
time rsync -av /nfs/CSDS2014/csd /local/osmart/csdslocal
sending incremental file list
created directory /local/osmart/csdslocal
csd/
csd/Nov13.ind
long file list edited out
time rsync -av /nfs/CSDS2014/data /local/osmart/csdslocal
sending incremental file list
data/
data/mogul535.path
data/Nov13/
long file list edited out
- The commands take about 30 seconds each on our system (but may well be slower for you). After this the local directory containing the csds data directories should occupy 5.8Gb disk space:
du -hsx /local/osmart/csdslocal/*
2.6G /local/osmart/csdslocal/csd
3.2G /local/osmart/csdslocal/data
tar xvf mogul_wrapper_datapath_csdpath.tar.gz
mogul_wrapper_datapath_csdpath.bash
- the script will now need to edited, altering lines 17,21 and 24 to give the locations of the mogul executable and the two data directories on your site.
- Once you have done the edit test that mogul_wrapper_datapath_csdpath.bash works:
- If you are on workstation that can support mogul running in graphical mode check that the wrapper launches interactive mogul.
- for an extensive test see GradeErrorMessageMogulKilledCPULimit
- To use mogul_wrapper_datapath_csdpath.bash with grade and buster-report simply set the environment variable BDG_TOOL_MOGUL to the full location of the wrapper on your system. For example
setenv BDG_TOOL_MOGUL /home/software/GPhL/extras/mogul_wrapper_datapath_csdpath.bash
- this can be done either as part of the normal Global Phasing Software configuration (in $BDG_home/setup_local.csh / $BDG_home/setup_local.sh) or afterwards as a user.
- In our tests using mogul_wrapper_datapath_csdpath.bash produces the same performance as a complete local installation.
- It would be possible to extend this solution so that local copies of the data and csd directories where made for each node of a cluster. Indeed if wished it would be possible to automated the copy/rsync process to occur on boot. Furthermore, if you have enough memory and want to avoid copies to a physical disk it would be possible to use a tmpfs filesystem.
Page by Oliver Smart. Original version 19th April 2014, last modified 1st May 2014. Address problems, corrections and clarifications to buster-develop@globalphasing.com