SHARP 2.0.1 - release notes

1. Introduction
2. List of changes
3. Timings
4. Installation
5. References

Introduction

The current production version of SHARP (2.0.1) is intended as a drop-in replacement of SHARP 1.4.0 (the previous production version). It can be run in the same way as SHARP 1.4.0 using the same set of scripts (SUSHI 3.0.x).

List of changes

1. General
   
   The algorithms implemented in this major revision of the code remain the same as in the relevant papers published previously [1], but some subtle changes have been introduced at various stages of the computations. However, the actual implementation is almost completely new. In the following sections the main changes are explained.
   
   
2. Double precision arithmetic
   
   All computations are now done in double precision. This transition was necessary to cope with very large range of values encountered in likelihood computations, a common phenomenon in statistics. Despite this, the new code is much faster and uses less memory.
   
   
3. Core libraries
   
   All low-level functions (mainly the likelihood functions and their derivatives) have been redesigned and rewritten from the ground-up. As mentioned above, they are the same as previously published. However, we anticipate changes there in the near future. A fairly sophisticated part of this section of the code is an algorithm to integrate various likelihood functions over the complex plane. Speed-up gains achieved initially has been deliberately sacrificed partly to achieve a higher numerical accuracy.
   
   
4. Middle layer of the code
   
   All computations are now orchestrated by a set of functions reflecting to a large extend the special nature of the optimization problem involved. This way of coding is related to problems like automatic differentiation [2] and use of sparsity of some matrices emerging at intermediate steps of the computations, but does not involve any approximations. Big memory savings and speed gains have been achieved here. The scalability wrt. model size has improved significantly. With the present version it is feasible to investigate structures containing hundreds of sites. The code is also very well suited for further extensions.
   
   
5. New optimizer
   
   The old minimizer (of the line-search type) has been replaced by a state-of-the-art bound-constrained trust region type Newton method algorithm [3],[4] which uses as a preconditioner a sparse incomplete Cholesky factorization with predictable memory requirements [5]. Although SHARP often solves relatively small problems (in terms of number of variables), this optimizer is well suited for very large ones.
   
   
6. Improvements in handling heavy atom model
   
   Model deficiencies caused by incorrect sites can now be detected at earlier stages making the optimization procedure more robust. A sparse approximation of the final Hessian can now be used to achieve further speedup for high resolution data and/or structures with many heavy atom sites.
   
   
7. This is a production release
   
   The code has been extensively tested on both simulated data and on all the real data sets at our disposal. It was in active beta testing for over 6 months. However, testing is never exhaustive and can only reveal bugs, but not the absence of problems. We would be grateful for any feedback from the users, especially those familiar with the previous versions of the program, about potential problems as well as overall code behaviour.
   
   

Timings

• SHARP 1.4.0                                  : 90.6

• SHARP 2.0.0                                  : 10.6

• SHARP 1.4.0                                  : 29.2

• SHARP 2.0.0                                  :  6.6

Installation

• If you have an existing SHARP installation (Sushi 3.0.x):
  
  
  1. download the following files into $BDG_home:
     
     

       installSHARP
       sushi.tar.gz
       sharp.*.tar.gz # for each system type you need
       

  2. if you want to keep your existing SHARP 1.4.0 binary:

       % cd $BDG_home
       % foreach exe (`ls bin/*_exe/sharp`)
         cp -pi $exe ${exe}-1.4.0
         end
       

  3. execute
     
     

       % cd $BDG_home
       % chmod +x installSHARP
       % adm/bin/kill_server
       % ./installSHARP -u
       % adm/bin/restart_server
       

  4. if you want to use various SHARP binaries (e.g. 1.4.0 and 2.0.1) within the same SHARP/autoSHARP installation, one way might be the following (but if in doubt please get in contact with us):
     
     
     1. make 2.0.1 the default version of SHARP, but save the 1.4.0 version:

          % cd $BDG_home
          % foreach exe (`ls bin/*_exe/sharp`)
            mv -i $exe ${exe}1.4.0
            ln -sf sharp-2.0.1 $exe
            end
          

     2. create a new user that will use the old 1.4.0 binary:

          % adm/bin/newuser johnOLD /where/ever user@domain.com
          % foreach exe (`ls bin/*_exe/sharp`)
            ln -sf sharp-1.4.0 ${exe}.johnOLD
            end
          

     3. if this new user (johnOLD) is going to share the sharpfiles directory structure (logfiles, datafiles, cardfiles) with another user (e.g. john): edit the file ~/sharpfiles/.sharpaccess.<server>.<port> and add all user to the 'require user' line.
        
        
• for a complete new installation follow the general installation instructions

References