################################################################### # # This is the May 08 2017 snapshot release of the autoPROC toolbox for # academic users. # ################################################################### # # Copyright (C) 2005-2017 by Global Phasing Limited # # All rights reserved. # # This software is proprietary to and embodies the # confidential technology of Global Phasing Limited # (GPhL). Possession, use, duplication or dissemination of # the software is authorised only pursuant to a valid # written licence from GPhL. # # Authors: G. Bricogne, C. Flensburg, P. Keller, W. Paciorek, # A. Sharff, O. Smart, C. Vonrhein, T. Womack # # Reference: C. Vonrhein, C. Flensburg, P. Keller, A. Sharff, # O. Smart, W. Paciorek, T. Womack & G. Bricogne # (2011). Data processing and analysis with the autoPROC # toolbox. Acta Crystallogr D 67, 293-302. # ################################################################### =================================================================== Changes since the last autoPROC stable release May 08 2017 =================================================================== * New features, fixes and improvements (released May 08 2017): * New features, fixes and improvements (released May 05 2017): * Create a file with REMARK 200 content related to the STARANISO MTZ file produced by the AIMLESS scaling path, to have an exact parallel to REMARK 200 file associated to the TRUNCATE MTZ file. * Give "table-1" like summary for each output MTZ file at end of autoPROC run (bothin in stdout and summary.html as well as a file with ending "*.table1"): this should make it clearer what statistics are relevant for which MTZ file. * Avoid writing empty shell information in MRFANA analysis when those shells are bordering high-resolution limit. * Added support for ALS 4.2.2 detector RDI1 (CMOS_8M). * Support in 'imginfo' and 'GPX2' for intermediate CBF files from Eiger data (i.e. only those CBF files are available, although they were generated by a tool intended for on-the-fly conversion as apart of XDS processing). * Take care of correct permissions during installation (especially for Ruby installation). * Some clarifications in output from aP_check (oe 'process -checkdeps'). * Added tool 'xparm_transform' totransform XPARM.XDS files so that either the rotation axis is along a lab frame axis and/or the detector X-axis is. * Added tools 'ana_cmpmat', 'aP_rotcal' and 'aP_wf_rotcal' to distribution: mainly useful for rotational calibration work. * For cases where image headers are not recording the exact settings of goniostat angles (Omega, Kappa/Chi, Phi), allow override via goniostat_{Omega,Kappa,Chi,Phi}_{axis,angle}_ parameters. * Simplified option to use all recorded images for background estimation (e.g. when shadowing needs to be taken into account as well): setting XdsNumImagesBackgroundRange to "All" will do that. * Update to support latest Diamond/I23 instrument (Kappa goniostat). * Improved generation of STARANISO plots (reciprocal planes) with better legend and represenation of reciprocal cell. * Support for writing STARANISO-related information (analysis of anisotropy) as well into XML file(s). * Update to support latest CCP4 (version/update 7.0.036, AIMLESS 0.5.32) and XDS (BUILT=20170215). * Added support for spherical/ellipsoidal statistics via MRFANA: the description of ellipsoid needs to be given via the '-ell' command-line argument. * Added option -L to 'find_images' tool to get arguments in form suitable for -Id flag of 'process' directly (reported by D. Reinert). * Added warning messages in case prediction pictures (via GPX2) could not be generated: this usually points to issues with headless cluster nodes. See also https://www.globalphasing.com/autoproc/wiki/index.cgi?FaqUsingGpx2OnHeadLessBox. * MRFANA now also computes the ellipsoidal completeness in resolution shells. This is the completeness within the ellipsoid fitted by STARANISO (to describe the anisotropy of diffraction within the data) for a given resolution range. * Some fixes to the computation of statistics within MRFANA that would have given small differences when comparing to other programs - mainly due to the inclusion of systematically absent reflections in those statistics and some inaccuracies when handling reflections at the border of resolution shells. * Fix when handling sparse (or very finely sliced) multi-sweep datasets: the unique renumbering of all datasets got confused when some images had no reflection centroids assigned to them. * New features, fixes and improvements (released Dec 07 2016): * GPX2: automatically detect and load image files compressed with either gzip for bzip2. Also, for miniCBF files protect against missing value of Polarization. * Fix to the plot of misfits as function of detector position: now also has correct aspect ratio of detector dimensions. * Fixed documentation in $autoPROC_home/docs/autoproc/manual/appendix1.html of AutoProcScaleStatsUseMrfanaIgnoreIceRingShells (reported by J. Benach). * Fixed problem with processing compressed (.gz/.bz2) images when using a subset of images for background estimation. * Updated to support XDS Nov 1, 2016 (BUILT=20161101), pointless 1.10.27, aimless 0.5.29 and CCP4 7.0.022. * Implemented workaround for problems in integrating weak data with XDS Nov 1, 2016 (BUILT=20161101): at least for Pilatus detectors setting GAIN=1 in XDS.INP seems to fix this. The problem manifests itself as a bias towards a non-zero intensity (we have seen values of +0.5 and -0.5) at high resolution when there really is no diffraction present. This has been reported to the XDS developers and the above workaround was suggested by K. Diederichs. * Added additional high resolution ice-ring values based on analysis of high-resolution datasets and comparing to [1]: 1.524 - 1.519 1.473 - 1.470 1.444 - 1.440 1.372 - 1.368 1.367 - 1.363 1.299 - 1.296 1.275 - 1.274 1.261 - 1.259 1.224 - 1.222 1.171 - 1.168 1.124 - 1.122 [1] Kumai, M. (1967). A study of hexagonal and cubic ice at low temperatures. Research Report 231, U.S. Army Materiel Command, Cold Regions Research & Engineering Laboratory, Hanover, NH. * Initial support for CBF files with imgCIF image headers (tested on Bruker Photon-II data; thanks to G. Bader, M. Benning and D. Reinert). * New features, fixes and improvements (released Jul 08 2016): * Fixed problem with PDF-report generation when running single-sweep dataset without using the -d flag: it was stopping with an error message (and exit status) at the last step of an autoPROC job (reported by D. Reinert). * Introduced the new parameter autoPROC_CreateReportStopOnError (default="no") in order to not stop with an error if the PDF-report generation fails. Since this is a still a very new feature we don't want any problems to upset users automated pipelines (or our own Pipedream software). * Added (optional) auto-refresh function to the generated summary.html file that is active as long as it is still written to (suggested by P.Lukat). This feature can be activated by setting the parameter autoPROC_CreateSummaryUseJavaScriptAutoRefresh (refresh rate in seconds) to a value greater than 0. * New features, fixes and improvements (released Jul 06 2016): * Change of defaults to AutoProcScale_RunStaraniso=yes => run STARANISO for each dataset autoPROC_CreateXml=yes => create ISPyB-compatible XML file for each merged dataset autoPROC_ScaleEarlyLateCreate=yes => create (if possible) amplitudes for F(early)-F(late) difference Fourier maps: these can then be computed automatically in BUSTER (setting parameter BusterEarlyLate=yes) to provide help in assessing the effects of radiation damage * First implementation of a PDF report generated for each merged datasets: the layout and content is a first design - any comments are very much welcomed. * Added support to hdf2mini-cbf (converter of Eiger HDF5 datasets into mini-cbf) for automatic detection of 32- versus 16-bit images. It now also supports GZIP compression as available as part of the HDF5 library (see: https://www.hdfgroup.org/HDF5/faq/compression.html). * Added further support to HDF5 metadata reader (in "hdf2mini-cbf" as well as "imginfo") for * reading "units" attribute for data items to ensure they are consistent with expectations; * check that sensor_thickness as written in HDF5 metadata file (*_master.h5) makes sense: if a value >= 320 mm is encountered, it will be reset to 1/1000th of this value. This works around a bug in images from * Eiger 9M at APS 21-ID-D (as of 20160426) * Eiger 9M at Soleil Proxima2 (known to be present between 20160131 to at least 20160415) * ensure that metadata arrays are read even if the apparent number of images and the number of items in those data arrays don't match. This works around an inconsistency found in images from * Eiger 9M at Soleil Proxima2 (20160415) * Added automatic analysis of statistics to give indication of likely high-resolution limit of anomalous signal. This should help users in getting an initial, rough estimate of anomalous signal. The parameter autoPROC_AnomalousSignal_AdjustAutomatically (default=yes) determines if subsequent stages of data processing should take a detected (or determined as being absent) anomalous signal into account. If the user runs with -ANO or -noANO command-line flags, that decision will always be used and no adaptation is done. * "imginfo" will now write rotation values (angles) with greater precision - required e.g. for extremely fine-sliced Eiger datasets. * Ensure also '<' and '>' characters render correctly in summary.html. * Newer AIMLESS versions (0.5.24 and later) have a changed default for the SDCORRECTION card (SAME instead of INDIVIDUAL) that requires a switch in logic for our aP_scale scaling module. * A table of statistics against resolution (in 20 bins, can be changed via the AutoProcScaleStatsUseMrfanaFinalNumBin parameter) is now also given for the AIMLESS scaling steps. * Added support for 'May 1, 2016' version of XDS. Please be aware that official XDS binaries after 20151015 up to 20160514 are known to show an issue in indexing (IDXREF): indexing might fail or produce incorrect solutions in a seemingly random way without any obvious reason. An updated/fixed binary (for academic users) has been provided by the XDS developers as of 20160517. If you can't update to this fixed binary, this problem can sometimes be averted by giving a slightly different direct beam position (changed by very small values - like 0.1 pixel). * Started adding links to autoPROC manual directly from the summary.html file (more will follow). * To avoid too much information (especially for multi-sweep datasets), a hide/show feature has been added to summary.html: this means some plots are initially hidden and can be show through simple clicks on a green "+" icon. * Plots for scaling statistics (Rmerge, completeness etc) are now placed after the relevant tables in summary.html. * Several fixes to the automatic high-resolution limit estimation via MRFANA: this should stabilise this feature in case of multi-sweep datasets. * Added new parameters to listing in Appendix 1 of manual (AutoProcScale_RunStaraniso AutoProcScale_RunStaranisoCards AutoProcScale_RunStaranisoHklPlotGrid AutoProcScale_RunStaranisoPqrPlotGrid AutoProcScale_RunStaranisoStopOnError RunIdxrefStopOnAnyError XdsSpotSearchNumImagesAngularRange XdsUpdateAlsoSegments). * Fixes to regression described at http://www.globalphasing.com/autoproc/wiki/index.cgi?IssuesPage201603#refinekwd. * Now should also work if reference MTZ file is in a directory without write permissions. * Removed out-of-date and unnecessary macros: XdsFast and HighRes2. The "automatic" macro is deprecated and will be removed in a future release. * New features, fixes and improvements (released May 01 2016): * Added feature to run STARANISO directly as part of autoPROC: - STARANISO is a program developed by Ian Tickle at Global Phasing to analyse the local mean I/sigI to then apply an anisotropic cut-off and an anisotropy correction on merged intensity data. - This is currently only implemented for the AIMLESS scaling paths (i.e. the default behaviour or when running with the -M ScalingA3 macro) and can be activated with the AutoProcScale_RunStaraniso=yes command-line argument. - Data scaled in the normal way (but without applying the isotropic high-resolution limit determined) are given to STARANISO as input. - All analysis resulting from STARANISO is presented in the summary.html file, including plots showing the (potentially) anisotropic nature of diffraction. - The reflection file after applying this anisotropic limit and anisotropy correction can be used as an alternative to the standard autoPROC output MTZ file for subsequent stages, e.g. model refinement with BUSTER. - For more background information, details and examples please visit the STARANISO server at http://staraniso.globalphasing.org/. * fixed problem in extracting Rmerge for REMARK 200 section in remark200.pdb when using XSCALE path ("-M ScalingX" macro) * fixes to formatting of summary.html (to avoid overflow in menu area and correct Angstroem symbol in text) =================================================================== Changes included in autoPROC 1.0.4 stable Mar 24 2016 =================================================================== * Fixes and improvements (released Mar 24 2016): * hdf2mini-cbf now also reports usage help when run without argument (reported by O. Svensson, ESRF). * Fixed note written by aP_convert_hdf5 regarding number of external link files found. * Added (currently) necessary imginfo wrappers for processing of Eiger/HDF5 datasets from following beamlines: Eiger 4M at ESRF, MASSIF-3 Eiger 9M at APS 21-ID-D Eiger 9M at Soleil Proxima2 Eiger 4M at Photon Factory BL1A For details and updates, please also check http://www.globalphasing.com/autoproc/wiki/index.cgi?DataProcessingHdf5 * A new parameter can be used to simplify masking of damaged or hot pixels (if those are not yet accommodated directly by the detector via an updated pixel-mask): autoPROC_Img2Xds_DamagedPixels can be set to a space-separated list of X,Y pairs a la process autoPROC_Img2Xds_DamagedPixels="1234,432 3210,2345" ... Note: pixel indexes start at (1,1) as shown by our GPX2 viewer. Other viewers might show the first pixel as (0,0) and values read off those need to be adjusted by +1. * Fixes and improvements (released Mar 04 2016): * Fixes to generation of XML file when running "process" without the -d command-line argument (reported by L. Muir, IMCA-CAT). * Avoid attempt to create prediction pictures when processing HDF5 (Eiger/Dectris) datasets: GPX2 does not yet support reading HDF5 format. This removes an irritating (and unnecessary) error message from standard output (reported by D. Reinert, BI). * Fixes and improvements (released Mar 02 2016): * "process -checkdeps" was warning about "staraniso" not being present when it should have ignored this (thanks to K. Bataille from IMCA-CAT for pointing this out). There will probably be a feature in autoPROC to run STARANISO directly as part of processing, but for the moment the recommended usage of STARANISO is to use the web service at http://staraniso.globalphasnig.org/. For that purpose a script (aimless_alldata.sh) is automatically generated within the output directory that can be used to create a reflection MTZ file adequate for the web server. * Fixed messages from HDF5 converter script (aP_convert_hdf5). * Fixes and improvements (released Feb 25 2016): * Added support for HDF5 datasets from Dectris/Eiger detector. * HDF5 datasets can be given to process via the new -h5 command-line argument or as part of the existing -Id flag. * The tool "hdf2mini-cbf" is used for extracting separate image files out of the HDF5 container. It can do this for a single image (emulating the H5ToXds tool from Dectris) or for a series of images. The output CBF images contain a fully populated mini-cbf header and can also be written directly as compressed *.gz files. Conversion and writing is parallelised. Handling of HDF5 files written with the bitshuffle compression is supported on both Linux and OsX. A utility (aP_convert_hdf5) is provided for simple and fast (e.g. by submission to a cluster of workstations) data image extraction and conversion. * GPX2: emit help text when an unknown option is met. * Fixed regression (introduced in snapshot release 20151125) resulting in unnecessary runs of INTEGRATE and CORRECT steps at first XDS invocation. * Consolidated behaviour and resulting file names in summary HTML and XML files under different run conditions. These include running with or without a -d argument and - if a -d argument is given - if it is an absolute or relative path or points outside of the current directory. There could still be unresolved issues when it comes to working in auto-mounted filesystems or when using symbolic links - in which cases 'pwd' and 'pwd -P' might return different results. * Fixes and improvements (released Dec 14 2015): * New feature to create early/late data values: * This is switched on by the new parameter autoPROC_ScaleEarlyLateCreate (default=no). * Currently this is only supported for AIMLESS scaling (default or ScalingA3 path) and single wavelength datasets. * The final MTZ file (truncate-unique.mtz etc) will contain F/SIGF columns with the extension "_early" and "_late". * For the definition of early and late datasets we use several points in the forward/backward completeness table. One point is of course the cross-over point, ie. where the two completeness curves cross over each other: we have to stay on either side of this. Optimal points for merging early/late images is then found by using the following parameters: * we want to have at least this completeness: autoPROC_EarlyLateCompletenessMin=0.85; * we stop once we reached that completeness: autoPROC_EarlyLateCompletenessMax=0.95; This means means we are now doing the early/late analysis a bit different to the first implementation in the previous snapshot release - where it was used only for marking the completeness-versus-batch plot. * the early/late border will not be shown if individual completeness is too low; * no need to divide all images into early/late if we can use much fewer images on either end with good completeness; Note: it is likely that the exact method for defining early/late datasets will change in future releases (additional criteria need to be considered). * Added support for RIGAKU/Eiger detectors via the new "RigakuEiger1M" macro. * Fixed regression: GPX2 was not packed up into the OSX tarball for the previous snapshot release. * Fixed generation of summary.html when job stops with an error message (reported by D. Reinert). * Improved writing of XML file: * new parameters autoPROC_CreateXml_ValTrue (default=1) and autoPROC_CreateXml_ValFalse (default=0) allow for different boolean types in local XML usage (True/1/true or False/0/false etc); * autoPROC.xml will be initialised with processingStatus False and only upon successful completion will we have True; * added support to inject externally defined XML elements via one of autoPROC_CreateXml_LocalElements="AutoProcContainer:AutoProcScalingContainer:AutoProcIntegrationContainer:Image:datasetID=12345" or (if value will contain spaces): autoPROC_CreateXml_LocalElements="AutoProcContainer:AutoProcScalingContainer:AutoProcIntegrationContainer:Image:datasetID" AutoProcContainer_AutoProcScalingContainer_AutoProcIntegrationContainer_Image_datasetID="12345" * Reworked feature to automatically select a subset of images for spot searching: the previous incarnation didn't really work. We now have * maximum number of images to use for spot-searching: XdsSpotSearchNumImages="999999"; * the images used for spot searching should cover this angular range: XdsSpotSearchNumImagesAngularRange="360"; * number of segments to use within XdsSpotSearchNumImagesAngularRange: XdsSpotSearchNumRanges="10"; * only images within this angular range (from first image) are considered: XdsSpotSearchAngularRange="360"; To use only a subset of images one could set XdsSpotSearchNumImagesAngularRange="10.0" XdsSpotSearchNumRanges=4 XdsSpotSearchAngularRange=180 CAVEAT: Please be aware that this might significantly hamper autoPROC's ability to detect multiple lattices and ice-rings (and take corrective measures). By using such settings the user might not become aware of serious problems with the dataset. * Allow the possibility to define pre-computed files (like {X,Y}-CORRECTIONS.cbf) to be reused even if they have an arbitrary filename. * Added new autoPROC_CreateGpxPicturesAtRotationAngles parameter (default="0 30 60 90") to define angles for GPX2/prediction pictures relative to first image. * Improved handling of reindexing requirement for multi-sweep datasets - when the SG symmetry allows this. This includes additional checks when running with or without a reference MTZ file and with EnsureConsistentIndexing=yes (default) or EnsureConsistentIndexing=no (non-default). * Corrections to handling of centro-symmetric spacegroups in MRFANA. * Ensure "xparm2simin" can be run stand-alone for multi-axis goniostats (as defined in our distributed $autoPROC_home/autoPROC/lib/kapparot.sites file) * The FRAME.cbf file written by COLSPOT is now saved as FRAME_COLSPOT.cbf: this can show some details about the spot-searching in XDS/COLSPOT. * Changed behaviour of aP_scale scaling module (invisible unless run explicitly with "aP_scale -id 0"): * by default output will have no prefix (before: "1_") * therefore there is no need to use "-id 0" in order to switch from "1_" to "" prefix. The value given with "-id" is now used as-is to generate "_" prefix for all output files; * Fixes and improvements (released Nov 25 2015): * Added support for writing of a HTML page (default = "summary.html" in the output directory) containing information about autoPROC run. This should simplify inspection of all the graphical output and plots produced. * it involves renaming some files or creating different file names (e.g. SPOT_never-indexed.XDS) in order to avoid the overwriting of files at different stages - which would confuse the HTML output; * a tarball is created (default = "summary.tar.gz" in the output directory) that will contain this HTML document as well as most files (plots etc) referenced therein; * Added support for XDS version "Oct 15, 2015", CCP4 6.5.018, POINTLESS 1.10.10 and AIMLESS 0.5.15. * Added support for the long-wavelength beamline I23 at the Diamond Light Source. * Added support for writing XML file(s): * triggered by command-line option "-xml"; * standard file name "autoPROC.xml" (for single wavelength data) and "autoPROC_.xml" for multi-wavelength data, all written into the output directory; * it follows (and expands slightly) the ISPyB examples as far as these are documented; * Changed default for GPX2 wrapper script (gpx.sh): after closing GPX2 the predictions will be removed. In order to keep them, the command-line argument "-keep" can be used. * Fixed bug in GPX2 wrapper script (gpx.sh): for very high-resolution and/or large unit cell datasets the prediction generation could have stop with a segmentation fault because the increase in stacksize limit was actually ignored. * Speedup generation of predictions with simcal_predict for large structures by enlarging the size of hash-tables. * Made xds_spots2res work for curved detectors by using anaspot to compute resolution of spots. * At the beginning of output the installation directory for autoPROC is now also shown. * Scaling module aP_scale: * new tool ana_mrfana gives more detailed plots as a function of resolution or image number (i.e. batch or dataset); * improved handling of data without anomalous data (-noANO flag to process or aP_scale): now also taken into account for XSCALE path and the potential conversion into *.sca files; * fix logic to avoid running MRFANA twice in all cases (even if not necessary); * Fixes in MRFANA regarding computations for centrosymmetric space groups which previously caused the printing of NaN. * A warning message will be displayed when using the MOSFLM path of autoPROC: this non-default option has not been developed or tested for quite some time and will be removed in one of the next releases. * The SPOT.XDS_orig file (original list of spots before doing any indexing) now doesn't contain any Miller indices - while SPOT.XDS will contain non-(0,0,0) indices for spots from the finally used indexing solution. * Added checks for consistent distance, wavelength, oscillation, 2-theta, chi, kappa and rotation angle within a set of images making up a sweep. * Improved handling of multi-angle goniostat values when dealing with data from such instruments (and image headers that contain those settings). * When giving only the space group name/number to process: add extension to iterative indexing output files for the two cycles that need to be run in such cases. * Skip initial IDXREF step if we are doing iterative-indexing anyway. * Better phrasing when going into iterative indexing to describe accurately why this step is performed. * Added fully functional Gnuplot 4.6.5 to distribution to package: this is intended only for running from within autoPROC jobs. A user can still provide their own (e.g. OS-provided) binary by setting it explicitly on the command-line, e.g. process gnuplot=/where/ever/bin/gnuplot ... * Added new parameter autoPROC_XdsMaxDistanceJitterWarning (default=0.2) to give a warning if the distance refinement results in results outside this range (relative to the mean value). * the parameter autoPROC_XdsMaxDistanceJitter (default = 1.0) is used to automatically trigger fixing of detector position refinement if needed; * parameter refinement resulting in the crystal-detector distance changing by more than 0.2 mm seems too large compared to the typical size of a crystal (on most beamlines the crystal would move out of the beam if it represented physical reality); * large changes in distance during parameter refinement mostly point to problematic diffraction patterns or poor spot shapes; * Changed the LowResOrTricky macro so that an additional iterative indexing is done after processing each sweep: this can highlight the exact nature and relation of multiple lattices better since a set of refined parameters (for describing the experiment) is now available. * A new macro (HighResCutOnCChalf) is included to base automatic high-resolution limit decision making solely on CC(1/2) value of 0.3. * A new macro (DiamondI04) for beamline I04 at Diamond is included. * At the start of processing (of each sweep/dataset) a short summary is given to show the information as read from the header of the first image. * Accommodate memory issues (either due to little physical memory or because of multiple jobs running simultaneously) when creating prediction pictures: give a warning message but don't make the full autoPROC job fail. * Fixes and improvements (released Sep 23 2015): * Introduced new parameter autoPROC_PointlessTolerance (default = 2.0): identical to parameter TOLERANCE as defined and used in POINTLESS. Increase this if the reference dataset is expected to deviate significantly from the dataset being processed. * Plots of misfits and spots now follow the usual image-viewing convention to be in sync with GPX2: origin is in top left corner. * Fix in GPX2-driving script (gpx.sh) regarding prediction file: ensuring it is removed after automatic picture generation and this removal is noticed when running the script interactively for manual inspection. * Better handling of inhouse (CuKa-wavelength) datasets when generating input to simcal_predict: if wavelength is between 1.5417 and 1.5419 A the polarisation defaults to an unpolarized beam. * Several fixes in MRFANA to stabilise the automatic decision making about an adequate high-resolution limit. This includes the use of a Bezier curve as a basis for testing the various cutoff-criteria. * Additional fixes in MRFANA regarding computation of completeness (especially anomalous completeness). * Fixed regression from previous snapshot (internal function aP_pre not being defined). * Fixes and improvements (released Aug 12 2015): * The computation of resolution within MRFANA has changed for MTZ files: instead of using the overall cell parameters the dataset-specific cell parameters are used now. This will have an effect on multi-sweep datasets, where each dataset comes with its own, slightly different set of cell parameters. It is then important to use the correct cell parameter for resolution computation in order to be consistent between stages (where we see only a single sweep or multiple sweeps simultaneously). * We now ensure that R-values in MRFANA (Rmerge, Rmeas and Rpim) are computed in the same way as in SCALA/AIMLESS. Please note that this is following the paper (Evans, P. (2006). Acta D62, 72-82), which is slightly different from others (Diederichs&Karplus/1997 and Weiss/1997) in the denominator. Additional information is given in the MRFANA output. * Added a table of statistics as a function of image/BATCH number in MRFANA. * Added more comments into various macro files to help user understanding the effect of each setting. * Fixed problem in analysis of iterative-indexing run (where failing IDXREF results were not excluded correctly and therefore sometime reported as a reference for the angle computation). * Fixed fetching of images when the user supplied an explicit image range (via the -Id flag): now only images within that range are used while before all images matching the given template were used. This was especially confusing during the GPX2 visualisation, since more images might have been presented than predictions were generated (if a subset of images was specified on the -Id flag). * Fixes and improvements (released Jul 15 2015): * GPX2 (prediction and image visualiser): * now also available for OS X (version 10.7 or higher) * introduced new GPX2 keyboard shortcuts to simplify interaction: Ctrl-Left : move to previous image Ctrl-Right : move to next image Ctrl-q : quit program * Ensure that the file pointed to by the autoPROC_PointlessExtra parameter is also included when running POINTLESS to check for consistent indexing against a user-supplied reference MTZ file. * Clarifications to the reporting of multiple indexing solutions (when entering iterative indexing). * Fixes to the reporting of minimum angle between multiple indexing solutions (and also mentioning the space group used at this point to determine the minimum angle). * Added parameter autoPROC_CreateGpxPicturesAndKeepFilterFile (default="no") to decide if we should keep the prediction files that are computed during generation of images showing diffraction and predictions. Since those files can become rather large, not keeping them seems a sensible default option. * Use a generic gpx.sh script (to visualise predictions with simcal_predict and GPX2) instead of generating one on-the-fly: this simplifies maintenance and allows introduction of new features. * the help messages (.../gpx.sh -h) will now also report the possible lattices (identifier and number of spots used) when this script is invoked in multi-lattice cases; * in multi-lattice cases, GPX2 will by default (i.e. when no explicit "-lat" argument is used) start with all lattices shown; * the "-lat" argument now also supports the use of "-lat all" to select all lattices in a simple way; * Added BeamCentreFrom=header:x,y to SlsPXIII macro to avoid unnecessary warning message (thanks to D. Reinert for pointing this out). * NEW FEATURE: incorporation of a visualisation capability for images and predictions (released Jun 16 2015): * The computation of the positions and shapes of reflections expected to appear on each image is performed in the newly added program simcal_predict. This tool is a particular use-mode of the simcal diffraction image simulation program written by Wlodek Paciorek and Claus Flensburg and extended as part of collaboration contract COL0044 between Global Phasing and the Diamond Light Source, specifically related to the I23 beamline. * Predicted spot positions and shapes are presented through ellipses superposed onto diffraction images using the new program GPX2, which was initially developed by Wolfgang Brehm (while in the group of Kay Diederichs at University of Konstanz/DE) during an internship with Global Phasing during the Summer of 2014 with generous support by the MX group (particularly PX-III) of the Swiss Light Source. At the moment, GPX2 is provided for Linux - a version of OS X is planned. * NEW FEATURE: new scaling options for using "pure" AIMLESS-only or XSCALE-only paths within autoPROC (released Jun 16 2015): * the AIMLESS-only scaling path can be selected via process -M ScalingA3 ... * the XSCALE-only scaling path is chosen with process -M ScalingX ... * the hybrid path (XDS/AIMLESS) is still the default * Updated documentation (released Jun 16 2015): * new sections about simcal_predict (computation of predicted positions and shapes) and GPX2 (display of those on top of the raw diffraction images) * new section about scaling modes implemented in scaling module aP_scale * new section "Interpreting autoPROC output" * enhanced examples and usage section and major reworking in all other parts * Changed defaults (released Jun 16 2015): * The diffraction image files will not be linked into the same directory the actual XDS processing is run, but into a subdirectory (autoPROC_LinkImagesIntoSubdir parameter, default = "Images"). * Ice-rings are now excluded automatically (if detected in the list of un-indexed spots): parameter XdsExcludeIceRingsAutomatically defaults to "yes". The width of the resolution range given to XDS for exclusion is enlarged slightly (parameter Xds_Spot2Res_IceRingWidthMultForXds with default 1.2). * Create pictures of diffraction images with predictions using GPX2 (parameter autoPROC_CreateGpxPictures defaults to "yes"). The creation of files for visualising with MOSFLM ("ipmosflm") is now disabled (parameter autoPROC_CreateMosflmScript="no"). * the parameters for divergence and mosaicity as determined by the XDS INTEGRATE step are reused in any subsequent integration pass (parameter XdsUpdateParKeepFixed="BEAM_DIVERGENCE:yes REFLECTING_RANGE:yes"). * Instead of using REBATCH and SORTMTZ we are now using our own programs MRFEDIT and MRFADD (parameter UseMrfadd=yes and UseMrfedit=yes). These should not suffer from the same size/array limitations. * a macro to switch back to previous defaults is provided: process -M Version103 ... * the "automatic" macro has been adapted accordingly * Clarified some command-line options in online-help given by "process -h" (released Jun 16 2015): * "-M" can also take a file name (and not only the name of a macro); therefore the "-D" flag is being deprecated since a duplication of that functionality. * the "-P" flag is deprecated (the MOSFLM path hasn't seen development for quite some time) * Support for (released Jun 16 2015): - CCP4 6.5 (update 011) - POINTLESS 1.9.31 - AIMLESS 0.5.9 * Removed "AutomaticChunking=yes" from "automatic" macro (released Jun 16 2015): if that feature is required it needs to be added manually as in process ... -M automatic AutomaticChunking=yes ... * Added a little jiffy to create image sets based on rotation angle instead of image file name: ln_images.sh (released Jun 16 2015). * Support for ALS beamline 4.2.2 including NOIR-1 detector (released Jun 16 2015). * Scaling module aP_scale (released Jun 16 2015): * resolution limits along reciprocal cell axes as reported by AIMLESS now uses the same criteria for CC(1/2) and I/sigI as the overall isotropic high-resolution limit. * NEW FEATURE: support for scaling path using XSCALE; one (or several) XDS_ASCII.HKL files can be given via the "-hkl" argument. * fixed erroneous halt when encountering -mtz :

,, syntax - found by D. Reinert * fixed logic for determining number of processors (to then use parallel AIMLESS feature) when running aP_scale standalone * some clean-up in the additional examples provided within the aP_scale.sh script generated automatically as an example for fine-tuning by the users * when re-running the scaling module aP_scale with the same identifier (prefix for output files): give more meaningful text so user knows better what to do next (thanks to S. Sheriff). * The SPOT.*.png files produced during the indexing stage (showing location of indexed and non-indexed spots on the detector surface) now use a variable radius for points depending on spot strength (released Jun 16 2015). * The remark200 tool will now also write versioning information for autoPROC and XSCALE into remark200.pdb file (released Jun 16 2015). * Added support to combine XDS INTEGRATE.HKL files via 'combine_files' tool (released Jun 16 2015). * When entering iterative indexing, also report angle between orientation matrices of each solution relative to the strongest one, ie the one with the most indexed spots (released Jun 16 2015). * Don't do detailed overload analysis by default: it takes time and will create more large files. This is under parameter control with autoPROC_XdsAnalyseOverloads=no being the default (released Jun 16 2015). * Added check for user-supplied -Id flags to 'process' command: ensure the first item (identifier) is unique (released Jun 16 2015). * Fixed corner case in MRFANA: if the input MTZ file contained batches outside of the ranges defined on RUN cards, an error was thrown. Since those batches will never be used in later analysis anyway we now report and skip them (released Jun 16 2015). * Also output CC(ano) (correlation of anomalous differences between random half-sets) into 'table 1' in MRFANA (released Jun 16 2015). * Fixed handling of slightly differing wavelength values (from BATCH header and/or dataset information) in our REBATCH-substitute program MRFEDIT (released Jun 16 2015). * Fixed spelling: parameter is called autoPROC_XdsMaxDistanceJitter and not autoPROC_XdsMaxDistanceChitter (released Jun 16 2015). * The need to obtain a separate licence key for each machine on which the software is to be run has been removed for academic users: a single valid licence key will now work on any number of machines (released Mar 16 2015). * Support for (released Mar 16 2015): - XDS version March 1, 2015 - CCP4 6.5 (including update 005): this includes all the change in logic due to the final digit in the reported version now corresponds to the update serial number - POINTLESS 1.9.27 - AIMLESS 0.5.4 * Read 'fraction of polarization' directly from Pilatus miniCBF-header if available (released Mar 16 2015). * Changes to MRFANA (released Mar 16 2015): * Major internal re-organisation of MRFANA code: this should not be visible to a user. * Implemented the computation of CC(DANO) between halfsets in MRFANA similar to AIMLESS. * MRFANA now writes a more compact version of the merging statistics table. The wide-format version can be produced with the "-wide-format" command-line argument. * Better detection/distinction between XDS_ASCII.HKL, INTEGRATE.HKL and unmerged XSCALE output files. * More detailed messages for POINTLESS and AIMLESS version checks during "checkdeps" stage (released Mar 16 2015). * Changes in scaling module aP_scale (released Mar 16 2015): * by default, each AIMLESS job will also write out the XMLOUT file (see http://www.ccp4.ac.uk/html/aimless.html#InputOutput). To switch this off set the parameter autoPROC_ScaleXmlout to "no". * New parameter autoPROC_ScaleWithAimlessRefineSdcorrSame (default="no") that allows to use the same SDCORRECTION values for all RUNs (subkeyword SAME on SDCORRECTION card) in AIMLESS path. * Changed setting AutomaticChunking_MaxRuns (default = 40) only for SCALA path. For the default AIMLESS path it is set to 9999. * Changed setting AutoProcScaleResolutionCompareDigits=3 to AutoProcScaleResolutionCompareDigits=0,2 to have different significance levels for low and high-resolution values when testing if convergence has been reached during scaling/merging with aP_scale. * Changed command-line argument in aP_scale from -reso to -R flag (more consistent with other programs/commands), but keep -reso as backwards compatibility (released Mar 16 2015). * Added check to -b flag: requires a range (-) pattern (otherwise stop with an error). * Try to catch an error in TRUNCATE (WIDTH OF BIN TOO LARGE) to then set the RANGES keyword explicitly (enforcing TRUNCATE to run through). * Create plots of Bdecay and Mn(k) as a function of image number as well. * Better explanation and additional checks for -x flag in aP_scale scaling module. * New parameter FindImages_MaxImageNumWidth (default = 6) to handle image file names where the prefix and the actual image serial number run into each other because of a missing delimiter character (released Mar 16 2015). * Added help regarding ice-ring exclusion to autoPROC reference card (released Mar 16 2015). * Added option to insert call to a program/script (parameter XdsRunScriptAfterColspot set to full path of this program) after COLSPOT (released Mar 16 2015). * New parameters UseKapparotForConsistentIndexingStopOnError (default = yes) and XdsGetRotationAxisViaKapparotStopOnError (default = yes) as a fallback to work with beamlines producing different image headers (e.g. with or without a mini-kappa goniostat installed). It should not be necessary to change these from default value (released Mar 16 2015). * Better error/warning message if image header is incomplete for consistent indexing between scans and/or when getting ROTATION_AXIS via Kapparot. Only relevant for multi-axis goniostat datasets (released Mar 16 2015). * anaspot: correct resolution computation for detector described with segments when the detector is not located at the origin (released Mar 16 2015). * If the new parameter XdsUpdateCorrectionTables (default = no) is set to yes, we will also update {X,Y}-CORRECTIONS.cbf with the G{X,Y}-CORRECTIONS.cbf files from CORRECT before re-running the integration step (released Mar 16 2015). * Nicer formatting of output from "imgdate.sh" jiffy: constant width for all file-names, dates and time-differences (released Mar 16 2015). * The analysis of and (possible) warning about image header inconsistencies is now under parameter control: autoPROC_ImginfoReportWarnings with a default of "yes" (released Mar 16 2015). * imginfo now also reads milliseconds from the timestamp in image headers if available (released Mar 16 2015). * Also handle runs where only the space group name is given (using the symm= argument) and we have to distinguish between hexagonal and rhombohedral settings of H3/H32 (released Mar 16 2015). * New parameter CombineFilesRunReindexOnAll in 'combine_files' utility (defaulting to yes): when running REINDEX on at least some MTZ files (to set correct space group), we also need to run it on all to ensure the SG operators are given in the same (CCP4/REINDEX) order. Otherwise SORTMTZ will get confused and stops with an error (because ISYM values would be wrong). This is due to POINTLESS using a different, clipper-based system of symmetry operators than other CCP4-based programs (released Mar 16 2015). * Also report ISa values after the XDS/CORRECT step (released Mar 16 2015). * The parameter XdsExcludeIceRingsAutomatically can now also be set to "all" - in which case all known ice-ring ranges will have their resolution ranges excluded in processing, no matter if those ice-rings are actually present/detectable or not (released Mar 16 2015). * Added more detailed notes and error messages during indexing stage to help spotting potential problems (released Mar 16 2015). * Renamed P13 and P14 beamline (PetraIII) macros to avoid warning message from macro reading routine (released Mar 16 2015). * Beamline macros for SLS (PX-III) and PetraIII (P13 and P14) will not set the silicon sensor thickness since that value is now read from the image header by imginfo (released Mar 16 2015). * improved 'ana_aimless' tool to deal with difference in table format between AIMLESS versions up to 0.1.32 and from 0.2.11. Also added a plot about cumulative normal and anomalous completeness and fixed handling of multi-wavelength (MAD) datasets (released Jul 26 2014). * improved the 'xds_stats' tool to give nicer plots (handling meaningful ranges, labels etc) and deal with old gnuplot 4.0 versions; non-square plots now default to XdsStats_size="800,600" (from previously "1200,800") to make them compatible with output from "ana_aimless" (released Jul 26 2014). * the 'imginfo' tool now also reads the sensor thickness (if given in mini-CBF header) from Pilatus-formatted images (released Jul 26 2014). * the 'rigaku2xds' tool (and therefore autoPROC) should now also support Pilatus 300K detectors on Rigaku (AFC-11) goniostats via the '-M RigakuPilatus300K' macro (released Jul 26 2014). * added support for P13 and P14 beamlines at Petra-III beamline via new macros (released Jul 26 2014). * fixed 'hkl2res.sh' jiffy when DETECTOR_DISTANCE is given on same line as ORGX/ORGY line (released Jul 26 2014). * fine-tuning the text when checking XDS version eg as part of the 'checkdeps' routine (released Jul 26 2014). * added jiffy ('imginfo_merge2cbf.sh') to process images that went through the XDS program MERGE2CBF (released Jul 26 2014). * changed default for range of images used in background determination: autoPROC will now use 3 ranges of 10 images each within the first 90 degree of data (released Jun 12 2014). * new macro (Version115) to move back to defaults for the previous stable version (released Jun 12 2014). * tested with latest POINTLESS version 1.9.8 (released Jun 12 2014). * give XDS references in standard output (released Jun 12 2014). * fixed problem when trying to print warning message (user requested more threads than available on machine, released Jun 12 2014). ------------------------------------------------------------------- <-- Below are the Release Notes for the previous stable release --> ------------------------------------------------------------------- =================================================================== Changes included in autoPROC version 1.0.3 stable May 12 2014 =================================================================== === === New features: === Installation instructions: * comprehensive guide how to install and test autoPROC on all major operating systems. These are available from the download page on www.globalphasing.com or included in each installation as $BDG_home/docs/installation/index.html (released May 07 2014). Parameter refinement: * when encountering instability of distance parameter refinement, detect and possible act upon it (released Mar 06 2014). If jitter is lower than half the autoPROC_XdsMaxDistanceChitter parameter value, keep it as a warning; if above this value: switch off distance refinement. Configuration: * a new tool ('aP_check') is available to test if all external dependencies (CCP4, XDS, POINTLESS, AIMLESS) are available and working. It can also be run via 'process -checkdeps' to give a short summary about those external dependencies (released Mar 06 2014). * the location of the XDS binary can be set via the environment variable BDG_TOOL_XDS (released Sep 13 2013). This is usually done in the files setup_local.sh and setup_local.csh located in $autoPROC_home (for a stand-alone installation) or $BDG_home (for all others). According to an existing general mechanism, this alternative way of specifying the location of the XDS binary can in turn be overridden on the command line (e.g. "process xds=/some/where/xds_par") or by a similar redefinition in a macro. External program support: * support for POINTLESS 1.92 as well as XDS "January 10, 2014 BUILT=20140307" (released May 07 2014). * support for CCP4 6.4.0 (including POINTLESS 1.8.17 and AIMLESS 0.2.17) as well as XDS "January 10, 2014 BUILT=20140211" (released Mar 06 2014). * support for POINTLESS 1.6.17, AIMLESS 0.1.24 and XDS "March 30, 2013 BUILT 20130706" (released Sep 13 2013). Simplifying support: * in case the user didn't save the standard output of the "process" or "aP_scale" command, we now report the filename where a copy of standard output was saved at the end of a run. These files are often the first information we might need to ask you for in order to help with user issues (released Sep 13 2013). Analysis: * the new parameter AutoProcScaleStatsUseMrfanaFinalNumBin (default = 20) is used at a final merging statistics calculation to use a coarser binning that is more useful for publications or deposition than the fine binning used for the automatic determination of the high-resolution cutoff (released Mar 06 2014). === === Changes: === Reciprocal space sampling: * reducing the values for parameters NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA and NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA from 21 to 15, since a too fine sampling doesn't seem necessary and this change reduces memory requirements (released May 07 2014). Iterative indexing: * the iterative indexing step will now output results into a subdirectory by default to reduce clutter in the main output directory (released Mar 06 2014). === === Improvements: === XDS support: * support of latest XDS version/build "January 10, 2014" (released Mar 06 2014). * better and more extensive checks on the XDS version (and build date) with improved feedback to the user (released Mar 06 2014). Beamline/Instrument support: * Support for Australian Synchrotron beamlines: new macros AustralianSyncMX1 and AustralianSyncMX2 (released Mar 06 2014). * Support for Soleil/PROXIMA1 Pilatus 6M images that mark the gaps not with -1 but leave them as 0 (released Mar 06 2014). The parameter autoPROC_Img2Xds_Pilatus6MGaps contains the gap specifications on a Pilatus 6M. With the new parameter autoPROC_Img2Xds_Pilatus6MIncludeUntrustedRectangle set to "yes" (default=no) those will be added to the list of UNTRUSTED_RECTANGLE. * improvements for non-square detectors (like the Dectris/Pilatus models) when the image header stores the beam centre in a non-standard way. The BeamCentreFrom parameter then needs to take this into account (released Mar 06 2014). On non-square detectors we swapped the (X,Y) beam-centre values and the NX/NY values at the same time. However, the NX/NY values are extremely unlikely to have been swapped by beamline software, since they relate directly to the fast/slow axis of the binary array. So now we actually calculate -y,-x = NX-y,NY-x as a default, while before we did -y,-x = NY-y,NX-x That latter calculation can also be used by setting the parameter BeamCentreFromNxNySwap to "yes". All this came to light from images at the Swiss-Norwegian beamline at the ESRF: a Pilatus 2M with the beam-centre stored as y,-x. Analysis: * MRFANA now also writes a dataset header for each merging statistics table: this helps identifying which images are included into the calculation of those statistics, what the average cell (and standard deviation) is etc (released Mar 06 2014). * calculation of <|DANO/sig(DANO)> in resolution shells added to MRFANA (released Mar 06 2014). * also write AIMLESS version into REMARK 200 section of remark20.pdb file (released Mar 06 2014). === === Fixes: === Scaling * when working with multiple scans that contain either more than 999 images or start with image numbers above 999: fix the assigning of the actual batch numbers for the final multi-sweep scaling run (released Mar 06 2014). General: * fix when updating XDS.INP file after POINTLESS analysis if the user gave an input spacegroup: the SPACE_GROUP_NUMBER keyword was lost (released Mar 06 2014). * fix in handling new versus old format of SPOT.XDS in our ANASPOT jiffy (released Mar 06 2014). * handle multi-axis Kappa goniostats better: sometimes those are described with 3 or 4 angles (including/excluding the Alpha angle) in the header (released Mar 06 2014). * if there are problems during parameter refinement in XDS/CORRECT: ensure that updating the XDS.INP and XPARM.XDS file takes this into account (released Mar 06 2014). =================================================================== Changes relative to 1.0.1 19 Jun 2013 =================================================================== === === New features: === Indexing: * When entering 'iterative indexing', also plot number of spots against images for all significant solutions (as fraction into run_idxref_spot_hkl_histfrc.png or as absolute values in run_idxref_spot_hkl_hist.png). A new parameter RunIdxref_SpotVersusImage_MinFrac (default=0.05) determines the significance level for lattices (with the default at least 5% of the number of spots for the most populated lattice). * Added feature to run new 'anaspot' program (which analyses SPOT.XDS files for ice-rings etc): if we have non-default resolution limits given by the user we need to intervene between the COLSPOT and the IDXREF stage. Calling anaspot will limit the found spots to the set resolution based on the current, idealised geometry. High-resolution criteria: * Implemented CC1/2 calculation according to Karplus & Diederichs (2012) paper. Final CC1/2 criteria to determine adequate high-resolution cutoff defaults to 0.3 (since 0.1 is too optimistic given the intrinsic noise a change in random seed already shows). Analysis: * Added plot combining divergence and mosaicity (since they are often related in interesting ways): this will not only print the estimated values (per image), but also the finally used values for the center region of the detector. * MTZANA tool for analysing (merged) MTZ files (MRFANA is the equivalent program for unmerged, multi-record MTZ files): added command-line arguments -colcmpl and -nocolcmpl to print (or not) completeness per column (for the resolution range of that column). * New tool (MTZANA) to provide very similar functionality to CCP4s MTZDUMP program - without some of the limitations in output format, that often make it difficult to parse the information correctly. === === Changes: === High-resolution criteria: * New set of defaults - same as previous HighRes2 macro: setvar ScaleAnaRmergeCut_123 "99.9999:99.9999" setvar ScaleAnaISigmaCut_123 "0.1:0.1 0.5:0.5 0.5:1.0 1.0:2.0" setvar ScaleAnaRpimallCut_123 "99.9999:99.9999 0.9:0.9 0.8:0.8 0.6:0.6" setvar ScaleAnaRmeasallCut_123 "99.9999:99.9999" setvar ScaleAnaCompletenessCut_123 "0.0:0.0" setvar ScaleAnaCChalfCut_123 "-1.0:-1.0 0.0:0.0 0.3:0.3" This means that Rmerge, Rmeas and Completeness are being ignored as a high-resolution cutoff-criterion. Rpim and CC1/2 are used very conservatively - and I/sigI (>= 2.0) is the main one. === === Improvements: === XDS support: * Put XDS parameters FIXED_SCALE_FACTOR and DATA_RANGE_FIXED_SCALE_FACTOR under XDS version control since availability depends on exact XDS version. * support for XDS version "March 30, 2013": - changes to format of XPARM.XDS file (contains all experiment settings): depending on the XDS version used, files generated by autoPROC will also follow the relevant format of this version. - additional keywords - changes to format of INTEGRATE.HKL (contains raw integrated intensities) - changes to various output log-files (mainly relevant to autoPROC are IDXREF.LP, INTEGRATE.LP) - changes in format of SPOT.XDS (contains list of found spots for indexing) - better versioning done by XDS (not just a version date, but now also a BUILT date is reported by XDS programs) Beamline/Instrument support: * Support for Pilatus images following the latest version of header specifications 1.3. See also https://www.dectris.com/technical_pilatus.html * Support for ALS 12.3.1 (SIBYLS) beamline. See also http://bl1231.als.lbl.gov/. Required settings (as of 201212) are: * 2-theta rotates opposite to the rotation axis: autoPROC_TwoThetaAxis="-1 0 0" * beam centre in header needs transforming: BeamCentreFrom="header:x,-y" * need to deal with beam centre in header stored including the 2-theta offset: imginfo="imginfo_ALS_12.3.1.sh" Running with process -M Als1231 ... should be sufficient. * Support for ALS 8.3.1 beamline: macro (Als831) and imginfo jiffy (imginfo_ALS_8.3.1.sh). Running with process -M Als831 ... should be sufficient. * Support for Pilatus 6M images from ID29@ESRF: these are marked as 'omega' scans while the current mini-CBF header from Dectris doesn't provide a specific Omega value. By using the EsrfId29 macro (-M EsrfId29) an appropriate override jiffy is called to dynamically correct those headers. Image file handling: * to enable handling of compressed (gz and bz2) images automatically, the new parameter FindImages_AllowCompressedImages should be set to "yes". Note: it might be a good idea to limit the number of threads for XDS if using this - otherwise there will be a lot of disk I/O (uncompression on-the-fly) by the XDS program, e.g with process autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_PROCESSORS=2 ... Please note that 'imginfo' can already work on compressed images directly. Spacegroup determination: * Extract scores for each symmetry element from the POINTLESS analysis: this will make it easier to detect lower symmetries (like pseudo-orthogonal monoclinic etc). Scaling: * Changes in MRFANA (analysis of unmerged reflection files) to work around Pointless/Aimless bug: currently these programs write incorrect information into the MTZ header (Laue group instead of Point group) which results in wrong setup of asymmetric-unit handling for reflection indices - which in turn results in incorrect handling of anomalous statistics calculations. * Changed autoPROC_ExcludeBadBatchesFromCycle from 2 to 3: to avoid excluding batches to early when scale factors, error-model and high-resolution limit are still being adjusted. * Changed parameter AutoProcScaleMinimumCycles from 2 to 4 to do at least 4 scaling cycles. * Also extract recommended anisotropic high resolution limits from AIMLESS (based on I/sigI > 2.0 and CC1/2 > 0.5 criteria. * When running with AIMLESS (default) we will allow wider gaps between batches (before classifying them as separate runs): this is mainly useful when dealing with fine-sliced data where sometimes no reflection is assigned to an intermediate image (therefore creating a gap). Therefore, the parameter AutoProcScaleMaxAllowedGapSize defaults to 5 with AIMLESS and 0 otherwise. * New option AutoProcScaleSetInitialHighRes (default = no) to apply an initial high-resolution cutoff based on of -0.2 (set by parameter AutoProcScaleSetInitialHighResIsigI) in the final shell. This should ensure that scaling doesn't fail because it sees a very large number of supposed spots that are actually pure noise - eg if the crystal-detector distance was set to a massively over-optimistic value. * We now also accept -Reso or -R instead of the standard -reso for setting resolution limits on the command-line. * When stopping (due to existing files that would otherwise be re-used and/or overwritten), also give a list of files that cause that error. This will make it easier for the user to remove those if wanted. * Create additional plots using "ana_aimless" command (in PNG format via gnuplot) that are useful: scaling B-factor (vs image/batch) and Rmerge (vs image/batch). The name of the created output files (prefix; default = "ana_aimless") can be set with the new (optional) argument -o. * Write script (aP_scale.sh) with actually used commands run: this should make it easier for a user to re-run the final scaling step. * Added parameter XdsScaleRunMultipleCriteria (default=no): additional scaling-only runs (in new aP_scale.sh) are thus switched off or on. * Using new MTZANA program instead of MTZDUMP: faster, better formatting and under our control (so easier to react and avoid surprises). Analysis: * Major reworking of the 'mkmovie' command (see -h output for details): usage of ffmpeg/avconv for encoding and MOSFLM/ADXV for image-generation still needs some testing and work. In general this is more a proof-of-concept than a robust way of generating movies for a set of diffraction images. * Improved generation of profile plots and movies in the 'mkmovie' command - including generation of gnuplot files that allow interactive handling (rotation) of 3D profile representations. Added command-line argument "-profile_only" to generate those. * New parameter XdsStats_StopOnGnuplotError (default=no) to control handling of errors during gnuplot: when set to "yes" autoPROC will stop if one of the plot-generating steps fails because of a gnuplot inconsistency. * Various changes to look-n-feel of plots: hopefully resulting in nicer plots. * Added diagnostic to warn if refined distance parameter moves more than autoPROC_XdsMaxDistanceChitter=1.0 mm from the mean. In those cases something else is most likely going on and at least the distance should be held fixed during integration (followed by several cycles of post-refinement and integration, eg. using the "-M LowResOrTricky" macro). General: * work around bug in AIMLESS/POINTLESS: unmerged MTZ files written by those programs (as of 20130517) have wrong content in the SYMINF record of the MTZ file header (Laue instead of Point group name). Contacted Phil Evans and CCP4 about that. * Extract POINTLESS program version via new generic function. * Better dealing with binning (for RSTATS-based statistics; shown when running with '-v' flag): setting CheckIndexing_RstatsBinningMaxNum to 50 (from 10) CheckIndexing_RstatsBinningMinWidth to 0.0001 (from 0.01) and adding CheckIndexing_RstatsBinningNumBin=10 CheckIndexing_RunMtzana="no" Also added command-line argument "-n " to set the number of bins to be used (beware: depending on completeness the actually used bins might still be different). Setting CheckIndexing_RunMtzana="yes" will run the data through the statistics part of MTZANA and calculate various CC values via this route too. * Better way to deal with differences between versions of POINTLESS: a generic function to extract version number from POINTLESS standard output will allow handling different versions of POINTLESS (and the different input they accept and output they produce). * Changed default conversion factor between XDS mosaicity (sigma) to MOSFLM (FWHM): from 2.0 to 2.3548 - which it should be for a Gaussian. * When updating parameters in XDS.INP file using "xds_update_pars" module: take different behaviour of XDS versions into account (see Version July 4, 2012 in XDS release notes at http://xds.mpimf-heidelberg.mpg.de/html_doc/Release_Notes.html). * Started adding support for generating commands that will use our new visualiser (gpx) to display diffraction images and predictions. This is still under construction and must await the release of the visualiser and prediction generator before it becomes fully functional. * Make sure some temporary files are removed to keep things tidy. === === Fixes: === Indexing: * When dealing with distributed background images: avoid problems if processed images don't follow a simple numbering (from 1 to N). Also: create pseudo-contiguous images for this part in subdirectory (to avoid confusion). * Fixed problem in getting the correct number of indexed spots out of an IDXREF.LP file (when this was reported with two different numbers in two places). This resulted in the wrong percentage being reported on standard output. Spacegroup determination: * Deal with a change in POINTLESS XML-output that caused the reindexing operator not being fetched correctly within "ana_pointless". Scaling * In MRFANA: fix to ensure correct sorting of ASCII files from XDS (INTEGRATE.HKL, XDS_ASCII.HKL or XSCALE output) to avoid wrong calculation of anomalous statistics like completeness. * Always set VRSET to -999999999 (SortmtzVrsetValue parameter) in calls to SORTMTZ: this should avoid errors - for datasets that have most likely some issues with detector areas (beam stop, dead modules or such). * Fixes to automatic adjustment of image ranges (-b argument): to deal with multi-scan experiments that are not using different filename templates, but rather a continuation of image numbers (e.g. when using different spots on the crystal). * Fixes to setting arguments for scaling module (aP_scale): to deal with multi-scan experiments that are not using different filename templates, but rather a continuation of image numbers (e.g. when using different spots on the crystal). * In MRFANA: fix initialisation in calls to CCP4 library. This should avoid bogus warning messages from the CCP4 library about LRIDX. * Write correct scaling software (SCALA or AIMLESS) into REMARK 200 section of automatically generated remark200.pdb files. High-resolution criteria: * In high-resolution cutoff criteria: explicitly set R-values (to be ignored) to 99.9999 to be in sync with MRFANA program (where this is the value to mark a value that can't be calculated). General: * Try to get the command-line arguments into a better formatted string (for reporting). * Clarified warning message when giving only symm card (without cell). * Give detailed information regarding the number of threads used at various stages (and for XDS or MOSFLM path). Also telling the user how to set or modify the automatic settings for the different stages. * Fixes in "check_indexing" to avoid different resolution limits being used for different re-indexing operators. * Fix problem where the final summary information (about cell, spacegroup, resolution etc) was given based on a wrong MTZ file (present in current directory and not subdirectory). Also fix extraction of resolution limits. Previously, overall limits of reflections was given, but now resolution limits of actually measured amplitudes is reported. * Fixed a bug which may cause the scaling module aP_scale to fail with no meaningful error message on some systems if mtzana is not available. For more details see: https://www.globalphasing.com/beta/wiki/index.cgi?SoftwareReleases201211KnownIssues#autoprocapscale Additional checks have been implemented to work around this issue. New features: * support for scaling with AIMLESS to support larger datasets (more than 5000 images). This is now the default (if AIMLESS is available in the users PATH), but can be switched off (to use SCALA instead) with autoPROC_ScaleWithAimless=no. This requires CCP4 version 6.3.0 onwards! By default, the scaling will run with multiple threads (can be controlled by the -nthreads command-line argument to 'process' or through the OMP_NUM_THREADS environment variable). * new jiffy add_freerflag.sh (to add existing test-set flags from one MTZ file to another) Improvements: * added AutoProcScaleMinimumCycles (default=2) parameter to the scaling module (aP_scale): this gives more control in macros like the HighRes2 one mentioned below) * give warning if only resolutions below 4A were used for space-group determination in POINTLESS. Users are advised to check the correct or most likely spacegroup in such cases of low resolution or poor diffraction by hand. * added new ways of using macros: - using 'process -M show' will not only give a description for a particular macro, but show also the detailed settings. This can help users fine-tune some of the distributed macros. - a user can now also give a filename as argument to '-M', so that own macros can be given without the need of setting the environment variable autoPROC_MacroDirs. The file still needs to follow the standard macro-format. * two new macros (HighRes2 and NoHighResCut) to either use only I/sigI and Rpim as high-resolution criteria or none at all. * new macro (LowResOrTricky) with a few settings that could help low-resolution and/or messy diffraction data: mainly trying to stabilise parameter refinement during integration. * easier way to keep mosaicity fixed during integration in XDS: setting the parameter XdsUpdateMosaicityScale to a positive (non-zero) value will keep the mosaicity constant and fixed during the second (and following) integration path. The actual value is the value recommended by XDS (as reported in INTEGRATE.LP) scaled by the XdsUpdateMosaicityScale value. * support for latest XDS version (September 26, 2012) * added parameter autoPROC_TestSetFraction (default=0.05) to the scaling module (aP_scale). This will be used when calling uniqueify to generate a fresh test-set. * imginfo tool now also deals with DATE items in ADSC images of format, such as 'DATE=Fri Apr 23 18:12:09 PDT 2010;' * support for POINTLESS 1.6.18 (from CCP4 6.3.0 release) * added plot showing detector position of misfits (according to XDS definition) * improvements in the generation of plots via new parameters XdsStats_sizeSquare, XdsStats_MaxSize, XdsStats_PositionInMm and XdsStats_PlotMissingImages * support for MOSFLM 7.0.9 (in CCP4 6.3.0 release) * extract additional information from POINTLESS step about potential twinning * added parameter ExpectLargeAnomalousSignal (default="no") and XdsSetStrictAbsorptionChiSquareCut (default=1.5): those are used to adjust settings in the CORRECT step - to deal better with datasets showing very large anomalous signals (e.g. with metal clusters at low resolution). * when dealing with datasets that have slight gaps (empty or non-existing images) together with automatic 'chunking': make use of new parameter AutoProcScaleMaxAllowedGapSize (default=0) to avoid weird chunking due to those gaps. * when working with multi-scan datasets that will get merged into separate files (e.g. multi-wavelength MAD): added parameter AutoProcScaleCombineCrystalDatasetInCrystal (default=yes) to work around CCP4 library feature (that will lose cell parameters for a specific dataset, since it assigns cell on a per-crystal basis). This should avoid problems with the iterative scaling (in SCALA), merging statistics and high-res recommendation (in MRFANA) - otherwise the calculation of resolution for a given HKL uses slightly different cells and would therefore be out-of-sync. * more sensible message about images with different scale factors: still based on global statistics, but at least doesn't get confused by missing images or image numbers above 999 * imginfo: - additional checks for odd Pilatus headers (wavelength values in A but unit in nm, distance given in mm without a unit or with 'm' unit) - added checks for weird wavelength value (below 0.5A) and distance value (>100m or <1mm) to automatically adjust these header specification issues - catch Proxima1/Soleil case where we have Kappa and Chi values but no Omega (and Chi is identical to Starting_angle): pretend that this is a Kappa goniostat This is due to Proxima1's capability of doing both Omega- and Phi-scans on a Kappa goniostat when the Pilatus header specification doesn't provide an entry for Omega (so the starting Omega value is stored in the Chi parameter). This might change with updates to software by Dectris and the Proxima1 beamline. - handle negative zero values in angles (reset to 0.0): otherwise the KAPPAROT tool to enforce consistent indexing and/or calculate general rotation axis (eg. phi-scans on multi-axis goniostats) gets confused. * when combining several reflection files (within autoPROC or with the command-line tool 'combine_files'): take value of EnsureConsistentIndexing into account to prevent POINTLESS from wrongly re-indexing the data into an equivalent (but not identical) setting. * new parameter SpacegroupIndistinguishableGroups to allow better message for I23/I213 and I222/I212121 cases * new parameter FindImages_ExcludeLastImage (default=no) to exclude the last image of a found scan: deals e.g. with sync issues at SLS-PXIII regarding shutter-closure * support for: - SLS-PXIII (Prigo) beamline configuration and Pilatus headers through macro: % process -M SlsPXIII ... - Soleil-Proxima1 (Kappa) beamline configuration and Pilatus headers through macro: % process -M SoleilProxima1 ... - NSLS-X25 (initial Pilatus 6M headers) through imginfo-wrapper: % process imginfo=imginfo_NSLS_X25.sh ... - support for Pilatus 6M images from ID29@ESRF: these are marked as 'omega' scans while the current mini-CBF header from Dectris doesn't provide a specific Omega value. By using the EsrfId29 macro an appropriate override jiffy is called to dynamically correct those headers: % process -M EsrfId29 ... * New tool (MTZANA) to provide very similar functionality to CCP4s MTZDUMP program (without some of the limitations in output format, that often make it difficult to parse the information correctly). * make sure some temporary files are removed to keep things tidy. Fixes: * handle multi-scan cases where one (or several) scans had more than 999 images (which could previously give an error in the final, multi-scan scaling step). * explicitly set VRSET in SORTMTZ to avoid odd failures when combining several MTZ files for scaling. However, those failures usually meant very poor or wrong beamstop masking, leading to wrongly integrated intensities behind the shadow. * added checks in MRFANA to avoid memory corruption. * added safety margin for low-resolution limit suggestion from MRFANA: avoids 'losing' low-resolution reflections due to slightly different methods of calculating resolution and/or rounding errors. This should also speed-up the scaling job (aP_scale) since it avoids unnecessary additional cycles of scaling with updated resolution limits. * fixed some wrong amounts of white-space in REMARK 200 generation (remark200.pdb file) * when combining files (using combine_files utility), take care to get correct batch numbering offset - relevant for large scans/datasets with more than 999 images. In the same combine_files utility, some reporting regarding necessary reindexing has been fixed (previously it might have reported reindexing when in fact it didn't do any). * better support for GnuPlot 4.4 (used in creating plots in PNG format) * fixes during indexing when iterative indexing is used (automatically determined or user-requested) and several indexing stages are necessary anyway (e.g. the user only supplied spacegroup but not cell parameters): pick the correct master list of spots for detection of (partially) empty datasets. * better messages when analysing indexed spots for potential high-resolution limit * fixed bug in the aP_scale module (when combining different analysis tools regarding automatic adjustment of high-resolution limits): the explanations why a certain high-resolution limit was picked were sometimes out-of-sync. * added check in iterative indexing to avoid continuing with obviously wrong solution: sometimes IDXREF seems to succeed whereas only very few spots were actually used for the indexing solution. The parameter XdsIndexingStopIfSpotsLessPercent (default=2) is used for that check. =================================================================== Changes relative to 1.0.0 20 Jul 2011 =================================================================== New features: Improvements and changes: * imginfo: - added support for Pilatus headers I03@Diamond - added support for ADSC/Rigaku combo headers - allow APS1 prefix in header items - additional checks when encountering odd/duplicate oscillation axis definitions * added some jiffies: - beam8.sh will report all 8 possible beam centre settings - imgdate.sh can be used to analyse collections of images based on the timestamp in the image header * better method for getting completeness (for overload analysis) from reflection files * added option autoPROC_XdsAcceptOverloads (default=no) to generate a second set of files with overloaded reflections being accepted instead of ignored. This is only meant to be a desperate and last measure! * MRFANA (merging statistics calculation): - better order of inclusion criteria of reflexions (resolution check now before sysabs) resulting in significant speedup. - ice-ring treatment improved through (among several changes): a) using ice-ring limits instead of mid-point (with fixed width) b) using experimentally observed values for ice-ring resolution limits from large number of JCSG datasets This results in much more reliable treatment of ice-rings. Statistics used in applying the various cut-off criteria (for the high-resolution limit) now use an interpolation across those ice-ring resolution shells. * aP_scale (scaling module): - significant speedup through changes in MRFANA (see above) and a faster invocation of MTZDUMP (to read header information from MTZ files) - changed default so that MRFANA will assume by default that the data could contain ice-rings (AutoProcScaleStatsUseMrfanaIgnoreIceRingShells=yes) - new command-line option (-noice) to revert back to previous default, ie. no ice-rings present in data * find_images should give nicer identifier values (by removing trailing '0' characters in obvious cases). * changed from old XDS keyword 'MINUTE' to new 'SECONDS' * unless the user saves standard output already, write a (uniquely named) log-file to ensure that standard output is not lost. * new option to distribute background images over larger range of images (in chunks): DistributeBackgroundImages (default="no"). This should give a better initial background (BKGINIT.cbf file from the XDS INIT step), while avoiding using a very large number of images (which would be slower). * added analysis for empty images: we now require at least 2 (XdsSpotSearchMinNumSpotsPerImage) spots per image but allow 5 (XdsSpotSearchNumAllowedEmptyImages) empty images within the range of images used for spot searching. * lowered starting value for automatic backstop masking in XDS (DEFPIX step) to 5000 (XdsOptimizeDefpixStart), to deal with very weak images. * added option (XdsSetHighResFromIndexedSpots=no) to set high-resolution limit based on indexed spots. Fixes: * add protection against dash failing on 'empty' .autoPROC files * avoid setting reslow/reshigh outside normal command-line system * MRFANA: - corrected bug in accessing cached values of completeness (per resolution bin), therefore avoiding duplication of calls - resulting in further speedup. * fixed wrong reporting of reindexing requirements (due to changes in POINTLESS log-file format) * ensure that 'process' exits with status 0 if all went well * fixed problem in passing empty strings (of non-zero length) into parameters, eg. refine autoPROC_XdsKeyword_REFINE_INTEGRATE=" " ... will now work. * when analysing the low- and high-resolution limit of indexed spots: ensure that the correct set of parameters is used when the iterative indexing was used (because of multiple lattices, ice-rings etc). * deal with scans of more than 999 images (when creating plots of scale, divergence and mosaicity) #### NOTES The 32-bit Linux distributions of Global Phasing's software will work on systems running relatively old kernels (versions 2.4.x). Most mainstream server and desktop Linux distributions that are currently under active maintenance are based on the newer 2.6.x kernel series. The current 32-bit release will be the last to support Linux systems based on the 2.4.x kernel. Future distributions (including patches and minor updates) of our software for all Linux systems released after this one can only be relied on to work on systems running kernels from the 2.6.x series. This is only a significant change for our 32-bit distribution: our 64-bit distributions for Linux have always required kernel version 2.6.x To check the version of the kernel on a particular system, run the following command: uname -r =================================================================== Any feedback and remarks should be send to the autoPROC developers: proc-develop@globalphasing.com ===================================================================