################################################################## # # This is the latest (Mar 16 2015) snapshot release of the autoPROC # toolbox. It is based on the previous 1.0.3 release and available # to academic users. # # Copyright (C) 2005-2015 by Global Phasing Limited # # All rights reserved. # # This software is proprietary to and embodies the # confidential technology of Global Phasing Limited # (GPhL). Possession, use, duplication or dissemination of # the software is authorised only pursuant to a valid # written licence from GPhL. # # Authors: G. Bricogne, C. Flensburg, P. Keller, W. Paciorek, # A. Sharff, O. Smart, C. Vonrhein, T. Womack # # Reference: C. Vonrhein, C. Flensburg, P. Keller, A. Sharff, # O. Smart, W. Paciorek, T. Womack & G. Bricogne # (2011). Data processing and analysis with the autoPROC # toolbox. Acta Crystallogr D 67, 293-302. # ################################################################### =================================================================== Changes since the last autoPROC stable release Mar 16 2015 =================================================================== * The need to obtain a separate licence key for each machine on which the software is to be run has been removed for academic users: a single valid licence key will now work on any number of machines (released Mar 16 2015). * Support for (released Mar 16 2015): - XDS version March 1, 2015 - CCP4 6.5 (including update 005): this includes all the change in logic due to the final digit in the reported version now corresponds to the update serial number - POINTLESS 1.9.27 - AIMLESS 0.5.4 * Read 'fraction of polarization' directly from Pilatus miniCBF-header if available (released Mar 16 2015). * Changes to MRFANA (released Mar 16 2015): * Major internal re-organisation of MRFANA code: this should not be visible to a user. * Implemented the computation of CC(DANO) between halfsets in MRFANA similar to AIMLESS. * MRFANA now writes a more compact version of the merging statistics table. The wide-format version can be produced with the "-wide-format" command-line argument. * Better detection/distinction between XDS_ASCII.HKL, INTEGRATE.HKL and unmerged XSCALE output files. * More detailed messages for POINTLESS and AIMLESS version checks during "checkdeps" stage (released Mar 16 2015). * Changes in scaling module aP_scale (released Mar 16 2015): * by default, each AIMLESS job will also write out the XMLOUT file (see http://www.ccp4.ac.uk/html/aimless.html#InputOutput). To switch this off set the parameter autoPROC_ScaleXmlout to "no". * New parameter autoPROC_ScaleWithAimlessRefineSdcorrSame (default="no") that allows to use the same SDCORRECTION values for all RUNs (subkeyword SAME on SDCORRECTION card) in AIMLESS path. * Changed setting AutomaticChunking_MaxRuns (default = 40) only for SCALA path. For the default AIMLESS path it is set to 9999. * Changed setting AutoProcScaleResolutionCompareDigits=3 to AutoProcScaleResolutionCompareDigits=0,2 to have different significance levels for low and high-resolution values when testing if convergence has been reached during scaling/merging with aP_scale. * Changed command-line argument in aP_scale from -reso to -R flag (more consistent with other programs/commands), but keep -reso as backwards compatibility (released Mar 16 2015). * Added check to -b flag: requires a range (-) pattern (otherwise stop with an error). * Try to catch an error in TRUNCATE (WIDTH OF BIN TOO LARGE) to then set the RANGES keyword explicitly (enforcing TRUNCATE to run through). * Create plots of Bdecay and Mn(k) as a function of image number as well. * Better explanation and additional checks for -x flag in aP_scale scaling module. * New parameter FindImages_MaxImageNumWidth (default = 6) to handle image file names where the prefix and the actual image serial number run into each other because of a missing delimiter character (released Mar 16 2015). * Added help regarding ice-ring exclusion to autoPROC reference card (released Mar 16 2015). * Added option to insert call to a program/script (parameter XdsRunScriptAfterColspot set to full path of this program) after COLSPOT (released Mar 16 2015). * New parameters UseKapparotForConsistentIndexingStopOnError (default = yes) and XdsGetRotationAxisViaKapparotStopOnError (default = yes) as a fallback to work with beamlines producing different image headers (e.g. with or without a mini-kappa goniostat installed). It should not be necessary to change these from default value (released Mar 16 2015). * Better error/warning message if image header is incomplete for consistent indexing between scans and/or when getting ROTATION_AXIS via Kapparot. Only relevant for multi-axis goniostat datasets (released Mar 16 2015). * anaspot: correct resolution computation for detector described with segments when the detector is not located at the origin (released Mar 16 2015). * If the new parameter XdsUpdateCorrectionTables (default = no) is set to yes, we will also update {X,Y}-CORRECTIONS.cbf with the G{X,Y}-CORRECTIONS.cbf files from CORRECT before re-running the integration step (released Mar 16 2015). * Nicer formatting of output from "imgdate.sh" jiffy: constant width for all file-names, dates and time-differences (released Mar 16 2015). * The analysis of and (possible) warning about image header inconsistencies is now under parameter control: autoPROC_ImginfoReportWarnings with a default of "yes" (released Mar 16 2015). * imginfo now also reads milliseconds from the timestamp in image headers if available (released Mar 16 2015). * Also handle runs where only the space group name is given (using the symm= argument) and we have to distinguish between hexagonal and rhombohedral settings of H3/H32 (released Mar 16 2015). * New parameter CombineFilesRunReindexOnAll in 'combine_files' utility (defaulting to yes): when running REINDEX on at least some MTZ files (to set correct space group), we also need to run it on all to ensure the SG operators are given in the same (CCP4/REINDEX) order. Otherwise SORTMTZ will get confused and stops with an error (because ISYM values would be wrong). This is due to POINTLESS using a different, clipper-based system of symmetry operators than other CCP4-based programs (released Mar 16 2015). * Also report ISa values after the XDS/CORRECT step (released Mar 16 2015). * The parameter XdsExcludeIceRingsAutomatically can now also be set to "all" - in which case all known ice-ring ranges will have their resolution ranges excluded in processing, no matter if those ice-rings are actually present/detectable or not (released Mar 16 2015). * Added more detailed notes and error messages during indexing stage to help spotting potential problems (released Mar 16 2015). * Renamed P13 and P14 beamline (PetraIII) macros to avoid warning message from macro reading routine (released Mar 16 2015). * Beamline macros for SLS (PX-III) and PetraIII (P13 and P14) will not set the silicon sensor thickness since that value is now read from the image header by imginfo (released Mar 16 2015). * improved 'ana_aimless' tool to deal with difference in table format between AIMLESS versions up to 0.1.32 and from 0.2.11. Also added a plot about cumulative normal and anomalous completeness and fixed handling of multi-wavelength (MAD) datasets (released Jul 26 2014). * improved the 'xds_stats' tool to give nicer plots (handling meaningful ranges, labels etc) and deal with old gnuplot 4.0 versions; non-square plots now default to XdsStats_size="800,600" (from previously "1200,800") to make them compatible with output from "ana_aimless" (released Jul 26 2014). * the 'imginfo' tool now also reads the sensor thickness (if given in mini-CBF header) from Pilatus-formatted images (released Jul 26 2014). * the 'rigaku2xds' tool (and therefore autoPROC) should now also support Pilatus 300K detectors on Rigaku (AFC-11) goniostats via the '-M RigakuPilatus300K' macro (released Jul 26 2014). * added support for P13 and P14 beamlines at Petra-III beamline via new macros (released Jul 26 2014). * fixed 'hkl2res.sh' jiffy when DETECTOR_DISTANCE is given on same line as ORGX/ORGY line (released Jul 26 2014). * fine-tuning the text when checking XDS version eg as part of the 'checkdeps' routine (released Jul 26 2014). * added jiffy ('imginfo_merge2cbf.sh') to process images that went through the XDS program MERGE2CBF (released Jul 26 2014). * changed default for range of images used in background determination: autoPROC will now use 3 ranges of 10 images each within the first 90 degree of data (released Jun 12 2014). * new macro (Version115) to move back to defaults for the previous stable version (released Jun 12 2014). * tested with latest POINTLESS version 1.9.8 (released Jun 12 2014). * give XDS references in standard output (released Jun 12 2014). * fixed problem when trying to print warning message (user requested more threads than available on machine, released Jun 12 2014). ------------------------------------------------------------------- <-- Below are the Release Notes for the previous stable release --> ------------------------------------------------------------------- =================================================================== Changes included in autoPROC version 1.0.3 stable May 12 2014 =================================================================== === === New features: === Installation instructions: * comprehensive guide how to install and test autoPROC on all major operating systems. These are available from the download page on www.globalphasing.com or included in each installation as $BDG_home/docs/installation/index.html (released May 07 2014). Parameter refinement: * when encountering instability of distance parameter refinement, detect and possible act upon it (released Mar 06 2014). If jitter is lower than half the autoPROC_XdsMaxDistanceChitter parameter value, keep it as a warning; if above this value: switch off distance refinement. Configuration: * a new tool ('aP_check') is available to test if all external dependencies (CCP4, XDS, POINTLESS, AIMLESS) are available and working. It can also be run via 'process -checkdeps' to give a short summary about those external dependencies (released Mar 06 2014). * the location of the XDS binary can be set via the environment variable BDG_TOOL_XDS (released Sep 13 2013). This is usually done in the files setup_local.sh and setup_local.csh located in $autoPROC_home (for a stand-alone installation) or $BDG_home (for all others). According to an existing general mechanism, this alternative way of specifying the location of the XDS binary can in turn be overridden on the command line (e.g. "process xds=/some/where/xds_par") or by a similar redefinition in a macro. External program support: * support for POINTLESS 1.92 as well as XDS "January 10, 2014 BUILT=20140307" (released May 07 2014). * support for CCP4 6.4.0 (including POINTLESS 1.8.17 and AIMLESS 0.2.17) as well as XDS "January 10, 2014 BUILT=20140211" (released Mar 06 2014). * support for POINTLESS 1.6.17, AIMLESS 0.1.24 and XDS "March 30, 2013 BUILT 20130706" (released Sep 13 2013). Simplifying support: * in case the user didn't save the standard output of the "process" or "aP_scale" command, we now report the filename where a copy of standard output was saved at the end of a run. These files are often the first information we might need to ask you for in order to help with user issues (released Sep 13 2013). Analysis: * the new parameter AutoProcScaleStatsUseMrfanaFinalNumBin (default = 20) is used at a final merging statistics calculation to use a coarser binning that is more useful for publications or deposition than the fine binning used for the automatic determination of the high-resolution cutoff (released Mar 06 2014). === === Changes: === Reciprocal space sampling: * reducing the values for parameters NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA and NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA from 21 to 15, since a too fine sampling doesn't seem necessary and this change reduces memory requirements (released May 07 2014). Iterative indexing: * the iterative indexing step will now output results into a subdirectory by default to reduce clutter in the main output directory (released Mar 06 2014). === === Improvements: === XDS support: * support of latest XDS version/build "January 10, 2014" (released Mar 06 2014). * better and more extensive checks on the XDS version (and build date) with improved feedback to the user (released Mar 06 2014). Beamline/Instrument support: * Support for Australian Synchrotron beamlines: new macros AustralianSyncMX1 and AustralianSyncMX2 (released Mar 06 2014). * Support for Soleil/PROXIMA1 Pilatus 6M images that mark the gaps not with -1 but leave them as 0 (released Mar 06 2014). The parameter autoPROC_Img2Xds_Pilatus6MGaps contains the gap specifications on a Pilatus 6M. With the new parameter autoPROC_Img2Xds_Pilatus6MIncludeUntrustedRectangle set to "yes" (default=no) those will be added to the list of UNTRUSTED_RECTANGLE. * improvements for non-square detectors (like the Dectris/Pilatus models) when the image header stores the beam centre in a non-standard way. The BeamCentreFrom parameter then needs to take this into account (released Mar 06 2014). On non-square detectors we swapped the (X,Y) beam-centre values and the NX/NY values at the same time. However, the NX/NY values are extremely unlikely to have been swapped by beamline software, since they relate directly to the fast/slow axis of the binary array. So now we actually calculate -y,-x = NX-y,NY-x as a default, while before we did -y,-x = NY-y,NX-x That latter calculation can also be used by setting the parameter BeamCentreFromNxNySwap to "yes". All this came to light from images at the Swiss-Norwegian beamline at the ESRF: a Pilatus 2M with the beam-centre stored as y,-x. Analysis: * MRFANA now also writes a dataset header for each merging statistics table: this helps identifying which images are included into the calculation of those statistics, what the average cell (and standard deviation) is etc (released Mar 06 2014). * calculation of <|DANO/sig(DANO)> in resolution shells added to MRFANA (released Mar 06 2014). * also write AIMLESS version into REMARK 200 section of remark20.pdb file (released Mar 06 2014). === === Fixes: === Scaling * when working with multiple scans that contain either more than 999 images or start with image numbers above 999: fix the assigning of the actual batch numbers for the final multi-sweep scaling run (released Mar 06 2014). General: * fix when updating XDS.INP file after POINTLESS analysis if the user gave an input spacegroup: the SPACE_GROUP_NUMBER keyword was lost (released Mar 06 2014). * fix in handling new versus old format of SPOT.XDS in our ANASPOT jiffy (released Mar 06 2014). * handle multi-axis Kappa goniostats better: sometimes those are described with 3 or 4 angles (including/excluding the Alpha angle) in the header (released Mar 06 2014). * if there are problems during parameter refinement in XDS/CORRECT: ensure that updating the XDS.INP and XPARM.XDS file takes this into account (released Mar 06 2014). =================================================================== Changes relative to 1.0.1 19 Jun 2013 =================================================================== === === New features: === Indexing: * When entering 'iterative indexing', also plot number of spots against images for all significant solutions (as fraction into run_idxref_spot_hkl_histfrc.png or as absolute values in run_idxref_spot_hkl_hist.png). A new parameter RunIdxref_SpotVersusImage_MinFrac (default=0.05) determines the significance level for lattices (with the default at least 5% of the number of spots for the most populated lattice). * Added feature to run new 'anaspot' program (which analyses SPOT.XDS files for ice-rings etc): if we have non-default resolution limits given by the user we need to intervene between the COLSPOT and the IDXREF stage. Calling anaspot will limit the found spots to the set resolution based on the current, idealised geometry. High-resolution criteria: * Implemented CC1/2 calculation according to Karplus & Diederichs (2012) paper. Final CC1/2 criteria to determine adequate high-resolution cutoff defaults to 0.3 (since 0.1 is too optimistic given the intrinsic noise a change in random seed already shows). Analysis: * Added plot combining divergence and mosaicity (since they are often related in interesting ways): this will not only print the estimated values (per image), but also the finally used values for the center region of the detector. * MTZANA tool for analysing (merged) MTZ files (MRFANA is the equivalent program for unmerged, multi-record MTZ files): added command-line arguments -colcmpl and -nocolcmpl to print (or not) completeness per column (for the resolution range of that column). * New tool (MTZANA) to provide very similar functionality to CCP4s MTZDUMP program - without some of the limitations in output format, that often make it difficult to parse the information correctly. === === Changes: === High-resolution criteria: * New set of defaults - same as previous HighRes2 macro: setvar ScaleAnaRmergeCut_123 "99.9999:99.9999" setvar ScaleAnaISigmaCut_123 "0.1:0.1 0.5:0.5 0.5:1.0 1.0:2.0" setvar ScaleAnaRpimallCut_123 "99.9999:99.9999 0.9:0.9 0.8:0.8 0.6:0.6" setvar ScaleAnaRmeasallCut_123 "99.9999:99.9999" setvar ScaleAnaCompletenessCut_123 "0.0:0.0" setvar ScaleAnaCChalfCut_123 "-1.0:-1.0 0.0:0.0 0.3:0.3" This means that Rmerge, Rmeas and Completeness are being ignored as a high-resolution cutoff-criterion. Rpim and CC1/2 are used very conservatively - and I/sigI (>= 2.0) is the main one. === === Improvements: === XDS support: * Put XDS parameters FIXED_SCALE_FACTOR and DATA_RANGE_FIXED_SCALE_FACTOR under XDS version control since availability depends on exact XDS version. * support for XDS version "March 30, 2013": - changes to format of XPARM.XDS file (contains all experiment settings): depending on the XDS version used, files generated by autoPROC will also follow the relevant format of this version. - additional keywords - changes to format of INTEGRATE.HKL (contains raw integrated intensities) - changes to various output log-files (mainly relevant to autoPROC are IDXREF.LP, INTEGRATE.LP) - changes in format of SPOT.XDS (contains list of found spots for indexing) - better versioning done by XDS (not just a version date, but now also a BUILT date is reported by XDS programs) Beamline/Instrument support: * Support for Pilatus images following the latest version of header specifications 1.3. See also https://www.dectris.com/technical_pilatus.html * Support for ALS 12.3.1 (SIBYLS) beamline. See also http://bl1231.als.lbl.gov/. Required settings (as of 201212) are: * 2-theta rotates opposite to the rotation axis: autoPROC_TwoThetaAxis="-1 0 0" * beam centre in header needs transforming: BeamCentreFrom="header:x,-y" * need to deal with beam centre in header stored including the 2-theta offset: imginfo="imginfo_ALS_12.3.1.sh" Running with process -M Als1231 ... should be sufficient. * Support for ALS 8.3.1 beamline: macro (Als831) and imginfo jiffy (imginfo_ALS_8.3.1.sh). Running with process -M Als831 ... should be sufficient. * Support for Pilatus 6M images from ID29@ESRF: these are marked as 'omega' scans while the current mini-CBF header from Dectris doesn't provide a specific Omega value. By using the EsrfId29 macro (-M EsrfId29) an appropriate override jiffy is called to dynamically correct those headers. Image file handling: * to enable handling of compressed (gz and bz2) images automatically, the new parameter FindImages_AllowCompressedImages should be set to "yes". Note: it might be a good idea to limit the number of threads for XDS if using this - otherwise there will be a lot of disk I/O (uncompression on-the-fly) by the XDS program, e.g with process autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_PROCESSORS=2 ... Please note that 'imginfo' can already work on compressed images directly. Spacegroup determination: * Extract scores for each symmetry element from the POINTLESS analysis: this will make it easier to detect lower symmetries (like pseudo-orthogonal monoclinic etc). Scaling: * Changes in MRFANA (analysis of unmerged reflection files) to work around Pointless/Aimless bug: currently these programs write incorrect information into the MTZ header (Laue group instead of Point group) which results in wrong setup of asymmetric-unit handling for reflection indices - which in turn results in incorrect handling of anomalous statistics calculations. * Changed autoPROC_ExcludeBadBatchesFromCycle from 2 to 3: to avoid excluding batches to early when scale factors, error-model and high-resolution limit are still being adjusted. * Changed parameter AutoProcScaleMinimumCycles from 2 to 4 to do at least 4 scaling cycles. * Also extract recommended anisotropic high resolution limits from AIMLESS (based on I/sigI > 2.0 and CC1/2 > 0.5 criteria. * When running with AIMLESS (default) we will allow wider gaps between batches (before classifying them as separate runs): this is mainly useful when dealing with fine-sliced data where sometimes no reflection is assigned to an intermediate image (therefore creating a gap). Therefore, the parameter AutoProcScaleMaxAllowedGapSize defaults to 5 with AIMLESS and 0 otherwise. * New option AutoProcScaleSetInitialHighRes (default = no) to apply an initial high-resolution cutoff based on of -0.2 (set by parameter AutoProcScaleSetInitialHighResIsigI) in the final shell. This should ensure that scaling doesn't fail because it sees a very large number of supposed spots that are actually pure noise - eg if the crystal-detector distance was set to a massively over-optimistic value. * We now also accept -Reso or -R instead of the standard -reso for setting resolution limits on the command-line. * When stopping (due to existing files that would otherwise be re-used and/or overwritten), also give a list of files that cause that error. This will make it easier for the user to remove those if wanted. * Create additional plots using "ana_aimless" command (in PNG format via gnuplot) that are useful: scaling B-factor (vs image/batch) and Rmerge (vs image/batch). The name of the created output files (prefix; default = "ana_aimless") can be set with the new (optional) argument -o. * Write script (aP_scale.sh) with actually used commands run: this should make it easier for a user to re-run the final scaling step. * Added parameter XdsScaleRunMultipleCriteria (default=no): additional scaling-only runs (in new aP_scale.sh) are thus switched off or on. * Using new MTZANA program instead of MTZDUMP: faster, better formatting and under our control (so easier to react and avoid surprises). Analysis: * Major reworking of the 'mkmovie' command (see -h output for details): usage of ffmpeg/avconv for encoding and MOSFLM/ADXV for image-generation still needs some testing and work. In general this is more a proof-of-concept than a robust way of generating movies for a set of diffraction images. * Improved generation of profile plots and movies in the 'mkmovie' command - including generation of gnuplot files that allow interactive handling (rotation) of 3D profile representations. Added command-line argument "-profile_only" to generate those. * New parameter XdsStats_StopOnGnuplotError (default=no) to control handling of errors during gnuplot: when set to "yes" autoPROC will stop if one of the plot-generating steps fails because of a gnuplot inconsistency. * Various changes to look-n-feel of plots: hopefully resulting in nicer plots. * Added diagnostic to warn if refined distance parameter moves more than autoPROC_XdsMaxDistanceChitter=1.0 mm from the mean. In those cases something else is most likely going on and at least the distance should be held fixed during integration (followed by several cycles of post-refinement and integration, eg. using the "-M LowResOrTricky" macro). General: * work around bug in AIMLESS/POINTLESS: unmerged MTZ files written by those programs (as of 20130517) have wrong content in the SYMINF record of the MTZ file header (Laue instead of Point group name). Contacted Phil Evans and CCP4 about that. * Extract POINTLESS program version via new generic function. * Better dealing with binning (for RSTATS-based statistics; shown when running with '-v' flag): setting CheckIndexing_RstatsBinningMaxNum to 50 (from 10) CheckIndexing_RstatsBinningMinWidth to 0.0001 (from 0.01) and adding CheckIndexing_RstatsBinningNumBin=10 CheckIndexing_RunMtzana="no" Also added command-line argument "-n " to set the number of bins to be used (beware: depending on completeness the actually used bins might still be different). Setting CheckIndexing_RunMtzana="yes" will run the data through the statistics part of MTZANA and calculate various CC values via this route too. * Better way to deal with differences between versions of POINTLESS: a generic function to extract version number from POINTLESS standard output will allow handling different versions of POINTLESS (and the different input they accept and output they produce). * Changed default conversion factor between XDS mosaicity (sigma) to MOSFLM (FWHM): from 2.0 to 2.3548 - which it should be for a Gaussian. * When updating parameters in XDS.INP file using "xds_update_pars" module: take different behaviour of XDS versions into account (see Version July 4, 2012 in XDS release notes at http://xds.mpimf-heidelberg.mpg.de/html_doc/Release_Notes.html). * Started adding support for generating commands that will use our new visualiser (gpx) to display diffraction images and predictions. This is still under construction and must await the release of the visualiser and prediction generator before it becomes fully functional. * Make sure some temporary files are removed to keep things tidy. === === Fixes: === Indexing: * When dealing with distributed background images: avoid problems if processed images don't follow a simple numbering (from 1 to N). Also: create pseudo-contiguous images for this part in subdirectory (to avoid confusion). * Fixed problem in getting the correct number of indexed spots out of an IDXREF.LP file (when this was reported with two different numbers in two places). This resulted in the wrong percentage being reported on standard output. Spacegroup determination: * Deal with a change in POINTLESS XML-output that caused the reindexing operator not being fetched correctly within "ana_pointless". Scaling * In MRFANA: fix to ensure correct sorting of ASCII files from XDS (INTEGRATE.HKL, XDS_ASCII.HKL or XSCALE output) to avoid wrong calculation of anomalous statistics like completeness. * Always set VRSET to -999999999 (SortmtzVrsetValue parameter) in calls to SORTMTZ: this should avoid errors - for datasets that have most likely some issues with detector areas (beam stop, dead modules or such). * Fixes to automatic adjustment of image ranges (-b argument): to deal with multi-scan experiments that are not using different filename templates, but rather a continuation of image numbers (e.g. when using different spots on the crystal). * Fixes to setting arguments for scaling module (aP_scale): to deal with multi-scan experiments that are not using different filename templates, but rather a continuation of image numbers (e.g. when using different spots on the crystal). * In MRFANA: fix initialisation in calls to CCP4 library. This should avoid bogus warning messages from the CCP4 library about LRIDX. * Write correct scaling software (SCALA or AIMLESS) into REMARK 200 section of automatically generated remark200.pdb files. High-resolution criteria: * In high-resolution cutoff criteria: explicitly set R-values (to be ignored) to 99.9999 to be in sync with MRFANA program (where this is the value to mark a value that can't be calculated). General: * Try to get the command-line arguments into a better formatted string (for reporting). * Clarified warning message when giving only symm card (without cell). * Give detailed information regarding the number of threads used at various stages (and for XDS or MOSFLM path). Also telling the user how to set or modify the automatic settings for the different stages. * Fixes in "check_indexing" to avoid different resolution limits being used for different re-indexing operators. * Fix problem where the final summary information (about cell, spacegroup, resolution etc) was given based on a wrong MTZ file (present in current directory and not subdirectory). Also fix extraction of resolution limits. Previously, overall limits of reflections was given, but now resolution limits of actually measured amplitudes is reported. * Fixed a bug which may cause the scaling module aP_scale to fail with no meaningful error message on some systems if mtzana is not available. For more details see: https://www.globalphasing.com/beta/wiki/index.cgi?SoftwareReleases201211KnownIssues#autoprocapscale Additional checks have been implemented to work around this issue. New features: * support for scaling with AIMLESS to support larger datasets (more than 5000 images). This is now the default (if AIMLESS is available in the users PATH), but can be switched off (to use SCALA instead) with autoPROC_ScaleWithAimless=no. This requires CCP4 version 6.3.0 onwards! By default, the scaling will run with multiple threads (can be controlled by the -nthreads command-line argument to 'process' or through the OMP_NUM_THREADS environment variable). * new jiffy add_freerflag.sh (to add existing test-set flags from one MTZ file to another) Improvements: * added AutoProcScaleMinimumCycles (default=2) parameter to the scaling module (aP_scale): this gives more control in macros like the HighRes2 one mentioned below) * give warning if only resolutions below 4A were used for space-group determination in POINTLESS. Users are advised to check the correct or most likely spacegroup in such cases of low resolution or poor diffraction by hand. * added new ways of using macros: - using 'process -M show' will not only give a description for a particular macro, but show also the detailed settings. This can help users fine-tune some of the distributed macros. - a user can now also give a filename as argument to '-M', so that own macros can be given without the need of setting the environment variable autoPROC_MacroDirs. The file still needs to follow the standard macro-format. * two new macros (HighRes2 and NoHighResCut) to either use only I/sigI and Rpim as high-resolution criteria or none at all. * new macro (LowResOrTricky) with a few settings that could help low-resolution and/or messy diffraction data: mainly trying to stabilise parameter refinement during integration. * easier way to keep mosaicity fixed during integration in XDS: setting the parameter XdsUpdateMosaicityScale to a positive (non-zero) value will keep the mosaicity constant and fixed during the second (and following) integration path. The actual value is the value recommended by XDS (as reported in INTEGRATE.LP) scaled by the XdsUpdateMosaicityScale value. * support for latest XDS version (September 26, 2012) * added parameter autoPROC_TestSetFraction (default=0.05) to the scaling module (aP_scale). This will be used when calling uniqueify to generate a fresh test-set. * imginfo tool now also deals with DATE items in ADSC images of format, such as 'DATE=Fri Apr 23 18:12:09 PDT 2010;' * support for POINTLESS 1.6.18 (from CCP4 6.3.0 release) * added plot showing detector position of misfits (according to XDS definition) * improvements in the generation of plots via new parameters XdsStats_sizeSquare, XdsStats_MaxSize, XdsStats_PositionInMm and XdsStats_PlotMissingImages * support for MOSFLM 7.0.9 (in CCP4 6.3.0 release) * extract additional information from POINTLESS step about potential twinning * added parameter ExpectLargeAnomalousSignal (default="no") and XdsSetStrictAbsorptionChiSquareCut (default=1.5): those are used to adjust settings in the CORRECT step - to deal better with datasets showing very large anomalous signals (e.g. with metal clusters at low resolution). * when dealing with datasets that have slight gaps (empty or non-existing images) together with automatic 'chunking': make use of new parameter AutoProcScaleMaxAllowedGapSize (default=0) to avoid weird chunking due to those gaps. * when working with multi-scan datasets that will get merged into separate files (e.g. multi-wavelength MAD): added parameter AutoProcScaleCombineCrystalDatasetInCrystal (default=yes) to work around CCP4 library feature (that will lose cell parameters for a specific dataset, since it assigns cell on a per-crystal basis). This should avoid problems with the iterative scaling (in SCALA), merging statistics and high-res recommendation (in MRFANA) - otherwise the calculation of resolution for a given HKL uses slightly different cells and would therefore be out-of-sync. * more sensible message about images with different scale factors: still based on global statistics, but at least doesn't get confused by missing images or image numbers above 999 * imginfo: - additional checks for odd Pilatus headers (wavelength values in A but unit in nm, distance given in mm without a unit or with 'm' unit) - added checks for weird wavelength value (below 0.5A) and distance value (>100m or <1mm) to automatically adjust these header specification issues - catch Proxima1/Soleil case where we have Kappa and Chi values but no Omega (and Chi is identical to Starting_angle): pretend that this is a Kappa goniostat This is due to Proxima1's capability of doing both Omega- and Phi-scans on a Kappa goniostat when the Pilatus header specification doesn't provide an entry for Omega (so the starting Omega value is stored in the Chi parameter). This might change with updates to software by Dectris and the Proxima1 beamline. - handle negative zero values in angles (reset to 0.0): otherwise the KAPPAROT tool to enforce consistent indexing and/or calculate general rotation axis (eg. phi-scans on multi-axis goniostats) gets confused. * when combining several reflection files (within autoPROC or with the command-line tool 'combine_files'): take value of EnsureConsistentIndexing into account to prevent POINTLESS from wrongly re-indexing the data into an equivalent (but not identical) setting. * new parameter SpacegroupIndistinguishableGroups to allow better message for I23/I213 and I222/I212121 cases * new parameter FindImages_ExcludeLastImage (default=no) to exclude the last image of a found scan: deals e.g. with sync issues at SLS-PXIII regarding shutter-closure * support for: - SLS-PXIII (Prigo) beamline configuration and Pilatus headers through macro: % process -M SlsPXIII ... - Soleil-Proxima1 (Kappa) beamline configuration and Pilatus headers through macro: % process -M SoleilProxima1 ... - NSLS-X25 (initial Pilatus 6M headers) through imginfo-wrapper: % process imginfo=imginfo_NSLS_X25.sh ... - support for Pilatus 6M images from ID29@ESRF: these are marked as 'omega' scans while the current mini-CBF header from Dectris doesn't provide a specific Omega value. By using the EsrfId29 macro an appropriate override jiffy is called to dynamically correct those headers: % process -M EsrfId29 ... * New tool (MTZANA) to provide very similar functionality to CCP4s MTZDUMP program (without some of the limitations in output format, that often make it difficult to parse the information correctly). * make sure some temporary files are removed to keep things tidy. Fixes: * handle multi-scan cases where one (or several) scans had more than 999 images (which could previously give an error in the final, multi-scan scaling step). * explicitly set VRSET in SORTMTZ to avoid odd failures when combining several MTZ files for scaling. However, those failures usually meant very poor or wrong beamstop masking, leading to wrongly integrated intensities behind the shadow. * added checks in MRFANA to avoid memory corruption. * added safety margin for low-resolution limit suggestion from MRFANA: avoids 'losing' low-resolution reflections due to slightly different methods of calculating resolution and/or rounding errors. This should also speed-up the scaling job (aP_scale) since it avoids unnecessary additional cycles of scaling with updated resolution limits. * fixed some wrong amounts of white-space in REMARK 200 generation (remark200.pdb file) * when combining files (using combine_files utility), take care to get correct batch numbering offset - relevant for large scans/datasets with more than 999 images. In the same combine_files utility, some reporting regarding necessary reindexing has been fixed (previously it might have reported reindexing when in fact it didn't do any). * better support for GnuPlot 4.4 (used in creating plots in PNG format) * fixes during indexing when iterative indexing is used (automatically determined or user-requested) and several indexing stages are necessary anyway (e.g. the user only supplied spacegroup but not cell parameters): pick the correct master list of spots for detection of (partially) empty datasets. * better messages when analysing indexed spots for potential high-resolution limit * fixed bug in the aP_scale module (when combining different analysis tools regarding automatic adjustment of high-resolution limits): the explanations why a certain high-resolution limit was picked were sometimes out-of-sync. * added check in iterative indexing to avoid continuing with obviously wrong solution: sometimes IDXREF seems to succeed whereas only very few spots were actually used for the indexing solution. The parameter XdsIndexingStopIfSpotsLessPercent (default=2) is used for that check. =================================================================== Changes relative to 1.0.0 20 Jul 2011 =================================================================== New features: Improvements and changes: * imginfo: - added support for Pilatus headers I03@Diamond - added support for ADSC/Rigaku combo headers - allow APS1 prefix in header items - additional checks when encountering odd/duplicate oscillation axis definitions * added some jiffies: - beam8.sh will report all 8 possible beam centre settings - imgdate.sh can be used to analyse collections of images based on the timestamp in the image header * better method for getting completeness (for overload analysis) from reflection files * added option autoPROC_XdsAcceptOverloads (default=no) to generate a second set of files with overloaded reflections being accepted instead of ignored. This is only meant to be a desperate and last measure! * MRFANA (merging statistics calculation): - better order of inclusion criteria of reflexions (resolution check now before sysabs) resulting in significant speedup. - ice-ring treatment improved through (among several changes): a) using ice-ring limits instead of mid-point (with fixed width) b) using experimentally observed values for ice-ring resolution limits from large number of JCSG datasets This results in much more reliable treatment of ice-rings. Statistics used in applying the various cut-off criteria (for the high-resolution limit) now use an interpolation across those ice-ring resolution shells. * aP_scale (scaling module): - significant speedup through changes in MRFANA (see above) and a faster invocation of MTZDUMP (to read header information from MTZ files) - changed default so that MRFANA will assume by default that the data could contain ice-rings (AutoProcScaleStatsUseMrfanaIgnoreIceRingShells=yes) - new command-line option (-noice) to revert back to previous default, ie. no ice-rings present in data * find_images should give nicer identifier values (by removing trailing '0' characters in obvious cases). * changed from old XDS keyword 'MINUTE' to new 'SECONDS' * unless the user saves standard output already, write a (uniquely named) log-file to ensure that standard output is not lost. * new option to distribute background images over larger range of images (in chunks): DistributeBackgroundImages (default="no"). This should give a better initial background (BKGINIT.cbf file from the XDS INIT step), while avoiding using a very large number of images (which would be slower). * added analysis for empty images: we now require at least 2 (XdsSpotSearchMinNumSpotsPerImage) spots per image but allow 5 (XdsSpotSearchNumAllowedEmptyImages) empty images within the range of images used for spot searching. * lowered starting value for automatic backstop masking in XDS (DEFPIX step) to 5000 (XdsOptimizeDefpixStart), to deal with very weak images. * added option (XdsSetHighResFromIndexedSpots=no) to set high-resolution limit based on indexed spots. Fixes: * add protection against dash failing on 'empty' .autoPROC files * avoid setting reslow/reshigh outside normal command-line system * MRFANA: - corrected bug in accessing cached values of completeness (per resolution bin), therefore avoiding duplication of calls - resulting in further speedup. * fixed wrong reporting of reindexing requirements (due to changes in POINTLESS log-file format) * ensure that 'process' exits with status 0 if all went well * fixed problem in passing empty strings (of non-zero length) into parameters, eg. refine autoPROC_XdsKeyword_REFINE_INTEGRATE=" " ... will now work. * when analysing the low- and high-resolution limit of indexed spots: ensure that the correct set of parameters is used when the iterative indexing was used (because of multiple lattices, ice-rings etc). * deal with scans of more than 999 images (when creating plots of scale, divergence and mosaicity) #### NOTES The 32-bit Linux distributions of Global Phasing's software will work on systems running relatively old kernels (versions 2.4.x). Most mainstream server and desktop Linux distributions that are currently under active maintenance are based on the newer 2.6.x kernel series. The current 32-bit release will be the last to support Linux systems based on the 2.4.x kernel. Future distributions (including patches and minor updates) of our software for all Linux systems released after this one can only be relied on to work on systems running kernels from the 2.6.x series. This is only a significant change for our 32-bit distribution: our 64-bit distributions for Linux have always required kernel version 2.6.x To check the version of the kernel on a particular system, run the following command: uname -r =================================================================== Any feedback and remarks should be send to the autoPROC developers: proc-develop@globalphasing.com ===================================================================