################################################################### # # This is the Jul 10 2024 release of the autoPROC toolbox for academic # users. # ################################################################### # # Copyright (C) 2005-2024 by Global Phasing Limited # # All rights reserved. # # This software is proprietary to and embodies the # confidential technology of Global Phasing Limited # (GPhL). Possession, use, duplication or dissemination of # the software is authorised only pursuant to a valid # written licence from GPhL. # # Authors: G. Bricogne, C. Flensburg, R. Fogh, P. Keller, W. Paciorek, # A. Sharff, O. Smart, I. Tickle, C. Vonrhein, T. Womack # # Reference: C. Vonrhein, C. Flensburg, P. Keller, A. Sharff, # O. Smart, W. Paciorek, T. Womack & G. Bricogne # (2011). Data processing and analysis with the autoPROC # toolbox. Acta Crystallogr D 67, 293-302. # ################################################################### # # Any feedback and remarks should be send to the autoPROC developers: # # proc-develop@globalphasing.com # ################################################################### =================================================================== Changes since the last autoPROC release Jul 10 2024 =================================================================== * New features, fixes and improvements: * Added tool (aP_fit_wvl_to_spots) to analyse processed datasets for a potential error in wavelength - by maximising the number of unindexed spots found in known ice-ring ranges of d*. Although an error in wavelength will have no impact on the actual reflection integration, it will (1) cause problems in automatic detection (and exclusion) of ice-rings during processing, and (2) introduce errors in unit-cell axis lengths and hence create strain in the atomic model during refinement (with a possible knock-on effect on refinement performance due to an incorrect automatic weighting between X-Ray and geometry terms). If a large number of ice-ring contaminated but otherwise good-quality datasets are available that have been collected during the same shift on a given beamline, this tool provides a convenient means of checking whether the energy/wavelength recorded in the metadata is physically correct. * Improved the handling of HDF5 datasets (via *_master.h5 files) when encountering different data types in various entries (e.g. double instead of integer) and the understanding of axis ordering in some arrays since NeXus/NXmx specifies this as slow-fast odering, i.e. C style and not Fortran (fast-slow). Thanks to Graeme Winter for highlighting some associated problems with this. The correct reading of meta data from HDF5 datasets has been improved via added checks. HDF5 datasets as produced by modern synchrotron beamlines and delivered to the user (or the software for data-processing), often undergo significant post-processing compared to the initial files written by the detector itself. This is mostly done for very good reasons, but the most popular tool for those transformations (h5py) requires some very careful handling - especially for character data (that can come in a multitude of variants). Our meta-data extracting program "imginfo" needed further developments to deal with the resulting changes compared to the initial formats used by the detector systems (Dectris' FileWriter). * Fixed a problem in pixel mask handling during conversion into (temporary) mini-cbf files by the hdf2mini-cbf program when those masks were stored in a separate file (and pointed to via an external link from the main master.h5 file). This did not impact data integration that uses the normal HDF5 plugin mechanism in XDS. * Removed the warning (from "imginfo") about sensor material being defined as "Si" instead of "Silicon": it seems more common than originally anticipated. * GEMMI has been updated to version v0.6.6-18-gd726fa1 (June 28th 2024). * Updated beamline/detector information for: ESRF BM07-FIP2, Pilatus-60-0106 at Soleil, ESRF ID29 (broken module in 2013) and Eiger-E-32-0117 for I03@DLS. * Updated information for beamline I04-1@DLS to include the correct wavelength for the period 16th Nov 2022 - 3rd July 2024. This is based on available data for that period and using the new aP_fit_wvl_to_spots tool (which reports an energy error of about 50-70 eV, resulting in a unit cell error of about 0.5%). We do not know if that incorrect meta-data is also present in datasets collected immediately before or after that period and advise users to check the autoPROC analysis in summary.html especially for datasets with significant ice-ring contamination (where any meta-data problem regarding energy/wavelength should become pretty obvious). * Fixed a problem in STARANISO that affected the data quality metrics ("Table 1") only, i.e. actual reflection data was unaffected: the cut-off surface written for low-symmetry spacegroups was treating some reciprocal-lattice points along the borders of the hemisphere incorrectly, leading to inaccurate handling of those few reflections when computing merging statistics within MRFANA. * Fixed a problem with GPX2 not displaying resolution values for predictions or showing generally confused prediction ellipsoids when running under a non-English locale (e.g. "de_DE.UTF-8") that interprets a decimal point differently - expecting a comma instead. * Fixed intermittently-occurring problems with DLS datasets containing axis specifications without a null character, described in our Known Issues at https://www.globalphasing.com/autoproc/wiki/index.cgi?IssuesPage20240123#dlsi03 =================================================================== * New features, fixes and improvements (released Jan 23 2024): * Added feature to extend detector module gaps depending on their position relative to the detector origin (e.g. to account for peculiarities at the module/chip edges). The new parameter autoPROC_Img2Xds_GapsAdd_HV (with default = "0,0 0,0 0,0 0,0") describes 4 gap extensions (horizontal-above, horizontal-below, vertical-left, vertical-right) as a pair of integer values: the first one is for gaps below (in pixel coordinates) and the second for those above the detector origin. So setting autoPROC_Img2Xds_GapsAdd_HV="2,0 0,2 1,0 0,1" would widen horizontal gaps above the detector origin (image origin top left with x horizontal, i.e. typical visualisation program view) by * 2 pixels at the top of a horizontal gap if that gap is above the detector origin; * 2 pixels at the bottom of a horizontal gap if that gap is below the detector origin; * 1 pixel to the left of a vertical gap if that gap is left of the detector origin; * 1 pixel to the right of a vertical gap if that gap is right of the detector origin; * Added mechanism to use (refined) parameters from first block of images (as defined by the XDS DELPHI parameter, which usually defaults to ca 5 degree of data) instead of the global, post-refined values from XDS/CORRECT. This can be useful, or even necessary, when there are significant crystal "slippages" or other changes at the early images of an experiment. Sets of parameters can be defined similarly to the REFINE(*) syntax of XDS, i.e. the parameter XdsUpdateFromFirstBlock can be set to one or several of "AXIS", "BEAM", "CELL", "ORIENTATION" or "POSITION". E.g. using process XdsUpdateFromFirstBlock="CELL ORIENTATION" would take cell dimensions and orientation matrix from the first block integrated instead of the overall value. In that case, also setting autoPROC_XdsKeyword_REFINEINTEGRATE="ORIENTATION CELL" would make sense in order to actually let those parameters refine during integration, especially when the instrument parameters are very accurately known and a crystal "slippage" is expected. This behaviour can also be triggered via the "-M SlippingCrystal" macro. * Added support for correct computation of absorption behaviour for CdTe detectors in XDS: a standalone program (calcmu) will compute values required for the XDS SILICON= keyword in case the sensor material is specified as "CdTe" in the image meta data. The imginfo program in autoPROC will extract the relevant sensor material information for mini-cbf or HDF5 datasets (so please make sure they are set correctly) and adapt not only the sensor material absorption parameter for XDS, but also adapt the known gaps for specific detectors - since these can differ between a Si and a CdTe variant. The use of CdTe specific behaviour can be enforced via the new parameter autoPROC_SensorMaterial (e.g. setting this to "CdTe"). * To help analysing datasets with incorrect image width recorded in the metadata, a new tool "aP_check_image_width" is provided that will scan a sensible range of possible values around the initial (nominal) image width in order to reveal any existing mismatch between that intended image width and what actually then happened during data collection. * Added option (autoPROC_Setup_IgnoreCaseForDetectorId with default value "yes") to always set detector identifiers to upper-case before the lookup into $autoPROC_home/lib/detector-site.def happens. This is to deal with variants like "E-32-0126" (Jul 2021) changing to "e-32-0126" (Nov 2022). >>>> =========================================================== >>>> Note to beamline staff: >>>> ----------------------- >>>> Please remember to notify us of any changes of that kind >>>> to detector meta-data (and also of any exchange of modules >>>> so we can update the database of known "damaged" pixels)! >>>> =========================================================== * Improved analysis for hot/damaged or flickering pixels in the initial spot list: a script calling our aP_detect_damaged_pixels tool can now be automatically generated to simplify the testing for those problem pixels (users need to explicitly set the parameter autoPROC_CleanupSpots_WriteJiffy=yes). This could be of interest especially for beamline staff wanting to monitor their detector behaviour over time. The aP_detect_damaged_pixels tool will now ensure that any exclusions (given via the -x flag) are active right from the start when doing the initial statistics. This should e.g. take care of situations where module gaps are not masked as such (by a negative number) in the input mini-cbf files to be analysed. * Centralised handling of known ice-ring resolution ranges to simplify maintenance and user interaction. The most important parameter settings for users now are XdsExcludeIceRingsAutomatically (default="yes") to switch ice-ring detection on/off, and autoPROC_IceRings_Sigma (default=3) to control the width of known ice rings. * STARANISO now handles systematically absent, unmeasured reflections (within the local averaging sphere) better by excluding them from the computation of the average. * Added support for HDF5 datasets containing the detector distance in /entry/instrument/detector/distance (e.g. I03@DLS from Oct 2023) instead of /entry/instrument/detector/detector_distance or /entry/instrument/detector_distance. Thanks to Graeme Winter for pointing this out. * Update to support XDS (Jun 30, 2023) and CCP4 8.0.15. * Updated information about detectors ADSC-402, ADSC-447, Pilatus-60-0135 (ALS 5.0.1), ADSC-423 (ALS 5.0.3), ADSC-905 (ALS 8.2.2), ADSC-907 (ALS 8.3.1), ADSC-415 (APS 14-BM-C) and Pilatus-60-0101 (SSRL 12-2), Eiger 9M (E-18-0127) at NSLS-II 19-ID (NYX). Added information about Pilatus detectors at beamlines BL18U1 and BL19U1 at the Shanghai Synchrotron Radiation Facility (SSRF) using publicly available diffraction images from 2021 and 2022 (including definition of an exclusion region for a damaged chip at BL19U1). The tool handling those detector/beamline-specific settings can now also list the full content of that database: running "x_setup -list" from the command-line. * Fixed a misunderstanding related to how XDS interprets the EXCLUDE_RESOLUTION_RANGE= keyword at the indexing (IDXREF) step: this led to computing percentages of indexed spots during iterative indexing relative to the non-ice ring spots (when they are still used in the final assignment of Miller indices). * Fixed a logic flaw in the analysis of unindexed spots within ice rings: in order to be faster, the loop over known ice-ring resolution ranges exited too early - resulting in an underestimation of the number of affected spots. * Fixed the generation of diffraction images for very short wavelengths (below 0.42 A) where an incorrect check on sensible limits was triggering an error. * Fix to the generation of diffraction images with shadowed reflections highlighted (if there is a shadowing model and shadowing occurs). * Fix to the feedback of updated beam divergence and/or crystal mosaicity parameters into subsequent integration steps when only one value is given, e.g. only crystal mosaicity (REFLECTING_RANGE_E.S.D.) via a parameter setting like autoPROC_XdsKeyword_REFLECTING_RANGE_ESD="0.08". * Fix to the generation of plots as a function of image number during analysis and automatic detection of poor image ranges (when smoothing created some artificial spikes" at the beginning/end of a sweep). * Various fixes to ensure that shadowing analysis is using the correct information and creates output/pictures only when sensible. * Fixes to the handling of runs where a full set of detector segment descriptions is given (e.g. when running as a result of a Global Phasing Workflow experiment): that information is now also taken into account when analysing the initial spot list for decisions about potentially present ice-rings even before indexing has been run. =================================================================== * New features, fixes and improvements (released Jul 26 2023): * The number of images present in any given set of consecutive images (or "scan") will now be printed into the same datablock that describes the settings of the first image (i.e. distance, collection date, angles etc). This will be present on standard output and in summary.html - and can be switched off via autoPROC_ReportNumImagesInSweep=no setting. Thanks to Steven Sheriff for for suggesting and discussing this feature. * We are now providing more options and control to aP_select_pdb: the use of amplitudes (instead of normalised structure factors, i.e. E-values) can be requested via the autoPROC_SelectPdb_UseEvalues parameter (default="yes") being set to "no". If the set of datasets (that a new MTZ file should be tested against) comes from quite similar processing and refinement procedures, setting autoPROC_SelectPdb_CellDiff_UseCorrespondingMtz to "yes" (to also use amplitudes from MTZ file instead of computing them afresh from a PDB file) might be a good idea. Thanks to Dirk Reinert for reporting and discussions. * In order to provide information about settings supplied by non-standard macros (given to "process" via the -M argument), their content can now be reported in stdout as well as in summary.html. Please set autoPROC_ReportNonStandardMacros="yes" to activate this feature. Thanks to Kate Smith and others for suggesting this. * The system to detect the (usually) faster runtime of OS-provided Gnuplot binaries has been extended to look at the full PATH at runtime. Remember that this feature can be switched off using a parameter setting of autoPROC_GnuplotUseOsVersion=no. * Fixed regression where the unscaled/unmerged file from XDS was converted into a temporary (and therefore lost after completion) MTZ file only. =================================================================== * New features, fixes and improvements (released Jun 14 2023): * Added functionality to achieve significant speedup when reading and writing large MTZ files in several programs (MRFADD, MRFANA, MRFEDIT and MTZANA). Furthermore, the computation of cumulative completeness in MRFANA has been improved for faster runtimes These features are automatically enabled but can be switched off with autoPROC_Use_Quick="no" setting. * The conversion of unmerged data from XDS HKL format to multi-record MTZ (in aP_scale and combine_files) now uses the new feature of the Gemmi program ("gemmi xds2mtz") instead of the much slower POINTLESS procedure. * Added a check to MRFANA to catch bogus data values in MTZ files (that can come sometimes from AIMLESS). * Updated support to CCP4 8.0.011. * Updated list of damaged/problem pixels for various beamlines: ALS 8.3.1 (Pilatus 6M), APS 21-ID-D (Eiger 9M), DLS I04 (Eiger 16M), ESRF ID23-2 (Eiger 9M), ESRF MASSIF-3 (Eiger 4M), SLS PX-I (Eiger 16M), SPring-8 BL32XU (Eiger 9M). * Added new detector to the list of known beamline/detector configurations: PILATUS3X 6M on Xaloc@ALBA * Bruker imgCIF/CBF images will now be automatically corrected - since the incorrect metadata (definition for the detector Y-axis) is still present and will possibly never be corrected. See also: https://www.globalphasing.com/autoproc/wiki/index.cgi?IssuesPage20170508#brukercbf This automatic and hard-wired correction can be switched off via autoPROC_Imgcif2Xds_CorrectBruker=no parameter setting. If at some point in the future the metadata will be corrected, the cut-off data for applying this fix can be changed via the autoPROC_Imgcif2Xds_CorrectBrukerMaxDate parameter (current default = 20990101). Unfortunately, we couldn't use the standard mechanism via a detector serial number and a collection date since those images don't contain a serial number (or other meaningful description that would point to a specific instrument). * Added parameter AutoProcCreateScaFiles (default="yes") to control default generation of *.sca files as part of scaling. These files are often used in experimental phasing (e.g. using autoSHARP via "run_autoSHARP.sh"), but can also easily be generated when needed using the CCP4 program "mtz2sca"). * The aP_detect_damaged_pixels tool (to find problem pixels on Eiger/Pilatus detectors) has now a parameter autoPROC_DetectDamagedPixels_MaxNumToAnalyse (default = 100) to avoid presenting the user with a large number of problem pixels that might just be due to a failed chip. Of course, we do assume here that only a maximum of 100 genuine problem pixels are left in the data - which is a reasonable assumption when the detector pixel masks are updated from time to time by beamline staff. * Fix to the generation of PDBx/mmCIF files for multi-wavelength data via aP_deposition_prep: different handling of the precision with which wavelength values are given potentially caused problems at this step. * Fixed aP_suggest_xds_parameters to actually print suggested command-line arguments for "process" command. * Fix to the way indexed spots (at the XDS/IDXREF stage) are counted: instead of extracting that number from the IDXREF.LP log file, we now look at the actual list of indexed spots - since these are not affected by resolution limits given to IDXREF for the index solution determination. After all, we want to know how many of the found spots are indexable for a given solution - not how many were used to come to that solution. * Fix to generation of various plots using Gnuplot in order to avoid the "warning: iconv failed to convert degree sign" messages on standard output. =================================================================== * New features, fixes and improvements (released Feb 22 2023): * Fixed a regression due to logic flaw that resulted in not writing ISPyB-compatible XML files, PDF reports or deposition-ready mmCIF files. =================================================================== * New features, fixes and improvements (released Feb 17 2023): * In order to write full-featured (extended) unmerged reflection data in PDBx/mmCIF format (following latest developments of the "Data Processing Subgroup" of the "PDBx/mmCIF Working Group"), we are now keeping track also of the ESDs associated with each refined cell parameter. These will come from the XDS/CORRECT step just before the final integration (XDS/INTEGRATE) when using the default scaling path (AIMLESS-only) and are therefore in sync with the cell parameters reported inside INTEGRATE.{HKL,mtz} and/or combine.mtz for multi-sweep data. These ESDs will now also be used during the generation of the prediction images instead of some default (fixed) values. * The interactive results from STARANISO included in the summary.html whith AutoProcScale_RunStaranisoCreate3D="yes" can now also display a hemisphere of points. Activate this by setting AutoProcScale_RunStaranisoCreate3DHemi="yes". * Updated detector information for ADSC-458 (19-ID@APS) and HDF5/Eiger-E-32-0119 (PX-II@SLS, thanks to Kate Smith for providing example data and discussions). * Updated list of damaged pixels for Eiger 16M (S/N E-32-0108) at APS beamline 24ID-E and for PILATUS 6M-F (S/N 60-0117-F) at Petra-III beamline P13. * Added information about HDF5/Eiger-E-18-0133 detector (ESRF ID23-2). * Added initial support for Rigaku HyPix-Arc 150 datasets in cbf format (requires an updated CAP software version to write additional mini-cbf header items describing the different modules). Thanks to Mathias Meyer for discussions. * Even for a (default) run where automatic rejection of poor image ranges might occur, allow user control if and when such a potentially segmented dataset should be treated as a single run in scaling and when should it be split into separate ones (each with their own error model adjustment and high-resolution cut-off for scaling). This is controlled by the parameter AutoProcScaleMaxAllowedGapSize which defaults to 99999 when automatic rejection is active (by default) or 5 otherwise. Thanks to Christian Becke for discussions. * Added support for CdTe Eiger2 intra-module gap definitions as part of generated XDS.INP files. This should never be necessary (since the pixel mask should already ensure those CdTe variant-specific gaps are marked by a negative pixel value), but for consistency with inter-module gaps we want to include those now as well. * If the scaling of all datasets has been switched off via ScaleAllDatasetsTogether=no, the required commands for (1) combining files and (2) scaling will still be written as scripts combine_files.sh and run_aP_scale.sh. This can be prevented via ScaleAllDatasetsTogether=NO setting. * Changed the default (max) number of bins when using equal-number binning in data-quality metric computations after STARANISO analysis from 20 to 100. This should give more meaningful statistics for very large and high-resolution datasets. Thanks to Ashwin Chari for discussions. * Fixed problem in aP_scale when trying to create plots of statistics against image number if only a single image ("Batch") was present - e.g. after converting deposited unmerged data from PDB entries. * Fixed misleading message (on Linux) about XDS binary format detection when the "xds_par" command picks up a symbolic link instead of the actual executable file. Thanks to Markus Rudolph for drawing our attention to this. * When a poorly aligned beamstop results in extreme low-resolution spots being present in the overall list of unindexed spots, a format overflow occured (resolution values above 99999 A) - resulting in aP_anaspot jobs hanging. A fix has been added - even if the better solution is to ensure correct beamstop alignment at the beamline itself. =================================================================== * New features, fixes and improvements (released Nov 21 2022): * Added initial support for different spacegroup settings P 2 21 21 or P 21 2 21 instead of P 21 21 2 P 21 2 2 or P 2 21 2 instead of P 2 2 21 I 2 instead of C2 in order to use a unique, XDS-supported space group number during indexing and integration. Before scaling (with AIMLESS) the original setting will be restored and used from then onwards. Please note that this has so far only been tested for cases where a reference MTZ file is given to autoPROC/process and the AIMLESS scaling path is used. * Switched to new method of detecting significant ice-ring contamination (and subsequent exclusion of all known ice-ring resolution ranges). The previous (now deprecated) method can be re-activated via the autoPROC_AnalyseIceRingsMethod=orig parameter setting. The whole feature (of automatic ice-ring detection and exclusion) can be switched off completely via: XdsExcludeIceRingsAutomatically=no On the other hand, exclusion of all ice-ring ranges can be triggered via: XdsExcludeIceRingsAutomatically=all We now also use a new list of ice-ring resolution ranges after re-analysing actually observed ice-ring data. This data comes in different widths (for each ice-ring) that can be triggered via the parameters autoPROC_{Anaspot,AnaCorrect,AnaMrfana,XdsSpots2res}_IceRings_202211_Class (values between 1 and 10 can be used to select from tight to wider resolution ranges - with a default of= 2, i.e. a farily tight analysis). Please note that this whole analysis depends on correct meta data (especially regarding wavelength/energy) to compute accurate resolution values for given spots as well as well-aligned experimental setup. It might also be confused by additional/other powder rings e.g. from soaked compounds or such. The actual effect of ice contamination on the final data quality depends - among others - on the nature (diffuse rings versus distinct spots), the effectiveness of outlier rejection during scaling/merging (multiplicity helps a lot) and the relative strength of ice versus actual crystal diffraction. If you feel that things are not working as expected (finding too many or not enough ice-rings), we'd be very happy to hear about those cases! * Change of defaults: * The AIMLESS-only scaling mode (previously triggered by "-M ScalingA3") is now the default. We expect this to give more sensible scales for weak data and significantly fewer outlier removals. * The isotropic high-resolution cutoff (as used during scale-determination) is now based mainly on the outer-shell CC1/2 value (previously triggered by "-M HighResCutOnCChalf") * The previous behaviour can be invoked via the "-M Version20220608" macro. * Updated information about beamlines and detectors: * updated list of damaged pixels for: Pilatus 6M at ALS 8.3.1 Eiger 16M at DLS I03 Eiger2 CdTe at Petra-III P14 Pilatus 6M at ALS 5.0.1 Pilatus 6M at ALS 5.0.2 Eiger 16M at APS 24-ID-C Pilatus 6M at APS 24-ID-C Pilatus 2M at BESSY 14.2 Pilatus 6M at CLSI 08ID-1 Eiger 16M at DLS I04 Pilatus 2M at ESRF ID23-2 Pilatus 6M at ESRF ID29 Pilatus 2M at LNLS MX2 Pilatus 6M at Soleil Proxima1 Pilatus 6M at SSRL BL9-2 Eiger 16M at SSRL 12-1 Pilatus 6M at SSRL 12-2 * use of pixel-mask to handle overload values correctly (when using compressed mini-cbf files) for: Eiger2 CdTe at Petra-III P14 Eiger 9M at APS 21-ID-D Eiger 16M at APS 22-ID Eiger 16M at APS 23-ID-B Eiger 16M at APS 24-ID-C Eiger 16M at APS 24-ID-E Eiger 16M at SLS PX-I Eiger 16M at SSRL 12-1 * added: Eiger2 16M at SSRL 12-1 Pilatus 6M at PAL/PLS 11C Pilatus 6M at PhotonFactory BL-5A Eiger 9M at SPring-8 BL32XU Eiger 4M at ESRF MASSIF-3 Eiger 16M at SLS PX-II Eiger 16M at SPring-8 BL44XU * Added values for high-resolution ice-rings based on ice data collected May 2022 P14@PETRA-III (Gleb Bourenkov). This adds 42 new values between 1.081 and 0.607 A. * Improved presentation of STARANISO results in autoPROC's summary.html file (pointed out by Steven Sheriff). * When using our distributed versions of the xds-zcbf plugin, the autoPROC_XdsPluginXdsZcbf_OverloadMask parameter can be used to support dynamic overload handling within that plugin. The parameter should then point to a pixel mask file (in mini-cbf format - optionally gzipped). This feature can be switched off via autoPROC_XdsPluginXdsZcbf_OverloadHandling="no" and the masks itself could be created via our cbf2stat program. We provide such a pixel mask e.g. for beamline P14@Petra-III (as of 20220720) which will be automatically activated for data from that detector (Dectris EIGER2 CdTe 16M, E-32-0129). To switch this off, set autoPROC_XdsPluginXdsZcbf_OverloadHandling=no. * Added fix-up function for images with ridiculous rotation angle settings outside the 0-360 degree range (or at least reasonably close to it). A too large value will create havoc in XDS by overflowing various format restrictions (e.g. in XPARM.XDS where it can only store values up to 999999999.9999 and autoPROC expects values with a maximum of 99999999.9999). * Updated support for CCP4 8.0 (currently 8.0.006) with Pointless 1.12.14 and Aimless 0.7.9. * Improved support for CdTe Eiger detectors: the relevant information should now be extracted correctly by imginfo for mini-cbf files (e.g. P14@Petra-III). * Improved handling of mini-cbf files where the detector information (leading up to the serial number) contains a string "EIGER" instead of the more common "Eiger". * Added support for Eiger2 detectors into aP_average_DXDY. * Added support for compressed *.cbf.gz images into aP_detect_damaged_pixels tool. * Added support for HDF5 datasets that contain the detector serial number as a variable-length string datatype (instead of a fixed-length string): this should fix "imginfo" writing odd characters as detector ID information (e.g. I03@DLS). * Added fallback when trying to determine the binary version of an executable (e.g. XDS) - which is eg. required to select the correct plugin object (ARM/M1 versus Intel on MacOS for example). * Improved handling of iterations within the aP_scale_multi_xtal_iterate tool (mainly to avoid unnecessary duplication of runs without any actual rejections) * Improved handling of HDF5 (Eiger) datasets with rather small number of images inside each individual *_data_N.h5 files: these might have previously been wrongly recognised as single-image files (since they can be rather small and a full dataset might have e.g. 72 of those). The parameter FindImages_DisallowedExtns is now set to "Z,bz2,jpg,pgm,h5" to avoid this. =================================================================== * New features, fixes and improvements (released Jun 03 2022): * Added support for mini-cbf images following the v2.0 specification that introduced additional parameters to describe the full instrument (detector, goniometer and incident beam). This is very much work-in-progress - thanks to Mathias Meyer (Rigaku) and Marcus Mueller (Dectris). The analysis of those extra keywords in a mini-cbf v2.0 header is done using a helper program (mini-cbf2xds) that can be somewhat controlled via parameters autoPROC_MiniCbf2Xds_IncidentBeamFactor (default = "1") autoPROC_MiniCbf2Xds_SwapDetXY (default = "no" autoPROC_MiniCbf2Xds_DetXFactor (default = "1" autoPROC_MiniCbf2Xds_DetYFactor (default = "1" autoPROC_MiniCbf2Xds_DetXFactors (default = "1 1 1" autoPROC_MiniCbf2Xds_DetYFactors (default = "1 1 1" autoPROC_MiniCbf2Xds_RotationAxisFactor (default = "1" to accommodate setups where the hardware doesn't follow the recorded instrument description (or vice-versa). * Added improved support for compressed mini-cbf files (*.cbf.gz) based on the xds-zcbf plugin (M. Nikolova, EMBL, https://git.embl.de/nikolova/xds-zcbf). The automatic use of this plugin can be switched off again using the FindImages_AllowCompressedImages="no" setting. We now distribute plugin versions for Linux/Intel as well as MacOS/Intel and MacOS/ARM and autoPROC should automatically detect the binary version of the XDS executable being used to pick the correct plugin accordingly. If you prefer using your own plugins, set the autoPROC_XdsKeyword_LIB parameter to the full path of your own (shared object) XDS plugin. * New tool (aP_suggest_xds_parameters) to help making sensible decisions regarding parallel processing with XDS - depending on number of images, image width and number of threads available. See output of "aP_suggest_xds_parameters -h" for more details. * The aP_anaspot tool now provides a recommendation regarding ice-ring exclusion during processing (if it detects strong indications of ice-rings). * Added new parameter autoPROC_XdsDistanceShift to adjust an incorrect distance parameter from dataset meta-data. Together with the already existing autoPROC_XdsDistanceFac value, any distance can be corrected via new = autoPROC_XdsDistanceFac * old + autoPROC_XdsDistanceShift * Improvements in handling database of known (dated) beamline and detector set-ups: * honor additional definitions for damaged pixels, problematic modules or manual beamstop masking when combining user-provided information and database information. * added support for referencing macro commands within the database file * The logic regarding a change of the fitness rejection limit at each scaling cycle has been stabilised within the aP_scale_multi_xtal_iterate tool. * Simplified the method for picking other than the default indexing solution: with XdsOptimizeIdxrefPickSolution=2nd (or "3rd", "4th" etc) less prominent solutions can be selected. The default has not changed: use the solution with the largest number of indexed spots ("MostSpots"). * Added support for latest XDS (January 10, 2022 version, BUILT=20220220), CCP4 7.1.018 and CCP4 8.0.001. * Added support for handling datasets where the individual image files have less than 3 numerical digits in the file name. This can be activated by setting the new parameter FindImages_MinImageNumWidth e.g. to 2 (to accommodate e.g. "test_01.cbf" file naming). The default is still to expect at least 3 numerals in the filename. * Changed minimum image file size in find_images (which is used unless an explicit list of images is given via the "process -Id" flag) from previously 512k to 256k. The previous behaviour can be reinstated via the FindImages_min_size=512k parameter. * Added code to detect (and test) OS-provided Gnuplot binaries (/usr/bin/gnuplot, version 5.X and supporting pngcairo terminal). This will then be used in preference of our own, distributed gnuplot binary - to avoid slow-down as experienced on some Linux distributions (like Ubuntu 22.04). This behaviour can be switched off via the parameter setting autoPROC_GnuplotUseOsVersion=no. * Updated database of known beamline/detectors combos for BESSY (http://www.helmholtz-berlin.de/bessy-mx) via data kindly provided by Thomas Hauss. Also added support for Petra3/P14 EIGER2 CdTe 16M, E-32-0129. * Based on the parameter autoPROC_XDS_MaxProcessors (default = 99) we now check that the setting of autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_PROCESSORS follows and accommodates the XDS limit of that parameter. * Switched to using HDF5 library version 1.10.8 (required for support of MacOS 11/arm64) for compilation of various internal tools. * Fix to imginfo wrapper for older Soleil/Proxima-1 (Pilatus 6M) data so that compressed images are also handled. Thanks to Marco Bellinzoni for reporting that issue. * Fix to "x_kappa -list" command (to show all available and defined goniostat settings). * Small fix to the extraction of per-run information (to standard output) when dealing with multi-sweep datasets. * Fix to handling processing with explicit segment definitions in initial spot analysis. * Fix to avoid warning message about ("awk: cmd. line:1: warning: regexp escape sequence `\!' is not a known regexp operator") triggered by newer versions of GNU awk (5.+). =================================================================== * New features, fixes and improvements (released Oct 20 2021): * Added improvements related to the initial indexing when no SG/cell (e.g. via a reference MTZ file) is given, introducing a series of fall-back options to help with really poor diffraction patterns. This is done mainly by removing the weakest spots found in a stepwise manner and is triggered either by an initial XDS/IDXREF failure mode or if that initial indexing uses less than RunIdxrefFailRemoveTailIfInitialPercentBelow=30 percent of initial spots. Large number of data sets collected during the CeBEM/CCP4 2021 workshop at Diamond Light Source provided very good feedback to test this feature - allowing for ca. 10% more datasets going past the initial indexing stage. * Improved the analysis of found/indexed spots (to define ranges of excluded images due to e.g. a miscentred crystal). * Added support to remove potentially spurious (weak) spots, e.g. at the edge of a beamstop shadow. The parameter autoPROC_CleanupSpots_RemoveLowestNum (default=0) can be used to define how many of the weakest spot/peak heights should be removed. Using for example a value of 3 here on an initial spot list where a lot of spots have values of 5, 4, 3 and 2, would remove all spots with values of 4, 3 or 2. * Added support for ReverseRotationAxis=yes setting even for datasets in d*TREK format (where we would expect the header to be fully self-contained and correct). This is required for ALS 4.2.2 CMOS-1 detector data. * Added support for compressed (gz, bz2) image processing for d*TREK (Rigaku) formatted images. * Added MASSIF-1/ESRF settings into kapparot.sites, to process datasets collected on MASSIF-1 that make use of the Kappa goniostat. Thanks to Matthew Bowler for providing a dataset. * Added support for several Rigaku in-house instruments with Eiger detectors writing mini-cbf images: the setting KapparotSite=Rigaku-202109 defines the goniostat setting found on these instruments. Thanks to Jim Thorpe, Tung-Chung Mou, Gabor Bunkoczi & Bob Nolte for reporting, discussions and providing test data. Please note that this is still work-in-progress to improve the provision of accurate instrument information via an enhanced set of meta-data items in the mini-cbf header. There are ongoing discussions with our colleagues at Rigaku (Matthias Meyer, Mark Del Campo) and Dectris (Clemens Schulze-Briese, Marcus Mueller). * Introduced better control of the indexing process when encountering Pilatus or Eiger detectors: two new parameters (autoPROC_Img2Xds_Pilatus_MaxErrorSpotPosition=2.0 and autoPROC_Img2Xds_Eiger_MaxErrorSpotPosition=4.0 can be used to adjust the XDS parameter MAXIMUM_ERROR_OF_SPOT_POSITION according to PAD detector type (i.e. pixel size) and/or source (synchrotron versus in-house). * Added fallback option when scaling fails due to over-optimistic crystal-detector distance (i.e. the integrated intensities passed on to scaling contain an overwhelming amount of noise). In that case, scaling will be attempted again with an initial high-resolution limit imposed. This new feature can be switched off via the autoPROC_ScaleRetryWithHighResLim=no parameter. * Added a new option to determine a hopefully sensible high-resolution limit after the initial integration and space-group determination. This is to avoid failures during integration and/or subsequent scaling when processing datasets collected in fully automatic mode when adequate adjustments of the crystal-detector distance might not be possible. This feature is not yet active by default, but can be triggered via the XdsSetHighResFromSecondCorrect=yes parameter. We would be very interested in any reports about the usefulness of this new option. * Updated detector-specific database information for ADSC-905, ADSC-471 (F1@CHESS), Pilatus-24-0109 and RIGAKU-08300127. * Updated list of defective pixels for Eiger2 XE 16M detector on I03@DLS (as of 20210923). * Updated list of defective pixels for Eiger detector on I04@DLS (as of 20210923). * Updated list of defective pixels for Eiger detector (E-18-0102) on Proxima2@Soleil (as of 20191106). * Updated detector-specific database information for Eiger 16M (S/N 0107), deployed at both P13 and P14 (which have differences in rotation axis orientation) at Petra-III/EMBL. * Added detector-specific database information for Eiger 16M (S/N 0121) at P11 (Petra-III/DESY): current HDF5 files contain inconsistent and incorrect definitions of detector and rotation axis vectors. Ideally, the rotation axis definition should be modified to follow the NXmx "Gold Standard". * Added support for HDF5 files with units given as "pixels" instead of "pixel". * Added support for compressed image files when it is necessary to analyse strings in the image header to determine the image format reliably (e.g. Rigaku IP/CCD detectors). * Added more control over some basic layout of the summary.html file generated. The two new parameters autoPROC_CreateSummaryFile_BodyOverflowX (default = "hidden") and autoPROC_CreateSummaryFile_CssSidebarPosition (default = "fixed") can be modified according to local requirements. * Some changes to the multi-xrystal strategy employed in the scaling tool (aP_multi_xtal_iterate): * the iterative rejection based on fitness parameter (autoPROC_AnaMrfana_FitnessCutoffs) has been changed to "0.25 0.34 0.42 0.5", i.e. one additional initial cycle at very low cut-off value of 0.25. * different scale parametrisations (used within AIMLESS) are used at different stages, starting with "CONSTANT", "ROTATION SPACING 5.0 BFACTOR OFF SECONDARY 0" and finally "ROTATION SPACING 5.0 BFACTOR ON SECONDARY 0" * Added further safety checks to avoid failures when generating report_staraniso.pdf files (triggered by seemingly random and rare failures during generation of the STARANISO 2D pictures). * Fixed handling of C2/I2 datasets within the aP_scale scaling module: the application of the cut-off mask (given in I2 setting) to the reflection data (given in C2) sometimes failed due to an incorrect checking of the mask extent. * Fixed generation of PDBx/mmCIF files when the traditional (isotropic) data for the so-called "early" and "late" analysis was taken even for the anisotropic (STARANISO) analysis: the correct data quality metrics are now included into the deposition-ready mmCIF file (Data_1_autoPROC_STARANISO_all.cif). * Fixed setting of _diffrn_refln.diffrn_id value (within scaled and unmerged data block) in aP_deposition_prep when using multi-sweep datasets. Thanks to Marcin Wojdyr for discussions. * Fixed extraction of detector serial number for mini-cbf images containing "PILATUS 6MF," string. * New features, fixes and improvements (released Jul 16 2021): * Added feature to process datasets in reverse order (from last to first image) via autoPROC_ProcessDataInReverse (default = "no") parameter. This requires the inversion of the rotation axis. - invert the rotation-axis definitions in HDF5 datasets via the parameter autoPROC_Hdf52Xds_RotationAxisFactor="-1" (default = 1) * When the XSCALE scaling path was requested (via "-M ScalingX"), autoPROC now writes a script ("scale_with_aimless.sh") that should provide the correct commands for running the default AIMLESS scaling path in a multi-sweep situation. * Added initial version of simple multi-crystal scaling that includes automatic exclusion of poor image ranges (or sweeps). So far, only scaling of MTZ files with AIMLESS is supported - see output of "aP_scale_multi_xtal_iterate -h". This has been tested on a few examples and is mainly intended for early adopters. Any feedback is highly appreciated. * Added feature in combine_files to help combining sparse datasets (where there might not be data overlap for all pairs of input files) while checking for consistency. If the reference file (first input data file is the default) has no overlap with a subsequent data file, the already processed files are used instead. * Added support for HDF5 datasets from beamline I04-1 (DLS, Eiger 2XE 9M) that created HDF5 *_master.h5 (and *.nxs) files with the rotation axis called "gonomega" instead of "omega". This has been reverted at DLS as of 25th June 2021 (thanks to Graeme Winter for reporting this issue and following it up at DLS). * Added support for 64-bit pixel mask (instead of 16- or 32-bit) in Eiger HDF5 datasets from e.g. beamlines I03 and IO4 (DLS). The reason for this change is not apparent and has been reverted (for I03) as of 25th June 2021 (thanks to Martin Welin for reporting this issue and to Graeme Winter for following this up at DLS). * Added automatic support for Eiger2 XE 16M (E-32-0117) at I03 (DLS) for data collected between 23rd April 2021 and 22nd June 2021: two modules are misbehaving with large numbers of problematic ("flickering") pixels. These are now excluded automatically via a autoPROC_XdsKeyword_UNTRUSTED_RECTANGLE="2079 4149 2749 3263" setting. Please note that data collected after 22nd June 2021 still shows the same problem, but the exclusion is now handled directly by the detector filewriter during data collection. A fix will require an exchange of those modules (Thanks to Graeme Winter for clarification). * Added further support for HDF5 datasets following closer the NXmx ("Gold Standard") model as opposed to the original Dectris/HDF5 style. * Added support for CCP4 7.1.015, including POINTLESS 1.12.10 and AIMLESS 0.7.7. * Added support to aP_select_pdb for specifying a directory (containing potential starting models as *.pdb files). See "aP_select_pdb -h" for details. * Added support for compressed PDB files (gz/bz2) in aP_select_pdb. * Improved generation of summary.tar.gz by (1) increasing the size limit for files to be included (to 50Mb from 20Mb) and (2) using a white-list of important files (deposition-ready mmCIF and result MTZ files). We encourage automatic processing systems as implemented at synchrotrons to provide easy access to this file for users. This way a single file should definitely contain everything relevant to the data processing (and can act e.g. as an input to Pipedream via the -autoprocdir flag). * Improved handling of data at the beginning and end of a sweep when determining potential image exclusion ranges. Because those statistics can be unreliable (due to the intrinsic partiality at those borders), we will also enforce a required minimum size for those exclusions to be activated. * Changed default to produce so-called PQR plots from STARANISO analysis: these will only be created if dealing with a centred space group (because there they might be easier to look at due to interpolation outside independent of the actual reciprocal lattice and the extinction conditions). This can be reverted via the autoPROC_StaranisoAlwaysCreatePqrPlots=yes parameter. * Fixed a browser dependent issue in the WebGL code used for viewing the color-coded reciprocal lattice scenes in 3-D as output from STARANISO. * Added support for mini-cbf datasets where the X- and Y-axis are flipped (e.g. because the detector has been mounted with a 90-degree rotation). This also requires a swapping of the module gap definitions. Thanks to Jim Thorpe for providing an example dataset. * If so-called "early" and "late" datasets were generated as part of the traditional (isotropic) analysis, the resulting amplitudes (F_early/SIGF_early and F_late/SIGF_late) will now also be added to the STARANISO result MTZ file. This will enable use of the anisotropically analysed data in refinement while at the same time provide the data for automatic analysis of F(early)-F(late) maps for detection and analysis of specific radiation damage within BUSTER (www.globalphasing.com/buster/). * Improved detection of intended scaling path within aP_scale to ensure that parameters in -hkl/-mtz flags are in sync and meaningful. * Fixed problem in aP_fixcbf (tool to adjust overloaded pixel handling in e.g. Eiger images written as mini-cbf CBF images) when running in directory different from input image directory. * Fixed problem with mini-cbf files from beamline 14.1 (Bessy, May 2017) that record(ed) Polarization as "None": this led to a segmentation fault in GPX2 and resulted in no prediction images being created or aP_detect_damaged_pixels failing. * Fixed handling of distance parameter refinement (and expected or too large jitter) during plot generation: this was sometimes corrupted when ranges of images were automatically excluded. * Fixed problem in MTZANA when all data items are associated with the dataset 0 (that usually is reserved to the Miller indices H, K and L only). That dataset often doesn't carry any wavelength information and we need to fall back to using the first non-zero dataset instead (if present). It seems that e.g. XDS_ASCII.HKL files converted by POINTLESS to MTZ show this issue. * New features, fixes and improvements (released Apr 20 2021): * Added reporting of a so-called "operational resolution" to facilitate comparison of data processing results including anisotropy analysis by STARANISO (that reports three diffraction limits along the principal axes of the ellipsoid fitted to the cut-off surface of the local distribution). These values are reported on standard output, in the summary.html file and in the individual *.table1 files. They take incompleteness of any kind into account, i.e. not just anisotropy but missing data due to a cusp or to module gaps, to ice-ring contamination, to overloaded pixels etc. To ensure that values are roughly comparable between the different types of analysis and of choice of cut-off criterion (traditional isotropic analysis based on average values of I/sig(I) or of CC1/2 within spherical shells; or anisotropic analysis with STARANISO based on local averages of I/sig(I)) or other criteria), separate values are given for all reflections in the input file and for those with a per-reflection I/sig(I) value above threshold values of 0, 1 and 2. Please note that those individual, per-reflection cut-offs should not be confused with the per-shell or local averages of I/sig(I) for the isotropic and anisotropic cases, respectively, that are used in defining a spherical/isotropic diffraction limit or an anisotropic cut-off surface. These per-reflection I/sig(I) cutoffs are used here to provide a basis for at least an approximate calibration across different types of data analysis. For the moment, our empirical observations lead us to recommend using the Operational Resolution value for the per-reflection threshold of I/sig(I)>=2 for such comparisons, being the most consistent across different types of input files. We would very much like to hear any additional views and observations from users about the usefulness of those various threshold values. The whole reporting can be switched off by setting the autoPROC_ReportOperationalResolution parameter to "no". The range of reported values can be controlled via the autoPROC_ReportOperationalResolution_cutoffs parameter. Remember that the standalone tool "aP_operational_resolution" can be used on any merged MTZ file to compute those values independently from an autoPROC processing run. We wish to thank Dirk Reinert and Steven Sheriff for feedback and discussions on these matters. * Added support in "imginfo" for CBF files (with mini-cbf header) from Rigaku/CrysalisPro: these use a different timestamp format to other mini-cbf formats. * Added support for detector serial numbers in mini-cbf files created via an Eiger ZeroMQ stream interface conversion. * To avoid ownership/permission issues outside of our control, the writing of so-called "harvesting files" by the CCP4 TRUNCATE program has been disabled. If you need this feature after all, please set the TruncateHarvest parameter to a single space (" "). However, all relevant information should already be provided in the PDBx/mmCIF files generated by autoPROC anyway. Thanks to Lewis Muir for feedback and discussions. * Improved handling of HDF5 datasets: extract detector serial number if provided now as well. * Improved handling of HDF5 datasets through the XDS plugins (Dectris/neggia and Diamond/Durin): overloaded pixels (i.e. those above the reported countrate_correction_count_cutoff value) will be marked as such to XDS by default. The previous behaviour (treating such pixels as if they were inactive) can be re-instated via autoPROC_XdsPluginNeggia_OverloadHandling=no autoPROC_XdsPluginDurin_OverloadHandling=no Please note that the correct (in our opinion) handling of overloaded pixels will not be available under the following conditions: * providing your own plugin shared object (e.g. via autoPROC_XdsKeyword_LIB=/where/ever/some.so) * processing Eiger datasets that were converted into mini-cbf files through a ZeroMQ/stream writer that doesn't handle those marked but unmasked (i.e. dynamic) pixels differently from explicitly masked (i.e. static) ones. * the countrate_correction_count_cutoff (HDF5) or Count_cutoff (mini-cbf) values are incorrect: they are usually set by the detector during "arming", but be careful with pre-generated mini-cbf headers. Our own converter (hdf2mini-cbf) will handle those overloaded pixels in the same way as described above: other converters might behave differently. * Improved support for Mar345 (Mar Image Plate) formats: the timestamp written into the image header is now also extracted by the "imginfo" tool. Thanks to Eugene Osipov for reporting. * Improved command-line checking for dataset specifications (-I, -Id and -h5 arguments). * Improved support for PDF report generation when running aP_scale scaling module standalone. * Improved support for full CBF image data (e.g. I19@DLS): the detector serial number is only present in the (potentially provided) mini-cbf part and needs to be extracted explicitly. * Added support for unusual serial number specification for P11@Petra-III Pilatus datasets (e.g. 6YNQ): these report "PILATUS 6MF-0109" instead of the more common separation of detector specification and serial number (e.g. "PILATUS 6M-F, S/N 60-0105-F", "PILATUS 6M, S/N 60-0116-F", "PILATUS 6MF, S/N 60-0102-F" or "PILATUS 6MF, SN 60-0001-F"). * Added support (for automatic parameter setting) for datasets collected on the following detectors Eiger-E-32-0101 (HDF5, 5RUQ claims to be a DECTRIS PILATUS3 6M on 8.3.1@ALS) Pilatus-E-32-0110 (mini-cbf format from Eiger 16M detector; 5RUR claims to be a DECTRIS PILATUS3 6M on 8.3.1@ALS) Pilatus-E-18-0110 (mini-cbf format from Eiger 9M detector; APS 17-ID, IMCA-CAT) Pilatus-E-18-0101 (mini-cbf format from Eiger 9M detector; APS 21-ID-D, LS-CAT) Pilatus-E-32-0115 (mini-cbf format from Eiger 16M detector; APS 22-ID, SER-CAT) Pilatus-E-32-0104 (mini-cbf format from Eiger 16M detector; APS 23-ID-B, GM/CA) Pilatus-E-32-0108 (mini-cbf format from Eiger 16M detector; APS 24-ID-E, NE-CAT) Pilatus-60-0126 (Pilatus 6M, DLS I03) Pilatus-60-0100 (Pilatus 6M, DLS I04) Pilatus-60-0116-F (Pilatus 6M, ESRF ID23-1) Pilatus-24-0118 (Pilatus 2M, ESRF ID23-2) Pilatus-24-0117 (Pilatus 2M, ESRF MASSIF-1; previously reported as "Pilatus-24-0118") Pilatus-24-0109 (Pilatus 2M, LNLS MX2) Pilatus-E-18-0104 (mini-cbf format from Eiger 9M detector; NSLS-II, 17-ID-1, AMX) HDF5/Eiger-E-32-0113 (HDF5, Eiger 16M detector; Photon Factory, BL-17A) Pilatus-E-32-0100 (mini-cbf format from Eiger 16M detector; SLS, X06SA, PX-I) * Added support to distinguish datasets where the detector serial number is non-unique (e.g. "PILATUS 6M, S/N 60-0104" on ESRF ID29 and ID30B, "PILATUS3 2M, S/N 24-0118" on ESRF ID23 and ID30, "PILATUS3 6M, S/N 60-0128" on ESRF ID29 and ID30). Those duplications are then also checked via complete string matching to provide adequate lookup within the detector/beamline configuraiton database. Thanks to Andrea Pica (ALPX) and Matthew Bowler for discussions! * Added functionality to test detector- and/or dataset-specific settings by running "x_setup" directly on a given image (or HDF5 master) file. The information given on standard output can help users that are updating their own local copy of the detector/setup information, e.g. synchrotron or beamline staff. See also $autoPROC_home/autoPROC/lib/detector-site.def (pointed to by the autoPROC_DetectorSiteDef_file parameter). * Added support for XDS version "Feb 5, 2021 (BUILT=20210323)" and CCP4 7.1.013. * Fixed specification to switch new auto-pruning feature off (via autoPROC_AnalyseForPoorImageRanges=no parameter). Thanks to Ernst terHaar for reporting. * Fixed regression in handling C2 spacegroups in STARANISO analysis: when the alternative I2 settings would provide a less oblique unit cell (beta angle closer to 90 degrees), the application of the cut-off surface (mask) to the full (scaled, but unmerged) data was not done correctly. This led to incorrect data quality metrics ("Table 1") for the anisotropic (STARANISO) analysis of those datasets. * Fixed PDBx/mmCIF item names for _reflns.pdbx_signal_type and _reflns.pdbx_observed_signal_threshold: these were still written with the "gphl_" (instead of "pdbx_") prefix. This should have no impact on the ability to deposit these mmCIF files - since the current dictionary contains appropriate aliases anyway. * Fixed problem when combining user-specified and automatically determined, instrument-specific settings: a user-defined item (e.g. "BeamCentreFrom=header:-y,x") would be ignored if the new instrument-specific setup routine was setting the same parameter (and no other) as well. If several parameters are automatically set by autoPROC no problem occurs. This is usually only a problem with historical datasets where (1) a particular detector has been moved between multiple beamlines with very different settings and (2) the distributed database of instrument settings is not up-to-date. Users can always disable the whole automatic feature via the do_setup=no parameter. * New features, fixes and improvements (released Feb 24 2021): * As part of our activities within the wwPDB's PDBx/mmCIF Deposition SubGroup on Data Collection/Processing, we have defined new dictionary items (over 50 in total), complete with detailed descriptions, that have now been incorporated into the official PDB Exchange (PDBx/mmCIF) Dictionary that supports deposition to the PDB - see : https://mmcif.wwpdb.org/dictionaries/mmcif_pdbx_v50.dic/Index/ Their purpose is to facilitate the deposition and archiving of certain types of diffraction data, as well as accommodate new quality metrics pertaining to these data. These extensions cover three areas: 1. merged reflection data that have been processed to take account of diffraction anisotropy, with descriptors for that anisotropy, for the cut-off criterion applied to the data, and for the ellipsoid fitted to the cut-off surface; 2. unmerged reflection data, providing extra item definitions aimed at improving support for the deposition of such data and at ensuring that they can be meaningfully reprocessed after retrieval from the archive; 3. anomalous diffraction data and associated statistics. All along this work, we have made sure that deposition-ready mmCIF files produced by our software (autoPROC and BUSTER) take advantage of these new definitions and validate against the updated mmCIF dictionary. As a consequence, users are very much encouraged to upload these files directly to the PDB for deposition, and thus avoid the MTZ + PDBEXTRACT route. We thank the PDBx/mmCIF Deposition SubGroup on Data Collection/Processing, and especially Ezra Peisach and John Westbrook, for their energetic and constructive engagement with this effort. * A new option has been introduced to automatically detect very poor image ranges and exclude them from processing. It is based on a careful analysis of indexed spots (taking multiple lattices into account) and a new metric to assess the importance of each image in terms of uniqueness, redundancy and contribution to the final merged data. This is still very much work-in-progress. This new option is active by default, but can be de-activated by the following parameter setting: autoPROC_AnalyseForPoorImageRanges=no. It has been tested and shown to work as expected on datasets that cover a large rotation range and have serious crystal-centering issues or exhibit extreme radiation damage. Any feedback from testing on other datasets would be very welcome! * By default, autoPROC will try and extract as much site-specific processing options via the dataset metadata (from image headers or HDF5 container). This should hopefully limit the amount of extra arguments and settings a user would be required to provide. If this feature misbehaves or is not wanted, setting do_setup=no (e.g. on the command-line) will switch it off again. The database storing the site/beamline-specific information is located at $autoPROC_home/autoPROC/lib/detector-site.def (where $autoPROC_home is the installation directory) and pointed to via the autoPROC_DetectorSiteDef_file parameter. There is additional information provided at the top of that file. Any feedback about instruments not (yet) covered or any misbehaviour would be much appreciated. * Thank you very much to the MX team at Diamond Light Source, D. Musil (Merck-Serono), J. Marquez (EMBL-HTX, Grenoble), ALPX (Grenoble), D. Reinert (Boehringer-Ingelheim), the SBGrid and proteindiffraction.org data repositories and all others for making large amounts of example data available for testing those new developments. * New tool (aP_anaspot) to enhance the analysis of initial (and indexed) spots - including a new automatic selection of image ranges without likely diffraction spots (e.g. due to a miscentred crystal). An improved ice-ring analysis feature is also provided (which will form the basis of better ice-ring exclusion in a future release). * When collecting more than 360 degree of data, the plots from iterative indexing (which by default will only use the first 360 degree of data) will now cover the full data range to highlight this difference. * Improved the diagnostics regarding fraction of indexed spots to better accommodate datasets with ice-rings and/or damaged pixels (that are excluded before the indexing stage itself). Additional checks are introduced to compute the right percentages (avoiding incorrectly reported values above 100%). * The safety net to stop processing if less than 2% of spots are indexed now takes ice-rings into account (since they can contribute a large number of spots to the overall count). * A new function will re-instate intensity data in the final output reflection files that TRUNCATE excluded because no amplitudes could be computed. This would only impact truncate-unique.mtz and only datasets where a significant number of very weak reflections (especially at high-resolution) is present. * XDS binaries are now run through a wrapper system to ensure non-ASCII characters coming from the LIB= plugin system (e.g. dectris-neggia.so plugin) are not preventing simple searching in log files. * All HDF5 datasets are now using the Durin plugin by default (https://github.com/DiamondLightSource/durin) within XDS: this avoids some failures during data extraction with the dectris-neggia (https://github.com/dectris/neggia) plugin that seemt o be related to differences in the bitshuffle implementation. To switch back to the previous behaviour, set autoPROC_Img2Xds_UseXdsPlugins_DectrisHdf5 to "dectris-neggia". * By default, the process and OpenMP stacksize is increased within autoPROC to 1 Gb. This can be controlled via the parameters autoPROC_IncreaseStacksize (default="yes"), autoPROC_IncreaseStacksize_MaxSize (default="1048576", i.e. 1 Gb) and autoPROC_IncreaseStacksize_OMP_STACKSIZE (default="1G"). * Fix to MRFANA (for computing data quality metrics during scaling/merging) in the setup of resolution bins when dealing with multiple scans/datasets that show significantly different overall quality. * When creating prediction pictures and a resolution limit was specified by the user, those predictions will now only cover the active resolution range (instead of going all the way to the corners of the detector). * Reduce disk footprint when using filter-file technology to handle dynamic shadows from the goniostat. * Added some initial support for "anonymising" autoPROC reports (summary.html and PDF reports), i.e. removing user and machine information, directory information etc as much as possible. For that you can e.g. change some variable settings to e.g. autoPROC_Summary_Title="something" aP_report_Footer="something" and switch off hostname reporting via ReportHostname="no". Please let us know which other items you would like to have direct control over (instead of editing the HTML and PDF files yourself). * New features, fixes and improvements (released Dec 11 2020): * New tool aP_operational_resolution to compute a so-called "operational resolution": this is the resolution at which a complete set of reflections would give the same number of reflections as were actually observed. It therefore takes spherical incompleteness into account (missing data due to cusps, detector gaps, anisotropy etc). The implementation was inspired by discussions during the "Data Quality Metrics Workshop" (April 2019) jointly organised by Global Phasing and ESRF/Grenoble - with special thanks to M. Weiss and K. Diederichs. Please note that the main application for this quantity is in comparison of different datasets in e.g. a compound screening campaign or when comparing different data processing procedures. * Initial support for Bruker SFRM images (e.g. Photon-II and Phonton-III). These are now supported in "imginfo" and "GPX2" as well as "process" itself. Thanks to V. Calderone and R. Byrne for example data and discussion. We suggest that data processing is done using those original SFRM images rather than those converted to a CBF variant: XDS handles the so-called baseline more accurately with this SFRM format. We would like to thank our colleagues at Bruker (J. Kaercher, V. Smith and M. Mrosek) for very helpful discussions and clarifications. * A new tool aP_auto_process to provide an API for automatic processing of a series of image directories, followed by association with a ligand description, selection of a likely starting model and subsequent Pipedream run. This is very much still work-in-progress and requires additional coding on the user-side. * Enhanced aP_select_pdb to handle different SG settings (like I2 vs C2). * Added support for deposition of unmerged data (within aP_deposition_prep tool): this follows the current PDBx/mmCIF (v50) dictionary, but work is underway - together with the other members of the PDBx/mmCIF working group - to provide files in so-called data-extension format (v5-next). * If generation of pictures for raw diffraction images failed, the number and size of warning/error messages printed on standard output is now reduced. * In anticipation of depositing (or archiving) unmerged+scaled data: separate multi-wavelength data will be combined into a single output file. * Support for CCP4 7.1.007. * Fixed error message about non-existing "register_file" command. * Fixed problem in creation of pictures for diffraction images when /tmp was not writable (e.g. on macOS). * Fixed regression for reporting of direct beam coordinates in ISPyB-compatible XML file (thanks to R. Gildea for reporting): the conversion from pixel coordinates to mm was done incorrectly (multiplying by pixel size twice). If you have any additional information or questions regarding the requirements and/or specifications of this ISPyB-compatible XML file, please let us know! * New features, fixes and improvements (released Sep 18 2020): * Added analysis of "stand-out" pixels, i.e. pixels that are not necessarily very "hot" (constantly having a very large count value) but that are (consistently) significantly higher than the surrounding pixels. These will now also be analysed within aP_detect_damaged_pixels. * Added plots of images with shadowed areas into summary.html - if an adequate shadowing model is available and has been used. This will allow checking of (a) the severity of the shadowing at e.g. high Kappa values and (b) control the accuracy of hte shadowing model itself. * To provide a simple check about the diffraction quality, a picture of the first image is now created (and also a close-up of the central region). This is also included into summary.html. * Support for CCP4 7.1.002 update. * Support for mini-cbf files (generated via the Eiger stream interface and some beamline-spcific tools) marked as following the _array_data.header_convention "Eiger". Obviously, this field should only actually contain a value related to the actual mini-cbf header specification (e.g. "PILATUS_1.4" as documented e.g. at https://www.dectris.com/support/downloads/header-docs/cbf/) and not a poorly defined string. * Two plots analysing R-values (Rmerge, Rpim and Rmeas) and completeness as a function of image number are now always shown in the summary.html. Other plots are still hidden by default (but can be shown via the green [+] button). * Improved the automatic analysis for "early" (least radiation damaged) and "late" (most damaged) data subsets: a better balance between completeness and the cost (in terms of narrowing the time/dose gap) should be made now. * Added plot of rejected "misfits" (as determined by XDS/CORRECT) as a function of image number. * Added support to create prediction images for HDF5 datasets: the parameter autoPROC_CreateGpxPicturesForHdf5 (default=no) can be set to activate this feature, which will extract images from the HDF5 container on-the-fly during the plot generation. * New tools (aP_gnuplot_multiplot aP_ana_aimless_scale) to help analysis of poor data ranges for automatic exclusion and decision making. * New standalone tool (aP_edit_mtz_history) to edit (add/replace/delete) MTZ history information. This can be used to set a specific information string (compound ID, protein name, software version etc) in an existing MTZ file. * Improved plots of number of (indexed, ice-ring) spots versus image number (run_idxref_spot_hkl_hist_smooth.png) and some fixes to the plot showing fraction of (indexed, ice-ring) spots versus image number (run_idxref_spot_hkl_histfrc_smooth.png). * Some initial twinning statistics - at the stage of anisotropy analysis via STARANISO - is now also presented (within the "Anisotropy analysis" part of summary.html for the moment until a more complete repesentation of the underlying data has been included into a separate analysis section). * An inconsistency regarding the data quality statistics (Table 1) within the "Anisotropy analysis" part of autoPROC has been fixed (thanks to M. Zeronian for reporting this issue). This involved the interplay between the merged data analysis in STARANISO and the data quality metrics computation in MRFANA, sometimes leading to the reporting of different numbers of unique reflections at the processing stage compared to the actual number of merged reflections at subsequent stages, e.g. during refinement. * Improve logic in GPX2 to find next image by skipping empty ranges. * Improve text placement of intensity values in GPX2. * Fixed regression in GPX2 such that offset is correctly scaled. * Fix to aP_fixcbf: works now correctly for directories containing multi-sweep datasets (defined by different file name template). * Fix of "check_indexing" regression regarding writing of output file (Thanks to E. Sprague for highlighting this problem). * Several improvements in preparation of deposition-ready mmCIF files (related to _software category and audit records). * Several fixes in plot generation and presentation for XDS/INTEGRATE results. * Several fixes in handling of multi-orientation datasets - especially when an unusual coordinate system is being used. * New features, fixes and improvements (released May 20 2020): * Improved layout of plot showing fraction of indexed spots for significant solutions achieved during iterative indexing ("run_idxref_spot_hkl_histfrc_smooth.png"): * highlighting the fraction of unindexed spots as a solid grey area * emphasising the solution picked at that stage (for multi-lattice cases) * reporting the angle between the orientations of the different indexing solutions and the main/selected one * New plot to show number of misfits as a function of image number. * To avoid the adjustment of error model within AIMLESS to result in extreme and unreasonable values, limits for the SdFac and SdAdd parameters (correspond to a/b in the XDS error model) have been introduced: autoPROC_AnaAimlessSdFacLimits="0.01 100.0" autoPROC_AnaAimlessSdAddLimits="0.0 0.5" These should prevent especially SdAdd to move into highly suspicious regions - and therefore stabilise the whole iterative scaling procedure in the aP_scale scaling module for extreme cases of very weak datasets, crystal moving out of the beam, severe radiation damage etc. * When running multiple integration (INTEGRATE) and post-refinement (CORRECT) steps (e.g. as part of the "LowResOrTricky" macro), the beam-divergence and mosaicity parameters are updated by default (XdsUpdateBeamDivergence=yes and XdsUpdateReflectingRange=yes). In case of multi-lattice diffraction (especially if they are close in orientation) or very streaky reflection shapes, these can drift throughout the process. To avoid this, limiting settings are introduced (XdsUpdateBeamDivergenceMaxShift=0.2 and XdsUpdateReflectingRangeMaxShift=0.2) to allow only shifts up to 20% away from the previous value. * Improved handling of Eiger detectors during indexing (especially if data are provided in the form of mini-cbf files) - taking the smaller pixel size compared to Pilatus detectors better into account. Thanks to G. Bourenkov for providing example data and input. * Additional checks regarding ice-ring reflections before going into (iterative) indexing: if the fraction of spots in ice-rings is above 10% (XdsExcludeIceRingsInIndexingIfAbovePercentage parameter), all known ice-ring resolution ranges will be excluded from indexing. At that point some additional analysis is done to highlight these potential problems to the user. * Added support for modified HDF5 datasets from beamline BL32XU (SPring-8) that contain compressed meta-data. Thanks to T. Fukami and K. Hirata for example data and discussions. * Added support for CCP4 7.1 (7.1.000) and XDS build 20200417. * If there are buffering issues with networked filesystems (very rare and should be looked at locally), the parameters XdsStats_waitntimes and autoPROC_AnaAimlessWaitNTimes (both default to 0) could be used to make some parts of autoPROC involved in producing graphs and analysis to wait and see if required files are "appearing" after a while. This is clearly not the best way of handling this, but might be a useful workaround in particular situations. * Fixed bug when setting specific tool locations via the BDG_TOOL_XYZ system (reported by J. Thorpe and B. Nolte, GSK; analysed in more detail by J. Key, SBGrid). * New features, fixes and improvements (released Mar 19 2020): * Additional interactive results from STARANISO can now be included in the summary.html file for viewing color-coded reciprocal lattice scenes in 3-D. This inclusion is controlled by a new parameter that defaults to AutoProcScale_RunStaranisoCreate3D="no". Set it to "yes" for activating this feature. * New analysis in aP_detect_damaged_pixels to not only detect hot/damaged pixels, but also "stuck" pixels, i.e. pixels with the same non-zero value on all analysed images. * New feature to allow generation of PDF reports also from scaling-only runs (using aP_scale): set parameter autoPROC_Scale_CreatePdf to yes, e.g. via aP_scale autoPROC_Scale_CreatePdf=yes ... * Plots of unindexed spots (highlighting e.g. ice-rings) and per-image scale factor (often highlighting potential instrumental problems) are now always shown within summary.html and not within the hidden (by default) sections. * In cases where XDS initially fails because it was unable to determine initial spot size parameters (e.g. if there are very few strong spots), autoPROC will now automatically adjust the block width used within the XDS/INTEGRATE step (autoPROC_AdjustIntegrateStepIfFailed=yes) by doubling it (autoPROC_AdjustIntegrateStepIfFailed_DelphiFac=2) up to a width of 20 degree (autoPROC_AdjustIntegrateStepIfFailed_DelphiMax=20). * New tool (aP_select_pdb) to help matching up newly processed data (MTZ file) with a sensible starting model out of a whole selection of potential models. Different indexing and screw-axes are taken into account when comparing actual reflection data to find a highly isomorphous pairing. See "aP_select_pdb -h" for help. Please let us know if there are additional features or capabilities that might be useful to have here. * New tool (aP_fixcbf) that tries to fix mini-CBF files containing erroneously masked pixels. This can happen eg. for data from Eiger detectors written as mini-cbf via the stream interface: * pixels with values abote the countrate cutoff value (i.e. seen as unreliable) should not be changed to a negative value (which defines a so-called "masked" pixel to subsequent programs). * software needs to know if a pixel is masked (module gaps, dmaged pixels) or just temporary "overloaded" * incorrect conversion from the Eiger stream interface into mini-cbf results in (1) overexposed datasets become invisible since not a single pixel will ever be defined as above the cut-off - and autoPROC will be unable to give a warning about excessive "overloads". (2) unreliable, strong ("overloaded") reflections will be treated by XDS as if they were only partially integrated (e.g. at the module edge): their intensity will then potentially be scaled up to (wrongly) accommodate for this - when instead they shuold be excluded from further scaling and merging as unreliable "overloaded" reflections. * We are aware of this problem for at least some beamlines using Eiger detectors and writing mini-cbf files (via the stream interface) - going back to at least 2016 judging from available, deposited raw image data. The aP_fixcbf tool tries to (1) detect those pixels in the images as they are and (2) replace the masked value with a value to define it as being above the count_cutoff value in the mini-cbf file header. * Fixed problem in automatic analysis of (potentially) damaged pixels ("hot" or flickering) that might have adversely affected the initial spot list: when using the "-M fast" macro or setting separate spot-searching image ranges manually (e.g. via the autoPROC_XdsKeyword_SPOT_RANGE parameter), this step triggered a failure in the previous release. * Fixed generation of example pictures for hot/damaged pixel detection using aP_detect_damaged_pixels: pictures close to the bottom and/or right of an image were previously truncated/shifted but should now be correct like for any other detector region. * Fixed generation of XML files associated with specific output MTZ files (i.e. truncate-unique.xml/staraniso_alldata-unique.xml for truncate-unique.mtz/staraniso_alldata-unique.mtz): if running autoPROC without the -d flag this resulted in an incorrect XML file. Thanks to K. Clark. * Fixed some image labeling error in summary.html. * Fixed handling of image sets with gaps (the last changes related to this didn't actually fix the problem). * Improved checking that files given to "combine_files" command are actually supported multi-record MTZ/HKL format. * New features, fixes and improvements (released Feb 06 2020): * Better method for extracting an adequate wavelength value from MTZ files within the MTZANA tool. * Improved diagnostics when encountering directories containing both *_master.h5 and *.nxs (HDF5/NXmx compliant) files. * Improvements to analysis of completeness to generate F(early) and F(late) datasets (that can then be used in BUSTER to compute F(early)-F(late) density maps to analyse for radiation damage). To avoid including too many images with little change to overall completeness (which would effectively reduce the gap between these two datasets in terms of exposure, ie. dose), the increase from the default minimum completeness (85%) to maximum completeness (95%) is compared in terms of cost (narrowing of the gap) vs benefit (increase in completeness). * Added functionality to fix AIMLESS buglet (where the dataset-specific cell dimensions are incorrect when it writes unmerged MTZ files). P. Evans is aware of that problem and we hope for a fix as soon as possible. Also added functionality to the merging statistic computation to handle multi-sweep datasets (each with its own set of cell parameters) in the context of STARANISO analysis (working on merged data with an average cell). Thanks to G. Verdon for noticing the resulting discrepancies in reported resolution values. * New tool ("aP_cleanup_spots") to detect and remove suspicious spots from a XDS-style SPOT.XDS file if (a) they are very strong and basically at the mid-point of the covered rotation range (often caused by damaged/hot pixels), and/or (b) there is an unexpectedly large cluster of spots at particular detector positions (the result of flickering damaged pixels. This will be used by default within autoPROC indexing step (autoPROC_DoCleanupSpots="yes"). * By default (autoPROC_RestrictLowResInIndexing=yes) the low-resolution will be restricted to 20A (set by parameter autoPROC_RestrictLowResInIndexingTo) during indexing: the danger of failed indexing due to inclusion of poor or incorrect spots that close to the beamstop is too great. Thanks to K. Diederichs for advice and discussion. * Improvements to checking for very weak diffraction patterns (throughout rotation or in particular orientations): we now only expect 1 spot per degree of data (within a smoothed, sliding window) before stopping with an error (set via parameter XdsSpotSearchMinNumSpotsPerDegree=1). * Improvements to creation of deposition-ready mmCIF files: (1) avoid gemmi writing comments (that seem to confuse the current deposition system) and (2) provide the correct wavelength value. * Added support for XDS versions March 15, 2019 (BUILT=20191211) and January 31, 2020 (BUILT=20200131). * Added option (-nodisplay) to aP_detect_damaged_pixels: this should enable more (semi-)automatic analysis and detection of potentially damaged pixels. * Fix to handling of overload value in hdf2mini-cbf (HDF5 to mini-cbf converter), which for NXmx-compliant Eiger/HDF5 datasets could result in incorrect overload value being written to the mini-cbf files. * Improvements and fixes to "check_indexing" regarding orthorhombic space groups (that allow for axis reordering), the scaling procedure used between sets of MTZ files and the final analysis. * Improvements in generation of inlined summary HTML page: mmCIF and PDF files should now also be included. * Added imginfo wrapper ("imginfo_SLS_crippled_miniCBF.sh") for handling mini-CBF files that were converted without describing the detector type (Pilatus/Eiger) in the Detector keyword. Thanks to J. Sack for highlighting this. * Fixed erroneous reporting of reindexing operators (from POINTLESS) even if no re-indexing was performed. * Fixed a regression regarding automatic handling of 2-Theta offsets introduced in the 20191129 release (thanks to A. Mulichak for reporting). * New features, fixes and improvements (released Nov 29 2019): * Tested against CCP4 7.0.078 and XDS (Mar 15, 2019, BUILT=20190806). * Support for XDS (Mar 15, 2019, BUILT=20191015), including the new XDS keywords SNRC and BATCHSIZE (but no extensive analysis of those changes has been made yet). * Improved handling of overloaded reflections (due to damaged/hot pixels that are not yet masked) in the AIMLESS-only scaling path ("-M ScalingA3"): we now ensure that those reflections are rejected before creating the input reflection MTZ file for AIMLESS. This should avoid numerical instabilities within the AIMLESS scaling program. * Better handling of explicitly excluded images (via the EXCLUDE_DATA_RANGE XDS keyword) in analysing for processing problems: user-specified images for exclusion will not trigger a warning message about problems reading those images. * Fixed bug in the Dectris/Neggia plugin to correctly handle 16-bit Eiger HDF5 datasets. For details see http://www.globalphasing.com/autoproc/wiki/index.cgi?DataProcessingHdf5Eiger16bit201910 * Improvement in detecting Eiger data in mini-cbf format (as opposed to HDF5): such datasets will be treated with the same defaults independently of the actual image format. * Fix in our HDF5 to mini-cbf converter (hdf2mini-cbf) to correctly determine and output the Count_cutoff value from HDF5 into the mini-cbf files). * Fix and improvements in aP_detect_damaged_pixels (detecting damaged and not yet masked pixels): pixels at the detector edges will be presented correctly in the pictutes. Also, pixels above the overload value (1048575, can be changed with -o flag) will also be included in the final list of damaged pixels by default. * New features, fixes and improvements (released Sep 23 2019): * New tool (aP_detect_damaged_pixels) to help analysing diffraction images (in mini-cbf format from standard datasets) for presence of damaged/hot pixels. It can be run on a random subset of images (typically 5-10 should be enough), e.g. using aP_detect_damaged_pixels *[05]1.cbf or aP_detect_damaged_pixels *[05]01.cbf For some example results see e.g. https://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings#hotpixels This tool requires the ImageMagick package to be installed (in order to create control pictures of the detected problem pixels). * Added support for version of Eiger2 HDF5/NeXus datasets (e.g. from beamline I04 at the Diamond Light Source, 20190904) that have the "image_nr_low" and "image_nr_high" attributes removed from the meta-data. The assumption than is that (1) data was collected with a starting image number of 1 and (2) applications return the list of external data files always in the order the images were actually collected. Thanks to G. Winter for providing example data. These assumptions might not always be correct (which is why those two attributes seem relevant and important) and data collections involving e.g. interleaving or image number offsets might well cause unexpected behaviour or failures. * Added support for direct use of Eiger2 HDF5/NeXus meta-data in NXmx format (*.nxs files) as currently defined e.g. at I04@DLS (20190904): this allows use of the *.nxs file in all places where a *_master.h5 file would have been used before. Please note that if both file types are present in a directory (as given to the -I argument of process), the *_master.h5 file(s) will take precedence and all *.nxs files are ignored. * Added more information about the actual (potentially restricted by lattice system) cell parameters used after the space group analysis done by POINTLESS. * Set DETECTOR keyword in XDS.INP to "EIGER" if appropriate (for XDS versions from 20160428 onwards). * Fixed regression regarding generation of F(early)/F(late) amplitudes introduced in the 20190131 release: the two amplitudes would have been the same by mistake. * Fixed some errors in checking for supported CCP4 version when running as part of checkdeps (e.g. 'process -checkdeps'). * Fixes in extracting all goniostat angles from HDF5/NeXus multi-axis datasets. * Fixes in "imginfo" when handling image or image directory names containing ":" characters. Thanks to C. Schleberger and M. Kroemer for notification. * Fixes to generation of mmCIF files (for reflection data): the _software category now contains the correct version information for each used package/program (as well as the location). * Fixes to mmCIF reflection data generation: now also include an explicit _refnl.status item (if a test-set flag column is present). * New features, fixes and improvements (released Jun 7 2019): * First version to support neutron data from the D19 instrument at the Institut Laue-Langevin (ILL) in Grenoble (F). Thanks to T. Forsyth, E. Mossou and K. Diederichs for getting this started. The new macro "ILL_D19" should set most of the relevant parameters - see "process -M list" for details. Please note that a working installation of ImageMagick is required (for handling the TIFF format data files). * Updated to support CCP4 7.0.074 (including POINTLESS 1.11.21). * New features, fixes and improvements (released May 29 2019): * Improved handling of HDF5 datasets (within find_images) with smaller *_master.h5 files (e.g. I04@DLS Eiger2 HDF5/NeXus files where the pixel mask(s) are no longer stored in the master file itself). When giving only a directory name ("process -I ...") those smaller files would not have been detected due to a default size limit of 512k. Now a separate parameter (FindImages_min_size_HDF5 with 32k as default value) is used to handle HDF5 datasets. Thanks to P. Matias for pointing this out. * Added Gemmi binary ("gemmi") to the autoPROC standalone distribution (as required for creation of deposition-ready mmCIF reflection files). Thanks to K. Battaile for pointing this out. * Fixed typo of F_meas_sigma label for mmCIF file. * Robustify handling of HDF5 file location. * Robustify some script constructions that could cause problems on Mac/OsX. * Updated to support latest XDS version "Mar 15, 2019" (20190315 build). * To allow systematic absences being carried through (as far as possible) within autoPROC, the new parameter autoPROC_CadSysAb can be set to "SYSAB_KEEP" (according to the CAD documentation). Thanks to M. Rudolph for suggesting that. * Fix to generation of mmCIF (reflection) files when a run without '-d' flag did create a subdirectory (because of required workaround regarding file/directory name length limitations in XDS). * Fixes and improvements to handling helical-scan data from Eiger detectors (HDF5 format) at Diamond beamlines - to ensure even without explicit specification of the rotation axis all relevant information can be read. Thanks to M. Zebisch for highlighting this issue. * Added improvements for (upcoming) Eiger2 detector at PX-II @ SLS to correctly extract the angle increment for each image. Thanks to J. Wojdyla for early access to example datasets. * New features, fixes and improvements (released Mar 01 2019): * Added generation of deposition-ready mmCIF files containing reflection data (can be switched off by setting autoPROC_CreateCif=no). Usually, two separate mmCIF files will be generated: (1) Data_1_autoPROC_STARANISO_all.cif (containing data from the STARANISO analysis, i.e. staraniso_alldata-unique.mtz) (2) Data_2_autoPROC_TRUNCATE_all.cif (containing data for the isotropic analysis, i.e. truncate-unique.mtz) Please note that (a) for multi-wavelength datasets there will be additional file-naming components based on the wavelength value, and (b) to produce these files as part of manually running the aP_scale scaling module you will need "aP_deposition_prep" manually. Each mmCIF file will contain not only the amplitudes and intensities of the final step (STARANISO or (C)TRUNCATE), but also the set of scaled and merged intensities that final analysis is based upon - without any restrictive high-resolution limit being imposed. We highly recommend using those mmCIF files during deposition: please let us know if you encounter problems with these by contacting us at proc-develop@globalphasing.com. * Further robustifying handling of HDF5/Eiger dataset variants: - allow for unit mismatches ("degree" versus "deg") - allow for image numbers not starting at 1 - handle new-style Eiger2/HDF5 metadata files (as produced e.g. by I04@DLS) better: * the change regarding image numbers caused problems with the /entry/data/omega dataset (in the same /entry/data location as external links to data files); * correctly handle the fact that non-chainging goniostat angles (chi/kappa, phi) are now not given as an array of size nimages, but only as a single value; * New features, fixes and improvements (released Feb 14 2019): * The content of the environment variable autoPROC_ReportComment will be reported at the top of standard output and the summary HTML file (in the section mentioning date, directory, user etc). This allows specification of "context", e.g. if autoPROC is run as part of Pipedream, some automatic synchrotron processing or an inhouse pipeline. * Further improvements regarding NeXus-style HDF5 (Eiger2) datasets from e.g. Diamond/DLS beamlines like I04: * will handle datasets with or without link to (non-existing) pixel mask * also extract kappa, chi and phi angle values stored within /entry/sample/sample_* (instead of /entry/sample/goniometer/*) * Initial support for HDF5 datasets with an image number offset, i.e. when the first image starts not as number 1. This allows for the same logic as for more traditional datasets (one file per image) where a dataset might start at image 1001 for internal bookkeeping reasons. This involved changes to 'imginfo' when analysing the meta data of a HDF5 dataset, 'hdf2mini-cbf' when extracting single images or image ranges (in CBF format with a full mini-cbf header) and several other autoPROC tools. * New features, fixes and improvements (released Jan 31 2019): * Improved handling of small molecule datasets (support for non-chiral SGs, handle non-existing anomalous signal, small unit cells etc). A single macro ("-M SmallMolecules") can be used to switch from the default mode (chiral SGs of macromolecules) to processing of small molecule X-Ray diffraction data. These improvements also includes * a new parameter (autoPROC_PointlessChirality, default = "CHIRAL") to set the POINTLESS keyword CHIRAL accordingly; * handling centrosymmetric SGs with no anomalous differences; * Also show per-image scale factor (from AIMLESS scaling) - which can be of interest in the AIMLESS-only scaling path ("-M ScalingA3"), but more difficult to interpret in the default path (because it represents scale factors on top of correction factors from XDS/CORRECT). * Improved automatic handling of multi-wavelength data processing when using e.g. an interleaved-wavelength protocol. The wavelength written into image headers (or other meta-data) can show small differences even for the nominally same energy - because of small drifts between sweeps. This could lead to autoPROC interpreting these as different datasets as that decision is based on wavelength value alone by default. The new parameter WavelengthSignificantEnergyDifferenceInEv with a default value of 2 (in eV) will combine sweeps that only differ in energy by a maximum of 2 eV. * Fixes for handling NeXus-style HDF5 (Eiger2) data (as e.g. encountered at DLS beamlines): * We also need to make the *_meta.hd5 file available to autoPROC. * Better handling of pixel mask data stored in external link file (*_meta.h5) - although e.g. I04/DLS data currently (20190128) are unable to hold the pixel mask data. DLS is aware of that and working with Dectris to achieve a solution. The data processing itself should not be impacted by that (since the image data itself already contains markers for any damaged/inactive pixel. Thanks to G. Winter and R. Flaig (DLS) as well as several Consortium members for providing background information, example data and help. * Removed problem with memory management within hdf2mini-cbf (possibly connected with previous item). An optional workaround for any issues with the hdf2mini-cbf application (when run as part of autoPROC processing HDF5 files via the Neggia or Durin interface) is to setDistributeBackgroundImagesForHdf5=no and see if that lets autoPROC process the HDF5 dataset successfully. * Added support for unusual Eiger HDF5 datasets from DLS (at least for I04 in January 2019) where detector distance is stored as /entry/instrument/detector_distance and not in /entry/instrument/detector/detector_distance. Thanks to R. Byrne (WuXi/Crelux) for pointing that out. * Better handling of image ranges that will not have any spots because we didn't actually look for spots on those. This will only have an effect if running autoPROC in non-default mode (settings intended for faster execution at the cost of accuracy). * Further improvements and fixes in the scaling module (aP_scale): * New parameter AutoProcScale_RunStaranisoUniqueifySaFlagOnly (default="yes"): when "uniqueifying" STARANISO MTZ files to add a test-set flag, this will only keep reflections with a valid SA_flag value. The SA_flag value is based on the ellipsoid fitted to the anisotropic cut-off surface and describes the diffraction limits in terms of observable and unobservable reflections. * New parameter AutoProcScaleStatsUseMrfanaUseFinalNumBin (default="yes"): ensure that we always use the same binning for the final *.{table1,stats,xml} files. By default this uses the AutoProcScaleStatsUseMrfanaFinalNumBin (default=20) binning. * New features, fixes and improvements (released Nov 27 2018): * Support for XDS plugin mechanism when processing HDF5 datasets (usually from Eiger detectors). Two plugins are provided as part of our distribution: dectris-neggia (https://github.com/dectris/neggia) for standard Eiger HDF5 datasets durin (https://github.com/DiamondLightSource/durin) for Diamond VMXi beamline data (and future I03 and I04 detector upgrades) These plugins will be activated by default (setting the adequate LIB= keyword for XDS). If that is not intended, please set autoPROC_Img2Xds_UseXdsPlugins=no to prevent this automated feature. * Support for extracting experiment descriptions (detector axes and incident beam direction) from HDF5 metadata file (master.h5). * Fixed problem with equal-observation-number binning of MRFANA which could have caused misjudging isotropic resolution limits and/or given unreliable results for very large number of reflections (and therefore bins). * Support for Diamond (DLS) beamline I24 images written in fullCBF format: compared to the I04 fullCBF images some vector/axis specifications have changed names (DET_Z had become DETECTOR_Z). Thanks to Richard Gildea (DLS) for pointing that out. * Support for extracting axis definitions from Eiger/HDF5 images, e.g. "/entry/sample/transformations/omega" as stored in HDF5 master files from the Eiger 16M at BioMAX (MaxIV). Thanks to Jie Nan for highlighting this. * Support for latest versions of XDS (20180808), CCP4 (7.0.066), POINTLESS (1.11.17) and AIMLESS (0.7.3). * Improvements in writing of REMARK 200 PDB files: new parameters (see below) allow the user to populate the relevant items as well. Thanks to D. Reinert for pointing this out. These default to "NULL": autoPROC_Remark200_TEMPERATURE autoPROC_Remark200_PH autoPROC_Remark200_NUMBER_OF_CRYSTALS_USED autoPROC_Remark200_BEAMLINE autoPROC_Remark200_MONOCHROMATOR autoPROC_Remark200_OPTICS autoPROC_Remark200_XRAY_GENERATOR_MODEL These default to adequate value (depending on wavelength, i.e. Cu, Cr, Au and Mo K-alpha defaults to non-synchrotron): autoPROC_Remark200_RADIATION_SOURCE autoPROC_Remark200_SYNCHROTRON * Fixed regression in previous release: report_staraniso.pdf was missing the information about resolution limits along principle axes of fitted ellipsoid (thanks to I. Cornaciu for reporting). * Fixed a typo in aP_summary2base64 that might have caused problems if a user-defined version of Imagemagick's "convert" program was used. * Fixed a problem in computing maps for ligand-detection (-L) mode that resulted in using the full-cell maps instead of maps only covering the asymmetric unit for cluster-analysis. There would have been no effect in the results (since the cluster analysis also handles symmetry to restrict itself to the asymmetric unit), but it would have run slightly slower. Thanks to M. Rudolph for highlighting this. * Fixed determination of early/late sections within STARANISO processed data. Please note that this does not yet result in generation of F_early/F_late columns in the STARANISO MTZ file (staraniso_alldata-unique.mtz): this will require additional work to have the common anisotropy mask being applied to those subsets of data. We hope to have that functionality ready in an upcoming release. For the moment you should use the truncate-unique.mtz file within BUSTER (with BusterEarlyLate=yes) to compute those F(early)-F(late) maps. * Better handling of HDF5 archives in "imginfo" utility: exposure time per image will now also be reported if available. * Detect situation where the post-refinement step (XDS/CORRECT) didn't produce an updated set of parameters (GXPARM.XDS): newer XDS versions (20171111 onwards) behave differently in this situation than older ones. * Improvements to ISPyB-compatible XML files produced by autoPROC (autoPROC.xml and autoPROC_staraniso.xml): * new parameters autoPROC_CreateXml_SeparateEarlyLate (default=no) to control if we should provide access to separate MTZ files containing F(early) and F(late) data. The default excludes those from the XML file since the default MTZ file will contain those columns anyway. * new parameter autoPROC_CreateXml_AdditionalFilesForMtz (default="table1 stats") to also provide access to the *.table1 (overall, inner and outer resolution bin merging statistics) and *.stats (merging statistics as a function of resolution) ASCII files. Thanks to O. Svenson for discussions and suggestions. * Fixed missing support for CLUSTER_NODES keyword (via autoPROC_XdsKeyword_CLUSTER_NODES) - thanks to K. Battaile for pointing that out. * New features, fixes and improvements (released May 15 2018): * Changes to the default behaviour of autoPROC: (1) The handling of Bijvoet pairs now assumes these to be identical. This does not prevent the analysis and output of anomalous data, but should cater better for the vast majority of datasets processed. If a strong anomalous signal is expected and high enough multiplicity is available, setting '-ANO' on the command-line for the 'process' or 'aP_scale' could be used. (2) The GAIN parameter within XDS will be set to a fixed value of 1.0 when encountering Pilatus or Eiger Pixel detectors. This can be switched off via one of autoPROC_Img2Xds_Pilatus_SetGainToOne="no" autoPROC_Img2Xds_Eiger_SetGainToOne="no" To go back to the previous default settings, the "Version20171219" macro can be used (via the '-M' flag to the 'process' command). * New functionality and control about the way processing information is summarised in HTML form: (1) Instead of opening the full-size image in a new window (or tab), a modal image will be created if the parameter autoPROC_CreateSummaryImageHrefLink (default="yes") is set to "no" (a modal is a dialog box/popup window that is displayed on top of the current page). This feature will also apply to certain types of ASCII files, like program log-files or tables. (2) There is the possibility to create a version of the summary.html with all images (as far as possible) inlined into the actual HTML page: summary_inlined.html. This will allow the full display of all plots and graphs even after moving autoPROC results to another computer or system - but be aware that the resulting HTML can become quite large (>100 MB). This work was undertaken in response to certain technical difficulties encountered at some synchrotron sites that made it impossible for them to provide their users with direct access to summary.html through their standard (often web-based) user interfaces. This resulted in either broken summary.html content (no images shown) or in requiring other workarounds that severely limited the usefulness and added value provided by the full content of this autoPROC summary. We expect and hope that these synchrotron sites will now adapt their systems so as to make the most out of this inlined summary.html as soon as possible, and thus give users full access to all autoPROC results in the same way that they do to those of other pipelines and programs. This version of summary.html with inlined (base64-encoded) image data can be created by using the new ReportingInlined macro or by setting autoPROC_Summary2Base64_Run to "yes". Additional fine-grained control is available via * autoPROC_Summary2Base64_Resize (default="no"): inlined image data is reduced from the original image size to the size required for the small (thumbnail) embedded in the HTML page. This will reduce the overall size of the resulting HTML page significantly, but a lot of plots and graphs will become rather difficult to analyse at such a reduced resolution. * autoPROC_Summary2Base64_Hide (default="keep"): determines how the "hidden" plots should be treated in the inlined-data version of summary.html. The default ("keep") leaves it as-is, while "yes" would ensure they are hidden or "no" that they are shown by default. If the resulting HTML file (with inlined data) is intended as the basis for conversion e.g. into PDF, then setting this parameter to "no" would be a good idea. * The parameter autoPROC_Summary2Base64_Format (default="JPG") can be used to select between JPG (small, but lossy compression) and "PNG" (larger, but lossless compression). * The default name for the inlined HTML file ("summary_inlined.html" in the same directory as summary.html) can be changed via the autoPROC_CreateSummaryInlinedFile parameter. A good setting for nice, inlined HTML page would be autoPROC_CreateSummaryImageHrefLink="no" autoPROC_Summary2Base64_Run="yes" which is what the new "ReportingInlined" macro provides. Therefore, just running with process -M ReportingInlined should be all that is needed for this new feature to work. * Added analysis of intensities (merged and unmerged) as well as as a function of image number. * More information will be written to the "Table 1" style summaries on stdout as well as within summary.html. This includes the active (isotropic) cut-off criteria where applicable, number of active ice-rings in the analysed resolution range and how many sweeps (RUNs) contribute to a particular dataset. * Avoid redoing the analysis as function of image number when switching between fine- and fixed-number binning (the former is for analysing high-resolution limits and ice-rings, while the latter is intended to give more standard binning for "Table 1" style statistics). These statistics are unaffected by a change of resolution binning, obviously. * Some small improvements in plot generation when we have the raw data (often giving quite noisy plots) and want to also show the smoothed/averaged version (which might be more useful and clearer). * Support for SSRL BL14-1 Eiger 16M images (written as mini-cbf images instead of HDF5 files). * New macro for the ID23-2 beamline at ESRF - for the Pilatus3 X 2M (S/N 24-0118) and MD3Up micro diffractometer setup as of 20180207. The vertical Omega axis (MD3Up) needs to be specified and the silicon thickness is recorded incorrectly in the image header (should be 0.45 mm). * Update to the I23/DLS beamline macro (DiamondI23): ensure we are using all images for background estimation to start with (required for proper shadow computation and correction). * Fix in generation of statistics as function of image/batch number at the STARANISO stage when handling multi-sweep datasets during the XSCALE scaling path. * Added check (and workaround) for TRUNCATE bug present in CCP4 7.0.039 to 7.0.056. This results in wrongly computed amplitudes/sigmas and is due to only partially correct changes by CCP4 on "Mon 2017-05-22 17:19:55 +0100". We have reported this problem (and included the required correction) to CCP4 for an upcoming update release of CCP4 - please watch http://www.ccp4.ac.uk/updates/ for details). Please note that the STARANISO produced files (staraniso_alldata-unique.mtz) are not affected. A new tool has been added to check existing MTZ files if they are affected by this bug: aP_mtz2hist -do_fq XYZ.mtz should create a file XYZ_hist_F-over-SIGF.png that shows a histogram of F/SIGF values. This should be smoothly varying - but if it shows a "gap" (often in the 10-20 range), it is most likely that the data was affected by that bug. * Support up to XDS version "Jan 26, 2018" (BUILT=20180319) and CCP4 7.0.057 (including POINTLESS 1.11.12 and AIMLESS 0.7.1). * New features, fixes and improvements (released Dec 19 2017): * Synchronisation of parameters SEPMIN and CLUSTER_RADIUS relevant for indexing via XDS/IDXREF: Pilatus detector now uses the same values as Eiger images (SEPMIN=4 and CLUSTER_RADIUS=2). This is different from the defaults but should help indexing very dense diffraction patterns. * Better handling of warning messages regarding image headers (incomplete, inaccurate or confusing information). * Fixes to automatic determination of non-shadowed image ranges (when re-doing initial background determination). * Update to support XDS Version November 11, 2017 (BUILT=20171111). * Automatically give gap definitions for known Pilatus/Eiger detectors based on detector dimensions. This will ensure correct processing even if the module gaps are wrongly marked as "0" (instead of the default negative number) - which is possibly due to a manual change of the default detector startup protocol at the given beamline. * Updates to the DiamondI23 macro, describing statically shadowed regions (beam stop holder, beamstop etc) as of 20171129. * New features, fixes and improvements (released Nov 14 2017): * Added support for SmarGon goniostat on I04 beamline (DLS): the standard "DiamondI04" macro now refers to the current status of the beamline (as of 20170906). Data collected on the previous incarnation (with a mini-Kappa goniostat) should be using the "DiamondI04-MK" macro instead. Thanks to G. Winter and R. Flaig (DLS) for making example data available. * Added support for full-CBF images (e.g. I04 or I19-1 at Diamond Light Source). These contain a mini-cbf plus imgCIF header. Thanks to G. Winter for making example data available. * Added initial support for the ADSC HF-4M detector at NSLS-II:19-ID NYX beamline. Thanks to Raman Jayaraman for making example data available. * Support in GPX2 for Bruker Photon-II images regarding visualisation and prediction generation. * Improve automatic detection of rotation axis name in the 'imginfo' application when some keywords are missing in image headers. * Added initial support for handling of negative oscillation angles (which requires inversion of rotation axis definition to achieve a positive rotation as required by XDS). * The option to apply high-resolution limits early on during data processing (instead of waiting for the final data scaling/merging for this decision making, as is the default) now uses a more relaxed value of 0.0 (instead 0.5) as default. Additionally, a CC1/2 criterion of 0.1 is used at this stage. This feature can be activated via settings of XdsSetHighResFromIndexedSpots=yes (default=no) XdsSetHighResFromFirstIntegrate=yes (default=no) XdsSetHighResFromFirstCorrect=yes (default=no) separately at different stages. Note that ideally the crystal-detector distance should have been chosen to make optimal use of the detector surface throughout the experiment. But in cases were a vastly optimistic distance was set (resulting in very large areas of the detector not recording any crystal diffraction spots), these features might help integration and scaling. * Added segment descriptions for Pilatus (100K, 200K, 300K and 1M) and Eiger (1M, 4M, 9M and 16M) detectors. * Updated to support CCP4 7.0.045 and Pointless 1.11.4. * Faster and more space-efficient way of generating and using predictions by using binary format. * Generalised the flag to notify autoPROC about a rotation axis running the opposite direction: use ReverseRotationAxis (default = "no") for both Omega and Phi scans. * Ensure we are removing the temporary files we create when using equally distributed images for background determination in XDS. For HDF5 files this creation of temporary files is required, but there is no need for keeping those afterwards. * We now automatically detect if we have access to X server (and switch off generation of prediction files via GPX2 if this is not the case instead of flooding standard output with error messages). * If no explicit list of image directories, file names and data ranges are given and we try to automatically determine the sweep list: when encountering unusual/unexpected sequence of image files (e.g. with gaps or odd timestamp ordering) ensure that no duplicate identifiers are extracted at all costs. * Update to support XDS Version June 1, 2017 (BUILT=20170923). * Various improvements in the PDF reports generated (report.pdf for isotropic data analysis and report_staraniso.pdf for anisotropic analysis) regarding content (STARANISO plots are now included), formatting and layout. * Fixes in generation of final XML files (autoPROC.xml and autoPROC_staraniso.xml): improved handling of single-/multi-wvl in combination with single-/multi-sweep dataset situations. * When working with Eiger HDF5 files (*_master.h5) and using a subset of images where the first image is not image number 1: ensure to extract the correct starting angle for this image explicitly. This is especially important for bookmarking datasets, i.e. when collecting at multiple positions on the same crystal in a pseudo-contiguous manner. * Initial support for handling minisets within an Eiger HDF5 dataset as currently implemented at the SLS (*_master.h5 file) in the 'imginfo' application. Please note that there are still various open questions about consistency of data, correct values for goniostat angles (including rotation axis) and how to distinguish different types of data (bookmarking, LCP crystals in wells etc). These are being discussed with developers at SLS and Dectris. * Improved handling and reporting for situations where no prediction images can be generated, e.g. when no X-server is available (running on cluster) or Eiger HDF5 datasets are being processed (the handling of which using GPX2 is not yet supported). * New features, fixes and improvements (released Sep 20 2017): * Added support for STARANISO in XSCALE path. This uses the full resolution data after the first XSCALE run, i.e. not 100% identical to the default AIMLESS path where we try to optimise the scaling before applying those optimised scales to the original, full-resolution data: this is not possible with XSCALE at the moment. * First implementation of a PDF report based on the STARANISO statistics (called report_staraniso.pdf) in parallel to the isotropic data (report.pdf). It still needs some further work (e.g. it doesn't yet contain the STARANISO pictures) and any comments or advice is welcome. * Updated list of parameters in docs/autoproc/manual/appendix1.html: removed unused parameters, added new ones and ensured the list is sorted correctly. * Work to reduce the footprint of autoPROC results: (1) Instead of linking individual files, we now link the parent directory instead (if possible). This can be prevented by setting autoPROC_LinkImagesIntoSubdirSeparate=yes to reinstate the previous behaviour. (2) remove XMGR files from AIMLESS run (3) added parameter AutoProcScale_RunStaranisoKeepOnlyAsymHkl (default=yes) to only keep *asym.hkl file after STARANISO step in scaling module (4) stop writing STARANISO files currently not used for plot-generation (this can be re-activated by setting AutoProcScale_RunStaranisoAlsoCreate="DWFACT IJMEAN RLAXES ELLIPS REDUND") * Ensure that each of the resulting *-unique.mtz files has a corresponding *.stats (statistics as a function of resolution in bins), *.xml and *.table1 file (Overall, Inner and Outer resolution shell statistics). The are also available via hyperlinks in the summary.html file. * Remember that the default high-resolution criterion for isotropic analysis is (mostly) based on an value of 2 in the last shell. This can easily be changed e.g. via process -M HighResCutOnCChalf ... (to a criterion based on CC1/2=0.3) or using process -M NoHighResCut ScaleAnaISigmaCut_123="1.0:1.0" ... (to a =1.0 criterion) if required. Please note that this is not a new feature in this release and that this has been available right from the start of autoPROC. * Added (initial) support to get F(early)/F(late) data for use in F(early)-F(late) maps also for STARANISO processed data. This uses parameters autoPROC_EarlyLateCompletenessEllMin=0.80 and autoPROC_EarlyLateCompletenessEllMax=0.95 to decide based on ellipsoidal completeness. * When creating plots for the various merging statistics (completeness, Rmerge, etc) we can try and run Gnuplot in parallel: this can be triggered by setting autoPROC_UseMakefileForSpeedup=yes (default=no). * Added tools for more detailed timing of autoPROC executions: this is triggered by setting environmental variables autoPROC_DetailedTiming=yes and having autoPROC_DetailedTimingFile to a file location. * Added support for anomalous ellipsoidal completeness as a funciton of image number in MRFANA. * Added parameter FindImages_ExcludeFirstImage (default=no) similar to already existing FindImages_ExcludeLastImage (default=no) parameter. This can be used for beamlines with known synchronisation issues for the first or last image in a series. * Added support for HDF5 (Eiger) datasets stored as *_master.h5 files within a directory: "process -I /some/directory" should now work in those cases too. This is based on adding the flag '-h5' to the "find_images" tool, which will switch from searching for separate image files to searching for *_master.h5 files. * Added Eiger 9M and 16M module gap definitions. Usually, the virtual pixels of module gaps in Eiger (or Pilatus) detectors should already be marked in the image array by a negative value. Therefore, an explicit specification of those module gaps should never be necessary. However, that automatic marking (by the detector firmware) could be inactive by mistake and an explicit masking is required. This can be requested by setting the corresponding value autoPROC_Img2Xds_Pilatus6MIncludeUntrustedRectangle autoPROC_Img2Xds_Pilatus2MIncludeUntrustedRectangle autoPROC_Img2Xds_Eiger9MIncludeUntrustedRectangle autoPROC_Img2Xds_Eiger16MIncludeUntrustedRectangle to "yes" (default=no). The gap definitions are stored in parameters autoPROC_Img2Xds_Eiger16MGaps etc. The parameter autoPROC_Img2Xds_GapsAdd (default=0) can be used to add padding around these module gap definitions - in case the bordering pixels (often special "virtual" pixels) should also be excluded. * Support in 'imginfo' and 'GPX2' for mini-CBF images that have an unspecified convention in the headers. * Setting autoPROC_ScaleEarlyLateCreate=no as default if running XSCALE path (this avoids warning messages since that feature is not available in the XSCALE path). * Better handling of MTZ2SCA (CCP4 program) failures. With the parameter autoPROC_AnaStaraniso_StopOnMtz2scaError (default=no) this will not trigger an error now. * Automatically adjust the width for the plot of indexed spots as a function of image number (useful to detect multiple lattices, poorly centred crystal, anisotropy or radiation damage) to ensure that even for large datasets it doesn't become too crowded. * Give some warning messages if run in non-default mode (e.g. switching off automatic ice-ring handling or skipping automatic adjustment of parameters for beamstop masking in DEFPIX). Such runs can be required due to local limitations of computing resources or time, but it should be clear in the final output that the data processing might be suboptimal as a result. * Added DistributeBackgroundImagesForHdf5 (default=yes) parameter to also use distributed images for background estimation when handling Eiger/HDF5 files. This requires extracting those files via hdf2mini-cbf. * Added reporting of program versions into summary.html as well as standard output (to simplify support). * Work with XDS developers to stabilise autoPROC/XDS when wrong SG/cell etc causes XDS/INTEGRATE jobs stalling (thanks to Herman Schreuder for contributing to this disucssion). See http://xds.mpimf-heidelberg.mpg.de/html_doc/Release_Notes.html namely Change of the refinement protocol and of defaults for the refined parameters at various stages of processing to reduce the risk of divergence of the procedure. * Fixes to determination of F(early) and F(late) datasets during analysis of incremental completeness as a function of image number. * Fixed specification of image subsets for HDF5/Eiger datasets specified by the -Id flag: previously this had no effect. * Fixes to extraction of all referenced files from summary.html (to include into summary.tar.gz file). Also increased autoPROC_CreateSummaryTarballFileMaxSize from 5 to 10 MB (the size of individual files within the tarball). * Fixes to running in current directory (for single or multi-sweep datasets): this should hopefully cover all situations, but please report any problems. * Fixes in aP_scale module when handling really large, multi-sweep jobs (whereby AIMLESS puts the name of reflecion files onto separate lines). * Fixes in computing statistics after STARANISO treatment when we have multi-sweep, multi-wvl data. * Fixes in equal-observation-number binning of MRFANA: - systematic absent reflections are now treated identically in binning setup and statistic computation - fix problem whereby rounding errors (for number of unique reflections in each bin) were pushed solely into the last bin * New features, fixes and improvements (released Jun 23 2017): * Fix in creation of XML file for multi-sweep datasets that are collected at a single wavelength. * New features, fixes and improvements (released June 20 2017): * Support for latest XDS (Version June 1, 2017, BUILT=20170615) - including a change to the default setting of MAXIMUM_NUMBER_OF_JOBS for this version: instead of leaving it at 0 (triggering usage of cluster nodes for processing) autoPROC defaults to a value of 1, i.e. running only on the current computer. To change this - depending on your local setup regarding cluster nodes - the user needs to explicitly set autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_JOBS to a value different from 1, e.g. process autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_JOBS=4 ... * Merging statistics for STARANISO-processed data now uses equal-observation-number binning by default. This will make the resulting tables (as a function of resolution) more useful than the standard equal-volume binning used for isotropic data analysis. * Take off trailing / from value to -d/-I argument to avoid potential issues with links in summary.html (thanks to R. Boer for reporting this). * Create XML report for data after anisotropy analysis with STARANISO. This file might not be completely ISPyB compatible (it contains new items related to the anisotropy analysis performed by STARANISO), but is intended as a starting point for discussions regarding description and presentation of anisotropy in diffraction data and processing. The original (isotropic) data analysis is still represented by the original autoPROC.xml file, while the new files are typically called autoPROC_staraniso.xml. * Fixes to generation of prediction images when processing compressed images. * Fixed missing labels for R{merge,pim,meas}-vs-batch plots. * Fixed odd/confusing text shown on standard output for STARANISO related notes. * Fixed misleading error/warning from "process -checkdeps" when 'montage' (from Imagemagick) was not available. * New features, fixes and improvements (released May 05 2017): * Create a file with REMARK 200 content related to the STARANISO MTZ file produced by the AIMLESS scaling path, to have an exact parallel to REMARK 200 file associated to the TRUNCATE MTZ file. * Give "table-1" like summary for each output MTZ file at end of autoPROC run (bothin in stdout and summary.html as well as a file with ending "*.table1"): this should make it clearer what statistics are relevant for which MTZ file. * Avoid writing empty shell information in MRFANA analysis when those shells are bordering high-resolution limit. * Added support for ALS 4.2.2 detector RDI1 (CMOS_8M). * Support in 'imginfo' and 'GPX2' for intermediate CBF files from Eiger data (i.e. only those CBF files are available, although they were generated by a tool intended for on-the-fly conversion as apart of XDS processing). * Take care of correct permissions during installation (especially for Ruby installation). * Some clarifications in output from aP_check (ie 'process -checkdeps'). * Added tool 'xparm_transform' to transform XPARM.XDS files so that either the rotation axis is along a lab frame axis and/or the detector X-axis is. * Added tools 'ana_cmpmat', 'aP_rotcal' and 'aP_wf_rotcal' to distribution: mainly useful for rotational calibration work. * For cases where image headers are not recording the exact settings of goniostat angles (Omega, Kappa/Chi, Phi), allow override via goniostat_{Omega,Kappa,Chi,Phi}_{axis,angle}_ parameters. * Simplified option to use all recorded images for background estimation (e.g. when shadowing needs to be taken into account as well): setting XdsNumImagesBackgroundRange to "All" will do that. * Update to support latest Diamond/I23 instrument (Kappa goniostat). * Improved generation of STARANISO plots (reciprocal planes) with better legend and represenation of reciprocal cell. * Support for writing STARANISO-related information (analysis of anisotropy) as well into XML file(s). * Update to support latest CCP4 (version/update 7.0.036, AIMLESS 0.5.32) and XDS (BUILT=20170215). * Added support for spherical/ellipsoidal statistics via MRFANA: the description of ellipsoid needs to be given via the '-ell' command-line argument. * Added option -L to 'find_images' tool to get arguments in form suitable for -Id flag of 'process' directly (reported by D. Reinert). * Added warning messages in case prediction pictures (via GPX2) could not be generated: this usually points to issues with headless cluster nodes. See also https://www.globalphasing.com/autoproc/wiki/index.cgi?FaqUsingGpx2OnHeadLessBox. * MRFANA now also computes the ellipsoidal completeness in resolution shells. This is the completeness within the ellipsoid fitted by STARANISO (to describe the anisotropy of diffraction within the data) for a given resolution range. * Some fixes to the computation of statistics within MRFANA that would have given small differences when comparing to other programs - mainly due to the inclusion of systematically absent reflections in those statistics and some inaccuracies when handling reflections at the border of resolution shells. * Fix when handling sparse (or very finely sliced) multi-sweep datasets: the unique renumbering of all datasets got confused when some images had no reflection centroids assigned to them. * New features, fixes and improvements (released Dec 07 2016): * GPX2: automatically detect and load image files compressed with either gzip for bzip2. Also, for miniCBF files protect against missing value of Polarization. * Fix to the plot of misfits as function of detector position: now also has correct aspect ratio of detector dimensions. * Fixed documentation in $autoPROC_home/docs/autoproc/manual/appendix1.html of AutoProcScaleStatsUseMrfanaIgnoreIceRingShells (reported by J. Benach). * Fixed problem with processing compressed (.gz/.bz2) images when using a subset of images for background estimation. * Updated to support XDS Nov 1, 2016 (BUILT=20161101), pointless 1.10.27, aimless 0.5.29 and CCP4 7.0.022. * Implemented workaround for problems in integrating weak data with XDS Nov 1, 2016 (BUILT=20161101): at least for Pilatus detectors setting GAIN=1 in XDS.INP seems to fix this. The problem manifests itself as a bias towards a non-zero intensity (we have seen values of +0.5 and -0.5) at high resolution when there really is no diffraction present. This has been reported to the XDS developers and the above workaround was suggested by K. Diederichs. * Added additional high resolution ice-ring values based on analysis of high-resolution datasets and comparing to [1]: 1.524 - 1.519 1.473 - 1.470 1.444 - 1.440 1.372 - 1.368 1.367 - 1.363 1.299 - 1.296 1.275 - 1.274 1.261 - 1.259 1.224 - 1.222 1.171 - 1.168 1.124 - 1.122 [1] Kumai, M. (1967). A study of hexagonal and cubic ice at low temperatures. Research Report 231, U.S. Army Materiel Command, Cold Regions Research & Engineering Laboratory, Hanover, NH. * Initial support for CBF files with imgCIF image headers (tested on Bruker Photon-II data; thanks to G. Bader, M. Benning and D. Reinert). * New features, fixes and improvements (released Jul 08 2016): * Fixed problem with PDF-report generation when running single-sweep dataset without using the -d flag: it was stopping with an error message (and exit status) at the last step of an autoPROC job (reported by D. Reinert). * Introduced the new parameter autoPROC_CreateReportStopOnError (default="no") in order to not stop with an error if the PDF-report generation fails. Since this is a still a very new feature we don't want any problems to upset users automated pipelines (or our own Pipedream software). * Added (optional) auto-refresh function to the generated summary.html file that is active as long as it is still written to (suggested by P.Lukat). This feature can be activated by setting the parameter autoPROC_CreateSummaryUseJavaScriptAutoRefresh (refresh rate in seconds) to a value greater than 0. * New features, fixes and improvements (released Jul 06 2016): * Change of defaults to AutoProcScale_RunStaraniso=yes => run STARANISO for each dataset autoPROC_CreateXml=yes => create ISPyB-compatible XML file for each merged dataset autoPROC_ScaleEarlyLateCreate=yes => create (if possible) amplitudes for F(early)-F(late) difference Fourier maps: these can then be computed automatically in BUSTER (setting parameter BusterEarlyLate=yes) to provide help in assessing the effects of radiation damage * First implementation of a PDF report generated for each merged datasets: the layout and content is a first design - any comments are very much welcomed. * Added support to hdf2mini-cbf (converter of Eiger HDF5 datasets into mini-cbf) for automatic detection of 32- versus 16-bit images. It now also supports GZIP compression as available as part of the HDF5 library (see: https://www.hdfgroup.org/HDF5/faq/compression.html). * Added further support to HDF5 metadata reader (in "hdf2mini-cbf" as well as "imginfo") for * reading "units" attribute for data items to ensure they are consistent with expectations; * check that sensor_thickness as written in HDF5 metadata file (*_master.h5) makes sense: if a value >= 320 mm is encountered, it will be reset to 1/1000th of this value. This works around a bug in images from * Eiger 9M at APS 21-ID-D (as of 20160426) * Eiger 9M at Soleil Proxima2 (known to be present between 20160131 to at least 20160415) * ensure that metadata arrays are read even if the apparent number of images and the number of items in those data arrays don't match. This works around an inconsistency found in images from * Eiger 9M at Soleil Proxima2 (20160415) * Added automatic analysis of statistics to give indication of likely high-resolution limit of anomalous signal. This should help users in getting an initial, rough estimate of anomalous signal. The parameter autoPROC_AnomalousSignal_AdjustAutomatically (default=yes) determines if subsequent stages of data processing should take a detected (or determined as being absent) anomalous signal into account. If the user runs with -ANO or -noANO command-line flags, that decision will always be used and no adaptation is done. * "imginfo" will now write rotation values (angles) with greater precision - required e.g. for extremely fine-sliced Eiger datasets. * Ensure also '<' and '>' characters render correctly in summary.html. * Newer AIMLESS versions (0.5.24 and later) have a changed default for the SDCORRECTION card (SAME instead of INDIVIDUAL) that requires a switch in logic for our aP_scale scaling module. * A table of statistics against resolution (in 20 bins, can be changed via the AutoProcScaleStatsUseMrfanaFinalNumBin parameter) is now also given for the AIMLESS scaling steps. * Added support for 'May 1, 2016' version of XDS. Please be aware that official XDS binaries after 20151015 up to 20160514 are known to show an issue in indexing (IDXREF): indexing might fail or produce incorrect solutions in a seemingly random way without any obvious reason. An updated/fixed binary (for academic users) has been provided by the XDS developers as of 20160517. If you can't update to this fixed binary, this problem can sometimes be averted by giving a slightly different direct beam position (changed by very small values - like 0.1 pixel). * Started adding links to autoPROC manual directly from the summary.html file (more will follow). * To avoid too much information (especially for multi-sweep datasets), a hide/show feature has been added to summary.html: this means some plots are initially hidden and can be show through simple clicks on a green "+" icon. * Plots for scaling statistics (Rmerge, completeness etc) are now placed after the relevant tables in summary.html. * Several fixes to the automatic high-resolution limit estimation via MRFANA: this should stabilise this feature in case of multi-sweep datasets. * Added new parameters to listing in Appendix 1 of manual (AutoProcScale_RunStaraniso AutoProcScale_RunStaranisoCards AutoProcScale_RunStaranisoHklPlotGrid AutoProcScale_RunStaranisoPqrPlotGrid AutoProcScale_RunStaranisoStopOnError RunIdxrefStopOnAnyError XdsSpotSearchNumImagesAngularRange XdsUpdateAlsoSegments). * Fixes to regression described at http://www.globalphasing.com/autoproc/wiki/index.cgi?IssuesPage201603#refinekwd. * Now should also work if reference MTZ file is in a directory without write permissions. * Removed out-of-date and unnecessary macros: XdsFast and HighRes2. The "automatic" macro is deprecated and will be removed in a future release. * New features, fixes and improvements (released May 01 2016): * Added feature to run STARANISO directly as part of autoPROC: - STARANISO is a program developed by Ian Tickle at Global Phasing to analyse the local mean I/sigI to then apply an anisotropic cut-off and an anisotropy correction on merged intensity data. - This is currently only implemented for the AIMLESS scaling paths (i.e. the default behaviour or when running with the -M ScalingA3 macro) and can be activated with the AutoProcScale_RunStaraniso=yes command-line argument. - Data scaled in the normal way (but without applying the isotropic high-resolution limit determined) are given to STARANISO as input. - All analysis resulting from STARANISO is presented in the summary.html file, including plots showing the (potentially) anisotropic nature of diffraction. - The reflection file after applying this anisotropic limit and anisotropy correction can be used as an alternative to the standard autoPROC output MTZ file for subsequent stages, e.g. model refinement with BUSTER. - For more background information, details and examples please visit the STARANISO server at http://staraniso.globalphasing.org/. * fixed problem in extracting Rmerge for REMARK 200 section in remark200.pdb when using XSCALE path ("-M ScalingX" macro) * fixes to formatting of summary.html (to avoid overflow in menu area and correct Angstroem symbol in text) =================================================================== Changes included in autoPROC 1.0.4 stable Mar 24 2016 =================================================================== * Fixes and improvements (released Mar 24 2016): * hdf2mini-cbf now also reports usage help when run without argument (reported by O. Svensson, ESRF). * Fixed note written by aP_convert_hdf5 regarding number of external link files found. * Added (currently) necessary imginfo wrappers for processing of Eiger/HDF5 datasets from following beamlines: Eiger 4M at ESRF, MASSIF-3 Eiger 9M at APS 21-ID-D Eiger 9M at Soleil Proxima2 Eiger 4M at Photon Factory BL1A For details and updates, please also check http://www.globalphasing.com/autoproc/wiki/index.cgi?DataProcessingHdf5 * A new parameter can be used to simplify masking of damaged or hot pixels (if those are not yet accommodated directly by the detector via an updated pixel-mask): autoPROC_Img2Xds_DamagedPixels can be set to a space-separated list of X,Y pairs a la process autoPROC_Img2Xds_DamagedPixels="1234,432 3210,2345" ... Note: pixel indexes start at (1,1) as shown by our GPX2 viewer. Other viewers might show the first pixel as (0,0) and values read off those need to be adjusted by +1. * Fixes and improvements (released Mar 04 2016): * Fixes to generation of XML file when running "process" without the -d command-line argument (reported by L. Muir, IMCA-CAT). * Avoid attempt to create prediction pictures when processing HDF5 (Eiger/Dectris) datasets: GPX2 does not yet support reading HDF5 format. This removes an irritating (and unnecessary) error message from standard output (reported by D. Reinert, BI). * Fixes and improvements (released Mar 02 2016): * "process -checkdeps" was warning about "staraniso" not being present when it should have ignored this (thanks to K. Bataille from IMCA-CAT for pointing this out). There will probably be a feature in autoPROC to run STARANISO directly as part of processing, but for the moment the recommended usage of STARANISO is to use the web service at http://staraniso.globalphasnig.org/. For that purpose a script (aimless_alldata.sh) is automatically generated within the output directory that can be used to create a reflection MTZ file adequate for the web server. * Fixed messages from HDF5 converter script (aP_convert_hdf5). * Fixes and improvements (released Feb 25 2016): * Added support for HDF5 datasets from Dectris/Eiger detector. * HDF5 datasets can be given to process via the new -h5 command-line argument or as part of the existing -Id flag. * The tool "hdf2mini-cbf" is used for extracting separate image files out of the HDF5 container. It can do this for a single image (emulating the H5ToXds tool from Dectris) or for a series of images. The output CBF images contain a fully populated mini-cbf header and can also be written directly as compressed *.gz files. Conversion and writing is parallelised. Handling of HDF5 files written with the bitshuffle compression is supported on both Linux and OsX. A utility (aP_convert_hdf5) is provided for simple and fast (e.g. by submission to a cluster of workstations) data image extraction and conversion. * GPX2: emit help text when an unknown option is met. * Fixed regression (introduced in snapshot release 20151125) resulting in unnecessary runs of INTEGRATE and CORRECT steps at first XDS invocation. * Consolidated behaviour and resulting file names in summary HTML and XML files under different run conditions. These include running with or without a -d argument and - if a -d argument is given - if it is an absolute or relative path or points outside of the current directory. There could still be unresolved issues when it comes to working in auto-mounted filesystems or when using symbolic links - in which cases 'pwd' and 'pwd -P' might return different results. * Fixes and improvements (released Dec 14 2015): * New feature to create early/late data values: * This is switched on by the new parameter autoPROC_ScaleEarlyLateCreate (default=no). * Currently this is only supported for AIMLESS scaling (default or ScalingA3 path) and single wavelength datasets. * The final MTZ file (truncate-unique.mtz etc) will contain F/SIGF columns with the extension "_early" and "_late". * For the definition of early and late datasets we use several points in the forward/backward completeness table. One point is of course the cross-over point, ie. where the two completeness curves cross over each other: we have to stay on either side of this. Optimal points for merging early/late images is then found by using the following parameters: * we want to have at least this completeness: autoPROC_EarlyLateCompletenessMin=0.85; * we stop once we reached that completeness: autoPROC_EarlyLateCompletenessMax=0.95; This means means we are now doing the early/late analysis a bit different to the first implementation in the previous snapshot release - where it was used only for marking the completeness-versus-batch plot. * the early/late border will not be shown if individual completeness is too low; * no need to divide all images into early/late if we can use much fewer images on either end with good completeness; Note: it is likely that the exact method for defining early/late datasets will change in future releases (additional criteria need to be considered). * Added support for RIGAKU/Eiger detectors via the new "RigakuEiger1M" macro. * Fixed regression: GPX2 was not packed up into the OSX tarball for the previous snapshot release. * Fixed generation of summary.html when job stops with an error message (reported by D. Reinert). * Improved writing of XML file: * new parameters autoPROC_CreateXml_ValTrue (default=1) and autoPROC_CreateXml_ValFalse (default=0) allow for different boolean types in local XML usage (True/1/true or False/0/false etc); * autoPROC.xml will be initialised with processingStatus False and only upon successful completion will we have True; * added support to inject externally defined XML elements via one of autoPROC_CreateXml_LocalElements="AutoProcContainer:AutoProcScalingContainer:AutoProcIntegrationContainer:Image:datasetID=12345" or (if value will contain spaces): autoPROC_CreateXml_LocalElements="AutoProcContainer:AutoProcScalingContainer:AutoProcIntegrationContainer:Image:datasetID" AutoProcContainer_AutoProcScalingContainer_AutoProcIntegrationContainer_Image_datasetID="12345" * Reworked feature to automatically select a subset of images for spot searching: the previous incarnation didn't really work. We now have * maximum number of images to use for spot-searching: XdsSpotSearchNumImages="999999"; * the images used for spot searching should cover this angular range: XdsSpotSearchNumImagesAngularRange="360"; * number of segments to use within XdsSpotSearchNumImagesAngularRange: XdsSpotSearchNumRanges="10"; * only images within this angular range (from first image) are considered: XdsSpotSearchAngularRange="360"; To use only a subset of images one could set XdsSpotSearchNumImagesAngularRange="10.0" XdsSpotSearchNumRanges=4 XdsSpotSearchAngularRange=180 CAVEAT: Please be aware that this might significantly hamper autoPROC's ability to detect multiple lattices and ice-rings (and take corrective measures). By using such settings the user might not become aware of serious problems with the dataset. * Allow the possibility to define pre-computed files (like {X,Y}-CORRECTIONS.cbf) to be reused even if they have an arbitrary filename. * Added new autoPROC_CreateGpxPicturesAtRotationAngles parameter (default="0 30 60 90") to define angles for GPX2/prediction pictures relative to first image. * Improved handling of reindexing requirement for multi-sweep datasets - when the SG symmetry allows this. This includes additional checks when running with or without a reference MTZ file and with EnsureConsistentIndexing=yes (default) or EnsureConsistentIndexing=no (non-default). * Corrections to handling of centro-symmetric spacegroups in MRFANA. * Ensure "xparm2simin" can be run stand-alone for multi-axis goniostats (as defined in our distributed $autoPROC_home/autoPROC/lib/kapparot.sites file) * The FRAME.cbf file written by COLSPOT is now saved as FRAME_COLSPOT.cbf: this can show some details about the spot-searching in XDS/COLSPOT. * Changed behaviour of aP_scale scaling module (invisible unless run explicitly with "aP_scale -id 0"): * by default output will have no prefix (before: "1_") * therefore there is no need to use "-id 0" in order to switch from "1_" to "" prefix. The value given with "-id" is now used as-is to generate "_" prefix for all output files; * Fixes and improvements (released Nov 25 2015): * Added support for writing of a HTML page (default = "summary.html" in the output directory) containing information about autoPROC run. This should simplify inspection of all the graphical output and plots produced. * it involves renaming some files or creating different file names (e.g. SPOT_never-indexed.XDS) in order to avoid the overwriting of files at different stages - which would confuse the HTML output; * a tarball is created (default = "summary.tar.gz" in the output directory) that will contain this HTML document as well as most files (plots etc) referenced therein; * Added support for XDS version "Oct 15, 2015", CCP4 6.5.018, POINTLESS 1.10.10 and AIMLESS 0.5.15. * Added support for the long-wavelength beamline I23 at the Diamond Light Source. * Added support for writing XML file(s): * triggered by command-line option "-xml"; * standard file name "autoPROC.xml" (for single wavelength data) and "autoPROC_.xml" for multi-wavelength data, all written into the output directory; * it follows (and expands slightly) the ISPyB examples as far as these are documented; * Changed default for GPX2 wrapper script (gpx.sh): after closing GPX2 the predictions will be removed. In order to keep them, the command-line argument "-keep" can be used. * Fixed bug in GPX2 wrapper script (gpx.sh): for very high-resolution and/or large unit cell datasets the prediction generation could have stop with a segmentation fault because the increase in stacksize limit was actually ignored. * Speedup generation of predictions with simcal_predict for large structures by enlarging the size of hash-tables. * Made xds_spots2res work for curved detectors by using anaspot to compute resolution of spots. * At the beginning of output the installation directory for autoPROC is now also shown. * Scaling module aP_scale: * new tool ana_mrfana gives more detailed plots as a function of resolution or image number (i.e. batch or dataset); * improved handling of data without anomalous data (-noANO flag to process or aP_scale): now also taken into account for XSCALE path and the potential conversion into *.sca files; * fix logic to avoid running MRFANA twice in all cases (even if not necessary); * Fixes in MRFANA regarding computations for centrosymmetric space groups which previously caused the printing of NaN. * A warning message will be displayed when using the MOSFLM path of autoPROC: this non-default option has not been developed or tested for quite some time and will be removed in one of the next releases. * The SPOT.XDS_orig file (original list of spots before doing any indexing) now doesn't contain any Miller indices - while SPOT.XDS will contain non-(0,0,0) indices for spots from the finally used indexing solution. * Added checks for consistent distance, wavelength, oscillation, 2-theta, chi, kappa and rotation angle within a set of images making up a sweep. * Improved handling of multi-angle goniostat values when dealing with data from such instruments (and image headers that contain those settings). * When giving only the space group name/number to process: add extension to iterative indexing output files for the two cycles that need to be run in such cases. * Skip initial IDXREF step if we are doing iterative-indexing anyway. * Better phrasing when going into iterative indexing to describe accurately why this step is performed. * Added fully functional Gnuplot 4.6.5 to distribution to package: this is intended only for running from within autoPROC jobs. A user can still provide their own (e.g. OS-provided) binary by setting it explicitly on the command-line, e.g. process gnuplot=/where/ever/bin/gnuplot ... * Added new parameter autoPROC_XdsMaxDistanceJitterWarning (default=0.2) to give a warning if the distance refinement results in results outside this range (relative to the mean value). * the parameter autoPROC_XdsMaxDistanceJitter (default = 1.0) is used to automatically trigger fixing of detector position refinement if needed; * parameter refinement resulting in the crystal-detector distance changing by more than 0.2 mm seems too large compared to the typical size of a crystal (on most beamlines the crystal would move out of the beam if it represented physical reality); * large changes in distance during parameter refinement mostly point to problematic diffraction patterns or poor spot shapes; * Changed the LowResOrTricky macro so that an additional iterative indexing is done after processing each sweep: this can highlight the exact nature and relation of multiple lattices better since a set of refined parameters (for describing the experiment) is now available. * A new macro (HighResCutOnCChalf) is included to base automatic high-resolution limit decision making solely on CC(1/2) value of 0.3. * A new macro (DiamondI04) for beamline I04 at Diamond is included. * At the start of processing (of each sweep/dataset) a short summary is given to show the information as read from the header of the first image. * Accommodate memory issues (either due to little physical memory or because of multiple jobs running simultaneously) when creating prediction pictures: give a warning message but don't make the full autoPROC job fail. * Fixes and improvements (released Sep 23 2015): * Introduced new parameter autoPROC_PointlessTolerance (default = 2.0): identical to parameter TOLERANCE as defined and used in POINTLESS. Increase this if the reference dataset is expected to deviate significantly from the dataset being processed. * Plots of misfits and spots now follow the usual image-viewing convention to be in sync with GPX2: origin is in top left corner. * Fix in GPX2-driving script (gpx.sh) regarding prediction file: ensuring it is removed after automatic picture generation and this removal is noticed when running the script interactively for manual inspection. * Better handling of inhouse (CuKa-wavelength) datasets when generating input to simcal_predict: if wavelength is between 1.5417 and 1.5419 A the polarisation defaults to an unpolarized beam. * Several fixes in MRFANA to stabilise the automatic decision making about an adequate high-resolution limit. This includes the use of a Bezier curve as a basis for testing the various cutoff-criteria. * Additional fixes in MRFANA regarding computation of completeness (especially anomalous completeness). * Fixed regression from previous snapshot (internal function aP_pre not being defined). * Fixes and improvements (released Aug 12 2015): * The computation of resolution within MRFANA has changed for MTZ files: instead of using the overall cell parameters the dataset-specific cell parameters are used now. This will have an effect on multi-sweep datasets, where each dataset comes with its own, slightly different set of cell parameters. It is then important to use the correct cell parameter for resolution computation in order to be consistent between stages (where we see only a single sweep or multiple sweeps simultaneously). * We now ensure that R-values in MRFANA (Rmerge, Rmeas and Rpim) are computed in the same way as in SCALA/AIMLESS. Please note that this is following the paper (Evans, P. (2006). Acta D62, 72-82), which is slightly different from others (Diederichs&Karplus/1997 and Weiss/1997) in the denominator. Additional information is given in the MRFANA output. * Added a table of statistics as a function of image/BATCH number in MRFANA. * Added more comments into various macro files to help user understanding the effect of each setting. * Fixed problem in analysis of iterative-indexing run (where failing IDXREF results were not excluded correctly and therefore sometime reported as a reference for the angle computation). * Fixed fetching of images when the user supplied an explicit image range (via the -Id flag): now only images within that range are used while before all images matching the given template were used. This was especially confusing during the GPX2 visualisation, since more images might have been presented than predictions were generated (if a subset of images was specified on the -Id flag). * Fixes and improvements (released Jul 15 2015): * GPX2 (prediction and image visualiser): * now also available for OS X (version 10.7 or higher) * introduced new GPX2 keyboard shortcuts to simplify interaction: Ctrl-Left : move to previous image Ctrl-Right : move to next image Ctrl-q : quit program * Ensure that the file pointed to by the autoPROC_PointlessExtra parameter is also included when running POINTLESS to check for consistent indexing against a user-supplied reference MTZ file. * Clarifications to the reporting of multiple indexing solutions (when entering iterative indexing). * Fixes to the reporting of minimum angle between multiple indexing solutions (and also mentioning the space group used at this point to determine the minimum angle). * Added parameter autoPROC_CreateGpxPicturesAndKeepFilterFile (default="no") to decide if we should keep the prediction files that are computed during generation of images showing diffraction and predictions. Since those files can become rather large, not keeping them seems a sensible default option. * Use a generic gpx.sh script (to visualise predictions with simcal_predict and GPX2) instead of generating one on-the-fly: this simplifies maintenance and allows introduction of new features. * the help messages (.../gpx.sh -h) will now also report the possible lattices (identifier and number of spots used) when this script is invoked in multi-lattice cases; * in multi-lattice cases, GPX2 will by default (i.e. when no explicit "-lat" argument is used) start with all lattices shown; * the "-lat" argument now also supports the use of "-lat all" to select all lattices in a simple way; * Added BeamCentreFrom=header:x,y to SlsPXIII macro to avoid unnecessary warning message (thanks to D. Reinert for pointing this out). * NEW FEATURE: incorporation of a visualisation capability for images and predictions (released Jun 16 2015): * The computation of the positions and shapes of reflections expected to appear on each image is performed in the newly added program simcal_predict. This tool is a particular use-mode of the simcal diffraction image simulation program written by Wlodek Paciorek and Claus Flensburg and extended as part of collaboration contract COL0044 between Global Phasing and the Diamond Light Source, specifically related to the I23 beamline. * Predicted spot positions and shapes are presented through ellipses superposed onto diffraction images using the new program GPX2, which was initially developed by Wolfgang Brehm (while in the group of Kay Diederichs at University of Konstanz/DE) during an internship with Global Phasing during the Summer of 2014 with generous support by the MX group (particularly PX-III) of the Swiss Light Source. At the moment, GPX2 is provided for Linux - a version of OS X is planned. * NEW FEATURE: new scaling options for using "pure" AIMLESS-only or XSCALE-only paths within autoPROC (released Jun 16 2015): * the AIMLESS-only scaling path can be selected via process -M ScalingA3 ... * the XSCALE-only scaling path is chosen with process -M ScalingX ... * the hybrid path (XDS/AIMLESS) is still the default * Updated documentation (released Jun 16 2015): * new sections about simcal_predict (computation of predicted positions and shapes) and GPX2 (display of those on top of the raw diffraction images) * new section about scaling modes implemented in scaling module aP_scale * new section "Interpreting autoPROC output" * enhanced examples and usage section and major reworking in all other parts * Changed defaults (released Jun 16 2015): * The diffraction image files will not be linked into the same directory the actual XDS processing is run, but into a subdirectory (autoPROC_LinkImagesIntoSubdir parameter, default = "Images"). * Ice-rings are now excluded automatically (if detected in the list of un-indexed spots): parameter XdsExcludeIceRingsAutomatically defaults to "yes". The width of the resolution range given to XDS for exclusion is enlarged slightly (parameter Xds_Spot2Res_IceRingWidthMultForXds with default 1.2). * Create pictures of diffraction images with predictions using GPX2 (parameter autoPROC_CreateGpxPictures defaults to "yes"). The creation of files for visualising with MOSFLM ("ipmosflm") is now disabled (parameter autoPROC_CreateMosflmScript="no"). * the parameters for divergence and mosaicity as determined by the XDS INTEGRATE step are reused in any subsequent integration pass (parameter XdsUpdateParKeepFixed="BEAM_DIVERGENCE:yes REFLECTING_RANGE:yes"). * Instead of using REBATCH and SORTMTZ we are now using our own programs MRFEDIT and MRFADD (parameter UseMrfadd=yes and UseMrfedit=yes). These should not suffer from the same size/array limitations. * a macro to switch back to previous defaults is provided: process -M Version103 ... * the "automatic" macro has been adapted accordingly * Clarified some command-line options in online-help given by "process -h" (released Jun 16 2015): * "-M" can also take a file name (and not only the name of a macro); therefore the "-D" flag is being deprecated since a duplication of that functionality. * the "-P" flag is deprecated (the MOSFLM path hasn't seen development for quite some time) * Support for (released Jun 16 2015): - CCP4 6.5 (update 011) - POINTLESS 1.9.31 - AIMLESS 0.5.9 * Removed "AutomaticChunking=yes" from "automatic" macro (released Jun 16 2015): if that feature is required it needs to be added manually as in process ... -M automatic AutomaticChunking=yes ... * Added a little jiffy to create image sets based on rotation angle instead of image file name: ln_images.sh (released Jun 16 2015). * Support for ALS beamline 4.2.2 including NOIR-1 detector (released Jun 16 2015). * Scaling module aP_scale (released Jun 16 2015): * resolution limits along reciprocal cell axes as reported by AIMLESS now uses the same criteria for CC(1/2) and I/sigI as the overall isotropic high-resolution limit. * NEW FEATURE: support for scaling path using XSCALE; one (or several) XDS_ASCII.HKL files can be given via the "-hkl" argument. * fixed erroneous halt when encountering -mtz :

,, syntax - found by D. Reinert * fixed logic for determining number of processors (to then use parallel AIMLESS feature) when running aP_scale standalone * some clean-up in the additional examples provided within the aP_scale.sh script generated automatically as an example for fine-tuning by the users * when re-running the scaling module aP_scale with the same identifier (prefix for output files): give more meaningful text so user knows better what to do next (thanks to S. Sheriff). * The SPOT.*.png files produced during the indexing stage (showing location of indexed and non-indexed spots on the detector surface) now use a variable radius for points depending on spot strength (released Jun 16 2015). * The remark200 tool will now also write versioning information for autoPROC and XSCALE into remark200.pdb file (released Jun 16 2015). * Added support to combine XDS INTEGRATE.HKL files via 'combine_files' tool (released Jun 16 2015). * When entering iterative indexing, also report angle between orientation matrices of each solution relative to the strongest one, ie the one with the most indexed spots (released Jun 16 2015). * Don't do detailed overload analysis by default: it takes time and will create more large files. This is under parameter control with autoPROC_XdsAnalyseOverloads=no being the default (released Jun 16 2015). * Added check for user-supplied -Id flags to 'process' command: ensure the first item (identifier) is unique (released Jun 16 2015). * Fixed corner case in MRFANA: if the input MTZ file contained batches outside of the ranges defined on RUN cards, an error was thrown. Since those batches will never be used in later analysis anyway we now report and skip them (released Jun 16 2015). * Also output CC(ano) (correlation of anomalous differences between random half-sets) into 'table 1' in MRFANA (released Jun 16 2015). * Fixed handling of slightly differing wavelength values (from BATCH header and/or dataset information) in our REBATCH-substitute program MRFEDIT (released Jun 16 2015). * Fixed spelling: parameter is called autoPROC_XdsMaxDistanceJitter and not autoPROC_XdsMaxDistanceChitter (released Jun 16 2015). * The need to obtain a separate licence key for each machine on which the software is to be run has been removed for academic users: a single valid licence key will now work on any number of machines (released Mar 16 2015). * Support for (released Mar 16 2015): - XDS version March 1, 2015 - CCP4 6.5 (including update 005): this includes all the change in logic due to the final digit in the reported version now corresponds to the update serial number - POINTLESS 1.9.27 - AIMLESS 0.5.4 * Read 'fraction of polarization' directly from Pilatus miniCBF-header if available (released Mar 16 2015). * Changes to MRFANA (released Mar 16 2015): * Major internal re-organisation of MRFANA code: this should not be visible to a user. * Implemented the computation of CC(DANO) between halfsets in MRFANA similar to AIMLESS. * MRFANA now writes a more compact version of the merging statistics table. The wide-format version can be produced with the "-wide-format" command-line argument. * Better detection/distinction between XDS_ASCII.HKL, INTEGRATE.HKL and unmerged XSCALE output files. * More detailed messages for POINTLESS and AIMLESS version checks during "checkdeps" stage (released Mar 16 2015). * Changes in scaling module aP_scale (released Mar 16 2015): * by default, each AIMLESS job will also write out the XMLOUT file (see http://www.ccp4.ac.uk/html/aimless.html#InputOutput). To switch this off set the parameter autoPROC_ScaleXmlout to "no". * New parameter autoPROC_ScaleWithAimlessRefineSdcorrSame (default="no") that allows to use the same SDCORRECTION values for all RUNs (subkeyword SAME on SDCORRECTION card) in AIMLESS path. * Changed setting AutomaticChunking_MaxRuns (default = 40) only for SCALA path. For the default AIMLESS path it is set to 9999. * Changed setting AutoProcScaleResolutionCompareDigits=3 to AutoProcScaleResolutionCompareDigits=0,2 to have different significance levels for low and high-resolution values when testing if convergence has been reached during scaling/merging with aP_scale. * Changed command-line argument in aP_scale from -reso to -R flag (more consistent with other programs/commands), but keep -reso as backwards compatibility (released Mar 16 2015). * Added check to -b flag: requires a range (-) pattern (otherwise stop with an error). * Try to catch an error in TRUNCATE (WIDTH OF BIN TOO LARGE) to then set the RANGES keyword explicitly (enforcing TRUNCATE to run through). * Create plots of Bdecay and Mn(k) as a function of image number as well. * Better explanation and additional checks for -x flag in aP_scale scaling module. * New parameter FindImages_MaxImageNumWidth (default = 6) to handle image file names where the prefix and the actual image serial number run into each other because of a missing delimiter character (released Mar 16 2015). * Added help regarding ice-ring exclusion to autoPROC reference card (released Mar 16 2015). * Added option to insert call to a program/script (parameter XdsRunScriptAfterColspot set to full path of this program) after COLSPOT (released Mar 16 2015). * New parameters UseKapparotForConsistentIndexingStopOnError (default = yes) and XdsGetRotationAxisViaKapparotStopOnError (default = yes) as a fallback to work with beamlines producing different image headers (e.g. with or without a mini-kappa goniostat installed). It should not be necessary to change these from default value (released Mar 16 2015). * Better error/warning message if image header is incomplete for consistent indexing between scans and/or when getting ROTATION_AXIS via Kapparot. Only relevant for multi-axis goniostat datasets (released Mar 16 2015). * anaspot: correct resolution computation for detector described with segments when the detector is not located at the origin (released Mar 16 2015). * If the new parameter XdsUpdateCorrectionTables (default = no) is set to yes, we will also update {X,Y}-CORRECTIONS.cbf with the G{X,Y}-CORRECTIONS.cbf files from CORRECT before re-running the integration step (released Mar 16 2015). * Nicer formatting of output from "imgdate.sh" jiffy: constant width for all file-names, dates and time-differences (released Mar 16 2015). * The analysis of and (possible) warning about image header inconsistencies is now under parameter control: autoPROC_ImginfoReportWarnings with a default of "yes" (released Mar 16 2015). * imginfo now also reads milliseconds from the timestamp in image headers if available (released Mar 16 2015). * Also handle runs where only the space group name is given (using the symm= argument) and we have to distinguish between hexagonal and rhombohedral settings of H3/H32 (released Mar 16 2015). * New parameter CombineFilesRunReindexOnAll in 'combine_files' utility (defaulting to yes): when running REINDEX on at least some MTZ files (to set correct space group), we also need to run it on all to ensure the SG operators are given in the same (CCP4/REINDEX) order. Otherwise SORTMTZ will get confused and stops with an error (because ISYM values would be wrong). This is due to POINTLESS using a different, clipper-based system of symmetry operators than other CCP4-based programs (released Mar 16 2015). * Also report ISa values after the XDS/CORRECT step (released Mar 16 2015). * The parameter XdsExcludeIceRingsAutomatically can now also be set to "all" - in which case all known ice-ring ranges will have their resolution ranges excluded in processing, no matter if those ice-rings are actually present/detectable or not (released Mar 16 2015). * Added more detailed notes and error messages during indexing stage to help spotting potential problems (released Mar 16 2015). * Renamed P13 and P14 beamline (PetraIII) macros to avoid warning message from macro reading routine (released Mar 16 2015). * Beamline macros for SLS (PX-III) and PetraIII (P13 and P14) will not set the silicon sensor thickness since that value is now read from the image header by imginfo (released Mar 16 2015). * improved 'ana_aimless' tool to deal with difference in table format between AIMLESS versions up to 0.1.32 and from 0.2.11. Also added a plot about cumulative normal and anomalous completeness and fixed handling of multi-wavelength (MAD) datasets (released Jul 26 2014). * improved the 'xds_stats' tool to give nicer plots (handling meaningful ranges, labels etc) and deal with old gnuplot 4.0 versions; non-square plots now default to XdsStats_size="800,600" (from previously "1200,800") to make them compatible with output from "ana_aimless" (released Jul 26 2014). * the 'imginfo' tool now also reads the sensor thickness (if given in mini-CBF header) from Pilatus-formatted images (released Jul 26 2014). * the 'rigaku2xds' tool (and therefore autoPROC) should now also support Pilatus 300K detectors on Rigaku (AFC-11) goniostats via the '-M RigakuPilatus300K' macro (released Jul 26 2014). * added support for P13 and P14 beamlines at Petra-III beamline via new macros (released Jul 26 2014). * fixed 'hkl2res.sh' jiffy when DETECTOR_DISTANCE is given on same line as ORGX/ORGY line (released Jul 26 2014). * fine-tuning the text when checking XDS version eg as part of the 'checkdeps' routine (released Jul 26 2014). * added jiffy ('imginfo_merge2cbf.sh') to process images that went through the XDS program MERGE2CBF (released Jul 26 2014). * changed default for range of images used in background determination: autoPROC will now use 3 ranges of 10 images each within the first 90 degree of data (released Jun 12 2014). * new macro (Version115) to move back to defaults for the previous stable version (released Jun 12 2014). * tested with latest POINTLESS version 1.9.8 (released Jun 12 2014). * give XDS references in standard output (released Jun 12 2014). * fixed problem when trying to print warning message (user requested more threads than available on machine, released Jun 12 2014). ------------------------------------------------------------------- <-- Below are the Release Notes for the previous stable release --> ------------------------------------------------------------------- =================================================================== Changes included in autoPROC version 1.0.3 stable May 12 2014 =================================================================== === === New features: === Installation instructions: * comprehensive guide how to install and test autoPROC on all major operating systems. These are available from the download page on www.globalphasing.com or included in each installation as $BDG_home/docs/installation/index.html (released May 07 2014). Parameter refinement: * when encountering instability of distance parameter refinement, detect and possible act upon it (released Mar 06 2014). If jitter is lower than half the autoPROC_XdsMaxDistanceChitter parameter value, keep it as a warning; if above this value: switch off distance refinement. Configuration: * a new tool ('aP_check') is available to test if all external dependencies (CCP4, XDS, POINTLESS, AIMLESS) are available and working. It can also be run via 'process -checkdeps' to give a short summary about those external dependencies (released Mar 06 2014). * the location of the XDS binary can be set via the environment variable BDG_TOOL_XDS (released Sep 13 2013). This is usually done in the files setup_local.sh and setup_local.csh located in $autoPROC_home (for a stand-alone installation) or $BDG_home (for all others). According to an existing general mechanism, this alternative way of specifying the location of the XDS binary can in turn be overridden on the command line (e.g. "process xds=/some/where/xds_par") or by a similar redefinition in a macro. External program support: * support for POINTLESS 1.92 as well as XDS "January 10, 2014 BUILT=20140307" (released May 07 2014). * support for CCP4 6.4.0 (including POINTLESS 1.8.17 and AIMLESS 0.2.17) as well as XDS "January 10, 2014 BUILT=20140211" (released Mar 06 2014). * support for POINTLESS 1.6.17, AIMLESS 0.1.24 and XDS "March 30, 2013 BUILT 20130706" (released Sep 13 2013). Simplifying support: * in case the user didn't save the standard output of the "process" or "aP_scale" command, we now report the filename where a copy of standard output was saved at the end of a run. These files are often the first information we might need to ask you for in order to help with user issues (released Sep 13 2013). Analysis: * the new parameter AutoProcScaleStatsUseMrfanaFinalNumBin (default = 20) is used at a final merging statistics calculation to use a coarser binning that is more useful for publications or deposition than the fine binning used for the automatic determination of the high-resolution cutoff (released Mar 06 2014). === === Changes: === Reciprocal space sampling: * reducing the values for parameters NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA and NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA from 21 to 15, since a too fine sampling doesn't seem necessary and this change reduces memory requirements (released May 07 2014). Iterative indexing: * the iterative indexing step will now output results into a subdirectory by default to reduce clutter in the main output directory (released Mar 06 2014). === === Improvements: === XDS support: * support of latest XDS version/build "January 10, 2014" (released Mar 06 2014). * better and more extensive checks on the XDS version (and build date) with improved feedback to the user (released Mar 06 2014). Beamline/Instrument support: * Support for Australian Synchrotron beamlines: new macros AustralianSyncMX1 and AustralianSyncMX2 (released Mar 06 2014). * Support for Soleil/PROXIMA1 Pilatus 6M images that mark the gaps not with -1 but leave them as 0 (released Mar 06 2014). The parameter autoPROC_Img2Xds_Pilatus6MGaps contains the gap specifications on a Pilatus 6M. With the new parameter autoPROC_Img2Xds_Pilatus6MIncludeUntrustedRectangle set to "yes" (default=no) those will be added to the list of UNTRUSTED_RECTANGLE. * improvements for non-square detectors (like the Dectris/Pilatus models) when the image header stores the beam centre in a non-standard way. The BeamCentreFrom parameter then needs to take this into account (released Mar 06 2014). On non-square detectors we swapped the (X,Y) beam-centre values and the NX/NY values at the same time. However, the NX/NY values are extremely unlikely to have been swapped by beamline software, since they relate directly to the fast/slow axis of the binary array. So now we actually calculate -y,-x = NX-y,NY-x as a default, while before we did -y,-x = NY-y,NX-x That latter calculation can also be used by setting the parameter BeamCentreFromNxNySwap to "yes". All this came to light from images at the Swiss-Norwegian beamline at the ESRF: a Pilatus 2M with the beam-centre stored as y,-x. Analysis: * MRFANA now also writes a dataset header for each merging statistics table: this helps identifying which images are included into the calculation of those statistics, what the average cell (and standard deviation) is etc (released Mar 06 2014). * calculation of <|DANO/sig(DANO)> in resolution shells added to MRFANA (released Mar 06 2014). * also write AIMLESS version into REMARK 200 section of remark20.pdb file (released Mar 06 2014). === === Fixes: === Scaling * when working with multiple scans that contain either more than 999 images or start with image numbers above 999: fix the assigning of the actual batch numbers for the final multi-sweep scaling run (released Mar 06 2014). General: * fix when updating XDS.INP file after POINTLESS analysis if the user gave an input spacegroup: the SPACE_GROUP_NUMBER keyword was lost (released Mar 06 2014). * fix in handling new versus old format of SPOT.XDS in our ANASPOT jiffy (released Mar 06 2014). * handle multi-axis Kappa goniostats better: sometimes those are described with 3 or 4 angles (including/excluding the Alpha angle) in the header (released Mar 06 2014). * if there are problems during parameter refinement in XDS/CORRECT: ensure that updating the XDS.INP and XPARM.XDS file takes this into account (released Mar 06 2014). =================================================================== Changes relative to 1.0.1 19 Jun 2013 =================================================================== === === New features: === Indexing: * When entering 'iterative indexing', also plot number of spots against images for all significant solutions (as fraction into run_idxref_spot_hkl_histfrc.png or as absolute values in run_idxref_spot_hkl_hist.png). A new parameter RunIdxref_SpotVersusImage_MinFrac (default=0.05) determines the significance level for lattices (with the default at least 5% of the number of spots for the most populated lattice). * Added feature to run new 'anaspot' program (which analyses SPOT.XDS files for ice-rings etc): if we have non-default resolution limits given by the user we need to intervene between the COLSPOT and the IDXREF stage. Calling anaspot will limit the found spots to the set resolution based on the current, idealised geometry. High-resolution criteria: * Implemented CC1/2 calculation according to Karplus & Diederichs (2012) paper. Final CC1/2 criteria to determine adequate high-resolution cutoff defaults to 0.3 (since 0.1 is too optimistic given the intrinsic noise a change in random seed already shows). Analysis: * Added plot combining divergence and mosaicity (since they are often related in interesting ways): this will not only print the estimated values (per image), but also the finally used values for the center region of the detector. * MTZANA tool for analysing (merged) MTZ files (MRFANA is the equivalent program for unmerged, multi-record MTZ files): added command-line arguments -colcmpl and -nocolcmpl to print (or not) completeness per column (for the resolution range of that column). * New tool (MTZANA) to provide very similar functionality to CCP4s MTZDUMP program - without some of the limitations in output format, that often make it difficult to parse the information correctly. === === Changes: === High-resolution criteria: * New set of defaults - same as previous HighRes2 macro: setvar ScaleAnaRmergeCut_123 "99.9999:99.9999" setvar ScaleAnaISigmaCut_123 "0.1:0.1 0.5:0.5 0.5:1.0 1.0:2.0" setvar ScaleAnaRpimallCut_123 "99.9999:99.9999 0.9:0.9 0.8:0.8 0.6:0.6" setvar ScaleAnaRmeasallCut_123 "99.9999:99.9999" setvar ScaleAnaCompletenessCut_123 "0.0:0.0" setvar ScaleAnaCChalfCut_123 "-1.0:-1.0 0.0:0.0 0.3:0.3" This means that Rmerge, Rmeas and Completeness are being ignored as a high-resolution cutoff-criterion. Rpim and CC1/2 are used very conservatively - and I/sigI (>= 2.0) is the main one. === === Improvements: === XDS support: * Put XDS parameters FIXED_SCALE_FACTOR and DATA_RANGE_FIXED_SCALE_FACTOR under XDS version control since availability depends on exact XDS version. * support for XDS version "March 30, 2013": - changes to format of XPARM.XDS file (contains all experiment settings): depending on the XDS version used, files generated by autoPROC will also follow the relevant format of this version. - additional keywords - changes to format of INTEGRATE.HKL (contains raw integrated intensities) - changes to various output log-files (mainly relevant to autoPROC are IDXREF.LP, INTEGRATE.LP) - changes in format of SPOT.XDS (contains list of found spots for indexing) - better versioning done by XDS (not just a version date, but now also a BUILT date is reported by XDS programs) Beamline/Instrument support: * Support for Pilatus images following the latest version of header specifications 1.3. See also https://www.dectris.com/technical_pilatus.html * Support for ALS 12.3.1 (SIBYLS) beamline. See also http://bl1231.als.lbl.gov/. Required settings (as of 201212) are: * 2-theta rotates opposite to the rotation axis: autoPROC_TwoThetaAxis="-1 0 0" * beam centre in header needs transforming: BeamCentreFrom="header:x,-y" * need to deal with beam centre in header stored including the 2-theta offset: imginfo="imginfo_ALS_12.3.1.sh" Running with process -M Als1231 ... should be sufficient. * Support for ALS 8.3.1 beamline: macro (Als831) and imginfo jiffy (imginfo_ALS_8.3.1.sh). Running with process -M Als831 ... should be sufficient. * Support for Pilatus 6M images from ID29@ESRF: these are marked as 'omega' scans while the current mini-CBF header from Dectris doesn't provide a specific Omega value. By using the EsrfId29 macro (-M EsrfId29) an appropriate override jiffy is called to dynamically correct those headers. Image file handling: * to enable handling of compressed (gz and bz2) images automatically, the new parameter FindImages_AllowCompressedImages should be set to "yes". Note: it might be a good idea to limit the number of threads for XDS if using this - otherwise there will be a lot of disk I/O (uncompression on-the-fly) by the XDS program, e.g with process autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_PROCESSORS=2 ... Please note that 'imginfo' can already work on compressed images directly. Spacegroup determination: * Extract scores for each symmetry element from the POINTLESS analysis: this will make it easier to detect lower symmetries (like pseudo-orthogonal monoclinic etc). Scaling: * Changes in MRFANA (analysis of unmerged reflection files) to work around Pointless/Aimless bug: currently these programs write incorrect information into the MTZ header (Laue group instead of Point group) which results in wrong setup of asymmetric-unit handling for reflection indices - which in turn results in incorrect handling of anomalous statistics calculations. * Changed autoPROC_ExcludeBadBatchesFromCycle from 2 to 3: to avoid excluding batches to early when scale factors, error-model and high-resolution limit are still being adjusted. * Changed parameter AutoProcScaleMinimumCycles from 2 to 4 to do at least 4 scaling cycles. * Also extract recommended anisotropic high resolution limits from AIMLESS (based on I/sigI > 2.0 and CC1/2 > 0.5 criteria. * When running with AIMLESS (default) we will allow wider gaps between batches (before classifying them as separate runs): this is mainly useful when dealing with fine-sliced data where sometimes no reflection is assigned to an intermediate image (therefore creating a gap). Therefore, the parameter AutoProcScaleMaxAllowedGapSize defaults to 5 with AIMLESS and 0 otherwise. * New option AutoProcScaleSetInitialHighRes (default = no) to apply an initial high-resolution cutoff based on of -0.2 (set by parameter AutoProcScaleSetInitialHighResIsigI) in the final shell. This should ensure that scaling doesn't fail because it sees a very large number of supposed spots that are actually pure noise - eg if the crystal-detector distance was set to a massively over-optimistic value. * We now also accept -Reso or -R instead of the standard -reso for setting resolution limits on the command-line. * When stopping (due to existing files that would otherwise be re-used and/or overwritten), also give a list of files that cause that error. This will make it easier for the user to remove those if wanted. * Create additional plots using "ana_aimless" command (in PNG format via gnuplot) that are useful: scaling B-factor (vs image/batch) and Rmerge (vs image/batch). The name of the created output files (prefix; default = "ana_aimless") can be set with the new (optional) argument -o. * Write script (aP_scale.sh) with actually used commands run: this should make it easier for a user to re-run the final scaling step. * Added parameter XdsScaleRunMultipleCriteria (default=no): additional scaling-only runs (in new aP_scale.sh) are thus switched off or on. * Using new MTZANA program instead of MTZDUMP: faster, better formatting and under our control (so easier to react and avoid surprises). Analysis: * Major reworking of the 'mkmovie' command (see -h output for details): usage of ffmpeg/avconv for encoding and MOSFLM/ADXV for image-generation still needs some testing and work. In general this is more a proof-of-concept than a robust way of generating movies for a set of diffraction images. * Improved generation of profile plots and movies in the 'mkmovie' command - including generation of gnuplot files that allow interactive handling (rotation) of 3D profile representations. Added command-line argument "-profile_only" to generate those. * New parameter XdsStats_StopOnGnuplotError (default=no) to control handling of errors during gnuplot: when set to "yes" autoPROC will stop if one of the plot-generating steps fails because of a gnuplot inconsistency. * Various changes to look-n-feel of plots: hopefully resulting in nicer plots. * Added diagnostic to warn if refined distance parameter moves more than autoPROC_XdsMaxDistanceChitter=1.0 mm from the mean. In those cases something else is most likely going on and at least the distance should be held fixed during integration (followed by several cycles of post-refinement and integration, eg. using the "-M LowResOrTricky" macro). General: * work around bug in AIMLESS/POINTLESS: unmerged MTZ files written by those programs (as of 20130517) have wrong content in the SYMINF record of the MTZ file header (Laue instead of Point group name). Contacted Phil Evans and CCP4 about that. * Extract POINTLESS program version via new generic function. * Better dealing with binning (for RSTATS-based statistics; shown when running with '-v' flag): setting CheckIndexing_RstatsBinningMaxNum to 50 (from 10) CheckIndexing_RstatsBinningMinWidth to 0.0001 (from 0.01) and adding CheckIndexing_RstatsBinningNumBin=10 CheckIndexing_RunMtzana="no" Also added command-line argument "-n " to set the number of bins to be used (beware: depending on completeness the actually used bins might still be different). Setting CheckIndexing_RunMtzana="yes" will run the data through the statistics part of MTZANA and calculate various CC values via this route too. * Better way to deal with differences between versions of POINTLESS: a generic function to extract version number from POINTLESS standard output will allow handling different versions of POINTLESS (and the different input they accept and output they produce). * Changed default conversion factor between XDS mosaicity (sigma) to MOSFLM (FWHM): from 2.0 to 2.3548 - which it should be for a Gaussian. * When updating parameters in XDS.INP file using "xds_update_pars" module: take different behaviour of XDS versions into account (see Version July 4, 2012 in XDS release notes at http://xds.mpimf-heidelberg.mpg.de/html_doc/Release_Notes.html). * Started adding support for generating commands that will use our new visualiser (gpx) to display diffraction images and predictions. This is still under construction and must await the release of the visualiser and prediction generator before it becomes fully functional. * Make sure some temporary files are removed to keep things tidy. === === Fixes: === Indexing: * When dealing with distributed background images: avoid problems if processed images don't follow a simple numbering (from 1 to N). Also: create pseudo-contiguous images for this part in subdirectory (to avoid confusion). * Fixed problem in getting the correct number of indexed spots out of an IDXREF.LP file (when this was reported with two different numbers in two places). This resulted in the wrong percentage being reported on standard output. Spacegroup determination: * Deal with a change in POINTLESS XML-output that caused the reindexing operator not being fetched correctly within "ana_pointless". Scaling * In MRFANA: fix to ensure correct sorting of ASCII files from XDS (INTEGRATE.HKL, XDS_ASCII.HKL or XSCALE output) to avoid wrong calculation of anomalous statistics like completeness. * Always set VRSET to -999999999 (SortmtzVrsetValue parameter) in calls to SORTMTZ: this should avoid errors - for datasets that have most likely some issues with detector areas (beam stop, dead modules or such). * Fixes to automatic adjustment of image ranges (-b argument): to deal with multi-scan experiments that are not using different filename templates, but rather a continuation of image numbers (e.g. when using different spots on the crystal). * Fixes to setting arguments for scaling module (aP_scale): to deal with multi-scan experiments that are not using different filename templates, but rather a continuation of image numbers (e.g. when using different spots on the crystal). * In MRFANA: fix initialisation in calls to CCP4 library. This should avoid bogus warning messages from the CCP4 library about LRIDX. * Write correct scaling software (SCALA or AIMLESS) into REMARK 200 section of automatically generated remark200.pdb files. High-resolution criteria: * In high-resolution cutoff criteria: explicitly set R-values (to be ignored) to 99.9999 to be in sync with MRFANA program (where this is the value to mark a value that can't be calculated). General: * Try to get the command-line arguments into a better formatted string (for reporting). * Clarified warning message when giving only symm card (without cell). * Give detailed information regarding the number of threads used at various stages (and for XDS or MOSFLM path). Also telling the user how to set or modify the automatic settings for the different stages. * Fixes in "check_indexing" to avoid different resolution limits being used for different re-indexing operators. * Fix problem where the final summary information (about cell, spacegroup, resolution etc) was given based on a wrong MTZ file (present in current directory and not subdirectory). Also fix extraction of resolution limits. Previously, overall limits of reflections was given, but now resolution limits of actually measured amplitudes is reported. * Fixed a bug which may cause the scaling module aP_scale to fail with no meaningful error message on some systems if mtzana is not available. For more details see: https://www.globalphasing.com/beta/wiki/index.cgi?SoftwareReleases201211KnownIssues#autoprocapscale Additional checks have been implemented to work around this issue. New features: * support for scaling with AIMLESS to support larger datasets (more than 5000 images). This is now the default (if AIMLESS is available in the users PATH), but can be switched off (to use SCALA instead) with autoPROC_ScaleWithAimless=no. This requires CCP4 version 6.3.0 onwards! By default, the scaling will run with multiple threads (can be controlled by the -nthreads command-line argument to 'process' or through the OMP_NUM_THREADS environment variable). * new jiffy add_freerflag.sh (to add existing test-set flags from one MTZ file to another) Improvements: * added AutoProcScaleMinimumCycles (default=2) parameter to the scaling module (aP_scale): this gives more control in macros like the HighRes2 one mentioned below) * give warning if only resolutions below 4A were used for space-group determination in POINTLESS. Users are advised to check the correct or most likely spacegroup in such cases of low resolution or poor diffraction by hand. * added new ways of using macros: - using 'process -M show' will not only give a description for a particular macro, but show also the detailed settings. This can help users fine-tune some of the distributed macros. - a user can now also give a filename as argument to '-M', so that own macros can be given without the need of setting the environment variable autoPROC_MacroDirs. The file still needs to follow the standard macro-format. * two new macros (HighRes2 and NoHighResCut) to either use only I/sigI and Rpim as high-resolution criteria or none at all. * new macro (LowResOrTricky) with a few settings that could help low-resolution and/or messy diffraction data: mainly trying to stabilise parameter refinement during integration. * easier way to keep mosaicity fixed during integration in XDS: setting the parameter XdsUpdateMosaicityScale to a positive (non-zero) value will keep the mosaicity constant and fixed during the second (and following) integration path. The actual value is the value recommended by XDS (as reported in INTEGRATE.LP) scaled by the XdsUpdateMosaicityScale value. * support for latest XDS version (September 26, 2012) * added parameter autoPROC_TestSetFraction (default=0.05) to the scaling module (aP_scale). This will be used when calling uniqueify to generate a fresh test-set. * imginfo tool now also deals with DATE items in ADSC images of format, such as 'DATE=Fri Apr 23 18:12:09 PDT 2010;' * support for POINTLESS 1.6.18 (from CCP4 6.3.0 release) * added plot showing detector position of misfits (according to XDS definition) * improvements in the generation of plots via new parameters XdsStats_sizeSquare, XdsStats_MaxSize, XdsStats_PositionInMm and XdsStats_PlotMissingImages * support for MOSFLM 7.0.9 (in CCP4 6.3.0 release) * extract additional information from POINTLESS step about potential twinning * added parameter ExpectLargeAnomalousSignal (default="no") and XdsSetStrictAbsorptionChiSquareCut (default=1.5): those are used to adjust settings in the CORRECT step - to deal better with datasets showing very large anomalous signals (e.g. with metal clusters at low resolution). * when dealing with datasets that have slight gaps (empty or non-existing images) together with automatic 'chunking': make use of new parameter AutoProcScaleMaxAllowedGapSize (default=0) to avoid weird chunking due to those gaps. * when working with multi-scan datasets that will get merged into separate files (e.g. multi-wavelength MAD): added parameter AutoProcScaleCombineCrystalDatasetInCrystal (default=yes) to work around CCP4 library feature (that will lose cell parameters for a specific dataset, since it assigns cell on a per-crystal basis). This should avoid problems with the iterative scaling (in SCALA), merging statistics and high-res recommendation (in MRFANA) - otherwise the calculation of resolution for a given HKL uses slightly different cells and would therefore be out-of-sync. * more sensible message about images with different scale factors: still based on global statistics, but at least doesn't get confused by missing images or image numbers above 999 * imginfo: - additional checks for odd Pilatus headers (wavelength values in A but unit in nm, distance given in mm without a unit or with 'm' unit) - added checks for weird wavelength value (below 0.5A) and distance value (>100m or <1mm) to automatically adjust these header specification issues - catch Proxima1/Soleil case where we have Kappa and Chi values but no Omega (and Chi is identical to Starting_angle): pretend that this is a Kappa goniostat This is due to Proxima1's capability of doing both Omega- and Phi-scans on a Kappa goniostat when the Pilatus header specification doesn't provide an entry for Omega (so the starting Omega value is stored in the Chi parameter). This might change with updates to software by Dectris and the Proxima1 beamline. - handle negative zero values in angles (reset to 0.0): otherwise the KAPPAROT tool to enforce consistent indexing and/or calculate general rotation axis (eg. phi-scans on multi-axis goniostats) gets confused. * when combining several reflection files (within autoPROC or with the command-line tool 'combine_files'): take value of EnsureConsistentIndexing into account to prevent POINTLESS from wrongly re-indexing the data into an equivalent (but not identical) setting. * new parameter SpacegroupIndistinguishableGroups to allow better message for I23/I213 and I222/I212121 cases * new parameter FindImages_ExcludeLastImage (default=no) to exclude the last image of a found scan: deals e.g. with sync issues at SLS-PXIII regarding shutter-closure * support for: - SLS-PXIII (Prigo) beamline configuration and Pilatus headers through macro: % process -M SlsPXIII ... - Soleil-Proxima1 (Kappa) beamline configuration and Pilatus headers through macro: % process -M SoleilProxima1 ... - NSLS-X25 (initial Pilatus 6M headers) through imginfo-wrapper: % process imginfo=imginfo_NSLS_X25.sh ... - support for Pilatus 6M images from ID29@ESRF: these are marked as 'omega' scans while the current mini-CBF header from Dectris doesn't provide a specific Omega value. By using the EsrfId29 macro an appropriate override jiffy is called to dynamically correct those headers: % process -M EsrfId29 ... * New tool (MTZANA) to provide very similar functionality to CCP4s MTZDUMP program (without some of the limitations in output format, that often make it difficult to parse the information correctly). * make sure some temporary files are removed to keep things tidy. Fixes: * handle multi-scan cases where one (or several) scans had more than 999 images (which could previously give an error in the final, multi-scan scaling step). * explicitly set VRSET in SORTMTZ to avoid odd failures when combining several MTZ files for scaling. However, those failures usually meant very poor or wrong beamstop masking, leading to wrongly integrated intensities behind the shadow. * added checks in MRFANA to avoid memory corruption. * added safety margin for low-resolution limit suggestion from MRFANA: avoids 'losing' low-resolution reflections due to slightly different methods of calculating resolution and/or rounding errors. This should also speed-up the scaling job (aP_scale) since it avoids unnecessary additional cycles of scaling with updated resolution limits. * fixed some wrong amounts of white-space in REMARK 200 generation (remark200.pdb file) * when combining files (using combine_files utility), take care to get correct batch numbering offset - relevant for large scans/datasets with more than 999 images. In the same combine_files utility, some reporting regarding necessary reindexing has been fixed (previously it might have reported reindexing when in fact it didn't do any). * better support for GnuPlot 4.4 (used in creating plots in PNG format) * fixes during indexing when iterative indexing is used (automatically determined or user-requested) and several indexing stages are necessary anyway (e.g. the user only supplied spacegroup but not cell parameters): pick the correct master list of spots for detection of (partially) empty datasets. * better messages when analysing indexed spots for potential high-resolution limit * fixed bug in the aP_scale module (when combining different analysis tools regarding automatic adjustment of high-resolution limits): the explanations why a certain high-resolution limit was picked were sometimes out-of-sync. * added check in iterative indexing to avoid continuing with obviously wrong solution: sometimes IDXREF seems to succeed whereas only very few spots were actually used for the indexing solution. The parameter XdsIndexingStopIfSpotsLessPercent (default=2) is used for that check. =================================================================== Changes relative to 1.0.0 20 Jul 2011 =================================================================== New features: Improvements and changes: * imginfo: - added support for Pilatus headers I03@Diamond - added support for ADSC/Rigaku combo headers - allow APS1 prefix in header items - additional checks when encountering odd/duplicate oscillation axis definitions * added some jiffies: - beam8.sh will report all 8 possible beam centre settings - imgdate.sh can be used to analyse collections of images based on the timestamp in the image header * better method for getting completeness (for overload analysis) from reflection files * added option autoPROC_XdsAcceptOverloads (default=no) to generate a second set of files with overloaded reflections being accepted instead of ignored. This is only meant to be a desperate and last measure! * MRFANA (merging statistics calculation): - better order of inclusion criteria of reflexions (resolution check now before sysabs) resulting in significant speedup. - ice-ring treatment improved through (among several changes): a) using ice-ring limits instead of mid-point (with fixed width) b) using experimentally observed values for ice-ring resolution limits from large number of JCSG datasets This results in much more reliable treatment of ice-rings. Statistics used in applying the various cut-off criteria (for the high-resolution limit) now use an interpolation across those ice-ring resolution shells. * aP_scale (scaling module): - significant speedup through changes in MRFANA (see above) and a faster invocation of MTZDUMP (to read header information from MTZ files) - changed default so that MRFANA will assume by default that the data could contain ice-rings (AutoProcScaleStatsUseMrfanaIgnoreIceRingShells=yes) - new command-line option (-noice) to revert back to previous default, ie. no ice-rings present in data * find_images should give nicer identifier values (by removing trailing '0' characters in obvious cases). * changed from old XDS keyword 'MINUTE' to new 'SECONDS' * unless the user saves standard output already, write a (uniquely named) log-file to ensure that standard output is not lost. * new option to distribute background images over larger range of images (in chunks): DistributeBackgroundImages (default="no"). This should give a better initial background (BKGINIT.cbf file from the XDS INIT step), while avoiding using a very large number of images (which would be slower). * added analysis for empty images: we now require at least 2 (XdsSpotSearchMinNumSpotsPerImage) spots per image but allow 5 (XdsSpotSearchNumAllowedEmptyImages) empty images within the range of images used for spot searching. * lowered starting value for automatic backstop masking in XDS (DEFPIX step) to 5000 (XdsOptimizeDefpixStart), to deal with very weak images. * added option (XdsSetHighResFromIndexedSpots=no) to set high-resolution limit based on indexed spots. Fixes: * add protection against dash failing on 'empty' .autoPROC files * avoid setting reslow/reshigh outside normal command-line system * MRFANA: - corrected bug in accessing cached values of completeness (per resolution bin), therefore avoiding duplication of calls - resulting in further speedup. * fixed wrong reporting of reindexing requirements (due to changes in POINTLESS log-file format) * ensure that 'process' exits with status 0 if all went well * fixed problem in passing empty strings (of non-zero length) into parameters, eg. refine autoPROC_XdsKeyword_REFINE_INTEGRATE=" " ... will now work. * when analysing the low- and high-resolution limit of indexed spots: ensure that the correct set of parameters is used when the iterative indexing was used (because of multiple lattices, ice-rings etc). * deal with scans of more than 999 images (when creating plots of scale, divergence and mosaicity) #### NOTES The 32-bit Linux distributions of Global Phasing's software will work on systems running relatively old kernels (versions 2.4.x). Most mainstream server and desktop Linux distributions that are currently under active maintenance are based on the newer 2.6.x kernel series. The current 32-bit release will be the last to support Linux systems based on the 2.4.x kernel. Future distributions (including patches and minor updates) of our software for all Linux systems released after this one can only be relied on to work on systems running kernels from the 2.6.x series. This is only a significant change for our 32-bit distribution: our 64-bit distributions for Linux have always required kernel version 2.6.x To check the version of the kernel on a particular system, run the following command: uname -r =================================================================== Any feedback and remarks should be send to the autoPROC developers: proc-develop@globalphasing.com ===================================================================