autoPROC Documentation	previous	next
Scaling options

autoPROC Documentation : Scaling options

Copyright © 2015-2018 by Global Phasing Limited

All rights reserved.

This software is proprietary to and embodies the confidential technology of Global Phasing Limited (GPhL). Possession, use, duplication or dissemination of the software is authorised only pursuant to a valid written licence from GPhL.

Documentation (2015-2018) Clemens Vonrhein, Claus Flensburg, Wlodek Paciorek & Gérard Bricogne

Contact proc-develop@GlobalPhasing.com

Introduction
Current default scaling mode
AIMLESS-only scaling path
XDS/XSCALE-only scaling path
Common features

Introduction

The correct path for scaling single or multiple sweep datasets both internally as well as relative to each other (before merging the related measurements), is an area of active investigation. The best mode will depend on one or several factors:

symmetry of crystal
multiplicity of data
experimental design (single or multiple wavelengths, single or multiple positions on crystal, interleaving energies or orientations)
radiation damage

Current default scaling mode

Each sweep is first scaled internally using the CORRECT module of XDS, producing an XDS_ASCII.HKL file of intensities. This includes all default CORRECTIONS:

absorption correction
modulation (as a function of detector position)
decay correction (to accommodate for radiation damage or varying strength of diffraction at different rotation angles)

At this point, error estimates are adjusted, outliers (misfits) are identified and polarisation correction is applied. The resulting XDS_ASCII.HKL file for this single sweep is then scaled in AIMLESS using a smoothly varying scale factor and scaling B-factor as well as a tightly restrained (to being spherical) absorption correction. Error estimates are adjusted using SdFac and SdAdd as described in the AIMLESS manual. This results in a series of files

File	Explanation
aimless_unmerged.mtz	internally scaled but unmerged intensities (a multi-record MTZ file)
aimless.mtz	internally scaled and merged intensities (MTZ format)
aimless.sca	internally scaled and merged intensities (Scalepack format)

The aimless.mtz file is then used to compute amplitudes and anomalous differences (plus their respective sigmas) with TRUNCATE:

File	Explanation
truncate.mtz	amplitudes, anomalous differences and intensities
truncate-unique.mtz	same as `truncate.mtz`, but including a test-set flag (if a reference MTZ file was given, those test-set flags are used and complemented if necessary)

When processing multiple sweeps within autoPROC, the separate XDS_ASCII.HKL files are combined into a multi-record MTZ file (using the combine_files tool) and then passed to AIMLESS via the aP_scale module. Here the same scaling model as above (smoothly varying scale and scaling B-factor plus restrained absorption correction) is applied, with the following characteristics:

scale and B-factor are restrained (smoothed) only within each "run" (i.e. each separate sweep of data processed)
absorption correction is done in the crystal frame and enforced to be the same over all sweeps: since autoPROC takes great care to ensure identical indexing between sweeps (knowing about the different goniostat settings and applying the relevant transformations for each sweep relative to the first one), all data are represented in the correct way - the X-ray beam "sees" the crystal exactly as it was (and not as a symmetry-equivalent orientation).
different levels of outlier classification are used within each "run" and between them, depending also on the final merging assignments (which "runs" are to be merged together after scaling)

The output files will consist of

File	Explanation
aimless_unmerged_<wvl>.mtz	internally and externally scaled but unmerged intensities (a multi-record MTZ file)
aimless_<wvl>.mtz	internally and externally scaled and merged intensities (MTZ format)
aimless_<wvl>.sca	internally and externally scaled and merged intensities (Scalepack format)
truncate_<wvl>.mtz	amplitudes, anomalous differences and intensities (based on `aimless_<wvl>.mtz`)
truncate_<wvl>-unique.mtz	same as `truncate.mtz`, but including a test-set flag (if a reference MTZ file was given, those test-set flags are used and complemented if necessary)

AIMLESS-only scaling path

By running "process" with the ScalingA3 macro as

process -M ScalingA3 ...

a pure AIMLESS path is provided with autoPROC. The difference to the above is that instead of using the XDS_ASCII.HKL file(s) from CORRECT, this uses the INTEGRATE.HKL file(s) directly after integration within INTEGRATE. The conversion of those reflection files into MTZ format (for use in AIMLESS via our aP_scale scaling module) is done with POINTLESS, using the correct polarisation value and description.

Those datasets for each sweep are then used in the same way as the default path described above - both for each sweep on its own as well as for the multi-sweep situation.

XDS/XSCALE-only scaling path

As an alternative to using AIMLESS as the final scaling program (either in the default/combined mode or the AIMLESS-only path), an XSCALE-only path is provided. This circumvents the use of AIMLESS completely and use only programs from the XDS package for scaling, outlier identification and error estimate adjustment. It can be activated using the ScalingX macro with

process -M ScalingX ...

Each sweep of data will be handled by the CORRECT module of XDS including all CORRECTIONS, outlier rejection and error estimate adjustment. If only a single sweep is processed by autoPROC, this internally scaled but unmerged data (XDS_ASCII.HKL) is first merged into intensities using XDSCONV and then converted into amplitudes using TRUNCATE. Therefore, the relevant output files are

for single sweep data (or multi-sweep data where all sweeps are being merged together at the end):

File	Explanation
XDS_ASCII.HKL	internally scaled but unmerged
xscale.ahkl	internally scaled and unmerged intensities
xscale_xdsconv.mtz	internally scaled and merged intensities (MTZ format)
xscale_xdsconv.sca	internally scaled and merged intensities (Scalepack format)
truncate.mtz	amplitudes, anomalous differences and intensities (based on `xscale_xdsconv.mtz`)
truncate-unique.mtz	same as `truncate.mtz`, but including a test-set flag (if a reference MTZ file was given, those test-set flags are used and complemented if necessary)

for multi-sweep data where different sweeps are merged together into separate, merged reflection files:

File	Explanation
XDS_ASCII.HKL	internally scaled but unmerged
xscale_<wvl>.ahkl	internally and externally scaled and unmerged intensities
xscale_<wvl>_xdsconv.mtz	internally and externally scaled and merged intensities (MTZ format)
xscale_<wvl>_xdsconv.sca	internally and externally scaled and merged intensities (Scalepack format)
truncate_<wvl>.mtz	amplitudes, anomalous differences and intensities (based on `xscale_<wvl>_xdsconv.mtz`)
truncate_<wvl>-unique.mtz	same as `truncate_<wvl>.mtz`, but including a test-set flag (if a reference MTZ file was given, those test-set flags are used and complemented if necessary)

If dealing with multi-sweep datasets, the various XDS_ASCII.HKL files are given to the scaling program XSCALE via our aP_scale module. All sweep datasets that are to be merged together (e.g. because they belong to the same wavelength) will be grouped under the same OUTPUT_FILE. This means that each sweep dataset within such a group "sees" data from the other sweeps of this group - while sweep datasets between groups only see each other prior to the computation of the correction factors (absorption, modulation and decay).

Common features

The above scaling modes share some common design and many common features:

internal scaling refers to the scaling applied to measurements that will finally be merged. This could be from a single sweep, different orientations of the crystal (making use of a multi-axis goniostat), different positions of the same crystal, inverse-beam data, a high- and low-resolution pass etc. Remember that by default autoPROC will merge all sweeps that have the same wavelength value.
external scaling refers to the scaling applied to measurements that will end up in separate merged reflection files. This mostly (but not exclusively) applies to multi-wavelength datasets (where each wavelength will obviously give rise to a separate output reflection file).
statistics (I/sigI, CC(1/2), Rmerge, Completeness etc) are always computed within our MRFANA program to ensure that identical algorithms are used, especially for random-number generation and binning.
the high-resolution decision-making is always based on the statistics as computed within MRFANA (see here for details)
the computation of amplitudes from intensities is done with the CCP4 program TRUNCATE
If the user provides a reference MTZ file containing amplitudes, these will be to ensure consistent indexing if the space-group allows alternative indexing schemes.
If this reference MTZ file also contains test-set flags, these will be transferred (and complemented if necessary) to the output MTZ file for cross-validation at later stages (e.g. Rfree computation during refinement). This transfer of test-set flags is recommended whenever a "parent" structure is used to phase identical crystal forms - like ligand screening by soaking crystals of a common apo-from.
See also the FAQ on the autoPROC Wiki.

Last modification: 27.03.2018