Content:
The notes below started in March 2016, when these datasets were starting to come out from various beamlines/synchrotrons. In the meantime, the HDF5 format (in it's original Dectris version or the NeXus format) can be considered standard and mainstream. So we would expect autoPROC to work with those datasets as-is and without issues.
However, there could still be issues - see e.g.
The Eiger detector is a new detector from Dectris that also produces data in a new format: HDF5. We added support for this into autoPROC based on Eiger 16M data provided by Dectris and SLS.
Notes:
The solutions shown below (via imginfo wrapper scripts) are not our preferred way of handling these newer detectors. We hope that updates to the software at beamlines will result in missing data to be added or incorrect items being corrected.
See also:
Running
% process -h5 some_master.h5 -d outdir | tee out.lis
should be all that is required for processing. All other standard autoPROC flags and options can be used.
Thanks to Dirk Reinert (Boehringer-Ingelheim), Meitian Wang (SLS), Andreas Foerster (Dectris), Joachim Diez (Expose), Stefan Brandstetter (Dectris), Zac Panepucci (SLS), Kay Diederichs (Konstanz) and many more.
See also:
Earlier datasets:
process dist=123.45 beam="1234.5 1234.6" ...
After downloading imginfo_ESRF_Eiger4M.sh and making it executable via
% chmod +x imginfo_ESRF_Eiger4M.sh
you can run something like
% process -h5 some_master.h5 \ autoPROC_XdsKeyword_UNTRUSTED_RECTANGLE="257 483 551 809" \ imginfo=`pwd`/imginfo_ESRF_Eiger4M.sh osc=0.05 \ -d outdir | tee out.lis
(obviously adapting to your local setup and dataset characteristics).
Thanks to David von Stetten (ESRF) for test data and discussions.
See also:
Important:
% process -h5 h.001_master.h5 imginfo=`pwd`/imginfo_APS_ID21D_Eiger9M.sh osc=0.5 -d 01 | tee 01.lis
Thanks to Joe Brunzelle (LS-CAT) for providing test data.
See also:
Important:
/entry/instrument/detector/detectorSpecific/eiger_fw_version = eiger-1.5.2-101.git5b1f904.release /entry/instrument/detector/detectorSpecific/software_version = 1.5.2 /entry/instrument/detector/detector_number = E-18-0102 ... /entry/instrument/detector/detectorSpecific/eiger_fw_version = eiger-1.6.1-104.git3f17ecb.release /entry/instrument/detector/detectorSpecific/software_version = 1.6.1 /entry/instrument/detector/detector_number = E-18-0102
% process -h5 Lyso_U2_1_master.h5 imginfo=`pwd`/imginfo_Soleil_Proxima2_Eiger9M.sh autoPROC_XdsKeyword_SENSOR_THICKNESS=0.45 -d 01 | tee 01.lis
Note: autoPROC releases from 20160324 onwards contain already this wrapper, so running
% process -h5 h.001_master.h5 imginfo=imginfo_Soleil_Proxima2_Eiger9M.sh autoPROC_XdsKeyword_SENSOR_THICKNESS=0.45 -d 01 | tee 01.lis
should work.
Thanks to Martin Savko and Bill Shepard (Soleil) as well as Nicholas Keep (Birkbeck, London) for providing test data.
See also:
Important:
% process -h5 h.001_master.h5 imginfo=`pwd`/imginfo_PF_BL1A_Eiger4M.sh osc=0.5 -d 01 | tee 01.lis
Note: autoPROC releases from 20160324 onwards contain already this wrapper, so running
% process -h5 h.001_master.h5 imginfo=imginfo_PF_BL1A_Eiger4M.sh osc=0.5 -d 01 | tee 01.lis
should work.
Thanks to Takaaki Fukami (Chugai Pharmaceutical) for providing feedback and test data.
Running
% process -h5 some_master.h5 ReverseRotationAxis=yes -d outdir | tee out.lis
should be all that is required for processing. All other standard autoPROC flags and options can be used.
Thanks to Keitaro Yamashita (SPring-8) for providing information and test data.
The master file contains the definition of the rotation axis, but autoPROC currently doesn't read this item. So one needs to run
% process autoPROC_XdsKeyword_ROTATION_AXIS="0 -1 0" -h5 some_master.h5 ...
for now.
Thanks to Jie Nan and Uwe Mueller for providing test data and help.
These issues have been fixed as of 20180129 (Thanks Jun Aishima frmo the Australian Synchrotron for letting us know)
As of 20170920, HDF5 datasets from the Eiger 16M at MX-2 of the Australian Synchrotron record inconsistent information about rotation axes in the master.h5 file: both "Omega" and "Phi" are marked as rotation axes. Both will contain start and end values according to the oscillation angle and the "imginfo" tool from autoPROC returns something like
>>> Image format detected as HDF5/Eiger Image number 1/2000 ===== Header information: date = 20 Sep 2017 15:31:54.976 distance [mm] = 380.032 wavelength [A] = 0.953729 sensor thickness [mm] = 0.450 Phi-angle (start, end) [degree] = 180.00000 180.10001 Oscillation-angle in Phi [degree] = 0.10001 Omega-angle (start, end) [degree] = 180.00000 180.10001 Oscillation-angle in Omega [degree] = 0.10001 Chi-angle [degree] = 0.00000 Kappa-angle [degree] = 0.00000 2-Theta angle [degree] = 0.00000 Pixel size in X [mm] = 0.075000 Pixel size in Y [mm] = 0.075000 Number of pixels in X = 4150 Number of pixels in Y = 4371 Beam centre in X [mm] = 155.187 Beam centre in X [pixel] = 2069.160 Beam centre in Y [mm] = 167.042 Beam centre in Y [pixel] = 2227.230 Overload value = 38079 WARNING: more than one angle marked with different start/stop values! WARNING: oscillation around more than one axis?
Obviously, not both axes (Omega and Phi) would have rotated during the experiment (since this would correspond to a total rotation of 0.2 degree in case of a Kappa-goniostat with Kappa=0.0 ... or of a total rotation of 0.0 if they rotate in opposite directions at that setting). So the above will confuse autoPROC and could result in failed or poor processing.
Until this has been fixed at the beamline - and in order to (re)process already collected datasets - a workaround is required to patch those header information up. Please download the imginfo_MX-2_AustralianSynchrotron_20171009.sh script, make it executable via
% chmod 0755 /where/ever/imginfo_MX-2_AustralianSynchrotron_20171009.sh
and use it in autoPROC:
% process imginfo=/where/ever/imginfo_MX-2_AustralianSynchrotron_20171009.sh ...
Please note that
% process BeamCentreFrom=header:y,x ReverseRotationAxis=yes ...
Thanks to Richard Birkinshaw (University Melbourne) for providing example datasets for testing. The above script will be included into the next autoPROC release.
Example data from May 2021 shows
imginfo -v -v -h5 some_master.h5
/entry/instrument/detector/description = "Dectris EIGER2 Si 16M" /entry/instrument/detector/detector_number = "E-32-0123" /entry/instrument/detector/sensor_material = "Si" ... /entry/instrument/detector/detectorSpecific/ntrigger = 1 /entry/sample/goniometer/omega[0] = 340.000000 /entry/sample/goniometer/omega[1] = 340.200000 /entry/sample/goniometer/omega[2] = 340.400000 /entry/sample/goniometer/omega_end[0] = 340.200000 /entry/sample/goniometer/omega_end[1] = 340.400000 /entry/sample/goniometer/omega_end[2] = 340.600000 /entry/sample/goniometer/omega_range_average = 0.200000 degree /entry/sample/goniometer/omega_range_total = 360.000000 degree /entry/sample/goniometer/kappa[0] = 0.000000 /entry/sample/goniometer/kappa[1] = 0.100000 /entry/sample/goniometer/kappa[2] = 0.200000 /entry/sample/goniometer/kappa_end[0] = 0.100000 /entry/sample/goniometer/kappa_end[1] = 0.200000 /entry/sample/goniometer/kappa_end[2] = 0.300000 /entry/sample/goniometer/kappa_range_average = 0.100000 degree /entry/sample/goniometer/kappa_range_total = 180.000000 degree /entry/sample/goniometer/chi[0] = 0.000000 /entry/sample/goniometer/chi[1] = 0.100000 /entry/sample/goniometer/chi[2] = 0.200000 /entry/sample/goniometer/chi_end[0] = 0.100000 /entry/sample/goniometer/chi_end[1] = 0.200000 /entry/sample/goniometer/chi_end[2] = 0.300000 /entry/sample/goniometer/chi_range_average = 0.100000 degree /entry/sample/goniometer/chi_range_total = 180.000000 degree /entry/sample/goniometer/phi[0] = 0.000000 /entry/sample/goniometer/phi[1] = 0.200000 /entry/sample/goniometer/phi[2] = 0.400000 /entry/sample/goniometer/phi_end[0] = 0.200000 /entry/sample/goniometer/phi_end[1] = 0.400000 /entry/sample/goniometer/phi_end[2] = 0.600000 /entry/sample/goniometer/phi_range_average = 0.200000 degree /entry/sample/goniometer/phi_range_total = 360.000000 degree
This is a very confused definition of rotation axes ... clearly we're not doing an experiment where all those angles are rotating at the same time (and a goniostat with both a Chi and Kappa axis is rather odd). This can easily lead to incorrect interpretation of both the starting angle (340 or 0?) and oscillation angle.
Further we then have
omega axis = -1.00000 0.00000 0.00000 fast pixel vector = -1.00000 0.00000 0.00000 slow pixel vector = 0.00000 -1.00000 0.00000
Which seems incorrect: the correct rotation axis is (1,0,0), i.e. we would need to run with
process ReverseRotationAxis=yes ...
However, because of those confusing goniostat angle information, autoPROC/imginfo is unable to uniquely determine the rotation axis and will not pick up that defined Omega axis vector anyway ... so all a bit messy and complicated for those datasets.
autoPROC provides a converter for HDF5 datasets into CBF format (including full mini-cbf headers). It supports both bitshuffle as well as LZ4 compression. Please note that is will only convert HDF5 datasets without applying any of the necessary fixes discussed above (which are necessary because of issues mostly at the metadata population stage). In order to obtain correct miniCBF headers, the metadata of the input HDF5 file needs to be correct.
The converter can be used to extract single images via e.g.
% hdf2mini-cbf -m some_master.h5 -image 47
or the complete dataset using
% hdf2mini-cbf -m some_master.h5
For additional help please see
% hdf2mini-cbf -h
USAGE: hdf2mini-cbf [-h] [-linkrange <i1> <i2>] [-image <imgnum>] -m <master.h5> [-o <prefix>] [-uncompressed|-compressed|-gzip-level <level>]
Additional options/features are
An additional tool is provided for very fast conversion by distributing the extraction/conversion to multiple compute nodes. See
% aP_convert_hdf5 -h
for details.
USAGE: aP_convert_hdf5 [-h] [-v] [-t] [-compressed] -m <master> -o <prefix> -h : show help -v : increase verbosity (default = 0) -t : don't actually run the command(s), but show what would be done -m <master> : HDF5 *_master.h5 file -o <prefix> : prefix (default=__169826_) for output mini-cbf files (default=__169826__######.cbf) -compressed : (optional) write *.cbf.gz files directly
Some settings - current default are adequate for SLS computing setup - can be given on the command-line (as par=value):
Note: the resulting miniCBF files take full advantage of the "signed 32-bit integer" data type used for storing the image array (see "X-Binary-Element-Type:" item). MOSFLM only supports pixel values from (and including) -2 and higher. It will therefore interpret other pixel markers below that threshold incorrectly, leading to wrong or completely failing processing (thanks to Harry Powell for confirmation). We are working on a MOSFLM-compatibility option for our hdf2mini-cbf converter for the next release. XDS will interpret the full "signed 32-bit integer" range correctly, but we have no information for any other data-processing package.
A collaboration between Dectris and the XDS developers one can now specify an external library, such that XDS can read HDF5 files directly. See eg. Dectris.
The use of this in autoPROC is not yet automatic. Meanwhile you can do:
% process -h5 data/abc_1_master.h5 \ autoPROC_XdsKeyword_LIB=/path/to/dectris-neggia.so \ -d aP-01 >& aP-01.txt
That is:
The dectris-neggia.so library is available from Dectris for CentOS6 and CentOS7. It is also available in source form from GitHub.
See also recent CCP4bb discussion.