Content:


Introduction

The notes below started in March 2016, when these datasets were starting to come out from various beamlines/synchrotrons. In the meantime, the HDF5 format (in it's original Dectris version or the NeXus format) can be considered standard and mainstream. So we would expect autoPROC to work with those datasets as-is and without issues.

However, there could still be issues - see e.g.


The Eiger detector is a new detector from Dectris that also produces data in a new format: HDF5. We added support for this into autoPROC based on Eiger 16M data provided by Dectris and SLS.

Notes:

The solutions shown below (via imginfo wrapper scripts) are not our preferred way of handling these newer detectors. We hope that updates to the software at beamlines will result in missing data to be added or incorrect items being corrected.


Eiger 16M at SLS, X06SA (PX-I)

See also:

Running

% process -h5 some_master.h5 -d outdir | tee out.lis

should be all that is required for processing. All other standard autoPROC flags and options can be used.

Thanks to Dirk Reinert (Boehringer-Ingelheim), Meitian Wang (SLS), Andreas Foerster (Dectris), Joachim Diez (Expose), Stefan Brandstetter (Dectris), Zac Panepucci (SLS), Kay Diederichs (Konstanz) and many more.


Eiger 4M at ESRF, MASSIF-3

See also:

Earlier datasets:

         process dist=123.45 beam="1234.5 1234.6" ...

After downloading imginfo_ESRF_Eiger4M.sh and making it executable via

% chmod +x imginfo_ESRF_Eiger4M.sh

you can run something like

% process -h5 some_master.h5 \
        autoPROC_XdsKeyword_UNTRUSTED_RECTANGLE="257 483 551 809" \
        imginfo=`pwd`/imginfo_ESRF_Eiger4M.sh osc=0.05 \
        -d outdir | tee out.lis

(obviously adapting to your local setup and dataset characteristics).

Thanks to David von Stetten (ESRF) for test data and discussions.


Eiger 9M at APS 21-ID-D

See also:

Important:

      % process -h5 h.001_master.h5 imginfo=`pwd`/imginfo_APS_ID21D_Eiger9M.sh osc=0.5 -d 01 | tee 01.lis

Thanks to Joe Brunzelle (LS-CAT) for providing test data.


Eiger 9M at Soleil Proxima2

See also:

Important:

        /entry/instrument/detector/detectorSpecific/eiger_fw_version = eiger-1.5.2-101.git5b1f904.release
        /entry/instrument/detector/detectorSpecific/software_version = 1.5.2
        /entry/instrument/detector/detector_number = E-18-0102

        ...

        /entry/instrument/detector/detectorSpecific/eiger_fw_version = eiger-1.6.1-104.git3f17ecb.release
        /entry/instrument/detector/detectorSpecific/software_version = 1.6.1
        /entry/instrument/detector/detector_number = E-18-0102
            % process -h5 Lyso_U2_1_master.h5 imginfo=`pwd`/imginfo_Soleil_Proxima2_Eiger9M.sh autoPROC_XdsKeyword_SENSOR_THICKNESS=0.45 -d 01 | tee 01.lis

Note: autoPROC releases from 20160324 onwards contain already this wrapper, so running

 % process -h5 h.001_master.h5 imginfo=imginfo_Soleil_Proxima2_Eiger9M.sh autoPROC_XdsKeyword_SENSOR_THICKNESS=0.45 -d 01 | tee 01.lis

should work.

Thanks to Martin Savko and Bill Shepard (Soleil) as well as Nicholas Keep (Birkbeck, London) for providing test data.


Eiger 4M at Photon Factory BL1A

See also:

Important:

      % process -h5 h.001_master.h5 imginfo=`pwd`/imginfo_PF_BL1A_Eiger4M.sh osc=0.5 -d 01 | tee 01.lis
 

Note: autoPROC releases from 20160324 onwards contain already this wrapper, so running

 % process -h5 h.001_master.h5 imginfo=imginfo_PF_BL1A_Eiger4M.sh osc=0.5 -d 01 | tee 01.lis

should work.

Thanks to Takaaki Fukami (Chugai Pharmaceutical) for providing feedback and test data.


Eiger 9M at SPring-8 BL32XU

Running

% process -h5 some_master.h5 ReverseRotationAxis=yes -d outdir | tee out.lis

should be all that is required for processing. All other standard autoPROC flags and options can be used.

Thanks to Keitaro Yamashita (SPring-8) for providing information and test data.


Eiger 16M at MAX IV (BioMAX)

The master file contains the definition of the rotation axis, but autoPROC currently doesn't read this item. So one needs to run

% process autoPROC_XdsKeyword_ROTATION_AXIS="0 -1 0" -h5 some_master.h5 ...

for now.

Thanks to Jie Nan and Uwe Mueller for providing test data and help.


Eiger 16M at Australian Synchrotron (MX-2)

These issues have been fixed as of 20180129 (Thanks Jun Aishima frmo the Australian Synchrotron for letting us know)

As of 20170920, HDF5 datasets from the Eiger 16M at MX-2 of the Australian Synchrotron record inconsistent information about rotation axes in the master.h5 file: both "Omega" and "Phi" are marked as rotation axes. Both will contain start and end values according to the oscillation angle and the "imginfo" tool from autoPROC returns something like

 >>> Image format detected as HDF5/Eiger

     Image number 1/2000

 ===== Header information:
 date                                = 20 Sep 2017 15:31:54.976
 distance                       [mm] = 380.032
 wavelength                      [A] = 0.953729
 sensor thickness               [mm] = 0.450
 Phi-angle (start, end)     [degree] = 180.00000 180.10001
 Oscillation-angle in Phi   [degree] = 0.10001
 Omega-angle (start, end)   [degree] = 180.00000 180.10001
 Oscillation-angle in Omega [degree] = 0.10001
 Chi-angle                  [degree] = 0.00000
 Kappa-angle                [degree] = 0.00000
 2-Theta angle              [degree] = 0.00000
 Pixel size in X                [mm] = 0.075000
 Pixel size in Y                [mm] = 0.075000
 Number of pixels in X               = 4150
 Number of pixels in Y               = 4371
 Beam centre in X               [mm] = 155.187
 Beam centre in X            [pixel] = 2069.160
 Beam centre in Y               [mm] = 167.042
 Beam centre in Y            [pixel] = 2227.230
 Overload value                      = 38079

 WARNING: more than one angle marked with different start/stop values!
 WARNING: oscillation around more than one axis?

Obviously, not both axes (Omega and Phi) would have rotated during the experiment (since this would correspond to a total rotation of 0.2 degree in case of a Kappa-goniostat with Kappa=0.0 ... or of a total rotation of 0.0 if they rotate in opposite directions at that setting). So the above will confuse autoPROC and could result in failed or poor processing.

Until this has been fixed at the beamline - and in order to (re)process already collected datasets - a workaround is required to patch those header information up. Please download the imginfo_MX-2_AustralianSynchrotron_20171009.sh script, make it executable via

% chmod 0755 /where/ever/imginfo_MX-2_AustralianSynchrotron_20171009.sh

and use it in autoPROC:

% process imginfo=/where/ever/imginfo_MX-2_AustralianSynchrotron_20171009.sh ...

Please note that

     % process BeamCentreFrom=header:y,x ReverseRotationAxis=yes ...

Thanks to Richard Birkinshaw (University Melbourne) for providing example datasets for testing. The above script will be included into the next autoPROC release.


Eiger 16M at CHESS

Example data from May 2021 shows

imginfo -v -v -h5 some_master.h5
     /entry/instrument/detector/description = "Dectris EIGER2 Si 16M"
     /entry/instrument/detector/detector_number = "E-32-0123"
     /entry/instrument/detector/sensor_material = "Si"
     ...
     /entry/instrument/detector/detectorSpecific/ntrigger = 1
     /entry/sample/goniometer/omega[0] = 340.000000
     /entry/sample/goniometer/omega[1] = 340.200000
     /entry/sample/goniometer/omega[2] = 340.400000
     /entry/sample/goniometer/omega_end[0] = 340.200000
     /entry/sample/goniometer/omega_end[1] = 340.400000
     /entry/sample/goniometer/omega_end[2] = 340.600000
     /entry/sample/goniometer/omega_range_average = 0.200000 degree
     /entry/sample/goniometer/omega_range_total = 360.000000 degree
     /entry/sample/goniometer/kappa[0] = 0.000000
     /entry/sample/goniometer/kappa[1] = 0.100000
     /entry/sample/goniometer/kappa[2] = 0.200000
     /entry/sample/goniometer/kappa_end[0] = 0.100000
     /entry/sample/goniometer/kappa_end[1] = 0.200000
     /entry/sample/goniometer/kappa_end[2] = 0.300000
     /entry/sample/goniometer/kappa_range_average = 0.100000 degree
     /entry/sample/goniometer/kappa_range_total = 180.000000 degree
     /entry/sample/goniometer/chi[0] = 0.000000
     /entry/sample/goniometer/chi[1] = 0.100000
     /entry/sample/goniometer/chi[2] = 0.200000
     /entry/sample/goniometer/chi_end[0] = 0.100000
     /entry/sample/goniometer/chi_end[1] = 0.200000
     /entry/sample/goniometer/chi_end[2] = 0.300000
     /entry/sample/goniometer/chi_range_average = 0.100000 degree
     /entry/sample/goniometer/chi_range_total = 180.000000 degree
     /entry/sample/goniometer/phi[0] = 0.000000
     /entry/sample/goniometer/phi[1] = 0.200000
     /entry/sample/goniometer/phi[2] = 0.400000
     /entry/sample/goniometer/phi_end[0] = 0.200000
     /entry/sample/goniometer/phi_end[1] = 0.400000
     /entry/sample/goniometer/phi_end[2] = 0.600000
     /entry/sample/goniometer/phi_range_average = 0.200000 degree
     /entry/sample/goniometer/phi_range_total = 360.000000 degree

This is a very confused definition of rotation axes ... clearly we're not doing an experiment where all those angles are rotating at the same time (and a goniostat with both a Chi and Kappa axis is rather odd). This can easily lead to incorrect interpretation of both the starting angle (340 or 0?) and oscillation angle.

Further we then have

 omega axis   = -1.00000  0.00000  0.00000
 fast pixel vector = -1.00000  0.00000  0.00000
 slow pixel vector =  0.00000 -1.00000  0.00000

Which seems incorrect: the correct rotation axis is (1,0,0), i.e. we would need to run with

process ReverseRotationAxis=yes ...

However, because of those confusing goniostat angle information, autoPROC/imginfo is unable to uniquely determine the rotation axis and will not pick up that defined Omega axis vector anyway ... so all a bit messy and complicated for those datasets.


HDF5 converter

autoPROC provides a converter for HDF5 datasets into CBF format (including full mini-cbf headers). It supports both bitshuffle as well as LZ4 compression. Please note that is will only convert HDF5 datasets without applying any of the necessary fixes discussed above (which are necessary because of issues mostly at the metadata population stage). In order to obtain correct miniCBF headers, the metadata of the input HDF5 file needs to be correct.

The converter can be used to extract single images via e.g.

% hdf2mini-cbf -m some_master.h5 -image 47

or the complete dataset using

% hdf2mini-cbf -m some_master.h5

For additional help please see

% hdf2mini-cbf -h
USAGE: hdf2mini-cbf [-h] [-linkrange <i1> <i2>] [-image <imgnum>] -m <master.h5> [-o <prefix>] [-uncompressed|-compressed|-gzip-level <level>]

Additional options/features are

An additional tool is provided for very fast conversion by distributing the extraction/conversion to multiple compute nodes. See

% aP_convert_hdf5 -h

for details.

USAGE: aP_convert_hdf5 [-h] [-v] [-t] [-compressed] -m <master> -o <prefix>

       -h                    : show help

       -v                    : increase verbosity (default = 0)

       -t                    : don't actually run the command(s), but show
                               what would be done

       -m <master>           : HDF5 *_master.h5 file

       -o <prefix>           : prefix (default=__169826_) for output mini-cbf
                               files (default=__169826__######.cbf)

       -compressed           : (optional) write *.cbf.gz files directly

Some settings - current default are adequate for SLS computing setup - can be given on the command-line (as par=value):

Note: the resulting miniCBF files take full advantage of the "signed 32-bit integer" data type used for storing the image array (see "X-Binary-Element-Type:" item). MOSFLM only supports pixel values from (and including) -2 and higher. It will therefore interpret other pixel markers below that threshold incorrectly, leading to wrong or completely failing processing (thanks to Harry Powell for confirmation). We are working on a MOSFLM-compatibility option for our hdf2mini-cbf converter for the next release. XDS will interpret the full "signed 32-bit integer" range correctly, but we have no information for any other data-processing package.


Using dectris-neggia.so to read HDF5 files in XDS directly.

A collaboration between Dectris and the XDS developers one can now specify an external library, such that XDS can read HDF5 files directly. See eg. Dectris.

The use of this in autoPROC is not yet automatic. Meanwhile you can do:

% process -h5 data/abc_1_master.h5 \
           autoPROC_XdsKeyword_LIB=/path/to/dectris-neggia.so \
           -d aP-01 >& aP-01.txt

That is:

The dectris-neggia.so library is available from Dectris for CentOS6 and CentOS7. It is also available in source form from GitHub.

See also recent CCP4bb discussion.