Content:


Introduction

The notes below started in March 2016, when these datasets were starting to come out from various beamlines/synchrotrons. In the meantime, the HDF5 format (in it's original Dectris version or the NeXus format) can be considered standard and mainstream. So we would expect autoPROC to work with those datasets as-is and without issues.

However, there could still be issues - see e.g.


The Eiger detector is a new detector from Dectris that also produces data in a new format: HDF5. We added support for this into autoPROC based on Eiger 16M data provided by Dectris and SLS.

Notes:

  • 20160307: the generation of diffraction images with overlaid predictions is not yet supported

The solutions shown below (via imginfo wrapper scripts) are not our preferred way of handling these newer detectors. We hope that updates to the software at beamlines will result in missing data to be added or incorrect items being corrected.


Eiger 16M at SLS, X06SA (PX-I)

See also:

Running

% process -h5 some_master.h5 -d outdir | tee out.lis

should be all that is required for processing. All other standard autoPROC flags and options can be used.

Thanks to Dirk Reinert (Boehringer-Ingelheim), Meitian Wang (SLS), Andreas Foerster (Dectris), Joachim Diez (Expose), Stefan Brandstetter (Dectris), Zac Panepucci (SLS), Kay Diederichs (Konstanz) and many more.


Eiger 4M at ESRF, MASSIF-3

See also:

  • 20160901: data collected after this date do not need any special scripts or flags to be processed by autoPROC (Thanks to G. Fischer, I. Cornaciu and D. von Stetten for informing us).

Earlier datasets:

  • Data prior to 20160301 had missing detector distance and beam centre information in the master.h5 file. In order to process the data these need to be set by hand, e.g. with (using the correct values of course):
         process dist=123.45 beam="1234.5 1234.6" ...
  • 20160307: the metadata in the master.h5 files is not yet complete (missing omega values and oscillation range). This requires the use of an imginfo wrapper script and setting the oscillation range by hand.
  • 20160307: one module is corrupted - this requires manually masking.

After downloading imginfo_ESRF_Eiger4M.sh and making it executable via

% chmod +x imginfo_ESRF_Eiger4M.sh

you can run something like

% process -h5 some_master.h5 \
        autoPROC_XdsKeyword_UNTRUSTED_RECTANGLE="257 483 551 809" \
        imginfo=`pwd`/imginfo_ESRF_Eiger4M.sh osc=0.05 \
        -d outdir | tee out.lis

(obviously adapting to your local setup and dataset characteristics).

Thanks to David von Stetten (ESRF) for test data and discussions.


Eiger 9M at APS 21-ID-D

See also:

Important:

  • at the moment (20160311), the Eiger 9M at APS 21-ID-D also requires a so-called imginfo-wrapper (to accomodate for missing/incorrect goniostat settings in the metadata section of the master.h5 HDF5 file)
  • download file imginfo_APS_ID21D_Eiger9M.sh (and make it executable)
  • run (with the correct osciallation range setting):
      % process -h5 h.001_master.h5 imginfo=`pwd`/imginfo_APS_ID21D_Eiger9M.sh osc=0.5 -d 01 | tee 01.lis

Thanks to Joe Brunzelle (LS-CAT) for providing test data.


Eiger 9M at Soleil Proxima2

See also:

Important:

  • as of 20170705 (possibly earlier) Eiger datasets work out-of-the-box in autoPROC
  • older datasets (up to 20160126) had both Phi and Omega marked as rotation axis; this is now fixed.
  • newer datasets (20160131 to at least 20160415) have the wrong sensor-thickness value recorded (or: the wrong unit for the given value):
        /entry/instrument/detector/detectorSpecific/eiger_fw_version = eiger-1.5.2-101.git5b1f904.release
        /entry/instrument/detector/detectorSpecific/software_version = 1.5.2
        /entry/instrument/detector/detector_number = E-18-0102

        ...

        /entry/instrument/detector/detectorSpecific/eiger_fw_version = eiger-1.6.1-104.git3f17ecb.release
        /entry/instrument/detector/detectorSpecific/software_version = 1.6.1
        /entry/instrument/detector/detector_number = E-18-0102
            % process -h5 Lyso_U2_1_master.h5 imginfo=`pwd`/imginfo_Soleil_Proxima2_Eiger9M.sh autoPROC_XdsKeyword_SENSOR_THICKNESS=0.45 -d 01 | tee 01.lis
    • Dectris are aware of that problem and it will be fixed in the next release of the EIGER firmware (thanks to Andreas Foerster, Dectris, for following this up)

Note: autoPROC releases from 20160324 onwards contain already this wrapper, so running

 % process -h5 h.001_master.h5 imginfo=imginfo_Soleil_Proxima2_Eiger9M.sh autoPROC_XdsKeyword_SENSOR_THICKNESS=0.45 -d 01 | tee 01.lis

should work.

Thanks to Martin Savko and Bill Shepard (Soleil) as well as Nicholas Keep (Birkbeck, London) for providing test data.


Eiger 4M at Photon Factory BL1A

See also:

Important:

  • at the moment (20160309), the Eiger 4M at PF BL1A also requires a so-called imginfo-wrapper (to accomodate for missing/incorrect goniostat settings in the metadata section of the master.h5 HDF5 file)
  • download file imginfo_PF_BL1A_Eiger4M.sh (and make it executable)
  • run (with the correct osciallation range setting):
      % process -h5 h.001_master.h5 imginfo=`pwd`/imginfo_PF_BL1A_Eiger4M.sh osc=0.5 -d 01 | tee 01.lis
 

Note: autoPROC releases from 20160324 onwards contain already this wrapper, so running

 % process -h5 h.001_master.h5 imginfo=imginfo_PF_BL1A_Eiger4M.sh osc=0.5 -d 01 | tee 01.lis

should work.

Thanks to Takaaki Fukami (Chugai Pharmaceutical) for providing feedback and test data.


Eiger 9M at SPring-8 BL32XU

Running

% process -h5 some_master.h5 ReverseRotationAxis=yes -d outdir | tee out.lis

should be all that is required for processing. All other standard autoPROC flags and options can be used.

Thanks to Keitaro Yamashita (SPring-8) for providing information and test data.


Eiger 16M at MAX IV (BioMAX)

The master file contains the definition of the rotation axis, but autoPROC currently doesn't read this item. So one needs to run

% process autoPROC_XdsKeyword_ROTATION_AXIS="0 -1 0" -h5 some_master.h5 ...

for now.

Thanks to Jie Nan and Uwe Mueller for providing test data and help.


Eiger 16M at Australian Synchrotron (MX-2)

These issues have been fixed as of 20180129 (Thanks Jun Aishima frmo the Australian Synchrotron for letting us know)

As of 20170920, HDF5 datasets from the Eiger 16M at MX-2 of the Australian Synchrotron record inconsistent information about rotation axes in the master.h5 file: both "Omega" and "Phi" are marked as rotation axes. Both will contain start and end values according to the oscillation angle and the "imginfo" tool from autoPROC returns something like

 >>> Image format detected as HDF5/Eiger

     Image number 1/2000

 ===== Header information:
 date                                = 20 Sep 2017 15:31:54.976
 distance                       [mm] = 380.032
 wavelength                      [A] = 0.953729
 sensor thickness               [mm] = 0.450
 Phi-angle (start, end)     [degree] = 180.00000 180.10001
 Oscillation-angle in Phi   [degree] = 0.10001
 Omega-angle (start, end)   [degree] = 180.00000 180.10001
 Oscillation-angle in Omega [degree] = 0.10001
 Chi-angle                  [degree] = 0.00000
 Kappa-angle                [degree] = 0.00000
 2-Theta angle              [degree] = 0.00000
 Pixel size in X                [mm] = 0.075000
 Pixel size in Y                [mm] = 0.075000
 Number of pixels in X               = 4150
 Number of pixels in Y               = 4371
 Beam centre in X               [mm] = 155.187
 Beam centre in X            [pixel] = 2069.160
 Beam centre in Y               [mm] = 167.042
 Beam centre in Y            [pixel] = 2227.230
 Overload value                      = 38079

 WARNING: more than one angle marked with different start/stop values!
 WARNING: oscillation around more than one axis?

Obviously, not both axes (Omega and Phi) would have rotated during the experiment (since this would correspond to a total rotation of 0.2 degree in case of a Kappa-goniostat with Kappa=0.0 ... or of a total rotation of 0.0 if they rotate in opposite directions at that setting). So the above will confuse autoPROC and could result in failed or poor processing.

Until this has been fixed at the beamline - and in order to (re)process already collected datasets - a workaround is required to patch those header information up. Please download the imginfo_MX-2_AustralianSynchrotron_20171009.sh script, make it executable via

% chmod 0755 /where/ever/imginfo_MX-2_AustralianSynchrotron_20171009.sh

and use it in autoPROC:

% process imginfo=/where/ever/imginfo_MX-2_AustralianSynchrotron_20171009.sh ...

Please note that

  • for those Eiger@MX-2 datasets no "ReverseRotationAxis=yes" or "BeamCentreFrom" keyword is required
  • that datasets from the ADSC detector at MX-1 still require
     % process BeamCentreFrom=header:y,x ReverseRotationAxis=yes ...

Thanks to Richard Birkinshaw (University Melbourne) for providing example datasets for testing. The above script will be included into the next autoPROC release.


Eiger 16M at CHESS

Example data from May 2021 shows

imginfo -v -v -h5 some_master.h5
     /entry/instrument/detector/description = "Dectris EIGER2 Si 16M"
     /entry/instrument/detector/detector_number = "E-32-0123"
     /entry/instrument/detector/sensor_material = "Si"
     ...
     /entry/instrument/detector/detectorSpecific/ntrigger = 1
     /entry/sample/goniometer/omega[0] = 340.000000
     /entry/sample/goniometer/omega[1] = 340.200000
     /entry/sample/goniometer/omega[2] = 340.400000
     /entry/sample/goniometer/omega_end[0] = 340.200000
     /entry/sample/goniometer/omega_end[1] = 340.400000
     /entry/sample/goniometer/omega_end[2] = 340.600000
     /entry/sample/goniometer/omega_range_average = 0.200000 degree
     /entry/sample/goniometer/omega_range_total = 360.000000 degree
     /entry/sample/goniometer/kappa[0] = 0.000000
     /entry/sample/goniometer/kappa[1] = 0.100000
     /entry/sample/goniometer/kappa[2] = 0.200000
     /entry/sample/goniometer/kappa_end[0] = 0.100000
     /entry/sample/goniometer/kappa_end[1] = 0.200000
     /entry/sample/goniometer/kappa_end[2] = 0.300000
     /entry/sample/goniometer/kappa_range_average = 0.100000 degree
     /entry/sample/goniometer/kappa_range_total = 180.000000 degree
     /entry/sample/goniometer/chi[0] = 0.000000
     /entry/sample/goniometer/chi[1] = 0.100000
     /entry/sample/goniometer/chi[2] = 0.200000
     /entry/sample/goniometer/chi_end[0] = 0.100000
     /entry/sample/goniometer/chi_end[1] = 0.200000
     /entry/sample/goniometer/chi_end[2] = 0.300000
     /entry/sample/goniometer/chi_range_average = 0.100000 degree
     /entry/sample/goniometer/chi_range_total = 180.000000 degree
     /entry/sample/goniometer/phi[0] = 0.000000
     /entry/sample/goniometer/phi[1] = 0.200000
     /entry/sample/goniometer/phi[2] = 0.400000
     /entry/sample/goniometer/phi_end[0] = 0.200000
     /entry/sample/goniometer/phi_end[1] = 0.400000
     /entry/sample/goniometer/phi_end[2] = 0.600000
     /entry/sample/goniometer/phi_range_average = 0.200000 degree
     /entry/sample/goniometer/phi_range_total = 360.000000 degree

This is a very confused definition of rotation axes ... clearly we're not doing an experiment where all those angles are rotating at the same time (and a goniostat with both a Chi and Kappa axis is rather odd). This can easily lead to incorrect interpretation of both the starting angle (340 or 0?) and oscillation angle.

Further we then have

 omega axis   = -1.00000  0.00000  0.00000
 fast pixel vector = -1.00000  0.00000  0.00000
 slow pixel vector =  0.00000 -1.00000  0.00000

Which seems incorrect: the correct rotation axis is (1,0,0), i.e. we would need to run with

process ReverseRotationAxis=yes ...

However, because of those confusing goniostat angle information, autoPROC/imginfo is unable to uniquely determine the rotation axis and will not pick up that defined Omega axis vector anyway ... so all a bit messy and complicated for those datasets.


HDF5 converter

autoPROC provides a converter for HDF5 datasets into CBF format (including full mini-cbf headers). It supports both bitshuffle as well as LZ4 compression. Please note that is will only convert HDF5 datasets without applying any of the necessary fixes discussed above (which are necessary because of issues mostly at the metadata population stage). In order to obtain correct miniCBF headers, the metadata of the input HDF5 file needs to be correct.

The converter can be used to extract single images via e.g.

% hdf2mini-cbf -m some_master.h5 -image 47

or the complete dataset using

% hdf2mini-cbf -m some_master.h5

For additional help please see

% hdf2mini-cbf -h
USAGE: hdf2mini-cbf [-h] [-linkrange <i1> <i2>] [-image <imgnum>] -m <master.h5> [-o <prefix>] [-uncompressed|-compressed|-gzip-level <level>]

Additional options/features are

  • directly writing compressed (gzip) CBF files (--compressed)
  • selecting a subset of images via their external link files (-linkrange)

An additional tool is provided for very fast conversion by distributing the extraction/conversion to multiple compute nodes. See

% aP_convert_hdf5 -h

for details.

USAGE: aP_convert_hdf5 [-h] [-v] [-t] [-compressed] -m <master> -o <prefix>

       -h                    : show help

       -v                    : increase verbosity (default = 0)

       -t                    : don't actually run the command(s), but show
                               what would be done

       -m <master>           : HDF5 *_master.h5 file

       -o <prefix>           : prefix (default=__169826_) for output mini-cbf
                               files (default=__169826__######.cbf)

       -compressed           : (optional) write *.cbf.gz files directly

Some settings - current default are adequate for SLS computing setup - can be given on the command-line (as par=value):

  • autoPROC_Hdf2Cbf_NodesFile: ASCII file with list of compute nodes (one per line, lines starting with '#' and empty lines are ignored); default = "/etc/xds_par_nodes".
  • autoPROC_Hdf2Cbf_Nodes: allow users to define the compute nodes directly as space-separated lis; default = "".
  • autoPROC_Hdf2Cbf_SshCommand: command to access each compute node; default = "ssh -x".

Note: the resulting miniCBF files take full advantage of the "signed 32-bit integer" data type used for storing the image array (see "X-Binary-Element-Type:" item). MOSFLM only supports pixel values from (and including) -2 and higher. It will therefore interpret other pixel markers below that threshold incorrectly, leading to wrong or completely failing processing (thanks to Harry Powell for confirmation). We are working on a MOSFLM-compatibility option for our hdf2mini-cbf converter for the next release. XDS will interpret the full "signed 32-bit integer" range correctly, but we have no information for any other data-processing package.


Using dectris-neggia.so to read HDF5 files in XDS directly.

A collaboration between Dectris and the XDS developers one can now specify an external library, such that XDS can read HDF5 files directly. See eg. Dectris.

The use of this in autoPROC is not yet automatic. Meanwhile you can do:

% process -h5 data/abc_1_master.h5 \
           autoPROC_XdsKeyword_LIB=/path/to/dectris-neggia.so \
           -d aP-01 >& aP-01.txt

That is:

  • use option "-h5" to specify the master file
  • tell autoPROC to add the "LIB=path..." instruction in the XDS.INP file

The dectris-neggia.so library is available from Dectris for CentOS6 and CentOS7. It is also available in source form from GitHub.

See also recent CCP4bb discussion.