Please also check the autoPROC home page and the news page for updates and snapshot releases.
As of 1st August 2024 (or maybe slightly earlier), a new XDS version has been published on the XDS page. Although we had access to the 20240712 binary since 24th July 2024, those few days didn't allow for a very detailed analysis of the performance of that version.
There are definitely some very significant changes in the resulting data that are most likely due to the changes in the background estimation hinted at in the Release Notes. We are still investigating these effects on the full software stack from integration, scaling, analysis of anisotropy, determination of a cut-off surface and refinement. This will take a fair amount of time - with the holiday season in full swing and the previous XDS binary stop working by the end of August 2024 ...
Since there is no mechanism to switch between different algorithms within the new XDS binary, we highly recommend that users keep their previous XDS binary available and run their own tests as soon as possible (August 2024 deadline!). This can be done by setting the variable BDG_TOOL_XDS to the relevant XDS binary, e.g. via
process BDG_TOOL_XDS=/where/ever/old/xds_par ... process BDG_TOOL_XDS=/where/ever/new/xds_par ...
We hope to analyse the changes in more detail - but it is highly likely this will not come to a definite conclusion by the time the previous XDS binary expired.
Recent datasets from those SPRING-8 beamlines showed a few problems due to specific ways of the way they are created.
The example datasets we had access to contain at least some attributes in UTF8 encoding instead of the much more common ASCII one. That causes at least the Durin plugin problems and results in an XDS failure:
... Durin plugin error: src/plugin.c - line 202 in plugin_open ... src/file.c - line 663 in det_visit_callback: H5OVisit callback: Error reading NX_class attribute on group entry/sample/transformations ... generic_getfrm: could not open
One can check this by running
h5dump -a entry/sample/transformations/NX_class your_master.h5which returns
ATTRIBUTE "NX_class" {
DATATYPE H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_UTF8; <<<<<
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "NXtransformations"
}
}
instead of the usual
ATTRIBUTE "NX_class" {
DATATYPE H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII; <<<<<
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "NXtransformations"
}
}
The exact provenance of those differences is not yet clear, but one can "fix" any master file by using a short h5py script:
#!/usr/bin/env python3
import sys
import h5py
f = h5py.File(sys.argv[1],'r+')
g = f['entry/sample/transformations']
# https://docs.h5py.org/en/stable/strings.html
if (isinstance(g.attrs['NX_class'],str)):
print("entry/sample/transformations/NX_class (str==UTF-8) : \"" + g.attrs['NX_class'] + "\"")
g.attrs['NX_class'] = str.encode(g.attrs['NX_class'])
that could be run e.g. as
fix.py your_master.h5
NOTE: this will overwrite the existing file!
Thanks to BL45XU staff for providing example data to trace that issue.
The pixel-mask information (at "/entry/instrument/detector/detectorSpecific/pixel_mask") here is stored as a bitshuffle-compressed dataset - which is not very common: most HDF5 datasets store the pixel-mask as uncompressed uint32 array (a few outliers use int32 or int64 here). The Durin plugin doesn't support (it seems) reading such compressed pixel masks - although it does support bitshuffle compression for the actual image data (but doesn't use "standard" HDF5 library filter mechanism as far as we can see).
We've created an initial version of the Durin plugin (for Linux only at the moment, Durin_20241216_bitshuffle_pixel-mask.zip) that should support those compressed pixel-masks. Within autoPROC, you can use this via
process autoPROC_XdsKeyword_LIB=/where/ever/Durin_20241216_bitshuffle_pixel-mask/durin-plugin.so ...which should show (in XDS output files) something like this:
...
[generic_data_plugin] - INFO - generic_open
+ library = <./durin-plugin.so>
+ template_name =
+ dll_filename = <./durin-plugin.so>
+ image_data_filename =
filter name ="bitshuffle; see https://github.com/kiyo-masui/bitshuffle"
bitshuffle filter is available now since H5Dread (of pixel-mask) triggered loading of the filter.
...
A new XDS version has been made available that we are currently looking into.
process autoPROC_AnalyseForPoorImageRanges=no ...