[sharp-discuss] autosharp not reading column headers in mtz files

Fri Aug 17 11:59:38 CEST 2012

The problem is I can't now reproduce the problem and I deleted all my files from yesterday in a clean-up exercise this morning (including the .mtzlog files, unfortunately)

- I can't see any significant difference between the output from different versions of mtzdump, apart from the version number

- when I run autoSharp now, the mtzlog file refers to mtzdump version 6.1 which must be an old version, but I don't know how Sharp decides which CCP4 version to use

so I'm puzzled

Phil


On 17 Aug 2012, at 10:26, Clemens Vonrhein wrote:

> Hi Phil et al,
> 
> looking at the source code of mtzdump.f, I can't see any difference
> apart from
> 
> * increasing maximum number of batches from 5000 to 10000 (but that
>   should only have an impact on unmerged multi-record files - not for the
>   files autoSHARP usually reads).
> 
> * introducing a version string (so that MTZDUMP can reports it's own
>   version and not the CCP4 version)
> 
> None of those should have any impact. Unless something deep in the
> CCP4 libraries has changed? Would it be possible to send me the output
> of
> 
>  % mtzdmp some.mtz > mtzdmp.log
> 
> for both versions (6.2.0 and 6.3.0) gor an MTZ file that gives such
> problems? I have been running SHARP/autoSHARP with 6.3.0 for quite
> some while now without problems.
> 
> Also: check if you haven't got a sharpfiles/datafiles/some.mtzlog file
> sitting there (a cached version of mtzdump output) with some bogus
> information.
> 
> And if you load up files through the interface (or by copying them
> into sharpfiles/datafiles):
> 
> * make sure to have new names for those files (not always using
>   "test.mtz" or "data.mtz")
> 
> * or delete any *.mtzlog file that might be the result of a previous
>   file ...
> 
> ------------------------------------------------------------------
> 
> The easiest solution is to not using MTZ files in autoSHARP but rather
> *.sca files. Those can easily be written out by SCALA or AIMLESS in
> addition to the MTZ files. Just add
> 
>  OUTPUT POLISH    # for SCALA
> 
> or
> 
>  OUTPUT SCALEPACK # for AIMLESS (does POLISH also work?)
> 
> The advantage of using *.sca files in autoSHARP:
> 
> * SHELXD can read them directly.
> 
>   So we don't need to convert back from F/SIGF (in MTZ) to I/SIGI (in
>   a temporary *.sca file) with some subtle differences: the F's will
>   have gone through some truncating procedure and just squaring them
>   doesn't give you back the original intensities (which could be
>   negative).
> 
> * you save yourself some typing: no need to give column labels in the
>   autoSHARP input form
> 
> Some additional tips that could be relevant to this:
> 
> * make sure you have the correct scalepack format: the 'merged'
>   version is known to work correctly - see
> 
>     http://www.globalphasing.com/sharp/manual/autoSHARP1.html#datafiles
> 
> * make sure you have anomalous data in them (if it is required for
>   SIRAS/MIRAS or SAD/MAD)
> 
> * it is very helpful to give your MAD wavelengths good identifiers:
>   infl, hrem, lrem or peak
> 
>   The reason: SHELXC doesn't use f'/f" values to distinguish the
>   different wavelengths and their corresponding atomic scattering
>   properties. It wants names like "HREM", "LREM", "PEAK" or
>   "INFL". autoSHARP tries to figure these names out from the f'/f"
>   values that you give in the interface - but this can often be
>   wrong. So by giving those identifier names, you are helping
>   autoSHARP to tell SHELXC the right names. See also
> 
>     http://www.globalphasing.com/sharp/manual/autoSHARP2.html#iden
> 
>   Of course, you should still give the correct f'/f" values
>   (especially for wavelengths close to the edge) - from a good
>   fluorescence scan.
> 
> Cheers
> 
> Clemens
> 
> On Fri, Aug 17, 2012 at 09:33:53AM +0100, Phil Evans wrote:
>> 
>> Yesterday I got this error, and it seemed to be fixed by replacing mtzdump with an old version. This morning I tried restoring the new failing mtzdump, but it still seems somehow to run the old one and then works OK. Also I can't see any significant differences between the output of the new & old mtzdump, so I'm not at all sure this was indeed the problem
>> 
>> I'm deeply confused at this point
>> 
>> Phil
>> 
>> 
>> On 17 Aug 2012, at 08:55, Phil Evans wrote:
>> 
>>> 
>>> I'm not now convinced about this as I'm not sure I can repeat it - investigating further
>>> 
>>> Phil
>>> 
>>> On 16 Aug 2012, at 18:15, Phil Evans wrote:
>>> 
>>>> 
>>>> OK got it
>>>> 
>>>> The new CCP4 has a version of mtzdump which subtly changes its output format, and Autosharp scrapes the output log file to extract the information which it's complaining about
>>>> 
>>>> Immediate fix is to replace mtzdump with an older version
>>>> 
>>>> Phil
>>>> 
>>>> 
>>>> On 16 Aug 2012, at 17:18, <Stephen.carr at rc-harwell.ac.uk> <Stephen.carr at rc-harwell.ac.uk> wrote:
>>>> 
>>>>> Dear All,
>>>>> 
>>>>> I am experiencing a frustrating error when trying to run autosharp.  After inputting the data via the httpd interface, the program is having trouble extracting information from the input datasets and keeps giving the error
>>>>> 
>>>>> No column "F_NAT" of type "F" found in file
>>>>> 
>>>>> mtzdump reveals that there is indeed a column of this name, yet autosharp cannot seem to read it.
>>>>> 
>>>>> I have tried this with several different data-sets and each gives the same sort of
>>>>> error.  All data sets which give this problem have been processed/scaled etc. using ccp4 6.3.0
>>>>> which appears to output each data-set in two parts, the first (sub)-data-set contains
>>>>> HKL and FreeR_Flag columns only, while the second contains all the other columns.  I think
>>>>> this might be the cause of the problem, since running a data-set scaled etc using an older
>>>>> version of ccp4 did not give this error and the data are read with no problems.
>>>>> 
>>>>> Has anyone else observed something similar and does anyone have any suggestions/workarounds
>>>>> that will allow me to run autosharp with this data?
>>>>> 
>>>>> Any help would be greatly appreciated.
>>>>> 
>>>>> Best wishes,
>>>>> 
>>>>> Steve
>>>>> 
>>>>> Dr Stephen Carr
>>>>> Research Complex at Harwell
>>>>> Rutherford Appleton Laboratory
>>>>> Harwell Oxford
>>>>> Didcot
>>>>> Oxon OX11 0FA
>>>>> United Kingdom
>>>>> Email stephen.carr at rc-harwell.ac.uk
>>>>> tel 01235 567717
>>>>> 
>>>>> 
>>>>> This email and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorized recipient of the addressee, please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to this email.
>>>>> 
>>>>> Any views or opinions presented are solely those of the author and do not necessarily represent those of the Research Complex at Harwell.
>>>>> 
>>>>> There is no guarantee that this email or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
>>>>> 
>>>>> We use an electronic filing system. Please send electronic versions of documents, unless paper is specifically requested.
>>>>> 
>>>>> This email may have a protective marking, for an explanation, please see:
>>>>> http://www.mrc.ac.uk/About/informationandstandards/documentmarking/index.htm.
>>>>> _______________________________________________
>>>>> sharp-discuss mailing list
>>>>> sharp-discuss at globalphasing.com
>>>>> https://www.globalphasing.com/mailman/listinfo/sharp-discuss
>>>> 
>>>> _______________________________________________
>>>> sharp-discuss mailing list
>>>> sharp-discuss at globalphasing.com
>>>> https://www.globalphasing.com/mailman/listinfo/sharp-discuss
>>> 
>> 
>> _______________________________________________
>> sharp-discuss mailing list
>> sharp-discuss at globalphasing.com
>> https://www.globalphasing.com/mailman/listinfo/sharp-discuss
> 
> -- 
> 
> ***************************************************************
> * Clemens Vonrhein, Ph.D.     vonrhein AT GlobalPhasing DOT com
> *
> *  Global Phasing Ltd.
> *  Sheraton House, Castle Park 
> *  Cambridge CB3 0AX, UK
> *--------------------------------------------------------------
> * BUSTER Development Group      (http://www.globalphasing.com)
> ***************************************************************