Results - September 2008


Participating in the CASP8 structure prediction assessment CASP8 (Critical Assessment of Techniques for Protein Structure Prediction) was the main goal for POEM@HOME in between May and August 2008.

CASP is a biannual event assessing the quality of structure prediction techniques for various classes of proteins. In this assessment predictor "groups" are given the full amino-acid sequence of proteins ("targets") accompanied with explanations about the experimental method used to obtain the (still unknown) reference protein structure. Predictors then have to return a prediction based on the sequence within a given timeframe. Once all targets have been released, independent assessors compare the predictions with the reference structure and sort them by quality.
CASP8 lasted from 05.05.08 to 23.08.08 releasing a total of 128 targets in the server category. 57 targets were also assigned to human predictors. Targets were assessed by 152 human and 126 (automatical) server groups. As of September 2008 the assessment has ended and several reference structures were already released as well as the predicted structures from the server groups.

POEM@HOME participated with two different groups named 'POEM' and 'POEMQA'.

  • The POEM group created structures with homology modeling and fragment methods and then evaluated them by relaxation runs on POEM@HOME.
  • POEMQA selects the structures corresponding to the best energy out of a pool of all structures returned by the server groups. Long relaxation runs on POEM@HOME were needed to obtain the best energy, because models are not 100% convertible between two different prediction strategies. Note that POEM-QA only ranks the conformations generated by other (server) groups. The credit for finding these conformations should go to the groups generating them, not to us. We participated with this protocol, because quality assessment, i.e. the relative and absolute ranking of the trustworthiness of the predicted structures, is one of the most important outstanding challenges. Biologists and pharmacists can tolerate the uncertainty of existing protocols, if they can obtain an accurate measure of the quality of the predicted structure. Good quality assessment would broaden the use of computational protein structure prediction for real-life biological and pharmaceutical applications.

Currently the official results of the assessment are not yet available. The following graphs give you an overview of how well the group strategies worked.

The first two plots present the outcome for the template based modeling targets (Targets, which contain segments, that also occur in other proteins (Templates selected by our group)). The plots show the best of all submitted server structures and the best of the five submitted POEM (or POEMQA) conformations. Additionally, the mean TM-Score over all submitted structures is shown.


The quality measure used in all these plots is the TM-Score: A TM-Score above 0.9 corresponds to a very good prediction within experimental resolution; a TM-Score below 0.17 indicates a random structure and therefore a failed prediction.







Summary


In nineteen of twenty template based modeling targets, both POEMQA and PPOEM submitted a model with a TM-Score better than the mean TM-Score over all models or in other words: Both groups succeeded in this section.




The plots above show two example structures T0423 (left, orange: POEM, blue: experiment) and T0457 (right). They display the first submitted conformation of the POEM group. While T0423 was a perfect prediction, the beta-sheet in front of T0457 is a bit too short; the prediction is still very good regardless.

Bar graphs of the results in the Free-Modeling section are available in the next two graphs (Free-Modeling means: no template could be found).






The free-modeling graphs show, that the structure generation step used in the POEM group still needs some refinement. POEMQA on the other hand predicted results comparable to the experimental structure very often. This means, that the correct structure could be identified using POEM@HOME.




The upper image shows the first prediction of POEMQA in the Free-Modeling section for target T0415. Also in this model substantial agreement with the experimental structure can be observed.

In the final three bar graphs, targets, which were only investigated by the POEMQA group are displayed.







The best target submitted by POEMQA is better than the mean TM-Score over all server targets in on 87 of 90 cases or in other words: POEMQA did quite well. POEM@HOME can therefore be used to rate protein structures coming from various sources by quality.