All members of the Scientific Community--at all levels of experience--are invited to participate as Challengers, and/or as Assessors.
In the Challenge Phase (July 2015-Apr 2016), 27 participants created and submitted 66 reconstructions of the challenge targets using the supplied raw image data. Challengers were encouraged to perform their own movie frame alignment, frame summation, and particle picking. Alternately, they could begin with pre-aligned, summed images and/or original author-provided particle positions. For each submission, challengers filled out a questionnaire (overview) and provided the following data:
final unmasked map with filtering/sharpening
final unmasked map without filtering or sharpening
half-maps and mask used for FSC calculation
CTF, coordinates, euler angles for each particle image used in the reconstruction
We are now in the Assessment Phase (Nov 2016-mid Feb 2017), and again welcome all interested to participate. Following initial review period by the map committee, the challenge data and files are now publicly available (entry authorship and software suppressed) for anyone to assess. Assessment methods could include statistical analyses, resolution estimation, or coordinate model fitting. A few suggestions gathered from software developers are summarized below. The intention is to enable comparisons of the various packages available and their options in a positive spirit. During this period, assessors are strongly encouraged to share their plans and short result summaries on this website using the Assessment Registration Form. Assessment results will be more fully presented and discussed via a workshop (~Spring 2017) as well as via manuscript submissions to a Journal special issue.
To sign up as an assessor and/or send a comment or question to the committee, click here. To review/download the submitted challenge entries, click here.
Map Challenge Assessment Guidance
We've gathered some suggestions here about how to proceed with comparisons of the map submissions.* These are not meant to be prescriptive; results from other approaches are also welcome.
FSC curves based on provided half-maps and masks have been prepared for each map challenge entry (link). In most cases the results are consistent with or very close to the submitter-reported resolution, but this initial analysis cannot be used to directly compare submissions, because of differences in masking and map sizes, and thus convolution effects. FSC is a fundamental similarity metric, but its use in standard cryoEM practice has been problematic because of the maps being compared. Many suggestions were made on how to carry out follow-up analyses:
Apply a single, common mask to all entries belonging to a target (e.g. 15-20 A average of all entries, with soft-edges or low-pass filtered).
Employ other methods/techniques such as:
post-process phase randomization (e.g., to investigate effects of different masking on FSC)
mask artifact compensation
determine FSC error
Calculate map-model FSCs
Map Density Analysis
Images of each map both by itself and aligned to a common model are provided for reference (link), but further investigation is warranted, as variations in density appearance may be due to differences in power spectra and/or filtering/sharpening schemes. Some suggestions:
Both overall images and close-up views are desirable; for comparison it is best to have the exact same view
Both well-ordered regions and not-so well ordered regions should be investigated
Views containing slices (slabs or grey-scale planes) could be useful
Apply a common filtering/sharpening scheme to the unfiltered (raw) map entries for a target, bringing power spectra to a “common denominator” for density comparison
Along this line, view density across maps attenuated at a common low resolution, and then walk the attenuation towards higher resolution
Density quality could be investigated by fitting defined portions of each map using modeling tools (e.g. compare rmsd's of multiple models).
*suggestion credits: Maya Holmdahl, Roberto Marabini, Sjors Scheres, Bernard Heymann, Niko Grigorieff, Pawel Penczek, Ed Egelman, Steve Ludtke, Scott Stagg, Marin van Heel
Map Committee meets monthly to identify challenge targets, goals, and parameters
Requests to 3DEM community members for public deposition of raw image Datasets; website development
Raw image data for all targets available for download at EMPIAR
JUL 2015-APR 2016
Pre-Challenge Announcement, Challenger and Assessor Registration Opens
Map Submission Opens
Registered Participants may apply for SDSC Gordon Supercomputer Usage
Challenge Data initial assessments, metadata extraction, preparation for release (Map Committee)
Review Period (Map Committee)
4 Nov 2016-17 Feb 2017
Assessors invited to perform analyses and comment
Spring 2017-date TBD
2 day Workshop for all challenge participants -- Committee, Challengers, Assessors
Challenge Writeups (multiple articles) for a Journal Special Issue
Six challenge targets are based on recently described 3DEM single particle structure determinations with data collected as multiple-frames-per-second movies, using the latest generation of detectors. One additional target is based on simulated (in silico) images. For each experimental target, the original raw micrograph movie frames are data available for download at EMPIAR, PDBe's raw 3DEM image data archive. Summed image data are also available, either as full micrographs or as picked particle stacks. In one case aligned frames are also deposited. Particle positions and defocus values from the raw data depositors are also available for download and may optionally be used by challengers in their reconstructions.