Data Cruncher default environment update

The default environment for Data Cruncher interactive analyses has been updated to include more up-to-date versions of Python (upgraded to 3.9) and R (upgraded to 4.1).

GDC Datasets version update

As of December 17, GDC datasets available through the Data Browser and the API correspond to GDC Data Release 30.0.

Recently published apps

• GRIDSS/PURPLE/LINX Workflow, used for somatic genomic rearrangement detection and classification on WGS data. This workflow takes a pair of matched tumor/normal BAM files and produces allele-specific copy number of every base of the genome, overall sample purity and ploidy, annotated SV clusters and gene fusion predictions. Moreover, it outputs detailed visualisations of the rearrangements in the tumor genome via integrated Circos plots showing copy number changes, clustered SVs, derivative chromosome predictions and impacted genes.

• PURPLE CNV Calling Workflow, used for somatic CNV calling and purity and ploidy estimation on WGS data. It is based on PURPLE 2.51, and consists of two additional tools - AMBER and COBALT. The workflow first calculates B-allele frequency (BAF) with AMBER and read depth ratios with COBALT, which is then used by PURPLE to estimate the purity, ploidy and copy number profile of a tumor sample.

Metadata editing using manifest files just got easier

CGC provides the capability to modify metadata for multiple files in a project by using the Export metadata manifest and Edit metadata with manifest options in the File Browser. This release brings some major improvements to this feature:

• Support for different manifest file formats. Besides CSV, we have added support for the TSV file format.

• Use either file name or ID to identify a file. Files whose metadata is being edited can be specified using only file ID or file name (along with path) in the manifest file used with the Edit metadata with manifest option.

• Support for folders. The name column can contain file path within the project (along with the file name) if the file is in a folder instead of the project root.

Click Read more below to see the full list of improvements.

Recently published apps

We have just published and upgraded versions (from 2.17 to 2.22) of minimap2, a sequence alignment program that aligns DNA or mRNA sequences against a reference database, and minimap2 build index, a reference indexer for minimap2 aligner.

This week’s publishing streak also includes METAL, a tool for meta-analysis genome-wide association scans. METAL can combine either (a) test statistics and standard errors or (b) p-values across studies (taking sample size and direction of effect into account). A METAL analysis is a convenient alternative to a direct analysis of merged data from multiple studies.

Recently published apps

We have just published Picard FastqToSam, a tool that converts FASTQ files to an unaligned SAM or BAM file, and a set of seven Delly tools:

• Delly CNV for calling copy-number variants

• Delly Call, a structural variants caller

• Delly LR, a structural variants caller for long reads data

• Delly Sansa Annotate for annotating structural variants

• Delly Classify for classifying somatic or germline copy-number variants

• Delly Filter, a tool that filters structural variants

• Delly Merge for merging of structural variants in BCF format

Recently published apps

We have just published the following apps:

• CrossMap, a tool that converts genomic coordinates between different assemblies, and CrossMap Viewchain that prints the chain file for two assemblies in a human-readable format.

• VerifyBamID2 that estimates contamination of DNA samples from read data, accounting for ancestry information.

Recently published apps

We have just published DRAGMAP, the open source DRAGEN mapper/aligner that can be used to align single or paired-end reads (FASTQ) or an input BAM file. The app is available in the Public Apps gallery.

Recently published apps

We have just updated the content of our public app galleries with new GATK releases:

• GATK Pre-Processing For Variant Discovery 4.2.0.0 workflow is used to prepare data for variant calling analysis. The workflow consists of two major segments: alignment to reference genome and data cleanup operations that correct technical biases. Resulting BAM files are ready for variant calling analysis and can be further processed by other BROAD best practice pipelines, like Generic Germline Short Variant Per-Sample Calling workflow, Somatic CNVs workflow, and Somatic SNVs + INDELs workflow.

• GATK Generic Germline Short Variant Per-Sample Calling 4.2.0.0 workflow that calls germline variants in a WGS sample with GATK HaplotypeCaller, starting from an analysis-ready BAM file.

And six GATK 4.2.0.0 tools:

• GATK GatherBQSRReports tool that gathers scattered BQSR recalibration reports into a single file.

• GATK BaseRecalibrator tool that generates a recalibration table based on various covariates for input mapped read data.

• GATK ApplyBQSR tool that recalibrates the base quality scores of an input BAM or CRAM file containing reads.

• GATK HaplotypeCaller tool for calling germline SNPs and indels from input BAM file(s) via local re-assembly of haplotypes.

• GATK VariantFiltration tool used for filtering variants in a VCF file based on INFO and/or FORMAT annotations.

• GATK MergeVcfs, used for combining multiple variant files.

Cancer Data Service Explorer for CDS data now available through the CGC's visual interface

Cancer Data Service Explorer is an integrated dataset file explorer on the CGC that allows you to filter and select the exact data that you want to analyze further, and then perform seamless import into a controlled project on the CGC. The explorer currently works with data available through CDS and is accessed by clicking Data > Cancer Data Service Explorer on the main menu bar, while on the CGC's main dashboard. Learn more about how to search and import CDS data to the CGC.

Recently published apps

We’ve just published four tools from the OncoGEMINI 1.0.0 toolkit:

• OncoGEMINI Bottleneck that identifies somatic variants with increasing allele frequency in longitudinal data.

• OncoGEMINI Loh, a command tool that performs loss of heterozygosity analysis.

• OncoGEMINI Truncal that recovers variants that appear in all tumor samples, but are absent in the normal sample.

• OncoGEMINI Unique tool for identifying somatic variants unique to a subset of samples.

Import via DRS now available on the CGC

Introduction of the DRS client on Cancer Genomics Cloud Powered by Seven Bridges enables import of DRS files from known or open external sources, similar to what's already available on CAVATICA Powered by Seven Bridges and NHLBI BioData Catalyst Powered by Seven Bridges since April 2021. A known source is a DRS endpoint which is known to the platform, while open external sources are DRS endpoints that don’t require authorization. This release enables interoperability between the following platforms, by making corresponding DRS endpoints available as known sources:

• NHLBI BioData Catalyst Powered by Seven Bridges

• CAVATICA Powered by Seven Bridges

• Cancer Genomics Cloud Powered by Seven Bridges

Recently published apps

SBG Image Processing Toolkit

SBG Image Processing Toolkit consists of apps that enable various stages of machine learning image processing. Seamless integration between the tools of this toolkit provides an easy and logical analysis flow, while enabling support of various data types, preprocessing steps and utilizing computation capabilities of the CGC.

1. SBG Deep Learning Image Classification Exploratory Workflow is an image classifier pipeline that relies on the transfer learning approach. This allows the use of pre-trained models as the starting point for building a model adjusted to given image datasets. Furthermore, the pipeline allows training of the model for a variety of hyperparameter combinations in parallel by utilizing multiple GPU instances, while detailed metrics and visualizations help determine the best configuration that can later be used to make predictions on new data instances.

2. SBG Deep Learning Prediction is an image classifier tool that classifies unlabeled images based on labeled data. It is intended as a final step after the SBG Deep Learning Image Classification Exploratory Workflow. Testing different configurations in parallel with the exploratory workflow and finding the best model configuration for the given dataset, then using SBG Deep Learning Prediction with that configuration and all available labeled images as the training data provides the optimal training conditions which lead to the best classification results.

3. SBG Histology Whole Slide Image Preprocessing takes SVS histopathology images, removes various artifacts, and outputs the desired number of best quality tiles in PNG format that consist of at least 90% tissue.

4. SBG X-Ray Image Preprocessing Workflow performs the selected X-ray image enhancement algorithm: unsharp masking (UM), high-frequency emphasis filtering (HEF) or contrast limited adaptive histogram equalization (CLAHE).

5. SBG Stain Normalization involves casting an array of images in the stain colors of a target image. Stain normalization is used as a histopathology image preprocessing step to reduce the color and intensity variations present in stained images obtained from different laboratories.

6. SBG Medical Image Convert performs medical image format conversion. If the input data are medical images in a non-standard format (e.g. SVS, TIFF, DCM or DICOM), SBG Medical Image Convert converts them to PNG format.

7. SBG Split Folders organizes an image directory into the train and test subdirectory structure. These directories are necessary inputs for SBG Deep Learning Image Classification Exploratory Workflow and SBG Deep Learning Prediction.

HistoQC

HistoQC is an open-source quality control tool for digital pathology slides. It performs fast quality control to not only identify and delineate artefacts but also discover cohort-level outliers (e.g., slides stained darker or lighter than others in the cohort). It outputs an interactive user interface for easy viewing and understanding of the results.

Minimac4

Minimac4 is a genetic imputation algorithm that can be used to impute genotypes in a genomic region starting from a reference panel in M3VCF format and pre-phased target GWAS haplotypes.

BOLT-LMM

BOLT-LMM is a tool that tests the association between genotypes and phenotypes using a linear mixed model.

Manifest-based import improvements and data update on the CGC

With this release, we have improved and simplified the process of importing PDC, ICDC and CDS data on the CGC. The update includes better user experience—the upload interface is more modern and streamlined, while manifest files are now automatically uploaded to the destination project and can be reused, shared, and organized as any other file in the project.
This release also includes addition of new studies and updates of existing data for the PDC, ICDC, and CDS datasets.

Recently published apps

GSEAPreranked Workflow performs Gene Set Enrichment Analysis (GSEA). It is generated with an assumption that a differential expression analysis has been done before using the DESeq2 tool which is publicly available on the CGC. The GSEAPreranked Workflow consists of two tools, GSEA Input Prepare and GSEAPreranked. The GSEAPreranked tool represents a wrapper around the command-line tool that was developed by the BROAD Institute. The GSEA Input Prepare tool is based on the Python script developed by the Seven Bridges team to prepare the required input file formats for the GSEAPreranked tool.

Amazon EC2 GPU G4dn instances available on the CGC

With this update you can now use the newest Amazon EC2 GPU G4dn instances, in task executions and Data Cruncher analyses, as the industry’s most cost-effective and versatile GPU instances for deploying machine learning models.

G4dn instances feature NVIDIA T4 GPUs and custom Intel Cascade Lake CPUs, and are optimized for machine learning inference and small scale training.

NVIDIA drivers come preinstalled and optimized according to the Amazon best practice for the specific instance family and are accessible from the Docker container.

Recently published apps

ENCODE ChIP-Seq Pipeline 2 analysis studies chromatin modifications and binding patterns of transcription factors and other proteins. It combines chromatin immunoprecipitation (ChIP) assays with standard NGS sequencing. The workflow is based on ChIP-Seq 2 pipeline, developed by the ENCODE Consortium.

ENCODE ATAC-seq Pipeline performs quality control and signal processing, producing alignments and measures of enrichment. The Assay for Transposase-Accessible Chromatin followed by sequencing (ATAC-seq) experiment provides genome-wide profiles of chromatin accessibility. Briefly, the ATAC-seq method works as follows: loaded transposase inserts sequencing primers into open chromatin sites across the genome, and reads are then sequenced. The ends of the reads mark open chromatin sites. The workflow is based on the ENCODE ATAC-seq pipeline, developed by the ENCODE Consortium.

PDC Embargo Date implementation and datasets update

When working with PDC data on the CGC, embargoed files are now clearly labelled in the visual interface, with their embargo date being stored as a metadata field and inherited by outputs when such files are used as task inputs. EMBARGO means that some NCI data are under an embargo for publication and/or citation until a specific date, known as embargo date.

In addition to the introduction of embargo date, PDC data available on the CGC has been updated to match the PDC Data Release V1.11 of March 10, 2021.

GDC Datasets version update

As of May 27, GDC datasets available through the Data Browser and the API correspond to GDC Data Release 29.0.

New Command-line Uploader released

The new Command-line (CLI) Uploader, just released as part of the existing Seven Bridges CLI tool, becomes the primary recommended tool for performing large scale uploads to the CGC. The Uploader is easy to install and use, and is a resilient and performant command line application that provides users with a secure and reliable way of uploading data to the CGC.

The legacy Command line uploader will remain functional until August 2021, before being officially deprecated. Along with the legacy CLI Uploader, Desktop Uploader is also planned to be deprecated in August 2021, as Web Uploader is available through the CGC’s visual interface (since September 2020). Find out more about the new CLI Uploader in our documentation.

Recently published apps

GENESIS Update Null Model for Fast Score Test updates null model file obtained with the GENESIS Null model workflow so that it can be used in the GENESIS Single Variant Association Testing workflow in fast score mode.

CWL v1.2 available on the CGC

The CGC now supports Common Workflow Language (CWL) version v1.2. The new version of CWL brings a major new functionality - conditional execution of workflow steps, as well as several minor features and improvements. For the detailed change log please see the CWL CommandLineTool specification and the CWL Workflow specification.

The new CWL version v1.2 is a backwards-compatible upgrade of version v1.1, meaning all v1.0 and v1.1 features are still supported in v1.2. To upgrade a v1.0 or v1.1 app to v1.2, simply edit the app and the next version you save can automatically be upgraded to v1.2. Note that upgrading a workflow CWL version to v1.2 this way will not upgrade the CWL version of the tools in the workflow.

Apps using CWL v1.0 and v1.1 versions are still supported and can be used in workflows in combination with CWL v1.2 apps.

Network access control per Project available on the CGC

The CGC has added another layer of security protecting your data. Researchers can now choose from two options for controlling network access for each Project. This feature defines the network access permissions for both Tasks (tools and workflow executions) and Data Cruncher analyses (interactive analysis environments).

When setting up a project, users can choose to deny network access for all executions, thus ensuring even higher security and compliance standards in the execution environment provided by Seven Bridges. This restricted option will be the default selection for all new Projects. This additional security feature will enhance the safety of data during analysis in the cloud for all apps and notebooks. This change will not affect pulling of externally hosted Docker images or access to project files that point to externally hosted datasets, which means that access to common public datasets such as TCGA will not change. Access to the CGC API will also be available from the execution environment.

Recently published apps

The following apps were published in CWL1.x:

• SRA Toolkit 2.10.8 - NCBI’s collection of tools and libraries for accessing data in Sequence Read Archives format (SRA).

• SRA Download and Set Metadata a workflow that allows for downloading full SRA datasets and populating any metadata information that goes with the dataset

• AnnotSV 3.0.7 - structural variant annotation and raking tool.

• IsoformSwitchAnalyzeR 1.12.0 - a tool for differential splicing analysis, it performs statistical identification of the isoform switching while comparing two sample groups.

• DRIMSeq 1.16.1 - performs differential transcript usage (DTU) analyses using Dirichlet-multinomial generalized linear models.

• DEXSeq 1.36.0 - toolkit for testing differential exon usage in comparative RNA-Seq experiments.

• Differential Exon Usage with DEXSeq 1.36.0 - a workflow constructed out of DEXSeq tools, meant for a comprehensive differential splicing analysis.

Recently published apps

The following apps were published in CWL1.x:

• Single Cell Multi Sample Pairwise Differential Expression Workflow - pipeline that performs differential expression analysis on single cell data between pairs of user defined conditions.

• Minimap2 v2.17 - a versatile sequence alignment program that aligns DNA or mRNA sequences against a large reference database, tailored for use with long read sequencing technologies.

• fastqValidator 0.1.1 - checks format correctness of paired-end and single-end FASTQ files.

• FastP 0.20.1 - ultra-fast FASTQ preprocessor with useful quality control and data-filtering features, including adapter trimming, quality filtering, per-read quality pruning and many other operations with a single scan of FASTQ data.

• SBG convert SRA/BAM to FASTQ - an all-in-one tool that converts SRA/SAM/BAM/CRAM files into FASTQ format.

• SBG Create Expression Matrix - creates aggregated matrices from various types of inputs, most typically from abundance estimates produced by tools like RSEM, Salmon, or Kallisto.

• SHAPEIT 4.2.1 - phasing tool for sequencing and SNP array data.

• Regenie 2.0.1 - tool for whole genome regression analysis.

• UMI-tools 1.1.1 - tools for dealing with Unique Molecular Identifiers (UMIs)/Random Molecular Tags (RMTs) and single cell RNA-Seq cell barcodes.