Bastien
Chevreux
bach@chevreux.org

> Solomon Short (modified)

> A baby is Life's way of insisting that the universe give it another chance.

# Assembly of Ion Torrent data with MIRA3

## Introduction

MIRA can assemble Ion Torrent type data either on its own or together with Sanger, 454 or Solexa type sequencing data (true hybrid assembly). Paired-end sequences coming from genomic projects can also be used if you take care to prepare your data the way MIRA needs it.

MIRA goes a long way to assemble sequence in the best possible way: it uses multiple passes, learning in each pass from errors that occurred in the previous passes. There are routines specialised in handling oddities that occur in different sequencing technologies

IonTorrent is - as of January 2014 - still a moving target: there are new protocols every few months and they've come a long way since their very first data sets. The data quality is now more or less comparable to better 454 data sets, at least from what I have seen. It might be that you need to fetch the latest MIRA version available to get the best possible assembly.

### Some reading requirements

This guide assumes that you have basic working knowledge of Unix systems, know the basic principles of sequencing (and sequence assembly) and what assemblers do.

While there are step by step walkthroughs on how to setup your Ion Torrent data and then perform an assembly, this guide expects you to read at some point in time

- the mira_usage introductory help file so that you have a basic knowledge on how to set up projects in mira for Sanger sequencing projects.

- and last but not least the mira_reference help file to look up some command line options.

## Walkthrough: unpaired data

This walkthrough will use two data sets for E. coli strain DH10B made available by IonTorrent and will show you the main steps you need to perform to get assemblies going.

### Preparing your file system

Note: this is how I set up a project, feel free to implement whatever structure suits your needs.

    arcadia:$ mkdir dh10b
    arcadia:$ cd dh10b
    arcadia:dh10b$ mkdir origdata data assemblies

Your directory should now look like this:

    arcadia:dh10b$ ls -l
    drwxr-xr-x 2 bach users 48 2011-08-12 22:43 assemblies
    drwxr-xr-x 2 bach users 48 2011-08-12 22:43 data
    drwxr-xr-x 2 bach users 48 2011-08-12 22:43 origdata

Explanation of the structure:

- the `origdata` directory will contain the 'raw' result files that one might get from sequencing. In our case it will be the ZIP files from the IonTorrent site.

- the `data` directory will contain the preprocessed sequences for the assembly, ready to be used by MIRA

- the `assemblies` directory will contain assemblies we make with our data (we might want to make more than one).

### Getting the data for this walkthrough

The data sets in question are

1. E. coli DH10B, PGM run B13-328 which you can download from (download the SFF). This data set, subsequently nicknamed B13, contains data from the 316 chip with reads of an average size of ~100bp.

2. E. coli DH10B, PGM run B14-387 which you can download from . This data set, subsequently nicknamed B14, contains data from the 314 chip which IonTorrent uses to show off the "longer reads" capability of its sequencer. "Longer" meaning in this case an average of ~220 bp, which is not bad at all.

Save the two ZIP files into the `origdata` directory should now look like this:

    arcadia:dh10b$ ls -l origdata
    -rw-r--r-- 1 bach bach 824002890 2011-08-15 21:43 B13_328.sff.zip
    -rw-r--r-- 1 bach bach 327926296 2011-08-14 20:32 B14_387_CR_0.05.sff.zip

Our data is still in ZIP files, let's get them out and put them into the `data` directory:

    arcadia:dh10b$ cd data
    arcadia:data$ unzip ../origdata/B13_328.sff.zip
    Archive:  ../origdata/B13_328.sff.zip
      inflating: B13_328.sff
       creating: __MACOSX/
      inflating: __MACOSX/._B13_328.sff
    arcadia:data$ unzip ../origdata/B14_387_CR_0.05.sff.zip
    Archive:  ../origdata/B14_387_CR_0.05.sff.zip
      inflating: R_2011_07_19_20_05_38_user_B14-387-r121336-314_pool30-ms_B14-387_cafie_0.05.sff
    arcadia:data$ ls -l
    -rw-r--r-- 1 bach bach 1721658336 2011-06-17 01:29 B13_328.sff
    drwxrwxr-x 2 bach bach       4096 2011-06-21 16:16 __MACOSX
    -rw-rw-r-- 1 bach bach  688207032 2011-07-28 23:31 R_2011_07_19_20_05_38_user_B14-387-r121336-314_pool30-ms_B14-387_cafie_0.05.sff

Oooops, quite some chaos ... IonTorrent included some unnecessary things (the `__MACOSX` directory) and gave their data files wildly different names. Let's clean up a bit here:

    arcadia:data$ rm -rf __MACOSX
    arcadia:data$ mv B13_328.sff B13.sff
    arcadia:data$ mv R_2011_07_19_20_05_38_user_B14-387-r121336-314_pool30-ms_B14-387_cafie_0.05.sff B14.sff
    arcadia:data$ ls -l
    -rw-r--r-- 1 bach bach 1721658336 2011-06-17 01:29 B13.sff
    -rw-rw-r-- 1 bach bach  688207032 2011-07-28 23:31 B14.sff

There, much nicer.

### Preparing the Ion Torrent data for MIRA

MIRA will need the base sequences, quality values attached to those bases and - if already present - clipping points for quality clips and sequencing adaptor clips.

The basic data type you will get from the sequencing instruments will be SFF files. Those files contain almost all information needed for an assembly, but SFFs need to be converted into more standard files before MIRA can use this information.

In former times this was done using 3 files (FASTA, FASTA quality and XML), but nowadays the FASTQ format is used almost everywhere, so we will need only two files: FASTQ for sequence + quality and XML for clipping information.

#### Tip

Use the sff_extract script from Jose Blanca at the University of Valencia. The home of sff_extract is: but I am thankful to Jose for giving permission to distribute the script in the MIRA 3rd party package (separate download on SourceForge).

The data sets B13 and B14 have short and long IonTorrent reads and we will want to assemble them together. For the sake of clarity, I want to name the file with a prefix: dh10b_b13b14.

    arcadia:data$ sff_extract -o dh10b_b13b14 B13.sff B14.sff
    Working on 'B13.sff':
    Converting 'B13.sff' ...  done.
    Converted 1687490 reads into 1687490 sequences.
    Working on 'B14.sff':
    Converting 'B14.sff' ...  done.
    Converted 350109 reads into 350109 sequences.

The parameters to `sff_extract` tell it to convert the SFFs named `B13.sff` and `B14.sff` into files where the names start with `dh10b_b13b14`. By default, a FASTQ file and a XML file in NCBI traceinfo will be created.

#### Do not hard clip sequences!

People "in the know" might want to tell `sff_extract` to simply dump hard-clip sequences into the FASTQ file via the -c argument. Hard-clipped means: the clipped sequence parts of a read are physically trimmed away, never to be seen again.

This is D I S C O U R A G E D !

Reason: unlike 454, IonTorrent actually uses actively the SFF feature to set different clipping points for quality and adaptor clips. This is useful information for MIRA. Furthermore, some of the quality control algorithms of MIRA use also the clipped part of a read to improve assembly quality with measurable effect. If a hard clip was performed on the sequences, these algorithms are not as effective anymore.

#### The XML TRACEINFO file

For the die hards out there who really do not want the XML TRACEINFO files: if MIRA gets only the sequence, it will use the usual 454/Roche convention to treat left and right lower case part of sequences as clipped and retain the uppercase middle part of a sequence. `sff_extract` adheres to this convention, and while the resulting assemblies are not quite as good as with the TRACEINFO XML, they're still better than with hard clipped sequences.

The conversion can take some time, the ~2 million IonTorrent reads from this example need approximately 2.5 minutes for conversion. Go grab a coffee, or tea, or whatever.

Welcome back. Your directory should now look something like this:

    arcadia:data$ ls -l
    -rw-r--r-- 1 bach bach  771462745 2011-08-19 22:24 dh10b_b13b14.fastq
    -rw-r--r-- 1 bach bach  441331004 2011-08-19 22:24 dh10b_b13b14.xml
    -rw-r--r-- 1 bach bach 1721658336 2011-06-17 01:29 B13.sff
    -rw-rw-r-- 1 bach bach  688207032 2011-07-28 23:31 B14.sff

Cool. Last step: we do not need the SFF files here anymore (we still have them as ZIP in the `origdata` folder), let's get rid of them:

    arcadia:data$ rm *sff
    arcadia:data$ ls -l
    -rw-r--r-- 1 bach bach  771462745 2011-08-19 22:24 dh10b_b13b14.fastq
    -rw-r--r-- 1 bach bach  441331004 2011-08-19 22:24 dh10b_b13b14.xml

### Writing a manifest configuration file

Good, we're almost there. Let's switch to the `assemblies` directory and create a subdirectory for our first assembly test.

    arcadia:data$ cd ../assemblies/
    arcadia:assemblies$ mkdir 1sttest
    arcadia:assemblies$ cd 1sttest

This directory is quite empty and the IonTorrent data is not present. And it does not need to be as we'll tell MIRA in the manifest configuration file where to find the data and what to do with it.

    # A manifest file can contain comment lines, these start with the #-character

    # First part of a manifest: defining some basic things

    # In this example, we just give a name to the assembly
    #  and tell MIRA it should assemble a genome de-novo in accurate mode
    # As special parameter, we want to use 4 threads in parallel (where possible)

    project = dh10b
    job = genome,denovo,accurate
    parameters = -GE:not=4

    # The second part defines the sequencing data MIRA should load and assemble
    # The data is logically divided into "readgroups", for more information
    #  please consult the MIRA manual, chapter "Reference"

    readgroup = SomeUnpairedIonTorrentReadsIGotFromTheLab
    technology = iontor
    data = ../../data/dh10b*

    # note the wildcard "dh10b*" part in the 'data' line above:
    #  if you followed the walkthrough and have the FASTQ and XML file,
    #  this will automatically load both files (which is what we want)

Save the above lines into a file, we'll use `manifest.conf` in this example.

    arcadia:/path/to/myProject$ ls -l
    -rw-r--r-- 1 bach users        756 2011-11-05 17:57 manifest.conf

### Starting the assembly

Starting the assembly is now just a matter of one line:

    arcadia:1sttest$ mira manifest.conf >&log_assembly.txt

Now, that was easy, wasn't it? In the above example - for assemblies having only Ion Torrent data and if you followed the walkthrough on how to prepare the data - everything you might want to adapt in the first time are the following entries in the manifest file:

- --project (for naming your assembly project)

- --job (perhaps to change the quality of the assembly to 'draft'

Of course, you are free to change any option via the extended parameters, perhaps change the default number of processors to use from 2 to 4 via -GE:not=4 or any other of the > 150 parameters MIRA has ... but this is covered in the MIRA main reference manual.

#### Low coverage

MIRA default parameters are optimised for assemblies with decent coverages. In the case of Ion Torrent, this means an average coverage of ≥ 20 to 25. If you are working on projects with lower coverage, please adapt a couple of parameters for low coverage data.

#### Do not use too high coverage

As with every sequencing technology, you should not use too much data as this is detrimental to assembly quality. Long story short: non-random sequencing errors will, at high coverage, occur frequently enough to look like valid repeat pattern, leading MIRA to create more contigs than necessary ... some of them with the sequencing errors looking like genuine sequence.

As a simple rule of thumb, coverage numbers between 50x and 80x are OK, coverage above can (and will) lead to problems.

## Walkthrough: paired data

### Getting the data

This walkthrough will use three data sets to assemble E.coli MG1655: one shotgun/fragment set and two paired-end data sets with different template sizes. Life has put the full data sets for download, but as using the full sets for a bacteria is absolute overkill (and also detrimental to assembly quality), they were also kind enough to prepare sub-sampled data sets which we will use.

You will find all files at

1. Set with single fragments (unpaired data), down-sampled to 40x coverage: C11-127_40X.sff.zip This data set, subsequently nicknamed C11, contains data from the 316 chip with reads of an average usable size of ~200bp.

2. Set with paired data from a long paired-end library with an average size of 3.5kb, down-sampled to 20x coverage: FRA-257_20X.sff.zip This data set, subsequently nicknamed FRA-257, contains data from the 316 chip with reads of an average usable size of ~xxxxxxxbp.

3. Set with paired data from a long paired-end library with an average size of 8.9kb, down-sampled to 20x coverage: C28-140_20X.sff.zip This data set, subsequently nicknamed C28, contains data from the 316 chip with reads of an average usable size of ~xxxxxxxbp.

4. File with the linker sequences used for the paired-end sequencing. This will be needed by `sff_extract`: LMP_Linkers.fasta.gz

Save all files into the `origdata` directory should now look like this:

    arcadia:origdata$ ls -l
    -r--r--r-- 1 bach bach 1826508536 Jun 14 00:04 C11-127_40X.sff.zip
    -r--r--r-- 1 bach bach  681739117 Jun 14 00:05 C28-140_20X.sff.zip
    -r--r--r-- 1 bach bach  537186774 Jun 14 00:06 FRA-257_20X.sff.zip
    -r--r--r-- 1 bach bach         91 Mar  7 20:05 LMP_Linkers.fasta.gz

Quickly unpack the tiny linkers file:

    arcadia:origdata$ gunzip LMP_Linkers.fasta.gz
    arcadia:origdata$ ls -l
    -r--r--r-- 1 bach bach 1826508536 Jun 14 00:04 C11-127_40X.sff.zip
    -r--r--r-- 1 bach bach  681739117 Jun 14 00:05 C28-140_20X.sff.zip
    -r--r--r-- 1 bach bach  537186774 Jun 14 00:06 FRA-257_20X.sff.zip
    -r--r--r-- 1 bach bach         89 Mar  7 20:05 LMP_Linkers.fasta

### Preparing the Ion Torrent data for MIRA

We will unzip all three sets with `unzip` and then extract with `sff_extract` into different data files, for the paired-end sets we will ask `sff_extract` to split the paired sequences in the SFF into single, but paired reads:

Unzip the data:

    arcadia:origdata$ cd ../data
    arcadia:data$ cd ../data
    arcadia:data$ unzip '../origdata/*zip'
    Archive:  ../origdata/C11-127_40X.sff.zip
      inflating: C11-127_40X.sff

    Archive:  ../origdata/C28-140_20X.sff.zip
      inflating: C28-140_20X.sff

    Archive:  ../origdata/FRA-257_20X.sff.zip
      inflating: FRA-257_20X.sff

    3 archives were successfully processed.
    arcadia:data$ ls -l
    -rw-r--r-- 1 bach bach 3617626646 Feb 10 00:32 C11-127_40X.sff
    -rw-rw-r-- 1 bach bach 1320383792 Dec 21 18:21 C28-140_20X.sff
    -rw-rw-r-- 1 bach bach 1159612116 Dec 16 20:48 FRA-257_20X.sff

Extract unpaired reads (will take ~2.5 minutes):

    arcadia:data$ sff_extract C11-127_40X.sff
    Working on 'C11-127_40X.sff':
    Converting 'C11-127_40X.sff' ... date
     done.
    Converted 1111060 reads into 1111060 sequences.

Extract 3.5kb paired reads (will take ~8 minutes):

    arcadia:data$ sff_extract C28-140_20X.sff
    Testing whether SSAHA2 is installed and can be launched ...  ok.
    Working on 'C28-140_20X.sff':
    Creating temporary sequences from reads in 'C28-140_20X.sff' ...  done.
    Searching linker sequences with SSAHA2 (this may take a while) ...  ok.
    Parsing SSAHA2 result file ...  done.
    Converting 'C28-140_20X.sff' ...  done.
    Converted 792280 reads into 1567302 sequences.

Extract 8.9kb paired reads (will take ~6.5 minutes):

    arcadia:data$ sff_extract C28-140_20X.sff
    Testing whether SSAHA2 is installed and can be launched ...  ok.
    Working on 'FRA-257_20X.sff':
    Creating temporary sequences from reads in 'FRA-257_20X.sff' ...  done.
    Searching linker sequences with SSAHA2 (this may take a while) ...  ok.
    Parsing SSAHA2 result file ...  done.
    Converting 'FRA-257_20X.sff' ...  done.
    Converted 618621 reads into 1224056 sequences.

Remove SFF files which are not needed anymore:

    arcadia:data$ rm *.sff
    arcadia:data$ ls -l
    -rw-rw-r-- 1 bach bach 1000477444 Jun 14 00:35 C11-127_40X.fastq
    -rw-rw-r-- 1 bach bach  246427864 Jun 14 00:35 C11-127_40X.xml
    -rw-rw-r-- 1 bach bach  355981807 Jun 14 00:44 C28-140_20X.fastq
    -rw-rw-r-- 1 bach bach  354995399 Jun 14 00:44 C28-140_20X.xml
    -rw-rw-r-- 1 bach bach  300577380 Jun 14 00:51 FRA-257_20X.fastq
    -rw-rw-r-- 1 bach bach  281069580 Jun 14 00:51 FRA-257_20X.xml

We're done here.

### Writing a manifest configuration file

Switch to the `assemblies` directory and create a subdirectory for our first assembly test.

    arcadia:data$ cd ../assemblies/
    arcadia:assemblies$ mkdir 1sttest
    arcadia:assemblies$ cd 1sttest

The manifest file itself should be pretty much self-explanatory, except perhaps the segment placement line, please look that up in the MIRA reference manual.

    project = mg1655
    job = genome,denovo,accurate
    parameters = -GE:not=4

    readgroup = TheShotgunReads
    technology = iontor
    data = ../../data/C11-127_40X.*

    readgroup = ThePE3.5kbLib
    technology = iontor
    data = ../../data/FRA-257_20X.*
    template_size = 2600 4400
    segment_placement = samedir backward

    readgroup = ThePE8.9kbLib
    technology = iontor
    data = ../../data/C28-140_20X.*
    template_size = 7800 10300
    segment_placement = samedir backward

    # note the wildcard "*" in the 'data' lines above:
    #  if you followed the walkthrough and have the FASTQ and XML files,
    #  this will automatically load both files for each data set
    #  (which is what we want)

Save the above lines into a file, we'll use `manifest.conf` in this example.

    arcadia:/path/to/myProject$ ls -l
    -rw-r--r-- 1 bach users        756 2011-11-05 17:57 manifest.conf

### Starting the assembly

Starting the assembly is now just a matter of one line:

    arcadia:1sttest$ mira manifest.conf >&log_assembly.txt

## What to do with the MIRA result files?

Please consult the corresponding section in the mira usage document, it contains much more information than this stub.

But basically, after the assembly has finished, you will find four directories. The `tmp` directory can be deleted without remorse as it contains logs and some tremendous amount of temporary data (dozens of gigabytes for bigger projects). The `info` directory has some text files with basic statistics and other informative files. Start by having a look at the `*_info_assembly.txt`, it'll give you a first idea on how the assembly went.

The `results` directory finally contains the assembly files in different formats, ready to be used for further processing with other tools.

If you used the uniform read distribution option, you will inevitably need to filter your results as this option produces larger and better alignments, but also more "debris contigs". For this, use the miraconvert which is distributed together with the MIRA package.

Also very important when analysing Ion Torrent assemblies: screen the small contigs ( < 1000 bases) for abnormal behaviour. You wouldn't be the first to have some human DNA contamination in a bacterial sequencing. Or some herpes virus sequence in a bacterial project. Or some bacterial DNA in a human data set. Or ...

Look whether these small contigs

- have a different GC content than the large contigs

- whether a BLAST of these sequences against some selected databases brings up hits in other organisms that you certainly were not sequencing.