Bastien
Chevreux
bach@chevreux.org

> Solomon Short

> You have to know what you're looking for before you can find it.

# Mapping assemblies

## Introduction

This guide assumes that you have basic working knowledge of Unix systems, know the basic principles of sequencing (and sequence assembly) and what assemblers do.

While there are step by step instructions on how to setup your data and then perform an assembly, this guide expects you to read at some point in time

- Before the mapping, to know what to do (or not to do) with the sequencing data before giving it to MIRA.

- Generally, the to know what to do with the results of the assembly. More specifically,

- And also to look up how manifest files should be written ( and and ), some command line options as well as general information on what tags MIRA uses in assemblies, files it generates etc.pp

- Last but not least, you may be interested in some observations about the different sequencing technologies and the traps they may contain, see for that. For advice on what to pay attention to before going into a sequencing project, have a look at .

## General steps

This part will introduce you step by step how to get your data together for a simple mapping assembly.

I'll make up an example using an imaginary bacterium: Bacillus chocorafoliensis (or short: Bchoc).

In this example, we assume you have two strains: a wild type strain of Bchoc_wt and a mutant which you perhaps got from mutagenesis or other means. Let's imagine that this mutant needs more time to eliminate a given amount of chocolate, so we call the mutant Bchoc_se ... SE for slow eater

You wanted to know which mutations might be responsible for the observed behaviour. Assume the genome of Bchoc_wt is available to you as it was published (or you previously sequenced it), so you resequenced Bchoc_se with Solexa to examine mutations.

### Copying and naming the sequence data

You need to create (or get from your sequencing provider) the sequencing data in either FASTQ or FASTA + FASTA quality format. The following walkthrough uses what most people nowadays get: FASTQ.

Create a new project directory (e.g. `myProject`) and a subdirectory of this which will hold the sequencing data (e.g. `data`).

    arcadia:/path/to mkdir myProject
    arcadia:/path/to cd myProject
    arcadia:/path/to/myProject$ mkdir data

Put the FASTQ data into that `data` directory so that it now looks perhaps like this:

    arcadia:/path/to/myProject$ ls -l data
    -rw-r--r-- 1 bach users 263985896 2008-03-28 21:49 bchocse_lane6.solexa.fastq
    -rw-r--r-- 1 bach users 264823645 2008-03-28 21:51 bchocse_lane7.solexa.fastq

I completely made up the file names above. You can name them anyway you want. And you can have them live anywhere on the hard disk, you do not need to put them in this `data` directory. It's just the way I do it ... and it's where the example manifest files a bit further down in this chapter will look for the data files.

### Copying and naming the reference sequence

The reference sequence (the backbone) can be in a number of different formats: GFF3, GenBank, MAF, CAF, FASTA. The first three have the advantage of being able to carry additional information like, e.g., annotation. In this example, we will use a GFF3 file like the ones one can download from the NCBI.

TODO: Write why GFF3 is better and where to get them at the NCBI.

So, let's assume that our wild type strain is in the following file: `NC_someNCBInumber.gff3`.

You do not need to copy the reference sequence to your directory, but I normally copy also the reference file into the directory with my data as I want to have, at the end of my work, a nice little self-sufficient directory which I can archive away and still be sure that in 10 years time I have all data I need together.

    arcadia:/path/to/myProject$ cp /somewhere/NC_someNCBInumber.gff3 data
    arcadia:/path/to/myProject$ ls -l data
    -rw-r--r-- 1 bach users   6543511 2008-04-08 23:53 NC_someNCBInumber.gff3
    -rw-r--r-- 1 bach users 263985896 2008-03-28 21:49 bchocse_lane6.solexa.fastq
    -rw-r--r-- 1 bach users 264823645 2008-03-28 21:51 bchocse_lane7.solexa.fastq

We're almost finished with the setup. As I like to have things neatly separated, I always create a directory called `assemblies` which will hold my assemblies (or different trials) together. Let's quickly do that:

    arcadia:/path/to/myProject$ mkdir assemblies
    arcadia:/path/to/myProject$ mkdir assemblies/1sttrial
    arcadia:/path/to/myProject$ cd assemblies/1sttrial

### Writing a simple manifest file

A manifest file is a configuration file for MIRA which tells it what type of assembly it should do and which data it should load. In this case we have unpaired sequencing data which we want to map to a reference sequence, the manifest file for that is pretty simple:

    # Example for a manifest describing a mapping assembly with
    # unpaired Illumina data

    # First part: defining some basic things
    # In this example, we just give a name to the assembly
    #  and tell MIRA it should map a genome in accurate mode

    project = MyFirstAssembly
    job = genome,mapping,accurate

    # The second part defines the sequencing data MIRA should load and assemble
    # The data is logically divided into "readgroups"

    # first, the reference sequence
    readgroup
    is_reference
    data = ../../data/NC_someNCBInumber.gff3
    strain = bchoc_wt

    # now the Illumina data

    readgroup = SomeUnpairedIlluminaReadsIGotFromTheLab
    data = ../../data/*fastq
    technology = solexa
    strain = bchoc_se

Please look up the parameters of the manifest file in the main manual or the example manifest files in the following section.

The ones above basically say: make an accurate mapping of Solexa reads against a genome; in one pass; the name of the backbone strain is 'bchoc_wt'; the data with the backbone sequence (and maybe annotations) is in a specified GFF3 file; for Solexa data: assign default strain names for reads which have not loaded ancillary data with strain info and that default strain name should be 'bchoc_se'.

### Starting the assembly

Starting the assembly is now just a matter of a simple command line:

    arcadia:/path/to/myProject/assemblies/1sttrial$ mira manifest.conf >&log_assembly.txt

For this example - if you followed the walk-through on how to prepare the data - everything you might want to adapt in the first time are the following thing in the manifest file: options:

- project= (for naming your assembly project)

- strain_name= to give the names of your reference and mapping strain

Of course, you are free to change any option via the extended parameters, but this is the topic of another part of this manual.

## Manifest files for different use cases

This section will introduce you to manifest files for different use cases. It should cover the most important uses, but as always you are free to mix and match the parameters and readgroup definitions to suit your specific needs.

Taking into account that there may be a lot of combinations of sequencing technologies, sequencing libraries (shotgun, paired-end, mate-pair, etc.) and input file types (FASTQ, FASTA, GenBank, GFF3, etc.pp), the example manifest files just use Illumina and 454 as technologies, GFF3 as input file type for the reference sequence, FASTQ as input type for sequencing data ... and they do not show the multitude of more advanced features like, e.g., using ancillary clipping information in XML files, ancillary masking information in SSAHA2 or SMALT files etc.pp.

I'm sure you will be able to find your way by scanning through the corresponding section on manifest files in the reference chapter :-)

### Mapping with shotgun data

Well, we've seen that already in the section above, but here it is again ... this time with Ion Torrent data though.

    # Example for a manifest describing a mapping assembly with
    # unpaired Ion data

    # First part: defining some basic things
    # In this example, we just give a name to the assembly
    #  and tell MIRA it should map a genome in accurate mode

    project = MyFirstAssembly
    job = genome,mapping,accurate

    # The second part defines the sequencing data MIRA should load and assemble
    # The data is logically divided into "readgroups"

    # first, the reference sequence
    readgroup
    is_reference
    data = ../../data/NC_someNCBInumber.gff3
    strain = bchoc_wt

    # now the Ion Torrent data

    readgroup = SomeUnpairedIonReadsIGotFromTheLab
    data = ../../data/someiondata.fastq
    technology = iontor
    strain = bchoc_se

### Manifest for data sets with paired reads

When using paired-end data in mapping, you must decide whether you want

1. use the MIRA feature to create long 'coverage equivalent reads' (CERs) which saves a lot of memory (both in the assembler and later on in an assembly editor). However, you then loose information about read pairs!

2. or whether you want to keep information about read pairs at the expense of larger memory requirements both in MIRA and in assembly finishing tools or viewers afterwards.

3. or a mix of the two above

The Illumina pipeline generally normally gives you two files for paired-end data: a `project-1.fastq` and `project-2.fastq`. The first file containing the first read of a read-pair, the second file the second read. Depending on the preprocessing pipeline of your sequencing provider, the names of the reads are either the very same in both files or already have a `/1` or `/2` appended. Also, your sequencing provider may give you one big file where the reads from both ends are present.

MIRA can read all FASTQ variants produced by various Illumina pipelines, be they with or without the /1 and /2 already appended to the names. You generally do not need to do any name mangling before feeding the data to MIRA. However, MIRA will shell out a warning if read names are longer than 40 characters.

When using paired-end data, you should know

1. the orientation of the reads toward each other. This is specific to sequencing technologies and / or the sequencing library preparation.

2. at which distance these reads should be. This is specific to the sequencing library preparation and the sequencing lab should tell you this.

In case you do not know one (or any) of the above, don't panic! MIRA is able to estimate the needed values during the assembly if you tell it to.

The following manifest shows you the most laziest way to define a paired data set by simply adding autopairing as keyword to a readgroup (using Illumina just as example):

    # Example for a lazy manifest describing a denovo assembly with
    # one library of paired reads

    # First part: defining some basic things
    # In this example, we just give a name to the assembly
    #  and tell MIRA it should map a genome in accurate mode

    project = MyFirstAssembly
    job = genome,mapping,accurate

    # The second part defines the sequencing data MIRA should load and assemble
    # The data is logically divided into "readgroups"

    # first the reference sequence
    readgroup
    is_reference
    data = ../../data/NC_someNCBInumber.gff3
    technology = text
    strain = bchoc_wt

    # now the Illumina paired-end data

    readgroup = DataIlluminaPairedLib
    autopairing
    data = ../../data/project_1.fastq ../../data/project_2.fastq
    technology = solexa
    strain = bchoc_se1

See? Wasn't hard and it did not hurt, did it? One just needs to tell MIRA it should expect paired reads via the autopairing keyword and that is everything you need.

If you know the orientation of the reads and/or the library size, you can tell this MIRA the following way (just showing the readgroup definition here):

    readgroup = DataIlluminaPairedEnd500Lib
    data = ../../data/project_1.fastq ../../data/project_2.fastq
    technology = solexa
    template_size = 250 750
    segment_placement = ---> <---

In cases you are not 100% sure about, e.g., the size of the DNA template, you can also give a (generous) expected range and then tell MIRA to automatically refine this range during the assembly based on real, observed distances of read pairs. Do this with autorefine modifier like this:

    template_size = 50 1000 autorefine

The following manifest file is an example for mapping a 500 bp paired-end and a 3kb mate-pair library of a strain called bchoc_se1 against a GenBank reference file containing a strain called bchoc_wt:

    # Example for a manifest describing a mapping assembly with
    # paired Illumina data, not merging reads and therefore keeping
    # all pair information

    # First part: defining some basic things
    # In this example, we just give a name to the assembly
    #  and tell MIRA it should map a genome in accurate mode
    # As special parameter, we want to switch off merging of Solexa reads

    project = MyFirstAssembly
    job = genome,mapping,accurate
    parameters = SOLEXA_SETTINGS -CO:msr=no

    # The second part defines the sequencing data MIRA should load and assemble
    # The data is logically divided into "readgroups"

    # first, the reference sequence
    readgroup
    is_reference
    data = ../../data/NC_someNCBInumber.gff3
    technology = text
    strain = bchoc_wt

    # now the Illumina data

    readgroup = DataForPairedEnd500bpLib
    autopairing
    data = ../../data/project500bp-1.fastq ../../data/project500bp-2.fastq
    technology = solexa
    strain = bchoc_se1

    readgroup = DataForMatePair3kbLib
    data = ../../data/project3kb-1.fastq ../../data/project3kb-2.fastq
    technology = solexa
    strain = bchoc_se1
    template_size = 2000 4000 autorefine
    segment_placement = <--- --->

Please look up the parameters used in the main manual. The ones above basically say: make an accurate mapping of Solexa reads against a genome. Additionally do not merge short short Solexa reads to the contig.

For the paired-end library, be lazy and let MIRA find out everything it needs. However, that information should be treated as "information only" by MIRA, i.e., it is not used for deciding whether a pair is well mapped.

For the mate-pair library, assume a DNA template template size of 2000 to 4000 bp (but let MIRA automatically refine this using observed distances) and the segment orientation of the read pairs follows the reverse / forward scheme. That information should be treated as "information only" by MIRA, i.e., it is not used for deciding whether a pair is well mapped.

Comparing this manifest with a manifest for unpaired-data, two parameters were added in the section for Solexa data:

1. `-CO:msr=no` tells MIRA not to merge reads that are 100% identical to the backbone. This also allows to keep the template information (distance and orientation) for the reads.

2. `template_size` tells MIRA at which distance the two reads should normally be placed from each other.

3. `segment_placement` tells MIRA how the different segments (reads) of a DNA template have to be ordered to form a valid representation of the sequenced DNA.

Note that in mapping assemblies, these `template_distance` and `segment_placement` parameters are normally treated as information only, i.e., MIRA will map the reads regardless whether the distance and orientation criterions are met or not. This enables post-mapping analysis programs to hunt for genome rearrangements or larger insertions/deletion.

If template size and segment placement checking were on, the following would happen at, e.g. sites of re-arrangement: MIRA would map the first read of a read-pair without problem. However, it would very probably reject the second read because it would not map at the specified distance or orientation from its partner. Therefore, in mapping assemblies with paired-end data, checking of the template size must be switched off to give post-processing programs a chance to spot re-arrangements.

### Mapping with multiple sequencing technologies (hybrid mapping)

I'm sure you'll have picked up the general scheme of manifest files by now. Hybrid mapping assemblies follow the general scheme: simply add as separate readgroup the information MIRA needs to know to find the data and off you go. Just for laughs, here's a manifest for 454 shotgun with Illumina paired-end

    # Example for a manifest describing a mapping assembly with
    # shotgun 454 and paired-end Illumina data, not merging reads and therefore keeping
    # all pair information

    # First part: defining some basic things
    # In this example, we just give a name to the assembly
    #  and tell MIRA it should map a genome in accurate mode
    # As special parameter, we want to switch off merging of Solexa reads

    project = MyFirstAssembly
    job = genome,mapping,accurate
    parameters = SOLEXA_SETTINGS -CO:msr=no

    # The second part defines the sequencing data MIRA should load and assemble
    # The data is logically divided into "readgroups"

    # first, the reference sequence
    readgroup
    is_reference
    data = ../../data/NC_someNCBInumber.gff3
    strain = bchoc_wt

    # now the shotgun 454 data
    readgroup = DataForShotgun454
    data = ../../data/project454data.fastq
    technology = 454
    strain = bchoc_se1

    # now the paired-end Illumina data

    readgroup = DataForPairedEnd500bpLib
    data = ../../data/project500bp-1.fastq ../../data/project500bp-2.fastq
    technology = solexa
    strain = bchoc_se1
    template_size = 250 750
    segment_placement = ---> <---

### Mapping with multiple strains

MIRA will make use of ancillary information present in the manifest file. One of these is the information to which strain (or organism or cell line etc.pp) the generated data belongs.

You just need to tell in the manifest file which data comes from which strain. Let's assume that in the example from above, the "lane6" data were from a first mutant named bchoc_se1 and the "lane7" data were from a second mutant named bchoc_se2. Here's the manifest file you would write then:

    # Example for a manifest describing a mapping assembly with
    # unpaired Illumina data

    # First part: defining some basic things
    # In this example, we just give a name to the assembly
    #  and tell MIRA it should map a genome in accurate mode

    project = MyFirstAssembly
    job = genome,mapping,accurate

    # The second part defines the sequencing data MIRA should load and assemble
    # The data is logically divided into "readgroups"

    # first, the reference sequence
    readgroup
    is_reference
    data = ../../data/NC_someNCBInumber.gff3
    technology = text
    strain = bchoc_wt

    # now the Illumina data

    readgroup = DataForSE1
    data = ../../data/bchocse_lane6.solexa.fastq
    technology = solexa
    strain = bchoc_se1

    readgroup = DataForSE2
    data = ../../data/bchocse_lane7.solexa.fastq
    technology = solexa
    strain = bchoc_se2

While mapping (or even assembling de-novo) with multiple strains is possible, the interpretation of results may become a bit daunting in some cases. For many scenarios it might therefore be preferable to successively use the data sets in own mappings or assemblies.

This strain information for each readgroup is really the only change you need to perform to tell MIRA everything it needs for handling strains.

## Walkthroughs

### Walkthrough: mapping of E.coli from Lenski lab against E.coli B REL606

TODO: Sorry, needs to be re-written for the relatively new SRR format distributed at the NCBI ... and changes in MIRA 3.9.x. Please come back later.

## Useful things to know about reference sequences

There are a few things to consider when using reference sequences:

1. MIRA is not really made to handle a big amount of reference sequences as they currently need inane amounts of memory. Use other programs for mapping against more than, say, 200 megabases.

2. Reference sequences can be as long as needed! They are not subject to normal read length constraints of a maximum of 32k bases. That is, if one wants to load one or several entire chromosomes of a bacterium or lower eukaryote as backbone sequence(s), this is just fine.

3. Reference sequences can be single sequences like provided in, e.g., FASTA, FASTQ, GFF or GenBank files. But reference sequences also can be whole assemblies when they are provided as, e.g., MAF or CAF format. This opens the possibility to perform semi-hybrid assemblies by assembling first reads from one sequencing technology de-novo (e.g. PacBio) and then map reads from another sequencing technology (e.g. Solexa) to the whole PacBio alignment instead of mapping it to the PacBio consensus.

   A semi-hybrid assembly will therefore contain, like a hybrid assembly, the reads of both sequencing technologies.

4. Reference sequences will not be reversed! They will always appear in forward direction in the output of the assembly. Please note: if the backbone sequence consists of a MAF or CAF file that contain contigs which contain reversed reads, then the contigs themselves will be in forward direction. But the reads they contain that are in reverse complement direction will of course also stay reverse complement direction.

5. Reference sequences will not not be assembled together! That is, even if a reference sequence has a perfect overlap with another reference sequence, they will still not be merged.

6. Reads are assembled to reference sequences in a first come, first served scattering strategy.

   Suppose you have two identical reference sequences and a read which would match both, then the read would be mapped to the first backbone. If you had two identical reads, the first read would go to the first backbone, the second read to the second backbone. With three identical reads, the first backbone would get two reads, the second backbone one read. Etc.pp.

7. Only in references loaded from MAF or CAF files: contigs made out of single reads (singlets) loose their status as reference sequence and will be returned to the normal read pool for the assembly process. That is, these sequences will be assembled to other reference sequences or with each other.

## Known bugs / problems

These are actual for version 4.0 of MIRA and might or might not have been addressed in later version.

Bugs:

1. mapping of paired-end reads with one read being in non-repetitive area and the other in a repeat is not as effective as it should be. The optimal strategy to use would be to map first the non-repetitive read and then the read in the repeat. Unfortunately, this is not yet implemented in MIRA.