Difference between revisions of "Preparing and loading data"

From GenomeView Manual
Jump to navigation Jump to search
Line 32: Line 32:
  
 
==Recommended file formats ==
 
==Recommended file formats ==
 +
This is a list of file formats that are recommended for different data types. See the full list of data types in the section below.
 +
 +
  
 
<table border=1>
 
<table border=1>
<tr><th>Data type</th><th>File format</th><th>Instructions</th></tr>
+
<tr><th>Data type</th><th>Recommended file format</th><th>Instructions</th></tr>
<tr><td>Reference sequence</td><td></td><td></td></tr>
+
<tr><td>Reference sequence</td><td>fasta</td><td>[[Preparing reference sequence]]</td></tr>
<tr><td></td><td></td><td></td></tr>
+
<tr><td>Annotation</td><td>gff3</td><td>[[Preparing annotation]]</td></tr>
<tr><td></td><td></td><td></td></tr>
+
<tr><td>Read a alignments</td><td>BAM</td><td>[[Preparing read alignments]]</td></tr>
 
<tr><td></td><td></td><td></td></tr>
 
<tr><td></td><td></td><td></td></tr>
 
<tr><td></td><td></td><td></td></tr>
 
<tr><td></td><td></td><td></td></tr>
 
<tr><td></td><td></td><td></td></tr>
 
<tr><td></td><td></td><td></td></tr>
 
</table>
 
</table>
 
  
 
== Supported data formats ==
 
== Supported data formats ==

Revision as of 00:26, 18 October 2013


There are several easy ways to load up data into GenomeView. Before you load your data, you may want to make sure you're using a supported format from the list below. Generally, GenomeView will notify you if it doesn't understand your data.

Loading data

You can load your data files ...

  • ... by selecting "work with my data" in the Genome Explorer
  • ... by dragging them onto GenomeView
  • ... by selecting the 'File' menu and then 'Load data...' (tutorial)
  • ... by pressing CTRL+O
  • ... by specifying them as argument on the command-line
  • ... by loading a session file


You can load preloaded data ...

Data preparation recipe

  1. Match identifiers: GenomeView uses the identifiers to link different sources, so make sure that the identifiers match (case-sensitive).
  2. Create indices for data files that need it (check table below)
  3. Convert file formats to get desired visuals (check table below)
  4. Load data (see above)

Why index files?

Indexing will create a look-up table for GenomeView to load data on-the-fly. This will will speed up browsing and loading speed, as well as significantly reduce the amount of memory you need. For some file formats we recommend you create indices, for other we do not. See the table below for more details and links to instructions.


Recommended file formats

This is a list of file formats that are recommended for different data types. See the full list of data types in the section below.


Data typeRecommended file formatInstructions
Reference sequencefastaPreparing reference sequence
Annotationgff3Preparing annotation
Read a alignmentsBAMPreparing read alignments

Supported data formats

Reference sequence

Data typeFile formatIndex*Max size**Comments
unindexed***indexed
Reference sequencefasta ¤Recommended
Index FASTA
50 MbunlimitedGenomeView will query the user create index for you if you don't have one and the file is very large.
embl, genbankNot possible50 Mb--EMBL and genbank are mixed file formats that can contain both annotation and reference sequence at the same time.

Annotation

Data typeFile formatIndex*Max size**Comments
Annotationgff ¤Not recommended Index GFF50 Mbunlimited
embl, genbankNot possible50 Mb--EMBL and genbank are mixed file formats that can contain both annotation and reference sequence at the same time.
bedNot recommended Index BED50 Mb or lessunlimitedBy default data from a bed file is added to the CDS track, if you want it in a different track, you have to add a line a the top of the file 'track name=Track_name'. No white-space is allowed in the track name.
ptt, tbl Not possible50 Mb or less--Other standard annotation formats GenomeView understands
various formatsNot possible50 Mb or less--GenomeView can directly parse the output of the following programs: Blast, GeneMark, TransTermHP, FindPeaks, MaqSNP, tRNA-scan

Whole genome alignments

Data typeFile formatIndex*Max size**Comments
Multiple genome alignmentmaf ¤Recommended100 MbunlimitedGenomeView will prompt you to create a compressed maf file and index it for you, if you're trying to load an unindexed maf file.
MAF is the recommended file format for whole genome alignemnt of large/complex genomes
multi-fasta ¤Not possible100 Mb--Recommended for small/simple genomes with a near 1:1 relationship.
aln, ClustalWNot possible100 Mb--

Read alignments

Data typeFile formatIndex*Max size**Comments
Sequence read alignmentbam ¤
Preparing read data
Required--unlimitedGenomeView will prompt you if there is no index and will create one for you. GenomeView can not automatically sort BAM files.
MAQ, MapView, BroadSolexaNot possible100 Mb--

Read coverage summary

Data typeFile formatIndex*Max size**Comments
Read coverage summary tdf ¤NativeunlimitedunlimitedTDF files can be created with the bam2tdf tool that is available for download.
bigwigNativeunlimitedunlimitedThis format can be used for any wig file, not just read coverage
pileupRequired--unlimitedThe pileup format becomes slow when you have extreme read depth (>5000 x coverage)
wigNot possible50 Mb--We strongly recommend to convert your wig files to TDF. GenomeView can automatically convert wig files to TDF. Caveats: 'track' information should all be on a single line, 'browser' lines will be ignored as the are specific to the UCSC Genome Browser. WIG files need to be sorted by chromosome and by genomic coordinate within the chromosome. BedGraph as well as Wiggle_0 format is supported. For the wiggle_0 type, both variableStep and fixedStep should work.

Genome variation and diversity

Data typeFile formatIndex*Max size**Comments
Genome variation vcf ¤Not recommended--unlimitedIt is recommended to run reducevcf on VCF prior to loading them, this will speed up the loading time significantly.


Allele diversity

Data typeFile formatIndex*Max size**Comments
Allele diversity summary pileup ¤Required--unlimitedThe pileup format becomes slow when you have extreme read depth (>5000 x coverage)
* Indicates whether this file format can/should be indexed. 
** Recommended maximum file size. First value is without index, the second with index. This values are only guidelines. When loading multiple data sets, you should add the sizes.
*** Unindexed data files can be gzip compressed.

¤ Recommended file format for this data type.



Output formats

(Modified) annotations can be saved as either GFF or EMBL.

All data that is loaded can be exported in their original format. This will not include modifications.

Converting formats

<a href="http://genomeview.org/loki/">We offer a few tools to convert files between formats.</a>

Previous documentation pages

Supported data formats Fasta files

Feature files

Read data

Coverage plots