Vamb inputs and outputs

Vamb inputs and outputs#

All modes of Vamb takes various inputs and produces various outputs. Currently, all modes take the following two central inputs:

  • The kmer-composition of the sequence (the composition).

  • The abundance of the contigs in each sample (the abundance).

For inputs that take significant time to produce, Vamb will serialize the parsed input to a file, such that future runs of Vamb can use that instead of re-computing it.

Composition#

The composition is computed from the input contig file in FASTA format (the ‘catalogue’). From command line, this looks like:

--fasta contigs.fna.gz

Where the catalogue may be either gzipped or a plain FASTA file.

Vamb produces the best results when run with the “multi-split” workflow, as demonstrated in the quickstart section in “how to run Vamb”. In this workflow, samples are assembled independently, and the resulting contigs are concatenated to a single FASTA file before binning. After binning, the bins can be split into sample-wise pure bins.

To do this splitting (termed “binsplitting”), Vamb needs to know which contig came from which sample. Therefore, it’s recommended that the FASTA headers are formatted in the following pattern: {samplename}C{contigname}

Where {samplename} is some text that uniquely identifies each sample, and {contigname} uniquely identifies each contig within a sample. For example, if the samples are named S1, S2, S3, etc., and the contigs are named 1, 2, 3, etc, a FASTA header may be S3C119.

After the composition.npz has been created, Vamb will write the composition in the output file composition.npz. Future runs of Vamb can then instead use the following option to load the composition directly:

--composition composition.npz

Abundance#

The abundance may be computed from either:

  • A TSV file with the header being “contigname” followed by one sample name per sample, and the values in the TSV file being precomputed abundances. These may be derived from pasteing together outputs from the tool strobealign --aemb, and concatenating the result to a TSV header.

  • A directory of sorted BAM files generated by mapping the reads of each sample to the contig catalogue.

On the command line, abundance input can be specified as:

--abundance_tsv abundance.tsv

or

--bamdir dir_with_bam_files

Once the abundance has been parsed, Vamb will produce the file abundance.npz, which can be used for future Vamb runs instead:

--abundance abundance.npz

Abundance TSV format#

This follows the ordinary .tsv format, with a mandatory header, and disallowing \t in contig names. The header of the first column name must be exactly contigname.

The abundance TSV file in the correct format can be created from the output of strobealign --aemb using the script in src/merge_aemb.py:

python src/merge_aemb.py input_dir abundance.tsv

Example file:

contigname	S1	S2	S3
S1C1	1.53	1.11	4.1e2
S1C2	0.94	9.2	5.1e2
S2C1	1.2e-3	0	9.2
S3C1	88.21	51.2	12.1e3
S3C2	14.1	90.1	13.01

Abundance as BAM files#

If you don’t want to compute abundance using strobealign --aemb and create a TSV file from its output (recommended), Vamb can compute abundances from BAM files.

To do this:

  • Create the FASTA contig catalogue as described in the section of “composition”.

  • Map the reads for each sample to the catalogue, to obtain on BAM file per sample.

Using the aligner minimap2 as well as samstrip and samtools, the commands may be:

# Index the FASTA file so it only has to be done once instead of on every mapping
minimap2 -I 32G -d catalogue.mmi catalogue.fasta;

# Map each sample. Here, using 8 threads, using paired short reads.
minimap2 -t 8 -ax sr catalogue.mmi s1.fw.fq.gz s1.rv.fq.gz | samstrip | samtools view -F 3584 -b - > s1.bam

Note that if you use minimap2 specifically, be aware of this bug in minimap2, where, if the index is not large enough, the output will be an invalid SAM file. To get around this, use enough RAM when indexing (e.g. set option -I appropriately)

Reference hash checking#

To ensure the integrity of the data, Vamb will compare the identifiers in the composition (ultimately: headers in the FASTA file) with the contig names from the abundance input (TSV file contig names, or BAM sequence names) and, if provided, that of the taxonomic input.

To do this efficiently, the identifiers are hashed to provide a reference hash (refhash), which is compared, and an error is thrown if they are different.

If you, for some reason, can’t create input files with matching identifiers, and you are 100% sure the order of the sequences is identical in the composition input and abundance input, you can disable this reference hashing with the --norefcheck option.

Taxonomy#

About taxonomy files#

Vamb takes one of two kinds of taxonomic files:

  • Unrefined taxonomies give the taxonomic annotation for each contig

  • Refined taxonomies gives the taxonomic annotation plus a probability estimate for each contig

Vamb’s Taxometer tool can be used to refine a taxonomy. It takes an unrefined taxonomy as input, and outputs a refined taxonomy. TaxVamb usually achieves better results if its taxonomy input is refined with Taxometer.

Both refined and unrefined taxonomies can be used for TaxVamb. By default, if TaxVamb gets an unrefined taxonomy, it will automatically refine it with Taxometer, unless --no_predict is passed.

Taxonomy files are TSV files with the following format:

  • Header: contigs\tpredictions for unrefined taxonomies and contigs\tpredictions\tscores for refined ones.

  • In the contigs column: The FASTA identifier for every contig in the catalogue.

  • In the predictions column: A semicolon-separated string with taxonomic levels, for each of the following seven ranks, in order: domain, phylum, class, order, family, genus, species. Lower ranks may be omitted. There is no requirement that the labels are actually meaningful, i.e. that they correspond to any real taxonomic clade.

  • In the scores column: A semicolon separated list of floats, one per element in the predictions column.

The following are examples of a VALID rows in the prediction column:

Bacteria;Bacillota;Clostridia
Bacteria;Bacillota;Bacilli;Bacillales
Bacteria;Pseudomonadota;Gammaproteobacteria;Moraxellales;Moraxellaceae;Acinetobacter;Acinetobacter sp. TTH0-4

The following are example of INVALID rows in the prediction column:

  • Invalid: Begins with class instead of domain: Clostridia;Eubacteriales;Lachnospiraceae;Roseburia;Roseburia hominis

  • Invalid: Skips the phylum: Bacteria;Gammaproteobacteria;Moraxellales;Moraxellaceae;Acinetobacter;Acinetobacter sp. TTH0-4

The following is an example of a valid, unrefined taxonomy file:

contigs	predictions
S18C13	Bacteria;Bacillota;Clostridia;Eubacteriales
S18C25	Bacteria;Pseudomonadota
S18C67	Bacteria;Bacillota;Bacilli;Bacillales;Staphylococcaceae;Staphylococcus

How to create a taxonomy file#

The taxonomy file format detailed above is independent of the annotation tool used to generate the taxonomy. Users may use any taxonomic classifier, and convert the output of that classifier to the TSV format above.

We recommend using MMseqs2 combined with GTDB for taxonomic classification, as that has given the best results in our tests.

Warning

The commands below only works for a given input format of MMseqs2. Since we can not control whether MMseqs2 changes their output formats, the code below may fail for future versions of MMseqs2. When running the code below, please verify that your MMseqs2 output format is as expected.

# Run MMseqs2 taxonomy with GTDB v 220 downloaded with mmseqs databases
mmseqs easy-taxonomy $(path_to_contigs) $(path_to_db) $(mmseqs_output_dir) --tax-lineage 1

# 2. Format taxonomy file for TAXVAMB
echo -e "contigs\tpredictions" > $(taxonomy_path)

# Replace taxonomy of unclassified contigs from the empty string to 'unknown'
sed 's/\t$/\tunknown/' $(mmseqs_output_dir)/gtdb_lca.tsv |  awk -F "\t" '{print $1 "\t" $9 }' >>$(taxonomy_path)

Outputs#

Vamb#

  • log.txt - A text file with information about the Vamb run. Look here (and at stderr) if you experience errors.

  • composition.npz: A Numpy .npz file that contain all kmer composition information computed by Vamb from the FASTA file. This can be provided to another run of Vamb to skip the composition calculation step. This is not produced if an existing composition.npz was used to run Vamb.

  • abundance.npz: Similar to composition.npz, but this file contains information calculated from the abundance TSV file (or BAM files). Using this as input instead of BAM files will skip re-parsing the BAM files, which take a significant amount of time. This file is not produced if an existing abundance.npz was used to run Vamb.

  • model.pt: A file containing the trained VAE model. When running Vamb from a Python interpreter, the VAE can be loaded from this file to skip training.

  • latent.npz: This contains the output of the VAE model, the embedding of each of the contigs.

  • bins: If --minfasta is set, this is a directory with one FASTA file per bin, after binsplitting.

  • vae_clusters_unsplit.tsv - A two-column TSV text with the header clustername\tcontigname, then one row per sequence: Left column for the cluster (i.e bin) name, right column for the sequence name. You can create the FASTA-file bins themselves using the script in src/create_fasta.py

  • (if binsplitting is enabled:) vae_clusters_split.tsv, similar to the unsplit version, but after binsplitting. See the section on binsplitting on the page”tips for running Vamb”.

  • vae_clusters_metadata.tsv: A file with some metadata about clusters.

    • Name: The name of the cluster

    • Radius: Cosine radius in embedding space. Small clusters are usually more likely to be pure.

    • Peak/valley ratio: A small PVR means the cluster’s edges is more well defined, and hence the cluster is more likely pure

    • Kind: Currently, Vamb produces three kinds of clusters:

      • Normal: Defined by a local density in latent space. Most good clusters are of this type

      • Loner: A contig far away from everything else in latent space.

      • Fallback: After failing to produce good clusters for some time, these (usually poor) clusters are created to not get stuck in an infinite loop when clustering

    • Bp: Sum of length of all sequences in the cluster

    • Ncontigs: Number of sequences in the cluster

    • Medoid: Name of contig used as the cluster’s medoid, i.e. the center of the cluster

TaxVamb#

  • log.txt, composition.npz and abundance.npz: Same as when running Vamb

  • predictor_model.pt and results_taxometer.tsv: If Taxometer was used to automatically refine TaxVamb. See the Taxometer output section.

  • vaevae_clusters_{split,unsplit,metadata}.tsv: Same as when running Vamb, but from TaxVamb’s VAEVAE model

  • vaevae_model.pt A PyTorch model with the trained VAEVAE model.

Taxometer#

  • log.txt, composition.npz and abundance.npz: Same as when running Vamb

  • predictor_model.pt: A PyTorch model file containing the trained predictor.

  • results_taxometer.tsv: A refined taxonomy file (see the section on files on the “how to run” page)

AVAMB#

Same as Vamb, but also:

  • aae_y_clusters_{split,unsplit}.tsv: The clusters obtained from the categorical latent space

  • aae_z_latent.npz: Like latent.npz, but of the adversarial Z latent space

  • aae_z_clusters_{metadata,split,unsplit}.tsv: Like the corresponding vae_clusters* files, but from the adversarial Z latent space

Taxonomy benchmarking#

  • log.txt: Same as when running Vamb

  • results_taxonomy_predicted_kfold.tsv: Taxonomy file that is predicted with cross-validation

  • file_tracking.tsv: Filepaths used for the provided and the predicted taxonomy

  • accuracy_report.tsv: The evaluation results. You are interested in “Accuracy” for the Domain level - which is usually “Level_0” for GTDB, and can be “Level_1” for NCBI. Please check your provided taxonomy. The higher value the better.

Contents