DOI

It’s been 10 months since I last posted an update to my Autocycler benchmarks, and there have since been a few changes to the long-read assembler landscape:

For this post, I re-ran the Autocycler paper benchmarks, covering all of the above.1

Updated results

All of the plots in this post use a pseudo-log transform on the y-axis.

Below is an updated Figure 2 from the Autocycler paper, with current versions for all tools.2 Error counts are shown on the y-axes (lower is better):

Autocycler benchmark updated results

For a detailed breakdown of error types, see this updated Figure S1: detailed results by error type.

This is an updated Figure S5, showing runtime and memory usage (lower is better):

Autocycler benchmark updated results

And for the tools with updated versions, here are some plots that show their metrics across versions: Hybracter, metaMDBG and Myloasm.3

Discussion

Of the tools with new versions, I was particularly impressed with metaMDBG. The current version (v1.4) produces fewer assembly errors than previous versions, and its low values for missing bases and extra bases make its total errors (bottom plot of Figure 2) the lowest of all the single-tool assemblers. It was already the most memory-efficient assembler I tested, and its runtime is now half of what it used to be. Very nice!

For both Hybracter and Myloasm, their current accuracy is about the same as when I last tested them. But Myloasm’s memory usage and runtime have improved, so it’s now tied with Raven as the fastest assembler tested.

Ilesta is the new kid on the block, and I’ve now added it to Autocycler helper. Like miniasm, Ilesta produces an unpolished assembly, so like miniasm, I ran its output through Minipolish. In the benchmark, it didn’t stand out in any particular metric, and its maximum total error count was the highest of the assemblers tested (it really struggled with the Shigella genome). So I’m not going to include it in my Autocycler pipeline yet, but the tool is young, so I’ll keep an eye on it and maybe reconsider if there are future versions.

Finally, this blog post introduces a new fast version of my Autocycler bash pipeline. An Autocycler assembly can take a long time to run, not because of Autocycler itself, but because of the time it takes to make the input assemblies. The autocycler_full.sh pipeline generates 36 input assemblies (9 assemblers × 4 read subsets), and much of that time is spent on Canu. The new autocycler_fast.sh pipeline generates just 12 input assemblies (6 assemblers × 2 read subsets). To save time, it does not run Canu (the slowest assembler), but it still runs Plassembler (also pretty slow) to recover small plasmids that other assemblers cannot. If you don’t care about small plasmids, you could drop Plassembler and save more time.

In this benchmark, the fast Autocycler pipeline was just as accurate as the full Autocycler pipeline, and it cut the runtime by more than three-quarters in most cases, making it comparable to Hybracter’s runtime. I still think the full pipeline will sometimes be beneficial, since I do encounter genomes where including Canu helps. But the fast pipeline should be perfectly suitable for most genomes, so some users may want to switch.

Footnotes

  1. But this time I didn’t run the additional assessment tools from the Autocycler paper: Inspector (Figure S2), CRAQ (Figure S3) and BUSCO (Figure S4). ↩

  2. I didn’t include Autocycler-curated results in this benchmark because they weren’t needed! The plasmid which was occasionally missing in earlier benchmarks (and thus necessitated manual intervention) was not missing in these runs. So the automated Autocycler assemblies were all fine on their own. ↩

  3. I didn’t test every version of these tools, just the current version plus any benchmark results from previous versions that I already had, either from the Autocycler paper or the last update post. ↩