fasterp

High-performance FASTQ preprocessing in Rust. A drop-in replacement for fastp with blazing fast performance.

Installation

cargo install fasterp

Or build from source:

git clone https://github.com/drbh/fasterp
cd fasterp
cargo build --release

Quick Start

# Single-end
fasterp -i input.fq -o output.fq

# Paired-end
fasterp -i R1.fq -I R2.fq -o out1.fq -O out2.fq

# With quality filtering
fasterp -i input.fq -o output.fq -q 20 -l 50

# With adapter trimming
fasterp -i input.fq -o output.fq -a AGATCGGAAGAGC

# Multi-threaded with memory limit
fasterp -i large.fq -o output.fq -w 8 --max-memory 4096

Quick Reference

Copy-paste commands for common tasks:

Pro tip: Combine flags freely. Example for production pipeline:

fasterp -i R1.fq.gz -I R2.fq.gz -o o1.fq.gz -O o2.fq.gz \
  -q 20 -l 50 --cut-tail --cut-mean-quality 20 \
  -c -w 16 --max-memory 8192

CLI Reference

Input/Output

FASTQ files contain sequence data as text. Each “read” is a short string of letters (A, C, G, T) representing a fragment of genetic material, plus a confidence score for each letter. fasterp reads these files, processes them, and writes cleaned versions.

Length Filtering

Reads that are too short often come from errors or failed sequencing. Think of it like filtering out incomplete sentences - if a read is shorter than your minimum threshold, it’s probably not useful data and gets discarded.

Quality Filtering

Each letter in a read has a confidence score (0-40+) indicating how certain the machine was about that letter. Higher is better. Quality filtering removes reads with too many low-confidence letters, too many unknown letters (N), or repetitive patterns that suggest errors. It’s like spell-checking - removing text that’s too garbled to trust.

Trimming - Fixed Position

Sometimes the beginning or end of reads contain known bad data (like primer sequences or low-quality regions). Fixed trimming removes a set number of characters from the start or end of every read - like cutting off the first and last few words of every sentence because you know they’re always wrong.

Trimming - Quality Based

Instead of cutting a fixed amount, this looks at actual quality scores and trims where the data gets bad. A sliding window moves along the read, checking average quality. When it drops below your threshold, everything past that point is cut off. It’s like editing a document and stopping where the text becomes unreadable.

PolyG/PolyX Trimming

Some sequencing machines produce false runs of the same letter (like “GGGGGGGG”) at the end of reads when they run out of real signal. These aren’t real data - they’re artifacts. This feature detects and removes these repetitive tails, like removing a stuck key’s output from a document.

Adapter Trimming

Adapters are short synthetic sequences added during sample preparation - they’re not part of the actual sample. They appear at the ends of reads and must be removed, like stripping metadata headers from a file. fasterp can auto-detect common adapters or you can specify them manually.

Paired-End Options

Paired-end sequencing reads the same fragment from both ends, giving you two reads that overlap in the middle. When they overlap, you can compare them: if one says “A” with high confidence and the other says “T” with low confidence, you trust the “A”. You can also merge overlapping pairs into a single, longer read. It’s like having two people transcribe the same audio and combining their best parts.

UMI Processing

A UMI (Unique Molecular Identifier) is a short random tag added to each original molecule before copying. Since copies of the same molecule have the same UMI, you can later identify and remove duplicates that came from the same original. Think of it like adding a unique serial number to each document before photocopying - you can then tell which copies came from which original.

Deduplication

During sample preparation, molecules get copied many times. Identical reads are likely copies of the same original, which can skew your analysis (like counting the same vote multiple times). Deduplication detects and removes these duplicates using a memory-efficient probabilistic filter. It estimates duplication rates without needing to store every read in memory.

Output Splitting

Large datasets can be split into multiple smaller files for parallel downstream processing. You can split by total file count (distribute reads evenly across N files) or by lines per file (each file gets up to N lines). This is useful when you want to process chunks in parallel on different machines.

Reporting

fasterp generates detailed statistics about your data before and after processing: total reads, quality distributions, how many were filtered out and why. The JSON report is machine-readable for pipelines; the HTML report has interactive charts for visual inspection.

Performance

Control how fasterp uses your system resources. More threads = faster processing but more CPU/memory usage. The batch size affects memory consumption - larger batches are more efficient but use more RAM. Use --max-memory to set a limit and let fasterp auto-tune the other settings.

Architecture

fasterp processes data in a pipeline with three stages running in parallel. The Producer reads chunks from disk. Multiple Workers process those chunks simultaneously (filtering, trimming, counting). The Merger collects results in order and writes output. This design keeps all CPU cores busy while maintaining correct output order.

fasterp uses a 3-stage pipeline for multi-threaded processing:

1. Producer: Reads input in 32MB batches, parses FASTQ records
2. Workers: Process batches in parallel (filtering, trimming, stats)
3. Merger: Writes output in order, aggregates statistics

Key optimizations:

• SIMD acceleration (AVX2/NEON) for quality statistics
• Zero-copy buffers with Arc<Vec<u8>>
• Lookup tables for base/quality checks
• Bloom filter for duplication detection

Examples

Quality Control Only

fasterp -i input.fq -o output.fq -A -G

Aggressive Trimming

fasterp -i input.fq -o output.fq \
  --cut-front --cut-tail --cut-mean-quality 20 \
  -q 20 -u 30 -l 50

Paired-End with Merging

fasterp -i R1.fq -I R2.fq -o out1.fq -O out2.fq \
  -m --merged-out merged.fq -c

High-Throughput Pipeline

fasterp -i large.fq.gz -o output.fq.gz \
  -w 16 --max-memory 8192 -z 4

With UMI

fasterp -i input.fq -o output.fq \
  --umi --umi-loc read1 --umi-len 12

Comparison with fastp

fasterp produces byte-exact output for standard parameters:

fastp -i input.fq -o fastp_out.fq
fasterp -i input.fq -o fasterp_out.fq
sha256sum fastp_out.fq fasterp_out.fq  # hashes match