Skip to content

MADCATS 0.14.0: genotype calling, paralog filter, tiled markers - #3

Merged
alex-sandercock merged 86 commits into
Breeding-Insight:developmentfrom
alex-sandercock:development
Nov 15, 2026
Merged

alex-sandercock merged 86 commits into
Breeding-Insight:developmentfrom
alex-sandercock:development

Conversation

@alex-sandercock

Copy link
Copy Markdown
Contributor

Summary

Version 0.14.0. Three additions. All tested, but Madcats call and filter-paralogs should be considered experimental.

  • madcats call (madcats_call.py) calls genotypes from a MADC and writes target, all-variant and haplotype VCFs. It has hwe, norm and structure priors, a selfing rate, an estimated overdispersion and background-read rate, an allelic bias (off, fit, auto), and MPP and PMC.
  • madcats filter-paralogs (madcats_paralogs.py) drops or flags the MADC rows that a reference genome places at another locus. It needs the optional mappy aligner (pip install 'madcats[genome]').
  • generate handles tiled markers (--tile-groups).

The docs include a genotype-calling tutorial, pages for both new commands, new validation sections and a style pass over every page. The 0.14.0 section of the changelog lists every change.

Branches this contains

filter-paralogs, generate-locus-group-ties and copilot/fix-github-actions-job (#2) are all inside this branch, so they can be closed once it merges.

Checks

  • All 257 tests pass on Python 3.9 and 3.13 with mappy installed.
  • The docs build with mdBook 0.5.2. The sdist and wheel build and pass twine check.
  • filter-paralogs on the two lettuce 3K MADCs reproduces the figures in the docs.
  • CI has not run on this branch before. The last Tests run on filter-paralogs failed in two jobs. The Python 3.9 job is fixed by the commit from Run Python 3.9 CI on Ubuntu 24.04 #2. The macOS job failed while installing mappy, and tests.yml now builds it for the runner's architecture. This pull request's run will confirm that.

After merging

Tag v0.14.0 to release. release.yml checks the tag against madcats.__version__.

alex-sandercock and others added 30 commits November 2, 2026 08:59
madcats filter-paralogs cleans a finished MADC (MADCATS output or a vendor
DArT MADC, with or without its starred preamble) against a reference genome.
Each marker is placed by aligning its Ref tag, with a CHROM_POS CloneID as
cross-check, tie-breaker and fallback. Each haplotype is labelled on_target,
paralog, ambiguous, unmapped or unplaced by comparing its best alignment score
at the marker's site with its best score elsewhere. Paralog and ambiguous rows
are dropped by default (--paralog-filter drop|flag, --ambiguous drop|flag|keep);
canonical Ref/Alt rows are never dropped. Flag mode writes labels into the
PICRef column, renamed GenomeLabel. Sidecars list labels per row and placement
per marker; the QC JSON reports rows and reads per label and outcome.

Alignment uses mappy (minimap2 sr preset), an optional extra installed with
pip install madcats[genome] and imported lazily. --genome-index loads or saves
a .mmi so the genome is indexed once.

Both MADC readers now warn on negative read counts and record count_overflow
in the QC JSON and the validate report, naming the signed 8-bit overflow
signature (negatives with a maximum of exactly 127). Counts are never altered.
filter-paralogs now lives in madcats_paralogs.py: alignment, marker
placement, labels, the table filter and its command line. madcats.py keeps
the MADC helpers validate shares with it and imports the module when it
builds the parser. Run as a script, madcats.py registers itself as madcats
so both modules share one MadcError. pyproject lists both modules.

The MADC readers find a header row whose AlleleID is not the first cell,
so filter-paralogs and validate read HapApp's fixed-allele-ID output
(Code_version first, IDs such as |Ref_0001, merged counts as floats). The
Code_version column is passed through unchanged and never read as a sample.

--paralog-filter is renamed --paralogs, matching --ambiguous: each option
names a label and takes its action. The QC parameter and provenance key
follow.

Help text and docs reworded in plainer terms.
CHROM_POS CloneIDs refer to the assembly a panel was designed on.
filter-paralogs reads them three ways: as a cross-check on the Ref-tag
placement, to choose between equal-best Ref-tag sites, and as a fallback
window when the Ref tag does not align. Against another assembly with the
same chromosome names, that window lands at the wrong place, and the
marker's non-canonical rows are dropped as paralogs.

--clone-positions ignore turns all three off: an unaligned marker is
unplaced and keeps its rows, and tied sites stay multi-copy. The default,
use, leaves output unchanged. The setting is recorded in the QC JSON and
provenance, the markers sidecar reports CloneIDCheck=ignored, and the
disagreement warning points to the option.

On the lettuce vendor MADC against v8, ignore changes 10 of 2,987 markers
(9 fallback windows, 1 tie); id95 does not change. The command page gains
a "Using Another Assembly" section with a recipe for comparing two runs.
Co-authored-by: alex-sandercock <39815775+alex-sandercock@users.noreply.github.com>
- CloneID positions under --clone-positions use are positions on the
  genome given. When no equal-best Ref-tag site covers the position, the
  Ref tag is realigned in the window around it; a covering alignment is
  the marker's site (clone_id_site), otherwise the marker is unplaced.
  Before, the Ref tag's genome-wide best site won, so a marker whose
  design site the search missed (Chr2_001081815) or whose assembly
  carries Alt at the target was judged at a paralog.
- Every haplotype is realigned at its own marker's sites, each marker
  with its own small index, and the target score is the higher of that
  and the genome-wide hit. The shared site index let markers crowd each
  other out of the hit limit, and a weaker genome-wide hit at the site
  hid a better realignment.
- One locus rule (half the shorter interval) for merging hits and for
  target membership, so adjacent tandem copies stay distinct.
- Ref tags keep every equal-best site for placement, not the first 8.
- A multi-part .mmi, of which mappy searches only the first part, is
  refused before anything is written.
- Docs: the CloneID rule, one-part indexes, and cross-assembly label
  changes as differences in support rather than proof of copy gain.
- read_madc and filter-paralogs share one count parser.

Lettuce: id95 labels unchanged; DArT keeps 12 more rows (994 reads),
drops none it kept, and bwa keep/drop agreement on Match/Other rows
goes from 94.8128% to 94.8136%.
madcats_call.py holds the statistical core for the planned 'madcats call'
command: beta-binomial dosage likelihoods in log space, uniform and
Hardy-Weinberg priors, posteriors, per-marker iteration of the allele
frequencies, and two ways to make a sample's allele dosages add up to its
ploidy (polyRAD's one-copy-at-a-time correction and an exact search).
Ploidy is per sample.  Standard library only.

tests/test_call.py checks the engine against polyRAD 2.0.1 values stored in
tests/fixtures/polyrad_parity.json, which scripts/make_polyrad_fixture.R
regenerates.  The command line and VCF output are not part of this commit.
The command reads a MADC from generate, filter-paralogs, a vendor or
HapApp, calls every sample with the polyRAD-derived model in
madcats_call.py, and writes one VCF per type chosen with --vcf:

  target     the designed variant of each marker (default); Other rows are
             excluded and reported as XD
  all        targets plus other base substitutions in the tags
  haplotype  whole tag sequences as alleles, with a table tying each allele
             number to its MADC rows

Two-allele records carry GQ, GP and PL.  Records with more alleles carry
MPC, the lowest marginal posterior among the called dosages.  Ploidy is
per sample (--ploidy, --ploidy-file).  Genotypes below --min-dp, --min-gq
or --min-marginal-posterior are written as missing with AD and DP kept.
The target is taken from the Ref and Alt tags only when it is
unambiguous, or from the reference tags' declared position.  Coordinates
are marker-local, and a QC JSON records the counts behind each file.
The beta-binomial overdispersion decides whether polyploid dosages can be
told apart, and polyRAD's default of 9 leaves most tetraploid and
hexaploid genotypes uncalled on simulated amplicon counts.

'--overdispersion auto' picks, from a grid of 2 to 500, the value under
which the markers' reference and alternate read counts are most probable,
with each sample's genotype summed out under Hardy-Weinberg proportions.
It recovered the simulated value in diploid, tetraploid, hexaploid and
mixed-ploidy populations.  It falls back to 9 with a warning when the
estimate ends at the lowest value, as it does for inbred lines.

polyRAD's TestOverdispersion was ported first and matched polyRAD's own
scores, but both underestimate the value in polyploids, so the port was
not kept.  docs/DEVELOPMENT.md records the measurements.

The default stays at 9.
Processed MADCs can carry a readCountSum column between AlleleSequence and
the samples; 'call' read it as a sample.

docs/DEVELOPMENT.md now records the comparison with polyRAD 2.0.1 on three
real tetraploid alfalfa MADCs (genotypes identical except at markers where
polyRAD stops iterating early) and agreement with the dosages published
for two F1 populations.
'--overdispersion auto' is now the default; a number sets the value, and
9 reproduces polyRAD.  On two real tetraploid alfalfa populations the
default of 9 left 59% of target genotypes uncalled, against 8-11% with
the estimated values, at the same agreement with published dosages.

The estimate needs at least 100 sample-by-marker cells with reads and
otherwise falls back to 9 with a note.

docs/DEVELOPMENT.md also records how polyRAD's population-structure and
mapping-population priors compare on the same real counts.
With --prior hwe, --selfing-rate sets the share of individuals that come
from self-fertilisation.  The prior becomes the genotype proportions a
mixed-mating population settles at (de Silva et al. 2005, equation 6, as
in polyRAD's selfing.rate), which moves probability from the intermediate
dosages to the two homozygous ones.  Samples of odd ploidy keep
random-mating proportions, as in polyRAD.  The overdispersion estimate
uses the same prior.

The stored polyRAD reference values gain a selfing section; the priors
match to 13 decimal places and the iterated results to 7.
Real alfalfa data and simulated tables with known genotypes: where the two
callers differ (allelic bias), and where the overdispersion estimate is
fragile (read errors above the contamination rate, populations far from
Hardy-Weinberg proportions).
Dosage probabilities proportional to a normal density at 0..ploidy, with a
mean and standard deviation fitted to the summed posteriors in every
iteration (Gerard and Ferrao 2020).  It does not assume random mating: it
can be narrower or wider than Hardy-Weinberg proportions and flattens out
for inbred lines.

The fit uses Newton's method on the two natural parameters and searches
the edges of updog's parameter limits when the peak lies outside them.
Each ploidy among the samples is fitted separately; a ploidy with fewer
than 10 samples uses Hardy-Weinberg proportions.

tests/fixtures/updog_norm.json (scripts/make_updog_fixture.R) holds updog
2.1.7 reference values: fitted priors for 14 sets of dosage weights and
four complete fits with updog's likelihood parameters held fixed.  Priors
agree to 2e-5 and every genotype is the same.

The default prior is unchanged (hwe), and the polyRAD parity tests pass
unchanged.
Allelic bias (--allele-bias fit).  Every record with two alleles gets
updog's bias (Gerard et al. 2018): the record is called at updog's five
starting values, each from scratch, and the best is refined by a
golden-section search; the result is the call with the highest
log-likelihood after updog's normal prior on the log of the bias.  With
the other settings equal, the fitted bias is within 0.1% of updog 2.1.7's
and the genotypes are the same (tests/fixtures/updog_norm.json).  The VCF
gains INFO/BIAS and PREFIX.bias.csv lists each fit.

Background-read rate (--background-rate, default auto).  The share of
reads that show an allele they did not come from, in the form updog uses
for sequencing error, estimated for the table together with the
overdispersion.  polyRAD's contamination rate is kept but is 0 unless
set: it cannot account for stray reads of an allele no other sample
carries, and with it too low the overdispersion estimate was pulled
down.  The bias is applied before the background reads, so samples that
do not carry an allele do not inform its bias.

The estimate of both parameters uses the norm prior, fits the bias when
the run does, and runs on the worker pool.

Two-allele records are computed for the alternate allele and mirrored
for the reference allele, and terms shared by the samples of a ploidy
are computed once per pass.  HWE results on real data are unchanged byte
for byte.

polyRAD's model as polyRAD runs it remains available:
--prior hwe --overdispersion 9 --contamination-rate 0.001 --background-rate 0
(a test checks its genotypes against the polyRAD reference values).
Bias search.  The search picked which starting value to refine by its
score before refinement.  At a marker where all samples share one or two
dosages, the reads fit equally well with every dosage shifted by one and
a strong bias, so that score could favour the wrong one.  Every starting
value that scores at least as well as its neighbours is now refined, and
the final scores are compared.  On an updog-simulated tetraploid F1
table with biased markers, accuracy with the norm prior rose from 90.7%
to 93.3% (updog: 94.0%).

Free-weights prior.  A prior with one free probability per dosage
(updog's 'flex'), fitted from the summed posteriors like the norm prior.
It is for estimating the read model and is not offered for calling.  The
estimate does not use it yet.
The estimate of the overdispersion and the background-read rate assumed
genotype proportions that inbred samples do not follow.  With
Hardy-Weinberg proportions or the norm shape, neither of which can
describe an excess of homozygotes, it ended at the lowest overdispersion
on a polyRAD-simulated table with inbreeding 0.5 (simulated value 20)
and on fully inbred tables.  With one free probability per dosage it
found 20 there, 40 on a fully inbred table simulated with 40, and the
same values as before on outbred, F1, two-subpopulation and hexaploid
tables.

The fallback to 9 remains for a table with no heterozygous samples and
no stray reads.
docs/DEVELOPMENT.md gains the background-read rate and its joint estimate
with the overdispersion, the free-weights prior the estimate uses, the
allelic bias and its agreement with updog, the finished model on the two
alfalfa populations, accuracy on twenty simulated tables (eight from the
earlier comparison, seven from updog's simulator, five from polyRAD's),
and wall time, CPU time and memory for polyRAD, updog and call on one
and four cores.

The command reference says when to fit the bias, and its timing and
estimate figures are those of the finished code.
The recovery table was measured with the first version of the estimate,
which assumed Hardy-Weinberg proportions. Re-run with free weights and
the background-read rate estimated alongside: every simulated value from
10 upward is recovered, F1 families included, and polyploid estimates at
5 are 6 to 8.
Eighty simulated tables (four kinds of population, three background
rates, two overdispersions, five combinations of ploidy and depth) were
estimated with the free-weights prior. The background rate is recovered
in 78; the overdispersion in 31 of 32 diploid and tetraploid tables with
20 or more reads. The command reference and the docstring of
estimate_read_model now state those results, which replace figures
measured with an earlier estimation prior.
…ings

A clean re-time on the benchmark table with the overdispersion and the
background-read rate given as numbers: calling alone takes 6 s (hwe),
10 s (norm) and 97 s (norm with the bias fitted) on one core, so the
estimate accounts for 27 s of a default run and about 200 s of a run
with the bias. Adds the four-core figure for polyRAD's settings (2 s).
The command reference says that giving the two values skips the
estimate.
Eighty-five tetraploid potato clones with DArTag read counts and
Infinium array genotypes (Endelman et al. 2024; the files ship with
polyBreedR) give call a truth that does not come from a read-count
caller. With polyRAD's settings call matches polyRAD (87.9% of dosages
equal the array's); the estimated read model gives 92.3%, the allelic
bias 96.6%, and updog on the same samples 96.7%.

The command reference and the changelog cite the result where they
describe when to fit the bias.
Each sample gets Hardy-Weinberg proportions at its own allele frequency,
the mean over the samples most like it across the table's markers with
the record's frequency counted as one more. The like samples are found
once for the table (find_neighbours) from a first pass with the hwe
prior; how many to use is chosen by how well neighbours found on one
half of the markers predict the samples at the other half, and is 0 (the
hwe prior) when that is no better than the table's own frequencies.
Records are then called independently as before (_neighbour_passes), so
the worker processes still apply and no dependency is added.

On thirteen tables (simulated groups, clines and families at 8 to 150
reads, potato DArTag counts thinned to 10 and 25 reads) the prior calls
as many genotypes correctly as polyRAD's IteratePopStruct or up to 0.7
points more, and gains 0.5 to 5.5 points over hwe at 8 to 25 reads. At
full DArTag depth it changes little.

It is refused together with --allele-bias fit and --selfing-rate, which
have not been tested with it. The read counts of the target records are
now gathered by one function for the read-model estimate and for this.
States the untested cases plainly and records the mixed-ploidy run with
all three VCF types, which completes and is read by bcftools.
An F1 prior for one biparental population with both parents on the plate
gains 0.1 to 0.7 points over the best of hwe, norm and structure on the
two alfalfa populations, at full depth and with reads thinned to 25 and
10. Only parental genotypes known from elsewhere help much, in many
small families, and that needs a pedigree. Records the figures so the
question need not be measured again.
…parents

Haplotype records: on simulated microhaplotype tables (3 to 6 alleles,
diploid and tetraploid; one population, groups, families) hwe matches
polyRAD's HWE prior, structure is never below hwe and gains 1 to 4
points where there are groups or families, norm gains at most 1.7, and
uniform is lowest. The default confidence filter keeps genotypes that
are 94 to 99% right under each prior. polyRAD publishes no accuracy
baseline for haplotype calls; on its own DArTag example call gives the
same genotypes.

Biparental population with both parents among the samples: the norm
prior is within 0.4 points of an F1 prior and of polyRAD's mapping
prior, and with the bias fitted within 0.3 points of updog's norm and
f1 models. Family priors cost 3 to 11 points on off-type samples.

The command reference states which priors were tested on haplotype
records and how.
Without a contamination rate, which is the command's default, a record's
likelihoods do not depend on the allele frequencies, yet they were
recomputed on every pass. They are now computed once per record. The
read-model estimate computes each sample's binomial coefficient once
instead of once per overdispersion tried, selects its markers once
instead of once per background rate, and the target read counts are
gathered once per run. The neighbour search multiplies plain lists.

Every output of twelve runs covering all priors and options is
byte-identical to before; the set ran in 1 min 38 s where it took
4 min 23 s (the earlier figure with other jobs running).
The README had two list items spliced into each other (the paralog
filter and call), and its command table had no filter-paralogs row.  The
documentation landing page listed neither command.  Both now say that a
MADC can go straight to call, and the workflow page has the optional
genotype-calling step beside the paralog filter.
A table near the top of the page says which settings to change for
polyploids on an amplicon panel, families, low-depth structured samples
and inbred lines, with the measured effect of each.

The page said to use --allele-bias auto when in doubt.  That is wrong for
inbred lines: on simulated inbred tetraploids every fitted bias ended at
the limit of the search, auto kept them all, and 71.7% of dosages were
right against 97.4% without a bias.  --selfing-rate is what helps them
(99.6% at 0.95, 99.9% at 0.99).  The guidance for families and for
hexaploids now gives the figures for auto as well as fit.
… inbred lines

The twenty simulated tables were called again with each prior and each
bias setting.  auto costs nothing on one population or an F1 family
without bias, keeps most of the gain at 150 reads and less at 25, and
fails on inbred lines as fit does, where every bias ends at a limit of
the search.  A selfing rate is what helps inbred lines.

Refusing a bias at a limit under auto was tried in a scratch copy: it
restores the inbred tables and costs 0.1 points on the potato data.  The
rule is unchanged.

One cell of the earlier simulator table did not reproduce (norm with a
fitted bias on updog's F1 table: 93.3% recorded, 93.9% now) and is
corrected; the other 79 repeated figures are the same.

The call page gains the depth figures for auto, and the changelog says
that auto does not protect inbred lines.
…cords without a bias as off does

Three changes to how a fitted allelic bias is used.

A bias at a limit of the search is not used under auto.  Samples with few
heterozygotes, such as inbred lines, drive every fit to a limit, where
homozygous samples pass for heterozygous ones; the fits gain far more
than the margin, so auto used them all.  call now warns, the bias table
has a Rejected column, and the QC JSON counts the records called without
their bias by reason and lists those rejected at a limit.  fit still
uses such a bias, and warns when a fifth or more of the fits end at one.

Under auto a record called without its bias is called under the read
model estimated without a bias, exactly as --allele-bias off calls it.
Only the records that keep their bias use the read model estimated with
the bias fitted.  Before, every record used the latter, which cost 1.9
points on a simulated hexaploid table where no record kept its bias and
left the calls over-confident.

The estimate with a bias fitted starts from the estimate without one.
Started from a fixed value its search could stop early: on that table
it chose an overdispersion of 10 where 20 is more probable by 105 log
units, with or without the bias.
The tutorial runs call on the bundled example and shows its output, then
covers ploidy, which settings to change for which samples, the record
types, what to check in the QC JSON, filtering markers afterwards and the
limits of the model.

The README opens with what the tool is for in four sentences and five
points, in place of twelve, and its quick start ends with a call step.
…e validation page

The settings section now separates the two changes that clearly help
(a bias for polyploids on an amplicon panel, a selfing rate for inbred
lines) from the two priors that helped only under particular conditions.
For norm in families and structure at low depth it says where each
helped and where it did not: norm gained in simulated families and at
about 10 reads, and was 0.8 points below hwe to 0.2 above on real
families at high depth.

The model, read-model and bias sections keep what the options do and
when to use them.  The measurements behind them are on the validation
page, under priors and read model.
…updog and polyRAD users

The settings step now says where to start (the defaults, with
--allele-bias auto for polyploids), which samples want a change and why,
and which options give the model a user of updog's norm or polyRAD's
IterateHWE and IteratePopStruct is used to.
…ttings

The validation page gains two sections on the allelic bias.  The first
gives the share of dosages right under off, fit and auto on the real
data sets and on simulated scenarios as means over four or five draws,
with what auto keeps of the gain of fit at each depth.  The second gives
how far the probability call states for its genotypes lies above the
share that are right.

The call page takes its figures from these: the conditions under which
norm and structure helped now carry the replicate results, and the
advice on when to use fit in place of auto names hexaploids and depths
under about 50 reads.
In simulated F1 families norm is ahead of hwe in every draw, and with it
auto keeps no bias that is not there.  In one randomly mating population
it is level with hwe without bias in the data and 0.5 to 0.7 points
below with one.
Twenty-five simulated scenarios from three simulators, four or five draws
each, and the real data sets, called with the rules of auto as built.
The record has the share of dosages right for each bias setting, the
difference of auto from off within each draw, the records whose bias
auto used, and how far the stated probability of the calls lies above
the share that are right; then the same for the norm prior on nine
scenarios, and norm and structure against hwe.

Where the data have no bias and the samples are one population or inbred
lines, auto is off in every draw.  Where they have one, it keeps 97 to
100% of the gain of fit at 150 reads and less at lower depth.  Under the
plain hwe prior it still uses a bias at 2 to 12 records in 100 for
families, subpopulations and half-inbred samples without bias, and loses
0.6 to 0.8 points; with a prior that fits them the loss is 0.2 or less.

The call page, the validation page and the tutorial say what to use for
partly inbred samples.
The introduction and the Highlights list go back to their wording and
layout from before the rewrite.  The content updates stay: call and
filter-paralogs in the Highlights and the command table, the call step
in the quick start, and the tutorials named in the documentation list.
Two-allele records carry FORMAT/MPP, the posterior probability of the most
probable genotype, and INFO/PMC, one minus the mean of MPP over the samples
with reads. These are updog's maxpostprob and prop_mis under the names
BIGr's updog2vcf writes, so BIGr's filterVCF can filter on them.

PMC does not depend on --min-dp or --min-gq. Samples without reads are left
out, which is what updog gives when they are passed as missing; given a
depth of 0, updog counts them at the largest probability of the prior.
Records with more than two alleles carry neither field.

The updog fixture now stores maxpostprob and prop_mis of its fits and one
fit with samples without reads; regenerating it left the earlier values
unchanged. The tests compare the written MPP and PMC with them.
The call page describes the two fields, how they relate to updog's values
and what they can and cannot say. The tutorial shows them in its example
record and in the section on filtering markers. The validation page has the
comparison with updog and with known genotypes.
What updog's code and documentation define, the choices made for call, the
agreement with updog's stored values, PMC against the true share of wrong
genotypes on the replicate tables beside updog's prop_mis, PMC as a marker
filter on the potato panels, and what of BIGr's filterVCF was run.
Part of a pass over the documentation that removes AI words and stock
phrases, uses fewer passive constructions, and replaces most colons and
semicolons inside sentences. Facts, numbers, code, links and tables stay
as they were. This commit holds the pages edited so far; the remaining
pages and the final check follow.
Finishes the pass started in the previous commit. Across the docs site
and the README, sentences use fewer passive constructions, colons inside
sentences and semicolons give way to separate sentences, AI words and
stock phrases are gone, and bold used for emphasis inside sentences and
table cells is plain text. Bold bullet lead-ins and paragraph titles stay.

The README keeps its introduction, Highlights and layout, with only the
sentence-level fixes. Facts, numbers, code, links and tables are as
before. The "Know the Limits" heading of the genotype-calling tutorial is
now "Limits"; nothing linked to it.
The same pass as the documentation site. Sentences use fewer passive
constructions, semicolons and colons inside sentences give way to
separate sentences, em dashes that joined clauses are gone, AI words and
stock phrases are removed, and bold inside sentences and table cells is
plain text. Bold paragraph titles stay, reworded where they held a colon.

Numbers, code, links, headings and tables are as before.
The Tests section still said 59.  It now gives the count per test file
and says which tests need mappy.  .gitignore lists *.mmi, since
filter-paralogs saves the index it builds under --genome-index.
mappy has no macOS wheel for Python 3.13, and its source does not compile
as a universal (arm64 + x86_64) binary.  The macOS test job failed at
that step on the last run of this workflow.  The step now sets ARCHFLAGS
to the runner's architecture on macOS, and the installation page gives
the same command for the universal Pythons from Apple and python.org.
Of the 113,667 rows kept from the vendor MADC, 5,687 are the designed
alleles with 87.5% of the kept reads.  Of the other 105,913, 84% never
reach 5 reads in a sample, 8% reach 5 only in 25 samples that stand apart
from the rest, and 1,697 recur in five or more of the other 351.

blastn on a sample of 18,763 rows makes the filter's decision for 99.4 to
99.6% of the rows dropped as paralog and for 69% of the well-supported
rows kept (89% of their reads).  bwa mem and blastn both keep 97% of the
kept rows within 3 differences of their site and 27% of those 6 or more
away.  Kept rows that bwa mem would drop hold 1.0% of the kept reads.

Recorded in the development record, the command page (limitations and how
it was checked), the validation page and the tutorial.  No code changes.
Copilot AI balanced review requested due to automatic review settings November 15, 2026 22:43
@alex-sandercock alex-sandercock self-assigned this Nov 15, 2026
@alex-sandercock alex-sandercock added the enhancement New feature or request label Nov 15, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The genotype truth checks can accept missing calls as homozygous-reference calls, and the documentation omits the odd-ploidy exception for a selfing rate of 1.

2 open issues
What changed in this PR

This PR prepares MADCATS 0.14.0, extending the FASTQ-to-MADC workflow with experimental genotype calling and genome-based paralog filtering, plus support for tiled markers.

Changes:

  • Documents genotype calling, paralog filtering, output formats, and validation results.
  • Adds tiled-marker tests, reference-fixture generators, and packaging for the new command modules.
  • Updates CI for Python 3.9 compatibility and optional mappy installation on macOS.
File Description
tests/​test_tiled_markers.py Adds end-to-end tiled-marker checks.
tests/​test_madcats.py Tests tile grouping, read credits, and caching.
scripts/​make_updog_fixture.R Generates updog reference fixtures.
scripts/​make_polyrad_fixture.R Generates polyRAD reference fixtures.
README.md Introduces calling and paralog filtering.
pyproject.toml Packages new modules and the genome extra.
docs/​src/​tutorials/​primer-free.md Clarifies primer-free workflow wording.
docs/​src/​tutorials/​paralog-filter.md Adds a genome-filtering tutorial.
docs/​src/​tutorials/​paired-end.md Clarifies paired-end and tiling guidance.
docs/​src/​tutorials/​genotype-calling.md Adds a genotype-calling tutorial.
docs/​src/​tutorials/​dartag.md Clarifies DArTag workflow wording.
docs/​src/​tutorials/​agriseq.md Clarifies normalization guidance.
docs/​src/​SUMMARY.md Links new tutorials and command pages.
docs/​src/​reference/​validation.md Adds caller and paralog-filter validation results.
docs/​src/​reference/​output-formats.md Describes genome labels, VCFs, and tile accounting.
docs/​src/​reference/​faq.md Explains optional dependencies and count overflow.
docs/​src/​reference/​evidence-model.md Documents tile assignment and read accounting.
docs/​src/​README.md Updates the documentation introduction and features.
docs/​src/​getting-started/​workflows.md Adds optional filtering and calling stages.
docs/​src/​getting-started/​quickstart.md Clarifies quick-start wording.
docs/​src/​getting-started/​installation.md Documents the genome extra and macOS installation.
docs/​src/​getting-started/​input-formats.md Defines tile families and grouping modes.
docs/​src/​getting-started/​example.md Clarifies examples and links genotype calling.
docs/​src/​development/​contributing.md Updates dependency and release guidance.
docs/​src/​development/​changelog.md Records 0.14.0 changes.
docs/​src/​development/​architecture.md Describes the three-module architecture.
docs/​src/​commands/​window-scan.md Clarifies scanning documentation.
docs/​src/​commands/​validate.md Documents HapApp headers and overflow reporting.
docs/​src/​commands/​overview.md Adds filtering and calling workflows.
docs/​src/​commands/​normalize-reads.md Clarifies normalization options.
docs/​src/​commands/​merge-pairs.md Clarifies paired-read merging behavior.
docs/​src/​commands/​infer-design.md Clarifies panel inference documentation.
docs/​src/​commands/​generate.md Documents tile handling and its option.
docs/​src/​commands/​filter-paralogs.md Adds the paralog-filter command reference.
docs/​src/​commands/​design.md Clarifies design command wording.
docs/​src/​commands/​call.md Adds caller models, options, and output details.
docs/​src/​citation.md Adds citations for filtering and calling methods.
.gitignore Ignores generated minimap2 indexes.
.github/​workflows/​tests.yml Pins the Python 3.9 runner and installs mappy.

Review effort: Balanced

Comment thread docs/src/commands/call.md Resolved
Comment thread tests/test_tiled_markers.py Resolved
… tile test

The call page said that samples of odd ploidy keep random-mating
proportions under --selfing-rate.  That holds below 1.  At 1 hwe_prior
gives every ploidy the two homozygous dosages only, as polyRAD does, and
the page now says so.

The tiled-marker test read a genotype that was not called (./.) as an
alternate dosage of 0, the same as 0/0, so an uncalled sample could pass
for a homozygous reference call.  It now reads None.
Copilot AI balanced review requested due to automatic review settings November 15, 2026 22:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

In auto mode, an undeclared same-window marker can merge families across declared group boundaries and credit reads across those boundaries, requiring a fix or explicit documentation of the limitation.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Prevent auto families from bridging declared groups

docs/​src/​getting-started/​input-formats.md:111

In auto mode, an undeclared same-window marker can bridge markers declared in amp1 and amp2, placing all three in one family and crediting a read across the stated group boundary. Check the declared groups of entire families before merging them, and add a three-marker regression test. If this behavior is intentional, document the limitation here and recommend declared when group boundaries must be preserved.

2 resolved since last review

Review effort: Balanced

@alex-sandercock
alex-sandercock merged commit 92aa346 into Breeding-Insight:development Nov 15, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants