Partitioning and (dis)similarity
Source:vignettes/articles/partitioning-and-dissimilarity.Rmd
partitioning-and-dissimilarity.RmdMeasuring how different samples are from one another is a separate question from how diverse each one is. The classic way to connect the two uses three quantities (Whittaker’s alpha/beta/gamma):
- alpha — the diversity within a typical single sample;
- gamma — the diversity of all samples pooled together;
- beta — how much the samples differ from one another, obtained from the other two.
hilldiv3 computes this split by
partitioning gamma into an alpha and a beta component,
then (optionally) converting beta into bounded (dis)similarity metrics
that are easy to compare across studies. All of this works for neutral,
phylogenetic and functional diversity — just add a tree or
dist as in the diversity-types article.
counts <- matrix(
c(10, 0, 5,
2, 8, 1,
3, 4, 0,
6, 2, 7),
nrow = 3,
dimnames = list(c("t1", "t2", "t3"), c("s1", "s2", "s3", "s4"))
)Alpha, gamma and beta
hillpart() returns the three components per diversity
order:
hillpart(counts)
#> Partitioning neutral Hill numbers of "q0", "q1", and
#> "q2".
#> <hilldiv3 result: neutral>
#> 9 rows x 3 cols
#>
#> q component value
#> 1 0 alpha 2.500000
#> 2 1 alpha 2.154269
#> 3 2 alpha 1.968927
#> 4 0 gamma 3.000000
#> 5 1 gamma 2.905735
#> 6 2 gamma 2.828374
#> 7 0 beta 1.200000
#> 8 1 beta 1.348827
#> 9 2 beta 1.436505- alpha — the effective diversity of a typical single sample.
- gamma — the effective diversity of all samples pooled.
-
beta —
gamma / alpha, the effective number of distinct communities. It ranges from1(all samples identical) toN(all samples completely distinct), whereNis the number of samples.
Beta is the multiplicative decomposition of Jost (2007): it is independent of alpha, which is what makes it comparable across datasets with different richness.
Multi-scale (hierarchical) partitioning
Real sampling designs are usually nested: replicates
within sites, sites within regions, regions within a study. A single
alpha/beta split throws that structure away — it cannot tell you whether
turnover happens between nearby replicates or between
distant regions. hillpart() solves this with a
hierarchy formula that decomposes gamma into one beta
per hierarchical level, so you can see at which spatial
(or temporal, or organisational) scale diversity actually turns
over.
Supply the nesting as a one-sided formula, coarsest to finest, plus a
metadata table giving each sample’s group memberships:
set.seed(1)
tab <- matrix(rpois(12 * 8, 5), nrow = 12,
dimnames = list(paste0("t", 1:12), paste0("s", 1:8)))
md <- data.frame(
region = rep(c("N", "S"), each = 4),
site = rep(c("a", "b", "c", "d"), each = 2),
row.names = paste0("s", 1:8)
)
hillpart(tab, hierarchy = ~ region / site, metadata = md)
#> Partitioning neutral Hill numbers across scales "sample < site < region <
#> total".
#> <hilldiv3 result: neutral>
#> 12 rows x 5 cols
#>
#> q scale n_units diversity beta
#> 1 0 sample 8 12.00000 NA
#> 2 0 site 4 12.00000 1.000000
#> 3 0 region 2 12.00000 1.000000
#> 4 0 total 1 12.00000 1.000000
#> 5 1 sample 8 11.11495 NA
#> 6 1 site 4 11.60076 1.043708
#> 7 1 region 2 11.77266 1.014817
#> 8 1 total 1 11.95901 1.015829
#> 9 2 sample 8 10.47242 NA
#> 10 2 site 4 11.24852 1.074109
#> 11 2 region 2 11.56260 1.027923
#> 12 2 total 1 11.92231 1.031110The result is one row per (q, scale), finest
(sample) to coarsest (total):
-
diversityis the effective diversity at that scale (A_k): the per-sample alpha atsample, the pooled gamma attotal, interpolating in between. -
betais the turnover gained at that scale,A_k / A_{k-1}— e.g. thesitebeta is turnover among samples within a site, theregionbeta among sites within a region, thetotalbeta among regions. -
n_unitsis how many units exist at that scale (8 samples → 4 sites → 2 regions → 1 total).
The betas multiply back to the overall beta exactly:
This telescoping identity is guaranteed by
construction (and checked in the package tests). With equal sample
weights, neutral and functional endpoints match plain
hillpart(); phylogenetic endpoints match on an ultrametric
tree at the default depth. Arbitrarily deep hierarchies work — just
chain more terms, e.g. ~ study / region / site. The
out = "matrix" form returns the same information as
alpha, one beta_<level> column per
transition, and gamma.
How it works (and what it assumes)
Advanced — skip on a first read. The formula below explains why the betas behave well; you do not need it to use
hillpart(). The practical takeaways are the bullet points on assumptions that follow it.
All three diversity types share one construction with sample weights (equal weights by default). For a unit , let and . Each scale uses a linear-weighted power sum,
with the entropy limit
at
,
where
.
Here
pools feature contributions,
is the per-feature measure, and
is a grand-total normaliser. Neutral diversity uses taxa with measure
;
phylogenetic diversity uses branches with measure
;
functional diversity uses taxa with measure
(attribute contribution at threshold tau). Coarsening takes
weighted means of the
,
so convexity or concavity keeps every
.
Phylogenetic and functional hierarchical partitioning simply add a
tree or dist, as everywhere else in
hilldiv3:
hillpart(tab, hierarchy = ~ region / site, metadata = md, tree = my_tree)
hillpart(tab, hierarchy = ~ region / site, metadata = md, dist = my_dist)Limitations and assumptions — worth knowing before interpreting the betas:
-
Sample weights.
weights = "equal"gives every sample weight1/n;"abundance"uses sample count totals, and a positive numeric vector supplies custom weights. Unit weights are sums of their sample weights. -
Phylogenetic: one shared tree depth across scales.
Alpha and gamma are reported in effective-branch-length (PD) units and
share a single mean depth
T_+over all scales by default. On an ultrametric tree,depthcan select the lastdepthunits before the tips. For ultrametric trees the finest alpha equals the single-level phylogenetichillpart()alpha exactly; for non-ultrametric trees the chain still telescopes, but the sharedT_+makes the per-scale PD values depth-pooled rather than strictly per-sample — prefer ultrametric (time-calibrated) trees here. -
Functional: one shared
tauandv_iper run. Eachtauis applied across every scale, withv_icomputed from the pooled data. A vector oftauvalues runs one full hierarchy per threshold; per-scaletauvalues are not supported. -
Hierarchical overlap and pairwise comparisons.
hilldiss()andhillsim()convert each transition’s beta to bounded dissimilarity metrics using its data-dependent ceiling.hillpair()compares weighted pooled units at a selected hierarchy level.
This nested multiplicative decomposition across neutral, phylogenetic and functional Hill numbers in one engine is, to our knowledge, not available in other packages; it generalises the single-level partition above rather than replacing it.
From beta to (dis)similarity
Beta lives on [1, N], which is awkward to compare across
studies with different sample counts. hilldiss() and
hillsim() rescale it to [0, 1] using the four
(dis)similarity measures of Chiu et al. (2014): two turnover types and
two overlap-complement types. The table describes the dissimilarities
returned by hilldiss(); hillsim() returns one
minus each.
| Code | Measure | Perspective |
|---|---|---|
S |
Jaccard-type turnover | turnover relative to the pooled assemblage (gamma) |
V |
Sorensen-type turnover | turnover relative to a single assemblage (alpha) |
U |
Jaccard-type overlap complement | proportion of non-shared units in the pooled assemblage |
C |
Sorensen-type overlap complement (1 − Morisita-Horn at
q = 2) |
effective average proportion of non-shared units in a single assemblage |
hilldiss(counts) # dissimilarities, long table (q, metric, value)
#> dissimilarity from neutral Hill numbers of "q0", "q1",
#> and "q2".
#> <hilldiv3 result: neutral>
#> 12 rows x 3 cols
#>
#> q metric value
#> 1 0 S 0.22222222
#> 2 1 S 0.34481987
#> 3 2 S 0.40515478
#> 4 0 C 0.06666667
#> 5 1 C 0.21585249
#> 6 2 C 0.40515478
#> 7 0 U 0.22222222
#> 8 1 U 0.21585249
#> 9 2 U 0.14550174
#> 10 0 V 0.06666667
#> 11 1 V 0.11627556
#> 12 2 V 0.14550174
hillsim(counts) # similarities = 1 - dissimilarity
#> similarity from neutral Hill numbers of "q0", "q1", and
#> "q2".
#> <hilldiv3 result: neutral>
#> 12 rows x 3 cols
#>
#> q metric value
#> 1 0 S 0.7777778
#> 2 1 S 0.6551801
#> 3 2 S 0.5948452
#> 4 0 C 0.9333333
#> 5 1 C 0.7841475
#> 6 2 C 0.5948452
#> 7 0 U 0.7777778
#> 8 1 U 0.7841475
#> 9 2 U 0.8544983
#> 10 0 V 0.9333333
#> 11 1 V 0.8837244
#> 12 2 V 0.8544983Request a single metric and order, and ask for the legacy matrix form
with out = "matrix":
hilldiss(counts, q = 1, metric = "C", out = "matrix")
#> dissimilarity from neutral Hill numbers of "q1".
#> [1] 0.2158525The choice of q matters as much as the metric:
q = 0 counts all taxa equally (presence/absence), while
q = 2 reflects shifts among dominant taxa. At
q = 0, U equals S and
C equals V; at q = 1,
U equals C.
Pairwise dissimilarity for ordination
hilldiss() summarises all samples at once. To
compare samples two-by-two — the input ordination methods expect — use
hillpair(), which returns a dist object (or a
long tibble):
d <- hillpair(counts, q = 1, metric = "C")
#> Computing neutral pairwise dissimilarity for 6 sample pairs.
d
#> s1 s2 s3
#> s2 0.52298484
#> s3 0.47119926 0.08924381
#> s4 0.09902103 0.29281463 0.33948330That dist drops straight into ordination and
clustering:
pcoa <- cmdscale(d, k = 2)
plot(pcoa, type = "n", xlab = "PCoA 1", ylab = "PCoA 2")
text(pcoa, labels = rownames(pcoa))
Ask for several orders/metrics at once and you get a named
list of dist objects (e.g. q0C,
q2U):
ds <- hillpair(counts, q = c(0, 2), metric = c("C", "U"))
#> Computing neutral pairwise dissimilarity for 6 sample pairs.
names(ds)
#> [1] "q0C" "q2C" "q0U" "q2U"Or a tidy long table, convenient for ggplot2 or
dplyr:
head(hillpair(counts, q = 1, metric = "C", out = "tibble"))
#> Computing neutral pairwise dissimilarity for 6 sample pairs.
#> first second q metric value
#> 1 s1 s2 1 C 0.52298484
#> 2 s1 s3 1 C 0.47119926
#> 3 s1 s4 1 C 0.09902103
#> 4 s2 s3 1 C 0.08924381
#> 5 s2 s4 1 C 0.29281463
#> 6 s3 s4 1 C 0.33948330Which taxa account for dissimilarity?
hillcontrib() ranks taxa by their contribution to the
same bounded dissimilarity reported by hillpair() and
hilldiss(). For each comparison, the
contribution column sums to the selected metric;
share is the fraction of that metric assigned to the taxon.
Set by = "collective" to examine the turnover among all
samples or among named groups of samples.
hillcontrib(counts, q = 1, metric = "C")
#> first second group q type metric set contribution share rank
#> 1 s1 s2 <NA> 1 neutral C t2 0.3636363636 0.695309574 1
#> 2 s1 s2 <NA> 1 neutral C t1 0.1062323151 0.203126951 2
#> 3 s1 s2 <NA> 1 neutral C t3 0.0531161576 0.101563475 3
#> 4 s1 s3 <NA> 1 neutral C t2 0.2857142857 0.606355548 1
#> 5 s1 s3 <NA> 1 neutral C t3 0.1666666667 0.353707403 2
#> 6 s1 s3 <NA> 1 neutral C t1 0.0188183079 0.039937049 3
#> 7 s1 s4 <NA> 1 neutral C t2 0.0666666667 0.673257663 1
#> 8 s1 s4 <NA> 1 neutral C t1 0.0243018651 0.245421254 2
#> 9 s1 s4 <NA> 1 neutral C t3 0.0080524973 0.081321083 3
#> 10 s2 s3 <NA> 1 neutral C t3 0.0454545455 0.509329954 1
#> 11 s2 s3 <NA> 1 neutral C t1 0.0370279144 0.414907370 2
#> 12 s2 s3 <NA> 1 neutral C t2 0.0067613499 0.075762676 3
#> 13 s2 s4 <NA> 1 neutral C t2 0.1626372003 0.555427164 1
#> 14 s2 s4 <NA> 1 neutral C t3 0.0999340079 0.341287617 2
#> 15 s2 s4 <NA> 1 neutral C t1 0.0302434233 0.103285219 3
#> 16 s3 s4 <NA> 1 neutral C t3 0.2333333333 0.687319037 1
#> 17 s3 s4 <NA> 1 neutral C t2 0.1057945504 0.311634036 2
#> 18 s3 s4 <NA> 1 neutral C t1 0.0003554141 0.001046926 3
hillcontrib(counts, q = 1, metric = "C", by = "collective",
groups = list(first = c("s1", "s2"), second = c("s3", "s4")))
#> first second group q type metric set contribution share rank
#> 1 <NA> <NA> first 1 neutral C t2 0.3636363636 0.695309574 1
#> 2 <NA> <NA> first 1 neutral C t1 0.1062323151 0.203126951 2
#> 3 <NA> <NA> first 1 neutral C t3 0.0531161576 0.101563475 3
#> 4 <NA> <NA> second 1 neutral C t3 0.2333333333 0.687319037 1
#> 5 <NA> <NA> second 1 neutral C t2 0.1057945504 0.311634036 2
#> 6 <NA> <NA> second 1 neutral C t1 0.0003554141 0.001046926 3You can also combine taxa with
sets = list(my_set = c("t1", "t2")), as in
hillshare(). Named sets may cover only some taxa or
overlap, so their rows need not sum to the total dissimilarity. With a
tree, shared branch turnover is distributed among descendant taxa
according to their own abundance variation; this is an explicit
allocation rule for phylogenetic dissimilarity.
For functional diversity, the partitioning engine uses raw counts. Unequal sample totals can therefore contribute to the functional dissimilarity and its taxon attribution even when relative composition is unchanged.
Larger datasets
hillpair() computes every pair through the same
partitioning engine, so cost grows with the number of pairs. For many
samples, run pairs in parallel by setting parallel = TRUE
with a future plan and the furrr package
installed:
Contrasts with other packages
Advanced — skip on a first read. This section is for users coming from other Hill-number tools who want to reconcile differing numbers. Newcomers can safely move on.
hillR and hilldiv2 also split Hill numbers
into alpha, gamma and beta. For neutral and
phylogenetic diversity their partitions match
hilldiv3 to numerical precision:
hilldiv2::hillpart(), hillR::hill_taxa_parti()
and hillR::hill_phylo_parti() return the same alpha, gamma
and beta as hillpart(). The functional
partition is where they part ways — and, as in the diversity types article, these are
convention differences, not bugs.
hillR::hill_func_parti
Three choices stack up, which is why the numbers look unrelated (≈1.5 vs several thousand) rather than merely rescaled:
-
What is reported.
hilldiv3returns the functional Hill numberqD— an effective number of functionally distinct species, bounded in[1, S]and in the same currency as its neutral and phylogenetic output.hill_func_parti()returns the functional attribute diversityFD = qD^2 * Q(whereQis Rao’s quadratic entropy); atq = 0this issum(d_ij), the total pairwise distance (Walker’s FAD). The squaring and theQfactor account for most of the apparent gap. -
The distance threshold
tau.hilldiv3usestau = max(d_ij)by default;hill_func_parti()fixestau = Q(≈ the abundance-weighted mean distance). A smallertaumakes species look more distinct and inflatesqD, so the threshold alone shifts the effective count several-fold.hilldiv3exposestauas an argument, so you can match either convention — or, as Chiu, Jost & Chao (2014) recommend, profile diversity across a range oftau. -
The scale of beta. Because
FDsquares the Hill number,hillR’s functional betaFD_gamma / FD_alphaequals(qD_gamma / qD_alpha)^2and lives in[1, N^2].hilldiv3’s functional beta is the Hill ratioqD_gamma / qD_alphain[1, N], directly comparable to its own neutral and phylogenetic betas. AhillRfunctional beta of1.6is therefore not on the same scale as a taxonomic beta of1.6.
hilldiv2::hillpart() shares hilldiv3’s
conventions exactly — the same tau = max(d_ij) threshold
and the same functional Hill number qD — so its functional
partition matches hilldiv3 to machine precision.
traits <- data.frame(body_size = c(1, 5, 9), row.names = rownames(counts))
d <- dist(traits)
# hilldiv3 reports the functional Hill number qD (effective number of species).
hillpart(counts, q = c(0, 1, 2), dist = d)
# hillR reports FD = qD^2 * Q (a different scale); it fixes tau = Rao's Q.
hillR::hill_func_parti(t(counts), traits, q = 1)Nested hierarchies
The multi-scale
partitioning above has no equivalent in either package: neither
hillR nor hilldiv2 accepts a nested
hierarchy, so telescoping
gamma = alpha * prod(beta) across region/site levels is
specific to hilldiv3.
| Want to match | Use in hilldiv3
|
|---|---|
hillR::hill_taxa_parti() (neutral) |
hillpart() (exact) |
hillR::hill_phylo_parti() (phylogenetic) |
hillpart(tree = ...) (exact) |
hillR::hill_func_parti() (functional) |
no exact match — it reports FD, not qD;
set tau = Q to align the threshold |
hilldiv2::hillpart() (any type) |
hillpart() (exact) |
In short: when functional partitions diverge, check (1) whether the
other package reports an effective number (qD) or attribute
diversity (FD), and (2) which tau it uses.
hilldiv3 keeps functional diversity in the same
effective-number currency as its neutral and phylogenetic facets, and
makes tau explicit.