Matching, dimensions and designs

Choosing stimuli is a constrained selection problem. One lexical property is manipulated, several others must not vary with it, and the words that satisfy every constraint at once are a small subset of the language. This vignette covers how lexsync states those constraints and how it resolves them: the dimensions it computes, the filters that narrow a lexicon to a candidate pool, the four matching methods and the situations each one suits, the tolerance windows that bound an acceptable match, the continuous alternative to a matched dichotomy, and the report that tells you what control the selection achieved.

The code in this vignette runs on the English example lexicon bundled with the package, a 3000-word slice that needs no download. It is cut from the screened English lexicon. The screen leaves out the forms SUBTLEX-US lacks or writes with a capital in more than 90% of their uses, such as proper names and acronyms. The slice’s N and OLD20 count only the forms it keeps. Taboo words and some abbreviations pass the screen, and Choosing and screening a lexicon below says what such forms do to a selection and how to screen them out. A lexicon of your own is taken as given.

library(lexsync)
schema <- yaml::read_yaml(
  system.file("extdata", "schema.yaml", package = "lexsync")
)
lex <- load_lexicon(
  system.file("extdata", "en_example.csv", package = "lexsync"),
  schema, language = "english"
)
head(
  lex[, c("word", "frequency", "length", "n_syllables", "n_density", "old20")],
  4
)
     word frequency length n_syllables n_density old20
1     aaa      3.77      3           1         7  1.65
2     aba      3.32      3           2         9  1.40
3   aback      2.99      5           2         0  1.95
4 abandon      4.02      7           3         0  2.85

The lexical dimensions

A dimension is any numeric column the matcher can equate. lexsync distinguishes those read straight from the lexicon file from those derived from the orthographic form, and the schema records the distinction along with each dimension’s unit. Units matter when reading a report: a raw difference of 1 means one letter on length but a factor of ten in corpus occurrence on frequency, which is why the comparisons are standardised.

dims <- schema$dimensions
knitr::kable(data.frame(
  dimension = names(dims),
  type = vapply(dims, function(d) d$type, character(1)),
  unit = vapply(dims, function(d) d$unit, character(1)),
  row.names = NULL
), caption = "The dimensions declared in schema.yaml")
The dimensions declared in schema.yaml
dimension type unit
length derived characters
frequency column Zipf
n_density derived neighbours
old20 derived mean Levenshtein distance
n_syllables derived syllables (orthographic estimate)
bigram_freq derived mean bigram probability (type-based, non-positional)

length is the character count of the orthographic form. frequency is the Zipf value read from the lexicon’s freq_zipf column, a log-scaled measure on which roughly 1 is a word occurring once per hundred million tokens and 7 a word occurring ten thousand times per million. The value is the base-10 logarithm of the frequency per billion words (van Heuven et al., 2014). It is preferred to raw counts because frequency effects are approximately linear on its log scale and because it is comparable across corpora of different sizes. n_density is Coltheart’s N, the count of same-length words differing by one letter substitution (Coltheart et al., 1977). old20 is the mean Levenshtein distance to the twenty nearest words in the reference set, which, unlike Coltheart’s N, stays informative for long words, where substitution neighbours run out (Yarkoni et al., 2008). A larger OLD20 means a more isolated word, so it runs in the opposite direction to n_density.

Like length, n_syllables is derived at load time. It counts maximal runs of vowels, an orthographic estimate with no pronunciation model behind it. No lexicon lexsync distributes carries bigram_freq, so it is computed on request, by add_bigram_frequency(). It is the mean corpus probability of a word’s adjacent letter bigrams, a proxy for phonotactic probability.

count_syllables(c("cat", "table", "rhythm", "being", "chocolate"))
[1] 1 2 1 1 4

The estimate errs in both directions. It undercounts being, whose ei is one run of vowel letters spanning two nuclei, and overcounts chocolate, whose silent final e forms a run of its own. It handles rhythm because y is treated as a vowel letter. The measure is adequate for equating two conditions on gross syllabic complexity, but it is not a substitute for a pronunciation dictionary.

Accented vowels count, because the word is decomposed and its diacritics set aside. The vowels of Polish, Czech, Hungarian, Romanian, Turkish and Vietnamese therefore count as those of English do. The estimate is defined for Latin orthographies only: a word holding a character outside the Latin script gets NA. run_pipeline() stops with an error when a design refers to n_syllables and such a word passes the other filters on the lexicon’s columns. The reproducibility vignette gives the exact letters and code points.

Computing dimensions on a pool

The bundled lexicon already carries n_density and old20, but neighbourhood measures depend on the reference set they are computed against, so a lexicon derived elsewhere may need them recomputed. add_neighbourhood() does that. Both measures are quadratic in the size of the reference set, so the pipeline computes them for the pool alone. The reference should still be the whole lexicon, because a word’s neighbours are its neighbours in the language, not among the handful of words that passed your filters.

run_pipeline() computes n_density, old20 and bigram_freq for a lexicon that lacks them whenever the design refers to them, in match_on, a condition’s define_by, pool_filters, a continuous predictor or control, or counterbalance.balance_on. A filter on a derived dimension is applied once the dimension is computed, after the other filters have narrowed the pool.

A lexicon can name the words that make up the reference with a reference_member column (1 for a member, 0 otherwise). The reference_words argument recomputes N and OLD20 against a list of your own, together with bigram_freq where the lexicon carries it or the design uses it. The recomputed values replace the lexicon’s columns, and the datasheet records the list with the columns it was applied to.

sample_words <- lex[lex$word %in% c("apple", "boat", "dog", "rhythm"), ]
add_neighbourhood(
  sample_words, reference = lex$word
)[, c("word", "n_density", "old20")]
       word n_density old20
123   apple         1  2.80
290    boat         2  1.85
782     dog         2  1.90
2211 rhythm         0  4.00

add_bigram_frequency() works the same way. merge_norms() joins any word-keyed norm table. Dimensions the corpus does not carry, such as concreteness, age of acquisition or behavioural measures from a megastudy, become matchable this way. Once a column is present, the matcher treats it like any other.

aoa <- data.frame(word = c("apple", "boat", "dog"), aoa = c(3.1, 3.0, 3.4))
merged <- merge_norms(lex, aoa)
merged[merged$word %in% aoa$word, c("word", "frequency", "aoa")]
     word frequency aoa
123 apple      4.76 3.1
290  boat      4.71 3.0
782   dog      5.10 3.4

The join is case- and whitespace-insensitive on both sides, keeps the lexicon’s own row and column order, and stops with an error on a norm column whose name already exists on the lexicon. A silent rename would leave the dimension the design matches on under a name the design does not refer to.

In a design, you do not call this yourself. Name the tables in a norms: block and the pipeline joins them before the pool is built, so a norm column can be filtered on, matched on or spanned like any other dimension.

norms:
  - path: norms/en_concreteness.csv
    join_on: word             # optional join key, default 'word'
    columns: [concreteness]   # optional; default every column but the key

pool_filters:
  concreteness: [2.0, 5.0]
match_on: [length, concreteness]

The join key is written join_on, because YAML reads a bare on as the boolean true. A bare on: is read as the join key all the same, with a note in the run log, and any other key in an entry is an error.

lexsync distributes no norm data, because licences vary and the citation is yours to honour. Every table you join appears in the materials datasheet with its checksum and its per-column coverage. The table is recorded because it can supply the variable a design manipulates, and a selection over columns of unstated origin cannot be reproduced from the datasheet. Coverage is recorded because a word the table does not cover gets a missing value. A filter on the column drops it, and the matcher excludes it before selecting whenever the column is a matched dimension, a continuous predictor or a control. The run log and the datasheet count those exclusions per condition.

Pool filters

A pool filter narrows the lexicon before any matching happens. It states what counts as an eligible word, independently of condition, and it is the right place for constraints you would apply to every item in the study.

pool_filters <- list(length = c(3, 7), frequency = c(3.8, 7.0))
pool <- build_pool(lex, pool_filters)
c(lexicon = nrow(lex), pool = nrow(pool))
lexicon    pool 
   3000     859 

A numeric pair is read as an inclusive range. Any other vector is read as a set of permitted values, so list(pos = c("noun", "verb")) would keep nouns and verbs if the lexicon carried a pos column. Rows missing a value on a filtered column are always dropped.

Called directly, build_pool() skips a filter naming a column the frame does not have, because it filters lexica, supplied pools and pair tables, which carry different columns. A misspelt key therefore silently widens the pool. In a design, the filters are checked first and a misspelt key stops the run. run_pipeline() stops with an error on a pool_filters key that names neither a column of the lexicon, counting any columns its norms: tables add, nor a dimension the pipeline derives (n_density, old20 or bigram_freq). match_stimuli() does the same on a define_by key the pool lacks.

Filtering is a separate step from matching. match_stimuli() reads a design’s conditions and match_on fields but never its pool_filters, so passing a raw lexicon where a pool was intended silently widens the selection. The pipeline calls build_pool() first and the examples here do the same.

Choosing and screening a lexicon

The lexica in the repository’s corpora/derived/ hold the most frequent forms wordfreq gives for a language, and wordfreq lists names, acronyms and internet forms beside words. Such forms enter a selection unevenly across conditions: when the en_andrews_repro demonstration read en.csv, they made up 24 of its 36 low-frequency items and none of its 36 high-frequency ones. N and OLD20 count them as neighbours too, so cat has 32 neighbours in en.csv, among them caa, cbt and cdt.

corpora/derived/en_v2.csv screens the English lexicon against SUBTLEX-US (Brysbaert & New, 2009). It holds the 30,000 forms and frequencies of en.csv and adds four columns. in_subtlex_us is 1 when SUBTLEX-US lists the form, and lowercase_share is the share of its SUBTLEX-US uses that begin with a lower-case letter, empty for a form SUBTLEX-US lacks. nonstandard is 1 for a form SUBTLEX-US lacks or whose share is below 0.10, which marks 6,565 forms, mostly names and acronyms, and reference_member is 1 for the other 23,435. The file’s n_density and old20 count neighbours among those only, so cat has 21, and n_density_raw and old20_raw keep the values of en.csv. To select among the unmarked forms, filter on the flag.

lexicon: corpora/derived/en_v2.csv
pool_filters:
  nonstandard: [0, 0]

The threshold is low because a capital at the start of a sentence lowers the lower-case share of ordinary words too. Even what, how, why, maybe, okay and yes begin with a lower-case letter in under half their uses, and a filter such as lowercase_share: [0.5, 1] would drop them all. At 0.10, the flag still marks the days of the week, the months other than march and may, most nationality words, god, ok and tv. It also marks every form the 2009 corpus lacks, such as selfie, emoji and covid. A design that needs some of those can leave the flag unfiltered and remove what it does not want by name. A range filter on lowercase_share would also drop every form SUBTLEX-US lacks, whose share is empty.

exclude_words removes words by name. It lists them in the design, or names a UTF-8 file with one word per line, in which blank lines and lines beginning # are skipped.

exclude_words: [adams, dubai, espn]
# or, from a file: exclude_words: {path: config/exclude_en.txt}

The words are trimmed, lower-cased and put in NFC, as a lexicon’s words are. They are removed from the candidates of a corpus, pool or generated design after any norms: join and before the pool filters. The neighbourhood reference still counts them, and a generated pseudoword may not equal one, because an excluded word remains a word of the language. In a design with an item table, exclude_words is an error, since its items are already a selection of your own. The datasheet records the list as materials_source$exclusions, with its checksum, the number of words it lists and the number of candidates it removed, and the Methods paragraph reports the removal.

The English demonstrations read en_v2.csv with the filter above. Their exclude_words file, config/exclude_en_demo.txt, leaves out the taboo words and the forms that pass the screen without being standard English words, such as abbreviations that subtitles write in lower case (mph, msg). Its header states the criteria, and the list was made by reading what the demonstrations selected. Spanish and Chinese have no screened lexicon, because no source of letter case or known words with a verified licence exists for them. Read a selection in either language before a study runs, and leave out what you find with exclude_words.

en.csv is unchanged byte for byte, so a design written for lexsync 0.1 selects what it selected once it sets matching.tie_break: bytes (see Ties between equally close candidates below). The exceptions are a design that uses the optimal method, whose selections differ from those of lexsync 0.1 under either rule, and a design whose pool holds candidates that lack a matched value, which are now left out.

The datasheet credits the lexicon a corpus or generated design read. A lexicon lexsync distributes is credited to wordfreq. en_v2.csv and the English example slice drawn from it are credited to wordfreq screened against SUBTLEX-US, with the screening rule. A corpus that fetch_corpus() put in the cache is credited by the registry’s citation, and any other file by its name and the first twelve digits of its checksum.

Supplied item pools

Narrowing a whole lexicon suits a study in which any word of the language will do, but not one in which the candidate set is itself a research decision: a list from a previous study, from a norming session, or one restricted to a semantic category that no lexical filter can express. Loaded as a lexicon, such a list would need an invented freq_zipf column. Given as items.source: table, it would skip matching. items.source: pool takes the list as it is.

items:
  source: pool
  path: items/pool_en_concrete_nouns.csv
  lexicon: corpora/derived/en_v2.csv

The list needs only a word column. Length and the syllable estimate are derived from the form, and items.lexicon supplies the rest, frequency above all, by lookup. A word the lexicon does not have raises an error, because a word missing a value would be left out of the candidates and the pool would then be smaller than it appears to be.

n_density and old20 are properties of a word in its language, so they are computed against the lexicon’s words. Computed against the 131 words of a supplied list, they would invalidate the controls without warning. Without a lexicon, the supplied pool is its own reference, which is why the example above gives one. Called directly outside a design, load_pool() returns both the pool and the reference word list.

The conditions, match_on, the report and the datasheet work as they do for a pool narrowed from a lexicon. Only the source of the candidates differs, and the datasheet records the list’s checksum. design_en_supplied_pool.yaml is a worked example.

Relational designs, where the item is a pair

A priming study’s theoretical variable is a property of the pair, such as distributional similarity or associative strength, while its nuisance variables are properties of each member: frequency, length, neighbourhood. A relational design therefore needs matching and its own item table at once.

items.members promotes an ordinary item table to a pair-keyed one.

items:
  source: table
  path: items/priming_pairs_en.csv
  members: [prime, target]             # the two word fields
  lexicon: corpora/derived/en_v2.csv   # where each member's norms are looked up
  anchor_condition: related            # whose row represents the pair during selection

Each member’s word-level norms are joined from the lexicon and prefixed, giving prime.frequency, target.length and so on. The member name comes first because of partial matching in R. prime.frequency is safe, since df$prime still exact-matches the bare prime column, whereas frequency.prime would risk a partial match. Those prefixed names are usable wherever a dimension is: in pool_filters, in match_on, as a continuous predictor or as a control.

pair.lev is the Levenshtein distance between the two members and pair.overlap is 1 - pair.lev / length of the longer form, a normalised orthographic similarity. It scores edit operations, so it is not a count of shared material: abc and cba share every letter but score 0.33. Orthographic overlap is the standard confound control in a priming design, since a related pair that also shares letters confounds semantic relatedness with orthographic similarity. The pipeline adds both automatically when the two members are named prime and target. Under any other member names, call add_pair_overlap() on the item table yourself. A predictor that cannot be computed from the forms alone, such as cosine similarity or forward associative strength, is supplied as an extra column on the item table and is used like any other.

n_per_condition: 8                  # counts PAIRS: 8 pairs, 16 rows, 8 per list
pool_filters:
  target.length: [3, 7]
  target.frequency: [3.0, 6.0]
continuous:
  predictor: target.frequency
  controls: [target.length, pair.overlap]
match_on: [target.length, pair.overlap]

Two rules follow from the item being a pair. A filter applies to a pair only if every one of its rows passes. Selection runs over one row per pair, the row of the anchor_condition, which defaults to the byte-first condition. The result is then re-expanded to every condition row of every chosen pair. Both rules keep each pair whole. A filter on prime.frequency applied row by row would keep a pair’s related row and drop its unrelated one, leaving a set the Latin-square counterbalancer cannot complete. Only a dimension that varies within the pair can split it that way. Prime-level and pair-level dimensions do, and target-level ones do not, since the target is the same word in both rows.

A member name may not be one of word, id, set, list, trial, condition, replicate or item, which the engines read directly, and a design using one is rejected with an error. The datasheet gains a relational block recording the members, the pair count, the member lexicon and its checksum, and the member-level dimensions separately from the relational ones. design_en_priming_continuous.yaml is the worked example, and its own comments go further into why each rule is there.

Conditions, the anchor and matched sets

A condition is a named region of the pool, given by define_by, which is read exactly as pool_filters is. The first condition listed is the anchor. Its items are chosen by an even spread across the sorted candidate subpool, so the anchor covers the whole of its region on the manipulated dimension. Every later condition is then matched to the anchor item by item, and the set column records the pairing. The matching needs at least one dimension in match_on. Without one, every candidate lies at distance zero from every anchor and the tie-break alone would choose the words, so a design with conditions and no match_on, or an empty one, is rejected with an error before anything is selected.

design <- list(
  name = "vignette_matching", language = "english", n_per_condition = 20,
  pool_filters = pool_filters,
  conditions = list(
    list(name = "high", define_by = list(frequency = c(5.2, 7.0))),
    list(name = "low",  define_by = list(frequency = c(3.8, 4.4)))
  ),
  match_on = list("length", "n_density", "old20")
)
stim <- match_stimuli(pool, design, schema)
head(
  stim[
    stim$set %in% 1:3,
    c("set", "condition", "word", "frequency", "length", "old20")
  ],
  6
)
   set condition    word frequency length old20
1    1      high perfect      5.20      7  2.55
2    2      high   issue      5.23      5  2.35
3    3      high    gets      5.25      4  1.40
21   1       low  herald      3.95      6  2.45
22   2       low   elbow      3.84      5  2.30
23   3       low    punk      4.20      4  1.40

Because the anchor is spread across its own subpool, the order of the conditions is a design decision with consequences. The anchor’s distribution on the manipulated dimension is inherited from the pool, and every other condition’s distribution is whatever matching to that anchor produces.

The four matching methods

The method is set with matching.method in a design, or left to the schema default. All four minimise a distance in standardised space, computed by z-scoring each match_on dimension against the whole pool, so that a letter and a Zipf unit become comparable. A candidate missing any match_on dimension has no place in that space, so every method starts from the rows complete on all of them. The run log and the datasheet count what each condition lost. The methods differ in what they minimise the distance over.

standardised_euclidean, the default, takes the anchor items in order and gives each one its nearest unused counterpart, greedily. Because the anchor is selected before any matching happens, it preserves the anchor’s even spread exactly. Like mahalanobis, it is anchored and so handles more than two conditions.

joint uses no anchor. It scores every cross-condition pair, then takes the best-matched pairs in ascending order of distance until it has enough, so an item with no good counterpart is never selected. This equates the controls more tightly than per-anchor matching when the manipulation is confounded with a control, but the design then no longer determines the manipulated dimension’s realised distribution. It requires exactly two conditions.

mahalanobis replaces the Euclidean metric with one weighted by the inverse of the pool’s correlation matrix, so that dimensions sharing variance are not counted twice. It suits the case where the matched dimensions are strongly intercorrelated, which is the normal case for orthographic measures. The correlations come from the pool’s rows that are complete on every matched dimension, and with k matched dimensions the pool needs at least k + 2 such rows.

round(cor(pool[, c("length", "n_density", "old20")]), 2)
          length n_density old20
length      1.00     -0.69  0.77
n_density  -0.69      1.00 -0.78
old20       0.77     -0.78  1.00

Length, neighbourhood density and OLD20 are strongly correlated in this pool. Under a plain Euclidean metric, matching on all three counts the shared length-neighbourhood variance three times over and matches an independent dimension less closely. Mahalanobis distance discounts that overlap (Rubin, 1980; Stuart, 2010).

optimal keeps the Euclidean metric and selects the n disjoint pairs whose summed distance is smallest. Choosing n pairs from two pools is the k-cardinality assignment problem, which lexsync solves exactly by padding the cost matrix with dummy rows and columns until every optimal assignment uses exactly n real pairs (Volgenant, 2004). Greedy matching can use up the one good counterpart of a hard item on an easy item that had alternatives. Optimal matching cannot, although on the pools tested joint came within a few per cent of the optimum. Gu and Rosenbaum (1993) compared optimal and greedy pair matching of every treated unit, a related problem, and found optimal matching no better at balancing the covariates.

When the conditions overlap, a word cannot be paired with itself or used twice. A pair lost that way is replaced by solving again over the unused words, and the selection is then no longer guaranteed to have the least summed distance. optimal requires exactly two conditions and the clue package.

methods <- c("standardised_euclidean", "joint", "mahalanobis")
if (requireNamespace("clue", quietly = TRUE)) methods <- c(methods, "optimal")

summarise_method <- function(method) {
  d <- design
  d$matching <- list(method = method)
  s <- match_stimuli(pool, d, schema)
  cmp <- match_report(
    s, c("frequency", "length", "n_density", "old20"), schema
  )$comparisons
  controls <- cmp[cmp$dimension != "frequency", ]
  high <- s$frequency[s$condition == "high"]
  low  <- s$frequency[s$condition == "low"]
  data.frame(
    method = method,
    worst_control_d = round(max(abs(controls$cohens_d)), 3),
    freq_d = round(cmp$cohens_d[cmp$dimension == "frequency"], 2),
    freq_raw_diff = round(mean(high) - mean(low), 2),
    freq_sd_high = round(sd(high), 2)
  )
}
do.call(rbind, lapply(methods, summarise_method))
                  method worst_control_d freq_d freq_raw_diff freq_sd_high
1 standardised_euclidean            0.05   4.58          1.60         0.46
2                  joint            0.00   9.11          1.30         0.11
3            mahalanobis            0.05   4.56          1.59         0.46
4                optimal            0.00   4.42          1.37         0.40

The comparison shows why a standardised difference should not be read as an effect size on its own. The two pairing methods reduce the control differences to zero, because a free choice over all pairs can find counterparts that match exactly on all three dimensions. The pool holds far more exact matches than the design needs, so each method’s rule for choosing among them decides the rest. joint takes them in tie-break order, and those pairs cluster in frequency. Its standardised frequency separation is the largest of the four, yet its raw separation in Zipf units is the smallest. Both follow from the clustering, which shrinks the SD of the high condition and with it the pooled SD that freq_d divides by. A narrower manipulation can therefore raise d. optimal finds equally exact pairs, but its assignment solver chooses among them by its own internal order, and here the high condition keeps more of its spread.

Whether that is a gain depends on the study. The anchored methods keep the high condition spread across its whole band by construction. That supports a claim about the band rather than about a cluster inside it, and the spread is variance you may want if the manipulation is later modelled as continuous. The pairing methods guarantee exact control of the nuisance dimensions where the pool allows it, but how much of the band they keep depends on which of the exact matches they take. These numbers describe this pool and this design. The example shows how to compare the methods, not which one is best.

The parity caveat on two methods

The R and Python engines select byte-identical stimuli under standardised_euclidean and joint. They do not guarantee it under mahalanobis or optimal. mahalanobis weights the distances by the inverse of the pool’s correlation matrix, and the last bits of that inverse depend on the platform’s linear-algebra implementation. Once two candidate distances differ in their final bits, the tie-break that would otherwise make the engines agree is never reached. optimal passes the distances to an assignment solver, and where several assignments have the same summed distance, clue::solve_LSAP() and SciPy’s solver can return different ones. mahalanobis usually still agrees exactly, and optimal often selects an equally optimal but different set. Each run’s datasheet records which case applies, under selection.cross_engine. If your project requires cross-engine reproducibility, use standardised_euclidean or joint. The reproducibility vignette explains why the exception remains.

Ties between equally close candidates

Lengths and neighbourhood counts are whole numbers and frequencies carry two decimals, so candidates often lie at exactly the same distance from an anchor. When en_freqcontrast read the unscreened English lexicon, en.csv, 71 of its 80 anchored picks had another candidate at the chosen distance, and the tie-break chose the word. matching.tie_break sets the rule, in a design or in the schema. Under bytes, the rule of lexsync 0.1, a tie goes to the word whose UTF-8 bytes sort first, and then to the lower id, so a condition decided by ties fills with early-alphabet words. Under that rule, 61% of the design’s low-frequency words began with a or b, against 13% of their candidates. A word’s initial phoneme biases voice-key measures of naming latency (Kessler et al., 2002), and the realised-control report cannot show such a skew, because it measures only the matched dimensions.

Under hash, the default, a tie goes to the smaller SHA-256 digest of lexsync-tie-v1| followed by the word. The salt is fixed, so the rule does not depend on the seed, and the digests favour no part of the alphabet. The same rule orders the tied pairs of joint and optimal and the rows tied at their candidate cap.

initial_share <- function(words) mean(substr(words, 1L, 1L) %in% c("a", "b"))
by_rule <- lapply(c(bytes = "bytes", hash = "hash"), function(rule) {
  d <- design
  d$matching <- list(tie_break = rule)
  match_stimuli(pool, d, schema)
})
low_candidates <- build_pool(pool, design$conditions[[2]]$define_by)
round(c(vapply(by_rule, function(s) initial_share(s$word[s$condition == "low"]),
               numeric(1)),
        candidates = initial_share(low_candidates$word)), 2)
     bytes       hash candidates 
      0.20       0.00       0.15 

On this small pool, the share says little. With 20 items, it moves by chance under either rule, and here it falls below the candidates’ share under hash. The skew is clear at the size of a whole lexicon, as the en_freqcontrast figures above show. Only the picks that ties decided can change, and here the matched differences stay the same.

sapply(by_rule, function(s) {
  cmp <- match_report(s, c("length", "n_density", "old20"), schema)$comparisons
  stats::setNames(round(cmp$cohens_d, 3), cmp$dimension)
})
          bytes  hash
length    0.050 0.050
n_density 0.027 0.027
old20     0.038 0.038

Under either rule, standardised_euclidean and joint give the same selection in the two engines, and bytes reproduces the selections of lexsync 0.1 except under optimal or where the pool holds candidates that lack a matched value. The even spreads that pick the anchor’s items and a continuous design’s items keep the byte order under both rules.

Tolerance windows

Matching by distance alone will always return the nearest available candidate, however far away it is. The tolerance window limits how far away that candidate can be. Before the distance ranking, each later condition’s candidates are filtered to those lying within the anchor’s mean plus or minus k standard deviations on every matched dimension, with k set per dimension.

str(schema$matching$tolerance_k)
List of 4
 $ length   : num 2
 $ frequency: num 1
 $ n_density: num 2
 $ old20    : num 2

Smaller k is stricter. The default of 2.0 on the orthographic dimensions is permissive, admitting most of a normal distribution, and it is meant only to exclude an extreme counterpart. The control comes from the distance ranking within the window.

A design tightens the window per dimension, and only the named dimensions are overridden. A dimension the schema does not list takes k = 2. The design’s values are merged over the schema’s by modifyList(). A null in the schema is ignored, and a null in the design removes the schema’s value for that dimension, so either leaves k at 2.

Tolerance windows apply to the anchored methods, standardised_euclidean and mahalanobis, which derive them from the anchor, and to continuous designs, which derive them from an even spread over the pool’s ranks on the predictor. joint and optimal never build an anchor and never read tolerance_k, so tightening a window leaves their selection unchanged. Both instead cap each subpool to the 1200 candidates nearest the other condition’s centroid before pairing, which bounds the cost of scoring every pair.

tight <- design
tight$matching <- list(tolerance_k = list(old20 = 0.25))
tight_stim <- match_stimuli(pool, tight, schema, verbose = TRUE)
round(match_report(tight_stim, "old20", schema)$comparisons$cohens_d, 3)
[1] -0.104

If a window leaves a condition with fewer candidates than there are anchor items, the matcher neither fails nor returns a short set. It relaxes that condition’s window and carries on. Tightening the window past what the pool can supply therefore looks exactly like a window that was never set. Under the default policy, only verbose = TRUE and the result’s "audit" attribute, which the pipeline writes to the run log and the datasheet, tell you which happened.

too_tight <- design
too_tight$matching <- list(tolerance_k = list(old20 = 0.05))
relaxed_stim <- match_stimuli(pool, too_tight, schema, verbose = TRUE)
lexsync: condition 'low' has 10 candidates within tolerance (< 20 needed); relaxing the window.

The relaxation is per condition and all-or-nothing. It drops that condition’s windows on every matched dimension at once, not only the one that proved unsatisfiable, so an over-tightened old20 also discards the length and n_density windows for that condition. The match degrades less than this might suggest, because the distance ranking still runs over the whole subpool and still prefers near counterparts. If the relaxed subpool is still too small, the matcher stops with an error naming the condition and the counts.

The schema’s matching block sets two policies. on_insufficient_tolerance governs the relaxation. Its default of relax keeps the behaviour above. Under error, the matcher stops with an error, which suits a design whose window is itself the criterion, as it is when reproducing a published study’s stated windows. shortfall governs what happens when fewer sets than n_per_condition can be selected, and it defaults to error because a smaller set changes the design that was planned. A design that accepts fewer sets says so with shortfall: allow. The datasheet then records the items each condition holds, and the Methods paragraph states them beside the request. Under the anchored methods, the policy applies to the anchor, and a later condition with fewer candidates than the anchor has items stops the run under either setting. Either key may be overridden under the design’s own matching: block, as tolerance_k was overridden above.

The repository’s config/design_es_gender_repro.yaml reproduces the windows of a published study. It sets tolerance_k to 0.111... for frequency, the window of the mean plus or minus SD/9 that González Alonso et al. (2025) used, and to 2 for length, the mean plus or minus 2 SD.

Continuous designs

A matched dichotomy is one way to study a graded property, and for a property like frequency it is often not the best one. Splitting a continuum into a high and a low band discards the variation within each band, costs statistical power and leaves the estimate dependent on where the split was placed. The selection can also create its own artefacts. Liben-Nowell et al. (2019) show that choosing a controlled subset from a large candidate set can produce an apparent effect from a set where none exists, because the selection is itself a search over the space of possible item sets. Kuperman (2015) argues that a megastudy’s variation can be used directly by sampling across a predictor’s range and modelling it.

A continuous block does that. Rather than conditions, it names one predictor to span and at least one control. match_on must equal continuous.controls, and the predictor must not appear among the controls.

cont_design <- list(
  name = "vignette_continuous", language = "english", n_per_condition = 60,
  pool_filters = pool_filters,
  continuous = list(
    predictor = "frequency",
    controls = c("length", "n_density", "old20")
  ),
  match_on = list("length", "n_density", "old20"),
  matching = list(
    tolerance_k = list(length = 1.5, n_density = 1.5, old20 = 1.5)
  )
)
cont <- select_continuous_stimuli(pool, cont_design, schema)
nrow(cont)
[1] 60
range(cont$frequency)
[1] 3.80 6.85

Selection runs in two passes. An even spread over the pool’s ranks on the predictor defines the control windows from its own distribution. The pool is filtered to those windows, and a second even spread over the ranks of the filtered pool is the selection. Because the spread is over ranks and not values, the selection follows the pool’s distribution of the predictor. In the en_freqcontinuous demonstration, its quantiles lie within 0.06 of the pool’s, and 95% of the items fall in the lower half of the frequency range, since frequent words are rare in any lexicon. The windows trim the controls’ tails without holding the controls constant, so how far each control is still correlated with the predictor has to be read from the report.

cont_report <- match_report_continuous(
  cont, "frequency",
  c("length", "n_density", "old20"), schema
)
cont_report$comparisons
  dimension      role pearson_r predictor_span    sd sd_ratio window_low
1 frequency predictor        NA           3.05 0.586    0.649         NA
2    length   control    -0.267           3.05 0.748    0.586      3.468
3 n_density   control     0.047           3.05 2.797    0.551     -2.795
4     old20   control    -0.138           3.05 0.263    0.512      1.209
  window_high
1          NA
2       6.965
3      10.262
4       2.491

The report is read differently from a matched one. predictor_span is the realised range of the predictor in the selected set and should be wide, since it is the variation the regression has to work with. Each control’s pearson_r is its correlation with the predictor and should be near zero, since a control that still correlates with the predictor is still a confound. A control that stays correlated belongs in the model as a covariate.

Tightening tolerance_k does not reliably lower those correlations. In en_freqcontinuous, k = 1.0 gives a correlation of 0.070 between frequency and n_density, against 0.301 at k = 1.5, but on the unscreened en.csv it gave 0.213 against -0.049. At k = 0.25, only 30 items fall within the windows, fewer than the 80 the design needs, so the selector relaxes them to the whole pool.

Each row also gives the dimension’s realised SD as sd. For the predictor, sd_ratio sets it against the SD of as many values spaced evenly over the same range. That SD is (max - min) * sqrt(n * (n + 1) / 12) / (n - 1), which approaches (max - min) / sqrt(12) as n grows, so an even spread has a ratio of 1 at any size. A ratio below 1 means the items bunch together more than an even spread would, and a ratio above 1 means they gather towards the ends of the range. en_freqcontinuous has a ratio of 0.59, an SD of 0.53 against 0.90.

For a control, sd_ratio sets the realised SD against the SD of the pool the selector drew from, and window_low and window_high give the window that the first pass set. In en_freqcontinuous, length keeps 0.76 of the pool’s SD, so the window trims the tails of a control and leaves much of its spread. The windows come from the "audit" attribute select_continuous_stimuli() attaches to its result, which match_report_continuous() reads unless it is given one as audit. A window that held too few items and was widened to the whole pool is recorded in the datasheet’s window_relaxations.

The datasheet for a continuous design reports the span, the correlations and a suggested regression model in which the controls enter as covariates. Its Methods paragraph sets the predictor’s SD against that of an even spread, gives each control’s window and its realised SD against the pool’s, and says when the windows were widened.

Reading the realised control

match_report() returns two data frames. The descriptives give per-condition n, mean, SD, minimum, median and maximum for each dimension, which is the table that belongs in a paper’s materials section.

report <- match_report(
  stim, c("frequency", "length", "n_density", "old20"), schema
)
knitr::kable(
  report$descriptives,
  caption = "Descriptive statistics per condition"
)
Descriptive statistics per condition
group dimension n mean sd min median max
high frequency 20 5.626 0.463 5.20 5.425 6.85
high length 20 4.600 1.046 3.00 4.000 7.00
high n_density 20 5.900 5.739 0.00 2.500 20.00
high old20 20 1.670 0.403 1.00 1.775 2.55
low frequency 20 4.028 0.170 3.82 3.960 4.40
low length 20 4.550 0.945 3.00 4.000 6.00
low n_density 20 5.750 5.379 0.00 2.500 17.00
low old20 20 1.655 0.386 1.00 1.775 2.45

The comparisons contrast every pair of conditions on every dimension. The rows against the anchor come first, and every further pair follows in condition order, with the earlier condition of the pair named in reference. A 2 x 2 design such as en_andrews_repro therefore shows the contrasts within each factor, such as its two low-frequency cells, beside the three against its first cell. The rows carry the statistics for judging whether the matching worked.

knitr::kable(
  report$comparisons,
  caption = paste(
    "Realised control: standardised differences with 90% intervals,",
    "variance ratios, TOST verdicts and raw differences"
  )
)
Realised control: standardised differences with 90% intervals, variance ratios, TOST verdicts and raw differences
condition reference dimension cohens_d d_ci_low d_ci_high var_ratio tost_p equivalent mean_diff role
low high frequency 4.584 3.569 5.599 0.135 1.0000 FALSE 1.598 NA
low high length 0.050 -0.483 0.583 0.815 0.0815 FALSE 0.050 NA
low high n_density 0.027 -0.506 0.560 0.878 0.0715 FALSE 0.150 NA
low high old20 0.038 -0.495 0.571 0.918 0.0761 FALSE 0.015 NA

cohens_d is the standardised mean difference, using the pooled SD. It is the mean of the condition named in reference minus the mean of the condition in that row, so a positive d means the reference scores higher. It should be large on the manipulated dimension and near zero on a controlled one. A condition with a single item has no spread, so d and its interval are then NA.

d_ci_low and d_ci_high bound it with a 90% confidence interval, built from the large-sample standard error of d (Hedges & Olkin, 1985). That standard error grows with d itself, so the interval on a manipulated dimension is much wider than on a well-matched control with the same number of items. The interval, not the point estimate, is the primary summary of a controlled dimension. Its upper limit is the largest imbalance still consistent with the stimuli you have, and with few items that limit can be substantial even when the point estimate is almost zero. Reporting the interval keeps the dependence on set size visible, where a bare ‘no significant difference’ hides it.

var_ratio is the condition’s variance over the reference’s. Two conditions can share a mean and still confound if one is far more spread out than the other, a difference that no mean-based statistic shows. A ratio near 1 is balanced and a common heuristic flags ratios outside 0.5 to 2 (Rubin, 2001, as cited in Stuart, 2010).

mean_diff is the same contrast in the dimension’s own unit: the reference mean minus the condition mean, in letters, Zipf units, neighbours or syllables, to three places. It is defined whenever each condition has an item, so it remains when d is NA.

role says what the design did with the dimension in that pair. A dimension is manipulated when the two conditions’ define_by windows on it differ, controlled when the design matches on it, and uncontrolled otherwise. The role belongs to the pair: in en_andrews_repro, frequency is manipulated between a high- and a low-frequency cell and uncontrolled between the two high-frequency cells, which share the window [5, 7]. run_pipeline() derives the roles from the design and passes them to match_report() as roles, a function of the row’s condition, reference and dimension. Called without it, as above, match_report() leaves role as NA.

The pipeline reports every dimension lexsync has for the items, whether or not the design matches on it. Length, frequency, N, OLD20, syllables and bigram frequency come first, then the matched dimensions, every define_by dimension and every joined norm column, so a norm that defines the conditions appears as the manipulated dimension it is. When the stimuli lack bigram frequency, it is computed for the report alone, and the stimuli file and the selection stay as they were. A dimension with no numeric value among the items, such as n_syllables for Chinese words, is left out. The uncontrolled rows are where a confound shows. In en_ndensity, the dense words have fewer estimated syllables than the sparse ones (d = -0.53, 90% CI [-0.79, -0.26]), which a report of the matched dimensions alone could not reveal.

tost_p and equivalent come from the two one-sided tests (TOST). A non-significant result from a conventional difference test is not evidence of equivalence, because failing to reject a null is not the same as supporting it, and the failure is easiest to achieve with few items and noisy measures. TOST inverts this by testing against an explicit bound. The reported p is the larger of two one-sided p-values against a difference of plus or minus bound_d pooled SDs, and a value below alpha supports equivalence within that bound (Lakens, 2017). The bound and alpha are the schema’s, so the verdict is only as meaningful as the bound you chose.

str(schema$equivalence)
List of 2
 $ bound_d: num 0.5
 $ alpha  : num 0.05

The schema’s default bound of 0.5 SD is a convention. Lakens (2017) does not propose it as a default and asks for the bound to be justified for the study at hand. Half a standard deviation is a permissive bound for a nuisance dimension in a design where that dimension has a strong effect on the response. If a particular confound matters to your study, set a bound you can defend. The verdict also depends on the number of items: at a bound of 0.5 and an alpha of .05, no dimension that varies can pass with fewer than 23 items per condition, even when the conditions are identical.

The 90% interval is the 1 - 2 x alpha interval, which corresponds to the TOST decision at alpha. The test declares equivalence when the interval falls inside the bounds, so the interval and the verdict are two readings of nearly the same thing. They differ slightly, because TOST tests the raw mean difference while the interval also allows for the sampling error of the pooled SD. For a control near zero, the two agree to well within the published rounding. For a limit close to the bound, tost_p is the verdict.

The interval and the two one-sided tests treat the selected items as a sample from a larger population, which they are not. Sassenhagen and Alday (2016) point out that a test on stimuli concerns a closed population, the items you chose. They recommend describing the stimuli, which the descriptives do, and entering potential confounds in the analysis as covariates.

The datasheet’s Methods paragraph follows that advice. It describes the realised control first: the largest difference on a matched dimension, in that dimension’s unit, with d and its interval. Raw differences in different units cannot be ranked against each other, so the largest is the one with the largest |d|. The interval is labelled by the schema’s alpha, 90% at .05 and 99% at .005. The two one-sided tests come second, as a check, with the number of matched comparisons they supported. When every condition holds fewer items than the bound needs, the paragraph adds that no dimension that varies can pass, however well matched. Last, it names, largest first, up to five comparisons of a dimension the design did not control whose |d| exceeds the bound, such as the syllable difference in en_ndensity, or says that none does. The suggested model enters the matched dimensions as centred covariates.

The individual statistics are exported and can be called directly on any two vectors.

high <- stim$old20[stim$condition == "high"]
low  <- stim$old20[stim$condition == "low"]
str(cohens_d_ci(high, low))
List of 3
 $ d      : num 0.038
 $ ci_low : num -0.495
 $ ci_high: num 0.571
str(tost_equiv(high, low, bound_d = 0.5))
List of 2
 $ p         : num 0.0761
 $ equivalent: logi FALSE
round(variance_ratio(low, high), 3)
[1] 0.918

balance_check() answers a different question, whether the levels of a column occur equally often. It returns human-readable warnings and an empty vector when all is well, and the pipeline logs whatever it returns.

balance_check(stim, "condition")
character(0)
balance_check(rbind(stim, stim[1, ]), "condition")
[1] "Column 'condition' is unbalanced: high=21, low=20"

Pseudowords

Lexical decision needs non-words, and the difficulty is that they must be orthographically plausible without being real. build_lexdec_stimuli() draws real words by an even spread across the byte-ordered pool, then generates a length-matched pseudoword for each.

lexdec_pool <- build_pool(lex, list(length = c(4, 7), frequency = c(3.5, 6.0)))
stim_lexdec <- build_lexdec_stimuli(
  lexdec_pool, n = 8, reference_words = lex$word
)
head(stim_lexdec[, c("target", "condition", "length", "set")], 4)
   target condition length set
1 abandon      word      7   1
2  cannot      word      6   2
3 efforts      word      7   3
4  herald      word      6   4

The presented string is target, the conditions are word and pseudoword, and set pairs each word with its non-word twin. reference_words should be the whole lexicon: it supplies the bigram statistics and the list of real forms a pseudoword must avoid, and it defaults to the pool when omitted, which is rarely what you want.

Two methods are available, chosen per design under items.generation.method.

letter_substitution, the default, changes as few letters as possible subject to three constraints: every resulting bigram must be attested in the corpus, the form must not be in the reference list or already generated, and the length must be preserved exactly. Single substitutions are searched first, then pairs as a fallback. Among the legal candidates, the most bigram-typical wins, with a byte-order tie-break.

subsyllabic splits each word into onset, nucleus and coda constituents and swaps whole constituents for attested alternatives of the same role and length, so the pseudowords keep their syllabic structure. Codas and nuclei are changed before onsets, which carry the most identifying orthography. Up to ceil(2k/3) of a word’s k constituents are replaced, each swap preserves length, and a word with no legal swap falls back to letter substitution, so every word yields a pseudoword. It is a deterministic orthographic approximation of Wuggy (Keuleers & Brysbaert, 2010), and it differs from Wuggy in more than the phonological model. Wuggy matches the template’s transition frequencies, matches letter length by default and has orthographic modules for several languages. By default, it also keeps two thirds of the segments, where lexsync replaces up to two thirds. In return, lexsync gains cross-engine reproducibility.

sub_stim <- build_lexdec_stimuli(
  lexdec_pool, n = 8, reference_words = lex$word, method = "subsyllabic"
)
# The real words come first in the frame, so pair each with its twin by `set`.
merge(
  sub_stim[sub_stim$condition == "word", c("set", "target")],
  sub_stim[sub_stim$condition == "pseudoword", c("set", "target")],
  by = "set", suffixes = c("_word", "_pseudoword")
)[1:4, ]
  set target_word target_pseudoword
1   1     abandon           oberder
2   2      cannot            cernin
3   3     efforts           arfings
4   4      herald            raring

In both methods, the scoring picks the most bigram-typical candidate and does not match the base word, so the pseudowords come out more bigram-typical than their words. The difference in mean bigram probability is about d = -1.0, words minus pseudowords, in en_lexdec and en_lexdec_wuggy alike.

Both generators process base words in byte order, so the set of already-used pseudowords evolves identically in the two engines and both select byte-identical stimuli. Their alphabet is the set of letters the reference words attest, in UTF-8 byte order, so a Spanish lexicon gives them á, é, í, ó, ú, ü and ñ, which one word in six carries in a typical Spanish band. build_lexdec_stimuli() keeps the pool’s words written wholly in those letters, and a generated design’s datasheet counts these words in the band as its candidates. A word with a hyphen, an apostrophe or a digit is left out. The subsyllabic segmenter takes its nuclei from the vowels of the syllable estimate, whatever their accent, and a word without one, such as a Russian or Greek word, is handled by letter substitution. Chinese words are rejected with an error, since Unicode counts their characters as letters but a form made by swapping one character for another is of unknown validity as a pseudoword. In the Spanish demonstration, design_es_lexdec.yaml, 8 of the 60 words and 4 of the pseudowords hold a letter outside a to z, and the pseudowords are checked against wordfreq’s Spanish word list.

The stimuli give each pseudoword its base word and its edit_distance from it, the Levenshtein distance, and give a word its own form and 0. A generated design’s report compares the words with their pseudowords on length, which they share by construction, and on bigram frequency, N and OLD20, which the generator does not hold constant. The report marks length as controlled and the other three as uncontrolled, and it computes those three against the lexicon on a copy of the stimuli. In en_lexdec, the bigram-frequency difference between the words and their pseudowords is d = -0.97 (90% CI [-1.29, -0.65]).

A pseudoword is a non-word only relative to the reference words, so a real word too rare for the lexicon can still come out as one. The bundled English and Spanish lexica stop at Zipf 2.95 and 3.11, and with the English lexicon alone the generators can produce real words such as crier and pall. exclude names further known word forms that no pseudoword may equal. They are trimmed, lower-cased and put in NFC like the lexicon’s words, and they take no part in the bigram statistics. In a design, the list goes under items.generation.exclude, and each entry is a file with one form per line, a CSV file with a word column or a registry corpus already fetched into the cache:

items:
  source: generate
  lexicon: corpora/derived/en_v2.csv
  generation:
    exclude: [corpora/derived/en_known_forms.txt]

Each file must be UTF-8 text, and no form in a one-form-per-line file may hold a tab, a comma or a semicolon. A file in which any form holds one of these characters is a table. Each of its rows would be kept whole as a form that matches nothing, so the file is rejected with an error. The datasheet records each list with its checksum and the number of forms it holds, and the Methods paragraph names it. In the lexsync repository, python corpora/fetch_corpora.py --known-forms en es writes wordfreq’s fuller lists to corpora/derived/ (289,023 English and 334,625 Spanish forms at Zipf 1.0 or above). They are not included in the package.

A long list makes ill-formed pseudowords more likely. The generators’ only check on form is that each letter pair occurs somewhere in the lexicon, so with fewer candidates left, the most bigram-typical one can be ill-formed. In the English demonstrations, 13 of the 140 pseudowords are ill-formed, among them fdes, gngs and ntena. Each lacks a vowel or opens or closes on a letter pair that opens or closes no screened English word of three letters or more with a vowel. Without the list, the same designs give 5 such pseudowords. Read the generated forms before a study runs.

generate_pseudowords() and make_pseudoword() can be called directly if you want the forms without a lexical-decision stimulus set.

generate_pseudowords(c("house", "table"), lex$word)
  base_word pseudoword
1     house      houre
2     table      talle

In a design, this is items.source: generate, with the method under items.generation.method. design_en_lexdec.yaml and design_en_lexdec_wuggy.yaml are the worked examples of the two methods, and design_en_lexdec_blocks.yaml adds practice and filler blocks to the same generated set.

Items as a random factor

A matched set is one sample from the language, and an analysis that treats its items as fixed generalises to those items rather than to the language they were drawn from (Clark, 1973; Yarkoni, 2022). Crossed random effects are the standard answer, and the datasheet suggests a model that includes them. resample_stimuli() draws several matched sets that share no items, each one an independent sample. A study can then run different item samples across participant groups or show that an effect survives the change of materials.

reps <- resample_stimuli(pool, design, schema, n_sets = 2)
table(reps$replicate, reps$condition)
   
    high low
  1   20  20
  2   20  20
length(intersect(
  reps$word[reps$replicate == 1],
  reps$word[reps$replicate == 2]
))
[1] 0

Each replicate is matched independently against the pool with the earlier replicates’ items removed, so the sets are disjoint by construction. Because later replicates draw from a depleted pool, match quality can degrade across replicates, and the report shows it if it does.

References

Brysbaert, M., & New, B. (2009). Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41(4), 977–990. https://doi.org/10.3758/BRM.41.4.977

Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335–359. https://doi.org/10.1016/S0022-5371(73)80014-3

Coltheart, M., Davelaar, E., Jonasson, J. T., & Besner, D. (1977). Access to the internal lexicon. In S. Dornic (Ed.), Attention and Performance VI (pp. 535–555). Erlbaum.

González Alonso, J., Bernabeu, P., Silva, G., DeLuca, V., Poch, C., Ivanova, I., & Rothman, J. (2025). Starting from the very beginning: Unraveling third language (L3) development with longitudinal data from artificial language learning and EEG. International Journal of Multilingualism, 22(1), 119–142. https://doi.org/10.1080/14790718.2024.2415993

Gu, X. S., & Rosenbaum, P. R. (1993). Comparison of multivariate matching methods: Structures, distances, and algorithms. Journal of Computational and Graphical Statistics, 2(4), 405–420. https://doi.org/10.1080/10618600.1993.10474623

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Academic Press. https://doi.org/10.1016/C2009-0-03396-0

Kessler, B., Treiman, R., & Mullennix, J. (2002). Phonetic biases in voice key response time measurements. Journal of Memory and Language, 47(1), 145–171. https://doi.org/10.1006/jmla.2001.2835

Keuleers, E., & Brysbaert, M. (2010). Wuggy: A multilingual pseudoword generator. Behavior Research Methods, 42(3), 627–633. https://doi.org/10.3758/BRM.42.3.627

Kuperman, V. (2015). Virtual experiments in megastudies: A case study of language and emotion. Quarterly Journal of Experimental Psychology, 68(8), 1693–1710. https://doi.org/10.1080/17470218.2014.989865

Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10.1177/1948550617697177

Liben-Nowell, D., Strand, J., Sharp, A., Wexler, T., & Woods, K. (2019). The danger of testing by selecting controlled subsets, with applications to spoken-word recognition. Journal of Cognition, 2(1), Article 2. https://doi.org/10.5334/joc.51

Rubin, D. B. (1980). Bias reduction using Mahalanobis-metric matching. Biometrics, 36(2), 293–298. https://doi.org/10.2307/2529981

Sassenhagen, J., & Alday, P. M. (2016). A common misapplication of statistical inference: Nuisance control with null-hypothesis significance tests. Brain and Language, 162, 42–45. https://doi.org/10.1016/j.bandl.2016.08.001

Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1–21. https://doi.org/10.1214/09-STS313

van Heuven, W. J. B., Mandera, P., Keuleers, E., & Brysbaert, M. (2014). SUBTLEX-UK: A new and improved word frequency database for British English. Quarterly Journal of Experimental Psychology, 67(6), 1176–1190. https://doi.org/10.1080/17470218.2013.850521

Volgenant, A. (2004). Solving the k-cardinality assignment problem by transformation. European Journal of Operational Research, 157(2), 322–331. https://doi.org/10.1016/S0377-2217(03)00205-4

Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, Article e1. https://doi.org/10.1017/S0140525X20001685

Yarkoni, T., Balota, D., & Yap, M. (2008). Moving beyond Coltheart’s N: A new measure of orthographic similarity. Psychonomic Bulletin & Review, 15(5), 971–979. https://doi.org/10.3758/PBR.15.5.971