Skip to contents

Changelog

Source: NEWS.md

XSLmodels (development version)

New Features

  • Added uncfam_general(), a single biased associative model that nests the uncfam() family, so variants can be compared as nested models (fix or free parameters) rather than as separate functions. A pair’s share of the trial’s weight is M^gamma * exp(B * (U(w) + U(o))) (normalized over the trial), plus eps times elimination support, and the trial’s weight is scaled by (relative object entropy)^kappa; finite K replaces deterministic allocation with sampled attention. Its defaults reproduce uncfam(). Setting gamma = 0, uncertainty = "novelty", a free gamma, a free eps, kappa = 1 or a finite K reproduces, respectively, uncfam(variant = "uncertainty-only"), uncfam(variant = "novelty"), uncfam_gamma(), uncfam_elimination(), uncfam_attention() and uncfam_sampling(). Each reproduces the original exactly (to the last bit, and draw for draw for sampling) on all 47 xsl_datasets conditions; this is checked in tests/testthat/test-uncfam_general.R. uncfam_predictive() is not nested, since its prediction-error update is unnormalized. Registered in show_models()/xsl_model_registry(), fitting X, B, C, gamma, eps, kappa with K = Inf. It also has an optional short-term trace: with rho > 0, every update also goes into a second matrix that decays by Cs per trial, and familiarity becomes (M + rho * S)^gamma, so recently strengthened pairings draw extra attention. Uncertainty, elimination and test performance still use M, and rho = 0 (the default) is unchanged
  • Every xsl_datasets condition now carries a response_matrix: for each test item, the number of participants choosing each object. This is the full confusion matrix behind accuracy, which it reproduces exactly, so models can be fit to which objects learners chose (e.g. by multinomial likelihood) rather than only to proportion correct. It is extracted from the raw response files (data-raw/raw_responses.R, data-raw/fix_freqcd_add_responses.R). data-raw/extract_response_matrices.R reports how each condition was matched to its raw data

Bug Fixes

  • tilles() now follows Tilles & Fontanari (2013) in three places where it departed from the paper. It matches an independent, literal transcription of the paper’s eqs 1-13 and procedure (tests/testthat/helper-tilles-literal.R) to 1e-12 on all 47 xsl_datasets conditions, including at the parameter bounds. This changes all tilles() results and fits.

    • Eq 7’s alpha is recomputed before every trial for every word seen so far, as the paper’s procedure (p. 4) specifies. Words absent from a trial use it in eqs 12-13. They used to keep the alpha from the last trial they appeared on, and a word seen only on trial 1 kept alpha_0.
    • Eqs 1 and 2 count objects. The paper’s contexts have C words and C objects. In the conditions with more objects than words (19 of 47, 201-222), eq 1 divided by the number of words and eq 2 by the number of new words, so a word’s associations summed to as much as 3.95 rather than 1 (eq 10). When a new word appears with no new objects (73 trials in 15 conditions, where its referent was shown earlier as a foil), eq 2’s b share now goes to all of the trial’s objects; it used to be dropped.
    • When a word has no association with any of the objects that eq 5 (or eq 11) is proportional to, the flux is distributed evenly over them, as the paper first describes eq 5. It used to be dropped, so the word’s associations summed to less than 1 (in condition 206 at any parameters, and in most of the 3x4-type conditions at b = 1).
    • Every word’s associations now sum to 1 at all parameters in [0, 1], bounds included, so b = 1 and x = alpha_0 = 1 no longer stop with “no associations left”.
    • At the v05 group-fit parameters (x = .129, b = .124, alpha_0 = .9855), SSE over the 47 conditions is .6466 (.6399 with only the eq 7 normalization fix below, .6125 before either), and model accuracy changes by up to .135 for a word. Refitting with v05’s settings gives SSE .6061 at x = .124, b = .129, alpha_0 = .979, the same as with only the eq 7 fix (.6060).

  • tilles()’s eq 7 now normalizes the entropy of a word’s associations to [0, 1], as Tilles & Fontanari (2013) specify, by dividing shannon_entropy() (in bits) by log2(n_o) rather than by log(n_o) in nats. The mismatch came from the original 2013 implementation, so the published tilles() results ran on it too. This changes all tilles() results and fits, except at alpha_0 = 1, where eq 7’s alpha is 1 whatever the entropy. Because alpha now stays in [alpha_0, 1], tilles() gives valid probabilities at every interior parameter vector. On all 47 xsl_datasets conditions at 1682 parameter vectors, all 72944 runs with x, b and alpha_0 in (0, 1) were valid, including the 23808 that had stopped with “associations became negative” (see below), and no association went negative even transiently. The only remaining stops were the boundary cases b = 1 and x = alpha_0 = 1, which no longer occur (see above). At the v05 group-fit parameters (x = .129, b = .124, alpha_0 = .985), SSE over the 47 conditions rises from .6125 to .6399. Refitting gives .6060 at almost the same parameters (x = .129, b = .126, alpha_0 = .979). When only one object has been seen, eq 7’s normalized entropy is now 0 instead of 0/0.

  • tilles() no longer returns NaN, or fails with “missing value where TRUE/FALSE needed” inside shannon_entropy(), at small alpha_0. Instead it stops with a clear error, tilles(): associations became negative by trial .... The cause is eq 7, carried over from the original 2013 implementation: it normalizes entropy in bits (log2) by log(n_o) in nats. Its uncertainty term can then reach 1/log(2) (about 1.44) rather than 1, so alpha falls below 0 when alpha_0 is below about .31. Eqs 8-13 can then push associations negative, and the next eq 7 entropy of a negative row is NA. tilles() now stops when a word’s associations are negative at its next eq 7 (where NaN used to start), or when final associations are negative (previously returned silently as negative probabilities). Separately, at the boundary values b = 1 (a novel word appearing with no novel objects gets no association) and x = alpha_0 = 1, a word could be left with no associations, giving the same NA (no longer, see above); this stopped with tilles(): word ... has no associations left by trial .... Every run that returned finite, nonnegative associations is unchanged, bit for bit. This was checked on all 47 xsl_datasets conditions at 553 parameter vectors, including every one in the v05 refit’s error log. Fitting already scored errors and NA results as an infinitely bad fit, so fits change only where tilles() previously returned negative probabilities, which are now also scored as infinitely bad. The normalization itself is corrected in the entry above, so small alpha_0 no longer causes this; the checks remain for parameters outside [0, 1] and for the boundary cases.

  • bayesian_decay()’s choice exponent ch_dec was ignored on conditions with a test (the 4AFC conditions 211-214 and the 12 filtering conditions): xsl_run() applied it as a power choice rule only on the get_perf() path, while mafc_test() scored test trials with plain Luce choice. predict_referent() and mafc_test() gain a d argument (the choice exponent; NULL, the default, leaves associations unchanged), and xsl_run() passes a model’s ch_dec to both paths. This changes bayesian_decay() fits involving any of these 16 conditions. Models without ch_dec are unchanged, bit for bit.

  • Two 4AFC conditions in xsl_datasets listed test alternatives that participants were not shown. This changes fits of every model involving conditions 211 and 214. See data-raw/fix_4afc_tests.R

    • 214 (3x4 6/8 4AFC): 175 of its 648 human responses, on all 18 words, were to objects not among the listed alternatives. All of them are among condition 213’s alternatives for the same word (and no other condition’s), so 214’s participants were evidently tested with 213’s displays. 214’s test is now 213’s. Its accuracy and response_matrix are unchanged.
    • 211 (1x3 6/18 4AFC), word 17: the alternatives listed object 17 twice and omitted foil 2, which participants chose; now c(17, 16, 5, 2).
    • Every human response in xsl_datasets is now to one of its item’s listed alternatives (checked in tests/testthat/test-data.R).

  • The frequency / contextual-diversity conditions in xsl_datasets are rebuilt from the original analysis data (data-raw/data/freqCD_item_data.Rdata), giving exact accuracies and correct sample sizes. This changes fits involving these conditions.

    • The stored accuracies were right, but n_subj had been attached to the wrong conditions. It is now 104 (was 26) for freq369-3x3hiCD, 66 (62) for freq369_36mx, 40 (66) for freq369_39mx and 62 (40) for cont_div6-12.
    • 3_x8_369_4x4 and freq369-3x3loCD had been pooled with the first test block of the “priming” experiments. Those participants trained on the same sequences but were tested four times, and the pooled values could not be reproduced. Both conditions now hold only their original experiments: n_subj 30 (was 74) and 34 (was 102), with item accuracies changing by up to .27 and .51.
    • Two duplicate conditions are dropped, leaving 54 conditions (was 56; 47 after the move below):
      • orig_3x3 was condition 224 (same training, identical accuracies from the same 36 participants) with n_subj 104.
      • orig_4x4’s 26 original participants are a subset of condition 223’s 77, and its training sequence differs from 223’s only in words 4 and 16 trading places between trials 11 and 24.
    • See data-raw/fix_freqcd_add_responses.R.

  • The Suanda, Mugwanya & Namy (2014) and Koehne, Trueswell & Gleitman (2013) conditions moved from xsl_datasets to new standalone held-out datasets, suanda2014 and koehne2013, registered in xsl_holdout_datasets, as in the original CogSci bake-off. Only their condition-level mean accuracies are available, so they can’t inform item-level group or cross-validated fits; they’re for testing generalization of mean accuracy. xsl_datasets now has 47 conditions, all with item-level accuracy and a response_matrix. See data-raw/move_condition_means_to_holdout.R

  • xslData‘s validator now expects one accuracy value (and one response_matrix row) per test item, i.e. per test trial when test is given, rather than per word. The filtering conditions’ 24-item tests failed the old check.

  • The 12 filtering conditions in xsl_datasets (filt*E_*L; Kachergis, Yu, & Shiffrin, 2012) were scored wrongly, and held only half their tests. Each word co-occurs in training with both its own object and its partner’s object (e.g. w1 with o1 and o7). It was tested twice, once with the partner object absent (AtoA, measuring w1-o1) and once with its own object absent (AtoB, measuring w1-o7). xsl_datasets held only the AtoA accuracies, scored as a choice among all 12 objects, so a model’s association with the absent partner object counted against it, and filt0E_* (symmetric pairs) was capped at .5. Each condition now has 24 test items scored by mafc_test(): the 12 AtoA items, then the 12 AtoB items. In the raw data, no participant ever chose the absent object on either test. Two participants in each *_6L condition had test codes shifted by 6, and are now excluded (n_subj 27 -> 23; the stored 27 was a per-word count inflated by the miscoding), which changes those conditions’ AtoA accuracies slightly. This changes all fits involving these conditions. See data-raw/fix_filtering_tests.R and ?xsl_datasets

  • mafc_test() (and xslData test specifications) accept an optional correct element giving each test trial’s correct object; it defaults to the tested word’s own object, as before

XSLmodels 0.4.0

New Features

  • Added uncfam_elimination(), a variant of uncfam() with learning by elimination (mutual exclusivity). Each on-trial pair’s mutual confidence P(o|w) * P(w|o) is computed from the current associations, and a pair gets elimination support to the extent that the trial’s other words and objects are already accounted for by each other. That support, weighted by a free parameter eps, is added to uncfam()’s familiarity-and-uncertainty share of the trial’s associative weight before renormalizing; eps = 0 reproduces uncfam() exactly. In uncfam(), an already-known pair on a trial draws weight away from the trial’s other pairs; with elimination, knowing one word directs learning to the remaining word-object pairing. Motivated by the initial-accuracy experiment, where learners were more accurate on an initially mis-paired word the more of its study trials it shared with an initially accurate word. Registered in show_models()/xsl_model_registry()
  • Added uncfam_gamma(), a variant of uncfam() that decouples familiarity’s contribution to a trial’s attentional allocation from uncertainty’s. uncfam()’s allocation weight is exp(B * entropy) * assocs – entropy gets a free exponent (B), but familiarity (assocs) enters linearly with an implicit weight of 1 and no free parameter of its own. uncfam_gamma() adds one parameter, gamma, raising familiarity to a free power instead (assocs^gamma); gamma = 1 reproduces uncfam() exactly. Cross-validated against all of xsl_datasets, it beats plain uncfam() in every fold of a 5-fold split (mean test SSE 0.48 vs. 0.56), with a best-fit gamma around 0.3 (familiarity’s pull dampens with diminishing returns). A separate analysis fitting this same free-gamma substrate to active cross-situational word learning (learners choosing which items to see named next, rather than a fixed passive sequence) instead found a best-fit gamma around 2 (amplified, not dampened) – see ?uncfam_gamma for the full comparison
  • Added minerva2(), todam(), and rem(), three distributed/episodic memory models ported from standalone prototypes that predated this package (Hintzman’s MINERVA2, 1984/1986; Murdock’s TODAM, 1982; Shiffrin & Steyvers’ REM, 1997) and were never wired into it or tested. All three represent words/objects as random feature vectors rather than tallying a single word-by-object association, and turn their native similarity/odds score into a Luce-choice-ready matrix via an exponentiated (X-temperature) or likelihood-ratio-based choice rule so they plug into get_perf()/mafc_test() like every other model here. Porting surfaced real bugs in the prototypes, fixed along the way (see each model’s @details/source comments): minerva2()‘s prototype compared a 2D echo-content vector against D-dim object vectors (silent recycling, not an error) instead of just the echo’s object half; todam()’s prototype computed trial-1 attention weights as cos_sim(item, 0) = NaN, poisoning memory from the first trial on, and both todam() prototypes called the long-removed as.real(); rem()’s prototype never actually implemented REM at all (it declared a geometric feature-vector generator but its model/test functions were an unmodified copy of the MINERVA2 prototype’s continuous-Gaussian code) – rem() here is a fresh adaptation of Shiffrin & Steyvers’ storage/odds equations to this package’s associative task. Benchmarking (see ?minerva2/?todam/?rem performance notes) found rem()’s raw per-simulation likelihood-ratio odds span several orders of magnitude (a lucky match on one rare feature value can dominate), which would otherwise let a single lucky simulation hijack xsl_run()’s sum-across-simulations aggregation; rem() now row-normalizes each simulation’s choice weights before returning them. All three are registered in show_models()/xsl_model_registry(), fitting only their choice-rule temperature (X) or storage parameters (g, u, c) by default – D/w (feature-vector dimensionality) are fixed representational hyperparameters, not free cognitive parameters. Speed varies widely: on the full xsl_datasets (53 conditions) at n_sim = 500, minerva2() takes ~30s, todam() ~8 min, and rem() ~20 min – todam()/rem() are impractical to xsl_fit() at full DEoptim settings (NP = 100, itermax = 100 would be tens of thousands of evaluations) without sharply reducing n_sim/NP/itermax, as the package’s own fitting examples already do for slow models
  • Added 3 more temporal-contiguity conditions to xsl_datasets from Kachergis, Yu, & Shiffrin (2009) – the same paper behind the existing 1_max_temp_spat_cont_orig/4_no_spat_orig_max_tc conditions. data-raw/XSL-dataset-fields.csv documented 6 “overlap” conditions (3 matched pairs of a control and non-control order at the same overlap degree) that were never pulled in; only the 3 control-order ones (1olap3tr_contr, temp_cont_1olap2tr_contr, temp_cont_2olap2tr_contr) have real subject-level accuracy data findable anywhere in the repo (in data-raw/agg_data/x13_1-21.txt, per export_data.R’s own comments mapping its generic condition labels to these named orders). The other 3 have order files but no discoverable human data, and are deliberately left out rather than guessed. See data-raw/add_temporal_contiguity_overlap.R
  • Added a fifth standalone dataset, benitez2020 (a named list of 7 xslData objects, one per condition x age-group combination) from Benitez et al. (2020), “The temporal structure of naming events differentially affects children’s and adults’ cross-situational word learning” (data: osf.io/2hmxr). data-raw/Benitez2020-rep/Benitez2020_XSLdata_extraction.R already extracted training orders, 2AFC test trials, and response-count matrices from the raw OSF data but stopped short of building xslData objects (a literal # ToDo in the file); data-raw/add_benitez2020.R picks up from there. 8 symmetric word-object pairs, 12 training trials, 8-item 2AFC test; unlike the other standalone datasets, real per-word accuracy is recoverable directly from the provided response matrices (not just an overall mean) – structurally it could join xsl_datasets directly, but is kept standalone by choice, as held-out generalization-test data (see xsl_holdout_datasets)
  • Added xsl_holdout_datasets, a registry of every dataset in the package that is not part of xsl_datasets and therefore never seen by get_group_model_fit()/get_crossvalidated_model_fit() – i.e. everything available for testing a fitted model’s generalization to genuinely held-out data, along with how to score each one (most need a different scoring approach than xsl_datasets’s standard diagonal convention, which is exactly why they’re standalone; benitez2020 and kachergis_initial_accuracy are the two directly-scorable exceptions, kept out of the fitting pool specifically to serve as held-out data)
  • Added a fourth standalone dataset, vlach_debrock2017 (Vlach & DeBrock, 2017, Journal of Memory and Language, “Remember dax? Relations between children’s cross-situational word learning, memory, and language abilities”). 12 symmetric word-object pairs, 2 pairs per training trial, 36 trials, with pairs varying in whether their 6 repetitions are massed or spaced (n = 47 children, ages 2-5). The source only reports an overall mean/SD across all pairs and participants, not a per-word breakdown, so accuracy is NA throughout rather than fabricated – kept standalone (not part of xsl_datasets) for the same reason as kachergis2012_highlighting/gangwani2011_category. See ?vlach_debrock2017 and data-raw/add_vlach_debrock2017.R
  • Added a third standalone dataset, gangwani2011_category (Experiment 1 of Gangwani, Kachergis & Yu, “Simultaneous Cross-situational Learning of Category and Object Names”; an undergraduate-journal version is Gangwani & Kachergis, Indiana Undergraduate Journal of Cognitive Science 5, 2010). A hierarchical design: each training trial shows 2 objects and names 3 words – the two objects’ own 1-to-1 names plus one 1-to-many “category” label naming those two objects and two others seen on other trials – so every object carries both a basic name and a category label, and the paper’s finding is that learners violate mutual exclusivity by learning both. A named list of three xslData objects (one per training block: ad hoc / natural / ad hoc), sharing one 36-trial co-occurrence schedule. Its 15 x 12 (word x object) association matrix breaks xslData’s diagonal-scoring convention the same way kachergis2012_highlighting does, so it is kept standalone; score it with score_gangwani2011_category() in the new tests/bakeoff_comparison/gangwani2011_category_fit.R, which reads 12-AFC name accuracy, 3-AFC category accuracy, and the paper’s headline ME-violation measure straight off any model’s matrix. See ?gangwani2011_category and data-raw/add_gangwani2011_category.R

Other changes

Documentation

  • The main vignette now covers every model and dataset added since 0.3.0 (uncfam_gamma(), uncfam_elimination(), minerva2()/todam()/rem(); vlach_debrock2017, benitez2020, xsl_holdout_datasets), and adds two sections: passing prior learning to a model as training trials vs. a start_matrix, and testing a fitted model’s generalization on the held-out datasets. Its description of the dataset structure used field names (trials, objs, perf) that no longer exist – so its create_cooc_matrix() example was silently printing an empty matrix – and said words/objects must be referred to by index, which no longer holds

Bug Fixes

  • xsl_fit() with a parallel DEoptim backend (DEoptim.control(parallelType = "parallel")) silently returned bestval = Inf when control was passed as a variable (e.g. ctrl <- xslControl(n_sim = 100); xsl_fit(..., control = ctrl)): control was never forced, so the objective shipped to the worker processes carried an unevaluated promise that each worker tried to evaluate in its own empty global environment, every evaluation errored, and the error was converted to Inf. control is now forced before fitting, and xsl_fit() warns when every evaluation of a fit failed instead of returning an all-Inf result silently. Serial fits were unaffected.
  • create_cooc_matrix() looped over the number of distinct words rather than the number of trials: with more words than trials it errored, and with fewer (e.g. its own documented example, xsl_datasets[[1]]: 18 words, 36 trials) it silently tallied only the first trials. It also indexed by raw label (the same issue as #14). Both fixed; plot_training_trials() now indexes by label too. No model uses either function, so no fits change
  • propose_but_verify() and pursuit() drew new hypotheses with sample(tr_o, ...). On a trial with a single object, tr_o is one value, and when that value is an integer k, R’s sample(k, n) draws from 1:k rather than returning k – so a word seen alone with an object usually hypothesized a different object, one not even on the trial (propose_but_verify() hypothesized the right object ~1/8 of the time in an 8-object toy). Both now index into tr_o (tr_o[sample.int(length(tr_o), n, replace = TRUE)], as guess_and_test() already did). No xsl_datasets condition has single-object trials, so fits to it are unchanged; kachergis_initial_accuracy (18 single-object familiarization trials per condition), kachergis2012_highlighting’s words-as-cues condition (28 trials naming two words with one object), and the naturalistic corpora (rollins_corpus, fm_corpus) are affected. Integer-labelled data (e.g. kachergis_initial_accuracy) was affected before this release too; the corpora were not – they reached these models as string labels, for which sample() behaves as intended – until #14 began handing models integer positions, so for the corpora this is a regression in 0.4.0.
  • Models indexed their association matrices by raw word/object value (m[tr_w, tr_o]) rather than by position in the sorted vocabulary, which is only correct for labels that are exactly 1..N (#14): a vocabulary with a gap crashed (“subscript out of bounds”), and a raw value that still fell in range silently updated the wrong row. Fixed once, in xsl_run(): models now receive each dataset’s words/objects as positions, and the original labels are restored on the returned matrix, perf, and trajectory. Test trials are scored by label, and a start_matrix with dimnames is aligned to the data by label. Checked against every registry model on every bundled dataset (23 models x 75 datasets, fixed seed): 1721 of 1725 runs are bit-identical, including all of xsl_datasets and the standalone datasets, whose labels are already 1..N. The 4 that differ are propose_but_verify() and pursuit() on rollins_corpus/fm_corpus, where the fix corrects a real error: both compared a column position against the trial’s object labels, which can never match for character labels, so on the corpora every hypothesis was treated as disconfirmed and none was ever verified. (Handing these models integer positions also exposed the single-object sample() bug above on the same corpora; both fixes are needed for correct corpus results.) The propose_but_verify and pursuit results previously in tests/bakeoff_comparison/corpus_report.md were therefore invalid (including pursuit’s reported collapse on FM); the report and vignette("corpora") have been rerun on the fixed code
  • uncfam_predictive() now caps associations at beta (1), its stated maximum, as well as flooring them at 0. Its unnormalized update could overshoot far above beta on a word’s first exposure: the prediction error is then ~beta, and it is scaled by exp(B * entropy) of the word’s row, which update_known() spreads over every object seen so far – so a word first seen after many others could reach an association in the thousands in a single trial (e.g. ~4600 for the 18th of 18 sequentially introduced words at X = .8, B = 1.6, C = .7). That association then dominated the word’s row and collapsed its entropy, locking in the first pairing it was seen with; in the initial-accuracy study this made the model predict learners would choose an initially inaccurate word’s familiarization referent 3-4x as often as they did. This changes uncfam_predictive()’s fits wherever the overshoot occurred, including the corpus-fit gains reported for it in tests/bakeoff_comparison/corpus_report.md (not rerun here)
  • get_perf() now treats negative associations as 0 before applying the Luce choice rule (as predict_referent() already did), and answers a word whose associations are then all 0 at chance (1 / ncol(m)) rather than returning NaN. Error-driven models (rescorla_wagner(), kalman_filter()) can learn negative associations, for which diag(m) / rowSums(m) gave per-item “probabilities” outside [0, 1] – which DEoptim could exploit when fitting. In the initial-accuracy study, fitted rescorla_wagner() had negative associations for about half of participants and out-of-range predictions for 25 of 61. This changes those models’ fits wherever negative associations occurred
  • kachergis_initial_accuracy put the familiarization pairing on the diagonal (the xslData “correct referent” slot), but the experiment’s test-correct answer is always the study pairing. For every initially inaccurate item these differ, so any model’s accuracy on those items was actually its probability of choosing the wrong, familiarization object. Its per-item accuracy was also keyed by init_word_ind (the word originally familiarized with the tested word’s correct object) rather than by the tested word. Both are fixed: objects are now indexed by the word they are studied with, so the study pairing is the diagonal and an initially inaccurate word’s familiarization object is off-diagonal, and accuracy is keyed by the tested word. It now also excludes the 4 participants whose median test response time was under 400 ms (as the accompanying manuscript does), leaving 34 High IA and 27 Low IA participants. tests/bakeoff_comparison/kachergis_initial_accuracy_fit.R and kachergis_initial_accuracy_persubject_fit.R are updated and rerun accordingly
  • kachergis_initial_accuracy’s familiarization trials were in word-index order; they now follow the order participants actually saw them (the same for every participant, recovered from the raw familiarization records). Order matters for the uncfam() family, whose first-exposure update depends on how many objects have already been seen via update_known() – for uncfam_predictive() it determined how far a word’s familiarization association overshot beta before the cap above. The bakeoff scripts are rerun accordingly

XSLmodels 0.3.0

New Features

  • Added two new model variants of uncfam(): uncfam_attention() (BAM+Attention), which scales a trial’s learning rate by that trial’s object uncertainty relative to the running mean (implementing Fitneva & Christiansen 2015’s system-level attention account), and uncfam_predictive() (BAM+Prediction), which replaces the normalized update rule with an item-level, Rescorla-Wagner-style prediction-error term (floored at 0 to prevent the unnormalized update from diverging over trials)
  • Added two standalone datasets, documented but deliberately not appended to xsl_datasets: kachergis2012_highlighting (Kachergis, 2012, CogSci) and kachergis_initial_accuracy (an unpublished MTurk initial-accuracy manipulation). kachergis2012_highlighting‘s training trials are genuine simultaneous 2-cue trials (2 words + 1 object, or 2 objects + 1 word, per trial, matching the paper’s Figure 2), read directly from the real per-trial ordering file for the real N=67 dataset behind the paper’s published statistics – confirmed by reproducing its exact reported response proportions and chi-square values (an earlier version of this dataset, built from a different, superficially similar file before that real data was located, did not reproduce them, and has been replaced). Each condition replicates the classic 3-role highlighting structure (PE, PL, I) twice; words-as-cues’ PL items have their real target at a different index than their own word index, and its I items have two legitimate targets – neither fits xslData’s one-correct-object-per-word accuracy convention, so only its two PE items get a real accuracy value (the rest are NA). Objects-as-cues has no such issue and gets a real accuracy value for all 4 of its items. Because of the partial-NA accuracy vector, kachergis2012_highlighting is kept standalone: an NA would silently poison every other dataset’s aggregate fit in get_group_model_fit()/get_crossvalidated_model_fit(), which sum/average SSE across all of xsl_datasets. kachergis_initial_accuracy has no such blocker but is kept standalone for now. See their ?help pages and data-raw/add_kachergis2012_highlighting.R/add_kachergis_initial_accuracy.R for full construction details, and tests/bakeoff_comparison/kachergis2012_highlighting_fit.R for a custom scorer that fits a model against all four of the paper’s reported proportions (PE-E, PL-L, I-E, I-L) directly off the association matrix, the way the paper’s own (non-package) fitting procedure did
  • Added kalman_filter(), a Kalman-filter generalization of rescorla_wagner()’s error-driven update: each word-object association is tracked as a Gaussian belief (mean and variance), giving an adaptive learning rate that starts high under uncertainty and shrinks with confidence, rather than a fixed rate (Dayan & Kakade, 2000; Kruschke, 2008)
  • Added softmax_rl(), a Q-learning-style reinforcement learning model. Unlike every other model in the package (which updates every word-object pair presented together on a trial), it makes a discrete choice per word – sampling a candidate referent from a softmax policy over its current values, then updating only that guess via a TD/delta-rule reward (1 if the guess was present in the scene, 0 otherwise). This mirrors propose_but_verify()/pursuit()’s propose-and-verify logic but replaces their threshold-based keep/discard rules with graded value learning and explicit softmax exploration
  • Added two naturalistic corpora imported from the wurwur package (github.com/mcfrank/wurwur), as standalone objects (not appended to xsl_datasets, which is scored by SSE against a human accuracy vector these don’t have): rollins_corpus – the CHILDES/Rollins corpus the Frank, Goodman & Tenenbaum (2009) model was fit to (619 mother-infant utterances, 416 words, 22 objects) – and fm_corpus – the larger Frank, Tenenbaum & Fernald corpus (4763 utterances, 1122 words, 30 objects), which additionally carries the speaker’s hand-coded referential intention per utterance (fm_corpus$intents). Each is a list bundling the training xslData ($data) with a gold-standard lexicon ($gold, plus $gold_variants for FM) to be scored with get_fscore()/get_roc()/get_tp() rather than SSE. Gaze/hand attentional cues in the FM source are not imported. See data-raw/add_wurwur_corpora.R
  • Added predict_referent(), a model-agnostic trial-level prediction: given any model’s word-object matrix, a heard word, and the objects present, it returns the probability distribution over those objects (P(object | word) proportional to m[word, object] * prior(object), normalized over the objects present) – the quantity to compare against a participant’s choice on a referent-selection trial. Supports a non-uniform prior (set an entry to 0 to remove an object the speaker has no epistemic access to) and a pragmatic = TRUE mode that resolves the objects with a one-step Rational Speech Act pragmatic listener (rsa_listener()/rsa_speaker(), also exported), yielding strong mutual exclusivity when a heard word is lexically ambiguous and a competitor word names one of the candidates. mafc_test() is now a thin wrapper over it (behavior unchanged; gains a ... passthrough for pragmatic/threshold)
  • Added fgt2009() and fgt2009_rsa(), a pure-R port of the intentional Bayesian word-learning model of Frank, Goodman & Tenenbaum (2009) and its Rational Speech Act (RSA) pragmatic extension, from the Python package wordlearn (github.com/mcfrank/wurwur). Unlike every other model in the package – which tally word-object associations incrementally, trial by trial – this is a batch model: it does joint MCMC inference (blocked per-object Gibbs plus MI-biased single-edge Metropolis moves) over the whole corpus at once, asking which lexicon best explains all the data, so xslFit$matrix holds posterior edge marginals P(word names object | corpus) rather than a running tally. It therefore has no learning trajectory and ignores control$reps. The scoring kernel is verified to match the Python reference to numerical precision. gamma/kappa default to the values used for the analogous cross-situational simulation in the source package (gamma = 1, since every word in a controlled XSL experiment is a referential label). The lexicon-size prior alpha scales with corpus size and its fit-vs-alpha curve is single-peaked, so it is not wired into xsl_model_registry()’s DEoptim bounds like the other models; use the new fgt2009_sweep_alpha() helper instead (which takes an optional gold argument to report the best-threshold F-score against a gold lexicon at each alpha, for corpora with no human accuracy vector)
  • Added a vignette, “Running and fitting models on naturalistic corpora”, covering how to run any model on rollins_corpus/fm_corpus, score the learned matrix against a gold lexicon with get_fscore()/get_roc()/get_roc_max(), and choose fgt2009()’s alpha by sweeping. It ends with a quick model comparison (bayesian_decay and fazly recover the gold lexicon best on both corpora, with fgt2009 close behind and highest-precision). The comparison script and its outputs are in tests/bakeoff_comparison/corpus_gold_comparison.R (every model at registry defaults); a companion, corpus_gold_fits.R, fits each model’s parameters to each corpus (parallel DEoptim maximizing gold-lexicon F, since these corpora have no accuracy vector; fgt2009() by an alpha sweep). Fitting lifts every model – dramatically for the ones whose registry defaults were tuned on controlled experiments (tilles +0.34, uncfam_predictive +0.15/+0.32) – and bayesian_decay fit to the FM corpus reaches F = 0.56. tests/bakeoff_comparison/corpus_report.md is a rendered write-up of all of this

Other changes

  • xsl_run() is now far more memory-frugal on long inputs (corpora). Previously it built the full list of every simulation’s xslFit – each carrying a word-by-object matrix per training trial – and only reduced them at the end, so a stochastic model on the 4763-utterance FM corpus at n_sim = 50 needed tens of GB and simply failed. Now: (1) the per-trial trajectory (xslFit$traj) is only recorded when xslControl(keep_traj = TRUE) – it is off by default and is not used by any function in the package; (2) xsl_run() accumulates the summed matrix one simulation at a time rather than holding all n_sim at once; (3) the full per-simulation list (fits[[i]]$sims) is dropped unless xslControl(keep_sims = TRUE), replaced by fits[[i]]$responses, a compact n_sim x n-words matrix of each simulated participant’s final per-word accuracy. matrix, perf, and sse are numerically unchanged.

Performance

  • uncfam_sampling()/multi_sampling(): every trial was allocating two full voc_sz x ref_sz matrices and computing a full outer product, even though only the handful of cells at [tr_w, tr_o] are ever nonzero. Entropy, weighting, and the update are now computed on that small per-trial submatrix instead. Behaviour-preserving – the sample() call’s probability vector is still built at full ref_sz length (zero outside tr_o), so the RNG draw sequence, and therefore every existing fit, is unchanged. 1.17x faster on a small lab condition, 1.97x on rollins_corpus (bigger vocabularies benefit more, since that’s the dimension the fix targets)
  • shannon_entropy() computed sum(p) twice (once for the <= 0 check, again for normalization); now computed once and reused
  • update_known()’s which(...)-based zero-filling replaced with direct logical-vector indexing, removing five which() calls per trial

Bug Fixes

  • update_known() computed which(m[w, fam_objects] == 0) – positions within the fam_objects-filtered subset – and then indexed m[w, zeros] as if those were absolute column positions, silently filling startval into the wrong cell whenever fam_objects (or fam_words, for the symmetric object-side fill) was a proper subset of all columns/rows – i.e. whenever some word or object was still wholly unfamiliar. Affects all 5 models that call it (uncfam(), uncfam_attention(), uncfam_predictive(), uncfam_sampling(), multi_sampling()). In practice this only ever mis-set a handful of near-zero cells during early training that get overwritten by real learning signal soon after – verified to make no difference to the final matrix on any of xsl_datasets’ 53 conditions or fm_corpus, and a single cell’s startval on rollins_corpus. Fixed as part of the logical-indexing rewrite above.
  • guess_and_test() errored (if with a zero- or multiple-length condition at the disconfirmation step) on any utterance that repeats a word – common in a naturalistic corpus, impossible in a controlled trial – because the repeated word was visited twice and the first pass could clear or double-set its hypothesis. It now de-duplicates a trial’s words and objects. Its disconfirmation check also compared a column position to the object labels (only equivalent when objects are labelled 1..N), and it applied new hypotheses to need_hypoths[w] rather than the actual stored word store[w] (setting hypotheses for the wrong words whenever only a subset rolled below the storage threshold); both are fixed.
  • tilles() coerced each trial’s words/objects with as.integer() (it indexes its state by position), turning any non-integer label into NA and leaving the matrix all-zero; it also never set dimnames on its matrix, so the result could not be scored against a gold lexicon by label. It now maps labels to positions with match() and names its matrix. xsl_datasets results are unchanged (the labels there are already 1..N).
  • softmax_rl() checked its sampled guess against the trial’s objects with proposal %in% tr_o, comparing a column position (1..ref_sz) against the objects’ labels. For xsl_datasets the object labels are 1..N so it happened to work, but on any dataset whose objects have non-integer (or non-contiguous) labels – e.g. either bundled corpus – every reward was 0 and the learned Q-matrix stayed identically zero. It now maps the objects present to positions (match(tr_o, ref)) before the check, and skips a trial with no objects present. xsl_datasets results are unchanged.
  • uncfam_sampling() and multi_sampling() errored (sample(): “too few positive probabilities”) on any dataset containing an utterance with no objects present – e.g. the ~320 non-referential utterances in fm_corpus, which made both models unable to run on a naturalistic corpus. Such a trial now leaves the association matrix unchanged apart from decay (as uncfam() already did), by skipping the per-word sampling when its weights are all zero and only applying the normalized update when something was sampled. Behaviour on datasets without objectless trials (all of xsl_datasets) is unchanged.
  • tilles() was essentially unfittable – three separate bugs (not the single one originally suspected) made a quick DEoptim search return Inf for every draw: (1) its alpha term was recomputed fresh every trial instead of persisting per-word across trials, so words absent from the current trial (needed by the “other old words” update) always had alpha = NA; (2) an internal matrix was sized voc_sz x voc_sz instead of voc_sz x ref_sz, a copy-paste bug that only breaks on asymmetric datasets (word count != object count); (3) a flux computation divided by a quantity that can legitimately be zero (a previously-seen word paired with an entirely new-to-it set of objects), producing Inf * 0 = NaN. One dataset failed on 100% of a 40-draw sweep regardless of parameters before these fixes; after them, a 100-draw sweep shows a residual ~2% failure rate confined to extreme parameter-space tails, on par with other models’ known numerical edges

XSLmodels 0.2.0

New Features

Bug Fixes

  • xslData() no longer rejects data with no response_matrix supplied – this was breaking the vignette, most function examples, and get_example_ambiguous_condition()/get_example_unambiguous_condition()
  • get_group_model_fit()/get_crossvalidated_model_fit() called uncfam_sampling(), multi_sampling(), tilles(), and fazly() with incorrect parameter names, causing them to error for those models
  • update_params() was dropping the parameter list’s names/type, which made xsl_run() error for every model except bayesian_decay() whenever it was called after fitting (e.g. from xsl_fit() or cross-validation)
  • xsl_fit() now treats a model erroring or returning NA/NaN SSE for a given parameter draw as an infinitely bad fit instead of aborting the whole optimization run
  • Fixed tilles() mislabeling its model name as "rescorla_wagner"
  • create_cooc_matrix() test had an incorrect expected value
  • plot_training_trials() now gives an informative error if the optional viridis/gganimate packages aren’t installed, instead of a raw namespace-loading error
  • xsl_run() checked !is.null(dat$test) to decide whether to use mafc_test(), but xslData()’s test defaults to list() (not NULL), so any dataset built without explicit test trials – including the package’s own get_example_ambiguous_condition()/get_example_unambiguous_condition() – was silently routed through mafc_test(mat, list()). That returned numeric(0) rather than erroring (an R quirk: 1:0 counts down instead of being empty, and vec[i] <- numeric(0) truncates the vector rather than erroring), producing a false sse = 0 “perfect fit” with no warning. Now checks length(dat$test) > 0.
  • guess_and_test() had an inverted guard clause (if (length(which(m[w,]==1))) next) that made its hypothesis-disconfirmation step permanently unreachable, since it was only reached for words that already have a stored hypothesis (guaranteeing the guard’s condition was always true). Hypotheses were therefore never disconfirmed once formed, silently disabling a core part of the model (“…on later encounters, learners attempt to retrieve this hypothesis from memory and test it against a new context, updating it only if it is disconfirmed”). Found by comparing the package’s fits against the original pre-package research code’s fits for the same 44-condition dataset.

Documentation

  • Enhanced README with working examples
  • Updated vignette with proper model categorization
  • Added comprehensive function documentation
  • Documented models and datasets