Six memories, eight seeds, on Amazon
The second Amazon study, planned in ADR 0001 after the replication. Same data split, same shared starting profile and same tuning budget as run 1. Three models kept their run-1 settings; the two-network memory and two no-memory baselines were tuned on the same 3 × 3 grid. Every model was then retrained on seeds 42 to 49 and scored on the 3,847 held-out users, with the memory-off checks, timings, memory size and a split by history length. This is exploration: no win rule was set and the test split stays sealed.
Learned memories beat plain averages by a wide margin (0.263 vs 0.170). Among them, the merged network averages 0.2677 against the grid's 0.2634, ahead on 6 of 8 seeds; two separate networks average 0.2649.
Held-out users · mean NDCG@10 over 8 seeds
Higher is better. Bars start at zero.
Per seed
| Seed | Grid memory (FWPRec) | Network, normalized | Network, additive | Two networks | Window mean, no memory | Signed EMA, no memory |
|---|---|---|---|---|---|---|
| Seed 42 | 0.26290 | 0.27348 | 0.27128 | 0.26711 | 0.17039 | 0.17040 |
| Seed 43 | 0.26403 | 0.26210 | 0.26168 | 0.26075 | 0.17038 | 0.17040 |
| Seed 44 | 0.26362 | 0.26898 | 0.26895 | 0.26383 | 0.17022 | 0.17039 |
| Seed 45 | 0.26058 | 0.26385 | 0.26010 | 0.26506 | 0.17024 | 0.17040 |
| Seed 46 | 0.26280 | 0.27326 | 0.26763 | 0.26804 | 0.17024 | 0.17033 |
| Seed 47 | 0.26351 | 0.27164 | 0.27087 | 0.26454 | 0.17024 | 0.17040 |
| Seed 48 | 0.26454 | 0.26099 | 0.26422 | 0.26434 | 0.17033 | 0.17039 |
| Seed 49 | 0.26489 | 0.26732 | 0.26609 | 0.26593 | 0.17024 | 0.17040 |
| Mean | 0.26336 | 0.26770 | 0.26635 | 0.26495 | 0.17028 | 0.17039 |
| Spread (SD) | 0.00134 | 0.00497 | 0.00412 | 0.00223 | 0.00007 | 0.00002 |
| Seeds ahead of the grid | - | 6 of 8 | 5 of 8 | 6 of 8 | 0 of 8 | 0 of 8 |
Lead over the grid memory, per seed
Same users scored by both models; brackets are 95% intervals from 2,000 user-level bootstrap samples.
| Model | Seed 42 | Seed 43 | Seed 44 | Seed 45 | Seed 46 | Seed 47 | Seed 48 | Seed 49 |
|---|---|---|---|---|---|---|---|---|
| Network, normalized | +0.0106 [+0.0073, +0.0136] | -0.0019 [-0.0048, +0.0009] | +0.0054 [+0.0020, +0.0088] | +0.0033 [+0.0005, +0.0063] | +0.0105 [+0.0072, +0.0138] | +0.0081 [+0.0043, +0.0119] | -0.0036 [-0.0064, -0.0007] | +0.0024 [-0.0003, +0.0053] |
| Network, additive | +0.0084 [+0.0047, +0.0120] | -0.0024 [-0.0056, +0.0008] | +0.0053 [+0.0017, +0.0091] | -0.0005 [-0.0036, +0.0026] | +0.0048 [+0.0012, +0.0085] | +0.0074 [+0.0037, +0.0110] | -0.0003 [-0.0034, +0.0026] | +0.0012 [-0.0018, +0.0041] |
| Two networks | +0.0042 [+0.0015, +0.0069] | -0.0033 [-0.0061, -0.0004] | +0.0002 [-0.0027, +0.0033] | +0.0045 [+0.0017, +0.0073] | +0.0052 [+0.0022, +0.0082] | +0.0010 [-0.0021, +0.0040] | -0.0002 [-0.0032, +0.0026] | +0.0010 [-0.0018, +0.0039] |
| Window mean, no memory | -0.0925 [-0.1001, -0.0851] | -0.0936 [-0.1011, -0.0864] | -0.0934 [-0.1009, -0.0862] | -0.0903 [-0.0976, -0.0831] | -0.0926 [-0.1002, -0.0849] | -0.0933 [-0.1009, -0.0858] | -0.0942 [-0.1017, -0.0871] | -0.0947 [-0.1022, -0.0876] |
| Signed EMA, no memory | -0.0925 [-0.1001, -0.0855] | -0.0936 [-0.1011, -0.0867] | -0.0932 [-0.1010, -0.0862] | -0.0902 [-0.0977, -0.0832] | -0.0925 [-0.1004, -0.0852] | -0.0931 [-0.1008, -0.0859] | -0.0942 [-0.1017, -0.0871] | -0.0945 [-0.1021, -0.0877] |
All metrics, mean over seeds
The two history columns are NDCG@10 on impressions whose user history fits in the 20-event memory window, and on longer ones.
| Model | AUC | MRR | NDCG@5 | NDCG@10 | ≤20 events (n=6,037) | >20 events (n=1,065) | Private bytes | Train s |
|---|---|---|---|---|---|---|---|---|
| Grid memory (FWPRec) | 0.7902 | 0.2227 | 0.2165 | 0.2634 | 0.2580 | 0.2938 | 32,768 | 82 |
| Network, normalized | 0.7963 | 0.2255 | 0.2211 | 0.2677 | 0.2623 | 0.2982 | 16,512 | 74 |
| Network, additive | 0.7938 | 0.2255 | 0.2206 | 0.2664 | 0.2605 | 0.2995 | 16,512 | 91 |
| Two networks | 0.7930 | 0.2231 | 0.2179 | 0.2649 | 0.2596 | 0.2954 | 32,768 | 108 |
| Window mean, no memory | 0.6926 | 0.1495 | 0.1331 | 0.1703 | 0.1635 | 0.2085 | 0 | 49 |
| Signed EMA, no memory | 0.7050 | 0.1490 | 0.1328 | 0.1704 | 0.1624 | 0.2155 | 0 | 49 |
| Static profile, no adaptation | 0.5038 | 0.0649 | 0.0448 | 0.0640 | 0.0509 | 0.1381 | 0 | - |
Noise check
The normalized network trained twice with seed 42 and identical settings scored 0.27332 and 0.27276 (gap 0.00055); weights differed, so GPU training is not exactly repeatable. Run 1's seed-42 score was 0.27332.
Development data only, previously examined; exploratory. No test split scored. Run on damlr-w03, 2 noise-check runs, 27 tuning runs, 48 retraining runs. Files: experiments/isolated/amazon_memory_exploration/ on branch DPFWM-HP; plan in docs/adr/0001-amazon-memory-exploration.md. Status: Complete.