Rerunning the professor's Amazon test
We reran the professor's continuous-memory study on Amazon, using his own script at commit cfc526c with one change: the starting profile for new users was rebuilt rather than loaded from an earlier study, because those files are not in Git. The data split matched his exactly. Every model got the same nine settings. Seed 42 picked the winner on tuning users, then all three seeds were scored on a separate group of users the tuning never saw. His handoff stopped before that last step, so the held-out numbers here are new.
The tuning result replicates. On held-out users the network leads by +0.0044 on average (1.7%), wins on two seeds and loses slightly on one. Not yet a reliable win.
Held-out users · mean NDCG@10 over 3 seeds
Higher is better. Bars start at zero, so the small gaps look small.
Per seed
The network moves about ten times more between seeds than the grid.
| Seed | Grid memory (FWPRec) | Network, normalized | Network, additive |
|---|---|---|---|
| Seed 42 | 0.26290 | 0.27332 | 0.27089 |
| Seed 43 | 0.26403 | 0.26205 | 0.26216 |
| Seed 44 | 0.26362 | 0.26842 | 0.26846 |
| Spread (SD) | 0.00057 | 0.00565 | 0.00450 |
Paired differences on held-out users
Same users scored by both models; intervals come from 2,000 user-level bootstrap samples. They don't include seed-to-seed variation.
| Comparison | Difference | 95% interval | Above zero? |
|---|---|---|---|
| Seed 42, network, normalized | +0.0104 | +0.0072 to +0.0136 | yes |
| Seed 42, network, additive | +0.0080 | +0.0043 to +0.0117 | yes |
| Seed 42, additive minus normalized | -0.0024 | -0.0057 to +0.0008 | no |
| Seed 43, network, normalized | -0.0020 | -0.0049 to +0.0010 | no |
| Seed 43, network, additive | -0.0019 | -0.0051 to +0.0013 | no |
| Seed 43, additive minus normalized | +0.0001 | -0.0031 to +0.0034 | no |
| Seed 44, network, normalized | +0.0048 | +0.0015 to +0.0082 | yes |
| Seed 44, network, additive | +0.0048 | +0.0011 to +0.0088 | yes |
| Seed 44, additive minus normalized | +0.0000 | -0.0033 to +0.0033 | no |
Tuning users · ours (professor's handoff)
Best score over learning rates and epochs at each temperature (T, the score scale). The grid memory matches to six decimals.
| Model | T 0.1 | T 0.2 | T 0.4 |
|---|---|---|---|
| Grid memory (FWPRec) | 0.2650 (0.2650) | 0.2587 (0.2587) | 0.2470 (0.2470) |
| Network, normalized | 0.2778 (0.2775) | 0.2675 (0.2684) | 0.2566 (0.2566) |
| Network, additive | 0.2768 (0.2770) | 0.2738 (0.2775) | 0.2732 (0.2727) |
Memory switched off · seed 42
With memory off, every model falls to the plain profile's score. The two checks agree exactly, as they should when memory is rebuilt for each request.
| Model | Normal | Writes off | Reset before read | Private bytes |
|---|---|---|---|---|
| Grid memory (FWPRec) | 0.26290 | 0.06398 | 0.06398 | 32,768 |
| Network, normalized | 0.27332 | 0.06398 | 0.06398 | 16,512 |
| Network, additive | 0.27089 | 0.06398 | 0.06398 | 16,512 |
Development data only, which earlier pilots had already looked at, so this is exploratory, not confirmation. No test split was touched. Run on damlr-w03; 27 tuning runs, 6 retraining runs, 53 minutes. Files: experiments/isolated/continuous_memory_replication_amazon/ on branch DPFWM-HP. Status: Complete.