EB-NeRD small · 8 seeds · held-out users · exploration · 3 machines · clicks only

The same six memories on Danish news

EB-NeRD
8 seeds
held-out users
6 models
exploration
clicks only

Same models, grid, seeds and reporting as the Amazon and MIND studies, on EB-NeRD small with the professor's frozen protocol: frozen Qwen article embeddings, histories of clicks only (so the like/dislike sign is always 'like' and the two-network model only writes its positive network). Development data is the 10% of training behaviours the loader holds out, split into tuning and held-out halves: 11,778 tuning and 11,511 held-out impressions from 3,318 users, matching the handoff's cohorts exactly. The official labelled validation directory is the test set and was never scored.

Takeaway

On EB-NeRD the merged network's lead over the grid is +0.0024 (7 of 8 seeds ahead); two separate networks: 0.5160; best plain average 0.4886. With no dislikes in the data, this isolates the memory mechanism itself.

Held-out users · mean NDCG@10 over 8 seeds

Higher is better. Bars start at zero, so small gaps look small.

Grid memory (FWPRec)

0.51430

Network, normalized

0.51666

Network, additive

0.51568

Two networks

0.51598

Window mean, no memory

0.48559

Signed EMA, no memory

0.48859

Per seed

Seed	Grid memory (FWPRec)	Network, normalized	Network, additive	Two networks	Window mean, no memory	Signed EMA, no memory
Seed 42	0.51437	0.51895	0.51866	0.51909	0.48555	0.48858
Seed 43	0.51037	0.51494	0.51418	0.51499	0.48560	0.48859
Seed 44	0.51466	0.51706	0.51464	0.51497	0.48561	0.48859
Seed 45	0.51832	0.51502	0.51563	0.51392	0.48560	0.48856
Seed 46	0.51389	0.51478	0.51410	0.51250	0.48555	0.48861
Seed 47	0.51205	0.51606	0.51474	0.51587	0.48568	0.48859
Seed 48	0.51380	0.51734	0.51632	0.51671	0.48558	0.48861
Seed 49	0.51692	0.51914	0.51712	0.51978	0.48554	0.48857
Mean	0.51430	0.51666	0.51568	0.51598	0.48559	0.48859
Spread (SD)	0.00251	0.00176	0.00161	0.00248	0.00004	0.00002
Seeds ahead of the grid	-	7 of 8	6 of 8	6 of 8	0 of 8	0 of 8

Lead over the grid memory, per seed

Same impressions scored by both models; brackets are 95% intervals from 2,000 user-level bootstrap samples.

Model	Seed 42	Seed 43	Seed 44	Seed 45	Seed 46	Seed 47	Seed 48	Seed 49
Network, normalized	+0.0046 [+0.0023, +0.0068]	+0.0046 [+0.0021, +0.0070]	+0.0024 [+0.0001, +0.0047]	-0.0033 [-0.0060, -0.0006]	+0.0009 [-0.0013, +0.0032]	+0.0040 [+0.0015, +0.0064]	+0.0035 [+0.0010, +0.0061]	+0.0022 [-0.0004, +0.0046]
Network, additive	+0.0043 [+0.0020, +0.0065]	+0.0038 [+0.0013, +0.0062]	-0.0000 [-0.0024, +0.0024]	-0.0027 [-0.0051, -0.0002]	+0.0002 [-0.0023, +0.0027]	+0.0027 [+0.0002, +0.0053]	+0.0025 [+0.0001, +0.0049]	+0.0002 [-0.0022, +0.0025]
Two networks	+0.0047 [+0.0023, +0.0071]	+0.0046 [+0.0022, +0.0071]	+0.0003 [-0.0021, +0.0028]	-0.0044 [-0.0072, -0.0016]	-0.0014 [-0.0038, +0.0010]	+0.0038 [+0.0013, +0.0063]	+0.0029 [+0.0006, +0.0054]	+0.0029 [+0.0003, +0.0052]
Window mean, no memory	-0.0288 [-0.0344, -0.0236]	-0.0248 [-0.0305, -0.0192]	-0.0291 [-0.0347, -0.0238]	-0.0327 [-0.0382, -0.0276]	-0.0283 [-0.0341, -0.0229]	-0.0264 [-0.0323, -0.0209]	-0.0282 [-0.0340, -0.0227]	-0.0314 [-0.0369, -0.0260]
Signed EMA, no memory	-0.0258 [-0.0313, -0.0204]	-0.0218 [-0.0276, -0.0162]	-0.0261 [-0.0318, -0.0205]	-0.0298 [-0.0353, -0.0245]	-0.0253 [-0.0311, -0.0198]	-0.0235 [-0.0293, -0.0180]	-0.0252 [-0.0310, -0.0195]	-0.0284 [-0.0339, -0.0229]

All metrics, mean over seeds

Training seconds are only comparable within a host; w06 has a faster GPU.

Model	AUC	MRR	NDCG@5	NDCG@10	≤20 events (n=296)	>20 events (n=11,215)	Private bytes	Host	Train s
Grid memory (FWPRec)	0.5939	0.4004	0.4460	0.5143	0.5138	0.5143	32,768	w02	60
Network, normalized	0.5964	0.4033	0.4498	0.5167	0.5259	0.5164	16,512	w03	56
Network, additive	0.5962	0.4022	0.4493	0.5157	0.5166	0.5157	16,512	w06	40
Two networks	0.5952	0.4026	0.4488	0.5160	0.5254	0.5157	32,768	w06	56
Window mean, no memory	0.5614	0.3690	0.4117	0.4856	0.4959	0.4853	0	w03	38
Signed EMA, no memory	0.5646	0.3728	0.4154	0.4886	0.5037	0.4882	0	w02	6
Static profile, no adaptation	0.5626	0.3701	0.4128	0.4866	0.4675	0.4871	0	-	-

Chosen settings

Model	Learning rate	Temperature	Epoch	Tuning NDCG@10
Grid memory (FWPRec)	0.003	0.2	8	0.5168
Network, normalized	0.003	0.1	7	0.5188
Network, additive	0.003	0.2	7	0.5187
Two networks	0.003	0.1	7	0.5179
Window mean, no memory	0.003	0.1	8	0.4888
Signed EMA, no memory	0.0003	0.1	1	0.4910

Noise check

The normalized network trained twice with seed 42 and identical settings scored 0.51808 and 0.51882 (gap 0.00074).

Development data only; exploratory. No test split scored. Hosts: w02 grid + signed EMA, w03 normalized network + window mean + noise check, w06 additive network + two networks. Files: experiments/isolated/ebnerd_memory_exploration/ on branch DPFWM-HP. Status: Complete.