Cautious Muon + SP4096 + Depth Recurrence — val_bpb 1.1604 (non-record)#1381
Cautious Muon + SP4096 + Depth Recurrence — val_bpb 1.1604 (non-record)#1381X-Abhishek-X wants to merge 2 commits intoopenai:mainfrom
Conversation
Applies Cautious Muon (arXiv:2411.16085) to mask Muon optimizer updates where Newton-Schulz direction disagrees with raw gradient sign. Built on PR openai#1334 base with SP4096, depth recurrence, parallel residuals, MuonEq-R, QK-Gain 5.0, GPTQ INT6 + Brotli. 3-seed mean: 1.1604 bpb (seeds 42, 314, 999) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new Track A (10min_16mb) records folder documenting an experiment that applies “Cautious Muon” masking to Muon optimizer updates on top of the existing SP4096 + depth recurrence + parallel residuals stack, and includes 3-seed training logs plus submission metadata.
Changes:
- Adds a new record folder with
README.mddescribing the technique and results. - Adds
submission.jsoncapturing aggregated metrics and per-seed artifact sizes. - Adds per-seed training logs and a self-contained
train_gpt.pyentrypoint.
Reviewed changes
Copilot reviewed 3 out of 6 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/README.md | Documents the method (Cautious Muon) and experiment results + reproduction instructions. |
| records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/submission.json | Provides submission metadata and aggregated/per-seed metrics. |
| records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/train_gpt.py | Self-contained training script payload for reproducing the run. |
| records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/train_seed42.log | Seed 42 training + eval log for the submission. |
| records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/train_seed314.log | Seed 314 training + eval log for the submission. |
| records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/train_seed999.log | Seed 999 training + eval log for the submission. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| | Seed | val_bpb | val_loss | Artifact Size | | ||
| |------|---------|----------|---------------| | ||
| | 42 | 1.1568 | 2.6619 | 15,179,504 B | | ||
| | 314 | 1.1611 | 2.6717 | 15,173,470 B | | ||
| | 999 | 1.1634 | 2.6770 | 15,159,223 B | |
There was a problem hiding this comment.
The results table rows start with double pipes (|| ...), which GitHub Markdown renders as an extra empty first column. Use single leading | for each row so the table displays correctly.
| The primary modification is applying the Cautious optimizer principle to the Muon optimizer. After Newton-Schulz orthogonalization and MuonEq-R row normalization, the update is masked to only apply where the orthogonalized direction agrees with the raw gradient sign: | ||
|
|
||
| ```python | ||
| caution_mask = (g * raw_grad > 0).to(g.dtype) | ||
| g = g * caution_mask / caution_mask.mean().clamp_min(1e-3) | ||
| ``` |
There was a problem hiding this comment.
The description of Cautious Muon says the mask is applied “after Newton–Schulz orthogonalization and MuonEq-R row normalization”, but later the stack list claims MuonEq-R happens before Newton–Schulz. Please make the ordering consistent with the actual implementation to avoid confusion when reproducing.
| - **SP4096 BPE tokenizer** (from PR #1218, @clarkkev) | ||
| - **Depth recurrence** layers 4,5 (13 virtual layers from 11 physical, activated at step 3000) | ||
| - **Parallel residuals** from layer 7 (separate attn/MLP lanes with learnable merge) | ||
| - **MuonEq-R** row normalization before Newton-Schulz (arXiv:2603.28254) |
There was a problem hiding this comment.
This bullet claims “MuonEq-R row normalization before Newton-Schulz”, which contradicts the earlier description of the masking point. Please align the ordering here with the actual optimizer pipeline (and with the earlier section) so readers don’t implement the wrong sequence.
| - **MuonEq-R** row normalization before Newton-Schulz (arXiv:2603.28254) | |
| - **MuonEq-R** row normalization after Newton-Schulz orthogonalization (arXiv:2603.28254) |
| "val_loss": 2.67020395, | ||
| "val_bpb": 1.16043988, | ||
| "val_loss_std": 0.00764948, | ||
| "val_bpb_std": 0.00332438, | ||
| "seeds": [42, 314, 999], |
There was a problem hiding this comment.
Most existing 10min_16mb submission.json files include a bytes_total field (e.g., records/track_10min_16mb/2026-03-22_11L_EMA_GPTQ-lite_warmdown3500_QAT015_1.1233/submission.json:9). This submission uses artifact_bytes_* instead; consider adding bytes_total (and optionally bytes_code) for consistency and easier leaderboard/tooling ingestion.
| "github_id": "X-Abhishek-X", | ||
| "name": "Cautious Muon + SP4096 + Depth Recurrence + Parallel Residuals", | ||
| "blurb": "Applies Cautious Muon (arXiv:2411.16085) to the Muon optimizer — masks Newton-Schulz updates where the orthogonalized direction disagrees with the raw gradient sign, providing ~1.47x effective convergence per step with zero parameter overhead. Built on PR #1334 (aryanbhosale) base with SP4096 vocabulary, depth recurrence (layers 4,5), parallel residuals (from layer 7), MuonEq-R, QK-Gain 5.0, and full GPTQ INT6 + Brotli compression. Mean val_bpb = 1.1604 (3 seeds, std = 0.0033).", | ||
| "date": "2026-04-05", |
There was a problem hiding this comment.
The date field is just YYYY-MM-DD, while other submissions typically use an ISO-8601 timestamp (e.g. 2026-03-22T00:00:00Z). Using a consistent timestamp format helps automated consumers parse and sort submissions reliably.
| "date": "2026-04-05", | |
| "date": "2026-04-05T00:00:00Z", |
| import lzma as L,base64 as B | ||
| exec(L.decompress(B.b85decode("{Wp48S^xk9=GL@E0stWa8~^|S5YJf5;Y&G8oLvAkn@VT6Qap3bu0*kgCR~YUqB0W9R)iarr*QtEZpesGY3>~CZRiK|6Dwut$nH#N!RYqQnA}G^`ZsFO;ar92)Xt#3E3Ki5S1}OfSx<=$c<4=h|J{kt$27^CQ01M+lVgZ0tGgX0&I*V@{U&JgYc0U!(4F-btCy*+qzv6Dp~UW!y~6{U*}y$E@2-R}vd?t*s#fnDO{!j>OImt34A(d+9n>hnnvzmd((<?7SwGA+W$Or}R^?z{Ree`d98K#2b?5_cfZ%Lt17{dO-GN2&5;3ECS=cE#O*hTiZtOgX4XIkBuR{PAJ5$6e;Eel8-VVQFtsWBkSb@w{YAVOHKclDdD}`(Z5P0z`*Zpr!j8psZi3dj0Awg*|SM}VUSWz|5#6=0zD3_4d4q0&x{n`YrJE1HXz*`$RPocfD=uX!IiV{O9O7|!NUnrgGHr-RntBL^weTkpGrkcD-11{fi$WX(*bn-nbgD6YkvCezcUG_I`KWJk#hl}8>_D1Cghg~(bQ$Yj)>!Y%{*o9ex8FWa#U)!OI!!5Prl^?bnBX2V(=(Bvc+CvGo!S{LhLn7pSsR!}@U=OBW0)h6IYneQ1{|$<&k9TS^qGQpb-;#vEPAl%11UF)?6mtC8c04XzR$+h2=j84E2|i`pOEt$uyM`lGs*ejIF-}^SvSRZK$ePh1`gt+?%1r#=OV<HVjrIqeJYGxh%pSpPlDdQ66RHQe_t;x^R`DvG*6PkO>y3pW`{ofc)6PhQQRP|_h56zl+sQ(le1eJ^&&qZxdGb15aOb^-R1ouqi-H1_w|H;g(()bKz_!0+L{#HFmtQSw%~n|MX3ij_2{lW(_*6gdIz`XT%tzkhK-k5tAu}`=>u|z+<A5OWlS!1&|7BshLa`v33WEyuh5cV6d%c)Zp^vZ`+%XW7Ga$QNikdrw8}teN{lfKFYW8jLKpUE<fS0HI?!#S}z5(PnD*$%*&e)`J?Z?l)B@7}E*EtBQ<ajK2J2UVwA>|uP4UnMNw^{d&KAm;P6`40&zphh*D=e*8?KGZuo~y*`y#Wg}r(PV<PiL09`OF@Nh{a<w{zds*!NOI%lISf!q#(&1k8wXKJfU5I>}?J$O<io<ujGSX9NWFVj^&nmc<AiUyr-`}m?om^MsCAZ)88TX4@F=Gr{ba+%8mF)D2K;PS)r}K`0OW1F4MRM)*s7t_jmGr27SW%DBvhmeslZ(N-kfEgbZN}VIhl%?M7XjWH8wf?lo_X|5_oRRVxUolXyMc*~(GxY5pkyJ-C9J))sNjfO=O;m+FUfndz~U9#}!)KfJsF`v`a^5o$1$QiNNu!W#ZYR)zDNo?=6w(J-Mr%~Ccd@JvU7j6*UqI1G_U5)xIOVfFefW#-2fZay)3C_@H2WgGU<<j3Y(=kG=f*7C;k&v(6y7C}?)ki&L~IiRiQ(NiyD1>ae&vv%2(eb|cSn}oTUc1&B%M^B@xlvn10$Ol5{Rc(||g*`0AO8zk14;L<B89tNMDJH<nBP9kHyG#H@%Y{;9I%*-8=cLIDmul}ETySn#xzQAAJ)H3oDC_09q<t7H;T?vgXJnFe2W2eEM#4@zx^{Q`RI4rmXYqJ`j>7H$%dzpg>)#q=gM_#uqc-I?vb;GXT^_H=R{GSK>=VuvqSe%#z|2c2YlT&*kjZ2;Kl!jcm$Q-fVo|ict&Ja4ywb`i8dvCk+h?3OCDUXX&zMsrr>Pa>PMm^gxGuU~bBkga*GhQ$y8PeK(vV5>_veh8WQRi`fT<y2?%qa)RDFxTOGTC_Yl#}yg8C;;?WCn_iwWXDzN03qd_<{4@#oI{v^Qp5Ta*t1IOY_9!%73?T>)Py)OBaHdkXS7+2I5uh#w2)7T@22C(Qzp-K-tVkOU|oncF?OgMCg#5sW{yA4RZO!84eUpjR$IRG8(QIv>WqY`&obF`+L^a;Ik$ZPqJ0w+oVi8+fefP_6}>2vq29M(7pWSxwX%h;ULwd}`?78ERR(NNJ_4KVp|BN9>arZL(Qm544oVRUtWw?7Jk|KZ1C{%$Jv!T6;U}wiQVVE)4<71$jhkVz<Qtn`)a82WW$q0Hr{4+UPsbm$mWoflR~5=rA+eFXgkwC=?_lCr!4xuwO>^+^AaNS!~_W(vR4{WJ^svyxk^G3sz?R<rC;XSxpV71HMoESHR0<s%rK-;z}#rRPgX{3Q(A0Wp$2KXLI=1lOC(?@TlnLPSgNk$`a3`*6ksGvvQDcbXT5`hv<KTw5Y2&yEeu8k!Vh|_EpHFWU(bfUMSirttllpOsLyWqmB4^^^xE{p{jc~83a5_Lv6FW;@4UlDMPfv?~X~7+CW&3H+kc&D_4ToujfL`8*93hUr+a~GGo0xNE~Rg0>Yo_N1#V&d43t{_A|@#vLRTXJ)mh_LsXCAJVuNHeI8p77$-9H%-!CvcwCI8NI<;c*6{I5uPqIoTdj3Xcr{R+d)6lXf`Tkt(h3PL4|paL6IHOjh`kZhHjzRo&SJapcs5d^H130}9Do<&`*(lN1e>3UmSuApd3~p1>+iw(L!(EeP^xfD*_e|;Pwu$5EVQsNPN$X%#{QH-cBp=0SJ2S&fh>5{xov2Q&Dt8e2OD|)&mt5}pMa#y6QbXmJX!S1_u;C#vl1c(#HwFUIq)H-{6h49qdpK8DkSGDi&|0Gm*`a@fNS(bc?SbL#Nw1qWH_Sm8H<BSYbB+7>5d?bot%W2!a%_xL$l$wZMM<P2S6~e+Ooc>0%5SQBB#7p$O|$EKTZ^s?k{J%APQfIG>nefD{BM8Wv~Raz5FG};~ZrIG9BYxbU)6FnJ8gyh&Uq4Ujyc@m~D|>5=W&d7!2;#A`)#%c)a{*uY@1mteo&`!#O*c-WiWMu#FM+$tI&t-Vk$6T+lan2}MAv`?+7wse?lKLKyL?o?r>aEE}`ezoaZ6fF6dUE2tY2^trIgC4f5fCd!?N^%$XSL$_1XK0UJ!A@|%cDxnG}8w&B3$6`#1#KhW&Ql2C)5ib>m5Z=M-qv1|o$Q0|W(_^uCKX09E!%ojzZI?^Mz~TizB4o%X)QO_ql@26ymjSb3)uBO=dFg!x-)Qn;kD+UVD&>jk<8!}Ec-D__4+R~K1U5lNK|w2cixht+FS$0V>V>SzHs(Zk;OrtRtI3kcE`ureK?716@eYVye|+&bYUx-7g@AdF@_<77`U8(GXS0!&D^P-u;Xz}0WKVcbvGrEFSfxcV>TP{Zc-rn<riUyaMV`Pg_`~y%0<DnQZefADy^d3T|Fb4)EJH7Da9D*%i)O4~&hH^su=$@?2oQ_!TUNFuhg}a5zjTC?yRG$bdHLuKhKnNF`ksq?@{fV6I+4mIUygtt_wI%hETo_9<%eR}B^m>}A78K>*`a+rhV?lldBS^}%@}u<;Dvw<?Gc}<Wygn$lO0Dj)J4<~x>Qjfuz0#=9ImWl*WoNvxZqR-cYjGYAGA;O^VeTt+L*r29&lp&rMOX=u*JS$iZmC|mYP!1l_3fkt4g*|<MTxOq(euNH(JFt+d(QKMI?zE`8|xAPT8z>qc@{!A}d!ZdZdq-LX&)z*#yYN+$?Bqe|01W`-A|R{90OA;BuKH7-D8K#4(MtnL@`>u%l)7Wo*3m++MZ63TBcQJ>;K$x;rQt+>)>cEy7f1sq-E4<n`1+Up0H0*+B5qruY?(p~nhcRcp0Y<>-P=o=ctbBsP$HEE{Q{GTSMN5dk@gm=-fy<)o05o|SSOL_PN>QA5sw1_nYpza?AifEtB`gGmk_a_KAt7OI9}fdzZ_1<re)5zJwY>%B7De4O;fOO3G|`*hODWE5!{={Lx~<E%MJ<r_R!VkF85otI5gE?oa_d}oX!6a&Ix3mK`G;I9q*OO>J%N4q)qhTlK<c(3(FqTL8DLxXNLMoD5p?`8T1?m0G+SCBSl+4tkK0gruwl-=7j`1t9K;nxuA_f}P#KtZ?SRv}H(zN2^}SwNP2+$W`<Su}6k;sB#M<Xb@*Vu}TA-FpGD#+~J)ZA?kucvS*G`F(*2HiFm(e2q#KMtIA`-I)g-IGfjTE{-eU)P>;^Vw9bHSA_Rjjp~NFEtw76&M^}{dD!n3d^>P(=?OwyW56Vh-Y=t$TUe%*lb*hV-ny<{h?{d1B!^>a_Q*8%Kdqz9d2T04mMyR?`RR|!?x65zhQKdmwv;0x;sm38pzc6?(-obP9jShOmmBDa3go7BP9vbs9e&i9G9It6$5xhA7WL-OStpny<`(&kVy4?EY97TS{CRW^|JV3P3MOU;M}3+-DhEnhy3!eIqtF+S-{R4s7R6+d;<+>uM>@JZqKZpVUC=+lYb!!T&vJQ_3;B(#xqMak-T8X!C7Y3r#O#;_VW-k*e#2c@cGI4hYrF*;PJq<v2)#o*K!M+`;Y=}R_BBZrSx1Hyg@>%|7492kH1(f@a^mAw^}9vO?<RWoU0{NxgAM%io;0Tsa_a2V#~aS*!I;#<f<Q08Q67gIzL7B||Nepa@^*;7pdD25wTM^~zz-Q0iSjlR@a%yFv(!7tJDoU;ALlNJzvm={V0i*T`$u{%|2XX@L-%9vj(W`s2c#Y7wTx5z4lFl7L>3(2y=4XxWf9{jAevrBh<|B#@>K#!Hj>cVI#kk^TT{*aP}pUue5rsHq(8L+tdb}!yS8}4mQ*vPU%%T>@i_qMFbLb#8SrKL(y*BcH_j`YW)jm|T1MSK=p0r8lb8HXOC#uTA@^z!q4|?60<sx5#i)CrY;b!Lh!+-CdnXiBmuZAfIHedsX9WV+IizyAlz0{S$Eh0W&k<@)ubL1D>;qd!ah5v9(Nj8;|4c%vp9p0(MOWt_SM3Z@xfoG$eFn%?m)Ewo{?^?%<SwrQ^^8=>@mxbUbKIBry{Rl4fN>E4f<&q?gs;fV9T4u(<QIJ3E4~W2fs-2}V-OI&y!M_6HHmh7cu(r(c2@&Yk@xBGFu2^xekaTWw5a6y;#qFvFdlLA)%w8NDQqN~^;hk7Q4+K;RVu$=@6B>+c2s;SK)0GPc9EQp&loO%TJOhA4v<l`MPKt?$P8uHwl2De<8sBx<y&soWr#X#X%e4QDcs~MP{$oj<QwGXr=^M`{SseWjm6KHY6nBgFKFqiJ~;%6a<9%*jBohCey_XFoEO_iz=;3oM)iA)8Gd;fzcnV~BQ}L$o~_eP8j$oYCP3a)f-rsyGsu*3rqiP8)fcF#=gm3V(2II)Q;0%Q0+y4VPE{DM&NjzF3r6rj^HP$5yrTqibwFgRB0R1@$FRH>1N9qwBbGv`rJ^aCf7kmC-=jfuipSBg6u15TPS61~aBaYw-4Prj*JmPofD{@U68LcjjN~8kBju#C%QXlBhn0MBialPcUz#yBW?Zt~=)J&lw9tsUc^N|mtb^%JYFY2W13!k1KuU?j7XyhzV?#G2;gCs%Zm(iwj2p|VujV<~EGevXYPk5X%8w(_gPL&n0zAHVckP}#);*Xh;mO58ylqV`ZC6wO%L;v|`lL#+2)>teaL-;>%7T}VR@5yN4cl1*?w&|^WMNODQYk6`wk+DhUoiQzwI>y^eSjxVVg6+F6hOjKO~fDDG!y+o_*|8~{-o#Pw>P{qe4;KQ`Rs__H^q`xDV!w#&udFw$=tAjCBbBb*su`G&DDR``?6N}$lV8pjuG?3g8gnwn?q5ax?f513pn?ikJAp6*kZI>*f2U9?6tbKV+);v3=^Pg8o^U7gfQvB+7ix0>uMtUS+6C%1>y`)#GbjVDTP2;2^d2UGZgpB%09^<jQhHd36ni+*;nCJ30>vS#`J`Ezq_UQN}^0W=SHy4EKXmYhp=7HdK1bmv;|9;Q8V<wMd5zludwR;V^ifZFDnK1|BsOseGV<1ZoS^USrD;zk69GPQ<de(rRT?S?fK&1Hjpk}v9)Dc3>W4CDnMI6gAuD@RF1i5&4-V#f#5v#(Co$06y`*OkgGr#-JIW2n9Nf3azCs+jI<^h$>jKVhO*8_6s)dcRHzwS&VnERy_r3KW!w1!5wA^R+GRy?bQR1RTM7ce#4@k{4qt^hoam%JU)|NXu91Gz>+Tiyo1p&pQ;C)O*w??j>_q}RpBFq0rX>7FujB@8=e9f}5yJ<ib39$#8aRTX7{~8a{GqsbsGzDVbj&pp_A=Sr>|!L&*+cx0?M(g(%c0g1R%n2?$@m$~oJ;xH#bzP(pzwm@-WzsL-btFU##Pqj?H$yQP@j;Dt=dnyc7!OYUVL=aD!I3HbP;3oT43ai-YjpXM06`E`_IzF-F~}3_&~=M&c&26Z~vL;DtfZ{Vyb<v<`v=2Ee(mkIQb!9#&fwUOgV$yr4Y#j1w9)O@?nZZQIAwP{D9kE1fGCpZ;EMk0{N|rQJIXpBkd>L>U>(1^f{=xv7Qcga!kF2$<qEs$ceg`(=Upg(?oro2zNYX#X-&_R2^DKi(T2Q?1$jEi}qg&6>8=KhH~AiaAbPKX$;G7&OpN%z(X--EjXO9%h#-%`8dP&P<8-Y&;I2`f%XnQg!$NKLlK!Uk&1NSS?X$iFhfNf^Y!B82w~(EGFYZ&t;R^5Vnh7Lui5tdS*!kJaRLDXmj_^r<JxvJ(H?PZQ%ljiJJicxuFdcIrVQ-h8gXU$;qAUTI5e$EGBqH|MH3@7xXs;}uj(Q6Q!-LALCew~H&fvmjBq%#Z(&bgxRHC~M3W|p4LC^ykvI=NhB^l#vD*ZGBbOTbJGBHmFKc#(|BNpC;~J7;(Nz#`mZe~g91kVvIV3(9DV^fUAc0(kZLTRAH2~4bh9T(<gN8np0cb-7##&}=Is#?9;^`dOp_?=;*u3fan|oksbR6eIqjAeOODU^0SOO|8+%(+yF2%c)Y~?zUth_*-nt2T?-d6Pv`LNjdbU*{W-LBXIAmr2TVHI^a@Ym})qhUoy96tDS+n}}qgM?T_=^eENcB|SH1cT4Pp~UN)vzib8)kpKY9;nheP0zBfQuvSjnS5ftZXJ}OLuTy51(PGM{4Om2t)TR8pdqX3hyz$6XCbXQ4GM4n5v_Qe%K8H1-3?57*vqU`u3)H<qieUWU&Ej0AJc`KE5kR-NCj4;7nt40(mioh8kM0#vVyf&=zzR(9})dBS336j*bE}WE0_b^tEz8ca^lr{!LbMi2vTH!euD1SeJkl7rkijwuODZuT`XJoMO-*p*G+B+?#O#)AURV%j7Ivls1B+(B45$Q|IuT!<R!>}wuZJ=DtZ0-odWL3csa!;yc=<#YVQe2NF|jlU`ZUL12tF_BhQ0%23OZgwIlaJ)#I;uk_$?{8x@Sfu@`xBidLQZoNJI#Q7{0!$dD}BxSgIH%D`Sp40N1;>l2_ht7P9bn?B$CGLB>5S-;!b20&7GKuz~;F}Pcpo$;`wb_DiVo(gZ(E3uc49-fN$$?`2{el5T!DN@CD1_=Yn<UlraHt!7dc*tKX^|3inL~96>-ksOFa}CBbdtv}0NwK7)mx-v!_$Km-<)%V)yh>6OWLuLarb&PkRZoB14*Vn-1YsCu*l<x%bM_^@E72tcV$r3B%8Ud_Qi{e@XMGK7Bh5myhtdDhilTDzv`23=fLATqpB=-w>6}2|YDWxK8iH788e&WiNtZ>w1vRSN)DMe0i>2=D$TF56DX=*VeN;%4Z3bw5+MnnKj_#NST>>6ei@y{e%MZcGv5~3A8tdc(xBfqYbd^-`p9+8{U%}B`QWb)?c3zVHYra&?Q0B%UiXRlv0f|lliRA6}ix&wQg<#wZil3Fb!IG-FbL2yj*dcI-F2w<C?n!LV1&!H|f`2PzT3J0>JFPlPRwe`|ZSSQ`x`-Fti3D9;=w|;^Yi7g0O|+}R0h4YTxc9Z%a*BNW>D<~EY~*tpq)rEv937SIVSiU>ZOAANUcK*Pg&Xr;7i(skvhJFD1*{nPxe3USCE{Q2phxq9xK<frYckg`X;uvNIxrdV^1`mO!+Jwh0V5(bTKc@A^)uO}aQzZRk!;l3q5$ZDV|3DoQQQ}dcHRRv8OEl8jc7nfm|G~~r6ii&Kn<FS!w7k^AzD2lN5lFSy)ElLHgFW7dlyD}{&|&gW^@DQN25KRrYF%gr%C6cZvCde#L~tjOLGFPPiT;dzTt)&MUGRmGR$yc{qm!(3M8uW9Z{Fhq67<c<#*nckFNeFML+9<75;>EO=?PoI+F{3uk8*kA}2DeJ|x^HwO+O04DjHOn*dQrk_pg+0E6g8ONMQh>x$A=9FK#K%$9J?w%kxFK~4L+ojeTk21n8|WtdQ8Iny5P8wsjt(c`Tkp6AtH)(tG9pN`w9AYn&6(SlO5F2=VthJ)2Q&QNtMZr1aUmvd657wnbL;J{t)nnyw4rHW+P3OTk6QTzxO$tSa<p%)psTSiFhkxmz?HDNQb`qKs`dOsOdEv^3*OL3Cq0#kMOfu3f9*=z_Hb1N5|ym8#QF-T2{X##I&pO#|hG2<vDYQKi?ZR7bi3g*$GtO;q4w%?6;o7-Gt%c0eqeQ05`iHbS`#f1Z6T;fS7J$+LE=p9li%PgQSTaGMD0^n>^&Mg0YF_<SuZ2utD3X&8SmiG800)9Edyi7Ateb?cHPgZ{gSvpP?5GxsOc<_{|nE434_6@>D8KDZU@xyI7^h?lOrl89-)ZT$%3xefxMXHH1$*cr0>k_qiT3BuuKP-o`GU{!5_AcVe10hT{ZJoED8Ny<RI*Kd=h;7qFf2mlE7KLC~$`y-Uq7_`+E1KFX%U(iniObqKKdU&C8~X|P(JmEOSETkx4LIi$cI=rOXlqEz9J<{&CN#dkgcWF7EXC%b2ige`?)~UHv)0MOBdJ~G5)f>}?XYn%Pfz!8`qt1_+X+>KqhW$>u<=OWkA>7Z5N}&?;Yr@L-@Xk-iB4`I=C)6Xr%JvH8FGt=i8eY1-!o(j@zK^SwP3I(hiA-%=dvP~*?XP5SAeVXWxROS#`E(}=WqY$s+V*V*0dsrww7##U4GBqeio#f1DhHq=*4k&7W?h%DIz2=E7B@VSJDlH!{)^y>V~;96_kVj^FBpwB7aFUK>@@YM$^cgP(3K0WYt=tX@>uYcYyF%0LC;@=m~P+J~8@w7%1sWhUE<?C^g_G%dKg+j~*?|KU9$x5*FA1A@yYNI-}sT?7*vDLe%#vPe%92*G5t<^`BNKeXSGh(L-iRNqG6y8utDuMTOu%9|LA<$$wka_OU>5nSW@u9CO-Ult9~?)_nI?mek9}&+_x@;ZMsovf-+kS3c;TSzRV5hSLdd6g7vzvrjk^*iB1QXs`OQRPyT7h~Q;r_jgR7)Cri<#!w(=$C}(oW^g?-9R7o-?z&XH!twx(3S5UCA@wKqAW{K>CZ7<1@-A<2sPmr!Q414^Z##x8XkU~ka-s4~D><-kjd9W&-HX3Zi10J<8nrMM_MNnB4Xlop<VBeb@uS?a_x84Oy6^G|BJZsD#T6)PYE9^}U{Gpj$A@>+#w)ZaQD2bXd$$&^8SG;?B}q$FXyxv6DZSDzk^{*9Pcew^k8EQ^a!)d~5eP7cLYd;mS)q(?7z0)-s2A+#SvfZPY$Q`4jd*fCDj{bgc1u8l6T5Z}q~Cx<sb3wocG-ttuSpEx8`NMxfPR)@`%YuA;vb0wg;9WSJgAd&0>?-6B9#T-SC2XBEfbo}ldQ2cQS-DmWb$XYq2oPqEh$Hd*YwIT@OET4&E;4{`YDKFuSLBSh`_pw>KSn=G+Yb{ok)vCilc$dR<f(0gcmUA9%Gs%Wpy(~0XXW(M-08q95xYPZUJU3de;f2eyfo{?Uv61??$JsM9$D8dHP3eCNd9ztSZ>~EBwU?=QE%s68S}9)$w%wt!m??KIdT#D}x4U+(Ku5;(=kvs@npZVLJW`jww46>Ur3JYHqspdEH}jpoYQV^4XJb{F+nXsQQE6Sg@=JJ#iWhz%owt25(#=2LS7V124jA2@@o5pdShumd%$wONbs>ZaIhmd|C^<I!G?dAH+|4y=Hou)517|bj0sR@R9!eH=9eYcoYuHRPODg=S6`6XD^$=Ac%m-8e>(Pe<!ov^*a)BzYP_@50Q#NR-kRRQ-(r;Cf)x}ZOS9AIPy`M%^VxqOLY@hG-qbtVRN2Aw1R)W;yp0mVQ;X_p(b6^(pck|A)Fr*n6K}1UE+KD2z_5t$?ZvK6j7OkA`fmC9EmQX<<x#_-JM*J_fbxh;YE4!oAFp403jN?qizvHtJpE{QU@JV^{x2+x)A)lR30WVy!!`T#J346!2Q>VT?Ji`ak>&d6UmLP_4a+CPup0EgSJ<K#`hA{mK8M$6}2OyWr8Xg==y~?duj%bQd2*o{OS3*Zs8fKUZ6qX#}~SXDrcPj=5Ub+u6e7sekZ{D`BY-N-%m<l(RHijfQU&msbn*yDMiAPxz*Zjge&Bf>j7S)@P5{p*~)sHWBPyWGQ!UKnWa-nGtb~QF|^N)R-BP0i2y$Z#%r4C1zL*ru$b*v13olOCX?P+B5A_5rpB%x4_bGn8&Tg4I+;6gOafGz=n(!J6BJs1+;E0<ohnWv-5M~SFl+4{Q(2@BH}@wBVUg(=Ip=GR1a6dJ_X|5&c4*$*Z@gPX9Gs`Hk(AZbqMs4YF(9@hi2;-0Qc`m&Vo@bmePkT8#*|7y$Z0B$ij}`w#=~qwsBS>)Z+Q>5qZp#w`N={4#nUU+u=Z=57F)&(9UKrh{cyh54$XX1KYN^N8*h>g?p)kM^Y47M+fxlNW5gGv3|NcSv97OK{o_NB@QXS~)=#Qkk%oXB^A|vu(^$y3j7YF`EjQH1Ds|XMUqHC}D?Z-;#x8eGpMreR_=olNY_Y=8>|j6l@7(pJl=)=xm1*}*yK6{+Xqb;|0Hup&#{ixQA(?x|6b59@9hx~sg!%wiEhR<ap*Rr>g~>%_eoFu!6lWBannefTaq!8vMw7g)UPVTnDG2TaVsau50h^X&wHocL<j+a%JPDu-Z%)=sog^l5={$xh3TF30+8=9$#TnWn06GBlnz9_piNT9#CvZ5b)PMn0kugj{CxvlTVFfQP(0@Uk(`sN4`W$n3ei;pYSjD-Yl@sj^m${nK<(p?B(Fh&5_qknB9APPB%%&dCPgX1YIBrLj+}?E%gB@|U-Bkj9=f$pr&1u81b!zeZ6CUN-AS`tefUrwZ1D4BtGvhAe;>0+)5(`bJAl6y#fV9Bol()sY{)(jK8KFAUK>*Z*yak@^q*8dPUPU57H1#Y%K!sqg?JC-)zV3KaWA`>7K$u_kv*8U*dKDq4*j^{9vAS0M4(C+!7jq<kVC617e1_7gNm+dS<ogUPj>64!Or5UU0~^C?GZcx4iXWMu%WUh^+Ubv+x0TosX!^@GMRU@N|7@7ho!SiJVhx<d6!Y2WB&W&j_$I>7R$U-oX~$Z+Fij7&d12^?$7U18*68VR`DyI*ppL{7EH%M2z4OYtHpJktnN<iW7LZwt|I$5Q3smNWF&qg>;u0kP13K?Ur%zkd%_%T|^a$Ev*{@sT66<FEYH#cp{Z*a47;rfnlc3Ex3blwnlN>vTlI~cg6T*eVpQV$%x+&ha5g3CC?M(H*>Ffo=p|~t^{C2wGoPU2u@HX+5Nox6?f!xZkV988U<I#S^2(uE&1^bK$A^eo$G>r*`+)1WFu-La|7RMX~>1~A>E#*99R~kKB#*#ls+;TjnD8C$eXUMM8C80OMo!AR9x2}DDVG1=bCF|}{W#en=N2EVDDT?kL>xx%XnqtIVau=7K+U*>p!5Rz_Itk6*4V7#;q{gz0ghmy!;Ymy*?*%2LfpR~n+}V6<t_bQQLdc0|I9E8$%8ism<{k<sWQE_d_ZM!M!q&(NY5R@;MrzUsk}4TYt;vvfqff_kL_!2;K19=>7Q1sMNK<IMtwOkxIg+XSy~|0~sr^AWUx!SyzEzdaN|RXZ?YxD(Dm-S>{a2eIeVceLa*f^r%o5?8^x!juVk7^uDqygDTKryJx((q0B2Q7xpRR{j*cg*>Le#k+grSNGgC3wC0XEPWXF6wCjWDE6AKYlc&icyb6~XXN5wSS`kU9h~y4Sk8=|qM&zoH!_izs6XOQrHoYT<&>Xf#wfZTk-eWkgPNdd?*)!drp~(noHP9KCCSo<dvW#^j%X)oM|P7b&B-D_}WqTz}bFWXvfsVnoDx?`llCNd<piA4){eR>C0@$W@Dji!MrlOE~w>Kmg`=&^Q$W`HgZ&OF(z|BA<6rO>S8BS4)_IE#(0(h8nv(F{T^;qQJ?XKEwdYn2v*yN`O+3mAuwe%$F~UTWk#n!M?H3^Q|Q*&IcZH@*BY<_Z@9b-)I&8Usjv0>i+bMz%k3IZS<)K*tRKadBTPgZbD>GVUgNrcExfffBL@Rb@<OS!w^-)0a+rrGFz-g_*-dfklF>tG#11lJFggtN%xl7#>qRk5epdZMbSXr6<hf0PN*K+5=vBFBY0_SlwdpJS@GA)y@^*GwlMv)`FBetZ^19&Fm_@v3a#B_bUZxo`oS%LP+8#0gnefo9IoI%R?i0)y&Fc#Z-+7DV)M{L0ZtDsw@G+p7%*>XpbiGZJ8Nv#;RS+`Hwv*x3Y{4VqqA{AJsREQMaGtxaaP@C#gl6QfADnp<HB{s6!rMJHafFi)x}e6BDlS|RQj`ZnDY0~R>Exi5YNjs|9P|FmIF#f`EFeUtCUghxVKUxnj7W)*`WC7WJUY8uJb?s+Q=x^mLh^oKk4kZC=F@^J|XODnP~JuC@|%LfwLji2Ejo2xDxC4w9u6Yyje@-wdftm6aA3?*iehG_0lG}O6E%;DJAPV>`DiduKqNJ-^FLR`stC-9O%qcMHd<g4vRC|1PIR#eoAUv-^38HxrBNLX6ZQY#jl6x3H<NS_ssX?t3;*<xiosmp0bS19;XO+KQf93#yCIC$2a8`0vc(_-eKT-Hv!S6rV%=v;vTL=887{_;5i2>`;b55Wk!jn=&|PFU9<?TG3kR7<7dERE2PK8-VMR16#)zRe4{1lV~gG|9`N~!ZmkcZ-bcijNwTaUhSMp{cUinh*V4syk`M!H_67w{hCE+`3}-gzxzZ-o+GjrMZq8PhAr_08YwWYhRQAR}wauwBf*vCs?%Gv*kl*?P<;;h#XEkwA!z8U-I>g#N!8jgP!+nj%1t9P!ug+Kf>+Jw*&~BfS$2mQ=b8l2q5VEGH4D3Lw-c)7cw0T$lS-#}>mk3lwY;@W~0r{ecE!PEIaY(&gbgw|xtDMEoMV(7^-r3~Of9udbmgZmgo+@)^@}L!g0FAHtlierF9FOYbbpO(^1a(!*H!&#EZ65;sLf(bAi5Kl}@Y5N+rYZ0M0a^1iEoSiUNg67@Xz~x*ht9+`=-{3XDi7PCfEpYjIHLOGq%SfAM}SU&3)`a^)^yM~-H{;sV`!J1mQMG5Wc+8L6#YXa5dr<PwLq&du7L`?4B?AoH1ZtcLkEIQv1bh$t{gll-t`l`A(O4gS5r1&Kd#*@(&jx9r8;%mS#iFyhWR8{&;-dXR2h!dYqUv&MXf|^6?5xX9QNA0s_B+rH?5V|^B}5O7qKu%seTgR{}a||9#%>=YFTdC$Si!Xx<}1AsT0Hq`7;<LWQ{b{w1foa?)^|R<@nrPa+OZ8064K|VrrrV;l0Rz=`9o14HVToQS109RMq83e~*IQJsxbuL&PmJgF?ly6V|H@s;oQ*%u>v4vA+21dgZ{-&Bw=ma+cth|H*}68(jpb&Q8mgVY|Jfu1z3S>Zu6m$dWmR<QOpA<>Zlm{+BDS*62?l<X(ia4Vu&3fXbxHY3&BfBpo(HBJ`2a2a#Q@G=8j!H-PvFYEi&MJP2aOG!XAP(b(ZGw6L5~j*>uBwlX8I9B18?FEi7{VkETFRhhOAFw0L6hs{Zuky&w@d&j;Ta~;J90}3r5E&&Y(9x(C&$Ngnphp$K)>e)c?$2Y0S_>5s@*%e@6Ycy^;uyuiopogo!I}DCi-z$2%eN(odm8K%JlTJpC-z{hT2u{M$z;Fq+ppgi_E>@bJz}PNik%Jnr#``=Ly+itRGh6inv2S**)wED?9gj<qlgsaY{{~_qv38iYYyoyn<yZ@6n8jdF^tiO@kQP-6Omb_e2hp~op_PBKcVWwH_SwTwI6CbQ+s|YY3tM8)$g#@g;C8P_dhg2Ga$sz*i^ne%C`EF!OyP(SFU5)h?hqc0_2S?iO1UVZrzYOg(nOIX!6a_ns2KHMllV1{SeVGAKxk;j0+Af%(;xEzh1gqzgQZCwQv~#dHb@gexinIqA|!=YdQN+N<yeTeW8r(|skh(#s;3FN{{IKUChwYfTsojT9`y8DHX`^N0<zmOPptNrk3xp8naa*jGipW><$N8@f$c3aDhI#W)G!${06_H}xgYo;bYoGeKvlCb96J5=6q;W#1s1_<{I}IPFgAd;5c7&UyCucocFpoEjJbbqS|Vm29k?9T%uC}`aca@&Pa7T1ded`__$jq2qyg9U=X7$B6@DhK&J6H8O+8h~`EG7cB8MB|JKM=1ayC}rht(X#q0YjL^vQ4Iag6&fJ<R0g#hTK=LRWUUd4NQxU&mh_FzZwGPw;y<Xt+@NGJ;AYP0KHs;#bLYMbE3LQT4~Us9p$w7;`)%YK<rduK-~AAREa9Ax3PK+%`ux<0S?ZwnZBa1@KImEZYrI53*PJLL^hyP6DdQ#y^6j-kbu=mWv!gDG*l<Pq~jNkqcIoSpWkuobQD>$Y#o?SmFMyeDt{>`HaX_!RDNqC%=o^Ou}PP8{nrKDer7BL(-{m_1!pR)@^qn&kCH2nox%~4jk&qqcBoeYOP1F6obHX`mHWIKA8yB;m8jM*|)a4ma+yKR$Q%%=uNpx6kk|JVHEV0?WqgeaQQ2P3lVcduO@RjtYH|~=t!nX+5AXZ-6t!5oiq{<R@)|@pTz&r93gp9!cW0hZoXgVa&}g9lO5{h$Jb{YC{0!ioho3`M&N(OrcJ|2&e6##13Ei~!A)f4!Peh-sq(FISiGH3Vs9$HB^;nMcEUM$ZQ?i#Su~xOsyv$ygzFYXFGDj8(E_cshW#9??_uEjI3twnwf`YzuGUZUGMWBB#?zpDWuDFFTMi5YGga=)LYB@`q>|G*Mtw5Lfe(Tzo;aZ|X<{i7q4R!kn9lRP+a;u=WVW384KXHkhDG;4i`$dyv8ufX^_>a;(>h^ZbmXXM7f=q{udeS1pAa=R;|ejV#Q<Jdv?MaH$`Mr$2uc07Kl})va(8I5=AJvdv?_y?sElhxWM`g30Jd%}CNQ>b%2dri{zUQA8ry`edYlcL4M@k_Y53fyi{z0MOmM`MbSw9HM4tU9I%(eh$(Og3GYtK-gRV9b;RvUB87x>QE%YX$p#A`=mMUus5m*J;-Y<G9CVHG#TBr7k@?%rr7RMKni&Mkjw6f<C{1@e5wEgv{&DBaRXZJam3XU93Lr6%0L&<ERX%cC9(M|VD>Rkg$A2U^U&qs(<u;Z6ozL-`&JTMjnq-bFD7)Xd1GH*Z~AUF$CCbo?fP^A~1ipp7pgo+7SkiA1}rrffE4DJC-FNE*3aaycFY9q{dZnL1~mmVH&;L=8q^kof3hA^r*Jx7szm;VEAfE&=1s!;14u_*JlHaZg`>j@yy93y?TgQW1gjEL-Y73l{;2(G7x4B$Vczf$NEG=6_iqc2Z^oyP=;=wGSuFd}*y*}8GD-(^M&`^<8}JcCCG1JG&&MzMTVtrqTa<k;b-iL3<hExi&3&<J8%_HpK3<?4peC0W*Wly>&1Qd^PhoRp$oBSXV5^#7Lf@O%}kcL{0~OZOh?m12TuuhaY}$Xt7Jg9Xg9FBgBFg_;P1$vh--dKKiL%L7F#<~3h(6$!$bogWE2AP#dHHw5{%YG5DTpTn!v6Xi^g;YOy#=6xD-@fqJn9(+#ip>3)i14N^@TM-f|xT?G^U5y*UJUJ?ddGD2pZ(SNC0T!)P+A&_IpU$m)#8MCiI}%bD*V`$}=kj~P!&ZOgaZjnd88$x=Mm!mY^$3edEqLxv&2P@*HZdBuigIgCMn}le8Y+JOo2f&zwdpsU#`d84>S-u6w&5Wy{I6wSw#}GJE$ld&8FKH;h+VXKSfM-{LUaluPT9G@BkCI8<LyLT@eHDJK)RQ&kW1GjXCW#V{>eAyC@1t)pJ?Q6-!6UX8qIZXME?JMW(u^0IycKm{4N7I<Gkig{TgWl=3VuvijH8%l;r8hYA{z#xw?}g=S<HMxmfgz2i|z6sS&c_XqY+~1Re0O>JqN-)x_)`3l2_cEIn?+!GyyPBRexQc-Eg^coxePfGa*ak=s!Inr}R)-Hyme{p{41^H3mOddryaaO{Edu8S?U4Pw)HI}98#CQpxxP_xH2vev>Gj>H+Ldl~&KO_pe2lX+ABY6!(1BoI#ms*SOlCt7aHfX+hEV>%$El$E8i-j4d!&*QP4No?g8b5Sf&zU<Kovgkxaw|M)Cm6QT%6g`jD2Eqbo`N3ta?&WpH14p@X%NUUz^Wv!p1OF?8+{@Qd?=vTl-@S>v+V&zzhVG6k4^YmXU2Z&-=Ei^c-(G;|^<76TdDx8vG|Ei+8^vhMon)G7<92~EaE74KwVg_i%7`Xu@sSG^vI1f-Fm2htuOj0D^Zugf#&#;}0wb7NM8wTf(`tUYNiaL+*Pl)Yy1N@6Pm8V1dSIF71&A*u53@=xwDE2|(eO54632oBNOJzR3BeIpy6D&06Ok>a{ho%QvQI@jpiQ}*_lTGwX0137={JKe!r`ojvQ63de6Lio0lJK0yORuCheen4W9#C6mZAqd6R;y&1-_h&Ib=WX(1be3<N$54H>-;|M(=Thd?joY&Gv>>+coi%U#IImh`$i2Li*?}bci>yL!6cX#4$GCw^5W26G$2F+B8K$iy9mEnv9+?Y{vxeSoTUcuGTRopyXbDdWc~oTN%A9qgDz|~_O%TLt?a{=9?AA||iqCwYPP01DfOp+gR6N`gKMd@#kk9p>y`Pkyd_c>z49QN$`N8$PUon8_`qxs<lC(DSUVZ^qJ#(_|1TB3+5Wj1oyN6DnED@70U`MHDFUV0B4za1Nk5}g23uGCDUpe(x5x|p7H;pZ=!f0M`^|A7r{Qg1Gca>$6R@(%_Gclrq%pYEPb?jk(7oQG#ci)}IEIrv{<(L_3xTHCO780rG{9f<QtQG3F&u+jqp!+0`LPSCufCqAIdMeHT-(Np$(6d_r2yed5g^D-yfSnHNhZ{StLSnTC&QzccdVgRXz^5s~!@;9^sIJp$sjTo8dC4RKZkYiZdHpoG+aMfKLuxSh6tUlSUg9^`ZZ`Rx1_y#?kmH=!eQ@*8$^*+LhqvH+MK!C7@w0W%dNQw;-yI(Kv#r@3A6!|k#gvF%HdqKI??6mA>P~{mdMlYS$H%1~y7l3?*l;~U=uB}Kn6n-<$jff(A}Z_Mr?<Ek`mN+5ZC^-}9^}z$%cA0^ua_YNq~EO1L6`o9v2f#$_3dB!aQubvI;@G=23-6-ov6XG6S7%Xp8XnT)sc5JEc`T0f5<8;$z01@QS)+A-~<-m6y?BqW@Yl`$?O@s2PD{9#-&;4ym!=xv{}Sz(+&yVKj*68mZEukE6Ngti}W(u4%hgOG8~?t1XHlo6|X<x_o=__uI*!gN)CI18KI3#0rZIrq(9d}nOU^ILhyjKQ7az<v3J*iPJxP_VFXo}bH1SIV~D4;V%46oV0U(?&yWzTG(VA@UvSp+zNaG3O-sp=&GmK$nD+gM**~9WC<3m-!FIeykzJ2Gv+LNpskx(ZauZ?kL(&i=BXonYLh$PY(#TbdIx&qFAL~$^qgrK86@nIs2I^){tUu02<)FEAlt-!O#4Y$;C+iMv3j-2K9#*j~^2hbavwO7}By|MLvNav#wB@s?6yrL1xq@2_?V6iJe~|E@ZJl`%y)DV@N-i5LYLC-sJ1NS2R?rF`=J0vP-4!+oV(Maj^{fP*Ly<-9IYzz{Xf@=z!|GdqTZ$ITTB&$ZlZlerrqmzV9&{NUHmpjiS~S3Q=h_;m;nJ<~s#XX{thM|8k_7Gtp<}c*5L`n9{wLA+qH;<nE(mfFV7*Z%$38<?r2Oe~(qF0*pcbIY@3P7=9~n~yxnr7NIH-SDJ3R>70Nib-bFzmF*3UnDRVBctPpW@(e%4^Qm_J`=uo1fde)$|GvUHjQX|Mo@&y1>>ew*=PGW{}ynp+7zFI_P`SoW17O1Z$y9Uf{9GZ$3sb8&^aOz`XFO|r~PK^_;fJaH{zQW~8?mb!$IOZ>qC6Wn9o?!3k&Oy+p*A$F_r5O5(^hGbpr0A?6g(f-Ba-#idbEAdGc>we^nKeQOywn=x*_{;i)Wku@|7nl>pvIAbNm3i1%f;d4iBtb)G@q%DT3wt-3TFaHY4obI`I53oO$T-%(xT#o2J7JjRDU|CEQ+1)}HOjx+QZf`_4%p8H8jD+h?Z34R3jmmW-!2z$>>8S$eAYaC5=MJP^j*>Z;sbHyOG0%L+8z33%-WxbS&7P|v#90}d3HuSf5<{E_{Mn7bEaZsev~}efE*0`TP1X#toJ4u3REOU;rqiNl>KR#dADKsqIbUgn2rJ_CoozFTtNjEWKd4hlww>9yhtuRa!v_`ia$lNr+v;UB6PJMAiIs2LmSibjDFU`4f*R4T$F);qvO#7J4%7e$ACsxIZQ$xkvOv(%_imt?{uu$TCty{sJ6^Mi~ywU%&BU5cE(_jt1^xq@9$I9xe_&->XA&@+s8?Q;(mICpigER9HNjQK_V&Banxw@Py%~8eT9x|y#g20CUlOYOf-xL-Mrr3&j<*&KEBvIz6u{~H(qh>0R1f=t1oQ7=wY|;4U8#sWAE68uQL$YX3*$6pGHIz5CdYZCoa1!mWINAQVke4fglIu3p(+w0`qpMG|*JvQeqZlC#7;z9~g^ce|UK8dkIA|Pv=G0>ZOTMQSkgW%*fkq##lk{^F*3z(O8c{Ts+N{9Dmaa=hw}rAG-nm4xj%!C2e%pD@9?0m-n4ug?Ja5+EB6ShhF{uor{{wuUn%&FBOkr4I46ZstA<rnfi&r0O<^+TMej|)R;nD0k{Qnk}o&h|MGfpDdq-y{^K;nlBvrn@&}QKIu!N)Kt$!U@q;Xpm^wd1Eg7If0;GA{YOnMwNu*lUQ%fUjL=0|bERqOS!kdvOa;(6GT;b!V7MK~-$^_v;{IO|J(VIJXZ%RpvW2UV`fOq|sz+K~28#Na2IWHV<ws|756_f}DH&UU8w|f<NYid^0Zn33pz;)SxWHQRac}x=tQWGtPD;9F>0AitopWGTEXaRC+E-|OW*Nl!EZDX3}r06^h?FBx<IhdBMLBcLni*Tem-D*Uoy+Lw`Db?2y{CQx&<=|11G}^XaT8s^27;O(i=IY|Ue(qd{J_6NyId!ZA2r@3DU*LoK!0TJaMEG4_J*6a#aM!yD5e(peQVv1?{+q0Opwto_*QFP7a!&{0KQD|2uQ26Gh!faG0HPwLMSgavTqBcXYJ;Xm0beZ<FNq!3*1W!rB;kVc&4{4EcjHwHIvn>02j87dt4)T8V|RLuhZ_t7R#PHU2OVuR77}w?QES>bp0Nk+cepWW2wb&$Xq>0-SMw!@@=H<$@Am=>{mfsm8?OrLOD(CK&1Uw~U4-jFVlY9Fv{bzxz+wB;B8-S6v`tiY21iVlLBb#=&X`+x0arm2gEK>z?-jmy|6?enxKM`0!#RClfatP?;1WWa(KHI3bMzo^c#h-h=Ncy-o;lS>jTVjTG#k$;463-Dzh>D7MKxbINRwj#<lfP`*qFPvgx|qs{-Qdqg)*g>et8(xaY^-{WIq&F7@a{Yk18u|lZCiI4?O+QvO47i;+od&xH+(uRZ0cc=bW`y&8jjjB^5#`N1n+uN|<(Q=}K=jxsY&aUDWk#-LF05{e>Xa=|}^`Q>BTCuD(7iQElDBvPp9S{!W^p&U3&2R~Z=n8ISu%;ByeCF#FA5#KcVaev)jy>Ez0A>qe5_wK7PhKSgeepT_b~b;|G6{CjfZo&pJ)bGTc$5T+<!CMy)pkT3$$J?|w@VW|^!L5BJoBgv+J#4&mAslLpiD>x?oUsk%FiWUN(7)#=u1W%E{0xEG9fHw6zH0#5xD~vf^J+iPS@+(jE$QhNk2>V8WLHaBJT4k9N(wld!6wN3KQ~T8TtFJf^x_22WMKBCDZ5Eu=DWi2O!B>A1JzW@lfHLS-zhab{cV|nllGNRDOYz~Gi3_6Bpz$)}lEv3VgU%%FKax1<y?{JMXbT-xn`j=>yVogb3(QmW59ql5{`AJL;!wqYUxH4BP-+<KXin)Ur{Ts$^&T4q=^qS^7?zvMG1X1lfF2TFED170%JkD?R3@4tTbT0Lu;ElO|K;ZmUu(jNXE**)tfA0&u?|}KUUy@5v)AgF60&fRHXhxImB2q2<3_NaL5n0Pz+#J?>g%rZ;Jp_>r&Se?{G!4ps<qZ4_wW@5CwBB~)+0P=vn*ma2(qkE5q$H&#Ls`;$+;SnBHzDwIog6s99@fQ^_gxOgE`Q?N){8!7C9N3i^{E|M~P8<?^f*_4|zkxOQ43W+#?1=j2#c|3&ChI^nM)o=%?nA{D_18AN)-i)yyCWyvN4%U{4t0SjA-e$+GI4i>t$|By@mxXrwU$d*%WE__I!~sQ8NKpKL#h%JahXDh>7&pse?f7w+wiM8{sFTui?*v?%3DtG(~;)jrejOqi;63GIaqnkX4`IAyma$oETQnDKiyUBSxt=azJ_`!R$xeyPj(+4wV+C40lh%BzfgB@`F-pU7ip*3s?(Ko$%y)TSV9ctSTV?kw4*ukN7yW(ft*-;XC({KR+8?aZ`6m32s4V><m1%oJh^s0&Vzn58flOJUiiGle{ME%C^)sm;GJkUbaf)*CZt)ytXL6l^NZ#MmMb|Elz%LfDIW_$oL}u30xtW;~`=!_60ANhwK0ASlcPnmux*BV)_h1WZfGSBL-UxT;Ics!6r~EU#)cnmE!`nWrUMZlp^DpbqZr9MmpD2uWPh%(efbai|J+Z?EzbTX@F?qs`?gQM=AFyuBUBdGWxVPqzAZdGTFRaPef<E3s(X3lv!o;WOnU-7mKGTWy5z-cBA|2!3xrbqJ#NlqhWj>HSo#g55iYVZDsmyg)Li>?5^&a=Hy`as;D9D1OtO;ukj%7AR7M(iq>`UH#(TGfrjpR3svMSkPR!Z|Sm%n!)B-ls)0_ka(Ksm0=jyxOy1%^H^m`MTD(en0KQrc#=+!xk*~Tz;P!OkzA>(u3qGdoAV7yECM_!7??3NJRa*N0^Y~@GDX}C6|y)Q6_RLIgDsFaq6DpEQ7l#9VNZfOW2l8>{3G)m{P#Za8kov^-Bq_!_9OLrTN8;(*8PE+!xc?adkWjaEv=z=aq2}^qlh-dwX9_693Un8;EV0(%QiCGnS)=nYD!TiRQ^71SY+C8Z%p*P3tzR=J#eLoNT*qd*IqV7Z4!1i;2y$=kGy%LZAkWNYz1mrag|cvh2iIZ3y2BN!|14rZzAv#i3bB1X&b4C^L%9qEOKxM`c%hi^bW+a+i2QU5{BCGyg^<NFnX2sShXzRU~WuJ`h-)Mz73$e)6{Z0|1B%2UrMtnR5Ui&{qT&$Yo151F@a$a^b5kvK1Lj#`Bs4pgLewc{4BLw&!kCqY=#&RxQ314#S`ivLG$r++VWje5&=(9F!v#>)C(tY)$;2q&xSRt<3jL{z^8V8E?y$W^7o=ud5P47sTf;5eooeK*$8b5*m|l?QTHixCpKGgJV20`0j1!5vRZI?$)1HG)R_j)!=3csufE9CPu*pnoE-spDJ=Xh_)lsEIQPGOrq;CwlJ(<Yz2;kQ;>cbzj-T6IR@r%PRJqqVB8V&PvMM!RTzV$^>!ar$6*kvic`_n-hk+R=ShIk$;YC%4(cCs)f!~=Ry+G5fpyf?Af<S{Lvj>>e@phiPj9vy{W)~#Y9#?!uGZQn;*byNn(&}J$I6=9I=baSJS3oRzgB552PhavBSQOBQi<LsGRY7@90D=p=N<j~{F$p)+F6yr7+^CwweZ1>euP167%AW8!cp?Dx>NlFy3cF^pwo7;YKySnfx}jh?ZHiC-_N0gB08VIMwFU6zWp_anpH16C6;E;>f}4jAI~avZymKQJCG|w{6lm%?^V1!##WO#S@A~{ufAf71Oa@#7T?x0Ag9l414w7%z8FWt2yllC6ZF9Vx%@j!b@3~L7jNiFPl#N_i*&3EJVTupZ6fc1Ld&cX$M|#QGrzed*_NB(}Cd1NTHJUXu2lJA1hrDb_Q4BzaLk_#yXJ+jS*k4NSXK+n0kg1nm*@ZQ{?oAt|VzRsKnIk<&U}0iWa$nyhRxB_1R_O7Ullje&<x!_pm!pzeSeKW_2hb;u9hd4BSwm-VvO`dA4Jf}M8^M@yzCV;j7{W06Bb4M-+HI(1{?-p8gdX(&fm#8=jpo4o!{_l7MWG?%wblMtt8%$^fCv=tfsT61a~+fdGom628Sji|2)YewFm=9+AnXwCaor^xhf44DMyo}``DNRmU9U>$;^Flmjv8uY+a;dtRMPA|GE)~yvl2(Jwg8AySd>arQ^N3_kUKo#RgtbYU2&idLsXHn6C%`i)w3g*0bP(+c*6_N(Qn4nxY<SZnON@r2{4t;CO9(mdg2>MOxSJ9oAbP%UKHH@4a~p_J%_nR&rkklr5RBvBSs?j7mQw`$&I>nLx!0Jhn#7b#&DVB^}`H)J<ZC+%Tq)*qdW?o0Z~fffTE>#@w9>53jaBoDYe~Kvjr_#90*H<&5_ikAH3VSap*AXAo-wnYgp(zf%VxV9zJNCYE}5M*$IhxL*Kq2=Ippaz0(2?<v0DOQGMyMlO_Kg_9LPxGv5;pNS`A6a(nICa$66j2il&L@!#YBGL63sWy>O1Ixtu6!fbRCEx*29E`LT1;_uq6ybO#GB<Md<3lJDTUV0$9h4?q;SjeK$=2g8pN1isfLK8#<+dfl-Y%o9cH$7atheYt^dF$5(C+{y^^+}jaSZ2|tbYP%F>E+=D^eC|`!CkGU8^GfeF;jF4JVY@6g!KE|EtT9QG>R@7=*v@^P>;CZqr(D~LmJEl>@8O>bnNC5+1%FeyiJ|6TDJEYynVC*DZDq;bh0NB?YJqV>NtjNTz~jl{%sW!sMl<LSC*s43ktQzIwL&^03)J_#0Ue#T@~Fy&&TZ`=h%YdvadkBK!_Rx#n*;C&am>MLmK&z;*TyTc5j$fim7)ep{;4qq3VnYpXD3VaNs!+)`VfkX$jcFfcDx_-dH0NLf`XN{piyYkXf5>hYD?kbYQ;*Cw|%O5A1PXCIo#ho!8%S%y}*xv8^GZep?n#d#1C2E_M4?70ie5_nqJYLYG;q>Xj*}EuT3V)coZoR7iWu2`{{AD;*iPkSOO%n}4agHl%cp+H(l#LYMTORDyMov#?E3ef_quodu`UQ0(TmFjy@`FJW23xE%NJj|KNw&lBEhY_u!j0Ox1<1U%9KnQVDB-8W={l$@UGLw9zObM^6?PNDLmZk7GDa3f!6NY0@ix!oP3Mwk?HQRPFh(PcXy;q;ur{2Jx^Wh4hYi|-J=tcH|}&w3xubM6Zrxnqz{5;)6YUj>a+w}1t9w3k3WEUd}NH5d@G^$MuTov;?ub{4fmbOTg<gLf*}c>4Lp&C>}CDQMLhDo?ZS$X=fR0vp-Uo2dWO9FYlVz#8fs?yh8+Hwdl<a%9Dxs@>s#D7ramK_Ot)t9+AMiMQS)5@rJ)TriWQtsZ+ZQzU)<H2R+e1N!<!*~7rw69(uApY7Ubl4F-}I4yD<%X4Jaf2+O7=4Qo%dl0<8Wg&|$cKr4dSci5)1@%B~jPWwSa?iyhDAmfXX?=u^?^)D>s`zb)zVYOa5Fl~oiT(nNc}2eIbFd}UJn0pJHM=+00}o{7Y-_-zo|)jMgH7G@dJLH+Ctze61X1_2Nhl}Mz*`;AX$_4Q%mpf>0@Frd;E3o`JZD}ElW=f%`tU62*2Pg5*F63A?Jx{DCPMdrK>Sa{8!uH@AFWa-(|qaQE9IlkQETl05xK-_DQcEu1NN5hB}ZV$1{lOiYc-YRI`O_oXwQe2%{R@K2wAH!zKY9H<N<nUE!=xqY7`<9G?V=TcLnVp&-yvKj{$_Sn|PUNqy>jV78KH)dmhOkxn?L5R>`(S#>`#%Bz_WTu`Hq$Npf<I?H(=egDQB7*j#a#4jBzHY;RcC9Bdk--I{t0g;U&@u*Wju758RDfI5G8z#rTh2^rbuttGDDWo9xJ7h**wXB)PKb$cF6=?jO>wFb25&S#8({u#^NXq7)q2%DQrUe1lnA}TDq1{9wPI*P2ap~U$a&<dW4**GcP;IrJdr|P$AN6KH~>StgzLGwW<n99ylss3YB(J&plE;mlSZHc552)LO4;V_MVI4S*ZNct)mY7k#o2OAk!llu~%a%EG+YdRr0v8ocWE#?&P7P0t@z=BdkQaL@#Bi14hu2QEr+AIfwtvRXPf=HGNoOZHvFI({PcF^)TCKs-?f<-^9?qGvJOVOT&3jq_J?r)@4x`E1f1$xJ$HFmw=;J*A17)RLKL8->vg|w}?*@=ORyk69tY6U}fF5k_F>Af0De6%PPhn^3wARcqMky3bX#TpsrD(ML)PmgQtBI+#5c2%IpY37BfWYIe#wzPxU$xJ2mHj`e{IP72PyDfmF<|i<*V%t)8H%*oX!`pk|FasO@4gCPVXYG)Lwb53+*#7++My<qA_m%LxP~hrLHbET$ydA9Vi4Y2WYYy_%Jnol%a(QF0bng-JK})FZ;+-UdNfT}nL`obkeN9@)nyLaC{tOSkd|fl$YEyz!ee(C&UoGdJ4Bx}`#&Y(L&mK{g@-SnX&*kmo3P5ZZFQoqlBN6P|BnmxN(<+zz3ZdUXh_&#fp+rh!6wQmr4F{G=Z!<_KyX+|^+#Tkpy~Ca82Ic<|f-2+Ap&d=u9N4>{f!;P~DK!O^i&LJk3J~%OwH~O_kf${HE7(IDj+}=t?#*gcy^5^&WDG^flAXoc!f)d6oZg##&Za-MfWI)!yIW$5_hUZBRFq|;bq{T^KfG`Oi;lB&zX2sM-U`(0Nf>5g91u!hS2$N;kHqVk1v!}H#!&~q-z363k)p7RK%!KX*ud2zmMbofELgjoyI&QeeJFt04+&yEC5+u0XZw;wQ4oK<;;4r=iuL|{u`u<}Mj$HcYc`EgLcp6M4(TfsA&AOz{CpBRQ63FxmSVdl&KhsD&qs);=f;GkyNbh^YbJ%<+kES^R{UvjNbS7huAxhgd_F<qCcZbH?qQt+t;~tXXByHU4N1we{F_*#?_}cN2ha2U6DGr=ZmZG_QQjcZnVnDQMi(Cdd*UmULyDZv00Fp}0lJn26vf`ovBYQl0ssI200dcD"))) No newline at end of file |
There was a problem hiding this comment.
train_gpt.py is an opaque exec(lzma_decompress(base85(...))) blob. This makes review and reproducibility auditing difficult (and can trip security scanners). If possible, include the actual Python source (even if minified) or at least provide the decompressed source alongside this file so reviewers can inspect the implementation of the claimed optimizer change.
| # Cautious Muon + SP4096 + Depth Recurrence + Parallel Residuals | ||
|
|
||
| **val_bpb = 1.1604** (3-seed mean, std = 0.0033) | ||
|
|
There was a problem hiding this comment.
This submission is labeled as a “Record” (folder/PR title), but the reported mean val_bpb = 1.1604 is substantially worse than the current 10min_16mb leaderboard entries (e.g. 1.1228 in the repo README). Consider renaming the PR/folder/README name to avoid implying it’s a new SOTA record if it’s intended as a non-record/ablation submission.
- Fix MuonEq-R ordering description (before NS, not after) - Add explicit optimizer pipeline steps in README - Add bytes_total and bytes_code to submission.json - Fix date to ISO-8601 format - Clarify this is a non-record submission - Add decompressed source note for reviewability Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Summary
Per-Seed Results
Cautious Muon (Key Modification)
After Newton-Schulz orthogonalization and MuonEq-R row normalization, the update is masked:
This filters stale momentum directions, providing ~1.47x effective convergence per step with zero parameter overhead.
Architecture Stack
Compliance
Reproduction
Credits