← Back to context Comment by msdz 7 hours ago TIL, that makes a lot of sense, and thanks for the correction. 1 comment msdz Reply HarHarVeryFunny 7 hours ago Just to add to that, a Transformer block consists of an attention part followed by a feed forward part. MoE only modifies the feed forward part (which basically contains declarative knowledge getting injected into the residual stream).
HarHarVeryFunny 7 hours ago Just to add to that, a Transformer block consists of an attention part followed by a feed forward part. MoE only modifies the feed forward part (which basically contains declarative knowledge getting injected into the residual stream).
Just to add to that, a Transformer block consists of an attention part followed by a feed forward part. MoE only modifies the feed forward part (which basically contains declarative knowledge getting injected into the residual stream).