Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
flipboard.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
Welcome to Flipboard on Mastodon. A place for our community of curators and enthusiasts to inform and inspire each other. If you'd like to join please request an invitation via the sign-up page.

Administered by:

Server stats:

1.2K
active users

flipboard.social: AboutProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.7

#attention

5 posts5 participants0 posts today
Public
Blender Dumbass ( J.Y.Amihud )

From: blenderdumbass . org

Doing those Dumbass Blender tutorials was supposed be a way for me to get noticed. But no matter how utterly insane my ideas were, I just fell deeper and deeper into despair.

Read or listen: blenderdumbass.org/articles/th

blenderdumbass . orgThe Impossible Task of Famousing Myself
More from Blender Dumbass ( J.Y.Amihud )
#YouTube#LBRY#Odysee
Public *
Alex Silver

Continuing to try to figure out the internet and information in 2025. Access to information has changed a lot in 20 years—it now seems that individuals have an intense responsibility as "executives" of their own attention.

I'm threading together my new place online: for my art, for my political discourse, for my support of quality press institutions, etc.

Mastadon is a decentralized social network: let's not let that mean "siloed."

#internet#attention#civicengagement
Continued thread
Public *
Coach Pāṇini ®

Young people today navigate unprecedented challenges: economic precarity, geopolitical instability, the erosion of traditional pathways in work and #education, AI's looming disruption of career prospects, and accelerating climate catastrophes.

Banning phones is a superficial solution to profound problems.

The evidence is unignorable and demands our #attention.

5/6

Public
Victoria Stuart 🇨🇦 🏳️‍⚧️

[2504.00927] Multi-Token Attention
arxiv.org/abs/2504.00927
news.ycombinator.com/item?id=4

arXiv logo
arXiv.orgMulti-Token AttentionSoft attention is a critical mechanism powering LLMs to locate relevant parts within a given context. However, individual attention weights are determined by the similarity of only a single query and key token vector. This "single token attention" bottlenecks the amount of information used in distinguishing a relevant part from the rest of the context. To address this issue, we propose a new attention method, Multi-Token Attention (MTA), which allows LLMs to condition their attention weights on multiple query and key vectors simultaneously. This is achieved by applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention. As a result, our method can locate relevant context using richer, more nuanced information that can exceed a single vector's capacity. Through extensive evaluations, we demonstrate that MTA achieves enhanced performance on a range of popular benchmarks. Notably, it outperforms Transformer baseline models on standard language modeling tasks, and on tasks that require searching for information within long contexts, where our method's ability to leverage richer information proves particularly beneficial.
#ML#TransformerArchitecture#NeuralNetworks
Public
Hacker News

Multi-Token Attention

arxiv.org/abs/2504.00927

#HackerNews #Multi-Token #Attention #research #AI #advancements #attentionmechanism #deeplearning #NLP

arXiv logo
arXiv.orgMulti-Token AttentionSoft attention is a critical mechanism powering LLMs to locate relevant parts within a given context. However, individual attention weights are determined by the similarity of only a single query and key token vector. This "single token attention" bottlenecks the amount of information used in distinguishing a relevant part from the rest of the context. To address this issue, we propose a new attention method, Multi-Token Attention (MTA), which allows LLMs to condition their attention weights on multiple query and key vectors simultaneously. This is achieved by applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention. As a result, our method can locate relevant context using richer, more nuanced information that can exceed a single vector's capacity. Through extensive evaluations, we demonstrate that MTA achieves enhanced performance on a range of popular benchmarks. Notably, it outperforms Transformer baseline models on standard language modeling tasks, and on tasks that require searching for information within long contexts, where our method's ability to leverage richer information proves particularly beneficial.
Public
Blender Dumbass ( J.Y.Amihud )

From: blenderdumbass . org

Doing those Dumbass Blender tutorials was supposed be a way for me to get noticed. But no matter how utterly insane my ideas were, I just fell deeper and deeper into despair.

Read or listen: blenderdumbass.org/articles/th

blenderdumbass . orgThe Impossible Task of Famousing Myself
More from Blender Dumbass ( J.Y.Amihud )
#YouTube#LBRY#Odysee
ExploreLive feeds
About