Yuanpeng Li

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

15

In this episode, we discuss GenHMR: Generative Human Mesh Recovery by Muhammad Usama Saleem, Ekkasit Pinyoanuntapong, Pu Wang, Hongfei Xue, Srijan Das, Chen Chen. The paper introduces GenHMR, a novel generative framework for human mesh recovery (HMR) that addresses uncertainties in converting 2D images to 3D mesh. It employs a pose tokenizer and an image-conditional masked transformer to learn distributions of pose tokens, improving upon deterministic and probabilistic approaches. The model also includes a 2D pose-guided refinement technique and demonstrates superior performance compared to current methods.

15

🚀 Supercharging Parameter Efficiency in Conv2D Layers 🚀 Welcome to Part 6 of my "Breaking Down LoRA" series. Today, I’m excited to share an advanced enhancement to our LoRA (Low-Rank Adaptation) implementation for Conv2D layers, achieving even greater parameter efficiency! 💡✨ 🔮 The Magic • The previous approach decomposed the a single channel of the weight matrix (size: ℎ × 𝑤). Focused on typical kernel sizes like 5×5 with a rank of 1, we can achieve 40% reduction in the number parameter at max. • In the revised approach, we "permute" the Δ𝑊 matrix from channels first to last format, i.e., [𝑚 × 𝑛 × ℎ × 𝑤] → [ℎ × 𝑤 × 𝑚 × 𝑛] • With typical channel dimensions ranging between 64 and 1024 or even higher, this rearrangement makes the decomposition much more efficient, resulting in matrices 𝐴 and 𝐵 of size [ℎ × 𝑤 × 𝑚 × 𝑟] and [ℎ × 𝑤 × 𝑟 × 𝑛] respectively. 🔢 A Concrete Example • ℎ = 𝑤 = 3 • 𝑚 = 𝑛 = 512 • 𝑟 = 8 Parameters Without Optimization: 512×512×3×3 = 2,359,296 With Enhanced LoRA: (3×3×512×8) + (3×3×8×512) = 73,728 (~3% of original) Δ𝑊 Reconstruction: • Δ𝑊 = permute_dims(𝐴⋅𝐵) • [Δ𝑊] = permute_dims([ℎ × 𝑤 × 𝑚 × 𝑟]⋅[ℎ × 𝑤 × 𝑟 × 𝑛])  = permute_dims([ℎ × 𝑤 × 𝑚 × 𝑛])  = [𝑚 × 𝑛 × ℎ × 𝑤] Code for the LoRA wrapper for Conv2D layers: https://lnkd.in/dub47uBj This sure adds a few matrix premutation steps for each pass but significantly drops the number of trainable parameters. Plus, pushing channels to last enables better logical, symmentically synchronous and easier extention to 1D & 3D convolution layers; which I shall take up next!? I’m thrilled about these strides in optimizing neural networks with LoRA! Let’s continue pushing the boundaries together. 💪✨ Feel free to reach out with your thoughts, questions, or collaboration ideas! 📚 Catch-up on previous parts: • Part 1: 🚀 Implementing LoRA (Low-Rank Adaptation) from Scratch in PyTorch! 🚀 https://lnkd.in/dEvGTZwi • Part 2: 🚀 Introducing a Learnable Bias Term, Δ𝑏, in LoRA! 🚀 https://lnkd.in/d7gHXd3J • Part 3: 🚀 From Pixels to Parameters: How LoRA Echoes a Classic Image Processing Technique! 🚀 https://lnkd.in/d3v_-Y_c • Part 4.1, 4.2: 🚀 Extending LoRA to Conv2D Layers! Boom! 🚀 https://lnkd.in/dQEwZMrx, https://lnkd.in/d2YTCKAE #MachineLearning #DeepLearning #AI #LoRA #ConvolutionalNeuralNetworks #FineTuning #Innovation #PyTorch #AIResearch #NeuralNetworks

7

The @IEEE Computer Society San Diego Chapter is proud o host Professor Nitin J Sanket of robotics engineering department of Worcester Polytechnic Institute for the 11th talk of the 2024 Invited Seminar Series. Professor Sanket will talk about TINY #ROBOT #AUTONOMY, #NAVIGATION, and #INTERACTION. Speaker: Dr. Nitin J Sanket of robotics engineering department of Worcester Polytechnic Institute. Title: FROM MYSTERY TO MASTERY: TINY ROBOT AUTONOMY USING THE MANIFESTATION OF THE UNKNOWN Date: September 25th - 05:30 PM to 06:30 PM PST Venue: Virtual Registration: https://lnkd.in/gWhqGGRA Abstract: The human fascination to mimic ultra-efficient living beings like insects and birds has led to the rise of small #autonomous #robots. Smaller robots are safer, more agile and are task distributable as swarms. One might wonder, why do we not have small robots deployed in the wild today? Smaller robots are constrained by a severe dearth of computation and sensor quality. To further exacerbate the situation, today's mainstream approach for #autonomy on small robots relies on building a 3D map of the scene that is used to plan paths for executing a control algorithm. Such a methodology has severely bounded the potential of small autonomous robots due to the strict distinction between #perception, #planning, and #control. Instead, we re-imagine each agent by drawing inspiration from insects at the bottom of the size and computation spectrum. Specifically, each of our agents comprises a series of hierarchical competencies built on bio-inspired sensorimotor #AI loops by utilizing the action-perception synergy. Here, the agent controls its movement and physical interaction to make up for its lack of computation and sensing. Such an approach imposes additional constraints on the data gathered to solve the problem using Active and Interactive Perception. To unify the class of motion problems, we present a method to exploit the unknown, i.e., the #uncertainty of the predictions to obtain additional informational cues in a new theory called Novel Perception that utilizes the statistics of motion fields to tackle various classes of problems from navigation and interaction. This method has the potential to be the go-to mathematical formulation for tackling the class of motion-field-based problems in robotics and made it into the cover of Science Robotics journal.

17

2 Comments

Excited to share the work that Chris Glaze Changho Shin Frederic Sala and I did on long context LLM evaluations. While there’s plenty of problems with LLM evals, a recurring one we’ve seen at Snorkel, working with enterprise clients, is having to rely on far too simple/toy benchmarks that do not relate to business problems. It is why we developed SWiM, a framework that lets users test long context model capabilities directly on their own data and tasks. Read the paper here: https://lnkd.in/g3uSuKcW

31

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

385

12 Comments

I'm excited about our case study! SAEs and interpretability have been cool and interesting, but haven't been useful for anything yet. I think we make a compelling case that SAEs can be the best tool for certain tasks, especially those that involve preventing rare edge cases or behaviors. The main advantage of an SAE is that we want conditional steering methods that are only active when the LLM is "thinking" about a particular concept. If we have this, we can have very fine grained, targeted interventions. When creating traditional datasets we don't have access to these ground truth labels for supervision, but an SAE extracts these in an unsupervised way.

8

6 Comments

New paper: Shared Memory Parallelism in Modern C++ and HPX https://lnkd.in/ghYcgGkQ

39

Sparse Autoencoders (SAEs) are one of the most promising and popular methods for LLM interpretability and explainability. I wrote a simple, intuitive introduction to SAEs, complete with diagrams and reference PyTorch code. Among other things, I cover why we use SAEs, how they are used for interventions, and challenges with using SAEs. https://lnkd.in/gzrvi6Kg

16

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

51

2 Comments

You Only Cache Once: Decoder-Decoder Architectures for Large Language Models Microsoft has just introduced the YOKO, designed to tackle challenges in large language models, especially when dealing with lengthy contexts. YOKO significantly reduces GPU memory consumption by combining two components: a cross-decoder and a self-decoder. YOCO cleverly combines self-attention and cross-attention mechanisms to allow efficient reuse of key-value caches, significantly reducing memory usage and speeding up the processing time. It's exciting to see such advancements both in theory and practice that make language models more scalable and efficient. The experimental results show that YOKO outperforms the Transformer architecture in terms of performance, with improved scalability for both model size and number of training tokens. Github: https://lnkd.in/dCsXf8Zv arXiv: https://lnkd.in/dKV7K9FB #MachineLearning #LLM #YOCO #AIResearch

4

My paper on Interpretable Tensor Fusion (https://lnkd.in/gjTxusR5) which can capture both linear combinations and multiplicative interactions of multimodal data is on Arxiv. Will also be presenting the work at IJCAI - 24 in Korea.

136

4 Comments

I'm thrilled to share a fascinating advancement in the integration of graph-based approaches with large language models (LLMs) to enhance reasoning abilities. The recent paper [GNN-RAG: Graph Neural Network and Retrieval-Augmented Generation]( https://lnkd.in/gwJyn5NM ) introduces a novel GNN-RAG approach that shows promising results in improving Knowledge Graph Question Answering (KGQA). This approach not only outperforms GPT-4 but also leverages multi-hop information retrieval to facilitate faithful LLM reasoning on complex questions. The potential of GNN-RAG to enhance the depth and accuracy of AI reasoning is truly inspiring. #AI #MachineLearning #GraphNeuralNetworks #LLM #KGQA #Innovation #Research #GNNRAG

17

🚀 𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗠𝗼𝗱𝗲𝗿𝗻𝗕𝗘𝗥𝗧: The Future of Encoder-Only Models in NLP The rise of large language models (LLMs) has brought impressive advancements in Natural Language Processing (NLP). However, tasks like retrieval, classification, and Named Entity Recognition (NER) still heavily rely on encoder-only models like BERT. But with the increasing demands of modern applications, traditional models are showing limitations. Enter ModernBERT, an innovative next-generation encoder-only model designed to tackle these constraints. By incorporating cutting-edge architectural improvements and diverse training methodologies, ModernBERT enhances performance, efficiency, and scalability. 𝗪𝗵𝘆 𝗠𝗼𝗱𝗲𝗿𝗻𝗕𝗘𝗥𝗧? Older models like BERT are limited by: - 𝘚𝘩𝘰𝘳𝘵 𝘴𝘦𝘲𝘶𝘦𝘯𝘤𝘦 𝘭𝘦𝘯𝘨𝘵𝘩𝘴 (up to 512 tokens) - 𝘐𝘯𝘦𝘧𝘧𝘪𝘤𝘪𝘦𝘯𝘵 𝘢𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵𝘶𝘳𝘦𝘴 - 𝘓𝘪𝘮𝘪𝘵𝘦𝘥 𝘵𝘳𝘢𝘪𝘯𝘪𝘯𝘨 𝘥𝘢𝘵𝘢 ModernBERT overcomes these challenges by: - Supporting sequences up to 𝟴,𝟭𝟵𝟮 tokens - Introducing 𝗥𝗼𝘁𝗮𝗿𝘆 𝗣𝗼𝘀𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 (RoPE) for better contextual understanding - Utilizing 𝗚𝗲𝗚𝗟𝗨 𝗮𝗰𝘁𝗶𝘃𝗮𝘁𝗶𝗼𝗻𝘀 and 𝗙𝗹𝗮𝘀𝗵 𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 for optimized performance - Ensuring 𝗵𝗮𝗿𝗱𝘄𝗮𝗿𝗲 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 with 149M to 395M parameters 𝗥𝗲𝗮𝗹-𝗪𝗼𝗿𝗹𝗱 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 ModernBERT outperforms older models in: - 𝘕𝘢𝘵𝘶𝘳𝘢𝘭 𝘓𝘢𝘯𝘨𝘶𝘢𝘨𝘦 𝘜𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥𝘪𝘯𝘨 (𝘕𝘓𝘜): Leading benchmarks like GLUE. - 𝘛𝘦𝘹𝘵 𝘙𝘦𝘵𝘳𝘪𝘦𝘷𝘢𝘭: Achieving state-of-the-art results in Dense Passage Retrieval (DPR). - 𝘊𝘰𝘥𝘦-𝘉𝘢𝘴𝘦𝘥 𝘈𝘱𝘱𝘭𝘪𝘤𝘢𝘵𝘪𝘰𝘯𝘴: Excelling in code-based tasks like CodeSearchNet and StackOverflow-QA. 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 ModernBERT is optimized for real-world deployment with: - 𝘍𝘢𝘴𝘵𝘦𝘳 𝘱𝘳𝘰𝘤𝘦𝘴𝘴𝘪𝘯𝘨 (14.5–30.9% quicker) - 𝘔𝘦𝘮𝘰𝘳𝘺 𝘰𝘱𝘵𝘪𝘮𝘪𝘻𝘢𝘵𝘪𝘰𝘯 for long sequences (up to 8,000+ tokens) Whether you're tackling text retrieval, classification, or code-based searches, ModernBERT provides the flexibility, speed, and power needed to drive next-gen NLP applications. 🔗 Know More About Modern-BERT: Paper: https://lnkd.in/gDZT2UXE Code: https://lnkd.in/g4MZMw5R Model Access: https://lnkd.in/gsUZbYYH #NLP #AI #DeepLearning #ModernBERT #MachineLearning #TextRetrieval #NamedEntityRecognition #CodeSearch

18