Jon Herlocker

Small models becoming smarter! Phi-3 reinforces the importance of data quality.

24

Robotics and AI will touch all aspects of life and society. This one “could” be a path in reducing or eliminating the herbicides used on crops. On the technical side, from the article, this is astounding. Who would have thought even 10 years ago GPUs would be utilized in this manner. “Powered by 24 NVIDIA GPUs, Carbon Robotics’ vehicles use deep-learning-based computer vision models to autonomously identify and eliminate weeds via CO2 lasers. LaserWeeder, as the product is called, processes 4.7 million high-resolution images per hour, zapping 5,000 weeds per minute, according to the company. To date, its dataset includes 25 million labeled plants and more than 30,000 crop and weed models” #AI #agritech #robotics #nvidia

3

1 Comment

🤖 The future of AI isn't about staring at screens—it's about living in intelligent environments. In my latest piece "The Symbiotic Future: Human-AI Integration in Spatial Computing," I explore how spatial AI will fundamentally transform our relationship with technology. Drawing from theories of extended cognition and environmental intelligence, I argue that we're moving beyond mere human-computer interaction toward true human-AI symbiosis. Key insights: —AI is evolving from tools we use to environments we inhabit —Intelligence will become distributed across our physical spaces —The future is not screen-based but world-embedded —Human and machine intelligence will form recursive feedback loops —This isn't just about better technology: it's about a new chapter in human evolution. Read the full article to discover how spatial computing might reshape what it means to be human in an AI-enabled world. #AI #SpatialComputing #FutureOfTech #Innovation #HumanAI #CognitiveScience #EmergingTech Thoughts? 🤔

9

1 Comment

Nearly a decade ago, I wrote out a complete and rigorous proof of Cantor's theorem that the set of sequences of natural numbers is bigger than the set of natural numbers...using only words of one syllable. https://lnkd.in/g4wWYVYc This is not the only time I revisited the result, much more recently I formally published a number of generalisations of this using different fundamental assumptions (for experts: taking Lawvere's fixed-point theorem and the contrapositive diagonal argument and showing how the proof works in a substructural type of logic). But this one-syllable-word proof was so much fun. I was reminded of this because someone got in touch with me, saying how it is something he periodically re-read for pure interest's sake, and asking if I could recover some material that was on the now-vanished Google+ to which I linked. I had in fact saved that discussion, which had contributions of similar pieces from a number of other mathematicians, including the Fields medal winner Tim Gowers. I will put a pdf of that thread somewhere public, soon, and link to it. But my correspondent was so taken by the exercise that he said he'd go so far as to purchase a physical book containing a collection of such "short words proofs", if one existed. The idea of publishing a slim volume of fun little proofs certainly is appealing, in some respects, because I love producing physical artifacts that reflect mathematics, but the market must surely be tiny!

7

2 Comments

Open source has always been a major part of my career. I spent 13 years building MongoDB and its open source developer ecosystem, and at Viam all of the software that runs on a user's device, our APIs, and our SDKs are open source. Now, I am excited to share that Viam is now an official member of the Open Source Robotics Alliance. Let’s talk more about why open source is important to us:  Security - If machines are inside your house, your office, etc. security and privacy are paramount, and you'd want to know exactly what’s happening with that data. Open source not only makes companies responsible to write good software, but lets the community easily vet it. Innovation - I have said before, no one wants to contribute to a closed source ecosystem. We want developers to bring their ideas to life using Viam. To do so, open source is necessary, as it’s harder to contribute to closed source, and not nearly as rewarding. It keeps corporations in check - When betting your business on a technology, you are responsible for making sure there are some checks in place. By making everything on-machine open source, it means that if something happens, there are ways for users to move forward using open source without Viam. This is all just the tip of the iceberg. Open source has the power to democratize robotics, remove barriers to interoperability, and make automation much easier than it is today. Read more here: https://lnkd.in/eTc8p_Dk Open Robotics #opensource #openrobotics #viam #mongodb #robotics #opensourceroboticsalliance #OSRA

181

3 Comments

Lin Guan and Yifan Zhou have led a very interesting study investigating the effectiveness of Vision Language Models in critiquing robot behaviors in terms of the desirability of their "style." If you are wondering how our work on LLM's inability to verify correctness squares with this study on using them as style critics, see my earlier post https://lnkd.in/gBYAr5qq ! This is the reason why being a style critic is one of the roles LLMs can play in LLM-Modulo framework: https://lnkd.in/gyAyKx4E Check out the details of the work @ https://lnkd.in/gfVZ4EYW ( Lin Guan also has a nice twitter thread on the work here: https://lnkd.in/gnieU-qM )

89

2 Comments

2023 was a breakout year for investment in generative AI startups, with equity funding of almost $22B across 426 deals. The 5 largest rounds all went to companies focused on core generative AI infrastructure: * OpenAI, AI poster child and maker of ChatGPT ($10B corporate minority) * Inflection AI, which focuses on human-computer interfaces ($1.3B Series B) * Anthropic, an AI model developer and research outfit ($1.8B across 2 corporate minorities) * Databricks, a data integration and analytics platform ($504M Series I) * Aleph Alpha, Germany-based LLM developer ($500M Series B) When coupled with the pace of change and the compression of time and cost, makes anyone wonder, given the apparent speed of the commoditization of these LLM engines, what will the ROIC on funds deployed actually look like over time?

1

Ethernet always wins. It has already adopted most of the features of Infiniband. Now, Meta reports Ethernet is more reliable.

12

📰 News: 📰 --Colorado's Quantum Tech Hub Secures Coveted Federal Award: Why SMBs Should Pay Attention!-- 🎉 Exciting news for the tech world! Colorado has officially been designated as a leading hub for quantum computing, scoring $40.5 million in federal funding and positioning itself at the forefront of this groundbreaking technology. This victory over Illinois confirms Colorado's role in pioneering the quantum frontier. Elevate Quantum, based in Denver, will command this initiative across Colorado, New Mexico, and Wyoming. With this substantial investment, the region aims to attract an influx of $2 billion in private investment and transform the Mountain West into the "Silicon Valley" of quantum computing. Here’s why small-to-medium-sized businesses -SMBs- should start exploring quantum solutions now: - 🚀 --Efficiency Gains:-- Quantum computing can solve complex problems exponentially faster than classical computing, significantly improving operational efficiency. - 🏆 --Early Adoption Advantage:-- Being an early adopter of quantum technology can provide a competitive edge, allowing SMBs to leverage advanced computational power for strategic advantage. - 💵 --Investment Potential:-- With over $125 million in federal and state funding, there are unprecedented opportunities to access resources that can drive innovation and growth. In the words of Gov. Jared Polis, "This decision shows that America is serious about being a global leader in quantum technology, the future of computing.” Join the conversation! How do you think quantum computing can transform your business? Comment below and follow Spin Quantum Tech for more updates! 🚀 -quantumcomputing -innovation -technews -businessleaders #Future #Futurism #Innovation #Business #ArtificialIntelligence #QuantumComputing #SQT https://lnkd.in/egznu_Gh

3

I am fascinated by graphs and networks, especially in the context of information storage and discovery (one of the reasons I'm a mega-fan of Obsidian). This article highlights an intriguing application of graph-based AI models coming out of Massachusetts Institute of Technology and Markus J. Buehler - https://lnkd.in/emAUS6Kv - I'm excited to see where this goes.

12

You are seriously underestimating the surprising capabilities of the current LLMs - very Large Language Models based on the Transformer design. Consider Stanford University’s comprehensive “Artificial Intelligence Index Report 2024”: https://lnkd.in/g24NDe7X On page 81, it discusses the State of AI Performance stating “As of 2023, AI has achieved levels of performance that surpass human capabilities across a range of tasks.” “Over the years, AI has surpassed human baselines on a handful of benchmarks, such as image classification in 2015, basic reading comprehension in 2017, visual reasoning in 2020, and natural language inference in 2021. As of 2023, there are still some task categories where AI fails to exceed human ability. These tend to be more complex cognitive tasks, such as visual commonsense reasoning and advanced-level mathematical problem-solving (competition-level math problems).” Then on page 115, the paper discusses the GPQA benchmark: “GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In the last year, researchers from NYU, Anthropic, and Meta introduced the GPQA benchmark to test general multi-subject AI reasoning. This dataset consists of 448 difficult multiple-choice questions that cannot be easily answered by Google searching. The questions were crafted by subject-matter experts in various fields like biology, physics, and chemistry. PhD-level experts achieved a 65% accuracy rate in their respective domains on GPQA, while nonexpert humans scored around 34%. The best-performing AI model, GPT-4, only reached a score of 41.0% on the main test set.” Note, that GPT-4 does better than highly skilled non-expert humans. But this was last year. There is an article published a few days ago that discusses the latest results: https://lnkd.in/g53hdRQR “Last month Anthropic announced that Claude 3 scored just under 60%” (using chain-of-thought prompting and 5 examples).” Note, this is almost at the level of PhD experts “in their respective domains”! And just days ago, OpenAI announced a new version of GPT-4 that they claim outperforms Claude 3 in reasoning abilities - reclaiming the lead its held for the last year.

Good discussion at the AI Conference about AI Regulations by Professor Ion Stoica (Berkeley Electrical Engineering and Computer Sciences) and Senator Scott Wiener (proponent of California Senate Bill 1047), sharing my key takeaways: California Senate Bill 1047, aimed at regulating AI safety, offers both opportunities and challenges in the world of artificial intelligence. 1. Enhanced AI Accountability: The bill requires developers to integrate safety protocols and shutdown mechanisms to prevent misuse of AI models, particularly in high-stakes environments like healthcare and finance. This accountability ensures AI systems are safer and less prone to harmful or unauthorized use. 2. Risk Mitigation: By mandating risk assessments before deployment, SB 1047 minimizes the chance that AI models will be weaponized or used in ways that could cause large-scale harm, such as cyberattacks or financial fraud. 3. Public and National Security: The bill’s focus on preventing catastrophic outcomes from AI misuse, including risks to national security, aligns with growing concerns over the potential misuse of advanced AI technologies. 4. Stifled Innovation: Many tech companies argue that the bill could hinder innovation by placing excessive restrictions on AI development. Open models, which rely on community input for improvements, might suffer due to liability concerns about post-training modifications. 2. Impact on Open-Source Development: Critics from academia and industry argue that the bill’s requirements, like the shutdown mandate, could disproportionately affect open-source and public sector AI research, discouraging innovation in those spaces. 3. State vs. Federal Jurisdiction: Some argue that AI regulation should be handled at the federal level rather than state-by-state. There are concerns that SB 1047 could create regulatory friction, especially if federal AI regulations emerge later. In my perspective, we need to have a strong foundation for AI safety, the debate underscores the need to balance innovation with security to ensure AI benefits society without imposing undue burdens on developers. www.rxsmeta.com

1

I continue to be alarmed at the progress of proposed California regulation SB 1047 and the attack it represents on open source and more broadly on AI innovation. As I wrote previously, this proposed law makes a fundamental mistake of regulating AI technology instead of AI applications, and thus would fail to make AI meaningfully safer. I’d like to explain why the specific mechanisms of SB 1047 are so pernicious to open source. To be clear, there are routes that regulators should pursue to improve safety. For example, I would welcome outlawing nonconsensual deepfake pornography, standardizing watermarking and fingerprinting to identify generated content, and investing more in red teaming and other safety research. Unfortunately, the proposed bill pursues a less beneficial and more harmful path. SB 1047’s purported goal is to ensure safety of AI models. It puts in place complex reporting requirements for developers who fine-tune models or develop models that cost more than $100 million to train. It is a vague, ambiguous law that imposes significant penalties for violations, creating a huge gray zone in which developers can’t be sure how to avoid breaking the law. This will paralyze many teams. You can read the latest draft of the law online. I’ve read through it carefully, and I find it ambiguous and very hard to follow. Developers who try to navigate the law’s complex requirements face what feels like a huge personal risk. It requires that developers submit, under penalty of perjury, a certification of compliance with the requirements of the law. But when the requirements are complex, hard to understand, and can even shift according to the whims of an unelected body (more on this below), how do we ensure we are in compliance? For example, the certification must include many different sections. One is an analysis of “the nature and magnitude of critical harms … the model might reasonably cause or enable.” But given that even leading AI researchers aren’t sure what harms models might cause or enable, how is a team of developers supposed to figure this out and declare — under penalty of perjury — that they meet this requirement? Further, some developers will be required to implement “protections to prevent … misuse of, or unsafe post-training modifications of, the covered model and all covered model derivatives … that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.” Even leading AI researchers don’t agree on how best to “protect” AI models against these supposed risks, or what would be “appropriate.” So how are developers supposed to figure out how to comply with this requirement? [Reached LinkedIn length limit. Rest of text here: https://lnkd.in/g_NhQpni ]

2,721

201 Comments

In the race to advance AI, we often focus on algorithms and data. But what if the most fundamental bottleneck lies in the design of GPUs? I recently spoke with Sean Lie, co-founder and CTO of Cerebras Systems, about their audacious approach to AI computing. Instead of iterating on existing chip designs, they're reimagining the entire system from the ground up. This end-to-end approach is pushing Cerebras to innovate in unexpected areas. Take server design, a field long thought to be settled. When you're working with a chip the size of a dinner plate—57x larger than NVIDIA’s H100—even servers need an overhaul. It's a reminder that breakthrough innovation in AI will not come from algorithms alone, but from rethinking every layer of the stack, from bare metal to high-level software. Our full conversation here: https://lnkd.in/gWZCi4zE

37

📜 A new blog post explaining the fundamentals of RL for LLMs—the perfect place to intuitively understand PPO and RLHF from first principles. 👇👇 Pre-trained LLMs are often compared to Lovecraft's Shoggoth, a monstrous, shape-shifting beast. PPO is the algorithm that brought it under control. Take a look! 🔗 https://lnkd.in/eXqDvrUV

29

1 Comment

"How does a vision LLM outperform OCR, and why?" I got this question from our customers a lot and we distill the answers into the blog: https://lnkd.in/guv8waCs Overall, Vision language models (VLM) address key OCR shortcomings by: 1. Accurately interpreting low-quality images and complex layouts 2. Understanding context and semantic meaning in documents 3. Handling multiple languages and scripts seamlessly 4. Reducing post-processing overhead #pdf #word #ocr #llm

36

3 Comments

Thanks Hamdi Amroun, PhD - this is a great paper to share. In the real world business context we often need AI to reason over large pieces of text, and often you need/want multi-document summarization (MDS). 'When testing five LLMs with benchmarks using news and conversation datasets, they found up to 75% of content in MDS summaries was hallucinated, with notable increases towards summaries' end. Alarmingly, even for non-existent topics, models like GPT-3.5-turbo and GPT-4 generated fabricated content 79% and 44% of the time, respectively. Analysis of 700+ generated insights showed that hallucinations often arose from failures to follow instructions or overly generic content.' I agree with Hamdi's sentiments that despite improvements in post-processing techniques, robust solutions are urgently needed to reduce hallucinations, as inaccurate summaries can lead to business risks (notably misinformation and misrepresentation). I would add that there is still a lot of utility in LLMs/Multimodal models but we need to be vigilant - this is not about AI malevolence or intentional deception. LLMs do not possess consciousness or intent; they generate content based on patterns in data they were trained on.

6

1 Comment

As a part of Google's Quantum announcement today, they mentioned "It lends credence to the notion that quantum computation occurs in many parallel universes, in line with the idea that we live in a multiverse, a prediction first made by David Deutsch." What crazy level of information is this 🫨 😵‍💫 Does it mean that whatever living being that exist in these multiverses use Quantum Computer? https://lnkd.in/dwbgXZ9S

8

1 Comment