Radar Trends to Watch: April 2024

There are lots of new models, including one from Apple, but that’s hardly news. AI news is infiltrating other sections of Trends (particularly Programming and Security)—but that’s also hardly news. NVIDIA CEO Jensen Huang has said that AI will replace coding—but again, he’s not the first. But what’s new is Devin: an AI software engineer from Cognition Labs. Its makers claim that it “can execute complex engineering tasks requiring thousands of decisions. Devin can recall relevant context at every step, learn over time, and fix mistakes.” Devin is in early access; what we’ve heard from those who’ve used it is that it’s far from finished—but even in that state, it’s very impressive. There’s also an open source OpenDevin project on GitHub.

A supply chain attack added a back door to Linux systems through the widely used xz package. Fortunately, this attack was discovered before the package was incorporated into the leading Linux distributions. However, the attack raises a lot of troubling questions about security—including the question of how we know software is trustworthy. The attack wasn’t discovered by security experts analyzing the code; social engineering may even have been used to prevent it from being tested adequately. The attack was discovered by an engineer who noticed some performance anomalies. Nobody knows who the maintainer who inserted the back door really is; it may not be a single person. This time, we were lucky.

Artificial Intelligence

• What does open source mean for AI? What does it include aside from code? Can there be restrictions on how the AI is used? Nobody knows. And it’s entirely too likely that the vacuum will be filled by a self-interested definition coming from one of the internet giants.
• Jan is a new way of packaging open source language models so that they can be run completely locally. It’s available for Windows, macOS, and Linux.
• Can AI be an aid to observability? Yes—both by recognizing normal and abnormal activity and by interpreting and summarizing log data and making suggestions for resolving problems.
• NVIDIA has announced that they intend to create an “embodied” AI: an AI incorporated into a humanoid robotic form. Is this “moonshot” just an attempt at realized science fiction or an important step on the road to general intelligence?
• At NVIDIA’s developer conference, their CEO outlined a vision for the future of programming in which AI systems replace the entire development pipeline. Humans remain in the loop and in control, but they will only use natural human languages.
• The LLM4Decompile project is building large language models for decompiling software (translating from assembly back into a higher level language like C). This would be a tremendous tool for reverse engineering. The models are available on Hugging Face.
• Now Apple has a large language model. It isn’t open to the public, but they’ve published a paper about it. In a Twitter post (that I can’t find) they claim performance similar to Gemini-1 at each model size.
• Answer.ai is releasing an open source system for fine tuning large language models with up to 70B parameters. It can run on a desktop computer with two commodity gaming GPUs.
• A month or two ago, we noted that attackers have proven that they can jailbreak large language models by using steganographic techniques to hide hostile prompts within an image. It turns out you don’t have to be subtle: ASCII art that spells out the hostile words will suffice.
• SudoLang is a programming language for interacting with large language models. It’s not the only attempt along these lines; we’ve also noted GPTScript. SudoLang is particularly interesting, though, because the language was designed in part by GPT-4.
• Simon Willison clarifies the distinction between prompt injection and jailbreaking. Prompt injection involves concatenating trusted and untrusted input in prompts. It is far more dangerous than jailbreaking and harder (perhaps impossible) to defend against.
• A generative AI platform called Lore Machine can take a short story and turn it into an illustrated comic.
• ToxicChat is a new benchmark for detecting toxic prompts sent to language models. It is based on actual prompts collected by language models rather than social media content.
• Anthropic’s latest series of models, Claude 3, is now available. The most advanced model, Opus, is only available through subscription. All of them feature a 200,000-token context window.
• Over the past few years, large models have reduced their data requirements by going from 32-bit floating point to 8 bits to 4 bits, in a process called “quantization.” The next step forward is single-bit models (actually, 1.58 bits).
• GPTScript is a simple programming language for automating interactions with GPT. It’s starting to look a lot like a formal informal language.

Programming

• GitHub now offers Code Scanning Autofix, a service that uses AI to detect vulnerabilities and suggest fixes to code written in Java, JavaScript, Python, and TypeScript. They claim that it can detect and correct over 90% of known vulnerabilities. They note that it’s still important for the programmer to verify that the suggestion actually fixes the vulnerability.
• JetBrains now offers TeamCity Pipelines, a CI/CD tool for small to midsized teams. It is currently in public beta. Simpler tools that solve the problems of smaller projects are a welcome addition to the tooling scene.
• Ravi is a new dialect of Lua that supports optional static typing. It has a just-in-time compiler and can also compile directly to machine code.
• BOINC is a project that lets you allow scientific computing tasks to run on your computer in the background. It’s similar to projects like SETI@Home, but more general; it isn’t associated with a specific research project. BOINC is based at UC Berkeley and supported by the NSF.
• Devin is “the world’s first fully autonomous AI software engineer.” The claims made for Devin are impressive: it can learn new technologies from a blog post, build deploy apps, fix bugs, train language models, and more. If it lives up to these claims, it will be very impressive.
• A startup has released open source libraries for fully homomorphic encryption. Homomorphic encryption is a set of codes and protocols for computing with encrypted data without first decrypting the data.
• We know that language models can assist in writing code. Can they also assist in building infrastructure as code?
• GitHub is being attacked by cybercriminals who are creating millions of repositories containing malware. The malicious repos have names similar to legitimate repos in hopes that programmers will use the wrong repo (often with the encouragement of social engineering).
• Github is offering Copilot Enterprise, a higher-priced version of Copilot that knows about a company’s codebase. Code completions are based on code in the company’s repositories, so they match the company’s practices. It can even learn proprietary in-house languages.
• Wax is an open source framework for building word processing software on the web. It facilitates change tracking, commenting, equations, basic text styling, managing citations, and other features you’d expect in a professional word processing system.

Operations

• Brendan Gregg has posted a list of Linux crisis tools: utilities that you are likely to need to diagnose and fix an outage and that your favorite distribution might not have.
• DBOS is a new cloud native operating system that is based on a high performance distributed database. It is intended to replace the Linux/Kubernetes combination that has become the basis for orchestrating complex distributed applications.
• Buoyant is now charging organizations with 50 or more users for access to the latest stable release of the linkerd service mesh. They have not changed linkerd’s licensing, which is still open source (Apache 2.0).
• Netflix has released bpftop, a command line tool for monitoring programs that use eBPF (extended Berkeley packet filters). bpftop gives users insight into their eBPF tools, preventing eBPF from inadvertently compromising performance while attempting to improve performance.

Web

• Facebook’s Threads now allows users to share their posts on Mastodon. The feature is currently opt-in. Threads users can’t yet view posts made by Mastodon users. How Facebook will handle Mastodon users’ private data and dislike of advertising remains to be seen.
• Ludic is a new lightweight web framework that is built to be used with htmx. It uses a component approach similar to React but does not require any JavaScript to build dynamic applications. It is based on Python 3.12.
• YouTube is requiring creators to disclose when they have used generative AI to create or modify otherwise realistic video. This rule does not apply to content that is “clearly unrealistic” (e.g., animations), color adjustments or beauty filters, and background effects (e.g., blur).
• LaVague is a large language model designed for controlling browser interactions. It would be ideal for controlling a testing framework like Selenium. And it may be useful for automating other “mundane tasks.”
• The Bluesky social network, created by Twitter founder Jack Dorsey, now allows federation: individuals and groups can now run their own servers, similar to Mastodon.

Security

• A supply-chain compromise added a backdoor to Linux’s xz package. The attack raises many questions. Social engineering may have prevented Google from testing it adequately; the maintainer was pressured into adding a second maintainer who was probably the attacker and  may have been state-sponsored; and it was discovered because of performance anomalies.
• Loop DoS is a new denial of service attack in which targeted computers send UDP packets back and forth in an infinite loop. Equipment from several major vendors, including Cisco, Microsoft, and Broadcom, is reported to be vulnerable.
• A new attack against LLMs allows attackers to recover the texts of chat sessions even if they are encrypted. The attack is based on observing the lengths of the tokens and matching the lengths to words. This vulnerability applies to all LLMs except for Google’s Gemini.
• Pixieboot (aka PXE boot) is a collection of attacks against UEFI firmware, a very low-level system-within-a-system that controls the boot process on most modern PCs. While this particular set of vulnerabilities is mostly of concern to cloud and datacenter operators, Cory Doctorow writes about the danger of nonupdateable subsystems that treat the user as a threat.
• Cloudflare is introducing an AI firewall product that, among other things, will eventually include a prompt validation feature that will detect and block prompt injection attacks. The feature may help with jailbreaking (a single hostile prompt), but it’s harder to see how it would be effective against true prompt injection (a hostile prompt concatenated with a trustworthy prompt).
• A paper analyzes over 600,000 prompt injection attacks to produce a taxonomy of vulnerabilities. The authors collected the attacks by running a global prompt hacking competition.
• Docker, Confluence, Redis, and Apache Yarn are being targeted by malware in a new set of attacks. The malware is written in Go, though it is clumsily disguised to look like shell scripts.
• Even more prompt injection attacks: Microsoft Copilot (distinct from Github Copilot) is vulnerable to conditional prompt injection attacks, where the hostile prompt is activated only for a specific user.
• Yes, there’s now a prompt injection worm. A hostile prompt is embedded in an email, which then gets sent to the AI-based email assistant through RAG. Along with stealing data, the prompt can instruct the email assistant to generate new emails that spread the worm.

Things

• Another Copilot, this time not from Microsoft, is a Raspberry Pi-based AI system for bicyclists that alerts them to approaching cars and cars that are driving erratically or getting too close. It’s a good example of Pete Warden’s TinyML.
• Want your own Klein Bottle? Made by Cliff Stoll, author of the cybersecurity classic The Cuckoo’s Egg, who will autograph your bottle for you (and may include other surprises).

Quantum Computing

• Google has published its threat model for quantum attacks against cryptography. The document is an excellent summary of the state of post-quantum cryptography.

Biology

• Can fungus be engineered to produce artificial meat products? Fungus and its relatives have long been the basis of many food products, including cheese and beer. And funguses can produce the molecule that gives meat its flavor.