A new paper argues that current LLMs are fundamentally broken because they're completely static. They call it "anterograde amnesia", which is honestly spot on. A model gets pre-trained, and from that moment on, its weights are frozen. It can't actually learn anything new. Sure, it has a context window, but that's just short-term memory. The model can't take new information from its context and permanently update its own parameters. The knowledge in its MLP layers is stuck in the past, and the attention mechanism is the only part that's live, but it forgets everything instantly.

The paper introduces what they term Nested Learning to fix this. The whole idea is to stop thinking of a model as one big, deep stack of layers that all update at the same time. Instead, they take inspiration from the brain, which has all kinds of different update cycles running at different speeds in form of brain waves. They represent the model as a set of nested optimization problems , where each level has its own update frequency. Instead of just deep layers, you have levels defined by how often they learn.

The idea of levels was then used to extend the standard Transformer which has a fast attention level that updates every token and the slow MLP layers that update only during pre-training. There's no in-between.

The paper presents a Hierarchical Optimizers and Parallel Extensible model with additional levels. You might have a mid-frequency level that updates its own weights every, say, 1,000 tokens it processes, and a slower-frequency level that updates every 100,000 tokens, and so on. The result is a model that can actually consolidate new information it sees after pre-training. It can learn new facts from a long document and bake them into that mid-level memory, all while the deep, core knowledge in the slowest level stays stable. It creates a proper gradient of memory from short-term to long-term, allowing the model to finally learn on the fly without just forgetting everything or suffering catastrophic forgetting.

The modern smartphone, laden with the corporate ecosystem pulsing underneath its screen, robs us of this feeling, conspires to keep us from “true” fullness. The swiping, the news cycles, the screaming, the idiocy — if anything destroys a muse, it’s this. If anything keeps you locked into a fetid loop of looking, looking, and looking once more at the train wreck, it’s this. I find it impossible to feel fullness, even in the slightest, after having spent just a bit of a day in the thralls of the algorithms.

The smartphone eradicates “space” in the mind. With that psychic loss of space, grace becomes impossible. You see the knock-on effects of this rippling out across the world politically.

Which is why these long walks of mine are so inspiring (to me), and I feel so compelled to head out on them, again and again: They are nothing if not “space generation” machines for the mind. They’re full-bodied reminders of what fullness is and how it can manifest. How close we are to it (it’s right there!!), every day, and how elusive it has become because of our digital habits, our diets of, mostly, garbage.

VPN company CyberGhost just sent Cloudflare a bogus DMCA takedown demand, claiming that our article about their last bogus copyright takedown demand, somehow violates their copyright.

I’m not sure I’d trust a VPN company that fucks up this badly.

There are a lot of sketchy VPN companies out there, and it’s sometimes tricky to tell which ones are legit, and which ones to be wary of. I would suggest that if your VPN company is running around sending totally bogus DMCA notices that’s a bad sign. But if your VPN company is sending bogus DMCA notices to take down stories about its bogus DMCA stories, well, then you really have found the worst of the worst.

Enter CyberGhost.

Almost exactly a year ago, we wrote about a bizarre copyright takedown involving CyberGhost. In that case, it had sent the takedown to Facebook because we had reposted the Daily Deal we had offered in 2016 for a CyberGhost subscription. As with all Techdirt posts, it had automatically reposted to our Facebook account.

For no clear reason, CyberGhost falsely claimed that Facebook post (but not our original post) violated its copyright (it does not). So yeah, this seemed like CyberGhost sending a copyright takedown of us running a promotion for their VPN from eight years earlier. How bizarre.

At the risk of sounding like a shill, Mullvad is the answer.

Worth noting ... the feds are coming for archival sites.

The Federal Bureau of Investigation is trying to unmask the operator of Archive.is, also known as Archive.today, a website that saves snapshots of webpages and is commonly used to bypass news paywalls.

The FBI sent a subpoena to domain registrar Tucows, seeking “subscriber information on [the] customer behind archive.today” in connection with “a federal criminal investigation being conducted by the FBI.” The subpoena tells Tucows that “your company is required to furnish this information.”

The subpoena is supposed to be secret, but the Archive.today X account posted the document on October 30, the same day the subpoena was issued. The X post contained a link to the PDF and the word “canary.”

“If you refuse to obey this subpoena, the United States Attorney General may invoke the aid of a United States District Court to compel compliance. Your failure to obey the resulting court order may be punished as contempt,” the document said. It gave a deadline of November 29.