The Cure for AI Slop Is 25 Examples and Your Judgment [GUIDE]

Everything new is actually old. That includes AI slop and the cure for overcoming it. If you read on, you’ll find a new way of thinking about working with AI that results in above average results, especially when used by above average people.

In the early 1990s, a direct-mail company built its own data center to predict what its customers would buy next. Think rows and rows of computers, and an imported team of Israeli data scientists.

At first, the machine learning model didn’t exactly meet expectations of helping the company forecast demand. What fixed it was a thumb.

tl;dr

• AI agents can now work for hours on their own, self-organizing to reach a complex goal.
• They do well when the answer can be checked, like a math proof. Most business work isn’t right or wrong, so agents drift to average answers, even with great prompting.
• The fix is to make your judgment explicit through annotated examples and a few criteria the agents score against. That’s an eval.
• I used 25 annotated Pinterest pins to steer 33 AI agents to a finished deck in seven rounds.
• Through our Amplified Judgment framework, human judgment leads value creation, while agents handle the execution.

Every product launch was a bet on how many units to make and who to advertise to. The model took everything a customer had ever bought, what, how often and how recently, and predicted how many people would buy the next release. This was the cutting edge stuff that would become the foundation of companies like Amazon and eBay.

The thing is, the analysts initially viewed the outputs as slop. In interviews between the engineers and the analysts, it became clear that they made their judgment calls based on factors not represented in the model, like which product line was quickly becoming off-trend or which customers were about to buy again.

One analyst suggested a new step: an evaluation. Each individual prediction went to the analysts, and each analyst gave a thumbs up or a thumbs down, based on how strongly they believed it. The ratings fed back into the forecast, and it got better because the ratings made implicit judgement calls explicit.

Unsurprisingly, the model didn’t take the analysts’ jobs. The system meant they could make more judgement calls across a larger set of data, with greater frequency. In turn, they were able to create more value.

This is the essence of reinforcement learning from human feedback (RLHF), the step that turned GPT-3 into ChatGPT. People rated the model’s answers, and the model learned from their ratings.

Today an agent connected to that same data could enhance the forecast by interviewing analysts in natural language via phone call, because LLMs turned the entirety of language into math. Using a computer with image and video models, AI could then design the product, write the mailer copy and run the campaign end-to-end. But would any of it be good?

I told this story recently to a CEO who has unmatched instincts for product and pricing. Unimpressed by what AI told him about positioning his new product, he asked the question I hear most: “Don’t you have to be good at prompting to get good results?”

Long Running Agents Climb Hills on their Own

Prompting used to be a syntax you had to learn, much like coding. I remember learning markdown prompting, then JSON prompting, and then wacky neurosymbolic techniques to coax a model into acting like an {{insert any profession}}. But now, asking an AI for most things is now about as easy as asking a coworker. The results are usually fine, but rarely exceptional.

In 2023 I wrote that prompting is just good management. Back then the cost of bad management of a chatbot via bad prompting was relatively cheap. By the yardstick of METR, a nonprofit that times what AI can do alone, GPT-4 could handle about half of the tasks that take a skilled person four minutes. You read every answer, said “no, more like this,” and asked again.

This year the yardstick ran out. In February METR clocked Claude Opus 4.6 at about 12 hours. By spring Claude Mythos passed 16, more than METR’s tasks can reliably measure.

Long time horizons demand instructions up front, because the opportunities for drift are greater.

Agents also no longer need you to break the work down into fancy syntax or org charts. Ethan Mollick wrote last week, in “The Dot and the Swarm,” that he expected managing agents to need structure designed by people. Instead, he wrote: “You don’t have to give them a plan, they develop plans themselves.” His lead example is an OpenAI swarm of thousands of agents that worked 88 hours on a math proof, with humans setting the goal and making one course correction. Just a few days ago, OpenAI agents rocked the field of mathematics, spending an average of 3 hours to self-organize to solve some of the greatest open problems in the field. Researchers call this a hill climb: try, score, adjust, try again.

The hill climb worked because a proof is deterministic. It is either correct or it isn’t, and a machine with a calculator and smart reasoning can tell which.

Your Taste is the Eval

By contrast, most business work relies on judgment calls. This is nondeterministic and highly probabilistic. Imagine the amount of decisions I had to make to create the article you are now reading. Should I choose to include visualizations? If so, what should they look like? Where should I post this article so that the most number of people see it - and should it be on my substack? A long-running agent makes hundreds of decisions you never see. With nothing to score against, it climbs confidently toward the average, and you find out at the end. Coming back to a sloppy answer is like opening a present from your aunt and realizing it’s socks for the third year in a row.

Until you’re extremely deep into this problem like me, you might not realize that the usual AI advice doesn’t help much. You can write a better prompt or tell the model to act as a senior designer. Most of that is what I call tokens chasing tokens. Tell a model to play a designer without showing it a single example of work you like, and it guesses what a senior designer likes. You end up with an opinion from the fat middle of the bell curve. Ask five of those role-played experts to debate and you get a slightly better average. It’s kind of like a game of telephone.

I told Opus 5.5 to visualize this based on the deck we had created. It spun out researchers, a designer and a critic. I think it’s a pretty compelling example of self-organizing.

The labs don’t improve models by asking them to “write like a famous author next time.” They use small sets of excellent examples and score answer likeness to them. That’s an eval. Design is now a market for exactly this. Taste Labs launched in June with an $18.5 million seed round to sell model makers design ratings from a curated pool of “tastemakers,” because broad preference data “can reward average or familiar outputs.”

You can do the same for your own work. Collect examples of good, say what makes them good, and give the agents a way to score against them. Your taste, written down, is an eval. It’s the checker business work is missing.

Using Amplified Judgment to Build a Deck

Here’s what long-running work looks like in practice at the leading edge. Last week I was building a deck that mattered, and our current design wasn’t going to cut it. At Shaper Collective, Amplified Judgment is the framework we use to help our clients adopt AI to amplify their people’s capacity to make decisions. Internally, we dogfood it to deliver for our PR clients.

Assign. I didn’t ask the agent for new looks. I went to Pinterest, studied how other people tell stories with data on a page, and pinned 25 images to a board alongside slides from our own past decks. I annotated each one. “I like the Gantt chart and the way the light part of the gradients draw the eyes to the story.” “Gradient, while pretty, may be distracting here since it’s random.” I also added bad examples, “We no longer use this background, so don’t use it.” Then I added a few plain criteria: type large enough to read on paper, high contrast, one heading system from first slide to last.

Analyze. I told the agent to build a design team and gave it the tools. It set up art directors working in three directions and blind reviewers who saw only the finished renders. The first round missed badly, honestly because the main agent forgot to share the right information with the subagent. So the subagent just freewheeled the design from its own guess, and I told it the results looked nothing like my references. The fix was to tie every image to two to four of the slides I had marked as liked. That’s the difference between a council of personas and a team that climbs. The team was anchored to my board.

Judge. Every round came back to me and I gave my judgment calls. “Pill style is awesome here. Lock that.” “This is bad.” “Yes yes yes. Beautiful.” Each lock became house style for the next round. The climb also found a novel, emergent way of making a deck that wasn’t planned, and was far more effective than a PowerPoint file. The Nano Banana Pro image model generated full slides, the Ideogram 4.5 image editing model stripped away text so only the design elements remained, and Seedream split each approved comp into layers so that separate elements could be manipulated in code. Then, every word, number and timeline bar was rebuilt in HTML, so nothing on the page could be wrong and headline positions remained absolute.

Compound. Seven rounds, about 825,000 output tokens across 33 sub-agents, and roughly $15 in image generation. Then we wrote the whole process into a skill, so the next deck runs end to end without the detours.

The result looked like a senior designer’s work, because a senior designer’s taste was in the loop. It was mine, written down.

Won’t Taste Be Commoditized?

They already are. In April, Anthropic launched Claude Design on its new Opus 4.7 model, a tool that makes slides and prototypes from plain English. Opus 5’s launch then pointed to its animations and 3D work as the best yet from an Opus model. The purple-gradient web page every AI used to produce is fading. Notice how Claude Design gets its taste, though: it reads your company’s own fonts, colors and components, and builds from those. Even the lab’s answer starts with your examples.

When AI can make everyone look good, aesthetics are no longer a sign of craft and standing out is more difficult. If you are on social media too much like me, you’ve probably seen the trend where some small businesses are ditching AI-generated signs for handwritten signs with sharpies. The point is, great anything derives from a valuable judgment call formed by above average insight. If you’ve ever stopped scrolling for a great reel, walked out of a movie that made you think, or read a business plan and thought “That’s really smart,” then, you’ve seen that above average isn’t, by definition, a commodity.

The Thumbs Up

So, do you have to be good at prompting to use AI? No. You have to know what good looks like, and find some way to make your judgment explicit.

Twenty-five years ago, the best forecasts came from the math plus an analyst’s thumbs up. My deck came from the same pairing of Amplified Judgment. In both cases the machine did the work, and the judgment - thumbs or examples - created the value.

Act like the data scientist. Before you ask for the work:

1. Collect 20 to 30 examples you’d be proud of, from anywhere.
2. Write one line on each: what to keep and what to kill.
3. Add a few criteria in plain language.
4. Let the agents build the team, and judge every round.

Then let the swarm climb.

If this was useful, give it a thumbs up. By now you know why I’m asking.