PJ Pereira has spent 30 years in advertising, co-founding Silverside and Pereira O’Dell and leading Emmy-winning work. He sat down with Hyde co-founder and CEO Anirudh Sanga to talk about why professional design needs its own benchmark, why a brand’s taste is its IP, and what that means for creative teams.
Ani: Well, thanks PJ for having us in your office. This is an incredible space. My name is Anirudh. I'm the co-founder and CEO of Hyde. You're PJ Pereira, co-founder of Silverside, co-founder of Pereira O'Dell. You've worked with brands like Coca-Cola, Lego, TikTok, over almost a 3-decade career in professional design. And we've jointly built AtelierBench, which is quickly becoming the frontier benchmark for professional design work. Excited to have this conversation with you, and let's get right into it.
Why does professional design need a benchmark?
Ani: First question, why build a benchmark for professional design at all?
PJ: So I have been at least 30 years in the business. I have been part of lots of discussions about what's good and what's not, what's acceptable, what's not, what's great and what's not. I've been president of jury or jury in all of the biggest festivals in the world. And so I spend a lot of time trying to understand what good work is. And I always felt that there was a— everything's so beautifully intuitive, but sometimes you need a little bit more precision. And our creative direction is not a very precise world, for the good and the bad. And having tools that allow us to see and score and understand what our impressions are really, really helps.
Ani: Yeah. We're seeing the same thing across any industry that is working in sort of non-verifiable domains. So non-coding, non-math related, broadly speaking. It's quite hard to kind of judge a stochastic model on things that are just very, very taste dependent. Most enterprise work is taste dependent and it's based on judgment of years and years of doing the work.
When we started working with you, the biggest challenge was how do we actually understand what is good AI output for creative work, for briefs that your clients are sending you and the sort of assets that you're producing for them. The way we started tackling that, as you know, is look at hundreds of real client briefs and outputs that your people generated and you generated. Break that down into claims, thousands of claims of, you know, why a certain image was made a certain way, why a logo went here versus there, and really create this sort of machine of: can we evaluate AI output as the models change, as new models are becoming available? Is there a way for us to test how good these models are for the kind of work we do? And eventually use that delta between where the model is and where the actual client-acceptable work is to build our own AI.
PJ: One of my favorite parts about this is how you actually separate the good and bad in enough dimensions so I can see things separately. And because every assignment is different from the other, the revolution comes from when you do this and then you look back and you start to understand your own understanding. And this metacognition, I think, is the big leap forward.
I remember a few years ago, I did a project with Nicholas Negroponte, founder of the Media Lab at MIT. He was sharing how programming was a revolution for humanity, not only because of what it does, but because it forced human beings to think about thinking. And that changed the way we understand our own thoughts. And I think that this wave of generative AI is allowing creatives to create creativity. But in order to do it right, we need to have some sort of mechanics underneath. That we can understand why we've made some cognitive intuitive decisions so we can tell the machine, don't do this, don't go there. For these kind of projects, this is more important than this. Otherwise, the machine is going to do what the machine thinks is better for the overall world.
Ani: Yeah, metacognition is a great way to put it. It's basically when we think about building AI or building AI for Silverside in the way that you think about it and not just leveraging someone else's AI. It's really about firstly thinking about: is our base AI generating against kind of our own judgment? Is it being evaluated against what we think about correctness? What is shippable to a client? But equally, can we encode that judgment of ‘what is correct’ into a judge model itself? So you have a generation model and a judge model, and they're teaching each other what's good, and they're over time evolving as they see more client briefs, as they see kind of more ways of working, as they generate taste from kind of the people that are working here.
PJ: One of the simplest benefits of this, which is pretty cool, is that my taste level, not me as an individual, but our collective taste here at Silverside becomes IP, right? It becomes something that we own and is documented, it turns into a dataset. It's not the whole taste. But it's a good piece of it, right? And allows us to determine and know how to get to the kind of things that we like. That is the other part that is not codified. It's still the things that can get much faster there. So one way or another, it works between the robots and the humans. Together we can get to the work that we like.
But the fact that we don't have to worry so much about what is the path to get to the work that we want, saves us so much time, so much energy, which means that we can spend more of the time that we have, the very limited time that we have, into pursuing the perfection, fine-tuning the things that we think are really important, and I think that is the baseline of everything. That is where the transformation happens.
But then when you imagine everyone doing that at the same time in different brands, different companies, and different creatives, different teams, it allows us to see more clearly as well what is the beauty of diversity of thinking. For some of my clients, cinematography, the style of the camera move is the most important thing for them together with the precision of the product. For others, precision of the product doesn't matter that much because they can always fix it in post if needed. What matters most is if the environment where they're being seen feels real.
Understanding that is a fascinating thing. It's always fascinating for me how if you put 10 people that I have tremendous respect for in the room judging the same work, they're all going to see it slightly different. I learned something by looking at it that way. I think we're going to become more self-aware of our own instincts and tastes. And self-awareness of taste, I think, is always a good thing.
Ani: It's interesting you mentioned those 3 or 4 dimensions. All of those are dimensions we test in AtelierBench. You touched on something interesting there around IP and kind of capturing that for the sort of AI-native world of design.
Why keep the benchmark private?
Ani: One of the questions we got when we released the benchmark was, why is it private? Why shouldn't we release this out to the public? And I think the sort of key answer to that is it is the company's IP. We support open source and we support open models, but the moment you kind of release this benchmark to the public, the frontier models just kind of start hill climbing this benchmark and start becoming very, very good at the specific things that this benchmark measures.
Quality of kind of keeping it private is twofold. One is it's your IP and it kind of becomes this asset that you evolve over time. It stays inside the company's four walls. Number two is you don't give the answer sheet to the model builders. So as new models are becoming available, you have your own kind of evaluation rubric to measure these models against that are independently developed from your workflows.
PJ: Of course, there's from a competitive standpoint as an entrepreneur, as an agency owner, as I want our knowledge to stay here, our dataset to be our dataset. Everything that AI produces comes from a very general dataset that is built by the behavior and the taste and the input from the whole universe, right? As creatives, we don't want to have the average of the whole universe. We want at the very least to have the average of the very best. But once you start to have the average of the very best, you realize that that's not enough. You actually want the average of your own vision because we creatives direct with our perspective. You can codify, you can put numbers, you can score it, but it's ours as individuals and as groups more than anything.
And even I don't want others to have access to mine because I feel protective of it. But I also don't think they want access to mine either. I want my things to look like my things. And if I had access to the mind and the brain of some of my favorite creatives in history, I don't want to do that because I'm not them. I know some people who are incredible writing humor. Everything they do is so funny and it's very specific kind of humor that is not my humor. I want the freedom, but I also want the style, I want some unity because that's what the industry comes to us for, a specific kind of taste, a specific kind of execution, and the way we work.
You know, I say that if 10% of a label of a product is wrong, the product is 100% wrong. Right? And the level of precision that we require on products, people from entertainment, they don't get it. For me, that's very important because I'm being paid to put that product front and center. I want to be able to have my models and the way I prompt them and the way I operate them to be perfect for what I do. I don't want the entertainment people to know exactly how I do it because otherwise they're going to take my lunch, but I don't want theirs either. Because there are things that I can learn from them. We can exchange information. I can invite them here and I can go there and talk about the things. We're super open in the way we do it. I want to make sure that our way is our way and not someone else's.
Ani: Yeah, that's a really strong point. Good taste is defined by what people aren't doing yet, and the moment everyone starts doing the thing that kind of defines good taste, it stops being good taste, which is the problem with general purpose models and creative work: it is general purpose. It's trained on what the sort of median is doing in that category of work.
PJ: Because taste, we were talking about how taste is IP, but taste is also identity. Right, and I think that we need to have our own. When we can teach our instruments to understand our identity, they become better instruments. If you look back to before artificial intelligence, it's like we're evolving from riding bikes to riding horses. Before, we wrote these things that would obey exactly what we were saying, but all the muscle, all the power came from our legs. Now we have these gigantic beasts with their own muscle, their own power. But if they decide to stop and drink water, they will. And they're much bigger than us. We cannot control, but we can learn to communicate. I can teach the horse to understand what I mean, and I can learn what the horse means. That communication, I think, is what you guys are codifying. You're actually helping the rider and the horse operate as a better team. We didn't need that for bikes, because bikes only obey. Horses make decisions.
What does it mean for brands and creative teams?
Ani: Great. So last question, you know, we built this benchmark, we have a way to now codify taste, codify IP. What does this mean for your clients? You know, how are we kind of expanding this to individual clients and helping them encode their own taste and their IP into their own design language?
PJ: I think the short answer for that is productivity, right? It's much faster, much easier. This chaotic world of artificial intelligence where you never know exactly if the tools are going to give us what we expect or not. Being able to learn what tool is better for what task and for what clients and based on what person accelerates the whole thing. Right, it makes everything much faster. And for a different client, it can be.
Now I can actually see all these things and I don't have to operate in a general standard by asking the world, okay, let me look at the ranking of what is the best model in the world. The best model in the world doesn't exist. You know, sometimes it will be the cost to get to a result that we can, that is usable. Sometimes it's going to be, you know, cost doesn't matter, it's actually the performance. Sometimes it's going to be the style. And every client, every project, every person in the team is going to be different. I have tools to make sure that my team always operates at a very high minimum level. Their minimal level of delivery is much higher because there's nothing that goes below that level. And now they can focus on the high level.
So the productivity is the short answer, but I think that the real game is what do we do with those gains? What do we do with what we are getting? I see a lot of them, the conversation about AI ends up get into, oh, this is much cheaper, much faster. And I think it's just kind of a little limiting and we're shrinking our own potential. When I started my career, I had to spend money on 3 things per year. Now I have 3,000 things per year to make. And I really need to make sure that I'm productive so those 3 things that really matter, I have time for them. So productivity is the ultimate tool so I can put the attention of my personal bandwidth on things that matter, and can allow me to build up to the ideas that I care about.
So I think that what we are doing here is trying to find a solution for the problem of our creative generation. It's almost like the gods of creativity have been hearing us crying and begging for a miracle for a while, and the gods of creativity couldn't find that miracle. One day they called the gods of engineering and said, "Hey, let's make a deal here." And they found a creative tool that allowed that miracle to happen so we can do all, take care of all these thousands of things that we need to do with a good quality so we can spend time enough on the important ones again. And then we got the miracle and the world is still complaining about it.
Ani: Yeah, no, humans definitely seem to be moving one level up the stack. We're seeing this every day now working with you guys. The core problem that I feel like this solved and some of the findings we found in building this benchmark was CMOs were just adding more tools to the repertoire, to sort of the toolset that their designers had, which wasn't really giving more sort of better output or gains in sort of the output that the creative AI-enabled team was producing. It was just like more draft style work that just led humans to like do more work and just the final like ship-or-redo calls, rather than actually one-shot the output and think more creatively about like, how do I amplify this, or what's kind of like a more leveraged thing that I can do with my time.
We saw that increasing the sort of harness length increased cost by 40% or something, but didn't really have any major impact on the output, especially with large models. With small models it did, but that also plateaus over time. So we think the sort of direction that this will go into is building these taste models, building judge models, and really kind of enabling the designers to one-shot the output that they want, without having to worry about, okay, is this going to be like more of the average output that I'll have to like spend more time fine-tuning and like figuring out how to make it shippable? And just move to that next level of the stack where like, okay, my tactical work is now automatable and I can trust that. What's next?
PJ: The problem in the creative world, the bandwidth problem, has never been with the ideas. It has always been with the decision, with the picking of the right idea. And you can see it in the structures of all the companies. Every creative company in the world has multiple creatives and then a few that work under a creative director. What is the bottleneck? By definition, the bottleneck is the creative director. So if that creative director is the only instrument of filtering what goes forward, you either increase the number of creative directors or you have to make these people below be able to make better decisions.
I think that once we bring more tools and more information to the system, conceptually what we're doing is that we're allowing the tools to alleviate some of that bottleneck. So the creative directors don't need to say no to things that he shouldn't even be seeing, the creative working on something should have something flagged, okay, this is better. This can be better. I'm going to work on this again. Or, okay, I don't know exactly how to make this, but I know that this other person has done this better than me, so I know that it can be done. I know that he can do it, or she can do it, and I'm going to ask for information. I'm going to make it. So when it gets to the bottleneck, it is probably ready to go. And that is a transformative mentality to the entire ecosystem.
Ani: Yeah, a world full of super creative directors. I'd love to live in that one. It feels like super intelligence for design is basically what we're going towards. Well, thank you, PJ. I think this is a good time to end it.
PJ: My pleasure.