00;00;04;11 - 00;00;27;21
Roland Ellison
Welcome to the latest edition of Lexicon with Interesting Engineering. I'm your host, Roland Ellison. This week we spoke to the partner and co-founder of Code Word, Karl Monson. Code Word is a digital marketing agency. As recently conducted an experiment with eight interns. We caught up with Karl to find out how Iko and Aiden had performed in their first 90 days on the job.
00;00;28;26 - 00;00;44;22
Roland Ellison
So welcome, Karl. Great to have you on the podcast to tell us about your recent experiment with Generative AI. I just wanted to kick off by if you could give us some background on code word, the type of work you do and some of the clients that you work with.
00;00;45;00 - 00;01;21;00
Kyle Monson
Sure. Yeah. And thanks for having me. Code word were a mid-sized marketing agency, about 100 people, mostly in the US and most of our clients are in the tech space. So we spend a lot of time working with tech companies, engineers, developers and helping them kind of turn very technical stories into content and social media and PR campaigns that people can understand, especially journalists can understand, but also enthusiast communities, fan communities and other like, you know, developer audiences and more technical audiences as well.
00;01;21;02 - 00;01;24;13
Roland Ellison
And is that mainly it's mainly consumer or a bit of both B2B and.
00;01;24;26 - 00;01;43;21
Kyle Monson
It's actually mostly not consumer. We do a lot of B to B stuff. We do a lot of like developer relations campaigns. We write a lot of speeches for big tech keynotes and things like that. The types of people that follow tech companies on Twitter and read Hacker News, that's kind of our our sweet spot.
00;01;44;01 - 00;01;52;28
Roland Ellison
Okay, what stuff? And so can you tell us a little more about the generative AI experiment that you've undertaken and really how it came about?
00;01;54;11 - 00;02;24;21
Kyle Monson
Yeah, sure. You remember back in November, these tools were really taking off. Everybody was talking about them, and there were all these kind of like deep existential conversations about how they were going to impact every different industry, really. And we decided that instead of pushing back and resisting, we were going to try and embrace it and see, you know, how do these tools actually exist in a creative environment like an advertising agency?
00;02;24;21 - 00;02;46;22
Kyle Monson
Can we use them? Are they a threat to our business model? Are they a threat to our employees careers? Or are these useful tools that we can you know, if we learn how to use them, they can be a competitive differentiator for us. So we went in that direction. You know, the team was a little bit apprehensive at first, but we've had a great time with it.
00;02;47;01 - 00;02;57;04
Kyle Monson
And I think we've we've come to realize some things about the tools and and really, you know, we see it as a we see it as a positive thing, at least for right now.
00;02;59;07 - 00;03;07;18
Roland Ellison
And why is it that you think your industry or your sector is particularly well-suited to to running an experiment like this? Yeah, that's a good question.
00;03;07;18 - 00;03;30;06
Kyle Monson
I mean, I think with your audience where, you know, you've got engineers solving very complex challenges and probably some particularly sensitive challenges through engineering and code and developing and science. We're not doing it for marketers right. So, you know, if we get it wrong answer from Bard or TATP or something, it's like, okay. Like, we'll rewrite it next time.
00;03;30;22 - 00;03;42;22
Kyle Monson
That's that's like, completely true. I'm being a little bit too flip with it. But, you know, there are lives hanging in the balance based on what we're doing with our generative A.I. tools. There are branded blog posts hanging in the balance, which is a very different thing.
00;03;42;29 - 00;03;45;01
Roland Ellison
Yeah, yeah. You know, interesting.
00;03;45;16 - 00;04;00;04
Kyle Monson
You know, our clients want our work to be accurate. Our audiences will certainly call us out if our work isn't accurate. But I think we're well positioned to have some fun here and and test the limits of these tools in a way that some industries probably shouldn't.
00;04;00;05 - 00;04;09;13
Roland Ellison
You know, I've received a lot of press coverage about the experiment. I just wondered what the what the reaction has been really to it from the sector generally.
00;04;09;19 - 00;04;14;05
Kyle Monson
It's funny. There's a lot of sort of anguish.
00;04;15;13 - 00;04;16;01
Roland Ellison
I think these.
00;04;16;01 - 00;04;37;22
Kyle Monson
Tools are really scary for people and force them to confront a lot of questions about their careers and the spaces that they work in and their own levels of creativity, which had always been true of tech. You know, that was true of the Industrial Revolution. That was true of the Gutenberg's Printing Press. You know, these tech breakthroughs have always kind of forced human conversations.
00;04;37;22 - 00;05;03;17
Kyle Monson
And I think we saw that with with our news cycle. We were one of the first companies to be like, hey, guys, we're really going to embrace this and experiment with it and see how it works. Yeah, not a lot of companies had done that in any kind of official way, like we had. And so it I think it prompted this kind of outburst of both positive both positive responses like, good for you, this is the exact right move.
00;05;03;17 - 00;05;11;04
Kyle Monson
And like, how could you do this? Are you going to lay off your junior staff? What does this mean for the future of writers and designers in the world?
00;05;11;14 - 00;05;12;28
Roland Ellison
You're killing creativity.
00;05;13;15 - 00;05;33;20
Kyle Monson
And creativity and it's like, sure, you know, we did have some fun with this. We kind of knew when we went into it that that that we were wading into some choppy waters. But it's been good to see the responses that I care most about our, my internal team and our clients. And they've actually been quite supportive and had a lot of fun with it and are learning.
00;05;33;24 - 00;06;05;12
Kyle Monson
So I think it's a good thing, you know, we we can either sit back and be behind on this stuff as other companies and organizations take the lead and learn how to do this stuff. Or we can take the lead and learn how to do it and kind of control our own destiny. Not to get on my soapbox, but I really think for my writers and designers, we've got about 60 writers and designers that code word and I think for them and for their careers look, if you're a writer in 2023, your career is already fairly precarious.
00;06;06;17 - 00;06;39;23
Kyle Monson
The opportunities are shrinking by the day for writers, whether human or anything else. A writer who can walk into their next job interview and say, I really know how to use these tools that work more efficiently. I'm really good at writing great generative AI prompts. I know the landscape and I'm going to do really good work for you by combining technology and my own human creativity and ingenuity, I think that that's a really great career move for them and it's going to open a lot of doors for writers at a time when writers need every open door we can get.
00;06;39;23 - 00;06;46;26
Kyle Monson
I'm a writer myself, so that's why I feel quite strongly about that. I don't know. I don't know. Do you agree with that? Do you think that that's the right move?
00;06;47;02 - 00;07;11;23
Roland Ellison
I think certainly, you know, the future of, you know, writing the right prompts and being able to interrogate this this these tools as best you can is definitely the future of work. You know, the huge amount of time that you can save from a creative point of view, just getting a general outline of a of an article or piece of work down, you know, in that sort of early brainstorming stage.
00;07;11;23 - 00;07;35;03
Roland Ellison
And I think, you know, particularly in your your industry and I have some experience in the credit sector as well where, you know, a lot of it for junior staff is Googling and trying to find imagery and things that actually I tools could probably do much faster and more effectively. It's an interesting one for sure. And I think, you know, stealing a march on the competition on it is the way to go.
00;07;35;19 - 00;07;45;09
Roland Ellison
Do you know of any other creative businesses who are sort of conducting similar experiments? I know some some publishers of have been running some some AI produced content.
00;07;45;17 - 00;08;09;21
Kyle Monson
Yeah. And that's way trickier to me than what we're doing. I think that's a which is one of the big tech publications. Our audience, those that came out about the same week that we made our announcement, it was discovered, I think, by futurism that that CNET was using generative tools without really telling anybody. That to me is a very different it.
00;08;09;21 - 00;08;36;09
Kyle Monson
It kind of changes the nature of the publisher reader relationship, in my opinion. And and then BuzzFeed committed to it as well, which I'm also much more on the fence about, because BuzzFeed, I think, is explicitly using generative tools to replace human staff, to churn out more content without having to hire more people, which that feels quite different than what we're doing and in much more of an ethical, shady area, in my opinion.
00;08;37;27 - 00;08;45;25
Kyle Monson
But you know, the media is a business too, and they're struggling. They're struggling to survive. So I guess I kind of get it. Any important in storm, right?
00;08;47;04 - 00;09;02;11
Roland Ellison
Yeah. And I think, you know as well, you've got that thing of robots writing for robots. So you've got, you know, AI writing for the search engines. And it's like I was actually conceiving this concept. We need sort of AI consumers as well, right, to give eyeballs to some of this stuff that can be a.
00;09;02;11 - 00;09;22;16
Kyle Monson
Good thing or a bad thing. You know, there's we got a lot of talk, people asking us, you know, what if a candidate wrote a cover letter using AI tools, I'd be like, Great. I don't really read the cover letter closely. They save time on the cover letters by writing them with generative AI. That's fantastic. They 100% do it.
00;09;23;17 - 00;09;39;18
Kyle Monson
They'll be they'll still be putting more thought into it than I am able to put into it. And but on the other hand, there's things like, you know, and when you've got the general AI tool, like helping you write your emails and someone else has a generative tool helping them read the emails.
00;09;39;27 - 00;09;40;06
Roland Ellison
Yeah.
00;09;41;14 - 00;09;43;05
Kyle Monson
The robots are writing for the robots.
00;09;43;15 - 00;09;43;24
Roland Ellison
Yeah.
00;09;43;24 - 00;09;51;12
Kyle Monson
But on the other hand, like their emails, I would love to have no more emails within the great world. A world where like robots take care of our emails for us. That sounds amazing.
00;09;52;26 - 00;09;55;25
Roland Ellison
There's a labor saving element that's definitely attractive. I'm sure.
00;09;56;20 - 00;10;04;06
Kyle Monson
Yeah, totally. And there's a downside, but I can't remember what it was. I guess I'm a tech enthusiast right now.
00;10;04;06 - 00;10;31;20
Roland Ellison
Okay. Well, I mean, just to talk a little bit more in detail about the experiment itself. Yeah. So you you basically had A.I. interns or you kind of had them as part of your teams. I think you had one on the design team and one on the on the content team. Can you tell us a little bit about what you learned through the experiment, which is I think nearly finished now, is that you've completed the 90 days yet or nearly it's finished.
00;10;32;12 - 00;10;53;20
Kyle Monson
We're about to go into chapter two of the experiment. Yeah, we learned a lot. I mean, one of the first things we learned is that these tools are not ready for production environments like there's not right yet. There are some things that they can do and save us a lot of time as creatives and there's a lot of stuff that we really can't rely on them for which, you know, no surprises there.
00;10;54;04 - 00;11;13;26
Kyle Monson
I wish I had something earth shattering to tell your listeners, but, you know, that's sort of where we're at with it right now. And we've been following all the all the new tools. You know, we got fired access a couple weeks ago we've been using for the design team has been using a mix of mid journey daily to and stable diffusion.
00;11;13;26 - 00;11;34;06
Kyle Monson
We've been using a mix of borrowed and chat shaped on the language model side. I think a lot of the stuff that we're getting from them isn't quite usable. The two things that we have learned positively are the space is changing so fast. Like the three months that we've been doing this experiment have felt like three years in terms of tech innovation in the space.
00;11;34;06 - 00;11;56;10
Kyle Monson
It's been wild to watch, which is another good reason for doing the experiment is like this space is changing so fast, like keeping up on these tools takes real time and intention. And the other thing that we learned is while they can't do necessarily production work, they can get us through lots of kind of creative roadblocks and hurdles.
00;11;57;26 - 00;12;18;08
Kyle Monson
You, you know, as a writer, sometimes the trickiest thing for me is staring at the white page and being like, I got a thing with words. Where do I start? The general chat tools are actually great at helping and just start, just kick the thing off. Here's some starter ideas to go with. You're going to need to do your own research, but like, you know, the page isn't blank anymore.
00;12;18;16 - 00;12;48;05
Kyle Monson
And the thing with the designers, you know, when we're doing like a logo exercise or a rebrand of a tech company, typically that kicks off with quite a long research phase where you're looking at the design elements of their competitors and comparing logos in the space and colors and fonts and creating mood boards and generative design tools can do that for you in seconds where it would take weeks of research to do it.
00;12;48;21 - 00;13;09;10
Kyle Monson
So, you know, it does provide meaningful shortcuts. And that's the nice thing about those shortcuts is like our human teams don't like doing that work. Designers know I'm doing research. They want to spend their time creating things. So if you can have the robots do the research for you, it really does like a shortcut quite a lot and can jump start your creativity at the same time.
00;13;10;07 - 00;13;15;26
Roland Ellison
So have you found that the it's been more useful for design than than content creation. That's a.
00;13;15;26 - 00;13;35;25
Kyle Monson
Good question. I guess so. Yeah, sure. Let's say yes for now. But I think the large language models are again advancing super fast. So I think that they've got a lot that's going to happen over the next even over the next six months. There's a lot coming down the pipeline that's going to that's going to improve the space quite dramatically.
00;13;37;12 - 00;13;48;16
Roland Ellison
Okay. And has there been anything that's really kind of surprised you about about the output, like things that you thought maybe they wouldn't be able to do well, that they have and things that you thought they definitely would be able to do well and maybe didn't.
00;13;49;00 - 00;14;10;08
Kyle Monson
Yeah. I mean, two of the most time sensitive types of creative work on the design side are hand-drawn illustrations and photo reel. Because if you're doing photo real stuff, like sometimes you actually have to get people in a room with a camera. Yeah. And the tools can actually replicate that fairly effectively already and they're going to get a lot better.
00;14;11;28 - 00;14;43;05
Kyle Monson
And then hand-drawn illustration this is ironic, but for brands that want that kind of handcrafted organic feeling, that takes quite a lot of manpower to do and it's quite expensive. So and the tools can replicate that quite quickly and make it look good enough. Like you're not going to hang it in an art gallery, but for a client presentation, like Perfect, it looks great, you know, it looks like we put a lot of thought and care and time into this thing.
00;14;43;07 - 00;14;44;01
Kyle Monson
You jokingly.
00;14;44;01 - 00;14;44;27
Roland Ellison
Crafted. Yeah.
00;14;45;10 - 00;14;56;16
Kyle Monson
Yeah. Which is sort of ironic. Like, that's sort of like the last refuge for human designers is like that handcrafted feel, but the robots are quite good at doing that.
00;14;58;01 - 00;15;11;11
Roland Ellison
And so, yeah, I just wanted to ask you about your clients and how, how they've responded to the knowledge that, you know, you might have been using generative tools on, on your work for them and whether there was any kind of pushback on that or concern.
00;15;12;23 - 00;15;36;14
Kyle Monson
Yeah, not yet. I think that there would be if we were using the tools in to overtly replace our creative teams. Our clients don't want that. And I will say, like our clients, many of them are leaders in the AI and machine learning space, like big one, big, big ones. And so, you know, they're not going to push back on us.
00;15;36;14 - 00;15;59;19
Kyle Monson
You know, in many cases, we're using the tools that they themselves are developing, you know. So yeah, we haven't gotten any pushback on it, but we haven't really pushed the limits of that either. And I don't think that we will. We're an agency like our the product that we sell is is the time of creative teams. I'm actually not looking to replace that in any meaningful way because it it undermines our business model.
00;15;59;19 - 00;16;25;14
Kyle Monson
And I think it undermines the work that we're doing for our clients. To replace full creative teams with robots would be quite bad for us. It also, you know, not to get too much into the business specifics, but there's a certain competitive moat that we have by being the ones who know how to use these tools. I don't want to make it too easy for our clients to just, like, excuse us and bring all this stuff in-house.
00;16;26;16 - 00;16;27;04
Roland Ellison
No. Right.
00;16;27;21 - 00;16;38;14
Kyle Monson
So so the mix of human and human and AI creativity is quite important for us to get right. It's actually like an existential question for our business, for our whole industry, really.
00;16;38;14 - 00;16;47;20
Roland Ellison
Well, I guess you're kind of, you know, helping them sort of road test them for for for real life kind of professional work and testing the tools. And I get.
00;16;47;28 - 00;16;53;23
Kyle Monson
That if we can be really good at this, that it's going to be a win for us and not a loss, which I think is a pretty safe bet.
00;16;54;14 - 00;17;16;11
Roland Ellison
Yeah, no, for sure. Sure. So now you say you should have completed the initial 90 day experiment you're moving into to phase two? Yeah. What does phase two look like and is it an ongoing thing or are you going to do another sort of time defined experiment?
00;17;16;11 - 00;17;33;25
Kyle Monson
We are. We're going to keep going with it. I think we've got a bit of a fork in the road right now for where we take the team. You know, our strategy team has already asked like, can we have a design intern? And I'm like, Yeah, go ahead. Like the tools are open, just open a browser tab and boom, you've got a, you've got, you're an intern.
00;17;33;25 - 00;17;58;09
Kyle Monson
And meanwhile our like kind of formal interns, Aiden and Iko. Yeah. I mean, they've completely complete successfully completed an internship, you know, what do you do with people like that? Either hire me or let them go, right? Or they go on to find new opportunities. So, you know, we're thinking about what it can mean to have them as more permanent team members.
00;17;58;09 - 00;18;01;22
Kyle Monson
You know, they're I think that they're also exploring other opportunities.
00;18;03;11 - 00;18;03;27
Roland Ellison
Did they.
00;18;06;04 - 00;18;06;11
Kyle Monson
What.
00;18;07;00 - 00;18;10;09
Roland Ellison
Did you conduct? Look, a performance review.
00;18;10;20 - 00;18;29;25
Kyle Monson
We did, yeah. And we're we're going to publish our next one in I think next week, I believe. But yeah, you know, they're ambitious. They've got big dreams for their careers. They've got a lot of dentist appointments on their calendar, which is a little special. So I think they might be interviewing at other places. We'll see.
00;18;30;03 - 00;18;30;28
Roland Ellison
These. Gen Z.
00;18;32;03 - 00;18;32;14
Kyle Monson
Right.
00;18;34;12 - 00;18;48;14
Roland Ellison
And so. So how do you see I mean, you touched on it a little bit earlier, but how do you see the future of generative AI tools in the creative sector and how do you they're going to change the way your industry works in the next sort of five, five years or so?
00;18;49;27 - 00;19;21;26
Kyle Monson
My hope is that my hope is that we get to a point where creatives, whether you're a designer or writer or whatever developer, we just kind of always have a tab open in our browser and we know how to use it. Basically, even right now, you know, I've got what do I have open? I've got chatty beta open, I've got my discord, my mid journey discord channel that I turn to from time to time.
00;19;21;26 - 00;19;36;06
Kyle Monson
And so those are just always open. They're just two tabs on my browser that I've always got, that I've always got open. And I'm, I'm still trying to figure out how to best use those tabs and I think we're all going to go through that over the next year. I think that's I think this is a near term.
00;19;36;06 - 00;19;51;00
Kyle Monson
A shift is like that's over the next year. At the same time, I'm putting on my like prediction at the Internet's going to fill up with like a 100 X level shit just like crap everywhere online.
00;19;51;03 - 00;19;51;29
Roland Ellison
No more.
00;19;52;19 - 00;20;19;03
Kyle Monson
Yeah, yeah, exactly. Right. A flood of content, the likes of which we've never seen. And it's all going to be low quality and it's going to be terrible, and you're going to know that it's coming from gendered sources. And honestly, I don't think it's bad. I that might be an okay thing because it's going to teach us. It's going to teach us that you're going to need to trust your sources of information maybe a little more responsibly, which I think in the UK and the US has been a real challenge for a lot of people over the past.
00;20;19;03 - 00;20;43;15
Kyle Monson
Let's say six years. So, you know, maybe the, the flood of shit content and misinformation and just absolute crap on the internet is going to finally end this kind of age of credulity we've been in since Gutenberg invented the printing press. We're going to learn, you know what? You can't trust anything you read, so you better pick your sources pretty carefully.
00;20;43;15 - 00;20;48;09
Kyle Monson
Yeah, I don't know. That's like a societal level prediction. Sorry. I don't know if that's the way you want to me.
00;20;48;09 - 00;21;17;02
Roland Ellison
No, that's that's absolutely the kind of question that we we were driving at. And I think, you know, do you think there will be big sort of ethical questions arising for businesses using these these tools? I mean, what is it about, you know, declaring their use on on on content, particularly? I'm thinking of publishers, but how do you how do you see the sort of path forward for the people who are using this in a public facing role?
00;21;17;02 - 00;21;32;05
Kyle Monson
I don't know how I feel about that. I, I think that there's get there's always technological pressure to keep innovating and to adopt new tools. There is a call maybe two weeks ago that we should have a six month suspension of any research.
00;21;32;13 - 00;21;36;16
Roland Ellison
With Elon Musk and a few other signatories. Yep.
00;21;36;16 - 00;21;57;20
Kyle Monson
And I'm like, come on. Like, who's going to enforce it? Like, even if you do enforce it in the US, that just means that some other like some other tech scene elsewhere in the world who doesn't have it is just going to pick it up and and keep moving, you know, push past this. So I do think that there's like real pressure to always be pushing forward and always adopting new tools.
00;21;57;20 - 00;22;31;09
Kyle Monson
And, you know, in the media industry and the tech industry and marketing, I don't think we're going to slow down. I do think that the tech companies are going to do more to integrate these tools into existing products that we use every day. And that's going to, I think, make this field safer for people. You know, when when when your generative design tools are in a discord and they're creating all kinds of stuff like that's a little bit scary when there's just the camera on your phone, it's going to feel a lot less scary.
00;22;31;09 - 00;22;46;20
Kyle Monson
And that's already been happening. You know, there's air all around us all the time. We just we don't always know it. Like we're talking on Google Meet right now, which has really great noise cancelation tools that are powered by machine learning. I don't know if you could hear my dog with the squeaky toy earlier, but I don't know.
00;22;46;20 - 00;22;49;12
Roland Ellison
Only caught one squeak. But yeah, not. Not yeah.
00;22;50;07 - 00;23;07;14
Kyle Monson
And if my dog barks and if there's construction outside and if there's birds chirping like the AI in Google Meet is going to cancel a lot of that stuff out on its own without me ever knowing it and without there being a big sign. That's like the robots are helping you right now. Yeah, yeah. So I think that that's much integration.
00;23;08;05 - 00;23;20;12
Kyle Monson
Yeah, it's, it's already been happening over the past ten years. It's probably going to speed up a little bit. And I think that the big tech companies are going to be fairly responsible about doing it in ways that don't feel threatening. But we'll see.
00;23;20;29 - 00;23;44;08
Roland Ellison
That they have such a great track record of being responsible with this stuff. As we approach the singularity, just wanted to ask finally, where can people find out more about the experiment and your findings? Like, I know you plan to sort of publicize some of the some of the results and some of your thinking on it.
00;23;44;08 - 00;24;04;06
Kyle Monson
Yeah. I mean, if after listening to this podcast, you still want to hear more about code words experiment, you can you can find us on LinkedIn. We're on medium. Medium is where we're posting most of the results for now. And yeah, you can keep up on the experiment. We're posting LinkedIn and Twitter updates as well or any marketing agencies can be found.
00;24;04;09 - 00;24;21;05
Roland Ellison
Exactly. And I'm sure people would love to see some examples of some of some of the outputs that you've had. That would be that would be great. Thanks very much for joining us today, Kyle. While we're on the subject of AI, we recently set out our own policy for the use of generative AI on the interesting engineering site.
00;24;21;21 - 00;24;38;16
Roland Ellison
We'll put a link to that article in the podcast description along with links to the code word blog. This has been Lexicon with Interesting Engineering and me your host, Roland Ellison. If you enjoyed the podcast, please subscribe and leave a review until next time. It's good bye for now.
Transcript
