When AI generates work, who’s accountable?

00:00:00,600 --> 00:00:31,040 [Eric] [upbeat music] Welcome to the Token Intelligence Show. AI is changing the way that we work, and Token Intelligence will help you understand the state-of-the-art, cut through all of the noise, and apply wisdom to become an effective leader in the age of AI. So we're gonna do that today by talking about accountability 00:00:32,160 --> 00:00:44,340 [Eric] in the workplace, uh, with AI really changing the way we do so many things and becoming an integral part of how a lot of people are performing their jobs. 00:00:44,340 --> 00:00:50,460 [John] Yeah. 'Cause this came up 'cause we were talking about how there's a lot of conversation about taste and judgment right now. 00:00:50,460 --> 00:00:51,140 [Eric] Mm-hmm. 00:00:51,140 --> 00:00:54,240 [John] And I haven't heard as much about this topic, so I'm excited to dive in. 00:00:54,240 --> 00:01:06,020 [Eric] Yeah. Let's... Okay, on the state-of-the-art side of things, w- why is the taste and judgment conversation so important, do you think? Or actually define that maybe for listeners- 00:01:06,020 --> 00:01:06,060 [John] Yeah 00:01:06,060 --> 00:01:14,340 [Eric] ... who haven't been as exposed to that, you know, as sort of us in, in the Silicon Valley heat of the moment with AI. 00:01:14,340 --> 00:01:24,539 [John] Right. Yeah, so I think taste and judgment were the, the two words that have come bubbled up to be the things humans contribute when working with AI. 00:01:24,540 --> 00:01:24,830 [Eric] Hmm. 00:01:24,830 --> 00:01:36,000 [John] Like human, the... So we'll start with taste. So a- at least so far, we've found that humans can do better with, let's, let's call it, like, aesthetics and kind of feel- 00:01:36,000 --> 00:01:36,220 [Eric] Mm-hmm 00:01:36,220 --> 00:01:39,140 [John] ... look and feel than an, than an AI just by itself. 00:01:39,140 --> 00:01:39,520 [Eric] Yep. 00:01:39,520 --> 00:01:48,340 [John] And then judgment, I would just think of evaluating outputs with some kind of human standard and then iterating with AI to achieve that output. 00:01:48,340 --> 00:01:55,870 [Eric] Right. Yep. Uh, I have a, an example of this, which is, uh, building a website, right? 00:01:55,870 --> 00:01:55,920 [John] Yes. 00:01:55,920 --> 00:02:03,780 [Eric] So I think every single company has gone through the process of redoing their website, and 00:02:04,880 --> 00:02:09,460 [Eric] there used to be a technological barrier to this. 00:02:09,460 --> 00:02:10,280 [John] Right. 00:02:10,280 --> 00:02:36,980 [Eric] And so you would hire an agency, or you would have engineers do this in-house, and you have all these competing needs of people who need to publish content, so you need some sort of content management system. You know, you have designers who wanna, you know, make the site look a certain way. You have marketers who want it to perform and convert a certain way. But the technological barrier was really significant. 00:02:36,980 --> 00:02:37,100 [John] Right. 00:02:37,100 --> 00:02:44,410 [Eric] Right? And so what ended up happening was you converge on frameworks. So WordPress is a great example. 00:02:44,410 --> 00:02:44,440 [John] Yeah. 00:02:44,440 --> 00:02:52,380 [Eric] The most popular, you know, website, um, platform in the world. I think it runs a third of the internet, uh- 00:02:52,380 --> 00:02:52,390 [John] Maybe 00:02:52,390 --> 00:03:05,680 [Eric] ... last time I looked at the statistics, which has been some time. Uh, and then within the WordPress ecosystem, you have templating systems, right? So then you don't have to, you know, do everything by hand, and then you sort of m- you know, modify templates, right? And so- 00:03:05,680 --> 00:03:06,000 [John] Right 00:03:06,000 --> 00:03:08,960 [Eric] ... that's why a lot of corporate websites end up looking similar. 00:03:08,960 --> 00:03:09,960 [John] Yeah, roughly the same. 00:03:09,960 --> 00:03:32,850 [Eric] Um, you know, which is interesting, right? So if you live in a world where AI can generate any sort of website you want, it could hook it up to any sort of CMS that you want, you know, so that your content team can publish, it can make it performant, then what used to be sort of this, this taste of like, "Ooh, that's a very avant-garde- 00:03:32,850 --> 00:03:32,880 [John] Mm-hmm 00:03:32,880 --> 00:03:38,440 [Eric] ... website," you know, there was a technical barrier keeping most people from investing a lot of their time there. 00:03:38,440 --> 00:03:38,740 [John] Right. 00:03:38,740 --> 00:03:48,240 [Eric] If that barrier's gone, then the taste of your design aesthetic, user experience becomes the new competitive advantage. 00:03:48,240 --> 00:03:48,620 [John] Right. 00:03:48,620 --> 00:03:50,300 [Eric] Right? Um, is sort of how 00:03:51,420 --> 00:04:16,740 [Eric] I've heard that discussed. On the judgment side, I think it's related, but a lot of times that can be more technical, where AI will do whatever you tell it to do. And so, um, and it can do almost anything, you know, building software. And so making a good judgment call on how to do something, again, becomes a competitive advantage versus just being able to accomplish it. 00:04:16,740 --> 00:04:16,959 [John] Yep. 00:04:18,060 --> 00:04:19,200 [John] So thinking back to school, 00:04:20,840 --> 00:04:30,360 [John] did you ever have any classes where you were just in a really unique class where there were just a t- a disproportionate number of, like, super smart people, like, in a particular class? 00:04:30,360 --> 00:04:30,840 [Eric] Hmm. 00:04:35,680 --> 00:04:48,340 [Eric] Yes. Um, statistics. So I really enjoyed statistics in college, and so I took a lot of statistics courses, which is probably why, you know, I ended up- 00:04:48,340 --> 00:04:48,740 [John] Yeah 00:04:48,740 --> 00:04:49,659 [Eric] ... you know, in part of my career- 00:04:49,660 --> 00:04:49,930 [John] Right. Yeah 00:04:49,930 --> 00:04:51,840 [Eric] ... you know, really getting into data. 00:04:53,260 --> 00:04:57,740 [Eric] But the l- the more advanced statistics courses, people were s- people were very smart- 00:04:57,740 --> 00:04:57,770 [John] Yeah 00:04:57,770 --> 00:04:58,320 [Eric] ... 'cause it was... 00:04:59,460 --> 00:05:02,760 [Eric] You know, and then maybe some economics too, I think, were the two classes- 00:05:02,760 --> 00:05:02,830 [John] Yeah 00:05:03,880 --> 00:05:06,720 [Eric] ... where I just thought, "Man, there are a ton of super smart people in here." 00:05:06,720 --> 00:05:08,320 [John] Yeah. So- 00:05:08,320 --> 00:05:08,620 [Eric] How about you? 00:05:08,620 --> 00:05:11,820 [John] My... Yeah, so chemistry is what sticks out for me. 00:05:11,820 --> 00:05:12,080 [Eric] Hmm. 00:05:12,080 --> 00:05:16,620 [John] I just had a really smart group that was in my, um, a couple of my chemistry classes. 00:05:16,620 --> 00:05:17,540 [Eric] Mm-hmm. 00:05:17,540 --> 00:05:23,700 [John] Um, which I took with my... It was pre-med and, and, uh, liberal arts college, so I would've taken probably some anyways. 00:05:23,700 --> 00:05:23,820 [Eric] Mm-hmm. 00:05:23,820 --> 00:05:33,320 [John] But chemistry is the one that sticks out to me. But I really think of, for the average company, AI is all of a sudden being in that class and raising the average grades. 00:05:33,320 --> 00:05:34,140 [Eric] Hmm. 00:05:34,140 --> 00:05:41,700 [John] So for the average company, I d- I do think the average grade point for work output goes up, at least some, 00:05:43,080 --> 00:05:52,500 [John] and then that happens kind of across the board. So the average becomes here. Let's just take writing quality. There's, there's not that many people in the world that are very good at writing. It's- 00:05:52,500 --> 00:05:52,900 [Eric] Agreed. 00:05:52,900 --> 00:05:54,219 [John] [laughs] Agreed. 00:05:54,220 --> 00:05:54,640 [Eric] Hard agree. 00:05:54,640 --> 00:05:58,900 [John] But, but people with AI definitely bumps the average grade. 00:05:58,900 --> 00:05:59,350 [Eric] Yes. 00:05:59,350 --> 00:06:05,320 [John] Let, let's, let, let's say we- we're moving from, like, a, a C to a B- or something. Maybe a D to a C. I don't know. 00:06:05,320 --> 00:06:05,740 [Eric] Yeah. 00:06:05,740 --> 00:06:06,040 [John] Um- 00:06:06,040 --> 00:06:07,120 [Eric] That's a whole other episode. 00:06:07,120 --> 00:06:07,140 [John] Yeah. [laughs] Right. 00:06:07,140 --> 00:06:09,984 [Eric] But we can discuss that. 'Cause that is my job. [laughs] 00:06:09,984 --> 00:06:10,744 [John] But yeah, yeah. 00:06:11,764 --> 00:06:12,884 [John] Yeah, so it's pretty bad. 00:06:12,884 --> 00:06:13,044 [Eric] Yeah. 00:06:13,044 --> 00:06:15,424 [John] Um, even people that are really good at other things. 00:06:15,424 --> 00:06:19,204 [Eric] But generally, generally, yes, right? 00:06:19,204 --> 00:06:19,364 [John] Right. 00:06:19,364 --> 00:06:22,724 [Eric] Uh, you know, the average person, 00:06:23,844 --> 00:06:24,164 [Eric] um, 00:06:25,464 --> 00:06:31,803 [Eric] if they are decent at prompting, it can really help them, you know, write more clearly. 00:06:31,804 --> 00:06:32,804 [John] Yeah, ex- exactly. 00:06:32,804 --> 00:06:33,984 [Eric] Especially for things like- 00:06:33,984 --> 00:06:34,264 [John] Yeah 00:06:34,264 --> 00:06:43,444 [Eric] ... email communications or other things like that, right? Board meeting agendas. Uh, you know, I mean, there are just so many things that it is genuinely helpful for. 00:06:43,444 --> 00:06:43,664 [John] Right. 00:06:44,724 --> 00:06:52,084 [John] So I think because you're in this new grade with this, like, at least kind of cluster of smart people, like bringing the average up- 00:06:52,084 --> 00:06:52,784 [Eric] Mm-hmm 00:06:52,784 --> 00:07:00,124 [John] ... the... And I'm applying this to the taste and judgment piece because now it's like, okay, well, we're all here- 00:07:00,124 --> 00:07:00,244 [Eric] Mm-hmm 00:07:00,244 --> 00:07:03,344 [John] ... which is a little higher than we used to be, and maybe a lot higher for some people. 00:07:03,344 --> 00:07:04,324 [Eric] Yep. 00:07:04,324 --> 00:07:17,324 [John] The taste and judgment is, has been like pushed as a differentiator. But I think, which is still true, but I think, and we're kind of about to get into this, accountability is another factor here that- 00:07:17,324 --> 00:07:17,524 [Eric] Huge 00:07:17,524 --> 00:07:35,764 [John] ... that's like under, that is undersold right now. Um, and I guess you could say the person or persons accountable for the output, and taste and judgment are two pieces. Judgment for sure a piece of the output. And, and so it... Like, I guess maybe taste is a piece of the input and judgment of the output. 00:07:35,764 --> 00:07:39,484 [Eric] Yep. Yeah, yeah. So we're in a world where 00:07:41,744 --> 00:07:47,184 [Eric] on average the floor has been raised. I think the ceiling has been raised significantly- 00:07:47,184 --> 00:07:47,684 [John] Yeah 00:07:47,684 --> 00:07:51,804 [Eric] ... way more than the floor has been raised, but that's another, that's another episode. 00:07:53,184 --> 00:08:03,144 [Eric] So the floor is raised for what the average employee can do as far as their work product. You know, let's say the quality, the quantity, right? Just in general as an aggregate, you know, as an aggregate- 00:08:03,144 --> 00:08:03,214 [John] Yeah 00:08:03,214 --> 00:08:28,284 [Eric] ... measure of their work output. But this introduces really, really hard questions about accountability because the reason that the floor is raised is that these people are using a tool that is non-deterministic and generative. And so I'm gonna define those two things, um, because they're important terms to be, to be accurate about, um, to be precise about. 00:08:29,784 --> 00:08:32,344 [Eric] So non-deterministic means that, um, 00:08:33,584 --> 00:08:46,304 [Eric] it is not a math equation where you put numbers in and you get numbers out, and because of the rules of mathematics, you reliably get the same output every single time, right? 00:08:46,304 --> 00:08:46,884 [John] Yep. 00:08:46,884 --> 00:09:08,084 [Eric] It is non-deterministic in that new information is, is being generated, right? And so... And there are factors that influence this. So because you have conversations with Claude, you know, all day about data stuff, and I have conversations with Claude all day about whatever I'm writing about at Vercel or whatever I'm building, 00:09:09,624 --> 00:09:13,954 [Eric] even if we ask Claude the same question, we will get different answers. 00:09:13,954 --> 00:09:13,984 [John] Sure. 00:09:13,984 --> 00:09:15,164 [Eric] That is non-deterministic- 00:09:15,164 --> 00:09:15,174 [John] Right 00:09:15,174 --> 00:09:27,644 [Eric] ... because the context of your previous conversations and the work that you've done, your system prompts everything influence the, influence the answer that you're gonna get, right? And so two people can ask the same question and get different answers. That's part of the design of the system, actually. 00:09:27,644 --> 00:09:27,834 [John] Right. 00:09:27,834 --> 00:09:52,473 [Eric] Right? So that's not necessarily a bug, but it is something to be very cognizant of. Um, and it's generative, meaning that if you and I both, you know, ask, uh, Claude to build a strategy, you know, or let's say a plan, you know, a quarterly plan, um, it will generate new information based on information that you've given it. 00:09:52,473 --> 00:09:52,484 [John] Yeah. 00:09:52,484 --> 00:09:56,624 [Eric] But it will, it will literally generate, you know, new text. You're not writing it. 00:09:56,624 --> 00:09:57,044 [John] Right. 00:09:57,044 --> 00:10:20,494 [Eric] Um, but that becomes part of your work product as an individual. So there are many things to discuss here specifically with AI, but I wanna briefly start with first principles. So let's talk about first principles, let's talk about specifics with AI, and then let's get practical and talk about, you know, how you and I manage teams and people with accountability. 00:10:20,494 --> 00:10:20,544 [John] Right. 00:10:20,544 --> 00:10:30,824 [Eric] So first principles with accountability, and I'd love the perspective of a manager. Like you as a manager, how do you think about accountability done really well? 00:10:30,824 --> 00:10:31,004 [John] Yeah. 00:10:32,744 --> 00:10:43,434 [John] I mean, I think you have to go back to, um, back to the beginning, and that's Peter Drucker is considered the father of modern management. 00:10:43,434 --> 00:10:43,464 [Eric] Mm-hmm. 00:10:43,464 --> 00:10:47,124 [John] And his methodology was management by objectives, MBOs. 00:10:47,124 --> 00:10:47,484 [Eric] Mm-hmm. 00:10:47,484 --> 00:10:51,744 [John] Which turned into and highly influenced OKRs- 00:10:51,744 --> 00:10:51,794 [Eric] Yep 00:10:51,794 --> 00:10:57,444 [John] ... which is kind of the tech version of that. And then there's more downstream stuff from that, but we can just keep it there, I think. 00:10:57,444 --> 00:10:58,004 [Eric] Yep. 00:10:58,004 --> 00:11:03,164 [John] And one of the big innovations, which hardly even feels like an innovation anymore, 00:11:04,524 --> 00:11:14,014 [John] is, is essentially multiple levels of management and it not just being like a bunch of people doing task- 00:11:14,014 --> 00:11:14,014 [Eric] Mm 00:11:14,014 --> 00:11:17,944 [John] ... with like one person in charge, um, telling people what to do. 00:11:17,944 --> 00:11:18,144 [Eric] Mm-hmm. 00:11:18,144 --> 00:11:34,284 [John] Like that... 'Cause that is still kind of the default. Like if you were to work, like tomorrow's, um, well like say this weekend you were working, you know, with the kids in the yard, you're like, like, you know, like you do this and you do this. Like that's ki- that is kind of a human default of like- 00:11:34,284 --> 00:11:34,594 [Eric] Here's the- 00:11:34,594 --> 00:11:34,844 [John] ... how to- 00:11:34,844 --> 00:11:35,064 [Eric] Yeah 00:11:35,064 --> 00:11:35,804 [John] ... how to do it. 00:11:35,804 --> 00:11:36,844 [Eric] We open the shed. 00:11:36,844 --> 00:11:37,344 [John] Yeah. 00:11:37,344 --> 00:11:38,404 [Eric] There's a bunch of yard tools. 00:11:38,404 --> 00:11:38,424 [John] There's a s- 00:11:38,424 --> 00:11:39,754 [Eric] Like let's get to work, right? 00:11:39,754 --> 00:11:39,903 [John] Right, right. 00:11:39,904 --> 00:11:41,584 [Eric] And so what is the result of that going to be? 00:11:41,584 --> 00:11:59,484 [John] Right, right. In the management by objective, the key piece there is the objective piece, is you now have goals, objectives, whatever you wanna call it, that a, um, group of people is responsible for, and then maybe there's another group even above that that's responsible for a bigger objective that flows into the smaller objective. 00:11:59,484 --> 00:11:59,804 [Eric] Mm-hmm. 00:11:59,804 --> 00:12:05,144 [John] And you have these layers of objectives that are not task level. They're, um, 00:12:06,324 --> 00:12:09,044 [John] like area of responsibility level. Like- 00:12:09,044 --> 00:12:09,284 [Eric] Yep 00:12:09,284 --> 00:12:11,676 [John] ... uh, an objective is like You know, 00:12:12,896 --> 00:12:15,536 [John] grow our sales from X number to Y. 00:12:15,536 --> 00:12:15,696 [Eric] Yep. 00:12:15,696 --> 00:12:19,895 [John] Not like, "I want you to, like, make 10 calls today." 00:12:19,896 --> 00:12:19,996 [Eric] Yep. 00:12:19,996 --> 00:12:28,656 [John] So that, and that, and then layering, and then connecting all that together in an organization is, is, you know, is a big component of management, for sure. 00:12:28,656 --> 00:12:33,256 [Eric] And so accountability in that context would be, 00:12:35,196 --> 00:12:40,646 [Eric] I think, clarity that every manager in the hierarchy has around 00:12:41,776 --> 00:12:43,166 [Eric] what the outcome needs to be. 00:12:43,166 --> 00:12:43,456 [John] Mm-hmm. 00:12:43,456 --> 00:12:49,076 [Eric] And so they hold their teams accountable. So they define the inputs. 00:12:49,076 --> 00:12:49,696 [John] Right. 00:12:49,696 --> 00:12:53,275 [Eric] Um, okay, we need to increase our sales by some number. 00:12:53,276 --> 00:12:53,646 [John] Right. 00:12:53,646 --> 00:13:00,566 [Eric] Right? And so you are the head of the sales team, or you're the marketing leader who needs to create pipeline or generate leads for the- 00:13:00,566 --> 00:13:00,566 [John] Right 00:13:00,566 --> 00:13:07,166 [Eric] ... sales team, right? And so you define the inputs. Okay, we need to... In order to make that sales number happen- 00:13:07,166 --> 00:13:07,166 [John] Mm-hmm 00:13:07,166 --> 00:13:09,646 [Eric] ... the marketing leader needs to generate this many leads, right? 00:13:09,646 --> 00:13:09,716 [John] Right. 00:13:09,716 --> 00:13:14,056 [Eric] Or the sales leader needs to actually add two headcount, you know, to increase- 00:13:14,056 --> 00:13:14,436 [John] Right 00:13:14,436 --> 00:13:23,096 [Eric] ... carrying capacity. And so everyone has clarity on, let's define the inputs and drive towards those and measure against the, the sales increase. 00:13:23,096 --> 00:13:32,656 [John] Yeah. Yeah. Yeah, I think that's a component of it, but depending on the level, like, some of the genius of it is the accountability is, is for the objective, not for the, um, inputs into the objective. 00:13:32,656 --> 00:13:33,816 [Eric] Yes, correct. 00:13:33,816 --> 00:13:33,876 [John] So- 00:13:33,876 --> 00:13:44,996 [Eric] But the, all of the inputs that are created, um, there's natural accountability baked in to those actually contributing towards the objective, right? 00:13:44,996 --> 00:13:45,536 [John] Yeah. Right. 00:13:45,536 --> 00:13:59,036 [Eric] And then on a practical day-to-day level, a manager is holding their individual contributors, let's say, you know, the, the people on their team, accountable to the inputs- 00:13:59,036 --> 00:14:00,516 [John] They believe accomplish- 00:14:00,516 --> 00:14:01,476 [Eric] They believe- 00:14:01,476 --> 00:14:01,486 [John] ... will move the needle 00:14:01,486 --> 00:14:02,746 [Eric] ... accomplish the- 00:14:02,746 --> 00:14:02,746 [John] Yeah 00:14:02,746 --> 00:14:03,516 [Eric] ... the objective, right? 00:14:03,516 --> 00:14:05,076 [John] Yeah. For sure. 00:14:05,076 --> 00:14:19,536 [Eric] How about on a very individual day-to-day level as a manager who is holding people accountable to the inputs that you, you know, that they believe or that have been agreed upon will accomplish the objective? 00:14:21,496 --> 00:14:22,516 [John] Yeah. I mean, I think... 00:14:24,616 --> 00:14:27,076 [John] I mean, I think a couple things I've learned on this is, one, 00:14:28,536 --> 00:14:33,416 [John] you have, you have to repeat the objective a lot. [laughs] 00:14:33,416 --> 00:14:33,996 [Eric] Every day. 00:14:33,996 --> 00:14:35,296 [John] Like, yeah, every day. 00:14:36,316 --> 00:14:55,386 [John] And um, and I think that's even more important than repeating the inputs because the inputs are important too, but generally if I'm a marketer or I'm a data analyst, like, since that's the career path somebody's chosen, they're, like, relatively familiar with the inputs. 00:14:55,386 --> 00:14:55,456 [Eric] Right. 00:14:55,456 --> 00:15:03,906 [John] I'm a marketer, like, I, I need to send emails. Like, say that's part of your job, and people get that. Like, they, they feel like they're being productive when they're sending emails, and when they're not, they don't. 00:15:03,906 --> 00:15:04,036 [Eric] Yeah. 00:15:04,036 --> 00:15:05,496 [John] Or pick a, pick a thing. 00:15:05,496 --> 00:15:09,796 [Eric] Or, or the inputs need to change because whatever current set of inputs- 00:15:09,796 --> 00:15:09,956 [John] Right 00:15:09,956 --> 00:15:12,146 [Eric] ... you know, that you had as your initial hypothesis- 00:15:12,146 --> 00:15:12,245 [John] Right 00:15:12,245 --> 00:15:15,056 [Eric] ... that would achieve the outcome aren't working, right? 00:15:15,056 --> 00:15:15,156 [John] Right. 00:15:15,156 --> 00:15:16,556 [Eric] And so we need to change the inputs. 00:15:16,556 --> 00:15:19,236 [John] Well, but, but most people come pre-baked with inputs. 00:15:19,236 --> 00:15:19,516 [Eric] Yes. 00:15:19,516 --> 00:15:21,305 [John] You hire them, they come pre-baked, and they have, like, 00:15:22,796 --> 00:15:24,316 [John] a playbook of, like, four or five inputs- 00:15:24,316 --> 00:15:24,346 [Eric] Mm-hmm 00:15:24,346 --> 00:15:29,585 [John] ... of what they know how to do, and that's what they know to do, and they can do more of less of each and kind of mix and match. 00:15:29,585 --> 00:15:29,596 [Eric] Mm-hmm. 00:15:29,596 --> 00:15:31,476 [John] But they, they just come with a baked thing- 00:15:31,476 --> 00:15:31,536 [Eric] Yep 00:15:31,536 --> 00:15:32,376 [John] ... of inputs. 00:15:32,376 --> 00:15:32,726 [Eric] Most of the time. 00:15:32,726 --> 00:15:36,616 [John] But they do not come baked with... Well, yeah, most of the time. But they don't come baked with objectives typically. 00:15:36,616 --> 00:15:37,476 [Eric] Yes. 00:15:37,476 --> 00:15:37,896 [John] Um- 00:15:37,896 --> 00:15:37,976 [Eric] Yeah 00:15:37,976 --> 00:15:43,896 [John] ... A, because those can change from company to company, which is fair, and then B, because I don't know, it's kind of not their job- 00:15:43,896 --> 00:15:44,236 [Eric] Mm-hmm 00:15:44,236 --> 00:15:44,236 [John] ... 00:15:45,506 --> 00:15:46,816 [John] to, to come up with objectives. 00:15:46,816 --> 00:15:46,946 [Eric] Yeah. 00:15:48,176 --> 00:15:52,176 [Eric] Two, two or three additional things that, that come to mind for me. 00:15:53,636 --> 00:16:00,516 [Eric] One is being really consistent and really early on holding people accountable. 00:16:00,516 --> 00:16:01,696 [John] Mm-hmm. 00:16:01,696 --> 00:16:18,836 [Eric] And I mean, at a basic level, it's actually surprising how common it is for managers to avoid just asking people to do their entire job, [laughs] right? Fulfill all of your job responsibilities. 00:16:18,836 --> 00:16:22,316 [John] You mean, like, they get happy with, like, "Ah, I mean, they do about three-fourths, and it's fine." 00:16:22,316 --> 00:16:22,936 [Eric] Yeah. 00:16:22,936 --> 00:16:23,076 [John] Yeah. 00:16:23,076 --> 00:16:24,396 [Eric] Yeah, exactly. 00:16:24,396 --> 00:16:25,056 [John] Or half. 00:16:25,056 --> 00:16:25,686 [Eric] Yeah. 00:16:25,686 --> 00:16:25,686 [John] Sometimes. 00:16:25,686 --> 00:16:25,686 [Eric] Or half. 00:16:25,686 --> 00:16:25,796 [John] Yeah. 00:16:25,796 --> 00:16:26,216 [Eric] Um, 00:16:28,016 --> 00:16:30,036 [Eric] you know, that's, so that's one thing, actually. 00:16:30,036 --> 00:16:30,696 [John] Right. 00:16:30,696 --> 00:16:34,566 [Eric] The other that comes to mind is just quality. 00:16:34,566 --> 00:16:34,686 [John] Yeah. 00:16:34,686 --> 00:16:36,326 [Eric] And I think the reason, 00:16:37,656 --> 00:16:52,936 [Eric] probably for the first one, you know, just, just getting people to do their entire job, but then also the quality aspect is it's very time-consuming to be proactive on reviewing all of the work product. 00:16:52,936 --> 00:16:53,355 [John] Mm-hmm. 00:16:53,416 --> 00:17:03,196 [Eric] You know, trying to understand what someone's doing and managing that really well so that people operate in an environment where there's a super high standard- 00:17:03,196 --> 00:17:03,356 [John] Right 00:17:03,356 --> 00:17:05,966 [Eric] ... right, for how much work you get done and the quality of that work. 00:17:06,976 --> 00:17:13,376 [Eric] And I'm not talking about micromanaging. I'm just talking about, you know, understanding- 00:17:13,376 --> 00:17:13,696 [John] Right 00:17:13,696 --> 00:17:17,806 [Eric] ... the throughput of your team on a detailed enough level 00:17:19,636 --> 00:17:21,396 [Eric] th- that ultimately the ideal outcome- 00:17:21,396 --> 00:17:21,406 [John] Right 00:17:21,406 --> 00:17:23,446 [Eric] ... is people grow. They get really better. 00:17:23,446 --> 00:17:23,475 [John] Right. 00:17:23,476 --> 00:17:27,225 [Eric] You know, they really better. Well, they get really good at their craft. 00:17:27,225 --> 00:17:27,236 [John] Right. 00:17:27,236 --> 00:17:32,956 [Eric] They become a better, um... You know, they become a better professional- 00:17:32,956 --> 00:17:32,966 [John] Right 00:17:32,966 --> 00:17:46,016 [Eric] ... as a result of that, and that's pretty rare. Um, I think the, the last thing that comes to mind is if you aren't proactive about that early, you get more of what you subsidize. 00:17:46,016 --> 00:17:46,176 [John] Right. 00:17:46,176 --> 00:17:52,756 [Eric] In that, you know, the worst version of this is you get to your quarterly review or, or worse- 00:17:52,756 --> 00:17:52,766 [John] Annual 00:17:52,766 --> 00:17:54,346 [Eric] ... your six-month review- 00:17:54,346 --> 00:17:54,346 [John] Yeah 00:17:54,346 --> 00:17:56,006 [Eric] ... or annual review- 00:17:56,006 --> 00:17:56,016 [John] Right 00:17:56,016 --> 00:17:58,886 [Eric] ... and there's a bunch of stuff that comes up that needs to change, right? 00:17:58,886 --> 00:17:58,896 [John] Right. 00:17:58,896 --> 00:18:04,626 [Eric] And it's like, no, uh, you, you need to address that essentially in real time as it comes up- 00:18:04,626 --> 00:18:04,626 [John] Right 00:18:04,626 --> 00:18:07,636 [Eric] ... in my experience. Uh, I mean, there's tact around that, but- 00:18:07,636 --> 00:18:14,296 [John] Sure. Yeah. Yeah, I think, I think one thing that I would add would be- 00:18:14,296 --> 00:18:29,846 [John] Having, having a team b- because that... You're absolutely right on the quality thing, but, but the way to sustain it is to have really high team standards and have everybody want to be accountable to the standard- 00:18:29,846 --> 00:18:29,846 [Eric] Yeah 00:18:29,846 --> 00:18:33,736 [John] ... and to have a lot of, um, peer review as a component of that standard. 00:18:33,736 --> 00:18:34,376 [Eric] Hmm. Mm-hmm. 00:18:34,376 --> 00:18:36,996 [John] 'Cause then you as the manager, you're not, like, a complete bottleneck for everything. 00:18:36,996 --> 00:18:37,976 [Eric] Yes. Totally. 00:18:37,976 --> 00:18:38,496 [John] I think that's the only way. 00:18:38,496 --> 00:18:40,035 [Eric] It's a culture that you build on your team. 00:18:40,036 --> 00:18:40,316 [John] Right. 00:18:40,316 --> 00:18:40,596 [Eric] For sure. 00:18:40,596 --> 00:18:41,316 [John] Right. 00:18:41,316 --> 00:18:43,776 [Eric] But to build that is an immense amount of work. 00:18:43,776 --> 00:18:44,196 [John] Yeah. 00:18:44,196 --> 00:18:45,456 [Eric] It's an immense amount- 00:18:45,456 --> 00:18:45,466 [John] Right 00:18:45,466 --> 00:18:45,796 [Eric] ... of work. 00:18:45,796 --> 00:18:46,176 [John] Right. 00:18:46,176 --> 00:18:46,496 [Eric] Um, 00:18:47,696 --> 00:18:49,596 [Eric] okay, switching gears, 00:18:51,416 --> 00:18:54,476 [Eric] accountability in the age of AI, and so let me give you, 00:18:56,096 --> 00:19:00,236 [Eric] let me give you a really specific example that I want you to field as a manager. 00:19:01,736 --> 00:19:02,236 [Eric] [lips smack] Um, 00:19:03,616 --> 00:19:07,476 [Eric] there... You are working hard to create a culture 00:19:08,716 --> 00:19:09,096 [Eric] of 00:19:10,136 --> 00:19:16,756 [Eric] peer-to-peer accountability, top-down accountability for you as a manager, where you're setting the tone for, you know, quality and throughput. 00:19:17,836 --> 00:19:21,676 [Eric] Um, and AI enters the picture, 00:19:22,996 --> 00:19:25,756 [Eric] and the results are not evenly distributed, 00:19:27,576 --> 00:19:31,256 [Eric] and this, I think, is a very real, real challenge. So- 00:19:31,256 --> 00:19:31,956 [John] Mm-hmm 00:19:31,956 --> 00:19:37,776 [Eric] ... I'll, I'll be very specific here with examples. So let's say that, um, 00:19:38,836 --> 00:19:45,636 [Eric] you know, someone on the team who maybe was, you know, sort of let's say a mid... You know, good, good employee. 00:19:45,636 --> 00:19:45,646 [John] Mm-hmm. 00:19:45,646 --> 00:19:45,816 [Eric] You know- 00:19:45,816 --> 00:19:45,876 [John] Mm-hmm 00:19:45,876 --> 00:19:47,416 [Eric] ... let's, let's talk about agreeable data. 00:19:47,416 --> 00:19:47,936 [John] Mm-hmm. 00:19:47,936 --> 00:19:52,816 [Eric] A good employee, like good analyst, you know, but not blowing you away. 00:19:52,816 --> 00:19:53,496 [John] Mm-hmm. 00:19:53,496 --> 00:19:59,036 [Eric] And they begin to use AI, and it's like, whoa, their work product is, is awesome. 00:19:59,036 --> 00:19:59,316 [John] Mm-hmm. 00:19:59,316 --> 00:19:59,476 [Eric] Right? 00:20:00,496 --> 00:20:00,976 [Eric] Um, 00:20:02,256 --> 00:20:11,596 [Eric] another situation where s- let's say someone who's really good starts to use AI, and maybe you start to see more mistakes in their work. 00:20:11,596 --> 00:20:12,096 [John] Hmm. 00:20:12,096 --> 00:20:12,296 [Eric] Right? 00:20:12,296 --> 00:20:12,816 [John] Mm-hmm. 00:20:12,816 --> 00:20:18,976 [Eric] So two scenarios. There are infinite, you know, number of scenarios. Address each one of those. How are you gonna deal with that? Um, 00:20:20,296 --> 00:20:24,216 [Eric] you know, because in the first scenario, it's obviously 00:20:25,436 --> 00:20:27,356 [Eric] a net benefit- 00:20:27,356 --> 00:20:27,626 [John] Right 00:20:27,626 --> 00:20:28,336 [Eric] ... right, I think. 00:20:28,336 --> 00:20:28,996 [John] Right. 00:20:28,996 --> 00:20:29,456 [Eric] Um, 00:20:31,396 --> 00:20:35,096 [Eric] how do you think about that, and then how do you think about the second person where there's more mistakes? 00:20:35,096 --> 00:20:40,276 [John] Yeah. I think for... I think it's actually this... I mean, there's nuances between the two, obviously. 00:20:40,276 --> 00:20:40,406 [Eric] Mm-hmm. 00:20:40,406 --> 00:20:44,756 [John] But the answer is the same, is I think AI rewards right thinking. 00:20:44,756 --> 00:20:45,206 [Eric] Hmm. 00:20:46,236 --> 00:20:49,176 [John] So we'll start with the second example 00:20:50,236 --> 00:20:52,756 [John] of, like, further mistakes, you know, et cetera. 00:20:52,756 --> 00:20:53,376 [Eric] Mm-hmm. 00:20:53,376 --> 00:21:04,105 [John] So there's probably some workflow tweaks, some, "Hey..." Actually, I talk about this a lot. Talk about a pie chart of, like, how you spend tokens and, and tell people- 00:21:04,105 --> 00:21:04,105 [Eric] Hmm 00:21:04,105 --> 00:21:12,296 [John] ... um, how do you spend tokens and, and then the, the default... I don't even think people verbalize this, but like, "Well, we spend tokens building, like doing stuff." 00:21:12,296 --> 00:21:12,816 [Eric] Mm-hmm. 00:21:12,816 --> 00:21:16,415 [John] It's like, well, you can spend tokens researching, you can spend tokens planning. 00:21:16,416 --> 00:21:16,736 [Eric] Mm-hmm. 00:21:16,736 --> 00:21:20,636 [John] You can spend tokens prototyping. You can spend tokens coding. You can spend- 00:21:20,636 --> 00:21:20,816 [Eric] Hmm 00:21:20,816 --> 00:21:24,256 [John] ... tokens doing QA. And people don't have enough buckets. 00:21:25,536 --> 00:21:30,616 [John] So I typically... The... I mean, the first person, maybe they're j- maybe they're doing great. Maybe there's not a lot there. 00:21:30,616 --> 00:21:30,626 [Eric] Mm-hmm. 00:21:30,626 --> 00:21:34,276 [John] But on the other one, I'd say your, like, your allocations are off. 00:21:34,276 --> 00:21:34,476 [Eric] Hmm. 00:21:34,476 --> 00:21:36,636 [John] Like, if there's a quality work product- 00:21:36,636 --> 00:21:36,716 [Eric] Yeah 00:21:36,716 --> 00:21:48,676 [John] ... quality problem, and it, and it... Sure, it could be like a QA thing and you need to spend more time here. Or it could be you need to spend more time in research and thinking because, 'cause there's, like, fundamental- 00:21:48,676 --> 00:21:48,896 [Eric] Yeah 00:21:48,896 --> 00:21:51,836 [John] ... things that get you off track 'cause you didn't start in the right place. 00:21:51,836 --> 00:21:52,496 [Eric] Yep. 00:21:53,816 --> 00:22:02,996 [Eric] How do you think about presenting work to a client of yours that you know has been, 00:22:04,416 --> 00:22:08,776 [Eric] um, generate, you know, a large amount of the work has been generated by AI, 00:22:09,936 --> 00:22:11,556 [Eric] you know, for one of your employees? 00:22:11,556 --> 00:22:11,606 [John] Mm-hmm. 00:22:11,606 --> 00:22:15,996 [Eric] Right? They're going to... Let's say they're building a data app or they're building a report or- 00:22:15,996 --> 00:22:16,296 [John] Right 00:22:16,296 --> 00:22:17,696 [Eric] ... you know, um, 00:22:18,776 --> 00:22:23,716 [Eric] or you implement a data agent, data analyst agent for one of your clients. 00:22:23,716 --> 00:22:23,806 [John] Right. 00:22:23,806 --> 00:22:23,956 [Eric] Right? 00:22:25,456 --> 00:22:29,336 [Eric] How does that work? How does the accountability work there, you know? 00:22:29,336 --> 00:22:29,416 [John] Yeah. 00:22:29,416 --> 00:22:42,016 [Eric] Or how do you... Let's say you have the person who is doing really, really well with AI, right? And then there's sort of a major problem. They ship something to a client- 00:22:42,016 --> 00:22:42,025 [John] Right 00:22:42,025 --> 00:22:43,796 [Eric] ... and there's a big problem with it. 00:22:43,796 --> 00:22:43,886 [John] Right. 00:22:43,886 --> 00:22:43,936 [Eric] Right? 00:22:45,056 --> 00:22:45,956 [Eric] How do you deal with that? 00:22:45,956 --> 00:22:52,076 [John] Yeah. So my... So super top of mind is that the data analyst agent, like managed agents- 00:22:52,076 --> 00:22:52,266 [Eric] Mm-hmm 00:22:52,266 --> 00:23:01,696 [John] ... things we're working on, and we're working on right now, one right now, and I'm super excited 'cause we started with, what do we want this to be able to do? We started with objectives. 00:23:01,696 --> 00:23:02,056 [Eric] Hmm. 00:23:02,056 --> 00:23:07,456 [John] So the agent comes with objectives of, like, what it should be able to accomplish with, like, a fair amount of granularity. 00:23:07,456 --> 00:23:08,356 [Eric] Hmm. 00:23:08,356 --> 00:23:14,476 [John] And then, and then we will work backwards and build evals for, like, what it should be able to accomplish. 00:23:14,476 --> 00:23:15,726 [Eric] Hmm. 00:23:15,726 --> 00:23:17,436 [John] And then iterate, like integrate systems- 00:23:17,436 --> 00:23:17,536 [Eric] Mm-hmm 00:23:17,536 --> 00:23:18,155 [John] ... connect things. 00:23:18,156 --> 00:23:18,856 [Eric] Mm-hmm. 00:23:18,856 --> 00:23:22,436 [John] And then every time that agent has changed, the evals run and they pass or fail- 00:23:22,436 --> 00:23:22,466 [Eric] Mm-hmm 00:23:22,466 --> 00:23:24,266 [John] ... based on what we said we wanted it to do. 00:23:24,266 --> 00:23:24,626 [Eric] Mm-hmm. 00:23:24,626 --> 00:23:26,436 [John] So I'm super excited about that. 00:23:26,436 --> 00:23:29,776 [Eric] And evals, just define evals very, really quickly- 00:23:29,776 --> 00:23:29,786 [John] Yeah 00:23:29,786 --> 00:23:32,306 [Eric] ... 'cause I think it's a super important concept. 00:23:32,306 --> 00:23:43,756 [John] Mm-hmm. Yeah, so, so it's, so, so we've got the most simple version of this, so I've got five question, uh, five to 10 questions I want this agent to be able to accurately answer every time. 00:23:43,756 --> 00:23:45,796 [Eric] So how many daily active users do I have? 00:23:45,796 --> 00:23:46,756 [John] Yeah. 00:23:46,756 --> 00:23:49,276 [Eric] You know, what was my margin- 00:23:49,276 --> 00:23:49,576 [John] Right 00:23:49,576 --> 00:23:53,296 [Eric] ... you know, from sales in this product category last week? 00:23:53,296 --> 00:23:56,616 [John] Right. Yep. So start with 10 and, and maybe over time it gets to be a lot more. 00:23:56,616 --> 00:23:57,456 [Eric] Mm-hmm. 00:23:57,456 --> 00:24:01,096 [John] And then you have it run through the 10. 00:24:01,096 --> 00:24:01,436 [Eric] Mm-hmm. 00:24:01,436 --> 00:24:04,275 [John] So you, like, hold the answer key over here where it can't see it. It's out of context. 00:24:04,276 --> 00:24:05,095 [Eric] Mm-hmm. 00:24:05,096 --> 00:24:12,016 [John] And then you run through the 10 and see how it does, and then compare it and, you know, did it get 10 out of 10, 9 out of 10, whatever. 00:24:12,016 --> 00:24:12,536 [Eric] Mm-hmm. 00:24:12,536 --> 00:24:24,572 [John] And then the idea is, you know, sh- you ship it the first time, 10 out of 10, like good. And anytime you change it to, like, quote, "make it better," we have to prove that we made it better 'cause it has to achieve that same- 00:24:24,572 --> 00:24:26,712 [Eric] Oh, right. It has to. Yep, yep, yep. 00:24:26,712 --> 00:24:31,512 [John] Yeah. And then the really tricky part right now is, um, agent 00:24:32,572 --> 00:24:39,162 [John] memory is becoming a thing. Like, people may have noticed this, like ChatGPT has this thing where it can kinda remember previous conversations- 00:24:39,162 --> 00:24:39,242 [Eric] Mm-hmm 00:24:39,242 --> 00:24:44,412 [John] ... across threads. Claude kinda has it. Um, and then a lot of these agents are gonna have it as a component. 00:24:44,412 --> 00:24:44,732 [Eric] Mm-hmm. 00:24:44,732 --> 00:24:56,302 [John] And that's actually the trickiest part, because the agent is changing in real time, and these evals, even though we didn't make any, like, core changes, the memory changed some, and it may affect the evals. 00:24:56,302 --> 00:24:56,312 [Eric] Hmm. 00:24:56,312 --> 00:24:58,392 [John] So I think that's an interesting, like- 00:24:58,392 --> 00:24:58,582 [Eric] So- 00:24:58,582 --> 00:24:59,472 [John] ... space right now 00:24:59,472 --> 00:25:04,732 [Eric] ... you, what you're saying is that you actually are implementing accountability 00:25:06,172 --> 00:25:07,512 [Eric] in the agent 00:25:08,612 --> 00:25:10,212 [Eric] workloads themselves. 00:25:10,212 --> 00:25:10,652 [John] Right. 00:25:10,652 --> 00:25:14,872 [Eric] So the agent workloads run, and they have accountability built into the system. 00:25:14,872 --> 00:25:15,212 [John] Right. 00:25:15,212 --> 00:25:25,952 [Eric] So, which is super cool. But then y- the agent's also generating work, right? Now we have multiple layers, right? It's not just the analyst- 00:25:25,952 --> 00:25:26,312 [John] Right 00:25:26,312 --> 00:25:28,122 [Eric] ... who's going in and writing SQL or Python and- 00:25:28,122 --> 00:25:28,122 [John] Right 00:25:28,122 --> 00:25:37,692 [Eric] ... performing an analysis, right? They are directing an agent who's running loops. There's accountability through evals, um, that benchmark the agent's work. 00:25:37,692 --> 00:25:38,032 [John] Right. 00:25:38,032 --> 00:25:43,222 [Eric] But then that still surfaces back up to the analyst and then eventually makes it to the client, right? 00:25:43,222 --> 00:25:43,252 [John] Right. 00:25:43,252 --> 00:25:48,772 [Eric] And so you're introducing risk there, or are you introducing risk? 00:25:48,772 --> 00:25:52,232 [John] Um, generally clients are directly interacting with, with the AI. Um- 00:25:52,232 --> 00:25:53,222 [Eric] Okay, they're directly interacting with the AI. 00:25:53,222 --> 00:25:54,792 [John] In, in this use case, yeah, yeah. 00:25:54,792 --> 00:25:55,032 [Eric] Yep, yep. 00:25:55,032 --> 00:26:02,522 [John] But yeah, sure. And the risk is the, um, is completely naive to think that your eval- evaluations are gonna cover all use cases. 00:26:02,522 --> 00:26:03,132 [Eric] Sure, right, right. 00:26:03,132 --> 00:26:03,142 [John] So- 00:26:03,142 --> 00:26:03,672 [Eric] That's, that's- 00:26:03,672 --> 00:26:03,852 [John] Yeah 00:26:03,852 --> 00:26:05,872 [Eric] ... yeah. It's, there are going to be mistakes that are made. 00:26:05,872 --> 00:26:14,352 [John] Yeah. And mistakes that are made and just, there's like a practical, like back to the token budget thing, like how much money do you wanna spend on the evals? 00:26:14,352 --> 00:26:15,852 [Eric] Mm-hmm. Mm-hmm. Right. Yeah, yeah, yeah. 00:26:15,852 --> 00:26:17,292 [John] It's just, there's some practical things there too. 00:26:17,292 --> 00:26:17,792 [Eric] Mm-hmm. 00:26:17,792 --> 00:26:18,912 [John] And time and, and whatever. 00:26:18,912 --> 00:26:19,092 [Eric] Mm-hmm. 00:26:19,092 --> 00:26:28,272 [John] So I think-- So that's what I'm thinking about when it comes to that. And then as far as deterministic things that are produced with AI, that's kind of that other bucket- 00:26:28,272 --> 00:26:28,322 [Eric] Mm-hmm 00:26:28,322 --> 00:26:29,782 [John] ... I think. Um, 00:26:31,012 --> 00:26:41,292 [John] I mean, yeah, back to the allocation thing, I would say. I was like, all right, how much time and effort do we spend on QA allocation? How much on research? How much on planning? How much on, um, actually like- 00:26:41,292 --> 00:26:41,472 [Eric] Mm 00:26:41,472 --> 00:26:47,972 [John] ... implementing? And nine times out of 10, it is not enough on research, not enough on planning, not enough on QA. 00:26:47,972 --> 00:26:48,512 [Eric] Hmm. 00:26:48,512 --> 00:26:51,032 [John] It's one, it's all three or like one of those things. 00:26:51,032 --> 00:27:01,032 [Eric] Yeah. Super interesting. There was a, a, a great blog post that we published at Vercel by one of our engineers called Agent Responsibly. 00:27:02,092 --> 00:27:02,792 [John] Okay. 00:27:02,792 --> 00:27:03,272 [Eric] And, 00:27:04,532 --> 00:27:15,491 [Eric] you know, one of the challenges I think that we have with implementing AI in our day-to-day work is that throughput can increase dramatically. 00:27:15,492 --> 00:27:16,112 [John] Yes. 00:27:16,112 --> 00:27:30,972 [Eric] And that means you sort of have, you know, two options, and this is a, this is an over simp- simplification, [chuckles] right? But it's like, okay, you just accept increa- you know, throughput's gonna increase dramatically, and some of it's gonna be non-deterministic, and that's fine. 00:27:30,972 --> 00:27:31,172 [John] Right. 00:27:31,172 --> 00:27:31,202 [Eric] Right? 00:27:31,202 --> 00:27:32,092 [John] Right. 00:27:32,092 --> 00:27:39,292 [Eric] Um, or you have to figure out how to deal with a lot more review of output. 00:27:39,292 --> 00:27:39,551 [John] Right. 00:27:39,552 --> 00:27:48,092 [Eric] Right? I mean, and that could be code, it could be, you know, an analysis, it could be writing, right? And so there's, um, 00:27:49,322 --> 00:28:04,792 [Eric] uh, you know, which is a, which is a very real challenge, right? How do... If, if everyone becomes significantly more productive, you know, is there an... There isn't an equal increase in productivity on every type of review, right? 00:28:04,792 --> 00:28:04,812 [John] Right. 00:28:04,812 --> 00:28:45,332 [Eric] And so this blog, Agent Responsibly, I think was really, really interesting because it was essentially, it was actually based on an internal memo, I think, or a, a s- a talk that this engineer gave in a team standup. And it sort of made the rounds internally because it was a very compelling, uh, um, you know, sort of charge for accountability. And, um, and so we ended up publishing it on the blog. But essentially, the jur-- it, it drew lines of jurisdiction, and it said, you know, at Vercel, if you're gonna push code to production, then it doesn't matter if you wrote it or an agent wrote it, you, you own it, right? 00:28:45,332 --> 00:28:45,532 [John] Right. 00:28:45,532 --> 00:28:49,592 [Eric] And so the production problem is your fault. Like- 00:28:49,592 --> 00:28:50,192 [John] Right 00:28:50,192 --> 00:29:04,002 [Eric] ... and you take full responsibility for that, right? And, and that's sort of drawing the lines of jurisdiction, right? And it's, you know, the nuance there is that not every project is production code, right? 00:29:04,002 --> 00:29:04,032 [John] Yep. 00:29:04,032 --> 00:29:06,192 [Eric] And even when you're changing production code, 00:29:07,312 --> 00:29:21,722 [Eric] not every change is as critical, right? And so if I'm doing a database migration, that's really different than changing the font size, you know, for an H3 on the blog, right? Those are dramatically- 00:29:21,722 --> 00:29:21,982 [John] Yeah 00:29:21,982 --> 00:29:23,852 [Eric] ... different [chuckles] levels of consequence- 00:29:23,852 --> 00:29:24,512 [John] Right 00:29:24,512 --> 00:29:26,002 [Eric] ... if something goes wrong. Um, 00:29:27,152 --> 00:29:28,812 [Eric] and so the 00:29:30,152 --> 00:29:35,292 [Eric] level of required accountability is not evenly distributed, and I think- 00:29:35,292 --> 00:29:35,572 [John] Right 00:29:35,572 --> 00:29:37,612 [Eric] ... maps to the, maps to the amount of consequence, right? 00:29:37,612 --> 00:29:37,951 [John] Right. 00:29:37,952 --> 00:29:46,672 [Eric] But it is a really interesting time for managers who need to review work, especially when there's way more work to review. 00:29:46,672 --> 00:29:47,152 [John] Right. 00:29:47,152 --> 00:29:50,252 [Eric] Um, and especially when that work needs to be reviewed by a human. 00:29:51,392 --> 00:30:09,072 [Eric] Uh, in my role, you know, one of the... I, I review and edit a lot of content, and we don't publish anything that has not gone through, um, a full, that has not been fully scrutinized and, uh, like, and reviewed by a human. 00:30:09,132 --> 00:30:09,222 [John] Right. 00:30:09,222 --> 00:30:18,702 [Eric] Right? And so, but we also use AI very heavily, and so the amount of review required is, is really significant. 'Cause before it was like- 00:30:18,702 --> 00:30:18,702 [John] Yeah 00:30:18,702 --> 00:30:20,972 [Eric] ... okay, people can only do this so fast. 00:30:20,972 --> 00:30:21,152 [John] Right. 00:30:21,152 --> 00:30:29,156 [Eric] You know, and so you add more people, you know, but now it, it's very different because the throughput of a very small team is super high. Um- 00:30:29,156 --> 00:30:37,136 [Eric] You know, which is a challenge. But we tend to treat it, things the same way on my team. Not tend to, but we, we have human review, right? 00:30:37,136 --> 00:30:37,436 [John] Right. 00:30:37,436 --> 00:30:37,566 [Eric] Um, 00:30:38,636 --> 00:30:56,086 [Eric] and that is tricky. But I- but that is, I, I think one of the challenges that leaders will increasingly face in the age of AI is defining jurisdiction and accountability along those lines, 00:30:57,316 --> 00:31:07,256 [Eric] especially related to consequence. And there, there are some areas that are very clear on that, right? A database migration, no question. 00:31:07,256 --> 00:31:07,475 [John] Right. 00:31:07,476 --> 00:31:12,916 [Eric] Um, but there are a lot of areas where it's, it's kind of tricky to answer that question. 00:31:12,916 --> 00:31:13,096 [John] Right. 00:31:13,096 --> 00:31:53,496 [Eric] Uh, and so why don't we end with some practical advice. So for a leader who... Let's think about a leader who is, like, well on their way to implementing AI in the workplace. Maybe they're at the, the place in the adoption curve where, you know, they're still in the learning phase, but throughput is probably going to increase pretty significantly. What h- what would you say to the, to the leader who's gonna be dealing with, one, a lot more throughput and review, and then two, where do I draw the lines of, you know, where do the lines of jurisdiction fall and how do I, how do I implement those well? 00:31:53,496 --> 00:31:53,756 [John] Right. 00:31:55,516 --> 00:32:03,586 [John] So I don't think management by objectives works for agents yet, and people are trying to do it. 00:32:03,586 --> 00:32:03,696 [Eric] Mm-hmm. 00:32:03,696 --> 00:32:07,296 [John] There's been a lot of, like, hype recently around, 00:32:08,576 --> 00:32:13,076 [John] uh... So there, there's this goal command, like give the agent a goal- 00:32:13,076 --> 00:32:13,246 [Eric] Mm-hmm 00:32:13,246 --> 00:32:19,056 [John] ... an objective, have it do a thing. Recently, I think loop engineering is a term, which is a similar- 00:32:19,056 --> 00:32:19,156 [Eric] Yeah 00:32:19,156 --> 00:32:19,976 [John] ... concept. 00:32:19,976 --> 00:32:19,986 [Eric] Yeah. 00:32:19,986 --> 00:32:21,656 [John] Before then somewhat, like- 00:32:21,656 --> 00:32:21,666 [Eric] Yeah 00:32:21,666 --> 00:32:26,616 [John] ... so people keep pushing up against this, like, give the agent an objective, give the agent an objective. And 00:32:27,736 --> 00:32:35,356 [John] sure, to some extent, I, I do think you can do that, but not in the same way that you would give a human an, an objective yet. 00:32:35,356 --> 00:32:36,136 [Eric] Yep. 00:32:36,136 --> 00:32:44,036 [John] And because of that, um, the human objective, I think, is still more important, and the human is still in charge. 00:32:44,036 --> 00:32:44,196 [Eric] Yeah. 00:32:44,196 --> 00:32:49,146 [John] Like [chuckles] we don't actually have any AIs managing humans at this point. Um [laughs] 00:32:49,146 --> 00:32:50,056 [Eric] [laughs] 00:32:50,056 --> 00:32:53,146 [John] S- despite what some people may be saying. Um- 00:32:53,146 --> 00:32:53,516 [Eric] We... I think, 00:32:54,696 --> 00:32:59,225 [Eric] I think we do have some humans asking AI to manage them. [laughs] 00:32:59,225 --> 00:33:00,416 [John] Yeah. Totally. Totally. 00:33:00,416 --> 00:33:00,796 [Eric] But that's different. 00:33:02,016 --> 00:33:02,676 [Eric] [laughs] 00:33:02,676 --> 00:33:16,076 [John] Yeah. Yeah. So the practical, though, is, is to remember that. It's like, wait, b- because if you work with these things for a long amount of time, like, you're totally right, that it can be like, "What, what do you think I should do?" You know what I mean? [laughs] You can just get down the road of like- 00:33:16,076 --> 00:33:16,776 [Eric] Oh, yeah 00:33:16,776 --> 00:33:17,546 [John] ... "What should I do?" 00:33:17,546 --> 00:33:17,556 [Eric] Yeah, absolutely. 00:33:17,556 --> 00:33:19,396 [John] Like, you just, you just tell me, right? 00:33:19,396 --> 00:33:19,505 [Eric] Right. 00:33:19,505 --> 00:33:21,975 [John] Like, and, and you can get a answer that sounds right. 00:33:21,976 --> 00:33:22,476 [Eric] Mm-hmm. 00:33:22,476 --> 00:33:23,796 [John] Um, sometimes it is, sometimes it's not. 00:33:23,796 --> 00:33:24,516 [Eric] Mm-hmm. 00:33:24,516 --> 00:33:24,676 [John] Um, 00:33:25,716 --> 00:33:29,866 [John] so I think, I think a really practical thing that I found there is, 00:33:30,976 --> 00:33:35,716 [John] um... 'cause back to the, like, time research, time planning, time doing QA. 00:33:35,716 --> 00:33:36,016 [Eric] Mm-hmm. 00:33:36,016 --> 00:33:42,116 [John] And then of course, like, some time building, is taking some of those other steps more analog. 00:33:42,116 --> 00:33:42,616 [Eric] Hmm. 00:33:42,676 --> 00:33:47,606 [John] So taking, "All right, let's do the planning," like whiteboard, audio. 00:33:47,606 --> 00:33:47,616 [Eric] Yeah. 00:33:47,616 --> 00:33:59,416 [John] Like, not gonna use computer. Or let's do, um... QA is the toughest one to take that because at that point, like you've usually generated a lot of work product that it needs to stay digital. Um, but planning and research both- 00:33:59,416 --> 00:34:00,476 [Eric] Mm-hmm 00:34:00,476 --> 00:34:07,156 [John] ... I think. And now AI is super helpful for research too, but I think both of those things... Like, just as a really practical example, 00:34:08,416 --> 00:34:15,176 [John] um, there's a few of us in the room the other day, we were whiteboarding something, and then somebody got out, uh, voice AI, I think it was ChatGPT or- 00:34:15,176 --> 00:34:15,436 [Eric] Mm-hmm 00:34:15,436 --> 00:34:25,096 [John] ... one of the other ones, and we're whiteboarding and drawing out, like, system diagram, and then literally, like, live chatting with AI, and it's going and doing some web searches and like- 00:34:25,096 --> 00:34:25,126 [Eric] Hmm 00:34:25,126 --> 00:34:29,456 [John] ... "Hey, I think we need, like, a, an email tool," and be like, "Hey, like, what do people use?" Like- 00:34:29,456 --> 00:34:29,856 [Eric] Right 00:34:29,856 --> 00:34:32,496 [John] ... something like SendGrid, and then it goes and finds three alternate. Like- 00:34:32,496 --> 00:34:32,796 [Eric] Mm-hmm 00:34:32,796 --> 00:34:34,416 [John] ... just that kind of thing- 00:34:34,416 --> 00:34:34,746 [Eric] Sure 00:34:34,746 --> 00:34:36,156 [John] ... is a really neat- 00:34:36,156 --> 00:34:36,656 [Eric] Totally 00:34:36,656 --> 00:34:37,626 [John] ... way to work. Um, 00:34:38,636 --> 00:34:44,596 [John] and another really neat way, which we've talked about a lot, I think you use WhisperFlow a lot, and use a similar thing- 00:34:44,596 --> 00:34:44,606 [Eric] Yeah 00:34:44,606 --> 00:34:58,736 [John] ... where you start with your ideas and you can, um, be as unorganized and as whatever as you want and just talk out the plan, and then easily have an organized outline of your thoughts that's, like, very coherent- 00:34:58,736 --> 00:34:59,356 [Eric] Yep 00:34:59,356 --> 00:35:00,436 [John] ... from, from the plan. 00:35:00,716 --> 00:35:00,726 [Eric] Yep. 00:35:00,726 --> 00:35:12,976 [John] And then just use those grounding starting points versus what people tend to do is just hop in and, like, um, just start. Like, "Hey, I need to build this thing or do this thing." Like, "Can you do it?" 00:35:12,976 --> 00:35:13,216 [Eric] Yep. 00:35:13,216 --> 00:35:18,876 [John] And then... And you do get some cool... Even if you work that way, you can still get some cool moments like, "Wow," like, "This is all," like, "This is so great." 00:35:18,876 --> 00:35:19,376 [Eric] Mm-hmm. 00:35:19,376 --> 00:35:25,376 [John] Um, but the consistency over time of, like, where I've gotten the best results 00:35:26,396 --> 00:35:33,436 [John] has been starting here with some kind of, like, longer input other than, like, "Hey, can you build the thing?" 00:35:33,436 --> 00:35:34,076 [Eric] Hmm. 00:35:34,076 --> 00:35:40,136 [John] Um, and, and I think because it occasionally works when you say, like, "Hey, can you build the thing?" 00:35:40,136 --> 00:35:40,336 [Eric] Mm-hmm. 00:35:40,336 --> 00:35:41,856 [John] It's really tempting to just work that way. 00:35:41,856 --> 00:35:42,136 [Eric] Hmm. 00:35:42,136 --> 00:35:43,116 [John] 'Cause it's easier. 00:35:43,116 --> 00:35:51,576 [Eric] Yep. Yep. So if I had to summarize that, you, you pointed out a couple things. One would be go analog, 00:35:53,436 --> 00:36:00,995 [Eric] which I, you know, I think my interpretation is that that's to continue to build muscle, right? Like, whiteboard- 00:36:00,996 --> 00:36:01,176 [John] Right 00:36:01,176 --> 00:36:07,396 [Eric] ... maintain your ability to collaborate and think creatively without AI. Um- 00:36:07,396 --> 00:36:19,376 [John] The best meetings I'm in are still multiple people, whiteboard, multiple people have markers, and we're, like, physically collaborating on a actual whiteboard. They're still the best meetings. 00:36:19,376 --> 00:36:29,916 [Eric] Yep. Yep. And then also doing the upfront work on the input and not getting lazy on, you know, just issuing a, a very basic command. 00:36:29,916 --> 00:36:30,656 [John] Right. 00:36:30,656 --> 00:36:30,966 [Eric] Right? 00:36:30,966 --> 00:36:31,076 [John] Right. 00:36:31,076 --> 00:36:32,236 [Eric] Um- 00:36:32,236 --> 00:36:38,992 [John] Which You can still do, like you can do that, and I've... I'm guilty of that too. But then I just spend a bunch of time editing, you know? 00:36:38,992 --> 00:36:39,572 [Eric] Right. [laughs] 00:36:39,572 --> 00:36:40,192 [John] And I think like- 00:36:40,192 --> 00:36:40,832 [Eric] The review. 00:36:40,832 --> 00:36:42,432 [John] Yeah. [laughs] Like I could have done... 00:36:43,532 --> 00:36:52,932 [John] Yeah, c- 'cause you can technically, and it's important to note this, 'cause I think you can get to that same quality output, and you get to pick which side you wanna put more work in. 00:36:52,932 --> 00:36:53,392 [Eric] Yeah. 00:36:53,392 --> 00:36:59,572 [John] And I would say 9 times out of 10 it's better to put more work in over here on the planning and research than more work on the QA and editing. 00:36:59,572 --> 00:37:00,152 [Eric] Yes. I 100% agree. 00:37:00,152 --> 00:37:05,732 [John] So if you find that you spend tons of time on QA and editing, like, try putting more effort into planning and research. 00:37:05,732 --> 00:37:11,972 [Eric] Yes. I totally agree. That was the third thing that you brought up, is make sure that your pie chart is balanced and heavily- 00:37:11,972 --> 00:37:11,981 [John] Right 00:37:11,981 --> 00:37:13,332 [Eric] ... weighted towards planning and research. 00:37:13,332 --> 00:37:13,832 [John] Right. 00:37:13,832 --> 00:37:14,012 [Eric] Yep. 00:37:15,052 --> 00:37:19,672 [Eric] One of the interesting things, I think, when it comes to accountability 00:37:20,732 --> 00:37:23,072 [Eric] in, in the age of AI is that 00:37:24,272 --> 00:37:30,232 [Eric] it's easy to use our preexisting model for work as- 00:37:30,232 --> 00:37:30,242 [John] Mm-hmm 00:37:30,242 --> 00:37:34,812 [Eric] ... the baseline, and forget that the way that we work is actually changing- 00:37:34,812 --> 00:37:35,292 [John] Right 00:37:35,292 --> 00:37:37,032 [Eric] ... pretty, pretty dramatically. 00:37:37,032 --> 00:37:37,122 [John] Right. 00:37:37,122 --> 00:37:42,352 [Eric] Right? So there are entirely new ways of doing things that we did before. 00:37:43,512 --> 00:37:43,912 [Eric] And 00:37:45,112 --> 00:38:01,132 [Eric] one, at least one of the major, um, at least one of the major shifts that I see, and I actually... Someone, uh, we'll put it in the show notes, I'll dig up the tweet. There's a guy named, um, LM Sacasas. How about that for a name? 00:38:01,132 --> 00:38:01,642 [John] Love it. 00:38:01,642 --> 00:38:08,672 [Eric] Uh, he's written about technology's impact on us as humans in society for a very long time. Unbelievable thinker. 00:38:09,912 --> 00:38:10,272 [Eric] And 00:38:11,712 --> 00:38:27,152 [Eric] he summarized, I think, the risk in a way that is very concise and really helpful for, for leaders. He said that, um, one thing that is unique about AI is that it has... It is, um, 00:38:28,452 --> 00:38:37,592 [Eric] it, it can very easily and almost even naturally erode your ability to use it well. 00:38:37,592 --> 00:38:38,632 [John] Hmm. 00:38:38,632 --> 00:38:41,962 [Eric] Which is a very unique characteristic of a technology. 00:38:41,962 --> 00:38:42,092 [John] Right. 00:38:42,092 --> 00:38:42,532 [Eric] Um, 00:38:44,092 --> 00:38:49,452 [Eric] and it's, and that describes exactly what you were talking about, right? Where my pie chart is way out of balance. 00:38:49,452 --> 00:38:49,612 [John] Right. 00:38:49,612 --> 00:38:57,152 [Eric] Um, which is interesting. Side note, if your pie chart's out of balance, it gets way more expensive, 'cause you have to do way, way more turns. 00:38:57,152 --> 00:38:57,352 [John] True. Yeah. 00:38:57,352 --> 00:38:57,882 [Eric] Which is interesting. 00:38:57,882 --> 00:38:58,432 [John] Good point. Yeah. 00:38:58,432 --> 00:39:00,292 [Eric] Um, so I would think about that as a leader as well. 00:39:01,352 --> 00:39:02,602 [John] Mm-hmm. 00:39:02,602 --> 00:39:06,452 [Eric] Um, so using AI well is good thinking, is actually cheaper. 00:39:06,452 --> 00:39:06,801 [John] Mm-hmm. 00:39:09,072 --> 00:39:20,752 [Eric] Um, uh, the other thing that I... And, and so in, in the, you know, with erosion of your ability to use AI as a result of using AI [laughs] with that as the backdrop- 00:39:20,752 --> 00:39:21,292 [John] Right 00:39:21,292 --> 00:39:34,452 [Eric] ... one thing that I actually believe that I'm going to start doing, because I've done a lot of this personally, and I'm, I'm going to talk to my team about it, is intentionally putting in practice sharpening your skills without AI. 00:39:34,452 --> 00:39:35,312 [John] Mm-hmm. 00:39:35,312 --> 00:39:38,152 [Eric] Um, because it makes you... It, it 00:39:39,172 --> 00:39:45,202 [Eric] dramatically increases your ability to wield AI, especially for writing, I think that's true- 00:39:45,202 --> 00:39:45,202 [John] Mm-hmm 00:39:45,202 --> 00:39:45,992 [Eric] ... generally. 00:39:45,992 --> 00:39:46,852 [John] Mm-hmm. 00:39:46,852 --> 00:40:00,352 [Eric] Um, but of course, you know, in my world, we use AI to write a lot, and I just find that the more that I practice writing without AI, the, the more helpful it seems to get when I use it. 00:40:00,352 --> 00:40:00,792 [John] Hmm. 00:40:00,792 --> 00:40:07,502 [Eric] Um, which is really interesting, and even changes the way that I use it or, like, where I choose to employ it. 00:40:07,502 --> 00:40:07,592 [John] Yeah. 00:40:07,592 --> 00:40:07,992 [Eric] Um, 00:40:09,192 --> 00:40:16,672 [Eric] and it's really fascinating. So I guess those are, those are my recommendations for the leaders out there thinking about accountability. 00:40:16,672 --> 00:40:21,092 [John] We got one more. Um, you know Dan Shipper from Every? 00:40:21,092 --> 00:40:21,392 [Eric] Yes. 00:40:21,392 --> 00:40:21,852 [John] He's on- 00:40:21,852 --> 00:40:22,232 [Eric] Mm-hmm 00:40:22,232 --> 00:40:23,212 [John] ... Lenny's podcast. 00:40:23,212 --> 00:40:23,552 [Eric] Yep, yep. 00:40:23,552 --> 00:40:24,342 [John] Great episode. 00:40:24,342 --> 00:40:24,872 [Eric] Yep. 00:40:24,872 --> 00:40:30,692 [John] You guys should check out. Um, but he, he's, um, he's kind of an AI thinker- 00:40:30,692 --> 00:40:30,772 [Eric] Mm-hmm 00:40:30,772 --> 00:40:33,332 [John] ... and, and very much a builder. Less on the, like, academic side, very much the practical. 00:40:33,332 --> 00:40:35,282 [Eric] Yeah, Every has, like, a bunch of products. 00:40:35,282 --> 00:40:37,512 [John] They have several pro- yeah, several products. Um- 00:40:37,512 --> 00:40:39,392 [Eric] Like a product studio. 00:40:39,392 --> 00:40:48,652 [John] Yeah, exactly. So one of the things he got me thinking about from that podcast was, um... So he was going through essentially how they use AI and, and talking about how his thinking has changed. 00:40:48,652 --> 00:40:49,452 [Eric] Mm-hmm. 00:40:49,452 --> 00:41:05,792 [John] And, um, it's just so interesting to, to dig in in their world and, and he talks about, like, them really, um, thinking, like moving to the agenti- like, working with agents versus prompts, right? 00:41:05,792 --> 00:41:05,941 [Eric] Mm-hmm. 00:41:05,941 --> 00:41:14,052 [John] Like, which I think a lot of people have gotten there at this point. But I think, I guess was Lenny, like, asking, like, well, you know, what's next? And, um, 00:41:15,232 --> 00:41:34,992 [John] and I... Well, I mean, I think the thing I took away from that with my team is, like, as we're thinking and planning and researching, like, the, the core thing is, is walking through, um, and this is kind of like what you were saying with writing, is walking through the steps without every time, like, having your hand held. 00:41:34,992 --> 00:41:35,552 [Eric] Mm-hmm. 00:41:35,552 --> 00:41:41,652 [John] Because that's like the A- the AI is, like, if you're used to having your hand held and you, like, just stay there- 00:41:41,652 --> 00:41:42,332 [Eric] Mm-hmm 00:41:42,332 --> 00:41:47,632 [John] ... then, like, it's, it's, like, too uncomfortable, like, when, when it's not there. 00:41:47,632 --> 00:41:47,992 [Eric] Yeah. 00:41:47,992 --> 00:41:48,572 [John] Right? 00:41:48,572 --> 00:41:48,872 [Eric] Yep. 00:41:48,872 --> 00:41:49,452 [John] And you get... 00:41:50,472 --> 00:41:50,872 [John] A- and then- 00:41:50,872 --> 00:41:51,912 [Eric] Yes, yes 00:41:51,912 --> 00:41:57,752 [John] ... and, and, and, and the way I would... And the other, like, component of this is not 00:41:59,212 --> 00:42:03,162 [John] the, the... Remember, we, we've talked a lot actually in the past of the blank page problem. 00:42:03,162 --> 00:42:03,432 [Eric] Mm-hmm. 00:42:03,432 --> 00:42:10,312 [John] Which, like, five seconds on that is essentially, like, people typically start using AI, they pull up a blank screen, and then it's like, "Well, what do I even type?" 00:42:10,312 --> 00:42:10,632 [Eric] Right. 00:42:10,632 --> 00:42:22,732 [John] Um, but I think for my team, it's almost, like, full circle of, like, don't, don't ever start with a blank page. Start with something, like, human-produced that's, like, as close as is reasonable to what you wanted- 00:42:22,732 --> 00:42:23,092 [Eric] Hmm. Mm-hmm 00:42:23,092 --> 00:42:24,792 [John] ... and then start using AI. 00:42:24,792 --> 00:42:27,732 [Eric] Okay, I have a very specific example of this 00:42:28,972 --> 00:42:37,952 [Eric] that I think will be great to close out on. So I hired a guy, I'll say, I'll say his name, I hope he doesn't mind, but his name is Kevin Sundstrom. 00:42:37,952 --> 00:42:38,452 [John] Yeah. 00:42:38,452 --> 00:42:46,087 [Eric] Uh, he was at GitHub. He, um, he built- ... the content agent that they use inside of, um, GitHub- 00:42:46,088 --> 00:42:46,448 [John] Sweet 00:42:46,448 --> 00:42:48,988 [Eric] ... uh, sort of built that out. And, um, 00:42:50,828 --> 00:42:59,538 [Eric] just a really, really sharp guy, and he's only been on the job for... He's been on the job two weeks. Okay? 00:42:59,538 --> 00:43:00,187 [John] Okay. 00:43:00,188 --> 00:43:13,678 [Eric] And we are in the process of sort of, you know, we had a very, very small team, and so we built out this content agent. Actually, another Kevin on my team, um, Kevin Corbett. I'm just gonna give these guys credit- 00:43:13,678 --> 00:43:13,798 [John] So many Kevins 00:43:13,798 --> 00:43:15,048 [Eric] ... because they're great. I only hire Kevins. 00:43:15,048 --> 00:43:16,008 [John] Do you have a third Kevin? [laughs] 00:43:16,008 --> 00:43:17,368 [Eric] No. Uh- 00:43:17,368 --> 00:43:18,278 [John] All right. I'll look for... I'll help you. I'll help you find one 00:43:18,278 --> 00:43:20,168 [Eric] ... no, the third hire was Amelia- 00:43:20,168 --> 00:43:20,178 [John] All right 00:43:20,178 --> 00:43:21,268 [Eric] ... uh, who's amazing- 00:43:21,268 --> 00:43:21,628 [John] All right 00:43:21,628 --> 00:43:22,337 [Eric] ... as well. But, um, 00:43:23,608 --> 00:43:29,508 [Eric] Kevin Corbett built out this really great agent, but it was sort of like we just- we used it, right? 00:43:29,508 --> 00:43:29,517 [John] Mm-hmm. 00:43:29,517 --> 00:43:31,678 [Eric] And so there's, like, rough edges, and it's super- 00:43:31,678 --> 00:43:31,678 [John] Right 00:43:31,678 --> 00:43:32,958 [Eric] ... powerful, but- 00:43:32,958 --> 00:43:32,958 [John] Right 00:43:32,958 --> 00:43:36,758 [Eric] ... there are just a lot of things, like the ergonomics weren't great, but it's, like, awesome, you know? 00:43:36,758 --> 00:43:37,408 [John] Yeah. Right. 00:43:37,408 --> 00:43:47,148 [Eric] And there was sort of another, like, lighter weight agent that, you know, other people in the company were using that was sort of like a derivative of this or whatever, but this was sort of, like, our content engineering agent. 00:43:47,148 --> 00:43:47,198 [John] Mm-hmm. 00:43:48,268 --> 00:43:55,748 [Eric] And so we've just been so busy 'cause we were such a small team that we hadn't had time to really, you know, spend cycles on- 00:43:55,748 --> 00:43:55,788 [John] Right 00:43:55,788 --> 00:43:56,508 [Eric] ... improving the agent- 00:43:56,508 --> 00:43:56,638 [John] Mm-hmm 00:43:56,638 --> 00:44:05,958 [Eric] ... and then rolling it out to the rest of the company, right? And so now that we have more people on the team, a big focus is let's, let's really augment this. Let's fix the ergonomics and the UX, and then let's start to expose it in surfaces so that- 00:44:05,958 --> 00:44:05,958 [John] Right 00:44:05,958 --> 00:44:10,468 [Eric] ... other people in the company can use it. And so Kevin Sundstrom, 00:44:11,788 --> 00:44:16,648 [Eric] in our first, in our first, like, one-on-one meeting, he wanted to talk a lot about the content agent, 00:44:17,688 --> 00:44:24,148 [Eric] and I was like, "Great, you know, we need... You know, let's, let's make this thing awesome. I mean, it is awesome, but let's make it, like- 00:44:24,148 --> 00:44:24,388 [John] Yeah 00:44:24,388 --> 00:44:25,458 [Eric] ... you know, usable and- 00:44:25,458 --> 00:44:25,828 [John] Yeah. Right 00:44:25,828 --> 00:44:30,288 [Eric] ... you know, uh, and, and get adoption throughout the company." And, um, 00:44:31,968 --> 00:44:38,107 [Eric] he said, "You know, w- I think that we're sort of missing, like, the most important piece of this thing." 00:44:38,108 --> 00:44:38,528 [John] Hmm. 00:44:38,528 --> 00:44:50,797 [Eric] And I was like, "What do you mean?" And he said, "Well, like, if we use this on our team, it feels like a really powerful tool," and it is a really powerful tool. 00:44:50,797 --> 00:44:51,718 [John] Mm-hmm. 00:44:51,718 --> 00:44:55,098 [Eric] And he said, "But if we... As we start to think about distributing this to the rest of the company," 00:44:56,428 --> 00:45:03,508 [Eric] he said, "it's, it's probably not going to produce the same results-" 00:45:03,508 --> 00:45:04,488 [John] Mm-hmm 00:45:04,488 --> 00:45:06,828 [Eric] "... because, um, 00:45:08,268 --> 00:45:13,858 [Eric] it doesn't, like, force or help people do the hardest part." 00:45:13,858 --> 00:45:13,868 [John] Right. 00:45:13,868 --> 00:45:27,748 [Eric] And I said, "Okay, well, that's really interesting. What do you mean?" And he said, "Well, like, if you boil really good writing down to its essence, it's that you're bringing, like, a very well-thought-out argument to the table." 00:45:27,748 --> 00:45:28,238 [John] Mm-hmm. 00:45:28,238 --> 00:45:35,208 [Eric] "And so if you do that, and you feed it into our agent, then it, it's feels really powerful, right?" 00:45:35,208 --> 00:45:35,238 [John] Right. 00:45:35,238 --> 00:45:38,638 [Eric] And he's like, "And that's why we love using it," you know? [laughs] 00:45:38,638 --> 00:45:38,668 [John] Right. 00:45:38,668 --> 00:45:40,628 [Eric] "And that's why, like, we're okay with the rough edges." 00:45:40,628 --> 00:45:40,808 [John] Right. 00:45:40,808 --> 00:45:41,048 [Eric] Um, 00:45:42,468 --> 00:45:50,908 [Eric] and he said, "But we're not helping other people do that." And he said, "But that's actually the thing that's the hardest that most people really struggle with in writing-" 00:45:50,908 --> 00:45:51,448 [John] Right 00:45:51,448 --> 00:45:56,308 [Eric] "... is really bringing clarity to the ideas that they're presenting-" 00:45:56,308 --> 00:45:56,388 [John] Right 00:45:56,388 --> 00:46:00,068 [Eric] "... you know, as an argument." And he said, "If we, 00:46:01,308 --> 00:46:12,228 [Eric] if we want this to produce great results for the rest of the company, we need to make it easier for people. We need to use AI to, like, draw that out of people." 00:46:12,228 --> 00:46:12,238 [John] Yeah. Yeah. 00:46:12,238 --> 00:46:16,088 [Eric] "Because if they don't put that into the system, we're not gonna get good results out, and they won't either." 00:46:16,088 --> 00:46:18,768 [John] Having, having a really powerful Q&A tool is, like, the best. 00:46:18,768 --> 00:46:26,708 [Eric] Yes, exactly. Uh, and so that just really... It was really interesting. He said that's... He said, "It's so hard to do that." And so- 00:46:26,708 --> 00:46:26,718 [John] Right 00:46:26,718 --> 00:46:31,808 [Eric] ... but he said, "But we can't. It won't be successful without that. And so we need to change the way that we're thinking about this 00:46:32,948 --> 00:46:36,078 [Eric] in that we need to make that part easier for people." 00:46:36,078 --> 00:46:36,108 [John] Yeah. 00:46:36,108 --> 00:46:37,528 [Eric] How do we, you know, sort of draw that out? 00:46:37,528 --> 00:46:38,588 [John] Yeah. 00:46:38,588 --> 00:46:42,508 [Eric] Anyways, I thought that was really great because it was sort of this, like, um... 00:46:43,788 --> 00:46:48,448 [Eric] It was a really great form of accountability where it's like, no, I mean, you have to do the hard work- 00:46:48,448 --> 00:46:48,788 [John] Right 00:46:48,788 --> 00:46:49,328 [Eric] ... up front- 00:46:49,328 --> 00:46:49,448 [John] Right 00:46:49,448 --> 00:46:52,038 [Eric] ... still, you know, no matter how powerful the agent is. 00:46:52,038 --> 00:46:52,348 [John] Right. 00:46:52,348 --> 00:46:52,668 [Eric] So. 00:46:52,668 --> 00:47:10,488 [John] So my quick example of this was working on, um, working with one of our interns actually, um, to summarize some meeting notes where we did some requirements with a client. And they... I was like, "Hey, like, you, you give it a shot. Like, here's the transcription, you know, here's some, like, rough guidance. Like, give it a shot." 00:47:10,488 --> 00:47:10,528 [Eric] Mm-hmm. 00:47:10,528 --> 00:47:25,107 [John] "Like, kind of summarize it, and let's, let's start working on, um, iterations of what we're gonna do for them." And they, you know, and they did fine, and you kind of had a basic outline and stuff. I was like, "Let me show you a secret." So we go, like... And this was just what came top of mind. I didn't have this, like, super planned out. 00:47:25,108 --> 00:47:25,428 [Eric] Mm-hmm. 00:47:25,428 --> 00:47:40,198 [John] And we'll probably refine this over time. I was like, "All right. Let's go find what is most analogous to what we're trying to do." So I was like, "All right. This is... We actually want an architectural overview," and we, like, got to some more specifics and, like, data architecture overview, and we kind of refined. 00:47:40,198 --> 00:47:40,208 [Eric] Mm-hmm. 00:47:40,208 --> 00:47:42,868 [John] Like, okay, this is the document we w- we're trying to make. 00:47:42,868 --> 00:47:43,288 [Eric] Mm-hmm. 00:47:43,288 --> 00:47:45,708 [John] We're not just making a generic SOW or whatever. 00:47:45,708 --> 00:47:45,928 [Eric] Mm-hmm. 00:47:45,928 --> 00:47:47,048 [John] Actually, we need this. 00:47:47,048 --> 00:47:47,628 [Eric] Mm-hmm. 00:47:47,628 --> 00:47:53,468 [John] So super helpful. And then number two was like, "All right. Let's go find one." So it's like McKenzie data architectural overview, blah, blah. 00:47:53,468 --> 00:47:53,828 [Eric] Mm-hmm. Mm-hmm. 00:47:53,828 --> 00:48:03,667 [John] And it's like, oh look, there's a PDF. It's like, all right, like, this is a decent starting point, and just helping somebody. If you can get to, like, an already good version of a thing, and then, like- 00:48:03,668 --> 00:48:03,948 [Eric] Mm-hmm 00:48:03,948 --> 00:48:07,648 [John] ... take your context and overlay it into the al- already good version of the thing. 00:48:07,648 --> 00:48:08,107 [Eric] Yeah. 00:48:08,108 --> 00:48:09,898 [John] Which is what humans already do mentally. 00:48:09,898 --> 00:48:09,908 [Eric] Yes. 00:48:09,908 --> 00:48:15,348 [John] Like, if you went to school, you just have a mental, like, list of templates that you can overlay- 00:48:15,348 --> 00:48:15,358 [Eric] Mm-hmm 00:48:15,358 --> 00:48:17,988 [John] ... into the context. Like, it's the same process. 00:48:17,988 --> 00:48:18,668 [Eric] Yep. 00:48:18,668 --> 00:48:22,948 [John] But now it's easier because you're, you've got this infinite list of templates. 00:48:22,948 --> 00:48:23,468 [Eric] Mm-hmm. 00:48:23,468 --> 00:48:27,108 [John] And as long as you have the ability to, to construct that right starting point- 00:48:27,108 --> 00:48:27,588 [Eric] Mm. 00:48:27,588 --> 00:48:30,428 [John] And then, and then, you know, we ran it back through, and it was like 100 times better. 00:48:30,428 --> 00:48:30,528 [Eric] Yep. 00:48:30,528 --> 00:48:31,648 [John] Like 100 times better. 00:48:31,648 --> 00:48:31,908 [Eric] Totally. 00:48:31,908 --> 00:48:35,948 [John] And, and, and the reaction I got was like, "Wow, like, this is, like, pretty sweet." You know what I mean? [laughs] 00:48:35,948 --> 00:48:35,978 [Eric] Yeah, yeah. 00:48:35,978 --> 00:48:37,908 [John] Just like got this from the, you know, from the guy. 00:48:37,908 --> 00:48:38,588 [Eric] Yeah. 00:48:38,588 --> 00:48:44,488 [John] And, and that's, and that's just one of those things where, um, I don't think that's necessarily obvious to people. 00:48:44,488 --> 00:48:45,968 [Eric] Yeah, I agree. I agree. 00:48:45,968 --> 00:48:46,478 [John] Yeah. 00:48:46,478 --> 00:48:53,668 [Eric] Yeah. And in fa- and in fact, going back to LM Sicases' point, I think the danger is that it... You're tempted to shortcut that, right? 00:48:53,668 --> 00:48:53,828 [John] Yeah. 00:48:53,828 --> 00:48:54,688 [Eric] And so 00:48:55,968 --> 00:49:02,608 [Eric] I, I think if we had to summarize everything that we've said on the show today, it's don't shortcut the process. 00:49:02,608 --> 00:49:02,948 [John] Right. 00:49:02,948 --> 00:49:13,508 [Eric] Maintain your ability to do the hard work up front, and that's a real unlock, and actually significantly decreases the risk of, you know, generating output that isn't what you want. 00:49:13,508 --> 00:49:14,808 [John] Yeah. Agreed. 00:49:14,808 --> 00:49:37,038 [Eric] All right. Well, thanks for joining the Token Intelligence Show. Uh, AI is changing the way we work. We help you understand the state-of-the-art, cut through the noise, and use wisdom to become a great leader in the age of AI, and we will catch you on the next show. [upbeat music]