Can AI actually replace an employee?

00:00:00,600 --> 00:00:15,070 [Eric] [upbeat music] Welcome back to Token Intelligence. John, I have been thinking a lot about- 00:00:15,070 --> 00:00:16,880 [John] You almost said the name of our other show, I think. 00:00:16,880 --> 00:00:17,760 [Eric] I almost did. 00:00:17,760 --> 00:00:18,160 [John] [laughs] 00:00:18,160 --> 00:00:19,410 [Eric] Our previous show. 00:00:19,410 --> 00:00:19,440 [John] Yeah. 00:00:19,440 --> 00:00:20,700 [Eric] Well, it was five years, so. 00:00:20,700 --> 00:00:21,840 [John] It was five years. [chuckles] 00:00:21,840 --> 00:00:23,040 [Eric] I said that many times. 00:00:23,040 --> 00:00:23,240 [John] Yeah. Sorry. 00:00:23,240 --> 00:00:24,000 [Eric] Hundreds of times. 00:00:25,120 --> 00:00:31,740 [Eric] Uh, I've been thinking a lot about i- hype around AI employees. 00:00:31,740 --> 00:00:31,940 [John] Yep. 00:00:33,020 --> 00:00:45,140 [Eric] And I saw a post on X that I think exemplifies this really well. Um, and it's from the founder of Sentry, David Kramer, 00:00:46,350 --> 00:01:19,260 [Eric] [lips smack] uh, you know, who's a really interesting guy and has-- He has great thoughts on a lot of different things. Uh, so I'll just read this. I'll, I'll just read this. He's quoting another tweet, um, that says, "Nobody's running 20 agents overnight and building stuff for actual users." So David Kramer comments on this and says, "Everyone is slowly coming to this realization, and I assure you, no one is running multitudes of agents overnight, no one that is doing anything of substance at least. There are people pretending to be scientists 00:01:20,300 --> 00:01:45,340 [Eric] or fully caught up in their drug-infused AI overdose that think their slop machines are changing the world. They're not, though, and they're just wasting a bunch of money and compute to create a lot of lines of code that will just get thrown away. The state-of-the-art is still, 'Can we even one-shot a production quality patch that we won't regret later?' And it's rarer than you'd expect based on discourse." 00:01:46,580 --> 00:01:58,160 [Eric] So this is interesting 'cause I think the news cycle around, you know, layoffs especially, you know. So there was a huge layoff at, uh, Block- 00:01:58,160 --> 00:01:58,650 [John] Block. Yeah 00:01:58,650 --> 00:02:19,630 [Eric] ... you know, which is a big financial company, um, you know, that does Square and, you know, Cash App and other, you know, other financial services. Um, you know, they laid off thousands of employees. And that's been in the news cycle a bunch, right? And so one of the headlines consistently is, you know, AI is replacing these employees. 00:02:19,630 --> 00:02:19,660 [John] Yeah. 00:02:19,660 --> 00:02:29,800 [Eric] And so then in the software world, like David Kramer said, you have people who are saying, "I'm running 20 agents overnight to, you know, accomplish all of this work." And 00:02:30,860 --> 00:02:44,240 [Eric] I have seen multiple posts like this where people say, "Okay, I mean, you're building something," but that's not actually a legitimate cycle in which production-level work- 00:02:44,240 --> 00:02:44,400 [John] Right 00:02:44,400 --> 00:02:51,799 [Eric] ... or real, you know, user, like, user-facing products are, um, are being generated that way. 00:02:51,800 --> 00:02:52,300 [John] Right. 00:02:52,300 --> 00:02:57,800 [Eric] And you came to mind as I-- E-every time I read these posts, you come to mind- 00:02:57,800 --> 00:02:57,810 [John] Right 00:02:57,810 --> 00:03:14,600 [Eric] ... because you have tried to create and have created AI employees, you know, or au- an autonomous fleet of agents, uh, for your business. And I wanted to ask you whether you think David Kramer 00:03:15,660 --> 00:03:16,320 [Eric] is correct. 00:03:17,580 --> 00:03:20,000 [John] I think he's mostly right 'cause we-- 00:03:21,220 --> 00:03:32,620 [John] I wouldn't say it's been hundreds, but we've done dozens of, like, internal app experiments of, "Okay, like, what if we could automate this, this thing?" 00:03:32,620 --> 00:03:32,900 [Eric] Yep. 00:03:32,900 --> 00:03:46,940 [John] Like, we do a lot of data engineering. So, like, a bunch of reps of like, all right, data engineering. And all of them have essentially [chuckles] stayed internal. Um, like we, like, we don't have any of these, like, autonomous agents, like, working on client projects. 00:03:46,940 --> 00:03:49,560 [Eric] [lips smack] Oh, interesting. Okay. 00:03:49,560 --> 00:03:49,920 [John] Um- 00:03:49,920 --> 00:03:55,600 [Eric] So what do they do? So give me an example. Describe one of your autonomous AI employees. 00:03:55,600 --> 00:03:56,460 [John] So- 00:03:56,460 --> 00:03:58,370 [Eric] Do you even call it that internally? 00:03:58,370 --> 00:03:59,180 [John] So I think- 00:03:59,180 --> 00:04:02,540 [Eric] Or did you call it that and you don't call it that [chuckles] anymore? 00:04:02,540 --> 00:04:12,500 [John] So I think there's, like, three different lev- different levels here. Um, like, one is what works at home personally for people. One- 00:04:12,500 --> 00:04:12,820 [Eric] Right 00:04:12,820 --> 00:04:21,719 [John] ... what works at work personally for people. Two, and then three, what works with a team at work. 00:04:21,720 --> 00:04:22,580 [Eric] Hmm. 00:04:22,580 --> 00:04:26,360 [John] So it's like personal me, like, for fun at home. 00:04:26,360 --> 00:04:26,600 [Eric] Yep. 00:04:26,600 --> 00:04:33,380 [John] Personal me at work, productivity, which, and then layer on some, like, security and some pieces there. And then, like, what works, uh, with a team at work. 00:04:33,380 --> 00:04:39,240 [Eric] Yep. Let, let's give examples of each of those because-- And I'll just frame it in terms of tooling. 00:04:39,240 --> 00:04:40,340 [John] Yep. 00:04:40,340 --> 00:04:41,940 [Eric] So what works for me at home, 00:04:43,170 --> 00:04:49,360 [Eric] there are three things that come to mind. So on the advanced-- on the technically advanced end of the spectrum, you have OpenClaw. 00:04:49,360 --> 00:04:49,920 [John] Right. 00:04:49,920 --> 00:04:51,080 [Eric] So you run it on a Mac Mini. 00:04:51,080 --> 00:04:51,640 [John] Yep. 00:04:51,640 --> 00:04:57,360 [Eric] You give it access to your text messages, your email, your calendar, everything, and it becomes, you know, personal, a personal assistant. 00:04:57,360 --> 00:04:57,480 [John] Right. 00:04:57,480 --> 00:04:58,780 [Eric] Right? And can do all these things for you. 00:04:58,780 --> 00:04:59,280 [John] Mm-hmm. 00:04:59,280 --> 00:05:02,180 [Eric] And people who go crazy, you know, will have it trade on Polymarket- 00:05:02,180 --> 00:05:02,280 [John] Yeah 00:05:02,280 --> 00:05:03,680 [Eric] ... and give it their credit card. 00:05:03,680 --> 00:05:04,060 [John] Yeah. 00:05:04,060 --> 00:05:09,540 [Eric] [lips smack] Uh, Zo Computer, I would say, is sort of middle ground. 00:05:09,540 --> 00:05:09,900 [John] Okay. 00:05:09,900 --> 00:05:21,780 [Eric] So it's AI running in a secure cloud computer. So you can kinda think about it as OpenClaw with the guardrails, um, that you would want, right? 00:05:21,780 --> 00:05:22,180 [John] Right. 00:05:22,180 --> 00:05:31,460 [Eric] Um, so it's running AI, but it's in a completely secure environment. I mean, you can give it access to certain things, but it's not nearly as, you know, sort of on the risk end of the spectrum, um, 00:05:32,760 --> 00:05:33,780 [Eric] as something like OpenClaw. 00:05:34,820 --> 00:05:52,219 [Eric] Uh, and because it's an actual computer, it, you know, you can have host files on it and do all sorts of things, right? So way more powerful to actually have a computer. And then there's this other tool that's, that's as consumer, like a full consumer product, not necessarily like prosumer like a Zo- 00:05:52,220 --> 00:05:52,250 [John] Right 00:05:52,250 --> 00:05:53,380 [Eric] ... or a, you know- 00:05:53,380 --> 00:05:53,390 [John] Right 00:05:53,390 --> 00:05:55,950 [Eric] ... OpenClaw, that's called Poke. 00:05:55,950 --> 00:05:56,020 [John] Okay. 00:05:56,020 --> 00:05:57,660 [Eric] And they don't even have an interface. 00:05:57,660 --> 00:05:57,880 [John] Okay. 00:05:57,880 --> 00:06:01,060 [Eric] You just use iMessage, Telegram, whatever. You just use a messaging service. 00:06:01,060 --> 00:06:01,440 [John] Yeah. Right. 00:06:01,440 --> 00:06:03,010 [Eric] And that's the only interface. 00:06:03,010 --> 00:06:03,039 [John] Okay. 00:06:03,040 --> 00:06:05,690 [Eric] And it will just do things for you. So you just text this agent all day. 00:06:05,690 --> 00:06:06,140 [John] Yeah. Nice. 00:06:06,140 --> 00:06:07,320 [Eric] And it can do all these different things. 00:06:07,320 --> 00:06:07,500 [John] Uh-huh. 00:06:07,500 --> 00:06:08,680 [Eric] And you can connect stuff, right? 00:06:08,680 --> 00:06:08,820 [John] Yeah. 00:06:08,820 --> 00:06:11,760 [Eric] So they're sort of the same idea, right? So that's personal. 00:06:11,760 --> 00:06:12,560 [John] Yep. 00:06:12,560 --> 00:06:21,090 [Eric] Um, using things individually at work is-Um, on the advanced technical end of the spectrum, you're using Claude Code to like generate- 00:06:21,090 --> 00:06:21,099 [John] Yeah 00:06:21,099 --> 00:06:22,460 [Eric] ... code or do a project or whatever. 00:06:22,460 --> 00:06:23,439 [John] Right. 00:06:23,440 --> 00:06:29,060 [Eric] You know, and then you have something like Claude Desktop or GPT, you know, where you can generate artifacts. 00:06:29,060 --> 00:06:29,140 [John] Right. 00:06:29,140 --> 00:06:31,890 [Eric] You can like, you know, let it read physical files. 00:06:31,890 --> 00:06:31,920 [John] Right. 00:06:31,920 --> 00:06:35,710 [Eric] You can upload things, you can give it access to Google Drive, and it can help you with a spreadsheet, right? 00:06:35,710 --> 00:06:35,720 [John] Right. 00:06:35,720 --> 00:06:39,300 [Eric] And so that's just using it for productivity, right? 00:06:39,300 --> 00:06:39,340 [John] Right. 00:06:39,340 --> 00:06:50,130 [Eric] And then at work on a team. So this is an area where I don't think a lot of people outside of those who are working, you know, in AI- 00:06:50,130 --> 00:06:50,130 [John] Yeah 00:06:50,130 --> 00:06:53,100 [Eric] ... every day understand what the tool set even is. 00:06:53,100 --> 00:06:53,480 [John] Yeah. 00:06:53,480 --> 00:06:59,380 [Eric] So how do you even think about this? Because I think most people are familiar with, "I use AI personally at work." 00:06:59,380 --> 00:06:59,580 [John] Right. 00:07:01,100 --> 00:07:11,640 [John] So yeah, I think it would be helpful just to go personally. So personally, I've played around a fair amount with and, and been able to do some cool things with OpenClaw- 00:07:11,640 --> 00:07:11,830 [Eric] Mm-hmm 00:07:11,830 --> 00:07:13,140 [John] ... Obsidian- 00:07:13,140 --> 00:07:13,360 [Eric] Mm-hmm 00:07:13,360 --> 00:07:20,240 [John] ... which is a note app similar to Notion, and, um, [lip smacks] GPT-4 and now GPT-5. 00:07:20,240 --> 00:07:20,760 [Eric] Mm-hmm. 00:07:20,760 --> 00:07:34,580 [John] Like it's really cool. Um, and that is something that [chuckles] this is silly, but the most productive, like enjoyable thing that it does is on a schedule multiple times a day, 00:07:35,660 --> 00:07:40,060 [John] it filters out advertisements from my personal email. That's what it does. 00:07:42,000 --> 00:07:42,660 [Eric] [laughs] 00:07:42,660 --> 00:07:44,239 [John] And it's great. 00:07:44,239 --> 00:07:45,340 [Eric] [laughs] Yeah. 00:07:45,340 --> 00:07:47,140 [John] Like it's awesome. 00:07:49,500 --> 00:07:50,200 [Eric] [laughs] 00:07:50,320 --> 00:07:51,000 [John] And it, it, it saves- 00:07:51,000 --> 00:07:52,350 [Eric] Yeah, I have a lot of thoughts on that becoming a feature 00:07:52,350 --> 00:07:53,160 [John] ... and over like a month. 00:07:53,160 --> 00:07:53,640 [Eric] Yeah, yeah. 00:07:53,640 --> 00:07:54,690 [John] And over like a month. 00:07:54,690 --> 00:07:55,220 [Eric] Yeah, sure. 00:07:55,220 --> 00:07:57,720 [John] Like it's several hours of savings. 00:07:57,720 --> 00:07:58,400 [Eric] Mm-hmm. 00:07:58,400 --> 00:08:03,740 [John] And no email provider that gets revenue from ads will ever build that in. 00:08:03,740 --> 00:08:04,040 [Eric] Mm-hmm. 00:08:04,040 --> 00:08:05,900 [John] So it's a legitimate long-term use case- 00:08:05,900 --> 00:08:05,960 [Eric] Mm-hmm 00:08:05,960 --> 00:08:07,760 [John] ... unless the model changes for email. 00:08:07,760 --> 00:08:08,120 [Eric] Mm-hmm. 00:08:08,120 --> 00:08:09,850 [John] Right. So I mean- 00:08:09,850 --> 00:08:10,060 [Eric] Yeah 00:08:10,060 --> 00:08:12,620 [John] ... that, that, that is the like single most useful thing. 00:08:12,620 --> 00:08:13,960 [Eric] That's, yeah, that's a whole- 00:08:13,960 --> 00:08:14,150 [John] It's silly 00:08:14,150 --> 00:08:15,570 [Eric] ... episode that we should discuss. 00:08:15,570 --> 00:08:15,600 [John] Yeah. 00:08:15,600 --> 00:08:16,330 [Eric] But yeah. 00:08:16,330 --> 00:08:19,580 [John] Yeah. Anyways, and then, and then there's the, like there's, there's plenty of other like small things. 00:08:19,580 --> 00:08:20,060 [Eric] Mm-hmm. 00:08:20,060 --> 00:08:23,850 [John] But, but the use case and like to drill into like what's actually happening there, 00:08:24,899 --> 00:08:30,760 [John] it's like a really, um, highly refined, lots of iterations prompt- 00:08:30,760 --> 00:08:31,300 [Eric] Mm-hmm 00:08:31,300 --> 00:08:33,420 [John] ... that's run on a schedule multiple times a day- 00:08:33,420 --> 00:08:34,240 [Eric] Mm-hmm 00:08:34,240 --> 00:08:35,420 [John] ... which is really useful. 00:08:35,420 --> 00:08:35,880 [Eric] Mm-hmm. 00:08:35,880 --> 00:08:40,820 [John] And, and I slightly modify it over time. So like if I know- 00:08:40,820 --> 00:08:41,070 [Eric] Hmm 00:08:41,070 --> 00:08:50,620 [John] ... it... 'Cause there's general like i- it's like decent at like separating ads versus not, but it's not perfect. So, uh, so I'm, I've get, I've gotten better at working with it. So I'll tell it like, "Hey, 00:08:51,660 --> 00:08:52,830 [John] I'm working with 00:08:53,920 --> 00:08:59,460 [John] my buddy Blake on a thing. Like just don't... Like make sure that Blake's emails don't get like lost." 00:08:59,460 --> 00:09:00,090 [Eric] Yeah, yeah, yeah. 00:09:00,090 --> 00:09:01,220 [John] And like you can productively tell it. 00:09:01,220 --> 00:09:01,280 [Eric] Mm-hmm. 00:09:01,280 --> 00:09:06,320 [John] And, and then, and it's just like little things like that where it's like it's not gonna know if that's spam or not. Like it might- 00:09:06,320 --> 00:09:06,400 [Eric] Yeah 00:09:06,400 --> 00:09:06,760 [John] ... you know. 00:09:06,760 --> 00:09:07,260 [Eric] Sure. 00:09:07,260 --> 00:09:07,499 [John] Um- 00:09:07,500 --> 00:09:07,600 [Eric] Yeah 00:09:07,600 --> 00:09:12,430 [John] ... or I signed up for this new service. Like I know that like it might look like spam, but I c- I, I'm expecting it- 00:09:12,430 --> 00:09:12,430 [Eric] Right 00:09:12,430 --> 00:09:13,320 [John] ... that it's gonna email me. 00:09:13,320 --> 00:09:13,820 [Eric] Right. Right. 00:09:13,820 --> 00:09:14,680 [John] Or like actually use- 00:09:14,680 --> 00:09:15,740 [Eric] I do want these updates- 00:09:15,740 --> 00:09:16,480 [John] Yeah 00:09:16,480 --> 00:09:17,480 [Eric] ... on the product newsletter- 00:09:17,480 --> 00:09:17,510 [John] Or like a blog or something 00:09:17,510 --> 00:09:18,050 [Eric] ... or whatever. Yeah. 00:09:18,050 --> 00:09:18,540 [John] Yeah, exactly. 00:09:18,540 --> 00:09:19,000 [Eric] Yep. Yep. 00:09:19,000 --> 00:09:25,240 [John] So I think the important piece there is the, um, get it like 90% and then iterate with it over time. 00:09:25,240 --> 00:09:25,420 [Eric] Yep. 00:09:25,420 --> 00:09:33,580 [John] And that's like true for anything. And then the next important piece is what can I do personally that could happen every day that would be super valuable, and it just happens. 00:09:33,580 --> 00:09:34,240 [Eric] Mm-hmm. 00:09:34,240 --> 00:09:40,860 [John] Um, all right. So that's then work, like a couple things. Yeah. One, you have to layer on security. Um- 00:09:40,860 --> 00:09:41,580 [Eric] Right 00:09:41,580 --> 00:09:44,680 [John] ... two, for us, like kinda client preference. Um- 00:09:44,680 --> 00:09:44,960 [Eric] Yep 00:09:44,960 --> 00:09:49,420 [John] ... which, which we've talked about before as far as some people really wanna use this model versus the other. 00:09:49,420 --> 00:09:49,840 [Eric] Yep. 00:09:49,840 --> 00:09:57,420 [John] Um, but I think the, uh, but, but the, the motion is similar. You're j- like you said, you're just making spreadsheets or like- 00:09:57,420 --> 00:09:57,840 [Eric] Sure 00:09:57,840 --> 00:09:58,580 [John] ... whatever. 00:09:58,580 --> 00:09:58,720 [Eric] Yep. 00:09:58,720 --> 00:09:59,490 [John] Um, the teams- 00:09:59,490 --> 00:10:05,340 [Eric] You're just-- Yeah, yeah. You're leveraging AI to perform activities that you performed before, but you're doing them faster, more comprehensively- 00:10:05,340 --> 00:10:05,350 [John] Yeah 00:10:05,350 --> 00:10:06,140 [Eric] ... whatever it is. 00:10:06,140 --> 00:10:09,550 [John] Right. Right. Yeah. Faster and slightly worse, right? 00:10:09,550 --> 00:10:09,610 [Eric] [laughs] 00:10:09,610 --> 00:10:14,200 [John] Um, [laughs] but, but, but sometimes that's like it doesn't matter. 00:10:14,200 --> 00:10:15,880 [Eric] Yeah. Yeah. Yeah. 00:10:15,880 --> 00:10:15,890 [John] Um. 00:10:15,890 --> 00:10:15,890 [Eric] Yeah. 00:10:15,890 --> 00:10:15,920 [John] Yeah. 00:10:15,920 --> 00:10:16,080 [Eric] Yeah. 00:10:16,080 --> 00:10:16,190 [John] Okay. 00:10:16,190 --> 00:10:19,059 [Eric] But still, yeah, you're just augmenting your existing workflow. 00:10:19,060 --> 00:10:21,050 [John] Yeah. So the teams thing I think is the most interesting. 00:10:21,050 --> 00:10:21,060 [Eric] Yep. 00:10:21,060 --> 00:10:39,800 [John] 'Cause I think it's the part people have the least figured out. Um, so for me, like the way I'm thinking about teams is you have to move the work into a space the team already like uses, which so far is chat, which practically means Slack or Microsoft Teams- 00:10:39,800 --> 00:10:39,910 [Eric] Yes 00:10:39,910 --> 00:10:40,960 [John] ... depending on your organization. 00:10:40,960 --> 00:10:41,880 [Eric] Yep. 00:10:41,880 --> 00:10:48,239 [John] So you have to move the work there, especially if you're gonna get non-technical people involved. They're not gonna log into another thing, like- 00:10:48,240 --> 00:10:49,100 [Eric] Yep 00:10:49,100 --> 00:10:57,310 [John] ... like i- if maybe, and especially if they're not using like AI tools like much for productivity, you have to get into the, the chat interface. 00:10:57,310 --> 00:10:59,120 [Eric] Hmm. Right. 'Cause yeah, that's an adoption. 00:10:59,120 --> 00:11:04,100 [John] Yeah. So there's that, but then there's the like h- how do you use it in public because- 00:11:04,100 --> 00:11:05,040 [Eric] In public within a company 00:11:05,040 --> 00:11:11,660 [John] ... within a company. Yeah, yeah. Not... Yeah, yeah. How do you, how do you use it in public within a company? And the answer is, is channels or groups. 00:11:11,660 --> 00:11:12,679 [Eric] Mm-hmm. 00:11:12,680 --> 00:11:16,840 [John] Because that's the only way to show people in a like really low, 00:11:18,060 --> 00:11:26,300 [John] um, risk, I don't know, like a, a really easy way to get to do something 'cause people either don't know that it can do things or they like- 00:11:26,300 --> 00:11:26,520 [Eric] Hmm 00:11:26,520 --> 00:11:28,900 [John] ... feel like they don't know how or they're nervous or whatever. 00:11:28,900 --> 00:11:29,650 [Eric] Interesting. 00:11:29,650 --> 00:11:33,580 [John] Um, especially non-technical people, which I don't have that, [chuckles] that many in my organization- 00:11:33,580 --> 00:11:33,700 [Eric] Right. Right 00:11:33,700 --> 00:11:34,290 [John] ... non-technical people. 00:11:34,290 --> 00:11:35,000 [Eric] Right. 00:11:35,000 --> 00:11:42,700 [John] But that to me, I think at least for now, is like, is the right play. And, and natively in Claude, like 00:11:43,800 --> 00:11:50,179 [John] chats between other people in group chat like doesn't even really exist, and it kind of exists in ChatGPT, but like- 00:11:50,180 --> 00:11:50,920 [Eric] Mm-hmm 00:11:50,920 --> 00:11:52,340 [John] ... uh, kind of. 00:11:52,340 --> 00:11:53,110 [Eric] Yep. 00:11:53,110 --> 00:12:02,600 [John] There's, I, I've, I've done a couple. I don't know if you've ever done one. You can do group chats now with humans and AI and, and GPT, but at work I, you know. 00:12:02,600 --> 00:12:03,020 [Eric] Yeah. 00:12:03,020 --> 00:12:05,160 [John] Not, I don't think too many people use it for whatever reason. 00:12:05,160 --> 00:12:07,140 [Eric] Right. Right. Yeah. Yeah. So, 00:12:08,160 --> 00:12:15,540 [Eric] okay, that is really interesting because you're framing this in part as an adoption problem. 00:12:15,540 --> 00:12:16,280 [John] Right. 00:12:16,280 --> 00:12:24,460 [Eric] So even if we assume that you could run 20 agents overnight and have, you know, these digital employees like doing things on your behalf- 00:12:24,460 --> 00:12:25,180 [John] Right 00:12:27,250 --> 00:12:30,150 [Eric] You know, getting a company to adopt that is actually hard. 00:12:30,150 --> 00:12:39,070 [John] Well, yeah. It is, but it's not... But the reason the adoption problem is hard is, is that the, is the, like, value. Like, I, I don't know what it can do- 00:12:39,070 --> 00:12:39,090 [Eric] Mm-hmm 00:12:39,090 --> 00:12:40,730 [John] ... reliably. 00:12:40,730 --> 00:12:42,290 [Eric] Ah. Okay, so- 00:12:42,290 --> 00:12:47,270 [John] They can theoretically know, like, "Oh, I, I hear it can connect to things and do neat things," but, like, what can it actually do? 00:12:47,270 --> 00:12:47,430 [Eric] Right. 00:12:47,430 --> 00:12:47,830 [John] You know? 00:12:47,830 --> 00:12:50,630 [Eric] Right. Is that, would you describe that as a blank page problem? 00:12:52,570 --> 00:13:03,590 [John] No, it's not, like, completely blank. But it's like there, there's, like, enough filled in. It's like a Mad Lib problem, and the, and you have, like, a group of people that's, like, kind of good at Mad Lib, and a group that's kind of bad at Mad Lib. 00:13:03,590 --> 00:13:04,310 [Eric] Mm. 00:13:04,310 --> 00:13:05,130 [John] That's how I think about it. [chuckles] 00:13:05,130 --> 00:13:20,530 [Eric] Yeah, that's a great analogy. So what, you know, where is the, where's the point of diminishing return that you have found, right? So, uh, and I'll frame this, I'll frame this as a, a theoretical use case. 00:13:20,530 --> 00:13:20,590 [John] Right. 00:13:20,590 --> 00:13:22,580 [Eric] You, you probably have already tried to implement this. 00:13:22,580 --> 00:13:22,610 [John] Yeah. 00:13:22,610 --> 00:13:22,870 [Eric] Okay. 00:13:24,290 --> 00:13:25,470 [Eric] You try to build 00:13:26,610 --> 00:13:29,390 [Eric] a, a data analyst agent- 00:13:29,390 --> 00:13:29,530 [John] Yes 00:13:29,530 --> 00:13:30,950 [Eric] ... for a company. 00:13:30,950 --> 00:13:31,550 [John] Yes. 00:13:31,550 --> 00:13:37,930 [Eric] Okay? So this is a digital employee, autonomous agent, whatever term you wanna use. 00:13:38,950 --> 00:13:47,550 [Eric] But it is a machine that performs the job of what a typical analyst would do. 00:13:47,550 --> 00:13:47,640 [John] Right. 00:13:47,640 --> 00:13:58,940 [Eric] And so this can show up in many forms. Just a couple that, you know, come to mind out of the gate are, uh, you can have it, like, run a report on a schedule, right? 00:13:58,940 --> 00:13:58,970 [John] Yeah. 00:13:58,970 --> 00:14:00,940 [Eric] So I can get my weekly sales report. 00:14:00,940 --> 00:14:01,229 [John] Sure. 00:14:01,230 --> 00:14:01,320 [Eric] Right? 00:14:01,320 --> 00:14:02,230 [John] Right. 00:14:02,230 --> 00:14:07,410 [Eric] Um, you can, uh, ask it ad hoc questions. 00:14:07,410 --> 00:14:07,820 [John] Right. 00:14:07,820 --> 00:14:16,620 [Eric] Right? So, um, you know, website traffic is, looks like it's down this week. You know, what are the potential reasons that it's down? 00:14:16,620 --> 00:14:16,630 [John] Right. 00:14:16,630 --> 00:14:27,590 [Eric] Like, what are the variables that changed and impacted the, impacted, you know, the delta the most? Uh, you could have it perform, you know, analyses like forecasting or other things like that, right? 00:14:28,690 --> 00:14:29,450 [John] Yep. 00:14:29,450 --> 00:14:30,340 [Eric] Is that, 00:14:32,030 --> 00:14:35,930 [Eric] you know, the, again, based on sort of what David Cramer said in the post, 00:14:37,249 --> 00:14:48,010 [Eric] people who are, you know, people are saying this is... You know, and like the layoff news and all that sort of stuff, right? What do you think in your experience is the actual state of that? 00:14:48,010 --> 00:14:48,129 [John] Yeah. 00:14:48,130 --> 00:14:52,890 [Eric] Is there huge adoption? Is that providing value? Is it replacing a data analyst as an employee? 00:14:52,890 --> 00:14:53,030 [John] Yeah. 00:14:54,530 --> 00:14:56,390 [John] So I think two interesting things. One, 00:14:57,710 --> 00:15:08,610 [John] on the, uh, the data analyst thing, and we, we were talking about this earlier. We... Well, a lot of the experimentation we've done has been internal and like, "Hey, I wonder if we can make a data analyst." 00:15:08,610 --> 00:15:08,910 [Eric] Mm-hmm. 00:15:08,910 --> 00:15:11,390 [John] And it's like, "Does anybody have time to work on this?" "No." 00:15:11,390 --> 00:15:11,510 [Eric] [laughs] 00:15:11,510 --> 00:15:24,570 [John] So, so, like, we'll have, like, uh, like I think I did a demo of this, like, for a group that we're a part of. So, like, we'll, we'll go find the craziest thing. Like, like, there's these zero-employee companies that, like, there's two or three of them out there- 00:15:24,570 --> 00:15:24,630 [Eric] Mm-hmm 00:15:24,630 --> 00:15:26,730 [John] ... that are like, "We're a zero-employee company, all run by AI." 00:15:26,730 --> 00:15:26,740 [Eric] Mm-hmm. 00:15:26,740 --> 00:15:33,090 [John] And it's like, fine, we'll try that. And, like, give it an objective. Like, "Hey, we want this cool AI data analyst," and then give it, like, a bunch of tokens- 00:15:33,090 --> 00:15:33,250 [Eric] Mm-hmm 00:15:33,250 --> 00:15:35,090 [John] ... and be like, "Hey, build the thing." Um- 00:15:35,090 --> 00:15:35,610 [Eric] Yep 00:15:35,610 --> 00:15:36,070 [John] ... so I think that's one- 00:15:36,070 --> 00:15:36,750 [Eric] You've done this. 00:15:36,750 --> 00:15:37,490 [John] Yeah, yeah, we've done this. 00:15:37,490 --> 00:15:37,650 [Eric] Yeah. 00:15:37,650 --> 00:15:37,659 [John] Yeah. 00:15:37,659 --> 00:15:37,849 [Eric] Yep. 00:15:37,850 --> 00:16:02,690 [John] One extreme vector, and it, like, is exactly what you would expect. Well, I don't know what you would expect. Like, it kind of works. Like, [chuckles] it's like it, it, it like y- you learn some cool things. You're like, "Oh, that's interesting it did that." And then, like, you start it up and like, of course it doesn't work right the first time. And then, like, and then, and then, like, I... And I couldn't tell you, like, the exact, like, timeframe between that and, like, hardening to make it, like, something really good- 00:16:02,690 --> 00:16:02,830 [Eric] Right 00:16:02,830 --> 00:16:06,950 [John] ... or if I would just wanna start over, 'cause we haven't, like, quite done that. 00:16:06,950 --> 00:16:07,680 [Eric] Yeah. 00:16:07,680 --> 00:16:08,130 [John] Um- 00:16:08,130 --> 00:16:10,230 [Eric] Well, let me ask you two questions. Um, 00:16:12,130 --> 00:16:13,110 [Eric] two questions come to mind. 00:16:14,210 --> 00:16:17,870 [Eric] In the current state with what you've tried, is the juice worth the squeeze? 00:16:19,330 --> 00:16:20,490 [John] On which piece? 00:16:20,490 --> 00:16:20,790 [Eric] On, 00:16:22,050 --> 00:16:24,930 [Eric] you know, running a zero em- 00:16:24,930 --> 00:16:25,040 [John] Oh, yeah, yeah. 00:16:25,040 --> 00:16:26,950 [Eric] Let's say a zero-employee- 00:16:26,950 --> 00:16:27,110 [John] Right 00:16:27,110 --> 00:16:28,550 [Eric] ... data team or something. 00:16:28,550 --> 00:16:29,069 [John] Yeah. 00:16:29,130 --> 00:16:29,370 [Eric] Right? 00:16:29,370 --> 00:16:30,010 [John] The- 00:16:30,010 --> 00:16:33,050 [Eric] Is the juice worth the squeeze? Like, you're not deploying this to all of your clients. 00:16:33,050 --> 00:16:33,070 [John] No. 00:16:33,070 --> 00:16:40,710 [Eric] And it sounds like the way that you talk about it is you say it kind of works, which means that you're running it internally, but you're not, you know- 00:16:40,710 --> 00:16:40,830 [John] Right 00:16:40,830 --> 00:16:42,090 [Eric] ... doing this in production- 00:16:42,090 --> 00:16:42,099 [John] Right 00:16:42,099 --> 00:16:43,390 [Eric] ... on behalf of a client. 00:16:43,390 --> 00:16:51,490 [John] Here, here's how I think about it, and, and this is, this is, um, this is almost, this is a shortcoming of, like, the design of a lot of the tools. 00:16:51,490 --> 00:16:51,589 [Eric] Okay. 00:16:51,590 --> 00:16:53,550 [John] I don't think it's a fundamental model problem. 00:16:54,690 --> 00:17:05,250 [John] Um, and I've had, uh, the, I had a chance to, like, describe this to, like, somebody on my team recently. You know, so helpful for me to frame it, and I was like, all right, think about, like, all the tokens you spend in a week. 00:17:05,250 --> 00:17:05,770 [Eric] Mm-hmm. 00:17:05,770 --> 00:17:07,029 [John] What do you do with them? 00:17:07,030 --> 00:17:07,720 [Eric] Mm-hmm. 00:17:07,720 --> 00:17:11,210 [John] Like, let's categorize it. Writing code. All right. What percentage of the time? 00:17:11,210 --> 00:17:12,330 [Eric] Mm-hmm. 00:17:12,330 --> 00:17:13,210 [John] Research. 00:17:13,210 --> 00:17:13,230 [Eric] Mm-hmm. 00:17:13,230 --> 00:17:14,610 [John] What percentage of the time? Planning. 00:17:14,610 --> 00:17:14,690 [Eric] Mm-hmm. 00:17:14,690 --> 00:17:17,960 [John] What percentage of the time? QA and quality, what percentage of the time? 00:17:19,230 --> 00:17:32,769 [John] I think that is one of the current fundamental problems is the misallocation, especially for technical people, of way too high a budget for writing code, way too low a budget for planning, way too low a budget for research- 00:17:32,770 --> 00:17:32,850 [Eric] Hmm 00:17:32,850 --> 00:17:34,790 [John] ... way too low a budget for quality. 00:17:34,790 --> 00:17:35,090 [Eric] Interesting. 00:17:35,090 --> 00:17:46,370 [John] So if you, like, inverted that and you're like, "Hey, token maxer, you have a token budget that's unlimited, but every day or every week, here's your breakdown. You need to spend- 00:17:46,370 --> 00:17:46,680 [Eric] Yes 00:17:46,680 --> 00:17:48,050 [John] ... X amount of your budget on research- 00:17:48,050 --> 00:17:48,210 [Eric] Yep 00:17:48,210 --> 00:17:51,830 [John] ... X amount on planning, X amount on quality, and the remainder you can spend on coding." 00:17:51,830 --> 00:17:52,330 [Eric] Interesting. 00:17:52,330 --> 00:17:54,010 [John] I think that would fundamentally change- 00:17:54,010 --> 00:17:54,019 [Eric] Yeah 00:17:54,019 --> 00:17:56,110 [John] ... a lot of, like, how people- 00:17:56,110 --> 00:17:56,140 [Eric] Yeah, yeah 00:17:56,140 --> 00:17:56,810 [John] ... like, use. 00:17:56,810 --> 00:17:57,320 [Eric] Yeah. 00:17:57,320 --> 00:17:59,440 [John] And I, and I think the results would be better. 00:17:59,440 --> 00:18:02,880 [Eric] W- I guess what I'm getting at is 00:18:04,850 --> 00:18:06,380 [Eric] it, it sounds like, 00:18:07,950 --> 00:18:16,710 [Eric] it sounds like you have not reached a point where you're super excited about these autonomous digital data analysts 00:18:17,730 --> 00:18:24,080 [Eric] suddenly providing some unbelievable amount of value based on your implementations of them. 00:18:24,080 --> 00:18:24,089 [John] Right. 00:18:24,090 --> 00:18:29,430 [Eric] But you also view it as a problem that will be solved in the near future. 00:18:30,590 --> 00:18:37,330 [John] I think, I think a lot of the tooling and everything that's being pushed right now is, is very developer centric. 00:18:37,330 --> 00:18:38,309 [Eric] Mm-hmm. 00:18:38,310 --> 00:18:47,790 [John] And in the real world, like what we've developed over time is all these different roles, like ops, dev ops, security, you know, [chuckles] Q- QA at different points. I feel like QA- 00:18:47,790 --> 00:18:47,800 [Eric] Mm-hmm 00:18:47,800 --> 00:18:50,230 [John] ... comes and goes as far as like fads. 00:18:50,230 --> 00:18:50,890 [Eric] Yep. 00:18:50,890 --> 00:18:54,950 [John] Um, qual- like, and we're just really lopsided right now. 00:18:54,950 --> 00:18:55,050 [Eric] Mm-hmm. 00:18:55,050 --> 00:19:02,590 [John] So e- so when I go to use like, to try out like an autonomous, like orchestrated zero person company, 00:19:03,650 --> 00:19:06,310 [John] who made that? QA people, ops people- 00:19:06,310 --> 00:19:06,570 [Eric] Right 00:19:06,570 --> 00:19:08,470 [John] ... or devs? Devs, of course. 00:19:08,470 --> 00:19:08,830 [Eric] Yeah, yeah. 00:19:08,830 --> 00:19:10,390 [John] They're not... Or database people. They're not- 00:19:10,390 --> 00:19:10,790 [Eric] Right 00:19:10,790 --> 00:19:12,850 [John] ... d- they're not data people ma- Like, they're devs. 00:19:12,850 --> 00:19:13,450 [Eric] Yeah. 00:19:13,450 --> 00:19:27,350 [John] And like there's just a bent that's like built into the framework and the, and the, and the software of l- of, of if I want to emphasize token spend on like... And, and you could absolutely do it, but there's, there's like a pre-bent to building. 00:19:27,350 --> 00:19:27,770 [Eric] Mm-hmm. 00:19:27,770 --> 00:19:29,240 [John] Whereas like you need extra like 00:19:30,510 --> 00:19:36,870 [John] work and thought to like kinda force it to do like, oh, you need to do more planning, more QA, more research, like. And you can do it, 'cause most of these- 00:19:36,870 --> 00:19:36,880 [Eric] Yeah 00:19:36,880 --> 00:19:43,450 [John] ... frameworks are fairly, are fairly flexible. But if you just want out of the box, like do something cool, like it's got that dev bent to it. Like it's- 00:19:43,450 --> 00:19:43,460 [Eric] Yeah 00:19:43,460 --> 00:19:44,470 [John] ... hard to explain- 00:19:44,470 --> 00:19:44,870 [Eric] Yep 00:19:44,870 --> 00:19:45,810 [John] ... any other way. 00:19:45,810 --> 00:19:46,290 [Eric] Yeah. 00:19:46,290 --> 00:20:00,900 [John] And, and that would be the, the case, that would be the case regardless of AI. If there's some like new thing and, and it's like brand new and devs built it, and you don't have the other layers that, that would be in a commercial organization of quality and ops and all the other things. 00:20:00,900 --> 00:20:00,909 [Eric] Mm-hmm. 00:20:00,910 --> 00:20:05,090 [John] Like, I think you'd run into the same problems, but AI like exaggerates the problem. 00:20:05,090 --> 00:20:12,430 [Eric] Hmm. It's super interesting. One of the other things that comes to mind is the challenge around... 00:20:13,750 --> 00:20:16,070 [Eric] We talked about the, you know, sort of the blank page problem- 00:20:16,070 --> 00:20:16,180 [John] Yeah 00:20:16,180 --> 00:20:17,590 [Eric] ... or the Mad Lib problem as you framed it- 00:20:17,590 --> 00:20:17,600 [John] Right, right 00:20:17,600 --> 00:20:18,370 [Eric] ... which I think is great. 00:20:19,530 --> 00:20:23,530 [Eric] But I think the other difficulty is that the... 00:20:24,550 --> 00:20:26,450 [Eric] If you think about a data analyst within a company, 00:20:28,230 --> 00:20:28,590 [Eric] um, 00:20:29,630 --> 00:20:35,890 [Eric] you know, let's say you build a... You know, we've seen these things actually be successful where you have a little data agent in Slack and you can- 00:20:35,890 --> 00:20:35,940 [John] Mm-hmm 00:20:35,940 --> 00:20:39,050 [Eric] ... or Microsoft Teams, and you can ask it questions, and that's super helpful. 00:20:39,050 --> 00:20:39,110 [John] Yeah. 00:20:39,110 --> 00:20:41,270 [Eric] It's like, it's very, you know, 00:20:42,290 --> 00:20:43,960 [Eric] what's the daily active user trend- 00:20:43,960 --> 00:20:43,960 [John] Right 00:20:43,960 --> 00:20:45,460 [Eric] ... for the last three months or something, right? 00:20:45,460 --> 00:20:45,460 [John] Right. 00:20:45,460 --> 00:20:46,730 [Eric] And it's just like amazing- 00:20:46,730 --> 00:20:46,820 [John] Yeah 00:20:46,820 --> 00:21:04,690 [Eric] ... to get those answers, you know, on demand. But another thing that I've noticed is that, you know, people will say, "Well, you know, those sorts of things are not necessarily accurate all the time." But I think it's because they are set up 00:21:05,990 --> 00:21:12,850 [Eric] even with a semantic layer and really clear definitions that only covers a certain amount of surface area- 00:21:12,850 --> 00:21:12,950 [John] Right 00:21:12,950 --> 00:21:14,750 [Eric] ... on the data- 00:21:14,750 --> 00:21:14,990 [John] Yeah 00:21:14,990 --> 00:21:22,130 [Eric] ... that's, you know, o- of the data and the questions that can be asked. But because it's op- an open-ended Mad Libs problem- 00:21:22,130 --> 00:21:22,950 [John] Right 00:21:22,950 --> 00:21:28,930 [Eric] ... people can ask questions, and the agent will try to answer those questions- 00:21:28,930 --> 00:21:29,330 [John] Yeah 00:21:29,330 --> 00:21:30,270 [Eric] ... by, you know, by default- 00:21:30,270 --> 00:21:30,280 [John] Yeah 00:21:30,280 --> 00:21:31,650 [Eric] ... it tries to answer the questions. 00:21:31,650 --> 00:21:32,050 [John] Yeah. 00:21:32,050 --> 00:21:44,050 [Eric] And so I think that's another challenge where you do have to, even though agents can, you know, technically reason autonomously, you actually do have to bake the guardrails in 00:21:45,590 --> 00:21:50,550 [Eric] to control for the fact that they will try to answer by default, even if they don't have- 00:21:50,550 --> 00:21:51,390 [John] Right 00:21:51,390 --> 00:21:55,330 [Eric] ... um, you know, the underlying context that they need to answer the question. 00:21:55,330 --> 00:21:56,090 [John] Right. 00:21:56,090 --> 00:22:03,030 [Eric] Which is challenging, and I think that's another limitation of, you know, whether it's an, a coding agent, it will try to build the feature, right? A data analyst- 00:22:03,030 --> 00:22:03,040 [John] Right 00:22:03,040 --> 00:22:08,610 [Eric] ... will try to answer the question. Um, it's actually a lot of work to hard code the guardrails in that- 00:22:08,610 --> 00:22:08,910 [John] Right 00:22:08,910 --> 00:22:12,090 [Eric] ... that keep it from doing things that it shouldn't do. 00:22:12,090 --> 00:22:12,430 [John] Right. 00:22:12,430 --> 00:22:13,150 [Eric] Right? 00:22:13,150 --> 00:22:21,630 [John] Yeah. Well, so if we talk about data analyst, I'm very bullish, and I think we're there on the data part, but not on the analyst part. 00:22:21,630 --> 00:22:21,750 [Eric] Yes. 00:22:21,750 --> 00:22:30,550 [John] So if you have a really good analyst that like basically just needs text to SQL, like we're there. It works great. Um, and they ha- if they have domain knowledge- 00:22:30,550 --> 00:22:30,560 [Eric] Mm-hmm 00:22:30,560 --> 00:22:32,760 [John] ... and understanding of the data, and they just- 00:22:32,760 --> 00:22:32,760 [Eric] Yes 00:22:32,760 --> 00:22:37,090 [John] ... like, they wouldn't be able to write SQL very fast or maybe at all, and like 00:22:38,170 --> 00:22:42,070 [John] they just have like some intuition, like that works great like today. 00:22:42,070 --> 00:22:51,769 [Eric] But that's a fundamentally... That is fundamentally different because that is augmenting an existing human role- 00:22:51,769 --> 00:22:52,269 [John] Yep 00:22:52,269 --> 00:22:53,820 [Eric] ... which is fundamentally different than a- 00:22:53,820 --> 00:22:53,860 [John] Mm-hmm 00:22:53,860 --> 00:22:55,450 [Eric] ... zero person company or a- 00:22:55,450 --> 00:22:55,460 [John] Yeah 00:22:55,460 --> 00:22:55,680 [Eric] ... or an autonomous employee. 00:22:55,680 --> 00:22:57,090 [John] Or, or even an AI employee. Yeah. 00:22:57,090 --> 00:22:57,870 [Eric] Yep, yep. 00:22:57,870 --> 00:23:07,930 [John] So let, let me end on this 'cause as far as like where I think we're going, one, I think autonomous employee like skips step. I mean, um, autonomous, like zero person company [chuckles] skips a bunch of steps. 00:23:07,930 --> 00:23:08,450 [Eric] Yes. 00:23:08,450 --> 00:23:25,950 [John] The, the thing that's most interesting me- to me right now is the, if you've heard of like co-digital twin, like basically working on a, a, a digital, a digital employee is what people are calling it, but, but it's like who's responsible for the employee? 00:23:25,950 --> 00:23:26,030 [Eric] Yep. 00:23:26,030 --> 00:23:41,490 [John] I think the best model right now is one-to-one and saying like, "All right. I'm gonna like be constructing this thing," and like all of my... And there's like basic framework stuff that like, like is gonna be solved, it's like already being solved by Model, by GPT and Anthropic and all of these. 00:23:41,490 --> 00:23:41,510 [Eric] Mm-hmm. 00:23:41,510 --> 00:23:49,710 [John] And then like all the effort goes into the guardrails and the prompting and the like baking in the knowledge, so the interactions with it are like easy. 00:23:49,710 --> 00:23:50,110 [Eric] Yes. 00:23:50,110 --> 00:24:01,450 [John] Like that, that to me is the next step. And if it says something dumb, like it's my co. Like co- co is the mo- like is the term a lot of people are using. It's my co, and I'm like responsible for fixing it. 00:24:01,450 --> 00:24:01,490 [Eric] Mm-hmm. 00:24:01,490 --> 00:24:03,020 [John] That to me is the most compelling model right now. 00:24:03,020 --> 00:24:05,100 [Eric] Interesting. Yeah, yeah. Yeah, yeah. 00:24:05,100 --> 00:24:08,610 [John] And, and it, and it has permission to do some things on my behalf, like I have to approve important things. 00:24:08,610 --> 00:24:08,770 [Eric] Mm-hmm. 00:24:08,770 --> 00:24:12,110 [John] Like that, that is most interesting to me right now. 00:24:12,110 --> 00:24:13,450 [Eric] Yeah, yeah. All right. 00:24:14,470 --> 00:24:15,930 [Eric] So I'm gonna summarize this. 00:24:15,930 --> 00:24:16,210 [John] All right. 00:24:17,510 --> 00:24:22,790 [Eric] We're not there yet. [both chuckling] But we are getting there pretty quickly. 00:24:22,790 --> 00:24:23,830 [John] Yeah. 00:24:23,830 --> 00:24:24,250 [Eric] Okay. 00:24:24,250 --> 00:24:25,090 [John] I think that's fair. 00:24:25,090 --> 00:24:26,770 [Eric] All right. AI employees. 00:24:26,770 --> 00:24:28,210 [John] Are we there yet? [both laughing] 00:24:28,210 --> 00:24:29,310 [Eric] Are we there yet? 00:24:29,310 --> 00:24:31,820 [John] Having young kids, that's a, that's a common question. 00:24:31,820 --> 00:24:33,610 [Eric] Yes, exactly. All right. Well, thanks for joining- 00:24:33,610 --> 00:24:33,700 [John] Yep 00:24:33,700 --> 00:24:44,070 [Eric] ... the Token Intelligence Show, and we'll catch you on the next one. [upbeat music]