Summary Sergey Brin on Gemini 1.5 Pro (AGI House; March 2, 2024) (Youtube) youtu.be
4,963 words - YouTube video - View YouTube video
Speaker 0 And they have to talk to you. That's huge. That's pretty exciting times. Well, thank you all for coming first. So thanks so much for, giving Jim and I go.
Speaker 0 What should I say? We actually have people who know what they're talking about. I think Similar to that. Okay. Okay.
Speaker 0 I was worried I would have to say something that, I'm not quite up to speed on. I'll just quickly say, look, it's very exciting times, this model that, I think we're playing with 1.5 Pro. We internally call it Goldfish. It's a little secret. I don't actually oh, I know why.
Speaker 0 It's because Goldfish have very short memories. It's kind of an ironic thing. But when we were training this model, we didn't expect it to have to come out nearly as powerful as it did or to have all the capabilities that it does. In fact, it was just part of the scaling ladder experiment. But when we saw what it could do, we thought, hey.
Speaker 0 We don't wanna wait. We want the world to try it out, and I'm grateful that all of you here are here to give it a go. What else do they say? Let me introduce it to somebody else. What's happening next?
Speaker 1 I think people will probably have a lot of questions.
Speaker 0 Oh, okay. Okay. Quick questions. I'm You know, we definitely messed up on the image generation, and I think it was mostly due to just, like, not thorough testing. And, it definitely, for good reasons, upset a lot of people, on the images as you might have seen.
Speaker 0 I think the the images prompted a lot of people to really deeply test the base text models. And the text models, have 2 separate effects. 1, you know, 1 thing is quite honestly, if you deeply test any text model out there, whether it's ours, rock, what have you, it'll say, you know, some pretty weird things that are out there that, you know, definitely feel far left, for example. And kind of any model, if you try hard enough, could be prompted to in that regime. But, also, just to be fair, there's definitely work in that model.
Speaker 0 So once again, we haven't fully understood, why it means left in many cases, and that's not our intention. But if you try it starting, over this last week, it should be at least, 80% better of the test cases that we've covered. So I'd like all of you to try it. This should be a big effect, to bother your time, the Gemini 1.5 pro, which isn't in the sort of public facing app, the thing we used to call BART, shouldn't have much of that effect except for that general effect that if you sort of red team any AI model, you're gonna get weird corner cases. But we're not, even though this 1 hasn't been sort of thoroughly tested that way, we don't expect it to have, strong particularly.
Speaker 0 I suppose you can give it a go, But we're more excited today to try to prolong context and some of the technical features. A video chat. You probably wanna call it that. But, no. I mean, multimodal, both in and out, It's very exciting with video, audio.
Speaker 0 And we run early experiments, and, it is I mean, it's an exciting field. Even the little do you guys remember the duck video that kinda got us in trouble? But to be fair, it was fully displayed within the video. It wasn't real time. But, but that that is something that we have actually done is fed up images and, you know, in, like, frame by frame doo doo doo doo doo, how to talk about it.
Speaker 0 So, yeah, that's super exciting. I don't, I don't think we have anything, like, real time to present, right now today. Yeah.
Speaker 1 Are you personally writing code for some projects?
Speaker 0 I have been actually writing code, to be perfectly honest. It's not like that you would be very impressed by. But yeah, every once in a while, I'm still, like, kind of debugging or just trying to understand for myself, how a model works or, you know, to just analyze performance in a slightly different area or something like that. But little bits and pieces that make me feel connected. It's once again, I don't think you would be very technically impressed by it.
Speaker 0 But it's nice to it's nice to be able to play that. And sometimes, I'll use the AI bots to write code for me, because I'm rusty and I they actually do a pretty good job. So I'm very pleased with that. Okay. Question.
Speaker 2 We'll be back. Yes. Oh. 1 Juan, hook that first. Yeah.
Speaker 2 Okay. So pre AI simulation so pre AI, the closest thing
Speaker 1 we got to simulators was game engines. What do you think the new advances in the field mean for us to create better games or game engines in general? Do you have a view on
Speaker 0 Sorry. I wasn't, like, a sigh because of of disapproval or anything. I think I mean, I, what can I say about game engines? I think, obviously, like, on the graphics, you can do new and interesting things with game engines. But I think maybe the more interesting is the interaction with I guess I guess these days, you know, you can call people or land MPCs or whatever.
Speaker 0 But in the future, maybe MPCs will be actually very powerful and interesting. Yeah. So I think that's a really rich possibility. Probably not not enough of a gamer to think through all the possible futures with AI, but, and it opens up many hospitals. Yeah.
Speaker 0 What kind of, like, applications are you most excited about
Speaker 1 for people, like, building on on Gemini?
Speaker 0 Yeah. What kind of application level was decided about? I mean, I I think just ingesting, right now or for the version we're trying to, you know, you know, 1.5 pro, the warm context is something we're really experimenting with. And whether you dump a ton of code there or video, I mean, I've just seen people do I I don't think the model could do this, to be perfectly honest. But but people, like, dump in their code, and do a video of the app, and say, here's the bug, and the model will figure out where the bug is in the code, which is kind of mind blowing, but that works at all.
Speaker 0 I honestly don't really understand how the model does that. But I'm not saying you should do exactly that thing. But, yeah, experimenting with things that really require the long context. We have the service to support all these people here banking on it. So
Speaker 1 We we have people who are on the service here as well.
Speaker 0 Okay. Yeah. My phone is buzzing. Everybody's really stressed out. It's customer Cooper.
Speaker 0 Because, you know, the million context queries do take a bit of computer time, but you should go for it. Yeah.
Speaker 1 You mentioned a few times that you're not sure how this model works or you you weren't sure that this could do the things that it does. Do you think we can reach a point where we actually understand how these models work, or will they remain black boxes that we just trust to make us a model to not mess up?
Speaker 0 No. I I I think you can learn to understand it. I mean, you know, the fact is that when, we train these things, they're a thousand different capabilities you could try out. So on the 1 hand, it's very surprising that it can do it. On the other hand, if it's any particular, what capability you couldn't go back.
Speaker 0 And, you know, we can look at, where the attention is going at each layer between, like, the code and the video. And, you know, we can't deeply analyze it. I've personally thought that I don't know how far along the researchers have gone to doing that kind of thing. But, you know, it takes a huge amount of time and study to really slice apart on why a model is able to do some things. And, honestly, most of the time that I see slicing, it's, like, why it's not doing something.
Speaker 0 So I guess I would say it's it's mostly because I I think we could understand it, and people probably are, but most of the effort is spent figuring out where it goes wrong, not where it goes wrong.
Speaker 1 Yeah. So in computer science, there's this concept of reflective programming. So, like, a program can look at its own source code, maybe modify its own source code. And then in AGI literature, there's, like, recursive self improvements. So what are your thoughts on the implications of extremely long context windows and the language model being able to modify its own prompts?
Speaker 1 And what that has to do with, like, autonomy and building towards AGI potentially?
Speaker 0 Yeah. I think it's very exciting to, you know, to have these things actually prove themselves. I remember when I was, I think in grad school, I wrote this game where, like, it was, like, a wall maze you're flying through, or you shot the walls. The walls corresponded to bits of memory, and we just, like, flip those bits. And the goal is to crash it as quickly as possible, which doesn't really answer your question.
Speaker 0 But that was an example of self modifying code. I guess not for a particularly useful purpose, but I have people, you know, play that until the computer crashed. Anyhow, on your positive example, I see today people just using a 12 vector. I think, you know, open loop could have worked for certain, I think for certain, very limited domains today. Like, if you without the human intervention to guide it, I bet it could actually do some kind of continued improvement.
Speaker 0 But I don't think we're quite at the stage where for I don't know, real serious things. And first of all, the known context is not actually enough for a bit code basis, to to to turn on the entire code base. But you could do, like, retrieval and then on patient editing. I guess I haven't personally played it enough. But I I haven't seen it be at the stage today where a complex sort of piece of code will just iteratively improve itself.
Speaker 0 But, but but it's it's a great tool. And like I said, with human assistance, we for sure do I mean, like, I will use Gemini to, like, try to do something with Gemini code even today. But not very open loop deep sophisticated things, I guess. I I
Speaker 3 I'm trying.
Speaker 1 I'm sick.
Speaker 0 Let me get somebody in the back just because of yes. Yeah. Cool. So you first and then, the lady who
Speaker 1 Thank you. So I'm curious. What's your take on some ultimate decision or plan
Speaker 0 at least to erase $7,000,000,000,000? Right? Just, because, you know, look, I saw the headline. I didn't get too deep into that. I see there was sort of a provocative pipeline or statement or something.
Speaker 0 I don't know. I don't know. I he hasn't asked me for some $1,000,000,000,000. I think it was, it was meant for, like, chip development or something like that. Right?
Speaker 0 Yeah. I don't I I don't get I'm not an expert in chip development, but I don't get the sense that it's just something you can, like, sort of pour money, like, even huge amounts of money in how to make chips. I'm not an expert, but I could go. Let's see. Let me try somebody reined up back.
Speaker 0 Is there okay. Yes. So oh, 0, the training cost of balls are super high. Yeah. The training costs are definitely high.
Speaker 0 And, you know, that's something companies like us have to cope with. But I think, you know, the long term utility is incomparably higher. Like, if you kind of measure it on a human productivity level, you know, if it saves somebody an hour of work over the course of the week, you know, that hour's worth a lot. There are a lot of people using these things. I will be using them.
Speaker 0 But you do it's a big bet on the future. Model training on device. Model training on device. Oh, model running on device. Yeah.
Speaker 0 Model running on device. We've shipped it to, I think, Android, Chrome, and, yeah. Our Pixel phones. I think even Chrome runs a pretty decent model these days. We just open sourced Gemma, which was pretty small, a couple of different parameters.
Speaker 0 I can't remember what Yeah. 7. 7. Yeah. Yeah.
Speaker 0 I mean, that's really useful. You know, it could be low latency. You're not dependent on connectivity. And, the small models can call bigger models in the cloud too. So, an IP com device is a really good idea.
Speaker 0 Yeah. It's Which industries do I think have a big opportunity? I think that it's just, like, very hard to predict. I mean, there's sort of the obvious industries that people think of, sort of customer service, kind of just, like, you know, analyzing, I don't know, like, different lengthy documents and kind of the workflow of automation, I guess. Those are obvious, but, I think they're gonna be non obvious ones, which I can't predict, especially as you look at these certain multimodal models and the surprising capabilities that they have.
Speaker 0 I mean, I mean, that's why we have all of you here. You guys want the creative ones to figure that out? Okay. You, sir.
Speaker 1 Oh, my name is Alex. And now it seems that g and I is another thing that really works. Thank you so much for this.
Speaker 0 I like that to hear it. Thank you. It seems like
Speaker 1 just planning to raise prices at some point? Or, you know,
Speaker 0 Well, not I I I'm actually not on top of a pricing claim. I don't expect that we will raise prices, however, because, I mean, there are fundamentally a couple of trends. 1 is just that these, you know, there's just optimizations and things around their friends, but they're constantly, like, all the time. So it's like, this 10% idea. This 20% idea.
Speaker 0 And, like, month after month, that adds up. I think our GPUs are actually pretty damn good, at, inferencing, not the the same GPUs. But, but for certain inference workloads, they're just configured really nicely. And the other big effect is actually we're able to make smaller models more and more effective just with new generations, just whatever architectural changes, training changes, all kinds of things like that. So the models are getting more powerful even at the same kind of size.
Speaker 0 So I would not expect prices to go. Yes, ma'am. Oh, AI, health care, and biotech. Well, I I think there are a couple very, you know, different ways. You know, on the biotech side, people look at, things like, alpha fold and things like that, just like understanding the fundamental mechanics of life.
Speaker 0 And I think you'll see AI do more and more of that, whether it's actual physical molecule and popping things or reading and summarize external articles, things like that. I also think for patients and this is kind of a tough area, honestly, because we're definitely not prepared for our just AI is like, go ahead, ask it any question. Like, we're not you know, AI is making mistakes and same things like that. But I think there is a future when you if you can overcome those kinds of issues where an AI can much more deeply spend time on the individual person and their history and all their scans, maybe mediated by a doctor or something. But, actually, it could be just better diagnosis, better recommendations, things like that.
Speaker 3 And, are you focusing on any, other non transformer architecture for, like, reasoning, planning, or any of to get to get better at
Speaker 0 the Okay. Are you focusing on any non transformer architectures? I mean, I think there's, like, so many sort of, variations, but I guess most people are they were still kind of transformer based. I mean, I'm sure somebody in the company here is speak to a bit more to it, would be looking. But, yeah, as much progress as transformers have made over the last, 6, 7, 7, 8 years, I guess, There's, you know, there's nothing to say.
Speaker 0 There's not gonna be some new revolutionary, architecture. And it's also possible that just, you know, incremental changes, for example, sparsity and things like that, that are still kind of the same transformer, also bring our oceans. So I don't know. I don't know about your cancer.
Speaker 3 But but is there some bottleneck for, like, reasoning kind of questions?
Speaker 0 So it's
Speaker 3 a bottleneck. But using this
Speaker 0 like transformers? Yeah. I mean, there's been lots of theoretical work showing the limitations of transformers. You know, we can't do this kind of thing, this many layers and things like that. I I I don't know how to extrapolate that to, like, contemporary transformers that usually don't meet the assumptions of the theoretical works.
Speaker 0 So it may not apply, but, I probably hedge my thoughts and try other architecture as well as being cool. Thank you. More than willing to do. No. No.
Speaker 0 But I I I feel like I made some bad decisions. Yeah. It was for sure early and early in 2 senses of the word. Maybe early in the overall evolution of technology, but also, I think I, like, in hindsight, I tried to push it as a product within itself. It was sort of more of a prototype, and I should've set those expectations around it.
Speaker 0 And I just didn't know much about sort of consumer hardware supply chains back then. Anyway, a bunch of things I wish I'd done differently. But I personally am still a fan of kind of the lightweight kind of minimal display that that offered, and you could just, like, wear LA versus the big heavy things that we have today. That's my personal preference. But the the Apple Vision and the Aquazzes for the matter, they're very impressive, like, having played with them.
Speaker 0 I mean, I'm just depressed with the way you can have a photo of your screen. That was what I was personally going for Back then. Yes, ma'am.
Speaker 1 So do you see, Gemini expanding capabilities into, like,
Speaker 0 Wow. It's a good question. To be honest, I haven't thought about it. But now that you say it, yeah, there's no reason sort of 3 like, it's kind of another mode, you know, 3 d data. So probably something interesting would happen.
Speaker 0 I mean, I I don't see why you would've, try to put that into a model that's already up. Got all the smarts with the text model, and I can turn on something else too. And by the way, maybe somebody's doing good at John and I don't know. It's probably what oh, yes. Because it's because I thought
Speaker 1 to be to it or I forgot about it. It doesn't mean it's not happening.
Speaker 0 Are you optimistic that we'll be able to bring in, text generating models' ability to hallucinate? And what do you think about the ethical issue of potentially spreading Big, problem right now. No question about it. I we have made them lose, like, less and less over time. But I would definitely be excited to see a breakthrough that brings it to near 0.
Speaker 0 I don't know. You know? That's not you can't just, like, count on breakthroughs. So I think we're gonna keep going with the incremental kinds of things that we do to just, like, bring all the loose station down, down, down over time. Like I said, I think breakthrough would be good.
Speaker 0 Misinformation, you know, misinformation is a complicated issue, I think. I mean, obviously, you don't want your AI bots to be just making stuff up. But they can also be kind of tricked into But there's a lot of, I guess, complicated, political issues in terms of what people consider what different people consider misinformation versus not. And it gets into kind of a broad social debate. I suppose none of them you could consider us about them sort of deliberately generating this information on behalf of another actor.
Speaker 0 From that point of view, I mean, unfortunately, it's like it's very easy to make a lousy AI, like, 1 that hallucinates a lot. And you can take, you know, any open source text model and probably tweak it to generate misinformation of all kinds. And if you're not concerned about, you know, the accuracy, it's just, like, kind of an easy thing to do. So I don't know. I I guess now I think about it.
Speaker 0 Detecting AI generated content is an important field and something that we work on and so forth. So you you just can maybe tell if something coming out of you was AI generated.
Speaker 1 Yeah. Hi, Alexandra. So the CEO of NVIDIA said that, basically, the future of writing code as a career is dead.
Speaker 0 Okay. Yeah. I thought that's. I mean, that's, like, we don't know where the future of AI is going broadly. I would you know, we don't know.
Speaker 0 You know, it seems to help across a range of many careers, whether it's graphic artists or customer support or doctors or or executives or, you know, what have you. I mean so I don't know that I would be, like, singling out, programming in particular. It's it's actually probably 1 of the more challenging tasks for an LLO today. But if you're talking about for, you know, decades in the future, what should you be kind of preparing for and so forth? I mean, it's it's hard to say.
Speaker 0 I mean, the AI could get quite good at programming. But you could say about kind of any field of human endeavor. So I guess I probably wouldn't have singled that out as, like, say, don't study specifically programming. I don't know if that's you know, I'm gonna answer it. Okay.
Speaker 0 Hand in the back.
Speaker 1 Checked for certain issues, or you could argue that, like, we're get better at writing test suites which cover all the cases. What are your opinions on this? Like,
Speaker 0 Oh, wow. You guys are all trying to choose careers, basically. But I I do think that, you know, using an AI today, right, let's say, unit tests is pretty straightforward. Yeah. Like, that's the kind of thing the AI does really quite well.
Speaker 0 So I guess my hope is that AI will make code more secure, not less secure. I mean, it's kind of it's usually and security is, to some extent, the effect of people being lazy. And they will think that AI is kind of put out as, you know, not being lazy. So if I had to bet, I would say there's probably enough benefit of the security with AI. But I wouldn't discourage you from pursuing a career ID security based company.
Speaker 1 Hey. Hey. Do you wanna build AGI?
Speaker 0 Do I want
Speaker 1 to build?
Speaker 0 Yeah. Yeah. Yeah. I mean, I think that's, yeah, different people need different things about that. But, to me, the reasoning aspects are really exciting and, amazing.
Speaker 0 And, you know, I kinda came out of retirement just because of which vector they have. It was so exciting. And as computer scientists, just seeing what these models can do year after year is astonishing. So yes.
Speaker 3 Any efforts on, like, humanoid robotics or, these because there was so much progress in Google X, like, in 20 15, 16.
Speaker 0 Oh, humanoid robotics. Boy, we've done a lot of humanoid, robotics over the years and sort of acquired and sold a bunch of companies for humanoid robotics. And now there are obsolete sorry. There are quite a few companies doing humanoid robotics. And internally, we sell groups that work on robotics in varying, varying forms.
Speaker 0 So what are my thoughts about that? I don't know if you know what? In general, I worked on x prior to this sort of new AI way of it. But their focus was more hardware projects, for sure. But, honestly, I guess I found the the hardware is not intended.
Speaker 0 Hardware is much more difficult, kind of on a technical basis, on business basis, in every way. So I'm not discouraging people from doing it. We need people, for sure, to do it. At the same time, while the software and the AIs are getting so much faster at such a high rate, I guess, to me, that feels like that's kind of the rocket ship. Mhmm.
Speaker 0 And I feel like if I get distracted in a way by making hardware for today's AI's, that might not be the best of use at times compared to, oh, what is the next kind of level of AI could be able to support? And for that matter, will it design a robot for me? That's my personal. There are a bunch of people at, Google now that who could work on hardware. Yes?
Speaker 0 Thanks. Yeah. The question about advertising? Yeah. I, love all people.
Speaker 0 I'm not too terribly concerned about business model shifts. I mean, I think it's a little bit I think it's wonderful that we've been now for 25 years or whatever, able to give just world class information, search, for free to everyone, and that's supported by advertising, which in my mind is great. It's great for the world. You know? Well, you know, a kid in Africa has just as much access to basic information as the president of the United States or what have you.
Speaker 0 So that's good. At the same time, I expect business models are going to evolve over time. And, yeah, maybe those will be advertising because whatever the advertising kinda works better, the AI is able to tailor it better or you like it. But even if it happens to move to you know, now we have, Gemini Advanced, other companies that have, you know, paid models. I think the fundamental issue is that you're delivering a huge amount of value, you know, displacing all of the mental effort that would have been, you know, required to take the place of that AI, or the airtime or labor or what have you is enormous.
Speaker 0 And the same thing was true in search. So I personally feel as long as there's huge value being generated, we'll figure out business models.
Speaker 1 Yeah. Which is a Chrome's 3rd party cookie application get Google ID advantage in, like, trading
Speaker 0 Well, it's a super exciting time, for search because your ability to answer questions the way I it's just so much greater. I think it's the bigger opportunity is in situations where you are, recall with it, or so. Like, you might ask a very specialized question or it's related to your own personal situation in a way that nobody out there, you know, on the Internet has already written up. And for the questions that the only people have written about already and thought deeply about, it's probably not a big deal. But the things that are very specific to what you might care about right now in a particular way, that's a huge opportunity.
Speaker 0 And, you know, you can imagine all kinds of products in your eyes with different ways to deliver that, but basically AI is an enabler just doing a much better job in that case. Okay. Last question. Okay. Who's gonna get the last question?
Speaker 0 Is it a
Speaker 1 better 1? Who's got a good 1 in the back?
Speaker 0 Look. I I I'm probably not as well versed as all of you are, but, I've definitely seen the, kind of the the molecular AI make huge amounts of progress. You could imagine that there would also be a lot of progress, maybe I haven't seen yet, on the epidemiology side of things to just be able to get kind of, I don't know, more honest, better control of kind of broader understanding of what's happening to people's health around the world. But, yeah, what kind of I don't know if you could could answer on the last 1. I don't know.
Speaker 0 I don't have, like, a really brilliant immortality piece of AI just like that. But, you know, it's the kind of field that for sure benefits from AI, whether you're a researcher or, like, you know, I want it to just summarize articles to me. That's 1. But in the future, you know, I would expect the AI would actually give you novel hypotheses to test. It does that today in the alphabets of the world, but maybe in more complex systems than just molecules.
Speaker 0 Okay.
Speaker 1 David, thank you. Thank you.
Speaker 2 Yeah. I think, really powerful to have