OpenAI New Patents, Gavin Uberti and the Path to More Intelligence
Added 2024-02-02 13:48:04 +0000 UTCExploring new patents from OpenAI that seemingly no other outlet else spotted, plus an exclusive interview with Gavin Uberti, CEO of Etched AI, on how verification and this kind of mass sampling could push models to the state of the art in intelligence.
Gavin Uberti - Etched.AI (they are hiring)
OpenAI Patent 1: https://www.freepatentsonline.com/y2024/0020116.html
Patent 2: https://www.freepatentsonline.com/y2024/0020096.html
OpenAI Codex: https://openai.com/blog/openai-codex
AlphaCode 2: https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf
Meta 350k: https://uk.pcmag.com/ai/150532/zuckerbergs-meta-is-spending-billions-to-buy-350000-nvidia-h100-gpus
Gemini Claim: https://deepmind.google/technologies/gemini/#capabilities
MMMU: https://arxiv.org/pdf/2311.16502.pdf
Gemini Paper: https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf
Altman on Ilya: https://www.youtube.com/watch?v=QFXp_TU-bO8&t=1147s
Verify: https://arxiv.org/pdf/2305.20050.pdf
Alignment Impact: https://openai.com/research/improving-mathematical-reasoning-with-process-supervision
Speculative Decoding: https://miro.medium.com/v2/resize:fit:1400/format:webp/0*7RxMK7wm3rJxKePy.png
Phi 2: https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
Comments
I am reading about Gavin and etched online & many seem to doubt him. 1 and a half year of since they started etched and still no working product i guess. Also, i see that groq already has a workin product. I would like to see Gavin succeed. Good luck to him :)
kavin vikram
2024-08-18 18:46:35 +0000 UTCWhat STEM jobs?
GGuy
2024-02-27 04:19:23 +0000 UTCWhat will inspiring students to pursue STEM jobs be like once there are no more incentives to learn to code?
Joshua Davis
2024-02-26 15:54:37 +0000 UTCThis video was absolutely incredible. As to this matter of the use of the word "may" as someone who owns several patents this is absolutely normal and is done as a matter of course by lawyers who write these patents. Even though the use of the word "may" may not appear in the claims it appears in the descriptions so that it can provide Open AI the ability to pursue continuation patents. So yes it is indicative of what *may* come next but it's also just a lawyers inclination to attempt to gain as broad a scope as they possibly can.
Joshua Davis
2024-02-26 15:52:37 +0000 UTCWhats the difference between groq and etched? are they direct competitors?
Dane Wagenhoffer
2024-02-22 01:14:24 +0000 UTCThis is why I am skeptical of the "foom" idea. The first AGI will likely be operating at the very edge of what is possible with available resources. While there's a chance it could develop a new architecture that puts it into AGI++ territory given its resources, it seems just as possible that it will only be able to self-improve a bit via efficiency optimizations. It won't be until the AGI can self-improve *and* solve resource scaling that we would get a runaway effect. Solving resource scaling (e.g., through a breakthrough in chip manufacturing) may or may not be within reach of the AGI itself. Seems more likely that it would need humans to develop and use specialized AIs, kind of how a very well rounded intellectual might still need to go to an expert for a deep problem in a specialized field.
James Kittock
2024-02-16 03:52:29 +0000 UTCCode generation is perhaps the single most important thing AI can do, because it unlocks everything else AI can do. This patent seems to have both defensive and offensive aspects. On the defensive side, OpenAI is likely trying to ensure that they will have rights to use AI to generate code to improve their own products. Filing now is their way to protect against another player swooping in and patenting the methodology and thus blocking OpenAI from using it, or forcing them to license it. On the offensive side, OpenAI may be trying to block other contenders (like Google) from engaging in certain kinds of code generation. For better or worse, this kind of patent warfare is very common in tech. It's also possible that OpenAI is planning to launch a code generation product and wants to protect future revenue of that product, but my guess would be that's a secondary consideration behind the above.
James Kittock
2024-02-16 03:34:18 +0000 UTCOpenAI....but not for you!
Christian Terry
2024-02-08 04:09:42 +0000 UTCGreat content thank you. I would like to learn more about the verification model. Maybe the verification step should be done at inference time, using the same network, i.e. reusing the KV cache. Other question, I would like to have an intuition of how much it cost in electricity to run those GPU. Like what is the ratio of gpu operation cost / gpu price.
Stéphane Collot
2024-02-04 11:02:40 +0000 UTCi think it is about specialst->generalist-> special generalist. Code is ohne core element of generating digital capabilities and virtualization of creativity. If you can handle that you can jump to combinations of vision and motion and building a world model
Markus Knaup
2024-02-04 07:20:08 +0000 UTCGreat comment. This is what I mean when I say demo-AGI say end of 2028. Not at all feasible for mass production at that point of course, or finished in any way either, but rather leveraging vast compute as the ultimate proof of concept.
Philip
2024-02-03 16:16:15 +0000 UTCYeah I was quite intrigued to see an attempt at patenting such a general method
Philip
2024-02-03 16:11:54 +0000 UTCWow… just wow, I’m bullish on the future that’s for sure.
Anouar Mansour
2024-02-03 03:17:25 +0000 UTCDo you think it helps with generating synthetic datasets 🤔?
GGuy
2024-02-03 02:09:01 +0000 UTCI also wonder this. Filter Github to all programmers that commits every few minutes with unique "descriptive" commit messages and CI results. Is this the closest thing to pure reasoning available on the internet?
GGuy
2024-02-03 01:16:55 +0000 UTCSpeculative decoding improves the performance of the inference phase. That might improve the scalability and cost problems when deployed to a large user base. However, it would also seem to give further potential to create a "one-off" special model, where an entire server complex focuses on answering a single question. That might have no commercial application, but the national security community might find it useful. According to the Villalobos paper (https://epochai.org/blog/trading-off-compute-in-training-and-inference) that technique could enable "peaking ahead" a full generation beyond current frontier models. If the improvement from speculative decoding is added, and coupled with enough dedicated hardware, it implies a further increased "model look ahead" ability. IOW any major AI breakthroughs will be seen internally on experimental configurations long before (maybe years before) they are publicly available. When people discuss possible dates for AGI, they tend to assume that will be the "big date" and it will be like CERN announcing the Higgs particle. I'm not sure it will unfold that way.
Joe Marler
2024-02-02 20:30:38 +0000 UTCPatent from OpenAI - how ironic :-) But if way of thinking for the machines could be patented what it will do to the whole AI industry. If AGI will be build does it mean everything developed by it is owned by the patent holder. This will literally put all knowledge and all wealth in the hands of single company...
Arek Stryjski
2024-02-02 17:59:20 +0000 UTCI am wondering why the patents of OpenAI are focused on code generation. A field where right now many players try to compete by building wrappers around LLMs. Yes for code the verification step is easy compared to other domain but I wonder if they are thinking a step further like the better the model is at coding the better it is at reasoning. What do you think?
SteveHaupt
2024-02-02 17:31:20 +0000 UTCAnyone else started their career change? 😂
GGuy
2024-02-02 15:32:51 +0000 UTC
