Do LLMs Boost Productivity? Harvard Business School Interview + 5 Papers Analysed
Added 2024-01-18 16:37:21 +0000 UTCLots of articles recently in the New York Times, Washington Post, Forbes and MIT about whether LLMs boost productivity for white-collar workers. Well, I've dug into the details behind 5 of the papers frequently cited, interviewing co-authors [Edward McFowland III from Harvard Business School] and revealing what no other outlet did. Plus, test yourself on a task that GPT-4 actually worsened productivity for, learn about the Simpsons Paradox, and hear what the IMF thinks about the future of work.
Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality: https://www.hbs.edu/ris/Publication%20Files/24-013_d9b45b68-9e74-42d6-a1c6-c72fb70c7282.pdf
NYT Article: https://www.nytimes.com/2023/12/28/science/chatgpt-business-consultants.html
Forbes Coverage: https://www.forbes.com/sites/danpontefract/2023/09/29/harvard-and-bcg-unveil-the-double-edged-sword-of-ai-in-the-workplace/?sh=6147400e3f9f
MIT Coverage: https://mitsloan.mit.edu/ideas-made-to-matter/how-generative-ai-can-boost-highly-skilled-workers-productivity
Washington Post Coverage: https://www.washingtonpost.com/opinions/2023/09/28/ai-chatgpt-robots-taking-jobs/
MIT Paper on productivity, Noy: https://economics.mit.edu/sites/default/files/inline-files/Noy_Zhang_1.pdf
Ideas are Dimes a Dozen: LLMS Paper - https://deliverypdf.ssrn.com/delivery.php?ID=293089090070118003027066013026092098004008058002025032090120112097083029086082088112122000022099013026035126087086095086112064059076021059086079092015025084097126077047095008092028027090110111069127021109114004072094068099068003107108106097015092024&EXT=pdf&INDEX=TRUE
Simpson’s Paradox: https://en.wikipedia.org/wiki/Simpson%27s_paradox
Muck Rack Public Relations Report on GenAI Adoption: https://4272994.fs1.hubspotusercontent-na1.net/hubfs/4272994/State%20of%20AI%202024/Muck%20Rack%20-%20State%20of%20AI%20in%20PR%202024.pdf?utm_source=www.theneurondaily.com&utm_medium=referral&utm_campaign=microsoft-copilot
Github Study: https://github.blog/2023-10-10-research-quantifying-github-copilots-impact-on-code-quality/
Alphacode 2 Paper: https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf
Bloomberg IMF Coverage: https://www.bloomberg.com/news/articles/2024-01-14/artificial-intelligence-will-affect-almost-40-of-jobs-imf-says?srnd=technology-ai
Comments
I agree using ChatGPT for idea generation saves an astronomical amount of time. But I think more broadly that's because idea generation falls into a category of tasks where users can provide almost instant feedback. Idea generation, image generation, code generation all fit into those categories. "This idea feels better", "This image looks better", "This code doesn't compile or provide correct output". It quickly becomes obvious to users which idea misses the mark, which "hand" looks wrong, which line of code is causing the a compile error. I ponder if business use will drastically increase when we focus less on LLMs to provide the answers but also provide the user with the ability to have "instant feedback". For example I believe the Simpson's paradox is easily solved if the AI simply highlighted the data for the user. AI can still attempt to answer, but the "highlighted table" would make it more obvious something doesn't feel right. Green Green Red Red Red Green It'd be interesting to run the experiment again with the simple addition of the AI also providing a highlighted version of the table as part of its justification and see what the results are 🤔.
GGuy
2024-01-23 01:21:56 +0000 UTC@r though I have to say, it's great, thank you
Viktor
2024-01-22 21:25:27 +0000 UTCi agree 100%. i am a researcher and developer in the humanities. we are able to semi-automate entire workflows.
Christopher Pollin
2024-01-22 07:16:02 +0000 UTCThe secret sauce is the synergy with LMM. To achieve it, you need to have broad-based subject knowledge (in my case, medicine and pharmacy) to guide and verify, as well as the ability to communicate with AI (involving data preparation, prompt engineering, and RAGs). Unfortunately, most people just ask questions to ChatGPT and are disappointed with the low-quality output.
Michal Babula
2024-01-20 18:06:13 +0000 UTCNice! And agreed
Philip
2024-01-20 13:23:27 +0000 UTCI think the replacement wouldn't happen abruptly though. You'd be working in a team of 10, and the company would eventually realise that 5 people can do all the work the company needs. And you'd be let go. When that happens at global scale, it'd eventually be hard for you to find full time jobs unless the market grows. If the market doesn't grow as much, maybe some regulation to reduce work hours could help getting everyone employed. Eventually, AGI could do everything, but I guess that if humans have no marketable skills, then we probably would have bigger problems than finding a job :).
Carlos Baraza
2024-01-19 22:02:52 +0000 UTCI just tried it out, used their proposed example for AI project to write a code for multiplayer tictactoe using react and express, it impressively created a lot of files, but in the end it doesn't compile. But if there is a future where it does compile, it will be awesome.
Viktor
2024-01-19 22:02:28 +0000 UTCFrom my experience, I've worked 20 years in pharmaceutical companies and recently spent 15 months learning about LLMs. It's amazing how I can now do in 3 hours what used to take 8 hours. Plus, I'm having much more fun at work. It's a big change!
Michal Babula
2024-01-19 21:39:19 +0000 UTCYou answered with my reply yourself! A huge boom until the moment that AI can do 100% of the tasks in a job. And I am not a big believer that 'a preference for humans' will last long in the market. Even if 90% of businesses are willing to act on that preference, the 10% will gradually outcompete them, until they win the whole pie or are regulated out of this (good luck doing that internationally), or the 90% join in the automation. At which point we see the job disappear. The ultimate question of which jobs can survive this process, and whether they are enough to form a healthy economy, is something I think about a lot.
Philip
2024-01-19 15:12:59 +0000 UTCWhy would there be a bust after the boom? What if the market grows at a similar pace as AI increases workers' productivity? If the market grows, workers could still be needed until 100% of their work is replaced. The question then is, would AI ever be able to take on 100% of what it means to be a human in the economy? For example, if more people can code, more people would create software, so more programming gigs would eventually be available for senior engineers when the new comers to the field get stuck. Even if the senior programmer becomes more efficient, the same amount of engineers or more might be needed to address the new bigger market. I wonder what would happen to the demand side. Would AI allow you to consume a bigger part of the market? That would justify market growth.
Carlos Baraza
2024-01-19 10:15:16 +0000 UTCDid you see my Holy Grail vid on here? Interviewed top experts as my best attempt to get to an answer!
Philip
2024-01-19 08:21:44 +0000 UTCWill check it out! Have you tried AlphaCodium?
Philip
2024-01-19 08:20:54 +0000 UTCVery similar for me Trenton, and thank you. I have spent almost a year squeezing what I can out of LLMs, using every technique under the sun, and still it isn't quite enough for many tasks. Interestingly Turbo was more of a step change than they announced it to be but still.
Philip
2024-01-19 08:19:54 +0000 UTCThis was a wonderful video. Bravo. But the question now is… Do you want to look under the hood of the future of LLM logic and reasoning beyond domain specific projects?
Stefan
2024-01-19 02:48:40 +0000 UTCI've been pondering whether the challenge in integrating AI into our workflows lies more in the difficulty of discovering AI tools that are effectively applicable, rather than AI's inherent limitations in enhancing our workflows. For instance, while GitHub Copilot has its merits in expediting boilerplate code, it didn't significantly transform my coding process. However, discovering cursor.sh was a game-changer for me, revealing the untapped potential of coding AI. This makes me curious: are there other groundbreaking AI applications like cursor.sh out there, waiting to be discovered? I'm eager to hear about any 'unknown unknowns' that others might have stumbled upon in this space.
r
2024-01-19 02:29:23 +0000 UTCTwo in one day? You spoil us Philip. I found your explanation of the worker productivity study very interesting. My firsthand experience agrees with their findings, it’s a massive boost for creative tasks but I have to be cautious when using it for detailed problem solving or critical thinking. I’m extremely hopeful for the next release, the consistency currently isn’t there for the type of tasks I really want to use it for.
Trenton Dambrowitz
2024-01-18 18:10:18 +0000 UTC
