How do machines think? What can we learn about human intelligence by studying? Can we understand ourselves better if we better understand the artificial intelligence systems that are becoming an increasingly important part of our everyday lives?
These questions may be deeply philosophical, but for Philip Isola, finding the answers is as much about calculation as it is about contemplation.
Isola, a newly appointed associate professor in the Department of Electrical Engineering and Computer Science (EECS), studies the fundamental mechanisms involved in human-like intelligence from a computational perspective.
While understanding intelligence is the paramount goal, his work focuses primarily on computer vision and machine learning. Isola is particularly interested in learning how intelligence emerges in AI models, how these models learn to represent the world around them, and what their “brains” share with the minds of their human creators.
“I see all the different types of intelligence as having a lot of similarities, and I want to understand those similarities. What is it that all animals, humans, and AI have in common?” Isola, who is also a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL).
According to Isola, a better scientific understanding of the intelligence possessed by AI agents will help the world integrate them into society safely and effectively, maximizing their potential to benefit humanity.
asking questions
Isola began considering scientific questions at an early age.
While growing up in San Francisco, he and his father would often go hiking on the Northern California coastline or camping in the hills around Point Reyes and Marin County.
He was fascinated by geological processes and often wondered how the natural world worked. In school, Isola was driven by an insatiable curiosity, and while he gravitated toward technical subjects like math and science, there were no limits to what he wanted to learn.
Isola didn’t fully know what to study as an undergraduate at Yale University until she came across cognitive science.
He says, “I was initially interested in nature – how the world works. But then I realized that the brain is even more interesting, and even more complex than the formation of planets. Now, I wanted to know what makes us tick.”
As a first-year student, he began working in the laboratory of his cognitive science professor and soon-to-be advisor, Brian Schall, a member of the Yale psychology department. He remained in that laboratory throughout his time as an undergraduate.
After spending a gap year working with some childhood friends at an indie video game company, Isola was ready to get back into the complex world of the human brain. He enrolled in the graduate program in brain and cognitive sciences at MIT.
He says, “Grad school was where I felt like I finally found my place. I had great experiences at Yale and in other stages of my life, but when I got to MIT, I realized that this is work I really love and these are people who think the same way as me.”
Isola credits his PhD advisor, Ted Adelson, the John and Dorothy Wilson Professor of Vision Science, as a major influence on his future path. He was inspired by Adelson’s focus on understanding fundamental principles rather than simply chasing new engineering benchmarks, which are formal tests used to measure a system’s performance.
A computational perspective
At MIT, Isola’s research moved toward computer science and artificial intelligence.
“I still love all those questions in cognitive science,” he says, “but I thought I could make more progress on some of those questions if I looked at it from a purely computational perspective.”
His thesis focused on perceptual grouping, which involves the mechanisms used by people and machines to organize separate parts of an image as a single, coherent object.
If machines can learn perceptual grouping on their own, it could enable AI systems to recognize objects without human intervention. This type of self-supervised learning has applications in areas such as autonomous vehicles, medical imaging, robotics, and automatic language translation.
After graduating from MIT, Isola completed a postdoc at the University of California at Berkeley so she could broaden her perspective by working in a laboratory focused entirely on computer science.
Isola recalls, “That experience helped make my work more impactful because I learned to balance understanding the fundamental, abstract principles of intelligence with exploring some more concrete criteria.”
At Berkeley, he developed the Image-to-Image Translation Framework, an early form of generative AI model that can, for example, transform a sketch into a photographic image, or a black-and-white photo into color.
He entered the academic job market and accepted a faculty position at MIT, but Isola deferred for a year to work at a then-small startup called OpenAI.
“It was a nonprofit, and I liked the idealistic mission at the time. They were really good at reinforcement learning, and I thought it was an important topic to learn more about,” he says.
He enjoyed working in the laboratory with so much scientific freedom, but after a year Isola was ready to return to MIT and start his own research group.
study human intelligence
He immediately fell in love with running a research laboratory.
“I really love the early stages of an idea,” he says. “I feel like I’m a kind of startup incubator where I’m constantly able to do new things and learn new things.”
Based on his interest in cognitive science and desire to understand the human brain, his group studies the fundamental computations involved in the human-like intelligence that emerges in machines.
A primary focus is representation learning, or the ability of humans and machines to represent and experience the sensory world around them.
In recent work, he and his colleagues observed that many different types of machine-learning models, from LLMs to computer vision models to audio models, appear to represent the world in similar ways.
These models are designed to perform very different functions, but there are many similarities in their architecture. And as they grow larger and are trained on more data, their internal structures become more similar.
This led Isola and his team to introduce the Platonic representation hypothesis (taking its name from the Greek philosopher Plato), which says that all these models that learn representations are converging toward a shared, underlying representation of reality.
“Language, images, sound – these are all different shadows on the wall from which you can infer that there is some kind of underlying physical process – some kind of causal reality –. If you train models on all these different types of data, they should eventually converge on that world model,” Isola says.
A related area that his team studies is self-supervised learning. This includes the ways in which AI models learn to group related pixels in an image or words in a sentence without labeled examples to learn from.
Because data is expensive and labels are limited, using only labeled data to train models can hinder the capabilities of AI systems. With self-supervised learning, the goal is to develop models that can create accurate internal representations of the world on their own.
“If you can have a good representation of the world, it makes it easier to solve the problem later,” he explains.
The focus of Isola’s research is on finding something new and surprising rather than building complex systems that can beat the latest machine-learning benchmarks.
Although this approach has achieved much success in uncovering innovative technologies and architecture, it means that the work sometimes lacks a concrete end goal, which can create challenges.
For example, when a lab is focused on discovering unexpected results, it can be difficult to keep the team together and funding flowing, he says.
“In a way, we’re always working in the dark. It’s high-risk, high-reward work. Every once in a while, we find some piece of truth that is new and surprising,” he says.
Apart from acquiring knowledge, Isola is passionate about imparting knowledge to the next generation of scientists and engineers. Among his favorite courses to teach is 6.7960 (Deep Learning), which he and several other MIT faculty members launched four years ago.
The class has seen rapid growth, from 30 students in the initial offering to more than 700 this fall.
And while the popularity of AI means there’s no shortage of students interested in it, the speed at which the field moves can make it difficult to separate the hype from actually significant progress.
He says, “I tell students that they have to take seriously everything we say in class. Maybe in a few years we will tell them something different. With this course we are really at the pinnacle of knowledge.”
But Isola also emphasizes to students that, despite all the hype surrounding the latest AI models, intelligent machines are much simpler than most people suspect.
He says, “Human ingenuity, creativity and emotions – many people believe that these can never be modeled. That may be true, but I think intelligence is quite simple once we understand it.”
Even though his current work focuses on deep-learning models, Isola is still fascinated by the complexity of the human brain and continues to collaborate with researchers studying cognitive science.
All the while, he remained mesmerized by the beauty of the natural world, which inspired his first interest in science.
Although she has less time for hobbies these days, Isola loves hiking and backpacking in the mountains or on Cape Cod, skiing and kayaking, or finding beautiful places to spend time when traveling to scientific conferences.
And while she’s eager to explore new questions in her lab at MIT, Isola can’t help but think about how the role of intelligent machines might change the course of her work.
He believes that artificial general intelligence (AGI), or the point where machines can learn and apply their knowledge like humans, is not far away.
“I don’t think AI will do everything for us and we’ll go to the beach and enjoy life. I think there will be this coexistence between smart machines and humans, who still have a lot of agency and control. Now, I’m thinking about interesting questions and applications when that happens. How can I help the world in a post-AGI future? I don’t have an answer yet, but it’s on my mind,” he says.
