Many engineering challenges boil down to the same headache – too many knobs to turn and too few opportunities to test them. Whether tuning the power grid or designing a safer vehicle, each assessment can be costly, and there can be hundreds of variables that may matter.
Consider car safety design. Engineers must integrate thousands of parts, and many design choices can affect how a vehicle will perform in a collision. Classic optimization tools can start to struggle when searching for the best combination.
MIT researchers have developed a new approach that rethinks how a classic method, known as Bayesian optimization, can be used to solve problems with hundreds of variables. In tests on realistic engineering-style benchmarks such as power-system optimization, the approach found top solutions 10 to 100 times faster than widely used methods.
Their technique leverages a foundation model trained on tabular data that automatically identifies the variables that are most important for improving performance, repeating the process on better and better solutions. Foundation models are huge artificial intelligence systems trained on huge, common datasets. This allows them to be adapted to different applications.
Researchers do not need to constantly retrain the tabular base model as it works toward a solution, increasing the efficiency of the optimization process. This technique provides greater speedup for more complex problems, so it may be particularly useful in demanding applications such as materials development or drug discovery.
“Modern AI and machine-learning models can fundamentally change the way engineers and scientists build complex systems. We have come up with an algorithm that can not only solve high-dimensional problems, but is also reusable, so it can be applied to many problems without the need to start all over,” says Rosen Yu, a graduate student in computational science and engineering and lead author of a paper on this technology.
The paper co-authors with Yu are Cyril Picard, a former MIT postdoc and research scientist, and Faiz Ahmed, an associate professor of mechanical engineering and a core member of the MIT Center for Computational Science and Engineering. This research will be presented at the International Conference on Learning Representations.
Improving a Proven Method
When scientists want to solve a multidimensional problem but have expensive ways to evaluate success, such as crash testing a car to find out how good each design is, they often use a tried-and-true method called Bayesian optimization. This iterative method finds the best configuration for a complex system by building a surrogate model that helps predict what to explore next while considering the uncertainty of its predictions.
But the surrogate model must be re-trained after each iteration, which can quickly become computationally difficult if the space of possible solutions is very large. Additionally, whenever scientists want to deal with a different scenario, they need to create a new model from scratch.
To overcome both shortcomings, MIT researchers used a generative AI system, known as a tabular foundation model, as a surrogate model inside a Bayesian optimization algorithm.
“A tabular foundation model is like a ChatGPT for spreadsheets. The input and output of these models are tabular data, which is much more common to see and use than language in the engineering domain,” says Yu.
Like large language models like ChatGPT, Cloud, and Gemini, the model is pre-trained on massive amounts of tabular data. This makes it well equipped to deal with a wide variety of forecasting problems. Furthermore, the model can be deployed as is without the need for any retraining.
To make their system more accurate and efficient for optimization, the researchers adopted a trick that enables the model to identify those features of the design space that will have the greatest impact on the solution.
“A car may have 300 design criteria, but not all of them are the main drivers of the best design if you’re trying to increase certain safety parameters. Our algorithm can smartly choose the most important features to focus on,” says Yu.
It uses a tabular base model to estimate which variables (or combinations of variables) most affect the outcome.
It then focuses the search on those high-impact variables instead of wasting time searching for everything equally. For example, if the size of the front crumple zone has increased significantly and the car’s safety rating has improved, that feature has likely played a role in the increase.
Bigger problems, better solutions
One of their biggest challenges, says Yu, was finding the best tabular foundation model for the task. They then had to combine this with a Bayesian optimization algorithm in such a way that it could identify the most salient design features.
“Finding the most salient dimension is a well-known problem in mathematics and computer science, but finding a method that took advantage of the properties of tabular basis models was a real challenge,” says Yu.
With the algorithmic framework in place, the researchers tested their method by comparing five state-of-the-art optimization algorithms.
On 60 benchmark problems, including realistic situations like power grid design and car crash testing, their method consistently found the best solution 10 to 100 times faster than other algorithms.
“When an optimization problem gets more and more dimensions, our algorithm really shines,” Yu said.
But their method did not outperform the baseline on all problems such as robotic path planning. Yu says this probably indicates that the scenario was not well defined in the model’s training data.
In the future, researchers want to study methods that can boost the performance of tabular foundation models. They also want to apply their techniques to problems with thousands or even millions of dimensions, such as naval ship design.
“At a high level, this work points to a broader shift: not just to perception or language, but to using foundation models as algorithmic engines inside scientific and engineering tools, allowing classical methods like Bayesian optimization to scale to regimes that were previously impractical,” says Ahmed.
“The approach presented in this work, using a pre-trained foundation model with high-dimensional Bayesian optimization, is a creative and promising way to reduce the heavy data requirements of simulation-based design. Overall, this work is a practical and powerful step toward making advanced design optimization more accessible and easier to implement in real-world settings,” says Wei Chen, the Wilson-Cook Professor in Engineering Design and chair of the department of mechanical engineering at Northwestern University, who was involved in this research. Were not.
