Should you catch your umbrella before getting out of the door? Checking the weather forecast in advance will only be helpful when that forecast is accurate.
Problems of spatial prediction, such as weather forecasting or air pollution, include predicting the value of a variable in a new location based on the values known at other places. Scientists usually use effort-and-truth verification methods to determine this to rely on these predictions.
But MIT researchers have shown that these popular verification methods may fail quite badly for spatial prediction tasks. This may motivate someone to believe that a forecast is accurate or that a new prediction method is effective when it is not really so.
Researchers developed a technique to assess predictive-satisfaction methods and used it to prove that two classical methods could be quite wrong on spatial problems. He then determined why these methods could fail and a new method designed to handle the types of data used for spatial predictions can be created.
In experiments with real and fake data, his new method provided more accurate beliefs than two most common techniques. Researchers evaluated each method using realistic spatial problems, including predicting air speed at Chicago O-Hare Airport and forecasting air temperatures at five American metro locations.
Their recognition method can be applied to many problems, which helps climate scientists to predict sea surface temperature for epidemiologists in assessing the effects of air pollution on certain diseases.
Tamara Broaderick, an associate professor at MIT’s Electrical Engineering and Computer Science (EECS), says, “Hopefully it will be more reliable evaluation when people are coming with new forecast methods and better understanding of how good ways are performing , “MIT’s Electrical Engineering and Computer Science (EECS), an associate professor Tamara Broaderick, says. , A member of the laboratory for information and decision systems and an associated with data, system, and institute for society, and computer science and artificial intelligence laboratory (CSAIL).
Brodarick to lead writer and MIT Postdock David R. Burt and EECS are included on paper by Graduate Student Uni Shane. Research will be presented at the International Conference on Artificial Intelligence and Statistics.
Evaluation of verification
The group of Broaderick has recently collaborated with oceanography and atmospheric scientists to develop machine-learning prediction models that can be used for problems with a strong spatial component.
Through this work, he noticed that traditional verification methods in spatial settings could be wrong. These methods catch a small amount of training data, called verification data, and are used to assess the accuracy of the prophet.
To find the root of the problem, he made a thorough analysis and determined that traditional methods create assumptions that are unfair to spatial data. Methods of evaluation depend on perceptions about how verification data and data want to predict one, related to test data, related.
Traditional methods believe that verification data and test data are distributed as independent and identity, which means that the value of any data point does not depend on other data points. But in a spatial application, this often does not happen.
For example, a scientific EPA can use verification data from the air pollution sensor to test the accuracy of the method that predicts air pollution in conservation areas. However, the EPA sensors are not independent – they sat based on the location of other sensors.
In addition, perhaps verification data cities have EPA sensors, while conservation sites are in rural areas. Because these data are from different places, they have different statistical properties, they are not distributed by identity.
“Our experiments showed that you get some wrong answers in spatial matters when these perceptions are broken by verification method,” called broadrich.
Researchers needed to come up with a new perception.
Especially spatial
In particular, thinking about a spatial context, where the data is collected from different places, they have prepared a method that changes recognition data and testing data smoothly into space.
For example, the level of air pollution is unlikely to change dramatically between two neighboring houses.
“This regularity perception is suitable for many spatial processes, and it allows us to create a way to evaluate spatial prophets in spatial domains. To do the best of our knowledge, no one has made a systematic theoretical evaluation that went wrong to come with a better approach, ”says Brodarick.
To use their evaluation technology, no one will input its prophet, in which they want to predict, and their verification data, then it automatically does the rest. In the end, it estimates how correct the forecast of the prophet for the location in the question. However, effectively assessing his verification technique proved to be a challenge.
“We are not evaluating a method, instead we are evaluating an evaluation. Therefore, we had to step back, think carefully, and should be creative about the appropriate experiments we could use, ”Broaderick explains.
First, he designed several tests using fake data, which had unrealistic aspects, but allowed them to carefully control significant parameters. Then, he created more realistic, semi-similated data by modifying the actual data. Finally, he used real data for many experiments.
Using three types of data from realistic problems, such as predicting the price of a flat in England and based on their location and forecasting the wind speed, enabled them to make a comprehensive evaluation. In most experiments, his technology was more accurate than the traditional method he compared it.
In the future, researchers plan to implement these techniques to improve the amount of uncertainty in spatial settings. They also want to find other areas where regularity perception can improve the performance of predictions, such as with time-chain data.
This research has been funded in part by the office of National Science Foundation and Naval Research.
