Five scores, one hidden formula, one try a day

By Rishi Garg, Taher Merchant, Aarush Mahajan

What we learned from six weeks in the NeurIPS 2026 RealPDE Competition, where our team finished in the top 20 of Track 2 by working out what a leaderboard rewards when the rules don't say.

From mid-August to the end of September 2026, three of us competed in the NeurIPS 2026 RealPDE Competition under the team name probability. We entered both tracks and finished in the top 20 of Track 2 with a final score of 81.82. This post is about what the competition asked for, what made it hard, and what we would tell a team starting one like it. You can find the results and leaderboard screenshots in our news post.

Why a vision lab entered a fluid dynamics competition

Our lab works on computer vision and interpretability, and RealPDE looks at first like a physics problem. The data changed our minds. The real measurements come from particle image velocimetry (PIV), where a laser lights up tracer particles in a wind tunnel and cameras photograph them. Software turns pairs of frames into a velocity field. Each sample is a grid of vectors over time, much like a short video.

Predicting how that video continues is a forecasting problem we know how to approach. Doing it on real measurements, with their noise and gaps, is the part that makes RealPDE different from most PDE benchmarks.

What the competition asked

RealPDE studies flow around a NACA4418 airfoil at several Reynolds numbers and angles of attack. For each condition the organizers released two versions of the flow. One came from the wind tunnel and the other from a computer simulation (CFD). The two never match exactly, and that gap is the subject of the competition.

The competition ran as two tracks on Codabench:

• Track 1 (Sim2Real) asks a model trained mostly on simulation to predict the real measured flow.
• Track 2 (Long-Term Test-Time Adaptation) asks a model to forecast the flow far into the future while it adapts to the data it sees during the test. Small errors compound over a long rollout, and the model has a strict time budget per step.

Track 2 alone drew more than 350 registered participants and over 1,600 submissions by the end of the development phase.

Five scores and a formula nobody published

Every submission came back with five scores:

1. Relative L2 measures how close the predicted flow field is to the real one.
2. TKE checks whether the prediction keeps the turbulent kinetic energy of the real flow. A forecast that blurs the eddies away scores badly here even when it looks accurate on average.
3. MVPE measures the error in the mean velocity profile.
4. Time rewards a model that runs fast.
5. SPS, the safe prediction score, asks the model to give an uncertainty band around its forecast. The band earns credit when it is narrow and still contains the true flow.

The leaderboard ranked teams on one final score that combined all five. The organizers never published the formula.

This shaped the whole competition. The five scores pull against each other. A smoother forecast lowers the L2 error and loses TKE. A wider uncertainty band covers more of the truth and loses SPS for being wide. A larger model predicts better and loses time. Without the weights, we couldn’t say in advance whether a change helped. We had to submit it and read the final score.

One try a day

During the development phase, each team could submit once per day. We joined in mid-August, which gave us about forty chances to learn from the real test set before the phase closed on 27 September.

That limit changed how we worked. A submission stopped being a way to show off a result and became an experiment with one question attached. Before each upload we wrote down what we expected the score to do and what we would conclude if it didn’t. When the result came back, we updated the plan the same day. The submissions that taught us the most were often the ones that scored worse than we expected, because they showed us which of our assumptions were wrong.

When local validation disagrees with the leaderboard

The hardest lesson was about our own validation. Several times a change improved every number we could measure locally and then lost points on the leaderboard. The hidden test set behaved differently from the data we had, and a random split of the training data hid that difference from us.

What helped was holding out whole flow conditions instead of random samples, and then checking the split against the leaderboard itself. We kept a validation setup only after its predictions had matched real leaderboard changes. Once one did, we could test ideas locally and save the daily submission for the ones that passed. Before that, a local gain was a guess.

Speed is part of the score

In Track 2 the clock counts. A model that needs twice as long per step gives up points it has to win back in accuracy. We spent real effort on how fast our code ran on the GPU, timed it the same way the organizers did, and checked the timing before every upload. Several ideas that improved accuracy didn’t survive once we paid for the extra time they took.

What we would tell the next team

• Read the scoring code before you design a model. Each score rewards a specific behaviour, and you can’t trade them off well until you understand all of them.
• Build a validation split you trust before you trust your validation scores, and check it against the leaderboard.
• Treat each submission as an experiment with a written prediction.
• Count runtime as part of the score from the first day.
• Write everything down. Even six weeks is long enough to forget why you made a decision.

What comes next

The development phase closed on 27 September. The organizers have invited us to the Track 2 final evaluation as a reserve team. In that round they retrain each method from scratch on the public training data and score it on a private test set. Final results come out on 10 November 2026.

We would like to thank the RealPDE organizers for a carefully run competition, Seifedine Kadry for his guidance throughout, and Abhinav Joshi and Siddharth Bharti for their help and contributions in Track 1.