There are regular competitions in which constraint solvers are compared against each other on unseen benchmarks.
The MiniZinc Challenge
The MiniZinc Challenge is an annual competition of constraint programming solvers on a variety of benchmarks. It has been held every year since 2008, with the winners being announced at the annual CP conference.
The XCSP3 Competition
The international XCSP3 competitions are organized to assess the performances of constraint solvers on Constraint Satisfaction Problems and Constraint Optimisation Problems. It was ran from 2017 - 2019 and might relaunch in the future.
Archive: the CSC Competitions
The CSC competitions were the predecessor to the XCSP3 Competitions. They used XCSP2 as specification language.