TCEC Season 16
The 16th season of the Top Chess Engine Championship began on 15 July 2019 and ended on 13 Oct 2019.
The season showed how fast neural network-based engines were progressing relative to traditional ones, with neural network-based engines making up half of Premier Division for the first time. Furthermore, AllieStein, a neural network-based engine that had reached Premier Division last season, made its first appearance in the superfinal after finishing second in Premier Division. Nonetheless, a traditional engine – Stockfish – won both Premier Division and the superfinal.
Overview
Structure
TCEC changed its format for this season again, with the aim of allowing more chances for new engines to climb the rankings. This is in turn necessary because of the fast pace of development in computer chess, with numerous very strong engines emerging quickly. The superfinal and Premier Division were kept, but the lower divisions were replaced by three leagues: a Qualification League, League Two, and League One. The top six engines from the Qualification League promote to League Two, while the top four engines from League Two promote to League One. Finally, the top four engines in League One contest a playoff, with the top two finishers advancing to Premier Division. The top two engines in Premier Division then contest the superfinal. All three lower-division leagues are single double-round robins, while Premier Division is 3x double-round robin, and the superfinal remains a 100-game match.Results
Qualification League
The qualification league comprised 18 engines; however, one engine - Leela Chess Zero's CPU version - would only be used as a benchmark, and its scores zeroed out to determine final standings.Right from the start it became apparent that two engines - the new engine Stoofvlees and ScorpioNN - were significantly stronger than everyone else. ScorpioNN had been plagued by crashes last season, but those issues had been fixed, and it had further improved rapidly. It defeated ten of its seventeen rivals 2-0, and did not drop a match. Stoofvlees turned in a similarly dominant performance, also defeating ten of its seventeen rivals 2-0, but it dropped a game to 3rd-place Marvin. Both engines comfortably claimed the first two spots, six points ahead of Marvin. The remaining promotion slots were more contested. Marvin and Wasp were able to distance themselves from the chasing pack, but Asymptote and Winter were able to keep up with Topple and chess22k up until the final rounds, eventually finishing two points off the pace.
League Two
Fresh from promotion and with an update before the start of League Two, Stoofvlees comfortably placed first with 22.5/30, two points ahead of ScorpioNN. Ominously however, it showed erratic play and lost three games. Second-placed ScorpioNN did not lose a game and promoted smoothly as well. In contrast, the remaining two promotion spots were closely contested. Pedone was whitewashed by Stoofvlees, but was able to defeat six engines 2-0. This eventually allowed it to finish third, one point ahead of Arasan and Fritz. Arasan and Fritz both finished with 18/30. Like ScorpioNN, Fritz did not lose a game, but it wasn't able to score many wins either. This allowed Arasan - who had narrowly missed promotion last season - to claim the final promotion spot on the third tiebreak.League One
In contrast to the dominating performances in the previous leagues, League One was closely contested, with every engine losing games. Facing their strongest opponents yet, Stoofvlees and ScorpioNN again managed to promote, although Stoofvlees again played erratically and lost three games. ScorpioNN, which had not lost any games in the previous divisions, finally dropped points to Stoofvlees and last season's Premier Division engine, Ethereal. Eventual second-placed finisher Laser crashed once against Ginkgo, but still mustered enough wins to finish half a point ahead of ScorpioNN and Ethereal, qualifying for the playoffs. The final results were marred by crashes: Booot crashed three times and was disqualified, while Ginkgo came close by crashing twice. The possibility of Ginkgo crashing a third time kept fifth-placed Xiphos in contention up until the final round, when it needed to beat Stoofvlees and hope for a Ginkgo crash, in which case it would qualify for the playoffs instead of Ethereal. Nonetheless, Ginkgo did not crash, and Ethereal finished third, just ahead of ScorpioNN on tiebreak.At the other end of the table, Fire, which had been relegated from Premier Division together with Ethereal last season, submitted a drastically different neural-network based version for this season, and was soundly outmatched. It finished last with 6.5/30, scoring only one win by forfeit against Booot.
The League One playoffs was very close, and each engine lost games. After 12 games, only two points separated first-placed Stoofvlees from last-placed Laser. Stoofvlees, which had won League 1 comfortably, was defeated by ScorpioNN in the very first game of the playoffs, but recovered with three wins to finish first. Laser started with a win against Ethereal, but lost consecutive games to Stoofvlees and ScorpioNN to finish last. Ethereal was within striking distance of ScorpioNN up until the final games, and even won a head-to-head game, but it was defeated by Stoofvlees twice and finished one point behind ScorpioNN.
Premier Division
After the straightforward Premier Division of TCEC S15, the S16 Premier Division turned out to be a roller-coaster. As in the previous season, multiple-time champion and S15 runner-up Stockfish cruised to a dominating lead. After three rounds, it had scored wins against every engine other than S15 champion Leela Chess Zero and third-place finisher Komodo, while soundly defeating newcomers Stoofvlees and ScorpioNN 2.5-0.5. Meanwhile Lc0 didn't manage to win nearly as many games as Stockfish, but did score a head-to-head win, and was second. Komodo had started brightly with two wins, but went down to AllieStein, who in turn was defeated by Stockfish.However, Stockfish dramatically crashed once in the second round against Houdini, and then again in the third round against AllieStein, both times in probably-drawn positions. Although Stockfish had built up such a lead that it was still joint first after the crashes, the crashes did mean that one more crash would lead to Stockfish being disqualified. Stockfish's developers eventually came to the conclusion that the crashes were caused by a bug in MinGW used in compiling Stockfish, which included a faulty.dll file. Although the source of the crashes was due to third-party code and not Stockfish itself, by TCEC rules, engines could not be updated during divisions. Nonetheless, in Season 8, Stockfish had been allowed to update during a division to correct a non-critical bug, which set a precedent to allow updates if all participating teams agreed. Tournament director Anton Mihailov therefore organized an anonymous vote among all competitors to see if Stockfish should be allowed to replace the.dll file. Stockfish would be allowed to update only if the vote was unanimously in favor. Most of the participating teams voted "yes", but one team voted "no". The "no" vote served as a veto and meant that Stockfish would not be allowed to update the.dll file.
In the rounds after the vote, AllieStein surged to win after win to seize first place, defeating ScorpioNN, Houdini, Komodo MCTS and Stoofvlees one after another. By the last round, AllieStein had amassed 14 wins, including 7 with Black – more Black wins than the rest of the division combined. This large number of wins gave it the tiebreaker advantage over Lc0. Stockfish, with two crashes, had the worst tiebreakers of the three rivals and needed to finish ahead in points. It managed to defeat AllieStein a further two times, which gave it an advantage over Lc0, who was still struggling to score wins. Nonetheless, with the specter of two crashes hanging over Stockfish, the race was not over. In the final round, Lc0 needed to beat Stoofvlees with Black while hoping for Stockfish to either crash or lose to Komodo MCTS with White. When Stockfish–Komodo MCTS ended in a draw, Lc0 was eliminated. It became one of three engines in TCEC to finish a season undefeated. The draw also meant Stockfish narrowly won the division, 0.5 points ahead of AllieStein, in spite of two crashes.
For the other competitors, Komodo turned in a credible performance, conceding only two losses: to AllieStein, which it avenged in their fifth head-to-head encounter, and to Leela. This allowed it to finish clear fourth, 3.5 points behind Lc0, but 3 points ahead of Stoofvlees. Stoofvlees continued to play erratically, but was one of only two engines to hold Leela to all draws. Houdini, which had not been updated for two years, showed it was still strong enough to compete, finishing sixth. Komodo MCTS had been competitive with Houdini last season, but performed poorly this season, losing ten games and scoring only one win against ScorpioNN. Finally, newcomer ScorpioNN was outmatched and finished last without scoring a win.