Special Judge

This page mainly introduces methods for writing spj programs for some judging tools/OJs.

Introduction

Special Judge (abbreviated as spj, also known as checker) is a program used to determine whether an answer is valid when a problem has multiple solutions.

Warning

An spj should also check whether there is extra content at the end of the file and whether the output format is correct (for example, if the problem requires one space between numbers but the contestant uses a newline). However, at present, only Testlib can conveniently do the former, while almost nobody deliberately checks the latter.

Be careful of NaN when judging floating-point numbers. An unreasonable judging method may make it possible to obtain AC simply by outputting NaN.

When reading the contestant's file, the program should check whether all required content has been read correctly, to prevent an spj runtime error. (Some OJs treat an spj runtime error as a system error.)

Note

The following examples all use C++ and take “requiring the difference between the standard answer and the contestant's answer to be less than 1e-3, with the file named num and a maximum score of 10 points for a single test case” as the example.

Testlib

See: Testlib/Introduction, Testlib/Checker

Testlib is a C++ library used to help problem setters write algorithm competition problems in C++.

Testlib must be used as the spj judging tool/OJ on Codeforces, Luogu, UOJ, and others.

Testlib can be used as the spj judging tool/OJ on LibreOJ (Lyrio), Lemon, Nowcoder, and others.

The modified version of Testlib required by SYZOJ 2 is hosted on pastebin1, but this modified version does not modify interactive mode. syzoj/testlib hosts a version of Testlib that can be used in interactive mode on SYZOJ 2.

The modified version of Testlib required by Lemon is hosted at GitHub - GitPinkRabbit/Testlib-for-Lemons. Note that this version should use registerLemonChecker() rather than registerTestlibCmd() when registering a checker. This version is inherited from the old version by matthew99 and adds some new Testlib features. If you use LemonLime, you can use the native Testlib.

The modified version of Testlib required by DOMJudge is hosted at cn-xcpc-tools/testlib-for-domjudge. This version of Testlib can serve both as a Special Judge checker and as an interactor for interactive problems.

The modified version of Testlib required by Arbiter is hosted at testlib-for-arbiter.

Most other judging tools/OJs require Testlib to be modified according to their spj writing format, and testlib.h uploaded together with the spj; or testlib.h placed in the include directory.

#include "testlib.h"
//
#include <cmath>

int main(int argc, char *argv[]) {
  /*
   * inf: input
   * ouf: contestant output
   * ans: standard output
   */
  registerTestlibCmd(argc, argv);

  double pans = ouf.readDouble(), jans = ans.readDouble();

  if (abs(pans - jans) < 1e-3)
    quitf(_ok, "Good job\n");
  else
    quitf(_wa, "Too big or too small, expected %f, found %f\n", jans, pans);
}

Lemon

Note

Lemon has a ready-made modified version of Testlib; using Testlib is recommended.

The latest version of LemonLime already supports using the original Testlib to write validators. If you use LemonLime, using Testlib is recommended.

#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: contestant output
   * argv[3]: standard output
   * argv[4]: score for a single test case
   * argv[5]: final score output (0 ~ argv[4])
   * argv[6]: error report output
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fout = fopen(argv[2], "r");
  FILE* fstd = fopen(argv[3], "r");
  FILE* fscore = fopen(argv[5], "w");
  FILE* freport = fopen(argv[6], "w");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(fscore, "%s", argv[4]);
    fprintf(freport, "Good job\n");
  } else {
    fprintf(fscore, "%d", 0);
    fprintf(freport, "Too big or too small, expected %f, found %f\n", jans,
            pans);
  }
}

Cena

#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * FILENAME.in: input
   * FILENAME.out: contestant output
   * argv[1]: score for a single test case
   * argv[2]: standard output
   * score.log: final score output (0 ~ argv[1])
   * report.log: error report output
   */
  FILE* fin = fopen("num.in", "r");
  FILE* fout = fopen("num.out", "r");
  FILE* fstd = fopen(argv[2], "r");
  FILE* fscore = fopen("score.log", "w");
  FILE* freport = fopen("report.log", "w");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(fscore, "%s", argv[1]);
    fprintf(freport, "Good job\n");
  } else {
    fprintf(fscore, "%d", 0);
    fprintf(freport, "Too big or too small, expected %f, found %f\n", jans,
            pans);
  }
}

CCR

#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * stdin: input
   * argv[2]: standard output
   * argv[3]: contestant output
   * stdout:L1: final score ratio output (0 ~ 1)
   * stdout:L2: error report output
   */
  FILE* fout = fopen(argv[3], "r");
  FILE* fstd = fopen(argv[2], "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    printf("%d\n", 1);
    printf("Good job\n");
  } else {
    printf("%d\n", 0);
    printf("Too big or too small, expected %f, found %f\n", jans, pans);
  }
}

Arbiter

#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: contestant output
   * argv[3]: standard output
   * /tmp/_eval.score:L1: error report output
   * /tmp/_eval.score:L2: final score output
   */
  FILE* fout = fopen(argv[2], "r");
  FILE* fstd = fopen(argv[3], "r");
  FILE* fscore = fopen("/tmp/_eval.score", "w");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(fscore, "Good job\n");
    fprintf(fscore, "%d", 10);
  } else {
    fprintf(fscore, "Too big or too small, expected %f, found %f\n", jans,
            pans);
    fprintf(fscore, "%d", 0);
  }
}

HUSTOJ

#include <cmath>
#include <cstdio>

#define AC 0
#define WA 1

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: standard output
   * argv[3]: contestant output
   * exit code: return judging result
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fout = fopen(argv[3], "r");
  FILE* fstd = fopen(argv[2], "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3)
    return AC;
  else
    return WA;
}

QDUOJ

QDUOJ is somewhat more troublesome by comparison. Problems with an spj have no standard output, so std can only be written into the spj, and the standard output generated by running it can then be used for judging.

#include <cmath>
#include <cstdio>

#define AC 0
#define WA 1
#define ERROR -1

double solve(...) {
  // std
}

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: contestant output
   * exit code: return judging result
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fout = fopen(argv[2], "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);

  jans = solve(...);
  if (abs(pans - jans) < 1e-3)
    return AC;
  else
    return WA;
}

HDOJ

HDOJ is basically the same as QDUOJ: std also needs to be implemented in the spj and compared with the contestant's output. However, unlike QDUOJ, HDOJ compares the answer with the spj output on standard output and then gives the final result. Therefore, when uploading output, only the output of the spj when correct needs to be uploaded.

HDOJ requires uploading a binary file compiled under Windows, rather than source code.

#include <cmath>
#include <cstdio>

double solve(FILE* fin) {
  // std, read input from fin
}

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * stdin: contestant output
   */
  FILE* fin = fopen(argv[1], "r");

  double pans, jans;
  if (scanf("%lf", &pans) != 1) {
    printf("WA\n");
    goto finish;
  }

  jans = solve(fin);
  if (abs(pans - jans) < 1e-3)
    printf("AC\n");
  else
    printf("WA\n");

finish:
  fclose(fin);
  return 0;
}

The corresponding answer file is:

AC

SYZOJ 2

Note

SYZOJ 2 has a ready-made modified version of Testlib; using Testlib is recommended.

The latest version of LibreOJ is no longer based on SYZOJ, but on Lyrio. Lyrio supports using the original Testlib to write validators, which is also the more general and recommended approach.

#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * in: input
   * user_out: contestant output
   * answer: standard output
   * code: contestant code
   * stdout: final score output (0 ~ 100)
   * stderr: error report output
   */
  FILE* fin = fopen("input", "r");
  FILE* fout = fopen("user_out", "r");
  FILE* fstd = fopen("answer", "r");
  FILE* fcode = fopen("code", "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    printf("%d", 100);
    fprintf(stderr, "Good job\n");
  } else {
    printf("%d", 0);
    fprintf(stderr, "Too big or too small, expected %f, found %f\n", jans,
            pans);
  }
}

Nowcoder

Note

Nowcoder has a ready-made modified version of Testlib; using Testlib is recommended.

See: How to Set Special Judge Programming Problems on Nowcoder

#include <cmath>
#include <cstdio>

#define AC 0
#define WA 1

int main(int argc, char* argv[]) {
  /*
   * input: input
   * user_output: contestant output
   * output: standard output
   * exit code: return judging result
   */
  FILE* fin = fopen("input", "r");
  FILE* fout = fopen("user_output", "r");
  FILE* fstd = fopen("output", "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3)
    return AC;
  else
    return WA;
}

DOMJudge

Note

DOMJudge supports spj programs written in any language. See: problemarchive.org output validator format.

DOMJudge has a ready-made modified version of Testlib; using Testlib is recommended.

Documentation for the Testlib used by DOMJudge and for importing Polygon problem packages: https://github.com/cn-xcpc-tools/testlib-for-domjudge

DOMJudge's default comparator has floating-point comparison with precision built in; simply add validator_flags with float_tolerance 1e-3 to the problem configuration.

#include <cmath>
#include <cstdio>

#define AC 42
#define WA 43
char reportfile[50];

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: standard output
   * argv[3]: folder for judging-information output
   * stdin: contestant output
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fstd = fopen(argv[2], "r");
  sprintf(reportfile, "%s/judgemessage.txt", argv[3]);
  FILE* freport = fopen(reportfile, "w");

  double pans, jans;
  scanf("%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(freport, "Good job\n");
    return AC;
  } else {
    fprintf(freport, "Too big or too small, expected %f, found %f\n", jans,
            pans);
    return WA;
  }
}

The header file validate.h provided by Kattis Problem Tools can also be used to write a more complex validator.

References

Footnotes

  1. LibreOJ supports the Testlib checker! ↩

Previous section
036 WSL (Windows 10)
Next section
Overview Introduction to Testlib

This content comes from OI Wiki: Special Judge. See the original for authors and revision history. AIOJ has adapted the formatting, in-site links, and resource paths.

Content is shared under CC BY-SA 4.0 and SATA license; additional terms are governed by the original notice. Thanks to the OI Wiki authors and contributors; per the SATA terms, please visit the OI Wiki repository and star it to show your support.