AIOJ

Arbiter

Arbiter is an evaluation tool developed by Beihang University for NOI Linux, and is now used for judging major NOI-series programming contests. According to Lu Kaifeng's lecture “The Next-Generation Evaluation System” at the 2016 winter camp, Arbiter was developed by a team from Beihang University (GAIT) under the leadership of Professor Yin Baolin.

After NOI Linux was updated to version 2.0, Arbiter was also recompiled with Qt 5.12.8 and released as Arbiter 2.0. Since all subsequent evaluation environments use NOI Linux 2.0, the Arbiter version introduced below is the Arbiter 2.0 bundled with NOI Linux 2.0.

This evaluation software can only be found on NOI Linux. The binary is located at /usr/local/arbiter/local/arbiter_local.

Usage

Configuring the program

Configure the contestants' source-code folders and contestant list. Create the contestant folders in the NOIP format:

players/
| -- day1
|    | -- <contestant_1's ID>
|    |     | -- <problem_1>
|    |     |   `-- <problem_1>.c/cpp/pas
|    |     | -- <problem_2>
|    |     |   `-- <problem_2>.c/cpp/pas
|    |     | ...
|    |     | -- <problem_x>
|    |        `-- <problem_x>.c/cpp/pas
|    | -- <contestant_2's ID>
|    |     | -- <problem_1>
|    |     ...
|    ...
| -- day2
|    | -- <contestant_1's ID>
|    |     | -- <problem_1>
|    |     |   `-- <problem_1>.c/cpp/pas
|    |     | -- <problem_2>
|    |     |   `-- <problem_2>.c/cpp/pas
|    |     | ...
|    |     | -- <problem_x>
|    |        `-- <problem_x>.c/cpp/pas
|    | -- <contestant_2's ID>
|    |     | -- <problem_1>
|    |     ...
|    ...
...

Here, day<x> contains <x>, the round number, and <contestant_x's ID> is the contestant ID, in the form <省份>-<编号>, such as HL-001 and JL-125; <problem_x> is the problem name. For self-testing, a combination of letters, hyphens (that is, -), and digits can be used as a contestant ID.

The contestant list has the following format:

<contestant_1's ID>,<contestant_1's name>
<contestant_2's ID>,<contestant_2's name>
...

Here, <contestant_x's name> is the contestant's name. Save this file as a plain-text file or CSV file; UTF-8 encoding can be used.

The contestant list can also be added manually after starting Arbiter.

Next, configure the test data. The naming format for each data set is as follows:

<problem_x><y>.in <problem_x><y>.ans

Here, <y> is the data number, starting from 1. The default suffix for test data is .ans, while the suffix for contestant output is .out; do not confuse them.

If you need to change previously generated out files to ans files, you can use the rename command to make the change in batches in NOI Linux 2.0, or the ren command in Windows. The usage of these commands is introduced later.

There is no need to put the test data for each problem in that problem's folder; simply put them together.

Then begin configuring the evaluation folder.

In the lower-left corner, select “Show Applications” - “All” - “Arbiter_local” to start Arbiter.

Arbiter_Home

Click OPEN to open an existing contest, and then select the setup.cfg file in the corresponding contest folder; click NEW to create a new contest and set its name and contest directory. Note that a folder must be created under the user's home directory, and selected as the contest directory; evaluation will not work if the contest directory is created on the desktop. It is very likely that this problem occurs because the contest folder path cannot contain Chinese characters.

add_problem

In the problem overview on the left, right-click and select “Add Exam”, then right-click the exam tab and select “Add Problem” to create the problem.

Click the down arrow to the left of the exam to display everything. Click the problem tab to change the problem name to the problem's English name, and also change the problem's time and memory limits and comparison method. It is strongly not recommended to use “exact full-text direct comparison” as the comparison method, as it is very unfriendly to data created under Windows. You can choose a comparator according to the problem, but note that a comparator must be selected; otherwise the evaluation result will be No Score..

problem_list

Click “File” - “Save”. This operation cannot be omitted, otherwise the program will not generate the problem configuration file. Note that every modification to the problem configuration must be saved.

At this point, when you open the contest folder, you will find the following contents:

<name>/
| -- data
| -- evaldata
| -- filter
| -- final
| -- players
| -- result
| -- tmp
`-- day1.info
`-- player.info
`-- setup.cfg
`-- task1_1.info
`-- task1_2.info
`-- team.info

The filter folder contains some comparators; the result folder stores the contestants' evaluation results; the tmp folder is the cache folder used during evaluation. Here, the day<x>.info files are round configuration files, where <x> is the round number; the task<x>_<y>.info files are problem configuration files, where <x> is the round number and <y> is the problem number.

Put the prepared contestant program folders in the players/ directory. Note that the outermost level should contain the corresponding day<x> folder according to the contest day. Put all test data (not inside folders) in evaldata. If a custom checker is used, put it in filter.

Official evaluation

Open the “Problem Evaluation” tab to see the following page:

Pretest

If the contestant list has already been created, select “Import List” on the right to import it directly. If there are relatively few contestants, select “Add Contestant” on the right to import them.

The imported page is shown below.

Test

The ID in the example is HL-001, and the program automatically identifies the “Affiliation” field. If the ID does not follow the NOIP specification, it cannot be identified.

Change evaluation round 0 to evaluation round 1 (or another round). Then select “Select All” on the right (or select designated contestants), select “Evaluate Selected Contestants” below, select the problems to evaluate (or all problems), and finally wait for the evaluation to finish.

Detailed test-point information needs to be viewed in the result folder. There will be a result folder for each contestant, and the result files have the .result suffix; view them as plain-text files. If the error No score file. appears, check whether the /tmp/_eval.score file was generated during evaluation.

Writing a custom checker

By decompiling other checkers, we can determine that the command to run a custom checker is <problem>_e <in> <out> <ans>. The last three parameters represent the input, the contestant output, and the answer file, respectively. The final score must be written to /tmp/_eval.score; the first line is evaluation information and the second line is the score, with 10 points being full marks.

After compilation, the name of a custom checker must be <problem>_e, where <problem> is the problem name. When configuring a problem, select the custom checker, and then select the required custom checker.

In the problem configuration area of problem management, change the submission method from source code to answer file, and then select a custom checker to test answer-only problems.

Notes

The following points have been confirmed as requiring attention:

Be sure to save the contest promptly; otherwise the program may crash. To prevent crashes, try saving the contest multiple times, or save the contest whenever you make a modification.
Do not click the score statistics above before any evaluation has been performed, as this will cause Arbiter to crash directly.
Due to the Linux runtime stack limit, if you want to enable an unlimited stack, enter ulimit -s unlimited in the terminal before executing arbiter_local to open the evaluator; otherwise, an Exceeding memory limit problem may occur.
For official evaluation, all problems should have the same memory limit when preparing the problems. During evaluation, replace unlimited in the command with the number of KiB in the problem's memory limit. For example, if the problem's memory limit is 512 MiB, the command is ulimit -s $((512 * 1024)). The main reason for this problem is that Arbiter is started directly, its parent process is GNOME, and child processes inherit the parent's stack-size limit.
It is not recommended that the software's working directory contain spaces. If it does, it is very likely that none of the default checkers can be copied into the filter directory when creating a contest (that is, the filter directory is empty). Evaluation will then produce all-zero scores, and the generated result files will contain the message Compile Failed..
When viewing code, the message “answer file not found” means that the contestant's source code was not found.

Uncertain content:

It is easy for the system to freeze; for example, moving the mouse during a large evaluation may cause it to freeze.
It crashes intermittently (partly because the contest was not saved promptly).
After changing the comparison method, the change may fail to take effect.
Pay attention to permission issues during configuration, but permission problems will not occur if the same user is used to create the contest, copy the data, and perform the evaluation.

Vulnerabilities

Due to a long lack of maintenance, the system has some vulnerabilities, such as allowing the use of #pragma G++ optimize("O2") and __attribute__((__optimize__("-O2"))). gcc-plugins-for-oi can be used to detect these commands at compile time.

Evaluation

Arbiter 1.0.2 has not received any substantive updates since its development was completed, resulting in an extremely poor evaluation experience and a UI detached from modern aesthetics. In NOI Linux 1.4.1, it, like the GUIDE bundled with NOI Linux, became an object of intense complaints from contestants and coaches. In NOI Linux 2.0, apart from removing the source code from the comparator and recompiling the software as a whole with Qt 5, there were no major changes; some stability issues remain unresolved.

Appendix: Usage of the ren and rename commands

The Windows operating system includes a command for changing file names: ren.

The command syntax is as follows:

ren [<drive>:][<path>]<filename1> <filename2>

If we need to modify all files in the current working directory, for example, changing all out files to ans files, we can execute the following command:

ren *.out *.ans

If performing this type of change in the NOI Linux 2.0 environment, the rename command seems to be relatively convenient at present. However, it is not included in the NOI Linux 2.0 environment, so you must install it first:

sudo apt install rename

Note: If execution produces E: Unable to locate package package_name, you need to execute this command first: sudo apt-get update

After installation is complete, you can use the rename command normally. The rename command is used similarly to direct text replacement. Its command syntax in the NOI Linux 2.0 environment is as follows:

rename 's/<old-text>/<new-text>/' <filename>

Here, <filename> can use the wildcard *, or can specify one type of file (for example, *.out).

Note that there is also a / at the end inside the quotation marks. If one / is omitted, the rename command will report the error: Substitution replacement not terminated at (user-supplied code).

If we now need to modify all files in the current working directory, for example, changing all out files to ans files, we can write:

rename 's/\.out/\.ans/' *

Here, \. means escaping ..

(Friendly reminder: If \. is omitted, suppose your file contains outtest.out; after this command is executed, the file will be changed to anstest.out.)

Similarly, if you need to uniformly modify all files named atmost<x>.ans (where <x> represents the test-point number) to test<x>.ans, you can write:

rename 's/atmost/test/' *.ans