Autograder Overview

CSCI 3310 uses a system for automatically checking your projects and ensuring their correctness. The autograder is designed to help you do the projects correctly and to provide feedback in a timely fashion. By design, you can submit each project to the autograder multiple times as you work (instead of just once at the very end).

When you submit a project to the autograder, your submission will be checked automatically using a series of test cases. In each test case, your program's output on the test is compared to the output of a known correct program; the test is passed if the outputs are identical. The results (i.e., tests that passed and failed) will be emailed back to you as soon as the autograder is finished checking your submission. Be warned, however, that passes and failures will be the only information you will get: the autograder results will not tell you what the test cases are doing, what your program's outputs were, or what the correct outputs are. This design is intentional: the autograder does not function as a debugger, but rather as a tool to let you when your project is working properly. This feedback will help you avoid erroneously thinking that your project is finished when there is still work to be done.

Rather than relying on the autograder to debug, the best way to debug your program is to write your own test cases, figure out the correct outputs (i.e., according to the project specification), and then compare your program's output to the correct answers. The only way you can be sure that your program follows a given part of the specification is if you have a test that checks that part. Designing test cases will also give you a better understanding of the concepts covered by the projects.

Later projects will require you to submit your test cases to the autograder along with your program, and your tests themselves will be checked to determine how well they exercise the program. The autograder checks submitted test cases by running them through known buggy programs; if the submitted test case output on the known buggy program differs from its output on a correct program, then the bug is considered "exposed" by the test case. If the outputs are the same, then the bug is not exposed. The autograder results will tell you which bugs were exposed by your test cases -- without, of course, telling you anything about what the bugs are.

Submission Policy

The submission policy of the autograder is as follows:

1. You may submit a project to the autograder as many times as you like (up to the project due date). The final submission that you make prior to the deadline is considered your project submission, and this is the only submission that will be graded by me. You are never penalized for earlier submissions.
2. Although you can submit as many times as you like, you will only receive autograder feedback on the first submission of each calendar day. This means that the earlier you start working and submitting to the autograder, the more submissions with feedback you will have!
3. In addition to the single feedback round per day, you are allotted three bonus submissions per project that will also provide feedback. Bonus submissions are applied automatically: any time you make a submission on a day in which you already made an earlier submission, one of your bonus submissions will be applied (if you still have any left). If your three bonus submissions are exhausted, the system will continue to provide feedback once per day, but further submissions each day will not provide feedback (i.e., rule #2).

As you might surmise, the autograder submission policy is designed to incentivize you to start the projects as early as possible, which will both give you more time and provide you with more feedback.

In addition to the autograder's evaluation of your program's correctness, I will manually evaluate your final submission for issues such as program design, efficiency, documentation, style, etc. Your final score on each project will be a combination of the autograder score and my hand-graded score.

Keeping the Autograder Happy

The autograder can be a bit picky about the programs that you submit. If you submit a program that the autograder doesn't like, it will refuse to check your submission and tell you so via your submission feedback. This will make you sad for two reasons: (1) you will not get useful feedback, and (2) you will have wasted an autograder submission. In particular, any of the following will cause the autograder to reject your submission:

1. Submitting C++ files that end in a file suffix other than .cc.
2. Submitting a program that does not compile on the class server using the -std=c++11 flag.
3. Submitting a program that outputs a warning (any warning!) when compiled on the class server using the -Wall and -std=c++11 flags.
4. Submitting a program that follows the previous two rules on a different machine (e.g., your laptop), but hasn't been tested on the class server (ok, so you might get lucky here, but...).

Avoid these potential missteps by following a few simple rules:

1. Always give your C++ source files the filename suffix .cc.
2. Always compile using -std=c++11 and -Wall (you should always be using the latter anyways). The former flag tells g++ to use the 2011 standard of C++ (aka C++11); don't use any language features introduced in later standards. Put your flags in a Makefile to save you time and ensure you won't forget to add them. The starter files for most projects already include a Makefile containing these flags.
3. Fix every single warning, even if they don't affect the operation of your program. Fixing warnings is often an educational process as well.
4. If developing on a machine other than the class server, test your code on the class server (both compiling and running) before submitting.

Using the Autograder

Submissions to the autograder are made using the submit3310 command on the class server, like so:

submit3310 <projectnum> <file1> <file2> ...

For example, to submit Project 1, you would execute the following:

submit3310 1 inverter.cc

The system will package up your project and send it to the autograder. Within an hour or so (a few minutes in most cases) you should receive an email telling you how your submission did. If you do not receive a response within an hour of submission, let me know (but check your spam filter first). The official time of submission for your project is the time that the project is received by the autograder, regardless of how long it takes to check. If you make any submission after the project due date, your project will be considered late (and will use up your flex days).

Past students have correctly noted that it would be nicer if the autograder sent Slack messages instead of emails. This effort is left to future work.

Acknowledgments

Many parts of the autograder were initially developed by Peter Chen, Brian Noble, Atul Prakash, Mark Corner, and others.