CS 8761 Natural Language Processing - Fall 2004 - Automated Essay
Grading
Final Project - Alpha Stage, due Mon Nov 15, before noon
This may be revised in response to your questions. Last update Weds
Nov 3, 2 pm
Objectives
To design, implement and deploy an alpha version of your essay grading
system.
This system should be web based, and include baseline functionality for
two of the following four components to your system:
- gibberish detection
- irrelevance measure
- identify statements of fact
- check statements of fact for accuracy
You will submit your system such that I can install and use it, and you
will also have a web site up and running that can be used. It is very
likely that I will demo your systems to the class on Mon, Nov 15.
Specification
Your system should have a simple web interface that provides the user
with a prompt (i.e, question) to answer. This question should be selected
from a file of possible prompts that you provide. The user should enter
their essay, and then submit it for grading. The system should respond
relatively quickly and be interactive. At this point do not try and
provide an overall score, just provide feedback from the above components.
Your system should be submitted "pre-trained", so that I can begin
using it immediately. In other words, if you do any calculations based
on large corpora, you should have already done those computations and
provide me with a data file that includes those results. Of course there
may be some dynamic computation required, and that is fine. For example,
you may look up information on the Web based on the users input. However,
if you use an LSA style approach, please create the co-occurrence matrix
ahead of time, and include that in your distribution.
It is up to your team to decide how you will approach each of these
problems. You may certainly use ideas from the published literature,
just make sure to acknowledge this in your documentation (described
below) by providing citations and references.
Installation
The installation of your system should simple, and follow the standard "3
step Perl" install. These steps consist of the following:
perl Makefile.PL PREFIX=/home/cs/tpederse etc.
make
make test
Note that if you rely on existing tools (like the Brill Part of Speech
Tagger, WordNet, etc.) you can simply provide me with detailed
instructions about where I can find those tools, and how to install them.
Do not assume that I know how to install those, and do not assume that I
already have them available. Also, do not simply refer me to the
instructions in that package, please provide a concise set of instructions
that includes any and all tools that might need to be available.
Note that the PREFIX variable indicates that I will not have supervisor
access, and I will install into my own personal directory. It is fine if
there are other directives required for Makefile.PL, simply indicate what
they are in your INSTALL file.
Your distribution should include a plain text file named INSTALL the
provides detailed instructions on how to install your system. You should
assume that the user of your system only has this documentation
available, and is not an expert in system administration. Thus, if any
other tools need to be installe, paths need to be set, Makefile.PL
variables need to be set, etc. please provide detailed instructions
about how to do that. Assume that your user does not have supervisor
access!
Documentation
Your distribution should include a file named README.pod that is written
in perldoc that describes your overall system.
You should assume that the user has no specialized knowledge of automated
essay grading. Thus, your README.pod should begin with a general
introduction to the problem of automated essay grading. This should
include a brief description of the history and related work in this area,
and it should be written in the style and form of a related work
section in a thesis or thesis proposal.
You should then introduce each of the four problems described above. In
other words, do not assume that the user will know what gibberish
detection, or relevance measurement is. Provide examples of text that
would "trigger" your system to provide guidance to the essay writer. You
should do this for all four systems, even though you only need to have
baseline implementations of two of them.
Then, you should describe the specific approach that you are taking for
your two baseline solutions, and also describe your overall plan for each
other components. You should clearly indicate what possible approaches you
are considering for each component, and also describe who is going to do
what.
Finally, compare and contrast your proposed approach with existing
techniques, and clearly credit any publications, systems, etc. that might
have given you ideas.
To summarize, your README.pod should consist of the following sections:
- INTRODUCTION TO AUTOMATED ESSAY GRADING (motivations, history, and
related work)
- OBJECTIVE (describe each problem /component that we hope to address,
and give examples both of the problem and the types of situations you
hope to be able to handle)
- IMPLEMENTATION OF BASELINE APPROACHES (describe in detail the methods
you have implemented in this release for two components)
- OVERALL PROPOSED SOLUTION (describe proposed solution for each of
the four solutions, and indicate who is going to do what)
- RELATION TO PREVIOUS WORK (specifically describe how your system
relates to others that already exist, and make sure to credit where your
ideas come from.)
Submission Guidelines
You should package your system as a single compressed tar file that is
named as TEAMNAME-0.01.tar.gz. This should include all of the code
needed to run your system, and the README and INSTALL documentation
described above. If you are using CPAN modules in your system, make
sure those dependencies are described in your Makefile.PL, and I will
obtain those modules via CPAN. The same is true of other outside packages.
If they are not available via CPAN (such as the Brill Tagger, WordNet)
provide detailed instructions as to where I can find those and how to
install them. You do not need to include those in your distribution.
Your system should unpack into a directory named
as TEAMNAME-0.01. TEAMNAME should be your assigned team name, written in
all capital letters. If there are embedded spaces in your name (e.g.,
BOCA JUNIORS) you should replace the space with a -, to result in the
name (e.g., BOCA-JUNIORS-0.01).
If you are using sourceforge, upload your system and I will download
from there. Please note that sourceforge provides a time stamp when
you upload. In your README, please prominently provide a URL where I
can run your system. Please note that I will also install it on my own
system as well.
Finally, make sure that you specifically address the issue of "terms
of use". In
the end, we will be posting your code on the Web, and you should
specifically address the issue of how the code can be used and
distributed. One option is the
GNU
CopyLeft. You are free to
investigate other licensing terms, but you must specifically address
this issue and follow the accepted standards for including licensing
information in your code and documentation. Also make sure to clearly
identify the authors of individual programs, and provide contact
information. If one person does all the work for a particular component,
then their name should be the only one on that particular component. In
cases where the effort is shared, then all team members should be
credited.
Policies
This is a team assignment. You are strongly advised to divide up the
work of the project into tasks that can be carried out in parallel by
various team members. All team members should be acknowledged in the
comments, etc. and all teammates will receive the same grade. Do not work
with other teams, and do not discuss your approach with other
teams. Each team should operate independent of all the other
teams. Make your own decisions as a team and do not be influenced by
the decisions of other teams if you happen to hear of them
accidentally.
by:
Ted Pedersen
- tpederse@umn.edu