Using "Annotator Rationales" to Improve Machine Learning for Text Categorization
Omar F. Zaidan, Jason Eisner, and Christine Piatko

Paper: in PDF and PS.
Presentation: here (used for NAACL HLT 2007 talk).
Handout PDF: here (a trimmed down version of the presentation, converted to PDF).
Dataset: here (~7 MB; here's a small sample, or just the README file).

BibTeX entry:
@InProceedings{zaidan-eisner-piatko:2007:MainConf,
  author    = {Zaidan, Omar  and  Eisner, Jason  and  Piatko, Christine},
  title     = {Using ``Annotator Rationales'' to Improve Machine Learning for Text Categorization},
  booktitle = {NAACL HLT 2007; Proceedings of the Main Conference},
  month     = {April},
  year      = {2007},
  pages     = {260--267}
}


NOTE: I'm still going to add more stuff/documentation, especially regarding the annotation
process, but feel free to download and use the dataset, which is availabe (clearly) and has been
finalized (except for some more documentation maybe).