600.226: Data Structures (Spring 2017)

Assignment 9: Hashing Out JHUgle

Overview

The ninth assignment is a little more open-ended than what you’re used to for this course. We no longer break things down into separate problems below; the entire assignment is the problem now. This also means that the “rubric” for the assignment is the grading breakdown above, and it all applies.

Your “one” task for this assignment is to write a simplified search engine that we call JHUgle.java, subject to the specifications and constraints detailed below. Deciding what data structures and implementations to use, and implementing them as efficiently as possible, will be critical in earning maximum performance points. You are expected to do this by (once again) implementing the Map interface, this time using various hash table techniques (your choice, see below).

Important: The 20% for Performance will be awarded by evaluating your submissions according to both time and space required by your search engine. Less time is better, less space is better, and both are considered separately for 10% each. Performance points are only awarded for programs that produce correct results!

Partners

You are allowed (strongly recommended, but not strictly required) to work in pairs for this assignment. Both partners will receive the same grade. Only one student should submit the actual code zip on Blackboard, but all students must submit a text note indicating their partner’s name and Blackboard ID. Students who fail to do this risk getting a zero. Make sure that both students names are in the code that is submitted as well.

Also, late days must be available to both students if the pair submits late. We suggest finding a partner in class or on Piazza, but make sure that you both have comparable late days left in case they are needed. Otherwise the assignment must be submitted within the late allowance for the student that comes soonest.

Paired programming is an excellent technique for working with a partner: you write the code together, taking turns acting as driver (typing) and navigator (reading, watching, guiding). This is way more effective than trying to split up the work and coding separately. Also, you will want to spend some time discussing the project and coming up with a game plan before you begin. Your discussion, analysis and approach decisions must be documented in the README file you submit. You must also document how the code was developed and who contributed to which parts.

JHUgle Search Engine

The “good news” is that you get to follow in some very famous footsteps and write an entire search engine! (Well, at least a basic one…) This will require applying the knowledge and techniques you’ve been learning all semester!

Your JHUgle.java program must have a single command-line argument which is the name of a file containing a list of websites and the keywords that are contained on each site. You’ll want to use this file to construct an index that maps each keyword to a collection of URLs in which it appears. Since reading in this file and constructing the index will take significant time, your program should output “Index Created” followed by a newline once that operation completes.

The file will be structured to have the URL for each site on one line, followed by a single line containing words found on that site, separated by whitespace. Note, this could be more than one space character, or a tab! There are plenty of methods in Java’s Scanner that will help you parse these lines. (You might also want to mimic the Words.java approach from the last assignment to use a BufferedReader with the string split method for better performance!) There will be multiple URLs in the file, alternating on each line between the URL and the word list. These files could be many many many lines long, but a very short example named urls.txt might look as follows:

http://www.cars.com/
car red truck wheel blue fast brown silver gray
https://en.wikipedia.org/wiki/Cat
cat feline small fur red brown fast gray
http://www.foobar.com/baz
foo bar baz lorem ipsum red blue yellow silver

The end-goal of this program is to provide a way for the user to ask for the set of URLs that contain some set of words. The sets of words will be specified as a logical expression. In the example above, if I were to ask for the pages containing “red AND fast”, JHUgle should return the first two URLs, but not the third one, as it does not contain “fast” even though it does contain “red”. Similarly, if I were to ask for “car OR baz”, the second URL would be omitted, as it contains neither of these words. Note that queries may contain many operands (words) and operators (AND, OR).

To make the input simpler, we will be specifying these queries in postfix notation similar to Assignment 4. We will also use symbols for the operations rather than words so that the search engine can differentiate words in queries from the operations. The queries above would be given as red fast && and car baz || respectively. The main JHUgle program will read queries as one word or logical operation at a time from standard input, in response to a > prompt.

There are two more operations your JHUgle implementation will need to handle. The first is ?, which requires your program to print the URLs corresponding to the expression at the top of the query stack, one per line with no additional formatting. Your program will loop until the quit operation is given, an !. When the user quits, simply exit the program without producing any additional output.

Here is simple sample run that demonstrates all aspects of the program operation, with results based on the short sample input file above.

$ java JHUgle urls.txt
Index Created
> ?
> baz
> red
> ?
http://www.cars.com/
https://en.wikipedia.org/wiki/Cat
http://www.foobar.com/baz
> &&
> ?
http://www.foobar.com/baz
> !
$

You can see how quickly your JHUgle program works using the xtime program from previous assignments. To do this you’ll want to make a file that contains words and commands in response to the prompts that can be used to run the program with standard input redirection. For example, if you put the above input sequence (? baz red ? && ? !) in a plain text file called ops.txt, you can time and run the program using:

$ ./xtime java JHUgle urls.txt <ops.txt >output.txt

We will make some small and large URL input files available on Piazza for you to use in testing your application. You might want to create a first iteration of this program using one of the Map implementations that we previously provided, or that you have from former assignments.

The Hash Table

You are expected to achieve excellent performance in JHUgle.java by developing and comparing various hashing techniques. Name your best version that is used in the JHUgle program HashMap.java. Obviously your HashMap.java must implement the Map interface, but beyond that you have quite a few options for how to proceed:

Make sure that you include extensive comments at the start of your HashMap implementation to clarify what type of collision resolution strategy it implements. This should go without saying, but you’ll need “all the usual pieces” for your HashMap implementation. In particular:

And that’s it. Yes, you’re really on your own for figuring out what kind of hash tables you should implement, and which one to use in the end and how to effectively do so in order to implement the search engine.

All critical map operations, except insert, must run in O(1) expected time (or better); insert can run in O(1) amortized time in case you have to grow the bucket array table to keep the load factor down.

Got Extra Hash Tables?

Depending on just how serious you are about those Performance points, you may well end up writing multiple different hash tables over the course of this assignment. However, you have to pick one of those to use in JHUgle.java, named HashMap.java. You are welcome to submit other implementations as well, each named accordingly to indicate what type of hash table technique it uses. Include all the benchmarking data, results and analysis that contributed to your final decision on which implementation to use for the search engine.

Iterative Development

You cannot know how fast your HashMap is until it’s actually written. You cannot improve your HashMap until you can tell how fast it is. So the worst mistake you can make is to “think about it” for days without writing any code. (Thinking ahead is good in principle, thinking ahead for too long is the problem here.)

We recommend you start right now by writing the simplest HashMap you can think of and making that work. For example you could write one based on separate chaining but with a fixed array size.

You want your test cases and benchmarks in place before you keep going. Make sure that your test cases are complete and that your benchmarks tell you how well the various Map operations work for that first version of HashMap. You should probably save a backup (or even submit early!) as soon as you get done with the first round.

From then on, it’s “try to improve things” followed by “see if the tests still pass” followed by “benchmark to see if things actually got better” followed by either “Woops, that was a bad idea, let’s undo that” or “Yay, I made progress, let’s save a backup of the new version” and so on and so forth. We predict that there will be a correlation between how well you do and how often you “went around” this iterative development cycle.

What Classes Are Allowed?

You may not use java.util.HashMap or java.util.LinkedHashMap to implement your HashMaps and you may not use those classes or java.util.HashSet in your JHUgle solution either. You also may not use other map implementations (yours or Java’s) as part of your own HashMap classes. However, in order to write the JHUgle.java application you may (and are expected to) reuse other interfaces and classes that have been provided, that you developed for other assignments, or from the Java library.

If in doubt about what is permitted, better to ask on Piazza first! You don’t want to find out minutes before the deadline that you used something that’s not okay…

What about the README?

You should use your README file to explain how you approached this assignment, what you did in what order, how it worked out, how your plans changed, etc. Try to summarize all the different ways you developed, evaluated, and improved your JHUgle application and various Hashmaps over time. If you don’t have a story to tell here, you probably didn’t do enough…

Random Hints

Deliverables

You must turn in a zipped (.zip only) archive containing all source code, your README file, and any other deliverables required by the assignment. The filename should be HW##-jhed.zip with ## replaced by the 2-digit number (use leading 0s) of this assignment (see above) and jhed replaced by your Blackboard login. (For example, Peter would use HW03-pfroehl1.zip for his submission of Assignment 3.) The zip should contain no derived files whatsoever (i.e. no .class files, no .html files, etc.), but should allow building all derived files. Include a plain text README file (not README.txt or README.docx or whatnot) that briefly explains what your programs do and contains any other notes you want us to check out before grading. Your answers to written problems should be in this README file as well. Finally, make sure to include your name and email address in every file you turn in (well, in every file for which it makes sense to do so anyway)!

Grading

For reference, here is a short explanation of the grading criteria; some of the criteria don't apply to all problems, and not all of the criteria are used on all assignments.

Packaging refers to the proper organization of the stuff you hand in, following both the guidelines for Deliverables above as well as the general submission instructions for assignments.

Style refers to Java programming style, including things like consistent indentation, appropriate identifier names, useful comments, suitable javadoc documentation, etc. Many aspects of this are enforced automatically by Checkstyle when run with the configuration file available on Piazza. Style also includes proper modularization of your code (into interfaces, classes, methods, using public, protected, and private appropriately, etc.). Simple, clean, readable code is what you should be aiming for.

Testing refers to proper unit tests for all of the data structure classes you developed for this assignment, using the JUnit 4 framework as introduced in lecture. Make sure you test all (implied) axioms that you can think of and all exception conditions that are relevant.

Performance refers to how fast/with how little memory your program can produce the required results compared to other submissions.

Functionality refers to your programs being able to do what they should according to the specification given above; if the specification is ambiguous and you had to make a certain choice, defend that choice in your README file.

If your programs cannot be built you will get no points whatsoever. If your programs cannot be built without warnings using javac -Xlint:all we will take off 10% (except if you document a very good reason; no, you cannot use the @SuppressWarnings annotation either). If your programs fail miserably even once, i.e. terminate with an exception of any kind, we will take off 10% (however we'll also take those 10% off if you're trying to be "excessively smart" by wrapping your whole program into a universal try-catch).