1 Task

You need to help Prof. Wiegley find the perfect rug for his office. (if he was allowed to have a rug that is. Unfortunately only big-wigs get carpeting in their office. *sigh*...)

There exists an online area rug store called http://www.arearugs2u.com/ that has a wide selection of area rugs but their search abilities are atrocious.

We don’t want to hammer that site though because we aren’t going to actually buy anything. Instead, use

http://rugs.cyte.com/

(Don’t attempt any other shenanigans against this site. It runs other services that other companies depend on for survival. And, on a teacher’s salary, I can only afford pitiful bandwidth.)

You are to write a spider program, written in Perl, that will search through the http://rugs.cyte.com/area-rugs.html site and builds up a list of standard SQL queries that will build and populate a database that can be searched for the perfect rugs.

The schema for the database consists of a single table named “rugs”. The SQL CREATE commands for properly creating this schema is:

DROP TABLE IF EXISTS rugs;       CREATE TABLE IF NOT EXISTS rugs (         url VARCHAR(256) NOT NULL,         collection VARCHAR(256) NOT NULL,         model VARCHAR(256) NOT NULL,         thickness INT NOT NULL,         density INT NOT NULL,         width INT NOT NULL,         length INT NOT NULL,         cost DECIMAL(7,2) NOT NULL       );

For instance, one insert that should be generated from the page

http://rugs.cyte.com/aubusson/493-Cannes.html

would be:

INSERT INTO rugs         (url, collection, model, thickness, density, width, length, cost)       VALUES         (           ’http://rugs.cyte.com/aubusson/493-Cannes.html’,           ’Aubusson’, ’493 Cannes’,           ’4’, ’5’, ’60’, ’96’, ’817.97’         );

More inserts from this page would, of course, be generated. One for each of the sizes available. They would all share the same thickness and density as this information is obtained from the parent page that leads to the example given.

The last SQL command output should be:

SELECT url FROM rugs     WHERE width>’96’ AND length>’120’ AND density>=’4’ AND COST<’600.00’     ORDER BY url;

2 Specifications

• You may work in teams of up to two members. (You may work alone if you desire to.) Team members should have full knowledge of how the project works because there is high probability that a more complicated Perl question related to this assignment will appear on the final exam.
• To stdout, spew only SQL commands. stderr can be used for anything else you wish. But the instructor is going to feed stdout directly to a MySQL client and any errors will cause decrease in points from your grade.
• You must include the above SQL commands to drop and create the table prior to outputting any INSERT commands.
• Ignore any rug combinations for which all fields cannot be determined. For example: if the price is something like “call for best price” then treat that rug as though it did not exist. (Prof. Wiegley hates having to contact people for information that should have been present in the first place. grrrrr.)
• All possible rug combinations must be accounted for.
• You must assume that the available models, collections, manufacturers, sizes and costs are subject to change at any time. (I.e. you cannot just generate a static list of INSERT commands. You must obtain the list dynamically through an active spider-type search.)
• You do not have to write a full “spider”. You can assume the following: (The first relevant link possible on each page was select as the examples given)
  • All rugs can be found from the http://rugs.cyte.com/area-rugs.html URL.
  • That you only need to spider the manufacturers.
  • That clicking on a manufacturer (e.g. http://rugs.cyte.com/mohawk-select.html ) brings up a page with all the collections for the manufacturer.
  • Clicking on a collection (e.g. http://rugs.cyte.com/antique/mohawk-select-antique-area-rugs.html) takes you to a page that has the thickness and density (e.g. 3 and 3 for the example) for that collection and all models for the collection.
  • Clicking on a model (e.g. http://rugs.cyte.com/antique/AutumnFoliage-Spice.html) leads you to a page containing the available sizes and costs for that model. This page would generate three entries since it has three sizes.
• A good Perl package to use is “LWP” which stands for “Library for WWW in Perl” it can be used for efficient browser-like behaviors in Perl. A lot of information can be found on the web. But basically if you want to grab a URL stored in the Perl scalar variable $url then:

  # Early in your program:       use LWP; # Loads all important LWP classes, and makes              #  sure your version is reasonably recent.       my $browser = LWP::UserAgent->new;       ...       # Then later, whenever you need to make a get request:     my $url = ’http://freshair.npr.org/dayFA.cfm?todayDate=current’;       my $response = $browser->get( $url );     die "Can’t get $url -- ", $response->status_line      unless $response->is_success;       die "Hey, I was expecting HTML, not ", $response->content_type      unless $response->content_type eq ’text/html’;        # or whatever content-type you’re equipped to deal with       # Otherwise, process the content somehow:       if($response->content =~ m/jazz/i) {       print "They’re talking about jazz today on Fresh Air!\n";     } else {       print "Fresh Air is apparently jazzless today.\n";     }

For a quick tutorial a good site is http://www.perl.com/pub/a/2002/08/20/perlandlwp.html. It is the instructor’s belief that the entire contents of the O’Reilly book “Perl & LWP” is available online for free through O’Reilly’s “Safari” service as long as you access Safari from a campus IP (i.e. use the campus VPN or be on campus).

You’ll probably need to install the LWP Perl modules.

If you are on a Linux distribution then your distribution probably has a package for LWP. (there is another “LWP” package for providing “light weight” processes. Don’t install that! Make sure you are getting the Library for WWW in Perl package.) Under Gentoo, Debian or Ubuntu the libwww-perl package is what you want. Under RedHat/Fedora it is the perl-libwww package.

If you fail to having a pre-built package, Perl has a system called the Comprehensive Perl Archive Network (CPAN). CPAN is an semi-automatic system that is platform independent. It’s sort of like a mini-distribution system that takes care of just Perl in a platform independent sort of way. Most Perl installations have CPAN installed by default so to install LWP you should be able to just do:

perl -MCPAN -e ’install Bundle::LWP’

Required dependencies and such will be pulled in and installed.

3 Deliverables

Email your files to jeffw@csun.edu

Files should be attached as individual attachments to the message.

Include the string “[COMP421] Project 2” in the subject line.