The Academic Web Link Database Project

Making available databases of academic web links to the world research community

This project was created in response to the need for research into web links: including web link mining, and the creation of link metrics. It is aimed at providing the raw data and software for researchers to analyse link structures without having to rely upon commercial search engines, and without having to run their own web crawler. This site will contain all of the following.

• Complete databases of link structures of collections of academic web sites.

• Files of summary statistics about the link databases.

• Software tools for researchers to extract the information that they are particularly interested in.

• Descriptions of the methodologies used to crawl the web so that the information provided can be critically evaluated.

• Files of information used in the web crawling process.

Slow Internet connection? We will send researchers the databases for free upon receipt of a self-addressed (unstamped) parcel containing an empty CD-case. We will pay postage and supply the CD without charge (it would be too little money to bother with anyway).

Databases - Tools for mining the data - Crawling methodology - Web link research - Research group

Database 9: UK university web sites June-July, 2002

• Full database of the link structure of 111 UK university institutions, as of June-July, 2002 (Huge file, 300Mb). Zip file needs an Unzip program to extract the link files. Expect to need 4Gb of disk space to extract this file and work with it.
• The list of the known domain names of the universities covered. University name list.
• List of URLs and partial URLs that were ignored by the crawler.
• Files of summary statistics will be added.

Database 8: Taiwan university web sites July, 2002

• Full database of the link structure of 46 Taiwan universities, as of July, 2002 (Huge file, 54Mb). Zip file needs an Unzip program to extract the link files. Expect to need 5Gb of disk space to extract this file and work with it.
• The list of the known domain names of the universities covered.
• Files of summary statistics will be added.

Database 7: Mainland China university web sites July, 2002

• Full database of the link structure of 76 UK universities, as of December, 2001 - January, 2002 (Huge file, 240Mb). Self-extracting zip file: run the program to extract the link files. Expect to need 3Gb of disk space to extract this file and work with it.
• The list of the known domain names of the universities covered.
• Files of summary statistics will be added.

Database 6: New Zealand university web sites January 2002 to February 2002

• Full database of the link structure of all 8 New Zealand universities, as of January 2002- February 2002 (Big file, 11Mb). Self-extracting zip file: run the program to extract the link files. Expect to need 0.5Gb of disk space to extract this file and work with it. This crawl was completed on the 6th February 2002 and posted here on the same day.
• The list of the known domain names of the universities covered.
• List of URLs and partial URLs that were ignored by the crawler. University name list.
• Files of summary statistics will be added.

Database 5: Australian university web sites October, 2001 to January 2002 (slow crawl)

• Full database of the link structure of 38 Australian universities, as of October, 2001 to January 2002 (Huge file, 98Mb). Self-extracting zip file: run the program to extract the link files. Expect to need 2.5Gb of disk space to extract this file and work with it. This crawl was completed on the 30th January 2002 and posted here the next day.
• The list of the known domain names of the universities covered. University name list.
• List of URLs and partial URLs that were ignored by the crawler.
• Files of summary statistics will be added.

Database 4: UK university web sites July, 2001

• Full database of the link structure of 109 UK universities, as of July, 2001 (Huge file, 195Mb). Self-extracting zip file: run the program to extract the link files. Expect to need 3Gb of disk space to extract this file and work with it.
• The list of the known domain names of the universities covered. University name list.
• List of URLs and partial URLs that were ignored by the crawler.
• Files of summary statistics will be added.

Database 3: UK university web sites June-July, 2000

• Full database of the link structure of 107 UK universities and similar institutions, as of July, 2000 (Huge file, 46Mb). Self-extracting zip file: run the program to extract the link files. Expect to need 850Mb of disk space to extract this file and work with it.
• A summary file containing counts of links to external URLs from each British university web site (Very big file, 14Mb).
• This data file is less reliable than the 2001 version because of one domain name root only per institution. This does not impact on places like Wolverhampton much (almost all pages use wlv.ac.uk) but would more on Sussex (a lot of pages use both sussex.ac.uk and susx.ac.uk). University name list.
• At the time of collection of this data it was not intended to make it publicly available, so is presented here 'as is' and has not been cleaned up to remove universities that were subsequently not used in any analysis, e.g. the federal University of Wales. The ignored URLs list is believed to have been lost but will be posted if found.

Database 2: New Zealand university web sites July-August, 2000

• Full database of the link structure of all NZ universities, as of July-August, 2000 (Big file, 3.3Mb). Self-extracting zip file: run the program to extract the link files. Expect to need 100Mb of disk space to extract this file and work with it.
• A summary file containing counts of links to external URLs from each New Zealand university web site (940k).
• This data file is less reliable than the 2001 version because of one domain name root only per institution. University name list.
• At the time of collection of this data it was not intended to make it publicly available, so is presented here 'as is' and has not been cleaned up. The ignored URLs list is believed to have been lost but will be posted if found.

Database 1: Australian university web sites July-August, 2000

• Full database of the link structure of all known Australian universities, as of July-August, 2000 (Very big file, 13.5Mb). Self-extracting zip file: run the program to extract the link files. Expect to need 300Mb of disk space to extract this file and work with it.
• A summary file containing counts of links to external URLs from each Australian university web site (Big file, 5Mb).
• This data file is less reliable than the 2001 version because of one domain name root only per institution. University name list.
• At the time of collection of this data it was not intended to make it publicly available, so is presented here 'as is' and has not been cleaned up. The ignored URLs list is believed to have been lost but will be posted if found.

Tools

These programs should run on most versions of Windows. Please email if there is any problem. Some of the programs may take a long time to run (days if you have a slow computer and are processing the large database files). Expect a more comprehensive collection of tools soon. We are sorry for the awful interfaces provided on the programs but are happy to advise researchers on which programs will be useful to conduct the type of analysis that they are interested in.

• A program to run on the link structure files to extract subsites. For example, you can run this program to extract just the pages from the scit.wlv.ac.uk domain from the wlv.ac.uk file.
• A program to extract summary statistics about counts of links between two sites covered by the database.

Description of Crawling Methodology

A link to an online journal article is expected shortly, based on this preprint. Additional crawling issues and techniques are discussed in the following article.

Thelwall, M. (2001) A Web Crawler Design for Data Mining, Journal of Information Science 27(5), 319-326.

Web Link Research

For our publications, please see the Statistical Cybermetrics Research Group home page. There is a large list of related work available on the web site of the e-journal Cybermetrics. A much bigger Unix-based archive that is similar in spirit is available at http://www.archive.org/.

About this project

This project is run by the Statistical Cybermetrics Research Group at the University of Wolverhampton. We do not charge for any of the data or tools placed here because we feel that we have an obligation to make our raw data available for free since we collected it for free from the Web sites covered. The crawling is resource intensive and time-consuming so we are unfortunately not able to respond to requests such as "please crawl country X". If any bodies, such as national research agencies, would like to see their countries' universities included, then this will involve a charge. We would expect, but not insist, that the data resulting from such an arrangement would be subsequently made available on this site, also without charge. We are currently bidding for funding for Web link mining research projects that involve crawling countries and expect that this site will grow as a result.

For more information or to notify errors please email m.thelwall@wlv.ac.uk