Adaptive technique for web page change detection using multi-threaded crawlers

Lakmal Meegahapola, Roshan Alwis, Eranga Heshan, Vijini Mallawaarachchi, Dulani Meedeniya, Sampath Jayarathna

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

3 Citations (Scopus)

Abstract

World Wide Web is getting dense as many new web pages and resources are created on a daily basis. Keeping track of the changes in the web content has become an immense challenge and is a research problem with a great significance. Even the search engines require to detect changes in the web to keep search indexes up to date. Numerous researches have been carried out on optimizing the change detection algorithms. This paper presents a methodology named Multi-Threaded Crawler for Change Detection of Web (MTCCDW), which is inspired from the producer-consumer problem. The suggested change detection process mainly analyses the performances and suggests a tread-based implementation process for the optimisation of the changed detection process. The experimental results show that the proposed methodology is capable of reducing the effective time to detect changes in a web page by 93.51%.

Original languageEnglish
Title of host publication7th International Conference on Innovative Computing Technology, INTECH 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages120-125
Number of pages6
ISBN (Electronic)9781509039883
DOIs
Publication statusPublished - 9 Nov 2017
Externally publishedYes
Event7th International Conference on Innovative Computing Technology, INTECH 2017 - Luton, United Kingdom
Duration: 16 Aug 201718 Aug 2017

Publication series

Name7th International Conference on Innovative Computing Technology, INTECH 2017

Conference

Conference7th International Conference on Innovative Computing Technology, INTECH 2017
Country/TerritoryUnited Kingdom
CityLuton
Period16/08/1718/08/17

Keywords

  • change detection
  • multi-threading
  • producer-consumer problem
  • web crawling
  • web page change detection

Fingerprint

Dive into the research topics of 'Adaptive technique for web page change detection using multi-threaded crawlers'. Together they form a unique fingerprint.

Cite this