Open Access Open Access  Restricted Access Subscription or Fee Access

Scrapple: a Flexible Framework to Develop Semi-Automatic Web Scrapers

(*) Corresponding author

Authors' affiliations



The World Wide Web is the biggest source of data that the general public has access to. Students and researchers working on data-related problems need a well-maintained source to get their data from. Most online services provide APIs to use their service. However, this may not provide all the data that they actually need even though it exists on the website. In this paper, "Scrapple" – a framework for creating semi-automatic web scrapers, which extract required data from the Web, is introduced. Users do not need to write the entire scraping programs - they only need to define a configuration file which is used to build the required scraper. The configuration provides support for CSS selectors and XPath expressions
Copyright © 2015 Praise Worthy Prize - All rights reserved.


Web Scraping; Web Wrapper; CSS Selector; XPath Expressions

Full Text:



Kei Kanaoka, YotaroFujii and Motomichi Toyama.Ducky: A Data Extraction Systemfor Various Structured Web Documents. In Proceedings of the 18th International Database Engineering & Applications Symposium, IDEAS ’14, pages 342-347, New York, NY, USA, 2014. ACM

Selectors.[Online] Available:

W3 Document Object Model.[Online] Available:

XPath support in ElementTree[Online] Available:

T. Furche, G. Gottlob, G. Grasso, C. Schallhart, and A. Sellers.Oxpath: A language for scalable data extraction, automation, and crawling on the deep web. The VLDB Journal, 22(1):47–72, Feb. 2013

V. Crescenzi, P. Merialdo, and D. Qiu. Alfred: Crowd assisted data extraction. In Proceedings of the 22Nd International Conference on World Wide Web Companion, WWW ’13 Companion, pages 297–300, Republic and Canton of Geneva, Switzerland, 2013. International World Wide Web Conferences Steering Committee.

F. Kokkoras, K. Ntonas, and N. Bassiliades.Deixto: A web data extraction suite. In Proceedings of the 6th Balkan Conference in Informatics, BCI ’13, pages 9–12, New York, NY, USA, 2013. ACM.

H. A. Sleiman and R. Corchuelo, "A survey on region extractors from web documents", IEEE Trans. Knowl. Data Eng., vol. 25, no. 9, pp.1960 -1981, July 2012

Scrapy: A fast and powerful scraping and web crawling framework. [Online] Available:

Boris Katz , Sue Felshin , DenizYuret , Ali Ibrahim , Jimmy Lin , Gregory Marton , Alton Jerome McFarland , BarisTemelkuran. Omnibase: Uniform Access to Heterogeneous Data for Question Answering. In Proceedings of the 7th International Workshop on Applications Of Natural Language To Information Systems. NLDB 2002.Stockholm. Sweden. June 2002.

lxml - Processing XML and HTML with Python [Online]. Available:

BeautifulSoup vs. lxml benchmark.[Online] Available:


  • There are currently no refbacks.

Please send any question about this web site to
Copyright © 2005-2024 Praise Worthy Prize