Offensive and defensive strategy of web crawler

Yuanshu Jiang; Wenzhong Tang; Liyong Guo

doi:10.1109/WCICA.2012.6357898

Offensive and defensive strategy of web crawler

Yuanshu Jiang^*
, Wenzhong Tang
, Liyong Guo

^*Corresponding author for this work

School of Computer Science and Engineering

Beihang University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

Crawling strategies of web crawler affect not only the quality of search engine, but also the working status of web server. Many web servers restrict the access of unknown crawler or the crawler with excessive visiting frequency. This paper analyzes these restrictions and proposes a strategy of proxybased, login by simulating verification code automatically; give some guidance on the design of web crawler.

Original language	English
Title of host publication	WCICA 2012 - Proceedings of the 10th World Congress on Intelligent Control and Automation
Pages	355-358
Number of pages	4
DOIs	https://doi.org/10.1109/WCICA.2012.6357898
State	Published - 2012
Event	10th World Congress on Intelligent Control and Automation, WCICA 2012 - Beijing, China Duration: 6 Jul 2012 → 8 Jul 2012

Publication series

Name	Proceedings of the World Congress on Intelligent Control and Automation (WCICA)

Conference

Conference	10th World Congress on Intelligent Control and Automation, WCICA 2012
Country/Territory	China
City	Beijing
Period	6/07/12 → 8/07/12

Keywords

proxy server
recognition of verification code
web crawler

Access to Document

10.1109/WCICA.2012.6357898

Cite this

@inproceedings{81588a97649b4d01979ddd1f0bfa2ae2,

title = "Offensive and defensive strategy of web crawler",

abstract = "Crawling strategies of web crawler affect not only the quality of search engine, but also the working status of web server. Many web servers restrict the access of unknown crawler or the crawler with excessive visiting frequency. This paper analyzes these restrictions and proposes a strategy of proxybased, login by simulating verification code automatically; give some guidance on the design of web crawler.",

keywords = "proxy server, recognition of verification code, web crawler",

author = "Yuanshu Jiang and Wenzhong Tang and Liyong Guo",

year = "2012",

doi = "10.1109/WCICA.2012.6357898",

language = "英语",

isbn = "9781467313988",

series = "Proceedings of the World Congress on Intelligent Control and Automation (WCICA)",

pages = "355--358",

booktitle = "WCICA 2012 - Proceedings of the 10th World Congress on Intelligent Control and Automation",

note = "10th World Congress on Intelligent Control and Automation, WCICA 2012 ; Conference date: 06-07-2012 Through 08-07-2012",

}

Jiang, Y, Tang, W & Guo, L 2012, Offensive and defensive strategy of web crawler. in WCICA 2012 - Proceedings of the 10th World Congress on Intelligent Control and Automation., 6357898, Proceedings of the World Congress on Intelligent Control and Automation (WCICA), pp. 355-358, 10th World Congress on Intelligent Control and Automation, WCICA 2012, Beijing, China, 6/07/12. https://doi.org/10.1109/WCICA.2012.6357898

Offensive and defensive strategy of web crawler. / Jiang, Yuanshu; Tang, Wenzhong; Guo, Liyong.
WCICA 2012 - Proceedings of the 10th World Congress on Intelligent Control and Automation. 2012. p. 355-358 6357898 (Proceedings of the World Congress on Intelligent Control and Automation (WCICA)).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Offensive and defensive strategy of web crawler

AU - Jiang, Yuanshu

AU - Tang, Wenzhong

AU - Guo, Liyong

PY - 2012

Y1 - 2012

N2 - Crawling strategies of web crawler affect not only the quality of search engine, but also the working status of web server. Many web servers restrict the access of unknown crawler or the crawler with excessive visiting frequency. This paper analyzes these restrictions and proposes a strategy of proxybased, login by simulating verification code automatically; give some guidance on the design of web crawler.

AB - Crawling strategies of web crawler affect not only the quality of search engine, but also the working status of web server. Many web servers restrict the access of unknown crawler or the crawler with excessive visiting frequency. This paper analyzes these restrictions and proposes a strategy of proxybased, login by simulating verification code automatically; give some guidance on the design of web crawler.

KW - proxy server

KW - recognition of verification code

KW - web crawler

UR - https://www.scopus.com/pages/publications/84872363268

U2 - 10.1109/WCICA.2012.6357898

DO - 10.1109/WCICA.2012.6357898

M3 - 会议稿件

AN - SCOPUS:84872363268

SN - 9781467313988

T3 - Proceedings of the World Congress on Intelligent Control and Automation (WCICA)

SP - 355

EP - 358

BT - WCICA 2012 - Proceedings of the 10th World Congress on Intelligent Control and Automation

T2 - 10th World Congress on Intelligent Control and Automation, WCICA 2012

Y2 - 6 July 2012 through 8 July 2012

ER -

Offensive and defensive strategy of web crawler

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this