A scalable crawler framework. It covers the whole lifecycle of crawler: downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler.
Features:
- Simple core with high flexibility.
- Simple API for html extracting.
- Annotation with POJO to customize a crawler, no configuration.
- Multi-thread and Distribution support.
- Easy to be integrated.
Install:
Add dependencies to your pom.xml:
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-core</artifactId>
<version>0.7.4</version>
</dependency>
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-extension</artifactId>
<version>0.7.4</version>
</dependency>
WebMagic use slf4j with slf4j-log4j12 implementation. If you customized your slf4j implementation, please exclude slf4j-log4j12.
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
Get Started:
First crawler:
Write a class implements PageProcessor. For example, I wrote a crawler of github repository infomation.
public class GithubRepoPageProcessor implements PageProcessor {
private Site site = Site.me().setRetryTimes(3).setSleepTime(1000);
@Override
public void process(Page page) {
page.addTargetRequests(page.getHtml().links().regex("(https://github\\.com/\\w+/\\w+)").all());
page.putField("author", page.getUrl().regex("https://github\\.com/(\\w+)/.*").toString());
page.putField("name", page.getHtml().xpath("//h1[@class='public']/strong/a/text()").toString());
if (page.getResultItems().get("name")==null){
//skip this page
page.setSkip(true);
}
page.putField("readme", page.getHtml().xpath("//div[@id='readme']/tidyText()"));
}
@Override
public Site getSite() {
return site;
}
public static void main(String[] args) {
Spider.create(new GithubRepoPageProcessor()).addUrl("https://github.com/code4craft").thread(5).run();
}
}
-
page.addTargetRequests(links)Add urls for crawling.
You can also use annotation way:
@TargetUrl("https://github.com/\\w+/\\w+")
@HelpUrl("https://github.com/\\w+")
public class GithubRepo {
@ExtractBy(value = "//h1[@class='public']/strong/a/text()", notNull = true)
private String name;
@ExtractByUrl("https://github\\.com/(\\w+)/.*")
private String author;
@ExtractBy("//div[@id='readme']/tidyText()")
private String readme;
public static void main(String[] args) {
OOSpider.create(Site.me().setSleepTime(1000)
, new ConsolePageModelPipeline(), GithubRepo.class)
.addUrl("https://github.com/code4craft").thread(5).run();
}
}
Docs and samples:
Documents: http://webmagic.io/docs/
The architecture of webmagic (refered to Scrapy)
There are more examples in webmagic-samples package.
Lisence:
Lisenced under Apache 2.0 lisence
Thanks:
To write webmagic, I refered to the projects below :
-
Scrapy
A crawler framework in Python.
-
Spiderman
Another crawler framework in Java.
Mail-list:
https://groups.google.com/forum/#!forum/webmagic-java
http://list.qq.com/cgi-bin/qf_invite?id=023a01f505246785f77c5a5a9aff4e57ab20fcdde871e988
QQ Group: 373225642 542327088
Related Project
-
A web console based on WebMagic for Spider configuration and management.
![[Bug] CountableThreadPool 类中 exec() 中是否存在线程安全问题](https://avatars.githubusercontent.com/u/30571851?v=4)
事实上我是每半小时从这个页面取到最热门的几条,我希望这个入口不会被过滤,但其他内容页的url照常过滤,应该怎么做啊?
2.5k Dec 31, 2022
4.3k Jan 3, 2023
20 Aug 22, 2020
1.6k Jan 4, 2023
69 Nov 27, 2022
9.9k Jan 4, 2023
1.2k Dec 21, 2022
3 Dec 2, 2021
13 Apr 15, 2022
5 Oct 14, 2022
396 Dec 17, 2022
776 Jan 2, 2023
6.5k Jan 4, 2023
1k Jan 1, 2023
450 Dec 23, 2022
289 Dec 18, 2022
939 Dec 23, 2022