爬虫框架分类 1. 分布式爬虫
Nutch
2. Java单机爬虫
Crawler4j、WebMagic、WebCollector
3. 非Java单机爬虫
Scrapy
开发思路 根据业务需求选择合适的爬虫框架
根据网站规则及业务需求抽取数据,保存到中间库
数据清洗/格式化,保存到目标库
基于Crawler4j + jsoup实现爬虫 用Crawler4j构建多线程的web爬虫来抓取页面内容。Crawler4j的使用分为两个步骤:
(1) 实现一个继承自WebCrawler的爬虫类
需要覆盖两个主要方法:
shouldVisit:这个方法决定了要抓取的URL及其内容。
visit:当URL下载完成会调用这个方法。使用jsoup解析HTML,可以采用jQuery选择器的语法。
(2) 通过CrawController调用实现的爬虫类。
指定抓取的种子(seed)、中间数据存储的文件夹、并发线程的数目等信息,实现控制器类。
示例代码
pom.xml中依赖包设置
edu.uci.ics
crawler4j
4.2
org.jsoup
jsoup
1.10.1
代码
(1) 爬虫类
public class MyCrawler extends WebCrawler {
private final static Pattern FILTERS = Pattern.compile(".*(\\.(css|js|gif|jpg"
+ "|png|mp3|mp3|zip|gz))$");
/**
* This method receives two parameters. The first parameter is the page
* in which we have discovered this new url and the second parameter is
* the new url. You should implement this function to specify whether
* the given url should be crawled or not (based on your crawling logic).
* In this example, we are instructing the crawler to ignore urls that
* have css, js, git, ... extensions and to only accept urls that start
* with "http://www.ics.uci.edu/". In this case, we didn't need the
* referringPage parameter to make the decision.
*/
@Override
public boolean shouldVisit(Page referringPage, WebURL url) {
String href = https://www.it610.com/article/url.getURL().toLowerCase();
return !FILTERS.matcher(href).matches()
&& href.startsWith("http://www.ics.uci.edu/");
}
/**
* This function is called when a page is fetched and ready
* to be processed by your program.
*/
@Override
public void visit(Page page) {
String url = page.getWebURL().getURL();
System.out.println("URL: " + url);
if (page.getParseData() instanceof HtmlParseData) {
HtmlParseData htmlParseData = https://www.it610.com/article/(HtmlParseData) page.getParseData();
String text = htmlParseData.getText();
String html = htmlParseData.getHtml();
Set links = htmlParseData.getOutgoingUrls();
System.out.println("Text length: " + text.length());
System.out.println("Html length: " + html.length());
System.out.println("Number of outgoing links: " + links.size());
}
}
}
(2) Controller调用
public class Controller {
public static void main(String[] args) throws Exception {
String crawlStorageFolder = "/data/crawl/root";
int numberOfCrawlers = 7;
CrawlConfig config = new CrawlConfig();
config.setCrawlStorageFolder(crawlStorageFolder);
/*
* Instantiate the controller for this crawl.
*/
PageFetcher pageFetcher = new PageFetcher(config);
RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);
/*
* For each crawl, you need to add some seed urls. These are the first
* URLs that are fetched and then the crawler starts following links
* which are found in these pages
*/
controller.addSeed("http://www.ics.uci.edu/~lopes/");
controller.addSeed("http://www.ics.uci.edu/~welling/");
controller.addSeed("http://www.ics.uci.edu/");
/*
* Start the crawl. This is a blocking operation, meaning that your code
* will reach the line after this only when crawling is finished.
*/
controller.start(MyCrawler.class, numberOfCrawlers);
}
}
【基于Crawler4j + jsoup实现爬虫】
推荐阅读
- activemq笔记
- MyEclipse中防止代码格式化时出现换行的情况的设置
- JAVA 8 lambda 表达式实现按某个属性值查找对象集合中符合条件的对象
- tomcat启动时报invalid LOC header (bad signature)错误
- docker中运行zabbix/zabbix-web-nginx-mysql镜象,前端页面显示时间不准确的问题(差5个小时)
- java.lang.NoSuchMethodError: 'feign.Request$Body feign.Request.requestBody()
- maven must be "pom" but is "jar" 填坑记
- npm 报错(permission denied Please try running this command again as root/Administrator.)
- IDEA中Lombok使用