锐客网

  1. 首页 > 睿知 > it技术 > >

核心代码java 核心代码怎么找

applicationIT技术

Java网络爬虫怎么实现？网络爬虫是一个自动提取网页的程序，它为搜索引擎从万维网上下载网页，是搜索引擎的重要组成 。\x0d\x0a传统爬虫从一个或若干初始网页的URL开始，获得初始网页上的URL，在抓取网页的过程中 ， 不断从当前页面上抽取新的URL放入队列，直到满足系统的一定停止条件 。对于垂直搜索来说，聚焦爬虫 ， 即有针对性地爬取特定主题网页的爬虫，更为适合 。\x0d\x0a\x0d\x0a以下是一个使用java实现的简单爬虫核心代码：\x0d\x0apublic void crawl() throws Throwable {\x0d\x0awhile (continueCrawling()) {\x0d\x0aCrawlerUrl url = getNextUrl(); //获取待爬取队列中的下一个URL\x0d\x0aif (url != null) {\x0d\x0aprintCrawlInfo();\x0d\x0aString content = getContent(url); //获取URL的文本信息\x0d\x0a\x0d\x0a//聚焦爬虫只爬取与主题内容相关的网页，这里采用正则匹配简单处理\x0d\x0aif (isContentRelevant(content, this.regexpSearchPattern)) {\x0d\x0asaveContent(url, content); //保存网页至本地\x0d\x0a\x0d\x0a//获取网页内容中的链接，并放入待爬取队列中\x0d\x0aCollection urlStrings = extractUrls(content, url);\x0d\x0aaddUrlsToUrlQueue(url, urlStrings);\x0d\x0a} else {\x0d\x0aSystem.out.println(url" is not relevant ignoring ...");\x0d\x0a}\x0d\x0a\x0d\x0a//延时防止被对方屏蔽\x0d\x0aThread.sleep(this.delayBetweenUrls);\x0d\x0a}\x0d\x0a}\x0d\x0acloseOutputStream();\x0d\x0a}\x0d\x0aprivate CrawlerUrl getNextUrl() throws Throwable {\x0d\x0aCrawlerUrl nextUrl = null;\x0d\x0awhile ((nextUrl == null)(!urlQueue.isEmpty())) {\x0d\x0aCrawlerUrl crawlerUrl = this.urlQueue.remove();\x0d\x0a//doWeHavePermissionToVisit：是否有权限访问该URL，友好的爬虫会根据网站提供的"Robot.txt"中配置的规则进行爬取\x0d\x0a//isUrlAlreadyVisited：URL是否访问过，大型的搜索引擎往往采用BloomFilter进行排重，这里简单使用HashMap\x0d\x0a//isDepthAcceptable：是否达到指定的深度上限 。爬虫一般采取广度优先的方式 。一些网站会构建爬虫陷阱（自动生成一些无效链接使爬虫陷入死循环） ， 采用深度限制加以避免\x0d\x0aif (doWeHavePermissionToVisit(crawlerUrl)\x0d\x0a(!isUrlAlreadyVisited(crawlerUrl))\x0d\x0aisDepthAcceptable(crawlerUrl)) {\x0d\x0anextUrl = crawlerUrl;\x0d\x0a// System.out.println("Next url to be visited is "nextUrl);\x0d\x0a}\x0d\x0a}\x0d\x0areturn nextUrl;\x0d\x0a}\x0d\x0aprivate String getContent(CrawlerUrl url) throws Throwable {\x0d\x0a//HttpClient4.1的调用与之前的方式不同\x0d\x0aHttpClient client = new DefaultHttpClient();\x0d\x0aHttpGet httpGet = new HttpGet(url.getUrlString());\x0d\x0aStringBuffer strBuf = new StringBuffer();\x0d\x0aHttpResponse response = client.execute(httpGet);\x0d\x0aif (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {\x0d\x0aHttpEntity entity = response.getEntity();\x0d\x0aif (entity != null) {\x0d\x0aBufferedReader reader = new BufferedReader(\x0d\x0anew InputStreamReader(entity.getContent(), "UTF-8"));\x0d\x0aString line = null;\x0d\x0aif (entity.getContentLength()0) {\x0d\x0astrBuf = new StringBuffer((int) entity.getContentLength());\x0d\x0awhile ((line = reader.readLine()) != null) {\x0d\x0astrBuf.append(line);\x0d\x0a}\x0d\x0a}\x0d\x0a}\x0d\x0aif (entity != null) {\x0d\x0ansumeContent();\x0d\x0a}\x0d\x0a}\x0d\x0a//将url标记为已访问\x0d\x0amarkUrlAsVisited(url);\x0d\x0areturn strBuf.toString();\x0d\x0a}\x0d\x0apublic static boolean isContentRelevant(String content,\x0d\x0aPattern regexpPattern) {\x0d\x0aboolean retValue = https://www.04ip.com/post/false;/x0d/x0aif (content != null) {/x0d/x0a//是否符合正则表达式的条件/x0d/x0aMatcher m = regexpPattern.matcher(content.toLowerCase());/x0d/x0aretValue = m.find();/x0d/x0a}/x0d/x0areturn retValue;/x0d/x0a}/x0d/x0apublic List extractUrls(String text, CrawlerUrl crawlerUrl) {/x0d/x0aMap urlMap = new HashMap();/x0d/x0aextractHttpUrls(urlMap, text);/x0d/x0aextractRelativeUrls(urlMap, text, crawlerUrl);/x0d/x0areturn new ArrayList(urlMap.keySet());/x0d/x0a}/x0d/x0aprivate void extractHttpUrls(Map urlMap, String text) {/x0d/x0aMatcher m = (text);/x0d/x0awhile (m.find()) {/x0d/x0aString url = m.group();/x0d/x0aString[] terms = url.split("a href=https://www.04ip.com/"");\x0d\x0afor (String term : terms) {\x0d\x0a// System.out.println("Term = "term);\x0d\x0aif (term.startsWith("http")) {\x0d\x0aint index = term.indexOf("\"");\x0d\x0aif (index0) {\x0d\x0aterm = term.substring(0, index);\x0d\x0a}\x0d\x0aurlMap.put(term, term);\x0d\x0aSystem.out.println("Hyperlink: "term);\x0d\x0a}\x0d\x0a}\x0d\x0a}\x0d\x0a}\x0d\x0aprivate void extractRelativeUrls(Map urlMap, String text,\x0d\x0aCrawlerUrl crawlerUrl) {\x0d\x0aMatcher m = relativeRegexp.matcher(text);\x0d\x0aURL textURL = crawlerUrl.getURL();\x0d\x0aString host = textURL.getHost();\x0d\x0awhile (m.find()) {\x0d\x0aString url = m.group();\x0d\x0aString[] terms = url.split("a href=https://www.04ip.com/"");\x0d\x0afor (String term : terms) {\x0d\x0aif (term.startsWith("/")) {\x0d\x0aint index = term.indexOf("\"");\x0d\x0aif (index0) {\x0d\x0aterm = term.substring(0, index);\x0d\x0a}\x0d\x0aString s = //"hostterm;\x0d\x0aurlMap.put(s, s);\x0d\x0aSystem.out.println("Relative url: "s);\x0d\x0a}\x0d\x0a}\x0d\x0a}\x0d\x0a\x0d\x0a}\x0d\x0apublic static void main(String[] args) {\x0d\x0atry {\x0d\x0aString url = "";\x0d\x0aQueue urlQueue = new LinkedList();\x0d\x0aString regexp = "java";\x0d\x0aurlQueue.add(new CrawlerUrl(url, 0));\x0d\x0aNaiveCrawler crawler = new NaiveCrawler(urlQueue, 100, 5, 1000L,\x0d\x0aregexp);\x0d\x0a// boolean allowCrawl = crawler.areWeAllowedToVisit(url);\x0d\x0a// System.out.println("Allowed to crawl: "url" "\x0d\x0a// allowCrawl);\x0d\x0acrawler.crawl();\x0d\x0a} catch (Throwable t) {\x0d\x0aSystem.out.println(t.toString());\x0d\x0at.printStackTrace();\x0d\x0a}\x0d\x0a}
java设置servletoutputstream的文件名字1. 核心代码
//设置Mime-Type
response.setContentType("application/msword");
//设置下载默认文件名
response.setHeader("Content-Disposition", "attachment;filename=123.doc");
登录后复制
完整代码
public RObject download(@PathVariable String id, HttpServletResponse response) {
File uploadFile = new File(id".doc");
//获取后缀名
String suffix = uploadFile.substring(uploadFile.lastIndexOf("."));
String responseType = ResponseTypeUtils.getResponseContentByType(suffix);
response.setContentType(responseType);
response.setHeader("Content-Disposition", "attachment;filename="System.currentTimeMillis()suffix);
//输入流,通过输入流读取文件内容
try (FileInputStream fileInputStream = new FileInputStream(uploadFile)) {
ServletOutputStream outputStream = response.getOutputStream();
int len;
byte[] bytes = new byte[1024];
while ((len = fileInputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, len);
//刷新一下
outputStream.flush();
}
//关闭资源
outputStream.close();
return R.ok();
} catch (Exception e) {
throw new RuntimeException(e.getMessage(), e);
}
}
登录后复制
当然也可以通过NIO的方式实现代码，具体可参见HttpServletResponse 返回图片的几种方法对比，核心代码如下：
fis.getChannel().transferTo(0,fis.available(),Channels.newChannel(response.getOutputStream()));
登录后复制
3 参考状态码
文件扩展名 Content-Type(Mime-Type) 文件扩展名 Content-Type(Mime-Type)
.*（ 二进制流，不知道下载文件类型） application/octet-stream .tif image/tiff
.001 application/x-001 .301 application/x-301
.323 text/h323 .906 application/x-906
.907 drawing/907 .a11 application/x-a11
.acp audio/x-mei-aac .ai application/postscript
.aif audio/aiff .aifc audio/aiff
.aiff audio/aiff .anv application/x-anv
.asa text/asa .asf video/x-ms-asf
.asp text/asp .asx video/x-ms-asf
.au audio/basic .avi video/avi
.awf application/vnd.adobe.workflow .biz text/xml
.bmp application/x-bmp .bot application/x-bot
.c4t application/x-c4t .c90 application/x-c90
.cal application/x-cals .cat application/vnd.ms-pki.seccat
.cdf application/x-netcdf .cdr application/x-cdr
.cel application/x-cel .cer application/x-x509-ca-cert
.cg4 application/x-g4 .cgm application/x-cgm
.cit application/x-cit .class java/*
.cml text/xml .cmp application/x-cmp
.cmx application/x-cmx .cot application/x-cot
.crl application/pkix-crl .crt application/x-x509-ca-cert
.csi application/x-csi .css text/css
.cut application/x-cut .dbf application/x-dbf
.dbm application/x-dbm .dbx application/x-dbx
.dcd text/xml .dcx application/x-dcx
.der application/x-x509-ca-cert .dgn application/x-dgn
.dib application/x-dib .dll application/x-msdownload
.doc application/msword .dot application/msword
.docx application/msword .xlsx application/vnd.ms-excel
.drw application/x-drw .dtd text/xml
.dwf Model/vnd.dwf .dwf application/x-dwf
.dwg application/x-dwg .dxb application/x-dxb
.dxf application/x-dxf .edn application/vnd.adobe.edn
.emf application/x-emf .eml message/rfc822
.ent text/xml .epi application/x-epi
.eps application/x-ps .eps application/postscript
.etd application/x-ebx .exe application/x-msdownload
.fax image/fax .fdf application/vnd.fdf
.fif application/fractals .fo text/xml
.frm application/x-frm .g4 application/x-g4
.gbr application/x-gbr .pptx application/vnd.ms-powerpoint
.gif image/gif .gl2 application/x-gl2
.gp4 application/x-gp4 .hgl application/x-hgl
.hmr application/x-hmr .hpg application/x-hpgl
.hpl application/x-hpl .hqx application/mac-binhex40
.hrf application/x-hrf .hta application/hta
.htc text/x-component .htm text/html
.html text/html .htt text/webviewhtml
.htx text/html .icb application/x-icb
.ico image/x-icon .ico application/x-ico
.iff application/x-iff .ig4 application/x-g4
.igs application/x-igs .iii application/x-iphone
.img application/x-img .ins application/x-internet-signup
.isp application/x-internet-signup .IVF video/x-ivf
.java java/* .jfif image/jpeg
.jpe image/jpeg .jpe application/x-jpe
.jpeg image/jpeg .jpg image/jpeg
.jpg application/x-jpg .js application/x-javascript
.jsp text/html .la1 audio/x-liquid-file
.lar application/x-laplayer-reg .latex application/x-latex
.lavs audio/x-liquid-secure .lbm application/x-lbm
.lmsff audio/x-la-lms .ls application/x-javascript
.ltr application/x-ltr .m1v video/x-mpeg
.m2v video/x-mpeg .m3u audio/mpegurl
.m4e video/mpeg4 .mac application/x-mac
.man application/x-troff-man .math text/xml
.mdb application/msaccess .mdb application/x-mdb
.mfp application/x-shockwave-flash .mht message/rfc822
.mhtml message/rfc822 .mi application/x-mi
.mid audio/mid .midi audio/mid
.mil application/x-mil .mml text/xml
.mnd audio/x-musicnet-download .mns audio/x-musicnet-stream
.mocha application/x-javascript .movie video/x-sgi-movie
.mp1 audio/mp1 .mp2 audio/mp2
.mp2v video/mpeg .mp3 audio/mp3
.mp4 video/mpeg4 .mpa video/x-mpg
.mpd application/vnd.ms-project .mpe video/x-mpeg
.mpeg video/mpg .mpg video/mpg
.mpga audio/rn-mpeg .mpp application/vnd.ms-project
.mps video/x-mpeg .mpt application/vnd.ms-project
.mpv video/mpg .mpv2 video/mpeg
.mpw application/vnd.ms-project .mpx application/vnd.ms-project
.mtx text/xml .mxp application/x-mmxp
.net image/pnetvue .nrf application/x-nrf
.nws message/rfc822 .odc text/x-ms-odc
.out application/x-out .p10 application/pkcs10
.p12 application/x-pkcs12 .p7b application/x-pkcs7-certificates
.p7c application/pkcs7-mime .p7m application/pkcs7-mime
.p7r application/x-pkcs7-certreqresp .p7s application/pkcs7-signature
.pc5 application/x-pc5 .pci application/x-pci
.pcl application/x-pcl .pcx application/x-pcx
.pdf application/pdf .pdf application/pdf
.pdx application/vnd.adobe.pdx .pfx application/x-pkcs12
.pgl application/x-pgl .pic application/x-pic
.pko application/vnd.ms-pki.pko .pl application/x-perl
.plg text/html .pls audio/scpls
.plt application/x-plt .png image/png
.png application/x-png .pot application/vnd.ms-powerpoint
.ppa application/vnd.ms-powerpoint .ppm application/x-ppm
.pps application/vnd.ms-powerpoint .ppt application/vnd.ms-powerpoint
.ppt application/x-ppt .pr application/x-pr
.prf application/pics-rules .prn application/x-prn
.prt application/x-prt .ps application/x-ps
.ps application/postscript .ptn application/x-ptn
.pwz application/vnd.ms-powerpoint .r3t text/vnd.rn-realtext3d
.ra audio/vnd.rn-realaudio .ram audio/x-pn-realaudio
.ras application/x-ras .rat application/rat-file
.rdf text/xml .rec application/vnd.rn-recording
.red application/x-red .rgb application/x-rgb
.rjs application/vnd.rn-realsystem-rjs .rjt application/vnd.rn-realsystem-rjt
.rlc application/x-rlc .rle application/x-rle
.rm application/vnd.rn-realmedia .rmf application/vnd.adobe.rmf
.rmi audio/mid .rmj application/vnd.rn-realsystem-rmj
.rmm audio/x-pn-realaudio .rmp application/vnd.rn-rn_music_package
.rms application/vnd.rn-realmedia-secure .rmvb application/vnd.rn-realmedia-vbr
.rmx application/vnd.rn-realsystem-rmx .rnx application/vnd.rn-realplayer
.rp image/vnd.rn-realpix .rpm audio/x-pn-realaudio-plugin
.rsml application/vnd.rn-rsml .rt text/vnd.rn-realtext
.rtf application/msword .rtf application/x-rtf
.rv video/vnd.rn-realvideo .sam application/x-sam
.sat application/x-sat .sdp application/sdp
.sdw application/x-sdw .sit application/x-stuffit
.slb application/x-slb .sld application/x-sld
.slk drawing/x-slk .smi application/smil
.smil application/smil .smk application/x-smk
.snd audio/basic .sol text/plain
.sor text/plain .spc application/x-pkcs7-certificates
.spl application/futuresplash .spp text/xml
.ssm application/streamingmedia .sst application/vnd.ms-pki.certstore
.stl application/vnd.ms-pki.stl .stm text/html
.sty application/x-sty .svg text/xml
.swf application/x-shockwave-flash .tdf application/x-tdf
.tg4 application/x-tg4 .tga application/x-tga
.tif image/tiff .tif application/x-tif
.tiff image/tiff .tld text/xml
.top drawing/x-top .torrent application/x-bittorrent
.tsd text/xml .txt text/plain
.uin application/x-icq .uls text/iuls
.vcf text/x-vcard .vda application/x-vda
.vdx application/vnd.visio .vml text/xml
.vpg application/x-vpeg005 .vsd application/vnd.visio
.vsd application/x-vsd .vss application/vnd.visio
.vst application/vnd.visio .vst application/x-vst
.vsw application/vnd.visio .vsx application/vnd.visio
.vtx application/vnd.visio .vxml text/xml
.wav audio/wav .wax audio/x-ms-wax
.wb1 application/x-wb1 .wb2 application/x-wb2
.wb3 application/x-wb3 .wbmp image/vnd.wap.wbmp
.wiz application/msword .wk3 application/x-wk3
.wk4 application/x-wk4 .wkq application/x-wkq
.wks application/x-wks .wm video/x-ms-wm
.wma audio/x-ms-wma .wmd application/x-ms-wmd
.wmf application/x-wmf .wml text/vnd.wap.wml
.wmv video/x-ms-wmv .wmx video/x-ms-wmx
.wmz application/x-ms-wmz .wp6 application/x-wp6
.wpd application/x-wpd .wpg application/x-wpg
.wpl application/vnd.ms-wpl .wq1 application/x-wq1
.wr1 application/x-wr1 .wri application/x-wri
.wrk application/x-wrk .ws application/x-ws
.ws2 application/x-ws .wsc text/scriptlet
.wsdl text/xml .wvx video/x-ms-wvx
.xdp application/vnd.adobe.xdp .xdr text/xml
.xfd application/vnd.adobe.xfd .xfdf application/vnd.adobe.xfdf
.xhtml text/html .xls application/vnd.ms-excel
.xls application/x-xls .xlw application/x-xlw
.xml text/xml .xpl audio/scpls
.xq text/xml .xql text/xml
.xquery text/xml .xsd text/xml
.xsl text/xml .xslt text/xml
.xwd application/x-xwd .x_b application/x-x_b
.sis application/vnd.symbian.install .sisx application/vnd.symbian.install
.x_t application/x-x_t .ipa application/vnd.iphone
.apk application/vnd.android.package-archive .xap application
java语言中如何获取字符串中汉字的个数【实例描述】
字符串中可以包含数字核心代码java，字母、汉字或者其他字符 。使用Character类的isDigit()方法可以判断字符中的某个字符是否为数字核心代码java，使用Character类的isLetter()方法可以判断字符中的某个字符是否为字母 。实例中将介绍一种方法用来判断字符串中的某个字符是否为汉字，通过此方法可以计算字符串中汉字的数量 。实例的运行效果如图4.24所示 。
【实现过程】
在Eclipse中新建项目ChineseCharacter，并在其中创建一个ChineseCharacter.java文件 。在该类的主方法中创建标准输入流的扫描器对象，接收用户输入的字符串 。我们在程序中使用matches()方法来统计该字符串中汉字的个数 。核心代码如下所示：
protectedvoiddo_button_actionPerformed(ActionEvente){Stringtext=chineseArea.getText();//获取用户输入intamount=0;//创建汉字数量计数器for(inti=0;itext.length();i){//遍历字符串每一个字符//使用正则表达式判断字符是否属于汉字编码booleanmatches=Pattern.matches("^[一-龥]{0,}$","" text.charAt(i));if(matches){//如果是汉字amount;//累加计数器}}umField.setText(amount "");//在文本框显示汉字数量}
【代码解析】
本实例的关键点在于正则表达式的使用 。Java提供了Pattern用于正则表达式的编译表示形式，该类提供的静态方法matches()可以执行正则表达式的匹配 。该方法编译给定正则表达式并尝试给定输入与其匹配 。如果要匹配的字符序列与正则表达式匹配则返回true，否则返回false 。其声明语法如下：
publicstaticbooleanmatches(Stringregex,CharSequenceinput);【知识扩展】
使用正则表达式可以方便地进行字符串操作，正则表达式经常被用来验证用户输入的信息，如可以判断用户输入的格式是否正确 。本实例中使用正则表达式来判断用户输入的字符串是否为汉字，如果为汉字则计数器加1 ， 最后得到字符串中所有汉字的数量 。
Java的核心代码是什么《Java 核心代码》 是一本书，这就是核心代码java你问题核心代码java的答案 。
这本书真的挺厚的，没办法给你打出来 。
你要知道 Java 核心代码是什么，就把这本书从头到尾看一遍吧！
【核心代码java 核心代码怎么找】关于核心代码java和核心代码怎么找的介绍到此就结束了，不知道你从中找到你需要的信息了吗 ？如果你还想了解更多这方面的信息 ， 记得收藏关注本站 。

    推荐阅读


    上一篇：怎么用链接去弄公众号，微信公众号怎样做链接

    下一篇：如何使用cns服务器？ cns服务器怎么用