h5作品网站,网站基本流程,关键词排名查询工具有什么作用?,企业信息公开网查询系统场景
Pythont通过request以及BeautifulSoup爬取几千条情话#xff1a;
Pythont通过request以及BeautifulSoup爬取几千条情话_爬取情话-CSDN博客
Node-RED中使用html节点爬取HTML网页资料之爬取Node-RED的最新版本#xff1a;
Node-RED中使用html节点爬取HTML网页资料之爬…场景
Pythont通过request以及BeautifulSoup爬取几千条情话
Pythont通过request以及BeautifulSoup爬取几千条情话_爬取情话-CSDN博客
Node-RED中使用html节点爬取HTML网页资料之爬取Node-RED的最新版本
Node-RED中使用html节点爬取HTML网页资料之爬取Node-RED的最新版本_node-red html-CSDN博客
Jsoup
Jsoup是一种Java 的HTML(html也是XML文档)解析器可直接解析某个URL地址、HTML文本内容。
它提供了一套易于操作的API可通过DOMCSS以及类似于jQuery选择器的操作方法来取出和操作数据。
使用jsoup就可以解析HTML。
Jsoup使用的是DOM解析方式把整个HTML文档XML文档加载到内存中形成一棵DOM树得到文档的Document对象。
HTML里的标签会转换成Element对象。
官网地址
jsoup: Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety
EasyExcel
Java解析、生成Excel比较有名的框架有Apache poi、jxl。但他们都存在一个严重的问题就是非常的耗内存
poi有一套SAX模式的API可以一定程度的解决一些内存溢出的问题但POI还是有一些缺陷
比如07版Excel解压缩以及解压后存储都是在内存中完成的内存消耗依然很大。
easyexcel重写了poi对07版Excel的解析一个3M的excel用POI sax解析依然需要100M左右内存
改用easyexcel可以降低到几M并且再大的excel也不会出现内存溢出03版依赖POI的sax模式
在上层做了模型转换的封装让使用者更加简单方便。
官网地址
关于Easyexcel | Easy Excel
注
博客https://blog.csdn.net/badao_liumang_qizhi
实现
1、引入依赖 !--Jsoup 是一个用于解析HTML和XML文档的Java库--dependencygroupIdorg.jsoup/groupIdartifactIdjsoup/artifactIdversion1.11.3/version/dependency!--EasyExcel是一个基于Java的、快速、简洁、解决大文件内存溢出的Excel处理工具--dependencygroupIdcom.alibaba/groupIdartifactIdeasyexcel/artifactIdversion3.0.5/version/dependency
2、找到需要爬取的网页内容
比如以下面为例
2023财富世界500强企业榜单 2023全球500强企业 世界500强排名一览表→买购网 这里要获取500强排名数据因为单次刷新网页只能返回100条数据所以只解析前100条。获取更多数据可根据其分页请求规则分别进行爬取。
打开F12找到要爬取的数据的dom结构
这里要获取到id为t_container的div元素大的第22个子元素(索引为21)的table元素的tr元素的td数据。 3、编写测试代码连接并解析html元素 String url https://www.maigoo.com/news/3jcNODk3.html;try {//读取url得到DocumentDocument document Jsoup.connect(url).ignoreContentType(true).userAgent(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3).timeout(30000).header(referer,https://www.maigoo.com).get();Elements select document.select(#t_container div:eq(21) table tr);} catch (IOException e) {e.printStackTrace();}
注意这里使用选择器的语法
#t_container 代表id为t_container
代表找父元素下的子元素
div:eq(21) 代表第22个元素
table tr 代表table 标签下tr标签
更多select选择器用法
Use CSS selectors to find elements: jsoup Java HTML parser
Selector overview
tagname: find elements by tag, e.g. div#id: find elements by ID, e.g. #logo.class: find elements by class name, e.g. .masthead[attribute]: elements with attribute, e.g. [href][^attrPrefix]: elements with an attribute name prefix, e.g. [^data-] finds elements with HTML5 dataset attributes[attrvalue]: elements with attribute value, e.g. [width500] (also quotable, like [data-namelaunch sequence])[attr^value], [attr$value], [attr*value]: elements with attributes that start with, end with, or contain the value, e.g. [href*/path/][attr~regex]: elements with attribute values that match the regular expression; e.g. img[src~(?i)\.(png|jpe?g)]*: all elements, e.g. *ns|tag: find elements by tag in a namespace prefix, e.g. fb|name finds fb:name elements*|tag: final elements by tag in any namespace prefix, e.g. *|name finds fb:name and name elements
Selector combinations
el#id: elements with ID, e.g. div#logoel.class: elements with class, e.g. div.mastheadel[attr]: elements with attribute, e.g. a[href]Any combination, e.g. a[href].highlightancestor child: child elements that descend from ancestor, e.g. .body p finds p elements anywhere under a block with class bodyparent child: child elements that descend directly from parent, e.g. div.content p finds p elements; and body * finds the direct children of the body tagsiblingA siblingB: finds sibling B element immediately preceded by sibling A, e.g. div.head divsiblingA ~ siblingX: finds sibling X element preceded by sibling A, e.g. h1 ~ pel, el, el: group multiple selectors, find unique elements that match any of the selectors; e.g. div.masthead, div.logo
Pseudo selectors
:has(selector): find elements that contain elements matching the selector; e.g. div:has(p):is(selector): find elements that match any of the selectors in the selector list; e.g. :is(h1, h2, h3, h4, h5, h6) finds any heading element:not(selector): find elements that do not match the selector; e.g. div:not(.logo):contains(text): find elements that contain the given text. The search is case-insensitive; e.g. p:contains(jsoup):containsOwn(text): find elements that directly contain the given text:matches(regex): find elements whose text matches the specified regular expression; e.g. div:matches((?i)login):matchesOwn(regex): find elements whose own text matches the specified regular expression:lt(n): find elements whose sibling index (i.e. its position in the DOM tree relative to its parent) is less than n; e.g. td:lt(3):gt(n): find elements whose sibling index is greater than n; e.g. div p:gt(2):eq(n): find elements whose sibling index is equal to n; e.g. form input:eq(1)Note that the above indexed pseudo-selectors are 0-based, that is, the first element is at index 0, the second at 1, etc 除使用select选择器之外还可使用XPath选择器用法
Use XPath selectors to find elements and nodes: jsoup Java HTML parser
4、解析dom数据并赋值到对象添加到list
新建实体对象并添加excel注解
import com.alibaba.excel.annotation.ExcelProperty;
import lombok.Builder;
import lombok.Data;import java.io.Serializable;Data
Builder
public class WealthEntity implements Serializable {private static final long serialVersionUID -1760099890427975758L;ExcelProperty(value 排名,index 0)private Integer index;ExcelProperty(value 公司名称,index 1)private String companyName;ExcelProperty(value 收入,index 2)private String income;ExcelProperty(value 利润,index 3)private String profit;}
进行dom解析和添加到list Elements select document.select(#t_container div:eq(21) table tr);ListWealthEntity list new ArrayList();for (int i 1; i select.size(); i) {Element tr select.get(i);Elements tds tr.select(td);Integer index Integer.valueOf(tds.get(0).text());String companyName tds.get(1).text();String income tds.get(2).text();String profit tds.get(3).text();WealthEntity wealthEntity WealthEntity.builder().index(index).companyName(companyName).income(income).profit(profit).build();list.add(wealthEntity);}
5、导出为excel String fileName D:/2023财富世界100强.xlsx;EasyExcel.write(fileName,WealthEntity.class).sheet(100强).doWrite(list);
6、完整示例代码 String url https://www.maigoo.com/news/3jcNODk3.html;try {//读取url得到DocumentDocument document Jsoup.connect(url).ignoreContentType(true).userAgent(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3).timeout(30000).header(referer,https://www.maigoo.com).get();Elements select document.select(#t_container div:eq(21) table tr);ListWealthEntity list new ArrayList();for (int i 1; i select.size(); i) {Element tr select.get(i);Elements tds tr.select(td);Integer index Integer.valueOf(tds.get(0).text());String companyName tds.get(1).text();String income tds.get(2).text();String profit tds.get(3).text();WealthEntity wealthEntity WealthEntity.builder().index(index).companyName(companyName).income(income).profit(profit).build();list.add(wealthEntity);}String fileName D:/2023财富世界100强.xlsx;EasyExcel.write(fileName,WealthEntity.class).sheet(100强).doWrite(list);} catch (IOException e) {e.printStackTrace();}
7、运行结果
