公司如何申请一个网站,中信建设有限责任公司湖南分公司,企业管理平台app安卓版,百度站长社区文章目录 1. 获取登录的请求2. 用postman模拟登录请求3. 用wget模拟登录请求并保存cookie4. 开始爬取网站5. 查看爬取结果6. 网站爬虫简易教程 爬取需要登录的网站的资源 背景#xff1a;对于一些网站需要使用用户名和密码登录并且使用了https#xff0c;我们如果不通过凭证将… 文章目录 1. 获取登录的请求2. 用postman模拟登录请求3. 用wget模拟登录请求并保存cookie4. 开始爬取网站5. 查看爬取结果6. 网站爬虫简易教程 爬取需要登录的网站的资源 背景对于一些网站需要使用用户名和密码登录并且使用了https我们如果不通过凭证将无法进行该网站的下载、爬虫而具体的凭证一般的是”cookies“形式的。 内容本文主要介绍了如何爬取需要登录网站的内容视频、图片、网页的简易教程。
postman文档地址https://learning.postman.com/docs/sending-requests/requests/
1. 获取登录的请求
首先需要使用用户名密码登录到网站查看f12找到登录的请求复制成Copy as CURL 登录请求uri一般是login或register等等认真找一找 2. 用postman模拟登录请求
导入请求到postman
将复制的内容导入到postman接口工具中 发送请求获取到wget代码片段
发送请求检查是否模拟登录成功如果请求发送成功则按下图获取到postman的wget代码片段。 3. 用wget模拟登录请求并保存cookie
在从postman复制的代码片段后追加如下cookie配置。
意思就是把cookie保存在cookies.txt中以及后续使用
--save-cookiescookies.txt --keep-session-cookies模拟登录请求并保存cookie
用命令行发送类似下面的wget命令。该命令就是postman复制的代码片段后追加--save-cookiescookies.txt --keep-session-cookies
wget --no-check-certificate --quiet --method GET --timeout0 --header authority: qvb111.xyz --header accept: text/html,application/xhtmlxml,application/xml;q0.9,image/avif,image/webp,image/apng,*/*;q0.8,application/signed-exchange;vb3;q0.9 --header accept-language: zh-CN,zh;q0.9 --header cache-control: max-age0 --header cookie: md10kdfjijf89485.online; _gaGA1.1.1107869110.1654255726; _ga_6DLS4FBHC6GS1.1.1654259056.2.1.1654260355.0; _nipple_sessionDZmMES3vGmHhXLnp9TnULezhbUhy%2FIqFyLNWNYot0S%2FCq7n73iJ1P7ypivBy4u8IPPYe6smeiP7I%2FttFSLEHeb6jEafg50to7ceYCtDLQdAVwnBRdGenEKtc7dODRRQn9FaVOS9ietmoMO0IAbcJ6%2B%2BypZestlQ9IIoAYyYmTvmzQltULHnuA2cQEGUyxlmJqwCF1nfYrhMtBqEgpFP2UwrBKEcBBcqYFL96klIQBOOCSdm8UueNKLZ9O%2BUAlN%2FEIRQgV229ziwy5kUVxBDYzJ9tmLbxrVtSKzKxESuQ1W9n6JefP64fB%2FC7l7kWfL0Vys%2BlCi57UkpuhHfM0IJhj33FOSy4iMtXcVGETor4NG2%2FHcUL2U974YCfPBX6Rc%2BoQ%2Bm8%2Fkyzdutme9AQS%2FPk--RkCe6gHEAt3X3JgH--j5UScZwkeVHIukpKpt6TGQ%3D%3D; _nipple_sessionGBgJoGvRuRJBkWfWwcoSDKiquxucPgj24AUTQQe%2FfPANRvWA6unhiGQFQ8SPqml271vlZwFtGra448GmgDKSnpX%2FCSUkwzEiqDr0ekV9oKw%2FKdrkk6ELO0Z3J8YqInUSiQKm04eVKJvHCRc5p0MH1jJ%2BZAcONVfvfh11Ai2TGpTzYOxZ%2BIi2uHqXn817GUFO7GkDB2VI%2FTIPMz%2B8J7Sxj2GJaEQU%2FKyROs5XN0BWCVhe9EF8CT8RKa1DP%2FrLzOosn33weZOCaPR%2Bbn7jwupxrxsCZ68Tg9oUl%2Ff4GrVTPoAyaWuoPlD0sKtteh9HKqg%2Fb%2BzJMS04US9OlztCm5rzJmV7xW6uoUX9%2BerYxZJB11haN%2Fquablym5VufyWURAZybjY7jEaCoSp94t4EBlPJ--SphXN3nrbR%2Fc3Yhu--G6JqS5oBVQSPdSCeXCf4lg%3D%3D --header referer: https://qvb111.xyz/users/sign_in --header sec-ch-ua: -Not.A/Brand;v8, Chromium;v102 --header sec-ch-ua-mobile: ?0 --header sec-ch-ua-platform: macOS --header sec-fetch-dest: document --header sec-fetch-mode: navigate --header sec-fetch-site: same-origin --header sec-fetch-user: ?1 --header upgrade-insecure-requests: 1 --header user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36 https://qvb111.xyz/ --save-cookiescookies.txt --keep-session-cookies4. 开始爬取网站
配置从cookies.txt中加载cookies并爬取网站https://qvb111.xyz/girl/show/2797
wget --load-cookies cookies.txt \--keep-session-cookies \
https://qvb111.xyz/girl/show/27975. 查看爬取结果
作者爬取了某个带颜色的网站后并用以下的命令查看爬取的内容
cd firefish
ls
cd show
ls
ls | wc -l
du -sh .6. 网站爬虫简易教程
1、正常登录目标网站
2、找到登录请求、复制、导入postman处理
3、复制postman生成wget代码片段并追加设置
--save-cookies cookies.txt --keep-session-cookies4、模拟登录并保存凭证
wget --no-check-certificate --quiet --method GET --timeout0 --header authority: qvb111.xyz --header accept: text/html,application/xhtmlxml,application/xml;q0.9,image/avif,image/webp,image/apng,*/*;q0.8,application/signed-exchange;vb3;q0.9 --header accept-language: zh-CN,zh;q0.9 --header cache-control: max-age0 --header cookie: md10kdfjijf89485.online; _gaGA1.1.1107869110.1654255726; _ga_6DLS4FBHC6GS1.1.1654259056.2.1.1654260355.0; _nipple_sessionDZmMES3vGmHhXLnp9TnULezhbUhy%2FIqFyLNWNYot0S%2FCq7n73iJ1P7ypivBy4u8IPPYe6smeiP7I%2FttFSLEHeb6jEafg50to7ceYCtDLQdAVwnBRdGenEKtc7dODRRQn9FaVOS9ietmoMO0IAbcJ6%2B%2BypZestlQ9IIoAYyYmTvmzQltULHnuA2cQEGUyxlmJqwCF1nfYrhMtBqEgpFP2UwrBKEcBBcqYFL96klIQBOOCSdm8UueNKLZ9O%2BUAlN%2FEIRQgV229ziwy5kUVxBDYzJ9tmLbxrVtSKzKxESuQ1W9n6JefP64fB%2FC7l7kWfL0Vys%2BlCi57UkpuhHfM0IJhj33FOSy4iMtXcVGETor4NG2%2FHcUL2U974YCfPBX6Rc%2BoQ%2Bm8%2Fkyzdutme9AQS%2FPk--RkCe6gHEAt3X3JgH--j5UScZwkeVHIukpKpt6TGQ%3D%3D; _nipple_sessionGBgJoGvRuRJBkWfWwcoSDKiquxucPgj24AUTQQe%2FfPANRvWA6unhiGQFQ8SPqml271vlZwFtGra448GmgDKSnpX%2FCSUkwzEiqDr0ekV9oKw%2FKdrkk6ELO0Z3J8YqInUSiQKm04eVKJvHCRc5p0MH1jJ%2BZAcONVfvfh11Ai2TGpTzYOxZ%2BIi2uHqXn817GUFO7GkDB2VI%2FTIPMz%2B8J7Sxj2GJaEQU%2FKyROs5XN0BWCVhe9EF8CT8RKa1DP%2FrLzOosn33weZOCaPR%2Bbn7jwupxrxsCZ68Tg9oUl%2Ff4GrVTPoAyaWuoPlD0sKtteh9HKqg%2Fb%2BzJMS04US9OlztCm5rzJmV7xW6uoUX9%2BerYxZJB11haN%2Fquablym5VufyWURAZybjY7jEaCoSp94t4EBlPJ--SphXN3nrbR%2Fc3Yhu--G6JqS5oBVQSPdSCeXCf4lg%3D%3D --header referer: https://qvb111.xyz/users/sign_in --header sec-ch-ua: -Not.A/Brand;v8, Chromium;v102 --header sec-ch-ua-mobile: ?0 --header sec-ch-ua-platform: macOS --header sec-fetch-dest: document --header sec-fetch-mode: navigate --header sec-fetch-site: same-origin --header sec-fetch-user: ?1 --header upgrade-insecure-requests: 1 --header user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36 https://qvb111.xyz/ --save-cookiescookies.txt --keep-session-cookies5、开始爬虫
wget --load-cookies cookies.txt \--keep-session-cookies \
https://qvb111.xyz/girl/show/27976、查看爬虫成果见视频 可以以个人网站测试或gitee个人仓库测试不合理使用
