上海网站建设制作微信,通了网站建设,视频营销模式有哪些,国外的网站模板类网站前面豆子学习了基本的urllib的模块#xff0c;通过这个模块可以写一些简单的爬虫文件。如果要处理大中型的爬虫项目#xff0c;urllib就显得比较low了#xff0c;这个时候可以使用scrapy框架来实现#xff0c;很多基本的处理在scrapy里面已经做好了。 首先来安装一下。推荐…前面豆子学习了基本的urllib的模块通过这个模块可以写一些简单的爬虫文件。如果要处理大中型的爬虫项目urllib就显得比较low了这个时候可以使用scrapy框架来实现很多基本的处理在scrapy里面已经做好了。 首先来安装一下。推荐的流程如下 首先升级一下pipC:\WINDOWS\system32python -m pip install --upgrade pip
Requirement already up-to-date: pip in c:\python36\lib\site-packages 安装wheel C:\WINDOWS\system32pip install wheel
Requirement already satisfied: wheel in c:\python36\lib\site-packages3.安装lxml C:\WINDOWS\system32pip install lxml
Collecting lxmlDownloading lxml-4.1.1-cp36-cp36m-win32.whl (3.2MB)100% |████████████████████████████████| 3.2MB 307kB/s
Installing collected packages: lxml
Successfully installed lxml-4.1.1 安装twisted如果在线安装报错的话可以考虑离线安装从https://www.lfd.uci.edu 上搜索twisted, 可以看见他有很多个版本 执行一下python命令看看当前的版本可以看见我的版本是3.6.2,32位的 c:\Users\yuan.li\Downloadspython
Python 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32
Type help, copyright, credits or license for more information. 这样的话可以下载对应的版本twisted-17.9.0-cp36-win32.whl就行了。下载之后手动安装 c:\Users\yuan.li\Downloadspip install Twisted-17.9.0-cp36-cp36m-win32.whl
Processing c:\users\yuan.li\downloads\twisted-17.9.0-cp36-cp36m-win32.whl
Requirement already satisfied: incremental16.10.1 in c:\python36\lib\site-packages (from Twisted17.9.0)
Requirement already satisfied: Automat0.3.0 in c:\python36\lib\site-packages (from Twisted17.9.0)
Requirement already satisfied: zope.interface4.0.2 in c:\python36\lib\site-packages (from Twisted17.9.0)
Requirement already satisfied: hyperlink17.1.1 in c:\python36\lib\site-packages (from Twisted17.9.0)
Requirement already satisfied: constantly15.1 in c:\python36\lib\site-packages (from Twisted17.9.0)
Requirement already satisfied: six in c:\python36\lib\site-packages (from Automat0.3.0-Twisted17.9.0)
Requirement already satisfied: attrs in c:\python36\lib\site-packages (from Automat0.3.0-Twisted17.9.0)
Requirement already satisfied: setuptools in c:\python36\lib\site-packages (from zope.interface4.0.2-Twisted17.9.0)
Installing collected packages: Twisted
Successfully installed Twisted-17.9.0 最后安装scrapyc:\Users\yuan.li\Downloadspip install scrapy
Collecting scrapyDownloading Scrapy-1.5.0-py2.py3-none-any.whl (251kB)100% |████████████████████████████████| 256kB 2.3MB/s
Collecting pyOpenSSL (from scrapy)Downloading pyOpenSSL-17.5.0-py2.py3-none-any.whl (53kB)100% |████████████████████████████████| 61kB 4.5MB/s
Collecting cssselect0.9 (from scrapy)Downloading cssselect-1.0.3-py2.py3-none-any.whl
Collecting parsel1.1 (from scrapy) 安装完成之后执行一下scrapy, 看看是否工作 c:\Users\yuan.li\Downloadsscrapy
Scrapy 1.5.0 - no active projectUsage:scrapy command [options] [args]Available commands:bench Run quick benchmark testfetch Fetch a URL using the Scrapy downloadergenspider Generate new spider using pre-defined templatesrunspider Run a self-contained spider (without creating a project)settings Get settings valuesshell Interactive scraping consolestartproject Create new projectversion Print Scrapy versionview Open URL in browser, as seen by Scrapy[ more ] More commands available when run from project directory
