Scrapy for Windows
安裝wheelD:SoftwarePython>pip install wheel
Requirement already satisfied: wheel in d:softwarepythonlibsite-packages (0.30.0)
https://www.lfd.uci.edu/~gohlke/pythonlibs/
根據python版本，下載下傳對應版本的lxml
D:SoftwarePython>pip install lxml #如果下載下傳了對應版本的lxml，就執行指令：pip install 檔案路徑
Requirement already satisfied: lxml in d:softwarepythonlibsite-packages (4.2.4)
如果可以import的話，lxml庫就已經安裝好了
D:SoftwarePython>python
Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml
https://pypi.org/project/pyOpenSSL/#downloads
https://pypi.org/project/pyOpenSSL/#files
D:SoftwarePython>pip install C:UsersAdministratorDownloadspyOpenSSL-19.0.0-py2.py3-none-any.whl
Processing c:usersadministratordownloadspyopenssl-19.0.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in d:softwarepythonlibsite-packages (from pyOpenSSL==19.0.0) (1.11.0)
Collecting cryptography>=2.3 (from pyOpenSSL==19.0.0)
Downloading https://files.pythonhosted.org/packages/31/1f/f158f5e83c934ff71cde6c54ebcac48547ec05c01836054ea721daf9840b/cryptography-2.7-cp36-cp36m-win_amd64.whl (1.5MB)
|████████████████████████████████| 1.5MB 64kB/s
Requirement already satisfied: asn1crypto>=0.21.0 in d:softwarepythonlibsite-packages (from cryptography>=2.3->pyOpenSSL==19.0.0) (0.22.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in d:softwarepythonlibsite-packages (from cryptography>=2.3->pyOpenSSL==19.0.0) (1.10.0)
Requirement already satisfied: pycparser in d:softwarepythonlibsite-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyOpenSSL==19.0.0) (2.18)
Installing collected packages: cryptography, pyOpenSSL
Found existing installation: cryptography 2.0.3
Uninstalling cryptography-2.0.3:
Successfully uninstalled cryptography-2.0.3
Found existing installation: pyOpenSSL 17.2.0
Uninstalling pyOpenSSL-17.2.0:
Successfully uninstalled pyOpenSSL-17.2.0
Successfully installed cryptography-2.7 pyOpenSSL-19.0.0
https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
D:SoftwarePython>pip install C:UsersAdministratorDownloadsTwisted-19.2.0-cp36-cp36m-win_amd64.whl
Processing c:usersadministratordownloadstwisted-19.2.0-cp36-cp36m-win_amd64.whl
Requirement already satisfied: incremental>=16.10.1 in d:softwarepythonlibsite-packages (from Twisted==19.2.0) (17.5.0)
Requirement already satisfied: zope.interface>=4.4.2 in d:softwarepythonlibsite-packages (from Twisted==19.2.0) (4.4.2)
Requirement already satisfied: Automat>=0.3.0 in d:softwarepythonlibsite-packages (from Twisted==19.2.0) (0.6.0)
Requirement already satisfied: hyperlink>=17.1.1 in d:softwarepythonlibsite-packages (from Twisted==19.2.0) (17.3.1)
Collecting attrs>=17.4.0 (from Twisted==19.2.0)
Downloading https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
Requirement already satisfied: constantly>=15.1 in d:softwarepythonlibsite-packages (from Twisted==19.2.0) (15.1.0)
Collecting PyHamcrest>=1.9.0 (from Twisted==19.2.0)
Downloading https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl (52kB)
|████████████████████████████████| 61kB 12kB/s
Requirement already satisfied: setuptools in d:softwarepythonlibsite-packages (from zope.interface>=4.4.2->Twisted==19.2.0) (28.8.0)
Requirement already satisfied: six in d:softwarepythonlibsite-packages (from Automat>=0.3.0->Twisted==19.2.0) (1.11.0)
Installing collected packages: attrs, PyHamcrest, Twisted
Found existing installation: attrs 17.2.0
Uninstalling attrs-17.2.0:
Successfully uninstalled attrs-17.2.0
Successfully installed PyHamcrest-1.9.0 Twisted-19.2.0 attrs-19.1.0
https://sourceforge.net/projects/pywin32/files/pywin32/
https://jingyan.baidu.com/article/bad08e1ed173d409c85121f8.html 參考
自動識别了python目錄進行安裝
安裝w3lib（如果已經安裝，可能會需要更新）https://pypi.org/project/w3lib/#files
C:UsersAdministrator>pip install C:UsersAdministratorDownloadsw3lib-1.20.0-py2.py3-none-any.whl
Processing c:usersadministratordownloadsw3lib-1.20.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.4.1 in d:softwarepythonlibsite-packages (from w3lib==1.20.0) (1.11.0)
Installing collected packages: w3lib
Found existing installation: w3lib 1.18.0
Uninstalling w3lib-1.18.0:
Successfully uninstalled w3lib-1.18.0
Successfully installed w3lib-1.20.0
C:UsersAdministrator>pip install scrapy
Requirement already satisfied: scrapy in d:softwarepythonlibsite-packages (1.6.0)
Requirement already satisfied: cssselect>=0.9 in d:softwarepythonlibsite-packages (from scrapy) (1.0.1)
Requirement already satisfied: Twisted>=13.1.0 in d:softwarepythonlibsite-packages (from scrapy) (19.2.0)
Requirement already satisfied: PyDispatcher>=2.0.5 in d:softwarepythonlibsite-packages (from scrapy) (2.0.5)
Requirement already satisfied: service-identity in d:softwarepythonlibsite-packages (from scrapy) (17.0.0)
Requirement already satisfied: six>=1.5.2 in d:softwarepythonlibsite-packages (from scrapy) (1.11.0)
Requirement already satisfied: queuelib in d:softwarepythonlibsite-packages (from scrapy) (1.4.2)
Requirement already satisfied: pyOpenSSL in d:softwarepythonlibsite-packages (from scrapy) (19.0.0)
Requirement already satisfied: parsel>=1.5 in d:softwarepythonlibsite-packages (from scrapy) (1.5.1)
Requirement already satisfied: w3lib>=1.17.0 in d:softwarepythonlibsite-packages (from scrapy) (1.20.0)
Requirement already satisfied: lxml in d:softwarepythonlibsite-packages (from scrapy) (4.2.4)
Requirement already satisfied: incremental>=16.10.1 in d:softwarepythonlibsite-packages (from Twisted>=13.1.0->scrapy) (17.5.0)
Requirement already satisfied: zope.interface>=4.4.2 in d:softwarepythonlibsite-packages (from Twisted>=13.1.0->scrapy) (4.4.2)
Requirement already satisfied: Automat>=0.3.0 in d:softwarepythonlibsite-packages (from Twisted>=13.1.0->scrapy) (0.6.0)
Requirement already satisfied: constantly>=15.1 in d:softwarepythonlibsite-packages (from Twisted>=13.1.0->scrapy) (15.1.0)
Requirement already satisfied: hyperlink>=17.1.1 in d:softwarepythonlibsite-packages (from Twisted>=13.1.0->scrapy) (17.3.1)
Requirement already satisfied: PyHamcrest>=1.9.0 in d:softwarepythonlibsite-packages (from Twisted>=13.1.0->scrapy) (1.9.0)
Requirement already satisfied: attrs>=17.4.0 in d:softwarepythonlibsite-packages (from Twisted>=13.1.0->scrapy) (19.1.0)
Requirement already satisfied: pyasn1 in d:softwarepythonlibsite-packages (from service-identity->scrapy) (0.3.4)
Requirement already satisfied: pyasn1-modules in d:softwarepythonlibsite-packages (from service-identity->scrapy) (0.1.4)
Requirement already satisfied: cryptography>=2.3 in d:softwarepythonlibsite-packages (from pyOpenSSL->scrapy) (2.7)
Requirement already satisfied: setuptools in d:softwarepythonlibsite-packages (from zope.interface>=4.4.2->Twisted>=13.1.0->scrapy) (28.8.0)
Requirement already satisfied: asn1crypto>=0.21.0 in d:softwarepythonlibsite-packages (from cryptography>=2.3->pyOpenSSL->scrapy) (0.22.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in d:softwarepythonlibsite-packages (from cryptography>=2.3->pyOpenSSL->scrapy) (1.10.0)
Requirement already satisfied: pycparser in d:softwarepythonlibsite-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyOpenSSL->scrapy) (2.18)
Scrapy for Centos
[[email protected] simon]# pip install scrapy
Collecting scrapy
Using cached https://files.pythonhosted.org/packages/3e/45/414e87ac8209d537c91575538c5307c20217a6943f555e0ee39f6db4bb0f/Scrapy-1.6.0-py2.py3-none-any.whl
Collecting Twisted>=13.1.0 (from scrapy)
Downloading https://files.pythonhosted.org/packages/f8/2b/a80a70f71eb2b86992ffa5aaae41457791ae67faa70927fd16b76127c2b7/Twisted-19.2.0.tar.bz2 (3.1MB)
|████████████████████████████████| 3.1MB 16kB/s
Requirement already satisfied: lxml in /usr/local/python/lib/python3.5/site-packages (from scrapy) (4.3.3)
Requirement already satisfied: six>=1.5.2 in /usr/local/python/lib/python3.5/site-packages (from scrapy) (1.12.0)
Requirement already satisfied: cssselect>=0.9 in /usr/local/python/lib/python3.5/site-packages (from scrapy) (1.0.3)
Collecting w3lib>=1.17.0 (from scrapy)
Downloading https://files.pythonhosted.org/packages/81/43/9dcf92a77f5f0afe4f4df2407d7289eea01368a08b64bda00dd318ca62a6/w3lib-1.20.0-py2.py3-none-any.whl
Collecting pyOpenSSL (from scrapy)
Using cached https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl
Collecting queuelib (from scrapy)
Using cached https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl
Collecting service-identity (from scrapy)
Downloading https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from scrapy)
Downloading https://files.pythonhosted.org/packages/cd/37/39aca520918ce1935bea9c356bcbb7ed7e52ad4e31bff9b943dfc8e7115b/PyDispatcher-2.0.5.tar.gz
Collecting parsel>=1.5 (from scrapy)
Downloading https://files.pythonhosted.org/packages/96/69/d1d5dba5e4fecd41ffd71345863ed36a45975812c06ba77798fc15db6a64/parsel-1.5.1-py2.py3-none-any.whl
Collecting zope.interface>=4.4.2 (from Twisted>=13.1.0->scrapy)
Downloading https://files.pythonhosted.org/packages/0c/dc/d715a3f2f9df4e0440e67b1dc654bcfb6dc11aac8296353eaf56164b8f01/zope.interface-4.6.0-cp35-cp35m-manylinux1_x86_64.whl (167kB)
|████████████████████████████████| 174kB 17kB/s
Collecting constantly>=15.1 (from Twisted>=13.1.0->scrapy)
Downloading https://files.pythonhosted.org/packages/b9/65/48c1909d0c0aeae6c10213340ce682db01b48ea900a7d9fce7a7910ff318/constantly-15.1.0-py2.py3-none-any.whl
Collecting incremental>=16.10.1 (from Twisted>=13.1.0->scrapy)
Downloading https://files.pythonhosted.org/packages/f5/1d/c98a587dc06e107115cf4a58b49de20b19222c83d75335a192052af4c4b7/incremental-17.5.0-py2.py3-none-any.whl
Collecting Automat>=0.3.0 (from Twisted>=13.1.0->scrapy)
Downloading https://files.pythonhosted.org/packages/a3/86/14c16bb98a5a3542ed8fed5d74fb064a902de3bdd98d6584b34553353c45/Automat-0.7.0-py2.py3-none-any.whl
Collecting hyperlink>=17.1.1 (from Twisted>=13.1.0->scrapy)
Downloading https://files.pythonhosted.org/packages/7f/91/e916ca10a2de1cb7101a9b24da546fb90ee14629e23160086cf3361c4fb8/hyperlink-19.0.0-py2.py3-none-any.whl
Collecting PyHamcrest>=1.9.0 (from Twisted>=13.1.0->scrapy)
Downloading https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl (52kB)
|████████████████████████████████| 61kB 17kB/s
Collecting attrs>=17.4.0 (from Twisted>=13.1.0->scrapy)
Downloading https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
Collecting cryptography>=2.3 (from pyOpenSSL->scrapy)
Downloading https://files.pythonhosted.org/packages/97/18/c6557f63a6abde34707196fb2cad1c6dc0dbff25a200d5044922496668a4/cryptography-2.7-cp34-abi3-manylinux1_x86_64.whl (2.3MB)
|████████████████████████████████| 2.3MB 4.8kB/s
Collecting pyasn1-modules (from service-identity->scrapy)
Downloading https://files.pythonhosted.org/packages/91/f0/b03e00ce9fddf4827c42df1c3ce10c74eadebfb706231e8d6d1c356a4062/pyasn1_modules-0.2.5-py2.py3-none-any.whl (74kB)
|████████████████████████████████| 81kB 8.7kB/s
Collecting pyasn1 (from service-identity->scrapy)
Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)
|████████████████████████████████| 81kB 11kB/s
Requirement already satisfied: setuptools in /usr/local/python/lib/python3.5/site-packages (from zope.interface>=4.4.2->Twisted>=13.1.0->scrapy) (20.10.1)
Requirement already satisfied: idna>=2.5 in /usr/local/python/lib/python3.5/site-packages (from hyperlink>=17.1.1->Twisted>=13.1.0->scrapy) (2.8)
Collecting asn1crypto>=0.21.0 (from cryptography>=2.3->pyOpenSSL->scrapy)
Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
|████████████████████████████████| 102kB 11kB/s
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyOpenSSL->scrapy)
Downloading https://files.pythonhosted.org/packages/62/76/135eeffe0089e6724bdd65c1bf9f1654db9b47783e65b8d9f1454c540d8b/cffi-1.12.3-cp35-cp35m-manylinux1_x86_64.whl (429kB)
|████████████████████████████████| 430kB 9.4kB/s
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyOpenSSL->scrapy)
Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
|████████████████████████████████| 163kB 4.5kB/s
Installing collected packages: zope.interface, constantly, incremental, attrs, Automat, hyperlink, PyHamcrest, Twisted, w3lib, asn1crypto, pycparser, cffi, cryptography, pyOpenSSL, queuelib, pyasn1, pyasn1-modules, service-identity, PyDispatcher, parsel, scrapy
Running setup.py install for Twisted ... done
Running setup.py install for pycparser ... done
Running setup.py install for PyDispatcher ... done
Successfully installed Automat-0.7.0 PyDispatcher-2.0.5 PyHamcrest-1.9.0 Twisted-19.2.0 asn1crypto-0.24.0 attrs-19.1.0 cffi-1.12.3 constantly-15.1.0 cryptography-2.7 hyperlink-19.0.0 incremental-17.5.0 parsel-1.5.1 pyOpenSSL-19.0.0 pyasn1-0.4.5 pyasn1-modules-0.2.5 pycparser-2.19 queuelib-1.5.0 scrapy-1.6.0 service-identity-18.1.0 w3lib-1.20.0 zope.interface-4.6.0
[[email protected] python]# scrapy version
Scrapy 1.6.0
![scrapy安裝_scrapy架構介紹1.Scrapy五大基本構成2.使用scrapy架構爬取網頁資料[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)