以后再也不用擔(dān)心寫(xiě)爬蟲(chóng)ip被封,不用擔(dān)心沒(méi)錢(qián)買(mǎi)代理ip的煩惱了
在使用python寫(xiě)爬蟲(chóng)時(shí)候,你會(huì)遇到所要爬取的網(wǎng)站有反爬取技術(shù)比如用同一個(gè)IP反復(fù)爬取同一個(gè)網(wǎng)頁(yè),很可能會(huì)被封。如何有效的解決這個(gè)問(wèn)題呢?我們可以使用代理ip,來(lái)設(shè)置代理ip池。
現(xiàn)在教大家一個(gè)可獲取大量免費(fèi)有效快速的代理ip方法,我們?cè)L問(wèn)西刺免費(fèi)代理ip網(wǎng)址
這里面提供了許多代理ip,但是我們嘗試過(guò)后會(huì)發(fā)現(xiàn)并不是每一個(gè)都是有效的。所以我們現(xiàn)在所要做的就是從里面提供的篩選出有效快速穩(wěn)定的ip。
以下介紹的免費(fèi)獲取代理ip池的方法:
優(yōu)點(diǎn):免費(fèi)、數(shù)量多、有效、速度快
缺點(diǎn):需要定期篩選
主要思路:
從網(wǎng)址上爬取ip地址并存儲(chǔ)
驗(yàn)證ip是否能使用-(隨機(jī)訪問(wèn)網(wǎng)址判斷響應(yīng)碼)
格式化ip地址
代碼如下:
1.導(dǎo)入包
import requests
from lxml import etree
import time
1
2
3
2.獲取西刺免費(fèi)代理ip網(wǎng)址上的代理ip
def get_all_proxy():
url = 'http://www.xicidaili.com/nn/1'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
}
response = requests.get(url, headers=headers)
html_ele = etree.HTML(response.text)
ip_eles = html_ele.xpath('//table[@id="ip_list"]/tr/td[2]/text()')
port_ele = html_ele.xpath('//table[@id="ip_list"]/tr/td[3]/text()')
proxy_list = []
for i in range(0,len(ip_eles)):
proxy_str = 'http://' + ip_eles[i] + ':' + port_ele[i]
proxy_list.append(proxy_str)
return proxy_list
1
2
3
4
5
6
7
8
9
10
11
12
13
14
3.驗(yàn)證獲取的ip
def check_all_proxy(proxy_list):
valid_proxy_list = []
for proxy in proxy_list:
url = 'http://www.baidu.com/'
proxy_dict = {
'http': proxy
}
try:
start_time = time.time()
response = requests.get(url, proxies=proxy_dict, timeout=5)
if response.status_code == 200:
end_time = time.time()
print('代理可用:' + proxy)
print('耗時(shí):' + str(end_time - start_time))
valid_proxy_list.append(proxy)
else:
print('代理超時(shí)')
except:
print('代理不可用--------------->'+proxy)
return valid_proxy_list
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
4.輸出獲取ip池
if __name__ == '__main__':
proxy_list = get_all_proxy()
valid_proxy_list = check_all_proxy(proxy_list)
print('--'*30)
print(valid_proxy_list)
1
2
3
4
5
技術(shù)能力有限歡迎提出意見(jiàn),保證積極向上不斷學(xué)習(xí)
————————————————
版權(quán)聲明:本文為CSDN博主「彬小二」的原創(chuàng)文章,遵循 CC 4.0 BY-SA 版權(quán)協(xié)議,轉(zhuǎn)載請(qǐng)附上原文出處鏈接及本聲明。
原文鏈接:https://blog.csdn.net/qq_39884947/article/details/86609930
標(biāo)簽:
python
ip
代理
防止
上傳時(shí)間:
2019-11-15
上傳用戶:fygwz1982
CH341系列編程器芯片usb轉(zhuǎn)串口Altium Designer AD原理圖庫(kù)元件庫(kù)CSV text has been written to file : 1.9 - CH341系列編程器芯片.csvLibrary Component Count : 56Name Description----------------------------------------------------------------------------------------------------CH311Q PC debug port monitorCH331T Mini USB Disk ControllerCH340G CH340H USB to TTL Serial / UART, USB to IrDACH340T USB to TTL Serial / UART, USB to IrDACH340R USB to IrDA, USB to RS232 SerialCH340S_P USB to Print Port / ParallelCH340S_S USB to TTL Serial / UART, pin compatible with CH341CH341A_S USB to TTL Serial / UART / I2C/IICCH341S_P USB to Print Port / ParallelCH341A_P USB to Print Port / ParallelCH341S_S USB to TTL Serial / UARTCH341S_X USB to EPP Parallel / SPI / I2C/IICCH341A_X USB to EPP Parallel / SPI / I2C/IICCH341T USB to TTL Serial / UART / I2C/IICCH345T USB to MidiCH352L_M PCI to 8255 mode 2 Parallel for MCU and 16C550 UART / IrDACH352L_P PCI to Print Port / Parallel and 16C550 UART / IrDACH352L_S PCI to Dual 16C550 UART, TTL Serial*2 / IrDA*1CH362L PCI Device / Slave only for RAM / Expansion ROMCH364F Member of CH364 chipsetsCH364P PCI Device / Slave Embedded Flash ROM, for Expansion ROMCH365P PCI Device / Slave, for I/O port or RAM / ROMCH372T USB Device / Slave for MCU, ParallelCH372A USB Device / Slave for MCU, ParallelCH372V USB Device / Slave for MCU, ParallelCH374S USB Host & Device / Slave for MCU, parallel / SPICH374T USB Host & Device / Slave for MCU, parallel / SPICH375S USB Host & Device / Slave for MCU, parallel / UART SerialCH375A USB Host & Device / Slave for MCU, parallel / UART SerialCH375V USB Host & Device / Slave for MCU, parallel / UART SerialCH411G FDC MFM encode and decodeCH421A Dual port bufferCH421S Dual port bufferCH423D I2C/IIC I/O expander, 16 GPO + 8 GPIO, 128 LEDs DriveCH423S I2C/IIC I/O expander, 16 GPO + 8 GPIO, 128 LEDs DriveCH423D_D I2C/IIC I/O expander, 16 GPO + 8 GPIO, 128 LEDs DriveCH423S_D I2C/IIC I/O expander, 16 GPO + 8 GPIO, 128 LEDs DriveCH423G I2C/IIC I/O expander, 6 GPO + 5 GPIOCH432Q Dual 16C550 UART with IrDA, parallel / SPICH432T SPI Dual 16C550 UART with IrDACH450K 6 Digits / 48 LEDs Drive & 8x6 Keyboard, I2C/IICCH450H 6 Digits / 48 LEDs Drive & 8x6 Keyboard, I2C/IICCH450L 8 Digits / 64 LEDs Drive & 8x8 Keyboard, I2C/IICCH451L 8 Digits / 64 LEDs Drive & 8x8 Keyboard, 4 Wire Interface, SPICH451S 8 Digits / 64 LEDs Drive & 8x8 Keyboard, 4 Wire Interface, SPICH451D 8 Digits / 64 LEDs Drive & 8x8 Keyboard, 4 Wire Interface, SPICH452L_2 8 Digits / 64 LEDs Drive & 8x8 Keyboard, I2C/IICCH452L_4 8 Digits / 64 LEDs Drive & 8x8 Keyboard, 4 Wire Interface, SPICH452S_2 8 Digits / 64 LEDs Drive & 8x8 Keyboard, I2C/IICCH452S_4 8 Digits / 64 LEDs Drive & 8x8 Keyboard, 4 Wire Interface, SPICH453S 16 Digits / 128 LEDs Drive, I2C/IICCH453D 16 Digits / 128 LEDs Drive, I2C/IICPCI 32Bit PCI Bus, simple / short cardPCI32 32Bit PCI BusUSB USB Port
標(biāo)簽:
ch341
編程芯片
usb
串口
altium designer
上傳時(shí)間:
2022-03-13
上傳用戶: