以后再也不用擔(dān)心寫爬蟲ip被封，不用擔(dān)心沒錢買代理ip的煩惱了 在使用python寫爬蟲時(shí)候，你會(huì)遇到所要爬取的網(wǎng)站有反爬取技術(shù)比如用同一個(gè)IP反復(fù)爬取同一個(gè)網(wǎng)頁，很可能會(huì)被封。如何有效的解決這個(gè)問題呢？我們可以使用代理ip，來設(shè)置代理ip池。 現(xiàn)在教大家一個(gè)可獲取大量免費(fèi)有效快速的代理ip方法，我們?cè)L問西刺免費(fèi)代理ip網(wǎng)址 這里面提供了許多代理ip，但是我們嘗試過后會(huì)發(fā)現(xiàn)并不是每一個(gè)都是有效的。所以我們現(xiàn)在所要做的就是從里面提供的篩選出有效快速穩(wěn)定的ip。 以下介紹的免費(fèi)獲取代理ip池的方法： 優(yōu)點(diǎn)：免費(fèi)、數(shù)量多、有效、速度快 缺點(diǎn)：需要定期篩選 主要思路： 從網(wǎng)址上爬取ip地址并存儲(chǔ) 驗(yàn)證ip是否能使用-（隨機(jī)訪問網(wǎng)址判斷響應(yīng)碼） 格式化ip地址 代碼如下： 1.導(dǎo)入包 import requests from lxml import etree import time 1 2 3 2.獲取西刺免費(fèi)代理ip網(wǎng)址上的代理ip def get_all_proxy(): url = 'http://www.xicidaili.com/nn/1' headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36', } response = requests.get(url, headers=headers) html_ele = etree.HTML(response.text) ip_eles = html_ele.xpath('//table[@id="ip_list"]/tr/td[2]/text()') port_ele = html_ele.xpath('//table[@id="ip_list"]/tr/td[3]/text()') proxy_list = [] for i in range(0,len(ip_eles)): proxy_str = 'http://' + ip_eles[i] + ':' + port_ele[i] proxy_list.append(proxy_str) return proxy_list 1 2 3 4 5 6 7 8 9 10 11 12 13 14 3.驗(yàn)證獲取的ip def check_all_proxy(proxy_list): valid_proxy_list = [] for proxy in proxy_list: url = 'http://www.baidu.com/' proxy_dict = { 'http': proxy } try: start_time = time.time() response = requests.get(url, proxies=proxy_dict, timeout=5) if response.status_code == 200: end_time = time.time() print('代理可用：' + proxy) print('耗時(shí):' + str(end_time - start_time)) valid_proxy_list.append(proxy) else: print('代理超時(shí)') except: print('代理不可用--------------->'+proxy) return valid_proxy_list 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 4.輸出獲取ip池 if __name__ == '__main__': proxy_list = get_all_proxy() valid_proxy_list = check_all_proxy(proxy_list) print('--'*30) print(valid_proxy_list) 1 2 3 4 5 技術(shù)能力有限歡迎提出意見，保證積極向上不斷學(xué)習(xí) ———————————————— 版權(quán)聲明：本文為CSDN博主「彬小二」的原創(chuàng)文章，遵循 CC 4.0 BY-SA 版權(quán)協(xié)議，轉(zhuǎn)載請(qǐng)附上原文出處鏈接及本聲明。 原文鏈接：https://blog.csdn.net/qq_39884947/article/details/86609930
上傳時(shí)間： 2019-11-15
上傳用戶：fygwz1982
使用matlab實(shí)現(xiàn)gibbs抽樣，MCMC: The Gibbs Sampler 多元高斯分布的邊緣概率和條件概率 Marginal and conditional distributions of multivariate normal distribution
上傳時(shí)間： 2019-12-10
上傳用戶：real_
Artificial Intelligence (AI) has undoubtedly been one of the most important buz- zwords over the past years. The goal in AI is to design algorithms that transform com- puters into “intelligent” agents. By intelligence here we do not necessarily mean an extraordinary level of smartness shown by superhuman; it rather often involves very basic problems that humans solve very frequently in their day-to-day life. This can be as simple as recognizing faces in an image, driving a car, playing a board game, or reading (and understanding) an article in a newspaper. The intelligent behaviour ex- hibited by humans when “reading” is one of the main goals for a subfield of AI called Natural Language Processing (NLP). Natural language 1 is one of the most complex tools used by humans for a wide range of reasons, for instance to communicate with others, to express thoughts, feelings and ideas, to ask questions, or to give instruc- tions. Therefore, it is crucial for computers to possess the ability to use the same tool in order to effectively interact with humans.
標(biāo)簽： Embeddings Processing Language Natural in
上傳時(shí)間： 2020-06-10
上傳用戶：shancjb
蟲蟲下載站版權(quán)所有 京ICP備2021023401號(hào)-1
