天天看點

使用Curl指令擷取網站Web伺服器類

Curl是一款用于在網絡上傳輸資料的工具,支援HTTP, HTTPS, FTP, FTPS, TFTP, DICT,TELNET,LDAP等協定。通過curl你既可以将伺服器的資料下載下傳下來,也可以将本地的資料上傳到伺服器。curl的選項很多,大家可以參考curl manual。

最簡單的用法,擷取一個網站頁面的源代碼:

[[email protected] ~]# curl www.g.cn

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">

<TITLE>301 Moved</TITLE></HEAD><BODY>

<H1>301 Moved</H1>

The document has moved

<A HREF="http://www.google.cn/webhp?source=g_cn">here</A>.

</BODY></HTML>

[[email protected] ~]# curl www.google.cn

 言歸正傳,下面将告訴大家如何使用-Is選項來擷取網站的web伺服器類型。

-I :擷取網站的HTTP-header資訊

-s :靜默模式。不顯示進度及錯誤資訊

--connect-timeout :設定連接配接逾時秒數

[[email protected] ~]# curl -Is www.163.com

HTTP/1.0 200 OK

Server: nginx

Date: Fri, 08 Jan 2010 14:40:23 GMT

Content-Type: text/html; charset=GBK

Vary: Accept-Encoding

Expires: Fri, 08 Jan 2010 14:41:43 GMT

Cache-Control: max-age=80

Vary: User-Agent

Vary: Accept

X-Cache: MISS from cache.163.com

Connection: close

首先建立一個site.txt檔案,裡面輸入相應的網站。

[[email protected] ~]# head site.txt

www.google.com

www.baidu.com

youa.baidu.com

post.baidu.com

hi.baidu.com

www.sogou.com

www.youdao.com

www.soso.com

www.sohu.com

www.sina.com

之後結合下面的curl.sh腳本,就可以獲知site.txt中網站的伺服器類型了。

#!/bin/sh

IIS=0

nginx=0

apache=0

other=0

if [ ! -f site.txt ]; then

   echo "ERROR:site.txt NOT exists!"

exit 1

fi

total=`wc -l site.txt|awk '{print $1}'`

for website in `cat site.txt`

do

   server=`curl -Is --connect-timeout 15 $website|awk -F":" '/^Server:/{print $2}'`

   echo -e $website":" $server

   if echo $server|grep -i "IIS">/dev/null

     then IIS=`expr $IIS + 1`

   elif echo $server|grep -i "Apache">/dev/null

     then Apache=`expr $Apache + 1`

   elif echo $server|grep -i "nginx">/dev/null

     then nginx=`expr $nginx + 1`

   else other=`expr $other + 1`

   fi

done

echo "--------------------------------------------"

echo -e "Total/tApache/tIIS/tnginx/tother"

echo -e "$total/t$Apache/t$IIS/t$nginx/t$other"

echo -e "100%/t"`echo "scale=5;$Apache/$total*100"|bc|cut -c1-5`"%/t"`echo "scale=5;$IIS/$total*100"|bc|cut -c1-5`"%/t"`echo "scale=5;$nginx/$total*100"|bc|cut -c1-5`"%/t"`echo "scale=5;$other/$total*100"|bc|cut -c1-5`"%/t"

echo "--------------------------------------------"

exit 0

輸出結果:

[email protected] ~]# bash curl.sh

www.google.com: gws

www.baidu.com: BWS/1.0

youa.baidu.com: Apache

post.baidu.com: apache 2.7.18.0

hi.baidu.com: apache 1.1.26.0

www.sogou.com: Apache

www.youdao.com: nginx

www.soso.com: SWS/1.0

www.sohu.com: SWS

www.sina.com: Apache/2.0.54 (Unix)

www.ifeng.com: DnionOS/1.0

www.alibaba.com: Apache/2.0.59 (Unix) mod_AliCookie(for apache2.x)/1.1 aliBeacon/1.0 mod_jk/1.2.25

www.163.com: nginx

www.126.com: Apache

www.263.com: Microsoft-IIS/6.0

www.tom.com: Apache

www.qq.com: nginx/0.6.39

qzone.qq.com: Apache

www.hotmail.com: Microsoft-IIS/6.0

www.wordpress.com: nginx

mail.163.com: Apache

mail.sina.com: Apache/2.2.9 (FreeBSD) mod_ssl/2.2.9 OpenSSL/0.9.7e-p1 PHP/5.2.6 with Suhosin-Patch

mail.tom.com: Apache/1.3.41 (Unix)

blog.sina.com: nginx/0.7.62

blog.sohu.com: nginx

blog.163.com: nginx

blog.hexun.com: Microsoft-IIS/6.0

www.hackbase.com: Microsoft-IIS/6.0

www.mydrivers.com: Microsoft-IIS/6.0

www.enet.com.cn: Apache

www.cnzz.com: Apache/2.2.4 (Unix) PHP/5.2.4

www.soufun.com: BIG-IP

www.chinaz.com: Microsoft-IIS/6.0

www.51la.com: nginx/0.6.35

www.download.com: Apache/2.2

www.godaddy.com: Microsoft-IIS/6.0

www.answers.com: Apache

www.sourceforge.net: BigIP

www.gmail.com: gws

www.msn.com: Microsoft-IIS/6.0

www.cctv.com: Apache

www.xinhuanet.com: Apache

www.renren.com: nginx/0.7.64

www.kaixin001.com: Apache

www.hao123.com: lighttpd

www.114la.com: Apache

www.douban.com: nginx

www.wikimedia.com: Apache/1.3 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.8f AuthPG/1.3 FrontPage/5.0.2.2635

www.jiangmin.com: DnionOS/1.0

www.rising.com.cn: Microsoft-IIS/6.0

www.kaspersky.com: nginx/0.8.15

www.kingsoft.com: Apache/2.0.53

www.avast.com: Apache

www.360.cn: Apache

www.micropoint.com.cn: Apache

www.free-av.com: Apache

www.avg.com: Apache

www.hexun.com: nginx

www.eastmoney.com: Microsoft-IIS/6.0

www.chinahr.com: Microsoft-IIS/6.0

www.51job.com: Apache/2.2.8 (Unix) PHP/5.2.5

www.zhaopin.com: Apache/1.3.37 (Unix)

www.taobao.com: Apache

www.paipai.com: nginx

www.dangdang.com: nginx/0.6.37

www.youku.com: Apache

www.tudou.com: tws/0.1

www.ku6.com: Apache

6.cn: nginx/0.6.14

www.cntv.cn: Apache

media.17173.com: Apache

www.gougou.com: Apache/2.2.8 (Unix) mod_fastcgi/2.4.6

www.xunlei.com: nginx/0.7.62

www.verycd.com: Apache

www.pps.tv: PPStream

www.tianya.cn: Microsoft-IIS/5.0

dzh.mop.com: Resin/3.0.19

www.pconline.com.cn: nginx

www.zol.com.cn: Apache

www.pcpop.com: Microsoft-IIS/6.0

www.it168.com: Microsoft-IIS/6.0

www.newhua.com: Microsoft-IIS/6.0

www.skycn.com: Apache

www.crsky.com: Microsoft-IIS/6.0

www.csdn.net: CWS/1.0.64

www.chinaitlab.com: Microsoft-IIS/6.0

www.51.com: Apache

www.58.com: Microsoft-IIS/6.0

www.jiayuan.com: nginx/0.7.62

www.marry5.com: Apache

www.icbc.com.cn: Microsoft-IIS/6.0

www.abchina.com: Microsoft-IIS/7.5

www.ccb.com: Apache

www.boc.cn: Microsoft-IIS/7.0

www.bankcomm.com: IBM_HTTP_SERVER

www.ebay.com: Apache-Coyote/1.1

www.myspace.com: Microsoft-IIS/7.5

www.aol.com: Apache-Coyote/1.1

www.cnn.com: Apache

www.reuters.com: Apache

www.wto.org: Microsoft-IIS/6.0

www.who.org: nginx/0.6.35

www.gov.cn: Apache

www.court.gov.cn: Apache/2.0.52 (CentOS)

www.moe.edu.cn: Apache/2.0.54 (Unix) mod_jk2/2.0.4

www.mps.gov.cn: IBM_HTTP_Server/6.0.2.23 Apache/2.0.47 (Win32)

www.most.gov.cn: Microsoft-IIS/6.0

www.pbc.gov.cn: Microsoft-IIS/6.0

www.moh.gov.cn: Apache/2.0.59 (Win32)

www.ccnt.gov.cn: Microsoft-IIS/6.0

www.mof.gov.cn: Apache/2.0.59 HP-UX_Apache-based_Web_Server (Unix) DAV/2

www.mod.gov.cn: Apache

www.microsoft.com: Microsoft-IIS/7.5

www.microsoft.com.cn: Microsoft-IIS/7.0

www.centos.org: Apache/2.0.52 (CentOS)

www.redhat.com: Apache

www.ubuntu.com: Apache/2.2.8 (Ubuntu) mod_python/3.3.1 Python/2.5.2 PHP/5.2.4-2ubuntu5.9 with Suhosin-Patch mod_ssl/2.2.8 Ope

nSSL/0.9.8g

www.gentoo.org: Apache

www.debian.org: Apache/2.2.9 (Debian) mod_ssl/2.2.9 OpenSSL/0.9.8g

www.novell.com: Apache

www.slackware.com: Apache/1.3.27 (Unix) PHP/4.3.1

www.opensuse.org: Apache/2.2.10 (Linux/SUSE)

www.wikipedia.org: Apache

www.hp.com: Apache

www.dell.com: Microsoft-IIS/7.0

www.hitachi.com: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.7d

www.toshiba.com: Sun-ONE-Web-Server/6.1

www.nokia.com: "Nokia"

www.sony.com: Apache

www.intel.com: IA Web Server/1.0

www.amd.com: AkamaiGHost

www.nvidia.com: Microsoft-IIS/6.0

www.sun.com: Sun-Java-System-Web-Server/7.0

www.ibm.com: IBM_HTTP_Server

--------------------------------------------

Total   Apache  IIS     nginx   other

134     64      30      19      21

100%    47.76%  22.38%  14.17%  15.67%

--------------------------------------------

統計的結果,似乎和實際使用情況還有些相近。是不是很簡單