运维进行时

一、部署LAMP/LNMP+CodeIgniter框架(略)
二、安装rrdtool
三、下载管理平台包
wget http://tianyalvs.googlecode.com/files/TianyaLVS.tar.gz
cd "web root"
tar -zxvf TianyaLVS.tar.gz
注意事项

1、要求webroot/TianyaLVS/LVS、webroot/TianyaLVS/images/LVS目录可写。
2、系统需.htaccess支持，除rewrite支持外apache还需添加：
<Directory "/www/TianyaLVS">             AllowOverride All   </Directory>
修改项
cd TianyaLVS/TianyaLVSsystem/application/config
vi config.php

安装步骤如下：
ln -s /usr/src/kernels/2.6.9-89.0.9.EL-smp-x86_64/  /usr/src/linux
2.6.9-89.0.9.EL-smp-x86_64与uname -a内核配对。
下载
wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz
wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz

安装
tar zxvf ipvsadm-1.24.tar.gz
cd ipvsadm-1.24
make && make install
cd ..
tar zxvf keepalived-1.1.15.tar.gz
cd keepalived-1.1.15
注意项

./configure
make && make install

Func简介
      Func是由红帽子公司以Fedora平台统一网络控制器 Func（Fedora Unified Network Controller https://fedorahosted.org/func），目的是为了解决这一系列统一管理监控问题而设计开发的系统管理基础框架。 它是一个能有效的简化我们多服务器系统管理工作的工具，它很容易学习、很容易使用、也很容易被扩展，它功能强大而我们只需要非常非常少的配置和维护。
    Func分为master及slave两部分，master为主控端，slave为被控端。以下为两部分的安装配置说明
    FUNC模块学习笔记

     由于第一版的LVS管理平台是整合到现有的运维系统中，现用CodeIgniter框架+certmaster+func重写了老的管理平台，比原系统更具有独立性，主要功能模块分为性能图表、数据中心、虚拟IP池、主机管理、监控模块、生成hosts、主机探测等功能，基本上是按F5－LTM管理平台思路来设计，下面只要对这几大块功能进行说明。
项目托管地址：http://code.google.com/p/tianyalvs/
welcome

1、性能图表
功能说明：以小时、日、星期、月、年的图表展示LVS SERVER、VIP、SERVER等流量情况，效果图如下：

    一个用于管理服务器的运维软件，可以很方便管理上千台Linux服务器。

项目托管地址：http://code.google.com/p/tianyaservmanager/

开源协议
Artistic License/GPL

一、开发包
Client
python 2.5 +
wxpython2.8+
rpyc3.0+
psyco 1.6+
MySQLdb
ConfigParser
pywin32 for py2.5 (windows only)

Server
func
certmaster
cfengine
mysql5.0+
rpyc3.0+

二、支持平台
Windows xp/2000/2003 Linux2.6+ MacOS

三、角色分配
客户端
服务器端

四、功能特点
跨平台、分级管理、实时监控、及时报警、远程操作、可扩展性、在线升级、安全可靠

五、系统架构图

开源客户端地址：http://code.google.com/p/tianyaservmanager/
代码比较简单就不加以说明了,大家有任何疑问可以留言。
服务代码
import time import os import re from cPickle import dumps from rpyc import Service from rpyc.utils.server import ThreadedServer #----------------------------------------------------------------------------  # # Name:        ServManagerServer.py                                               # # Purpose:     system service Module                                                 # # Author:      Liutiansi                                                               # # Email:       liutiansi@gamil.com                                                  # # Created:     2009/03/03                                                            # # Copyright:   (c) 2009                                                              # #-----------------------------------------------------------------------------  # class ManagerService(Service):     #Login--------------------------------------------------------------     def exposed_login(self,user,passwd):         if user=="admin" and passwd=="123456":             self.Checkout_pass=True         else:             self.Checkout_pass=False          #group1---------------------------------------------------------------     def exposed_Runcommands(self,Parameter1="",Parameter2="",Parameter3="",Parameter4=""):         pinglist=""         try:             if self.Checkout_pass!=True:                 return dumps("Illegal User!")         except:                 return dumps("Illegal Login!")      #明码传输                             pinglist = os.popen("/usr/local/bin/python /var/system/ModuleClass.py "+str(Parameter1)+" '"+Parameter2+"' "+str(Parameter3)+" "+str(Parameter4)).readlines()   #加密传输,Decryption方法不提供，用户可自定义。         #pinglist = os.popen("/usr/local/bin/python /var/system/ModuleClass.py "+self.Decryption(str(Parameter1))+" '"+self.Decryption(str(Parameter2))+"' "+self.Decryption(str(Parameter3))+" "+str(Parameter4)).readlines()            return dumps(eval(pinglist[0])) s=ThreadedServer(ManagerService,port=7758,auto_register=False) s.start()

      写爬虫是一项复杂、枯噪、反复的工作，考虑的问题包括采集效率、链路异常处理、数据质量(与站点编码规范关系很大)等。整理自己写一个爬虫程序，单台服务器可以启用1~8个实例同时采集，然后将数据入库。
#-*- coding:utf-8 -*- #!/usr/local/bin/python import sys, time, os,string import mechanize import urlparse from BeautifulSoup import BeautifulSoup import re import MySQLdb import logging import cgi from optparse import OptionParser #----------------------------------------------------------------------------# # Name:        TySpider.py                                                            # # Purpose:     WebSite  Spider Module                                         # # Author:      刘天斯                                                                     # # Email:       liutiansi@gamil.com                                                  # # Created:     2010/02/16                                                            # # Copyright:   (c) 2010                                                                # #----------------------------------------------------------------------------# """ |-------------------------------------------------------------------------- | 定义 loging class; |-------------------------------------------------------------------------- | | 功能：记录系统相关日志信息。 | | """ class Pubclilog():     def __init__(self):         self.logfile = 'website_log.txt'     def iniLog(self):         logger = logging.getLogger()         filehandler = logging.FileHandler(self.logfile)         streamhandler = logging.StreamHandler()         fmt = logging.Formatter('%(asctime)s, %(funcName)s, %(message)s')         logger.setLevel(logging.DEBUG)         logger.addHandler(filehandler)         logger.addHandler(streamhandler)         return [logger,filehandler] """ |-------------------------------------------------------------------------- | 定义 tySpider class; |-------------------------------------------------------------------------- | | 功能：抓取分类、标题等信息 | | """ class BaseTySpider:     #初始化相关成员方法     def __init__(self,X,log_switch):         #数据库连接         self.conn = MySQLdb.connect(db='dbname',host='192.168.0.10', user='dbuser',passwd='SDFlkj934y5jsdgfjh435',charset='utf8')         #分类及标题页面Community         self.CLASS_URL = 'http://test.abc.com/aa/CommTopicsPage?'         #发表回复页         self.Content_URL = 'http://test.bac.com/aa/CommMsgsPage?'         #开始comm值         self.X=X         #当前comm id取模，方面平均到表         self.mod=self.X%5         #Community文件下载页         self.body=""         #self.bodySoup对象         self.soup=None         #发表回复页下载内容变量         self.Contentbody=""         #发表回复页内容self.ContentbodySoup对象         self.Contentsoup=None         #日志开关         self.log_switch=log_switch     #======================获取名称及分类方法==========================     def _SpiderClass(self,nextpage=None):         if nextpage==None:             FIXED_QUERY = 'cmm='+str(self.X)         else:             FIXED_QUERY = nextpage[1:]         try:             rd = mechanize.Browser()             rd.addheaders = [("User-agent", "Tianya/2010 (compatible; MSIE 6.0;Windows NT 5.1)")]             rd.open(self.CLASS_URL + FIXED_QUERY)             self.body=rd.response().read()             #rd=mechanize.Request(self.CLASS_URL + FIXED_QUERY)             #response = mechanize.urlopen(rd)             #self.body=response.read()         except Exception,e:             if self.log_switch=="on":                 logapp=Pubclilog()                 logger,hdlr = logapp.iniLog()                 logger.info(self.CLASS_URL + FIXED_QUERY+str(e))                 hdlr.flush()                 logger.removeHandler(hdlr)                 return         self.soup = BeautifulSoup(self.body)         NextPageObj= self.soup("a", {'class' : re.compile("fs-paging-item fs-paging-next")})         self.cursor = self.conn.cursor()         if nextpage==None:             try:                 Ttag=str(self.soup.table)                 #print Ttag                 """                 ------------------分析结构体-----------------                 <table cellspacing="0" cellpadding="0">                     <tr>                         <td>                             <h1 title="Dunhill">Dunhill</h1>                         </td>                        <td valign="middle">                             <div class="fs-comm-cat">                                 <span class="fs-icons fs-icon-cat"> </span> <a href="TopByCategoryPage?cid=211&ref=commnav-cat">中国</a> » <a href="TopByCategoryPage?cid=211&subcid=273&ref=commnav-cat">人民</a>                             </div>                         </td>                     </tr>                 </table>                 """                 soupTable=BeautifulSoup(Ttag)                 #定位到第一个h1标签                 tableh1 = soupTable("h1")                 #print self.X                 #print "Name:"+tableh1[0].string.strip().encode('utf-8')                 #处理无类型的                 try:                     #定位到表格中符合规则“^TopByCategory”A链接块，tablea[0]为第一个符合条件的连接文字，tablea[1]...                     tablea = soupTable("a", {'href' : re.compile("^TopByCategory")})                     if tablea[0].string.strip()=="":                         pass                     #print "BigCLass:"+tablea[0].string.strip().encode('utf-8')                     #print "SubClass:"+tablea[1].string.strip().encode('utf-8')                 except Exception,e:                     if self.log_switch=="on":                         logapp=Pubclilog()                         logger,hdlr = logapp.iniLog()                         logger.info("[noClassInfo]"+str(self.X)+str(e))                         hdlr.flush()                         logger.removeHandler(hdlr)                     self.cursor.execute("insert into baname"+str(self.mod)+" values('%d','%d','%s')" %(self.X,-1,tableh1[0].string.strip().encode('utf-8')))                     self.conn.commit()                     self._SpiderTitle()                     if NextPageObj:                         NextPageURL=NextPageObj[0]['href']                         self._SpiderClass(NextPageURL)                         return                     else:                         return                 #获取链接二对象的href值                 classlink=tablea[1]['href']                 par_dict=cgi.parse_qs(urlparse.urlparse(classlink).query)                 #print "CID:"+par_dict["cid"][0]                 #print "SubCID:"+par_dict["subcid"][0]                 #print "---------------------------------------"                                  #插入数据库                 self.cursor.execute("insert into class values('%d','%s')" %(int(par_dict["cid"][0]),tablea[0].string.strip().encode('utf-8')))                 self.cursor.execute("insert into subclass values('%d','%d','%s')" %(int(par_dict["subcid"][0]),int(par_dict["cid"][0]),tablea[1].string.strip().encode('utf-8')))                 self.cursor.execute("insert into baname"+str(self.mod)+" values('%d','%d','%s')" %(self.X,int(par_dict["subcid"][0]),tableh1[0].string.strip().encode('utf-8')))                 self.conn.commit()                                  self._SpiderTitle()                 if NextPageObj:                     NextPageURL=NextPageObj[0]['href']                     self._SpiderClass(NextPageURL)                 self.body=None                 self.soup=None                 Ttag=None                 soupTable=None                 table=None                 table1=None                 classlink=None                 par_dict=None             except Exception,e:                 if self.log_switch=="on":                     logapp=Pubclilog()                     logger,hdlr = logapp.iniLog()                     logger.info("[ClassInfo]"+str(self.X)+str(e))                     hdlr.flush()                     logger.removeHandler(hdlr)         else:             self._SpiderTitle()             if NextPageObj:                 NextPageURL=NextPageObj[0]['href']                 self._SpiderClass(NextPageURL)     #====================获取标题方法=========================     def _SpiderTitle(self):         #查找标题表格对象(table)         soupTitleTable=self.soup("table", {'class' : "fs-topic-list"})                  #查找标题行对象(tr)         TitleTr = soupTitleTable[0]("tr", {'onmouseover' : re.compile("^this\.className='fs-row-hover'")})         """         -----------分析结构体--------------         <tr class="fs-alt-row" onmouseover="this.className='fs-row-hover'" onmouseout="this.className='fs-alt-row'">             <td valign="middle" class="fs-hot-topic-dots-ctn">                 <div class="fs-hot-topic-dots" style="background-position:0 -0px" title="点击量：12"></div>             </td>             <td valign="middle" class="fs-topic-name">                 <a href="CommMsgsPage?cmm=16081&tid=2718969307756232842&ref=regulartopics" id="a53" title="【新人报到】欢迎美国人民加入" target="_blank">【新人报到】欢迎美国人民加入</a>                 <span class="fs-meta">                 <span class="fs-icons fs-icon-mini-reply"> </span>0                   /                   <span class="fs-icons fs-icon-pageview"> </span>12</span>             </td>             <td valign="middle">                 <a class="fs-tiny-user-avatar umhook " href="ProfilePage?uid=8765915421039908242" title="中国人"><img src="http://img1.sohu.com.cn/aa/images/138/0/P/1/s.jpg" /></a>             </td>             <td valign="middle" style="padding-left:4px">                 <a href="Profile?uid=8765915421039908242" id="b53" title="中国人" class="umhook">中国人</a>             </td>             <td valign="middle" class="fs-topic-last-mdfy fs-meta">2-14</td>         </tr>         """         for CurrTr in TitleTr:             try:                 #初始化置顶及精华状态                 Title_starred='N'                 Title_sticky='N'                 #获取当前记录的BeautifulSoup对象                 soupCurrTr=BeautifulSoup(str(CurrTr))                 #BeautifulSoup分析HTML有误，只能通过span的标志数来获取贴子状态，会存在一定误差                 #如只有精华时也会当成置顶来处理。                 TitleStatus=soupCurrTr("span", {'title' : ""})                 TitlePhotoViewer=soupCurrTr("a", {'href' : re.compile("^PhotoViewer")})                 if TitlePhotoViewer.__len__()==1:                     TitlePhotoViewerBool=0                 else:                     TitlePhotoViewerBool=1                 if TitleStatus.__len__()==3-TitlePhotoViewerBool:                     Title_starred='Y'                     Title_sticky='Y'                 elif TitleStatus.__len__()==2-TitlePhotoViewerBool:                     Title_sticky='Y'                 #获取贴子标题                 Title=soupCurrTr.a.next.strip()                 #获取贴子ID                 par_dict=cgi.parse_qs(urlparse.urlparse(soupCurrTr.a['href']).query)                 #获取回复数及浏览器                 TitleNum=soupCurrTr("td", {'class' : "fs-topic-name"})                 TitleArray=string.split(str(TitleNum[0]),'\n')                 Title_ReplyNum=string.split(TitleArray[len(TitleArray)-4],'>')[2]                 Title_ViewNum=string.split(TitleArray[len(TitleArray)-2],'>')[2][:-6]                 #获取贴子作者                 TitleAuthorObj=soupCurrTr("td", {'style' : "padding-left:4px"})                 Title_Author=TitleAuthorObj[0].next.next.next.string.strip().encode('utf-8')                 #获取回复时间                 TitleTime=soupCurrTr("td", {'class' : re.compile("^fs-topic-last-mdfy fs-meta")})                 """                 print "X:"+str(self.X)                 print "Title_starred:"+Title_starred                 print "Title_sticky:"+Title_sticky                 print "Title:"+Title                 #获取贴子内容连接URL                 print "Title_link:"+soupCurrTr.a['href']                 print "CID:"+par_dict["tid"][0]                 print "Title_ReplyNum:"+Title_ReplyNum                 print "Title_ViewNum:"+Title_ViewNum                 print "Title_Author:"+Title_Author                 print "TitleTime:"+TitleTime[0].string.strip().encode('utf-8')                 """                 #入库                 self.cursor.execute("insert into Title"+str(self.mod)+" values('%s','%d','%s','%d','%d','%s','%s','%s','%s')" %(par_dict["tid"][0], \                                                                          self.X,Title,int(Title_ReplyNum),int(Title_ViewNum),Title_starred,Title_sticky, \                                                                          Title_Author.decode('utf-8'),TitleTime[0].string.strip().encode('utf-8')))                 self.conn.commit()                 self._SpiderContent(par_dict["tid"][0])             except Exception,e:                 if self.log_switch=="on":                     logapp=Pubclilog()                     logger,hdlr = logapp.iniLog()                     logger.info("[Title]"+str(self.X)+'-'+par_dict["tid"][0]+'-'+str(e))                     hdlr.flush()                     logger.removeHandler(hdlr)     #======================获取发表及回复方法=======================     def _SpiderContent(self,ID,nextpage=None):         if nextpage==None:             FIXED_QUERY = 'cmm='+str(self.X)+'&tid='+ID+'&ref=regulartopics'         else:             FIXED_QUERY = nextpage[9:]         rd = mechanize.Browser()         rd.addheaders = [("User-agent", "Tianya/2010 (compatible; MSIE 6.0;Windows NT 5.1)")]         rd.open(self.Content_URL + FIXED_QUERY)         self.Contentbody=rd.response().read()         #rd=mechanize.Request(self.Content_URL + FIXED_QUERY)         #response = mechanize.urlopen(rd)         #self.Contentbody=response.read()         self.Contentsoup = BeautifulSoup(self.Contentbody)         NextPageObj= self.Contentsoup("a", {'class' : re.compile("fs-paging-item fs-paging-next")})         try:             Tdiv=self.Contentsoup("div", {'class' : "fs-user-action"})             i=0             for Currdiv in Tdiv:                 if i==0:                     Ctype='Y'                 else:                     Ctype='N'                 #发表时间                 soupCurrdiv=BeautifulSoup(str(Currdiv))                 PosttimeObj=soupCurrdiv("span", {'class' : "fs-meta"})                 Posttime=PosttimeObj[0].next[1:]                 Posttime=Posttime[0:-3]                 #IP地址                 IPObj=soupCurrdiv("a", {'href' : re.compile("CommMsgAddress")})                 if IPObj:                     IP=IPObj[0].next.strip()                 else:                     IP=''                 #发表／回复内容                 ContentObj=soupCurrdiv("div", {'class' :"fs-user-action-body"})                 Content=ContentObj[0].renderContents().strip()                 """                 print "ID:"+str(self.X)                 print "ID:"+ID                 print "Ctype:"+Ctype                 print "POSTTIME:"+Posttime                 print "IP:"+IP                 print "Content:"+Content                 """                 self.cursor.execute("insert into Content"+str(self.mod)+" values('%s','%d','%s','%s','%s','%s')" %(ID,self.X,Ctype,Posttime,IP,Content.decode('utf-8')))                 self.conn.commit()                 i+=1         except Exception,e:             if self.log_switch=="on":                 logapp=Pubclilog()                 logger,hdlr = logapp.iniLog()                 logger.info("[Content]"+str(self.X)+'-'+ID+'-'+str(e))                 hdlr.flush()                 logger.removeHandler(hdlr)         #如“下一页”有链接刚继续遍历         if NextPageObj:             NextPageURL=NextPageObj[0]['href']             self._SpiderContent(ID,NextPageURL)     def __del__(self):         try:             self.cursor.close()             self.conn.close()         except Exception,e:             pass #遍历comm范围 def initapp(StartValue,EndValue,log_switch):     for x in range(StartValue,EndValue):         app=BaseTySpider(x,log_switch)         app._SpiderClass()         app=None if __name__ == "__main__":          #定义命令行参数     MSG_USAGE = "TySpider.py [ -s StartNumber EndNumber ] -l [on|off] [-v][-h]"     parser = OptionParser(MSG_USAGE)     parser.add_option("-s", "--set", nargs=2,action="store",             dest="comm_value",             type="int",             default=False,             help="配置名称ID值范围。".decode('utf-8'))     parser.add_option("-l", "--log", action="store",             dest="log_switch",             type="string",             default="on",             help="错误日志开关".decode('utf-8'))     parser.add_option("-v","--version", action="store_true", dest="verbose",             help="显示版本信息".decode('utf-8'))     opts, args = parser.parse_args()     if opts.comm_value:         if opts.comm_value[0]>opts.comm_value[1]:             print "终止值比起始值还小？"             exit();         if opts.log_switch=="on":             log_switch="on"         else:             log_switch="off"         initapp(opts.comm_value[0],opts.comm_value[1],log_switch)         exit();     if opts.verbose:         print "WebSite Scider V1.0 beta."         exit;

如大家有什么疑问或感兴趣的话题可以通过weibo与我交流：http://t.qq.com/yorkoliu

      同一个分析日志的awk脚本，在Centos、ubuntu操作系统运行效率存在巨大差异。即在ubuntu中只需1分钟，而在Centos中则需要20分钟。以下为我的排查步骤：
1、检查服务器内核版本

2、检查内核ulimit参数

3、优化awk脚本

4、检查gawk版本