标题：构建高性能缓存推送平台(添加squid支持)[原创]
出处：运维进行时
时间：Fri, 15 Oct 2010 23:13:10 +0000
作者：刘天斯
地址：https://blog.liuts.com/post/217/

内容：
       在web2.0网站系统架构中，缓存无处不在。如一个大的CDN平台，一个普通的页面级缓存等。合理的利用缓存可以提高用户下载速度、降低源服务器性能负载。如何高效去维护一个缓存集群？除了提高命中率便是缓存的推送管理。现与大家分享构建一个高性能的缓存推送平台思路及实现。

一、平台架构图

点击在新窗口中浏览此图片
https://blog.liuts.com/attachment.php?fid=224

       说明：平台使用Django+mysql开发，支持POS/GET两种模式，POST方式用于管理员的手工推送，GET方式则用于接口模式。平台将接收到的URL进行校验、分类后统一向所属缓存服务器组进行telnet pureg请求。遍历完所有主机后返回结果串。

二、平台截图
1、管理UI


2、接口说明


3、接口调用



三、功能说明
 1、普通模式
     只限制于文件或目录的推送
  例：推送http://www.domain.com/web/index.shtml及http://www.domain.com/js/下的全部文件。

  1.1、手工方式
  输入框中输入：
引用

   http://www.domain.com/web/index.shtml
   http://www.domain.com/js/


后点［提交推送］按钮。

  1.2、接口方式
引用

  http://www.domain.com/push/executive/?PushURLid=http://www.domain.com/web/index.shtml,http://www.domain.com/js/&Advancedmode=



 2、高级模式
     支持正则表达式的URL规则(适合专业人员使用)
  例：推送http://www.domain.com/main/下文件名为index，扩展名任意的文件及http://www.domain.com/images/下所有*.swf,*.gif文件。

  2.1、手工方式
  输入框中输入：
引用

   http://www.domain.com/main/index..*$
   http://www.domain.com/images/.*.(swf|gif)


选中［高级模式］后点［提交推送］按钮。

  2.2、接口方式
引用

  http://www.domain.com/push/executive/?PushURLid=http://www.domain.com/main/index..*$,http://www.domain.com/images/.*.(swf|gif)&Advancedmode=1



 3、注意事项
   手工模式中禁止在输入框中出现空行、多个URL连接。
注：squid主机不支持高级模式及目录推送
三、性能测试
#ab -c 1000 -n 5000 http://sys.domain.com/push/executive/?PushURLid=http://1.domain.com/1.js,http://2.domain.com/img/&Advancedmode=
3.1、Nginx+Django+fastcgi模式测试数据

点击在新窗口中浏览此图片
https://blog.liuts.com/attachment.php?fid=219


3.2、Nginx+Django+UWSGI模式测试数据

点击在新窗口中浏览此图片
https://blog.liuts.com/attachment.php?fid=226


结论：对接口进行了压力测试，以5000次1000并发进行请求，fastcgi模式速度很快，但存在很多返回错误，从后台日志来看，都是NGINX 502 Bad Gatewa信息，说明后端fastcgi已经处理超时了，UWSGI模式速度不及fastcgi，但表现得很稳定。最终选择了UWSGI+Nginx组合。

三、平台部署
1、部署环境平台
参考：Centos5.4+Nginx-0.8.50+UWSGI-0.9.6.2+Django-1.2.3搭建高性能WEB服务器[原创]

2、部署代码
#cd /opt/www
#wget http://blog.liuts.com/Purgesys.tar.gz
#tar -zxvf Purgesys.tar.gz

#cd Purgesys
#vi setting.py
红色为修改项
引用

# Django settings for Purgesys project.

DEBUG = True
TEMPLATE_DEBUG = DEBUG

ADMINS = (
     ('Liuts', 'liutiansi@gmail.com'),
)

MANAGERS = ADMINS


#数据库信息
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'Purgesys',                      # Or path to database file if using sqlite3.
        'USER': 'dbuser',                      # Not used with sqlite3.
        'PASSWORD': 'dbpass',                  # Not used with sqlite3.
        'HOST': '192.168.100.215',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '3306',                      # Set to empty string for default. Not used with sqlite3.
    }
}


#varnish管理端口与varnish启动参数.. -T 127.0.0.1:2603保持一致。
VARNISHADMINPORT=2603
SQUIDADMINPORT=80


# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'Asia/Shanghai'

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'zh-cn'

SITE_ID = 1

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale
USE_L10N = True

# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = '/opt/www/Purgesys/static/'

# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash if there is a path component (optional in other cases).
# Examples: "http://media.lawrence.com", "http://example.com/media/"
MEDIA_URL = '/static/'

# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
ADMIN_MEDIA_PREFIX = '/static/'

# Make this unique, and don't share it with anybody.
SECRET_KEY = 'tj)*mv!pc_)!zsk*cki$u_f9fg9&^)2^64qi#8!c8xjji--sm7'

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.Loader',
    'django.template.loaders.app_directories.Loader',
#     'django.template.loaders.eggs.Loader',
)

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
)

ROOT_URLCONF = 'Purgesys.urls'

TEMPLATE_DIRS = (
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    '/opt/www/Purgesys/templates'
)

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    # Uncomment the next line to enable the admin:
    # 'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',
    'Purgesys.purgeapp',
    'Purgesys.publicclass',
)





3、导入数据库文件
cd Purgesys/sql

#mysql -u dbuser -pdbpass dbname


Generated by Bo-blog 2.1.1 Release