构建高性能缓存推送平台(添加squid支持)[原创]
在web2.0网站系统架构中,缓存无处不在。如一个大的CDN平台,一个普通的页面级缓存等。合理的利用缓存可以提高用户下载速度、降低源服务器性能负载。如何高效去维护一个缓存集群?除了提高命中率便是缓存的推送管理。现与大家分享构建一个高性能的缓存推送平台思路及实现。
一、平台架构图
说明:平台使用Django+mysql开发,支持POS/GET两种模式,POST方式用于管理员的手工推送,GET方式则用于接口模式。平台将接收到的URL进行校验、分类后统一向所属缓存服务器组进行telnet pureg请求。遍历完所有主机后返回结果串。
二、平台截图
1、管理UI
2、接口说明
3、接口调用
三、功能说明
1、普通模式
只限制于文件或目录的推送
例:推送http://www.domain.com/web/index.shtml及http://www.domain.com/js/下的全部文件。
1.1、手工方式
输入框中输入:
http://www.domain.com/web/index.shtml
http://www.domain.com/js/
后点[提交推送]按钮。
1.2、接口方式
http://www.domain.com/push/executive/?PushURLid=http://www.domain.com/web/index.shtml,http://www.domain.com/js/&Advancedmode=
2、高级模式
支持正则表达式的URL规则(适合专业人员使用)
例:推送http://www.domain.com/main/下文件名为index,扩展名任意的文件及http://www.domain.com/images/下所有*.swf,*.gif文件。
2.1、手工方式
输入框中输入:
http://www.domain.com/main/index..*$
http://www.domain.com/images/.*.(swf|gif)
选中[高级模式]后点[提交推送]按钮。
2.2、接口方式
http://www.domain.com/push/executive/?PushURLid=http://www.domain.com/main/index..*$,http://www.domain.com/images/.*.(swf|gif)&Advancedmode=1
3、注意事项
手工模式中禁止在输入框中出现空行、多个URL连接。
注:squid主机不支持高级模式及目录推送
三、性能测试
#ab -c 1000 -n 5000 http://sys.domain.com/push/executive/?PushURLid=http://1.domain.com/1.js,http://2.domain.com/img/&Advancedmode=
3.1、Nginx+Django+fastcgi模式测试数据
3.2、Nginx+Django+UWSGI模式测试数据
结论:对接口进行了压力测试,以5000次1000并发进行请求,fastcgi模式速度很快,但存在很多返回错误,从后台日志来看,都是NGINX 502 Bad Gatewa信息,说明后端fastcgi已经处理超时了,UWSGI模式速度不及fastcgi,但表现得很稳定。最终选择了UWSGI+Nginx组合。
三、平台部署
1、部署环境平台
参考:Centos5.4+Nginx-0.8.50+UWSGI-0.9.6.2+Django-1.2.3搭建高性能WEB服务器[原创]
2、部署代码
#cd /opt/www
#wget http://blog.liuts.com/Purgesys.tar.gz
#tar -zxvf Purgesys.tar.gz
#cd Purgesys
#vi setting.py
红色为修改项
# Django settings for Purgesys project.
DEBUG = True
TEMPLATE_DEBUG = DEBUG
ADMINS = (
('Liuts', 'liutiansi@gmail.com'),
)
MANAGERS = ADMINS
#数据库信息
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'Purgesys', # Or path to database file if using sqlite3.
'USER': 'dbuser', # Not used with sqlite3.
'PASSWORD': 'dbpass', # Not used with sqlite3.
'HOST': '192.168.100.215', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '3306', # Set to empty string for default. Not used with sqlite3.
}
}
#varnish管理端口与varnish启动参数.. -T 127.0.0.1:2603保持一致。
VARNISHADMINPORT=2603
SQUIDADMINPORT=80
# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'Asia/Shanghai'
# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'zh-cn'
SITE_ID = 1
# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True
# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale
USE_L10N = True
# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = '/opt/www/Purgesys/static/'
# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash if there is a path component (optional in other cases).
# Examples: "http://media.lawrence.com", "http://example.com/media/"
MEDIA_URL = '/static/'
# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
ADMIN_MEDIA_PREFIX = '/static/'
# Make this unique, and don't share it with anybody.
SECRET_KEY = 'tj)*mv!pc_)!zsk*cki$u_f9fg9&^)2^64qi#8!c8xjji--sm7'
# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
# 'django.template.loaders.eggs.Loader',
)
MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
)
ROOT_URLCONF = 'Purgesys.urls'
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
'/opt/www/Purgesys/templates'
)
INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
# Uncomment the next line to enable the admin:
# 'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
'Purgesys.purgeapp',
'Purgesys.publicclass',
)
3、导入数据库文件
cd Purgesys/sql
一、平台架构图
说明:平台使用Django+mysql开发,支持POS/GET两种模式,POST方式用于管理员的手工推送,GET方式则用于接口模式。平台将接收到的URL进行校验、分类后统一向所属缓存服务器组进行telnet pureg请求。遍历完所有主机后返回结果串。
二、平台截图
1、管理UI
2、接口说明
3、接口调用
三、功能说明
1、普通模式
只限制于文件或目录的推送
例:推送http://www.domain.com/web/index.shtml及http://www.domain.com/js/下的全部文件。
1.1、手工方式
输入框中输入:
引用
http://www.domain.com/web/index.shtml
http://www.domain.com/js/
1.2、接口方式
引用
http://www.domain.com/push/executive/?PushURLid=http://www.domain.com/web/index.shtml,http://www.domain.com/js/&Advancedmode=
2、高级模式
支持正则表达式的URL规则(适合专业人员使用)
例:推送http://www.domain.com/main/下文件名为index,扩展名任意的文件及http://www.domain.com/images/下所有*.swf,*.gif文件。
2.1、手工方式
输入框中输入:
引用
http://www.domain.com/main/index..*$
http://www.domain.com/images/.*.(swf|gif)
2.2、接口方式
引用
http://www.domain.com/push/executive/?PushURLid=http://www.domain.com/main/index..*$,http://www.domain.com/images/.*.(swf|gif)&Advancedmode=1
3、注意事项
手工模式中禁止在输入框中出现空行、多个URL连接。
注:squid主机不支持高级模式及目录推送
三、性能测试
#ab -c 1000 -n 5000 http://sys.domain.com/push/executive/?PushURLid=http://1.domain.com/1.js,http://2.domain.com/img/&Advancedmode=
3.1、Nginx+Django+fastcgi模式测试数据
3.2、Nginx+Django+UWSGI模式测试数据
结论:对接口进行了压力测试,以5000次1000并发进行请求,fastcgi模式速度很快,但存在很多返回错误,从后台日志来看,都是NGINX 502 Bad Gatewa信息,说明后端fastcgi已经处理超时了,UWSGI模式速度不及fastcgi,但表现得很稳定。最终选择了UWSGI+Nginx组合。
三、平台部署
1、部署环境平台
参考:Centos5.4+Nginx-0.8.50+UWSGI-0.9.6.2+Django-1.2.3搭建高性能WEB服务器[原创]
2、部署代码
#cd /opt/www
#wget http://blog.liuts.com/Purgesys.tar.gz
#tar -zxvf Purgesys.tar.gz
#cd Purgesys
#vi setting.py
红色为修改项
引用
# Django settings for Purgesys project.
DEBUG = True
TEMPLATE_DEBUG = DEBUG
ADMINS = (
('Liuts', 'liutiansi@gmail.com'),
)
MANAGERS = ADMINS
#数据库信息
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'Purgesys', # Or path to database file if using sqlite3.
'USER': 'dbuser', # Not used with sqlite3.
'PASSWORD': 'dbpass', # Not used with sqlite3.
'HOST': '192.168.100.215', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '3306', # Set to empty string for default. Not used with sqlite3.
}
}
#varnish管理端口与varnish启动参数.. -T 127.0.0.1:2603保持一致。
VARNISHADMINPORT=2603
SQUIDADMINPORT=80
# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'Asia/Shanghai'
# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'zh-cn'
SITE_ID = 1
# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True
# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale
USE_L10N = True
# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = '/opt/www/Purgesys/static/'
# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash if there is a path component (optional in other cases).
# Examples: "http://media.lawrence.com", "http://example.com/media/"
MEDIA_URL = '/static/'
# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
ADMIN_MEDIA_PREFIX = '/static/'
# Make this unique, and don't share it with anybody.
SECRET_KEY = 'tj)*mv!pc_)!zsk*cki$u_f9fg9&^)2^64qi#8!c8xjji--sm7'
# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
# 'django.template.loaders.eggs.Loader',
)
MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
)
ROOT_URLCONF = 'Purgesys.urls'
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
'/opt/www/Purgesys/templates'
)
INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
# Uncomment the next line to enable the admin:
# 'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
'Purgesys.purgeapp',
'Purgesys.publicclass',
)
3、导入数据库文件
cd Purgesys/sql
wnian
2010/10/16 10:39
支持一下,到时候研究下
分页: 2/2 1 2