MyDLP Blog

If your users generally upload file with larger sizes then usual, you may need to modify Squid timeout options. Because, when MyDLP is inspecting big files (generally several times compressed files ), total inspection time increases naturally. And as a result, Squid could give timeout errors instead of waiting MyDLP.

To make Squid wait MyDLP, in /etc/squid/squid3.conf you should modify these parameters as;

read_timeout 30 minutes
request_timeout 30 minutes
persistent_request_timeout 30 minutes
pconn_timeout 30 minutes

Today, these worked great for us.

Yesterday, in our Squid cache optimization trials “aufs” created problems. We have seen warning lines about “queue congestion in disk I/O operations” in cache.log and at the same time we have detected unreturned HTTP requests.

Then, we have tried “diskd” engine for cache storage system. Results were wonderful. After switching to “diskd”,  warning lines or unreturned requests were disappeared.

Difference between “aufs” and “diskd”, in order to avoid blocking Squid main process with disk I/O requests, “aufs” delegates these requests to new POSIX threads, “diskd” delegates them to another separate process. And in our site, “diskd” gives better results.

I guess we’ll continue with “diskd”.

Several documents about how to optimize Squid cache are available all around the web. Obviously, there is no silver bullet, there is no definite solution.

But these days, we are trying to optimize a Squid cache which is using MyDLP as ICAP server. In our site there are about 1,5K users which are very active web users. Access times are very important and delays create problems.

Our servers have 8-core Xeon CPUs and 16GB of memory.

To improve performance and decrease access times, I have searched through the web and created myself a startup recipe. I’ll also write about results and further changes in this recipe.

Linux commands for setting up OS limits;

echo 1024 32768 > /proc/sys/net/ipv4/ip_local_port_range
echo 8192 > /proc/sys/net/ipv4/tcp_max_syn_backlog
ulimit -HSd unlimited
ulimit -HSn 16384

Parameter changes in /etc/squid3/squid.conf file;

icp_port 0
htcp_port 0
icp_access deny all
htcp_access deny all
snmp_port 0
snmp_access deny all
dns_nameservers 127.0.0.1
cache_mem 2048 MB
pipeline_prefetch on
memory_pools on
memory_pools_limit 2048 MB
maximum_object_size 2048 KB
maximum_object_size_in_memory 1024 KB
ipcache_size 4096
ipcache_low 90
ipcache_high 95
cache_dir diskd /var/spool/squid3 4096 16 256
cache_store_log none
log_fqdn off
half_closed_clients off
cache_swap_high 100%
cache_swap_low 80%

Few hours later from now, I’ll push this configuration to one of our servers in production.

Everybody who played with Squid on Ubuntu, have probably encountered with this problem; Ubuntu Squid packages had been compiled without SSL option. Therefore, it is not possible to proxy HTTPS connections with Squid on Ubuntu Server.

Have a good day

The upcoming MyDLP SaaS will be working with your preferred ICAP enabled proxy server. So we are going to post ICAP configuration examples for widely used proxies, starting with Squid. Squid is an open source caching proxy which has a very large user base. It is open source and very stable.

Squid 3.0 and 3.1 have out of the box ICAP support with slightly different configuration parameters. Unfortunately, older versions have no built-in ICAP support but it is possible to enable it using patches.

For Squid 3.1 enter following lines into squid.conf file to enable request adaptation:

icap_enable on

icap_service service_req reqmod_precache bypass=0 icap://127.0.0.1:1344/dlp
adaptation_access service_req allow all

For Squid 3.0 enter following lines into squid.conf file to enable request adaptation:

icap_enable on

icap_service service_req reqmod_precache 0 icap://127.0.0.1:1344/dlp
icap_class class_req service_req
icap_access class_req allow all


There are more parameters for configuring ICAP which are explained here.