I am writing this blog post because when I setup Varnish is very painful to learn, because varnish does not work out of the box. It needs to be configured to even start on redhat. Although there are some great posts out there on how to setup, they all fail to mention key details that every newb wants to know and ends up digging all over the net to find. So I have decided to save everyone the trouble and I\u2019m writing it from beginning to end with descriptions and why and how it all works.<\/p>\n
Understanding The Architecture and process model<\/p>\n
<\/p>\n Varnish has two main processes: the management process and the child process. The management process apply configuration changes (VCL and parameters), compile VCL, monitor Varnish, initialize Varnish and provides a command line interface, accessible either directly on the terminal or through a management interface.<\/p>\n The management process polls the child process every few seconds to see if it\u2019s still there. If it doesn\u2019t get a reply within a reasonable time, the management process will kill the child and start it back up again. The same happens if the child unexpectedly exits, for example from a segmentation fault or assert error.<\/p>\n This ensures that even if Varnish does contain a critical bug, it will start back up again fast. Usually within a few seconds, depending on the conditions.<\/p>\n The child process<\/strong><\/span><\/p>\n The child process consist of several different types of threads, including, but not limited to: It is important to remember that even if you have 5MB of session workspace and are using 1000 threads, the actual memory usage is not 5GB. The virtual memory usage will indeed be 5GB, but unless you actually use the memory, this is not a problem. Your memory controller and operating system will keep track of what you actually use.<\/p>\n To communicate with the rest of the system, the child process uses a shared memory log accessible from the file system. This means that if a thread needs to log something, all it has to do is grab a lock, write to a memory area and then free the lock. In addition to that, each worker thread has a cache for log data to reduce lock contention.<\/p>\n The log file is usually about 90MB, and split in two. The first part is counters, the second part is request data. To view the actual data, a number of tools exist that parses the shared memory log. Because the log-data is not meant to be written to disk in its raw form, Varnish can afford to be very verbose. You then use one of the log-parsing tools to extract the piece of information you want – either to store it permanently or to monitor Varnish in real-time.<\/p>\n All of this is logged to syslog. This makes it crucially important to monitor the syslog, otherwise you may never even know unless you look for them, because the perceived downtime is so short.<\/p>\n VCL compilation<\/p>\n Configuring the caching policies of Varnish is done in the Varnish Configuration Language (VCL). Your VCL is then interpreted by the management process into to C and then compiled by a normal C compiler – typically gcc. Lastly, it is linked into the running Varnish instance.<\/p>\n As a result of this, changing configuration while Varnish is running is very cheap. Varnish may want to keep the old configuration around for a bit in case it still has references to it, but the policies of the new VCL takes effect immediately.<\/p>\n Because the compilation is done outside of the child process, there is no risk of affecting the running Varnish by accidentally loading an ill-formated VCL.<\/p>\n A compiled VCL file is kept around until you restart Varnish completely, or until you issue vcl.discard from the management interface. You can only discard compiled VCL files after all references to them are gone, and the amount of references left is part of the output of vcl.list.<\/p>\n Storage backends<\/p>\n Varnish supports different methods of allocating space for the cache, and you choose which one you want with the -s argument.<\/p>\n file They approach the same basic problem from two different angles. With the malloc-method, Varnish will request the entire size of the cache with a malloc() (memory allocation) library call. The operating system divides the cache between memory and disk by swapping out what it can\u2019t fit in memory.<\/p>\n The alternative is to use the file storage backend, which instead creates a file on a filesystem to contain the entire cache, then tell the operating system through the mmap() (memory map) system call to map the entire file into memory if possible.<\/p>\n The file storage method does not retain data when you stop or restart Varnish! This is what persistent storage is for. When -s file is used, Varnish does not keep track of what is written to disk and what is not. As a result, it\u2019s impossible to know whether the cache on disk can be used or not \u2014 it\u2019s just random data. Varnish will not (and can not) re-use old cache if you use -s file.<\/p>\n While malloc will use swap to store data to disk, file will use memory to cache the data instead. Varnish allow you to choose between the two because the performance of the two approaches have varied historically.<\/p>\n The persistent storage backend is similar to file, but experimental. It does not yet gracefully handle situations where you run out of space. We only recommend using persistent if you have a large amount of data that you must cache and are prepared to work with us to track down bugs.<\/p>\n Tunable parameters<\/p>\n In the CLI:<\/p>\n param.show -l<\/p>\n Varnish has many different parameters which can be adjusted to make Varnish act better under specific workloads or with specific software and hardware setups. They can all be viewed with param.show in the management interface and set with the -p option passed to Varnish – or directly in the management interface.<\/p>\n Remember that changes made in the management interface are not stored anywhere, so unless you store your changes in a startup script, they will be lost when Varnish restarts.<\/p>\n The general advice with regards to parameters is to keep it simple. Most of the defaults are very good, and even though they might give a small boost to performance, it\u2019s generally better to use safe defaults if you don\u2019t have a very specific need.<\/p>\n A few hidden commands exist in the CLI, which can be revealed with help -d. These are meant exclusively for development or testing, and many of them are downright dangerous. They are hidden for a reason, and the only exception is perhaps debug.health, which is somewhat common to use.<\/p>\n The shared memory log<\/strong><\/span><\/p>\n Varnish\u2019 shared memory log is used to log most data. It\u2019s sometimes called a shm-log, and operates on a round-robin capacity.<\/p>\n There\u2019s not much you have to do with the shared memory log, except ensure that it does not cause I\/O. This is easily accomplished by putting it on a tmpfs.<\/p>\n This is typically done in \u2018\/etc\/fstab\u2019, and the shmlog is normally kept in \u2018\/var\/lib\/varnish\u2019 or equivalent locations. All the content in that directory is safe to delete.<\/p>\n The shared memory log is not persistent, so do not expect it to contain any real history.<\/p>\n The typical size of the shared memory log is 80MB. If you want to see old log entries, not just real-time, you can use the -d argument for varnishlog: varnishlog -d.<\/p>\n Warning: Some packages will use -s file by default with a path that puts the storage file in the same directory as the shmlog. You want to avoid this.<\/p>\n Threading model <\/p>\n The child process of Varnish is where the magic takes place. It consists of several distinct threads performing different tasks. The following table lists some interesting threads, to give you an idea of what goes on. The table is not complete.Thread-name\u00a0\u00a0\u00a0 Amount of threads\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Task<\/p>\n cache-worker\u00a0\u00a0\u00a0 One per active connection\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Handle requests Most of the time, we only deal with the cache-worker threads when configuring Varnish. With the exception of the amount of thread pools, all the other threads are not configurable.<\/p>\n For tuning Varnish, you need to think about your expected traffic. The thread model allows you to use multiple thread pools, but time and experience has shown that as long as you have 2 thread pools, adding more will not increase performance.<\/p>\n The most important thread setting is the number of worker threads.<\/p>\n Note: If you run across tuning advice that suggests running one thread pool for each CPU core, res assured that this is old advice. Experiments and data from production environments have revealed that as long as you have two thread pools (which is the default), there is nothing to gain by increasing the number of thread pools.<\/p>\n <\/p>\n Threading parameters Details of threading parameters<\/span><\/strong><\/p>\n <\/p>\n While most parameters can be left to the defaults, the exception is the number of threads.Varnish will use one thread for each session and the number of threads you let Varnish use is directly proportional to how many requests Varnish can serve concurrently.The available parameters directly related to threads are:Parameter thread_pool_add_delay\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 2 [milliseconds] Among these, thread_pool_min and thread_pool_max are most important. The thread_pools parameter is also of some importance, but mainly because it is used to calculate the final number of threads.<\/p>\n Varnish operates with multiple pools of threads. When a connection is accepted, the connection is delegated to one of these thread pools. The thread pool will further delegate the connection to available thread if one is available, put the connection on a queue if there are no available threads or drop the connection if the queue is full. By default, Varnish uses 2 thread pools, and this has proven sufficient for even the most busy Varnish server.<\/p>\n For the sake of keeping things simple, the current best practice is to leave thread_pools at the default 2 [pools].<\/p>\n Number of threads<\/p>\n Varnish has the ability to spawn new worker threads on demand, and remove them once the load is reduced. This is mainly intended for traffic spikes. It\u2019s a better approach to try to always keep a few threads idle during regular traffic than it is to run on a minimum amount of threads and constantly spawn and destroy threads as demand changes. As long as you are on a 64-bit system, the cost of running a few hundred threads extra is very limited.<\/p>\n The thread_pool_min parameter defines how many threads will be running for each thread pool even when there is no load. thread_pool_max defines the maximum amount of threads that will be used per thread pool.<\/p>\n The defaults of a minimum of 5 [threads] and maximum 500 [threads] threads per thread pool and 2 [pools] will result in:<\/p>\n At any given time, at least 5 [threads] * 2 [pools] worker threads will be running<\/p>\n No more than 500 [threads] * 2 [pools] threads will run.<\/p>\n We rarely recommend running with more than 5000 threads. If you seem to need more than 5000 threads, it\u2019s very likely that there is something not quite right about your setup, and you should investigate elsewhere before you increase the maximum value.<\/p>\n For minimum, it\u2019s common to operate with 500 to 1000 threads minimum (total). You can observe if this is enough through varnishstat, by looking at the N queued work requests (n_wrk_queued) counter over time. It should be fairly static after startup.<\/p>\n Timing thread growth<\/p>\n Varnish can use several thousand threads, and has had this capability from the very beginning. Not all operating system kernels were prepared to deal with this, though, so the parameter thread_pool_add_delay was added which ensures that there is a small delay between each thread that spawns. As operating systems have matured, this has become less important and the default value of thread_pool_add_delay has been reduced dramatically, from 20ms to 2ms.<\/p>\n There are a few, less important parameters related to thread timing. The thread_pool_timeout is how long a thread is kept around when there is no work for it before it is removed. This only applies if you have more threads than the minimum, and is rarely changed.<\/p>\n An other is the thread_pool_fail_delay, which defines how long to wait after the operating system denied us a new thread before we try again.<\/p>\n System parameters<\/strong><\/span><\/p>\n As Varnish has matured, fewer and fewer parameters require tuning. The sess_workspace is one of the parameters that could still pose a problem. Workspaces are some of the things you can change with parameters. The session workspace is how much memory is allocated to each HTTP session for tasks like string manipulation of incoming headers. It is also used to modify the object returned from a web server before the precise size is allocated and the object is stored read-only.<\/p>\n Some times you may have to increase the session workspace to avoid running out of workspace.<\/p>\n <\/p>\n As most of the parameters can be left unchanged, we will not go through all of them, but take a look at the list param.show gives you to get an impression of what they can do.<\/p>\n TimersParameter\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Default Description\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Scope The timeout-parameters are generally set to pretty good defaults, but you might have to adjust them for strange applications. The connection timeout is tuned for a geographically close web server, and might have to be increased if your Varnish server and web server are not close.<\/p>\n Keep in mind that the session timeout affects how long sessions are kept around, which in turn affects file descriptors left open. It is not wise to increase the session timeout without taking this into consideration.<\/p>\n The cli_timeout is how long the management thread waits for the worker thread to reply before it assumes it is dead, kills it and starts it back up. The default value seems to do the trick for most users today.<\/p>\n Now that you have read this you can go read My I put this post together because you kind of need to understand these things before you try and setup varnish, otherwise you will be trial and error like I was which took a bit a longer. If I had known these things it would of helped. Varnish 3.0 How it works I am writing this blog post because when IRead More …<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[58,138,1],"tags":[],"class_list":["post-49","post","type-post","status-publish","format-standard","hentry","category-centos","category-linux","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/49","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/comments?post=49"}],"version-history":[{"count":6,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/49\/revisions"}],"predecessor-version":[{"id":1626,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/49\/revisions\/1626"}],"wp:attachment":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/media?parent=49"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/categories?post=49"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/tags?post=49"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"http:\/\/www.nicktailor.com\/wp-content\/uploads\/2012\/09\/architecture.png\" "\\"architecture\\"")
<\/a><\/p>\n
\nAcceptor thread to accept new connections and delegate them.\u00a0Worker threads – one per session. It\u2019s common to use hundreds of worker threads.\u00a0Expiry thread, to evict old content from the cache\u00a0Varnish uses workspaces to reduce the contention between each thread when they need to acquire or modify memory. There are multiple workspaces, but the most important one is the session workspace, which is used to manipulate session data. An example is changing www.example.com to example.com before it is entered into the cache, to reduce the number of duplicates.<\/p>\n
\nmalloc
\npersistent (experimental)
\nRule of thumb: malloc if it fits in memory, file if it doesn\u2019t
\nExpect around 1kB of overhead per object cached<\/p>\n
\nThe child process runs multiple threads
\nWorker threads are the bread and butter of the Varnish architecture
\nUtility-threads
\nBalance<\/p>\n
\ncache-main\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 One\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Startup
\nban lurker\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 One\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Clean bans
\nacceptor\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 One\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Accept new connections
\nepoll\/kqueue\u00a0\u00a0\u00a0 Configurable, default: 2\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Manage thread pools
\nexpire\u00a0\u00a0 One\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Remove old content
\nbackend poll\u00a0\u00a0\u00a0\u00a0\u00a0 One per backend poll\u00a0\u00a0\u00a0\u00a0 Health checks<\/p>\n
\nThread pools can safely be ignored
\nMaximum: Roughly 5000 (total)
\nStart them sooner rather than later
\nMaximum and minimum values are per thread pool<\/p>\n
\nDefault value<\/p>\n
\nthread_pool_add_threshold\u00a0\u00a0\u00a0\u00a0\u00a0 2 [requests]
\nthread_pool_fail_delay\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 200 [milliseconds]
\nthread_pool_max\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 500 [threads]
\nthread_pool_min\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 5 [threads]
\nthread_pool_purge_delay\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1000 [milliseconds]
\nthread_pool_stack\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 65536 [bytes]
\nthread_pool_timeout\u00a0\u00a0\u00a0 300 [seconds]
\nthread_pools\u00a0\u00a0\u00a0\u00a0 2 [pools]
\nthread_stats_rate\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 10 [requests]<\/p>\n
\nsess_workspace – incoming HTTP header workspace (from client)
\nCommon values range from the default of 16384 [bytes] to 10MB
\nESI typically requires exponential growth\u00a0Remember: It\u2019s all virtual – not physical memory.<\/p>\n
\nconnect_timeout\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0.700000 [s]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 OS\/network latency\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Backend
\nfirst_byte_timeout\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 60.000000 [s]\u00a0\u00a0\u00a0\u00a0\u00a0 Page generation?\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Backend
\nbetween_bytes_timeout\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 60.000000 [s]\u00a0\u00a0\u00a0\u00a0\u00a0 Hiccoughs?\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Backend
\nsend_timeout\u00a0\u00a0 60 [seconds]\u00a0\u00a0\u00a0\u00a0\u00a0 Client-in-tunnel\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Client
\nsess_timeout\u00a0\u00a0\u00a0 5 [seconds]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 keep-alive timeout\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Client
\ncli_timeout\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 10 [seconds]\u00a0\u00a0\u00a0\u00a0\u00a0 Management thread->child\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Management<\/p>\n
\nVarnish Configuration for Drupal in HA on Redhat\u00a0<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"