Optimizing Kallithea performance¶
When serving a large amount of big repositories, Kallithea can start performing slower than expected. Because of the demanding nature of handling large amounts of data from version control systems, here are some tips on how to get the best performance.
Fast storage¶
Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) and plenty of RAM is usually more important than a fast CPU.
Caching¶
Tweak beaker cache settings in the ini file. The actual effect of that is questionable.
Note
Beaker has no upper bound on cache size and will never drop any caches. For memory cache, the only option is to regularly restart the worker process. For file cache, it must be cleaned manually, as described in the Beaker documentation:
find data/cache -type f -mtime +30 -print -exec rm {} \;
Database¶
SQLite is a good option when having a small load on the system. But due to locking issues with SQLite, it is not recommended to use it for larger deployments.
Switching to PostgreSQL or MariaDB/MySQL will result in an immediate performance increase. A tool like SQLAlchemyGrate can be used for migrating to another database platform.
Horizontal scaling¶
Scaling horizontally means running several Kallithea instances (also known as worker processes) and let them share the load. That is essential to serve other users while processing a long-running request from a user. Usually, the bottleneck on a Kallithea server is not CPU but I/O speed - especially network speed. It is thus a good idea to run multiple worker processes on one server.
Note
Kallithea and the embedded Mercurial backend are not thread-safe. Each worker process must thus be single-threaded.
Web servers can usually launch multiple worker processes - for example mod_wsgi
with the
WSGIDaemonProcess
processes
parameter or uWSGI
or gunicorn
with
their workers
setting.
Kallithea can also be scaled horizontally across multiple machines. In order to scale horizontally on multiple machines, you need to do the following:
- Each instance’s
data
storage needs to be configured to be stored on a shared disk storage, preferably together with repositories. Thisdata
dir contains template caches, sessions, whoosh index and is used for task locking (so it is safe across multiple instances). Set thecache_dir
,index_dir
,beaker.cache.data_dir
,beaker.cache.lock_dir
variables in each .ini file to a shared location across Kallithea instances - If using several Celery instances, the message broker should be common to all of them (e.g., one shared RabbitMQ server)
- Load balance using round robin or IP hash, recommended is writing LB rules that will separate regular user traffic from automated processes like CI servers or build bots.
Serve static files directly from the web server¶
With the default static_files
ini setting, the Kallithea WSGI application
will take care of serving the static files from kallithea/public/
at the
root of the application URL.
The actual serving of the static files is very fast and unlikely to be a problem in a Kallithea setup - the responses generated by Kallithea from database and repository content will take significantly more time and resources.
To serve static files from the web server, use something like this Apache config snippet:
Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/
Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/
Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/
Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/
Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/
Then disable serving of static files in the .ini
app:main
section:
static_files = false
If using Kallithea installed as a package, you should be able to find the files
under site-packages/kallithea
, either in your Python installation or in your
virtualenv. When upgrading, make sure to update the web server configuration
too if necessary.
It might also be possible to improve performance by configuring the web server to compress responses (served from static files or generated by Kallithea) when serving them. That might also imply buffering of responses - that is more likely to be a problem; large responses (clones or pulls) will have to be fully processed and spooled to disk or memory before the client will see any response. See the documentation for your web server.