Scalability of TWiki

Sometimes we get the question on how well TWiki can scale. This blog post compiles scalability related information so that you can plan your TWiki deployment effectively.

Scaling Across Teams and Departments

TWiki was designed as an enterprise wiki from its inception. You find features specifically designed to support large deployments. Other wiki engines have a different focus and may lack some of these features. Wikis typically flourish in grassroots. Once at the radar screen of the CTO/CIO, grassroots wikis often get consolidated into a central TWiki. That is when scalability comes into play. Key scaling features of TWiki:

  • Multiple webs (workspaces):
    • You can create as many webs as you need. Some large TWiki deployments have over 1000 webs. Think of a web like a wiki within TWiki. Each team can get their own wiki. People need to register only once, then they can create content in their own space. If needed you can link across webs, such as to reference a registered user or an entry in the Glossary web.
  • Fine grained access control:
    • You can create TWikiGroups and restrict access to content for view and edit based on those groups. Although it is possible to restrict access on a topic (page) level, it is typically done on a web level for ease of administration.
  • Authentication:
    • In a large deployment it is advisable to authenticate users against your directory server, such as Active Directory or LDAP. That reduces the workload on registration/login questions.
  • File attachments:
    • TWiki has a per-topic namespace for file attachments. That means, if one team uploads a file called inventory.xls to their team page, and another team uploads a file of the same name to a different page, they will not collide. Try that with Mediawiki or other wikis.
    • You can limit the maximum size of attachments that can be uploaded. This can be done for the whole site and also on a web level.
  • Organize content:
  • Web 2.0 platform:
  • Create your own applications:
    • TWiki goes beyond Web 2.0: The TWiki platform is about user generated application logic. Your users can create situational applications that solve specific business needs, such as a bug tracker, a employee news portal, TWiki's Support web and more. You do not need to be a programmer; all application logic is done in TML (TWiki Markup Language) using TWiki forms, reports and optionally some HTML and JavaScript.
    • The IT department is in charge of the wiki dial tone and wants to have some control over the wiki deployment. With TWiki you allow users to experiment in a controlled environment. That is, IT can get the dreaded "shadow IT" under control.
  • Integrate:
    • TWiki has a plugin API and ready made plugins to connect to external databases. That way you can run a query in MySQL and other RDBMS and display the result in TWiki pages. Useful to show CRM data to sales teams and bug trends to engineering teams.

Server Selection, Caching, Load Balancing

Plan for adequate server hardware when you deploy TWiki. The following are ballpark figures for an sample TWiki deployment serving 1000 employees and 50,000 pages:

  • Enterprise class Linux
  • Dual core CPU 2.6 GHz
  • 2 GB RAM
  • RAID 1 or RAID 5 for redundancy
  • Dual power supply for redundancy
  • Plan disk space:
    • Page content: 15MB per 1000 page (yes, MB, not GB)
    • File attachments: 1GB per 1000 pages

If you have a high read to write ratio (such as a TWiki on the public internet) consider a caching solution and/or a load balanced setup.

  • Load balancing:
    • For high volume traffic sites it is possible to put TWiki on a load balanced setup. Here is an example:
      • Cisco Ace load balancer.
      • 3 webservers.
      • NAS storage back-end.
      • Webservers share data on NAS for pages, file attachments and log files.
    • In the early days, TWiki.org was on a load balanced server setup while hosted at SourceForge.net. Now it is on a single server. The TWiki community plans to move TWiki.org again to a load balanced server setup which will improve the performance considerably.

Scalability of Search

TWiki uses the Unix grep command to search content in real time. This enables flexible and powerful searches in real time, which is important for TWiki applications. Search is covered in related pages in the TWiki web: SearchHelp, VarSEARCH, QuerySearch, FormattedSearch, and on TWiki.org at TWiki:TWiki/SearchSupplement.

The real time search has a performance impact. Searching all webs in a TWiki sites with more than 50,000 pages can be slow. If you have a large TWiki deployment of more than 50,000 pages it is advisable to index TWiki content with a search engine. This can be done with a commercial search engine such as the Google Search Appliance or an open source search engine. TWiki currently has three open source search engine integrations: TWiki:Plugins.SearchEnginePluceneAddOn, TWiki:Plugins.SearchEngineSwishEAddOn and TWiki:Plugins.SearchEngineKinoSearchAddOn. See more extensions on search.

To scale the queries of TWikiForms based TWiki applications look into TWiki:Plugins.DBCacheContrib and TWiki:Plugins.DBCachePlugin.

Flat File Back-end

TWiki currently requires only a flat file storage back-end. This has several advantages:

  • Simple installation
  • Simple backup and restore
  • Simple migration of content between TWiki installations (think of grassroots wiki consolidations, spin-offs and acquisitions of companies)
  • Well understood caching and replication technologies available
  • Resilient to data corruption

Some people express concerns that TWiki's flat file back-end does not scale. TWiki.org performs well on a single webserver with currently 40,000 users and 90,000 pages. For larger deployments you need a load balanced setup. We know of a number of large TWiki deployments, such as 300,000 pages at a Silicon Valley high-tech company, 1000+ webs at a major telco company, and 10,000+ users at a major financial institution in USA.

Some scaling factors:

  • TWiki scales well on number of webs, e.g. it does not matter much if you have 3 webs or 3000 webs.
  • TWiki has a limit on the number of pages in a web. You will see a performance impact if you have more than 20,000 pages in a single web. This depends on the file system/configuration used, on the bandwidth of your server I/O and on the memory installed.
  • TWiki scales well on the number of registered users. We have not done tests on the upper limit. It is also feasible to not register users in TWiki, e.g. to rely solely on LDAP login.

As stated above, performance can be addressed to some degree with caching and/or load balancing. The TWiki community is working on a pluggable storage back-end for DB support, see TWiki:Codev.TWikiRoadMap.

pencil 008-03-25 | Peter Thoeny | Category Best Practices

TWIKI.NET Blog

Recent Posts