This site hosts historical documentation. Visit www.terracotta.org for recent product information.

Fast Restartability

Introduction

This page covers cache persistence, fast restartability, and using the local disk as a cache storage tier. While Ehcache offers various disk usage choices (summarized here), as of version 2.6, the recommended option for persistence is the Fast Restart store.

Fast Restart provides enterprise-ready crash resilience with an option to store a fully consistent copy of the cache on the local disk at all times. The persistent storage of the cache on disk means that after any kind of shutdown — planned or unplanned — the next time that the application starts up, all of the previously cached data is still available and very quickly accessible. The advantages of the Fast Restart store include:

  1. A persistent store of the cache on disk survives crashes, providing the fastest restart. Because cached data does not need to be reloaded from the data source after a crash, but is instead loaded from the local disk, applications can resume at full speed after restart. Recovery of even terabytes of data after a failure will be very fast, minimizing downtime.

  2. A persistent store on disk always contains a real-time copy of the cache, providing true fault tolerance. Even with BigMemory, where terabytes of data can be held in memory, the synchronous backup of data to disk provides the equivalent of a hot mirror right at the application and server nodes.

  3. A consistent copy of the cache on local disk provides many possibilities for business requirements, such as working with different datasets according to time-based needs or moving datasets around to different locations. It can range from a simple key-value persistence mechanism with fast read performance, to an operational store with in-memory speeds during operation for both reads and writes.

Cache Persistence Implementation

Ehcache has a RestartStore which provides fast restartability and options for cache persistence. The RestartStore implements an on-disk mirror of the in-memory cache. After any restart, data that was last in the cache will automatically load from disk into the RestartStore, and from there the data will be available to the cache.

Cache persistence on disk is configured by adding the <persistence> sub-element to a cache configuration. The <persistence> sub-element includes two attributes: strategy and synchronousWrites.

<cache>
   <persistence strategy="localRestartable|localTempSwap|none|distributed" 
      synchronousWrites="false|true"/>
</cache>

Strategy Options

The options for the strategy attribute are:

  • "localRestartable" — Enables the RestartStore and copies all cache entries (on-heap and/or off-heap) to disk. This option provides fast restartability with fault tolerant cache persistence on disk. This option is available for standalone Enterprise Ehcache only.

  • "distributed" — Defers to the <terracotta> configuration for persistence settings. This option is for clustered Ehcache. For more information, refer to Terracotta Clustering Configuration Elements.

  • "localTempSwap" — Enables temporary local disk usage. This option provides an extra tier for storage during cache operation, but this disk storage is not persisted. After a restart, the disk tier is cleared of any cache data.

  • "none" — Does not offload cache entries to disk. With this option, all of the cache is kept in memory. This is the default mode.

Synchronous Writes Options

If the strategy attribute is set to "localRestartable", then the synchronousWrites attribute may be configured. The options for synchronousWrites are:

  • synchronousWrites="false" — This option provides an eventually consistent copy of the cache on disk at all times. Writes to disk happen when efficient, and cache operations proceed without waiting for acknowledgement of writing to disk. After a restart, the cache is recovered as it was when last synced. This option is faster than synchronousWrites="true", but after a crash, the last 2-3 seconds of written data may be lost.

    If not specified, the default for synchronousWrites is "false".

  • synchronousWrites="true" — This option provides a fully consistent copy of the cache on disk at all times. As elements are put into the cache, they are synchronously written to disk. The write to disk happens before a put returns to the caller. After a restart, the cache is recovered exactly as it was before shutdown. This option is slower than synchronousWrites="false", but after a crash, it provides full cache consistency.

    For transaction caching with synchronousWrites, soft locks are used to protect access. If there is a crash in the middle of a transaction, then upon recovery the soft locks are cleared on next access.

Note: The synchronousWrites attribute is also available in the <terracotta> sub-element. If configured in both places, it must have the same value.

DiskStore Path

The path to the directory where any required disk files will be created is configured with the <diskStore> sub-element of the Ehcache configuration.

  • For "localRestartable", a unique and explicitly specified path is required.

  • For "localTempSwap", if the DiskStore path is not specified, a default path is used for the disk storage tier, and the default path will be auto-resolved in the case of a conflict with another CacheManager.

Configuration Examples

This section presents possible disk usage configurations for standalone Ehcache 2.6.

Options for Crash Resilience

Note: Crash resilience options are for Enterprise Ehcache only.

The following configuration provides fast restartability with fully consistent cache persistence:

<ehcache>
  <diskStore path="/path/to/store/data"/>
  <cache>
    <persistence strategy="localRestartable" synchronousWrites="true"/>
  </cache>
</ehcache>  

The following configuration provides fast restartability with eventually consistent cache persistence:

<ehcache>
  <diskStore path="/path/to/store/data"/>
  <cache>
    <persistence strategy="localRestartable" synchronousWrites="false"/>
  </cache>
</ehcache>  

Clustered Caches

If a cache is clustered, then persistence strategy should be set to "distributed". Persistence settings for clustered Ehcache are configured in the <terracotta> sub-element.

<cache>
  <persistence strategy="distributed"/>
  <terracotta clustered="true" consistency="eventual" synchronousWrites="false"/>
</cache>    

Temporary Disk Storage

The "localTempSwap" persistence strategy allows the cache to use the local disk during cache operation. The disk storage is temporary and is cleared after a restart.

<ehcache>
  <diskStore path="/auto/default/path"/>
  <cache>
    <persistence strategy="localTempSwap"/>
  </cache>
</ehcache>  

Note: With the "localTempSwap" strategy, you can use maxEntriesLocalDisk or maxBytesLocalDisk at either the Cache or CacheManager level to control the size of the disk tier.

In-memory Only Cache

When the persistence strategy is "none", all cache stays in memory (with no overflow to disk nor persistence on disk).

<cache>
  <persistence strategy="none"/>
</cache>    

Programmatic Configuration Example

The following is an example of how to programmatically configure cache persistence on disk:

Configuration cacheManagerConfig = new Configuration()
    .diskStore(new DiskStoreConfiguration()
    .path("/path/to/store/data"));
CacheConfiguration cacheConfig = new CacheConfiguration()
    .name("my-cache")
    .maxBytesLocalHeap(16, MemoryUnit.MEGABYTES)
    .maxBytesLocalOffHeap(256, MemoryUnit.MEGABYTES)
    .persistence(new PersistenceConfiguration().strategy(Strategy.LOCALRESTARTABLE));

cacheManagerConfig.addCache(cacheConfig);

CacheManager cacheManager = new CacheManager(cacheManagerConfig);
Ehcache myCache = cacheManager.getEhcache("my-cache");

Fast Restart Performance

When configured for fast restartability ("localRestartable" persistence strategy), the cache becomes active on restart after all the data is loaded. The amount of time until a cache is restarted is proportionate to the amount of data in the cache and the speed of the underlying infrastructure. Generally, recovery can occur as fast as the disk speed. With an SSD, for example, if you have a read throughput of 1 GB per second, you will see a similar loading speed during cache recovery.

Fast Restart Limitations

The following recommendations should be observed when configuring caches for fast restartability:

  1. The size of on-heap or off-heap caches should not be changed during a shutdown. If the amount of memory allocated to a cache is reduced, elements will be evicted upon restart.

  2. Restartable caches should not be removed from the CacheManager during a shutdown.

  3. If a restartable cache is disposed, the reference to the cache is deleted, but the cache contents remain in memory and on disk. After a restart, the cache contents are once again recovered into memory and on disk. The way to safely dispose of an unused restartable cache is to call clear on the cache and then dispose, so it doesn't take any space in disk or memory.

Compatibility with Previous Versions

Comparison of Disk Usage Options

The following table summarizes the configuration options for disk usage in Ehcache 2.6 and previous versions.

Disk Usage Ehcache 2.6 Ehcache 2.5 and Earlier
Fast Restartability with Strong Consistency persistence strategy="localRestartable" synchronousWrites="true" (Enterprise only) Not available
Fast Restartability with Eventual Consistency persistence strategy="localRestartable" synchronousWrites="true" (Enterprise only) Not available
Persistence for Clustered Caches persistence strategy="distributed" Remove or edit out any disk persistence configuration elements
Non-Fault-Tolerant Persistence Use one of the fault-tolerant options above* overflowToDisk="true" diskPersistent="true"**
Temporary Storage Tier persistence strategy="localTempSwap" overflowToDisk="true" diskPersistent="false"
In-memory Only (no disk usage) persistence strategy="none" overflowToDisk="false"

*It is recommended to use one of the fault-tolerant options, however non-fault-tolerant persistence is still available. If <persistence> has not been specified, you can still use overflowToDisk="true" diskPersistent="true".

**In Ehcache 2.5 and earlier, cache persistence on disk for standalone Ehcache is configured with the overflowToDisk and diskPersistent attributes. If both are set to "true", cached data is saved to disk asynchronously and can be recovered after a clean shutdown or planned flush. To prevent corrupt or inconsistent data from being returned, checking measures are performed upon a restart, and if any discrepancy is found, the cache that was stored on disk is emptied and must be reloaded from the data source.

Upgrading to Ehcache 2.6

After upgrading to Ehcache 2.6, it is strongly recommended to add the <persistence> sub-element to your cache configuration, and to delete, disable, or edit out disk persistence configuration elements from previous versions. The previous elements include:

  • overflowToDisk
  • diskPersistence
  • DiskStoreBootstrapCacheLoaderFactory

Note: If any of the elements above are specified in the same configuration with either the <persistence> sub-element or the <terracotta> sub-element, it will cause an Invalid Configuration Exception.

After upgrading, however, it is not mandatory to add the <persistence> sub-element. In Ehcache 2.6, disk persistence configuration elements from previous Ehcache versions will continue to be available with the same functionality, as long as the <persistence> sub-element has not been specified.

Upgrading for Open Source Ehcache

All persistence strategy options are available in Open Source Ehcache 2.6, except "localRestartable". In addition, all disk persistence configuration elements from previous Ehcache versions continue to be available with the same functionality (as long as the <persistence> sub-element has not been added).

For cache persistence on disk, you should continue to use the overflowToDisk and diskPersistent attributes. For more information, refer to Persistence in the Ehcache 2.5 documentation.