This site hosts historical documentation. Visit www.terracotta.org for recent product information.

Troubleshooting the Terracotta Management Server

The Terracotta Management Server (TMS) is easy to set up and operate. In cases where an error occurs, consult this page for more information.

Each section below addresses an error or set of errors that you might encounter.

Setup Errors

500 Problem Accessing the Keychain File

Problem: After configuring and attempting to use the LDAP or AD URL, you see a message similar to the following:

Problem accessing /tmc/setupAuth. Reason:

impossible to initialize the keychain
...
~/.tc/mgmt/keychain doesn't point to a valid file

Cause: The keychain file does not exist in the expected location.

Solution: Create the file keychain in $(user.home)/.tc/mgmt while adding the first entry:

bin/keychain.sh -c -O -S ~/.tc/mgmt/keychain ldap://admin@localhost:1389

Cannot Retrieve Entry for LDAP or Active Directory User

Problem: After configuring and attempting to use the LDAP or AD URL, you see a message similar to "Impossible to retrieve systemUsername password from the keychain : ldap://admin@localhost:1389".

Cause: The keychain does not contain an entry matching the system user specified.

Solution: Create a correct entry for the specified user. For the example above, the password in the keychain file should be keyed with "ldap://admin@localhost:1389".

Number of Clients Impacts Performance

Problem: You might observe a performance degradation when the number of clients is 200 or more.

Cause: It is possible to have an insufficient number of Java Management Extensions (JMX) threads because the default value of 64 JMX threads is designed for the common scenario in which there are no more than 256 client connections.

Solution: Stop the cluster, adjust the value of l2.remotejmx.maxthreads in tc-config.xml for your particular environment to ensure there is a JMX thread for every four L1 nodes, then restart the cluster. The following example sets the number of threads high enough to support 1024 clients.

<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-8.xsd">
  <tc-properties>
    <property name="l2.remotejmx.maxthreads" value="256"/>
  </tc-properties>
  <servers>
    ...
  </servers>
</tc:tc-config>

Connections Errors

Connection Refused

Problem: An attempt to add a connection to a managed agent is rejected.

Cause: The agent is unreachable or not running.

Solution: Check the following:

  • The URI in the connection setup is correct.
  • The network connection to the node running the agent is working.
  • The agent process is running on the expected node.

404 Connection Not Found

Problem: An attempt to add a connection returns a 404 status code.

Cause: The URI used in the connection setup is incorrect or malformed.

Solution: Check the following:

  • The URI in the connection setup is correct.
  • The port used in the URI is correct (by default: 9888 for BigMemory Go, 9530 for BigMemory Max).

"A message body reader for Java class ... was not found"

Problem: An attempt to add a connection causes the exception "A message body reader for Java class java.util.Collection, and Java type java.util.Collection, and MIME media type unknown/unknown was not found"

Cause: The URI used in the connection setup is incorrect or malformed.

Solution: Check the following:

  • The URI in the connection setup is correct.
  • The port used in the URI is correct (by default: 9888 for BigMemory Go, 9530 for BigMemory Max).

Connection Timed Out

Problem: An attempt to add a connection fails because the TMS has failed to reach the agent within the timeout limit.

Cause: The agent is unreachable or not running.

Solution: Check the following:

  • The URI in the connection setup is correct.
  • The network connection to the node running the agent is working.
  • The agent process is running on the expected node.

Unexpected End of File From Server

Problem: An attempt to add a connection fails with an EOF error.

Cause: An unsecure connection is being attempted but the agent is set up to use SSL.

Solution: Ensure that the URI is using "https://" not "http://".

"Unrecognized SSL Message, plaintext connection?"

Problem: An attempt to add a connection fails with the error "Unrecognized SSL Message, plaintext connection?".

Cause: A secure connection is being attempted but the agent is not set up to use SSL.

Solution: Ensure that the URI is using "http://" not "https://", or set up SSL on the agent.

Missing Keychain Entry

Problem: An attempt to add a connection fails with the error "Missing keychain entry for URL <agent-url>".

Cause: A connection is being attempted to an agent with identity assertion, but the TMS keychain cannot find that agent's entry.

Solution: Add an entry for the agent using the add-tc-agent script.

401 Unauthorized

Problem 1: An attempt to add a connection to an agent configured with identity assertion returns a 401 status code.

Cause: The agent's public key cannot be found in the the TMS truststore.

Solution: Import the agent's public key into the TMS truststore.

Problem 2: An attempt to add a connection to an agent configured with identity assertion over SSL returns a 401 status code. Errors containing unknown_certificate (in the agent log) and

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target

(in the TMS log) appear in the logs (see the list of SSL-related logged errors).

Cause: The TMS public key cannot be found in the the agent's truststore.

Solution: Import the TMS public key into the agent's truststore.

Logged SSL Connection Errors

Certain issues can cause exceptions to appear in the logs when an SSL-enabled connection is attempted. The following list shows parts of log messages that indicate specific exceptions:

  • keyMaterial=null
    The connection URI has not been added to the keychain.
  • unknown_certificate (in the agent log) and
    PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
    unable to find valid certification path to requested target
    (in the TMS log)
    The agent is not using (or cannot find) its keystore.
  • unknown_certificate (in the agent log) and
    the counterpart is not ssl compliant (in the tms log)
    The agent is not configured to use SSL (or not configured correctly). Confirm that SSL is set up as shown above.
  • unknown_certificate (in the TMS log)
    Identity assertion (basic TMS security, or IA) is being used over SSL, but the IA URI has not been added to the keychain file. For example:

    bin/keychain.sh ~/.tc/mgmt/keychain https://localhost:9443/tmc/api/assertIdentity
    

    In addition, ensure that the TMS container is configured to use tms-keystore and tms-truststore.

Runtime Errors

If CacheException is being thrown as a result of an attempt to perform certain operations in the TMC, see Bad Cache or CacheManager Names.

Display Errors

Bad Cache or CacheManager Names

Using the following characters in the names of caches or CacheManagers causes display and runtime errors: % | ; , / # & * " < ?

Issues caused can include statistics not appearing correctly in the Overview panel, pop-up TMC error messages in response to an attempt to view cache configuration, and runtime CacheException errors.

Sizing Errors

The TMC displays cache sizing information on certain panels. If there appear to be errors in the way sizing information is displayed for nonstop caches, the sizing operation might be timing out (it uses the nonstop timeout value). You can tune the sizing operation's timeout value by setting the bulk-loading timeout multiplier.