HP Diagnostics Database Symbol Table Corruption and Restore

Symbol Table Corruption

When the hard drive fills up on an HP Diagnostics Mediator, the most common impact to Diagnostics is a symbol table corruption in some database files that might have been open when the filesystem filled.

Depending on the level or type of corruption, and how long the Mediator has been running, you can often restore a "backup" version of the symbol table(s), which are created and maintained by HP Diagnostics. Ideally you would restore from a formal system backup, but if you don't have that, then this is the next best option.

Every data object presented, tracked, reported on within HP Diagnostics has some kind of object name/string label associated with it in the HP Diagnostics data stores. Some of those names/strings can get quite long. For example “com.sleepycat.je.log.DbChecksumException” is a symbol that HP Diagnostics manages.

In order to help limit the disk space that all of this uses, and also to help find things based on strings faster, HP Diagnostics puts each string in the ‘symbol table – Database’ and give it an index or hash number to refer to it. Then the actual sample data is stored in the Database using the hash number instead of the string. It makes querying things much faster and a single long or double value in the Database is much shorter than that string.

Without the symbol table, especially an accurate one, when you tell Diags to go find the probe named "Frank," it can't convert the string to a hash, so it can’t find any data.

The Symbol table is ‘backed up’ on disk and with the Mediator shutdown you can try to replace the current files with the backup. I’ve personally not had a great amount of luck this because the corruption has been so long running that I find the backups are also corrupted.  Yet it might work...  but it is manually intensive...­

How to Restore Symbol Table Backup Files

During each (daily?) aggregation, the server copies the local symbol table (.jdb files) and pathed symbol table (.db file) to symboltable/backup/<granularity>. In case the symbol table becomes corrupted you can install the files from the backup directory with the following procedure:

  1. Stop the server
  2. Rename the symboltable directory
  3. Create a new symboltable directory and copy all the files with the same timestamp in the filename to the symboltable directory E.g. in a backup, we could end up having files from multiple backups, like 00000000_2009_4_19_0.jdb and 00000000_2009_3_19_0.jdb. Use the latest ones that have 2009_4_19_0 in the name and remove all the others that have 2009_3_19_0 in the name
  4. Rename the PathedSymbolTable.db. E.g. if it is named PathedSymbolTable_2009_4_19_0.db, rename it to PathedSymbolTable.db
  5. To rename the *.jdb files (timestamp must not be included in the filename), use the following perl script: @files = <*>; foreach $file (@files) {if($file =~ /(.*)_2.*\.jdb/) { # print"$1.jdb\n"; rename($file,"$1.jdb"); } }
  6. Run this script inside the symboltable directory with perl -w <filename>. You can download perl from http://www.activestate.com/activeperl/downloads/
  7. After all files are in the form <number>.jdb and free of the timestamp (like 000003f0.jdb), restart the server via the server.cmd script in the bin directory. Running the server outside of the nanny provides more debugging information on the console.

If that is too much work and the data is not that valuable, then you can uninstall HPD; remove the directories and then reinstall.

Have more questions? Submit a request


Please sign in to leave a comment.