OpenVMS Notes: RMS File Tuning

  1. The information presented here is intended for educational use by qualified OpenVMS technologists.
  2. The information presented here is provided free of charge, as-is, with no warranty of any kind.

RMS for OpenVMS

RMS Links

RMS File Tuning

Example #1 (Vanilla Convert)

Legend for the following examples:

<ur> = user response <sr> = system response <enter> = hit the enter key
 
<sr> $
<ur> dir troubledb2200.dat/nohead/notrail/size=all/width=file=45	! see all file sizes before we start
<sr> CSMIS$ROOT4:[DAT]TroubleDB2200.dat;120 250908/250911		! 250908 blocks used; 250911 blocks allocated
     $
<ur> convert/create troubledb2200.dat *.*				! convert troubledb2200.dat into a newer version
<sr> $
<ur> dir troubledb2200.dat/nohead/notrail/size=all/width=file=45	! see the file sizes after we're done
<sr> CSMIS$ROOT4:[DAT]TroubleDB2200.dat;121 250920/250920		! slightly more blocks used / allocated
     CSMIS$ROOT4:[DAT]TroubleDB2200.dat;120 250908/250911		!
     $

Example #2 (FDL-based Convert with a vanilla FDL)

<sr> $
<ur> ana/rms/fdl TroubleDB2200.dat					! create an FDL for this file
<sr> $									!
<ur> convert/create.fdl=troubledb2200.fdl troubledb2200.dat *.*		! convert troubledb2200.dat into a newer version
<sr> $
<ur> dir troubledb2200.dat/nohead/notrail/size=all/width=file=45	! see the file sizes after we're done
<sr> CSMIS$ROOT4:[DAT]TroubleDB2200.dat;122 250920/250920		! no change from version 121
     CSMIS$ROOT4:[DAT]TroubleDB2200.dat;121 250920/250920		!
     CSMIS$ROOT4:[DAT]TroubleDB2200.dat;120 250908/250911		!
     $
<ur> type troubledb2200.fdl						! hacker's delight: peek into the FDL but do not change anything

Example #3 (FDL-based Convert with feedback (a.k.a. tuning)

<ur> $ana/rms/fdl TroubleDB2200.dat					! create an FDL for this file
<sr> $									!
<ur> sh sym edit							! Is EDIT redefined by a DCL symbol?
<sr>   ED*IT == "EDIT/EDT"						! Yes. Notice "two equals" so we need to delete a global symbol
     $
<ur> del/sym/glo edit							! delete global DCL symbol for current process
<sr> $									! 
<ur> edit/fdl TroubleDB2200						! invoke the FDL editor (notice that I did not enter an extension)
									! continued in next yellow block...

========== Start Of Recorded Procedure ==========

  • my responses are in red (with some attention grabbing using a green highlighter)
  • people using this tool for the first time should stick with the default responses unless you've got ANY segmented keys (i.e. keys composed of more than one field)
  • the indexed data file demoed has four keys (this procedure would be much shorter for fewer keys)
  • in this case, only the first 2 keys are segmented but I will always answer yes to the "segmented?" prompt for all keys
    • Why? A bug in the OPTIMIZE script defaults to change segmented keys into single-field keys rather than leaving things as-is. I am not sure if this bug existed on VMS for VAX but it always existed on OpenVMS for Alpha.
  • if you get skittish then you can always hit <ctrl-y> to go to the main menu then type "QUIT <enter>"
  • Warnings (again):
    • Always do your hacking on an offline development platform
    • Always make sure your production disks are fully backed up
    • Never purge FDL files until you've used them to do a "$convert/create/fdl" to produce a data file that is functional with your application software (you can almost always use an old FDL to restore your old indexes)
    • Never purge converted data files until you've used them with your application software (you can always delete or rename a new data file version thus exposing the old one to your application software)
  • Learning how to use this tool is really worth the effort. With it, you can make your system fly... 
Parsing Definition File
Definition Parse Complete

OpenVMS FDL Editor (menu)

Add     to insert one line into the FDL definition
Delete  to remove one line from the FDL definition
Exit    to leave the FDL Editor after creating the FDL file
Help    to obtain information about the FDL Editor
Invoke  to initiate a script of related questions
Modify  to change an existing line in the FDL definition
Quit    to abort the FDL Editor with no FDL file creation
Set     to specify FDL Editor characteristics
View    to display the current FDL Definition

Main Editor Function            (Keyword)[Help] : i <enter>

Script Title Selection (menu)

Add_Key         modeling and addition of a new index's parameters
Delete_Key      removal of the highest index's parameters
Indexed         modeling of parameters for an entire Indexed file
Optimize        tuning of all indices' parameters using file statistics
Relative        selection of parameters for a Relative file
Sequential      selection of parameters for a Sequential file
Touchup         remodeling of parameters for a particular index

Editing Script Title            (Keyword)[-]    : o <enter>

An Input Analysis File is necessary for Optimizing Keys.

Analysis File file-spec (1-512 chars)[null]
: troubledb2200 <enter>		(notice that I did not enter a file extension)

Parsing Analysis File
Analysis Parse Complete

The Definition of Key  0 will be replaced.

Press RETURN to continue (^Z for Main Menu) <enter>

Key  0 Graph Type Selection

Line    Bucket Size vs Index Depth      as a 2 dimensional plot
Fill    Bucket Size vs     Load Fill Percent     vs Index Depth
Key     Bucket Size vs         Key Length        vs Index Depth
Record  Bucket Size vs        Record Size        vs Index Depth
Init    Bucket Size vs Initial Load Record Count vs Index Depth
Add     Bucket Size vs  Additional Record Count  vs Index Depth

Graph type to display           (Keyword)[Line] : <enter>

Number of Records that will be Reloaded
into the File                 (0-2Giga)[203843] : <enter>

(Fast_Convert NoFast_Convert RMS_Puts)
File Reloading Method           (Keyword)[Fast] : <enter>

Number of Additional Records to be Added After
the Reloading the File        (0-2147279802)[0] : <enter> (input the record number you will add before next convert)

Key  0 Segmentation desired     (Yes/No)[No]    : yes <enter> Note: you should always answer yes

Key  0 Length           SEG1    (1-255)[-]      : 0 <enter> (asking about SEG1 because SEG0 was skipped; Note: this
						  means key#0 was not segmented and lying did not break anything)

      *|
      9|
      8|
Index 7|
      6|
Depth 5|
      4|
      3|      3 3 3
      2|            2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
      1|
       +- + - - - + - - - - + - - - - + - - - - + - - - - + - - - - + - +
          1       5        10        15        20        25        30  32
       Bucket Size (number of blocks)

PV-Prolog Version       3 KT-Key  0 Type     String  EM-Emphasis  Flatter ( 9)
DK-Dup Key  0 Values  Yes KL-Key  0 Length        9 KP-Key  0 Position      0
RC-Data Record Comp   60% KC-Data Key Comp      27% IC-Index Record Comp  25%
BF-Bucket Fill       100% RF-Record Format Variable RS-Mean Record Size  1200
LM-Load Method  Fast_Conv IL-Initial Load    203843 AR-Added Records        0
Which File Parameter    (Mnemonic)[refresh]     : fd <enter>

Text for FDL Title Section      (1-126 chars)[null]
:<enter>

Emphasis Used In Defining Default:      (     Flatter_files     )
Suggested Bucket Sizes:                 (       3      9     18 )
Number of Levels in Index:              (       3      2      2 )
Number of Buckets in Index:             (     593     66     18 )
Pages Required to Cache Index:          (    1779    594    324 )
Processing Used to Search Index:        (     288    588   1178 ) <<< ATTENTION

Key  0 Bucket Size              (3-63)[9]       :<enter>

+--------------------------------------------------------------------------------------------------
| Quote: The primary index structure consists of the file's data records and a key pathway based
| on the primary key (key 0). The base of a primary index structure is the data records themselves,
| arranged sequentially according to the primary key value. The data records are called level 0 of
| the index structure.
|
| Reference: http://h41379.www4.hpe.com/doc/73final/4506/4506pro_008.html#apps_rms_index_structure
| Interpretation: choosing a lower value here requires more memory but will reduce disk i/o 
+--------------------------------------------------------------------------------------------------

The Depth of Key  0 is Estimated to be No Greater
than 2 Index levels, which is 3 Total levels.

Press RETURN to continue (^Z for Main Menu) <enter>

The Definition of Key  1 will be replaced.

Press RETURN to continue (^Z for Main Menu) <enter>

Key  1 Graph Type Selection

Line    Bucket Size vs Index Depth      as a 2 dimensional plot
Fill    Bucket Size vs     Load Fill Percent     vs Index Depth
Key     Bucket Size vs         Key Length        vs Index Depth

Graph type to display           (Keyword)[Line] : <enter>

Key  1 Segmentation desired     (Yes/No)[No]    : yes <enter> Note: you should always answer yes

Key  1 Length           SEG2    (1-255)[-]      : 0  <enter> (asks about SEG2 because SEG0 and SEG1 were skipped;
								Note: apparently this key was segmented)

      *|
      9|
      8|
Index 7|
      6|
Depth 5|
      4|
      3|
      2|      2 2 2 2 2 2 2 2 2 2 2
      1|                            1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
       +- + - - - + - - - - + - - - - + - - - - + - - - - + - - - - + - +
          1       5        10        15        20        25        30  32
       Bucket Size (number of blocks)

PV-Prolog Version       3 KT-Key  1 Type     String  EM-Emphasis  Flatter (18)
DK-Dup Key  1 Values  Yes KL-Key  1 Length        8 KP-Key  1 Position     95
RC-Data Record Comp    0% KC-Data Key Comp      60% IC-Index Record Comp  20%
BF-Bucket Fill       100% RF-Record Format Variable RS-Mean Record Size  1200
LM-Load Method  Fast_Conv IL-Initial Load    203812 AR-Added Records        0
Which File Parameter    (Mnemonic)[refresh]     : fd <enter>

Emphasis Used In Defining Default:      (     Flatter_files     )
Suggested Bucket Sizes:                 (       3     18     27 )
Number of Levels in Index:              (       2      1      1 )
Number of Buckets in Index:             (      21      1      1 )
Pages Required to Cache Index:          (      63     18     27 )
Processing Used to Search Index:        (     192    589    885 )

Key  1 Bucket Size              (1-63)[18]      : <enter>

The Depth of Key  1 is Estimated to be No Greater
than 1 Index levels, which is 2 Total levels.

Press RETURN to continue (^Z for Main Menu) <enter>

The Definition of Key  2 will be replaced.

Press RETURN to continue (^Z for Main Menu) <enter>

Key  2 Graph Type Selection

Line    Bucket Size vs Index Depth      as a 2 dimensional plot
Fill    Bucket Size vs     Load Fill Percent     vs Index Depth
Key     Bucket Size vs         Key Length        vs Index Depth

Graph type to display           (Keyword)[Line] : <enter>

Key  2 Segmentation desired     (Yes/No)[No]    : yes <enter> Note: you should always answer yes

Key  2 Length           SEG2    (1-255)[-]      : 0 <enter> (asks about SEG2 because SEG0 and SEG1 were skipped;
								Note: apparently this key was segmented)

      *|
      9|
      8|
Index 7|
      6|
Depth 5|
      4|
      3|
      2|      2 2 2 2 2 2 2 2 2 2 2
      1|                            1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
       +- + - - - + - - - - + - - - - + - - - - + - - - - + - - - - + - +
          1       5        10        15        20        25        30  32
       Bucket Size (number of blocks)

PV-Prolog Version       3 KT-Key  2 Type     String  EM-Emphasis  Flatter (18)
DK-Dup Key  2 Values  Yes KL-Key  2 Length        8 KP-Key  2 Position     99
RC-Data Record Comp    0% KC-Data Key Comp      60% IC-Index Record Comp  20%
BF-Bucket Fill       100% RF-Record Format Variable RS-Mean Record Size  1200
LM-Load Method  Fast_Conv IL-Initial Load    203814 AR-Added Records        0
Which File Parameter    (Mnemonic)[refresh]     : fd <enter>

Emphasis Used In Defining Default:      (     Flatter_files     )
Suggested Bucket Sizes:                 (       3     18     27 )
Number of Levels in Index:              (       2      1      1 )
Number of Buckets in Index:             (      21      1      1 )
Pages Required to Cache Index:          (      63     18     27 )
Processing Used to Search Index:        (     192    589    885 )

Key  2 Bucket Size              (1-63)[18]      : <enter>

The Depth of Key  2 is Estimated to be No Greater
than 1 Index levels, which is 2 Total levels.

Press RETURN to continue (^Z for Main Menu) <enter>

The Definition of Key  3 will be replaced.

Press RETURN to continue (^Z for Main Menu)

Key  3 Graph Type Selection

Line    Bucket Size vs Index Depth      as a 2 dimensional plot
Fill    Bucket Size vs     Load Fill Percent     vs Index Depth
Key     Bucket Size vs         Key Length        vs Index Depth

Graph type to display           (Keyword)[Line] : <enter>

Key  3 Segmentation desired     (Yes/No)[No]    : yes <enter> Note: you should always answer yes

Key  3 Length           SEG1    (1-255)[-]      : 0 <enter> (asks about SEG1 because SEG0 was skipped; Note: this
						  means key#3 was not segmented and lying did not break anything)
      *|
      9|
      8|
Index 7|
      6|
Depth 5|
      4|
      3|
      2|     2 2 2 2 2 2 2 2 2 2
      1|                         1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
      +- + - - - + - - - - + - - - - + - - - - + - - - - + - - - - + - +
         1       5        10        15        20        25        30  32
      Bucket Size (number of blocks)

PV-Prolog Version       3 KT-Key  3 Type     String  EM-Emphasis  Flatter (18)
DK-Dup Key  3 Values  Yes KL-Key  3 Length       15 KP-Key  3 Position    304
RC-Data Record Comp    0% KC-Data Key Comp      58% IC-Index Record Comp  31%
BF-Bucket Fill       100% RF-Record Format Variable RS-Mean Record Size  1200
LM-Load Method  Fast_Conv IL-Initial Load    110857 AR-Added Records        0
Which File Parameter    (Mnemonic)[refresh]     : fd <enter>

Emphasis Used In Defining Default:      (     Flatter_files     )
Suggested Bucket Sizes:                 (       3     18     27 )
Number of Levels in Index:              (       2      1      1 )
Number of Buckets in Index:             (      19      1      1 )
Pages Required to Cache Index:          (      57     18     27 )
Processing Used to Search Index:        (     148    450    676 )

Key  3 Bucket Size              (1-63)[18]      : <enter>

The Depth of Key  3 is Estimated to be No Greater
than 1 Index levels, which is 2 Total levels.

Press RETURN to continue (^Z for Main Menu) <enter>

The Definition of Key  4 will be replaced.

Press RETURN to continue (^Z for Main Menu) <enter>

Key  4 Graph Type Selection

Line    Bucket Size vs Index Depth      as a 2 dimensional plot
Fill    Bucket Size vs     Load Fill Percent     vs Index Depth
Key     Bucket Size vs         Key Length        vs Index Depth

Graph type to display           (Keyword)[Line] : <enter>

Key  4 Segmentation desired     (Yes/No)[No]    : yes <enter> Note: you should always answer yes

Key  4 Length           SEG1    (1-255)[-]      : 0 <enter> (asks about SEG1 because SEG0 was skipped)

      *|
      9|
      8|
Index 7|
      6|
Depth 5|
      4|
      3|
      2|
      1|      1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
       +- + - - - + - - - - + - - - - + - - - - + - - - - + - - - - + - +
          1       5        10        15        20        25        30  32
       Bucket Size (number of blocks)

PV-Prolog Version       3 KT-Key  4 Type     String  EM-Emphasis  Flatter ( 3)
DK-Dup Key  4 Values  Yes KL-Key  4 Length        8 KP-Key  4 Position    265
RC-Data Record Comp    0% KC-Data Key Comp      43% IC-Index Record Comp  37%
BF-Bucket Fill       100% RF-Record Format Variable RS-Mean Record Size  1200
LM-Load Method  Fast_Conv IL-Initial Load      1307 AR-Added Records        0
Which File Parameter    (Mnemonic)[refresh]     : fd <enter>

Emphasis Used In Defining Default:      (     Flatter_files     )
Suggested Bucket Sizes:                 (       3      3     18 )
Number of Levels in Index:              (       1      1      1 )
Number of Buckets in Index:             (       1      1      1 )
Pages Required to Cache Index:          (       3      3     18 )
Processing Used to Search Index:        (     115    115    696 )

Key  4 Bucket Size              (1-63)[3]       : <enter>

The Depth of Key  4 is Estimated to be No Greater
than 1 Index levels, which is 2 Total levels.

Press RETURN to continue (^Z for Main Menu) <enter>

OpenVMS FDL Editor (menu)

Add     to insert one line into the FDL definition
Delete  to remove one line from the FDL definition
Exit    to leave the FDL Editor after creating the FDL file
Help    to obtain information about the FDL Editor
Invoke  to initiate a script of related questions
Modify  to change an existing line in the FDL definition
Quit    to abort the FDL Editor with no FDL file creation
Set     to specify FDL Editor characteristics
View    to display the current FDL Definition

Main Editor Function            (Keyword)[Help] : exit <enter>

========== End of Recorded Procedure ==========

Example #4: (using your tuned FDL to perform a convert)

<sr> CSMIS$ROOT4:[DAT]TroubleDB2200.FDL;2  123 lines
     $
<ur> convert/create/fdl=TroubleDB2200 TroubleDB2200.dat *.*
<sr> $
<ur> dir troubledb2200.dat/nohead/notrail/size=all/width=file=40
<sr> CSMIS$ROOT4:[DAT]TroubleDB2200.dat;123 273690/273690		! more blocks used / allocated but will be much faster
     CSMIS$ROOT4:[DAT]TroubleDB2200.dat;122 250920/250920 
     CSMIS$ROOT4:[DAT]TroubleDB2200.dat;121 250920/250920 
     CSMIS$ROOT4:[DAT]TroubleDB2200.dat;120 250908/250911
     $

Example5: (Additional Hacking)

RMS File Tuning Links


Back to Home
Neil Rieck
Waterloo, Ontario, Canada.