OpenVMS Notes: Apache HTTPd

Our intranet site employs AJAX and certain apps continually (every minute) pull back three little gifs. What is worse is this: these gifs are sent over the encrypted channel (which adds additional overhead) and the problem is multiplied by the fact that more than 120 employees are using the app at the same time. You can't control how a user sets up his browser "cache-wise", but the following tweak has greatly reduced the problem on my system:

1. I have removed the 2012 hack which didn't work well in 2013 after most of our clients upgraded their browsers from IE7 to IE8
   1. why use such old browsers when Microsoft is just introducing IE11? Many of our corporate HR systems were built with IE8 dependencies so our employees work in a locked-down environment unable to upgrade.
   2. I THINK a very good case could be made for having our employees use IE10 or IE11 in IE7 compatibility mode. For example, many people do not know that IE8 can only utilize ONE CORE on a multicore platform. All modern versions of IE, Chrome and Firefox can utilize multiple cores in 2014 including IE10 in IE7 compatibility mode (it is only compatible in the way it renders HTML; not the bugs or limitations).
2. I have removed the 2013 hack (ruthless caching) because I have found a better way in web ecosystems where THE MAJORITY OF PEOPLE are using browsers less than 5 years old (IE8 was released in 2009)
   • newer browsers prefer header responses with CACHE-CONTROL
   • older browsers prefer header responses with EXPIRES
   • ruthless caching employed both EXPIRES and CACHE-CONTROL which required the browser to use the correct method (VARY). There is quite a bit of controversy about how different browsers do this with many pundits thinking browsers wrongly choose EXPIRES over CACHE-CONTROL
   • the 2014 hack which follows only employs header responses with CACHE-CONTROL and a reduced FileEtag. If nothing else, just removing the EXPIRES line from the header of each web page component will slightly reduce the processing overhead of both client and server. Savings of a million lines-per-day at the sever end would be typical.
3. This following tweak has been tested to work as-is with Apache/2.0.63 on OpenVMS
   1. Primary testing:
      1. I no longer test with WireShark which was useless when testing encrypted connections over HTTPS
      2. I use the Network Profiler built into Firefox (click on the monkey-wrench icon; click CUSTOMIZE if you do not see it)
      3. I use the Network Profiler built into FireBug (a Firefox add-on). Once you get used to the time-colored responses this tool is hard to give up.
      4. I also use the optional YSlow profiler which can be installed into FireBug
         • caveat: even though I have removed EXPIRES on my site, YSLOW still reports an "A" on the EXPIRES test provided a CACHE-CONTROL statement exists with a minimum time entry of 72-hours. This is normal behavior because EXPIRES is less important this side of 2009 
      5. I also use the Network Profiler built into Chrome
      6. I also use the Network Profiler built into IE10 (good) and IE11 (better)
         • caveat: when the IE Network Profiler comes up, the third button (Always Refresh From Server) is depressed by default which will always force the browser to bypass local cache. You will think that caching is not working when it actually is
   2. Secondary testing:
      1. hopefully you are rotating your Apache logs every day. If so, make changes in Apache config then allow the system to run for a day before using $SEARCH (a VMS app) to collect stats. You want to compare the ratio of 200 responses ("here is the file you requested") to 304 responses ("use the file in your cache").
         • $search *log*.* " 200 " /noout/stats
         • $search *log*.* " 304 " /noout/stats
      2. On my system I collected three numbers:
         • number of 200 messages divided by total messages
         • number of 304 messages divided by total messages
         • number of 200 messages divided by 304 messages
         and could see a large block of messages moving from "category 200" to "category 304"
   3. If you are running Apache on OpenVMS in case-insensitive mode (which is typical) then you might think that "file.ext" is the same as "FILE.EXT". While this is true on the server, the browser caches by case-sensitive URL so it is always best if your URLs are down-cased or camel-cased. If you are setting up a new instance of Apache on OpenVMS then I recommend always using case-sensitive mode (the norm in the UNIX world where this technology was first developed)

#
# file: [.conf]httpd.conf
#

[...snip...]

#----------------------------------------------------------------------------------------
# 2014 tweak to reduce data sent to the browser
#
#	send less text to the browser (repeated for every page component)
#
# (default)         send: Server: Apache/2.0.63 (OpenVMS) mod_ssl/2.0.63 OpenSSL/0.9.8w
# ServerTokens OS   send: Server: Apache/2.0.63 (OpenVMS)
# ServerTokens Min  send: Server: Apache/2.0.63
# ServerTokens Prod send: Server: Apache
#
#----------------------------------------------------------------------------------------
#ServerTokens	Min
ServerTokens	Prod

#----------------------------------------------------------------------------------------
#	consider sending less caching information to the browser
#	note: ETag is not sent with dynamic content
#
#	FileETag All		sends something like this:	"8f0101a-ec2-2de84100"
#	FileETag MTime Size	sends something like this:	"ec2-2de84100"
#----------------------------------------------------------------------------------------
FileETag	All

#----------------------------------------------------------------------------------------
# force browsers to cache more stuff but don't go crazy
#
# notes:
# 1) verified with Apache/2.0.63 on OpenVMS
# 2) Expires directives were removed so don't load mod_expire
# 3) Header directives require mod_header
# 4) Reduce the size of each response header as much as possible
# 5) typing CTRL-R or clicking page reload will force browsers to do cache revalidation
#----------------------------------------------------------------------------------------
#ExpiresActive	On	# bowsers after IE7 prefer Cache-Control rather than Expires
#ExpiresDefault	now	# bowsers after IE7 prefer Cache-Control rather than Expires
#
#	note: when enabled, this next line blocks IE8 from downloading DOCX files
# Header set Cache-Control "no-cache, max-age=0, s-maxage=0"
#
#	this next line is for all documents (dynamic and static) so...
#	always use a small value (good) or disable the directive (better)
#
# Header set Cache-Control "public, max-age=0"
#----------------------------------------------------------------------
#	this will override (if a static file) something set above
#----------------------------------------------------------------------
# a-block
<FilesMatch "\.(ico|gif|jpg|jpeg|png|pdf)$">
Header unset Cache-Control
Header set Cache-Control "max-age=86400, public"
#	note: the next line must not be used in production
# Header set MyHeader "trigged Neil's a-block"
</FilesMatch>
#----------------------------------------------------------------------
#	this will override (if a static file) something set above
#----------------------------------------------------------------------
# b-block
<FilesMatch "\.(htm|html|js|css)$">
Header unset Cache-Control
Header set Cache-Control "max-age=3600, public"
#	note: the next line must not be used in production
# Header set MyHeader "trigged Neil's b-block"
</FilesMatch>
#----------------------------------------------------------------------
#	this will override (if a static file) something set above
#----------------------------------------------------------------------
# c-block
# comments:
# 1) we are trying to better support the client-server model for clients around the world
#    (some of the clients to this Canadian system are in India; others are across Canada)
# 2) really big files that don't change much should be cached for 24-hours
# 3) experiments with regex wildcarding
#    regex reference: The wildcard . matches any character. For example,
#    a.b matches any string that contains an "a", then any other character and then a "b"
#    while a.*b matches any string that contains an "a" and a "b" at some later point.
<FilesMatch "(jquery|angular|ddsmooth).*\.(css|js)$">
Header unset Cache-Control
Header set Cache-Control "max-age=86400, public"
#	note: the next line must not be used in production
# Header set MyHeader "trigged Neil's c-block"
</FilesMatch>
#-----------------------------------------------------------------------------------------

{ ...snip... }

Better Apache config (for corporate use on an intranet)

#
# file: [.conf]httpd.conf
#

{ ...snip... }

#
#	enable HTTP/1.1 keepalives so clients do not open/close on every page component
#
KeepAlive		On

#
#	more efficient than the default of 15 (should never exceed 60 seconds)
#
KeepAliveTimeout	30

#
#	more efficient than the default of 100
#
MaxKeepAliveRequests	999

#
#	since Apache only grows once per second, start off with MaxSpareServers
#
StartServers		9
MinSpareServers		6
MaxSpareServers		12

#
#	you will never have more server processes than MaxClients
#
# If you set MaxClients too large then you might run out of VMS process slots when
# hackers use Apache to probe our system (I have seen this happen).
# Consider using SYSGEN to increase MaxProcessCnt when you increase MaxClients
#
MaxClients		100

#	MaxRequestsPerChild defaults to 0 (which is good when there are no memory leaks)
#
MaxRequestsPerChild	999

#
# send less text on the server line (affects every web page, and page component)
#
# "ServerTokens Full" (default) sends this:
#			Server: Apache/2.0.63 (OpenVMS) mod_ssl/2.0.63 OpenSSL/0.9.8w
# "ServerTokens Min" sends this:
#			Server: Apache/2.0.63
# "ServerTokens Prod" sends this:
#			Server: Apache
#
ServerTokens		Prod

[...snip...]

Changes to OpenVMS SYSGEN (for Apache)

Apache needs resources for interprocess communications (VMS mailboxes are like UNIX pipes)

legend:	<sr> = system response
	<ur> = user response
-------------------------------------------------------------------------
	recording your current sysgen settings to a file
<sr>	$
<ur>	def/user sys$output sysgen_20131031.txt	! output will be diverted
<sr>	$
<ur>	mcr sysgen
<sr>	SYSGEN>
<ur>	sho /all				! output goes to file
<sr>	SYSGEN>
<ur>	exit
<sr>	$
<ur>	type sysgen_20131031.txt
<sr>	...file contents are displayed...
-------------------------------------------------------------------------
	making changes to your running system (DANGER)
<sr>	$
<ur>	mcr sysgen
<sr>	SYSGEN>
<ur>	sho maxbuf
<sr>	Parameter Name Current Default    Min.    Max.  Unit Dynamic
        -------------- ------- ------- ------- -------  ---- -------
        MAXBUF            8192    8192    4096   64000 Bytes       D
<ur>	set maxbuf 64000
<sr>	SYSGEN>
<ur>	set defmbxbufquo 64000
<sr>	SYSGEN>
<ur>	set defmbxmxmsg 64000
<sr>	SYSGEN>
<ur>	write current
<sr>	SYSGEN>
<ur>	write active
<sr>	SYSGEN>
<ur>	exit
<sr>	$
--------------------------------------------------------------------------
	ensure you add these overrides to file sys$system:modparams.dat

Webpage hit-counters (2013-11-29)

Image-based Counters

These seem to be the gold-standard in counters. Between 1995 and 2000 (when web servers were powerful platforms serving up mostly static webpages to underpowered desktop PCs sporting Pentium-2 or Pentium-3 processors) most sites used a bit of freeware written by Muhammad Muquit named "count". Needless to say, I am envious of his programming skills. But this solution places a more-than-a-trivial computational burden on the server. Why? Updating a count-file is the easy part so no problem here. However, using the count-file digits to reference a library of individual digits graphics then assemble binary slices scan-line by scan-line into a resultant GIF is the harder part.

Needless to say that Muquit's program works on many webserver flavors including those for OpenVMS. Some of those distributions can be found here

I'm running an overworked decade-old server (an AlphaServer DS20e installed in 2002) and now think the time has come to shift the computational burden from the server to the client's browser. I have a little "C" program ready which increments the counter file then sends back the plain-text result. The calling webpage uses a small amount of AJAX (~20 lines) to send the increment request, receive the plain-text count, then inject the value into the browser's DOM for rendering. The browser may not have access to all the cool image libraries seen in Muhammad's offering but that may not be as big a deal as you might think. At least your boss (and user community) will get instant feedback.

On my systems, most users can't tell one font from another so never noticed anything other than a faster response.

What's up with "browserconfig.xml"?

Okay so it looks like IE-11 is always asking for file browserconfig.xml from your server root directory

• If the file is sent then IE-11 will cache it (so will not ask for the file again during the current session)
• If the file is not found then IE-11 will keep asking for it -AND- Apache will be continually writing file-not-found messages to the error log

So the best way forward is to create a minimal version of this file in the server root (on my system we have "/" mapped to "/apache$Documents/main/")

<?xml version="1.0" encoding="utf-8"?>
<browserconfig>
  <msapplication>
  </msapplication>
</browserconfig>

 ACL's on OpenVMS

• Most Apache files in OpenVMS have been tagged for auditing via an ACL (see yellow block just below)
• This is done with DCL command "$SET SECURITY/ACL=(stuff) file.ext"
• This means that whenever any of these files are touched in anyway by anyone or anything, the touch event will be sent to the AUDIT_SERVER for logging.
• This is a lot like running the system with full ACCOUNTING enabled except that auditing attaches and additional compute burden only to the stuff being audited (but is is seen my all)
• Auditing is most likely a good idea since Apache is Open Source and most of us have not inspected every line of code.
• This is especially true when we are using third-party plug-ins and add-ons.
• However...
  • If your system is very busy (or too busy at critical peak times-of-the-day)
  • And you trust the software
  • And you trust your system admins
  • Then consider removing ACLs like so:
    
    SET SECURITY/ACL/DELETE=ALL apache$common:[000000...]*.*;*

$ dir/full APACHE$HTTPD.EXE

Directory APACHE$COMMON:[000000]
APACHE$HTTPD.EXE;5            File ID:  (13942,118,0)         
Size:           25/32         Owner:    [AP_HTTPD,APACHE$WWW]
Created:    22-AUG-2012 18:00:19.83
Revised:     6-DEC-2013 14:13:29.47 (3)
Expires:    <None specified>
Backup:     <No backup recorded>
Effective:  <None specified>
Recording:  <None specified>
Accessed:    6-DEC-2013 14:13:26.49
Attributes:  6-DEC-2013 14:13:29.47
Modified:   23-AUG-2012 09:27:27.75
Linkcount:  1
File organization:  Sequential
Shelved state:      Online 
Caching attribute:  Writethrough
File attributes:    Allocation: 32, Extend: 0, Global buffer count: 0, Version limit: 15, Contiguous best try
Record format:      Fixed length 512 byte records
Record attributes:  None
RMS attributes:     None
Journaling enabled: None
File protection:    System:RWED, Owner:RWED, Group:, World:
Access Cntrl List:  (IDENTIFIER=APACHE$EXECUTE,ACCESS=READ+EXECUTE)
ClClient attributes:  None

Getting rid of .HTACCESS and INDEXES

• The official Apache docs are full of reasons why you should get rid of .HTACCESS files by moving directives into file httpd.conf
• By implementing directive "AllowOverride None" Apache will no longer search individual directories looking to see if an .HTACCESS file has been recently added, removed or changed (.HTACCESS was developed in the 1990s for use by customers who wanted to modify access rules on "their" individual directories but did not have access to the common httpd.conf file)
• But this problem gets much worse when INDEXES (provides you with a directory listing when DEFAULT.HTML is not found) is enabled on a directory requiring authentication
• I was recently experimenting with my own MOD_AUTH authentication files and had inserted some trace code which would log all activities including date-time
• With browser cache always initialized before each test...

  accessing this path	ran the authentication code this many times	(file: default.html exists?)
  http://node/yada/	6	N
  http://node/yada/	2	Y
  http://node/yada/file.ext	1	n/a

• You can only image the additional load this would place on a system if each execution required a single database lookup.

Shift authentication from Apache to your apps -AND- remove unnecessary plugins

• On many Apache servers you will see numerous authentication modules loaded from httpd.conf which may include several of this small subset:

  mod_auth mod_auth_basic	supports Basic access authentication using a base64 Authorization string mod_auth was renamed to mod_auth_basic in Apache-2.2
  mod_auth_digest	better than Basic in that the username + password are message digest encrypted
  mod_auth_kerberos	authenticates by dipping into Active Directory (a.k.a. Microsoft stuff)
  mod_auth_ldap	authenticates by dipping into LDAP
  mod_auth_openvms	authenticates by dipping into SYSUAF (a.k.a. OpenVMS stuff)

• many times these modules were enabled by a well-meaning webmaster who needed to solve an urgent problem while thinking that he would only enable the desired module in paths where it would be required. But the truth is actually a little more sinister:
  • every module which is loaded will create "per-directory OFF settings" for every directive supported by that plugin
  • every auth module will most likely contain a line similar to this:
    ap_hook_check_user_id(authenticate_whatever,NULL,NULL,APR_HOOK_MIDDLE);
    which will register routine "authenticate_whatever" into the check-user-id execution queue.
  • When Apache scans the check-user-id execution queue it will execute every hook until one registered plugin responds with SUCCESS (allow), FAIL (reject) or DECLINE (defer to the next hook).
    • Loading five authentication modules means that up to five hooks will be fired (hopefully each plugin author did the best job possible)
    • While I was tracing "all activity" (this includes the DECLINE) associated with an auth module I was writing for Apache 2.0, I was loggin all the Apache authentication events in the system. This happens for every authenticated transaction so only load the auth modules you really need then develop a plan to reduce the number of modules to one (mod_auth_basic) or none.
    • I got my system down to one by doing Basic access authentication in my own plugin. So it is my intention to:
      1. do cookie-based application authentication in all my apps
      2. use my custom plugin to limit access to certain restricted directories
• If you shift authentication from "Apache proper (where some of this stuff would be passed to an application via CGI)" into your "Apache applications" -AND- unload the unnecessary Apache modules, you will greatly reduce per-transaction overhead.

Freeing Up Disk Space with Log Rotation

Note: this will aid in analyzing your Apache log files so you can improve your web server system. Failure to do this places all your transactions in one huge file.

<sr>	$
<ur>	sh time
<sr>	27-NOV-2005 20:52:11
	$ 
<ur>	dir [...]*.*/siz=all/date/sel=siz=min=10000
<sr>	Directory APACHE$COMMON:[000000.SPECIFIC.KAWC15.LOGS]

	ACCESS_LOG.;1        1061068/1061095  16-SEP-2003 19:48:46.00
	ERROR_LOG.;1          455636/455700   16-SEP-2003 19:48:44.50
	SSL_ENGINE_LOG.;1     238230/238280   16-SEP-2003 19:48:44.56

	$
<ur>	del APACHE$COMMON:[000000.SPECIFIC.KAWC15.LOGS]*_log*.*;*

Wow! These files have been growing 26 months

Notes:

1. These files grow forever. I have not been able to get "built-in Apache log rotation" working on OpenVMS (it would have added overhead to Apache anyway) so it is probably a good idea to stop the server every month to delete the files
2. Here is a better method (you do not need to stop the server)
   • start by creating 31 subdirectories named [.log01] to [.log31] which would hold log files for each day of the month
   • run a batch job every night (perhaps at one second after midnight) to do the following:
     1. rename the files into the desired subdirectory while they are open (this is legal in OpenVMS and writing will continue in the new location)
     2. now execute this job (a DCL command template for day 31)
        

        $ set def sys$common:[000000.specific.www.logs]	!
        $ ren *_log*.* [.log31]				! okay to do this while files are open
        $ @APACHE$COMMON:[000000]APACHE$SETUP		! create Apache symbols for our process
        $ httpd -k flush				! tell Apache to flush the log buffers to disk
        $ httpd -k new					! tell Apache to close current files then open new ones

3. or you could just use this automated DCL script    <<<---***

Character Sets (why servers must never lie to clients)

Overview: A few common character sets

Single Byte Characters

• In the early days of mainframe and minicomputers, 128 characters were represented by one single 8-bit byte usually represented by 7-bit ASCII. One exception to this was EBCDIC which was created and promoted by IBM for its mainframe business
• With the sale of minicomputers into a worldwide marketplace, additional Western European characters were added by supporting 8-bit ISO-8859
  • the lower 128-characters mapped to ASCII
  • the upper 128-characters mapped to the extended set
    • ISO-8859-1 (also called latin1) was the most popular character set used in the western world
    • ISO-8859-2 to ISO-8859-9 was used through out the rest of the world
• With the popularity of Windows-based personal computers, Microsoft added 32 additional characters to ISO-8859-1 which they called Windows-1252 (now commonly referred to as ANSI). This decision predates the internet and therein lies today's problem.

Wide Characters + Multibyte

• With the advent of internet applications including email and web browsers, customer-facing software needed to move beyond the Latin Character Set and did so by supporting Unicode
• Unicode began as Unicode88 which was nothing more than a 16-bit character set co-developed by Apple and Xerox. It was further developed by Sun Microsystems and Microsoft.
• For programmers, the simplest way to represent Unicode is to use wide character sets where all characters are coded as 2-byte entities, 3-byte entities, and 4-byte entities (the choice depends on how many characters you are willing to support)
  • Wide Character Sets and Multibyte support has been available in C/C++ for a long while
    • the number of bytes used determines how many Unicode characters you are prepared to support
      • UCS-4 for Linux
      • UCS-2 for Windows (although UCS-2 is now defunct in the wild)
    • Here are two examples of many:
      • printf is used to output single byte characters
      • wprintf is used to output wide character sets
  • Unicode is built into Java
  • Unicode is built into some JavaScript technologies (JSON springs to mind)

Sending Unicode over a communications channel

Back in the early 1990s, most data communications occurred via modems so engineers were always looking for ways to send it so that it so that it would not lockup the modem. Many standards were developed including UCS-2 which precedes UTF-16, UCS-4 which precedes UTF-32 but UTF-8 became the most popular. Too bad its implementation in internet applications was so ad-hoc.

1. HTML meta statements incorrectly use the word CHARSET where UTF-8 support was added with very little thought

   <meta charset="UTF-8">

   The creators of the "charset kludge" probably meant "character set will be Unicode but will follow UTF-8 encoding rules"
   
2. HTTP 1.1 Response Headers are a little less confusing:

   HTTP/1.1 200 OK
   Date: Mon, 23 May 2005 22:38:34 GMT
   Content-Type: text/html; charset=UTF-8
   Content-Encoding: UTF-8
   Content-Length: 138
   Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
   Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux)
   Connection: close

3. XML documents properly use the phrase ENCODING

   <?xml version="1.0" encoding="UTF-8"?>

4. MySQL and MariaDB have some confusing settings. While you can store the data as "Unicode binary", you select the internal variable size with these directives:
   •  utf8 = 3-bytes
   •  utf8mb4 = 4-bytes
   On the flip side, if your client connects using charset=latin1 then sends latin1 data to a database expecting utf8, the database will convert the data before storing it. It is almost impossible to convert utf8 back to latin1 without loosing something (unless you resort to HTML entities) which is why most web apps today only support utf-8

Failover Logic in Browsers

While the browser wars are over, browser vendors are still going out of their way to make sure documents render properly no matter what the server told them. For example, most versions of IE will render documents even though they are not HTML compliant (e.g. a missing tags like: <html> , <head> , and  <body>). On top of this, all browsers tend to support only two character set flavors:

1. utf-8 (which is not really a character set; it is a Unicode encoding)
   
2. everything else fails over to windows-1252 (which is a superset of iso-8859-1 which is a super set of ASCII)

• If you send ASCII data to a browser expecting iso-8859-1, everything will work properly because ASCII is mapped over the lower half of the iso-8859-1 character set
  
• If you send iso-8859-1 data to a browser expecting iso-8859-1, everything will work properly (obviously)
  
• If you send windows-1252 (also called ANSI) data to a browser expecting iso-8859-1, everything will work properly because all browsers support windows-1252 even though iso-8859-1 was specified (huh?)

  quote: browsers will change to Windows-1252 when ISO-8859-1 is declared. This is done for any DOCTYPE: HTML4, HTML5, and XHTML.
  reference-1: http://www.w3schools.com/charsets/ref_html_ansi.asp
  
  quote: Most modern web browsers and e-mail clients treat the MIME charset ISO-8859-1 as Windows-1252 to accommodate such mislabeling. This is now standard behavior in the HTML5 specification, which requires that documents advertised as ISO-8859-1 actually be parsed with the Windows-1252 encoding.
  reference-2: https://en.wikipedia.org/wiki/Windows-1252

• However, if you send UTF-8 data to a browser expecting iso-8859-1 then any characters between 128 and 255 will be interpreted incorrectly
• Likewise, if you send iso-8859-1 data to a browser expecting UTF-8 then any characters between 128 and 255 will be interpreted incorrectly

Apache Directives

• Apache configuration file httpd.conf is usually installed with directive: AddDefaultCharacterSet iso-8859-1 or AddDefaultCharacterSet On which both mean the same thing. This is instructive in that it appears that Apache was developed with iso-8859-1 in mind (a character set encoding first published in 1987)
  
• Some newer Apache distributions provide a default httpd.conf where AddDefaultCharacterSet is set to windows-1252 but uniformed webmasters sometimes change this back to iso-8859-1 while others change this to utf-8
  
• According to Apache documentation, DefaultCharacterSet is only used when you, the CGI (common gateway interface) programmer, do not supply a CHARSET in the HTTP response header. Here is an excerpt from the O'Reilly book Apache: The Definitive Guide which I highly recommend:

  This directive specifies the name of the character set that will be added to any response that does not have any parameter on the content type in the HTTP headers. This will override any character set specified in the body of the document via a META tag. A setting of AddDefaultCharset Off disables this functionality. "AddDefaultCharset On" enables Apache's internal default charset of iso-8859-1 as required by the directive. You can also specify an alternate charset to be used; e.g. "AddDefaultCharset utf-8"

Sending data back to the server

Both these HTTP methods...

• GET (these are mostly data requests)
• POST (these are mostly data pushes)

...can be coupled to a SUBMIT button on your HTML form -and- will usually (at least before 2008) default to the character set defined for the currently rendered web page. But this is not the case for AJAX-based pushes which now default to UTF-8 no matter what character set was used in the currently rendered web page. To confuse things further:

1. people continually interchange the words Unicode and UTF-8
   
2. UTF-8 transmits 7-bit ASCII (codes: 0-127) as-is but converts all codes above 127 to multi-byte encodings.
   
3. If UTF-8 is declared, sending certain single byte characters (iso-8859-1 or windows-1252) as-is will...
   • be displayed incorrectly in most browsers
   • stop an XML parser and might throw an error message

More details on character sets

• Unicode is a character set while UTF-8 is a Unicode encoding
  
• What's the difference? Think about UTF-8 as a dialup-safe compression method for sending Unicode
  • some online references incorrectly show bytes 2-3-4 as "1xxxxxxx". That material comes from the initial proposal published in 1992. This was changed to "10xxxxxx" in the 1993 standard. The resulting change is a little less efficient but is better able to to detect encoding errors (e.g. detecting a first byte pattern like this "11110xxx" means that three bytes must follow based upon a pattern like this "10xxxxxx")
  • some online references incorrectly show 5 and 6 byte codings. These were dropped from the official spec in 2003 (see RFC-3629)
  • this table describes how UTF-8 is currently implemented on the internet

    UTF-8 (2003)

    Total Bytes	Data Bits	First code point	Last code point	Byte 1	Byte 2	Byte 3	Byte 4
    1	7	U+0000	U+007F	0xxxxxxx
    2	11	U+0080	U+07FF	110xxxxx	10xxxxxx
    3	16	U+0800	U+FFFF	1110xxxx	10xxxxxx	10xxxxxx
    4	21	U+10000	U+10FFFF	11110xxx	10xxxxxx	10xxxxxx	10xxxxxx

• At this point, you might want to inspect this page showing four character sets side-by-side: 
  • reference: http://www.w3schools.com/html/html_charset.asp (caveat: here "UTF-8" is listed as a character set when the author meant "Unicode")

Behind the scenes (information for programmers)

• As long as computers were using single byte character sets (ASCII, ISO-8859-1, Windows-1252) only a single byte of memory or storage was required
  • here is a list of c functions for doing byte i/o:
    • fgetc fputc fscanf fprintf ungetc fgets fputs scanf printf fread getc putc vfprinf fwrite gets puts vprintf getchar putchar
• But switching to Unicode brings new problems. Do we...
  • represent Unicode characters and strings internally as multi-byte characters? (e.g. UTF-8)    -or-
  •  as wide characters which is also known as binary? (limited to one of: 2-bytes, 3-bytes or 4-bytes)
• While UTF-8 would require much less memory, wide characters and wide strings would be easier to compare and collate
• Here is a short list of C library functions for converting characters including converting between multibyte and wide:
  • ecvt fcvt toupper gcvt mbtowc towctrans mbrtowc wctrans mbsrtowcs wcrtomb toascii wcsrtombs tolower
• Here is a list of c functions for doing wide i/o:
  • fgetwc fputwc fwscanf fwprintf ungetwc fgetws fputws wscanf wprintf getwc putwc vfwprintf getwchar putwchar vwprintf
    caveat: This are implementation dependent. For example, a wchar_t is definded as:
    • 32-bit wide on Linux holding UCS-4/UTF-32 encoded Unicode
    • 16-bit wide on Windows holding UTF-16 Unicode (originally it held UCS-2 Unicode, but UCS-2 is now officially obsolete)
• Modern databases like MySQL and MariaDB can store data in whatever way you declare
  • latin1 means single character
  • UTF8 means 3-byte binary (for character-based languages)
  • UTF8mb4 means 4-byte binary (RFC-3629)

• Almost every text file used by System32 on Windows contains 16-bit characters (UCS-2). Most IDEs (Visual Studio, Eclipse, NetBeans) write characters as 16-bit binaries which is why you had to edit via an IDE because common text editors were only single byte aware. For people wanting to learn more, try playing with this free editor named Notepad++ making sure check out the Encoding menu.
  
  p.s. it can open ANY file including corrupt spreadsheets etc.

Additional Technical Info

• Unicode in C and C++: What You Can Do About It Today
  • http://www.cprogramming.com/tutorial/unicode.html
• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
  • http://www.joelonsoftware.com/articles/Unicode.html
• UTF-8: The Secret of Character Encoding (this should be your bible)
  • http://htmlpurifier.org/docs/enduser-utf8.html
  • http://htmlpurifier.org/docs/enduser-utf8.html#fixcharset
  • http://htmlpurifier.org/docs/enduser-utf8.html#whyutf8-forms (all about the SUBMIT method associated with HTML forms)
    • quote: This is the Content-Type that GET requests must use, and POST requests use by default. It involves the ubiquitous percent encoding format that looks something like: %C3%86. There is no official way of determining the character encoding of such a request, since the percent encoding operates on a byte level, so it is usually assumed that it is the same as the encoding the page containing the form was submitted in. (RFC 3986 recommends that textual identifiers be translated to UTF-8; however, browser compliance is spotty.) You'll run into very few problems if you only use characters in the character encoding you choose.
  • http://htmlpurifier.org/docs/enduser-utf8.html#migrate-db
• UTF-8 on Wikipedia
  • https://en.wikipedia.org/wiki/UTF-8
• Unicode HOWTO
  • http://www.tldp.org/HOWTO/Unicode-HOWTO-1.html
  • http://www.tldp.org/HOWTO/Unicode-HOWTO-6.html Making your programs Unicode aware
• What is UTF-8
  • http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
• Some people reading this page might find this book in their office library: Digital UNIX - Writing Software for the International Market - March 1996
  please read chapter-2 including:
  • Using Codesets
  • Manipulating Multibyte Characters
  • Converting Between Multibyte-Character and Wide-Character Data
  • Binding a Locale to the Run-Time Environment

Affects all of XML as well

• I recently received an XML document where the encoding directive on line-1 was set to "utf-8" but this document could not be parsed.
• I inspected the data and noticed this French character: é (e-acute) in the CDATA block
  • What? up until this point I thought anything could be placed in the CDATA block. Wrong!
• Obviously this French data was coming from a database which was holding windows-1252 data but some boilerplate had defaulted the XML encoding to utf-8
• Just using a plain-text editor to modify the encoding to "windows-1252" fixed the problem. All the JavaScript parsers built into IE, Firefox, and Chrome now could read the XML document.

Interim Solution for OpenVMS

• In order to edit UTF-8 encoded data files on OpenVMS, you will require:
  1. a terminal emulator able to directly deal with UTF-8
     • this is the default for Tera Term where encoding defaults to "UTF-8" (in fact, only UTF-8 is available without optional plugin modules)
       • Menu:Setup Item:Terminal Selector:Coding (receive) and Selector:Coding (transmit)
     • Reflection-v14
       • Menu:Setup Item:Terminal Tab:Emulation Selector:Host Character Set
     • you terminal driver set to eight bit like so
       • $set term/eight
  2. an editor able to directly deal with UTF-8 (so changing one character position on your screen moves the file insertion pointer by 1-4 bytes)
• Get a copy of VIM for OpenVMS from this cool site in Stockholm, Sweden
  • http://www.polarhome.com/vim/
• add these two lines to file VIM:defaults.vim
  • set encoding=utf-8
  • setglobal fileencoding=utf-8
• refer to this reference:
  • http://vim.wikia.com/wiki/Working_with_Unicode

CSWS-2.2-1 (2015-07-28)

• Okay so I didn't see this one coming
• I'm in the middle of migrating our business system from an AlphaServer-DS20e to an Itanium rx2800-i2
• About 99% of the work so far is pretty-much a no-brainer (copy/compile/link/next) but my web services are unavailable
  • did I mention we transitioned all of our green-screen apps to the web in 2013 then forced all our users to move over
  • did I mention we now support ~ 1400 web users?
• Most pages at our site also make extensive use of jQuery and AJAX so if the home page has problems then so will everything else.
• Weird browser behavior
  • IE11 seemed to work properly
  • Firefox-39 rendered a page but there was no sliding menu, just a linked list
  • Chrome-44 didn't render anything at all but further examination with Developer Tool's (hit F12) showed a problem with incomplete chunked transfers
• Rather than waste your time telling you all the things I did which did not work, let me say that CSWS-2.2-1 now requires text files to be stored in stream_lf format
  • http://support.hp.com/sg-en/document/c04739414
• We were running CSWS-2.2 on the Alpha which did not require stream_lf format text files so it looks like we're in a jam for a while with only two options:
  1. convert all text files to stream_lf                 -OR-
     • this includes rewriting software which generates static html-based reports
     • this includes rewriting software interfaces responsible for capturing file uploads
     • don't forget to convert css files (you will need to update the script)
     • don't forget to convert js files but you might not want to convert minified js files (oops, another wrinkle)
  2.  roll back to CSWS-2.1 then reapply update patches #1 and #2
     • however, the version of OpenSSL baked-into this release is vulnerable to numerous exploits

Update: 2015-07-29

• Good news. I requested a patch from HP/HPE today via my OpenVMS support account. HP responded within an hour by placing the patch in a 24-hour drop box. I installed the patch (a replacement version of APACHE$HTTPD_SHR.EXE) and now everything works properly.
   
• Bad news. This bug also affects the Alpha version of CSWS-2.2-1 but you can only get the patch if you have an "OpenVMS on Alpha" support contract. We dropped our Alpha support contract when we moved to Itanium (probably a mistake since we still have one production AlphaServer running but hey, we needed to do this to get approval for the Itanium business case). So I suppose we'll have to wait until HP/HPE releases a publically available patch at their CSWS site. Meanwhile it would appear that our site which is running a fully patched version of CSWS-2.2 got an "F" when I tested it on 2015-12-20 via this tool: https://www.ssllabs.com/ssltest/

Update: 2016-09-xx

• Good news. We're an all Itanium shop now. Too bad no one wants our AlphaServer machines. They were rockets 15-years ago but no longer.

Links:

• CSWS (Apache Web Server) from the good folks at VSI
• minimum requirements:
• OpenVMS-8.4-1H1 which is only available from VSI
• SSL111 which is only available from VSI (mod_ssl dynamically links SSL111)
• Apache HTTP Server Documentation
• my OpenVMS Notes
  • includes OpenVMS Notes: WASD (a very high quality free web server from VSM Software Services in Australia)
• Y2k20 in 2020 - are your computer systems ready for it?