NAME
Net::Clacks::UpgradeGuide - Notes on upgrading Net::Clacks
SYNOPSIS
This document lists important information when upgrading from previous versions of Net::Clacks.
Please be sure to read all relevant sections of this short guide. While i myself don't like breaking/incompatible changes, bugfixes and major enhancements sometimes make them the most painless way of handling things.
On the bright side, after you've gone through this document and implemented required changes (if any), you get to enjoy a newer, shinier version of your favourite real time interprocess messaging system. While i strive to remove bugs in each release, sometimes new ones creep in. Just keep bugging me about them until i fix them as well.
SECURITY AND BACKWARD COMPATIBILITY
WARNING: The author of this module only tests against stable Perl versions that are still considered "in maintenance". Since <Net::Clacks> contains network code, the author of this module strongly recommends to always keep your Perl installation as well as Net::Clacks up-to-date for security reasons.
Net::Clacks may work with older versions of Perl that are out of maintenance, but these may have unfixed security flaws.
IMPORTANT UPGRADE NOTES
VERSION 6
Version 6 (and higher) of Net::Clacks::Server imlements a smarter interclacks sync. Make sure to upgrade all nodes on your local clacks network at the same time! While the protocol itself is mostly backward compatible, interclacks sync will fail otherwise.
Version 6 also includes a smarter shutdown sequence for Net::Clacks::Client and the ability to persistantly store the clackscache data in a file in Net::Clacks::Server. The file format has also changed somewhat from previous beta versions of this feature due to the implementation of smarter sync. If you have used persistance before, you might (or might not) have to reset/remove the persistance file.
This change to "smarter" syncing has increased stability of some of my systems using "Net::Clacks", but has been in use for only a limited time. You should test with your own software before upgrading.
VERSION 7
Version 7 (and higher) of Net::Clacks::Server and Net::Clacks::Client implement a lot of bugfixes and improvements. This includes authentication timeouts, somewhat smarter automatic reconnects and stuff like that.
While the protocol in theory is backwards compatible to version 6, it is strongly recommended that you upgrade ALL nodes (clients and servers) in your network at the same time. I have done only limited testing with backward compatibility and i would not recommending a mix&match approach on critical systems.
VERSION 8
On Systems that support Unix domain sockets and have IO::Socket::UNIX installed, Net::Clacks::Server and Net::Clacks::Client can now use Unix domain sockets for local communication. See the examples for. This might or might not drastically lower CPU usage, depending on your hardware, software, weather on moon phase.
Version 8 also includes a number of bugfixes and improvements. This includes a better way of detecting closed/broken connection which should prevent servers and clients from holding on to closed connection until it times out. This should also lower CPU usage under certain circumstances. This is far from perfect, though and may lead to some false positives (e.g. accidental closure of perfectly fine connections) thanks to the combination of SSL and non-blocking sockets.
Version 8 is fully backwards compatible with Version 7 on the network layer (no protocol change), but as always it is recommended to update all servers and clients at the same time if possible.
One important client API change (sort of) is the generation of messages with type "reconnected" after a connection has been re-established. After receiving this message, a client application must resend any LISTEN calls it wants to make. While in previous versions, this was accomplished by checking for type "disconnect" messages, this was unreliable at best. The "reconnected" message is generated internally after a new connection has been established, except on the first ever connection. Technically, at that point in time Net::Clacks::Client has spooled the Auth request to the server, but may not have recieved the answer yet, but i'll assume here that you have configured your client correctly.
VERSION 9
Version 9 added the socketchmod option to chmod() the socket file so other users can connect to the socket, too.
VERSION 10
WARNING: BREAKING CHANGES! It is required to update all servers and clients at the same time.
Version 10 disables SSL on Unix Domain Sockets for better local performance. This version also adds the ability to run interclacks through Unix Domain Sockets, in case you want to run a master/slave setup locally. This should not really affect security, though. If an attacker has enough permissions to spy on a Unix Domain Socket, they will most likely also have the ability to gain access to the configuration files and SSL keys of the Clacks server running on the same host..
This version also has a slightly improved timeout handling when a slave reconnects to the master at the cost of a bit more traffic during the initial setup phase.
In the protocol, error messages have been implemented via the "E" flag in the "OVERHEAD" command. Net::Clacks::Client forwards this to the client software via a clacks message of type "error_message".
In theory, the only *breaking* incompatible change is contained in handling Unix Domain Sockets. You could (try to) upgrade to Version 10 client-by-client if you don't use Unix Domain Sockets, but this is neither tested nor recommended.
VERSION 11
Under certain circumstances Net::Clacks::Client would hang or otherwise error out during DESTROY(). This is due to the fact that the underlying socket might already have been DESTROYed by perl. I could reliably produce hangs in certain applications when calling exit(0) in the main loop.
In Version 11, the DESTROY method now uses eval() to try and catch any errors. It is recommended that you use the new disconnect() function in Net::Clacks::Client BEFORE exiting the program or deleting the last reference to the instance. This will then try to send out any remaining commands in the output buffer. Simply exiting the program will still work, but there is a good chance that anything still remaining in the output buffer after your last call to doNetwork() will NOT be sent.
Version 11 also implements a Unicode/UTF8 bugfix. This should have no visible side effects in pretty much all use-cases of Net::Clacks
Upgrading to version 11 on a client-by-client basis should work fine since there have been no incompatible changes, but this has only seen limited testing. As always, it is recommended to upgrade everything in one go.
VERSION 12
This is mostly an internal bugfix release. It now again allows in install Net::Clacks directly, instead of requiring the user to name the full sub-module name ("Net::Clacks::Client").
In this release, a new disconnect() method was added to Net::Clacks::ClacksCache. This will allow to disconnect cleanly from the server.
VERSION 13
Before this version, Net::Clacks::Server will always cache deleted keys for one week. Caching is needed for clean reconnection of interclacks links.
But the long caching time can lead to all sorts of problems, especially with things like cached web sessions or temporary cookies, that sort of thing. Cache time of deleted keys is really a trade off between the ability of interclacks links to sync cleanly after long interruptions of the connection and the memory and bandwidth required to do so.
Version 13 now defaults to 1 hour (3600 seconds) of caching deleted keys. This can be changed in the "deletedcachetime" config setting.
Syncing now also better handles timeout situations during long sync by avoiding them. Well, i hope it does. Fingers crossed, really...
This release is backwards compatible with Version 12. But it is highly recommended to upgrade at least all server nodes to this version as soon as possible to avoid all sorts of problems.
VERSION 14
Version 14 has only internal bugfixes and no externally incompatible changes.
The main "feature" of this release is more internal error checking. This is to detect broken connections sooner and a bit more reliable.
VERSION 15
Version 15 adds the configuration variable "persistanceinterval" (defaults to 10 seconds for backward compatibility). This defined the interval in which a persistance file is written. You could lower it for more frequent checkpointing, but more importantly you could increase to longer time spans to reduce disk activity.
VERSION 16
Version 16 Server now croaks with a descriptive message if it encounters an invalid persistance file.
This version does not introduce any incompatible changes.
VERSION 17
Version 17 Server has better checks for Net::Clacks::Client call arguments and croaks or carps where appropriate.
This version does not introduce any incompatible changes but it will produce callback traces when called with missing arguments. Calling new() now croaks/dies when arguments are missing (as it should have from the beginning).
VERSION 18
WARNING: BREAKING CHANGES regarding persistance files.
Version 18 provides better reliability for the Clacks Server when using a persistance file, including startup problems and prevention of data loss.
Here is the problem in older versions: If the server process is killed while writing the persistance file, the file on disk may be invalid (partial data) or completely blank. Blank files at least prevented the server from starting up, but otherwise incomplete files could lead to data corruption in cached data and/or all other sorts of problems. This relates to the fact that the server only does very minimal checks on its internal cache during runtime for performance reasons. The original persistance file was just a quick hack to dump/restore the cache between runs and did not account for system crashes or the process being killed while writing to disk.
This new version of Net::Clacks::Server tackles this in in a multi-stage approach:
1.) The persistance file now includes the ENDBYTE string as a third line. This is used to check if the file is complete.
2.) Before overwriting the current persistance file, it is COPIED to a file with '_bck' added to the configured name.
3.) The file is written under a new name (with '_' added at the end) and then MOVED over the correct file name. File move (rename) within the same directory should be an atomic operation of the operating system, so it either works or doesn't (no stopping half-way inbetween).
4.) An invalid persistance file doesn't prevent the clacks server startup. If it detects an invalid file, it first tries to load the previous version ('_bck'), then as a last desperate measure to prevent data loss, it tries to load the temporary file ('_'). If that also fails, it starts 'blankety-blank'.
Clacks caching was always designed as a CACHE, not as the final storage for important data. This upgrade to Version 18 changes the file format of the persistance file slighty. The older format is now detected as invalid, resulting in a "blankety-blank" startup. If you really want or need (?!?!?!) to preserve the cached data for some reason, there are two ways to accomplish this:
1.) Stop the clacks server and upgrade the Net::Clacks package. Edit the persistance file and add a third line with the word "ENDBYTES" without quotes as its only content.
or
2.) If you are running a master/slave setup with interclacks, you can stop ONLY the server that has the persistance file. Then upgrade Net::Clacks and start the server. It should start "blankety-blank" and automatically resyncronize cached data from the interclacks network. It works the same if all your clacks servers use persistance files for some weird reason, just restart them one-by-one, waiting inbetween for the interclacks sync to finish. You can use the example rawclient.pl. Connect to the server you keep running and start the MONITOR command. While interclacks sync is in progress, you will see KEYSYNC commands flying between the servers. If you stop seeing them for a few seconds, KEYSYNC is finished.
VERSION 19
Version 19 allows for more flexible config file handling. You can define the environment variable PC_CONFIG_PATHS to define the search paths for the config file and the use a the config file name without path. This is very handy for deploying the same start scripts and config files to multiple systems with different environments and installation paths.
for example:
export PC_CONFIG_PATHS=/home/username/src/chatserver/configs:/usr/share/chatserver/configs
perl chatserver.pl minecraftchat.xmlThis would search the config file minecraftchat.xml in /home/username/src/chatserver/configs first, and fall back to /usr/share/chatserver/configs.
Version 19 also allows you to define other environment variables starting with PC_.* and those will replace the corresponding text strings in the config file (only in memory, not on disk).
Say you want to change the TCP port where the server is listening when you are debugging. If you define the port in the config like this:
<port>PC_CLACKSPORT</port>then you can say export PC_CLACKSPORT=18889 perl chatserver.pl minecraftchat.xml
or even
PC_CLACKSPORT=18889 perl chatserver.pl minecraftchat.xmlThis change is largely backwards compatible. If you don't change anything on your system, the only change in behaviour will be a few additional messages on STDOUT during startup.
VERSION 20
Bugfix release: Fix package name in Net/Clacks.pm
Net::Clacks is dummy class for documentation purposes, so there should be no functional changes (except to fix "cpan install Net::Clacks" and some issues with MetaCPAN)
VERSION 21
WARNING: BREAKING CHANGES! It is required to update all servers and clients at the same time.
This version implements a network-optimized change on how setAndStore works by introducing a decicated SETANDSTORE command. This halfs the network load for all setAndStore operations, because the command and payload gets only send once instead of twice. This is especially useful on slow links with larger payloads.
This version is dedicated to Sven Guckes, who passed away on 20th February 2022. Accordingly, his name will now occasionally passed in the overhead of the Clacks protocol.
"Do you not know that man is not dead while his name is still spoken?" (Terry Pratchett, Going Postal)
VERSION 22
WARNING: BREAKING CHANGES! It is REQUIRED to update all servers and clients at the same time.
Version 22 fixes some warnings and wrong print statements regarding persistance files.
This version also makes sure that the persistance file is saved on exit, to make sure we have the latest data on file, instead of something potentially outdated.
Another important change is the way interclacks sync works. Before sending the keys to the interclacks partner, every interclacks node sends an "OVERHEAD T" timestamp. This allows the other side to calculate (roughly) the time offset between the nodes and adjust the recieved timestamps accordingly. If the nodes are both synced to the same system time, this is pretty much of no importance. But if one of the nodes has a large offset (think "Raspberry Pi when it can't reach a time server during boot"), syncing via time stamps can get messed up. Version 22 of Net::Clacks tries its best to rectify that. Of course, this can't replace proper time sync of the interclacks nodes, but it's better than nothing, i guess.
This version also implements "forget stale entries". That means, it removes cached entries which have not been accessed (STORE, SETANDSTORE, INCREMENT, DECREMENT, RETRIEVE) in a long time. The default timeframe if 24 hours, but this can be configured with the "stalecachetime" parameter (in seconds). This means that the persistancefile format changes, but this will be upgraded transparently by the software during startup. This changes the interclacks KEYSYNC command to include the last access time.
VERSION 23
This purely internal change does not break backward compatibility.
Version 23 switches Net::Clacks to use function signatures (features 'signatures'). This makes the code a bit clearer to read, properly enforces the correct number of function arguments and will allow slimming down the argument checking in future releases.
WARNING: While this change should work with Perl 5.20 and up, the author of this module only tests against stable Perl versions that are still considered "in maintenance". As of this module release, these are 5.32.x and 5.34.x. Since <Net::Clacks> contains network code, the author of this module recommends to always keep your Perl installation as well as Net::Clacks up-to-date for security reasons.
VERSION 24
WARNING: BREAKING CHANGES! It is REQUIRED to update all servers and clients at the same time.
This is a bugfix and stability enhancement release.
Version 24 fixes interclacks KEYSYNC problems. Some keys failed to sync after the clacks servers were running for some time (affects versions 22 and 23, possibly earlier versions as well). This was caused by a badly designed cache cleaning algorithm that wrongly removed the "cache time" while keeping the actual data, which in turn lead to sending out invalid KEYSYNC commands to the interclacks partner.
I rewrote the way clacks cache entries are stored internally. This also changes the persistance file format, which will be upgraded in-place automatically (in nearly all cases).
The caching algorithm should now be much more stable and easier to maintain (fingers crossed).
The automatic cache cleaning interval can now be set by the "cachecleaninterval" config, which defaults to 60 seconds. "deletedcachetime" sets how long deleted cache entries are kept for interclacks sync (default 1 hour), and "stalecachetime" sets how long to keep cache entries that have not been accessed (default 1 day).
This release also has a lot of code refactoring and other internal changes.
This release is dedicated to the dog Sheila, faithful companion of my friend @NightStorm_KPC. Rest in peace, Sheila.
VERSION 25
This version is (in theory) fully backward compatible. The only change is the addition of the Net::Clacks::PostgreSQL2Clacks module.
This Version is dedicated to Tobias, my coworker whos software always wants to know (in real time!) if critical data has changed in the database. WITHOUT polling the database constantly. Displaying up-to-date data to the user? What a strange and novel concept.
VERSION 26
This fixes some documentation issues.
VERSION 27
This fixes some small design issues:
* Unix domain sockets don't use SSL. If only UDS are used in a Server, there is no requirement to specify server certificates.
* runOnce() in Net::Clacks::PostgreSQL2Clacks now returns the lines it would have formerly printed to STDERR. The non-returning run() still prints to STDERR.
VERSION 28
This version is (in theory) fully backward compatible (i hope).
Version 28 of Net::Clacks::Server now supports multiple users with different permissions. The primary user (same as in previous versions of Net::Clacks) is the only one that can run Interclacks connections.
The additional users can have their permissions limited to read/write/manage (or a combination thereof). See "clacks_master.xml" in examples for details.
* read permission allows to read cached variables and listen to events.
* write permission allows to add/alter/delete cached variables and transmit events
* manage permission allows to manage the server (MONITOR, disconnect other clients, etc)
It's important to note that a write permission does NOT imply a read permission. Basically, you can have, for example, sensors that only transmit their state. They don't need the ability to read data or listen to other events.
From Version 28 onwards, Makefile.PL and the documentation also clearly state the minimum Perl version required, which currently is v5.36. As a general rule, i only support the two most recent stable versions of Perl, similar to the developers of Perl itself. If no new Perl feature is required, Net::Clacks will be able to run on older Perl versions as well, but i will only test on supported versions.
Net::Clacks::Server also has an additional way to call the event loop, runOnce(). This allows you to run your own event loop and just call runOnce() every few milliseconds or so.
Special thanks goes to kcott from PerlMonks for the MIN_PERL_VERSION stuff in Makefile.PL
VERSION 29
Increase the Listen queue (=how many new connections can be handled per cycle) from 1 to 20. This should increase reliability of initial startup (and reconnects after a network problem) of clients more reliable.
This version is fully backwards compatible to Version 28
AUTHOR
Rene Schickbauer, <cavac@cpan.org>
Source code repository
The official source code repository is located at: https://cavac.at/public/mercurial/Net-Clacks/
COPYRIGHT AND LICENSE
Copyright (C) 2008-2024 Rene Schickbauer
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.10.0 or, at your option, any later version of Perl 5 you may have available.