/ 
App-Netdisco-2.034001
River stage one • 1 direct dependent • 1 total dependent
/ App::Netdisco::Manual::Developing

NAME

App::Netdisco::Manual::Developing - Notes for contributors

DEVELOPER NOTES

This document aims to help developers understand the intent and design of the code within Netdisco. Patches and feedback are always welcome :-)

TLDR; I want to clone git and run the web server.

First do a normal App::Netdisco install into a dedicated user's home, as per the documentation. Then:

su - netdisco && cd $HOME
mkdir git && cd git

git clone git://git.code.sf.net/p/netdisco/netdisco-ng netdisco-ng
cd netdisco-ng/Netdisco

DBIC_TRACE=1 ~/bin/localenv plackup -R share,lib -p 5001 bin/netdisco-web-fg

The above creates you a git clone (change the URL if you're a Netdisco Developer) and runs the web server:

  • In the foreground

  • Using a single process only (no forking)

  • With DBIx::Class tracing

  • On port 5001 so it won't conflict with any already-running web frontend

  • Restarts the web server when you save a file in the share or lib directory

You might also want to set check_userlog to false in your config to quieten some of the web client callbacks.

For the daemon, it's very similar:

DBIC_TRACE=1 ~/bin/localenv bin/netdisco-daemon-fg

You can point at a different database without editing deployment.yml:

NETDISCO_DBNAME=testdb DBIC_TRACE=1 ~/bin/localenv plackup -R share,lib -p 5001 bin/netdisco-web-fg
NETDISCO_DBNAME=testdb DBIC_TRACE=1 ~/bin/localenv bin/netdisco-daemon-fg

It's recommended to delete the "~/perl5/lib/perl5/App/Netdisco" directory to avoid accidentally picking up old Netdisco code. For working on SNMP::Info you can similarly delete "~/perl5/lib/perl5/SNMP/Info*" and then symlink from "Info.pm" and "Info" to your git repo. If you pull from upstream and the dependencies have changed, you can install them without re-installing Netdisco itself:

cd netdisco-ng/Netdisco
~/bin/localenv cpanm --installdeps .

Happy hacking!

Introduction

This release of Netdisco is built as a Dancer application, and uses many modern technologies and techniques. Hopefully this will make the code easier to manage and maintain in the long term.

Although Dancer is a web application framework, it provides very useful tools for command line applications as well, namely configuration file management and database connection management. We make use of these features in the daemon and deployment scripts.

Overall the application tries to be as self-contained as possible without also needing an excessive number of CPAN modules to be installed. However, Modern Perl techniques have made dependency management almost a non-issue, and Netdisco can be installed by and run completely within an unprivileged user's account, apart from the PostgreSQL database setup.

Finally the other core component of Netdisco is now a DBIx::Class layer for database access. This means there is no SQL anywhere in the code, but more important, we can re-use the same complex queries in different parts of Netdisco.

The rest of this document discusses each "interesting" area of the Netdisco codebase, hopefully in enough detail that you can get hacking yourself :-)

Versioning

This is Netdisco major version 2. The minor version has six digits, which are split into two components of three digits each. It's unlikely that the major version number (2) will increment. Each "significant" release to CPAN will increment the first three digits of the minor version. Each "trivial" release will increment the second three digits of the minor version.

Beta releases will have a a suffix with an underscore, to prevent CPAN indexing the distribution. Some examples:

2.002002     - "significant" release 2, second "trivial" release
2.002003     - a bug was found and fixed, hence "trivial" release 3
2.003000_001 - first beta for the next "significant" release
2.003000_002 - second beta
2.004000     - the next "significant" release

The words "significant" and "trivial" are entirely subjective, of course.

Global Configuration

Dancer uses YAML as its standard configuration file format, which is flexible enough for our needs, yet still simple to edit for the user. We no longer need a parser as in the old version of Netdisco.

At the top of scripts you'll usually see something like:

use App::Netdisco;
use Dancer ':script';

First, this uses App::Netdisco, which is almost nothing more than a placeholder module (contains no actual application code). What it does is set several environment variables in order to locate the configuration files.

Then, when we call "use Dancer" these environment variables are used to load two YAML files: config.yml and <environment>.yml where <environment> is typically either deployment or development.

The concept of "environments" allows us to have some shared "master" config between all instances of the application (config.yml), and then settings for specific circumstances. Typically this might be logging levels, for example. The default file which App::Netdisco loads is deployment.yml but you can override it by setting the "DANCER_ENVIRONMENT" environment variable.

The file is located in an environments folder which defaults to being in the user's home directory. The name (or full path) of the folder can be overriden using the "DANCER_ENVDIR" environment variable. The location of the folder alone can be overridden using the "NETDISCO_HOME" environment variable.

Dancer loads the config using YAML, merging data from the two files. Config is made available via Dancer's setting('foo') subroutine, which is exported. So now the foo setting in either config file is easily accessed.

Another line commonly seen in scripts is this:

use Dancer::Plugin::DBIC 'schema';

This plugin saves a lot of effort by taking some database connection parameters from the configuration file, and instantiating DBIx::Class database connections with them. The connections are managed transparently so all we need to do to access the Netdisco database, with no additional setup, is:

schema('netdisco')->resultset(...)->search({...});

DBIx::Class Layer

DBIx::Class, or DBIC for short, is an Object-Relational Mapper. This means it abstracts away the SQL of database calls, presenting a Perl object for each table, set of results from a query, table row, etc. The advantage is that it can generate really smart SQL queries, and these queries can be re-used throughout the application.

The DBIC layer for Netdisco is based at App::Netdisco::DB. This is the global schema class and below that, under App::Netdisco::DB::Result is a class for each table in the database. These contain metadata on the columns but also several handy "helper" queries which can be called. There are also ResultSet classes which provide additional "pre-canned" queries.

Netdisco's DBIx::Class layer has excellent documentation which you are encouraged to read, particularly if you find it difficult to sleep.

Results and ResultSets

In DBIC a Result is a table and a ResultSet is a set of rows retrieved from the table as a result of a query (which might be all the rows, of course). This is why we have two types of DBIC class. Items in the Result generally relate to the single table directly, and simply. In the ResultSet class are more complex search modifiers which might synthesize new "columns" of data (e.g. formatting a timestamp) or subroutines which accept parameters to customize the query.

However, regardless of the actual class name, you access them in the same way. For example the device table has an App::Netdisco::DB::Result::Device class and also an App::Netdisco::DB::ResultSet::Device class. DBIC merges the two:

schema('netdisco')->resultset('Device')->get_models;

Virtual Tables (VIEWs)

Where we want to simplify our application code even further we can either install a VIEW in PostgreSQL, or use DBIx::Class to synthesize the view on-the-fly. Put simply, it uses the VIEW definition as the basis of an SQL query, yet in the application we treat it as a real table like any other.

Some good examples are a fake table of only the active Nodes (as opposed to all nodes), or the more complex list of all ports which are connected together (DeviceLink).

All these tables live under the App::Netdisco::DB::Result::Virtual namespace, and so you access them like so (for the ActiveNode example):

schema('netdisco')->resultset('Virtual::ActiveNode')->count;

Versioning and Deployment

To manage the Netdisco schema in PostgreSQL we use DBIx::Class's deployment feature. This attaches a version to the schema and provides all the code to check the current version and do whatever is necessary to upgrade. The schema version is stored in a new table called dbix_class_schema_versions, although you should never touch it.

The netdisco-db-deploy script included in the distribution performs the following services:

* Installs the dbix_class_schema_versions table
* Upgrades the schema to the current distribtion's version

This works both on an empty, new database, and a legacy database from the existing Netdisco release, in a non-destructive way. For further information see DBIx::Class::Schema::Versioned and the netdisco-db-deploy script.

The files used for the upgrades are shipped with this distribution and stored in the .../App/Netdisco/DB/schema_versions directory. They are generated using the nd-dbic-versions script which also ships with the distribution.

Foreign Key Constraints

We have not deployed any FK constraints into the Netdisco schema. This is partly because the current poller inserts and deletes entries from the database in an order which would violate such constraints, but also because some of the archiving features of Netdisco might not be compatible anyway.

Web Application

The Netdisco web app is a "classic" Dancer app, using most of the bundled features which make development really easy. Dancer is based on Ruby's Sinatra framework. Its style is for many "helper" subroutines to be exported into the application namespace, to do things such as access request parameters, navigate around the "handler" subroutines, manage response headers, and so on.

Pretty much anything you want to do in a web application has been wrapped up by Dancer into a neat helper routine that does the heavy lifting. This includes configuration and database connection management, as was discussed above. Also, templates can be executed and Netdisco uses the venerable Template::Toolkit engine for this.

Like most web frameworks Dancer has a concept of "handlers" which are subroutines to which a specific web request is routed. For example if the user asks for "/device" with some parameters, the request ends up at the App::Netdisco::Web::Device package's "get '/device'" handler. All this is done automatically by Dancer according to some simple rules. There are also "wrapper" subroutines which we use to do tasks such as setting up data lookup tables, and handling authentication.

Dancer also supports AJAX very well, and it is used to retrieve most of the data in the Netdisco web application in a dynamic way, to respond to search queries and avoid lengthy page reloads. You will see the handlers for AJAX look similar to those for GET requests but do not use Template::Toolkit templates.

Compared to the current Netdisco, the handler routines are very small. This is because (a) they don't include any HTML - this is delegated to a template, and (b) they don't include an SQL - this is delegated to DBIx::Class. Small routines are more manageable, and easier to maintain. You'll also notice use of modules such as NetAddr::MAC and NetAddr::IP::Lite to simplify and make more robust the handling of data.

In fact, many sections of the web application have been factored out into separate Plugin modules. For more information see the App::Netdisco::Web::Plugin manual page.

Running the Web App

Dancer apps conform to the "PSGI" standard interface for web applications, which makes for easy deployment under many stacks such as Apache, FCGI, etc. See Dancer::Deployment for more detail.

At a minimum Netdisco can run from within its own user area as an unprivileged user, and actually ships with a fast, preforking web server engine. The netdisco-web script uses Daemon::Control to daemonize this simple web server so you can fire-and-forget the Netdisco web app without much trouble at all. This script in turn calls netdisco-web-fg which is the real Dancer application, that runs in the foreground if called on its own.

Authentication

Session and authentication code lives in App::Netdisco::Web::AuthN. It is fully backwards compatible with the existing Netdisco user management, making use of the database users and their MD5 passwords.

There is also support for unauthenticated access to the web app (for instance if you have some kind of external authentication, or simply trust everyone). See App::Netdisco::Manual::Configuration for further details.

Authorization

Every Dancer route handler must have proper role based access control enabled, to prevent unauthorized access to Netdisco's data, or admin features. This is done with the Dancer::Plugin::Auth::Extensible module. It handles both the authentication using Netdisco's database, and then protects each route handler. See App::Netdisco::Manual::WritingPlugins for details.

Templates

In the share/views folder of this distribution you'll find all the Template::Toolkit template files, with .tt extensions. Dancer first loads share/views/layouts/main.tt which is the main page wrapper, that has the HTML header and so on. It then loads other templates for sections of the page body. This is a typical Template::Toolkit "wrapper" configuration, as noted by the [% content %] call within main.tt that loads the template you actually specified in your Dancer handler.

All templates (and Javascript and Stylesheets) are shipped in the App::Netdisco distribution and located automatically by the application (using the environment variables which App::Netdisco set up). The user doesn't have to copy or install any files.

There's a template for the homepage called index.tt, then separate templates for searching, displaying device details, and showing inventory. These are, pretty much, all that Netdisco ever does.

Each of these pages is designed in a deliberately similar way, with re-used features. They each can have a "sidebar" with a search form (or additional search parameters). They also can have a tabbed interface for sub-topics.

Here's where it gets interesting. Up till now the page content has been your typical synchronous page load (a single page comprised of many templates) in response to a GET request. However the content of the tabs is not within this. Each tab has its content dynamically retrieved via an AJAX request back to the web application. Javscript triggers this automatically on page load.

This feature allows the user to search and search again, each time refreshing the data they see in the tab but without reloading the complete page with all its static furniture. AJAX can, of course, return any MIME type, not only JSON but also HTML content as in this case. The templates for the tabs are organised below share/views/ajax/... in the distribution.

Stylesheets

The main style for Netdisco uses Twitter Bootstrap, which is a modern library of CSS and javascript used on many websites. It does a lot of heavy lifting, providing simple CSS classes for all of the standard web page furniture (forms, tables, etc). Check out the documetation at the Twitter Bootstrap web site for more information.

These stylesheets are of course customised with our own netdisco.css. We try to name all CSS classes with a prefix "nd_" so as to be distinct from Twitter Bootstrap and any other active styles.

All stylesheets are located in the share/public/css folder of the distribution and, like the templates, are automatically located and served by the Netdisco application. You can also choose to serve this content statically via Apache/etc for high traffic sites.

Although Twitter Bootstrap ships with its own set of icons, we use an alternative library called Fontawesome. This plugs in easily to Bootstrap and provides a wider range of scaleable vectored icons which are easy to use.

Javascript

Of course many parts of the Netdisco site use Javascript, beginning with retrieving the page tab content itself. The standard library in use is jQuery, and the latest version is shipped with this distribution.

Many parts of the Netdisco site have small Javscript routines. The code for these, using jQuery as mentioned, lives in two places. The main netdisco.js file is loaded once in the page HTML header, and lives in share/public/javascripts/netdisco.js. There's also a netdisco_portcontrol.js which is included only if the current user has Port Control rights.

Netdisco also has Javascript routines specific to the device search or device details pages, and these files are located in share/views/js/... because they're loaded within the page body by the templates. These files contain a function inner_view_processing which is called each time AJAX delivers new content into a tab in the page (think of it like a callback, perhaps).

Job Daemon

The old Netdisco has a job control daemon which processes "port control" actions and also manual requests for device polling. The new Netdisco also has a daemon, although it is a true separate process and set of libraries from the web application. However, it still makes use of the Dancer configuration and database connection management features mentioned above.

The job daemon is backwards compatible with the old Netdisco database job requests table. All code for the job daemon lives under the App::Netdisco::Daemon namespace and like the rest of Netdisco is broken down into manageable chunks.

Running the Job Daemon

Like the web application, the job daemon is fully self contained and runs via two simple scripts shipped with the distribution - one for foreground and one for background execution (see the user docs for instructions).

The netdisco-daemon script uses Daemon::Control to daemonize so you can fire-and-forget the Netdisco job daemon without much trouble at all. This script in turn calls netdisco-daemon-fg which is the real application, that runs in the foreground if called on its own.

Daemon Engineering

The job daemon is based on the MCE library, which handles the forking and management of child processes doing the actual work. This actually runs in the foreground unless wrapped with Daemon::Control, as mentioned above. MCE handles four flavours of "worker" for different tasks.

One goal that we had designing the daemon was that sites should be able to run many instances on different servers, with different processing capacities. This is both to take advantage of more processor capability, but also to deal with security zones where you might only be able to manage a subset of devices from certain locations. Netdisco has always coped well with this via its discover_* and similar configuration, and the separate poller process.

So, the single Manager "worker" in the daemon is responsible for contacting the central Netdisco database and booking out jobs which it's able to service according to the local configuration settings. Jobs are "locked" in the central queue and then copied to a local job queue within the daemon.

There is support in the daemon for the workers to pick more than one job at a time from the local queue, in case we decide this is worth doing. However the Manager won't ever book out more jobs from the central Netdisco job queue than it has workers available (so as not to hog jobs for itself against other daemons on other servers). The user is free to configure the number of workers in their config.yml file (zero or more).

The fourth kind of worker is called the Scheduler and takes care of adding discover, macsuck, arpnip, and nbtstat jobs to the queue (which are in turn handled by the Poller worker). This worker is automatically started only if the user has enabled the "schedule" section of their deployment.yml site config.

SNMP::Info

The daemon obviously needs to use SNMP::Info for device control. All the code for this has been factored out into the App::Netdisco::Util namespace.

The App::Netdisco::Util::SNMP package provides for the creation of SNMP::Info objects along with connection tests. So far, SNMPv3 is not supported. To enable trace logging of the SNMP::Info object simply set the INFO_TRACE environment variable to a true value. The Connect library also provides routines to map interface and PoE IDs.

Configuration for SNMP::Info comes from the YAML files, of course. This means that our mibhome and mibdirs settings are now in YAML format. In particular, the mibdirs list is a real list within the configuration.

Other Noteable Technology

local::lib

This is the system used to install Netdisco and all its Perl dependencies into a folder independent of the system's Perl libraries. It means Netdisco can be self-contaned and at the same time relocated anywhere. The local::lib module is responsible for re-setting Perl's environment to point at the new library.

App::cpanminus

This is simply a sane replacement for the CPAN shell. Don't ever bother with the CPAN shell again, just use the cpanm client which comes with this distribution. We install Netdisco using cpanm.

App::local::lib::helper

This is a companion to local::lib which provides the localenv script you see referenced in the documentation. It's run automatically by Netdisco to locate its local::lib folder (that is, works around the bootstrapping problem where the shipped app doesn't know to where it is relocated). We can help things along by setting the NETDISCO_HOME environment variable.

Try::Tiny

A replacement for eval which provides proper try/catch semantics. You have to take a bit of care unfortunately over things like return statements though. However it's a lot cleaner than eval in many cases. See the documentation for further details.

Role::Tiny

Anyone familiar with the concept of an interface from other programming languages might understand what a role is. It's class functionality, often also called a "trait", which is composed into a class at run-time. This module allows the Daemon workers to dynamically assume different roles according to configuration.