host_history.pl 0.0.1 released

Discussion in 'Host History' started by Ian.H, Jun 19, 2004.

  1. Ian.H

    Ian.H Administrator Staff Member

    Hi all..

    I've hacked up a small Perl script to generate some activity graphs for my server. Here's an example the 2 graphs it produces:

    Small:
    [​IMG]

    Large:
    [​IMG]

    I wrote some scripts to read these graphs so I could display them on my server but I happen to run Apache on the same box as my LFS server runs so I have that "advantage". For others, you would probably have to implement some kind of automagic FTP upload or similar.

    The script runs standalone.. but designed to run from a cronjob every minute (the processing time is very short on my 350Mhz PII (maybe 3 seconds) from start to finish).

    Please read the comments in the script for how to configure the small section at the top and the pre-req's. Also please note that I haven't enough free time to explain how to install each of the 3 pre-reqs if you don't have them already. The process isn't difficult, but please see the relevant sites / accompanying README files for those instructions =)

    This could probably be coded better.. but it was a quick 10 min jobby =)


    The script:
    Code:
    #!/usr/bin/perl
    ###########################################################################
    #            host_history.pl
    #
    #  15/06/2004 14:18
    #
    #  Copyright (c) 2004,  Ian.H <ian.h@dsrc.digiserv.net>
    #  Web    : <http://digiserv.net/>
    #         : <http://dsrc.digiserv.net/>
    #
    # License : BSD
    ###########################################################################
    #
    # A simple script that will use the host{port}.txt file to generate an
    # overall log of your server activity and then generate 2 graphs (small
    # and large ) representing 24 hours usage for drivers count.
    #
    ###########################################################################
    #
    # NOTE: This was coded to run on a *nix platform but it should also run
    #       on a windoze box.
    #
    ###########################################################################
    #
    # PRE-REQUISITES:
    # ---------------
    #
    # Perl5 ( <http://www.perl.com/> | <http://www.activestate.com/> )
    # GD1 or GD2 libs ( <http://www.boutell.com/gd/> )
    # tail ( windoze version at: <http://unxutils.sourceforge.net/> )
    #
    #
    # Please DO NOT ask how to install these if you do not currently have
    # them, see their accompanying README files for installation information.
    #
    ###########################################################################
    #
    # CONFIGURATION:
    # --------------
    #
    # Change the details below in the 'my %CONF = (...)' section to match
    # your requirements / filesystem. PLEASE NOTE: the *_file values MUST BE
    # fully-qualified paths / filename. The output_file and hourly_log_file
    # will be created upon execution. The host_operator should be the name of
    # your server connection that displays in the 'connections list' in LFS.
    #
    # The graph entries are the fully-qualified path / filename of where the
    # graphs should be created
    #
    # The tail entry should be a fully-qualified path/ filename to your
    # tail or tail.exe binary
    #
    ###########################################################################
    
    
    use strict;
    use warnings;
    use GD;
    
    my %CONF = (
    
        host_file               => '/path/to/LFS/host63392.txt',
        output_file             => '/path/to/host_history.log',
        hourly_log_file         => '/path/to/host_history_hour.log',
        host_operator           => 'B.O.F.H',
        large_graph_output_file => '/path/to/driver_history_large.png',
        small_graph_output_file => '/path/to/driver_history_small.png',
        tail                    => '/path/to/tail(.exe)',
        server_name             => 'Your server name'
    );
    
    
    my @drivers = ();
    my $track = '';
    my $cars = '';
    
    my $time = time;
    
    open(HOST, "<$CONF{host_file}");
        while (<HOST>) {
            if (/^trackcfg=(.+)$/s) {
                ($track = $1);
            }
    
            if (/^cars=(.+)$/s) {
                ($cars = $1);
            }
    
            push @drivers, /^conn=(.+)$/s;
        }
    
        $track =~ s/[\n|\r|[\r\n]$//g;
        $cars =~ s/[\n|\r|[\r\n]$//g;
    close(HOST);
    
    open(LOG, ">>$CONF{output_file}");
        print LOG "$time|$track|$cars|";
        foreach (@drivers) {
            s/[\n|\r|[\r\n]$//g;
            print LOG "$_\x00" unless $_ eq $CONF{host_operator} or $_ eq '';
        }
        print LOG "\n";
    close(LOG);
    
    system("$CONF{tail} -n 1440 $CONF{output_file} > $CONF{hourly_log_file}");
    
    
    create_small_driver_history_graph();
    create_large_driver_history_graph();
    
    
    
    sub create_small_driver_history_graph {
        my @data = ();
        my @hourly_log_contents = ();
    
        open(LOG, "<$CONF{hourly_log_file}");
            @hourly_log_contents = <LOG>;
        close(LOG);
    
        foreach (@hourly_log_contents) {
            /^[0-9]+\|[A-Z1-9]+\|[A-Z46]+\|(.*)$/s;
            my $driver_list = $1;
            my @drivers = split("\x00", $driver_list);
            pop @drivers;
            push @data, scalar @drivers;
        }
    
        my $width = 1440;
        my $height = 80;
        my $out_width = 170;
        my $out_height = 95;
        my $graph = new GD::Image($width, $height);
        my $out_graph = new GD::Image($out_width, $out_height);
    
        my $white = $graph->colorAllocate(255, 255, 255);
        my $black = $graph->colorAllocate(0, 0, 0);
        my $light_blue = $graph->colorAllocate(134, 155, 191);
        my $background = $graph->colorAllocate(214, 224, 235);
        my $int_lines = $graph->colorAllocate(184, 194, 205);
        my $int_lines_light = $graph->colorAllocate(204, 214, 225);
    
        my $out_white = $out_graph->colorAllocate(255, 255, 255);
        my $out_black = $out_graph->colorAllocate(0, 0, 0);
        $out_graph->filledRectangle(0, 0, $width, $height, $out_white);
    
        $out_graph->string(gdTinyFont, 162, 60, '2', $out_black);
        $out_graph->string(gdTinyFont, 162, 40, '4', $out_black);
        $out_graph->string(gdTinyFont, 162, 20, '6', $out_black);
        $out_graph->string(gdTinyFont, 162, 1, '8', $out_black);
        $out_graph->string(gdTinyFont, 128, 82, get_time() . '>', $out_black);
        $out_graph->string(gdTinyFont, 1, 82, $CONF{server_name}, $out_black);
    
        $graph->filledRectangle(0, 0, $width, $height, $background);
    
        $graph->line(0, 70, $width, 70, $int_lines_light);
        $graph->line(0, 60, $width, 60, $int_lines);
        $graph->line(0, 50, $width, 50, $int_lines_light);
        $graph->line(0, 40, $width, 40, $int_lines);
        $graph->line(0, 30, $width, 30, $int_lines_light);
        $graph->line(0, 20, $width, 20, $int_lines);
        $graph->line(0, 10, $width, 10, $int_lines_light);
    
        my $i = 0;
        foreach (@data) {
            $graph->line($i, $height - ($_ * 10), $i, $height, $light_blue);
            $i++;
        }
    
        $out_graph->copyResized($graph, 0, 0, 0, 0, 160, 80, $width, $height);
        $out_graph->setAntiAliased($light_blue);
    
        $out_graph->interlaced('true');
    
        open(GRAPH, ">$CONF{small_graph_output_file}");
            binmode GRAPH;
            print GRAPH $out_graph->png;
        close(GRAPH);
    
        undef $graph;
        undef $out_graph;
    }
    
    
    sub create_large_driver_history_graph {
        my @data = ();
        my @hourly_log_contents = ();
    
        open(LOG, "<$CONF{hourly_log_file}");
            @hourly_log_contents = <LOG>;
        close(LOG);
    
        foreach (@hourly_log_contents) {
            /^[0-9]+\|[A-Z1-9]+\|[A-Z46]+\|(.*)$/s;
            my $driver_list = $1;
            my @drivers = split("\x00", $driver_list);
            pop @drivers;
            push @data, scalar @drivers;
        }
    
        my $width = 1440;
        my $height = 160;
        my $out_width = 350;
        my $out_height = 200;
        my $graph = new GD::Image($width, $height);
        my $out_graph = new GD::Image($out_width, $out_height);
    
        my $white = $graph->colorAllocate(255, 255, 255);
        my $black = $graph->colorAllocate(0, 0, 0);
        my $light_blue = $graph->colorAllocate(134, 155, 191);
        my $background = $graph->colorAllocate(214, 224, 235);
        my $int_lines = $graph->colorAllocate(184, 194, 205);
        my $int_lines_light = $graph->colorAllocate(204, 214, 225);
    
        my $out_white = $out_graph->colorAllocate(255, 255, 255);
        my $out_black = $out_graph->colorAllocate(0, 0, 0);
        $out_graph->filledRectangle(0, 0, $width, $height, $out_white);
    
        $out_graph->string(gdSmallFont, 1, 5, 'Server activity over the past 24 hours', $out_black);
    
        $out_graph->string(gdSmallFont, 342, 140, '2', $out_black);
        $out_graph->string(gdSmallFont, 342, 100, '4', $out_black);
        $out_graph->string(gdSmallFont, 342, 60, '6', $out_black);
        $out_graph->string(gdSmallFont, 342, 20, '8', $out_black);
        $out_graph->string(gdSmallFont, 300, 182, get_time() . '>', $out_black);
        $out_graph->string(gdTinyFont, 1, 82, $CONF{server_name}, $out_black);
    
        $graph->filledRectangle(0, 0, $width, $height, $background);
    
        $graph->line(0, 140, $width, 140, $int_lines_light);
        $graph->line(0, 120, $width, 120, $int_lines);
        $graph->line(0, 100, $width, 100, $int_lines_light);
        $graph->line(0, 80, $width, 80, $int_lines);
        $graph->line(0, 60, $width, 60, $int_lines_light);
        $graph->line(0, 40, $width, 40, $int_lines);
        $graph->line(0, 20, $width, 20, $int_lines_light);
    
        my $i = 0;
        foreach (@data) {
            $graph->line($i, $height - ($_ * 20), $i, $height, $light_blue);
            $i++;
        }
    
        $out_graph->copyResized($graph, 0, 20, 0, 0, 340, 160, $width, $height);
        $out_graph->setAntiAliased($light_blue);
    
        $out_graph->interlaced('true');
    
        open(GRAPH, ">$CONF{large_graph_output_file}");
            binmode GRAPH;
            print GRAPH $out_graph->png;
        close(GRAPH);
    
        undef $graph;
        undef $out_graph;
    }
    
    
    sub get_time {
    	my ($sec, $min, $hour, $mday, $mon, $year) = localtime(time);
    	return sprintf("%02d:%02d", $hour, $min);
    }
    Downloadable version attached.

    Hope someone else finds a use for this too =)



    Regards,

    Ian
     

    Attached Files:

    Last edited: Jun 20, 2004
  2. Bladerunner

    Bladerunner [dSRC] Member

    Re: Server activity (code)

    Like the idea for the new add-on...are you planning on incorporating it into the main site, and is there a script I need to run so you can get info from my server too?
     
  3. Ian.H

    Ian.H Administrator Staff Member

    Re: Server activity (code)

    Yup and yup =D

    The top connected drivers is currently only on the dev site as I have a few other changes present on that version too, but they're not 100% complete.. so rather than me upload a few files each time, I'll finish off the few additions and upload the new version complete =)

    As for your stats, to get both additions you'll need to run both of the above scripts. The host_history one generates the hourly log for the graphs.. that and the driver_visits script work off the _main_ log produced by host_history.

    You will need to make sure you have a few things installed though...

    Perl (grab a windoze copy free from the activestate.com site in the code comments). It's a windoze installer.. easy to install and takes a few mins.. make sure you associate .pl wiles with Perl when asked as this'll save you hassles later with the shebang line etc.

    You've probably got the GD2 libs installed from your PHP install (IIRC, they're compatible) and I you'll need to install the Perl GD module via activestate's Perl Package Manager (if you get everything installed, I'll go through the Perl module stuff with ya if needed.. it's _very_ simple.. but if ya never used it before......).

    I'll drop you a mail a little later with a copy of tail.exe to save downloading the entire tools collection.

    Once that part is done, I have another PHP script that parses the log files loke the read_host file does for the LFS host file. I'll sort those out for you too.

    driver_visits should then run once the above is sorted without any extra requirements for installing.. although there is an additional PHP script to parse that log file too.

    I have an updated version of the driver_visits script I need to post.. which has another 2 options (number of drivers to output and whether to output with or without the ^# colour codes (yes ideally for Web displaying, no if you were just displaying it as plaintext)). The one above works fine if you get if you run it.. but new version is a bit better =)

    Grab the installs and we'll install the Perl module and see what we can get =D



    Regards,

    Ian
     
  4. Bladerunner

    Bladerunner [dSRC] Member

    Re: Server activity (code)

    Ian..the 'host_history.pl' zip *seems* to be corrupted...cant unzip it!..showing as 2.2kb, shows file inside zip..but wont extract :(

    (gonna try and copy the code from the box above, and save it as....see if it werks! :) )
     
  5. Ian.H

    Ian.H Administrator Staff Member

    Re: Server activity (code)

    :eyepop: didn't extract for me either once I'd downloaded it.

    Re-archived and uploaded and this one extracts for me now... thanks for pointing this out =)

    I've attached a copy of tail.exe here too. Personally I think your $WINDOZE dir is the best dir to extract it to.. it's a great tool for monitoring the likes of Apache logfiles "real-time" too =)



    Regards,

    Ian
     

    Attached Files:

    • tail.zip
      File size:
      16.5 KB
      Views:
      315
  6. franky500

    franky500 New Member

    Hey Ian.

    Not sure if you would still be reading this area but meh.

    I have grabbed the Host_History.pl 0.0.2 release that you done a long time ago,

    I have installed Perl etc (as far as im awear.. correctly).

    When i run the hosthistory.pl file i get the HostHist.log, hourly.log and the 2 graphs created, The Log files both contain entries as expected, however the graph simply puts up the graph without any line data on it. It shows numbers, Time and server name on the axis, but just does not actually draw a line of the actual number of racers.

    The server at the time had 30 People on it and i ran the script for 5 minutes as a test.

    I know this is all very old but its still exactly what i need so if you have any ideas that would be appreciated.

    Regards,
    Dean
     
  7. Ian.H

    Ian.H Administrator Staff Member

    Hi Dean..

    Could you email me a couple of the 'host#.txt' files (preferably with some drivers connected etc)? I don't have an LFS server to test this myself, but if you can send something over to me, I'll run the script agianst that and see if I can find out why it's not working and fix it if I can :)

    There maybe a change in the host#.txt file format since I originally coded it causing the script to fail.. but I'll have a gander.

    ian.h@dsrc.digiserv.net should reach me :)



    Regards,

    Ian
     
  8. franky500

    franky500 New Member

    Cheers Ian,

    Just e-mailed some to you :)

    Regards,
    Dean
     
  9. Ian.H

    Ian.H Administrator Staff Member

    Hi Dean..


    Thanks for that. All received.

    Got a meeting with a client in a bit, but will have a look at this once I'm back home :thumb3d:



    Regards,

    Ian
     
  10. franky500

    franky500 New Member

    No Worries :)

    Enjoy the meeting :)

    Regards,
    Dean
     
  11. Ian.H

    Ian.H Administrator Staff Member

    Hi Dean..

    Meeting went well... got another contract signed :D

    Anyways... worked a bit on the script and have found a much more efficient way of producing them now (the script here used to get the list of driver names, split them into an array, count the number of array elements to get the number of drivers and then use that info to plot the graph. It seems that now, a new 'guests=#' line has been added to the host#.txt file meaning I can just grab that number) :)

    I have this working for both graphs now and just need to finish off the small graph. I had to make adjustments as the current script would only display up to 8 drivers (IIRC, this was the max I had my server set to way back in the day). It now has 0-50 as the y-axis as I believe the grid sizes now can be 48?

    Once I have the scaling sorted for the small graph sorted, I'll send you over a copy to test (should be done tomorrow as today's a bit busy) :)



    Regards,

    Ian
     
  12. franky500

    franky500 New Member

    Fantastic stuff Ian.

    Just got back from a family BBQ.

    Quick question whilst im here. Is there a way for it to pick up the server name at the same time ( i believe that is also in the file). Rather than having to set it in the .pl file?.

    Regards,
    Dean
     
  13. franky500

    franky500 New Member

    Just re read your post there Ian.

    And just FYI, i'm not planning on using the small graph (not that im aware of anyway!) So if it makes it easier for you to ignore that section, please do so :)

    Regards,
    Dean
     
  14. Ian.H

    Ian.H Administrator Staff Member

    Hi Dean..

    Apologies for the delay.

    I'd already changed the code to pick up the server name from the host file :D

    Just sent you an email with an updated version. Let me know how it goes. If all's well, I'll re-publish the code on clue-by-4.org :)



    Cheers..

    Ian
     
  15. franky500

    franky500 New Member

    Hi Ian,

    All received and sounds fantastic. Still got that 1 issue though.

    Currently populated server, updating every minute at the moment:

    http://81.19.209.205/History/serverhistory_37445/driver_history_large.png

    Still not drawing the line's like the old one did in the day. This could easily be a config problem with the perl, but i have never used perl before on windows so if you have any pointers (if you think it might be that) that would be appreciated.

    Using the command line:

    Code:
    hosthist.pl -hostfile="C:\path\to\folder\host37445.txt" -log-dir="C:\path\to\folder\log_37445" -chart-dir="C:\path\to\folder\serverhistory_37445"
     
    Last edited: Jun 19, 2010
  16. Ian.H

    Ian.H Administrator Staff Member

    Hmm.. the cmdline data looks fine.

    Can you check the 2 log files Dean to see if there's anything in them?

    I've just run it again withe files you sent to me and although it takes 5 mins to initially see anything drawn in the graph (this is due to the graph being drawn fullsize (1440px wide) and then shrunk to an acceptable size for output), it "works for me™".

    If there is data in the log files, can you email them over to me please? I'll force some tests to read those without editing them and see what I can generate here. I don't think it's a Perl config issue as the main arse-ache to getting this all working is the GD library.. and you have that sorted else it wouldn't create any image at all :)



    Cheers..

    Ian
     
  17. franky500

    franky500 New Member

    1 log has something in it, but the hourly_log does not.


    http://81.19.209.205/History/serverhistory_37445/log_37445.zip

    Seems like a strange one then if its not the Perl / GD Setup!.

    Appreciate the help Ian.

    Regards,
    Dean

    P.S i should actually note that with that command line the script still runs and creates everything (obviously) but does come up with a standard windows "this is not a valid executable or program" rubbish it kicks out.
     
  18. Ian.H

    Ian.H Administrator Staff Member

    Ahah.. that kinda helps (to narrow things down at least).

    Do you have the 'tail.exe' file Dean?

    Seeing as the global log file is being generated, the script is parsing the host#.txt file properly.. but after that, it calls the tail.exe file to generate the hourly log file. This part seems to be the bit that it's falling over on. If you do have the tail.exe file, have you edited the host_history.pl file %CONF() hash to point to the full path to tail.exe? (I suggest dropping tail into your 'C:\Windows' dir and editing the script to make tail.exe point to 'C:\Windows\tail.exe'.



    Regards,

    Ian
     
  19. franky500

    franky500 New Member

    I am such an idiot!!.

    Yes i did have tail.exe but it was in a big windows utilities package.. which was in Program files.. which of course meant there were a good few spaces in the path!!!.

    Just grabbed the tail.exe from the previous page and chucked that in c:\windows as suggested and that has instantly solved it!! Nice one.

    Now that its working i have only one question, (although perhaps after the initial 24 hours it will be as expected)

    The graph looks like it is drawing from the left, should it not draw from the right (current time)?

    Again, Night one and thanks Ian.

    Regards,
    Dean
     
  20. Ian.H

    Ian.H Administrator Staff Member

    Glad that it was an easy fix. Actually, glad you had the issue in the first place as it gave me a chance to rewrite some of the code in a much better way and to make it more configurable :D

    You're right, it does write the data from the left to right. It's right (and will look as epxetced once 24 hours has passed). Basically, the number of drivers gets fed into an array.. that's then looped through and the lines are drawn, so once the hourly log file has a complete hours worth of entries.. the "now" data will always be the last entry of the file, and will draw the line against the right edge.

    I think I did this as the "easy way out", heh.. but I'll have a look to see what's involved in reversing the drawing routine (keeping the current time against the right edge, but making the empty space until full on the left instead).



    Cheers..

    Ian
     

Share This Page