Thursday 31 May 2012

Google Talk add users to roster via Perl

Since we started moving to Google Apps we're having to manually add all other users in our corporation to our Google Talk roster. This is a nuisance for our users especially at the beginning as there is a large number of users that need to be added.


Thanks to Perl and a few modules from CPAN it shouldn't be that hard to get around this issue. I've got a spreadsheet with usernames and passwords in the following format from our initial setup.


"username@domain.com","123password","John Smith"


The script below will add all of those email addresses to each others rosters.



#!/usr/bin/perl
#

use Text::CSV;
use Net::XMPP;
use Data::Dumper;
use warnings;

my $config;
$config->{'resource'} = 'perl';
$config->{'debug'} = 1;
$config->{'company name'} = "Company";

my @accounts;
my $file = $ARGV[0];
if ( !$file ) {
print
"Call this script as shown: add_xmpp_contacts.pl email_accounts.csv.\n";
print "The format of the csv script shall be as follows:\n";
print '"username1@domain.com","password1","John Doe"' . "\n";
print '"username2@domain.com","password2","John Smith"' . "\n";
exit(0);
}

my $csv = Text::CSV->new();
open( CSV, "<", $file ) or die $!;
while (<CSV>) {
if ( $csv->parse($_) ) {
my @columns = $csv->fields();
print "USERNAME: " . $columns[0] . "\n";
my $account->{username} = $columns[0];
$account->{display} = $columns[2];
push( @accounts, $account );
}
else {
my $err = $csv->error_input;
print "Failed to parse line: $err";
}
}
close CSV;

open( CSV, "<", $file ) or die $!;
while (<CSV>) {
if ( $csv->parse($_) ) {
my @columns = $csv->fields();
if ( $config->{'debug'} == 1 ) {
print "USERNAME: " . $columns[0] . "";
print "PASSWORD: " . $columns[1] . "\n";
}
else {
print "USERNAME: " . $columns[0] . "\n";
}
&dump_data( $columns[0], $columns[1], @accounts );
}
else {
my $err = $csv->error_input;
print "Failed to parse line: $err";
}
}
close CSV;

##############################################################################
# Subroutines, don't touch unless you know what you're doing.
##############################################################################

sub dump_data() {
my ( $email, $password, @accounts ) = @_;
my ( $username, $domain ) = split( /\@/, $email );
print "Connecting for " . $username . "\n";
my $Connection = new Net::XMPP::Client();

# Connect to talk.google.com
my $status = $Connection->Connect(
hostname => 'talk.google.com',
componentname => $domain,
port => 5222,
tls => 1
);

if ( !( defined($status) ) ) {
print "ERROR: XMPP connection failed.\n";
print " ($!)\n";
exit(0);
}
else {
print "Connected\n";
}

# Change hostname
my $sid = $Connection->{SESSION}->{id};
$Connection->{STREAM}->{SIDS}->{$sid}->{hostname} = 'gmail.com';

# Authenticate
$Connection->AuthSend(
username => $username,
password => $password,
resource => $config->{'resource'}
);

if ( $Connection->Connected() == 0 ) {
print "ERROR: Authorization failed: $res - $msg\n";
exit(0);
}
else {
print "Connected OK for " . $username . "\n";
}
$Connection->PresenceSend( show => 'available' );
my $Roster = $Connection->Roster();
my %userlist;
my %roster = $Connection->RosterGet();
print Dumper( \%roster ) if $config->{'debug'} == 1;
while ( my ( $id, $data ) = each %roster ) {
print "ID: "
. $id . " '"
. $data->{'name'}
. "' Groups: '"
. @{ $data->{groups} } . "'\n"
if $config->{'debug'} == 1;
$userlist->{$id} = 1;
}
print Dumper($userlist) if $config->{'debug'} == 1;
foreach my $contact (@accounts) {
print "Checking: "
. $contact->{display} . " "
. $contact->{username} . "\n";
if ( exists $userlist->{ $contact->{username} } ) {
print "Already Exists: " . $contact->{display} . "\n";
}
else {
print "Adding: " . $contact->{display} . "\n";
$Connection->Roster->add(
jid => $contact->{username},
name => $contact->{display},
groups => []
);
$Connection->Subscription(
type => "subscribe",
to => $contact->{username}
);
}
sleep 1;
}
}



Sharing Google Contact Groups

We have been working on migrating from MDaemon to Google Apps. One of the issues we've run is Google Apps Sync not syncing contact groups and shared contacts to Outlook. We've worked around this limitation in the web base version by using Ginza Contact Sharing.  For those users using Outlook we’re going to have them link to Sharepoint and have our Sharepoint contacts lists and our Google Apps contacts lists synchronized.  Source code will be posted when this has been completed.

Migration from MDaemon to Google Apps

We have been using MDaemon since 2007. Since the beginning we've had stability issues with it, especially with Comagent, that we've not been able to get resolved. This spring we decided it was time to move to a different platform.


Here are some of our requirements:



  1. Integrated instant messaging.

  2. Blackberry, Android, and iPhone support.

  3. Email for around 100 people.

  4. Shared contacts

  5. Must be able to move all of our existing email, contacts, calendars to new platform.

  6. Minimal/No downtime with migration.




After studying the options, especially Exchange and Google Apps we decided to move ahead with Google Apps. I did not find a seamless tool to migrate from MDaemon to Google Apps. However, after putting together a few different tools and some Perl scripting, most of the process has been automated. Here's the list of tools I used:



  1. Google Active Directory Sync (http://support.google.com/a/bin/answer.py?hl=en&answer=106368)

  2. ImapSync (http://imapsync.lamiral.info/) This one is well worth buying.

  3. GAM - Google Apps Manager (http://code.google.com/p/google-apps-manager/) - This tool is very useful if you need to make changes to all of your users.




Here's what happened:



  1. The signup and domain confirmation is painless as I'm used to it being.

  2. Time to start adding users. MDaemon lets you export a CSV will all your user details including passwords which is very handy to have around during the migration. Keep this, you'll need it later. I got all the users added using GADS as I want to keep my Google Apps accounts synchronized with our active directory. This tool is easy to use if you take your time getting it setup. For now we're not going to use the LDAP contacts functionality. We might yet but not for now. Unfortunately the import process did not leave me with passwords we could use. This was probably due to an error in the way I ran it but this is where GAM came in very handy.

  3. GAM takes a bit to get setup but their documentation will walk you through it. This script comes from the GAM website but it did exactly what I needed. I updated the MDaemon .csv export to only show usernames and passwords as that's all I need.



    1. Here's a sample from the email_accounts.csv file:


      username,password


      user123@domain.com,1234567


      user1234@domain.com,123313

    2. Here's the script:


      $list = Import-Csv '.\email_accounts.csv'


      foreach ($entry in $list)


      {


      .\gam.exe update user $($entry.username) password $($entry.password)


      }



  4. Once I ran that script all my users now had the same passwords in Google Apps as in MDaemon. That makes the rest of this process a lot easier.

  5. At this point it was time to turn "webclips" off. Once again GAM comes to the rescue: gam all users webclips off

  6. Time to start migrating email. There are a few gotchas with this process. The biggest one is that MDaemon uses a different layout than Google Apps does and you probably don't want any odd names in your folders. If your users do not have the same passwords in Google Apps as they do in MDaemon the following script will not work.

  7. Run it like this if you want to do it per user. Also, please be aware that it will take a long time to run if you have large amounts of email. imapsync.sh 'mdaemonimapip' 'username@domain.com' 'password'





    1. imapsync --host1 $1 \
      --user1 $2 --password1 $3 \
      --user2 $2 --password2 $3 \
      --port1 143 --authmech1 PLAIN --host2 imap.gmail.com \
      --authmech2 LOGIN --port2 993 --ssl2 \
      --exclude "sent-items|Shared Folders|Sent|Drafts|Infected Items|Junk E-mail|Outbox|Trash|Public Folders|Deleted Items|Sent Items" --prefix1 "INBOX/" \
      --skipsize \
      --nofoldersizes \
      --tmpdir /var/tmp \
      --useuid

      imapsync --host1 $1 \
      --user1 $2 --password1 $3 \
      --user2 $2 --password2 $3 \
      --port1 143 --authmech1 PLAIN --host2 imap.gmail.com \
      --authmech2 LOGIN --port2 993 --ssl2 \
      --folder "Sent Items" --prefix2 '[Gmail]/' --regextrans2 's/Sent\ Items$/Sent Mail/' \
      --folder "Sent" --prefix2 '[Gmail]/' --regextrans2 's/Sent$/Sent Mail/' \
      --folder "sent-items" --prefix2 '[Gmail]/' --regextrans2 's/sent-items$/Sent Mail/' \
      --folder "Drafts" --prefix2 '[Gmail]/' --regextrans2 's/Drafts$/Drafts/' \
      --skipsize \
      --nofoldersizes \
      --tmpdir /var/tmp \
      --useuid


    2. If you want to run that script on a csv file in the "email_address,password" format you can use the following script:



      1. #!/usr/bin/perl
        #

        use Text::CSV;
        use warnings;

        my $file = $ARGV[0];
        my $ip = $ARGV[1];
        if (!$file || !$ip) {
        print "Call this script as shown: imapsync_bulk.pl email_accounts.csv mdaemon_ip.\n";
        print "The format of the csv script shall be as follows:\n";
        print 'username1@domain.com,password1'."\n";
        print 'username2@domain.com,password2'."\n";
        exit(0);
        }
        my $csv = Text::CSV->new();
        open (CSV, "<", $file) or die $!;
        while (<CSV>) {
        if ($csv->parse($_)) {
        my @columns = $csv->fields();
        print "USERNAME: " . $columns[0] . " ";
        print "PASSWORD: " . $columns[1] . "\n";
        &dump_data($columns[0],$columns[1]);
        } else {
        my $err = $csv->error_input;
        print "Failed to parse line: $err";
        }
        }
        close CSV;


        ##############################################################################
        # Subroutines, don't touch unless you know what you're doing.
        ##############################################################################

        sub dump_data() {
        my ( $username, $password ) = @_;
        my $result = system("./imapsync.sh '" . $ip . "' '" . $username . "' '" . $password . "'");
        }







  8. Now comes the fun... Migrating individual contacts and calendars... There is a company that will do this for $15/mailbox but I figured I'd spend a little time at this myself first especially as this migration will probably take some time and who knows how many times I'll end up redoing it. MDaemon lets you export your contacts and calendars into a CSV file that Google Apps will accept for import. I do not have a script to import these files into Google Apps, so far I'm doing it manually but you will have the data in csv files. Please be aware that if you run this on a csv file containing all your users and passwords they will all get kicked out of Worldclient due to the 1 simultaneous login issues.



    1. Here's the script. It comes with no warranties or guarantees of any sort. This script is known to have issues with some entries containing quotation marks in the events themselves. We had few enough of these that I just manually fixed them as they came up. I also have not tested this script with files containing a list of accounts to process as I did these one at a time.





      1. #!/usr/bin/perl
        #

        use LWP::UserAgent;
        use URI::URL;
        use HTTP::Headers;
        use HTTP::Cookies;
        use Text::CSV;
        use Text::CSV_XS;
        use WWW::Google::Contacts;
        use Net::Google::Calendar;
        use Data::Dumper;
        use Date::Format;
        use warnings;

        ##############################################################################
        # Configure Variables;
        ##############################################################################
        # WARNING - WARNING - WARNING - WARNING
        # There are some very important settings here. Be sure to read before
        # using.
        ##############################################################################

        my ( $cookie_jar, $config, $hdrs, $ua, $url, $req, $resp );
        $config->{'WorldClient'} = "https://mail.com/WorldClient.dll";
        $config->{'time_zone'} = 'America/Edmonton';
        $config->{'delete_existing_contacts'} =
        0; #Set to 0 if you don't want all your existing contacts deleted!!!
        $config->{'print_contact_groups'} = 1;
        $config->{'delete_existing_calendar'} =
        1; #Set to 0 if you don't want all your existing calendar entries deleted!!!
        $config->{'use_proxy'} = 0; #Change to 1 if you want to use a proxy.
        $config->{'Proxy'} = 'http://192.168.1.1:8888/';

        # Check program parameters;
        my $file = $ARGV[0];
        if ( !$file ) {
        print "Call this script as shown: mdaemon_migrate.pl email_accounts.csv.\n";
        print "The format of the csv script shall be as follows:\n";
        print 'username1@domain.com,password1' . "\n";
        print 'username2@domain.com,password2' . "\n";
        exit(0);
        }

        # Prepare connections
        $ua = LWP::UserAgent->new();

        $ua->proxy( ['http'], $config->{'Proxy'} ) if $config->{'use_proxy'} == 1;

        my $csv = Text::CSV->new( { binary => 1, auto_diag => 1, } );
        open( CSV, "<", $file ) or die $!;

        while (<CSV>) {
        if ( $csv->parse($_) ) {
        my @columns = $csv->fields();
        print "USERNAME: " . $columns[0] . " ";
        print "PASSWORD: " . $columns[1] . "\n";
        &dump_data( $columns[0], $columns[1] );
        }
        else {
        my $err = $csv->error_input;
        print "Failed to parse line: $err";
        }
        }
        close CSV;

        ##############################################################################
        # Subroutines, don't touch unless you know what you're doing.
        ##############################################################################

        sub dump_data() {
        my ( $username, $password ) = @_;
        $hdrs = HTTP::Headers->new( 'User-Agent' => 'WorldClient Client' );
        $ua->cookie_jar( HTTP::Cookies->new() );
        $url = URI::URL->new( $config->{'WorldClient'} . '?View=Main' );
        $req = HTTP::Request->new( POST, $url, $hdrs );
        $resp = $ua->post( $url, [ User => $username, Password => $password ] );

        my ( $junk, $session_string1 ) = split( m/Session=/, $resp->content );
        my ( $session_id, $junk1 ) = split( m/'/, $session_string1 );
        print STDERR "SESSIONID: " . $session_id . "\n";

        $url =
        URI::URL->new( $config->{'WorldClient'}
        . '/calendar.csv?Session='
        . $session_id
        . '&View=CalendarExport&Filename=calendar.csv&FolderID=1' );
        $req = HTTP::Request->new( POST, $url, $hdrs );
        $resp = $ua->post(
        $url,
        [
        STARTTIMEMONTH => '1',
        STARTTIMEDAY => '1',
        STARTTIMEYEAR => '2002',
        STARTTIMEHOUR => '00',
        STARTIMEMINUTE => '00',
        ENDTIMEMONTH => '12',
        ENDTIMEDAY => '31',
        ENDTIMEYEAR => '2021',
        ENDTIMEHOUR => '23',
        ENDTIMEMINUTE => '55',
        EXPORT => 'Export'
        ]
        );
        my $calendar_csv = $resp->content;
        #Replace the embedded line breaks with <br>
        $calendar_csv =~ s/\r\r\n/<br>/g;
        #Replace embedded " with '
        $calendar_csv =~ s/(?<=( |\w))"(?=( |\w))/'/g;

        open( CALENDAR_CSV, '>' . $username . "-calendar.csv" );
        print CALENDAR_CSV $calendar_csv;
        close(CALENDAR_CSV);
        $cal = Net::Google::Calendar->new;
        $cal->login( $username, $password );
        #Get a list of all events in the calendar.
        for ($cal->get_events()) {
        print "EXISTING CALENDAR ENTRY: " . $_->title."\n";
        if ($config->{'delete_existing_calendar'} == 1) {
        $cal->delete_entry($_);
        print "DELETING CALENDAR ENTRY: " . $_->title."\n";
        }
        }
        # my @rows;
        # my $csv = Text::CSV_XS->new ({ binary => 1, eol => $/,auto_diag => 1 }) or
        # die "Cannot use CSV: ".Text::CSV_XS->error_diag ();
        # open my $fh, "<", $username . "-calendar.csv" or die $!;
        # while (my $row = $csv->getline ($fh)) {
        # print STDERR "NEW LINE: " . Dumper($row) . "END NEW LINE\n";
        # push @rows, $row;
        # }
        # $csv->eof or $csv->error_diag ();
        # close $fh;

        my $csv = Text::CSV->new( { binary => 1, auto_diag => 1, } );
        open( CSV, "<", $username . "-calendar.csv" ) or die $!;
        while (<CSV>) {
        # print STDERR "NEW LINE: " . $_ . "END NEW LINE\n";
        if ( $csv->parse($_) ) {
        my @columns = $csv->fields();
        if ($columns[0] ne "Subject") {
        print "Adding Calendar Event: " . $columns[0] . "\n";
        my $entry = Net::Google::Calendar::Entry->new();
        $entry->title( $columns[0] );
        $entry->content( $columns[12] );
        $entry->location( $columns[14] );
        $entry->transparency('transparent');
        $entry->status('confirmed');
        if ($columns[15] eq "TRUE") {
        $entry->visibility('private');
        }
        if ($columns[6] eq "TRUE") {
        my ($alert_month, $alert_day, $alert_year) = split(m!/!, $columns[7]);
        my ($alert_hour, $alert_minute, $alert_second) = split(m/:/, $columns[8]);
        my $alert = DateTime->new(
        year => $alert_year,
        month => $alert_month,
        day => $alert_day,
        hour => $alert_hour,
        minute => $alert_minute,
        second => $alert_second,
        nanosecond => 0,
        time_zone=> $config->{'time_zone'}
        );
        $entry->reminder('alert','absoluteTime',$alert);
        }
        my ($start_month, $start_day, $start_year) = split(m!/!, $columns[1]);
        my ($start_hour, $start_minute, $start_second) = split(m/:/, $columns[2]);
        my ($end_month, $end_day, $end_year) = split(m!/!, $columns[3]);
        my ($end_hour, $end_minute, $end_second) = split(m/:/, $columns[4]);
        my $allday = 0;
        if ($columns[2] eq "0:00:00" && $columns[4] eq "23:59:00") {
        print "ALL DAY EVENT\n";
        $end_hour = "00";
        $end_minute = "00";
        $end_second = "00";
        $allday = 1;
        }
        my $start = DateTime->new(
        year => $start_year,
        month => $start_month,
        day => $start_day,
        hour => $start_hour,
        minute => $start_minute,
        second => $start_second,
        nanosecond => 0,
        time_zone=> $config->{'time_zone'}
        );
        my $end = DateTime->new(
        year => $end_year,
        month => $end_month,
        day => $end_day,
        hour => $end_hour,
        minute => $end_minute,
        second => $end_second,
        nanosecond => 0,
        time_zone=> $config->{'time_zone'}
        );
        $entry->when($start,$end, $allday);
        $cal->add_entry($entry);
        }
        }
        else {
        my $err = $csv->error_input;
        print "Failed to parse line: $err";
        }
        }
        close CSV;

        ############################################################
        # Start processing Contacts
        ############################################################
        $url =
        URI::URL->new( $config->{'WorldClient'}
        . '/contacts.csv?Session='
        . $session_id
        . '&View=ContactExport&Filename=contacts.csv&FolderID=2' );
        $req = HTTP::Request->new( POST, $url, $hdrs );
        $resp = $ua->post( $url, [ EXPORT => 'Export' ] );
        my $contacts_csv = $resp->content;
        $contacts_csv =~ s/\r\r\n/<br>/g;
        open( CONTACTS_CSV, '>' . $username . "-contacts.csv" );
        print CONTACTS_CSV $contacts_csv;
        close(CONTACTS_CSV);
        my $csv = Text::CSV->new( { binary => 1, auto_diag => 1 } );
        open( CSV, "<", $username . "-contacts.csv" ) or die $!;
        my $google = WWW::Google::Contacts->new(
        username => $username,
        password => $password
        );

        if ( $config->{'print_contact_groups'} == 1 ) {
        my $groups = $google->groups;
        while ( my $gr = $groups->next ) {
        print "You got a contact group called " . $gr->title . "\n";
        }
        }
        if ( $config->{'delete_existing_contacts'} == 1 ) {
        my @contacts = $google->contacts->search();
        foreach my $c (@contacts) {
        print STDERR "Deleting Contact: " . $c->full_name . "\n";
        $c->delete;
        }
        }

        while (<CSV>) {
        if ( $csv->parse($_) ) {
        my @columns = $csv->fields();
        my $contact = $google->new_contact;
        $contact->given_name( $columns[1] );
        $contact->additional_name( $columns[2] );
        $contact->family_name( $columns[3] );
        $contact->name_prefix( $columns[0] );
        $contact->name_suffix( $columns[4] );
        $contact->full_name( $columns[59] );
        $contact->add_email( { type => "work", value => $columns[57] } )
        if $columns[57] ne "";
        $contact->add_email( { type => "other", value => $columns[60] } )
        if $columns[60] ne "";
        $contact->phone_number( { type => "work", value => $columns[31] } )
        if $columns[31] ne "";
        $contact->phone_number(
        { type => "work 2", value => $columns[32] } )
        if $columns[32] ne "";
        $contact->phone_number(
        { type => "work fax", value => $columns[30] } )
        if $columns[30] ne "";
        $contact->phone_number( { type => "home", value => $columns[37] } )
        if $columns[37] ne "";
        $contact->phone_number(
        { type => "mobile", value => $columns[40] } )
        if $columns[40] ne "";
        $contact->phone_number( { type => "other", value => $columns[42] } )
        if $columns[42] ne "";
        $contact->im();
        $contact->organization(
        {
        name => $columns[5],
        title => $columns[7],
        department => $columns[6]
        }
        );
        $contact->postal_address(
        [
        {
        type => "work",
        street => $columns[8],
        street2 => $columns[9],
        street3 => $columns[10],
        city => $columns[11],
        region => $columns[12],
        postcode => $columns[13],
        country => $columns[14]
        },
        {
        type => "home",
        street => $columns[15],
        street2 => $columns[16],
        street3 => $columns[17],
        city => $columns[18],
        region => $columns[19],
        postcode => $columns[20],
        country => $columns[21]
        },
        {
        type => "other",
        street => $columns[22],
        street2 => $columns[23],
        street3 => $columns[24],
        city => $columns[25],
        region => $columns[26],
        postcode => $columns[27],
        country => $columns[28]
        }
        ]
        );
        $contact->billing_information();
        $contact->birthday();
        $contact->hobby();
        $contact->add_group_membership("System Group: My Contacts");
        $contact->create;
        }
        else {
        my $err = $csv->error_input;
        print "Failed to parse line: $err";

        }
        }
        close CSV;
        }