We have been using MDaemon since 2007. Since the beginning we've had stability issues with it, especially with Comagent, that we've not been able to get resolved. This spring we decided it was time to move to a different platform.
Here are some of our requirements:
- Integrated instant messaging.
- Blackberry, Android, and iPhone support.
- Email for around 100 people.
- Shared contacts
- Must be able to move all of our existing email, contacts, calendars to new platform.
- Minimal/No downtime with migration.
After studying the options, especially Exchange and Google Apps we decided to move ahead with Google Apps. I did not find a seamless tool to migrate from MDaemon to Google Apps. However, after putting together a few different tools and some Perl scripting, most of the process has been automated. Here's the list of tools I used:
- Google Active Directory Sync (http://support.google.com/a/bin/answer.py?hl=en&answer=106368)
- ImapSync (http://imapsync.lamiral.info/) This one is well worth buying.
- GAM - Google Apps Manager (http://code.google.com/p/google-apps-manager/) - This tool is very useful if you need to make changes to all of your users.
Here's what happened:
- The signup and domain confirmation is painless as I'm used to it being.
- Time to start adding users. MDaemon lets you export a CSV will all your user details including passwords which is very handy to have around during the migration. Keep this, you'll need it later. I got all the users added using GADS as I want to keep my Google Apps accounts synchronized with our active directory. This tool is easy to use if you take your time getting it setup. For now we're not going to use the LDAP contacts functionality. We might yet but not for now. Unfortunately the import process did not leave me with passwords we could use. This was probably due to an error in the way I ran it but this is where GAM came in very handy.
- GAM takes a bit to get setup but their documentation will walk you through it. This script comes from the GAM website but it did exactly what I needed. I updated the MDaemon .csv export to only show usernames and passwords as that's all I need.
- Here's a sample from the email_accounts.csv file:
username,password
user123@domain.com,1234567
user1234@domain.com,123313 - Here's the script:
$list = Import-Csv '.\email_accounts.csv'
foreach ($entry in $list)
{
.\gam.exe update user $($entry.username) password $($entry.password)
}
- Here's a sample from the email_accounts.csv file:
- Once I ran that script all my users now had the same passwords in Google Apps as in MDaemon. That makes the rest of this process a lot easier.
- At this point it was time to turn "webclips" off. Once again GAM comes to the rescue: gam all users webclips off
- Time to start migrating email. There are a few gotchas with this process. The biggest one is that MDaemon uses a different layout than Google Apps does and you probably don't want any odd names in your folders. If your users do not have the same passwords in Google Apps as they do in MDaemon the following script will not work.
- Run it like this if you want to do it per user. Also, please be aware that it will take a long time to run if you have large amounts of email. imapsync.sh 'mdaemonimapip' 'username@domain.com' 'password'
imapsync --host1 $1 \
--user1 $2 --password1 $3 \
--user2 $2 --password2 $3 \
--port1 143 --authmech1 PLAIN --host2 imap.gmail.com \
--authmech2 LOGIN --port2 993 --ssl2 \
--exclude "sent-items|Shared Folders|Sent|Drafts|Infected Items|Junk E-mail|Outbox|Trash|Public Folders|Deleted Items|Sent Items" --prefix1 "INBOX/" \
--skipsize \
--nofoldersizes \
--tmpdir /var/tmp \
--useuid
imapsync --host1 $1 \
--user1 $2 --password1 $3 \
--user2 $2 --password2 $3 \
--port1 143 --authmech1 PLAIN --host2 imap.gmail.com \
--authmech2 LOGIN --port2 993 --ssl2 \
--folder "Sent Items" --prefix2 '[Gmail]/' --regextrans2 's/Sent\ Items$/Sent Mail/' \
--folder "Sent" --prefix2 '[Gmail]/' --regextrans2 's/Sent$/Sent Mail/' \
--folder "sent-items" --prefix2 '[Gmail]/' --regextrans2 's/sent-items$/Sent Mail/' \
--folder "Drafts" --prefix2 '[Gmail]/' --regextrans2 's/Drafts$/Drafts/' \
--skipsize \
--nofoldersizes \
--tmpdir /var/tmp \
--useuid- If you want to run that script on a csv file in the "email_address,password" format you can use the following script:
#!/usr/bin/perl
#
use Text::CSV;
use warnings;
my $file = $ARGV[0];
my $ip = $ARGV[1];
if (!$file || !$ip) {
print "Call this script as shown: imapsync_bulk.pl email_accounts.csv mdaemon_ip.\n";
print "The format of the csv script shall be as follows:\n";
print 'username1@domain.com,password1'."\n";
print 'username2@domain.com,password2'."\n";
exit(0);
}
my $csv = Text::CSV->new();
open (CSV, "<", $file) or die $!;
while (<CSV>) {
if ($csv->parse($_)) {
my @columns = $csv->fields();
print "USERNAME: " . $columns[0] . " ";
print "PASSWORD: " . $columns[1] . "\n";
&dump_data($columns[0],$columns[1]);
} else {
my $err = $csv->error_input;
print "Failed to parse line: $err";
}
}
close CSV;
##############################################################################
# Subroutines, don't touch unless you know what you're doing.
##############################################################################
sub dump_data() {
my ( $username, $password ) = @_;
my $result = system("./imapsync.sh '" . $ip . "' '" . $username . "' '" . $password . "'");
}
- Now comes the fun... Migrating individual contacts and calendars... There is a company that will do this for $15/mailbox but I figured I'd spend a little time at this myself first especially as this migration will probably take some time and who knows how many times I'll end up redoing it. MDaemon lets you export your contacts and calendars into a CSV file that Google Apps will accept for import. I do not have a script to import these files into Google Apps, so far I'm doing it manually but you will have the data in csv files. Please be aware that if you run this on a csv file containing all your users and passwords they will all get kicked out of Worldclient due to the 1 simultaneous login issues.
- Here's the script. It comes with no warranties or guarantees of any sort. This script is known to have issues with some entries containing quotation marks in the events themselves. We had few enough of these that I just manually fixed them as they came up. I also have not tested this script with files containing a list of accounts to process as I did these one at a time.
#!/usr/bin/perl
#
use LWP::UserAgent;
use URI::URL;
use HTTP::Headers;
use HTTP::Cookies;
use Text::CSV;
use Text::CSV_XS;
use WWW::Google::Contacts;
use Net::Google::Calendar;
use Data::Dumper;
use Date::Format;
use warnings;
##############################################################################
# Configure Variables;
##############################################################################
# WARNING - WARNING - WARNING - WARNING
# There are some very important settings here. Be sure to read before
# using.
##############################################################################
my ( $cookie_jar, $config, $hdrs, $ua, $url, $req, $resp );
$config->{'WorldClient'} = "https://mail.com/WorldClient.dll";
$config->{'time_zone'} = 'America/Edmonton';
$config->{'delete_existing_contacts'} =
0; #Set to 0 if you don't want all your existing contacts deleted!!!
$config->{'print_contact_groups'} = 1;
$config->{'delete_existing_calendar'} =
1; #Set to 0 if you don't want all your existing calendar entries deleted!!!
$config->{'use_proxy'} = 0; #Change to 1 if you want to use a proxy.
$config->{'Proxy'} = 'http://192.168.1.1:8888/';
# Check program parameters;
my $file = $ARGV[0];
if ( !$file ) {
print "Call this script as shown: mdaemon_migrate.pl email_accounts.csv.\n";
print "The format of the csv script shall be as follows:\n";
print 'username1@domain.com,password1' . "\n";
print 'username2@domain.com,password2' . "\n";
exit(0);
}
# Prepare connections
$ua = LWP::UserAgent->new();
$ua->proxy( ['http'], $config->{'Proxy'} ) if $config->{'use_proxy'} == 1;
my $csv = Text::CSV->new( { binary => 1, auto_diag => 1, } );
open( CSV, "<", $file ) or die $!;
while (<CSV>) {
if ( $csv->parse($_) ) {
my @columns = $csv->fields();
print "USERNAME: " . $columns[0] . " ";
print "PASSWORD: " . $columns[1] . "\n";
&dump_data( $columns[0], $columns[1] );
}
else {
my $err = $csv->error_input;
print "Failed to parse line: $err";
}
}
close CSV;
##############################################################################
# Subroutines, don't touch unless you know what you're doing.
##############################################################################
sub dump_data() {
my ( $username, $password ) = @_;
$hdrs = HTTP::Headers->new( 'User-Agent' => 'WorldClient Client' );
$ua->cookie_jar( HTTP::Cookies->new() );
$url = URI::URL->new( $config->{'WorldClient'} . '?View=Main' );
$req = HTTP::Request->new( POST, $url, $hdrs );
$resp = $ua->post( $url, [ User => $username, Password => $password ] );
my ( $junk, $session_string1 ) = split( m/Session=/, $resp->content );
my ( $session_id, $junk1 ) = split( m/'/, $session_string1 );
print STDERR "SESSIONID: " . $session_id . "\n";
$url =
URI::URL->new( $config->{'WorldClient'}
. '/calendar.csv?Session='
. $session_id
. '&View=CalendarExport&Filename=calendar.csv&FolderID=1' );
$req = HTTP::Request->new( POST, $url, $hdrs );
$resp = $ua->post(
$url,
[
STARTTIMEMONTH => '1',
STARTTIMEDAY => '1',
STARTTIMEYEAR => '2002',
STARTTIMEHOUR => '00',
STARTIMEMINUTE => '00',
ENDTIMEMONTH => '12',
ENDTIMEDAY => '31',
ENDTIMEYEAR => '2021',
ENDTIMEHOUR => '23',
ENDTIMEMINUTE => '55',
EXPORT => 'Export'
]
);
my $calendar_csv = $resp->content;
#Replace the embedded line breaks with <br>
$calendar_csv =~ s/\r\r\n/<br>/g;
#Replace embedded " with '
$calendar_csv =~ s/(?<=( |\w))"(?=( |\w))/'/g;
open( CALENDAR_CSV, '>' . $username . "-calendar.csv" );
print CALENDAR_CSV $calendar_csv;
close(CALENDAR_CSV);
$cal = Net::Google::Calendar->new;
$cal->login( $username, $password );
#Get a list of all events in the calendar.
for ($cal->get_events()) {
print "EXISTING CALENDAR ENTRY: " . $_->title."\n";
if ($config->{'delete_existing_calendar'} == 1) {
$cal->delete_entry($_);
print "DELETING CALENDAR ENTRY: " . $_->title."\n";
}
}
# my @rows;
# my $csv = Text::CSV_XS->new ({ binary => 1, eol => $/,auto_diag => 1 }) or
# die "Cannot use CSV: ".Text::CSV_XS->error_diag ();
# open my $fh, "<", $username . "-calendar.csv" or die $!;
# while (my $row = $csv->getline ($fh)) {
# print STDERR "NEW LINE: " . Dumper($row) . "END NEW LINE\n";
# push @rows, $row;
# }
# $csv->eof or $csv->error_diag ();
# close $fh;
my $csv = Text::CSV->new( { binary => 1, auto_diag => 1, } );
open( CSV, "<", $username . "-calendar.csv" ) or die $!;
while (<CSV>) {
# print STDERR "NEW LINE: " . $_ . "END NEW LINE\n";
if ( $csv->parse($_) ) {
my @columns = $csv->fields();
if ($columns[0] ne "Subject") {
print "Adding Calendar Event: " . $columns[0] . "\n";
my $entry = Net::Google::Calendar::Entry->new();
$entry->title( $columns[0] );
$entry->content( $columns[12] );
$entry->location( $columns[14] );
$entry->transparency('transparent');
$entry->status('confirmed');
if ($columns[15] eq "TRUE") {
$entry->visibility('private');
}
if ($columns[6] eq "TRUE") {
my ($alert_month, $alert_day, $alert_year) = split(m!/!, $columns[7]);
my ($alert_hour, $alert_minute, $alert_second) = split(m/:/, $columns[8]);
my $alert = DateTime->new(
year => $alert_year,
month => $alert_month,
day => $alert_day,
hour => $alert_hour,
minute => $alert_minute,
second => $alert_second,
nanosecond => 0,
time_zone=> $config->{'time_zone'}
);
$entry->reminder('alert','absoluteTime',$alert);
}
my ($start_month, $start_day, $start_year) = split(m!/!, $columns[1]);
my ($start_hour, $start_minute, $start_second) = split(m/:/, $columns[2]);
my ($end_month, $end_day, $end_year) = split(m!/!, $columns[3]);
my ($end_hour, $end_minute, $end_second) = split(m/:/, $columns[4]);
my $allday = 0;
if ($columns[2] eq "0:00:00" && $columns[4] eq "23:59:00") {
print "ALL DAY EVENT\n";
$end_hour = "00";
$end_minute = "00";
$end_second = "00";
$allday = 1;
}
my $start = DateTime->new(
year => $start_year,
month => $start_month,
day => $start_day,
hour => $start_hour,
minute => $start_minute,
second => $start_second,
nanosecond => 0,
time_zone=> $config->{'time_zone'}
);
my $end = DateTime->new(
year => $end_year,
month => $end_month,
day => $end_day,
hour => $end_hour,
minute => $end_minute,
second => $end_second,
nanosecond => 0,
time_zone=> $config->{'time_zone'}
);
$entry->when($start,$end, $allday);
$cal->add_entry($entry);
}
}
else {
my $err = $csv->error_input;
print "Failed to parse line: $err";
}
}
close CSV;
############################################################
# Start processing Contacts
############################################################
$url =
URI::URL->new( $config->{'WorldClient'}
. '/contacts.csv?Session='
. $session_id
. '&View=ContactExport&Filename=contacts.csv&FolderID=2' );
$req = HTTP::Request->new( POST, $url, $hdrs );
$resp = $ua->post( $url, [ EXPORT => 'Export' ] );
my $contacts_csv = $resp->content;
$contacts_csv =~ s/\r\r\n/<br>/g;
open( CONTACTS_CSV, '>' . $username . "-contacts.csv" );
print CONTACTS_CSV $contacts_csv;
close(CONTACTS_CSV);
my $csv = Text::CSV->new( { binary => 1, auto_diag => 1 } );
open( CSV, "<", $username . "-contacts.csv" ) or die $!;
my $google = WWW::Google::Contacts->new(
username => $username,
password => $password
);
if ( $config->{'print_contact_groups'} == 1 ) {
my $groups = $google->groups;
while ( my $gr = $groups->next ) {
print "You got a contact group called " . $gr->title . "\n";
}
}
if ( $config->{'delete_existing_contacts'} == 1 ) {
my @contacts = $google->contacts->search();
foreach my $c (@contacts) {
print STDERR "Deleting Contact: " . $c->full_name . "\n";
$c->delete;
}
}
while (<CSV>) {
if ( $csv->parse($_) ) {
my @columns = $csv->fields();
my $contact = $google->new_contact;
$contact->given_name( $columns[1] );
$contact->additional_name( $columns[2] );
$contact->family_name( $columns[3] );
$contact->name_prefix( $columns[0] );
$contact->name_suffix( $columns[4] );
$contact->full_name( $columns[59] );
$contact->add_email( { type => "work", value => $columns[57] } )
if $columns[57] ne "";
$contact->add_email( { type => "other", value => $columns[60] } )
if $columns[60] ne "";
$contact->phone_number( { type => "work", value => $columns[31] } )
if $columns[31] ne "";
$contact->phone_number(
{ type => "work 2", value => $columns[32] } )
if $columns[32] ne "";
$contact->phone_number(
{ type => "work fax", value => $columns[30] } )
if $columns[30] ne "";
$contact->phone_number( { type => "home", value => $columns[37] } )
if $columns[37] ne "";
$contact->phone_number(
{ type => "mobile", value => $columns[40] } )
if $columns[40] ne "";
$contact->phone_number( { type => "other", value => $columns[42] } )
if $columns[42] ne "";
$contact->im();
$contact->organization(
{
name => $columns[5],
title => $columns[7],
department => $columns[6]
}
);
$contact->postal_address(
[
{
type => "work",
street => $columns[8],
street2 => $columns[9],
street3 => $columns[10],
city => $columns[11],
region => $columns[12],
postcode => $columns[13],
country => $columns[14]
},
{
type => "home",
street => $columns[15],
street2 => $columns[16],
street3 => $columns[17],
city => $columns[18],
region => $columns[19],
postcode => $columns[20],
country => $columns[21]
},
{
type => "other",
street => $columns[22],
street2 => $columns[23],
street3 => $columns[24],
city => $columns[25],
region => $columns[26],
postcode => $columns[27],
country => $columns[28]
}
]
);
$contact->billing_information();
$contact->birthday();
$contact->hobby();
$contact->add_group_membership("System Group: My Contacts");
$contact->create;
}
else {
my $err = $csv->error_input;
print "Failed to parse line: $err";
}
}
close CSV;
}
No comments:
Post a Comment