Do It Yourself .Mac
How I created my own .mac replacement
by Matt Simerson
2006.02.08 - misc updates.
2005.07.11 - v2 updated for use with Mac OS X 10.4 (Tiger).
version 1 available
Is .mac worth it to me?
Many of the reasons I do not find .mac useful are the same reasons I encourage others to use it. My needs are different than the average computer user. This is not an "I hate .mac" article but rather an explanation of the motivation and methods I used to provide myself with comparable services that are more useful to me. I publish it so that others may benefit from what I have learned.
Please don't expect free support from the author. Support requests that arrive without monetary compensation are typically ignored. Instead, try using the support forums.
You can also read about my use of .mac
Project Goals
- Retain the useful features: Regardless of whether or not I renew my subscription, I want the features I have found most useful (iDisk, iSync (between computers), iCal sharing, and Backup).
- Enhance the useful features: The most value can be found in enhancing each feature. For iDisk, speed and disk space are impediments to its usefulness. iSync works quite well. iCal sharing works well but publish and subscribe updates are terribly slow. Backup is hamstrung by the disk space issue.
Plan of Attack
- Since most of the the .mac services revolve around the use of of webdav, the first step seems rather obvious: set up a webdav server. There are numerous documents published on this subject so I only give a brief summary of the steps I took. Then things get more complicated as I convince my desktop to use my webdav server instead of Apple's. That creates a number of problems which I successively tackle and attempt to beat into submission.
Getting it Done
Set up webdav
The first order of business is to get webdav installed and configured. I run Apache 2 on FreeBSD, so I already had mod_dav installed. It was just a matter of configuring it. I did so by creating two directories for use by WebDav. The /home/idisk directory is used by webdav clients, /var/run is used internally by Apache.
mkdir /home/idisk/html /var/run/webdav chmod 755 /var/run/webdav chown www:www /home/idisk/html /var/run/webdav
Note that my Apache server runs as the user www. If you run Apache as some other user (nobody is common), then make sure to alter the chown command to suit.
I then added these config lines to the Apache config files:
DavLockDB /var/run/webdav/DavLock
<VirtualHost *:80>
ServerName idisk.cadillac.net
DocumentRoot "/home/idisk/html"
</VirtualHost>
<Directory "/home/idisk/html">
Dav on
AuthType Digest
AuthName iTools
AuthDigestDomain "/"
AuthDigestFile /home/idisk/WebDavUsers
AuthGroupFile /home/idisk/WebDavGroups
Options None
AllowOverride None
<LimitExcept GET HEAD OPTIONS>
require valid-user
</LimitExcept>
Order allow,deny
Allow from All
</Directory>
The next task is creating the password digest file that Apache will use for authentication.
htdigest -c /home/idisk/WebDavUsers iTools mattsimerson
Finally, I created a DNS record for idisk.cadillac.net and restarted Apache. I was able to use Finder.app's "Connect To Server" menu item and connected directly to http://idisk.cadillac.net.
Now I have something resembling my own iDisk. Since my web server is also on my home LAN, I am connected to it via gigabit Ethernet, which is unbelievably faster than .mac. Apple's iDisk is painfully slow, even when I had an extremely fast T3 internet connection.
Add Quota Support
One thing you may notice is that KB available is empty. The Apache 2 dav module does not include working quota support.
Andreas Amann wrote to point me at a mod_dav patch by William Carrel with quota support. The patch is available here and I've mirrored it as well. I've tested the patch and it reports back your free disk space on the server. When I enabled quotas on my FreeBSD server, it does not honor them but does report them. This is not quite ideal but better than nothing. To apply this patch you will have to rebuild Apache and apply the patch to the dav module.
Once you have installed Apache with the patch, reconnect and you should have quota results shown.
Configuring WebDav accounts
Now that webdav is enabled, we want to allow others to connect as well. In the following example, I set up two user accounts for myself and my wife.
cd /home/idisk/html mkdir mattsimerson jen chdir mattsimerson mkdir Backup Documents Library Movies Music Pictures Public Sites Sites/.calendars chdir ../jen mkdir Backup Documents Library Movies Music Pictures Public Sites Sites/.calendars chown -R www:www /home/idisk/html
Repeat the previous steps for each additional user you want to create.
Restrict access to "personal" directories by addding the following to your Apache config:
<Directory "/home/idisk/html/*/Public">
Options +Indexes
</Directory>
<Directory "/home/idisk/html/mattsimerson">
<LimitExcept GET HEAD OPTIONS>
require user mattsimerson matt
</LimitExcept>
</Directory>
<Directory "/usr/home/idisk/html/jen">
<LimitExcept GET HEAD OPTIONS>
require user jen
</LimitExcept>
</Directory>
Do the same for each user that you add.
Publish iCal calendars on webdav
My wife and I enjoy being able to share each others calendars. The next step was to get iCal to publish to my new server. For consistency, I recommend publishing your calendars to the same location that Apple uses for .mac: Sites/.calendars. You can test now by setting up a test iCal calendar:
After clicking publish, you should get a success notice that looks like this:
After updating our calendars, I also modified them so that they auto-publish after each change. When using .mac, I had them update every hour, as updating would take quite a number of seconds to complete. Now the publishing of changes is transparent. Once our calendars were all published, it was a trivial exercise to get PHP iCalendar to publish our calendars.
Emulate www.mac.com
Now things get more complicated. If you do not have experience with using Server Apache and configuring DNS servers, you may want to find someone with these skills to help you out.
The first step was to convince my mac that my new webdav server was Apple's iDisk server. I began by using tcpdump on my firewall and see where the connections where going. As of Mac OS 10.4 (Tiger), when I access the .mac control panel, it retrieves the configuration and settings from four different servers:
configuration.apple.com syncmgmt.mac.com idisk.mac.com www.mac.com
Backup.app connects to both idisk.mac.com and www.mac.com. The next step is to determine exactly what the clients are looking for from the server and provide it (to the extent possible). The simplest way to redirect the traffic to my web server is to map the DNS for those four hostnames to my own server.
Configure DNS
I manage my own DNS servers so it is quite easy for me to tell them that they are authoritative for mac.com, and resolve the addresses to my servers IP. If you only need this to work on one system, simply add an entry to /etc/hosts to do the same thing. After adding the entries to /etc/hosts, run "lookupd -flushcache". You can test by running a network command like "ping www.mac.com" and you should see the traffic going to your host.
Configure Apache
To get Apache to act like idisk.apple.com, I considerably expanded the Apache directives as shown in ["http://www.tnpi.net/computing/mac/tips/idisk/idisk.cadillac.net.txt this apache include file]. I also created another include file to simulate www.mac.com. Finally, I generated a SSL certificate for www.mac.com. I did it the same way I always do, RTFM on the Apache web site. I also signed the new SSL cert with my CA key which is already trusted by all the macs on my LAN. You can also use the Mac OS Cerficate Assistant which you will find in the menu of Keychain Access on OS X 10.4.
Create emulation scripts
Next up was creating the scripts that respond as Apple's do. This was quite easy and you can read all about it on Otto's site if you're interested. The nuts and bolts are pretty simple. Since you cannot packet sniff the https connection, get the info from your Apache logs to see what URLs are being asked for. Once you have that, install a script there that dumps the POST info to a temp file. You end up with something like this in that temp file:
SERVER_SOFTWARE = Apache/2.0.55 (FreeBSD) SERVER_NAME = www.mac.com GATEWAY_INTERFACE = CGI/1.1 SERVER_PROTOCOL = HTTP/1.1 SERVER_PORT = 443 REQUEST_METHOD = POST SCRIPT_NAME = /WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration REMOTE_ADDR = 10.0.1.218 CONTENT_LENGTH = 160
{
body = {relativePath = Public; };
function = retrieveDiskConfiguration;
header = {
password = ******; username = mattsimerson; version = 1;
};
}
I copied and pasted the {} info into the file foo and then used lynx to see what Apple's server returns. When I sent this request:
lynx -source -post_data -useragent="InternetPref/Version-10.2" \ https://www.mac.com/WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration < foo
I got back this response:
{
payload = {
guestReadEnabled = Y;
guestWriteEnabled = N;
hasGeneralPassword = N;
iDiskQuotaInBytes = 104857600;
iDiskUsedBytes = 11383808;
relativePath = Public;
};
statusCode = success;
}
Knowing what the script needs to return, create the script:
mkdir -p /home/www.mac.com/WebObjects/Info.woa/wa/Query vi /home/www.mac.com/WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration
The contents of that file should be something like what follows:
#!/bin/sh
echo Content-type: text/plain
echo
cat << EOT
{
payload = {
guestReadEnabled = Y;
guestWriteEnabled = N;
hasGeneralPassword = N;
iDiskQuotaInBytes = 1048576000;
iDiskUsedBytes = 339338752;
relativePath = Public;
};
statusCode = success;
}
EOT
Now when you access the iDisk tab in the .mac control panel, it gets its values from the script you just created.
Dominic Rivera contributed a script that returns the disk free output instead of using a static value for disk space.
Notice that I now have 1GB of space available. It is, in effect, infinite because no matter how much data I upload, I will always have 663 MB free. If I were a service provider, I would rewrite the script in Perl or C, have it parse the POST data, verify the authentication parameters, and return actual disk quota values. Since I am only doing this for myself and my wife, that is not important.
I can connect to my iDisk using Finder.app "Go->iDisk->My iDisk" menu, or the Cmd-Shift-I keyboard shortcut. When I connect, it is wicked fast and I have ample disk space available.
Get Backup.app working
Upon running Backup.app, a check of my Apache log files revealed that it was checking the URL https://www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo. As before, we install a script there to capture the POST data and see what it is looking for. We end up with this in the temp file:
SERVER_SOFTWARE = Apache/2.0.55 (FreeBSD)
SERVER_NAME = www.mac.com
GATEWAY_INTERFACE = CGI/1.1
SERVER_PROTOCOL = HTTP/1.1
SERVER_PORT = 443
REQUEST_METHOD = POST
HTTP_ACCEPT = image/gif, image/jpeg, image/pjpeg, */*
PATH_INFO =
PATH_TRANSLATED =
SCRIPT_NAME = /WebObjects/Info.woa/wa/Query/accountInfo
QUERY_STRING =
REMOTE_HOST =
REMOTE_ADDR = 10.0.1.218
REMOTE_USER =
AUTH_TYPE =
CONTENT_TYPE = text/xml
CONTENT_LENGTH = 165
{
body = {keys = (iToolsBackupActivated, trialAccountDaysLeft); };
function = accountInfo;
header = {password = *******; username = mattsimerson; };
}
Once again, save the contents within the {} brackets into a temp file and post that to https://www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo Apple's URL. I did so with the following Lynx command:
lynx -source -post_data -useragent="Backup 2.0.2" \ https://www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo < foo
and got back the following results:
{
payload = {
iToolsBackupActivated = Y;
};
statusCode = success;
}
Now set up the accountInfo script to return that value when Backup.app queries it. I did so by editing the following file:
vi /home/www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo
The contents of that file should be something like this:
#!/bin/sh
echo Content-type: text/plain
echo
cat << EOT
{
payload = {
iToolsBackupActivated = Y; trialAccountDaysLeft = -1;
};
statusCode = success;
}
EOT
After making this change, the first time you run Backup.app, it will check your server, see that you have Backup activated and let you back up your Mac(s) to your local iDisk server. I don't know how often it performs this check but after having done so, I have not seen it check again (unless I delete the Backup prefs file).
I now have and Backup access on all three of my computers (dual G5], PowerBook, and wifes iMac) without purchasing multiple .mac accounts. I also have a backup disk large enough to be useful. This is much better than .mac!
Configure a Proxy
Redirecting www.mac.com to my server creates a new problem in that now I cannot visit www.mac.com from my LAN. Since I still do have a .mac account, I wish to retain that ability. To do so, I adjusted my Apache config file a little more by adding some proxy directives.
I edited my httpd.conf and uncommented the proxy_module, proxy_connect_module (for ssl), and proxy_http_module modules. I also added the following block:
<Proxy *> Order Deny,Allow Deny from all Allow from 10.0 </Proxy>
The Proxy statement prevents the rest of the world from being able to access my proxy server. Publicly available proxy servers are a hazard to their owners and the rest of the internet. Be sure to secure yours!
I also added the following commands to the www.mac.com virtualhost containers:
ProxyRequests On ProxyVia On ProxyPass /WebObjects/Info.woa/wa/Query/accountInfo ! ProxyPass /WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration ! ProxyPass / http://www.mac.com/
Here is my completed www.mac.com vhost config file for your reading enjoyment.
The last loose end is adding iSync support so that I can use my own server to sync Address Book, iCal, and Safari bookmarks between my systems. Jeremy Baker has headed down that road so I expect to spend some time tinkering with that in the future.
NOTES
Platform Independent
This solution is not even remotely dependent on FreeBSD. I could have just as easily implemented this solution on my Linux or Mac OS X systems. I chose my FreeBSD server because it is the gateway between my LAN and the internet. Because it is dual homed, I can access it locally on the LAN as well as remotely with my PowerBook without playing silly network tricks (like VPN or SSL tunnels).
I do have a few .mac FreeBSD specific build notes
iPhoto Homepage publishing
It is broken. I do not know why. I intend to figure out why at some point but don't hold your breath because it is not important to me. I use Gallery with the iPhotoToGallery plugin and BetterHTMLExport with a template I customized.
iSync
iSync still works with .mac synchronization disabled but I lose the ability to sync between computers if I drop my .mac account. That would be a major loss.
iDisk disk images
Dominic suggested using disk images for users DAV space. Combine this with his script for fetching disk usage and you have working per-user disk space reporting.
.mac, iCal, iDisk, iSync, and a few other things listed here are probably all registered trademarks of Apple Computer.