2005.07.11 - v2 by Matt Simerson
version 1 available here
Version 2 is updated for use with Mac OS X 10.4 (Tiger).

My .mac subscription is 60 days from renewal so I have to ask myself, "how useful is .mac to me?

Is .mac worth it to me? Many of the reasons I don't find .mac useful are the same reasons I encourage others to use .mac. One has to keep in mind that I'm not an "average" computer user. My needs are different and Apple wouldn't make any money trying to sell a .mac like service to guys like me. This is not an "I hate .mac" site but rather an explanation of the motivation and methods I used to provide myself with comparable services that are more usable to me. I publish it so that others may benefit from what I have learned.

This is published to help others, but don't expect free support from the author. Support requests that arrive without monetary compensation for my time will almost certainly be ignored. Instead, try using the support forums and maybe someone will help you out.

To understand why I did this, you might want to read about my use of .mac services.

Project Goals:

Retain the useful features: Regardless of whether or not I renew my subscription, I want to retain the features I have found most useful (iDisk, iSync (between computers), iCal sharing, and Backup).

Enhance the useful features: Simply retaining the useful features would be an utter failure. The most value can be found in addressing the shortcomings of each feature. For iDisk, speed and disk space are the impediments to it's usefulness. iSync already works quite well. iCal sharing works well but publish and subscribe updates are sloooow. Backup is hamstrung by the iDisk space issue.

Plan of Attack

Since most of the the .mac services revolve around the use of of webdav, the first step seems rather obvious: set up a webdav server. There are quite a few documents published on this subject so I only give a brief summary of the steps I took to accomplish that task. Then things get a little more complicated as I had to convince my systems to use my webdav server instead of Apple's. That creates a number of problems which I successively tackle and attempt to beat into submission.

Set up webdav:

The first order of business is to get webdav installed and configured. I run Apache 2 on FreeBSD, so I already had mod_dav installed. It was just a simple matter of configuring it. I did so by first creating two directories for use by WebDav. The /home/idisk directory is the one to be used by webdav clients, the /var/run one is used internally by Apache.

• mkdir /home/idisk/html /var/run/webdav
• chmod 755 /var/run/webdav
• chown www:www /home/idisk/html /var/run/webdav

Note that my Apache server runs as the user www. If you run Apache as some other user (nobody is common), then make sure to alter the chown command to suit.

I then added these config lines to Apache's config files:

• DavLockDB /var/run/webdav/DavLock
•  
• <VirtualHost *:80>
• ServerName idisk.cadillac.net
• DocumentRoot "/home/idisk/html"
• </VirtualHost>
•  
• <Directory "/home/idisk/html">
• Dav on
• AuthType Digest
• AuthName iTools
• AuthDigestDomain "/"
• AuthDigestFile /home/idisk/WebDavUsers
• AuthGroupFile /home/idisk/WebDavGroups
• Options None
• AllowOverride None
•  
• <LimitExcept GET HEAD OPTIONS>
• require valid-user
• </LimitExcept>
•  
• Order allow,deny
• Allow from All
• </Directory>

The next task is creating the password digest file that Apache will use for authentication.

• htdigest -c /home/idisk/WebDavUsers iTools mattsimerson

Finally, I created a DNS record for idisk.cadillac.net and restarted Apache. I was able to use the Finder's "Connect To Server" menu item and connected directly to http://idisk.cadillac.net.

Very nifty, now I have something resembling my own iDisk. Since my web server is also on my home LAN, I'm connected to it via Gigabit Ethernet so it's way faster than using Apple's iDisk. Apple's iDisk is painfully slow, despite my extremely fast T3 internet connection. One thing you'll notice is that KB available is empty. The Apache2 dav module doesn't include working quota support.

mod_dav quota support: Andreas Amann wrote to point me at a mod_dav patch by William Carrel with quota support. The URL is here and I've mirrored it here. I've tested the patch and it reports back your free disk space on the server. When I enabled quotas on my FreeBSD server, it doesn't honor them. This is not quite ideal but better than nothing. To apply this patch you'll have to rebuild Apache and apply the patch to the dav module.

Once you've installed Apache with the patch, reconnect and you should have quota results shown.

Configuring WebDav accounts

Now that you have WebDav enabled, you'll want to allow others to connect to it as well. In the following example, I set up two user accounts, one for myself and one for my wife (Jen).

Create the home directories:

• cd /home/idisk/html
• mkdir mattsimerson jen
• chdir mattsimerson
• mkdir Backup Documents Library Movies Music Pictures Public Sites Sites/.calendars
• chdir ../jen
• mkdir Backup Documents Library Movies Music Pictures Public Sites Sites/.calendars
• chown -R www:www /home/idisk/html

You get the idea. Repeat the previous steps for each addition user you might want to create.

Restrict access to "personal" directories by addding the following to your Apache config:

• <Directory "/home/idisk/html/*/Public">
• Options +Indexes
• </Directory>
•  
• <Directory "/home/idisk/html/mattsimerson">
• <LimitExcept GET HEAD OPTIONS>
• require user mattsimerson matt
• </LimitExcept>
• </Directory>
•  
• <Directory "/usr/home/idisk/html/jen">
• <LimitExcept GET HEAD OPTIONS>
• require user jen
• </LimitExcept>
• </Directory>

You'll do the same for each user that you add. That was the easy part.

Publish iCal calendars on webdav: My wife and I enjoy being able to share each others calendars.

The next step was to get iCal to publish to my new server. For consistency, I recommend publishing your calendars to the same location that Apple uses for .mac: Sites/.calendars. You can test now by setting up a test iCal calendar:

After clicking publish, you should get a success notice that looks like this:

After updating all our calendars, I also modified them so that now they auto-publish after each change. When using .mac, I had them update every hour, as updating would take quite a number of seconds to complete. Now the publishing of changes is so fast that it's transparent. Once our calendars were all published, it was a trivial exercise to get PHP iCalendar to publish our calendars via a web server. Click here to see how.

Emulate www.mac.com

The next step was to convince my mac that my new webdav server was Apple's iDisk server. The first step was to use tcpdump on my firewall and see where the connections where going. As of 10.4 (Tiger), when I access the .mac control panel, it retrieves the configuration and settings from four different servers:

• configuration.apple.com
• syncmgmt.mac.com
• idisk.mac.com
• www.mac.com

The Backup client software connects to both idisk.mac.com and www.mac.com. The next step is to determine exactly what the clients are looking for from the server and replicate it (to the extent possible). The simplest way to redirect the traffic to my Apache server is to map the DNS for those four hostnames to my own server.

Configure DNS: I manage my own DNS servers so it's quite easy for me to tell my DNS servers that they are authoritative for mac.com, and resolve the addresses to my FreeBSD servers IP. If you only need this to work on one system, simply add an entry to /etc/hosts to do the same thing. After adding the entries to /etc/hosts, run lookupd -flushcache. Then if you run a network command like ping www.mac.com, you should see the traffic going to your host.

Configure Apache: To get Apache to act like idisk.apple.com, I considerably expanded the Apache directives as shown in this apache include file. I also created another include file to simulate www.mac.com. You'll also need to generate a SSL certificate for www.mac.com as well. I did it the same way I always do, by RTFM on Apache's site. I also signed the new SSL cert with my CA key which is already trusted by all the macs on my LAN.

Create emulation scripts: Next up was creating the scripts that respond as Apple's do. This was quite easy and you can read all about it on Otto's site if you're interested. The nuts and bolts are pretty simple. Since you can't packet sniff the https connection, get the info from your Apache logs to see what URL's are being asked for. Once you know that, install a script there that dumps the POST info to a temp file. You end up with something like this in that temp file:

• SERVER_SOFTWARE = Apache/2.0.50 (FreeBSD)
• SERVER_NAME = www.mac.com
• GATEWAY_INTERFACE = CGI/1.1
• SERVER_PROTOCOL = HTTP/1.1
• SERVER_PORT = 443
• REQUEST_METHOD = POST
• SCRIPT_NAME = /WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration
• REMOTE_ADDR = 10.0.1.218
• CONTENT_LENGTH = 160
• {
• body = {relativePath = Public; };
• function = retrieveDiskConfiguration;
• header = {
• password = ******; username = mattsimerson; version = 1;
• };
• }

I copied and pasted the {} info into the file foo and then used lynx to see what Apple's server returns. When I sent this request:

lynx -source -post_data -useragent="InternetPref/Version-10.2" https://www.mac.com/WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration < foo

I got back this response:

• {
• payload = {
• guestReadEnabled = Y;
• guestWriteEnabled = N;
• hasGeneralPassword = N;
• iDiskQuotaInBytes = 104857600;
• iDiskUsedBytes = 11383808;
• relativePath = Public;
• };
• statusCode = success;
• }

So, now that we know what the script needs to return, let's create the script:

mkdir -p /home/www.mac.com/WebObjects/Info.woa/wa/Query

cd /home/www.mac.com/WebObjects/Info.woa/wa/Query

vi retrieveDiskConfiguration

The contents of that file should be something like what follows:

• #!/bin/sh
•  
• echo Content-type: text/plain
• echo
•  
• cat << EOT
• {
• payload = {
• guestReadEnabled = Y;
• guestWriteEnabled = N;
• hasGeneralPassword = N;
• iDiskQuotaInBytes = 1048576000;
• iDiskUsedBytes = 339338752;
• relativePath = Public;
• };
• statusCode = success;
• }
• EOT

Now when you access the iDisk tab in the .mac control panel, it gets it's values from the script you just created.

UPDATE: Dominic Rivera contributed a script that returns the disk free output instead of using a static script.

Very, very cool. Notice that I now have 1GB of space available. It is, in effect, infinite because no matter how much data I upload, I'll always have 663 MB free. If I were a service provider, I'd rewrite the script in perl or C, have it parse the POST data, verify the authentication parameters, and return actual disk quota values. Since I'm only doing this for myself and my wife, that's not important.

Now I can connect to my iDisk using the Finder "Go->iDisk->My iDisk" menu, or the Cmd-Shift-I keyboard shortcut. When I connect, it is wicked fast and I have tons of disk space available.

Upon running Backup.app, a check of my Apache log files revealed that it was checking the URL "https://www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo". As before, we install a script there to capture the POST data and see what it's looking for. We end up with this in the temp file:

• SERVER_SOFTWARE = Apache/2.0.50 (FreeBSD)
• SERVER_NAME = www.mac.com
• GATEWAY_INTERFACE = CGI/1.1
• SERVER_PROTOCOL = HTTP/1.1
• SERVER_PORT = 443
• REQUEST_METHOD = POST
• HTTP_ACCEPT = image/gif, image/jpeg, image/pjpeg, */*
• PATH_INFO =
• PATH_TRANSLATED =
• SCRIPT_NAME = /WebObjects/Info.woa/wa/Query/accountInfo
• QUERY_STRING =
• REMOTE_HOST =
• REMOTE_ADDR = 10.0.1.218
• REMOTE_USER =
• AUTH_TYPE =
• CONTENT_TYPE = text/xml
• CONTENT_LENGTH = 165
•  
• {
• body = {keys = (iToolsBackupActivated, trialAccountDaysLeft); };
• function = accountInfo;
• header = {password = *******; username = mattsimerson; };
• }

Once again, save the contents within the {} brackets into a temp file "foo" and post them to Apple's URL.. I did so with the following Lynx command:

• lynx -source -post_data -useragent="Backup 2.0.2" https://www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo < foo

I got back the following results:

• {
• payload = {
• iToolsBackupActivated = Y;
• };
• statusCode = success;
• }

So, now we need to set up the accountInfo script to return that value when Backup queries it. I did so by editing the following file:

• vi /home/www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo

The contents of that file should be something like what follows:

• #!/bin/sh
•  
• echo Content-type: text/plain
• echo
•  
• cat << EOT
• {
• payload = {
• iToolsBackupActivated = Y; trialAccountDaysLeft = -1;
• };
• statusCode = success;
• }
• EOT

After making this change, the first time you run Backup, it'll check your server, see that you have Backup activated and let you back up your Mac(s) to your local iDisk server. I don't know how often it performs this check but after having done so, I haven't seen it check again (unless I delete the Backup prefs file).

I now have iDisk and Backup access on all three of my computers (dual G5, PowerBook, and wifes iMac) without purchasing multiple .mac accounts. Excellent, this is better than .mac!

Configure a Proxy

Redirecting www.mac.com to my server creates a new problem in that now I can't visit www.mac.com from my LAN. Since I still do have a .mac account, I wish to retain that ability. To do so, I adjusted my Apache config file a little more by adding some proxy directives.

First, I edited my httpd.conf and uncommented the proxy_module, proxy_connect_module (for ssl), and proxy_http_module modules. I also added the following block:

• <Proxy *>
• Order Deny,Allow
• Deny from all
• Allow from 10.0
• </Proxy>

The Proxy statement prevents the rest of the world from being able to access my proxy server. Publicly available proxy servers are a hazard to their owners and the rest of the internet. Be sure to secure yours!

I also added the following commands to the www.mac.com virtualhost containers:

• ProxyRequests On
• ProxyVia On
• ProxyPass /WebObjects/Info.woa/wa/Query/accountInfo !
• ProxyPass /WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration !
• ProxyPass / http://www.mac.com/

Here is my completed www.mac.com vhost config file for your reading enjoyment.

The one last loose end is adding iSync support so that I can use my own server to sync Address Book, iCal, and Safari bookmarks between my systems. Jeremy Baker has headed down that road so I expect to spend some time tinkering with that in the future.

NOTES

Platform Independent: This solution is not even remotely dependent on FreeBSD. I could have just as easily implemented this solution on my Linux or Mac OS X systems. I chose my FreeBSD server because it's already the gateway between my LAN and the internet. Because it's dual homed, I can access it locally on the LAN as well as remotely with my PowerBook without playing silly network tricks (like VPN or SSL tunnels).

iPhoto Homepage publishing: It's broken. I don't know why yet. I intend to figure out why at some point but don't hold your breath because it's not very important to me. I use Gallery with the iPhotoToGallery plugin and BetterHTMLExport with a template I customized.

iSync still works with .mac synchronization disabled but I lose the ability to sync between computers. That's a major loss.

Dominic also mentioned using disk images for users DAV space. Combine this with his script for fetching disk usage and you've got working per-user disk space reporting.

.mac, iCal, iDisk, iSync, and a few other things listed here are probably registered trademarks of Apple Computer.

Last modified on 2/8/06.