How I created my own .mac replacement
2004.08.16 - Published by Matt Simerson
2004.09.10 - Added iCal integration notes
2004.09.23 - Updated mod_dav patch
2004.11.24 - Added Dominic's retrieveDiskConfiguration script
2004.12.?? - Added Support Forums link and note.
2005.01.26 - Added Order directive for Apache config (for v2.0.52+)
My .mac subscription is 60 days from renewal so I have to ask myself, "how useful is .mac to me? Is it worth another $100?"
Is .mac worth it to me? Many of the reasons I don't find .mac useful are the same reasons I encourage others to use .mac. One has to keep in mind that I'm not an "average" computer user. My needs are different and Apple wouldn't make any money trying to sell a .mac like service to guys like me. This is not an "I hate .mac" site but rather an explanation of the motivation and methods I used to provide myself with comparable services that are more usable to me. I publish it so that others may benefit from what I have learned.
This is published to help others, but don't expect free support from the author. Support requests that arrive without monetary compensation for my time will almost certainly be ignored. Instead, try using the support forums and maybe someone will help you out.
My use of .mac services
Mail: I have used it, and have recently discontinued using it principally because of inadequate spam filtering. About 80% of the spam in my in-box arrives via my .mac account. I have hotmail, yahoo, gmail, and several mailboxes I host on my own servers. That tells me Apple has a serious problem. Their filtering is still behind their competitors (most of which are FREE) and even that of my own Mail::Toaster project. I hate spam, so getting rid of my .mac account (1 of 9 primary email accounts) is just another way to reduce the spam I see.
Address Book: I don't use .mac Address Book because I have a Palm & iPod and both have that data in them. I prefer not to publish the personal data of every person I know on any web site.
Homepage: I like the integration and ease of use but the limits are far too severe to be useful to me. My personal web site takes up 350 MB, mainly because of photo albums like Alaska that I publish. Some of my sites require database backends and other fun CGI stuff I can't do with .mac. Apple also has been know to shut down sites that exceed certain unpublished bandwidth threshholds. I'm not anxious to discover that has happened to my site.
iDisk: iDisk: I regularly use my iDisk to toss files onto and then download them at a client site. However, this habit often costs me wasted time because access can be interminably slow. In the time it takes to download a 30MB file from my iDisk, I can drive home from anywhere in town, burn the file to a CD or store it on a USB flash drive, and drive back before the 30MB download is complete. Putting that same file on my WebDav enabled server takes seconds and I can download it on any broadband connection in just a few minutes.
Backup: I don't use Backup to my iDisk mainly because I don't trust others with my personal data. In order for me to consider using Backup, I'd insist on seeing my data encrypted as it passes across the internet and stored in an encrypted format such that it's technically not feasible for anyone to access my data without my private key. Also, 100MB is a good start, and a reasonable 1GB is $350/year. Did I mention how slow iDisk access is? For these reasons, Backup is next to useless to me in it's current form.
iSync: I use iSync to sync my Address Book and iCal calendars between multiple macs, my Palm, and iPod.
iCal: I like and use the iCal support for publishing my calendars and subscribing to my wife's. She also subscribes to mine, this is quite useful.
Misc: I never use the following features: iCards, Virex, Support. I do often download the freebies, and these are occasionally of some value.
Project Goals:
Retain the useful features: Regardless of whether or not I renew my subscription, I want to retain the features I have found most useful (iDisk, iSync (between computers), iCal sharing, and Backup).
Enhance the useful features: Simply retaining the useful features would be an utter failure. The most value can be found in addressing the shortcomings of each feature. For iDisk, speed and disk space are the impediments to it's usefulness. iSync already works quite well. iCal sharing works well but publish and subscribe udpates are sloooow. Backup is hamstrung by the iDisk space issue.
Is it possible: First I had to determine if I could even achieve comparable results using resources I already possess. My first endeavor was scrub the net looking for folks that have already done similar things. Apparently, SpyMac is where .mac refugees who don't have their own servers go. While it doesn't appear to be impossible, it certainly doesn't seem like well charted territory.
Plan of Attack
Since most of the the .mac services revolve around the use of of webdav, the first step seems rather obvious: set up a webdav server. There are quite a few documents published on this subject so I only give a brief summary of the steps I took to accomplish that task. Then things get a little more complicated as I had to convince my systems to use my webdav server instead of Apple's. That creates a number of problems which I successively tackle and attempt to beat into submission.
Set up webdav: The first order of business is to get webdav installed and configured.
I run Apache 2 on FreeBSD, so I already had mod_dav installed. It was just a simple matter of configuring it. I did so by first creating two directories for use by WebDav. The /home/idisk directory is the one to be used by webdav clients, the /var/run one is used internally by Apache.
mkdir /home/idisk /var/run/webdav
chmod 755 /var/run/webdav
chown www:www /home/idisk /var/run/webdav
Note that my Apache server runs as the user www. If you run Apache as some other user (nobody is common), then make sure to alter the chown command to suit.
I then added these config lines to Apache's config files:
DavLockDB /var/run/webdav/DavLock
<VirtualHost *:80>
ServerName idisk.cadillac.net
DocumentRoot "/home/idisk"
</VirtualHost>
<Directory "/home/idisk">
Dav on
AuthType Digest
AuthName iTools
AuthDigestDomain "/"
AuthDigestFile /usr/local/www/WebDavUsers
<LimitExcept GET OPTIONS>
require valid-user
</LimitExcept>
Order allow,deny
Allow from All
</Directory>
The next task is creating the password digest file that Apache will use for authentication.
htdigest -c /usr/local/www/WebDavUsers iTools mattsimerson
Finally, I created a DNS record for idisk.cadillac.net and restarted Apache. I was able to use the Finder's "Connect To Server" menu item and connected directly to http://idisk.cadillac.net.
Very nifty, now I have something resembling my own iDisk. Since my web server is also on my home LAN, I'm connected to it via full-duplex 100 Base-T so it's way faster than using Apple's iDisk. Apple's iDisk is painfully slow, despite my extremely fast T3 internet connection.
Getting WebDav to work was the easy part.
Publish iCal calendars on webdav: My wife and I enjoy being able to share each others calendars. Until now, we had shared them via .mac. To get them published via webdav was trivial. I first created a directory on the server to store them:
mkdir /home/idisk.cadillac.net/Calendars
It was then a simple matter to select each calendar in iCal and choose "change location". A dialog box will pop up and you simply fill in the new details.
After updating all our calendars, I also modified them so that now they auto-publish after each change. When using .mac, I had them update every hour, as updating would take quite a number of seconds to complete. Now the publishing of changes is so fast that it's transparent. From there it was a trivial exercise to get PHP iCalendar to publish our calendars via a web server. I simply configured it to view the calendars in /home/idisk/Calendars.
iCal .mac integration: The next step was to get iCal to publish to my server with .mac settings. I created a test calendar and tried publishing it to .mac. It failed because the Sites/.calendars folder didn't exist. After I created that directory, iCal happily exported my test calendar to my new iDisk. This is obviously much better as calendar space is no longer "shared." Now we can both publish a calendar named "Work". This change does complicate web access via iCal. Now PHP iCalendar needs to know your username in order to know which calendar to display.
To solve this problem I did two things. The first was requiring HTTP authentication to view calendars. This is arguably something that should be done anyway. I did so by updating the following block of code in httpd.conf:
<Directory "/usr/local/www/phpicalendar">
Options Indexes ExecCGI
AllowOverride Limit
Order deny,allow
deny from all
AuthType Digest
AuthName iTools
AuthDigestDomain "/"
AuthDigestFile /usr/local/www/WebDavUsers
require valid-user
satisfy any
</Directory>
Now when I visit my iCal site, I have to authenticate using my webdav credentials. The last little tidbit was getting PHP iCalendar to know where to find my calendar files. I added the following little code snippet to config.inc.php, immediately after the $calendar_path declaration:
if (isset($_SERVER['REMOTE_USER'])) {
$calendar_path = "/home/idisk/".$_SERVER['REMOTE_USER']."/Sites/.calendars";
}
If I haven't passed HTTP authentication credentials, I get the normal icalendar behavior. If I authenticate, my published calendars are displayed. The only remaining anomoly is that the publish URL that iCal shows is on the ical.mac.com server. Obviously, I'd need to point ical.mac.com to my server, or just re-write the URL to point to my server. The latter requires less effort.
Patch mod_dav for quota support: After a bit more poking around the net, I found a great site detailing how to use Backup without .mac. After a bit of reading on Otto's site, I decided to get quota support working with mod_dav since that was going to be a requirement for Backup.app to work properly. Otto has a patch for mod_dav 1 but it doesn't work with mod_dav 2. Since mod_dav 2 is included with Apache 2, I made some alterations and built a working patch which I have posted here. This patch is tested against Apache 2.0.50, the latest version as of this writing. Now, when I connect to my WebDav server, I get quota results returned. Yay.
mod_dav quota UPDATE! Andreas Amann wrote to point me at a better mod_dav patch by William Carrel with better quota support. The URL is here and I've mirrored it here. I've tested the patch and it reports back your free disk space on the server. However, when I enabled quotas on my FreeBSD server, it doesn't honor them. This is not quite ideal but better than the previous patch though so it's linked here.
Emulate www.mac.com: The next step was to convince my mac that my new webdav server was Apple's iDisk server. This was not quite so easy but I had Otto's site as a guide which helped tremendously. The first thing I did was the normal server stuff so that my FreeBSD server would act like Apple's iDisk server. This involved the following steps:
Configure DNS: I manage my own DNS servers so it's quite easy for me to tell my DNS servers that they are authoritative for www.mac.com and idisk.mac.com, and resolve both addresses to 10.0.1.1 (my FreeBSD servers internal IP). If you don't have a LAN full of macs and only want it to work on one, simply add an entry to /etc/hosts to do the same thing.
Configure Apache: To get Apache to act like idisk.apple.com, I considerably expanded the Apache directives as shown in this apache include file. I also created another include file to simulate www.mac.com. You'll also need to generate a SSL certificate for www.mac.com as well. I did it the same way I always do, by RTFM on Apache's site. I also signed the new SSL cert with my CA key which is already trusted by all the macs on my LAN.
Create emulation scripts: Next up was creating the scripts that respond as Apple's do. This was quite easy and you can read all about it on Otto's site if you're interested. The nuts and bolts are pretty simple. Since you can't packet sniff the https connection, get the info from your Apache logs to see what URL's are being asked for. Once you know that, install a script there that dumps the POST info to a temp file. You end up with something like this in that temp file:
SERVER_SOFTWARE = Apache/2.0.50 (FreeBSD)
SERVER_NAME = www.mac.com
GATEWAY_INTERFACE = CGI/1.1
SERVER_PROTOCOL = HTTP/1.1
SERVER_PORT = 443
REQUEST_METHOD = POST
SCRIPT_NAME = /WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration
REMOTE_ADDR = 10.0.1.218
CONTENT_LENGTH = 160
{
body = {relativePath = Public; };
function = retrieveDiskConfiguration;
header = {
password = ******; username = mattsimerson; version = 1;
};
}
I copied and pasted the {} info into the file foo and then used lynx to see what Apple's server returns. When I sent this request:
lynx -source -post_data -useragent="InternetPref/Version-10.2" https://www.mac.com/WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration < foo
I got back this response:
{
payload = {
guestReadEnabled = Y;
guestWriteEnabled = N;
hasGeneralPassword = N;
iDiskQuotaInBytes = 104857600;
iDiskUsedBytes = 11383808;
relativePath = Public;
};
statusCode = success;
}
So, now that we know what the script needs to return, let's create the script:
mkdir -p /home/www.mac.com/WebObjects/Info.woa/wa/Query
cd /home/www.mac.com/WebObjects/Info.woa/wa/Query
vi retrieveDiskConfiguration
The contents of that file should be something like what follows:
#!/bin/sh
echo Content-type: text/plain
echo
cat << EOT
{
payload = {
guestReadEnabled = Y;
guestWriteEnabled = N;
hasGeneralPassword = N;
iDiskQuotaInBytes = 1048576000;
iDiskUsedBytes = 339338752;
relativePath = Public;
};
statusCode = success;
}
EOT
Now when you access the iDisk tab in the .mac control panel, it gets it's values from the script you just created.
UPDATE: Dominic Rivera contributed a script that returns the disk free output instead of using a static script.
Very, very cool. Notice that I now have 1GB of space available. It is, in effect, infinite because no matter how much data I upload, I'll always have 663 MB free. If I were a service provider, I'd rewrite the script in perl or C, have it parse the POST data, verify the authentication parameters, and return actual disk quota values. Since I'm only doing this for myself and my wife, that's not important.
Now I can connect to my iDisk using the Finder "Go->iDisk->My iDisk" menu, or the Cmd-Shift-I keyboard shortcut. When I connect, it is wicked fast and I have tons of disk space available.
Get Backup.app working:
Upon running Backup.app, a check of my Apache log files revealed that it was checking the URL "https://www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo". As before, we install a script there to capture the POST data and see what it's looking for. We end up with this in the temp file:
SERVER_SOFTWARE = Apache/2.0.50 (FreeBSD)
SERVER_NAME = www.mac.com
GATEWAY_INTERFACE = CGI/1.1
SERVER_PROTOCOL = HTTP/1.1
SERVER_PORT = 443
REQUEST_METHOD = POST
HTTP_ACCEPT = image/gif, image/jpeg, image/pjpeg, */*
PATH_INFO =
PATH_TRANSLATED =
SCRIPT_NAME = /WebObjects/Info.woa/wa/Query/accountInfo
QUERY_STRING =
REMOTE_HOST =
REMOTE_ADDR = 10.0.1.218
REMOTE_USER =
AUTH_TYPE =
CONTENT_TYPE = text/xml
CONTENT_LENGTH = 165
{
body = {keys = (iToolsBackupActivated, trialAccountDaysLeft); };
function = accountInfo;
header = {password = *******; username = mattsimerson; };
}
Once again, save the contents within the {} brackets into a temp file "foo" and post them to Apple's URL.. I did so with the following Lynx command:
lynx -source -post_data -useragent="Backup 2.0.2" https://www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo < foo
I got back the following results:
{
payload = {
iToolsBackupActivated = Y;
};
statusCode = success;
}
So, now we need to set up the accountInfo script to return that value when Backup queries it. I did so by editing the following file:
vi /home/www.mac.com/WebObjects/Info.woa/wa/Query/accountInfo
The contents of that file should be something like what follows:
#!/bin/sh
echo Content-type: text/plain
echo
cat << EOT
{
payload = {
iToolsBackupActivated = Y; trialAccountDaysLeft = -1;
};
statusCode = success;
}
EOT
After making this change, the first time you run Backup, it'll check your server, see that you have Backup activated and let you back up your Mac(s) to your local iDisk server. I don't know how often it performs this check but after having done so, I haven't seen it check again (unless I delete the Backup prefs file).
I now have iDisk and Backup access on all three of my computers (dual G5, PowerBook, and wifes iMac) without purchasing multiple .mac accounts. Excellent, this is better than .mac!
Configure a Proxy
Redirecting www.mac.com to my server creates a new problem in that now I can't visit www.mac.com from my LAN. Since I still do have a .mac account, I wish to retain that ability. To do so, I adjusted my Apache config file a little more by adding some proxy directives.
First, I edited my httpd.conf and uncommented the proxy_module, proxy_connect_module (for ssl), and proxy_http_module modules. I also added the following block:
<Proxy *>
Order Deny,Allow
Deny from all
Allow from 10.0
</Proxy>
The Proxy statement prevents the rest of the world from being able to access my proxy server. Publicly available proxy servers are a hazard to their owners and the rest of the internet. Be sure to secure yours!
I also added the following commands to the www.mac.com virtualhost containers:
ProxyRequests On
ProxyVia On
ProxyPass /WebObjects/Info.woa/wa/Query/accountInfo !
ProxyPass /WebObjects/Info.woa/wa/Query/retrieveDiskConfiguration !
ProxyPass / http://www.mac.com/
Here is my completed www.mac.com vhost config file for your reading enjoyment.
The one last loose end is adding iSync support so that I can use my own server to sync Address Book, iCal, and Safari bookmarks between my systems. Jeremy Baker has headed down that road so I expect to spend some time tinkering with that in the future.
NOTES
Platform Independent: This solution is not even remotely dependent on FreeBSD. I could have just as easily implemented this solution on my Linux or Mac OS X systems. I chose my FreeBSD server because it's already the gateway between my LAN and the internet. Because it's dual homed, I can access it locally on the LAN as well as remotely with my PowerBook without playing silly network tricks (like VPN or SSL tunnels).
iPhoto Homepage publishing: It's broken. I don't know why yet. I intend to figure out why at some point but don't hold your breath because it's not very important to me. I use Gallery with the iPhotoToGallery plugin and BetterHTMLExport with a template I customized.
iSync still works with .mac synchronization disabled but I lose the ability to sync between computers. That's a major loss.
Dominic also mentioned using disk images for users DAV space. Combine this with his script for fetching disk usage and you've got working per-user disk space reporting.
.mac, iCal, iDisk, iSync, and a few other things listed here are probably registered trademarks of Apple Computer.
Last modified on 7/11/05.
|
