As it turns out, the typical install avoids popup blocker issues and does return you to your site. With the latest version, you can even append name and icon_src params to include more context about the web application you are coming from.
The generic install with the custom icon and name looks like this (not a real web application, just some placeholder text and image):
The actual page is here, if you want to check it out yourself, just view source. The code with the extra branding params looks like this:
location.href = "http://gears.google.com/?action=install&message=Please install Google Gears to enable offline access to my imaginary ultrasaurus application." + "&return=" + location.href + "&name=Ultrasaurus&icon_src=/dino.gif";
While I think the icon and name is a nice addition, the prominence of Google branding vs. the website using Gears seems like it should be inverted. PayPal does a good job of this. When you buy something from a site using PayPal, you end up on the PayPal site for some part of the transaction. However, the website branding is in the top header with the PayPal branding inside the content. For example, here’s a purchase page from ThinkGeek using PayPal:
Of course, what I really want to do is not direct the person using my app to whole different webpage, but just prompt for the download within my site. Maybe someday if they get the whole install working from an ActiveX control, it’ll be more seamless.