Posted by: digdug2k | July 10, 2012

Detecting taps using touch events

We recently shipped our first release of Firefox for Android with full multi-touch touchevents support, and I’d like to keep hearing from people if they find bugs or sites that don’t work. I recently had to debug and fix (for a second time) a bug involving Google maps, touch events, and tap detection. I thought it might be worth blogging here as well for future authors looking at this problem. Plus, Maps code is super obfuscated which made it tough to find the problem (both times).

Sidenote to try and avoid questions: Our users also don’t get some other nice Google Maps features like pinch-zoom and nice looking buttons. We fully support multitouch touch events in Firefox for Android, and pinch zoom works (last I checked) if you change your UA to match the stock browser or iPhone. The CSS problems are mostly due to the use of -webkit-background-size without other vendor prefixes. If someone wants to port Stylish to Fennec, it’d be a start to a prettier Google Maps experience for many of our users!

The Maps devs at some point realized they had all this nice code written to handle mouse events for their setup, and wanted to reuse it as much as possible. Ideally, touchevents and mouseevents would be interchangable and things would just work, but this is the system we’re kinda stuck with now. So the Maps team built functions that take touch events, convert them to mouse events, and rebroadcast them:

var sendConvertedTouch = function (eventType, touchEvent) {
  if (1 != b.changedTouches.length) return k;
    var mouseevent = document.createEvent("MouseEvents"),
             touch = b.changedTouches[0];
    c.initMouseEvent(eventType, true, true, window, true,
                     touch.screenX, touch.screenY,
                     touch.clientX, touch.clientY,
                     l, l, l, l, 0, null);
    c.translated = true;
    touch.target.dispatchEvent(mousevent);
  }
}

I doubt this is crazy performant, but it works! I’m guessing there are similar hacks all around the webs. This makes sending mouseup/down/move events simple.

It takes a bit more work to actually fire clicks when the user taps. There’s numerous tools and utilities out there to help with these things. But in this case the team decided to do their own thing. A super simplified version of the result looks something like:

var dragging = false;
document.addEventListener("touchmove", function(event) {
  dragging = true;
}, false);

document.addEventListener("touchend", function(event) {
  if (!dragging)
    sendConvertedTouch("click", event);
  dragging = false;
}, false);

(in their case, dragging actually holds a count of the number of taps).

This fails in current builds of Fennec. Why? Well essentially we don’t do a whole lot of filtering on the input we send to content. The system sends us a touch event, we send it on to content (we do some event coalescing if our internal event handling/drawing is slow the entire system doesn’t get backed up, but that’s about it). But touches on a screen actually jitter quite a bit especially on hdpi devices. Even when it feels like your finger is completely stationary, you’re probably still sending tons of touchmove events to the pages you’re viewing. So even though your finger hasn’t moved far, we send a touchmove event and maps refuses to click.

In addition, Fennec actually supports some of the Level 2 draft for touch events, which includes things like force and radius (NOTE: This spec is still in development and is likely to change, so use those with caution). So if a user presses on a single spot on the screen and then continues pressing harder and harder, we fire touchmove events for all those force/radius changes even though the users finger is still at (approximately) the same place.

All of that means the above code doesn’t actually work for us unless we filter out at least this very first touchmove event and don’t fire it until the user has actually moved their finger a significant distance on the screen. For now, that’s what we’ve done to make maps work for our users.

The correct code should check for some minimum distance after the initial touchstart event. If you get a touchmove that’s further away than that, you’re dragging! Ideally you’d have access to the device dpi here and you could use units of mm or something, but I can’t find a webapi for device dpi (yet?). This post is already getting too long, so I’ll leave that as an exercise to the reader.

“tap” events with some sort of number indicating how many taps have occurred in approximately the same place would probably be a useful addition to the touch events spec. (I noticed the other day that we actually have an internal MozGestureTap event that isn’t implemented in Fennec and isn’t exposed to web content (yet)). For now the EASIEST way to get around some of these problems is to use something like zepto, jquerymobile, or pointer.js which abstract this and other useful “gestures” for you.

Advertisements
Posted by: digdug2k | May 19, 2012

Porting Cleary to Native Fennec

I’ve had some requests to port Cleary over to Native Fennec (aka the hot new Beta for Android), and thought it would be a good chance to blog about addons in the native Fennec world. Cleary is a pretty simple addon (based on mfinkle’s BootstrapJones tutorials) that gave Fennec some more advanced clear history options, like the ability to only clear certain things:

  1. History (cache,history,downloads,formdata)
  2. Passwords (cookies,passwords,sessions,syncAccount)
  3. Site Preferences (offlineApps,geolocation,siteSettings)

and only clear them for certain time frames (One hour, Two hours, Four hours, a Day, and Forever). Porting it gives a chance to show off some new things we can do. Turned out, porting it was really easy though. In fact, a lot of the code it was using in XUL Fennec isn’t even necessary anymore thanks to some additions to the platform! Quick overview:

Step One: Updating compatibility

This steps pretty easy. We just need to mark Cleary as compatible on our install.rdf

<em:targetApplication>
  <Description>
    <em:id>{aa3c5121-dab2-40e2-81ca-7ea25febc110}</em:id>
    <em:minVersion>14.0b1</em:minVersion>
    <em:maxVersion>15.0</em:maxVersion>
  </Description>
</em:targetApplication>

Updating our UI

Cleary (luckily), is a bootstrapped extension and doesn’t include any overlays. While you could potentially use an overlay in Native Fennec, I’d highly discourage addon authors from doing it.

So to update our UI here basically means altering some scripts so that they only run on XUL Fennec. I decided an easy way to do that here was just to check the app id:

if (Services.appInfo.ID == "{a23983c0-fd0e-11dc-95ff-0800200c9a66}") {
  // setup xul fennec
} else {
  // do nothing!
}

Inside here the XUL Fennec code creates some buttons and rows for the Preferences pane. Overlaying things into prefs isn’t possible in Native yet, so we can basically do nothing! In some cases you might opt to use our NativeWindow interface to add UI elements. Instead, I’m opting to include some inline options for the addon.

settings.xul also happened to already be in Cleary. I just used some trickery to display it in the normal settings list. It also included some things we didn’t want, like a <settings> element, which I’m pretty sure Native Fennec will not like, so I removed that and just created a new options.xul in the root of the addon.

We also need to re hook up our click listeners to the clear buttons in the dialog. This requires a fix for Fennec that hopefully we can push into the beta. In bootstrap.js we need to add a listener for “addon-options-displayed” notifications:

Services.obs.addObserver(observer, "addon-options-displayed", false)

and a little observer to see them trigger:

var observer = {
  observe: function(aSubject, aTopic, aData) {
    if (aTopic == "addon-options-displayed" && aData == "cleary@digdug.org") {
      var doc = aSubject;
      var control = doc.getElementsByClassName("clearButton");
      for (var i = 0; i < control.length; i++) {
        var c = doc.defaultView.gChromeWin.cleary;
        control[i].addEventListener("command", c.doClear.bind(c), false);
      }
    }
  }
};

The old settings.xul was also localized using some JS trickery, which won’t work now. BUT!!! bootstrapped addons now support a chrome.manifest file, so we can add it and save ourselves some work. At the same time, I can remove a bunch of junk I had to register a resource URL and load style sheets. Yay for simplifying! We just add a chrome.manifest file with:

content cleary content/
locale cleary en-US locales/
style chrome://browser/content/browser.xul chrome://cleary/content/dialog.css

Functionality

Most of the functionality we previously had will just work in Native Fennec. Passwords, Form history, Cache, Cookies, Downloads, etc. All of its the same (and the same as desktop).

Except! browsing history is no longer handled in Gecko at all. Its all done in Java. Eventually Fennec should expose an interface for this, ideally that one that mirrors nsIBrowserHistory. I filed bug 731888 awhile ago for a slightly less hacky fix as well, but need to put up a correct patch there (volunteers welcome!).

For now, I had to hack into the database. This is NOT recommended. It will probably break eventually, and it will definitely make kittens cry but you can open the history database using the mozStorage service and tinker around in it. That may crash some phones where Android’s sqlite assuredly won’t match Gecko’s, so be warned! For history I added:

let file = FileUtils.getFile("ProfD", ["browser.db"]);
let dbConn = Services.storage.openDatabase(file);
if (this.range) {
  var statement = dbConn.createStatement("DELETE FROM history WHERE :from_date <= date AND date <= :to_date "
  statement.params.from_date = this.range[0];
  statement.params.to_date = this.range[1];
  statement.executeAsync();
} else {
  dbConn.executeSimpleSQL("DELETE FROM history");
}

And we’re done! There’s a build with this on Dropbox here, but it will not work until bug 756689 is fixed an checked in (and be warned its not on AMO because I’m still testing! Dataloss definitely possible!) There’s some bugs here I need to look into (clicks in the options dialog seem to fire multiple times and there’s some oddity with menulists), and I haven’t tested on XUL Fennec yet. For things that delete your data, there’s always lots and lots and LOTS of testing to do. But for the most part the port was pretty easy.

I’m pumped about Native Fennec. Its a great browser, and having extensions work in it is just icing on the cake. I’m excited to see what some good add-on developers can come up with to do with it in order to fix and enhance the mobile web.

Posted by: digdug2k | September 30, 2011

Downloadable Locales in Fennec

A new Firefox Mobile Aurora build has left, and with it the Locale Picker!

We noticed during development of Firefox Mobile that we were a bit larger than some other Android apps. We’ve written about why that is before, but one thing that’s eating up space in that little .apk file is locales. Android doesn’t let us filter what you download based on language (although they’ve started to add more options that way). So for our past few releases, we’ve put out a multi-locale version. At run-time, we look at which language your system is in. If we have it, we use it. Unfortunately, we can’t ship every language we support.

To get around this we’ve been working on a mechanism to let you download your favorite locale dynamically. Recently the last piece of this landed for users on our Nightly and Aurora channels to start testing. To turn this on, you need to point Firefox Mobile to a list of downloadable locales compatible with your version. Eventually, this will come from AMO, but I’ve put up temporary lists at:

To use these lists:

  1. Open about:config
  2. Search for the pref “extensions.getLocales.get.url” (it should show up if you search for “locales”)
  3. Change the pref to one of the above links (now that we support copy and paste, it might be easier to just open them and copy and paste it from the urlbar).
  4. You’ll have to restart Fennec for this pref to take effect (fixed in nightly – bug 689702)

Everything should continue like normal, but now when you go into our preferences section and select to change your locale, you will (after a second to download) see a list of downloadable locales. You can then pick the locale you want, click “Continue”, and the locale will be downloaded, installed, and set up for you automatically. Because these locales are actually distributed as add-ons, using them right now requires a restart of the browser (although we’re going to remove that soon), and voila! you should have Firefox  in the language of your choice.

Flow of the Fennec locale pickerThe Update problem

There are some things to worry about here. Nightly builds of Firefox Mobile are not string frozen. New strings can appear or disappear from day to day, and if you’re using a locale with missing strings, that can lead to a completely broken version (aka the yellow screen of death). To help avoid this, we disable the installed locales each day before we update, and try to download a new version if its available. Aurora users shouldn’t be seeing string breakage from day to day (Aurora is string frozen), but this is the time we encourage localizers to start updating their locales, so Aurora users should see the same disable and update occurring each day when they update.

When this feature moves to Beta and Release builds of Firefox Mobile , we distribute updates through the Android Market. This means that our updater code never runs to disable and update add-ons, but because the locales are only marked compatible with one particular build of Firefox, they’ll be marked incompatible by the Add-ons Manager. To help keep this from becoming to annoying, we also try to update any locales marked disabled by the add-ons manager each startup as well.

The last few pieces of this just recently landed, and its time to start letting our Nightly and Aurora users beat on the feature to test it out. Feel free to use the above prefs and localization files, and try Firefox Mobile out in a new language. Find problems. Report bugs (or email me, or send us a message in the #mobile channel on IRC). If your language isn’t available, feel free to jump in and help! Help us make sure that Firefox is great for users on mobile devices in every language. Thanks!

Posted by: digdug2k | May 2, 2011

Touch Events in Fennec Nightly Builds

This past Friday we checked in the initial frontend patches to Fennec to turn on single-touch TouchEvents in the mobile browser. As a result, you can now pan around on sites like Google or Bing Maps in Fennec! Apple added support for touchevents in mobile Safari early in the iPhone’s lifetime, but the W3C TouchEvent specification has only recently starting coming together. Mozilla’s mozTouchEvents will likely be deprecated in favor of the W3C standard specification.

Touchevents are necessary on browsers designed to be used with your fingers like Fennec. Our panning and double tap detection can make it difficult to forward typical mousedown, mouseup, and mousemove events to the client in a timely manner. For instance, in Fennec, we currently set the active pseudoclass on elements when they are first tapped (and use it for tap highlighting). If the user lifts their finger, without moving it very far from the initial mousedown, we wait a short time to make sure there isn’t a second (‘double’) tap, and if not we fire mousedown, mouseover, mouseup, and click simultaneously on the element. The delay makes it impossible to create pages that quickly respond to user input.

To help with this, in Fennec 4 we removed the time spent waiting for a double tap if the page has disabled user scaling in the metaviewport tag. This allows the page to quickly receive click events, but since we pan the page as your finger moves, we still couldn’t send mousemove events to the client. Touchevents provide the bridge between the two modes of interaction. Instead of registering for mouse events pages can register for touchstart, touchmove and touchend events. These seem very similar to typical mouse events, but have a few key differences. Touchevents don’t contain clientX/Y, pageX/Y etc coordinates directly on the event. Instead, the mouse coordinates are stored as an array of touchpoints and added to the touches property of the event. So, for instance, to use just the first touch or an event, you can call:

window.addEventListener("touchstart", function(aEvent) {
  var x = aEvent.touches[0].clientX;
  var y = aEvent.touches[0].clientY;
}, true);

In addition, touchevents also target themselves differently than regular mouse events. While touchdown is fired on the initially touched element, touchmove events will always remain targetted at that same target, even if the user moves their finger onto a different element. So if you want to write touchevents to drag an object around the screen, you can register a touchdown and touchmove listener on the object, and you will always receive the move events on the object, even if the user moves their finger onto a different element on the page. That’s how this example from quirksmode works.

When a page registers touchevents we attempt to detect it in the Fennec chrome and disable panning until the page has had  its chance to handle touchmove events. If the page author decides to call preventDefault on either touchstart or on the very first touchmove event, it will prevent Fennec from panning the page (we still allow panning the sidebars and urlbar off of the page, but will not allow you to pan them back on). It will also prevent any clicks that would normally be fired. This can pose a problem for Fennec where panning the page is necessary in order to gain access to the urlbar, and to the tab- and control-sidebars.

In order to continue to allow you to access the UI, and still allow pages to allow complex touchevents, we’ve currently implemented two escape hatches into Fennec:

  1. Pressing the menu button will show the urlbar, allowing you to move to a different page
  2. Pans starting at the very top of the page (within the top 1/2 inch) can not stop the panning of chrome

These will always bring in the chrome, and so you can trust that they are showing valid content.

Single-touch events are supported in nightly builds of Fennec (Androidother platforms) and should ship (fingers crossed) with Fennec 6. Support for multi-touch on Android is currently targeted at Fennec 7. Feedback is welcome. In particular, we’d like to ensure that turning these on hasn’t introduced noticeable regressions in panning for anyone, or if you can find places where our handling of preventDefault doesn’t agree with what is present in other modern mobile browsers. Test them out. Enjoy! Google Maps is likely the primary place where you will notice the work (NOTE: Maps has display problems partly due to scaling issues in Fennec, and partly because of Google’s use of webkit prefixed css without providing fallbacks for other browsers). I wrote a quick (and buggy) finger painting program to demo as well, but we’re excited to see what kind of nifty web applications and games people use this stuff for!

Special thanks to Matt Brubeck and Olli Pettay for their hard work within the WebEvents work group. Olli (smaug) also implemented the backend pieces necessary for Fennec to support touchevents. Matt has been hard at work documenting how touchevents work in different mobile browser (hint: they’re not all the same), and filing/fixing bugs to help us continue to bring Fennec’s implementation inline with as many as possible.

Posted by: digdug2k | March 25, 2011

Mozilla!

This is my first post as a Mozilla blog.

Posted by: digdug2k | February 13, 2011

Weekly Wesley Update 1/15

This week has been pretty uneventful around the house. I have continued my obsession with getting media I own into a form that’s easy to view on my TV. I spent a lot of the week at Mozilla fixing a “bug” where Gecko buffers 10 frames of an video element you insert in a page. That’s fine unless you’re on a resource limited device (Android phones don’t have much RAM to start with, and Android itself likes to eat what does exist), and loading a page with HD video (which PMO does frequently). Along the way I found a few other little bugs in Android and Fennec frontend code and fixed them. We now download files to a more appropriate location, and hopefully crash a little less.We’re mostly focusing on our upcoming Beta 5 release.

Valentines day is coming up, and I love my wife. Not a huge fan of the day myself. One year, for some reason we actually had to celebrate a few days after Valentines day, and happily the cost of everything she wants drops by 50% after the holiday. I’ve tried to make this a traidition, but it hasn’t quite caught on with her yet. Instead I think she would rather just celebrate twice. But like I said, I love her, so I’m gonna end this with a nice picture from when we were dating:

Posted by: digdug2k | February 7, 2011

Hell is Pfeiffer State Parks

This is my first post on WordPress. Not a lot to write about to start, but I want to put something, I’m working on a rant about Google TV, but its turning out super long, so this first post will not be about Google TV. In the interest of full disclosure, my wife works at Google. I work at Mozilla. Sometimes the companies are best friends. Sometimes they are fierce rivals. There isn’t any real reason those can’t both be true at the same time, but its confusing, especially in a house divided. I see the same thing with lots of families in the Bay area, and feel pretty confident we’ll survive.

The pic above is from last weekend when we drove down the coast to see Big Sur. There’s a funny story there. Might as well write it to start this off. There is a beautiful waterfall down that coast at a place named Julia Pfeiffer Burns State Park. The park is really pretty, and there’s this really easy nice trail leading to this:

You can’t really see the waterfall here at all, but it’s there off to the left, spilling off that cliff and onto the beach, and then trickling into the ocean. We made it barely at sunset on a pretty cloudy day, and while it was super pretty for us, our camera isn’t that nice. So this is what you get.

The story involves us being late, and the fact that, just 10 minutes north of Julia Pfeiffer Burns State Park is Pfeiffer Big Sur State Park, a place that is quite different. Pfeiffer Big Sur is kinda pretty too. It’s a bunch of forest, some camp sites, and a Ranger station manned by an angry park ranger who spends more time chatting with friends than helping you out. It’s also got a few signs, but most of them don’t face the direction you’re facing, are tiny, and probably don’t contain the information you want. Some of them tell you that trails are closed, but don’t have maps. Those that have maps don’t have helpful, “You are here” stickers, so you really can’t read them anyway. Pfeiffer Big Sur is a place for those looking to frustrate themselves. And if you manage to survive the frustration and find the trail, at the very end you are greeting by a separate, also beautiful waterfall:

Frustrated and tired after a long walk to this place, we asked the manager of a gift shop where the waterfall was. We specifically mentioned that this waterfall flows into the ocean. I even mentioned the name of the trail it is on (“Easy Waterfall Trail”) according to a book. Unfortunately, I forgot to bring the tour book and picture of the waterfall, and instead she sent us back up the same trail again. Luckily, I”m not dumb and didn’t fall for this trickery. We then found out that I am also stupid and had typed the wrong name into our GPS during the original trick. Luckily all worked out. We had a pretty day, but also one full of reminders that good UX design applies far beyond consumer electronics and vehicles.

« Newer Posts

Categories