Timing it right

Status quo

During the Flash Player 10.1 time frame, I was tasked with taking a look at the timing system we use in the Flash Player. Until now the Flash Player has been using a poll based system. Poll based means that everything which happens in the player is served from a single thread and entry point using a periodic timer which polls the run-time. In pseudo code the top level function in the Flash Player looked like this:

while ( sleep ( 1000/120 milliseconds ) ) {
// Every browser provides a different timer interval
...
if ( timerPending ) { // AS2 Intervals, AS3 Timers
handleTimers();
}
if ( localConnectionPending ) {
handleLocalConnection();
}
if ( videoFrameDue ) {
decodeVideoFrame();
}
if ( audioBufferEmpty ) {
refillAudioBuffer();
}
if ( nextSWFFrameDue ) {
parseSWFFrame();
if ( actionScriptInSWFFrame ) {
executeActionScript();
}
}
if ( needsToUpdateScreen ) {
updateScreen();
}
...
}

The periodic timer is not driven by the Flash Player, it is driven by the browser. In case of Internet Explorer there is an API for this purpose. In the case of Safari on OS X is it hard coded to 50 frames/sec. Every browser implements this slightly differently and things become very complex quickly once you go into details. This has been causing a lot of frustration among designers who could never count on a consistent cross platform behavior.

Another challenging issue with this approach has been that limiting the periodic timer to the SWF frame rate is not acceptable. The problem becomes more obvious when you think of a SWF with a frame rate of let’s say 8 and play back a video inside which runs at 30 frames/sec. To get good video playback you really need to drive the periodic timer at a very high frequency to get good playback otherwise video frames will appear late. In the end the Flash Player always used the highest frequency available on a particular platform and/or browser environment.

The wrong path

The obvious way to re-architect this is to get rid of the polling and instead design an event based system. The new player code would have looked like this, with different subclasses of a Event base class encapsulating what the polling code had done before:

Event e;
while ( e=waitForNextEvent() ) {
e.dispatch();
}

This approach failed miserably:

  • CPU usage turned out to be much higher than expected due to the abstraction involved.
  • In some cases the queue would grow unbounded.
  • The queue needed a prioritization scheme which turned out to be almost impossible to tune properly.
  • Most SWF content out there depends on a certain sequence logic. Out of order events broke the majority of the SWFs out there.

It’s not all bad

Back to the drawing board. This time my focus was on the actual problem: The Flash Player polls up to 120 times second even if nothing is happening. Modifying the original code slightly I came up with this:


while ( sleepuntil( nextEventTime ) OR externalEventOccured() ) {
...
if ( timerPending ) { // AS2 Intervals, AS3 Timers
handleTimers();
nextEventTime = nextTimerTime();
}
if ( localConnectionPending ) {
handleLocalConnection();
nextEventTime = min(nextEventTime , nextLocalConnectionTime());
}
if ( videoFrameDue ) {
decodeVideoFrame();
nextEventTime = min(nextEventTime , nextVideoFrameTime());
}
if ( audioBufferEmpty ) {
refillAudioBuffer();
nextEventTime = min(nextEventTime , nextAudioRebufferTime());
}
if ( nextSWFFrameDue ) {
parseSWFFrame();
if ( actionScriptInSWFFrame ) {
executeActionScript();
}
nextEventTime = min(nextEventTime , nextFrameTime());
}
if ( needsToUpdateScreen ) {
updateScreen();
}
...
}

This approach is solving several problems:

  • There is no abstraction overhead.
  • In most cases it reduces the polling frequency to a fraction.
  • It is fairly backwards compatible.

More importantly, I replaced the browser timer with a cross platform timer which can wait for a particular time code. This not only yields better cross platform behavior, it also allows us to tune it in a way I could not do before. Which leads me to the most important change you will see in Flash Player 10.1: The way we behave when a SWF is not visible.

Implications for user experience

In Flash Player 10.1 SWFs on hidden tabs are limited resource wise. Whereas they would run at full speed in Flash Player 10.0 and before (note though that we NEVER rendered, we only continued to run ActionScript, audio decoding and video decoding), we now throttle the Flash Player when a SWF instance is not visible. Doing this change was not easy as I had to add many exceptions to avoid breaking old content. Here is a list of some of the new rules:

Visible:

  • SWF frame rates are limited and aligned to jiffies, i.e. 60 frames a second. (Note that Flash Playe 10.1 Beta 3 still has an upper limit of 120 which will be changed before the final release)
  • timers (AS2 Interval and AS3 Timers) are limited and aligned to jiffies.
  • local connections are limited and aligned to jiffies. That means a full round trip from one SWF to another will take at least 33 milliseconds. Some reports we get say it can be up to 40ms.
  • video is NOT aligned to jiffies and can play at any frame rate. This increases video playback fidelity.

Invisible:

  • SWF frame rate is clocked down to 2 frames/sec. No rendering occurs unless the SWF becomes visible again.
  • timers (AS2 Interval and AS3 Timers) are clocked down to 2 a second.
  • local connections are clocked down to 2 a second.
  • video is decoded (not rendered or displayed) using idle CPU time only.
  • For backwards compatibility reasons we override the 2 frames/sec frame rate to 8 frames/sec when audio is playing.

This marks a pretty dramatic change from previous Flash Player releases. It’s one of those changes which are painful for designers and developers but are unavoidable for better user experience. Let me show you a CPU usage comparisons with content running on a hidden tab (test URL was this CPU intensive SWF):

Flash Player 10.0

Flash Player 10.1

In this test case the frame rate in the background tab tab has been reduced to 8 frames/sec as audio effects are playing. If there was no audio the improvement would be even more pronounced. The test machine was a Acer Aspire Revo AR1600.

PS: You’ll notice in the two screen shots that the memory usage shows a quite dramatic difference also. That’s for another blog post.

Leave a comment

49 Comments.

  1. Great post thanks for the details!

    This should help a lot with many tabs open and stupid amounts of inefficient banner ads running

  2. Unreality 泡影

    thanks for this important info!

  3. Josh McDonald

    It's great to get some insight into Player internals! Just a couple of questions: Are all communications (http, and sockets) bound to jiffies now? Also does this limit affect Player in AIR?

  4. Thanks for this. I spent many hours trying to figure out what was going on under the hood (and doing experiments across 100s of machines to understand the differences between browsers). Now this explains it all, and makes programming Flash easier too!

  5. Thanks for this information. Question: do we get any event to know the swf is now going to go into this "throttled" state because its on a non active tab? It could also give a hook for developers to write more performant code, like pause timers or animation if the user is not looking at it anyway.

  6. Thanks for your explanations.

    It's so important to understand the way the player handles the core functionalities.

    Is the "invisibilty" of a SWF will be notified through an Event (we could cut sound, pause video or stop complex tasks) ?

    Denis

  7. I would love if embeds that are below the fold and not visible could do the same!

  8. @Arpit & Chiron: not yet but we are looking at providing a predictable API through a browser plugin API extension (that means we need to get buy in and implementation from all browser vendors). Right now the behavior is fuzzy and unreliable.

    @Geoff: if you scroll SWFs out of view the same behavior applies. It is not only SWF on hidden tabs.

  9. @ Ticnic

    Great to see this, but i think the "test URL" you provided is actualy not a good test example.

    Even on FP10, the CPU is down to 4% if the "tab is hidden" or if you are not moving your mouse, but thats only because the programers of this test implemented performance optimizations to stop rendering if the mouse doesent move.

    Also could you be so kind and explain why does "screen tearing" heappen in FP10, and how to prevent it?

  10. This may be a silly question… but what happens with hidden SWFs that get some message through the ExternalInterface? will they be affected?

  11. Hi Tinic,
    Thanks for the detail in this post. I’d like to check out swf’s will still operate as we intend please. E.g.
    http://www.vebra.com/vebra/property/20013990 – these swf’s need to keep running (animation and sound) even when the user scrolls the swf out of sight (they may inspect elements of the page below the swf). Can you please confirm whether this will be the case with 10.1 production release?
    Thanks Nick

  12. thanks for that, The comment that Arpit & Chiron are what I think would help. Especially with intensive activities like a video streaming, it would be great to be able to stop/close off a netconnection or sound if we had events to work off.

    At the same time if your swf isn't being used it should be scripted not to do anything hence use no or very little cpu.

    Thank again.

  13. Finally! This is awesome, thanks for this!

  14. Rhytms of Innovation

    Interesting, I just tested that on Win 7, chrome. While on the tab, it was 67%, off that tab, it grounded to 3%, still audio was playing. Excellent work!

  15. It would be nice to have a flag in DisplayObject that indicates whether the object is visible or not e.g. is it visible on the stage (so visible to the user). With such a property we could gain so much more performance. Is this even possible to have something like this? I've read somewhere that there was a bug/issue about this added to jira by Colin Moock.

    Best regards! Great work with fp 10.1!

  16. Richard Leggett

    Are you expecting a big disparity between 10.1 b3 on Safari vs Chrome (OSX).

    If I visit http://www.agencynet.com/#/home/

    Leaving it on the home page sees ~100% CPU usage in Chrome, ~120% in Safari (when switching to another tab this drops to ~38%).

    The interesting thing is on Chrome it is extremely jerky, I've noticed this on a lot of sites since installing the latest drop of 10.1, is this timing code causing a problem here? Safari may use more CPU but it's incredibly smooth throughout.

  17. Richard Leggett

    I should add Firefox behaves much like Safari, higher CPU usage, but no constant stuttering.

  18. Mario Klingemann

    Will there be any trick or workaround to keep a swf that is on a non-active tab or scrolled out of view from being throttled? I am thinking about situations where I have an application that is rendering or calculating something in the background that takes longer. Typically it will show a "processing…" progress bar which will make people want to switch over to a new tab to check their mail or browse around to pass the time. Of course with the new timer this will be fatal since the calculation will go to a crawl.

    I think what is needed for these cases is something like a PriorityTimer or a hasPriority flag on the regular Timer which makes them non-trottlable first class citizens like the Video class.

    As far as I understood the new "hasPriority" swf parameter does not keep a swf from being deactivated or does it?

  19. Unless I've misread this article, dropping intervals/timers to 2 f/s is going to cause problems. If code is relying on an interval callback to be invoked every 50ms or so, dropping everything to 2 f/s will result in that callback being invoked every 500ms. This means we will no longer be able to rely on intervals or timers, and will have to constantly calculate the delay time between callback invocations. If existing Flash apps/sites aren't already calculating the delay then the chances of them breaking when viewed with FP10.1+ is pretty high.

    While the intention of this change to the Flash Player is a good I don't think it's sensible.

    Please correct me if I'm wrong.

  20. It's really interesting to hear whats going on under the hood and how the player is being improved.

    Keep the posts coming!

  21. Thanks Tinic.

    How does this affect animations that are governed using timers rather than enterframes?

    Does this mean that timer based animations will "slow down" when the SWF is not visible?

  22. Yes, very glad to see this! Will minimizing a browser also reduce the framerate?

  23. Will minimizing the browser also reduce SWF framerates?

  24. It really explains the performance boost in 10.1. 10.1 should make on of the best Flash Player releases ever.
    Great job!

  25. I hope that it would be possible to de-ectivate this feature… If not how it would be possible, for instance, to use a Flash Based live audio player (live radio or audio player embedded in blog like the one from last.fm, or …). As soon as the user scrolls down or keeps the player in another hidden window, with the new approach the player will stop playing?

  26. Thanx for the update.

    BTW, it's already possible to tell if the user leaves the tab.
    Here is a good example:
    http://stackoverflow.com/questions/1760250/how-to-tell-if-browser-tab-is-active

    It's easy to find if the content is scrolled out of view, as well.

    A Flash API will make it easier though.

  27. Thanx for the update.

    BTW, it's already possible to tell if the user leaves the tab.
    Here is a good example:
    http://stackoverflow.com/questions/1760250/how-to-tell-if-browser-tab-is-active

    It's easy to find if the content is scrolled out of view, as well.

    A Flash API will make it easier though.

  28. Great info. Looking forward to hearing more about memory.

  29. Great stuff, couple of questions:

    Is this currently in 10.1 beta 3? Does stage.frameRate reflect the throttling?

  30. I would like to second the request for information on "screen tearing" in fp10.

  31. Will RTMP streaming be affected by this?

  32. Will this type of background throttling also effect AIR applications when the next runtime is released?

  33. If I read that correctly, it means we can't have more than 60 FPS (which you can right now – I've gotten up to 120).

    This isn't a big deal, but I wonder how that affects video. Is that painted independently from the main rendering loop's 60fps updates?

    Some of us have monitors that can do a multiple of 24fps refresh rates (like the new "3D" 120Hz LCDs or older CRTs at 72Hz), which can display 24fps content more smoothly than 60Hz displays can.

    Just curious. :-)

  34. i dont get it

  35. This changes will produce any glitch in background audio MP3 AAC decoding?

  36. Nate Chatellier

    As usual, genious Tinic!

  37. Is it possible to have loaded swfs (MovieClips) running at their own framerate?

    In the same way that Video is able to maintain fidelity, if you load a Flash banner ad into a Flash application will it be possible for that banner to run at a separate framerate to the main application?

    There was nothing in the post that lead me to think this is possible, but I figured you're the man to ask : )

  38. Chris Velevitch

    Does this mean you can now have a frame rate of zero frames per second or specify the number of seconds per frame?

  39. If the timing could be more consistent through the different browsers then a huge pain would be lifted from all of us game developers working in flash. This sounds promising indeed!

  40. Here is another suggestion:
    Is it possible to separate the loop into two threads?

    As you said, there are positives and negatives if it is turned into a total event-driven pattern. But what if turn part of it and take the positives while avoiding the negatives?

    For example, take the "UpdateScreen" method in to another thread, and trigger an event to it when the "logic" part is ready.

    Hence, it might be even possible to analyse which part should be optimized.

  41. @FantasyDog: Tried and failed. You end up spending large amount of time in kernel synchronization functions. In some rare cases you are faster overall but will suck even harder on that battery life for all normal cases. Avoiding contested mutexes and in the end sequential code is key on modern CPU designs.

  42. Tinic – I share Mario's concerns. Have you given any thought as to how these types of scenarios can be addressed? It would be nice if the new behavior was treated as default behavior, but with developers having the ability to override this for certain use cases.

  43. @Ryan: If you have a use case which can be isolated and checked for without developer control I am definitely willing to listen. My fear is that any workarounds for developers are ripe for abuse and become a default quickly, because 'it fixes something sometimes'.

  44. Very inspiring read. Having flash player responsibility to keep frame rate low in user specific scenarios is definitely good move. It obviously has its drawbacks but that makes me thinking, It would be interesting if fp would have different performance modes which developer could switch to. So in let’s say “low” mode it would shut itself while off the screen and would be restrictive in memory usage/battery consumption, threading and so on what would be ideal for advertisement. On other hand there could be “high” mode which would access most of resources to get performance boost what would be ideal for games or also having some special mode just for video etc. Just loudly thinking:)

  45. I wonder if it would make sense to cap the max fps to the users' monitor's refresh rate instead of the arbitrary 60fps? Most users would get the benefit of having it capped at 60 as you have said you will do, while others still retain the possibility of utilizing 120fps, or 72 or even 50 if that's what their screens run at.

    To address concerns about background throttling from Arpit, Mario and others, maybe we could get an event that fires when the content goes into low priority mode, that developers could use to warn their users (playing a sound or something) somehow that switching tabs or scrolling content is going to increase waiting times when processing is occurring.

  46. There are problems with mouse and keyboard events in FP10.1b3. I tried to play a couple of social games on my EeePC with beta 3 and had a stuck. FPS was good, but interactions was in process for 3-4 minutes(!!!).

    For example, you can see it here:
    http://blog.alternativaplatform.com/en/2009/12/25/mipmapping/

    I don't know, is it connected with the new timing system, but you should know :)

  47. @Tinic: A use case I can see is the ability for some invisible "helper" applications to run at full speed. An example of an application that would be degraded is a WebSocket implementation using Flash for the communications that would likely be hidden and thus throttled.
    There should be a way for the developer to specifically declare that an application requires full speed. Maybe the solution is to prevent throttling in some scenarios, like a 1 pixel application, network communications, ExternalInterface or LocalConnection calls.

  48. This could be problematic for applications that load hundreds of assets in sequence underneath a load screen. You probably know that the Event.COMPLETE from an asset load is only received at the moment an ENTER_FRAME event is happening. If you're trying to load 300 little files in sequence, there's at minimum a 300 frame wait. At 30fps, that's a bare minimum of 10 seconds for the load time. At 2fps, you're talking 150 seconds, which is two and a half minutes.

    How many users actually stick around and sit on a load screen, waiting for the load to finish? What you're proposing increases their wait time by 1500% assuming a stantard framerate of 30fps.

    Running a silent, endless sound in the background only lessens the damage by increasing the wait time by 375%.

  49. The 10.1 rc1 player results in not playable HD videos also on 10.6.3 and recent webkit on a Core2Duo 2,8 machine. The player 10.0 allowed to watch hd videos without problems on this machine.

    Can you fix this?

    Also rc1 is a mess on 10.5-systems… There are lot of screen desyncs. The graphics is stuttering.

    Do you think this will give Adobe a good reputation?