Report on Mozilla's Memory Management Development Practices

This report is a fresh perspective on the state of certain development practices at Mozilla. Specifically, I report on the development practices associated with memory management in the code for the Firefox web browser and the SpiderMonkey JavaScript engine.

I wrote this in May 2008 after four months of involvement with Mozilla.

Analysis of Mozilla's Memory Management Development Practices

I want Mozilla to make better software, faster. This report identifies opportunities for Mozilla to improve some of its development practices. These recommendations are all related to issues of memory management, my area of specialty.

I have only a little experience with Mozilla code, so I am humble in my recommendations. I make them in the hope that I can give the issues a fresh perspective.

The first section describes what Mozilla is already doing well. The second section gives some recommendations for future direction.

Already Accomplished

Here, I highlight some of Mozilla's established best practices.

Best Practice: Respect memory management problems

Memory leaks seems to be a priority at Mozilla. This aspect of development culture is important.

The organization has made custom tools (http://wiki.mozilla.org/Performance:Leak_Tools) for finding memory leaks, and has begun to document those tools and techniques for using them (http://developer.mozilla.org/en/docs/Debugging_memory_leaks). As a whole, the tools have worked to reduce Mozilla leakage (http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/).

Best Practice: Use a custom malloc

Already, Mozilla has taken this step. Using a custom malloc(3) subsystem helps promote consistent behaviour across platforms. In addition, the 'jemalloc' subsystem selected by Mozilla has significantly superior performance to the standard malloc subsystem on all platforms (except maybe Mac OS X). It is faster, and memory fragmentation is reduced.

Also, it is wise for Mozilla to have access to a specialist (Jason Evans, the 'JE' in 'jemalloc') who can modify the subsystem easily in response to the organizations development and debugging needs.

It seems that Jason is in the process of adding (in May 2008) robust memory checking code in the allocator - memory walls, setting the contents of freed memory to a known constant, etc. This effort is an investment which will pay off in the form of fewer memory-related problems in the future. Developers will catch problems earlier.

I hope that Mozilla will eventually make the jemalloc subsystem its own well-maintained open source project, separate from Firefox.

Best Practice: Use more garbage collection

In code which is not performance sensitive, it is helpful to use a garbage collector to manage the memory and to prevent leaks and fragmentation. It frees developers from most of the worries of memory management. Further, the garbage collector can be instrumented more easily than widely dispersed malloc/free (or new/delete) code. (The instrumentation can help with leak finding, memory metrics, and performance metrics.)

Mozilla is moving in this direction, also. New code which is written in JavaScript satisfies this recommendation, since the JavaScript runtime is garbage collected.

Future Direction and Opportunities for Improvement

This section describes what Mozilla should improve.

Recommendation: Improve leak-finding tools and docs

Improve and augment documentation about the various leak-finding tools at http://wiki.mozilla.org/Performance:Leak_Tools and
http://developer.mozilla.org/en/docs/Debugging_memory_leaks.

Support the 'leaky' tool (http://lxr.mozilla.org/mozilla/source/tools/leaky/leaky.html), or a similar one, on all platforms, instead of just Intel/Linux, and add it to a regression suite.

Improved tools and documentation will make it more likely that developers and QA personnel will use the tools. The tools will be faster to use, resulting in less developer time spent chasing leaks. Documentation is essential to helping developers understand what the tools can and can't do.

Recommendation: Improve Talos's Memory Consumption Metrics

Currently, the techniques for measuring Firefox's memory consumption are primitive. While using performance counters on Windows might be appropriate, using the output of 'top' on Mac OS X is not. There are more meaningful metrics to collect. Further, the metrics could be made more fine-grained. For example, we could differentiate between memory consumed by SpiderMonkey, versus NSPR, versus Gecko, versus NSS, versus chrome, etc. (See "Recommendation: Use memory colouring", below.)

Modifying Talos to track regressions in the newly-finely-grained memory consumption metrics would be an important part of this.

Recommendation: Use memory colouring

Memory colouring is a technique for assigning "owners" to each memory allocation. Each owner is a colour. One can assign different colours to different allocations based on the purpose of the allocation. This requires instrumenting the code with "colour" annotations. There are ways to automate this process.

Memory colouring allows the fine-grained analysis of memory consumption. We get to see who's consuming how much memory. We get to track changes in memory consumption over time component-wise (by colour). We know who to blame when memory consumption increases.

Recommendation: Use more static analysis (DeHydra) scripts

Taras Glek (http://blog.mozilla.com/tglek) and David Mandelin (http://blog.mozilla.com/dmandelin) are working on projects which involve automated compile-time analysis of Mozilla code. Changing memory management practices in large code bases benefit from static analysis. These efforts should be supported. In particular, the analysis should be extended to perform automated transformation, also. (Some other tools do this, for example TXL .) Already, static analysis is helping change memory management code in Mozilla (eg. https://bugzilla.mozilla.org/show_bug.cgi?id=427115).

Recommendation: Investigate use of zone allocators

Experiment with "zone allocators" for grouping allocations with similar lifetimes closer together in memory. This may result in much less fragmentation due to freeing large amounts of memory, such as when a Firefox window/tab is closed.

Recommendation: A Mozilla-backed memory allocation library

Make the jemalloc subsystem its own well-maintained open source project, separate from Firefox. Mozilla should support the project. Enterprise-class features in a malloc(3) subsystem are important, but are lacking in all open-source malloc(3) libraries. Most such libraries exist to exhibit different performance (speed and fragmentation) characteristics, as opposed to exhibiting important enterprise-class features like variable-sized memory walls, memory colouring, or zoned allocations.

Recommendation: Make a decision about NSPR

Firefox is heavily dependent on the NSPR library, a separate Mozilla project. NSPR seems to have a development schedule which is not keeping pace with its dependents, unfortunately. Perhaps that is one reason that external ports of NSPR (http://nspr.darwinports.com/) and other open source platform abstraction layer libraries have come into existence (http://apr.apache.org/).

Firefox needs changes from NSPR faster than NSPR can provide them. This happens because NSPR has too many impediments to development, and too few developers and development resources.

NSPR has been ported to many more platforms than has Firefox. So, to maintain the cross-platform portability, more work is required to add features in NSPR than is required in Firefox. Sadly, NSPR seems to lack the build/test facilities to test on all of the platforms that it supports. I recommend that a decision be made to follow one of the following courses of action.

Option 1: slowly begin a migration away from dependence on NSPR. Give up on NSPR's development. Add new features only in Firefox, even if their logical place would otherwise have been NSPR. Firefox should aim to reduce or eliminate dependence on NSPR.

Option 2: Invest in NSPR's development by allocating more resources to that. NSPR needs testing machines and attention. Make NSPR's test suites more robust. Create an NSPR performance regression suite, especially for locking and memory management code. Provide performance stats to help developers make decisions. Add the performance and feature enhancements that Firefox developers need. Ensure that NSPR development keeps pace with Firefox development.

To fail to choose would mean that Firefox's development would remain impeded whenever the development is related to NSPR.

Recommendation: Develop Prism

This is an excellent long-term direction to take. Invest in Prism (http://labs.mozilla.com/2007/10/prism/). With Prism, web apps that would otherwise all share the same address space (Firefox's) can use separate address spaces. Separate address spaces would relieve the memory pressure on Firefox.

Problems with fragmentation and memory consumption within each process would be reduced. This is an excellent way to respond to the fact that users are keeping more and more browser windows open - let the user decide which window gets its own process.

Prism provides other benefits, all well-explained at the above URL.

Update: Mozilla has moved forward with Prism. See http://prism.mozilla.org.

Epilogue

Those in the Mozilla development community should feel free to contact me if they want some more information about these recommendations, or if I can clarify anything in this report.