What's the expected max memory usage?

[ssigwart]

It seems like when I do project wide finds, then open and edit a bunch of files, Komodo Edit gets really slow. It’s using about 1.1GB of memory, and virtual memory size is almost 12GB:

I have captures of the anon-memory-report.json.gz file, but I’m not sure I want to publicly upload that. In it, there are a bunch of lines about breadcrumbs.js . Here’s one using 76MB:

  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/add-ons/breadcrumbs@activestate.com/js-non-window/zones/zone(0x7f8e498ce400)/compartment([System Principal], chrome:\\\\breadcrumbs\\content\\sdk\\breadcrumbs.js (from: resource:\\\\gre\\modules\\commonjs\\toolkit\\loader.js:236))/classes/class(Function)/objects/gc-heap",
    "kind": 0,
    "units": 0,
    "amount": 80460576,
    "description": "Objects, including fixed slots."
  }

Also, I see a bunch of lines on findResultsTab.xul. For example:

{
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(none)/detached/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161952,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(none)/detached/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161952,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(none)/detached/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161952,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(none)/detached/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161952,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(none)/detached/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161952,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(none)/detached/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161952,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(none)/detached/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161952,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(none)/detached/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161952,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(none)/detached/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161952,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(chrome:\\\\komodo\\content\\komodo.xul, id=3)/active/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161968,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(chrome:\\\\komodo\\content\\komodo.xul, id=3)/active/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161968,
    "description": "Memory used by style sheets within a window."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(chrome:\\\\komodo\\content\\komodo.xul, id=3)/active/window(chrome:\\\\komodo\\content\\find\\findResultsTab.xul)/style-sheets",
    "kind": 1,
    "units": 0,
    "amount": 1161968,
    "description": "Memory used by style sheets within a window."
  },

Note that this is sorted by size:

jq '.reports' < memory-report.json | jq 'sort_by(.amount)'

@careyh, my main question is whether the amount of memory being used seems excessive and if it’s something I should try to investigate. If so, I have a few more questions:

1. Is the amount listed in each row just for that component or is it cumulative? I wonder because the bottom of the list shows this:

{
    "process": "Main Process (pid 1336)",
    "path": "page-faults-soft",
    "kind": 2,
    "units": 2,
    "amount": 28490173,
    "description": "The number of soft page faults (also known as 'minor page faults') that have occurred since the process started.  A soft page fault occurs when the process tries to access a page which is present in physical memory but is not mapped into the process's address space.  For instance, a process might observe soft page faults when it loads a shared library which is already present in physical memory. A process may experience many thousands of soft page faults even when the machine has plenty of available physical memory, and because the OS services a soft page fault without accessing the disk, they impact performance much less than hard page faults."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "js-main-runtime/compartments/classes/objects/malloc-heap/slots",
    "kind": 1,
    "units": 0,
    "amount": 29643712,
    "description": "Non-fixed object slots."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(chrome:\\\\komodo\\content\\komodo.xul, id=3)/active/window(chrome:\\\\komodo\\content\\komodo.xul)/js-compartment([System Principal], about:blank)/cross-compartment-wrapper-table",
    "kind": 1,
    "units": 0,
    "amount": 33554432,
    "description": "The cross-compartment wrapper table."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "js-main-runtime/compartments/cross-compartment-wrapper-table",
    "kind": 1,
    "units": 0,
    "amount": 40899456,
    "description": "The cross-compartment wrapper table."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "window-objects/style-sheets",
    "kind": 2,
    "units": 0,
    "amount": 57241824,
    "description": "This is the sum of all windows' 'style-sheets' numbers."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/xpconnect/runtime",
    "kind": 1,
    "units": 0,
    "amount": 67401584,
    "description": "The XPConnect runtime."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/add-ons/breadcrumbs@activestate.com/js-non-window/zones/zone(0x7f8e498ce400)/compartment([System Principal], chrome:\\\\breadcrumbs\\content\\sdk\\breadcrumbs.js (from: resource:\\\\gre\\modules\\commonjs\\toolkit\\loader.js:236))/classes/class(Function)/objects/gc-heap",
    "kind": 0,
    "units": 0,
    "amount": 80460576,
    "description": "Objects, including fixed slots."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "js-main-runtime/compartments/classes/objects/gc-heap",
    "kind": 0,
    "units": 0,
    "amount": 166260336,
    "description": "Objects, including fixed slots."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "js-main-runtime-gc-heap-committed/used/gc-things",
    "kind": 2,
    "units": 0,
    "amount": 245087464,
    "description": "GC things: objects, strings, scripts, etc."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "explicit/window-objects/top(chrome:\\\\komodo\\content\\komodo.xul, id=3)/active/window(chrome:\\\\komodo\\content\\komodo.xul)/dom/orphan-nodes",
    "kind": 1,
    "units": 0,
    "amount": 252631744,
    "description": "Orphan DOM nodes, i.e. those that are only reachable from JavaScript objects."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "window-objects/dom/orphan-nodes",
    "kind": 1,
    "units": 0,
    "amount": 252701920,
    "description": "Orphan DOM nodes, i.e. those that are only reachable from JavaScript objects."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "resident",
    "kind": 2,
    "units": 0,
    "amount": 2154266624,
    "description": "Memory mapped by the process that is present in physical memory, also known as the resident set size (RSS).  This is the best single figure to use when considering the memory resources used by the process, but it depends both on other processes being run and details of the OS kernel and so is best used for comparing the memory usage of a single process at different points in time."
  },
  {
    "process": "Main Process (pid 1336)",
    "path": "vsize",
    "kind": 2,
    "units": 0,
    "amount": 12887932928,
    "description": "Memory mapped by the process, including code and data segments, the heap, thread stacks, memory explicitly mapped by the process via mmap and similar operations, and memory shared with other processes. This is the vsize figure as reported by 'top' and 'ps'.  This figure is of limited use on Mac, where processes share huge amounts of memory with one another.  But even on other operating systems, 'resident' is a much better measure of the memory resources used by the process."
  }

That adds up to well over 12GB, so I’m thinking it’s combining some stuff. I’m wondering if that means the breadcrumbs and find results XUL lines aren’t really problems.

2. Do you have any idea how many breadcrumb lines should show per open file? How about findResultsTab.xul lines per search. I’m wondering if there’s a memory leak once files are closed.

3. I see https://github.com/Komodo/KomodoEdit/commit/ad03354b98167a7a8785acbe8c6cb3c777f29015 which is supposed to fix a memory leak in breadcrumbs. I see many more addEventListener calls than removeEventListener. Should there be the same number? If an event listener is not removed, will it be stuck in memory once the tab is closed?

4. Do you have any tips on tracking down memory leaks?

[careyh]

To answer your main question I’d say relatively speaking, for what Komodo does, 2GBs isn’t that bad when you consider some electron applications that don’t seem to be doing much use around the same or more. Having said that though, we know there are memory leaks in Komodo that we haven’t been able to track down so if you’re interested in digging, it wouldn’t be a waste of time.

1. If these are memory reports using Help > Troubleshooting > Memory Usage then I’d review them using that as well. Those last few that you listed look like “totals” though i’m not quite nerdy enough to know what they all represent. I note that the last two are the actual memory sizes reported in your screenshot though. ~2GB real memory and ~12GB Virtual Memory. 2GBs is high but not unheard of.

2. I just tried it and I have 16 entries for one files open. The entries break down every component of the breadcrumb so that seems normal. Again, i’d review the UI provided in Komodo. You can find breadcrumbs in the add-ons section near the bottom of the Explicit Allocations section. I didn’t try Find but i’m sure it’s the same idea and I imagine find tabs use more memory. I have no evidence telling me that there is a memory leak there. Review what it says in the Komodo UI and mess around with number of files/find tabs open and see what happens.

3. I only see one missing removeEventListener which shouldn’t amount to that big of a memory leak but you’re right, that would cause one. Can’t hurt to try adding the missing view_closed listener. I’d be interested in what you find there if you do it. If you do, could you open an issue for it; “investigating possible memory leak in breadcrumbs” or something like that.

4. You found the main one; Using the memory usage feature in increments as you push a specific feature of Komodo. eg. open 1 file, get memory usage dump, open a ton of files, get dump, close all but 1 file, get another dump, then see if the memory allocation recovers properly in the last one. You could also eliminate certain built in features completely by disabling them in Tools > Add-ons (Legacy).

I hope that’s help!

• Carey

[ssigwart]

Thanks for the tips, Carey.

2. That helps. I found it and it shows this:
   
   
   
   I still had files open at that point, but 196MB for breadcrumbs seems like a lot, especially when it’s nowhere near that large when throwing a ton of files at it in a test.
   From Per-class JS object and shape measurements in Firefox’s about:memory – Nicholas Nethercote, I see this:

If you have lots of memory used by Function objects and shapes, it suggests that the code is creating excessive numbers of closures.

3. There are a bunch of addEventListener calls in loops setting up menu items. That probably contributes to the number of functions using up memory. I’m trying to convert it to delegated events. I’ll open a pull request when I have it finished.

---

I haven’t looked much into these yet, but if you know off the top of your head, maybe you can help:

• Do you know what orphan-nodes are and if that might signal a leak?
  
• I’m assuming top(none)/detached means the elements are either a template or an old panel that is no longer shown, but still in memory. If the latter case, that might be a memory leak to investigate, right?
  

[ssigwart]

I added a pull request for the breadcrumbs.

Try To Reduce Memory Usage of Breadcrumbs

Looking into top(none)/detached, I wasn’t able to actually help anything yet, but I did notice that just opening and then closes the “About” or “Run” dialogs seem to wind up in here and leak memory each time they’re opened. I’m wondering if it’s more of a Mozilla issue than Komodo.

[careyh]

@ssigwart, Thanks for digging into this. Apologies for the delayed response. I was away from the office for a bit.

Will continue the conversation in the PR.

• Carey