A Bug's Life
Apr. 28th, 2020 12:18 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
sassa_nfI may just as well tell a story of one curious bug that fairly literally turned my life around.
I have only just embarked on a "serious job" outside academia. It was a compromise between me having to travel to Bristol, or my wife having to travel to London. My immediate team had three people in it, if you don't count the neighbours that used to borrow my shuttlecock at lunch time. The rest of the team, maybe 20 people, were all in the US. This is a Java shop.
At first it was some boring project that I cannot recall anything about. Everyone was doing something. Every Monday we had a "sync up" about things that left no impression on long-term memory. And then one day we had to "drop down and do twenty": there was an out of memory error and no one had a clue how to diagnose.
It is peculiar how senior engineers shove off complex problems they don't know how to solve to junior engineers. So I had to work it out.
I probably googled for what to do. Turned out usually you use a profiler to collect and analyse a heap dump. At this time a voice from above said: "Thou shalt not covet a license for a profiler, not even a floating license of thy neighbour."
There is some weird satisfaction in reinventing the wheel. You get to be the human that actually invented the wheel, but without the frustration of not knowing whether that's a good idea. On the other hand, if not for this experience, I would not know what the existing wheels are doing wrong.
The most important finding has been a fairly simple one, although very generic in its application. When you don't have the things you need, you need to work out what can be done with the things you have. Instead of asking the data to explain something you are curious about, ask what the data that you can see is trying to tell you.
In this case I was looking for ways to extract any data that does not require a profiler to take a look at. The JVM exposes a very simple thing, class histogram. After grappling with this for a little while, I realized that using profilers for leak detection is actually the wrong thing to do in the first place. A profiler can tell you where the bulk of the memory is retained, that's all. But it can't tell you where it is actually growing. All because it is looking at a snapshot of the state of the application - it is not looking at the evolution of the system state.
Two heap dumps can give you a difference.
Ten heap dumps can give you a trend. Well, you can't easily get ten heap dumps. But you can get trends from class histograms, and you can plenty of those at will.
Then you ask what types of trend analysis (distribution-free) is justified (total is a linear combination of instance sizes (constant (with a caveat)) and their corresponding instance counts (leak is here)). You don't want to perform analysis just because you can add and multiply. Of course, I thought about this after I added and multiplied and saw that it appears to be profitable.
Then you ask questions of the trends. You don't ask what you are after (what's consuming the memory? well, mostly byte[], char[], String, HashMap$Entry; generic things - generic answers). You ask why it looks like that, and what makes it different from any other application - other applications don't leak, despite using byte[], char[], String and HashMap$Entry just as much. Turns out the things you want to see are normally at the bottom of the histogram, due to their insignificant footprint. But the behaviour of the system can be inferred by watching whose trends match.
Then there is also an approach of tracking who allocates the objects responsible for the leak. That is also far from the answer. What you want to know, is who did not deallocate the objects. Unfortunately, no amount of object lifecycle tracking or code scan can tell you that. Because the line of code that should have deallocated the objects (or dropped the reference in the world with a GC), does not exist.
All because of the mean management that would not buy us a license.
I have only just embarked on a "serious job" outside academia. It was a compromise between me having to travel to Bristol, or my wife having to travel to London. My immediate team had three people in it, if you don't count the neighbours that used to borrow my shuttlecock at lunch time. The rest of the team, maybe 20 people, were all in the US. This is a Java shop.
At first it was some boring project that I cannot recall anything about. Everyone was doing something. Every Monday we had a "sync up" about things that left no impression on long-term memory. And then one day we had to "drop down and do twenty": there was an out of memory error and no one had a clue how to diagnose.
It is peculiar how senior engineers shove off complex problems they don't know how to solve to junior engineers. So I had to work it out.
I probably googled for what to do. Turned out usually you use a profiler to collect and analyse a heap dump. At this time a voice from above said: "Thou shalt not covet a license for a profiler, not even a floating license of thy neighbour."
There is some weird satisfaction in reinventing the wheel. You get to be the human that actually invented the wheel, but without the frustration of not knowing whether that's a good idea. On the other hand, if not for this experience, I would not know what the existing wheels are doing wrong.
The most important finding has been a fairly simple one, although very generic in its application. When you don't have the things you need, you need to work out what can be done with the things you have. Instead of asking the data to explain something you are curious about, ask what the data that you can see is trying to tell you.
In this case I was looking for ways to extract any data that does not require a profiler to take a look at. The JVM exposes a very simple thing, class histogram. After grappling with this for a little while, I realized that using profilers for leak detection is actually the wrong thing to do in the first place. A profiler can tell you where the bulk of the memory is retained, that's all. But it can't tell you where it is actually growing. All because it is looking at a snapshot of the state of the application - it is not looking at the evolution of the system state.
Two heap dumps can give you a difference.
Ten heap dumps can give you a trend. Well, you can't easily get ten heap dumps. But you can get trends from class histograms, and you can plenty of those at will.
Then you ask what types of trend analysis (distribution-free) is justified (total is a linear combination of instance sizes (constant (with a caveat)) and their corresponding instance counts (leak is here)). You don't want to perform analysis just because you can add and multiply. Of course, I thought about this after I added and multiplied and saw that it appears to be profitable.
Then you ask questions of the trends. You don't ask what you are after (what's consuming the memory? well, mostly byte[], char[], String, HashMap$Entry; generic things - generic answers). You ask why it looks like that, and what makes it different from any other application - other applications don't leak, despite using byte[], char[], String and HashMap$Entry just as much. Turns out the things you want to see are normally at the bottom of the histogram, due to their insignificant footprint. But the behaviour of the system can be inferred by watching whose trends match.
Then there is also an approach of tracking who allocates the objects responsible for the leak. That is also far from the answer. What you want to know, is who did not deallocate the objects. Unfortunately, no amount of object lifecycle tracking or code scan can tell you that. Because the line of code that should have deallocated the objects (or dropped the reference in the world with a GC), does not exist.
All because of the mean management that would not buy us a license.
no subject
Date: 2020-07-16 10:33 pm (UTC)> turned my life around
И?
no subject
Date: 2020-07-17 01:25 pm (UTC)