Tutorial: Memory Management and Calculations for Clusters That Only Run MR1 Services

In this post, you will learn how memory is sliced and diced from the time you start warden, how memory is allocated to each MapR service installed on the node, and what memory is made available for MR1 for running jobs.   

MR1 MapR cluster would normally have JT and TT services running on the cluster apart from core MapR services. It's important that you understand how memory is managed and divided among the services installed on the node, and how to predict the outcome of tweaking config parameters to obtain desired results.  

Environment Details: My test setup (using default configurations):

  • MapR version =
  • Number of CPU = 32
  • Memory = 60 Gb
  • Number of Disks=6
  • Cluster (MR1 only)

You can check all the MapR services installed on the node by listing the services in the roles directory.


Memory is reserved and calculated for each service via memory parameters in warden.conf. Values for each service in warden.conf will determine how much memory will be committed to the configured services, as well as how much is reserved for general OS/administration. The leftover memory on the node will be assigned to MR1 for concurrently running Jobs.

On our test node, we have ZooKeeper, CLDB, Fileserver, JobTracker, TaskTracker and Webserver configured. Granted, that's a lot of services for one node, considering the memory on my test setup, but it’s fine for demonstrating how memory gets assigned to each service. Note that this is not recommended for production setup. Each service requires memory, so warden will be allocating percentages of the memory to each service chipping away and leftover memory is assigned for MR1 jobs on the node .

The figure below is a snapshot from the MCS screen, which provides a quick view of memory assignment to each service:

Dashboard => Node Properties => Manager Node services (Section)

memory management MR1

Understanding the Memory Calculations for Each Service:

So, how did warden decide to give the above memory as stated by MCS to each service? For our demonstration, let's look at JT: why was it assigned 4.9 GB? What was the calculation/formula behind this?

memory management mr1

Warden uses the formula below to internally calculate actual heap size depending on parameters set in warden.conf. If any changes are made in warden.conf, then warden needs to restart for the new setting to apply.

Actual heap size for service =max(heapsize.min, min(heapsize.max, total-memory * heapsize.percent / 100))

                                                   = Max (256 , min (5000 ,61440 * 10/100 ))

                                                   = Max (256 , Min (5000, 6144))

                                                   =5000 MB = ~4.9 GB

Similarly, we obtain memory for all individual services. From the calculations, memory assignment for each service would be:

  • OS : 3.9 GB  MFS  : 20.6 GB  Webserver : 750 MB  CLDB : 3.9 GB JT : 4.9 GB  TT : 325 MB
  • ZK : 615 MB  Warden : 615 MB
  • Total = ~ 35.5 GB  (Calculated value for memory consumed by all services)

Fileserver Memory Calculation:

One thing to note is that Fileserver memory in MCS is reported as 21.5 GB, but as per our calculation, it should be 20.6 GB, which is in line with values printed in warden.log below. We will see later in the post where this extra memory came from.

2015-05-26 18:33:54,953 INFO  com.mapr.warden.service.KVStoreService [main]: Before adjustment, memory for KVStore = 21097.0

Total Memory Consumed by Configured Services:

The lines below in warden.log verify total memory consumed by all MapR configured services = 36378 MB = ~35.5 GB which matches our earlier calculations for Total Memory consumed (35.5 GB) by the services.

2015-05-26 18:33:54,967 INFO  com.mapr.warden.WardenManager [main]: Total node memory: 61722920

2015-05-26 18:33:54,967 INFO  com.mapr.warden.WardenManager [main]: Total consumed memory: 36378

Total Available Free Memory:

Available free memory (totalGlobalAvailableMemory) for MR1 is 23898 MB, as indicated in the log line below. This value is calculated by the difference between “Total node memory” and “Total consumed memory”: 61722920 - 36378 = 23898 MB

2015-05-26 18:33:54,967 INFO  com.mapr.warden.WardenManager [main]: Still available free memory: 23898

There is one formula which is executed if the node is a 'zknode' or 'cldb' node (we are being conservative in reserving memory):

totalFreeMemoryInMB=round(max(totalGlobalAvailableMemory * 0.75, totalGlobalAvailableMemory - 1500))

                                            = round ( max (23898*0.75 , 23898 -1500 ))

                                            = round ( max (17923 , 22398 ))

                                            = 22398 MB

Memory for MR1:

The log line below in warden.log does validate total memory for running MR1 is “memoryForTasks = 22398” while memory for service in log line is memory which is allocated to the fileserver before any adjustment.

2015-05-26 18:33:55,285 INFO  com.mapr.warden.WardenMapRed [kvstore_monitor]: Before adjustment, memoryForTasks = 22398, memory for service = 21097.0

Resources on the Node:

Now that we have memory calculated for running MR1 jobs, while warden starts up, it also looks for CPU and disks available on the node, which can be used while running jobs. This is done in order to take all resources into account while doing slot calculations, to make sure there are no resource bottlenecks when jobs are run in parallel. The below log lines are from warden logs:

2015-05-26 18:33:54,967 INFO  com.mapr.warden.WardenManager [main]: Total node cpu cores : 32

2015-05-26 18:33:54,967 INFO  com.mapr.warden.WardenManager [main]: Total node mfs cpu cores : 4

2015-05-26 18:33:54,967 INFO  com.mapr.warden.WardenManager [main]: Total available cpu cores : 28

2015-05-26 18:33:54,967 INFO  com.mapr.warden.WardenManager [main]: Total node disks : 7

2015-05-26 18:33:54,967 INFO  com.mapr.warden.WardenManager [main]: Total node mfs disks : 6

This value is written to file /opt/mapr/server/data/cpu_mem_disk

memory management mr1

The next task for warden is to determine what resources should be available for MapReduce v1 jobs and MR2 jobs, respectively. This check is done since the below config in the warden.conf file is present by default in the 4.x MapR version (since MR1 and MR2 are supported on the MapR Data Platform). However, since this node doesn’t have Nodemanager configured, the config values in warden.conf are ignored, and TaskTracker will get 100% of the resources available to process MapReduce jobs.

memory management mr1

Note: I have a separate post where both MR1 and MR2 services are installed. I would recommend reading that post if you would like to understand memory allocation, in case both MR1 and MR2 services are installed on the same node.

Final Memory Calculations (MR1):

Since TT will get 100% resources, TT has 22398 MB memory to use for further calculation. However, we also want to make sure we don't have a bottleneck on disk or on CPU. For this, Warden internally calls the below command in the background. We can run the below command manually on the shell any time, as well to get optimum MR1 memory.

maprcli tasksmemory -noheader -memorymb <mem in mb>

memory management mr1

The output of this command gives the memory that MR1 tasks will use without causing any bottlenecks on the system resources when attempts are run in parallel. This is the value which is logged in warden.log as “After adjustment, memory for MR1 tasks = 21504” 

2015-05-26 18:33:57,161 INFO  com.mapr.warden.WardenMapRed [kvstore_monitor]: After adjustment, memory for MR1 tasks = 21504, memory for service = 21991.0

The steps involved when the above command runs is logged in /opt/mapr/logs/maprcli-mapr*.log


Note: If needed, I can also write a separate post explaining the steps involved in calculating how this command determines memory for MR1.

Final Fileserver Memory:

Now that MR1 memory is decided, warden calculates the difference between memory for tasks (before adjustment, memoryForTasks) and actual memory for MR1 (after adjustment, memory for MR1 tasks) and assigns this excess memory back to the fileserver.

Excess Memory = 22398-21504 = 894 MB

Hence, Fileserver Memory After calculation =  21097 + 894 = 21991 which does match below log lines from warden.log

2015-05-26 18:33:59,032 INFO  com.mapr.warden.WardenMapRed [kvstore_monitor]: After adjustment, memory for MR1 tasks = 21504, memory for service = 21991.0

2015-05-26 18:33:59,032 INFO  com.mapr.warden.service.KVStoreService [kvstore_monitor]: After adjustment, memory for KVStore = 21991.0

Key Takeaways:

1. Memory for each service is calculated from warden min, max and percent settings.

2. Warden will allocate percentages of the memory to each service chipping away memory; the leftover memory is for MR1 jobs.

3. If the node is a CLDB or ZK node, some memory is reserved based on formula below:

totalFreeMemoryInMB=round(max(totalGlobalAvailableMemory * 0.75, totalGlobalAvailableMemory - 1500))

4. Warden will determine what resources should be available for MapReduce v1 jobs and MR2 jobs respectively, based on config (mr1.memory.percent, mr1.cpu.percent, mr1.disk.percent) in warden.conf file. If NM is not configured on the node, 100% resources are available to MR1.

5. MR1 memory is calculated based on the formula below, which ensures that there is no resource bottleneck:

maprcli tasksmemory -noheader -memorymb <mem in mb>

6. The difference between memory for tasks and actual memory assigned to MR1 is given back to the Fileserver:

Fileserver Memory After calculation = Before adjustment, memory for Fileserver +  [(Before adjustment, memoryForTasks) - (After adjustment, memory for MR1 tasks)]

In this tutorial, you learned how memory is managed from the time you start warden, how memory is allocated to each MapR service installed on the node, and what memory is made available for MR1 for running jobs. Let us know if you have any feedback on this tutorial, or if you are running into any issues.


Streaming Data Architecture:

New Designs Using Apache Kafka and MapR Streams




Download for free