RFE: Specify worker/executor memory under 'advanced cluster configuration'

[erikerlandson]

Executor memory defaults to 1 GB, which is a good default but it would be nice to have a convenient way to increase that when creating a cluster on the webui.

[crobby]

Is this something that is typically done via a configmap on the worker nodes? If so, you can specify a worker configmap on the advanced form already. Is this RFE just to make that process easier (ie: specify the executor memory on the form, which in turn will create the configmap under the covers and use that)?

[erikerlandson]

It should be configurable via config-maps as spark.executor.memory. It may be that we want to adhere to that convention. I should try it and see how easy it is.

[erikerlandson]

Just thinking out loud here... on one hand it feels a bit heavy-weight to generate a config-map and apply it, for the purpose of setting a single property. On the other hand, we don't really want to get sucked into the business of adding dozens of fields in the UI, corresponding to a large and evolving list of spark properties.

I wonder if it is possible to just add a text input where a person could add spark property settings:

spark.executor.memory = 4096
spark.executor.cores = 4
spark.ui.enabled = false
... etc

Then have the logic of config-file creation run automatically. Easy button style, I guess.

[crobby]

Maybe we can bake in some extra param on the oshinko-cli (which is what the webui uses under the covers) to specify configs like that. In turn, they can be used to either create a new configmap, or maybe even append to another configmap that is in use.

[NecromuncherDev]

Is there a possibility to define these criteria on the cli?
If not, where sould I look in the code in order to integrate such capabilities?

[crobby]

I don't believe there is currently a specific flag in the cli to make this happen. I think that executor memory can be set in the spark configuration though and that functionality (overriding spark config defaults) is supported by both the cli and webui (the configmap would need to be created outside of the webui, either via the oc cli or the openshift console).

[crobby]

Ok, here's a partial "how to" for at least part of this. (We'll be creating another issue to make this answer complete and more obvious in the near future).
https://github.com/radanalyticsio/oshinko-cli/blob/master/rest/docs/ClusterConfigs.md documents how a configMap can be created to override some or all of the spark configuration.
https://spark.apache.org/docs/latest/configuration.html has a listing of all the possible configs. You should be able to include any of those in the configmap that you create and then ask the cli or webui to use that configmap to spin-up your cluster.

[NecromuncherDev]

I'm sorry in advance if this sounds stupid but after generating a new configmap and using it with the cli like this:

oc create configmap m1-c4  --from-literal=spark.executor.memory=1g --from-literal=spark.executor.cores=4

oshinko create my-m1-c4 --workers=2 --workerconfig=m1-c4

After seeing that the pods span-up alright I peaked at the spark master webui and it said there that a total of only 4 cores are available instead of the 8 configured (4 cores * 2 workers).

Am I doing something wrong?

P.S. - if in your oppinion this should be under a seperate issue just say the word

[elmiko]

i think this has to do with how oshinko is provisioning the workers and the way that openshift assigns the cores to each container.

from the spark docs:

The number of cores assigned to each executor is configurable. When spark.executor.cores is explicitly set, multiple executors from the same application may be launched on the same worker if the worker has enough cores and memory. Otherwise, each executor grabs all the cores available on the worker by default, in which case only one executor per application may be launched on each worker during one single schedule iteration.

i have a feeling that since we are not using kubernetes as the cluster scheduler that the workers in your case are doing what is described here. since we do not change the underlying container spawn resources, those containers will still have the default number of cores (and memory) defined by openshift policy.

i'm not sure what the proper solution here is, but this sounds like more about the way spark interacts with openshift for creation of resources. this is a good gap to highlight though, thanks!

[tmckayus]

@ThatBeardedDude

Really happy to have you using oshinko, it's especially great to get feedback from someone who is looking with fresh eyes.

Our cluster configmaps for the master and worker are organized around spark config files, so you can't actually drop config values directly in a configmap, they have to be contained in a file. The easiest way to do this in this case is to put your configs in a file and use --from-file

1. create spark-defaults.conf file that contains what you want
2. oc create configmap workerconfig --from-file=spark-defaults.conf

This will cause OpenShift to create a configmap where the filename is the key and the file contents are the value, and when it's mounted as a configmap volume the files will be written to /opt/spark/conf.

Likewise, you can put multiple config files in a directory and do

oc create configmap workerconfig --from-file=myconfigdir

and each file will be added to the configmap

Let us know if this works for you.

[NecromuncherDev]

I will! Thank you @tmckayus very much for this clarification!
I can't imagine how much wasted time is saved with this simple change in configmap creation method.

Maybe this should be added to the propper README as well :)

[tmckayus]

@ThatBeardedDude

noted :) We have a README about this but it needs to move, probably to the rad.io landing page in a HowTo. I'll add one

[tmckayus]

@ThatBeardedDude as promised, this will show up on the landing page. Good?

radanalyticsio/radanalyticsio.github.io#187

[NecromuncherDev]

Amazing!

[NecromuncherDev]

This post is a problem I am facing while using Oshinko v0.4.6 - if this is solved at a current unreleased state, I'll be glad to see the solution

Regarding the gap @elmiko and I discussed earlier, I set up a cluster (following step by step after the README @tmckayus put up) with this configmap:

spark.executor.memory 4g
spark.executor.cores   1

Sadly, I was disappointed to see this in the openshift pod logs:

18/04/23 13:52:30 [INFO] Utils: Successfully started serveice 'sparkWorker' on port 35716.
18/04/23 13:52:30 [INFO] Worker: Starting Spark worker x.x.x.x:35716 with 2 cores, 1024.0 MB RAM.

Meaning the oshinko create simply ignored my configmap. Any ideas?