KEP-2170: Implement JobSet, PlainML, and Torch Plugins #2308

andreyvelich · 2024-10-24T22:25:25Z

Part of: #2290.

I implemented JobSet and PlainML plugins, and I was able to submit our first TrainJob and ClusterTrainingRuntime 🎉

Working ClusterTrainingRuntime

apiVersion: kubeflow.org/v2alpha1
kind: ClusterTrainingRuntime
metadata:
  name: first-runtime
spec:
  mlPolicy:
    numNodes: 5
  template:
    spec:
      replicatedJobs:
        - name: trainer-node
          template:
            spec:
              template:
                spec:
                  containers:
                    - name: trainer
                      image: python:alpine3.20
                      command:
                        - "/bin/sh"
                        - -c
                        - "echo 'Hello From Training Runtime'"
---
apiVersion: kubeflow.org/v2alpha1
kind: TrainJob
metadata:
  name: first-job
spec:
  runtimeRef:
    name: first-runtime
    kind: ClusterTrainingRuntime
    apiGroup: kubeflow.org
  trainer:
    numNodes: 2

A few details:

We discussed with @tenzen-y that Info object doesn't need to have Obj class, since we can directly pass the JobTemplateSpec into RunComponentBuilderPlugins() API.
I added the Trainer struct in Info object which represents values combined from MLPolicy and TrainJob. Those values will be inserted into JobSet.
I need to update the unit tests in this PR.
I need to implement torch plugin.
I need to implement env variables enforcement.

/assign @kubeflow/wg-training-leads @varshaprasad96 @akshaychitneni @deepanker13 @helenxie-bit @Electronic-Waste @saileshd1402 @kannon92 @kuizhiqing @shravan-achar

andreyvelich · 2024-10-24T22:25:50Z

/hold for review

pkg/runtime.v2/runtime_test.go

coveralls · 2024-10-25T22:25:22Z

Pull Request Test Coverage Report for Build 11617265788

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals
Change from base Build 11542660312:	0.0%
Covered Lines:	77
Relevant Lines:	77

💛 - Coveralls

pkg/runtime.v2/core/trainingruntime_test.go

andreyvelich · 2024-10-26T16:22:48Z

Torch Runtime is working 🎉

Simple Torch runtime

apiVersion: kubeflow.org/v2alpha1
kind: ClusterTrainingRuntime
metadata:
  name: torch-runtime
spec:
  mlPolicy:
    numNodes: 2
    torch:
      numProcPerNode: "1"
  template:
    spec:
      replicatedJobs:
        - name: trainer-node
          template:
            spec:
              template:
                spec:
                  containers:
                    - name: trainer
                      image: docker.io/andreyvelichkevich/test-torch
---
apiVersion: kubeflow.org/v2alpha1
kind: TrainJob
metadata:
  name: first-job
spec:
  runtimeRef:
    name: torch-runtime
    kind: ClusterTrainingRuntime
    apiGroup: kubeflow.org
  trainer:
    numNodes: 3
    numProcPerNode: "5"
    env:
      - name: TRAIN_JOB
        value: env

tenzen-y

Sorry for the delay review.
I did not review any testings yet.

pkg/constants/constants.go

pkg/runtime.v2/framework/core/framework.go

pkg/runtime.v2/framework/plugins/coscheduling/coscheduling.go

pkg/runtime.v2/framework/plugins/jobset/builder.go

pkg/runtime.v2/framework/plugins/torch/torch.go

pkg/runtime.v2/framework/interface.go

tenzen-y · 2024-10-28T18:57:54Z

pkg/runtime.v2/runtime.go

 	TotalRequests map[string]TotalResourceRequest
 }

+type Trainer struct {


Do we still need the Policy?

Actually, I am still getting some data from Policy in the EnforceMLPolicy: https://github.com/kubeflow/training-operator/blob/f00c1d37ef03d89214ddd9a7189f98f81fce2276/pkg/runtime.v2/framework/plugins/plainml/plainml.go#L50

Do you think, that we should just remove Policy API from Info object and bypass the TrainingRuntime to the enforcement APIs ?

I refactored the Info object to remove Policies: https://github.com/kubeflow/training-operator/blob/e5165cbff046dba921a1f372293b73ad52888ae6/pkg/runtime.v2/runtime.go#L26-L31.
Right now, we pass MLPolicy and PodGroupPolicy spec from the Runtime directly to the Enforce() APIs.

I think that the Trainer needs to have scheduler since we need to create the PodGroup and do some else scheduling logic.
If we create the PodGroup against a whole of TrainJob, that will not schedule to any Node for ever. Because we need to start the initializer before the Trainer.

That makes sense.
Does it mean that we should apply the coscheduling PodLabel only to the Trainer Job, not to the Initializer Job ?

Does it mean that we should apply the coscheduling PodLabel only to the Trainer Job, not to the Initializer Job ?

Yes. When users want to take the intializers separate from trainer, they want to specify another PodGroup name to initializers.

I see, do you want to update this change in the followup PR once I add the builder for Dataset and Model ?

I see, do you want to update this change in the followup PR once I add the builder for Dataset and Model ?

This can be followed up.
I will take it during you impl Datase initializer since this is obviously regressions as I mentioned the above.
The current impl bring us a whole of Job will be stuck forever.

pkg/util.v2/testing/wrapper.go

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Update error return Use ptr.Deref() for nil values Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

pkg/runtime.v2/core/trainingruntime.go

pkg/runtime.v2/core/trainingruntime_test.go

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

pkg/runtime.v2/framework/plugins/plainml/plainml.go

pkg/runtime.v2/framework/plugins/torch/torch.go

akshaychitneni · 2024-10-30T00:09:40Z

pkg/runtime.v2/core/trainingruntime_test.go

 					MinResources(corev1.ResourceList{
-						corev1.ResourceCPU: resource.MustParse("60"),
+						corev1.ResourceCPU: resource.MustParse("31"), // Every replica has 1 CPU = 31 CPUs in total.


I think we should have a utility function to calculate the quantity passing in the obj

We already have it here to calculate PodRequests based on the PodSpec: https://github.com/kubeflow/training-operator/blob/master/pkg/runtime.v2/runtime.go#L121-L124.

Do you mean having another utility just for the unit tests ?

yes utility to compute value in tests

I was not sure the reason why @akshaychitneni requests to implement the util.

@akshaychitneni Please can you give an example of code snippet that you want to use instead of:

corev1.ResourceCPU: resource.MustParse("31")

I was referring to calculating 31. Probably it is simpler as it is (numNodes + 1) * (Num of CPU configured in the runtime) in this case

I see, I think it depends on the unit test. For example, we have unit tests where we add CPU + Memory for the Pod requests: https://github.com/kubeflow/training-operator/blob/85d0cf1fca0e56d206e579f77d68f33aca1a0f87/pkg/runtime.v2/framework/core/framework_test.go#L359-L360
Additionally, in the future we might want to define different resources for initializer containers and trainer to verify that we correctly calculate MinResource for the PodGroup.

Let me add the TODO statement to see if we can create helper function for our tests to calculate these values based on number of replicas and resources per replica.

Actually, container resource computation is not easy, we need to consider runtimeClass (not training runtime, it's core API runtimeClass), limitRange, and initContainer and so on. So, I would like to keep manually resource creation for the testing.

If we provide such utils with complex computing, the lib could potentially bug, and we will miss the bugs in main testing cases.

tenzen-y

I submitted the production code review comments, first.

pkg/constants/constants.go

hack/python-sdk-v2/openapi-generator-cli.jar

pkg/constants/constants.go

pkg/runtime.v2/core/trainingruntime.go

tenzen-y · 2024-10-30T18:55:07Z

pkg/runtime.v2/core/trainingruntime.go

+
+	info := runtime.NewInfo(opts...)
+
+	if err := r.framework.RunEnforceMLPolicyPlugins(info, trainJob, mlPolicy); err != nil {


We do not want to surface the mlPlicy to the JobPipeline framework directly since the framework is not only for the TrainingRuntime.
So, could you move the mlPolicy propagation into the runtime initialization as we disscussed before?

tenzen-y · 2024-10-30T18:59:30Z

pkg/runtime.v2/framework/plugins/coscheduling/coscheduling.go

-func (c *CoScheduling) Build(ctx context.Context, info *runtime.Info, trainJob *kubeflowv2.TrainJob) (client.Object, error) {
-	if info == nil || info.PodGroupPolicy == nil || info.PodGroupPolicy.Coscheduling == nil || trainJob == nil {
+func (c *CoScheduling) Build(ctx context.Context, runtimeJobTemplate client.Object, info *runtime.Info, trainJob *kubeflowv2.TrainJob) (client.Object, error) {
+	if info == nil || info.Scheduler == nil || info.Scheduler.ScheduleTimeoutSeconds == nil || trainJob == nil {


Suggested change

if info == nil || info.Scheduler == nil || info.Scheduler.ScheduleTimeoutSeconds == nil || trainJob == nil {

if info == nil || info.Scheduler == nil || trainJob == nil {

As we discussed berore, they need to specify the empty cosheduler parameters when they do not want to modify the schedulerTimeoutSeconds, right?

Do we really want to allow this ? Otherwise, runtime might look as follows:

spec: mlPolicy: numNodes: 10 torch: podGroupPolicy: coscheduling:

Is it a valid object from the API perspective ?

As we discussed before, they can specify the empty object like this:

spec: mlPolicy: numNodes: 10 torch: podGroupPolicy: coscheduling: {}

I reverted it back to your original condition check: https://github.com/kubeflow/training-operator/blob/894cded764ddddb08c5271872631d0687a34750d/pkg/runtime.v2/framework/plugins/coscheduling/coscheduling.go#L92

pkg/runtime.v2/framework/plugins/plainml/plainml.go

tenzen-y · 2024-10-30T19:05:09Z

pkg/runtime.v2/framework/plugins/plainml/plainml.go

-			PodRequests: info.TotalRequests[rName].PodRequests,
+		// For other Jobs like the Initializer, replica is always equal to 1.
+		// TODO (andreyvelich): Add support for total requests from the TrainJob's ResourcesPerNode.
+		if rName == constants.JobTrainerNode {


I meant that the reason why we do not store thre totalRequests for the initializers?

tenzen-y

Comments for the unit testings.

tenzen-y · 2024-10-30T19:22:02Z

pkg/runtime.v2/runtime.go

 	TotalRequests map[string]TotalResourceRequest
 }

+type Trainer struct {


I think that the Trainer needs to have scheduler since we need to create the PodGroup and do some else scheduling logic.
If we create the PodGroup against a whole of TrainJob, that will not schedule to any Node for ever. Because we need to start the initializer before the Trainer.

tenzen-y · 2024-10-30T19:30:04Z

pkg/runtime.v2/core/trainingruntime_test.go

 					MinResources(corev1.ResourceList{
-						corev1.ResourceCPU: resource.MustParse("60"),
+						corev1.ResourceCPU: resource.MustParse("31"), // Every replica has 1 CPU = 31 CPUs in total.


I was not sure the reason why @akshaychitneni requests to implement the util.

pkg/runtime.v2/core/trainingruntime_test.go

tenzen-y

Comments for the Integration testings

tenzen-y · 2024-10-30T19:43:57Z

test/integration/controller.v2/trainjob_controller_test.go

-					g.Expect(rJob.Template.Spec.Template.Spec.Containers[0].Image).Should(gomega.Equal(originImageName))
+					if rJob.Name == constants.JobTrainerNode {
+						g.Expect(rJob.Template.Spec.Template.Spec.Containers[0].Image).Should(gomega.Equal(originImageName))
+					}
 				}
 			}, util.Timeout, util.Interval).Should(gomega.Succeed())
 		})


Could you add a case if we can build and deploy Torch JobSet like ginkgo.It("Should succeed to create TrainJob with Torch TrainingRuntime", func() {

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

pkg/runtime.v2/framework/plugins/torch/torch.go

deepanker13 · 2024-10-31T07:49:33Z

pkg/runtime.v2/framework/core/framework.go

 	var objs []client.Object
 	for _, plugin := range f.componentBuilderPlugins {
-		obj, err := plugin.Build(ctx, info, trainJob)
+		obj, err := plugin.Build(ctx, runtimeJobTemplate, info, trainJob)


ctx context is not getting used anywhere throughout the code, is it necessary to pass?

I think, we use it here to get the existing object and compare it
https://github.com/kubeflow/training-operator/blob/master/pkg/runtime.v2/framework/plugins/jobset/jobset.go#L82-L84

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich · 2024-10-31T16:16:50Z

pkg/runtime.v2/runtime.go

 type Info struct {
-	Obj         client.Object
+	// Labels and Annotations to add to the RuntimeJobTemplate.
 	Labels      map[string]string
-	PodLabels   map[string]string
 	Annotations map[string]string
-	Policy
-	TotalRequests map[string]TotalResourceRequest
+	// Original policy values from the runtime.
+	RuntimePolicy RuntimePolicy
+	// Trainer parameters to add to the RuntimeJobTemplate.
+	Trainer
+	// Scheduler parameters to add to the RuntimeJobTemplate.
+	*Scheduler
 }


I keep the original runtime policies values in the Info object for now, since we can't pass the specific policies to the EnforceMLPolicy and EnforcePodGroupPolicy() APIs. Similar to what we do today: https://github.com/kubeflow/training-operator/blob/62a058db0c445db1b7e16572d83fdace56f59476/pkg/runtime.v2/runtime.go#L40

In the future, we will discuss how we can optimize the Info object structure.

cc @tenzen-y

Yeah, that is at least a requirement. Let's revisit here later. I can take that in the next week.

pkg/runtime.v2/framework/plugins/coscheduling/coscheduling.go

tenzen-y · 2024-10-31T17:35:50Z

pkg/runtime.v2/framework/plugins/coscheduling/coscheduling.go

-	}
-	info.PodLabels[schedulerpluginsv1alpha1.PodGroupLabel] = trainJob.Name
+
+	info.Scheduler.PodLabels = make(map[string]string, 1)


What is the reason that we forcefully override the Pod labels?

We only use PodLabels in the Scheduler right now, that is why we just creating a new Map here.
In the future, if we should do any Pod labels merge for the Info object, we should revisit it.

Basically, this is pluggabble mechanism, so we should consider the other plugin.
So, as before initialization impl, we should initialize map only when the pod label is emtpy

Alright, let me change it.

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

tenzen-y

Thank you for this effort!
Let's open some follow-up issue!
/lgtm
/apprve

tenzen-y · 2024-10-31T18:33:32Z

/approve

tenzen-y · 2024-10-31T18:33:41Z

/hold cancel

google-oss-prow · 2024-10-31T18:33:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* KEP-2170: Implement JobSet and PlainML Plugins Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix nil pointer exception for Trainer Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix unit tests in runtime package Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix unit tests Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix integration tests Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix lint Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Implement Torch Plugin Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Use list for the Info envs Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix golang ci Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix Torch plugin Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Use K8s sets Update error return Use ptr.Deref() for nil values Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Use client.Object for Build() call Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Remove DeepCopy Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Remove MLPolicy and PodGroupPolicy from the Info object Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Inline error Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Remove SDK jar file Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add integration test for Torch plugin Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add TODO to calculate PodGroup values in unit tests Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Revert the change to add original Runtime Policies to Info Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Create const for the DefaultJobReplicas Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Check if PodLabels is empty Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

* Added test for create-pytorchjob.ipynb Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * fix yaml syntax Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Fix uses path Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Add actions/checkout Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Add bash to action.yaml Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Install pip dependencies step Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Add quotes for args Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Add jupyter Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Add nbformat_minor: 5 to fix invalid format error Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Fix job name Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * test papermill-args-yaml Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * testing multi line args Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * testing multi line args1 Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * testing multi line args2 Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * testing multi line args3 Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Parameterize sdk install Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Remove unnecessary output Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * nbformat normailze Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * [SDK] Training Client Conditions related unit tests (#2253) * test: add unit test for get_job_conditions function of training client Signed-off-by: Bobbins228 <mcampbel@redhat.com> * test: add unit test for is_job_created function of training client Signed-off-by: Bobbins228 <mcampbel@redhat.com> * test: add unit test for is_job_running function of training client Signed-off-by: Bobbins228 <mcampbel@redhat.com> * test: add unit test for is_job_restarting function of training client Signed-off-by: Bobbins228 <mcampbel@redhat.com> * test: add unit test for is_job_failed function of training client Signed-off-by: Bobbins228 <mcampbel@redhat.com> * test: add unit test for is_job_succeded function of training client Signed-off-by: Bobbins228 <mcampbel@redhat.com> * test: improve job condition unit tests efficiency Signed-off-by: Bobbins228 <mcampbel@redhat.com> --------- Signed-off-by: Bobbins228 <mcampbel@redhat.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * [SDK] test: add unit test for list_jobs method of the training_client (#2267) Signed-off-by: wei-chenglai <qazwsx0939059006@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Generate clientset, openapi spec for the V2 APIs (#2273) Generate clientset, informers, listers and open api spec for v2alpha1 APIs. Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * [SDK] Use torchrun to create PyTorchJob from function (#2276) * [SDK] Use torchrun to create PyTorchJob from function Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Update PyTorchJob SDK example Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add consts for entrypoint Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add check for num procs per worker Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * [SDK] test: add unit test for get_job_logs method of the training_client (#2275) Signed-off-by: wei-chenglai <qazwsx0939059006@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * [v2alpha] Move GV related codebase (#2281) Move GV related codebase in v2alpha Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Implement runtime framework (#2248) * KEP-2170: Implement runtime framework interfaces Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Remove grep dependency Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * KEP-2170: Implement ValidateObjects interface to the runtime framework Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * KEP-2170: Expose the TrainingRuntime and ClusterTrainingRuntime Kind Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * KEP-2170: Remove unneeded scheme field from the internal TrainingRuntime Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Rephrase the error message Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Distinguish TrainingRuntime and ClusterTrainingRuntime when creating indexes for the TrainJobs Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Propagate the TrainJob labels and annotations to the JobSet Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Remove PodAnnotations from the runtime info Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Implement TrainingRuntime ReplicatedJob validation Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Add TODO comments Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Replace queueSuspendedTrainJob with queueSuspendedTrainJobs Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> --------- Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Add DeepSpeed Example with Pytorch Operator (#2235) Signed-off-by: Syulin7 <735122171@qq.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Rename TrainingRuntimeRef to RuntimeRef API (#2283) * KEP-2170: Rename TrainingRuntimeRef to RuntimeRef API Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Rename RuntimeRef in runtime framework Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Adding CEL validations on v2 TrainJob CRD (#2260) Signed-off-by: Akshay Chitneni <achitneni@apple.com> Co-authored-by: Akshay Chitneni <achitneni@apple.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Upgrade Deepspeed demo dependencies (#2294) Signed-off-by: Syulin7 <735122171@qq.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Add manifests for Kubeflow Training V2 (#2289) * KEP-2170: Add manifests for Kubeflow Training V2 Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix invalid name for webhook config in cert Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix integration tests Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Move kubebuilder markers to runtime framework Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Use Kubernetes recommended labels Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: sailesh duddup E6A3 udi <saileshradar@gmail.com> * FSDP Example for T5 Fine-Tuning and PyTorchJob (#2286) * FSDP Example with PyTorchJob and T5 Fine-Tuning Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Modify text Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Implement TrainJob Reconciler to manage objects (#2295) * KEP-2170: Implement TrainJob Reconciler to manage objects Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Mode dep-crds to manifests/external-crds Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Rename run with runtime Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> --------- Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Remove Prometheus Monitoring doc (#2301) Signed-off-by: Sophie <sophy010017@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Decouple JobSet from TrainJob (#2296) Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Strictly verify the CRD marker validation and defaulting in the integration testings (#2304) Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Initialize runtimes before the manager starts (#2306) Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Generate Python SDK for Kubeflow Training V2 (#2310) * Generate SDK models for the Training V2 APIs Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Create pyproject.toml config Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Remove comments Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix pre-commit Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Create model and dataset initializers (#2303) * KEP-2170: Create model and dataset initializers Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add abstract classes Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add storage URI to config Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Update .gitignore Co-authored-by: Kevin Hannon <kehannon@redhat.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix the misspelling for initializer Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add .pt and .pth to ignore_patterns Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Co-authored-by: Kevin Hannon <kehannon@redhat.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Implement JobSet, PlainML, and Torch Plugins (#2308) * KEP-2170: Implement JobSet and PlainML Plugins Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix nil pointer exception for Trainer Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix unit tests in runtime package Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix unit tests Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix integration tests Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix lint Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Implement Torch Plugin Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Use list for the Info envs Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix golang ci Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix Torch plugin Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Use K8s sets Update error return Use ptr.Deref() for nil values Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Use client.Object for Build() call Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Remove DeepCopy Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Remove MLPolicy and PodGroupPolicy from the Info object Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Inline error Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Remove SDK jar file Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add integration test for Torch plugin Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add TODO to calculate PodGroup values in unit tests Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Revert the change to add original Runtime Policies to Info Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Create const for the DefaultJobReplicas Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Check if PodLabels is empty Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Implement Initializer builders in the JobSet plugin (#2316) * KEP-2170: Implement Initializer builder in the JobSet plugin Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Update the SDK models Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Remove Info from Initializer builder Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Update manifests Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Update pkg/constants/constants.go Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Use var for envs Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Remove check manifests from GitHub actions Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Move consts to JobSet plugin Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Add the TrainJob state transition design (#2298) * KEP-2170: Add the TrainJob state transition design Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Replace actual jobs with TrainJob Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Remove the JobSet conditions propagation and Add expanding runtime framework interfaces for each plugin Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Expand the Creation Failed reasons Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Rename Completed to Complete Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> --------- Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Update tf job examples to tf v2 (#2270) * mnist with summaries updaetd to TF v2 Signed-off-by: yelias <yossi.elias@nokia.com> * tf_sample updaetd to TF v2 Signed-off-by: yelias <yossi.elias@nokia.com> * Add mnist_utils and update dist-mnist Signed-off-by: yelias <yossi.elias@nokia.com> * Add mnist_utils and update dist-mnist Signed-off-by: yelias <yossi.elias@nokia.com> * Remove old example - estimator-API, this example has been replaced by distribution_strategy Signed-off-by: yelias <yossi.elias@nokia.com> * Small fix Signed-off-by: yelias <yossi.elias@nokia.com> * Remove unsupported powerPC dockerfiles Signed-off-by: yelias <yossi.elias@nokia.com> * Fix typo in copyright Signed-off-by: yelias <yossi.elias@nokia.com> --------- Signed-off-by: yelias <yossi.elias@nokia.com> Co-authored-by: yelias <yossi.elias@nokia.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Add TrainJob conditions (#2322) * KEP-2170: Implement TrainJob conditions Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Fix API comments Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Make condition message constants Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Stop connecting condition type and reason in JobSet plugin Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> --------- Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Pin Gloo repository in JAX Dockerfile to a specific commit (#2329) This commit pins the Gloo repository to a specific commit (43b7acbf) in the JAX Dockerfile to prevent build failures caused by a recent bug introduced in the Gloo codebase. By locking the version of Gloo to a known working commit, we ensure that the JAX build remains stable and functional until the issue is resolved upstream. The build failure occurs when compiling the gloo/transport/tcp/buffer.cc file due to an undefined __NR_gettid constant, which was introduced after the pinned commit. By using this commit, we bypass the issue and allow the build to complete successfully. Signed-off-by: Sandipan Panda <samparksandipan@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * [fix] Resolve v2alpha API exceptions (#2317) Resolve v2alpha API exceptions by adding necessary listType validations. Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Upgrade Kubernetes to v1.30.7 (#2332) * Upgrade Kubernetes to v1.30.7 Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr> * Use typed event handlers and predicates in job controllers Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr> * Re-organize pkg/common/util/reconciler.go Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr> * Update installation instructions in README Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr> --------- Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Ignore cache exporting errors in the image building workflows (#2336) Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * KEP-2170: Add Torch Distributed Runtime (#2328) * KEP-2170: Add Torch Distributed Runtime Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Add pip list Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Refine the server-side apply installation args (#2337) Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Add openapi-generator CLI option to skip SDK v2 test generation (#2338) Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Upgrade kustomization files to Kustomize v5 (#2326) Signed-off-by: oksanabaza <obazylie@redhat.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Pin accelerate package version in trainer (#2340) * Pin accelerate package version in trainer Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com> * include new line to pass pre-commit hook Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com> --------- Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com> Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Replace papermill command with bash script Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Typo fix Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Move Checkout step outside action.yaml file Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Add newline EOF in script Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Pass python dependencies as args and pin versions Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Update Usage Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Install dependencies in yaml Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * fix ipynb Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * set bash flags Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * Update script args and add more kubernetes versions for tests Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * add gang-scheduler-name to template Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * move go setup to template Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> * remove -p parameter from script Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> --------- Signed-off-by: sailesh duddupudi <saileshradar@gmail.com> Signed-off-by: Bobbins228 <mcampbel@redhat.com> Signed-off-by: wei-chenglai <qazwsx0939059006@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Syulin7 <735122171@qq.com> Signed-off-by: Akshay Chitneni <achitneni@apple.com> Signed-off-by: Sophie <sophy010017@gmail.com> Signed-off-by: yelias <yossi.elias@nokia.com> Signed-off-by: Sandipan Panda <samparksandipan@gmail.com> Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr> Signed-off-by: oksanabaza <obazylie@redhat.com> Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com> Co-authored-by: Mark Campbell <mcampbel@redhat.com> Co-authored-by: Wei-Cheng Lai <qazwsx0939059006@gmail.com> Co-authored-by: Varsha <varshaprasad96@gmail.com> Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Co-authored-by: yu lin <735122171@qq.com> Co-authored-by: Akshay Chitneni <akshayadatta@gmail.com> Co-authored-by: Akshay Chitneni <achitneni@apple.com> Co-authored-by: Sophie Hsu <112261858+sophie0730@users.noreply.github.com> Co-authored-by: Kevin Hannon <kehannon@redhat.com> Co-authored-by: YosiElias <73485442+YosiElias@users.noreply.github.com> Co-authored-by: yelias <yossi.elias@nokia.com> Co-authored-by: Sandipan Panda <87253083+sandipanpanda@users.noreply.github.com> Co-authored-by: Antonin Stefanutti <astefanutti@users.noreply.github.com> Co-authored-by: Oksana Bazylieva <61097730+oksanabaza@users.noreply.github.com> Co-authored-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

google-oss-prow bot requested review from jinchihe and kuizhiqing October 24, 2024 22:25

google-oss-prow bot added the size/L label Oct 24, 2024

google-oss-prow bot added the do-not-merge/hold label Oct 24, 2024

andreyvelich changed the title ~~KEP-2170: Implement JobSet and PlainML Plugins~~ KEP-2170: Implement JobSet, PlainML, and Torch Plugins Oct 24, 2024

google-oss-prow bot added size/XL and removed size/L labels Oct 25, 2024

akshaychitneni reviewed Oct 25, 2024

View reviewed changes

pkg/runtime.v2/runtime_test.go Outdated Show resolved Hide resolved

google-oss-prow bot added size/XXL and removed size/XL labels Oct 26, 2024

andreyvelich commented Oct 26, 2024

View reviewed changes

pkg/runtime.v2/core/trainingruntime_test.go Show resolved Hide resolved

tenzen-y reviewed Oct 28, 2024

View reviewed changes

andreyvelich added 11 commits October 29, 2024 14:53

KEP-2170: Implement JobSet and PlainML Plugins

c217f96

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Fix nil pointer exception for Trainer

7f631ab

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Fix unit tests in runtime package

661884c

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Fix unit tests

40e6fdd

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Fix integration tests

c3bd73a

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Fix lint

9d04515

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Implement Torch Plugin 6D47

1ea546e

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Use list for the Info envs

a17327a

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Fix golang ci

0398e88

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Fix Torch plugin

131e849

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Use K8s sets

cf694c5

Update error return Use ptr.Deref() for nil values Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

deepanker13 reviewed Oct 29, 2024

View reviewed changes

pkg/runtime.v2/core/trainingruntime.go Show resolved Hide resolved

deepanker13 reviewed Oct 29, 2024

View reviewed changes

pkg/runtime.v2/core/trainingruntime.go Outdated Show resolved Hide resolved

deepanker13 reviewed Oct 29, 2024

View reviewed changes

pkg/runtime.v2/core/trainingruntime_test.go Show resolved Hide resolved

andreyvelich force-pushed the issue-2290-torch-plugin branch from 545a6c0 to cf694c5 Compare October 29, 2024 15:50

andreyvelich added 2 commits October 29, 2024 22:49

Remove MLPolicy and PodGroupPolicy from the Info object

e5165cb

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Inline error

85d0cf1

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

akshaychitneni reviewed Oct 29, 2024

View reviewed changes

pkg/runtime.v2/framework/plugins/plainml/plainml.go Show resolved Hide resolved

akshaychitneni reviewed F438 Oct 30, 2024

View reviewed changes

pkg/runtime.v2/framework/plugins/torch/torch.go Show resolved Hide resolved

akshaychitneni reviewed Oct 30, 2024

View reviewed changes

tenzen-y reviewed Oct 30, 2024

View reviewed changes

andreyvelich added 3 commits October 30, 2024 22:47

Remove SDK jar file

3ada8c1

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Add integration test for Torch plugin

0899039

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Add TODO to calculate PodGroup values in unit tests

f189a82

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

deepanker13 reviewed Oct 31, 2024

View reviewed changes

pkg/runtime.v2/framework/plugins/torch/torch.go Outdated Show resolved Hide resolved

deepanker13 reviewed Oct 31, 2024

View reviewed changes

andreyvelich added 2 commits October 31, 2024 16:09

Revert the change to add original Runtime Policies to Info

1f627b4

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Create const for the DefaultJobReplicas

894cded

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich commented Oct 31, 2024

View reviewed changes

tenzen-y reviewed Oct 31, 2024

View reviewed changes

Check if PodLabels is empty

8149c68

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

tenzen-y reviewed Oct 31, 2024

View reviewed changes

google-oss-prow bot assigned tenzen-y Oct 31, 2024

google-oss-prow bot added the lgtm label Oct 31, 2024

google-oss-prow bot added approved and removed do-not-merge/hold labels Oct 31, 2024

google-oss-prow bot merged commit 7c5ea70 into kubeflow:master Oct 31, 2024
43 checks passed

andreyvelich deleted the issue-2290-torch-plugin branch October 31, 2024 19:03


		info := runtime.NewInfo(opts...)

		if err := r.framework.RunEnforceMLPolicyPlugins(info, trainJob, mlPolicy); err != nil {

	if info == nil \|\| info.Scheduler == nil \|\| info.Scheduler.ScheduleTimeoutSeconds == nil \|\| trainJob == nil {
	if info == nil \|\| info.Scheduler == nil \|\| trainJob == nil {

KEP-2170: Implement JobSet, PlainML, and Torch Plugins #2308

KEP-2170: Implement JobSet, PlainML, and Torch Plugins #2308

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pull Request Test Coverage Report for Build 11617265788

Details

💛 - Coveralls

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment