Could it be that you can't find the pod because it was deleted when reached final state? Did you set is_delete_operator_pod=False?

The pod executing the task is not deleted, it is visible in kubernetes but with a different name.

Pod creation happens in the scheduler, kubernetes_executor.run_pod_async. Here the real name is generated that is sent to kubernetes.

However, if i understand the flow correctly, the RenderedTaskInstanceFields is later created inside the pod itself, via the local executor when it calls taskinstance._execute_task_with_callbacks

            rtif = RenderedTaskInstanceFields(ti=self, render_templates=False)
            RenderedTaskInstanceFields.write(rtif)
            RenderedTaskInstanceFields.delete_old_records(self.task_id, self.dag_id)

RenderedTaskInstanceFields calls ti.render_k8s_pod_yaml() which calls construct_pod, where the pod name is regenerated using make_unique_pod_id, which as the name suggests is random and a new unique value :)

So it seems that this effects KubernetesExecutor but not using KPO as stand alone
cc @jedcunningham @dstandish

I found out that the correct pod name is shown under Task Instance Details -> Hostname.
BTW, issue also affects Airflow 2.4.3

ah right. so this is not a bug.. maybe we just need to clarify what pod is shown in render tab for users of Kubernetes executor

@csp33 would you like to raise PR ?

@eladkal sure! However, I still do not understand what is the pod name listed under the "K8s Pod Spec" tab.
I think its value should match the task hostname.

That rendered pod spec is essentially the result of regenerating the pod spec on the fly. You'll notice if you refresh you get a new pod name every time.

Similarly, if you look at the pod spec for an old, historical task, you see what the pod would look like if it was ran now, not the spec it used when it was ran. For example, say you updated Airflow versions/images since it was ran, you'd see the new image not the old image in the spec.

i guess the issue is that, RTIF gets stamped within _run_raw_task. and by the time it gets here, we're already within the pod. so we no longer have the actual pod details.

probably this pod spec does not belong in RTIF. (it's not a templated field, after all) probably it was just a convenient / easy way to implement this feature, nowithstanding the limitations pointed out here.

i suppose one thing we could do is, create the RTIF record earlier for KE (just with pod spec), then in run raw update it with the RTIFs.

another option would be, put more KE info (or if you want to make an abstraction, executor info) on the TI record. we could include spec here. adding namespace and pod name to the TI record would also help with certain logging scenarios.

another option would be add separate table like TIExecutorInfo which is 1-1 with TI and has arbitrary executor info such as this.