8000 ⚠️ Add metrics to infrastructure.json by shlomitk1 · Pull Request #1562 · fybrik/fybrik · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
8000

⚠️ Add metrics to infrastructure.json #1562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Jun 23, 2022
Merged

Conversation

shlomitk1
Copy link
Contributor
@shlomitk1 shlomitk1 commented Jun 19, 2022

Signed-off-by: Shlomit Koyfman shlomitk@il.ibm.com
Fixes #1536
Fixes #1550

Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
Copy link
Member
@simanadler simanadler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comments. If necessary please schedule a meeting to discuss.

Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
@shlomitk1 shlomitk1 changed the title Add attribute definitions to infrastructure.json Add metrics to infrastructure.json Jun 20, 2022
@shlomitk1 shlomitk1 marked this pull request as ready for review June 20, 2022 06:36
@shlomitk1 shlomitk1 requested review from zivnevo and roytman June 20, 2022 06:36
infraElement := dpc.env.AttributeManager.GetAttribute(attr, cluster.Name)
if infraElement == nil {
return "", "", fmt.Errorf("attribute %s is not defined for cluster %s", attr, cluster.Name)
for _, instanceType := range instanceTypes {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably not how we want to handle bandwidth-like attributes. We have to check len(instanceTypes). If it's 1, then it's a simple optimization goal and the current code is ok. If it's 2, then it's bandwidth-like optimization goal and we have to return "", "", nil

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not related to bandwidth-like attributes. The use-case is a single "cost" attribute defined for both clusters and storage-accounts. When this cost should be minimized it means minimizing everything on the data path that has "cost".
I saw that the code fills an array, so it should not be a problem, or should it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Policies could be about storage cost, cluster cost, or all costs. Minimization should be done on the relevant costs indicated in the policy. Right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Policies could be about storage cost, cluster cost, or all costs. Minimization should be done on the relevant costs indicated in the policy. Right?

Yes. Minimization policy is related to an attribute. The same attribute can be defined for storages, clusters,...
It is also possible to define separately storage-cost and cluster-cost as different attributes, in which case policy will define whether to minimize any of them, or both.

Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
@shlomitk1 shlomitk1 requested review from simanadler and zivnevo June 22, 2022 11:41
Copy link
Collaborator
@zivnevo zivnevo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid the suggested change to setSimpleGoalVarArray() is not going to work.
We really need a different paramArray for each instance type, because the i-th entry in the paramArray means the attribute-value for module/cluster/storage number i. So we cannot just put them all into one param array, bur rather need to call getAttributeMapping() for each instanceType.

In fact, we need to call setSimpleGoalVarArray() for each instanceType. Each such call will result in a variable-array holding the costs of the selected clusters/modules/storage-accounts along the data-path. We then need to sum each such array, and finally sum the sums 😢.

I suggest to delay this "total cost" feature for a later PR. For now, forbid this feature and open a new issue.

Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
@shlomitk1
Copy link
Contributor Author

I suggest to delay this "total cost" feature for a later PR. For now, forbid this feature and open a new issue.

OK. I have left the array in GetInstanceTypes but only the first element is used in the solver.

@shlomitk1 shlomitk1 requested a review from zivnevo June 22, 2022 18:19
@shlomitk1
Copy link
Contributor Author

@roytman This PR changes infrastructure.json format as well as the corresponding validation schema and taxonomy json.
Need to be reflected in the release notes as changing user interfaces.

@shlomitk1 shlomitk1 changed the title Add metrics to infrastructure.json ⚠️ Add metrics to infrastructure.json Jun 23, 2022
@shlomitk1 shlomitk1 merged commit b1b43b5 into fybrik:master Jun 23, 2022
@shlomitk1 shlomitk1 deleted the attributes branch June 23, 2022 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimizer: add a multi-regional attribute sample Provide a better syntax for attribute definitions
3 participants
0