8000 Enable smoother sbt bsp integration. by ckipp01 · Pull Request #2154 · scalameta/metals · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Enable smoother sbt bsp integration. #2154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Oct 30, 2020
Merged

Enable smoother sbt bsp integration. #2154

merged 29 commits into from
Oct 30, 2020

Conversation

ckipp01
Copy link
Member
@ckipp01 ckipp01 commented Oct 19, 2020

📢 EDIT: A lot has changed since this was first opened, but I'm leaving this here to show my thought process about how I started this vs what it's like now. So if you want to know the full history of how this started, feel free to read this, if not scroll on down to the other large section with gifs and I'll explain the approach I landed on.

This pr is the first steps in addressing #2049 and #2152 by enabling the sbt bsp support to be much more integrated into Metals if a user would like choose it instead of using Bloop.

This is still heavily a WIP, I still have a bunch of TODOs as you'll see in the code, I need a bunch of testing, and even some docs updated. However, it's getting a bit large, so I'd like to throw this up here as it is working, and I'd love any feedback about the approach or things I may have missed before I get even farther into this.

Following along with the detailed conversations in the two threads listed above and also the POC that @kpbochenek started on (👍 thanks for that), I split the intended support into 4 paths.

[Path 1] - User clones down a project, has never ran sbt on it before, opens it, and would like to use the sbt server.

Current behavior

The current behavior for this is that they would have to ignore the prompt to import the build, start up the sbt shell with -Dsbt.semanticdb=true, and then do a build-connect command to connect to it.

New behavior

Again, following the discussion, until all the kinks get smoothed out, we have proper testing, and the UX is on par with that of the Bloop integration, we'll keep Bloop as the default choice. That part doesn't change, so upon opening the project, the user will still get a prompt to import the build (with Bloop). If they'd like to instead use sbt server, they'd just ignore the prompt, and execute a new command (that may be named different in the client) build-server-start with a single parameter "sbt". The idea in the future would be for this new command to be the way we can trigger to start a server manually for whatever reason. In the case of sbt, this is sort of special-cased as there is a SbtServer.scala class that will handle this by ensuring the sbt version is > 1.4.0, starts up a process of sbt, and then establishes a bsp connection.

Here is a gif showing that. The time is accurate to give you an idea of how long it will take. There will "progress" added so the user knows what's going on.
2020-10-19 15 43 48

If you were to look at the logs in this situation, you'd see something like this, which will give you a good idea of what's going on behind the scenes:

2020.10.19 16:38:15 INFO  Started: Metals version 0.9.5-SNAPSHOT in workspace '/Users/ckipp/Documents/scala-workspace/bsp-test' .
2020.10.19 16:38:15 INFO  time: initialize in 0.31s
2020.10.19 16:38:16 INFO  Attempting quick connect to the build server
2020.10.19 16:38:16 WARN  Build server is not auto-connectable.
2020.10.19 16:38:17 INFO  no build target: using presentation compiler with only scala-library
2020.10.19 16:38:26 INFO  Attempting to connect to sbt BSP server...
2020.10.19 16:38:26 INFO  sbt 1.4.0 found for workspace.
2020.10.19 16:38:26 INFO  Suitable version of sbt found, attempting to connect...
2020.10.19 16:38:26 INFO  Installalling plugin to /Users/ckipp/Documents/scala-workspace/bsp-test/project/MetalsSbtBsp.scala
2020.10.19 16:38:26 INFO  Starting background sbt process...
2020.10.19 16:38:26 INFO  sbt process started: true
2020.10.19 16:38:37 INFO  sbt up and running, attempting to start a bsp session...
2020.10.19 16:38:38 INFO  tracing is enabled: /Users/ckipp/Library/Caches/org.scalameta.metals/bsp.trace.json
2020.10.19 16:38:38 INFO  Connected to Build server v1.4.0
2020.10.19 16:38:38 INFO  Processing
2020.10.19 16:38:38 WARN  There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2020.10.19 16:38:39 INFO  Processing
2020.10.19 16:38:39 INFO  Processing
2020.10.19 16:38:39 INFO  Processing
2020.10.19 16:38:39 INFO  Processing
2020.10.19 16:38:39 INFO  Processing
2020.10.19 16:38:39 INFO  Processing
2020.10.19 16:38:39 WARN  There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2020.10.19 16:38:40 INFO  time: imported build in 1.68s
2020.10.19 16:38:39 INFO  Processing
2020.10.19 16:38:42 INFO  time: indexed workspace in 1.72s
2020.10.19 16:38:42 INFO  Completed connection with a success
2020.10.19 16:38:42 INFO  Processing
2020.10.19 16:38:42 INFO  compiling bsp-test

[Path 2] - User re-opens the workspace after they have previously chosen to use the sbt bsp server support

Current behavior

This would currently be no different than above, the user would have to go through all the steps of connecting again.

New behavior

The users choice was saved up above, so that will be detected, sbt will be triggered to start, and then a bsp connection will be made. Some more work probably needs to happen here to ensure we don't have to many shells open that don't need to be.

2020-10-19 15 45 01

[Path 3] - User has already imported their build with Bloop, but now want to switch to sbt bsp

Current behavior

Again, same as the others, they would need to basically remove their .metals/ and .bloop/ directories, and then repeat the process from the beginning to connect.

New behavior

Now the user is able to simply again trigger the new start-build-server command. As an example in the gifs you're seeing it bound to a start-sbt-server command, which sends in the necessary command to Metals. I'll open that pr and one in VS Code shortly after this if someone wants to take it for a test drive.

2020-10-19 15 46 25

[Path 4] - User wants to switch back from sbt BSP to Bloop.

New behavior

I'm just going to skip the old behavior because that flow just doesn't exist right now. But there is now a couple ways to do this. Firstly, you could trigger an reset-choice to forget the build server choice, close your workspace, re-open, and then you'd be prompted again to import your build with Bloop.

2020-10-19 15 55 28

Switching back and forth is also supported, as you can see here where I use bsp-switch to jump from the sbt server back to bloop (sbt is shutdown here):

2020-10-19 16 07 55

There are also some other notable things:

  • Existing commands like build-disconnect, build-restart, etc now have changed behavior to detect if the user is using Bloop or another bsp server and issue the commands accordingly.
  • workspace/reload is now introduced so when a build change is detected while using sbt bsp, it's fired. This also h 8000 appens for example if the user triggers a manual build-import, but they are using sbt server rather than Bloop. Note that we do need to figure out some stuff with this and notifying the user if they reload a bad definition ([BSP] - unexpected workspace/reload response if given an invalid build definition sbt/sbt#5988) (This has been fixed)
  • The sbt plugin is currently just a scala file. I'm going to assume we want to actual publish that as a plugin, and then instead include a file including the plugin like we do with the sbt-bloop plugin?
  • I refactored a lot of the BSP related stuff into a bsp package.

I also want to add while I'm a avid Bloop user and fan, I've been very impressed with the BSP support in sbt. I remember first trying it very early on and was worried because of some issues I hit on, but I didn't notice anything like that while playing around with this. This coming week I'll probably be using this exclusively at work to really test drive it. Looking forward to feedback.

@ckipp01 ckipp01 marked this pull request as draft October 19, 2020 15:01
@ckipp01 ckipp01 added improvement Not a bug or a feature, but something general we can improve sbt server Relates to sbt BSP server support labels Oct 19, 2020
@ckipp01 ckipp01 changed the title Enable for smoother sbt bsp integration. Enable smoother sbt bsp integration. Oct 19, 2020
@tgodzik
< 8000 /summary> Copy link
Contributor
tgodzik commented Oct 19, 2020

Thanks for the great work @ckipp01 ! I will give it a go and see if there is anything I could suggest. Overall, it looks like a very good approach.

@ckipp01
Copy link
Member Author
ckipp01 commented Oct 24, 2020

So from the original approach up above, I changed a few things that started to worry me the more I looked at them. Mainly, I didn't want to special-case sbt so much, especially seeing how the bsp support in mill was merged, I kept trying to think of a nice way to after we smooth out the sbt bsp support to look ahead to the mill bsp support.

Ultimately this lead to the decision to get rid of the original SbtServer class that I had, and instead create a BuildServerProvider to be mixed in to a BuildTool the same way that BloopPluginBuildTool (which is now BloopInstallProvider) is done. So essentially all the build tools that support Bloop Install are now <BuildToolName> extends BuildTool with BloopPluginProvider and if they also provide bsp support, with BuildServerProvider also gets mixed in.

Another decision related to the above one was that instead of the build-server-start command, I instead landing on a different command called generate-bsp-config. This command is really only mean to be used in the case that you don't have a .bsp/<entry> file yet. This command will then check if the current build tool is a BuildServerProvider, and if so, generate the build server config, and then attempt to connect to it. If there are multiple build tools for the workspace (for example in the future Mill and sbt), then you'd be able to decide which to use as a build server as well. So to outline the 4 paths from above again:

[Path 1] - User clones down a project, has never ran sbt on it before, opens it, and would like to use the sbt server.

The user will ignore the first import prompt, and instead trigger the generate-bsp-config. sbt is the only build tool in the workspace, so it runs sbt bspConfig, and after the successful generation of it, connects to the sbt build server.

2020-10-24 16 02 20

[Path 2] - User re-opens the workspace after they have previously chosen to use the sbt bsp server support

Same as above, the decision was saved when the user did the generate-bsp-config as sbt, and when you reopen the workspace, it will auto-connect to the sbt server.

[Path 3] - User has already imported their build with Bloop, but now want to switch to sbt bsp

User has three choice.

  1. They can execute the generate-bsp-config (but probably not necessary because it will already probably exist), but if it didn't they could execute this, sbt version would be checked, .bsp/sbt.json created, the decision saved, and then connected.

  2. Since they .bsp/sbt.json already exists, the can simply do a metals.bsp-switch, which would detect sbt, ensure the plugin is there, and then connect to it while also saving the choice for next time.

  3. While a user probably won't do this, the can use the metals.build-disconnect command, followed up by a metals.build-connect command as well.

[Path 4] - User wants to switch back from sbt BSP to Bloop.

Same as the first examples, the user can again use the metals.bsp-switch command to switch back to Bloop. Or, they can also do a metals.reset-choice an select build-server, open the workspace again, and then they'll either be prompted to reimport if necessary or automatically connect to Bloop.

This seems to all be working pretty smoothly at this point, so dare I say the core part of this is now done. I'll focus in on cleaning a few things up ensuring things are properly tested and documented.

@ckipp01 ckipp01 force-pushed the sbtBsp branch 3 times, most recently from df08fde to b68c1e4 Compare October 25, 2020 11:25
@ckipp01 ckipp01 marked this pull request as ready for review October 25, 2020 12:20
@ckipp01
Copy link
Member Author
ckipp01 commented Oct 25, 2020

So I think this is in a pretty good place now to have some more eyes on it and to be taken for a test drive. I've been playing around with it for a while, using it this last week some at work, and I'm overall pretty happy with it. I know this is a pretty large change, so from a functionality point of view, I've made an asciicast to show me starting with a fresh workspace, switching bsp servers to sbt, switching back to bloop again, reloading the workspace, playing with the metals.build-* commands, etc. Just basically showing you how it functions fully with either chosen build server. I'll also go through the code and leave some comments where I feel they might be needed for better explanations.

There are also prs open in the VS Code and coc-metals repos to add in the extra command to help explore this as well.

asciicast

@eatkins
Copy link
Contributor
eatkins commented Oct 26, 2020

I was curious about this and I tried publishing a local version of this branch but it didn't seem to work. In a new project, there doesn't seem to be any option for metals.bsp-switch in the coc.nvim drop down after loading the project. Bloop was also clearly used for compilation. I did verify that the metals server was running the latest code.

As a side note, in addition to the metals.sbtScript, it would be nice to have the option of passing in the --client flag so that metals doesn't start a new sbt server if there is already one running. It actually would be nice to make sbt --client the default way of launching sbt from metals. Looking at the history, it seemed that the embedded launcher jar was used because of some problems with the launcher in windows back in 2018. sbt --client for sure won't work with sbt 1.4.1 and windows, but it should work on the other platforms and it should also work with windows in sbt 1.4.2. If not, we should probably investigate what the problem is on the sbt side. Even if sbt --client doesn't work on windows with metals with 1.4.2, I still think it might be a good idea to default to sbt --client on mac and linux for sbt versions >= 1.4.1 rather than using the embedded jar.

@ckipp01
Copy link
Member Author
ckipp01 commented Oct 27, 2020

I was curious about this and I tried publishing a local version of this branch but it didn't seem to work. In a new project, there doesn't seem to be any option for metals.bsp-switch in the coc.nvim drop down after loading the project. Bloop was also clearly used for compilation. I did verify that the metals server was running the latest code.

Hey! Thanks for trying to take it for a spin @eatkins! As mentioned up above, there are actually prs open in scalameta/coc-metals#320 and scalameta/metals-vscode#426 to help people explore the support since not all the command have been added to the extensions. Since you're using coc-metals, I actually just went ahead and merged in the metals.bsp-switch command since that has been in Metals for quite some time, just not used. The only command in the video you won't have access to yet (unless you want to build coc-metals from source) is the new metals.generate-bsp-config command. However, since you'll already have the .bsp/sbt.json generated, you should be fine with just starting the project, and then using the metals.bsp-switch command. So if you're willing, pull in the latest master of coc-metals and you should be good to go.

As a side note, in addition to the metals.sbtScript, it would be nice to have the option of passing in the --client flag so that metals doesn't start a new sbt server if there is already one running. It actually would be nice to make sbt --client the default way of launching sbt from metals. Looking at the history, it seemed that the embedded launcher jar was used because of some problems with the launcher in windows back in 2018. sbt --client for sure won't work with sbt 1.4.1 and windows, but it should work on the other platforms and it should also work with windows in sbt 1.4.2. If not, we should probably investigate what the problem is on the sbt side. Even if sbt --client doesn't work on windows with metals with 1.4.2, I still think it might be a good idea to default to sbt --client on mac and linux for sbt versions >= 1.4.1 rather than using the embedded jar.

Probably @olafurpg, @gabro, or @tgodzik will know a lot more since the decision was made so long ago, but it's a good point to bring up. I'll have to think about this a bit, and maybe this is a good discussion can start in another thread unless you think it's integral to consider those changes in this pr. I'm a bit hesitant at the moment since I imagine a lot of our users are still on sbt 1.3.x, so switching to default to --client should maybe happen, but I'm honestly unsure the best time to do that. What do you think?

@tgodzik
Copy link
Contributor
tgodzik commented Oct 27, 2020

Probably @olafurpg, @gabro, or @tgodzik will know a lot more since the decision was made so long ago, but it's a good point to bring up. I'll have to think about this a bit, and maybe this is a good discussion can start in another thread unless you think it's integral to consider those changes in this pr. I'm a bit hesitant at the moment since I imagine a lot of our users are still on sbt 1.3.x, so switching to default to --client should maybe happen, but I'm honestly unsure the best time to do that. What do you think?

The launcher is used if we don't know that the user has sbt installed and I think -client is valid also with the launcher (not sure). So this seems to be a separate problem altogether. We can start a new issue and discuss it there. I imagine using sbtn might also be something useful especially if we start the server itself before.

@eatkins
Copy link
Contributor
eatkins commented Oct 27, 2020

Thanks for trying to take it for a spin @eatkins! As mentioned up above, there are actually prs open in scalameta/coc-metals#320 and scalameta/metals-vscode#426 to help people explore the support since not all the command have been added to the extensions.

Whoops. I guess I glazed over that part.

The launcher is used if we don't know that the user has sbt installed and I think -client is valid also with the launcher (not sure). So this seems to be a separate problem altogether. We can start a new issue and discuss it there. I imagine using sbtn might also be something useful especially if we start the server itself before.

The launcher does support -client but it still spins up a jvm and takes about a second to start up (still better than spawning a new sbt process). The newer launchers scripts delegate sbt --client to sbtn so sbt --client bloopInstall can run very quickly. I manually overrode metals to run sbt --client instead of just sbt and it was very snappy. The reason to prefer sbt --client vs sbtn is that it should work even when sbt is not on the path, which isn't the case with sbtn (of course if sbtn is on the path, it's very unlikely that sbt isn't). That being said, I do appreciate that there is a bunch of work that might need to be done to transition metals to using sbt --client by default in some, if not all, cases.

@tgodzik
Copy link
Contributor
tgodzik commented Oct 27, 2020

@eatkins Ach, you are talking about using sbt script instead of the launcher. I missed that, we could for sure try to change it (at least for linux/macos), but I would still leave it for another issue.

Copy link
Contributor
@tgodzik tgodzik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow! Thanks for the amazing A+ work you did! I tested it out locally and it seems that the UX is good if not perfect. I left a couple of first comments, I will try to take a longer and closer look tomorrow.

Previously we weren't really capturing the capabilities. Since now we need to check
to see if the server supports reload requests, we carry it forward and check against
it later.

The `workspace/reload` refactor basically just allows now for all connections to be
reload, not just the main one. Plus it captures the error that the CompletableFuture
may have to display to the user that they tried to reload a workspace that isn't working.
Before this we have a SbtServer class to handle the initial and running and connecting
of the sbt BSP process. Instead now, we have a BuildServerProvider that is meant to be
mixed in to a BuildToold and provide the necessary functionality to generate a BSP
config file, and then connect to it.
@tgodzik
Copy link
Contributor
tgodzik commented Oct 30, 2020

I noticed that if you first connected to a BSP server it is no longer possible to generate Bloop config. Import build does not seem to run. Any idea what that is about?

@ckipp01
Copy link
Member Author
ckipp01 commented Oct 30, 2020

I noticed that if you first connected to a BSP server it is no longer possible to generate Bloop config. Import build does not seem to run. Any idea what that is about?

So part of this is expected. If you ignore the first prompt to import, the .bloop directory is never created. Then after you run generate-bsp-config, your choice of sbt has been saved. So at that point if you do a build-import, you're actually just triggering a workspace reload. So you'd need to reset the build server choice, which then when you do a build-import it will trigger you to import the build with Bloop.

However, I do notice that if .bloop doesn't exist and you try to do a bsp-switch, it won't allow you to switch back to Bloop. I should probably have it so that even in that scenario if Bloop has never been used and the user tries a bsp-switch, they still get an option for Bloop even though a .bloop doesn't' exist. wdyt?

@ckipp01
Copy link
Member Author
ckipp01 commented Oct 30, 2020

2020-10-30 16 40 19

This is what I explained above.

  • Build import doesn't trigger a bloopInstall but just reloads
  • bsp-switch warns that there is no other build server
  • reset-choice resets the build server choice
  • Build import now does a bloopInstall

@tgodzik
Copy link
Contributor
tgodzik commented Oct 30, 2020

So part of this is expected. If you ignore the first prompt to import, the .bloop directory is never created. Then after you run generate-bsp-config, your choice of sbt has been saved. So at that point if you do a build-import, you're actually just triggering a workspace reload. So you'd need to reset the build server choice, which then when you do a build-import it will trigger you to import the build with Bloop.

Oh, it makes sense. I was experimenting with a workspace switching the server back and forth - that's probably not something most people do, so it's a pretty niche use case.

However, I do notice that if .bloop doesn't exist and you try to do a bsp-switch, it won't allow you to switch back to Bloop. I should probably have it so that even in that scenario if Bloop has never been used and the user tries a bsp-switch, they still get an option for Bloop even though a .bloop doesn't' exist. wdyt?

Sounds good! I did run build import since that seemed analogous to generating config and then switching. I imagine it could also be a bit confusing to some users if it was difficult to switch back.

@ckipp01
Copy link
Member Author
ckipp01 commented Oct 30, 2020

Sounds good! I did run build import since that seemed analogous to generating config and then switching. I imagine it could also be a bit confusing to some users if it was difficult to switch back.

Sounds good. I'll special case this then and then a user can no matter what always do a bsp-switch and get back to using Bloop.

Copy link
Contributor
@tgodzik tgodzik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one small comment aside from the one before. We should be almost ready to merge. Great work! 🥇

@ckipp01
Copy link
Member Author
ckipp01 commented Oct 30, 2020

Alright, so I switched the logic up a bit. Now if a user has never imported their build and there is no .bloop but there is a .bsp/sbt.json they can do a bsp-switch which will show Bloop as an available server. If they choose it in this situation, we then trigger a bloopInstall. You can see this in the gif below:

2020-10-30 18 57 29

  • no .bloop exists
  • execute a bsp-switch and choose Bloop
  • bloopInstall happens

Copy link
Contributor
@tgodzik tgodzik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I think this is ready to let users test it out!

@ckipp01
Copy link
Member Author
ckipp01 commented Oct 30, 2020

Here we go!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Not a bug or a feature, but something general we can improve sbt server Relates to sbt BSP server support
Projects
None yet
5 participants
0