[pull] develop from twitter:develop #128
Open
+37,395
−10,368
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem/Solution: Lets add JVM expressions to the expressions endpoint JIRA Issues: CSL-11133 Differential Revision: https://phabricator.twitter.biz/D710867
Problem We want to release the next versions of our Twitter OSS libraries 21.8.0 - util - scrooge - finagle - twitter-server - finatra Solution Prepare libraries for their next releases. JIRA Issues: CSL-11162 Differential Revision: https://phabricator.twitter.biz/D722632
Problem/Solution: We currently need an entire ExpressionSchemaKey to query expressions. Lets add a utility to get a list of all expressions with a particular label Differential Revision: https://phabricator.twitter.biz/D722942
Problem
We want to release the next versions of our Twitter OSS libraries 21.8.0
util
scrooge
finagle
twitter-server
finatra
Solution
Prepare libraries for their next releases.
JIRA Issues: CSL-11162
Differential Revision: https://phabricator.twitter.biz/D723532
Problem We want to update to the next SNAPSHOT version of our Twitter OSS libraries 21.9.0 - util - scrooge - finagle - twitter-server - finatra Solution Prepare libraries for their next SNAPSHOT version. JIRA Issues: CSL-11162 Differential Revision: https://phabricator.twitter.biz/D722646
Problem: Sonatype expects a compiled javadoc JAR during the release and errors otherwise. Solution / Result: Allow sbt to publish an empty javadoc artifact JIRA Issues: CSL-11162 Differential Revision: https://phabricator.twitter.biz/D724170
Problem: Sonatype requires a javadoc jar, even if empty to release. Solution / Result: Revert the post-release commit to act as the new SHA to which will be released. This SHA will come after D724170 which doesn't skip the creation of the empty util javadoc JAR for Scala 3. JIRA Issues: CSL-11162 Differential Revision: https://phabricator.twitter.biz/D724302
Problem We want to update to the next SNAPSHOT version of our Twitter OSS libraries 21.9.0 - util - scrooge - finagle - twitter-server - finatra Solution Prepare libraries for their next SNAPSHOT version. JIRA Issues: CSL-11162 Differential Revision: https://phabricator.twitter.biz/D724396
**Problem** We're migrating util-cache to scala3. Dotty has new syntax rules that need to be addressed. **Solution** Use migration flags to solve syntax warnings. Add parentheses to make sure method call and definition match. JIRA Issues: CSL-11106 Differential Revision: https://phabricator.twitter.biz/D714304
**Problem** Build util-zk-test with Scala 3 **Solution** # Change util-zk-test project settings. # Compile project + test with Scala 3 migration flags # Remove flags # Fix incompatibilities # Run cross-building tests JIRA Issues: CSL-11093 Differential Revision: https://phabricator.twitter.biz/D720603
Problem We're migrating util-cache-guava to scala3. Dotty has new syntax rules that need to be addressed. Solution Use migration flags to solve syntax warnings. Add parentheses to make sure method call and definition match. JIRA Issues: CSL-11109 Differential Revision: https://phabricator.twitter.biz/D716101
**Problem** We're migrating util-slf4j-api to scala3. Dotty has new syntax rules that need to be addressed. This project also depends on util-mock and therefore on mockito-scala, which is incompatible with scala3. **Solution** Use migration flags to solve syntax warnings. Remove dependency on mockito-scala and use mockito-core instead in LoggingTest and MarkerLoggingTest. JIRA Issues: CSL-11110 Differential Revision: https://phabricator.twitter.biz/D725491
**Problem** We're migrating util-routing to scala3. Dotty has new syntax rules that need to be addressed. **Solution** Use migration flags to solve syntax warnings. Add parentheses to make sure method call and definition match. JIRA Issues: CSL-11113 Differential Revision: https://phabricator.twitter.biz/D727120
**Problem** We're migrating util-slf4j-jul-bridge to scala3. Dotty has new syntax rules that need to be addressed. **Solution** Use migration flags to solve syntax warnings. In this case there was no need for any changes other that the sharedScala3EnabledSettings JIRA Issues: CSL-11111 Differential Revision: https://phabricator.twitter.biz/D726468
**Problem** Build util-stats with Scala 3 **Solution** # Change util-stats project settings. # Compile project + test with Scala 3 migration flags # Remove flags # Fix incompatibilities # Run cross-building tests **Result** //Auto-rewrite fixes:// Scala 3 requires parentheses around the parameter of a lambda: - DelegatingStatsReceiverTest.scala //Other fixes:// Method `filterKeys` in trait MapOps is deprecated, should use .view.filterKeys: - CachedRegex.scala - CategorizingExceptionStatsHandlerTest.scala - MultiCategorizingExceptionStatsHandlerTest.scala `schemaKey` method must be called with `()`: - InMemoryStatsReceiver.scala Implicit conversion `long2float` in object Long is deprecated, need to be explicit: - Stat.scala - InMemoryStatsReceiverTest.scala - ExpressionSchemaTest.scala Cannot access apply method of case classes without specifying package: - Expression.scala Types in implicit definitions need to be given explicitly: - DelegatingStatsReceiverTest.scala Note: `print` changed to `println` in InMemoryStatsReceiver.scala to account for a Scala 3 bug where the split method does not work properly in string interpolation with special symbols. See https://jira.twitter.biz/browse/CSL-11259 JIRA Issues: CSL-11092 Differential Revision: https://phabricator.twitter.biz/D720514
Problem We're on an old version of jackson Solution Upgrade!! Differential Revision: https://phabricator.twitter.biz/D727879
Problem `App` is a trait that exposes non-final fields as `val`, which can result in access ordering issues when fields are overridden and then accessed by the base trait. While attempting to solve this problem, `Flags` overrides introduce other complications, as there are multiple ways to hook into flag parsing (via both `App#parseArgs` and `Flags#parseArgs`. Solution We move our `val` fields to `def` and expose the necessary properties that can be overridden to achieve the existing behavior, just more consistently. We make the `App#flags` def final and `Flags` class becomes final. The `App#name` val is similarly made a `def`. JIRA Issues: CSL-11231 Differential Revision: https://phabricator.twitter.biz/D723956
Problem/Solution Add metric description APIs in StatsReceiver and hydrate some key indicators. JIRA Issues: CSL-9962 Differential Revision: https://phabricator.twitter.biz/D615481
json4s: bump to 3.6.11 Differential Revision: https://phabricator.twitter.biz/D729764
Problem When parsing flags, we escape on the first parsing failure. This behavior results in poor user experience and odd side effects in certain scenarios. For the user, if they have multiple flag errors, they are forced to iterate and fix the errors 1-by-1. Without knowing that there are other parse failures to fix - this is a slow and time consuming feedback loop. In instances where an app references flag values as part of the lifecycle (i.e. as part of cleaning up/closing the app), a parse failure can result in remaining flags not being parsed, which cascades when we fail when flags are referenced, but not yet parsed, resulting in unclean shutdown of the app. Solution We will ensure that all flags get parsed consistently and that we roll-up errors as a single error result. We will also print a help message as part of a parse error, so that users are given the reason their flags were failed to be parsed, along with the expected usage for defined/available flags. Result A more streamlined experience for users attempting to remedy mistaken flag definitions and better behavior when flags are referenced during various phases of the application lifecycle. JIRA Issues: CSL-11231 Differential Revision: https://phabricator.twitter.biz/D729700
**Problem** Build util-jvm with Scala 3 **Solution** # Change util-jvm project settings. # Compile project + test with Scala 3 migration flags # Remove flags # Fix incompatibilities # Run cross-building tests **Result** //Auto-rewrite fixes:// Scala 3 requires parentheses around the parameter of a lambda: - Jvm.scala - JvmStatsTest.scala //Other fixes:// DurationOps expects a Long for as an input for operations. The conversion to long used to happen implicitly, but now needs to be explicit in order to work. - CpuProfile.scala - CpuProfileTest.scala Implicit conversion `long2float` in object Long is deprecated, need to be explicit: - JvmStats.scala getter `Stream` is deprecated, use `compat.LazyList` instead: - CpuProfileTest.scala Widening conversion from Long to Double is deprecated: - EstimatorApp.scala Scala 3 infers that executor is of type `ScheduledExecutorService` when it should be of type `MockScheduledExecutorService`. Explicitly give type to avoid error: - JvmTest.scala JIRA Issues: CSL-11094 Differential Revision: https://phabricator.twitter.biz/D728602
Problem/Solution/Result The link to Util's open source documentation was incorrect. Update the link. JIRA Issues: CSL-11285 Differential Revision: https://phabricator.twitter.biz/D732859
…ce.MethodPerEndpoint Problem Both `Service[Future]` and `Service.FutureIface` have been deprecated in Scrooge for a few years. Solution Migrate to a modern-day alternative - `Servive.MethodPerEndpoint`. Authors: Moses Nakamura, Yufan Gong, Joy Bestourous, Vladimir Kostyukov, Ian Bennett NO_USER_HOOK=1 Differential Revision: https://phabricator.twitter.biz/D741246
Problem
Reader.fromFuture has bug where duplicate calls to `read`
can race against the internal state. This was causing code
like this to fail:
```
test("Reader.fromFuture - unsatisfied") {
val p = Promise[Int]()
val r1 = Reader.fromFuture(p)
val f1 = r1.read()
val f2 = f1.flatMap(_ => r1.read())
p.setValue(1)
assert(await(f1) == Some(1))
// we already consumed the single item via the operation
// in `f1`, so `f2` should be empty since it's strictly
// sequenced after `f1`.
assert(await(f2) == None)
}
```
Solution
Ensure the internal state is set before returning a derivative
Future inside of `read`.
JIRA Issues: CSL-11326
Differential Revision: https://phabricator.twitter.biz/D744629
Problem Histogram expressions can represent a bunch of metrics: max/min/p99 etc. which means we have to either name them differently or put a label in expression. And using labels would be a better solution in terms of rendering expressions in dashboards. Solution We hold the Histogram bucket information upon creation, fill the label by defualt. JIRA Issues: CSL-11239 Differential Revision: https://phabricator.twitter.biz/D743299
Problem: If the future returned from Reader.readAllItems is interrupted, the interrupt is ignored. It can be desirable to instead propagate the interrupt back to the reader itself. Solution: If propagateInterrupts is set, discard the reader if the returned future is interrupted. Differential Revision: https://phabricator.twitter.biz/D745511
Problem ExpressionSchema#register() is mutable and can hide the errored operation. Solution Rename it to `build`. Change the return type to Try. Result Finagle doesn't fail when clients/servers are created using duplicated labels, hence we shouldn't fail with the same expression schema key. Return a Try to delegate error handling to the service owners who instrument their expressions. JIRA Issues: CSL-11284 Differential Revision: https://phabricator.twitter.biz/D747728
Problem: Regardess of having set `.withCounterishGauge()`, when building a gauge via the `MetricBuilder.gauge()` api, we always return a gauge schema. This makes the `.withCounterishGauge()`, in effect, non-functional. Solution: Rather than copying a `MetricBuilder` and resetting its `metricType`, simply verify that we're only calling the `.gauge()` `.counter()` or `.stat()` api on a `MetricBuilder` with the appropriate `metricType` JIRA Issues: CSL-11280, CSL-11336 Differential Revision: https://phabricator.twitter.biz/D749274
Problem/Solution: Update Jackson deps from 2.14.2 to 2.14.3 Differential Revision: https://phabricator.twitter.biz/D1069160
[Twitter-oss] bump release version Differential Revision: https://phabricator.twitter.biz/D1100185
Problem: /bin/bash is not available on non-FHS distros, such as NixOS Solution: Replace /bin/bash with /usr/bin/env bash Result: Closes #314 Differential Revision: https://phabricator.twitter.biz/D1107606
Problem/Solution: Add `deserializeAndFilterOutInvalidCertificates` which wraps the `deserializeX509` call in a `Try` (as `certificate.checkValidity()` can return `CertificateExpiredException`, `CertificateNotYetValidException`) and separates out any expired or not yet valid certificates detected. JIRA Issues: PSEC-16977 Differential Revision: https://phabricator.twitter.biz/D1107551
[Twitter-oss] bump release version Differential Revision: https://phabricator.twitter.biz/D1107767
Problem/Solution: Cut version 23.11 of: - util - scrooge - finagle - twitter-server - finatra Differential Revision: https://phabricator.twitter.biz/D1107956
Problem/Solution: Cut snapshot 24.2.0 of: - util - scrooge - finagle - twitter-server - finatra Differential Revision: https://phabricator.twitter.biz/D1108018
…n the application encounters an error on startup Problem When the application exits prematurely due to an exception encountered on startup, this exception is not printed to stderr, only the message "Exception thrown in main on startup". The exception stack trace is printed to stdout, but this is unintuitive for the user. Solution Include the exception and stack trace in the exception printed to stderr. Differential Revision: https://phabricator.twitter.biz/D1116753
[source] Update PROJECT owner references to reflect deactivated ldap Differential Revision: https://phabricator.twitter.biz/D1125334
Problem We want to release the next versions of our Twitter OSS libraries 24.2.0 - util - scrooge - finagle - twitter-server - finatra Solution Prepare libraries for their next releases. Differential Revision: https://phabricator.twitter.biz/D1140781
This reverts commit 996858fd593b43d101d3c546d5a9968dc5408855. There were issues with the release. Will be retrying it Differential Revision: https://phabricator.twitter.biz/D1141484
Problem We want to release the next versions of our Twitter OSS libraries 24.5.0 util scrooge finagle twitter-server finatra Solution Prepare libraries for their next releases. Differential Revision: https://phabricator.twitter.biz/D1141504
Problem We want to update to the next SNAPSHOT version of our Twitter OSS libraries 24.8.0-SNAPSHOT - util - scrooge - finagle - twitter-server - finatra Solution Prepare libraries for their next SNAPSHOT version. Differential Revision: https://phabricator.twitter.biz/D1141538
The `PrettyPrinter` configured for a Jackson `ObjectWriter` is shared between all downstream `JsonGenerator`s. To make this thread-safe (in order to prevent multiple generators from corrupting the shared indentation level), a class derived from `PrettyPrinter` can also inherit from `Instantiable`, which basically turns the `ObjectWriter`-level `PrettyPrinter` into a factory that produces a new `PrettyPrinter` for each generator. Since `util-jackson`'s `ArrayElementsOnNewLinesPrettyPrinter` inherits from `DefaultPrettyPrinter`, which has state that must be protected from corruption, we need to abide by this factory pattern. Differential Revision: https://phabricator.twitter.biz/D1167308
Unapply in pattern matching creates a tuple and unnecessarily boxes integers. Rewriting it in a more imperative style for efficiency. Differential Revision: https://phabricator.twitter.biz/D1173579
In a preparation for arm builds that require java 11 toolchains (as it is arm unlike java 8), migrating java/scala and code generation to java 11. (arm follow up diff https://phabricator.twitter.biz/D1171895) Main idea of the diff is to propagate -release 8 flag into `javac` and `scalac` that would compile classes into java 8 bytecode level as well as will use only java 8 API and won't leak java 11 APIs. One limitation of it is one cannot open up private packages like `sun.annotation.processor`. This makes targets that are using those private jdk packages uncompilable so I am just using java 8 for compilation of those (should not be a big deal and long term we should just get rid of java 8). Code generation tools are using java 11 as the runtime platform. In this diff: - adding java 11 toolchain with release 8 and propagating it through target_field_transforms.bzl that takes care of jdk transformations - using java8-do-not-use for targets that have private jdk packages. Have to override java8-do-not-use to java8 for scala_library as it has some REPL modules where one cannot override runtime_platform - making scrooge code generation use java 11 at runtime cls is failing because of unrelated diff: https://phabricator.twitter.biz/D1173582 Differential Revision: https://phabricator.twitter.biz/D1173098
Problem When a map is deserialized in scrooge we create it with default capacity, which means that for larger maps we need to resize it multiple times and each time recalculate key hashes Solution Allocate a map with sufficient capacity from the start. Differential Revision: https://phabricator.twitter.biz/D1174445
…shMap Differential Revision: https://phabricator.twitter.biz/D1175381
Problem ConsistentHashingDistributor uses TreeMap to find a node for a key. TreeMap is relatively slow. Solution Replace the TreeMap with two arrays, one sorted for keys and one matching the first for values. Use binarySearch to find the matching key. Because keys stored in array are not scattered in memory and are primitive values binary search is much faster. Based on ad-hoc benchmarking the gain is at least 50% Differential Revision: https://phabricator.twitter.biz/D1175980
Problem Currently we only have gc pause stats for the ParNew collector, but we'd like to be able to see this information for the G1 pools. Solution Add collection duration stats for all collector pools. Keep the jvm/gc/eden/pause_msec stat for compatibility, but this will be the same as jvm/gc/ParNew/pause_msec. Differential Revision: https://phabricator.twitter.biz/D1176049
…rFactory to track request burstiness Problem While we have a counter for the number of requests, we're limited by the frequency of metrics collection to see how these requests are spread out; that is, how "bursty" they are. Solution Introduce a HistogramCounter, created via a HistogramCounterFactory, that can be used to track request burstiness. The factory is configured on a client/server, and, if configured, is used in StatsFilter to track request burstiness. Differential Revision: https://phabricator.twitter.biz/D1180751
Problem Currently we only support ParNew collector for eden allocation stats. Solution Support G1 as well. Differential Revision: https://phabricator.twitter.biz/D1182243
1. making strato related tests concurrent as they are timing out on CI at this point 2. adding syncronized to Hotspot snapshot method as it is not thread safe. in particular when we lookup GC name we call PerfStringCounter that calls super PerfByteArrayCounter that resets buffer position to 0. if called concurrently one thread will read the counter resetting position to buffer length and we get BufferUnderflow i.e. race condition https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.management/share/classes/sun/management/counter/perf/PerfByteArrayCounter.java#L53 https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.management/share/classes/sun/management/counter/perf/PerfStringCounter.java#L64 Stack trace: https://ton.pdxa.twitter.com/scoot/prod_d70f787a-276c-4db8-7e76-36016e1ef442_08fbec44-75f9-482d-7f87-b240c9af3c1e/stdlog ``` generated columnDescs match generated ops(com.twitter.strato.config.compile_tests.ColumnDescTest) java.nio.BufferUnderflowException at java.base/java.nio.DirectByteBuffer.get(DirectByteBuffer.java:294) at java.base/java.nio.ByteBuffer.get(ByteBuffer.java:770) at java.management/sun.management.counter.perf.PerfByteArrayCounter.byteArrayValue(PerfByteArrayCounter.java:57) at java.management/sun.management.counter.perf.PerfStringCounter.stringValue(PerfStringCounter.java:64) at java.management/sun.management.counter.perf.PerfStringCounter.getValue(PerfStringCounter.java:58) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at com.twitter.jvm.Hotspot.$anonfun$getGc$5(Hotspot.scala:90) at scala.Option.map(Option.scala:230) at com.twitter.jvm.Hotspot.$anonfun$getGc$4(Hotspot.scala:90) at com.twitter.jvm.Hotspot.$anonfun$getGc$4$adapted(Hotspot.scala:89) at scala.Option.flatMap(Option.scala:271) at com.twitter.jvm.Hotspot.$anonfun$getGc$2(Hotspot.scala:89) at com.twitter.jvm.Hotspot.$anonfun$getGc$2$adapted(Hotspot.scala:88) at scala.Option.flatMap(Option.scala:271) at com.twitter.jvm.Hotspot.getGc(Hotspot.scala:88) at com.twitter.jvm.Hotspot.snap(Hotspot.scala:149) at com.twitter.strato.config.loading.ConfigResourceMonitoring$.collect(ConfigResourceMonitoring.scala:55) at com.twitter.strato.config.loading.CompiledStratoCatalog.<init>(CompiledStratoCatalog.scala:56) at com.twitter.strato.config.loading.CompiledStratoCatalog$.apply(CompiledStratoCatalog.scala:493) at com.twitter.strato.test.config.ConfigTestCatalog$$anon$1.mkConfigCatalog(ConfigTestCatalog.scala:333) at com.twitter.strato.test.config.ConfigTestCatalog$.apply(ConfigTestCatalog.scala:159) at com.twitter.strato.test.config.StratoConfigSuite.mkCatalog(StratoConfigSuite.scala:164) at com.twitter.strato.config.compile_tests.ColumnDescTest.$anonfun$new$1(ColumnDescTest.scala:18) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.twitter.servo.testing.prop.PropertySuite.$anonfun$test$1(PropertySuite.scala:71) ``` Differential Revision: https://phabricator.twitter.biz/D1188541
When Bufs are built by concatenation they are usually represented as a `Buf.Composite`.
Since this class is private, it is difficult to consume the underlying discontiguous Bufs without
copying. Making the class public allows constructions like:
```
def consume(buf: Buf)(eat: Array[Byte] => Unit): Unit = buf match {
case array: Buf.ByteArray => eat(array.unsafeRawByteArray) // [unsafeArrayOffset + length]
case Buf.ByteBuffer(bb) if bb.hasArray => eat(bb.array()) // [+ bb.arrayOffset()]
case Buf.Composite(bufs, _) => bufs.foreach(consume(_)(eat))
case _ => eat(Buf.ByteArray.Owned.extract(buf))
}
```
Differential Revision: https://phabricator.twitter.biz/D1198118
The first call to `LocalScheduler::get` registers the activation in a `Map[Activation, Boolean]` by writing a value of `java.lang.Boolean.TRUE`; this requires a boxing call which has negligible performance impact but can pollute profiles with many async dispatches. Differential Revision: https://phabricator.twitter.biz/D1209771
==== Problem Finagle doesn't have comprehensive support for dimensional metrics === Solution 1. Promote the visibility of `hierarchicalScope`, `dimensionalScope`, and `label` in `StatsReceiver` to public. This allows consumers to define their metric schemas for dual-consumption by the flat JSON & Prometheus exporters 2. Add the flag `com.twitter.finagle.stats.fullTranslation`. This allows service owners to express that raw `.scope` calls on `StatsReceiver` instances to propagate both hierarchical & dimensional metrics 3. Add the flag `com.twitter.finagle.stats.biasToFull`. This allows service owners to express that metrics with an indeterminate identity (have not been explicitly set as dimensional) to be exported through Prometheus anyways. 4. Remove the redundant dimensional scopes set on default Finagle client/service metrics 5. Remove the redundant `app` dimensional scope set on metrics exporter through `DefaultStatsReceiver` 6. Adjust, backwards-compatibly, the JVM metrics to utilize the new APIs and export the memory manager, pool, etc as a label instead of a hierarchical scope 7. Adjust, backwards-compatibly, some other internal use-cases to use labels instead of hierarchical scope Differential Revision: https://phabricator.twitter.biz/D1218090
=== Problem 1. Finagle & Finatra exceptions are reported only as hierarchical metrics. 2. Finatra route metrics are reported only as hierarchical metrics. 3. Certain Thrift client metrics are reported only as hierarchical metrics. === Solution 1. Continues to add first-class dimensional metrics support to Finagle by translating some hierarchically-scoped exception stats to labelled metrics. 2. Add dimensional equivalents to Finatra per-route metrics. 3. Report `ServicePerEndpoint` Thrift metrics dimensionally. Exceptions here are still hierarchical, for now. Differential Revision: https://phabricator.twitter.biz/D1218341
=== Problem `BroadcastStatsReceiver` hides the underlying receivers' translation mode. === Solution Just translate all metrics with `FullTranslation`. Differential Revision: https://phabricator.twitter.biz/D1220816
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )