feat: TruncateFront & TruncateBack support truncating all entries #31
Conversation
|
This looks good on the surface. |
I'll add some tests, might also adjust current test cases if needed. |
|
While I was writing tests, a question raised that needs your help to answer. With this change, the WAL may be in a state without any entries during use. As a result, the behavior of In the following two comments, these two functions are expected to return 0 when the log is empty. Lines 484 to 486 in 56126d5 Lines 502 to 504 in 56126d5 However, if the WAL is in a continuous read/write state, returning 0 might cause some confusion for users. Let's temporarily ignore the special handling of empty entries in these two functions and consider the following usage scenario: There are two Go routines—one continuously appends entries to the WAL, while the other continuously retrieves entries from the WAL. The code is as follows: func main() {
notify := make(chan struct{}, 1)
l, _ := wal.Open("wal", nil)
go func() {
for {
time.Sleep(1 * time.Second)
idx, _ := l.LastIndex()
idx++
l.Write(idx, []byte(strconv.FormatUint(idx, 10)+"hello world"))
select {
case notify <- struct{}{}:
default:
}
}
}()
go func() {
for {
select {
case <-notify:
}
for {
idx, _ := l.FirstIndex()
data, err := l.Read(idx)
if errors.Is(err, wal.ErrNotFound) {
break
}
fmt.Println(string(data))
l.TruncateFront(idx + 1)
}
}
}()
select {}
}Relatively speaking, this is a straightforward implementation of the producer-consumer pattern, where each Go routine only needs to focus on one end of the queue. In this case, Now, returning to the special handling of empty entries in these functions, the above code is actually just one step away from running perfectly: Since idx, _ := l.FirstIndex()
if idx == 0 {
idx = 1
}This isn't particularly complicated, but it may confuse users until they carefully read the source code of Currently, the implementation of these two functions only includes special handling for one specific case: when For Disregarding historical reasons, a feasible implementation would be to return an additional boolean value to indicate whether the WAL is empty. So the key question that needs to be clarified is:
|
Currently, it seems like there is no way to truncate entire log, which causes the duplicate transaction issue (#20).
This PR provides the ability of truncating all entries, by extending the index range 1 step further than the last(TruncateFront) or 1 step before the first(TruncateBack) entry.
For example, if the log has 3 entries
[10, 11, 12]:TruncateFront(13)will truncate all entries, and leftfirstIndexas 13 andlastIndexas 12.TruncateBack(9)will truncate all entries, and leftfirstIndexas 10 andlastIndexas 9.Why this feature is needed
While I was working on a project that uses this package as WAL implementation, I've met the situation that described in #20. I can't simply mark all jobs as done if the program exits unexpectedly, and next time it starts, it will duplicate the last job. So I need a way to remove this last job from wal once it's done.
With this PR, I can call
TruncateFront(index + 1)when I've poppedindexth job from wal, and let the program wait for incomingindex + 1th job.What's the difference between #29 and this PR
Thanks to @DStrand1 for providing a solution. However, #29 can't fulfill my requirements, by providing a
Clearfunction.Complexity of usage
In my use case, I'm running a long-lived goroutine as a consumer(
Read), and a gin server as producers(Write).With
Clearfrom #29, after the consumer finishes all jobs, it will first detect whether wal is empty (by either callingRead(index + 1)withErrNotFoundreturned, or checking a seperate counter), and then callClearto remove all entries. The problem is that, these two steps are not atomic, so I have to add another lock to protect them which is complicated.With this PR, both
TruncateFrontandTruncateBackhold internal lock, so they are atomic. And caller don't need to care about how many jobs left in wal, just callReadandTruncateFrontin a loop untilReadreturnsErrNotFound, then wait for incoming jobs.Corruption-safty
Clearfunction in #29 isn't corruption-safe, like #29 (comment) pointed out.This PR modifies
TruncateFrontandTruncateBack, following the original corruption-safe mechanism.Index continuity
#29 clears all entries, also resets
firstIndexandlastIndexto 1 and 0. BothWriteandReadhave to reset its own index counter afterClear.This PR keeps
firstIndexandlastIndexcontinuous,WriteandReadcan keep using previous index counter.