8000 feat: messages and contacts history fetch by CayoPOliveira · Pull Request #71 · fazer-ai/chatwoot · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: messages and contacts history fetch #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
a5a9c89
feat: add messaging history set handler for WhatsApp and restore cont…
CayoPOliveira Jun 18, 2025
b247261
feat: enhance messaging history processing and message handling in Wh…
CayoPOliveira Jun 19, 2025
d773f1c
feat: include MessagingHistorySet handler in IncomingMessageBaileysSe…
CayoPOliveira Jun 19, 2025
49c1ac9
feat: add options to sync messages and contacts in WhatsApp settings
CayoPOliveira Jun 19, 2025
2a33ae7
feat: implement initial sync for contacts and message history in setu…
CayoPOliveira Jun 19, 2025
83dc898
chore: correct phone number variable usage in create_contact method
CayoPOliveira Jun 25, 2025
a999f8d
refactor: rename methods for avoid impact other baileys handlers
CayoPOliveira Jun 25, 2025
b08d42a
fix: correct argument order in history_handle_attach_media method call
CayoPOliveira Jun 25, 2025
76c2184
fix: improve error logging for attachment download failure
CayoPOliveira Jun 25, 2025
455c69a
fix: correct history_filename method in messaging_history.set
CayoPOliveira Jun 25, 2025
ae06267
chore: remove unused helper includes in messaging_history_set
CayoPOliveira Jun 25, 2025
d00f5e8
fix: variables names in attachment handling methods
CayoPOliveira Jun 25, 2025
48e159c
fix: update fetch_message_history method to include phone_number para…
CayoPOliveira Jun 25, 2025
35e4d2e
feat: add BAILEYS_MESSAGE_HISTORY_COUNT environment variable for mess…
CayoPOliveira Jun 25, 2025
a584148
fix: rename message_content method to history_message_content for cla…
CayoPOliveira Jun 26, 2025
23d88ed
feat: implement fetch_message_history method to retrieve message hist…
CayoPOliveira Jun 26, 2025
7f8275d
fix: simplify setup_channel_provider method by removing unnecessary s…
CayoPOliveira Jun 26, 2025
0ec7602
fix: update process_messaging_history_set to conditionally sync conta…
CayoPOliveira Jun 26, 2025
91ff533
fix: update create_contact method to include inbox parameter in Conta…
CayoPOliveira Jun 26, 2025
345f3cf
fix: rename params name in message creation and attachment creation
CayoPOliveira Jun 26, 2025
ad35f31
fix: reduce default message history count to 5 in fetch_message_histo…
CayoPOliveira Jun 27, 2025
2d01f7f
fix: refactor fetch_message_history methods to streamline parameters …
CayoPOliveira Jun 27, 2025
5e534d2
fix: validate presence of key, message, and messageTimestamp in histo…
CayoPOliveira Jun 27, 2025
16c0c1e
fix: add error handling for contact and message processing in process…
CayoPOliveira Jun 27, 2025
e2b634b
fix: add presence validation for key, message, and messageTimestamp i…
CayoPOliveira Jun 27, 2025
40e7cda
fix: update history_create_message method to use raw_message directly…
CayoPOliveira Jun 27, 2025
b841a1b
fix: add contact attributes to ContactInboxWithContactBuilder in crea…
CayoPOliveira Jun 27, 2025
80c5bcf
fix: update history_message_content_attributes to handle multiple uns…
CayoPOliveira Jun 27, 2025
682d263
fix: remove media attachment handling methods from history_handle_att…
CayoPOliveira Jun 27, 2025
8f03dbc
fix: remove redundant WhatsApp sync labels from inbox management loca…
CayoPOliveira Jun 27, 2025
471b441
fix: update sync label order in inbox management localization file
CayoPOliveira Jun 27, 2025
8745ec0
fix: correct phone number attribute in fetch_message_history method
CayoPOliveira Jun 27, 2025
48a06ad
fix: enhance message flooding prevention and update fetch_message_his…
gabrieljablonski Jun 30, 2025
d86cf1e
chore: update fetch_message_history method to accept oldest_message p…
CayoPOliveira Jun 30, 2025
456ff17
chore: add fetch_message_history to error handling methods in Whatsap…
CayoPOliveira Jun 30, 2025
23a9246
fix: handle blank contact IDs and update name assignment logic in cre…
CayoPOliveira Jun 30, 2025
ffcc625
fix: update history_cache_message_source_id_in_redis to use set with …
CayoPOliveira Jun 30, 2025
6327952
test: create spec in message model for skip message flooding validati…
CayoPOliveira Jun 30, 2025
e93d18a
fix: skip processing of blank contact IDs in messaging history set
CayoPOliveira Jul 1, 2025
8cd5da3
refactor: remove unused process_status method from MessagingHistorySet
CayoPOliveira Jul 1, 2025
1859630
refactor: simplify process_messaging_history_set method by removing u…
CayoPOliveira Jul 1, 2025
0d47da5
refactor: simplify process_messaging_history_set and create_contact v…
CayoPOliveira Jul 1, 2025
5f15fa4
fix: exclude unsupported message types from ignore processing list in…
CayoPOliveira Jul 1, 2025
05a6569
refactor: update conversation creation to use history_conversation_pa…
CayoPOliveira Jul 1, 2025
b1cf9c8
test: add specs for messaging-history.set event handling
CayoPOliveira Jul 1, 2025
e38f893
refactor: streamline process_messaging_history_set by consolidating c…
CayoPOliveira Jul 1, 2025
1537b1d
refactor: enhance history_handle_message and history_message_valid me…
CayoPOliveira Jul 1, 2025
8cc5b26
test: add specs for sync_full_history and sync_contacts in setup_chan…
CayoPOliveira Jul 1, 2025
dde9ed4
chore: update fetch_message_history to ensure request body is properl…
CayoPOliveira Jul 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -266,3 +266,6 @@ BAILEYS_PROVIDER_DEFAULT_URL=http://localhost:3025
BAILEYS_PROVIDER_DEFAULT_API_KEY=

RESEND_API_KEY=

# Baileys API Whatsapp configurations
BAILEYS_MESSAGE_HISTORY_COUNT=50
4 changes: 4 additions & 0 deletions app/javascript/dashboard/i18n/locale/en/inboxMgmt.json
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,10 @@
"MARK_AS_READ": {
"LABEL": "Send read receipts"
},
"SYNC_FULL_HISTORY": {
"ONLY_CONTACTS_LABEL": "Sync only contacts",
"LABEL": "Sync messages and contacts"
},
"ADVANCED_OPTIONS": "Advanced options",
"BAILEYS": {
"SUBTITLE": "Click below to setup the WhatsApp channel using Baileys.",
Expand Down
4 changes: 4 additions & 0 deletions app/javascript/dashboard/i18n/locale/pt_BR/inboxMgmt.json
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,10 @@
"MARK_AS_READ": {
"LABEL": "Enviar confirmações de leitura"
},
"SYNC_FULL_HISTORY": {
"ONLY_CONTACTS_LABEL": "Sincronizar apenas contatos",
"LABEL": "Sincronizar contatos e mensagens"
},
"ADVANCED_OPTIONS": "Opções avançadas",
"BAILEYS": {
"SUBTITLE": "Clique abaixo para configurar o canal do WhatsApp usando o Baileys.",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ export default {
providerUrl: '',
showAdvancedOptions: false,
markAsRead: true,
syncContacts: false,
syncFullHistory: false,
};
},
computed: {
Expand All @@ -53,6 +55,8 @@ export default {
try {
const providerConfig = {
mark_as_read: this.markAsRead,
sync_contacts: this.syncContacts,
sync_full_history: this.syncFullHistory,
};

if (this.apiKey || this.providerUrl) {
Expand Down Expand Up @@ -177,6 +181,34 @@ export default {
</div>
</label>
</div>

<div
v-if="!syncFullHistory"
class="w-[65%] flex-shrink-0 flex-grow-0 max-w-[65%]"
>
<label>
<div class="flex mb-2 items-center">
<span class="mr-2 text-sm">
{{
$t(
'INBOX_MGMT.ADD.WHATSAPP.SYNC_FULL_HISTORY.ONLY_CONTACTS_LABEL'
)
}}
</span>
<Switch id="syncContacts" v-model="syncContacts" />
</div>
</label>
</div>
<div class="w-[65%] flex-shrink-0 flex-grow-0 max-w-[65%]">
<label>
<div class="flex mb-2 items-center">
<span class="mr-2 text-sm">
{{ $t('INBOX_MGMT.ADD.WHATSAPP.SYNC_FULL_HISTORY.LABEL') }}
</span>
<Switch id="syncFullHistory" v-model="syncFullHistory" />
</div>
</label>
</div>
</template>

<div class="w-full">
Expand Down
6 changes: 6 additions & 0 deletions app/models/channel/whatsapp.rb
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,12 @@ def received_messages(messages, conversation)
provider_service.received_messages(conversation.contact.phone_number, messages)
end

def fetch_message_history(oldest_message)
return unless provider_service.respond_to?(:fetch_message_history)

provider_service.fetch_message_history(oldest_message)
end

delegate :setup_channel_provider, to: :provider_service
delegate :send_message, to: :provider_service
delegate :send_template, to: :provider_service
Expand Down
3 changes: 2 additions & 1 deletion app/models/message.rb
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ class Message < ApplicationRecord
}.to_json.freeze

before_validation :ensure_content_type
before_validation :prevent_message_flooding
before_validation :prevent_message_flooding, unless: :skip_prevent_message_flooding
before_save :ensure_processed_message_content
before_save :ensure_in_reply_to

Expand All @@ -78,6 +78,7 @@ class Message < ApplicationRecord

# when you have a temperory id in your frontend and want it echoed back via action cable
attr_accessor :echo_id
attr_accessor :skip_prevent_message_flooding

enum message_type: { incoming: 0, outgoing: 1, activity: 2, template: 3 }
enum content_type: {
Expand Down
207 changes: 207 additions & 0 deletions app/services/whatsapp/baileys_handlers/messaging_history_set.rb
F438
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
module Whatsapp::BaileysHandlers::MessagingHistorySet # rubocop:disable Metrics/ModuleLength
private

def process_messaging_history_set
provider_config = inbox.channel.provider_config

return unless provider_config['sync_contacts'].presence || provider_config['sync_full_history'].presence

process_contacts(params)
process_messages(params) if provider_config['sync_full_history'].presence
end

def process_contacts(params)
contacts = params.dig(:data, :contacts) || []
contacts.each do |contact|
create_contact(contact)
end
end

def process_messages(params)
messages = params.dig(:data, :messages) || []
messages.each do |message|
history_handle_message(message)
end
end

def create_contact(contact)
return unless contact[:id].present? && jid_user?(contact[:id])

phone_number = history_phone_number_from_jid(contact[:id])
name = contact[:verifiedName].presence || contact[:notify].presence || contact[:name].presence || phone_number
::ContactInboxWithContactBuilder.new(
# FIXME: update the source_id to complete jid in future
source_id: phone_number,
inbox: inbox,
contact_attributes: { name: name, phone_number: "+#{phone_number}" }
).perform
end

# TODO: Refactor jid_type method in helpers to receive the jid as an argument and use it here
def jid_user?(jid)
server = jid.split('@').last
server == 's.whatsapp.net' || server == 'c.us'
end

# TODO: Refactor this method in helpers to receive the jid as an argument and remove it from here
def history_phone_number_from_jid(jid)
jid.split('@').first.split(':').first.split('_').first
end

def history_handle_message(raw_message)
return unless history_message_valid?(raw_message)

id = raw_message.dig(:key, :id)
jid = raw_message.dig(:key, :remoteJid)

history_cache_message_source_id_in_redis(id)
begin
contact_inbox = find_contact_inbox(jid)
unless contact_inbox.contact
Rails.logger.warn "Contact not found for message: #{id}"
return
end

history_create_message(raw_message, contact_inbox)
ensure
history_clear_message_source_id_from_redis(id)
end
end

def history_message_valid?(raw_message) # rubocop:disable Metrics/CyclomaticComplexity
id = raw_message.dig(:key, :id)
jid = raw_message.dig(:key, :remoteJid)

id.present? &&
jid.present? &&
raw_message[:message].present? &&
raw_message[:messageTimestamp].present? &&
jid_user?(jid) &&
!history_message_type(raw_message[:message]).in?(%w[protocol context]) &&
!history_find_message_by_source_id(id) &&
!history_message_under_process?(id)
end

# TODO: Refactor this method in helpers to receive the raw message as an argument and remove it from here
def history_message_type(message_content) # rubocop:disable Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity,Metrics/MethodLength
if message_content.key?(:conversation) || message_content.dig(:extendedTextMessage, :text).present?
'text'
elsif message_content.key?(:imageMessage)
'image'
elsif message_content.key?(:audioMessage)
'audio'
elsif message_content.key?(:videoMessage)
'video'
elsif message_content.key?(:documentMessage) || message_content.key?(:documentWithCaptionMessage)
'file'
elsif message_content.key?(:stickerMessage)
'sticker'
elsif message_content.key?(:reactionMessage)
'reaction'
elsif message_content.key?(:editedMessage)
'edited'
elsif message_content.key?(:protocolMessage)
'protocol'
elsif message_content.key?(:messageContextInfo)
'context'
else
'unsupported'
end
end

# TODO: Remove this method when include helpers in this module, after update the methods to receive arguments
def history_find_message_by_source_id(source_id)
return unless source_id

Message.find_by(source_id: source_id).presence
end

def find_contact_inbox(jid)
phone_number = history_phone_number_from_jid(jid)
::ContactInboxWithContactBuilder.new(
# FIXME: update the source_id to complete jid in future
source_id: phone_number,
inbox: inbox,
contact_attributes: { name: phone_number, phone_number: "+#{phone_number}" }
).perform
end

# TODO: Refactor this method in helpers to receive the source_id as an argument and remove it from here
def history_message_under_process?(source_id)
key = format(Redis::RedisKeys::MESSAGE_SOURCE_KEY, id: source_id)
Redis::Alfred.get(key)
end

# TODO: Refactor this method in helpers to receive the source_id as an argument and deprecate setex, then remove it from here
def history_cache_message_source_id_in_redis(source_id)
key = format(Redis::RedisKeys::MESSAGE_SOURCE_KEY, id: source_id)
::Redis::Alfred.set(key, true, nx: true, ex: 1.day)
end

# TODO: Refactor this method in helpers to receive the source_id as an argument and remove it from here
def history_clear_message_source_id_from_redis(source_id)
key = format(Redis::RedisKeys::MESSAGE_SOURCE_KEY, id: source_id)
::Redis::Alfred.delete(key)
end

def history_create_message(raw_message, contact_inbox)
conversation = get_conversation(contact_inbox)
inbox = contact_inbox.inbox
message = conversation.messages.build(
skip_prevent_message_flooding: true,
content: history_message_content(raw_message),
account_id: inbox.account_id,
inbox_id: inbox.id,
source_id: raw_message[:key][:id],
sender: history_incoming?(raw_message) ? contact_inbox.contact : inbox.account.account_users.first.user,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add safety check for account users to prevent nil reference error.

The code assumes inbox.account.account_users.first.user will always exist, but this could return nil if no account users are present.

 message = conversation.messages.build(
   content: history_message_content(raw_message),
   account_id: inbox.account_id,
   inbox_id: inbox.id,
   source_id: raw_message[:key][:id],
-  sender: history_incoming?(raw_message) ? contact_inbox.contact : inbox.account.account_users.first.user,
+  sender: history_incoming?(raw_message) ? contact_inbox.contact : (inbox.account.account_users.first&.user || contact_inbox.contact),
   sender_type: history_incoming?(raw_message) ? 'Contact' : 'User',
   message_type: history_incoming?(raw_message) ? :incoming : :outgoing,
   content_attributes: history_message_content_attributes(raw_message),
   status: process_status(raw_message[:status]) || 'sent'
 )
🤖 Prompt for AI Agents
In app/services/whatsapp/baileys_handlers/messaging_history_set.rb at line 148,
the code assumes inbox.account.account_users.first.user always exists, which can
cause a nil reference error if account_users is empty. Add a safety check to
verify that account_users.first and its user are present before accessing user,
and provide a fallback value or handle the nil case appropriately to prevent
errors.

sender_type: history_incoming?(raw_message) ? 'Contact' : 'User',
message_type: history_incoming?(raw_message) ? :incoming : :outgoing,
content_attributes: history_message_content_attributes(raw_message),
status: 'read'
)

message.save!
end

# NOTE: See reference in app/services/whatsapp/incoming_message_base_service.rb:97
def get_conversation(contact_inbox)
return contact_inbox.conversations.last if contact_inbox.inbox.lock_to_single_conversation

# NOTE: if lock to single conversation is disabled, create a new conversation if previous conversation is resolved
return contact_inbox.conversations.where.not(status: :resolved).last.presence ||
::Conversation.create!(history_conversation_params(contact_inbo 1C6A x))
end

# TODO: Refactor this method in helpers to receive the contact_inbox as an argument and remove it from here
def history_conversation_params(contact_inbox)
{
account_id: contact_inbox.inbox.account_id,
inbox_id: contact_inbox.inbox.id,
contact_id: contact_inbox.contact.id,
contact_inbox_id: contact_inbox.id
}
end

# TODO: Refactor this method in helpers to receive the raw message as an argument and remove it from here
def history_incoming?(raw_message)
!raw_message[:key][:fromMe]
end

# TODO: Refactor this method in helpers to receive the raw message as an argument and remove it from here
def history_message_content(raw_message)
raw_message.dig(:message, :conversation) ||
raw_message.dig(:message, :extendedTextMessage, :text) ||
raw_message.dig(:message, :imageMessage, :caption) ||
raw_message.dig(:message, :videoMessage, :caption) ||
raw_message.dig(:message, :documentMessage, :caption).presence ||
raw_message.dig(:message, :documentWithCaptionMessage, :message, :documentMessage, :caption) ||
raw_message.dig(:message, :reactionMessage, :text)
end

def history_message_content_attributes(raw_message)
{
external_created_at: baileys_extract_message_timestamp(raw_message[:messageTimestamp]),
is_unsupported: history_message_type(raw_message[:message]).in?(%w[image file video audio sticker unsupported]) || nil
}.compact
end
end
1 change: 1 addition & 0 deletions app/services/whatsapp/incoming_message_baileys_service.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ class Whatsapp::IncomingMessageBaileysService < Whatsapp::IncomingMessageBaseSer
include Whatsapp::BaileysHandlers::ConnectionUpdate
include Whatsapp::BaileysHandlers::MessagesUpsert
include Whatsapp::BaileysHandlers::MessagesUpdate
include Whatsapp::BaileysHandlers::MessagingHistorySet

class InvalidWebhookVerifyToken < StandardError; end

Expand Down
28 changes: 25 additions & 3 deletions app/services/whatsapp/providers/whatsapp_baileys_service.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,18 @@ class MessageNotSentError < StandardError; end
DEFAULT_API_KEY = ENV.fetch('BAILEYS_PROVIDER_DEFAULT_API_KEY', nil)

def setup_channel_provider
provider_config = whatsapp_channel.provider_config

response = HTTParty.post(
"#{provider_url}/connections/#{whatsapp_channel.phone_number}",
headers: api_headers,
body: {
clientName: DEFAULT_CLIENT_NAME,
webhookUrl: whatsapp_channel.inbox.callback_webhook_url,
webhookVerifyToken: whatsapp_channel.provider_config['webhook_verify_token'],
webhookVerifyToken: provider_config['webhook_verify_token'],
# TODO: Remove on Baileys v2, default will be false
includeMedia: false
includeMedia: false,
syncFullHistory: provider_config['sync_contacts'].presence || provider_config['sync_full_history'].presence
}.compact.to_json
)

Expand Down Expand Up @@ -155,6 +158,24 @@ def unread_message(phone_number, message) # rubocop:disable Metrics/MethodLength
process_response(response)
end

def fetch_message_history(oldest_message)
response = HTTParty.post(
"#{provider_url}/connections/#{whatsapp_channel.phone_number}/fetch-message-history",
headers: api_headers,
body: {
count: ENV.fetch('BAILEYS_MESSAGE_HISTORY_COUNT', 50).to_i,
oldestMsgKey: {
id: oldest_message[:key][:id],
remoteJid: oldest_message[:key][:remoteJid],
fromMe: oldest_message[:key][:fromMe]
},
oldestMsgTimestamp: oldest_message[:messageTimestamp]
}.to_json
)

process_response(response)
end

def received_messages(phone_number, messages)
@phone_number = phone_number

Expand Down Expand Up @@ -277,5 +298,6 @@ def handle_channel_error
:update_presence,
:read_messages,
:unread_message,
:received_messages
:received_messages,
:fetch_message_history
end
10 changes: 10 additions & 0 deletions spec/models/message_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,16 @@
expect(conv_new_message.errors[:base]).to eq(['Too many messages'])
end
end

it 'skips message flooding validation if skip_prevent_message_flooding is true' do
with_modified_env 'CONVERSATION_MESSAGE_PER_MINUTE_LIMIT': '2' do
conversation = message.conversation
create(:message, conversation: conversation)
conv_new_message = build(:message, conversation: message.conversation, skip_prevent_message_flooding: true)

expect(conv_new_message.valid?).to be true
end
end
end
end

Expand Down
Loading
0