telego: Bad Request: entity begins in a middle of a UTF-16 symbol at byte offset ...

💬 Telego version

0.22.0

👾 Issue description

entitys := []tu.MessageEntityCollection{} entitys = append(entitys, tu.Entity("🌗")) bot.SendMessage(tu.MessageWithEntities(tu.ID(ctm.Chat.ID), entitys...).WithReplyToMessageID(ctm.MessageID)) //[Thu Apr 13 13:07:41 MSK 2023] DEBUG API call to: "https://api.telegram.org/botBOT_TOKEN/sendMessage", with data: {"chat_id":1,"text":"🌗Last quarter","entities":[{"type":"code","offset":1,"length":12}],"reply_to_message_id":1} //[Thu Apr 13 13:07:41 MSK 2023] DEBUG API response sendMessage: Ok: false, Err: [400 "Bad Request: entity begins in a middle of a UTF-16 symbol at byte offset 4"] Please look: https://core.telegram.org/bots/api#messageentity MessageEntity … offset Integer Offset in UTF-16 code units to the start of the entity length Integer Length of the entity in UTF-16 code units … https://core.telegram.org/api/entities#entity-length Computing entity length Code points in the BMP (U+0000 to U+FFFF) count as 1, because they are encoded into a single UTF-16 code unit Code points in all other planes count as 2, because they are encoded into two UTF-16 code units (also called surrogate pairs) … However, since UTF-8 encodes codepoints in non-BMP planes as a 32-bit code unit starting with 0b11110, a more efficient way to compute the entity length without converting the message to UTF-16 is the following:

If the byte marks the beginning of a 32-bit UTF-8 code unit (all bytes starting with 0b11110) increment the count by 2, otherwise If the byte marks the beginning of a UTF-8 code unit (all bytes not starting with 0b10) increment the count by 1. Example:

length := 0 for byte in text { if (byte & 0xc0) != 0x80 { length += 1 + (byte >= 0xf0) } } Note: the length of an entity must not include the length of trailing newlines or whitespaces, rtrim entities before computing their length: however, the next offset must include the length of newlines or whitespaces that precede it.

⚡️ Expected behavior

{code 2 12 <nil> }

🧐 Code example

`					entitys := []tu.MessageEntityCollection{}
					entitys = append(entitys, tu.Entity("🌗"))
					t, es := tu.MessageEntities(entitys...)
					stdo.Printf("%U %v %v\n", []rune(t)[0], []byte(t), es) //15:46:48 main.go:471: U+1F317 [240 159 140 151] []
					entitys = append(entitys, tu.Entity("Last quarter").Code())
					t, es = tu.MessageEntities(entitys...)
					stdo.Printf("%s %v\n", t, es) //15:46:48 main.go:474: 🌗Last quarter [{code 1 12  <nil>  }]
					bot.SendMessage(tu.MessageWithEntities(tu.ID(ctm.Chat.ID), entitys...).WithReplyToMessageID(ctm.MessageID))
//[Thu Apr 13 13:07:41 MSK 2023] DEBUG API call to: "https://api.telegram.org/botBOT_TOKEN/sendMessage", with data: {"chat_id":1,"text":"🌗Last quarter","entities":[{"type":"code","offset":1,"length":12}],"reply_to_message_id":1}
//[Thu Apr 13 13:07:41 MSK 2023] DEBUG API response sendMessage: Ok: false, Err: [400 "Bad Request: entity begins in a middle of a UTF-16 symbol at byte offset 4"]
`