IMAP connections dropping

[casner]

I use the email client alpine that attempts to maintain a continuous IMAP connection with the server at imap.sonic.net. I have my computer and the alpine client running continuously all day. Beginning a day or two ago, several times a day alpine has reported that the connection has dropped and I need to connect again. Has a timeout been imposed on IMAP connections?

[gkeller]

Are you still seeing this issue? We have not changed any behavior on the imap server recently, but we did migrate the service to a new network on 4/6. We have not seen reports of other issue like this since those changes where made. Anything interesting in your logs?

[casner]

Yes, this is still happening, many times per day. Sometimes the drop occurs when I am active in the email client, which suggests that the problem is not an inactivity timeout either on your end or in my home router's NAT state.

In the alpine journal there is just the message "MAIL FOLDER "INBOX" CLOSED DUE TO ACCESS ERROR". What other kinds of logs would you suggest?

I have verified that an ssh connection to sh.sonic.net remains solid while these imap drops are occurring.

[casner]

This problem of dropped IMAP connections is still happening.

In addition, a new problem started last evening. Two messages from March 2019 appeared in my message index as if they had been newly added amongst the genuinely current messages. This morning a block of about 50 messages from June and July 2019 was inserted in the middle of the list of new messages for today. When I picked one message from 7/18/2019 and then selected all messages on that date, the search took a long time, as if it was searching the entire (admittedly huge) mailbox. Some other message selections that have always been fast have also been slow like a full mailbox search.

What IMAP server is the Sonic mail server running? What mailbox format? Is there a way for me to look at the mailbox files on the server?

[gkeller]

Interesting. Unfortunately we no longer have user access to the mailspool. When you say you noticed yesterday evening, around what time? If it was after midnight, then it was likely due to an issue when your connection closed last night while we rebooted the cluster for maintenance. Shouldn't have happened, but it would help me troubleshoot. Messages duplicating like that whould be due to either the client for some reason duplicating them, or an issue with the index files on our side. To rule out the latter, I purged your indexes, which will cause them to be rebuilt the next time you connect. An issue with the index files may have caused the other problem, but that wouldn't have been my first thought. Let me know if clearing the indexes changed anything for you. Dovecot 2.1.17, currently running in maildir format.

[casner]

When you say you noticed yesterday evening, around what time? If it was after midnight, then it was likely due to an issue when your connection closed last night while we rebooted the cluster for maintenance.

No, before midnight.

Messages duplicating like that whould be due to either the client for some reason duplicating them, or an issue with the index files on our side. To rule out the latter, I purged your indexes, which will cause them to be rebuilt the next time you connect.

To be clear, the messages were not duplicated, they were just out of order with respect to arrival time. Your purge of the indexes did make a difference. The messages from 2019 that were mixed in with today's messages are no longer appearing there in my message list. A single message from 1-1-2018, which may be the message that should be first in the list, appeared near the end of my message list, just before the message telling me about your post here.

An issue with the index files may have caused the other problem, but that wouldn't have been my first thought. Let me know if clearing the indexes changed anything for you.

When I attempt to look at the list of messages starting from 1 in the index, I only get part of a screenful and then the IMAP connection closes. First time about 5 messages, then I reconnected and got 10, then most recently 35. So the index seems to build slowly, and the activity of building the index may have some effect on the connection closing.

[casner]

Hmmm, now more messages from the beginning of 2018 are showing up at the end of the message list.

[casner]

Each time I use alpine to try to look at messages early in the index only a partial page is shown, then blank entries below that, and the IMAP connection closes. That is still happening now, a few hours after you purged the index. It appears that each time I try to look at this list of early messages the next message after the partial list gets moved to the end of the full index after today's messages received so far.

It looks like the server's indexing mechanism is broken. Has the server code changed recently?

[casner]

Is it possible to purge the index and then force the server to do a complete rebuild on its own without any access from my client? That might be a way to get the index rebuilt without any messages being out of sequence. As it is now, I'm afraid to try checking whether the index is rebuilt because doing so it likely to cause another old message to be moved to the end.

[casner]

@gkeller, I called tech support but was informed that I need assistance from you or someone in your group. My plan would be to move my emails from 2018 to an offline file to reduce the mailbox size, but I can't do that if the server breaks the connection whenever I try to access emails from 2018. As I asked earlier, is there a way to purge the indexes again and then force the server to completely rebuild the index before I try to connect?

Or is there a server log that you could check for error messages regarding my account? I could coordinate with you to attempt access while you watch.