mail: only copy attributes for new IMAP folders

When mail_daemon associates a subscribed IMAP folder with a local
folder, ~/config/settings/Tracker/DefaultQueryTemplates/text_x-email
attributes are duplicated to the folder. Any customizations (column
layout, window size / location, etc) are overwritten by the values set
for text_x-email. The text_x-email template folder was originally meant
to be used by the 'New E-mail' query. In particular, the columns that
might be set for the query view aren't directly useful for an IMAP
folder - Account is useful for the query view, but not for an IMAP
folder, Status is not useful for the query view, but useful for an IMAP
folder.

This behavior is now modified such that attributes are only copied the
first time an IMAP folder is created. Subsequent launches of mail_daemon
will not overwrite per-folder customizations.

Fixes #18297

Change-Id: Iacdc78403681d5546668f28f7f0e34d1a57fd410
Reviewed-on: https://review.haiku-os.org/c/haiku/+/6202
Tested-by: Automation <automation@haiku-os.org>
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

mail: fix crash when associating IMAPFolder

_CreateFolder returns NULL when it's unable to create a folder on the
local filesystem to represent an IMAP folder. This can happen when a
file has the same name as an IMAP folder that has been subscribed to on
the remote server.

Handle this condition and do not try to add the folder to a connection worker.

Change-Id: I197d577fabe45360ffa9ce4ac07da34652e78fdd
Reviewed-on: https://review.haiku-os.org/c/haiku/+/6186
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

inbound_protocols/imap: Fix PVS V773

Fix IMAPProtocol::_CreateFolder() was exited
without releasing 'folder' pointer.

Change-Id: I80249694b7a977a220cfb0853b169bae83129f76
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2908
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

IMAP: sync fixes

This patch should fix:

- deadlocks while manual syncing/shutdown;
- crashes while manual syncing/shutdown.

Change-Id: I0962ff5580c19cbf740d002b6f62681ed5f558ab
Reviewed-on: https://review.haiku-os.org/758
Reviewed-by: Stephan Aßmus <superstippi@gmx.de>

mail: revert ~IMAPProtocol

Waiting of worker threads in IMAPProtocol destructor introducred new deadlocks.

For example: fWorkerLock grabbed in destructor leads to deadlock with
CheckSubscribedFoldersCommand (which grabs this lock with flock grabbed).

So, currently it should be reverted.

Change-Id: Ibc4b4f85300352e045d4ce72deb804e0e613f25c
Reviewed-on: https://review.haiku-os.org/714
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

mail: IMAP inbound protocol destructor fix

This should fix crashes of mailer_daemon in IMAP workers.
This patch makes IMAPConnection destructor wait for all working threads to finish.
Also, do not call _Connect for QuitCommand (it speedups shutdown procedure).

Change-Id: Idffcc00d3459a96a85a8a073a343bcf4cd4984be
Reviewed-on: https://review.haiku-os.org/686
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

BMailProtocol: Removed delete/append methods.

* The mail system is file based which makes these methods pretty much
superfluous.

IMAP: New folders will now adopt default mail attributes.

* Added a function CopyMailFolderAttributes() that copies the attribute
layout from the text/x-email default query folder.
* This using the new CopyAttributes() method in libshared that is pretty
much a copy of a similar method from copyattr. However, I did not
replace the latter, as that one allows for more fine grained error
reporting (and attribute filtering).
* Closes ticket #3498.

IMAP: setting flags, and body fetching works now.

* The direct methods in BMailProtocol now forward the request to the
looper; it's no longer the mail_daemon's responsibility to know
anything about that protocol.
* It's in desperate need of refactoring, but it doesn't hurt to add
it to the repository as is.

IMAP: Use BStringList instead of STL vector.

SMTP works again.

* The message to send the mails never made it to the add-on looper.
* Mail protocol threads now have names.
* Added a "public" BOutboundMailProtocol::SendMessages() call that sends
itself a message (even the correct one this time).

IMAP: fixed GCC4 build.

IMAP: downloading mails is now working.

* Changed the way the attributes are written to make sure that everything
that can be written once is in fact written just once.
* The rename code in BMailProtocol::_ProcessFetchedHeader() was broken,
and caused the hang of the last commit.

Mail Kit: Changed how filter changes work.

* BMailFilter::HeaderFetched() now only alters the entry_ref, and returns
B_MOVE_MAIL_ACTION to move a mail.
* Instead of potentially moving the file around several times, the
BMailProtocol now takes care of carrying out the filter action just once,
including trying to make the file name unique.
* This also allows the IMAP add-on to know the final location of the mail,
and thus downloading a message actually works.
* However, with my test inbox, it currently hangs, and a current Debugger does
not work on my older system -- I guess I need to update.
* Replaced the duplicated space mechanism within the "HaikuMailFormatFilter"
that is substantially faster, and handles all whitespace, not just spaces.
It will also replace tabs with spaces.

IMAP: work in progress of downloading the mail body.

* Most things are in place now, we just try to download the body to the
wrong file, as the final location is currently unknown.
* Added local only kPartialMessage flag for mails, but it's not being
used yet.

IMAP: sync when requested.

IMAP: Use useful default without destination.

* We now use the account name as destination directory.
* Turns out that an empty string is written as destination which worked
around the GetString() solution.

imap: Fetching headers is now working.

* No state is stored yet, though, so that all messages will be downloaded
over and over again.

imap: Testing for new messages is now working.

* There is now a CheckSubscribedFoldersCommand that is issued on the main
connection that triggers everything.
* The new CheckMailboxesCommand divides the new mail check into several
portions, and requeues itself until the next step, ie. the
FetchHeadersCommand.
* The headers of the new mails are downloaded, but nothing is done with
them yet.
* The actual check for the new mails doesn't scale that well yet, not sure
how to properly do this without having to rely on the mail indices. Might
be sensible to solve this via some simple heuristic.

imap: Fleshed out command processing.

* Still doesn't do anything useful, though.

imap: Work in progress of connection setup.

* SelectCommand now correctly encodes the mailbox name again.
* Hierarchy separators from the LIST/LSUB commands are now properly parsed,
and taken into account.
* Folders in the file system are now created for mailboxes on the server,
supporting hierarchical mailbox names.
* A connection worker now has an IMAPFolder that handles the local side of the
mailbox, and an IMAPMailbox that handles the server side.
* Connection workers are now created/deleted, and setup correctly. They will
now also wait in case they don't have a mailbox until they get some.

IMAP: The connection workers are now started.

* On SyncMessages(), the main connection worker is started if it is not already
running (it may only run already in idle mode).
* This will then list the subscribed folders, and create as many more connection
workers as allowed and make sense.
* Finally, it will distribute the folders to the workers which don't do anything
further yet.

IMAP: Early work in progress of main protocol class.

BMailProtocol: Removed delete/append methods.

* The mail system is file based which makes these methods pretty much
superfluous.

IMAP: New folders will now adopt default mail attributes.

* Added a function CopyMailFolderAttributes() that copies the attribute
layout from the text/x-email default query folder.
* This using the new CopyAttributes() method in libshared that is pretty
much a copy of a similar method from copyattr. However, I did not
replace the latter, as that one allows for more fine grained error
reporting (and attribute filtering).
* Closes ticket #3498.

IMAP: setting flags, and body fetching works now.

* The direct methods in BMailProtocol now forward the request to the
looper; it's no longer the mail_daemon's responsibility to know
anything about that protocol.
* It's in desperate need of refactoring, but it doesn't hurt to add
it to the repository as is.

IMAP: Use BStringList instead of STL vector.

SMTP works again.

* The message to send the mails never made it to the add-on looper.
* Mail protocol threads now have names.
* Added a "public" BOutboundMailProtocol::SendMessages() call that sends
itself a message (even the correct one this time).

IMAP: fixed GCC4 build.

IMAP: downloading mails is now working.

* Changed the way the attributes are written to make sure that everything
that can be written once is in fact written just once.
* The rename code in BMailProtocol::_ProcessFetchedHeader() was broken,
and caused the hang of the last commit.

Mail Kit: Changed how filter changes work.

* BMailFilter::HeaderFetched() now only alters the entry_ref, and returns
B_MOVE_MAIL_ACTION to move a mail.
* Instead of potentially moving the file around several times, the
BMailProtocol now takes care of carrying out the filter action just once,
including trying to make the file name unique.
* This also allows the IMAP add-on to know the final location of the mail,
and thus downloading a message actually works.
* However, with my test inbox, it currently hangs, and a current Debugger does
not work on my older system -- I guess I need to update.
* Replaced the duplicated space mechanism within the "HaikuMailFormatFilter"
that is substantially faster, and handles all whitespace, not just spaces.
It will also replace tabs with spaces.

IMAP: work in progress of downloading the mail body.

* Most things are in place now, we just try to download the body to the
wrong file, as the final location is currently unknown.
* Added local only kPartialMessage flag for mails, but it's not being
used yet.

IMAP: sync when requested.

IMAP: Use useful default without destination.

* We now use the account name as destination directory.
* Turns out that an empty string is written as destination which worked
around the GetString() solution.

imap: Fetching headers is now working.

* No state is stored yet, though, so that all messages will be downloaded
over and over again.

imap: Testing for new messages is now working.

* There is now a CheckSubscribedFoldersCommand that is issued on the main
connection that triggers everything.
* The new CheckMailboxesCommand divides the new mail check into several
portions, and requeues itself until the next step, ie. the
FetchHeadersCommand.
* The headers of the new mails are downloaded, but nothing is done with
them yet.
* The actual check for the new mails doesn't scale that well yet, not sure
how to properly do this without having to rely on the mail indices. Might
be sensible to solve this via some simple heuristic.

imap: Fleshed out command processing.

* Still doesn't do anything useful, though.

imap: Work in progress of connection setup.

* SelectCommand now correctly encodes the mailbox name again.
* Hierarchy separators from the LIST/LSUB commands are now properly parsed,
and taken into account.
* Folders in the file system are now created for mailboxes on the server,
supporting hierarchical mailbox names.
* A connection worker now has an IMAPFolder that handles the local side of the
mailbox, and an IMAPMailbox that handles the server side.
* Connection workers are now created/deleted, and setup correctly. They will
now also wait in case they don't have a mailbox until they get some.

IMAP: The connection workers are now started.

* On SyncMessages(), the main connection worker is started if it is not already
running (it may only run already in idle mode).
* This will then list the subscribed folders, and create as many more connection
workers as allowed and make sense.
* Finally, it will distribute the folders to the workers which don't do anything
further yet.

IMAP: Early work in progress of main protocol class.

IMAP: fixed GCC4 build.

IMAP: downloading mails is now working.

* Changed the way the attributes are written to make sure that everything
that can be written once is in fact written just once.
* The rename code in BMailProtocol::_ProcessFetchedHeader() was broken,
and caused the hang of the last commit.

Mail Kit: Changed how filter changes work.

* BMailFilter::HeaderFetched() now only alters the entry_ref, and returns
B_MOVE_MAIL_ACTION to move a mail.
* Instead of potentially moving the file around several times, the
BMailProtocol now takes care of carrying out the filter action just once,
including trying to make the file name unique.
* This also allows the IMAP add-on to know the final location of the mail,
and thus downloading a message actually works.
* However, with my test inbox, it currently hangs, and a current Debugger does
not work on my older system -- I guess I need to update.
* Replaced the duplicated space mechanism within the "HaikuMailFormatFilter"
that is substantially faster, and handles all whitespace, not just spaces.
It will also replace tabs with spaces.

IMAP: work in progress of downloading the mail body.

* Most things are in place now, we just try to download the body to the
wrong file, as the final location is currently unknown.
* Added local only kPartialMessage flag for mails, but it's not being
used yet.

IMAP: sync when requested.

IMAP: Use useful default without destination.

* We now use the account name as destination directory.
* Turns out that an empty string is written as destination which worked
around the GetString() solution.

imap: Fetching headers is now working.

* No state is stored yet, though, so that all messages will be downloaded
over and over again.

imap: Testing for new messages is now working.

* There is now a CheckSubscribedFoldersCommand that is issued on the main
connection that triggers everything.
* The new CheckMailboxesCommand divides the new mail check into several
portions, and requeues itself until the next step, ie. the
FetchHeadersCommand.
* The headers of the new mails are downloaded, but nothing is done with
them yet.
* The actual check for the new mails doesn't scale that well yet, not sure
how to properly do this without having to rely on the mail indices. Might
be sensible to solve this via some simple heuristic.

imap: Fleshed out command processing.

* Still doesn't do anything useful, though.

imap: Work in progress of connection setup.

* SelectCommand now correctly encodes the mailbox name again.
* Hierarchy separators from the LIST/LSUB commands are now properly parsed,
and taken into account.
* Folders in the file system are now created for mailboxes on the server,
supporting hierarchical mailbox names.
* A connection worker now has an IMAPFolder that handles the local side of the
mailbox, and an IMAPMailbox that handles the server side.
* Connection workers are now created/deleted, and setup correctly. They will
now also wait in case they don't have a mailbox until they get some.

IMAP: The connection workers are now started.

* On SyncMessages(), the main connection worker is started if it is not already
running (it may only run already in idle mode).
* This will then list the subscribed folders, and create as many more connection
workers as allowed and make sense.
* Finally, it will distribute the folders to the workers which don't do anything
further yet.

IMAP: Early work in progress of main protocol class.