[Alpine-info] O365 XOAUTH2 via fetchmail

[Carl Edquist]

Hi Eduardo,

Thanks a bunch for taking the time to give a thoughtful reply!

I appreciate getting your philosophical take on these things.

It strikes right to the heart on a number of points!

...


> This is somewhat off-topic.


I wanted to ask here in part because i thought i had remembered seeing 
something in the past about a script for fetchmail with oauth2 in 
connection with alpine, but likely i had some details mixed up in my 
memory.



> I hope the maintainer of fetchmail decides to include OAUTH2 support.


Actually (and maybe this will be potentially interesting to some others 
here), apparently the development branch for the unreleased fetchmail 7 
does contain some amount of oauth2 support, despite the maintainer's 
original reluctance.  I had heard it was added for gmail, and i wasn't 
sure if it worked for O365 - though i knew Eduardo had gotten it working 
for alpine (because i've been using it successfully).

(I will probably give the development version of fetchmail 7 a try (it 
seems to come with an oauth2 renewal script), though it seemed worth a try 
to see if anyone here had any tips before wasting too much time probably 
not getting it right.)

...

Notably, i am very happy with alpine and its oauth2 support.  My problem 
is not with alpine but with O365's IMAP access, which is really painfully 
slow and seems to be broken when it comes to searching (eg, an IMAP search 
for participants only seems to match the names and not the addresses). 
Fetching to a local mbox file first solves this problem for me completely. 
Alpine is very snappy with mbox files and search isn't broken.

What i do currently is move everything over manually from within alpine, 
from the O365 IMAP inbox to a local spool mbox. (";aas[enter]adx[enter]") 
That works fine, but having an external daemon do it is nice for a couple 
reasons.

My *terminal windows* will charmingly let me know that "You have mail in 
/var/mail/$USER" when new mail arrives (rather than just the alpine 
window).  (And i will continue to get these charming messages even after 
closing alpine.)  And then i can open up as many alpine windows as i like, 
without having to enter my alpine password, and without having to worry 
about multiple alpine processes fighting to do the rule for new mail 
arrival to move it from the IMAP inbox to a local mbox file for me.

(The fetchmail daemon also prompts for a password on startup, but this is 
once per process lifetime, which equates to once per system boot.)

I had tried out another approach - to make two separate alpine configs. 
The first one with a rule to move mail on arrival from an IMAP inbox to a 
local spool mbox file.  The idea is this would sit open as some kind of 
an, um, foreground daemon that does what i was used to getting out of 
fetchmail.  This could even sit inside a screen(1) session or something. 
Then a second config that would not use IMAP at all (and not need any 
password), which would use the local spool mbox as my inbox.  This would 
be the one i'd use for normal alpine driving (reading, composing, etc), 
and i could open up as many of these as i like in separate windows.

The main trouble with this approach (besides it feeling a little clunky, 
the fetching instance needing to live in a separate pty somewhere), is if 
the instance connected to IMAP doing the fetching gets a [MAIL FOLDER 
"INBOX" CLOSED DUE TO ACCESS ERROR], i'm afraid it might get stuck there 
unnoticed, waiting for user input before it attempts to reconnect.  On the 
other hand, fetchmail polls in a loop, so transient failures don't really 
cause any noticeable problems.

I'd be open to suggestions if anyone has a better way to accomplish this 
in alpine.

...

Jason, thanks also for the tip about OfflineIMAP, and the links.  It looks 
like it might be promising, except that for the time being i am mostly 
just interested in output to an mbox file, which does not look to be 
supported or planned:

http://www.offlineimap.org/doc/FAQ.html#does-offlineimap-support-mbox-mh-or-anything-else-other-than-maildir


> * Does offlineimap support mbox, mh, or anything else other than

>   Maildir?

>

> Not directly. Maildir was the easiest to implement. We are not planning 

> to write an mbox-backend, though if someone sends well-written mbox 

> support and pledged to support it, it would be committed it to the tree.


So it's probably not the first choice for me at the moment.


Thanks all for the input!

Carl


PS, is it time to start a national "Tell your admin how much you love 
Alpine" day?  (perhaps "Alpine Appreciation Day", for short)


On Tue, 19 Apr 2022, Eduardo Chappa wrote:


> On Tue, 19 Apr 2022, Andrew C Aitchison wrote:

>

>>>  Has anyone gotten something like this working with fetchmail + XOAUTH2

>>>  for O365?  A big thank you in advance if anyone has any links, or

>>>  personal tips, etc on getting this working...

>>

>>  This is somewhat off-topic.

>>  https://lists.sourceforge.net/lists/listinfo/fetchmail-users

>>  would be an appropriate place to ask.

>

> I thought initially the same but then i realized that Alpine users might look 

> for answers to their questions about Alpine in an Alpine list instead of a 

> fetchmail list. There are many programs that relate to the use of Alpine and 

> this is one of them, so I reconsidered and thought it was appropriate too.

>

>>  [...]

>>  Matthias Andree, the fetchmail maintainer, is unhappy with the hoops

>>  gmail make him jump through to "register" fetchmail

>>  https://sourceforge.net/p/fetchmail/mailman/fetchmail-users/?viewmonth=202204&viewday=16

>>  If he cannot get fetchmail to use XOAUTH2 *without* registering the "app"

>>  he would appear to be considering whether dropping the feature is an

>>  option.

>

> This portion is both related and unrelated to Alpine.

>

> There is nothing to register when you register really. Let me say it this 

> way. Anyone can go to Google and register Alpine or fetchmail or mutt or 

> firefox, etc. because they are open source applications and what you need is 

> a client-id and client-secret to run your app. That is all.

>

> I went through the process of registering Alpine not because I like Google 

> but because Alpine users need it. It does not matter how I feel about the 

> abuses of Google, Alpine users care about reading their email and not my 

> feelings about Google. I ended up giving Alpine users the chance the get 

> their own client-id and client-secret because that is what a Google employee 

> told me that we were going to come down to.

>

> The real problem with Google is not the registration. It is the verification 

> (of the app). It costs $75000 to verify an app every year. That is the 

> minimum. I do not make money to give it to Google. I do not make money out of 

> selling anything Alpine related to give it to Google.

> Worse, no other company requires this. This is an abuse.

>

> On the Google side they told me that it was the lawyers who did this, as if 

> it was a logical conclusion of some sort and it could not be therefore 

> modified. It guarantees security, they said, which is something that Google 

> sells (in its advertisements). By now it is too late to do anything. No one 

> can go against the giant, and above all I am sorry people support Google by 

> using their products. However, despite my despise for Google, I will not make 

> Alpine users make my feelings be part of their experience, and I think the 

> same should be said about other programs that people depend on, such as 

> fetchmail.

>

> If there is one thing that I think XOAUTH2 is doing to programs like Alpine, 

> fetchmail, etc., is that they are being replaced by other commercial apps 

> completely. The requirement that a users authorizes an app to access their 

> email also is trumped by the requirement that the administrator authorizes 

> the app to access their server, and that is a big issue today as many 

> administrators prefer not to allow apps with which they are unfamiliar for 

> the sake of security and privacy.

>

> The real issue is that IMAP and SMTP are being deprecated by the fact that 

> OAUTH2 over HTTPS is sold as a secure/modern authentication, while IMAP and 

> SMTP are not. While it makes no sense to have this discussion in this forum, 

> it is an argument being used today to not to allow users to turn on IMAP and 

> SMTP, and that is an issue for Alpine users.

>

> Let me say it differently. The world is changing with the excuse of security 

> and privacy. With that excuse programs like Alpine are being left out. It is 

> important that all of us communicate to other people that Alpine is a safe 

> program to use, that respects your privacy and makes no effort to track you 

> or steal information from anyone. I am working on modernizing Alpine, but the 

> real issue is not if IMAP and SMTP will be killed, the real issue is if 

> Alpine will be given access to IMAP and SMTP by administrators, and that is a 

> bigger issue, because chances are that the administrator that you have to ask 

> this question to will say no.

>

> I hope the maintainer of fetchmail decides to include OAUTH2 support. We need 

> programs like fetchmail, mutt, alpine, etc. to keep working in the future. 

> Some Alpine users prefer fecthmail and I hope they will be able to continue 

> using it for many years to come.

>

> -- 

> Eduardo

>