Re: rgw: feedback on auth engine selection

Re: rgw: feedback on auth engine selection

[Radoslaw Zarzynski]

Hi Casey!

I think we could make that with currently existing
AuthEngines. Although they have some internal
state, rework of authenticate() methods to return
a pair consisted with enum (GRANTED, DENIED,
TRY_NEXT) and AuthApplier would allow to strip
it down.

Where I see problems is AWS Auth v4. It's truly
state-full. And when it comes to handling state,
I would prefer to use the language assistance for
it (non-static class members) instead of letting
internals (context) leak to req_state.

Anyway, before we start discussing the whole
auth stuff further, I guess it would be helpful to
write in one place all requirements/plans that
might affect this area. For instance, the idea of
RGWHandler rework can have such impact.

Let me start the requirements with an item found
after a session with Swift's FormPost middleware:

* the auth infrastructure must allow for easy-adoption
  of new AuthEngines as even FormPost would need
  one :-).

Regards,
Radek


On Tue, Sep 6, 2016 at 11:49 PM, Casey Bodley <cbodley@redhat.com> wrote:
> Hi Radek and friends,
>
> Matt and I have been discussing the auth strategy work in progress at
> https://github.com/ceph/ceph/pull/10925.
>
> With respect to the original auth rework, I really like the distinction
> between the Engine and the Applier, with only the Applier having access to
> the req_state. But I feel like we could have gone a step further with
> Engine, to remove all per-request state so that a single Engine instance
> could be reused for each request.
>
> It looks like this could be accomplished pretty easily by adding req_state*
> as an argument to the authenticate() function. Token extraction, if
> necessary, would happen internally to authenticate() so we shouldn't need to
> store the token in RGWTokenBasedAuthEngine. It doesn't look like the other
> engines need to store much either (let me know if I'm missing something!),
> but if there is important state that's unique to a given engine, we might
> consider moving it into the associated Applier.
>
> Having an Engine without per-request state means that, on initialization, we
> just need to generate a vector of Engines (pointers or unique_ptrs because
> they're polymorphic). RGWHandler_REST_SWIFT::authorize() works pretty much
> exactly as it is, except you loop over that vector instead of the std::array
> of Engines on the stack.
>
> I feel like the current AuthStrategy design is trying to get here, but by
> adding extra abstractions instead of collapsing the existing layers. Do you
> buy that?
>
> Thanks,
> Casey

Re: rgw: feedback on auth engine selection

[Casey Bodley]

On 09/07/2016 01:19 PM, Radoslaw Zarzynski wrote:
> Anyway, before we start discussing the whole
> auth stuff further, I guess it would be helpful to
> write in one place all requirements/plans that
> might affect this area. For instance, the idea of
> RGWHandler rework can have such impact.
>
> Let me start the requirements with an item found
> after a session with Swift's FormPost middleware:
>
> * the auth infrastructure must allow for easy-adoption
>    of new AuthEngines as even FormPost would need
>    one :-).

Some specific requirements that I recall from our past discussions include:

- don't heap allocate auth state per request
- don't construct or call into AuthEngines that shouldn't be enabled for 
the given handler

I'll add some of the other refactoring goals we've discussed, but I 
don't think they place any strict requirements on the auth subsystem:

- RGWHandler: being able to reuse handler instances, instead of 
allocating a new one for each request
- req_state: contains too many unrelated fields, some of which are 
transformed in confusing ways over the lifetime of the request. needs to 
be broken up into smaller, better encapsulated sub-objects
- concurrency: move away from a synchronous process_request() to build 
on asio frontend work

Re: rgw: feedback on auth engine selection

[Casey Bodley]

On 09/07/2016 01:19 PM, Radoslaw Zarzynski wrote:
> Where I see problems is AWS Auth v4. It's truly
> state-full. And when it comes to handling state,
> I would prefer to use the language assistance for
> it (non-static class members) instead of letting
> internals (context) leak to req_state.
AWS v4 is a good example to look at, yeah. It's a special case, because 
there's a separate 'complete' step after reading the body of the 
request, so we need to track auth state between the calls to authorize() 
and complete(). The current implementation stores this stuff in a 
'struct rgw_aws4_auth' member of req_state.

So far, all of the AuthEngine instances have been constructed on the 
stack in their handler's authorize() method, so their state has been 
limited to that scope. If a new AWS4AuthEngine were to encapsulate v4 
auth this state, the engine would have to live somewhere outside of the 
authorize() method - most likely in req_state. But we could just as 
easily have a stateless AWS4AuthEngine that accessed the existing 
rgw_aws4_auth via req_state. Encapsulation of rgw_aws4_auth could be 
improved by making everything private except for a complete() method 
(and call it  AWS4AuthCompletion, for example).

Re: rgw: feedback on auth engine selection

[Radoslaw Zarzynski]

On Wed, Sep 7, 2016 at 11:09 PM, Casey Bodley <cbodley@redhat.com> wrote:
>
> AWS v4 is a good example to look at, yeah. It's a special case, because
> there's a separate 'complete' step after reading the body of the request, so
> we need to track auth state between the calls to authorize() and complete().
> The current implementation stores this stuff in a 'struct rgw_aws4_auth'
> member of req_state.

Yup, AWS v4 was a game changer - not only from the perspective
of auth subsystem but for RGWOps as well. Before its introduction
the flow was pretty straightforward:

  1. authenticate - currently RGWHandler::authorize
  2. authorize - RGWOp::verify_permissions
  3. execute - RGWOp::execute

As you mentioned, in such situation AuthEngines can be truly
stateless. In the post-AWSv4 era things are more complicated,
unfortunately. This is the cost we're paying for assuring data
integrity without in-memory buffering on the application level.
Now the flow does look like:

  A. authenticate - RGWHandler::authorize
  B. authorize - RGWOp::verify_permissions
  C. execute_prepare - some parts of RGWOp::execute
  D. authenticate_complete - some parts of RGWOp::execute
  E. execute_commit - some parts of RGWOp::execute

The typical example is RGWPutObj::execute that actually does
point C, D, E. I can fully understand why we have this actually.
Though, in the future I would like to restore the clear separation
of concerns (the single responsibility principle). The new auth
infrastructure was designed to accommodate v4 as well.

> So far, all of the AuthEngine instances have been constructed on the stack
> in their handler's authorize() method, so their state has been limited to
> that scope. If a new AWS4AuthEngine were to encapsulate v4 auth this state,
> the engine would have to live somewhere outside of the authorize() method -
> most likely in req_state. But we could just as easily have a stateless
> AWS4AuthEngine that accessed the existing rgw_aws4_auth via req_state.
> Encapsulation of rgw_aws4_auth could be improved by making everything
> private except for a complete() method (and call it  AWS4AuthCompletion, for
> example).

I don't want to treat AWSv4 in any special way. I would prefer
to have simply an reference to AuthEngine in RGWHandler or
req_state if we want to make the first one stateless. The specific
implementation would encapsulate all required state inside.
Of course, the AuthEngine interface needs to be extended with
something like complete_authentication() but this was expected.
Moreover, AWSv4 was the main factor behind making AuthEngine
state-full.

Regards,
Radek

Re: rgw: feedback on auth engine selection

[Radoslaw Zarzynski]

On Wed, Sep 7, 2016 at 11:05 PM, Casey Bodley <cbodley@redhat.com> wrote:
> Some specific requirements that I recall from our past discussions include:
>
> - don't heap allocate auth state per request

Great. I understood that we don't need to optimize the early
run-time (setup, initialization) to eradicate dynamic allocations.
This would allow to make the things easier to read through
skipping some crazy CRTP/SFINAE stuff.

> - don't construct or call into AuthEngines that shouldn't be enabled for the
> given handler

This sounds like a step further in comparison with the current
solution. In the past we had a lot of "ifs" executed per request.
Currently, the logic is split into per-engine pieces and encapsulated
in is_applicable method of AuthEngines. In my understanding
the goal is to introduce init-time preselection. However, we need
to discuss dealing with run-time config changes (operator injecting
new configuration without restart). Do we want to cover this use
case at all?

> I'll add some of the other refactoring goals we've discussed, but I don't
> think they place any strict requirements on the auth subsystem:
>
> - RGWHandler: being able to reuse handler instances, instead of allocating a
> new one for each request

At the moment RGWHandler is dynamically allocated on each
request. Do we plan to remove only the allocations or go even
further and make handlers entirely state-less?

> - req_state: contains too many unrelated fields, some of which are
> transformed in confusing ways over the lifetime of the request. needs to be
> broken up into smaller, better encapsulated sub-objects

I'm 100% behind that.

> - concurrency: move away from a synchronous process_request() to build on
> asio frontend work

Sound really interesting.


Let me propose another requirement:
 - critical components should facilitate unit testing. In the future
   they should be covered by reasonable set of unit tests.

What do you think?

Regards,
Radek

Re: rgw: feedback on auth engine selection

[Pritha Srivastava]

----- Original Message -----
> From: "Casey Bodley" <cbodley@redhat.com>
> To: "The Sacred Order of the Squid Cybernetic" <ceph-devel@vger.kernel.org>
> Sent: Thursday, September 8, 2016 2:35:34 AM
> Subject: Re: rgw: feedback on auth engine selection
> 
> 
> On 09/07/2016 01:19 PM, Radoslaw Zarzynski wrote:
> > Anyway, before we start discussing the whole
> > auth stuff further, I guess it would be helpful to
> > write in one place all requirements/plans that
> > might affect this area. For instance, the idea of
> > RGWHandler rework can have such impact.
> >
> > Let me start the requirements with an item found
> > after a session with Swift's FormPost middleware:
> >
> > * the auth infrastructure must allow for easy-adoption
> >    of new AuthEngines as even FormPost would need
> >    one :-).
> 
> Some specific requirements that I recall from our past discussions include:
> 
> - don't heap allocate auth state per request
> - don't construct or call into AuthEngines that shouldn't be enabled for
> the given handler
> 
> I'll add some of the other refactoring goals we've discussed, but I
> don't think they place any strict requirements on the auth subsystem:
> 
> - RGWHandler: being able to reuse handler instances, instead of
> allocating a new one for each request
> - req_state: contains too many unrelated fields, some of which are
> transformed in confusing ways over the lifetime of the request. needs to
> be broken up into smaller, better encapsulated sub-objects
> - concurrency: move away from a synchronous process_request() to build
> on asio frontend work

One more requirement to add to the list:

- In case of AWS, two different methods (RGWPostObj_ObjStore_S3::get_policy() and RGW_Auth_S3::authorize), will be making use of the Auth Engines to authenticate a request and the auth key extraction method will be different for both of them. The auth infrastructure needs to take care of this.

Re: rgw: feedback on auth engine selection

[Casey Bodley]

On 09/08/2016 01:19 PM, Radoslaw Zarzynski wrote:
> On Wed, Sep 7, 2016 at 11:05 PM, Casey Bodley <cbodley@redhat.com> wrote:
>> Some specific requirements that I recall from our past discussions include:
>>
>> - don't heap allocate auth state per request
> Great. I understood that we don't need to optimize the early
> run-time (setup, initialization) to eradicate dynamic allocations.
> This would allow to make the things easier to read through
> skipping some crazy CRTP/SFINAE stuff.
>
>> - don't construct or call into AuthEngines that shouldn't be enabled for the
>> given handler
> This sounds like a step further in comparison with the current
> solution. In the past we had a lot of "ifs" executed per request.
> Currently, the logic is split into per-engine pieces and encapsulated
> in is_applicable method of AuthEngines. In my understanding
> the goal is to introduce init-time preselection. However, we need
> to discuss dealing with run-time config changes (operator injecting
> new configuration without restart). Do we want to cover this use
> case at all?

Right, this was the requirement that motivated the work on preselection, 
so we would only try to authenticate against engines that are properly 
configured and relevant to the given handler.

With respect to changing the auth strategy (i.e. the list of engines 
configured for a given handler) at runtime, I'm not sure it's worth the 
complexity at this point, and we certainly don't want to add locking to 
do this safely. [note that we do have 'dynamic reconfiguration' which 
pauses the frontend while it reloads RGWRados, so we could use that 
mechanism to change auth strategy without needing new locks]

>
>> I'll add some of the other refactoring goals we've discussed, but I don't
>> think they place any strict requirements on the auth subsystem:
>>
>> - RGWHandler: being able to reuse handler instances, instead of allocating a
>> new one for each request
> At the moment RGWHandler is dynamically allocated on each
> request. Do we plan to remove only the allocations or go even
> further and make handlers entirely state-less?
>
>> - req_state: contains too many unrelated fields, some of which are
>> transformed in confusing ways over the lifetime of the request. needs to be
>> broken up into smaller, better encapsulated sub-objects
> I'm 100% behind that.
>
>> - concurrency: move away from a synchronous process_request() to build on
>> asio frontend work
> Sound really interesting.
>
>
> Let me propose another requirement:
>   - critical components should facilitate unit testing. In the future
>     they should be covered by reasonable set of unit tests.
>
> What do you think?
>
> Regards,
> Radek

Re: rgw: feedback on auth engine selection

[Casey Bodley]

On 09/08/2016 11:46 AM, Radoslaw Zarzynski wrote:
> On Wed, Sep 7, 2016 at 11:09 PM, Casey Bodley <cbodley@redhat.com> wrote:
>> AWS v4 is a good example to look at, yeah. It's a special case, because
>> there's a separate 'complete' step after reading the body of the request, so
>> we need to track auth state between the calls to authorize() and complete().
>> The current implementation stores this stuff in a 'struct rgw_aws4_auth'
>> member of req_state.
> Yup, AWS v4 was a game changer - not only from the perspective
> of auth subsystem but for RGWOps as well. Before its introduction
> the flow was pretty straightforward:
>
>    1. authenticate - currently RGWHandler::authorize
>    2. authorize - RGWOp::verify_permissions
>    3. execute - RGWOp::execute
>
> As you mentioned, in such situation AuthEngines can be truly
> stateless. In the post-AWSv4 era things are more complicated,
> unfortunately. This is the cost we're paying for assuring data
> integrity without in-memory buffering on the application level.
> Now the flow does look like:
>
>    A. authenticate - RGWHandler::authorize
>    B. authorize - RGWOp::verify_permissions
>    C. execute_prepare - some parts of RGWOp::execute
>    D. authenticate_complete - some parts of RGWOp::execute
>    E. execute_commit - some parts of RGWOp::execute
>
> The typical example is RGWPutObj::execute that actually does
> point C, D, E. I can fully understand why we have this actually.
> Though, in the future I would like to restore the clear separation
> of concerns (the single responsibility principle). The new auth
> infrastructure was designed to accommodate v4 as well.
>
>> So far, all of the AuthEngine instances have been constructed on the stack
>> in their handler's authorize() method, so their state has been limited to
>> that scope. If a new AWS4AuthEngine were to encapsulate v4 auth this state,
>> the engine would have to live somewhere outside of the authorize() method -
>> most likely in req_state. But we could just as easily have a stateless
>> AWS4AuthEngine that accessed the existing rgw_aws4_auth via req_state.
>> Encapsulation of rgw_aws4_auth could be improved by making everything
>> private except for a complete() method (and call it  AWS4AuthCompletion, for
>> example).
> I don't want to treat AWSv4 in any special way. I would prefer
> to have simply an reference to AuthEngine in RGWHandler or
> req_state if we want to make the first one stateless. The specific
> implementation would encapsulate all required state inside.
> Of course, the AuthEngine interface needs to be extended with
> something like complete_authentication() but this was expected.
> Moreover, AWSv4 was the main factor behind making AuthEngine
> state-full.
>
> Regards,
> Radek

Thanks for the extra background! You've convinced me that it makes sense 
to extend the auth interfaces to support this instead of treating it as 
a special case. But I still don't think that AuthEngine is the right 
place to manage that state - mainly because each request will be trying 
to authenticate against multiple different engines, and only one of them 
will be successful. So with stateful AuthEngines, we'll have to 
construct and destruct instances for all of the engines that fail, and 
then once we find one that succeeds, we have to move or copy that 
instance to extend its lifetime to that of the req_state. I don't see a 
generic way to do that without an extra allocation.

I pointed out AuthApplier (which is what we return from a successful 
call to AuthEngine::authorize()) earlier, because it's an existing 
example of auth state that we store with req_state for later access. 
That seems like a much better pattern for something like AuthCompletion. 
I suggested combining this with AuthApplier, because a) then we don't 
have to allocate the completion state separately, and b) the completion 
is a noop for most auth engines.

Re: rgw: feedback on auth engine selection

[Matt Benjamin]

Hi,

inline

----- Original Message -----
> From: "Radoslaw Zarzynski" <rzarzynski@mirantis.com>
> To: "Casey Bodley" <cbodley@redhat.com>
> Cc: "The Sacred Order of the Squid Cybernetic" <ceph-devel@vger.kernel.org>
> Sent: Thursday, September 8, 2016 1:19:11 PM
> Subject: Re: rgw: feedback on auth engine selection
> 

> 
> At the moment RGWHandler is dynamically allocated on each
> request. Do we plan to remove only the allocations or go even
> further and make handlers entirely state-less?

That's been on my list of things to eventually do, for a while--in the librgw front-end, it already is.

> 
> > - req_state: contains too many unrelated fields, some of which are
> > transformed in confusing ways over the lifetime of the request. needs to be
> > broken up into smaller, better encapsulated sub-objects
> 
> I'm 100% behind that.

Me too.  Eventually, I'd like avoid needing separate allocations for any subobjects.  This was one of the main places I hoped we copuld use the polymorphic union idea that you called "static pointer." :)

> 
> > - concurrency: move away from a synchronous process_request() to build on
> > asio frontend work
> 
> Sound really interesting.
> 
> 
> Let me propose another requirement:
>  - critical components should facilitate unit testing. In the future
>    they should be covered by reasonable set of unit tests.

sure

> 
> What do you think?
> 
> Regards,
> Radek
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

Re: rgw: feedback on auth engine selection

[Radoslaw Zarzynski]

On Thu, Sep 8, 2016 at 7:21 PM, Pritha Srivastava <prsrivas@redhat.com> wrote:
> One more requirement to add to the list:
>
> - In case of AWS, two different methods (RGWPostObj_ObjStore_S3::get_policy() and RGW_Auth_S3::authorize), will be making use of the Auth Engines to authenticate a request and the auth key extraction method will be different for both of them. The auth infrastructure needs to take care of this.

Hi Pritha,

Thanks for pointing this out! FormPost implementations
in both S3 (RGWPostObj) and Swift require workflow
very similar to AWS v4. They need to parse fragments
of HTTP body to verify form's integrity (using signature)
before making the ultimate decision whether a request
is authenticated or not.

I think that after extending the infrastructure to cover
AWSv4 we would be also able to:
 1. eradicate the get_policy() method of RGWPostObj,
 2. avoid implementing similar thing in Swift's FormPost.

Regards,
Radek

Re: rgw: feedback on auth engine selection

[Radoslaw Zarzynski]

Hi Casey,

On Thu, Sep 8, 2016 at 10:49 PM, Casey Bodley <cbodley@redhat.com> wrote:
> Thanks for the extra background! You've convinced me that it makes sense to
> extend the auth interfaces to support this instead of treating it as a
> special case. But I still don't think that AuthEngine is the right place to
> manage that state - mainly because each request will be trying to
> authenticate against multiple different engines, and only one of them will
> be successful. So with stateful AuthEngines, we'll have to construct and
> destruct instances for all of the engines that fail, and then once we find
> one that succeeds, we have to move or copy that instance to extend its
> lifetime to that of the req_state. I don't see a generic way to do that
> without an extra allocation.

I see your concerns. I think we can deal with the issues of additional
dynamic allocations and moving/copying memory as well.

At the moment ::authorize() of the Swift API's handler instantiates
multiple engines on stack only to form the array of interface-typed
pointers. I'm pretty sure we can create them in the step-by-step
manner and perform the real job having only one instance alive
at the same time. The mockup of AuthStrategy::Visitor interface is
designed to utilize this pattern.

We can pair the step-by-step processing together with the static_ptr,
that already has the emplace<ConreteClass>(ctor, args) machinery,
to create AuthEngine instance directly in its final location.

This way we can avoid creation of disabled auth engines, dynamic
memory allocations, std::moving/copying between ::authorize() scope
and req_state and even optimize the on-stack memory usage.

> I pointed out AuthApplier (which is what we return from a successful call to
> AuthEngine::authorize()) earlier, because it's an existing example of auth
> state that we store with req_state for later access. That seems like a much
> better pattern for something like AuthCompletion. I suggested combining this
> with AuthApplier, because a) then we don't have to allocate the completion
> state separately, and b) the completion is a noop for most auth engines.

What makes me a bit uncomfortable with putting the authenticate_complete()
into AuthApplier is mixing responsibilities. Applier operates in the area
of identity - it verifies whether someone owns an account, has the admin
status, loads its account info and so on. AuthEngine in turn is supposed
to encapsulate the request authentication process.

Verifying message integrity looks like a part of request authentication
and thus conceptually seems to be the Engine's job.

Regards,
Radek

Re: rgw: feedback on auth engine selection

[Radoslaw Zarzynski]

On Thu, Sep 8, 2016 at 8:23 PM, Casey Bodley <cbodley@redhat.com> wrote:
> Right, this was the requirement that motivated the work on preselection, so we would only try to authenticate against engines that are properly configured and relevant to the given handler.

Which component should be responsible for verifying correctness
of configuration? I would say the best candidate is an AuthEngine
itself. Some time ago the static can_be_enabled() was proposed.
If we need that feature, maybe similar (but better named) method
is a way to go?

> With respect to changing the auth strategy (i.e. the list of engines configured for a given handler) at runtime, I'm not sure it's worth the complexity at this point, and we certainly don't want to add locking to do this safely. [note that we do have 'dynamic reconfiguration' which pauses the frontend while it reloads RGWRados, so we could use that mechanism to change auth strategy without needing new locks]

Great. It looks that at this phase we don't need to have the feature onboard.
It's enough to just don't prohibit implementing it in the future.

P.S.
Sorry for resending this letter, Casey. I had accidentally sent HTML
on my mobile client. The message was refused by vger.kernel.org.

Regards,
Radek

Re: rgw: feedback on auth engine selection

[Casey Bodley]

On 09/12/2016 05:48 AM, Radoslaw Zarzynski wrote:
> On Thu, Sep 8, 2016 at 8:23 PM, Casey Bodley <cbodley@redhat.com> wrote:
>> Right, this was the requirement that motivated the work on preselection, so we would only try to authenticate against engines that are properly configured and relevant to the given handler.
> Which component should be responsible for verifying correctness
> of configuration? I would say the best candidate is an AuthEngine
> itself. Some time ago the static can_be_enabled() was proposed.
> If we need that feature, maybe similar (but better named) method
> is a way to go?

I think this responsibility needs to be shared between the auth method 
and the module (s3, swift, or nfs) that's trying to configure it. The 
objection to a static can_be_enabled() method as part of the AuthEngine 
interface was that it doesn't allow different modules to provide a 
custom configuration - it could only decide based on global state like 
ceph.conf. That said, I do think it makes sense for the concrete 
AuthEngine types to provide their own methods that take whatever 
configuration parameters are relevant to that auth type.

As an example, here's how I see s3 constructing its auth strategy:

using AuthEnginePtr = std::unique_ptr<AuthEngine>;
using AuthStrategy = std::vector<AuthEnginePtr>;

static AuthStrategy get_s3_auth_strategy()
{
   AuthStrategy strategy;

   // try to configure LDAP
   LDAPAuthEngine ldap;
   if (ldap.initialize(ldap_uri, ldap_binddn, ldap_bindpw, 
ldap_searchdn, ldap_dnattr)) {
     AuthEnginePtr ptr{new LDAPAuthEngine(std::move(ldap))};
     strategy.emplace_back(std::move(ptr));
   }

   ...
   return strategy;
}

(this assumes the 'stateless' version of AuthEngine, where its only 
state is configuration, rather than the state needed to authenticate a 
single request. so initialize() is a non-static member function)

>> With respect to changing the auth strategy (i.e. the list of engines configured for a given handler) at runtime, I'm not sure it's worth the complexity at this point, and we certainly don't want to add locking to do this safely. [note that we do have 'dynamic reconfiguration' which pauses the frontend while it reloads RGWRados, so we could use that mechanism to change auth strategy without needing new locks]
> Great. It looks that at this phase we don't need to have the feature onboard.
> It's enough to just don't prohibit implementing it in the future.
>
> P.S.
> Sorry for resending this letter, Casey. I had accidentally sent HTML
> on my mobile client. The message was refused by vger.kernel.org.
>
> Regards,
> Radek